Harvey and colleagues deserve recognition for their recent paper: “Validating the measurement of real world functional outcomes: Phase 1 results of the Valero Study.” Certainly their acronym alone is worthy of commendation, making a large-scale effort of measure validation sound as appealing as a snappy Italian tune popularized by Dean Martin. But more importantly, they took a straightforward and well-powered approach to addressing which functional outcome scale maximally overlaps with a latent trait representing a mixture of cognitive ability/functional capacity. And the analysis appears to have yielded a clear rating scale “winner,” the Specific Levels of Function Scale (with the far less appealing acronym, the "SLOF").
Clearly, replication is needed in other hands, in other types of patient samples before the SLOF enters the hallowed halls of “gold standard” instruments. However, it enters the race for the hallowed halls with a very strong empirical foundation.
Is there any reason to have reservations about these findings? While the SLOF emerged victorious, it also has some intrinsic limitations. Ratings are done through consulting with an informant. Clearly, these informants have variable knowledge concerning the day-to-day functioning of patients, and varying approaches to how to use the rating scale, so it is very likely that SLOF ratings contain a fair amount of rater “noise” and are most accurate at the extreme ends of the scale. Thus, there is reason to be concerned that subtle but important and real differences (either between patients or within a patient over time as a function of treatment) may be a challenge to detect.
Note that the method of Harvey et al. was to identify functional outcome measures that maximally correlated with cognitive measures, and this shared variance is a very small portion of total outcome variance. Thus, the SLOF provides the best assessment of the general cognition-associated outcome variance. Left unaddressed and therefore unknown is the question of whether there might be specific cognition functions (with modest correlations with general ability) that have more powerful relationships to general level of functional outcome, or more specific predictors of different aspects of outcome. Might the approach to validating the SLOF therefore make it a measure that is harder to use in the search of more specific predictors? Harvey et al. had a different intent as they deliberately created a general cognitive ability latent trait measure by capturing shared variance across three cognitive tasks. This is clearly an easily defended approach, as the evidence for more specific cognition-function prediction is weak in the available literature. However, by selecting the SLOF on the basis of its relationship with general cognitive ability, Harvey et al. may have provided a tool that might cause interpretive problems for others who are seeking to demonstrate more specific relationships. For example, if one were to find that specific cognitive measure X fails to correlate with the SLOF, can that be taken at face value as evidence that the specific cognitive process is unrelated to functional outcome? That is, might the SLOF “signal” be a general cognition signal, whereas other functional outcome measures could plausibly be more sensitive to other, more specific forms of cognitive impairment? Ideally, a more specific predictor would account for some (all?) of the more general variance as well as additional, unique variance. Hopefully, someone will produce such evidence of powerful specific prediction and allow for an evaluation of the SLOF in that context.
In closing, Harvey et al. have done the field an important service by providing evidence that the SLOF provides a meaningful measure of functional outcome that appears to be well suited to a variety of clinical and research applications. It is not a perfect tool, but this area of assessment poses such enormous challenges that it is hard to imagine how a perfect tool could be developed. Hopefully, this publication will not dissuade others from trying to do better, thinking that the matter appears to be settled. This publication does establish the hurdle that other measures need to get over: they need to do better than the SLOF.
View all comments by James Gold