As stated in the CATIE and CAFÉ neurocognition manuscripts, it is possible that the small improvements in neurocognitive performance following randomization to one of the antipsychotic treatments in these studies are due solely to practice effects or expectation biases. This statement is affirmed by the excellent recent study by Goldberg et al. in which improvements in cognitive performance were almost identical in magnitude to the practice effects found in healthy controls. While these data may be perhaps disappointing to the hope that second-generation medications improve cognition, they may also suggest that cognitive performance is less recalcitrant to change than previously expected.

In the context of a double-blind study design, the degree of cognitive enhancement observed for each treatment group is a function of three major variables: treatment effect, placebo effect, and practice effect. In studies of antipsychotic medications without a placebo control group, practice and placebo effects in schizophrenia cannot be disentangled from treatment effects. They also cannot be disentangled from each other. Recent data from a double-blind study comparing the effects of donepezil hydrochloride and placebo in a highly refined sample of 226 patients with schizophrenia stabilized while taking second-generation antipsychotics suggested that patients taking placebo had neurocognitive effect size improvements (0.22 SD after being tested twice over 6 weeks; 0.45 SD after the third assessment at 12 weeks) on the same test battery used in the CATIE and CAFÉ studies, suggesting a practice or placebo effect (Keefe et al., Neuropsychopharmacology, in press) consistent with the improvements reported in the CATIE and CAFÉ treatment studies. These cognitive improvements are in contrast to test-retest data collected in patients with schizophrenia tested with the MATRICS Consensus Cognitive Battery (MCCB; Nuechterlein et al., in press) and the Brief Assessment of Cognition in Schizophrenia (BACS; Keefe et al., 2004), which showed very little practice effects. The contrast of the data from these test-retest studies that did not involve the initiation of new treatments with cognitive improvements following the initiation of antipsychotic treatment or placebo suggests that attribution biases beyond simple practice effects may be at work.

Test-retest data from patients tested twice within a briefer period than the test interval in the four treatment studies discussed above suggest that schizophrenia patients demonstrate relatively small improvements in executive functions (Keefe et al., 2004; Nuechterlein et al., in press) and the WAIS digit-symbol test (Nuechterlein et al., in press), and medium improvements on tests of verbal memory only when identical versions are repeated (Hawkins and Wexler, 1999; Keefe et al., 2004) but not on tests of verbal fluency (Keefe et al., 2004; Nuechterlein et al., in press). In the donepezil/placebo study, patients who received placebo improved substantially across several cognitive domains. Although not tested directly, this series of results suggests that the magnitude of placebo effects in cognitive enhancement trials may exceed the reported size of practice-related improvements in studies of schizophrenia patients tested twice without the prospect of the initiation of a cognitive intervention.

The greater improvements in cognition found in the context of a placebo-controlled trial could be due to a variety of psychological factors. When a patient enters into a trial or is treated with a medication that is believed to contribute beneficially to cognitive performance, rater bias and expectation bias can have strong effects on performance. Patients who are told that their cognitive abilities might improve may be able to perform better on the test batteries used in the study simply because their expectations become more optimistic. Second, testers who believe that a patient will have cognitive improvement, or hope for such improvement, could administer the tests in a more hopeful, positive manner, which can help the patient raise his or her expectations for performance and thus engage motivational systems that were previously disengaged (Keefe, 2006). Such expectation bias can also lead to inaccuracies in scoring; since many cognitive tests require the use of judgment to determine final scores, hopeful testers are more likely to give the “benefit of the doubt” to patients after they have entered into a study in which the treatment is potentially cognitively enhancing. Third, this same type of expectation could have an impact on the support that a patient receives in his or her community/living situation. If the people who interact regularly with the patient begin looking for better performance on cognitively related tasks, these expectations could become self-fulfilling in that they may raise the confidence and motivation of the patient to perform well on such tasks, including cognitive testing.

The factors associated with improvement during a placebo-controlled trial are indeed complex, and it is difficult to distinguish practice effects from placebo effects. However, the relatively small clinical improvement in test-retest designs without treatment or placebo intervention suggests that any potential practice effects may at least be potentiated by placebo effects.

The implications for this series of results include a methodological caution and a reason for optimism. Regarding the caution, future trials of cognitive-enhancing compounds might need to be designed in such a way that practice and placebo are reduced. Very few treatment studies of patients with schizophrenia have employed a priori methodological strategies to reduce the magnitude of potential practice effects, such as the use of a placebo run-in period with one or more administrations of the cognitive battery prior to randomization. Regarding the optimism, these studies suggest that schizophrenia cognition (perhaps especially when freed from the dampening effects of large doses of high potency medications such as haloperidol) could be more plastic that had been previously assumed; it is possibly as sensitive to experience-dependent learning in schizophrenia patients as healthy controls, and it may benefit from improved psychological expectations. While this is a methodological nuisance for clinical trial designs, it may also reveal an unexpectedly large potential gain for psychological interventions such as cognitive remediation, cognitive-behavioral therapy, and even encouragement.

References:

Goldberg TE, Goldman RS, Burdick KE, Malhotra AK, Lencz T, Patel RC, Woerner MG, Schooler NR, Kane JM, Robinson DG. Cognitive improvement after treatment with second-generation antipsychotic medications in first-episode schizophrenia: Is it a practice effect? Arch Gen Psychiatry. 2007 Oct;64:1115-1122. Abstract

Hawkins KA, Wexler BE (1999). California Verbal Learning Test practice effects in a schizophrenia sample. Schizophr Res 39: 73-78. Abstract

Keefe RSE. Missing the sweet spot: Disengagement in schizophrenia. Psychiatry, 2006; 3: 36-41.

Keefe RSE, Malhotra AK, Meltzer H, Kane JM, Buchanan RW, Murthy A, Sovel M, Li, C, Goldman R. Efficacy and safety of donepezil in patients with schizophrenia or schizoaffective disorder: Significant placebo/practice effects in a 12-week, randomized, double-blind, placebo-controlled trial. Neuropsychopharmacology, 2007 [Epub ahead of print]. Abstract

Keefe RSE¸ Goldberg TE, Harvey PD, Gold JM, Poe M, Coughenour L. The Brief Assessment of Cognition in Schizophrenia: Reliability, sensitivity, and comparison with a standard neurocognitive battery. Schizophrenia Research, 2004; 68: 283-297. Abstract

Nuechterlein KH, Green MF, Kern RS, Baade LE, Barch D, Cohen J, Essock S, Fenton WS, Frese FJ, Gold JM, Goldberg T, Heaton R, Keefe RSE, Kraemer H, Mesholam-Gately R, Seidman LJ, Stover E, Weinberger D, Young AS, Zalcman S, Marder SR. The MATRICS consensus cognitive battery: Part 1. Test selection, reliability, and validity. The American Journal of Psychiatry (in press).

View all comments by Richard Keefe

  I recommend the Primary Papers

This article questions the prevailing notion that antipsychotic medication (particularly second-generation antipsychotics) improve cognitive functioning in individuals with schizophrenia. As the authors rightly note, practice effects should be taken into account before attributing improvements to drug effects.

View all comments by Narsimha Pinninti

  I recommend the Primary Papers

I propose that future studies should use computational cognitive assessment tools like CANTAB or CogTest, which have at least two advantages. These tools have multiple similar test modules, so on each testing during one study, participants get a similar but not the same test to assess the same cognitive function. Besides, computational assessment also reduces chances of subjective bias on the part of investigator.

References:

Levaux MN, Potvin S, Sepehry AA, Sablier J, Mendrek A, Stip E. Computerized assessment of cognition in schizophrenia: promises and pitfalls of CANTAB. Eur Psychiatry. 2007 Mar;22(2):104-15. Review. Abstract

View all comments by Saurabh Gupta

One remedy would be repeated practice over time before the actual baseline, sufficient to reach asymptotic ability. Computerized testing of reaction time measures, short-term memory span, etc. would all be quite cheap and easy to implement, for example, as a weekly session.

View all comments by Sebastian Therman

  I recommend the Primary Papers

We recently completed a meta-analysis on "Longitudinal studies of cognition in schizophrenia" (to be published in the British Journal of Psychiatry) based on 53 studies providing data for 31 cognitive variables. When enough data were available (19 variables from eight cognitive tests), we compared the results of schizophrenic participants to those of normal controls.

Given the differences in methods and the fact that most of the studies included in our meta-analysis reported results of patients being past their first episode (FE), it is surprising how close our results and conclusions are compared to those of Goldberg et al. In our analysis we found that, with two exceptions (semantic verbal fluency and Boston naming test, which were stable), participants with schizophrenia improved their performances. The improvement was statistically significant for 19 variables (out of 29). However, controls also showed improvement in most of the variables due to the practice effect. A significant improvement (definite practice effect) was present for 10 variables, an improvement that did not reach significance (possible practice effect) was present in six more variables, and three variables showed no improvement. When compared with schizophrenic patients, controls showed similar improvement for 11 variables, significantly more improvement for seven variables (six of them from the “definite practice effect” group, one from the “possible practice effect”) and for one variable less improvement (the Stroop interference score). Thus, these results suggest that for most of the cognitive variables, improvement seen in schizophrenic subjects does not exceed improvement due to the practice effect.

It is interesting to mention that in our analysis only two variables improved significantly more when patients had a change in their medication from first-generation antipsychotics (FGAs) to second-generation antipsychotics (SGAs). These variables were time to complete TMT B and the delayed recall of the Visual Reproduction (from the WMS). In the Goldberg et al. study the only two tests that showed more improvement in schizophrenic subjects than in controls were also the TMT and visual reproduction. Although in our study schizophrenic subjects did not improve more than controls, the two results (Goldberg’s and ours) taken together could be an indirect argument for a differential, specific effect of SGAs on those two (visuo-spatial) tasks. The placebo effect—see the comment by Richard Keefe—could explain why improvement in the study by Goldberg et al. was greater than in our meta-analysis. Studies of effects of changing medication in the opposite direction, from SGAs to FGAs, could contribute to validate or invalidate these hypotheses.

Goldberg et al. suggested that there could be a set of task characteristics that could be used to develop tasks resistant to the practice effect. Our own results are less optimistic as they show that phonemic verbal fluency, despite a very similar format, does not share the “practice resistance” with the semantic verbal fluency. However, we think that there is already a wealth of data that could be used to select the best cognitive tests. An alternative solution is the use of scales and questionnaires for evaluating cognition (that are sensible to the placebo effect but not to the practice effect).

References:

Szoke A, Trandafir A, Dupont M-E, Meary A, Schurhoff F, Leboyer M. Longitudinal studies of cognition in schizophrenia. British Journal of Psychiatry (in press).

View all comments by Andrei Szoke

  I recommend the Primary Papers