The paper by Koutsouleris et al. offers a neuroimaging pattern classification system to identify subjects at risk for developing psychosis. Overall, I found that around the dependent variables, i.e., the images, the paper offered more than I would ever want to know, but that around the independent variables, i.e., the individuals meeting various at-risk clinical definitions, the paper offered far less than I wanted to know. It is a complicated sample with inclusion criteria covering more than half a page in the Archives. Nevertheless, the definitions are operationalized and clear. What is missing is any proof that these definitions were used reliably, including the classification of transition from prodromal to psychotic. Reference is made on page 701 for a paper containing details of the recruitment protocol, but I still could find nothing on reliability. This is a oversight which I find uncharacteristic of the Archives.

View all comments by Thomas McGlashan

Koutsouleris and colleagues’ recent article in Archives of General Psychiatry provides a robust and enlightening challenge to those who would look for the etiology of schizophrenia in isolated networks or circumscribed brain regions. Through the application of supervised learning algorithms (Noble, 2006; Vapnik, 1999) and multivariate classification strategies, the authors convincingly demonstrate extensive gray matter distribution differences among various categories of participants at risk for psychosis. They also demonstrate significant morphological pattern differences between those who transitioned to psychosis and those who did not. This study extends and expands findings previously published by Davatzikos and associates (Davatzikos et al., 2005). It also uses tools similar to those applied by the Philadelphia group.

Because the central nervous system is inherently multidimensional, and because many diseases affect the nervous system slowly and unpredictably, it is reasonable to study disease progression and disease variation with support vector machine methods. These approaches will likely be used extensively in the future by those determined to wrestle with the high dimensional data sets that characterize brain research. A thoughtful and systematic assessment of pattern changes associated with disease onset and progression should provide helpful clues regarding the molecular substrate that propels schizophrenia.

References:

Noble WS. What is a support vector machine? Nat Biotechnol. 2006;24:1565-1567. Abstract

Vapnik VN. An overview of statistical learning theory. IEEE Trans Neural Netw. 1999;10:988-999. Abstract

Davatzikos C, Shen D, Gur RC, Wu X, Liu D, Fan Y, Hughett P, Turetsky BI, Gur RE. Whole-brain morphometric study of schizophrenia revealing a spatially complex set of focal abnormalities. Arch Gen Psychiatry. 2005;62:1218-1227. Abstract

View all comments by Henry Holcomb

It was with great interest that we read the paper by Koutsouleris et al. in the July 2009 issue of the Archives in which they report on a method to identify among individuals at high risk of developing psychosis those that go on the reach full-blown psychosis. Psychiatric research in general and schizophrenia research in particular have been set up as a comparison between groups of affected and unaffected individuals. Although this has improved our understanding in the etiology and pathophysiology of the illness, results so far have failed to be of diagnostic or prognostic significance. For many researchers, the ultimate goal would be to find a disease-specific (set of) biological marker(s) to support the diagnostic process or predict the onset or course of illness.

In this well-written and elegant study, Koutsouleris et al. propose a statistical method, specifically, a multivariate machine learning algorithm (support vector machine) that is able to estimate group membership from effective integration of, in this case, neuroanatomical information. Two important issues can be raised. First, the sensitivity and specificity are only useful when at least one of the two is very high (typically >95 percent). In the current paper, impressive percentages are reported in the range of 76 to 94 percent. However, the authors did not report confidence intervals. If sensitivity is estimated to be at least 90 percent for adequate decision making, the lower boundary of the 95 percent confidence interval should be at least 90 percent. The confidence intervals can be reduced by increasing the sample size, and great effort should be made to do so. In this respect, sample size is important (as it is in neuroanatomical measures at the voxel level). This is particularly relevant in studies focusing on transition rates where the number of transitions is usually (very) small, as is the case here.

The second issue relates to the process by which two and three group classifications were estimated, always including a group of healthy comparisons. Two validation methods were used, a fivefold cross validation, which is a relatively straightforward way of testing generalizability, and a classification of an independent sample of healthy individuals. As a statistical test, the latter procedure might make sense and is informative. However, one may argue that clinically this is not very useful. It would have strengthened the paper if the authors validated the classification on an independent ARMS sample, preferably from another MRI scanner (to learn about the generalizability of MRI machine and scanning protocols).

Koutsouleris et al. provide an important first attempt to investigate the feasibility of predicting illness outcome in a psychotic disorder. This approach is both novel and important, as it is one of the first attempts to develop biological markers that can be useful in clinical practice. One can think of many other important clinical questions that could be answered using this approach: Can we predict outcome based on neuroanatomical measurements after psychosis onset? Can we distinguish between different axis-I diagnoses at illness onset based on these parameters? So far, these questions cannot be addressed with the currently used statistical techniques and study designs. However, if the imaging community’s goal is to develop neuroanatomical pattern recognition techniques that could be used along with other types of information (e.g., cognition, genetics, symptomatology) in clinical practice, we need to redesign our studies. To be of prognostic value in clinical practice, the method should be able to predict transition to a particular mental illness in a help-seeking population consisting of symptomatic or genetically high-risk people.

View all comments by Neeltje E.M. van Haren