We currently observe a disconcerting phenomenon in machine learning studies in psychiatry: While we would expect larger samples to yield better results due to the availability of more data, larger machine learning studies consistently show much weaker performance than the numerous small-scale studies. Here, we systematically investigated this effect focusing on one of the most heavily studied questions in the field, namely the classification of patients suffering from Major Depressive Disorder (MDD) and healthy controls based on neuroimaging data. Drawing upon structural MRI data from a balanced sample of N = 1868 MDD patients and healthy controls from our recent international Predictive Analytics Competition (PAC), we first trained and tested a classification model on the full dataset which yielded an accuracy of 61%. Next, we mimicked the process by which researchers would draw samples of various sizes (N = 4 to N = 150) from the population and showed a strong risk of misestimation. Specifically, for small sample sizes (N = 20), we observe accuracies of up to 95%. For medium sample sizes (N = 100) accuracies up to 75% were found. Importantly, further investigation showed that sufficiently large test sets effectively protect against performance misestimation whereas larger datasets per se do not. While these results question the validity of a substantial part of the current literature, we outline the relatively low-cost remedy of larger test sets, which is readily available in most cases.

Transgender individuals (TIs) show brain-structural alterations that differ from their biological sex as well as their perceived gender. To substantiate evidence that the brain structure of TIs differs from male and female, we use a combined multivariate and univariate approach. Gray matter segments resulting from voxel-based morphometry preprocessing of N = 1753 cisgender (CG) healthy participants were used to train (N = 1402) and validate (20% holdout N = 351) a support-vector machine classifying the biological sex. As a second validation, we classified N = 1104 patients with depression. A third validation was performed using the matched CG sample of the transgender women (TW) application sample. Subsequently, the classifier was applied to N = 26 TW. Finally, we compared brain volumes of CG-men, women, and TW-pre/post treatment cross-sex hormone treatment (CHT) in a univariate analysis controlling for sexual orientation, age, and total brain volume. The application of our biological sex classifier to the transgender sample resulted in a significantly lower true positive rate (TPR-male = 56.0%). The TPR did not differ between CG-individuals with (TPR-male = 86.9%) and without depression (TPR-male = 88.5%). The univariate analysis of the transgender application-sample revealed that TW-pre/post treatment show brain-structural differences from CG-women and CG-men in the putamen and insula, as well as the whole-brain analysis. Our results support the hypothesis that brain structure in TW differs from brain structure of their biological sex (male) as well as their perceived gender (female). This finding substantiates evidence that TIs show specific brain-structural alterations leading to a different pattern of brain structure than CG-individuals.

Childhood maltreatment is associated with cognitive deficits that in turn have been predictive for therapeutic outcome in psychiatric patients. However, previous studies have either investigated maltreatment associations with single cognitive domains or failed to adequately control for confounders such as depression, socioeconomic environment, and genetic predisposition. We aimed to isolate the relationship between childhood maltreatment and dysfunction in diverse cognitive domains, while estimating the contribution of potential confounders to this relationship, and to investigate gene-environment interactions. We included 547 depressive disorder and 670 healthy control participants (mean age: 34.7 years, SD = 13.2). Cognitive functioning was assessed for the domains of working memory, executive functioning, processing speed, attention, memory, and verbal intelligence using neuropsychological tests. Childhood maltreatment and parental education were assessed using self-reports, and psychiatric diagnosis was based on DSM-IV criteria. Polygenic scores for depression and for educational attainment were calculated. Multivariate analysis of cognitive domains yielded significant associations with childhood maltreatment (η²p = 0.083, P < 0.001), depression (η²p = 0.097, P < 0.001), parental education (η²p = 0.085, P < 0.001), and polygenic scores for depression (η²p = 0.021, P = 0.005) and educational attainment (η²p = 0.031, P < 0.001). Each of these associations remained significant when including all of the predictors in one model. Univariate tests revealed that maltreatment was associated with poorer performance in all cognitive domains. Thus, environmental, psychopathological, and genetic risk factors each independently affect cognition. The insights of the current study may aid in estimating the potential impact of different loci of interventions for cognitive dysfunction. Future research should investigate if customized interventions, informed by individual risk profiles and related cognitive preconditions, might enhance response to therapeutic treatments.