This study is a secondary data analysis of an existing database. The original database has been derived from a completed case-control study, investigating electroencephalogram (EEG) delta/alpha frequency activity in patients with a diagnosis of psychotic disorder, compared to controls without a psychotic illness.¹⁴

In the original study, participants were recruited using word of mouth methods, referral from clinicians, and media advertisements. Clinically stable outpatients from the Western Cape province of South Africa were selected as cases. Controls were from similar socio-economic backgrounds as the participants, but without a history or diagnosis of psychosis or bipolar disorder (BMD). Cases were required to have a diagnosis of schizophrenia (SZP), BMD with psychosis, or methamphetamine induced psychotic disorder (MPD), diagnosed as per the Structured Clinical Interview for DSM-IV-TR Axis I Disorders (SCID-I). They were required to be between 19 and 40 years of age and to be fluent in English. Exclusion criteria included any general medical conditions that required chronic treatment; history of learning disability; history of major brain injury or surgery; history of cardiovascular insult; individual or family history of epilepsy; and any medical implants or metal within their bodies. Females who were pregnant or lactating were excluded; and patients with a diagnosis of BMD or SZP were excluded if their disorder was deemed to be substance induced. However, co-morbid substance use per se was not excluded (including for the non-psychotic control group). The patients with MPD were required not to have evidence for another primary psychotic disorder.¹⁴

For the current analysis, patients with bipolar type II disorder who may have had psychotic symptoms as part of a major depressive episode, but not part of hypomania, were included. All participants in the dataset who completed the ASSIST were included. The sample size of this study was determined and limited by the data available in the parent study.

On the day of assessment all participants completed both the SCID-I and ASSIST. The SCID-I is a semi-structured interview, designed to detect Axis I disorders as defined in the DSM-IV. It is commonly used in research settings and its reliability has been demonstrated.¹⁵ The principle bipolar or psychotic disorder diagnosis as well as the DSM-IV SUD diagnosis (either substance abuse or substance dependence) was determined by the SCID-I, utilising modules A, B, C, D, and E. It uses a combination of closed and open-ended questions and is administered by trained interviewers. It takes an average of 3 h to administer.

The ASSIST version 3.0 is a self-report screening tool which was developed by the WHO to screen for psychoactive substance use and related problems in primary care patients.¹⁰ It consists of eight items that measure recent (i.e. past 3 months) and lifetime use of 10 substances. The ASSIST can be administered in 5–10 min, and is conducted in an empathic, non-judgmental manner, using open ended questions. The ASSIST begins with a broad question regarding lifetime substance use; it goes on to cover frequency of use, cravings, frequency of substance related problems (health, social, legal, or financial) and effects on role responsibilities, in the last 3 months. Regarding lifetime use, it asks whether others are concerned, past attempts to cut down on substance use, and any intravenous drug use.¹⁰ Each question on the instrument is differentially weighted with a Likert-type scale. A total substance involvement score (TSI) is obtained by summation of scores on Q1 through to Q8 that measure both recent and lifetime substance use across all substances used. It can also assess past 3-month substance use only (TSI-3-month; the sum of Q2–Q5). In addition, Specific Substance Involvement (SSI) scores can be calculated separately for each substance by summation of score across items Q2 to Q7.

The relevant variables, notably the ASSIST scores and the DSM-IV substance use diagnoses, were extracted from the data and tabulated. The SCID-I was used as the gold standard to determine the presence or absence of a DSM-IV SUD – either substance abuse or substance dependence. Cronbach’s alpha was used to determine the internal consistency, or reliability, of the ASSIST. The discriminant validity of the ASSIST was determined by comparing the TSI and SSI scores on the ASSIST obtained from patients with a SUD and to those without a SUD. Receiver Operating Characteristic (ROC) curves were used to evaluate the screening properties of the ASSIST. Optimal cut off scores were calculated to maximise sensitivity and specificity. The sensitivity, specificity, and positive and NPVs were calculated. We determined the normality of the data using histograms and the Shapiro Wilk’s test for normality. Skewed data were normalised using logarithmic transformations where appropriate. Normal data were analysed using Student’s t-test and Pearson’s correlation coefficient. For non-normal data, Wilcoxon-rank sum test or Spearman’s rank correlation coefficient were used. Categorical data were analysed using Chi-square test with Fisher’s exact test where appropriate. For comparisons across three groups on the ASSIST TSI and SSI scores (no SUD, abuse, and dependence), Kruskal–Wallis analysis of variance (ANOVA), with a Bonferroni correction for post-hoc pairwise comparisons, was used. Following the determination of optimal cut-points on the TSI scores, a logistic regression model was constructed to determine the unadjusted odds of having a SUD after testing positive (scoring at or above the cut-score) and the adjusted odds after accounting for covariates (sex, education level, and diagnostic group). Two-tailed tests were used throughout, with p-values <0.05 considered as statistically significant. Stata version 16 for Windows was used to analyse data.

The University of Cape Town’s Human Research Ethics Committee (HREC) granted approval for the study (HREC 833/2019). As this is a secondary analysis of an existing dataset, the ethical considerations and risks were considered minimal.

Of the 124 participants, 58 (46.8%) had a SUD diagnosis on the SCID-I, and 66 (53.2%) did not. Alcohol and methamphetamine were the most commonly abused substances, followed by cannabis and ‘other drugs’ (which includes methaqualone, cocaine, hallucinogens, opioids, and inhalants). A SUD diagnosis was significantly associated with gender (higher in males), having less than 12 years of education, and a diagnosis of either SZP, BMD, or MPD or non-psychotic control status (Table 1).

For patients with a major mood or psychotic disorder diagnosis, the duration of illness as reaching threshold for diagnosis was of a fairly long duration for most patients (BMD median = 72 months; SZP median = 72 months; MPD median = 12 months).

Cronbach’s alpha for the TSI lifetime score (Q1–Q8) was 0.9, while the 3-month TSI was 0.8. We were unable to calculate Cronbach’s alpha for methaqualone, hallucinogens, opioids, and sedatives because of low prevalence. For the lifetime SSI scores, alcohol had a Cronbach’s alpha of 0.7, cannabis of 0.8, and methamphetamine of 0.9. Cronbach’s alpha for the SSI scores for the last 3 months were 0.6 for alcohol, 0.7 for cannabis, and 0.9 for methamphetamine, respectively.

Those with a SUD diagnosis on the SCID-I had significantly higher TSI scores compared to those without a SUD diagnosis (Table 2). The average lifetime TSI score in those diagnosed with a SUD was 31.3 with a standard deviation (s.d.) of 23.5, while for those without a SUD diagnosis it was 10.2 (s.d. = 9.4), which was a statistically significant difference (Table 2).

The 3-month mean TSI score in those diagnosed with a SUD was 22.7 (s.d. = 18.8), and for those without a SUD diagnosis it was 5.2 (s.d. = 7.4), which was also statistically significant. The SSI scores for alcohol, cannabis, methamphetamine, and other drugs were also significantly higher for those participants who had a SUD diagnosis.

The mean scores on the ASSIST TSI and SSI were then compared across the three categories of no-SUD, substance abuse, and substance dependence (Table 3). Bonferroni corrected post-hoc analysis for the TSI showed good discrimination between no-SUD and substance abuse (p < 0.001) and dependence (P < 0.001), respectively. However, discrimination between substance abuse and dependence was not significant (P = 0.564). This was also true for alcohol (no-SUD vs. abuse, p = 0.003; no-SUD vs. dependence, p < 0.001; abuse vs. dependence, p = 0.538). The ASSIST did not distinguish well between no-SUD and abuse for cannabis (p = 0.18), or between abuse and dependence (p = 0.533). However, it discriminated well between no-SUD and dependence (p < 0.001). Regarding methamphetamine, the ASSIST discriminated well between no-SUD and abuse (p = 0.003), no-SUD and dependence (p < 0.001), and between abuse and dependence (p = 0.039). For other drugs, the ASSIST discriminated between no-SUD and dependence (p < 0.001), however not between no-SUD and abuse (p = 0.189), or between abuse and dependence (p = 0.562).

We then determined the optimal cut-points for the TSI and SSI’s, with maximal balance between sensitivity and specificity (Table 4). It was found that a cut-off score of 13 on the TSI correctly classified 74.1% of cases and 74.2% of non-cases of SUD. The 3-month TSI score cut-off of 11 was found to correctly identify 88.9% of cases and 86.1% of non-cases. For alcohol, cannabis, methamphetamine, and ‘other drugs’, SSI lifetime cut-off scores of 4, 3, 3, and 3 respectively were found to have optimal balance between sensitivity and specificity. However, the cut-off scores for 3-month SSI scores for alcohol, cannabis, methamphetamine, and ‘other drugs’ were found to be higher, at 7, 10, 13, and 10 respectively. The area under the curve (AUC) was 0.7 or above for both the TSI (Figure 1 and Figure 2) and SSI’s.

Univariate logistic regression analyses were then performed. A TSI of above 13 was found to be a significant predictor of the likelihood of any SUD, with individuals scoring at or above 13, having an 8.3-fold increase in the odds of having a SUD. When adjusted for sex, education level, and diagnosis (controls as reference vs. SZP, vs. BMD, vs. MPD), the OR was slightly lower at 6.5. This cut-off correctly classified 80.7% of cases. For the 3-month TSI score cut-off of 11, the OR was 49.5, with an adjusted OR of 66.8; 95.2% were correctly classified (Table 5).

Limitations of this study include the exclusive use of clinical interviews (the SCID-I) to determine the presence or absence of a SUD, without any confirmatory biological measures of substance use. Other limitations include the sample size that was limited to that of the parent study, possibly leading to diminished power in some analyses. While we were able to perform analyses for alcohol, cannabis, and methamphetamine, the sample size for methaqualone, hallucinogens, opioids, and sedatives was too low to be able to analyse data for these substances independently. Given that alcohol and cannabis are among the most abused substances in South Africa,²⁴ the ASSIST remains very relevant to local conditions despite this limitation.

The 3-month ASSIST score was compared to a 1-month (‘current’ SUD) measure on the SCID-I. Thus, the time frames were slightly different, possibly affecting the results relating to these measures. However, this would not apply to lifetime measures.

Although all participants were ambulatory, stable outpatients, this was a heterogenous group with differences in illness duration and illness severity that could have affected reporting of substance use. Future studies may need to consider the impact of variables such as psychosis severity and may benefit from fuller characterisation of psychotic symptom course. As this study is a secondary analysis, the sample and controls were chosen with different study aims in mind. This may affect the generalisability of these findings to the real-world clinical populations.

The strengths of the study include the fact that it is the first to examine the validity of the ASSIST in a population with multi-episode bipolar or psychotic illness, with a diverse range of diagnoses, as well as the inclusion of a control group without bipolar or psychotic illness. The patient sample included a diverse range of illness (SZP, bipolar and methamphetamine induced psychosis).

Further research is needed to address some of the limitations of this study, including validating the ASSIST in an inpatient setting and using a larger sample size especially for patients using methaqualone, hallucinogens, opioids and sedatives. A primary study rather than a secondary one may allow the findings to be more generalisable.