Blog

Diagnostic Unreliability of the DSM

The American Psychiatric Association (APA) has proposed a roadmap for the future of its Diagnostic and Statistical Manual of Mental Disorders (DSM). Five papers have been written by members of the Future DSM Strategic Committee. “They propose a forward-looking model for the evolution of the DSM.” One of their suggestions is to change the name of the DSM from Diagnostic and Statistical Manual to Diagnostic and Scientific Manual, “to better reflect its scientific and global scope.” But whether the DSM is truly scientific is debatable.

One of the current critics of the DSM is Nassir Ghaemi, a psychiatrist who has authored over 300 articles in the scientific literature and multiple books. He’s been critical of the DSM diagnostic system and its emphasis on reliability of diagnosis over scientific validity. Along with Mark Ruffalo, he wrote “Future DSM Plans Overlook Core Problem of Validity” for Psychiatric Times. Ghaemi and Ruffalo noted the Future DSM Strategic Committee proposed 4 organizational adjustments: revisions to its architecture, attention to biomarkers, contextual factors and measurement of functioning and quality of life. However, the authors thought the committee largely overlooked the DSM’s core scientific problem—”Its lack of validity.”

Before reliability, before the debate over categories vs dimensions, and before we add biomarkers or social determinants, the central question remains: Are the diagnoses real? DSM‑III acknowledged that most diagnoses lacked validation; DSM‑IV repeated the structure; DSM‑5 avoided the question; and the future DSM appears poised to do the same.

If the diagnostic system is valid, psychiatry can advance scientifically. If it is invalid, everything that depends on it—such as biomarkers, epidemiology, therapeutics, reimbursement, and research—rests on an uncertain foundation. One of us (S.N.G.) raised these concerns directly with leaders of the Future DSM Strategic Committee, but they received little weight. This article explains why the DSM remains scientifically weak, why the new proposals do not address the underlying issue, and why the field of psychiatry must adopt a scientific model based on empirical data rather than consensus.

Ghaemi and Ruffalo said we have been repeating essentially the same DSM system since 1980, where the leaders of the DSM Task Force were members of the baby boomer generation. But now, the leadership has shifted to Generation X, those who came of age in the 1980s and 1990s. “Yet despite this shift, the central problem that has limited progress in psychiatric diagnosis remains unchanged: The diagnostic system is not grounded primarily in validity—that is, in the reality or truthfulness of its definitions.” They argued that the only viable option was to adopt the standard approach used in other areas of medicine: “Diagnostic definitions should be determined by scientific experts on the basis of empirical research, rather than by consensus or preference.”

Before the DSM-III was published, “Research diagnostic criteria: rationale and reliability” described the development and initial reliability studies of a set of specific diagnostic criteria for a selected group of functional psychiatric disorders known as the Research Diagnostic Criteria (RDC). Ghaemi and Ruffalo said the authors argued that psychiatric diagnosis should be based on validity and reliability, in that order; and 14 valid diagnoses were identified. “Diagnostic validators were proposed, based on the fact that an external validator is always needed for symptoms.” The RDC group evolved 4 basic diagnostic validators: symptoms, genetics, course of illness, and biological markers. “No single validator is sufficient to make a diagnosis valid. The more validators point in the same direction, identifying a clinical condition that is distinguishable from other conditions and from normality, the more valid the diagnosis.”

The RDC group was subsumed into the DSM-III Task Force, and in 2 years, 14 valid diagnoses were expanded to 292 diagnoses, which means that 278 (95.2% of the total) were invalid. They were invented, created out of thin air, based on the wishes and beliefs [in other words, consensus] of clinical leaders in psychiatry. The DSM-III leadership admitted this fact and argued that if the 95% invalid diagnoses could be defined well (reliability), then future studies could change them to make them more valid. That did not happen, of course, because the leaders of DSM-IV in 1994 refused to make major changes to DSM-III, and the leaders of DSM-5 in 2013 refused to make major changes to DSM-IV. Reliability has not led to validity, and yet the future DSM leadership continues to claim that reliability matters more than validity. A dictionary is useful if the definitions in it are true (valid). A false dictionary is worse than useless; it is misleading. [The DSM 5 now contains over 300 distinct mental health diagnoses]

Mark Ruffalo said there were three possible paths forward for a future DSM with a validity-based hierarchy where diagnoses are ranked according to their empirical support. First, diagnoses would be ranked with those with robust validity at or near the top of the hierarchy, and diagnoses with limited or poor validity at the bottom of the hierarchy. Another path would be to replace the DSM system with a new system of clinical research diagnostic criteria, using the four diagnostic validators to identify and define diagnoses as done with the RDC of the 1970s. The third path would be to eliminate the DSM altogether. Clinicians would rely upon textbooks and scientific research to guide diagnosis, “as is done in every other branch of medicine.”

Insurance reimbursement would depend on ICD-10 or ICD-11, as with the rest of medicine. “There is no need for a DSM coding at all for anything.”

DSM-III was a social construction; it was not a scientific document. It was created for the social and political purposes of the American psychiatric profession. That remains the case. DSM revisions happen when the American Psychiatric Association needs to renew its main source of income—sales of DSM. The economic motivation for continuing the DSM enterprise is hard to ignore.

Instead, psychiatry should reconsider the continued centrality of DSM to clinical practice and research. The current system remains heavily shaped by administrative, economic, and legal considerations, particularly insurance reimbursement, rather than by the scientific question of diagnostic validity. Yet diagnostic concepts should be determined primarily by empirical evidence concerning the nature of psychiatric illness, not by professional consensus or institutional convenience. If psychiatric diagnoses are to advance scientifically, they must be grounded in reality rather than maintained for pragmatic or bureaucratic reasons. Biology and nature do not care what we want; they won’t adjust to our DSM wishes. We need to adjust to reality, not the other way around, and find out which psychiatric illnesses really exist. The DSM approach is abetted by a widespread antiscientific postmodernist attitude that we will never know how to reach validity, and thus we can make things up as we like. Our patients deserve better.

Poor Overall Reliability of Psychiatric Diagnoses

A new article published in Frontiers in Psychiatry, “Reliability of psychiatric diagnoses in the 21st century,” found that psychiatric diagnoses were often inconsistent and unreliable, “with clinicians often disagreeing on what diagnosis a set of systems should receive.” Using standardized clinical case vignettes, they conducted a large-scale international study of diagnostic reliability among medical doctors. They found a poor overall reliability of psychiatric diagnoses that corresponded to a situation where two clinicians, assessing the same written clinical case, agreed on diagnosis about half of the time. The authors said their conclusion echoed that of Spitzer and Fleiss when they said in their 1974 article, “The reliability of psychiatric diagnosis as it has been practiced since at least the late 1950s is not good” in “A Re-analysis of the Reliability of Psychiatric Diagnosis.”

Spitzer and Fleiss were lobbying for the development of structured interviews and specifying all diagnostic criteria in pre-DSM-III time. They said these changes “will result not only in improved reliability, but in improved validity, which is, after all our ultimate goal.” In “Reliability of psychiatric diagnoses in the 21st century,” the authors said: “Despite the introduction of operational diagnostic criteria and rule-based classification, the overall reliability of psychiatric diagnosis, assessed on written cases, is modest.” The authors of “Reliability of psychiatric diagnoses in the 21st century” said:

From a clinical perspective, the level of diagnostic agreement is not merely a statistical concern but has direct implications for patient care. If two clinicians are likely to assign different diagnoses to the same patient, this raises questions about the stability of treatment decisions, prognostic expectations, and communication across clinical settings. Diagnostic categories are often treated as if they reflect stable, clearly bounded entities, but the present findings suggest that their application in clinical practice may be more variable. Notably, diagnostic agreement did not differ significantly among the doctors, neither when stratified into position (psychiatrist, residents, and other medical doctor) nor when stratified into years of clinical experience. Thus, the observed diagnostic variability cannot be explained simply by varying levels of clinical experience.

They concluded their findings suggested that refinement of diagnostic criteria alone may not be sufficient for achieving high levels of diagnostic agreement. “Further empirical research is needed to better understand the sources of diagnostic variability and how they may be addressed to improve clinical practice and psychiatric research.” For further discussion on this study, see “When Doctors Disagree: New Study Raises Questions About the Reliability of Psychiatric Diagnosis.” For a deeper discussion of the historical context surrounding the Spitzer and Fleiss study, see “The Quest for Psychiatric Dragons, Part 1 and Part 2. 

About Anselm Ministries

Drawing its name from an eleventh century monk and theologian who had a profound impact on Christianity, Anselm Ministries is a church-based teaching organization whose purpose is to support the pastoral care of the local church. It seeks to help individuals grow in their faith and their understanding of how to live godly, Christ-centered lives.

Share This Post

X
Facebook
LinkedIn
Pinterest
Email
Print

Discussion

Charles Sigler

D.Phil., Licensed Counselor, Addiction & Recovery Specialist

Share This Post

Recent Posts

If psychiatric diagnosis is invalid, everything that depends on it rests on an uncertain foundation.
Rescheduling cannabis could lead to research showing the adverse medical and psychiatric effects of cannabis.
Wallnau's allusion of Trump as Cyrus may tickle the ears of his charismatic hearers, but it is not a legitimate use of Isaiah 45.
You restrict it, and then they just move on to the next new thing in opioids.

Favorite Posts

The bottom line is The Passion Translation (TPT) is not really a bible translation. Bible Gateway had good reasons to justify its removal.
There does seem to be a “fuzzy boundary” between Substance Abuse and Substance Dependence. Allen Frances suggests we simply ignore the DSM-5 change.
Marijuana researchers like Stacie Gruber are concerned that “policy has outpaced science” when it comes to lawmakers making public health decisions about recreational and medical marijuana.
If researchers and academic psychiatrists never believed the chemical imbalance theory of depression, why weren’t they as assertive challenging this urban legend?
The Niebuhrian version of the Serenity Prayer seems to have clearly come from Reinhold Niebuhr’s 1943 sermon.
“The kingdom is the whole of God’s redeeming activity in Christ in this world; the church is the assembly of those who belong to Jesus Christ.”

Related Posts

Search this Site