Author: Shewring, Elianne1
Published in National Security Journal, 02 October 2024
Download full PDF version – The Application of the Devil’s Advocacy Technique to Intelligence Analysis (377 KB)
Abstract
Understanding evolving situations and predicting future events are significant challenges in the intelligence sphere. These necessitate advanced techniques for intelligence collection and robust evaluation procedures to enhance the accuracy of predictions. The Devil’s Advocacy technique can be employed to improve analysis procedures by challenging prevailing views and mitigating the risk of cognitive biases. This paper examines the effectiveness of the Devil’s Advocacy technique when applied to intelligence analysis, by evaluating its strengths, weaknesses, and applicability, through a comprehensive literature review and empirical research involving experienced practitioners in New Zealand. Previous studies examining the Devil’s Advocacy technique primarily focused on its contribution to decision-making on the final intelligence product and on the perceptions of the devil’s advocate. However, this paper is also concerned with the analysts’ experience of practically applying the technique for intelligence analysis. This paper contends that the Devil’s Advocacy technique is an attractive complementary technique which is effective in specific situations. It also reveals that New Zealand intelligence practitioners define, perceive, and apply the Devil’s Advocacy technique differently across their organisations. It therefore argues that New Zealand practitioners may have an inconsistent understanding of the technique, indicating potential gaps in the training and education of New Zealand intelligence professionals. However, the inherently subjective nature of the technique itself could also be a contributing factor.
Keywords: Structured Analytic Techniques (SATs), Devil’s Advocacy Technique, intelligence analysis, cognitive biases
Introduction
The task of predicting the future and understanding the dynamics of situations as they unfold is a key challenge in the intelligence sphere.1 It has become even more challenging, as the nature of warfare has expanded from traditional conflicts between states to long-running confrontations involving non-state actors such as terrorist and guerrilla organisations. Globalisation has also compounded this problem. The availability and convenience of international travel, coupled with the ubiquity of internet access, have magnified the spread of information and disinformation. As a result, the kinds of threats we encounter today are more complex to identify, evaluate, and anticipate than before.
Against this backdrop, intelligence practitioners recognise that the task of accurately anticipating events is highly demanding.2 Analysts cannot fully predict the future but can increase the accuracy of their predictions by developing robust research procedures to evaluate intelligence. One such technique which aims to improve the research procedures for evaluating intelligence and reduce the potential for conceptual collectivism or groupthink is the Devil’s Advocacy technique.3 According to Coulthart, this technique is the most effective and has the strongest evidence base for improving the analysis process amongst the 12 Structured Analytic Techniques (SATs) he evaluated.4 Its main purpose is to question the dominant view held by analysts and decision makers to enable them to critically evaluate the robustness and validity of their assessments.5
This paper aims to gain a better understanding of how the Devil’s Advocacy technique assists analysts in conducting intelligence analysis, and how much weight is given to the devil’s advocate’s position when making decisions. This investigation will include an examination of the technique’s strengths, weaknesses, applicability, and overall effectiveness when applied to the intelligence domain. The technique’s effectiveness is evaluated by assessing whether it is applicable to a wide range of intelligence problems, whether its application leads to plausible alternatives, and whether it is worthwhile in terms of time and cost.
This paper is divided into two main sections. The first section includes a comprehensive literature review on the use of the Devil’s Advocacy technique in the intelligence realm, covering two primary case studies in Israel and the Netherlands. The second section draws on recent empirical research with experienced New Zealand practitioners to examine the following question: How is the Devil’s Advocacy technique applied in the New Zealand intelligence analysis context? Rather than evaluating intelligence cases to which the Devil’s Advocacy technique had been applied where much of the literature focuses, this section investigates how analysts understand and practically apply this technique in their work environments.
This paper establishes that the Devil’s Advocacy technique is an attractive complementary technique, as its application is effective in certain circumstances. This is attributed to the technique’s potential to add significant value at a relatively low cost. Furthermore, this paper argues that New Zealand practitioners employ the Devil’s Advocacy technique in an ad hoc manner, without necessarily having a comprehensive understanding of the workings of the technique. This paper contends that this may be ascribed to the technique’s subjective dimension. Although the focus of this paper is the Devil’s Advocacy technique, it is conceivable that other SATs are also employed in a similar manner across the New Zealand intelligence community. This could suggest that New Zealand analysts may not be receiving sufficient or consistent training in the application of SATs.
__________________________
1 Elianne Shewring holds a Master of International Security from the Centre of Defence and Security Studies, Massey University, New Zealand, and is currently a Researcher at Mitvim, The Israeli Institute for Regional Foreign Policies. She wishes to acknowledge her supervisor Dr John Battersby for his insights, support, and encouragement during this research. Correspondence with the author can be addressed to the Managing Editor NSJ at CDSS@massey.ac.nz.