Much of how our analysts and contributors are trained stems from time at Mercyhurst University. Training administered by the Ridge College for Intelligence Studies and Applied Science have gone through pain-staking lengths and hours spend on theses, have generated an effective means for evaluating sources found online and/or how to portray likelihood. Both Source Reliability Matrices as I will call them, and Kesselman Words of Estimate Probability, commonly referred to as WEPs are the bread and butter of our techniques in evaluation.
Lets dive a little deeper so we know whats going on:
Source Reliability Matrices || Trust Worksheet
Analysts working in Law Enforcement, Business, even National Security need effective means of evaluating sources (even those online). This is increasingly important in a world where both nefarious and well intended contributors can hook up to an internet connection and begin generating content at all corners of the world.
Dax Norman an individual I first heard about from Kristan Wheaton, a professor and friend, developed a Trust Scale and Website Evaluation Worksheet in 2001 while working on his thesis at the Joint Military Intelligence College/National Defense Intelligence College. (At some point, he was the Curriculum Manger for the National Cryptologic School too).
That said, his evaluation techniques are still being taught to intelligence practitioners and aspiring intelligence analysts alike. We like numbers, we like quantifying things. What once was un-quantifiable now is. Norman’s Trust Evaluation Worksheet is based on the survey results received from 66 information researchers in business, academia, and government. To implement this Trust Worksheet-copy the image to the right and evaluate your source with the corresponding values applied if the source adheres to the criteria. See figure.
Kesselman Words of Estimate Probability (WEPs)
Words of Estimate Probability (WEP) are terms that intelligence professionals use in the production process of analytic products to convey the likelihood of some future event occurring. The analysts confidence will also support the WEPs.
For instance, an analyst may be looking at some aerial footage of Russian troop movements. She pulls the imagery for 10 days straight and only sees heavy military equipment being transported near a border region for 2 of the 10 days analyzed, then no movement. She would perhaps make the analytic judgement that, “It is probable that Russian forces are providing maintenance to heavy machinery. Analytic confidence is low as at the time of evaluation information from inside the facility could not be corroborated with human sources on the ground.”
We take the Kesselman WEP approach to analytic probability. These WEPs are always used in conjunction with information that may support the intelligence question provided in the requirement. If less information supports the question, the WEPs classify into 1-45% (Remote, Highly Unlikely, Unlikely). If the chances are indistinguishable due to lack of information or inconclusive evidence they will change from 46-55% (Changes a Little Better [or Less]). Then finally, if the information is coming together and painting a clearer picture of the operating environment then we get into the 56-99% intervals (Likely, Highly Likely, Almost Certain), where Certainty is 100%. Once analyzed like in the example above, to support the analyst’s judgement; High, Moderate, and Low analytic confidence need to be ascribed to voice the analyst’s feelings towards the evaluation.
This is How We Operate
At the Intelligence Observer we are committed to providing intelligence briefs from open sources and we hope to provide insightful analysis utilizing the two principles discussed above to reduce your uncertainty on global events. We hope you enjoyed this piece about our methods and hope it shows you–the reader–how we evaluate and advocate our analysis!
Sources and Methods – For additional resources on methods please check out our friends at Sources and Methods