In his book Black Box Thinking, Matthew Syed describes a program called Scared Straight where seventeen kids, each having been arrested for serious crimes, were sent to a maximum security prison for three hours. They witnessed and interacted with hardened criminals, many of whom were in jail for life. The idea was to introduce the kids to what life was like in prison. The experience, in theory, would shock them into good behavior.
The system initially seemed to work. Of the seventeen kids, sixteen stayed clean for the subsequent three months. That’s a short period of time, but it was encouraging, especially since each of the kids going into the program was a juvenile delinquent convicted of at least one previous crime. The program was continued. Eventually 8,000 kids went through the program.
The communities sending kids to the program reported an 80 to 90 percent success rate. According to a 1978 documentary, the program was “unequalled by traditional rehabilitation methods”. The documentary, in fact, won an Academy Award. It was shown in 200 major cities around the world.
In reality, according to Syed and rigorous subsequent statistical analysis, the program actually increased the likelihood of the kids committing future crimes. The program not only didn’t work, but actually did the opposite of its intended purpose. Communities were sending their kids to three hours of counsel by hardened criminals.
The flaw, it turned out, was the study only considered the results of people in the program that responded to a survey. Clearly the straight people for whom the program worked were more likely to return the questionnaire than the people for whom the program didn’t work, who were most likely either in jail or dead. The survey resulted in an incredibly biased sample. Oops.
In Deceptive Definitive Studies we described how people with an agenda can rig studies to show whatever conclusion they want and they can do so in very convincing fashion. The lesson here is more subtle. There was no intent to deceive anyone. The flaw in the analysis went undetected because no one bothered to look at the study in a critical fashion.
It was a feel-good story. Everyone loved the conclusion. There were no natural critics present to raise questions about the underlying methodology of the study behind the story. Some of the worst disasters in history have resulted when there were no bona fide critics in the room to poke holes in the methodology behind the narrative.
In September of 1938, Neville Chamberlain, then Prime Minister of the United Kingdom, traveled to Munich along with Edouard Daladier (Prime Minister of France) and Benito Mussolini (Prime Minister of Italy) to sign an agreement with Adolf Hitler that handed Germany control of a strategic piece of Czechoslovakia. Czechoslovakia, interestingly, wasn’t a party to the agreement. The only thing England, France, and Italy got in return for ceding control of a piece of land that didn’t even belong to them was Hitler’s promise that it would be his last territorial claim in Europe.
In March of 2003, George W. Bush announced the beginning of Operation Iraqi Freedom: “the people of the United States and our friends and allies will not live at the mercy of an outlaw regime that threatens the peace with weapons of mass destruction”. It was the beginning of a quick war with Iraq and a very long occupation of the country. No weapons of mass destruction were ever found.
In April of 1961, 1,400 Cuban exiles landed on Cuba’s southern coast in an attempt to overthrow Fidel Castro. The exiles were backed by the U.S. CIA which thought ludicrously that the 1,400 exiles could defeat Castro’s 20,000 troops. The exiles were destroyed where they landed: a small swampy cove called the Bay of Pigs.
Each of these foreign policy blunders was made by smart, competent people with good intentions, but the decisions were made by a group of people that desperately wanted a certain conclusion. Chamberlain and his co-signers desperately wanted to stay out of war. Bush and his allies needed an excuse for war. Kennedy and his cabinet wanted Fidel Castro gone. All agreed on the desired outcomes. No one checked the method for faults. The result in each case was catastrophic, the desired outcome simply wishful thinking.
Analysis conducted with a universally-accepted outcome in mind breeds apathy with respect to methodology. That leads occasionally to tragic results. As asset managers, we see studies and narratives every day that were prepared with the outcome in mind. Some are intentionally deceptive. Most are mere wishful thinking.
