Superforecasting – A skill for Black Belts

I am reading a book called Superforecasting: The art of science and prediction by Philip E. Tetlock.

This book is about making predictions and how by finding people who are very good at it, you can learn how to better your own skills.  This is another book, that is not about Lean Six Sigma, that can teach you skills that will help your Lean Six Sigma performance.

Forecasting, as used in this book, is about the collection of existing information  and then estimating the chance of something occurring.  Most of the research discussed in the book was performed by the US government after the belief that there were WMDs in Iraq was found to be wrong.

The intelligence community wanted to understand how they could be so wrong.  To answer this question, they brought together a bunch of academics and specialists in forecasting.  This group decided to understand forecasting through specific testing.  The goal was to find people that can forecast the future quite well.

Testing to find superforecasters

The collected a few thousand folks and then provided them questions to be answered with a forecast.  Things like “Will there be a border conflict between China and Vietnam in the next six months?” or “Who will be elected as the president of Ethiopia?” and other questions.  Each one had a short time until the answer was known and the topics may or may not be general knowledge to the participants.  Each person would provide their forecast and the probability that they would be right.  Plus, each person was allowed to revise their conclusion or the probability of being correct at any time, up to the deadline or the truth was known.  Scoring was using an interesting correlation scale that I am not sure that I completely understand, but it was on a scale from 0 to 2, where 0 was perfectly correct and 2 was the equivalent to perfectly wrong.  Scoring a 1 would be close to 50:50 or randomness.

As you might expect, the performance was distributed somewhat like a bell curve on the accuracy.  Yes, some were doing very well and a lot of folks were near the measure of pure randomness.  The first interesting fact was that the top forecasters were normal people.  They were not analysts, news junkies, Mensa winners, or in a profession that you would expect them to be good forecasters.  The testing ran a few years with lots of questions each year.  As you may imagine, the top people stayed at the top every year.

Randomness in forecasting

If Forecasting was a random process, we would expect the best forecasters in one year would be not at the top in other years.  This would be assignable to the concept of Regression to the Mean, where a person may have a great start, but after enough inputs are made, the long term performance will approach to the persons true overall skill level.

Consider if you were flipping a coin to pick your forecast and every head result would be correct and a tail result would be wrong.  We could easily experience an initial performance of 8 heads and 2 tails, and 80% success rate for the first 10 tosses, but if we continued to toss the coin 100 or even 1000 times, we know that the overall performance would be close to 50%.  This is what is meant by regressing to the mean of the process.

What they found is that for many of the participants, there was no regressing to 50%.  The testing found people that were consistently at the top of the performance.  This book, Superforecasting, examines the behaviors and methods of the top forecasters.  There were two primary skills found in the best forecasters that we LSS practitioners can learn from and use to estimate the improvement our project may attain.

Baselining and Bayes Theorem

The Superforecasters always seemed to estimate a result using what the author called an Inside and outside estimate.  They would start with the outside estimate, which is like the baseline in a LSS project.  This is the base estimate for the question without considering any specifics about the question.  An example of estimate the chance of a border conflict between China and Vietnam in the next 6 months, you would lookup how many have happened over the past 30 or 50 years to come up with a conflicts per year number.  Now you have the outside or baseline estimate.

Lets consider that the overall estimate is 1 every 4 years, or a 12.5% chance of a conflict in any six month period.  OK, this is our start.  Next they would consider the specifics around the current state and the next six months.  Based on what they find in the current environment, the forecaster would adjust the estimate up or down from the 12.5% baseline.  This adjustment is called the inside estimate.

An example would be the current dispute about the islands in the South China Sea between Vietnam and China, along with the fact that China is having conflicts with the Philippines and Indonesia right now.  This is a relatively new issue and should increase the chance of a conflict.  So we find that China has had disputes with half of the countries that lay claim to these island, that is a 50% conflict rate, so I would say that this so I could say that it doubles the chance from the historical level, so I would estimate the chance of a conflict at 25%.

The key to this estimating concept is that you make estimates based on a historical average and then adjust the estimate down or up small amounts based on specific knowledge of this case and time period.  Because we start with a historical average rate, the estimate will be close to the truth.

Bayesian Statistics

This concept aligns with the concepts of Bayesian statistics, where you are taught that the current state data is not adequate to make an estimate of a hypothesis test result.  You should consider all prior knowledge of a process as part of a hypothesis test conclusion.

An example of this Bayesian view would be something like the estimate of the long term yield of a new product.  Right now the product has a 12% yield loss and you are to estimate the yield loss in 12 months.  Is a 12% yield correct?  Probably not, because yields improve as products are produced.  To make this estimate, we research past new products and find that the average yield improvement in 12 months and find a value of 50% reduction in scrap in 12 months.  This is my outside estimate, because it considers all products to be equal.  When we look at this product we find that it is a very simple product, when compared to our average product, which would adjust the estimate to be better than 50% gain, say a 30% improvement on the 50% or 50/3 = 16.6% better, so we would estimate a 66.6% improvement in yields which corresponds to a 4% yield loss in a year.

Fermi’s Method

This is the concept of breaking the big question into a lot of small ones that we are able to answer or at least estimate with a reasonable answer.  The example in the book is to estimate the number of piano tuners in Chicago.  How would we directly answer that question?  guess.  But what if we Fermi-tized it.  We could ask the population of Chicago, which is about 3 million.  Assume 3 per family, 1 million families.  The number of pianos per family, of which we have no idea, so we could say 1 in 1000, or 1000 pianos.  Plus how many pianos are not owned by families in Chicago, lets say another 1000.  We are now at 2000 pianos.  So how often are they tuned, lets pick every other year, which leads to 1000 tunings a year.  How fast can a piano be tuned?  Say it is a 1.5 hrs per piano (found on google).  Since the tuner must travel to the piano in the big city of Chicago, they may only get two per day per piano tuner.  Assume it is a part time job (50%) because a piano tuner will fix pianos too.   Now assume 200 work days in a year which comes to 200 pianos a year per tuner.  Divide 1000 per year by a 200 per tuner.  This comes up to 5 piano tuners in Chicago.

Now I made up all of these numbers and the ones used in the book are different (I did not want to copy the book) but you can see how the Fermi method works.  Did we know many of these numbers?  No, but you could look up estimates of many of them.  The question for you would be if this Fermi method would be better than estimating it in one big step.

Another advantage of the Fermi method is that you can easily adjust your estimate as you learn more about any of the small questions combined to provide the overall prediction.

Applicability to Lean Six Sigma

As I was reading the book, I realized that that I have used both of these concepts, Bayes and Fermi, during the DMAIC process.  You can use the Bayes concept to look at yield improvements.  My project is addressing a specific 4% yield loss problem, history has shown me that when the the specific yield problem has not been found our yields were only 2% better overall, meaning that when the specific cause created a defect, there may have been other defects in the product that were not identified. My team believes that we have eliminated 75% of the specific yield loss.  We would estimate the new overall yield will be about 1.5% better than the current yield.  [(4*.75)*.5]

A potential use of the Fermi method might be to estimate the overall yield of a new product by estimating the defect rate found in similar products at each step in the process and then combining these results to estimate the overall yield.  These estimates can be used to target improvement areas along with what the overall yield will be after eliminating defect causes in specific steps.

Summary

The book has far more information in it to help us make predictions, I only covered the two that I liked the most.  This is a great book to read and you could find applicability in your job and your life.  I did not include the best finding in the book, which may be why you should read it, that these great forecasters got better at it as the years went by because it is a skill that can be learned.  You can learn from this book!