I am involved with statistical work to understand a process failure. We know there was a process breakdown somewhere. It is easy to recognize the artifact that was developed because of the problem. But can the problem be fixed without understanding the generating cause. Probably not.
Lean Six Sigma training and Root Cause Analysis provides great tools to understand how to narrow down a problem to a single step or a short period of time that something happened to create the problem. OK, there is the easy part. How do we remove the true causes?
The risk to a lightly trained improvement leader is to jump on the first “Cause like” factor that shows a correlation to the period that the problem occurred. Take this moment to remember what your mentors told you…
correlation is not causation
You know what I mean. Just because there is a small p-value does not create causation.
In the situation I am working with, we can find a number of things that were somewhat unique during the time of the problem. Testing shows a low p-value for each event. Is this enough to make changes and move on? probably not.
One of my mentors shared his thoughts on pilot testing: If the pilot test appears successful, resist the drive to just leave it in place. You have not really proven it better until the removal of the pilot actions allows the problem to return. I think this thought should be considered when ever we have strong correlations for a problem. Unless we verify that our proposed cause will actually create the problem when introduced, and then avoid the problem when it is removed, we have not proved it. In Lean Six Sigma that is part of the improve process that many of us skip, validation of the solution. Just because the problem is gone does not provide a validation of causation.
This topic came up to me as I am working on an issue that cannot really be evaluated with a simple pilot test. We are approaching the validation in a more theoretical way. With two paths, one being a look at the the physical theory behind the process at that moment. Would theory support the belief of each cause? Unless theory will support the belief, a cause will be dropped.
The second method, which is fun for many of us statistical types, is to build a simulation of the process and see how it behaves. Is this testing theory, well sort of, but not really. In a simulation, we simulate the single components that we understand and let the basic model determine the outcome. This also gives us a chance to look at the probability of outliers and extreme events. Our belief is 1) if the simulation supports a cause, 2) the theory supports the cause, and 3) the observational analysis indicates the cause did exist then all three support a belief strong enough to adopt a change without any physical confirmation.
What do you think? good enough?