Being a Master Black Belt has always involved a lot of activities outside of your primary skills. We are so good at solving business problems and figuring out process issues, that our business may begin to believe we can figure out almost anything.
Well, in the past few months Forrest has tasked me with developing a software validation method for the Enterprise Performance Reporting System (EPRS) chart generation macros. In the beginning, the macro functionality was simple and I could test it by just running it a few times.
As the complexity grew, testing software functionality began to be more complicated than just some ad-hoc tests. I knew this because as the usage of the EPRS has grown, we have found more and more bugs in the macros.
Formally testing software functionality
I talked with Forrest and Gaylen about this issue and Forrest reminded me about Pass-Fail Functional Testing (PFFT) methods that we used to teach in our Master Black Belt course and the Design for Six Sigma course but was dropped when we moved to the IEE textbook from the Implementing Six Sigma text book. This method uses the design of experiments methods (DOE) to create a functional test plan that will identify functional combinations of settings that fail to work properly.
The PFFT method creates a 16 or 32 trial test matrix that involves all of the possible settings and then provides a simple method to identify which combinations caused failures. When we taught this in our courses it was more about hardware testing but it worked well for software too.
Pass-Fail Functional Testing
The class example discussed bringing a new pc operating system into a company. How would that IT organization verify that it would not cause problems. To use the method, you would list all of the existing options that are involved. This could be a list such as;
Communication protocols: TCPIP, FTP, HTTP, IMAP,..
Operating systems in use: WIN7, WIN10, OSX, LINUX, Win Server 2010
Output devices: HP printers, Canon multifunction printers, Postscript printers, Large format plotters
Scanners: Canon multifunction scanners, dedicated scanners, business card scanners
PC types: HP desktops, Dell Desktops, SGI workstations, IBM laptops
and so on….
Then you input this into a simple software program (which we sell) and you receive the test plan. Execute the test plan and then work through any failures.
EPRS use of the PFFT methods
I must say that I thought my MBB skills gave be an ability to use ad-hoc methods to produce a valid testing of the software. I was proved to be wrong! We kept finding little coding errors and I would not face up to the problem being my validation plan. So I used the PFFT for each of the five reporting functions in the EPRS. Forrest help me build the PFFT plan and then I coded it to be executed using the EPRS. I probably spent about an hour setting up the test for each macro (5) and then it took about 5 minutes to run the gamut of tests for each macro. Even though I believed the macros to be error free, three of the five showed internal coding errors that I had to resolve.
I have fixed all of the coding errors and now the macro’s are error free. Being a Lean Six Sigma practitioner, I know that this is not where you stop a project. I needed to institute a process for using the PFFT after every macro update effort to make sure we do not impact the other functions when a change is made.
Making the test of software functionality permanent
I documented all of the work to develop the software options to be tested along with the final test plan, but also the input files for the PFFT software. We may need to add to the option list and generate a new test plan in the future. Then I archived the actual code used to execute the test plan so that we can reuse it or modify it to include more functions. Along with this, I needed to archive a special minitab worksheet that was created that allowed for testing of all of the functions so that the next tester will not begin with nothing.
All of these pieces were then documented as part of a testing process instruction that would allow someone else to run the validation testing. Now I am done, or am I?
What else needed to be done?
The Pass-Fail functional testing is built around an assumption that all combinations should function properly. I also needed to perform testing that known data entry errors would produce errors that would generate the proper error message and allow the EPRS to complete its operations safely. Safe means that there is no crashes, hang-ups or other problem that causes user angst. For this, I just created a second test file that included every type of entry error and then a verify that each produced a proper error message that a user could understand and fix on their own. An example of the type of errors that were being tested would be like specifying the wrong column for the charting data, in that it was a text column or a date column rather than numbers. Or referencing a column of data in the chart that has a different column length than the data column.
I then archived these test files with the others. Now I am really done.
Summary
Assuming you are smart enough to not need to follow a formal process is risky. LSS practitioners should follow best practices and processes rather than shortcutting things. I learned a lesson, again, that I warn students about.