John F. Bergin
Bellevue, WA
(425) 242-1394
John.F.Bergin@Comcast.net Resume
Word Formatted Resume (Right Click and "Save Target As")
Cover Letter White Papers Code Examples
Senior
Software Engineer
How to Assure Software Quality
by John F. Bergin
The Universe of User Actions
If Sherlock Holmes were a Software Engineer, he might say something like this:
"If you can reproduce everything that a user could possibly do and not break your software product, you have assured quality"!
The first thing I hear from those who do not believe it is possible to assure, i.e. "guarantee" quality is that you cannot possibly do everything a user
might do given the constraints of the release (time, manpower, experience, etc). Believers know that you can get very close, close enough to eliminate
any doubts that the product will meet and surpass the users needs and expectations.
The Universe of possible user actions is infinite, which is one reason many people doubt the ability to reliably and repeatedly assure software quality.
With that said, there is a finite set of actions within this universe which represents the things a real user would do. For example, one possible user action
could be holding the "A" key down for a day, but it's doubtful this action would be encountered in the real world.
The set of valid user actions can be further simplified by by describing them as actions and permutations of actions. For example, a 3D game developer
might perform the action of coding a scene that uses "Depth Testing", which renders the object closer to the eye, and obscures objects that are behind it.
Permutations of that action would be "Triangle in front of a Triangle", "Triangle in front of a Line", etc.
For the purposes of Software Quality Assurance, the finite set of simplified actions that a user might perform can be captured via test
documentation, and the permutations of these actions can be exploited via test automation!
Test Documentation
Documenting test procedures is the single best way to capture the list of valid user actions, which can later be used as a roadmap for developing test
automation. The test procedures should documented according to the natural level of formality1 of the organization. For most organizations, a simple
checklist of valid user actions will suffice. The test documents should be used for manual testing of the software release, and should be revised after each
release to reflect changes in the product and optimizations of the test procedures. As test automation is developed for each test procedure, the manual test
steps can be minimized or eliminated. The test procedures replaced by test automation should not be removed from the test procedure document, instead
the steps should be grayed out and notated, indicating they are being tested by the automation.
Test Automation
Once the list of valid user actions is captured and described using test documentation, test automation can be developed to reproduce the permutations of the
list of actions. I cannot emphasize enough the importance of first developing test documentation! The biggest mistake you can make is hauling off and developing
test automation without first accurately analyzing and capturing what to test. Unfortunately in today's industry, this is exception to the rule, and developers of test
automation rarely document a complete system test before embarking on test development. The reason for this is the industries overvaluation of Computer Scientists
at the expense of Software Engineers. A good way to look at this is that Computer Scientists are mechanics, while Software Engineers are drivers, and although
you cannot diminish the value of mechanics in auto racing, you wouldn't put one behind the wheel of your multi-million dollar race car!
A Real World Example - Microsoft's "PIX for Windows" Testing
Over the last 3 years I have implemented the strategy I've described in this article with tremendous success. During a series of contracts with Microsoft's Xbox
and DirectX Performance Tools Test Group, I was responsible for the quality of PIX (Performance Investigator for Xbox) and PIX for Windows. Over 11
product releases I enjoyed a 100% quality success record!
To emphasize the value of test documentation, during one of my mandatory 100-day breaks2, the Microsoft personnel who took over my testing duties failed to
consult the test documents that I had prepared, and subsequently allowed a severity 1 bug slip into production, costing Microsoft a considerable amount of money
and embarrassment. It was later discovered that the action that failed was listed on one of the test procedure documents!
The manual test procedure document uses a checklist of major feature areas to be tested (i.e. the list of simplified user actions), trusting the expertise of the tester
to manually explore the permutations of each feature area. Taking the different operating systems into consideration (Windows XP, Vista, and Win7) as well as
32 and 64-bit configurations, the manual test consists of approximately 300 test cases, and takes about 1 week to execute.
After the manual test document for PIX for Windows had been used reliably for two product releases, a decision was made to automate the testing. The first
candidate to automate was obvious, the PIX for Windows Sanity Test. The Sanity Test was documented in the manual test procedure document, and tested
the most basic features of the tool, as well as all forms of input and output. The Sanity Test was used to give a base level of confidence for releasing the tool
for internal consumption. For example, The XNA Developer Connection (XDC), which acts as a liaison between Microsoft and Microsoft's major game
development customers, would request the latest version of PIX for Windows, with the soon to be released new features. The requests usually came without
warning, and I would have to drop everything and perform the 4-hour manual sanity test before releasing the product internally.
Using the test documentation as a roadmap, the PIX for Windows Sanity test was automated in relatively short order, and once the test was integrated into the
product test lab, it would launch automatically at the conclusion of the daily build. Instead of me dropping everything for a 4 hour test, the results of the sanity test
were being emailed to interested parties and available when they came into work in the morning!
During another mandatory 100-day break, the PIX for Windows Sanity Test was the primary test agent for the successful March 2008 product release, finding
and reporting multiple severity 1 bugs!
The PIX for Windows Sanity Test was a good first choice for automating since it was a high level horizontal test of the product, which tries to test the major functions
from end-to-end. The next automation candidate was a vertical test, which attempts to delve deep into the permutations of a single feature. Probably the most valuable feature
of PIX for Windows is the "Shader Debugger", which allows a game developer to look at and step through machine language code on the computer's graphics card. The
PIX for Windows Shader Debugger Test was the second automated test to be developed. Using the Shader Debugger Test as a template, additional vertical tests were
developed including the "Mesh Test", the "Objects Test" and the "Data Validation Test", all of which could be grayed out in the manual test document.
At the conclusion of PIX for Windows Test Automation development, the 250 of the 300 manual test cases were replaced with 10,000 automated test cases (80% automated),
and the 50 remaining manual test cases can be executed in a single day!
by John F. Bergin
1. - See "Natural Level of Formality" in "Software Engineering Strategies"
2. - See "End A-Dash Contracts"
____________________________
Copyright © 2010 by John F. Bergin.