Importance of Testing in Envision and Planning Phase

When you are working as a Test Consultant and if you are involved in the early phases of the engagement, it is quite important to deliver the test artifacts on a regular basis to show measurable progress. That can help a long way to win the confidence of the customer and the stakeholders involved. Arguably, the other disciplines (like developers, architects, business analysts) have relatively less to do to prove their worth as they own the key artifacts like Vision scope document & Functional specifications, which are perceived as the most critical ones for the success of the project. Though the test planning is an equally important activity as any other development activity of envisioning and planning phase it doesn’t get the same attention. Overall it gives a wrong impression that the testing team is not adding as much value as the development team in the initial phases of the project. I believe it is important to write a quality test plan that is not verbose but highly customized and designed, keeping in the mind the specific requirements and technology stack for the given project. It is critical for us to educate the customers by evangelizing the importance of testing in the initial phases.

I was in a discussion with the customers and they had this strong opinion that they are paying a lot of money for testing teams to just review requirements and write test cases. They recommended us to write automation scripts only without spending any time on designing test scenarios and test cases. I highlighted our role and contribution in requirement elicitation process to identify the gaps and save the effort upfront rather than discovering them in the build or stabilization phase. To accomplish that goal we need to spend effort in coming up with a detailed test plan and strategy that best fits the unique needs and requirements of the project in hand. We methodologically identify the areas that are best candidates for automation based on the ROI (Return on Investment) to save effort and time during the test execution phase. The quality of the automation scripts depends on the coverage of the underlying test cases / test scenarios and hence it is important to have them. At the end of the discussion I could sense that the customers were still skeptical and I didn’t expect them to get totally convinced either. I bought time and promised them to give a small demonstration of our capabilities by doing some quick POCs (Proof of Concepts).

From the requirements, I picked the most important ones and studied the pattern, the behavior and looked at the testability aspect of them and prepared a list of test techniques. I wanted to reduce my test cases to a number that is manageable and maintainable. I figured out that they had lot of complex process workflows where there are multiple jumps, branches and loops. I couldn’t cover them even if I wrote a whole bunch of test cases and even after doing that it will not guarantee that I didn’t miss an important path. We had the Visio's of those workflows so why not use a “State Transition Table”? I came up with a quick template to represent their most complex workflow and ensured I covered at least every state and every transition once. I had added a flag to be able to filter the main flows/alternate flows.

Then there were a few complex business rules that had lot of conditional clauses and the result depended on the outcome of all those clauses. Again I realized that there can be lot of permutation here and I didn’t want to take chances. “Decision Table” just sounded right to me. I quickly prepared one covering positive and negative flows and here I had a table that was covering it all.

For lot of data entry forms, there were many fields and especially drop downs, radio buttons and checkboxes. I didn’t want my team to randomly pick values and tomorrow miss double-mode types of defects. “Orthogonal array of testing” can be helpful and I thought of using its Pairwise implementation to solve that problem. I used a free tool from Microsoft called PICT and voila from 72 permutations I had now arrived at just 9 permutations scientifically that ensures every pair is covered at least once.

After a couple of days I had put together a few running POCs of these techniques in addition to test cases that used standard techniques like EP & BVA (Equivalence Partitioning and Boundary Value Analysis). I quickly came up with the best way to structure and organize my test artifacts using Microsoft Test Manager. I created a sample Requirement Traceability Matrix using Test Manager. I decided on my test suites and also defined different test configurations as the application is being targeted for multiple platforms.

In 3 days, we were in a conference room with all the customers and stakeholders including our project team. I ran the demos of my test techniques on the real requirements. I showed them we would ensure test coverage and provide them the requirement traceability reports using Test Manager. PICT was like magic to them. State transition tables taught them that testing is much more than randomly clicking buttons and following plain steps. Automation can be done only after you perform these key activities and test cases are not just plain word documents.

The meeting went well and ended with encouraging remarks like “Can you send me the link of that tool? “, “Can you share these with us for Unit testing?”, “We want you to demo this to our test team", “This sounds great”. It was evident from their response that we had their confidence.

Stay tuned for more post. I can be reached at rajkamal@microsoft.com

Disclaimer: This blog is my personal opinion and I apologize if it hurt your sentiments. In the words of my friend Arun (aruncha@microsoft.com) , “When a developer starts in a project he is already 100 miles high whereas a tester is 100 miles buried under the ground and it takes the tester lot of work to reach up there and prove his value before his work is acknowledged “