Tester Testifies

Do you have it in you? Technical skills are no more OPTIONAL skills for Software Testing

rajkamal@microsoft.com (Raj) — Wed, 07 Oct 2009 10:11:00 +0000

Words of Caution: When i say “Technical skills” please don't confuse it with “Test Automation” knowledge only. It more and the less has become a standard in the industry to presume that knowing QTP or RFT is same as having technical skills.

To start with, for a tester to be successful following are the 3 most important components which you should never forget:

In this blog post, we will talk about the second most important component “Technical Skills” which is only next to “Testing Skills & Aptitude” for a tester to be successful in the next decade.

In last decade or so, testing has been tagged and given the title of an easy profession where non-technical candidates (who aren’t capable of writing or understanding the code) can write some test cases and execute them to find issues. It was considered having domain knowledge to understand the business is sufficient to be a successful tester.

In last 5 yrs, automation has become the talk of the town as it has become a successful marketing tool to win more projects by showing huge ROIs benefits of million of dollars to the customers. We have also witnessed the same trend by the emergence of specialized commercial testing tools by big players like IBM, HP & Microsoft in this arena.

Lot of candidates in testing do have technical skills but they don't hone it or have stopped using them because of the popular perception that “its not test‘s job to write/read code”. The max they do is automation (where record/playback is most popular and only a few percentage (<20%) can truly write automation code to validate the application)

Because testing is considered a second fiddle to Development and the best minds get/shift into development, testing community is lagging seriously on candidates with strong technical skills.

Test automation is a subset of a broader category “Technical Skills” and now (more than ever), importance is being given to other technical skills other than test automation which are becoming mandatory to become a successful test professionals at organizations.

Following 10 are the skills / activities / tasks where technical skills of a tester is tested

(you might have read many of these in your testing training / process tutorials but now time has come to start exercising these)

1. Requirement Feasibility:

Don’t get surprised, there is not typo error here. You read it all right that requirement analysis ALSO requires you to have technical skills to understand if the requirements expected by the customer are POSSIBLE or not. Lot of times there are “technology constraints” and unless you understand the underlying technology you run a risk of committing on a requirement which can’t be implemented and you must know that telling customer on the last day wont be such a great idea.

Its not only dev job to call out technical constraints. In fact test can call out if a particular requirement is too unrealistic or ambitious to be implemented if they understand the technology well.

For ex. I can tell the business upfront that a specific multi select tree control is not present in c# and we need to think of an alternative.

2. Scalability Study (Hardware/Software):

Your app used to work fine and then a new functionality was added and then in the next release all of a sudden the app started hanging or the batch job is taking infinite time. If you have had experience something similar to this, you are a victim of “hardware/software constraints” which was not thought in advance.

Lets say the users have increased to million from thousand and you hardware RAM and Processor is not enough to handle so much traffic

Your RDBMS cant handle more than 10000 concurrent connections.

So it is must for you to understand your platform (hardware + software) and their boundary limits so that you can avoid surprised and downtimes in production leading to SLA misses and customer dissatisfaction.

3. Design consideration

When is the last time you were in a design meeting where you contributed to the translation of the requirements into design or more importantly, when you helped avoiding developers to commit a design blunder where let’s say they misunderstood the requirement and the flow was incorrect.

Unless you understand the development models and don't participate in high level and low level designs meetings you wont understand what’s happening “behind-the-scenes”.

It can also help you to write better test cases by understanding the flow of the system and how modules are integrated. Black box testing is not sufficient always when the architecture & the flow becomes complex as not all implementation details are captured in the functional specifications.

Technical design documents should also be evaluated by the testing team where say why Approach A is preferred over other options considered like Approach B or C.

4. Code walkthrough & Inspection

This might give nightmares to many test professions. They just hate looking at the code but believe me you can find lot of bugs by inspecting the code even before the code is dropped to testing. This static testing can save lot of time by avoiding a buggy code to be dropped to test.

You must understand the syntax and semantics of the language being used (C#, java etc) and ask the right questions to find out any memory leak or incorrect loop/condition clause being used in the code

5. Unit Testing & Integration Testing

Okay I wont start a debate by saying test should do unit testing but you will surely come across a situation in your long testing career when you will agree that knowing unit testing help a big deal.

For example, how do you do code coverage now? Unless you know all the inputs/outputs, all the main flows and branches, all the conditions and exception handling code, we cant write good scenarios to exercise critical code paths.

6. Performance tuning

With more and more focus now given on customer satisfaction, performance testing is something now which goes without saying and every customer expects it by default. It is extremely critical now for test team to ensure that the performance is well within customer’s expectations and with high traffic OR volume of data the application would respond within the SLA agreed at the time of signing-off requirements (as part of acceptance criteria).

Unless test teams are familiar with performance counters of the operating system and the the software / technology being used, bottlenecks and potential issues cant be determined.

Also when you want to simulate real-world scenario say to simulate 100000 concurrent users, the knowledge of session management and other technical concepts are must.

Many a times the queries / code are not optimized and it is also important to understand if the code written is efficient or not and also if there is a better way of writing the code which can improve the product’s performance.

8. Security testing

With the increasing number of virus attacks and the risk involved with losing highly confidential customer data or any possible frauds, tester’s contribution has become very crucial.

For example:

How can you test your application for SQL injection if you aren’t good at SQL?

How would you test your application for exposing any confidential data if you aren’t good at encryption/decryption algorithms or understanding different masking techniques?

How do you ensure which port shouldn’t be open/closed unless you understand the networking concepts well.

These are just some high level examples trust me security testing is much more than that and now constitutes of many modern & advanced techniques like fuzzing etc

9. Deployment & Prod Support

We are the gatekeepers of the application. It goes from our hand to Support folks for UAT & Production deployment. If we don't understand the configurations setting or environment pre-requisites to run our applications, chances are there that business would lose lot of their precious time waiting for your dev team to come and rescue you.

I am sure you don't want to end up there and its critical to understand the installation and configuration setting, troubleshooting tips and common FAQs

For example, there was a permission issue which was holding your user from using the application and you just don't know how to give the appropriate permission. Believe me it can be very embarrassing and frustrating at times.

That’s your lucky chance to build credibility with business and Support folks and help them get-going without wasting any amount of time.

10. Automation

And just because I haven’t kept at top of the list doesn't mean its less important than any of the above activities. Its the most important activity only next to manual testing but lot has been written about it and there is much more understanding and awareness about test automation in the industry now and hence I am keeping this short.

With this I am closing this post and i hope this will provide a quick checklist to the testers to become ideal testers-of-future.

Testing:: Chosen-By-Profession turned Chose-the-Profession.

rajkamal@microsoft.com (Raj) — Thu, 01 Oct 2009 15:52:00 +0000

It was morning of 12th Nov, 2004 when I found myself standing in a classroom and for the first time it was my back facing the black board and a new batch of future software professionals sitting in front of me and most of them were as anxious as me (obviously for different reasons). These young professionals fresh out of top colleges in India were campus hired ELTs (Entry Level Trainees) of Cognizant Technology Solutions.

Since that day and the day when I am typing this I have come across many Software Testers (99.9% of them where chosen by the profession and not otherwise) and the questions I have been asked or I could read it on their face is:

“What am I doing in Software Testing Batch? OR Why am I not present in .Net/Java Batch running in next classroom? Am I not good enough?”
“Don’t I have required technical skills to be in Development? OR Based on my aptitude have I been put into Testing?”
“Do people get lesser raise / bonus in Testing over Development?”
“If I perform well and prove my skills, will they shift me into development?”
“My elder brother/ cousin/parents/uncle etc told me that testing is not a good profession and its just about validating someone’s one. Is that true?”

Anyone who was present in one of those classrooms is reading this blog would know that raising above questions is definitely no sin and in fact I had the same questions myself as i was “chosen-by-the-profession” too. Please make a note here (I said “I was” and not “I am”).

I would have moved into development way back in 2004 but i CHOSE to stay in Software Testing and that was BY-CHOICE. Just to give you a brief background (no boasting here), I used to and still think that i am amazingly good in development (and yes i did all graduation lab & major projects on my own –be it C, C++,Computer Graphics Linux, Oracle, Java and got 3 scholarships for topping the college)

Times have changed since 2003-04 and now testing has turned out to be a lucrative and easy career option for candidates who find it difficult to crack into development but in next few paragraphs you will read “How I fell in love with Software Testing” and “Why Software testing is as good as Building software”

Software Testing is fun

If as a kid you enjoyed playing with a new electronic gadget more than building sand castles you will know what am i talking about?

If you had the curiosity and strong urge to see how watch or a transistor works, you will be able to correlate the itching that you get when you want to understand the new unexplored & unseen thing

so guys, testing is like seeing a product and then letting yourself go with the flow and try to understand how it works and where it doesn’t?

Human mind always wants to find out what you CANT do with any object in hand. and let’s say getting paid for it. Isn’t it a bonus?

Software Testing is an ART

Software Testing is art and so is software development as well but think of it that specifications and requirements say “what needs to be done” and design done by the architects & modellers says “how is it to be done” so development is left with very LITTLE room for innovation. I have known developers whose jobs are so monotonous and life is limited to just convert the plain English into a programming language code.

Now in Software Testing nobody says “HOW IT NEEDS TO BE TESTED?” :)

To an extent test plan and test strategy covers the scope and approaches to be followed but as a tester you can think innovative and challenge the very requirement or a workflow or a specific behaviour. You have absolute power to think of a undocumented alternative path and ultimately discover a MISSING/EXTRA requirement.

Software Testing is Advocacy

You like fighting for a cause ? You like making sure that justice is done?

You are customer’s advocate and most of the time he might not be even knowing it. Its YOU who has to ensure that justice is served and your customer is getting service / experience for the amount paid. Usability / Accessibility / Security / Performance are few non-explicit expectations which often get ignored by Developers, Managers, Business Analyst and that’s something can make a huge difference in a great customer experience.

Software Testing is Progressive

You want to be an entrepreneur? You want to understand your business?

There are lot of customer-facing professionals / business analyst who have been testers in their early career. The obvious reason why they are so successful is that they GET TO SEE A BIGGER PICTURE. They understand business better than developers because:

1. They clarify requirements / bugs directly or indirectly with customers

2. They understand the whole system (multiple modules and their integration)

3. They don’t get distracted by getting too deep in technology

Software Testing is Respectable

You have the key responsibility. You got to to take the call here. GO / NO-GO?

Customer & Management relies on your work to decide if we are ready to go into the wild and compete with the world.

You are the one who drives the Quality. You are the one who ensure processes are adhered to. You are the gate keeper. You are the important one here.

Software Testing is Challenge

If you are the one who don't like life to be boring or predictable, then join Testing. One life is not enough to learn everything in testing. Sometimes reproducing a simple-yet-important bug becomes a nightmare and at times understanding the customer requirement becomes so tough when you are working in an offshore model or when the dev slips and you are the one who has to still deliver quality without compromising on the timelines.

Software Testing is Technical

My favourite. This is exactly opposite to the popular belief that testing don’t need good testing skills. In fact, if you are not technical:

1. You won’t be able to repro many bugs because unless you understand what's-happening-behind-the-scenes you wont understand the problem

2. You wont be able to find the root cause many times. You will just be able to say “there is a problem” but “don’t-know-why”

3. You wont be able to suggest a better solutions / alternative

In the end, software testing is one profession that requires great communication skills, strong technical skills and most importantly a testing mind (passion to learn and understand new things)

Remember, Testing will Test you FOREVER.

Reduced / Selective Test Data Execution to SAVE Time & Effort - Introduction

rajkamal@microsoft.com (Raj) — Mon, 06 Apr 2009 17:37:00 +0000

Context: Data Warehouse / Business Intelligence (ETL Execution)

Background:

If you have ever involved in DW/BI Testing, waiting for ETL execution to complete won't be an alien subject to you. Its not ONLY frustating to discover ETL fails at Step X after running for say 8 long hours but its ALSO a show stopper for you to continue with your functional testing and directly impacts your efficiency and effectiveness.

If there are performance lags (like low configuration h/w, network bandwidth etc), the ETL job is only going to take longer than expected and often times this adds to higher WAIT Time (a.k.a NON Productive time).

Root Cause:

If dev / test team wants to run the ETL with Production data, which can be really huge (in GBs or sometimes even TBs), due to the volume its obviously takes a long time.
Trade Off is REAL vs SIMULATED data. Testing with REAL data comes with a cost.

Is testing team the only victim?

Answer is NO. Even Dev team has to wait for entire ETL to run to see the changes made in the code while doing integration testing.

Why don't we create Smaller test data ALWAYS?

-> Due to the data integration and transformational complexities, the team has to have very deep domain knowledge of the business rules to create test data which satisfy the ETL conditions (often times lack of this results in ETL failure or ZERO o/p after the run)

-> If the complexity is high (consider data coming from multiple disparate sources) then creating data while maintainig logical relationship can turn out to be quite tedious and time consuming.

Why dont we SELECT REDUCED / SUBSET of Test data from Production?

This is the heart of this blog and we are going to emphasize on this in detail. Though the above sounds simple and the obvious solution but it has been observed that teams dont find it very popular time because of following important reasons:

1. Random sampling of data doesnt work. Many times it will result in ORPHAN RECORDS (where you might end up pulling child records without parents records from the source)

2. ETL might break if the keys are not found.

3. ETL might succeed but can result in ZERO output records as none of the business conditions got satisfied for the the sample data.

4. Business / Domain knowledge is required

What do we recommend to select SUBSET test data with success?

1. Select a pattern:

a) Master Data Driven

b) Transaction Data Driven:

2. Identify Business Conditions

3. Select Sampling Method

4. Copy the SELECTED test data to a test db

5. Modify Test Data to cover Boundary Value conditions

6. Run the ETL

What are the advantages?

1. Save Time

2. Save Effort

3. Save Hardware Cost (Less Data, Less Space and computational resources required)

4. Better Test Coverage (High Requirement Coverage)

5. Better Code Coverage

The subsequent post will focus on the above methods in detail....Please share your feedback and suggestions.

SSAS Cube Testing

rajkamal@microsoft.com (Raj) — Fri, 12 Dec 2008 19:20:00 +0000

1.1 BVT / Smoke Testing Scenarios:

1. Ensure only the user with the desired permission is able to connect to the cube

2. Validate Data Source Connection string for cube

E.g. Provider=SQLOLEDB.1;Data Source=<Test ETL Server>.redmond.corp.microsoft.com;Integrated Security=SSPI;Initial Catalog=dbCPRMart

3. Validate your are able to successfully process the cube

4. Validate your are able to browse the cube

5. Configure and validating Analysis Services Query Logging

1.2 Data Validation Scenarios:

Approach 1: Browse the cube using BIDS or SQL Server Management Studio and compare the output against Mart using SQL query

If it’s a new development of a Cube, Browse the Cube, drag and drop measures on the Page and Dimension on the Y-axis. You need to write T-SQL that does similar aggregation that brings similar output.

Eg:

OWC Output:

                                 SALES        PROFIT

AP

      CHINA                 200                 60

      INDIA                   250                 50

US

      CHICAGO              500               65

       WASHINGTON      550               65

Equivalent T-SQL:

SELECT REGION, CITY, SUM(SALES ), SUM(PROFIT) FROM GEOG G JOIN SALES S FACT ON G.ID=S.ID

GROUP BY REGION, CITY ORDER BY 2

Approach 2: MDX query to compare data with SQL Query executed against Mart

2.1 For simpler Cube & Dimensional Models

Write your MDX query and compose SQL query for the same conditions and results should match

2.2 Automatically generate MDX using SQL Profiler & Reporting tools and compare it against SQL

SQL Profiler & Reporting Tools like PPS, ProClarity provide the capability of generating MDX on Cube conditions. Use these MDX against the SQL queries formed by you based on the relational / dimensional model on Mart.

Approach 3: Black box testing using sample test data

Insert / Update / Delete test data in your backend and calculate the outcome value based on the desired functionality / requirement without going to cube and ensuring that your expected output value matches the cube output in the final reports.

Approach 4: AMOMD object Compare

Create Automatic Verification Mechanism between OLAP and SQL objects

http://msdn.microsoft.com/en-us/library/cc281460.aspx

http://msdn.microsoft.com/en-us/library/cc280975.aspx

Figure: shows a high-level view of the AMO object model for OLAP objects

Reference:

http://microsoft.apress.com/feature/74/introduction-to-analysis-management-object-amo-programming

1.3 Cube Design Scenarios:

1. Validate all measures

Open Visual Studio analysis services DB

Browse and Open Cube DB

Double Cube and browse to "Measure" pane

Select measure <measure name>

Go to Properties, and check "source" field

     Expected Result:

Measure source should be set to correct table as source table and correct column as source column in Mart

Ensure measure has all the required fields / columns as present in the mart

2. Validate all dimensions & dimension hierarchies

Open Visual Studio analysis services DB

Browse and Open Cube DB

Double click Cube and open data source views and Open <Dimension name> Dimension under "Dimensions"

   Expected Result:

Make sure Table columns in Mart are mapped to Dimension correctly

Make sure Dimension key is correctly mapped to Dimension key column of the dimension Table in Mart

Make sure all the required columns / fields are present in dimension as present in mart

Ensure Hierarchy is correctly defined

Fiscal Year -> Fiscal Month -> Fiscal Week -> Calendar date

3. Validate all calculated measures

Open Visual Studio analysis services DB

Browse and Open Cube and go to "Calculated" tab

Check expression for "Calculated measure name"

    Expected Result

    The MDX expression defined should be accurate as per your requirements.

     Eg. [Measures].[ChangePoint Total Backlog] + [Measures].[Siebel High Pipe] + [Measures].[Siebel Low Pipe]

4. Validate “Data Source Views” of your cube against your design

1.4 Security Testing Scenarios:

1. Ensure each user belonging to a cube role has appropriate access

2. User with read permission should only have access to browse the cube

Read Definition checkbox should be selected

3. User with Process permission should be able to process the cube as well.

Process Database checkbox should be selected

4. User with Admin permission should be able to browse, process, make changes to the cube as well.

Full control (Administrator) checkbox should be selected

5. Cube roles should be mapped to correct users and group

6. Cube roles should be having restricted access or unlimited access to dimension data based on the design and project needs

1.5 Miscellaneous Scenarios:

1. Backup and Restore:

Take the backup of the cube and try restoring. The functionality should remain working as earlier.

1.6 Performance Scenarios:

1. Optimize Cube Aggregations

Before running your stress tests, you’ll want to ensure that you OLAP design has optimized aggregations. To optimize your aggregations, you’ll need to first turn on SQL Server Analysis Services query logging, then run the cube optimization wizard.

2. Using load simulator

Reduce number of invalid defects -> Improve Test Productivity and Efficiency & Let your Developers be happy

rajkamal@microsoft.com (Raj) — Wed, 12 Nov 2008 08:35:00 +0000

Just the other day we were discussing that how can we (testing team) reduce number of invalid defects. I was in deep thinking that why is it really important to reduce the number of invalid defects.

Isn't it a tester's fundamental job to log every potential defect and let it go through normal defect life cycle and let business and management take call if it is a valid defect or not?
Isn't it right that tester shouldn't assume that it is not a bug and then regret later because of a false assumption made?
Isn't a tester taught to think negative and always be suspicious and uncover what is not seen by someone like dev?

The point here is what is a big fuss if testing team raises 'invalid' defects unknowingly. At least they don't leave anything to assumptions which is far more dangerous. There primary job is to find defects, whether that is a 'valid' defect or not is a secondary question.

One of the tester I am mentoring complained that his testing team had found 108 defects out of which 104 were valid and only 4 were invalid but his management didn't seem to appreciate them for the the number of valid defects found as they were expecting number of invalid defects to be zero.

My take on this:

Yes, it is important to reduce number of invalid defects.

Why?

-> Test Metrics gets screwed (Test Effectiveness or lets say Test Productivity goes down with no. of invalid defects)

Test Effectiveness = No .of Valid defect / (No. of Valid Defects + No of invalid Defects) X 100 %

Example for above: 104 / (104 + 4) X 100 % = ~96.6 %

-> Time lost in tracking and logging invalid bug

When you raise a bug in your reporting tool like Test Director, it has to go through complete bug lifecycle. Say you raised a bug spending effort in recording it and then it turned out to be invalid, you developer rejects it. Finally you have to close it.

-> Management don't like invalid bugs

You bet me that "invalid" bugs doesn't please any manager. It is a human behavior to criticize something that is not right. It sets them off.

-> Developers stop taking you seriously

When they observe that you raise many invalid bugs, they start expecting that every time. They stop paying due attention to valid bugs considering them to be invalid. Quality over Quantity concept.

-> Time lost in triage meeting to discuss invalid bugs

When you log invalid bug, it not only your time getting lost, developers waste their time reading them, then testers and developers waste time arguing on that as it has been officially logged, and most importantly business waste their time in triage meeting to take call.

-> Spoiling terms with development team

Developers are under pressure to reduce number of defect found in their code by the test team. Now if you log it, they go defensive and try their best to prove your bug an invalid bug so that it doesn't spoil their commitments.

Now that we have a problem. Let me propose something which we successfully implemented

Now with this process, every bugs gets verified online by the Development team even before we officially log it. They update the sheet saying that they are okay with so and so bugs and we log only those bugs in the Bug Management tool and hence all "VALID" bugs.

Bugs which they update as "INVALID" or "REJECTED", we update the SHARED SHEET with more information like repro steps etc and they change the status in the sheet accordingly. Now if it was our fault and it was actually INVALID defect, we update the SHEET and close it there itself rather than logging it in bug management tool and going through the entire process.

Now our metrics always say 100 % valid bugs. We don't miss any bugs because we record it anyway in the Shared Sheet and triage it with development team online. Development team feels good as they get a chance to to repro bug and confirm before it is actually logged against them. We don't have to waste time in the triage meetings discussing whether its a bug or not. Now business only take call on functional bugs which are more important to end user.

Butterfly Effect in Software Testing (its happening all the time, did you ever notice)

rajkamal@microsoft.com (Raj) — Wed, 05 Nov 2008 17:17:00 +0000

"Small variation in the initial condition of a dynamic system may produce large variation in the long term behavior of the system"

Example: The phrase refers to the idea that a butterfly's wings might create tiny changes in the atmosphere that may ultimately alter the path of a tornado or delay, accelerate or even prevent the occurrence of a tornado in a certain location.

and for we guys "Small variation in the initial requirement of a dynamic application / product / system may produce large defects in the long term usage of the system"

Example: A tiny misinterpretation or misrepresentation of a customer expectation in the form of requirement can ultimately impact the analysis, design and code, testing in such a way that it might result in a product failure or in worst case business failure sometime

here tiny change in requirement can be compared with butterfly's flapping of wings and because this change was not detected in the requirement stage and it keeps on passing through analysis, design and coding phase and finally gets so much amplified that the impact becomes a huge loss to the business.

One of my friend had been asked in an interview "what is Domino effect in software testing?" He didn't know and hence couldn't answer.

It is again related to Butterfly effect and Chaos theory that a small change anywhere in the system (requirement, analysis, design, code, testing) can cause a similar change nearby and that will another similar change and so on in a linear sequence.

Example: A wrong change in requirement -> similar change in analysis to accommodate that -> similar change in design for that analysis-> similar change in code for that analysis -> similar change in test cases for that requirement and so on.

Interesting fact: The term is best known as a mechanical effect, and is used as an analogy to a falling row of dominoes.

Resurrecting the Prodigal Son – Data Quality (Presented at Test2008)

rajkamal@microsoft.com (Raj) — Sat, 01 Nov 2008 11:16:00 +0000

Conference Trip Report Test 2008 – New Delhi, India

rajkamal@microsoft.com (Raj) — Sat, 01 Nov 2008 11:09:00 +0000

On Oct 15^th and 16^th, I attended the Test 2008 conference in New Delhi, India. Test2008 is the first international test conference being organized by Pure Conferences in India. It had speakers from around 10 countries and the theme of the conference was “agility in testing”. Conference was primary organized by Vipul Koacher who is the head of Indian Testing Board and also the founder of Pure Conferences.

IMO, the key take away from the conference were the two animated panel discussions on “Agile Testing– Support Vs Against” and “Schools of Testing – Good or Bad”. In first panel discussion, the distinguished panel members had mixed reaction to the question “Is the agile development the way to go in future?” . William Rollison (BJ) gave a great example of successful agile testing happening at Microsoft in OfficeLab team

But the key message is that Agile testing is a great thing but it can’t replace other approaches and techniques especially in life and mission critical applications like aviation. When one panel member asked “How many of us would want to travel by an aircraft which is agile tested?”, everybody got the point made.

Another panel discussion on much blogged and talked about various “Schools of Testing” like Context-Driven school of testing, Analytic School, Standard School, Quality School & Agile School, promoted by many leading test practitioners like James Bach and Michael Bolton became quite intense. Famous test practitioner Rex Black and BJ made it very clear that they don’t support any such schools of testing and it is really unfair to create a rift in the testing industry by creating these mutually exclusive buckets and emphasized that the software testers should continue to use whichever good test techniques and approaches are available irrespective of which school they belong to.

Like most of the conferences there were three parallel tracks for audiences to choose from. “Interactive Bug Hunt” by Klaus Olsen was very popular among the attendees as it had 20 minutes hands on to find maximum bugs by forming small teams and ringing bell every time a new bug is discovered. “Building a fuzzing framework for software testers” by Rahul Verma from McAfee was a technical presentation on various security testing techniques and approaches.

There were couple on presentations made by Microsoft India products team on the role of “Virtualization” for agile development (Vinod Malhotra) and “Breaking the dev/test barrier” using Visual Studio Team System 2010 by Tanuj Vohra, Partners Director PM, Visual Studio Team, which were very well taken by the audience. This is the first test conference I have attended where Microsoft’s presence and dominance in software test industry was clearly visible.

I had the opportunity to present a paper with my co-speaker Bhoomika Goyal on “Resurrecting the Prodigal Son – Data Quality” which went quite well. We have received compliments from many attendees for choosing this relatively newer topic at the software test conference. Many participants agreed that Data Quality testing is an industry wide problem with very high impact but it has still remained ignored for a very long time and now the trends (investment made in Data Profiling, Data governance by industry leaders) show that having good data quality is extremely important for making accurate and timely decisions which is the most critical factor for the success of our customers. We presented a case study on DQ Test Automation Framework from our actual learning at Microsoft Business Intelligence COE, India.

Finally, this conference had quite a few sessions on Agile testing which gave me a new insight on how it is practiced industry-wide and how we can leverage that better at work.

Resurrecting the Prodigal Son – Data Quality “Rise from the Ashes: Battle of Data Quality Testing”

rajkamal@microsoft.com (Raj) — Sat, 01 Nov 2008 11:07:00 +0000

Authors:

Raj Kamal
Microsoft India (R&D) Private Limited,
Microsoft Campus,
India
rajkamal@microsoft.com
Bhoomika Goyal
Microsoft India (R&D) Private Limited,
Microsoft Campus,
India
Goyal.Bhoomika@microsoft.com

Abstract:

There is perhaps no other activity that is as much a bane of our existence in decision making as “reconciling” data. Data quality testing is a sea with a few large fish (data integrity, data consistency, data accuracy, redundancy-related issues etc.) and many minnows.

Every firm, if it looks hard enough, can uncover a host of costs and missed opportunities caused by inaccurate or incomplete data. This is where Data Quality Testing pitches in and can be instrumental in helping businesses achieve their true potential by reaping the benefits of the timely availability of high quality of data.

After all, who would like to buy yellow pages which contain outdated contact numbers or fly on an airplane that does not conduct preflight checks?

Our mission is to provide information about the costs that testing teams incur due to the lack of data quality testing versus the benefits of taking the actions we propose in this paper.

We would like to answer some important questions which might already be popping into your mind, such as “What is DQ Testing?” If it exists, “What’s wrong with the current approach?”, “What’s new in this paper?”, “What is there in it for me?” and “Where and how can I apply it?”

The objective is to share the key lessons regarding the importance of DQ Testing and present a step-by-step generic test strategy which will help the testers and test managers adopt and apply it when they go back to work, and enjoy the benefits.

As data quality is a strategic IT initiative which involves a strong commitment and a huge investment from management, including a steering committee, we will keep the scope of this paper restricted to the contribution the test team can make – an initiative to significantly improve the data quality by incorporating DQ in the testing process which will help your organization by detecting possible DQ issues earlier than your customer reports it.

The unique, yet simple, approach suggested here is to have an automated metadata-driven approach to continuously monitor the health of the application by automating DQ checks which will provide the test team and users with a DQ summary report containing DQ test metrics.

This will create a win-win situation for the testing staff and your enterprise where the testing team can earn the well-deserved credit for improving the DQ of the application by using an effective testing approach and helping the users of the application feel confident of the data health while making critical business decisions.

1 Introduction:

Data Quality: Data are of high quality "if they are fit for their intended uses in operations, decision making and planning" (J.M. Juran).

Poor data quality can seriously hinder or damage the efficiency and effectiveness of organizations and businesses. If not identified and corrected early on, defective data can contaminate all downstream systems and information assets. The growing awareness of such repercussions has led to major public initiatives like the “Data Quality Act” in the USA and the “European 2003/98” directive of the European Parliament. Enterprises must present data that meet very stringent data quality levels, especially in the light of recent compliance regulations standards.

2 Fable:

The task at hand is to test a large data warehouse and CRM implementation of a global bank which has a huge volume of transactional data (on the scale of terabytes) scattered among various disparate sources in different formats like flat files, DB2, Oracle, SQL server, MS Access, Excel etc. To make things even more complex, a medium-sized regional bank is acquired and merged which brings in additional gigabytes of data such as customer information, loan information, account details etc.

The test team faces the challenge of ensuring the quality of the data which is as complex as the data for a subject area, since a customer might be stored in more than one source in different formats and the underlying business rules vary among the systems. The integration and transformation of data performed by the development team has a higher probability of missing significant data, duplicating existing data and introducing latencies, inconsistencies and inaccuracies into the system.

The experienced test team manually verifies and validates the integrated data showing up in the reports coming out of data warehouse against the business rules. Few test cases are written to test the sample data flow from sources to data warehouse to reports and the test team certifies the application.

A couple of days after the production release, the customer comes back with a list of DQ issues, for example they can’t find some important data or some of the data is repeated multiple times in the drop-downs and the KPI (key performance indicators) are showing incorrect results. Later it is discovered that there are flaws in the transformation and integration logic and data was missed, duplicated and incorrectly manipulated resulting in many DQ issues.

Finally the customer rejects the application and goes back to manual reporting based on the legacy application and the IT team starts working on fixing and testing the DQ issues which could have been avoided by a focused DQ test approach for testing various dimensions of data qualities.

3 DQ Testing Problems / Sore Points:

DQ Testing has always been given less importance than it deserves, and for more than one reason.

3.1 DQ Requirements are “missed” in the Functional Specifications

Typically, DQ Requirements (DQ Definitions, Scope, Acceptance Criteria, Measurement criteria) are not covered in the Functional Specs and are not explicitly articulated by the business.

Fig. 1: Ishikawa Diagram

3.2 White box Testing / Technical knowledge is required

The testing team is often expected to perform black-box testing or functional testing based alone on the functional specifications given by the business. It was often assumed that the underlying architecture and the design (which also includes processing, transformation as well as flow of data) fall under the development team’s responsibility and have to be tested by developers during unit testing.

3.3 Management overconfidence on quality of their data

It was often the case that management felt that their data couldn’t have inconsistencies or inaccuracies and that activities like data integration can’t impact the overall quality of data.

3.4 Complexity grows with CRM / ERP Implementations & Mergers / Acquisitions

In the last decade, as a result of globalization along with the worldwide CRM and ERP implementations trend and strategic decisions like mergers/acquisitions, data integrations has become a bigger challenge for testers due to the exponential increase in the complexity of data.

Testing Data warehousing, CRM and e-business projects often involves testing poor quality data because they require companies to extract and integrate data from multiple operational systems. Data that is sufficient to run individual operations is often riddled with errors, missing values and integrity problems that do not show up until someone tries to summarize or aggregate the data.

3.5 Dynamic nature of the data which quickly becomes obsolete over time

Experts say that 2% of the records in a customer file become obsolete in a month because customers die or divorce, marry and move. In addition, data-entry errors, system migrations, system integrations and changes to source systems, among other things, generate numerous errors such as inconsistencies, inaccuracies, latencies etc.

4 DQ Adoption Guide

If your project falls under any of the categories shown in the diagram below then you enter the realm of DQ Testing. Basically whenever data is created, modified, integrated or processed from one form to another, there is a probability of introducing DQ issues and hence DQ testing is a must.

Fig. 2: Project Categories

5 DQ Test Strategy:

Data Quality Testing is the process of verifying the reliability and effectiveness of data. Maintaining data quality requires monitoring the data periodically and reporting it as early as possible. DQ Test Strategy should be prepared during the requirement analysis phase as it is important to define and measure DQ requirements which are expected by the customer.

DQ Testing can be done in a “Preventive” as well as “Reactive” manner. Focusing on DQ from the envisioning phase ensures that DQ issues are prevented by means of verification/static testing. As explained in Section 5 below, a Reactive approach has been developed which can identify the DQ issues that have crept into the system by monitoring the DQ health on a continuous basis.

Fig. 3: DQ Strategy

5.1 Test Prioritization (Kano Analysis):

This approach suggests prioritizing the performance of DQ checks depending on their importance to the customer which results in increased performance and satisfaction.

Fig. 4: Kano Analysis

As shown in the Kano analysis in the figure above, the DQ Dimensions/Audits are classified in terms of the following categories:

Must be’s (Expected Quality):

The highest test priorities should be given to the typical DQ checks shown in the figure above as these are the basic explicit customer requirements

One Dimensional (Desired Quality):

The next important requirements are the DQ checks like Timeliness, Availability etc. as these are most often the promised requirements.

Delighters (Excited Quality):

DQ checks which are not requested by the customer but which can increase the satisfaction by meeting these implicit requirements.

5.2 DQ Testing Approach

The involvement of the testing team starts right from the requirements phase and continues throughout the life span of the project. The test team has to ensure that DQ is appropriately defined and the measurement criteria are included in the requirement document along with the threshold and acceptance criteria defined by the business for each DQ check.

Fig. 5: DQ LifeCycle

DQ Test Planning:

· Define Data Quality test approach & the scope of testing by taking inputs from the business.

· Decide on the test metrics to monitor & measure DQ.

DQ Test Design:

· Create DQ Test Scenarios with the expected test results (refer to Section 6 below).

· Have it reviewed by the business and against the threshold values for each DQ check.

· Prepare DQ Metadata from Data dictionary, Business Rules from FS and Input from Business.

DQ Test Execution & Reporting: (refer to Figure under Section 5)

· Automate DQ test scenarios for checks which can be scheduled to run for the DQ metadata over a period of time.

· Log the DQ test scenarios result (manual/automation) and report DQ discrepancies to the users.

DQ Test Monitoring:

· Run DQ Test scenarios on a scheduled basis to continuously monitor the health of the applications.

· Send the automated reports/notifications to the users with the discrepancy summary of the DQ Test Metrics (refer to Section 7) which are violating the defined threshold during planning.

6 DQ Test Implementation / Solution:

To ensure information quality, the test team has to address it from the very beginning. The test team is required to define data quality in a broad sense by taking inputs from the business, establish metrics to monitor and measure it, and determine what should be done if the data fails to meet these metrics. Data quality issues need to be detected as early in the process as possible and dealt with as defined in the business requirements.

DQ Inputs:

· Metadata: contains list of server name, database name, table names, and column name against which DQ checks need to be performed.

· Business Rules: contain the functional logic and formulas which are used to process and transform the data.

· Thresholds: the values which define the valid range or accepted deviation for various DQ Metrics.

DQ Engine:

· Automated Scripts for DQ Checks: These can be your SQL code to validate consistency, accuracy, validity, row count etc. against the metadata which is keyed as input.

· Discrepancy/Results Tables: Automation scripts log the discrepancies in these tables by comparing the output against the discrepancies based on the threshold defined.

· Scheduling for DQ Checks to run: Automation scripts are run in an “unattended” mode on a continuous basis to track the DQ health of the application over a period of time.

DQ Output:

· DQ Discrepancy Summary Reports with Metrics: Based on the template defined, DQ metrics are calculated from the discrepancies logged by the DQ Engine and reported to customers and management in the form of DQ Notifications/e-mails.

· DQ Issues/Bugs: DQ issues like application bugs are triaged with the development team and the customers and pass through the typical defect lifecycle.

DQ Feedback & Continuous Process Improvement:-

· DQ Automatic Feedback & Continuous Improvement: The IT team along with the business decides the corrective actions and the priority.

Some of the corrective actions can be:

o Fixing the code to rectify the DQ issue

o Change in requirement or functional spec if it was a functional DQ issue

o Change in design, architecture if it was an environment or performance related issue

Following are the important “DQ Components” which can be automated by the test team to make the DQ test monitoring a continuous process:

Fig. 6: DQ Implementation

7 DQ Test Scenarios:

Fig. 7: DQ Checks

7.1 Row Counts:

Count of records at Source and Target should be the same at a given point of time.

Fig. 8: Row Count Example

7.2 Completeness:

All the data under consideration at the Source and Target should be the same at a given point of time satisfying the business rules.

Fig 9: Completeness Check

7.3 Consistency:

This ensures that each user observes a consistent view of the data, including changes made by transactions. There is data inconsistency between the Source & Target if the same data is stored in different formats or contains different values in different places.

Fig 10: Consistency Check Example

There is a need for tools that can handle international data differences, that require such features as support for Unicode and rules engines that can deliver local address validation and other functions across multiple languages and formats.

Fig 11: Consistency Check Example #2

7.4 Validity:

Validity refers to the correctness and reasonableness of data. A valid measure must be reliable, but a reliable measure need not be valid.

Questions:

-> Is the information reliable?

-> How is the information measured?

Fact: A large bank discovered that 62% of its home-equity loans were being calculated incorrectly, with the principal getting larger each month.

Fig 12: Validity Check

7.5 Redundancy/Duplicates Detection:

Consider what happens when a single customer is included in a company's database multiple times, each time with a different value for the customer identifier. In such a case, your company would be unable to determine the true volume of this customer's purchase. You could even be placed in the embarrassing situation of attempting to sell the customer an item that the customer has already purchased from you.

Physical Duplicates: All the column values repeating for at least 2 records in a table.

Logical Duplicates: Business Key (list of column) values are repeating for at least 2 records in a table.

Fig 13: Duplicates Check

7.6 Referential Integrity:

If there are child records for which there are no corresponding parent records then they are called “Orphan Records”. Logical relationship rules between parent & child tables should be defined by the business.

Fig 14: Referential Integrity Check

7.7 Domain Integrity:

Fig 15: Domain Integrity Check

Fig 16: Domain Integrity Check

7.8 Accuracy:

Fig 17: Accuracy Check

7.9 Usability:

Fig 18: Usability Check

7.10 Timeliness

Fig 19: Timeliness Check

8 Key DQ Test Metrics:

Data Quality metrics, which helps in quantifying the quality of data over different dimensions, can be derived on a scheduled basis from the DQ Test Scenarios execution or result logs for various DQ checks manually/automatically by having a DQ test implementation in place (similar to the one proposed above)

Fig 20: DQ Test Metrics

These DQ metrics are critical so that management may determine the DQ health of the integrated system. Depending on the project needs and the criticality, the frequency of calculating and reporting these metrics can be determined.

9 Conclusion:

It is critical to measure the effectiveness of the test strategy as it is the key to “continuous process improvement”. We will talk about this in detail in our next white paper. Stay tuned.

Today we have introduced many DQ test problems and recommended a test strategy to tackle them. We can assure you that the complexity and the scope of data quality testing in large enterprises are difficult for a layman to understand. Most test teams do not prepare or adopt a dedicated test strategy to build quality into their data in a proactive, systematic and sustained manner. Many potential DQ issues can be avoided by implementing the proposed DQ Test strategy as part of the overall test process. It is often not until you discover a major problem in production that could have been avoided through quality control of your data that you recognize the importance of data quality. As a consequence, the business may lose revenue, opportunities, even customers, and in the worst case can even get sued.

Don’t end up like them. Be different. Always remember to put quality first.

9. References:

[1] The Kano Analysis: Customer Needs Are Ever Changing by J. DeLayne Stroud

http://finance.isixsigma.com/library/content/c071017a.asp

[2] Be Prepared to Duel with Data Quality – Rick Sherman

http://www.athena-solutions.com/bi-brief/2006/jan06-issue24.html

[3] Trends in Data Quality – Lou Agosta

http://www.dmreview.com/issues/20050201/1018111-1.html

[4] Gartner ranks data quality management software, reveals trends - Hannah Smalltree

http://searchdatamanagement.techtarget.com/news/article/0,289142,sid91_gci1263861,00.html

[5] Data Warehousing Special Report: Data quality and the bottom line -Wayne W. Eckerson

http://www.adtmag.com/article.aspx?id=6321&page=

[6] The Importance of Quality Control: How Good Is Your Data? - Andrew Greenyer

http://www.customerthink.com/article/importance_quality_control_how_good_data

[7] Garbage In, Garbage out: The Importance of quality ideas – Jennifer Hanson

http://www.fastcompany.com/blog-post/garbage-garbage-out-importance-quality-ideas

10. Authors:

Raj is a Test consultant specializing in different types of testing techniques, test automation and testability in different domains like Manufacturing, Healthcare and Higher Education. He holds an APICS certification in Supply Chain Management. Expertise with Rational and Mercury testing tools, he has helped teams develop test automation strategies and architectures for such companies as Cognizant Technology Solutions and Oracle Corporation. He also provides training in automated testing architectures and design. He is QAI (CSTE) & ISTQB Certified. He has a master's degree in Computer Applications. He is currently working at Microsoft, India, Business Intelligence COE. He has earlier represented Microsoft and Oracle at International test conferences as a Speaker.

His Blog (http://www.itest.co.nr/ )

Bhoomika has been working with Microsoft for more than a year now as a part of the APEX program. She has completed her first two rotations in Development and Test discipline. Has done her B.E. – Information Technology from Vivekanand Education Society’s Institute of Technology, Chembur, Mumbai. She has mainly worked in technologies like C, C++, Visual Basic, Java, SQL Server, ASP.NET, C# and SharePoint. Her hobbies include Playing Chess, Solving Puzzles and Reading.

Resurrecting the Prodigal Son – Data Quality (Test 2008 International Conference)

rajkamal@microsoft.com (Raj) — Sat, 01 Nov 2008 09:58:00 +0000

http://www.slideshare.net/raj.kamal13/test2008-resurrecting-the-prodigal-son-data-quality-share-presentation

This is Bugs Story

rajkamal@microsoft.com (Raj) — Sat, 23 Feb 2008 17:44:00 +0000

| View | Upload your own

Million Dollar Question - 20 % Brand new test cases ?

rajkamal@microsoft.com (Raj) — Sat, 23 Feb 2008 11:55:00 +0000

When is the last time you wrote a test case or a scenario which you had never written before?

When is the last time you came across a scenario you had never thought in your testing career?

If % of above scenarios (for a given project) is not more than 20 % then:

You are not adding value to the customer by rewriting the old stuff.
You are not realizing your true potential.
You are reinventing the wheel

While doing test case creation, 80 % of times we write fewer than 20 % brand new test cases or scenarios (not trying to fit in pareto just because it looks cool)

I know by now you must have understood where am i coming from and you have would have possibly concluded that it is something not new what I am going to tell you.

But if I don't tell what I am going to tell you then there is no ways that you can find out what am I going to tell you:) so read on.

I promise you that two words I am not going to use in this post are "reuse" and "library", doesn't how much I will be successful because will be tempted to use them.

Reason: Reuse and Library are two most abused words not just in testing industry but IT as a whole.

Disclaimer: Author is not saying that he doesn't believe in reuse and library, its it just that we want to look at the true meaning of the words without using them.

Technique:

Next time you get an assignment of writing test cases, ensure following things:

First Step: TC Creation

1. Your test case / scenario contains two different logical sections - Generic Vs Application Specific.

2. Generic section should be written in plain English considering what that test case means to someone in real world generic step of execution and expected outcome for those

3. Application Specific section should contain details specific about your application only like exact steps, exact expected result, test data

4. Add a field specifying the "Area" - Web Application (.Net/ Java etc) / Desktop application / Database Application / ERP etc

5. Go through the complete cycle and keep on refining you Generic and Application specific areas.

Effort: (120 %: 100 % (Application specific sections) + 20 % (Generic sections) )

Second Step: Central Repository formulation

1. Pick only logical generic sections from the above set and classify them with different areas.

2. Spread this to other fellow testers and team.

3. Each times someone add any test case / scenario to this repository then quickly look for duplicates:

a) If duplicate, then merge both the test cases to have an enhanced test case.

b) If not duplicate, then make an entry into the repository.

Additional Effort: (20 % - Identifying duplicates and upgrade of repository)

Third Step: Think Hard - Test Case Creation

1. Now for this project (similar to above) ensure the following:

a) Take the generic test cases from the repository based on the "Area" filter.

b) Work on expanding the generic test cases

c) Now ensure that you think of at least 20 % new scenarios which are not present in the repository.

d) Go through the cycle and keep refining step b and c

Effort: ( Iteration 1: 80 %, Iteration 2: 60 %, Iteration 3: 40 %)

Fourth and Final Step: Paying back to the community

1. From Third Step, 1 (b), update the central repository to make the test cases more generic

2. From Third Step, 1 (c), add the test cases to the repository.

Additional Effort: (20 %, identifying duplicate and upgrade of repository)

Saturation Point:

After following Step 2,3 and 4 continuously over a period of time, you will reach a state of high TCM (Test Case Maturity) when it will start becoming identifying new scenarios and the effort will continue to go down and you get ROI of the work.

One man initiative:

It can be started by one man but can't be accomplished without strong management support and contribution from the community.

Million Dollar Question: Is it always possible to write 20 % brand new scenario's for every project which were not covered in past for a similar project?

Answer:

Scenario 1: If you haven't followed the above practice then yes have a very high chances of writing more than 20 % brand new scenarios very easily for a similar project.

Scenario 2:If you have followed the practices described above by author over a period of time (>3 Iterations) then the possibility of finding another set of 20 % brand new scenarios will become difficult and difficult with every iteration. And one day,you even after applying all your imagination / thinking and even after covering all the requirements, you wont be able to find 20 % more brand new scenarios and that is when you have reached saturation point.

Note: The more the difficultly in identifying brand new scenarios, the more (higher) is your requirement coverage and test case maturity.

New Testing term coined: Test Case Maturity

You must have heard of CMM levels which define the maturity of an organization's process. TCM is a term which defines the maturity of your test case.

Each time you write a test case , it has certain maturity associated with it which can be calculated by using the

1. Is the test case covering a most obvious requirement? - Level 0

2. Is test covering the very high level requirement ? - Level 1

3. Is the test case was already written by past and can be used as it is ? Level 2

4. Is it test case written in past can be re-adopted and can be customized for your project? Level 3

5. Is the test case can re-adopted and extended and made generic for other projects too ? Level 4

6. Is your test was a not present in repository and can be added to the central repository for other to adopt ? Level 5

7. Did your test case cover any unique aspect which can impact the existing test cases in central repository ? Level 6

Benefits:

1. Value addition to customer

2. Lesser effort required, lesser cost (> 3 Iterations, not overnight :) )

3. Challenging task for testers, keep them 'thinking'

4. Contributing to the community as a whole

5. Able to get time to think of scenarios and find bugs which never got time for.

6. Standardization of test case across community.

7. Easy for new comers, end users to execute and perform testing.

8. More accurate effort estimation.

Ending note:

Saturation point doesn't mean "stop there", but it means that you got to find our other ways to take this to the new level.

Example: You can dissolve 2 teaspoon of sugar in a cup of water in a matter of 20 secs,

another teaspoon of sugar, may be 30 secs, one more teaspoon , its becoming difficult.

just one more tea spoon, not it doesn't work . Is that is what we call a saturation point?

Thinking: The saturation point is different at different temperatures. The higher the temperature, the more sugar that can be held in solution.

So that's not the end, you can heat the sugar solution and it can hold more amount of sugar so keep trying.

But infinite amount of sugar, may be no.

BTW, did you do a CTRL + F to verify that if I did use the term "reuse" and "library" again :)

Please feel free to write your comments !!!! Contact: raj.kamal13@gmail.com

Rational Requisite Pro - Requirement Management (Advanced Training)

rajkamal@microsoft.com (Raj) — Sun, 17 Feb 2008 02:48:00 +0000

| View | Upload your own

Test Automation Change Control Process - Sounds Interesting?

rajkamal@microsoft.com (Raj) — Sun, 17 Feb 2008 02:00:00 +0000

This change control process is to ensure that in a "big" automation project whenre "Modular" Library architecture is followed, libraries are not mistakenly / intentionally changed by the automation testers and every change in the library goes through a well defined "Automation Workflow" depending upon the role and the access rights.

Changing libraries without this process can even cause the other automation scirpts to fail or having a possibility of redundant code being written for same functionality which is often the reason for automation failure.

Thought of depicting this process graphically so that everyone here can understand this process.

Library Change Control Process

Performance Testing - VU Scripting using IBM Rational Robot & Test Manager

rajkamal@microsoft.com (Raj) — Sun, 17 Feb 2008 01:19:00 +0000

| View | Upload your own

Rational Robot - Training

rajkamal@microsoft.com (Raj) — Sun, 17 Feb 2008 00:43:00 +0000

| View | Upload your own

Priortizing Test Activities - A Presentation

rajkamal@microsoft.com (Raj) — Sun, 17 Feb 2008 00:22:00 +0000

| View | Upload your own

Advanced Rational Robot - A Tribute

rajkamal@microsoft.com (Raj) — Sat, 16 Feb 2008 23:33:00 +0000

| View | Upload your own

Business Values of Testing (Published in TestQuest Magazine)

rajkamal@microsoft.com (Raj) — Mon, 01 Oct 2007 10:34:00 +0000

PONC - Those who speak of progress measure it by “Quantity” and not by “Quality”

rajkamal@microsoft.com (Raj) — Sat, 29 Sep 2007 21:13:00 +0000

Test Manager A: Already 3 days of testing is over and you have found just 8 bugs. I am disappointed with the way …….

Tester B: but if you look at the bugs you will find…

Test Manager A: Don't make excuses…when I was a tester, I found "101" bugs in just 2 days. Take example of Tester B007, he has already found 22 bugs and…

Tester B: I understand you point and let me explain you that I have found 5 Severity one bugs and they are really critical.

Test Manager A: That's fine but I want you to file at least 10 more bugs before you leave today.

Tester B: Okay L

Rings any bells? Sounds familiar? You are very lucky if you are not tester and it didn't happen to you ever J

Okay folks, I am not telling you a story, this happened to one of my friend in a CMM Level 5 company.

Why Quantity kills Quality? L

There is an article on "Quantity, Quality, and the Function of Knowledge "by Hartley B. Alexander

http://links.jstor.org/sici?sici=0160-9335%2819050817%292%3A17%3C459%3AQQATFO%3E2.0.CO%3B2-Y&size=LARGE&origin=JSTOR-enlargePage

I am presenting an interesting excerpt from this wonderful piece of work on philosophy and human psychology to prove my point:

'The human mind knows only numbers adequately.', 'The mind only grasp fully and clearly which is reducible to the term of ratio', 'we reduce our impressions to mathematical formulas, and simply by means of this reduction of a confused state to a clear idea, we manage adequately to grasp the real'

What do we understand by Quantity?

Quantity is a kind of property which exists as magnitude or multitude. Quantity was first introduced as quantum, an entity having quantity. Being a fundamental term, quantity is used to refer to any type of quantitative properties or attributes of things. In our simple industry terms "Quality is conformance to requirements."

Where to Apply?

Defects / Bugs: Finding Quality bugs / defects are more important than finding lots of not that severe bug.
Test Cases: Writing few quality test cases is more effective than writing 10001 test cases? In the end a good test case is one which has a high probability of find defects.
Automated Test Scripts: Automating few test cases which can give you high ROI are better to automate rather than trying to automate 100 % test cases and then ending up doing lot of rework.

Some on the common ways to measure / improve quality are:

KAIZEN (Contribution Process Improvement)
Quality Management System
Total Quality Management
Six Sigma
Zero Defect
SPC (Statistical Process Control)

Suggestions:

Severity: When you prepare test plans, define "Severity" definition very clearly.
Prioritization: Attack high risk areas first and something which is too critical for customer / end user.
Unit Testing: Encourage your dev team to perform a good unit testing so that you can find quality bugs in Integration and System testing phase.
Impact: For each bug logged state "Business Impact" of that bug if not been fixed.
Worst Case Scenario: Always identify and mention the "WORST CASE SCENARIO" of the bug.
Quality of the Build: If the bug is Sev 1 and that is critical enough and might be possible cause of many other bugs then immediately stop testing and reject the build. Business users don't have time to analyze your complex bug matrix.
Reports with GRAPHS: Give daily reports / weekly report / build summary / test coverage reports with "Graphs" with bug classification as Sev 1, Sev 2 and Sev 3.
Triage: Always include the stakeholders (recommended – Customer, Business User) in the BUG TRIAGE meetings and let them know that you have found something critical.
Talking with Customer always helps: If you feel testing estimation are too aggressive then always let the customer know that if test team is given more time then you can discover many more quality bugs and that way over a period of time you will have "many quality bugs and that will increase the bugs count too J"

My viewpoint: "Quality is reconcilable with Quantity" J There is tradeoff that exists between quality and quantity.

Final Verdict: Left to readers J

Lesson Learnt: I should try to write "Quality" blogs over "Quantity" L

If you are among the "vital few" who are still thinking what is PONC ...then it's "PRICE OF NON CONFORMANCE" and that signifies the importance of quality over just quantity.

Disclaimer: There is absolutely nothing wrong with assessing progress by looking at the no. of bugs but I would rather give more importance to **severity** of those bugs.

The ‘Omerta’ of Testers (Mafia)

rajkamal@microsoft.com (Raj) — Sun, 23 Sep 2007 11:44:00 +0000

Show me one tester with a considerable exposure to IT industry, who never engaged himself into “The Blame Game” intentionally or unintentionally with Developers or Analyst.

“Tick, Tock, and Testimony”
Each test team and opponents are asked for justification or tell their side of the story. Old mails are sought out for evidences to prove their point until both were out of time.

“You did It, Now Admit it!”
Now team held responsible will be penalized for committing that horrendous crime

“Jury of your management”
Management would decide to show mercy or no mercy and the convicted to be added to “Do Not Trust This Blame Game Loser”

Let’s not dig deep into what causes quality failures or defects in the production, we are trying to understand lot of psychological problems and people issues that often lead to slinging mud at each other.

After a detailed discussion testing teams from different leading organizations, following are identified as main culprits:

1. DREAD:
a) Fear of missing a bug.
b) An anxious feeling of not able to find all the bugs.
c) A lack of confidence due to nature of our job where exhaustive testing is not possible in the time given.
d) Scared of the no. of possible permutation and combinations that exist.
e) Panicked by the scarcity of time or resources.

2. BALL IS NOT IN MY COURT:
Most of the time testers think only when the build comes to them and during the execution phase they are “Accountable” for the quality. That means they can relax and chill out at the time of requirement, design, coding and deployment. Testing and Production is the only phase they can be held responsible.

3. ACCOUNTABILITY:
If anything goes wrong, testers are usually choked by “The Family” (Dev, Analyst, and Management). It’s not that testers are not accountable for quality but the common misconception is that they are the ONLY discipline responsible for QUALITY against the principle “Quality is the responsibility of each employee in the team”

4. PRESSURE
Even Indian cricket team dies under pressure then think of a testing team shouldering the responsibility of a product that means both business and satisfaction for the end users.

Sometime imperativeness takes a toll on the testing efforts and affects the quality of the effort being put by the tester.

Continuously being bugged by managers often restrict the tester from thinking “out-of-the-box” scenarios and trying out some end user events by putting themselves into end user’s shoes.

5. QUANTITY OVER QUALITY
I don’t know how many times you would have seen that management want to know the no. of bugs to decide upon the performance of the testing team. Priority / Severity take a back seat and Dev / Management shrugs their shoulders to say “Is that all you found?”

Ultimately it boils down to quantity over quality in performance appraisal of testers.

The ‘Omerta of Testers’: STOP the blame game!!!

“Whenever a tester appeals to the management against his fellow developer or analyst is either a fool or a coward. A tester who cannot take care of himself without management protection is both. It is as cowardly to blame an offender to justice, even though his offences are against yourself”

Remember Test and Dev are like wheels of a two-wheeler vehicle and only way to move forward is to make a clear distinction of each other’s responsibility and understanding of most important fact that we are working here as a team and for end user what matters the most is a high quality product at a low cost. The Industry, Client, End users don’t care if it is the dev fault or a miss from testing team.

Application And Script Independent Framework The Need For Data Normalization

rajkamal@microsoft.com (Raj) — Fri, 07 Sep 2007 12:32:00 +0000

From: guest0efb5e, 24 minutes ago

Why do you think you are a good testing professional?

rajkamal@microsoft.com (Raj) — Wed, 05 Sep 2007 07:46:00 +0000

I believe that we (testers) are end user's advocate and are often looked at like devils by our devs folks across the globe :). My aim in life is not to find a million bugs or to show that every day I come up with a new or an advanced automation framework. I believe we are here to prevent bugs from happening by constantly focusing on continuous process improvements, defect prevention techniques and ensuring only one thing that our customer’s expectations are not only met but exceeded by shipping high quality products keeping the cost of quality at a minimum.

Mapping Rational Unified Process to Rational Testing Tools

rajkamal@microsoft.com (Raj) — Mon, 03 Sep 2007 12:57:00 +0000

Abstract

Rational Unified Process raises everyone’s hope yet often confuses testers to follow it. IBM Rational Tools are used extensively but there is no documentation available, stating when and how to use Rational Testing tools in different RUP phases i.e. Inception, Elaboration, Construction and Transition. It frustrates and disappoints testers. Although Rational Unified Process provides “Tool Mentors” to explain the usage of all the tools but still there is no direct mapping between usage of the Rational Testing Tools and Testing Life Cycle which explain the responsibility of the Testing team through out the SDLC in different Iterations.. This paper presents the guidelines which will suggest the role and responsibility of testing team in different phases of Rational Unified Process. Follow these guidelines as you staff, tool, or schedule your test automation project, and you will be well on your way to success.

A Fable

I have worked in many testing projects, big and small. I have seen different projects following RUP model and I have personally talked to many testing teams. I am presenting this paper to avoid these problems faced by them. But first we need to understand it. Let me illustrate with a fable.

Once upon a time, we have a big project following RUP for a very important client. Client provides the team with Rational Testing Tools. Sujata is a Test Manager; she pulls the experienced testers. Everyone is very excited as they are going to work with best tools in the market and no more excel sheet to write test cases, track defects and to generate report.

Testers have been trained on Rational Unified Process. Inception Phase, project starts and lot of testing tools available like ClearQuest, RequisitePro, Robot and Test Manager. Testers know how to use tools, thanks to Sujata once again for sending them for Rational Tools Training before putting them into project. They know Rational Unified process too but they don’t know how to use different tools in different phase (Inception, Elaboration, Construction and Transition).

They use RequisitePro for requirement understanding, test manager for writing test cases but still not sure when to use them and in which phase. Elaboration phase comes followed by construction phase and finally transition phase and Sujata realizes that nothing is in place. All the requirements are not documented, changes are not documented, test cases mapped to requirement have become obsolete, automation was pushed too late, and version was not maintained after a certain period.

Sujata couldn’t find the problem, she provided the best training to the best testers and they used the best tools in the market but still they couldn’t manage the project. She quits and hand over to Ashish, new test managers. Project is a failure.

That's my fable. Perhaps parts of the story sound familiar to you. But I hope you haven't seen a similar ending. This paper will suggest some ways to avoid the same fate

The Problem

This fable illustrates several problems that plague testing project following RUP:

What to use when?
This is the major problem which hinders the success of the projects and many times result in total disaster. In the worst case, testers don’t know RUP, usage of automation tools which stops project from doing well. If we consider the best case as mentioned by me in the previous section that even if testers are trained in RUP and usage of testing tools like Robot & TestManager still project might not succeed because they should know when to use Robot, when to use Test Manager and when to use other testing tools like ClearQuest and ClearCase in different phases in RUP.

How to use?
Other big question is the functionalities and features of the tools those should be given more focus in different phases of RUP.

Who should use?
This question can be answered if the tester knows what he is supposed to do and what manager is responsible of doing.

Guidelines- usage of Rational Testing Tools in a RUP Project

This paper will be organized by the normal steps that we all use for our testing projects, making special notes of the considerations and challenges that are particular to RUP.

1. IBM Rational Administrator
2. IBM Rational ClearCase
3. IBM Rational RequisitePro
4. IBM Rational TestManager
5. IBM Rational Robot
6. IBM Rational ClearQuest

All the above mentioned tools are explained with respect to their usage in different phases of RUP. I haven’t covered Rational Rose as I wanted to keep the focus restricted to the tools which are used in software testing.

I am giving a brief introduction about RUP phases for the audiences who are new to RUP.

Inception – Focus of this phase is the understanding the scope of the project.

Elaboration: The architecture as wells as the requirement of the products being built must be understood by the end of the phase.

Construction & Integration: Product must be constructed in this phase.

Transition –The product must be rolled out to customers in this phase.

Step 1: IBM Rational Administrator

a) Phase: Inception
During this phase, a shared repository is created using either MS-Access or SQL Sybase Central. A new project folder is created in the share with UNC (Unified Naming Convention). The project should be UCM (Unified Change Management) Enabled to ensure integrity with Rational ClearCase for Configuration Management. Rational Administrator provided the platform to integrate all the rational tools together to achieve end-to-end mapping.

Activities:
Create a project
Connect to a project
Create a test datastore
Create integration between products
Decide whether to associate the administrator project with UCM project.

Fig.1. Mapping of RUP phases to Rational Testing Tools

b) Phase: Elaboration

During Elaboration phase, Rational Administrator is used to create new users with different access rights depending upon the requirement of the project

c) Phase: Construction & Integration

Like Construction phase, Rational Administrator is used for managing users and access rights. It also maintains the integrity between different tools used.

d) Phase: Transition

Like Elaboration & Construction phase, Rational Administrator is used for managing users and access rights. It also maintains the integrity between different tools used.

Step 2: IBM Rational ClearCase

a) Phase: Inception
ClearCase provides life cycle management and control of software development assets. With integrated version control, automated workspace management, parallel development support, baseline management, and build and release management, Rational ClearCase provides the capabilities needed to create, update, build, deliver, reuse and maintain business-critical assets.

Fig.2.Rational Clear Case Workflow

Activities:
Create the configuration management repositories that represent the subsystem defined by your architecture.
Import existing files and directories into the repositories to create an initial set of development configuration items.
Enable your ClearQuest schema to work with UCM
Enable your clear case project to work with ClearQuest.
Set Policies for new projects

b) Phase: Elaboration

During this phase too new assets created are base lined and updated. When new test cases are created using Test Manager and new scripts are created using Robot are base lined using UCM concept.

Activities:
Updating your project work area using Rational ClearCase.
Checking in and checking out configuration items.
Delivering your work with ClearCase
Promoting project baselines using Rational ClearCase
Comparing baselines using Rational ClearCase.

Fig.3.Rational Clear Case detailed diagram

c) Phase: Construction & Integration

Out-of-the-box integration of Rational Test Manager, RequisitePro and Rational ClearQuest with Rational ClearCase LT delivers full base lining capabilities to accommodate the natural evolution of a project, moving from one build or release to the next.

d) Phase: Transition

In transition phase, all the artifacts are delivered and existing one are moved to baseline to maintain version control.

Step 3: IBM Rational RequisitePro

a) Phase: Inception

During this phase RequisitePro is mainly used for Managing Requirements. Inception phase focuses mainly on managing the requirements collected.

“Manage Requirement” is one of the best practices in RUP. Research from Standish Report shows 31 % of projects gets failed because of poor requirements managements, incorrect definition of the requirements from the start of the project and poor requirement management throughout the project life cycle.
Requirements involve the transitions of stakeholder requests into a set of key stakeholder needs and system features. These are detailed into specifications for functional and non-functional requirements.

Traceability helps us to assess the project impact of a change in requirement, impact of a failure of a test on requirements, manage the scope of project, verify the implementation of requirements and manage changes.

With IBM Rational RequisitePro there is little to no learning curve as requirements can be created and updated directly in Microsoft Word, a familiar environment for most people. Documents contain requirements in addition to contextual text that is vital for understanding requirements.
IBM Rational RequisitePro is integrated with IBM Rational Rose XDE, IBM Rational TestManager, IBM Rational SoDA, IBM Rational Unified Process and IBM Rational ClearQuest, as well as Microsoft Project. Integrated tools provide a smooth workflow and eliminate errors due to requirement discrepancies across tools. Integrations keep the entire project team in sync and improve team productivity. Easy access to requirements from other tools encourages all team members to view the requirements before starting their work.

IBM Rational RequisitePro support for traceability allows you to easily set up and track relationships between requirements to verify whether high-level requirements are associated with detailed software requirements. Querying these relationships provides coverage analysis to ensure completeness and to make sure time is not wasted building irrelevant functionality.

With IBM Rational RequisitePro, when traceability is established between requirements, and one of the requirement changes, suspect links automatically appear. This allows you to identify other requirements that may be affected by a change.

Fig.4.Rational RequisitePro – Shows Coverage Analysis, Features and Vision, Glossary, Impact Analysis, Supplementary Requirements, Use Cases and Requirement Management Plan

Activities:
Setting up RequisitePro for a project.
Adding Templates to your Rational RequisitePro project.
Capturing a common vocabulary.
Developing a vision using Rational RequisitePro project.
Detailing a Use case.
Detailing a Business use case.
Reviewing Requirement.

b) Phase Elaboration

During Elaboration phase, RequisitePro is used to set priorities for the Requirement and the attributes for the requirements to be implemented.

In the Elaboration phase, we identify system imperatives and requirements. The product we were developing is part of a larger offering package whose products must comply with system imperatives. Although these requirements did not always add value to our product or our specific client base, we had to invest the time and resources required to implement, test, and document compliance features. This meant cutting our feature list, which required another round of evaluations. At the end of Elaboration, we enter the remaining requirements into IBM® Rational RequisitePro®, our requirements management tool.

When using IBM Rational Rose XDE, suspect links are displayed between changing requirements and their associated design elements, so that it is clear what parts of the system are affected. Understanding the impact of change allows you to more accurately assess the potential cost in time and money of changing requirements.

With IBM Rational RequisitePro, defining attributes like priority, difficulty, and cost and having the ability to sort and filter requirements based on these attribute values allows project managers to objectively identify the highest value requirements

Having carried out the risk-to-use-case mapping for all of the technical risks, we can remove use cases without arrows as candidates for development during Elaboration; we can safely postpone these risks until the Construction phase.

Then we can look for the risk and identify all the intersecting use cases; implementing any of those use cases will attack that risk. If we continue this exercise for each risk, it soon becomes apparent that we can attack a number of high risks by implementing just a couple of use cases during Elaboration.

However, remember that earlier we said the goal of Elaboration is to implement the smallest amount of functionality that will confirm whether the high-priority risks we identified either

Activities:
To set priorities for requirements and attributes.
Analyzing the impact of changes by showing suspect links.
Assessing the potential cost and time required for requirement changes.
Sorting and filtering requirements.
Attack requirements with technical risk first.

Create and Manage Traceability

Fig.5.Rational RequisitePro shows traceability between Features and Use Cases

c) Phase: Construction & Integration

Out-of-the-box integration with Rational RequisitePro ensures seamless traceability between requirements and test cases, resulting in clear test coverage metrics and sensitivity to requirement modification.

d) Phase: Transition

In transition phase, progress reports can be generated using requisite pro to measure requirement coverage and traceability between the requirements.

Step 4: IBM Rational TestManager

a) Phase: Inception

Use TestManager in this phase to design test cases and analyze test logs to do POC (Proof of Concept) for the very few requirements to be implemented. We will be covering this tool exclusively in Elaboration, Construction & Transition Phases.

Activities:
Creating a test plan

b) Phase: Elaboration

Rational TestManager, which ships with Rational Robot, is the central console for test activity management, execution and reporting. Built for extensibility, it supports everything from pure manual test approaches to various automated paradigms including unit testing, functional regression testing, and performance testing.

Rational TestManager is meant to be accessed by all members of a project team, ensuring the high visibility of test coverage information, defect trends, and application readiness.

Establish and manage traceability: Requirements are linked to test cases, ensuring proper test coverage. In addition, suspicion analysis ensures that when requirements change, test cases traced to the requirement are automatically flagged as possible candidates for modification.

Rational TestManager puts the test team in control of the testing effort, focusing their effort by automating and simplifying crucial tasks. The tool allows teams to start from, keep track of, and test all the required functionality of an application, helping to ensure no critical business requirements go untested.

Activities

 Test Folder Creation
 Test Case creation
 Test Case Design
 Associating test inputs and attaching External documents are given more emphasis.

Test implementation both Manual and Automated is done for the requirement in scope for elaboration phase.

Fig.6.Create a Test Plan, Test Case Folder and Test Cases under that in hierarchical fashion

c) Phase: Construction & Transition

Support for all test types: Plan, manage and execute functional, performance, manual, integration, regression, configuration and component testing from the same interface.

Support for local and remote test execution: Run tests on the local machine or on remote machines in the test lab. Parallel test execution is limited only by the number of system resources at your disposal.

Detailed test evaluation: An integrated log viewer constructs a log for each test run, including test status and environmental information.

Out-of-the-box integration with Rational ClearQuest enables in-context defect submission directly from the TestManager Log Viewer, automatically porting test run data into the defect report to ensure accuracy.

d) Phase: Transition
Test manager is also used to generate coverage reports to analyze the testing effort and progress for the current phase or iteration. Test plan can be exported to a parcel file and can be delivered as an artifact.

Step 5: IBM Rational Robot

a) Phase: Inception

User Robot to create scripts for the requirements implemented in Inception phase as a part of POC (Proof of Concept). Even Robot will be explored in the subsequent sections.

Activities:
Setting up the test environment in Rational Robot
POC (Proof of Concept)

b) Phase: Elaboration

Test Scripts are created for the requirements covered in elaboration phase. Test Execution is carried out for the same.

Identifying and developing reusable parts is difficult. It is during the iterations in the elaboration phase that common solutions for common problems are found. This is the phase were many reusable functions or procedures are written in the common libraries to be used effectively in subsequent iterations.

c) Phase: Construction & Integration

In construction and integration phase, Rational Robot is exclusively used to execute regression test scripts (both manual as well as automation scripts) to do regression testing and new test scripts are developed for the new requirements being implemented in this current release so that the same can be used in the subsequent iterations to perform regression testing. As we know majority of the requirements are implemented in this phase so for these requirements automation is only possible in this release.

IBM Rational Robot can be used to automate and perform regression, functional and configuration testing for e-commerce, client/server and ERP applications. It's used to test applications based upon a wide variety of user interface technologies, and is integrated with the IBM Rational TestManager

I am listing down the advantages of using Rational Robot for functional testing:

Simplifies configuration testing.
Tests many types of applications
Ensures testing depth
Tests custom controls and objects
Provides an integrated programming environment
Helps you analyze problems quickly
Enables reuse

Fig.7.Create modular test scripts and include the VP’s and libraries used (depends on the automation framework being used)

Activity: Creating Test Suite

Implementing each Test Suite, and managing all subsequent changes to it
Ensuring that the Test Suite accurately reflects the test idea being realized
Ensuring that the Test Suite is implemented according to defined standards so as to be compatible and maintainable with other Test Suites, and with any Test Scripts it is dependent on.
Ensuring that the Test Suite makes reasonably efficient use of the available resources

d) Phase: Transition

In transition phase, all the scripts developed previously are verified by grouping them into shell scripts or as a suite and are executed in sequence to verify the functionalities before delivering the scripts.

Test data is exported from Test manager and test logs are also taken from test manager to deliver along with the suite, shell scripts and scripts.

Step 6: IBM Rational ClearQuest

a) Phase: Inception
During this phase Schema Repository and Defect Database are created using Rational ClearQuest Maintenance Tool and Rational ClearQuest Designer respectively.

Activities:
Establishing the change request process.
Defining change and review notifications.

Fig.7.Showing defects or change requests in Rational ClearQuest

b) Phase: Elaboration
Defects are raised based on the test logs generated by either executing manual or automation scripts.

c) Phase: Construction & Transition

Rational ClearQuest is also used exclusively in this phase because most of the requirement can only be tested only after implementation. In this release manual and automation scripts are executed to uncover defect in the application and they are logged in ClearQuest for tracking and closure. Once all the defects are raised in ClearQuest, Development team undertakes all valid defects and after fixing delivers a new build or iterations. If the defects rose in ClearQuest is a change request which is enhancement or change in requirement, automation script need to be modified accordingly.

d) Phase: Transition

Reports are generated to find all the change requests still not being resolved.
Depending upon the priority and severity of the change request, appropriate decision is taken to go for release or to go for one more iteration. If team wants to go for release then all the defects should be moved to different buckets to have a track like deferred or waived. ClearQuest changes are also integrated with RequisitePro requirements.

What are the top things you should do?
• Get an expert in RUP & automation consultants work together. A RUP expert can provide guidance in the usage of RUP in different phases of software testing.

• Integrate all the tools to get the maximum benefits: Make end to end implementation of all the rational tools from RequisitePro to ClearQuest with the help of Rational Administrator to derive the maximum benefits and cost cutting in long run.

• Usage of Right tool at the Right time: It’s very important to understand the significance of using the right tool at the right time (RUP phase) to avoid rework and maintain all the information right in place.

• Adopt RUP best practices and be guided by the spirit of RUP. For success, you need to be guided by the spirit of RUP and you want to follow RUP best practices for the best possible outcome.

• Generate different Metrics: Using this approach you can get different reports (statistics) for development and test managers using Rational Test Manager, ClearQuest and Rational RequisitePro, which can help to determine the correct progress of the project and the state of the artifacts at any point of time.

Conclusion

In conclusion, I hope that this article will spur "inclusive thinking" among software testing organizations, encouraging them to practice RUP with respect into Software testing right from the beginning -- rather than treating them as an afterthought late in the development lifecycle. We also hope that tool vendors and third-party suppliers for RUP and other development frameworks will begin embedding an integrated approach for accessibility into their products.

Testing team should be aware of the usage of different rational testing tools in all phases of RUP. If the project is following RUP where requirements are delivered in multiple phases and multiple iterations are expected, using these tools efficiently and effectively can lead to smooth and successful completion of project. This paper can be treated as a guideline document by testing teams working in a RUP project to know what to do and when to do in different phases of RUP.

For all software development organizations, we believe that the approach we describe in this article will ultimately increase productivity and application quality. In making products that are accessible and practical for all potential users, progressive organizations can take advantage of new market opportunities and gain a strong competitive edge.

Reference

Rational Unified Process
Bach, James. 1996. “Test Automation Snake Oil.” Windows Technical Journal, (October): 40-44.
http://www.satisfice.com/articles/test_automation_snake_oil.pdf
Marick, Brian. 1998. “When Should a Test Be Automated?”. http://www.testing.com/writings/automate.pdf
Kaner, Cem. 1997. “Improving the Maintainability of Automated Test Suites. http://www.kaner.com/lawst1.htm
Dustin, Elfriede. 1999. “Lessons in Test Automation.” Software Testing and Quality Engineering (September): 16-22.
http://www.stickyminds.com/sitewide.asp?ObjectId=1802&ObjectType=ART&Function=edetail
Fewster, Mark and Dorothy Graham. 1999. Software Test Automation, Addison-Wesley.
Groder, Chip. “Building Maintainable GUI Tests” in [Fewster 1999].
Hendrickson, Elisabeth. 1999. “Making the Right Choice: The Features you Need in a GUI Test Automation Tool.” Software Testing and Quality Engineering Magazine (May): 21-25.
http://www.qualitytree.com/feature/mtrc.pdf

Iterative Automation Process using Rational Unified Process

rajkamal@microsoft.com (Raj) — Mon, 03 Sep 2007 12:48:00 +0000

Test automation is mantra of success in today’s IT world and is used as a strong qualifier to win new projects yet often doesn’t add much value to client. Although automation promises to save cost and effort dramatically but they rarely succeed. Project where RUP is followed, requirements are implemented in phased manner, automation can be very fruitful but it is only possible if RUP is understood and if the same is applied to automation too. Testing team should follow ATLM (Automation Testing Life-cycle Methodology) along with Rational Unified Process. The key is to start automation right from the inception phase and to continue till transition to get ROI. This paper presents the process and guidelines to be followed by Managers and Automation Test Leads. Follow these steps as you staff, tool, or schedule your test automation project, and you will be well on your way to success.

A Fable

I have seen lot of many test automation problems. I have worked in many automation projects, big and small. I have talked to many people from many other projects. I am presenting this paper to avoid these problems faced by them. But first we need to understand it. Let me illustrate with a fable.

Once upon a time, we have a pilot test automation project for a very important client. Client is very quality conscious and provides the team with Rational Testing Tools. Development team has decided to follow Rational Unified Process. Mr. Rakesh is a Test Manager who hires experienced testers with good programming knowledge. Everyone is very excited as they are going to work with best tools in the market and no more excel sheet to write test cases, track defects and to generate report.

Project starts and Rational RequisitePro is used for requirement management, development team uses Rational Rose to design and ClearQuest for configuration management. Testing team doesn’t have any clue what so ever what going on. Then Rakesh realized and instructed team to start writing test cases using Test Manager. Testers found TestManager as great tools and wrote some test cases based on the requirements. By the time development team releases the first build to test. Testing has started executing test cases manually and some how ended up.

Project has moved to next phase, elaboration and they are designing the data model along with coding for critical modules. Testing doesn’t know what to do it in this phase and they are waiting for the build to come. Testing team expands test cases in Test Manager and leave early to house.

Project moves to Construction phase and build is given again with most of the functionalities implemented and testing time is finding difficult to manage time efficiently. Defects are found and development team starts fixing the defects. Then Rakesh instructs all the testing members to automate the application and they do some quick record and playback. After spending lot of effort, testing team is ready is record and playback suite. Again, build comes and testing team fails to run even a single script properly, comes to the conclusion that slight maintenance is required but once again they fail and then again fail.
Client feels organization is incapable of doing automation; Rakesh feels that team is inefficient, team feels testing tool is not up to the mark. Rajesh asks for release, testing team members feel automation is of no use, manual testing is better. Project is a failure.

That's my fable. Perhaps parts of the story sound familiar to you. But I hope you haven't seen a similar ending. This paper will suggest some ways to avoid the same fate

The Problems

This fable illustrates several problems that plague test automation projects:

Automation at last moment: There is a general trend and people think that automation should come at the end of the project. Managers don’t focus on automation right from the initial phase.

Automating only regression candidates: This is a myth that only automation candidates should be automated and regression testing come late during the project. It pushes the automation to the last moment.

Short of time: Testers often feel that time given is insufficient for testing. It is because testing is manually all the time and automation comes into picture only in the end very near to deadline.

Automation takes less time: Managers think that automation takes less time and can be done in couple of days and then time can be saved. Reality is automation takes more time than manual testing and is only useful in long run.

Testing RUP project is no different: Managers think RUP project is just any other development model like Waterfall where product will be given as a whole.

Any tool will do: Managers don’t consider whether tool can be used for the AUT, they just go by the popularity and later realize that it is not good.

Follow the Rules of Iterative Automation

This paper will be organized by the normal steps that we all use for our test automation projects, making special notes of the considerations and challenges that are particular to test automation:

1. Test Tool Acquisition
2. Test Framework Selection
3. Test Planning and Development
4. Test Execution
5. Analysis & Assessment

Abstract

Step 1: Test Tool Acquisition

Role: Test Designer & Test Manager
Activity: Define Test Environment Configuration
Artifact: Test Environment Configuration
Phase: Inception

Test Tool Acquisition is an activity which should be done early in the Inception phase. During Inception phase, very few requirements are implemented to verify the feasibility. This is the phase when the automation tools should be evaluated and POC should be done. In this paper, out of other tools like QTP, Winrruner and Robot, Rational Robot came out as winner and was selected to automate the application-under-test.

Step 2: Test Framework Selection

Role: Test Designer
Activity: Select Automation Framework
Artifact: Test Automation Architecture
Phase: Elaboration

It is very important to select an appropriate automation framework. Different options available are Record/Playback, Module centric framework, Data-Driven framework and Keyword-Driven framework. It is preferable to start analyzing the option right from the inception phase. POC done in the inception phase will help Test Designer to select the framework which best suite the need. No. of iterations planned, size of the project and nature of the requirements are also considered as valuable inputs while selecting an automation framework.

Elaboration phase is the one when automation framework selected is frozen and making any changes after this can be risky and can involve lot of rework. In this feasibility of the framework is validated by automating the most critical requirements in Elaboration phase. A composition of various test automation design and implementation elements and their specifications that embody the fundamental characteristics of the test automation software system.

This artifact is particularly useful when the automated execution of software tests must be maintained and extended through multiple test cycles. This artifact is most useful as single artifact per project.

Step 3: Test Planning & Development

Role Test Analyst
Activity: Identify Test Ideas, Identify targets of test
Artifact: Test Idea List, Test Cases, Test Data
Phase: Elaboration

Test Analyst identifies the test cases to be automated and selects the automation candidates based on the following criteria:

Identifying and defining each Test Case, and approving all subsequent changes to it.
Ensuring that changes are communicated to affected downstream roles.
Ensuring that sufficient Test Cases have been identified to provide satisfactory evaluation of the Target Test Items.
Ensuring that sufficient detail has been provided to implement and conduct the test.
Managing and maintaining appropriate traceability relationships.
Managing the appropriate scope of the Test Cases in a given iteration.

Test Analyst is also responsible for Test Data:

Identifying potential data sources.
Gathering basic candidate Test Data.
Verifying the completeness, fitness for purpose and accuracy of the Test Data.

Test cases to be automated are present in Rational Test Manager and this activity is generally completed in Elaboration phase. Once all the test cases to be automated and test data is ready, automation can be done in full fledge manner in construction phase.

Role Tester
Activity: Implement Test, Implement Test Suite
Artifact: Test Script
Phase: Construction

This is most important activity which is carried out exclusively in Construction phase but you must have read in earlier sections that POC is done in inception phase and critical test cases are automated in Elaboration phase. Henceforth, automation is done for other automation candidate including the most important regression set of test cases.

Identifying and defining each Test Script, and managing all subsequent changes
Ensuring the Test Script accurately reflects the required test, identified by one or more a Test Ideas or defined in one or more Test Cases
Ensuring the Test Script is implemented according to defined standards to be compatible and maintainable with the other Test Scripts
Ensuring the Test Script makes reasonably efficient use of the available resources
Developing the Test Script with a focus on economy of effort and identifying opportunities for reuse and simplification
Developing the Test Script so that it can be used as part of a Test Suite

Step 4: Test Execution

Role Tester
Activity: Execute Test Suite, Analyze Test Failure
Artifact: Test log
Phase: Construction & Integration

Though script execution is something which even at Inception and Elaboration phase too but in construction phase all the scripts automated are executed as suites and overall functionality of the application is validated.

Different suites can be executed like BVT Suite which can be ran to do smoke testing and log can be used to take the decision to accept or reject the build. Regression suites can be ran to ensure that there is no regression impact of the fixes/changes made to the application.

Ensuring the accurate recording of the observed outcome of each test executed in the test cycle.
Ensuring the Test Logs are uniquely and accurately identified, and stored against the correct test cycle or test run.
Actively monitoring for anomalous and erroneous occurrences in the Test Log, and taking appropriate recovery and reporting actions.

Step 5: Analysis & Assessment

Role: Test Analyst
Activity: Determine Test Results
Artifact: Test Results
Phase: Construction, Integration & Transition

This activity is carried out exclusively in Construction phase and elaboration phase. The test logs are analyzed in Rational Test Manager and defects can be raised in ClearQuest.

If the defect raised is because of mistake or miscommunication, the same is fixed and the script is ran again in next iteration but if the failure was because of desirable change / new requirement / change in requirement, then script is modified and ran again in the same iteration and again test logs are analyzed.

Reviewing Test Logs and Change Requests
Actively monitoring for anomalous and erroneous occurrences in the Test Log, investigating and reporting a conclusion
Ensuring the accurate analysis of the observed outcome of each test conducted in the test execution cycle
Ensuring the Test Results are uniquely and accurately identified and recorded against the correct test execution cycle

Role: Test Manager
Activity: Assess and Advocate Quality, Assess and Improve Test Effort
Artifact: Test Evaluation Summary
Phase: Integration & Transition

In construction and transition phase, test manager performs the following activities:
Reviewing the Test Results, change request statistics, and coverage statistics.
Reviewing important Change Request and Issue details.
Presenting an accurate and fair assessment of the software based on the defined Evaluation Mission.

Overall Iterative Automation Process

An iterative process facilitates reuse of project elements because it is easier to identify common parts as they are partially designed or implemented instead of identifying all commonality in the beginning.

Iterative automation process includes the following steps:

Iterative automation has the following advantages:

1. The automated test suite should be executed for all the iterations. Doing this is very important to “Continuously Verify Quality” which is one of the best practice followed by RUP.

2. Doing automation in an iterative way allows flaws to be detected and resolved earlier in the product life cycle. It saves significant amount of time spent to perform manual testing for each and every build in each and every release.

3. Iterative automation provides better view of the quality and helps in correcting the major issues without jeopardizing target costs and schedules.

4. We cannot stop change from being introduced into our project. However we must control how and when changes are introduced into project artifacts, and who introduces the changes. Unified Change Management (UCM) is Rational Software’s approach to managing change in software system development, from requirement to release. “Manage Change” is one more important best practice followed by RUP.

5. Better Overall Quality: Automation scripts developed from an iterative process will be of better overall quality than are the products that result from a conventional sequential process. The script will have been tested several times, improving the quality of testing. At the time of delivery scripts will have been running longer.

6. Stage Containment: Iterative script development helps to contain the defect in the same phase and doesn’t let it go to the last phase. Design bugs can be identified in the elaboration phase and developers don’t need to wait for construction phase to find out design issues.

Conclusion

Automation should be iterative in nature and should be performed right from the beginning of the project. If the project is following RUP where requirements are delivered in multiple phases and multiple iterations are expected, automation can be very handy in reducing the testing effort and cost to a great extent. The effort saved can be better utilized in other areas. This paper can be considered as a reference document by automation testing teams to know what to do and how in different phases of RUP.

Reference

Rational Unified Process

Kaner, Cem. 1997. “Improving the Maintainability of Automated Test Suites.” http://www.kaner.com/lawst1.htm

Fewster, Mark and Dorothy Graham. 1999. Software Test Automation, Addison-Wesley.