Monday, October 1, 2007
Sunday, September 30, 2007
PONC - Those who speak of progress measure it by “Quantity” and not by “Quality”
Test Manager A: Already 3 days of testing is over and you have found just 8 bugs. I am disappointed with the way …….
Tester B: but if you look at the bugs you will find…
Test Manager A: Don't make excuses…when I was a tester, I found "101" bugs in just 2 days. Take example of Tester B007, he has already found 22 bugs and…
Tester B: I understand you point and let me explain you that I have found 5 Severity one bugs and they are really critical.
Test Manager A: That's fine but I want you to file at least 10 more bugs before you leave today.
Tester B: Okay L
Rings any bells? Sounds familiar? You are very lucky if you are not tester and it didn't happen to you ever J
Okay folks, I am not telling you a story, this happened to one of my friend in a CMM Level 5 company.
Why Quantity kills Quality? L
There is an article on "Quantity, Quality, and the Function of Knowledge "by Hartley B. Alexander
I am presenting an interesting excerpt from this wonderful piece of work on philosophy and human psychology to prove my point:
'The human mind knows only numbers adequately.', 'The mind only grasp fully and clearly which is reducible to the term of ratio', 'we reduce our impressions to mathematical formulas, and simply by means of this reduction of a confused state to a clear idea, we manage adequately to grasp the real'
What do we understand by Quantity?
Quantity is a kind of property which exists as magnitude or multitude. Quantity was first introduced as quantum, an entity having quantity. Being a fundamental term, quantity is used to refer to any type of quantitative properties or attributes of things. In our simple industry terms "Quality is conformance to requirements."
Where to Apply?
- Defects / Bugs: Finding Quality bugs / defects are more important than finding lots of not that severe bug.
- Test Cases: Writing few quality test cases is more effective than writing 10001 test cases? In the end a good test case is one which has a high probability of find defects.
- Automated Test Scripts: Automating few test cases which can give you high ROI are better to automate rather than trying to automate 100 % test cases and then ending up doing lot of rework.
Some on the common ways to measure / improve quality are:
- KAIZEN (Contribution Process Improvement)
- Quality Management System
- Total Quality Management
- Six Sigma
- Zero Defect
- SPC (Statistical Process Control)
Suggestions:
- Severity: When you prepare test plans, define "Severity" definition very clearly.
- Prioritization: Attack high risk areas first and something which is too critical for customer / end user.
- Unit Testing: Encourage your dev team to perform a good unit testing so that you can find quality bugs in Integration and System testing phase.
- Impact: For each bug logged state "Business Impact" of that bug if not been fixed.
- Worst Case Scenario: Always identify and mention the "WORST CASE SCENARIO" of the bug.
- Quality of the Build: If the bug is Sev 1 and that is critical enough and might be possible cause of many other bugs then immediately stop testing and reject the build. Business users don't have time to analyze your complex bug matrix.
- Reports with GRAPHS: Give daily reports / weekly report / build summary / test coverage reports with "Graphs" with bug classification as Sev 1, Sev 2 and Sev 3.
- Triage: Always include the stakeholders (recommended – Customer, Business User) in the BUG TRIAGE meetings and let them know that you have found something critical.
- Talking with Customer always helps: If you feel testing estimation are too aggressive then always let the customer know that if test team is given more time then you can discover many more quality bugs and that way over a period of time you will have "many quality bugs and that will increase the bugs count too J"
My viewpoint: "Quality is reconcilable with Quantity" J There is tradeoff that exists between quality and quantity.
Final Verdict: Left to readers J
Lesson Learnt: I should try to write "Quality" blogs over "Quantity" L
If you are among the "vital few" who are still thinking what is PONC ...then it's "PRICE OF NON CONFORMANCE" and that signifies the importance of quality over just quantity.
Disclaimer: There is absolutely nothing wrong with assessing progress by looking at the no. of bugs but I would rather give more importance to **severity** of those bugs.
Sunday, September 23, 2007
The ‘Omerta’ of Testers (Mafia)
Show me one tester with a considerable exposure to IT industry, who never engaged himself into “The Blame Game” intentionally or unintentionally with Developers or Analyst.
“Tick, Tock, and Testimony”
Each test team and opponents are asked for justification or tell their side of the story. Old mails are sought out for evidences to prove their point until both were out of time.
“You did It, Now Admit it!”
Now team held responsible will be penalized for committing that horrendous crime
“Jury of your management”
Management would decide to show mercy or no mercy and the convicted to be added to “Do Not Trust This Blame Game Loser”
Let’s not dig deep into what causes quality failures or defects in the production, we are trying to understand lot of psychological problems and people issues that often lead to slinging mud at each other.
After a detailed discussion testing teams from different leading organizations, following are identified as main culprits:
1. DREAD:
a) Fear of missing a bug.
b) An anxious feeling of not able to find all the bugs.
c) A lack of confidence due to nature of our job where exhaustive testing is not possible in the time given.
d) Scared of the no. of possible permutation and combinations that exist.
e) Panicked by the scarcity of time or resources.
2. BALL IS NOT IN MY COURT:
Most of the time testers think only when the build comes to them and during the execution phase they are “Accountable” for the quality. That means they can relax and chill out at the time of requirement, design, coding and deployment. Testing and Production is the only phase they can be held responsible.
3. ACCOUNTABILITY:
If anything goes wrong, testers are usually choked by “The Family” (Dev, Analyst, and Management). It’s not that testers are not accountable for quality but the common misconception is that they are the ONLY discipline responsible for QUALITY against the principle “Quality is the responsibility of each employee in the team”
4. PRESSURE
Even Indian cricket team dies under pressure then think of a testing team shouldering the responsibility of a product that means both business and satisfaction for the end users.
Sometime imperativeness takes a toll on the testing efforts and affects the quality of the effort being put by the tester.
Continuously being bugged by managers often restrict the tester from thinking “out-of-the-box” scenarios and trying out some end user events by putting themselves into end user’s shoes.
5. QUANTITY OVER QUALITY
I don’t know how many times you would have seen that management want to know the no. of bugs to decide upon the performance of the testing team. Priority / Severity take a back seat and Dev / Management shrugs their shoulders to say “Is that all you found?”
Ultimately it boils down to quantity over quality in performance appraisal of testers.
The ‘Omerta of Testers’: STOP the blame game!!!
“Whenever a tester appeals to the management against his fellow developer or analyst is either a fool or a coward. A tester who cannot take care of himself without management protection is both. It is as cowardly to blame an offender to justice, even though his offences are against yourself”
Remember Test and Dev are like wheels of a two-wheeler vehicle and only way to move forward is to make a clear distinction of each other’s responsibility and understanding of most important fact that we are working here as a team and for end user what matters the most is a high quality product at a low cost. The Industry, Client, End users don’t care if it is the dev fault or a miss from testing team.
“Tick, Tock, and Testimony”
Each test team and opponents are asked for justification or tell their side of the story. Old mails are sought out for evidences to prove their point until both were out of time.
“You did It, Now Admit it!”
Now team held responsible will be penalized for committing that horrendous crime
“Jury of your management”
Management would decide to show mercy or no mercy and the convicted to be added to “Do Not Trust This Blame Game Loser”
Let’s not dig deep into what causes quality failures or defects in the production, we are trying to understand lot of psychological problems and people issues that often lead to slinging mud at each other.
After a detailed discussion testing teams from different leading organizations, following are identified as main culprits:
1. DREAD:
a) Fear of missing a bug.
b) An anxious feeling of not able to find all the bugs.
c) A lack of confidence due to nature of our job where exhaustive testing is not possible in the time given.
d) Scared of the no. of possible permutation and combinations that exist.
e) Panicked by the scarcity of time or resources.
2. BALL IS NOT IN MY COURT:
Most of the time testers think only when the build comes to them and during the execution phase they are “Accountable” for the quality. That means they can relax and chill out at the time of requirement, design, coding and deployment. Testing and Production is the only phase they can be held responsible.
3. ACCOUNTABILITY:
If anything goes wrong, testers are usually choked by “The Family” (Dev, Analyst, and Management). It’s not that testers are not accountable for quality but the common misconception is that they are the ONLY discipline responsible for QUALITY against the principle “Quality is the responsibility of each employee in the team”
4. PRESSURE
Even Indian cricket team dies under pressure then think of a testing team shouldering the responsibility of a product that means both business and satisfaction for the end users.
Sometime imperativeness takes a toll on the testing efforts and affects the quality of the effort being put by the tester.
Continuously being bugged by managers often restrict the tester from thinking “out-of-the-box” scenarios and trying out some end user events by putting themselves into end user’s shoes.
5. QUANTITY OVER QUALITY
I don’t know how many times you would have seen that management want to know the no. of bugs to decide upon the performance of the testing team. Priority / Severity take a back seat and Dev / Management shrugs their shoulders to say “Is that all you found?”
Ultimately it boils down to quantity over quality in performance appraisal of testers.
The ‘Omerta of Testers’: STOP the blame game!!!
“Whenever a tester appeals to the management against his fellow developer or analyst is either a fool or a coward. A tester who cannot take care of himself without management protection is both. It is as cowardly to blame an offender to justice, even though his offences are against yourself”
Remember Test and Dev are like wheels of a two-wheeler vehicle and only way to move forward is to make a clear distinction of each other’s responsibility and understanding of most important fact that we are working here as a team and for end user what matters the most is a high quality product at a low cost. The Industry, Client, End users don’t care if it is the dev fault or a miss from testing team.
Friday, September 7, 2007
Wednesday, September 5, 2007
Why do you think you are a good testing professional?
I believe that we (testers) are end user's advocate and are often looked at like devils by our devs folks across the globe :). My aim in life is not to find a million bugs or to show that every day I come up with a new or an advanced automation framework. I believe we are here to prevent bugs from happening by constantly focusing on continuous process improvements, defect prevention techniques and ensuring only one thing that our customer’s expectations are not only met but exceeded by shipping high quality products keeping the cost of quality at a minimum.
Monday, September 3, 2007
Mapping Rational Unified Process to Rational Testing Tools
Abstract
Rational Unified Process raises everyone’s hope yet often confuses testers to follow it. IBM Rational Tools are used extensively but there is no documentation available, stating when and how to use Rational Testing tools in different RUP phases i.e. Inception, Elaboration, Construction and Transition. It frustrates and disappoints testers. Although Rational Unified Process provides “Tool Mentors” to explain the usage of all the tools but still there is no direct mapping between usage of the Rational Testing Tools and Testing Life Cycle which explain the responsibility of the Testing team through out the SDLC in different Iterations.. This paper presents the guidelines which will suggest the role and responsibility of testing team in different phases of Rational Unified Process. Follow these guidelines as you staff, tool, or schedule your test automation project, and you will be well on your way to success.
A Fable
I have worked in many testing projects, big and small. I have seen different projects following RUP model and I have personally talked to many testing teams. I am presenting this paper to avoid these problems faced by them. But first we need to understand it. Let me illustrate with a fable.
Once upon a time, we have a big project following RUP for a very important client. Client provides the team with Rational Testing Tools. Sujata is a Test Manager; she pulls the experienced testers. Everyone is very excited as they are going to work with best tools in the market and no more excel sheet to write test cases, track defects and to generate report.
Testers have been trained on Rational Unified Process. Inception Phase, project starts and lot of testing tools available like ClearQuest, RequisitePro, Robot and Test Manager. Testers know how to use tools, thanks to Sujata once again for sending them for Rational Tools Training before putting them into project. They know Rational Unified process too but they don’t know how to use different tools in different phase (Inception, Elaboration, Construction and Transition).
They use RequisitePro for requirement understanding, test manager for writing test cases but still not sure when to use them and in which phase. Elaboration phase comes followed by construction phase and finally transition phase and Sujata realizes that nothing is in place. All the requirements are not documented, changes are not documented, test cases mapped to requirement have become obsolete, automation was pushed too late, and version was not maintained after a certain period.
Sujata couldn’t find the problem, she provided the best training to the best testers and they used the best tools in the market but still they couldn’t manage the project. She quits and hand over to Ashish, new test managers. Project is a failure.
That's my fable. Perhaps parts of the story sound familiar to you. But I hope you haven't seen a similar ending. This paper will suggest some ways to avoid the same fate
The Problem
This fable illustrates several problems that plague testing project following RUP:
What to use when?
This is the major problem which hinders the success of the projects and many times result in total disaster. In the worst case, testers don’t know RUP, usage of automation tools which stops project from doing well. If we consider the best case as mentioned by me in the previous section that even if testers are trained in RUP and usage of testing tools like Robot & TestManager still project might not succeed because they should know when to use Robot, when to use Test Manager and when to use other testing tools like ClearQuest and ClearCase in different phases in RUP.
How to use?
Other big question is the functionalities and features of the tools those should be given more focus in different phases of RUP.
Who should use?
This question can be answered if the tester knows what he is supposed to do and what manager is responsible of doing.
Guidelines- usage of Rational Testing Tools in a RUP Project
This paper will be organized by the normal steps that we all use for our testing projects, making special notes of the considerations and challenges that are particular to RUP.
1. IBM Rational Administrator
2. IBM Rational ClearCase
3. IBM Rational RequisitePro
4. IBM Rational TestManager
5. IBM Rational Robot
6. IBM Rational ClearQuest
All the above mentioned tools are explained with respect to their usage in different phases of RUP. I haven’t covered Rational Rose as I wanted to keep the focus restricted to the tools which are used in software testing.
I am giving a brief introduction about RUP phases for the audiences who are new to RUP.
Inception – Focus of this phase is the understanding the scope of the project.
Elaboration: The architecture as wells as the requirement of the products being built must be understood by the end of the phase.
Construction & Integration: Product must be constructed in this phase.
Transition –The product must be rolled out to customers in this phase.
Step 1: IBM Rational Administrator
a) Phase: Inception
During this phase, a shared repository is created using either MS-Access or SQL Sybase Central. A new project folder is created in the share with UNC (Unified Naming Convention). The project should be UCM (Unified Change Management) Enabled to ensure integrity with Rational ClearCase for Configuration Management. Rational Administrator provided the platform to integrate all the rational tools together to achieve end-to-end mapping.
Activities:
Create a project
Connect to a project
Create a test datastore
Create integration between products
Decide whether to associate the administrator project with UCM project.
Fig.1. Mapping of RUP phases to Rational Testing Tools
b) Phase: Elaboration
During Elaboration phase, Rational Administrator is used to create new users with different access rights depending upon the requirement of the project
c) Phase: Construction & Integration
Like Construction phase, Rational Administrator is used for managing users and access rights. It also maintains the integrity between different tools used.
d) Phase: Transition
Like Elaboration & Construction phase, Rational Administrator is used for managing users and access rights. It also maintains the integrity between different tools used.
Step 2: IBM Rational ClearCase
a) Phase: Inception
ClearCase provides life cycle management and control of software development assets. With integrated version control, automated workspace management, parallel development support, baseline management, and build and release management, Rational ClearCase provides the capabilities needed to create, update, build, deliver, reuse and maintain business-critical assets.
Fig.2.Rational Clear Case Workflow
Activities:
Create the configuration management repositories that represent the subsystem defined by your architecture.
Import existing files and directories into the repositories to create an initial set of development configuration items.
Enable your ClearQuest schema to work with UCM
Enable your clear case project to work with ClearQuest.
Set Policies for new projects
b) Phase: Elaboration
During this phase too new assets created are base lined and updated. When new test cases are created using Test Manager and new scripts are created using Robot are base lined using UCM concept.
Activities:
Updating your project work area using Rational ClearCase.
Checking in and checking out configuration items.
Delivering your work with ClearCase
Promoting project baselines using Rational ClearCase
Comparing baselines using Rational ClearCase.
Fig.3.Rational Clear Case detailed diagram
c) Phase: Construction & Integration
Out-of-the-box integration of Rational Test Manager, RequisitePro and Rational ClearQuest with Rational ClearCase LT delivers full base lining capabilities to accommodate the natural evolution of a project, moving from one build or release to the next.
d) Phase: Transition
In transition phase, all the artifacts are delivered and existing one are moved to baseline to maintain version control.
Step 3: IBM Rational RequisitePro
a) Phase: Inception
During this phase RequisitePro is mainly used for Managing Requirements. Inception phase focuses mainly on managing the requirements collected.
“Manage Requirement” is one of the best practices in RUP. Research from Standish Report shows 31 % of projects gets failed because of poor requirements managements, incorrect definition of the requirements from the start of the project and poor requirement management throughout the project life cycle.
Requirements involve the transitions of stakeholder requests into a set of key stakeholder needs and system features. These are detailed into specifications for functional and non-functional requirements.
Traceability helps us to assess the project impact of a change in requirement, impact of a failure of a test on requirements, manage the scope of project, verify the implementation of requirements and manage changes.
With IBM Rational RequisitePro there is little to no learning curve as requirements can be created and updated directly in Microsoft Word, a familiar environment for most people. Documents contain requirements in addition to contextual text that is vital for understanding requirements.
IBM Rational RequisitePro is integrated with IBM Rational Rose XDE, IBM Rational TestManager, IBM Rational SoDA, IBM Rational Unified Process and IBM Rational ClearQuest, as well as Microsoft Project. Integrated tools provide a smooth workflow and eliminate errors due to requirement discrepancies across tools. Integrations keep the entire project team in sync and improve team productivity. Easy access to requirements from other tools encourages all team members to view the requirements before starting their work.
IBM Rational RequisitePro support for traceability allows you to easily set up and track relationships between requirements to verify whether high-level requirements are associated with detailed software requirements. Querying these relationships provides coverage analysis to ensure completeness and to make sure time is not wasted building irrelevant functionality.
With IBM Rational RequisitePro, when traceability is established between requirements, and one of the requirement changes, suspect links automatically appear. This allows you to identify other requirements that may be affected by a change.
Fig.4.Rational RequisitePro – Shows Coverage Analysis, Features and Vision, Glossary, Impact Analysis, Supplementary Requirements, Use Cases and Requirement Management Plan
Activities:
Setting up RequisitePro for a project.
Adding Templates to your Rational RequisitePro project.
Capturing a common vocabulary.
Developing a vision using Rational RequisitePro project.
Detailing a Use case.
Detailing a Business use case.
Reviewing Requirement.
b) Phase Elaboration
During Elaboration phase, RequisitePro is used to set priorities for the Requirement and the attributes for the requirements to be implemented.
In the Elaboration phase, we identify system imperatives and requirements. The product we were developing is part of a larger offering package whose products must comply with system imperatives. Although these requirements did not always add value to our product or our specific client base, we had to invest the time and resources required to implement, test, and document compliance features. This meant cutting our feature list, which required another round of evaluations. At the end of Elaboration, we enter the remaining requirements into IBM® Rational RequisitePro®, our requirements management tool.
When using IBM Rational Rose XDE, suspect links are displayed between changing requirements and their associated design elements, so that it is clear what parts of the system are affected. Understanding the impact of change allows you to more accurately assess the potential cost in time and money of changing requirements.
With IBM Rational RequisitePro, defining attributes like priority, difficulty, and cost and having the ability to sort and filter requirements based on these attribute values allows project managers to objectively identify the highest value requirements
Having carried out the risk-to-use-case mapping for all of the technical risks, we can remove use cases without arrows as candidates for development during Elaboration; we can safely postpone these risks until the Construction phase.
Then we can look for the risk and identify all the intersecting use cases; implementing any of those use cases will attack that risk. If we continue this exercise for each risk, it soon becomes apparent that we can attack a number of high risks by implementing just a couple of use cases during Elaboration.
However, remember that earlier we said the goal of Elaboration is to implement the smallest amount of functionality that will confirm whether the high-priority risks we identified either
Activities:
To set priorities for requirements and attributes.
Analyzing the impact of changes by showing suspect links.
Assessing the potential cost and time required for requirement changes.
Sorting and filtering requirements.
Attack requirements with technical risk first.
Create and Manage Traceability
Fig.5.Rational RequisitePro shows traceability between Features and Use Cases
c) Phase: Construction & Integration
Out-of-the-box integration with Rational RequisitePro ensures seamless traceability between requirements and test cases, resulting in clear test coverage metrics and sensitivity to requirement modification.
d) Phase: Transition
In transition phase, progress reports can be generated using requisite pro to measure requirement coverage and traceability between the requirements.
Step 4: IBM Rational TestManager
a) Phase: Inception
Use TestManager in this phase to design test cases and analyze test logs to do POC (Proof of Concept) for the very few requirements to be implemented. We will be covering this tool exclusively in Elaboration, Construction & Transition Phases.
Activities:
Creating a test plan
b) Phase: Elaboration
Rational TestManager, which ships with Rational Robot, is the central console for test activity management, execution and reporting. Built for extensibility, it supports everything from pure manual test approaches to various automated paradigms including unit testing, functional regression testing, and performance testing.
Rational TestManager is meant to be accessed by all members of a project team, ensuring the high visibility of test coverage information, defect trends, and application readiness.
Establish and manage traceability: Requirements are linked to test cases, ensuring proper test coverage. In addition, suspicion analysis ensures that when requirements change, test cases traced to the requirement are automatically flagged as possible candidates for modification.
Rational TestManager puts the test team in control of the testing effort, focusing their effort by automating and simplifying crucial tasks. The tool allows teams to start from, keep track of, and test all the required functionality of an application, helping to ensure no critical business requirements go untested.
Activities
Test Folder Creation
Test Case creation
Test Case Design
Associating test inputs and attaching External documents are given more emphasis.
Test implementation both Manual and Automated is done for the requirement in scope for elaboration phase.
Fig.6.Create a Test Plan, Test Case Folder and Test Cases under that in hierarchical fashion
c) Phase: Construction & Transition
Support for all test types: Plan, manage and execute functional, performance, manual, integration, regression, configuration and component testing from the same interface.
Support for local and remote test execution: Run tests on the local machine or on remote machines in the test lab. Parallel test execution is limited only by the number of system resources at your disposal.
Detailed test evaluation: An integrated log viewer constructs a log for each test run, including test status and environmental information.
Out-of-the-box integration with Rational ClearQuest enables in-context defect submission directly from the TestManager Log Viewer, automatically porting test run data into the defect report to ensure accuracy.
d) Phase: Transition
Test manager is also used to generate coverage reports to analyze the testing effort and progress for the current phase or iteration. Test plan can be exported to a parcel file and can be delivered as an artifact.
Step 5: IBM Rational Robot
a) Phase: Inception
User Robot to create scripts for the requirements implemented in Inception phase as a part of POC (Proof of Concept). Even Robot will be explored in the subsequent sections.
Activities:
Setting up the test environment in Rational Robot
POC (Proof of Concept)
b) Phase: Elaboration
Test Scripts are created for the requirements covered in elaboration phase. Test Execution is carried out for the same.
Identifying and developing reusable parts is difficult. It is during the iterations in the elaboration phase that common solutions for common problems are found. This is the phase were many reusable functions or procedures are written in the common libraries to be used effectively in subsequent iterations.
c) Phase: Construction & Integration
In construction and integration phase, Rational Robot is exclusively used to execute regression test scripts (both manual as well as automation scripts) to do regression testing and new test scripts are developed for the new requirements being implemented in this current release so that the same can be used in the subsequent iterations to perform regression testing. As we know majority of the requirements are implemented in this phase so for these requirements automation is only possible in this release.
IBM Rational Robot can be used to automate and perform regression, functional and configuration testing for e-commerce, client/server and ERP applications. It's used to test applications based upon a wide variety of user interface technologies, and is integrated with the IBM Rational TestManager
I am listing down the advantages of using Rational Robot for functional testing:
Simplifies configuration testing.
Tests many types of applications
Ensures testing depth
Tests custom controls and objects
Provides an integrated programming environment
Helps you analyze problems quickly
Enables reuse
Fig.7.Create modular test scripts and include the VP’s and libraries used (depends on the automation framework being used)
Activity: Creating Test Suite
Implementing each Test Suite, and managing all subsequent changes to it
Ensuring that the Test Suite accurately reflects the test idea being realized
Ensuring that the Test Suite is implemented according to defined standards so as to be compatible and maintainable with other Test Suites, and with any Test Scripts it is dependent on.
Ensuring that the Test Suite makes reasonably efficient use of the available resources
d) Phase: Transition
In transition phase, all the scripts developed previously are verified by grouping them into shell scripts or as a suite and are executed in sequence to verify the functionalities before delivering the scripts.
Test data is exported from Test manager and test logs are also taken from test manager to deliver along with the suite, shell scripts and scripts.
Step 6: IBM Rational ClearQuest
a) Phase: Inception
During this phase Schema Repository and Defect Database are created using Rational ClearQuest Maintenance Tool and Rational ClearQuest Designer respectively.
Activities:
Establishing the change request process.
Defining change and review notifications.
Fig.7.Showing defects or change requests in Rational ClearQuest
b) Phase: Elaboration
Defects are raised based on the test logs generated by either executing manual or automation scripts.
c) Phase: Construction & Transition
Rational ClearQuest is also used exclusively in this phase because most of the requirement can only be tested only after implementation. In this release manual and automation scripts are executed to uncover defect in the application and they are logged in ClearQuest for tracking and closure. Once all the defects are raised in ClearQuest, Development team undertakes all valid defects and after fixing delivers a new build or iterations. If the defects rose in ClearQuest is a change request which is enhancement or change in requirement, automation script need to be modified accordingly.
d) Phase: Transition
Reports are generated to find all the change requests still not being resolved.
Depending upon the priority and severity of the change request, appropriate decision is taken to go for release or to go for one more iteration. If team wants to go for release then all the defects should be moved to different buckets to have a track like deferred or waived. ClearQuest changes are also integrated with RequisitePro requirements.
What are the top things you should do?
• Get an expert in RUP & automation consultants work together. A RUP expert can provide guidance in the usage of RUP in different phases of software testing.
• Integrate all the tools to get the maximum benefits: Make end to end implementation of all the rational tools from RequisitePro to ClearQuest with the help of Rational Administrator to derive the maximum benefits and cost cutting in long run.
• Usage of Right tool at the Right time: It’s very important to understand the significance of using the right tool at the right time (RUP phase) to avoid rework and maintain all the information right in place.
• Adopt RUP best practices and be guided by the spirit of RUP. For success, you need to be guided by the spirit of RUP and you want to follow RUP best practices for the best possible outcome.
• Generate different Metrics: Using this approach you can get different reports (statistics) for development and test managers using Rational Test Manager, ClearQuest and Rational RequisitePro, which can help to determine the correct progress of the project and the state of the artifacts at any point of time.
Conclusion
In conclusion, I hope that this article will spur "inclusive thinking" among software testing organizations, encouraging them to practice RUP with respect into Software testing right from the beginning -- rather than treating them as an afterthought late in the development lifecycle. We also hope that tool vendors and third-party suppliers for RUP and other development frameworks will begin embedding an integrated approach for accessibility into their products.
Testing team should be aware of the usage of different rational testing tools in all phases of RUP. If the project is following RUP where requirements are delivered in multiple phases and multiple iterations are expected, using these tools efficiently and effectively can lead to smooth and successful completion of project. This paper can be treated as a guideline document by testing teams working in a RUP project to know what to do and when to do in different phases of RUP.
For all software development organizations, we believe that the approach we describe in this article will ultimately increase productivity and application quality. In making products that are accessible and practical for all potential users, progressive organizations can take advantage of new market opportunities and gain a strong competitive edge.
Reference
Rational Unified Process
Bach, James. 1996. “Test Automation Snake Oil.” Windows Technical Journal, (October): 40-44.
http://www.satisfice.com/articles/test_automation_snake_oil.pdf
Marick, Brian. 1998. “When Should a Test Be Automated?”. http://www.testing.com/writings/automate.pdf
Kaner, Cem. 1997. “Improving the Maintainability of Automated Test Suites. http://www.kaner.com/lawst1.htm
Dustin, Elfriede. 1999. “Lessons in Test Automation.” Software Testing and Quality Engineering (September): 16-22.
http://www.stickyminds.com/sitewide.asp?ObjectId=1802&ObjectType=ART&Function=edetail
Fewster, Mark and Dorothy Graham. 1999. Software Test Automation, Addison-Wesley.
Groder, Chip. “Building Maintainable GUI Tests” in [Fewster 1999].
Hendrickson, Elisabeth. 1999. “Making the Right Choice: The Features you Need in a GUI Test Automation Tool.” Software Testing and Quality Engineering Magazine (May): 21-25.
http://www.qualitytree.com/feature/mtrc.pdf
Rational Unified Process raises everyone’s hope yet often confuses testers to follow it. IBM Rational Tools are used extensively but there is no documentation available, stating when and how to use Rational Testing tools in different RUP phases i.e. Inception, Elaboration, Construction and Transition. It frustrates and disappoints testers. Although Rational Unified Process provides “Tool Mentors” to explain the usage of all the tools but still there is no direct mapping between usage of the Rational Testing Tools and Testing Life Cycle which explain the responsibility of the Testing team through out the SDLC in different Iterations.. This paper presents the guidelines which will suggest the role and responsibility of testing team in different phases of Rational Unified Process. Follow these guidelines as you staff, tool, or schedule your test automation project, and you will be well on your way to success.
A Fable
I have worked in many testing projects, big and small. I have seen different projects following RUP model and I have personally talked to many testing teams. I am presenting this paper to avoid these problems faced by them. But first we need to understand it. Let me illustrate with a fable.
Once upon a time, we have a big project following RUP for a very important client. Client provides the team with Rational Testing Tools. Sujata is a Test Manager; she pulls the experienced testers. Everyone is very excited as they are going to work with best tools in the market and no more excel sheet to write test cases, track defects and to generate report.
Testers have been trained on Rational Unified Process. Inception Phase, project starts and lot of testing tools available like ClearQuest, RequisitePro, Robot and Test Manager. Testers know how to use tools, thanks to Sujata once again for sending them for Rational Tools Training before putting them into project. They know Rational Unified process too but they don’t know how to use different tools in different phase (Inception, Elaboration, Construction and Transition).
They use RequisitePro for requirement understanding, test manager for writing test cases but still not sure when to use them and in which phase. Elaboration phase comes followed by construction phase and finally transition phase and Sujata realizes that nothing is in place. All the requirements are not documented, changes are not documented, test cases mapped to requirement have become obsolete, automation was pushed too late, and version was not maintained after a certain period.
Sujata couldn’t find the problem, she provided the best training to the best testers and they used the best tools in the market but still they couldn’t manage the project. She quits and hand over to Ashish, new test managers. Project is a failure.
That's my fable. Perhaps parts of the story sound familiar to you. But I hope you haven't seen a similar ending. This paper will suggest some ways to avoid the same fate
The Problem
This fable illustrates several problems that plague testing project following RUP:
What to use when?
This is the major problem which hinders the success of the projects and many times result in total disaster. In the worst case, testers don’t know RUP, usage of automation tools which stops project from doing well. If we consider the best case as mentioned by me in the previous section that even if testers are trained in RUP and usage of testing tools like Robot & TestManager still project might not succeed because they should know when to use Robot, when to use Test Manager and when to use other testing tools like ClearQuest and ClearCase in different phases in RUP.
How to use?
Other big question is the functionalities and features of the tools those should be given more focus in different phases of RUP.
Who should use?
This question can be answered if the tester knows what he is supposed to do and what manager is responsible of doing.
Guidelines- usage of Rational Testing Tools in a RUP Project
This paper will be organized by the normal steps that we all use for our testing projects, making special notes of the considerations and challenges that are particular to RUP.
1. IBM Rational Administrator
2. IBM Rational ClearCase
3. IBM Rational RequisitePro
4. IBM Rational TestManager
5. IBM Rational Robot
6. IBM Rational ClearQuest
All the above mentioned tools are explained with respect to their usage in different phases of RUP. I haven’t covered Rational Rose as I wanted to keep the focus restricted to the tools which are used in software testing.
I am giving a brief introduction about RUP phases for the audiences who are new to RUP.
Inception – Focus of this phase is the understanding the scope of the project.
Elaboration: The architecture as wells as the requirement of the products being built must be understood by the end of the phase.
Construction & Integration: Product must be constructed in this phase.
Transition –The product must be rolled out to customers in this phase.
Step 1: IBM Rational Administrator
a) Phase: Inception
During this phase, a shared repository is created using either MS-Access or SQL Sybase Central. A new project folder is created in the share with UNC (Unified Naming Convention). The project should be UCM (Unified Change Management) Enabled to ensure integrity with Rational ClearCase for Configuration Management. Rational Administrator provided the platform to integrate all the rational tools together to achieve end-to-end mapping.
Activities:
Create a project
Connect to a project
Create a test datastore
Create integration between products
Decide whether to associate the administrator project with UCM project.
Fig.1. Mapping of RUP phases to Rational Testing Tools
b) Phase: Elaboration
During Elaboration phase, Rational Administrator is used to create new users with different access rights depending upon the requirement of the project
c) Phase: Construction & Integration
Like Construction phase, Rational Administrator is used for managing users and access rights. It also maintains the integrity between different tools used.
d) Phase: Transition
Like Elaboration & Construction phase, Rational Administrator is used for managing users and access rights. It also maintains the integrity between different tools used.
Step 2: IBM Rational ClearCase
a) Phase: Inception
ClearCase provides life cycle management and control of software development assets. With integrated version control, automated workspace management, parallel development support, baseline management, and build and release management, Rational ClearCase provides the capabilities needed to create, update, build, deliver, reuse and maintain business-critical assets.
Fig.2.Rational Clear Case Workflow
Activities:
Create the configuration management repositories that represent the subsystem defined by your architecture.
Import existing files and directories into the repositories to create an initial set of development configuration items.
Enable your ClearQuest schema to work with UCM
Enable your clear case project to work with ClearQuest.
Set Policies for new projects
b) Phase: Elaboration
During this phase too new assets created are base lined and updated. When new test cases are created using Test Manager and new scripts are created using Robot are base lined using UCM concept.
Activities:
Updating your project work area using Rational ClearCase.
Checking in and checking out configuration items.
Delivering your work with ClearCase
Promoting project baselines using Rational ClearCase
Comparing baselines using Rational ClearCase.
Fig.3.Rational Clear Case detailed diagram
c) Phase: Construction & Integration
Out-of-the-box integration of Rational Test Manager, RequisitePro and Rational ClearQuest with Rational ClearCase LT delivers full base lining capabilities to accommodate the natural evolution of a project, moving from one build or release to the next.
d) Phase: Transition
In transition phase, all the artifacts are delivered and existing one are moved to baseline to maintain version control.
Step 3: IBM Rational RequisitePro
a) Phase: Inception
During this phase RequisitePro is mainly used for Managing Requirements. Inception phase focuses mainly on managing the requirements collected.
“Manage Requirement” is one of the best practices in RUP. Research from Standish Report shows 31 % of projects gets failed because of poor requirements managements, incorrect definition of the requirements from the start of the project and poor requirement management throughout the project life cycle.
Requirements involve the transitions of stakeholder requests into a set of key stakeholder needs and system features. These are detailed into specifications for functional and non-functional requirements.
Traceability helps us to assess the project impact of a change in requirement, impact of a failure of a test on requirements, manage the scope of project, verify the implementation of requirements and manage changes.
With IBM Rational RequisitePro there is little to no learning curve as requirements can be created and updated directly in Microsoft Word, a familiar environment for most people. Documents contain requirements in addition to contextual text that is vital for understanding requirements.
IBM Rational RequisitePro is integrated with IBM Rational Rose XDE, IBM Rational TestManager, IBM Rational SoDA, IBM Rational Unified Process and IBM Rational ClearQuest, as well as Microsoft Project. Integrated tools provide a smooth workflow and eliminate errors due to requirement discrepancies across tools. Integrations keep the entire project team in sync and improve team productivity. Easy access to requirements from other tools encourages all team members to view the requirements before starting their work.
IBM Rational RequisitePro support for traceability allows you to easily set up and track relationships between requirements to verify whether high-level requirements are associated with detailed software requirements. Querying these relationships provides coverage analysis to ensure completeness and to make sure time is not wasted building irrelevant functionality.
With IBM Rational RequisitePro, when traceability is established between requirements, and one of the requirement changes, suspect links automatically appear. This allows you to identify other requirements that may be affected by a change.
Fig.4.Rational RequisitePro – Shows Coverage Analysis, Features and Vision, Glossary, Impact Analysis, Supplementary Requirements, Use Cases and Requirement Management Plan
Activities:
Setting up RequisitePro for a project.
Adding Templates to your Rational RequisitePro project.
Capturing a common vocabulary.
Developing a vision using Rational RequisitePro project.
Detailing a Use case.
Detailing a Business use case.
Reviewing Requirement.
b) Phase Elaboration
During Elaboration phase, RequisitePro is used to set priorities for the Requirement and the attributes for the requirements to be implemented.
In the Elaboration phase, we identify system imperatives and requirements. The product we were developing is part of a larger offering package whose products must comply with system imperatives. Although these requirements did not always add value to our product or our specific client base, we had to invest the time and resources required to implement, test, and document compliance features. This meant cutting our feature list, which required another round of evaluations. At the end of Elaboration, we enter the remaining requirements into IBM® Rational RequisitePro®, our requirements management tool.
When using IBM Rational Rose XDE, suspect links are displayed between changing requirements and their associated design elements, so that it is clear what parts of the system are affected. Understanding the impact of change allows you to more accurately assess the potential cost in time and money of changing requirements.
With IBM Rational RequisitePro, defining attributes like priority, difficulty, and cost and having the ability to sort and filter requirements based on these attribute values allows project managers to objectively identify the highest value requirements
Having carried out the risk-to-use-case mapping for all of the technical risks, we can remove use cases without arrows as candidates for development during Elaboration; we can safely postpone these risks until the Construction phase.
Then we can look for the risk and identify all the intersecting use cases; implementing any of those use cases will attack that risk. If we continue this exercise for each risk, it soon becomes apparent that we can attack a number of high risks by implementing just a couple of use cases during Elaboration.
However, remember that earlier we said the goal of Elaboration is to implement the smallest amount of functionality that will confirm whether the high-priority risks we identified either
Activities:
To set priorities for requirements and attributes.
Analyzing the impact of changes by showing suspect links.
Assessing the potential cost and time required for requirement changes.
Sorting and filtering requirements.
Attack requirements with technical risk first.
Create and Manage Traceability
Fig.5.Rational RequisitePro shows traceability between Features and Use Cases
c) Phase: Construction & Integration
Out-of-the-box integration with Rational RequisitePro ensures seamless traceability between requirements and test cases, resulting in clear test coverage metrics and sensitivity to requirement modification.
d) Phase: Transition
In transition phase, progress reports can be generated using requisite pro to measure requirement coverage and traceability between the requirements.
Step 4: IBM Rational TestManager
a) Phase: Inception
Use TestManager in this phase to design test cases and analyze test logs to do POC (Proof of Concept) for the very few requirements to be implemented. We will be covering this tool exclusively in Elaboration, Construction & Transition Phases.
Activities:
Creating a test plan
b) Phase: Elaboration
Rational TestManager, which ships with Rational Robot, is the central console for test activity management, execution and reporting. Built for extensibility, it supports everything from pure manual test approaches to various automated paradigms including unit testing, functional regression testing, and performance testing.
Rational TestManager is meant to be accessed by all members of a project team, ensuring the high visibility of test coverage information, defect trends, and application readiness.
Establish and manage traceability: Requirements are linked to test cases, ensuring proper test coverage. In addition, suspicion analysis ensures that when requirements change, test cases traced to the requirement are automatically flagged as possible candidates for modification.
Rational TestManager puts the test team in control of the testing effort, focusing their effort by automating and simplifying crucial tasks. The tool allows teams to start from, keep track of, and test all the required functionality of an application, helping to ensure no critical business requirements go untested.
Activities
Test Folder Creation
Test Case creation
Test Case Design
Associating test inputs and attaching External documents are given more emphasis.
Test implementation both Manual and Automated is done for the requirement in scope for elaboration phase.
Fig.6.Create a Test Plan, Test Case Folder and Test Cases under that in hierarchical fashion
c) Phase: Construction & Transition
Support for all test types: Plan, manage and execute functional, performance, manual, integration, regression, configuration and component testing from the same interface.
Support for local and remote test execution: Run tests on the local machine or on remote machines in the test lab. Parallel test execution is limited only by the number of system resources at your disposal.
Detailed test evaluation: An integrated log viewer constructs a log for each test run, including test status and environmental information.
Out-of-the-box integration with Rational ClearQuest enables in-context defect submission directly from the TestManager Log Viewer, automatically porting test run data into the defect report to ensure accuracy.
d) Phase: Transition
Test manager is also used to generate coverage reports to analyze the testing effort and progress for the current phase or iteration. Test plan can be exported to a parcel file and can be delivered as an artifact.
Step 5: IBM Rational Robot
a) Phase: Inception
User Robot to create scripts for the requirements implemented in Inception phase as a part of POC (Proof of Concept). Even Robot will be explored in the subsequent sections.
Activities:
Setting up the test environment in Rational Robot
POC (Proof of Concept)
b) Phase: Elaboration
Test Scripts are created for the requirements covered in elaboration phase. Test Execution is carried out for the same.
Identifying and developing reusable parts is difficult. It is during the iterations in the elaboration phase that common solutions for common problems are found. This is the phase were many reusable functions or procedures are written in the common libraries to be used effectively in subsequent iterations.
c) Phase: Construction & Integration
In construction and integration phase, Rational Robot is exclusively used to execute regression test scripts (both manual as well as automation scripts) to do regression testing and new test scripts are developed for the new requirements being implemented in this current release so that the same can be used in the subsequent iterations to perform regression testing. As we know majority of the requirements are implemented in this phase so for these requirements automation is only possible in this release.
IBM Rational Robot can be used to automate and perform regression, functional and configuration testing for e-commerce, client/server and ERP applications. It's used to test applications based upon a wide variety of user interface technologies, and is integrated with the IBM Rational TestManager
I am listing down the advantages of using Rational Robot for functional testing:
Simplifies configuration testing.
Tests many types of applications
Ensures testing depth
Tests custom controls and objects
Provides an integrated programming environment
Helps you analyze problems quickly
Enables reuse
Fig.7.Create modular test scripts and include the VP’s and libraries used (depends on the automation framework being used)
Activity: Creating Test Suite
Implementing each Test Suite, and managing all subsequent changes to it
Ensuring that the Test Suite accurately reflects the test idea being realized
Ensuring that the Test Suite is implemented according to defined standards so as to be compatible and maintainable with other Test Suites, and with any Test Scripts it is dependent on.
Ensuring that the Test Suite makes reasonably efficient use of the available resources
d) Phase: Transition
In transition phase, all the scripts developed previously are verified by grouping them into shell scripts or as a suite and are executed in sequence to verify the functionalities before delivering the scripts.
Test data is exported from Test manager and test logs are also taken from test manager to deliver along with the suite, shell scripts and scripts.
Step 6: IBM Rational ClearQuest
a) Phase: Inception
During this phase Schema Repository and Defect Database are created using Rational ClearQuest Maintenance Tool and Rational ClearQuest Designer respectively.
Activities:
Establishing the change request process.
Defining change and review notifications.
Fig.7.Showing defects or change requests in Rational ClearQuest
b) Phase: Elaboration
Defects are raised based on the test logs generated by either executing manual or automation scripts.
c) Phase: Construction & Transition
Rational ClearQuest is also used exclusively in this phase because most of the requirement can only be tested only after implementation. In this release manual and automation scripts are executed to uncover defect in the application and they are logged in ClearQuest for tracking and closure. Once all the defects are raised in ClearQuest, Development team undertakes all valid defects and after fixing delivers a new build or iterations. If the defects rose in ClearQuest is a change request which is enhancement or change in requirement, automation script need to be modified accordingly.
d) Phase: Transition
Reports are generated to find all the change requests still not being resolved.
Depending upon the priority and severity of the change request, appropriate decision is taken to go for release or to go for one more iteration. If team wants to go for release then all the defects should be moved to different buckets to have a track like deferred or waived. ClearQuest changes are also integrated with RequisitePro requirements.
What are the top things you should do?
• Get an expert in RUP & automation consultants work together. A RUP expert can provide guidance in the usage of RUP in different phases of software testing.
• Integrate all the tools to get the maximum benefits: Make end to end implementation of all the rational tools from RequisitePro to ClearQuest with the help of Rational Administrator to derive the maximum benefits and cost cutting in long run.
• Usage of Right tool at the Right time: It’s very important to understand the significance of using the right tool at the right time (RUP phase) to avoid rework and maintain all the information right in place.
• Adopt RUP best practices and be guided by the spirit of RUP. For success, you need to be guided by the spirit of RUP and you want to follow RUP best practices for the best possible outcome.
• Generate different Metrics: Using this approach you can get different reports (statistics) for development and test managers using Rational Test Manager, ClearQuest and Rational RequisitePro, which can help to determine the correct progress of the project and the state of the artifacts at any point of time.
Conclusion
In conclusion, I hope that this article will spur "inclusive thinking" among software testing organizations, encouraging them to practice RUP with respect into Software testing right from the beginning -- rather than treating them as an afterthought late in the development lifecycle. We also hope that tool vendors and third-party suppliers for RUP and other development frameworks will begin embedding an integrated approach for accessibility into their products.
Testing team should be aware of the usage of different rational testing tools in all phases of RUP. If the project is following RUP where requirements are delivered in multiple phases and multiple iterations are expected, using these tools efficiently and effectively can lead to smooth and successful completion of project. This paper can be treated as a guideline document by testing teams working in a RUP project to know what to do and when to do in different phases of RUP.
For all software development organizations, we believe that the approach we describe in this article will ultimately increase productivity and application quality. In making products that are accessible and practical for all potential users, progressive organizations can take advantage of new market opportunities and gain a strong competitive edge.
Reference
Rational Unified Process
Bach, James. 1996. “Test Automation Snake Oil.” Windows Technical Journal, (October): 40-44.
http://www.satisfice.com/articles/test_automation_snake_oil.pdf
Marick, Brian. 1998. “When Should a Test Be Automated?”. http://www.testing.com/writings/automate.pdf
Kaner, Cem. 1997. “Improving the Maintainability of Automated Test Suites. http://www.kaner.com/lawst1.htm
Dustin, Elfriede. 1999. “Lessons in Test Automation.” Software Testing and Quality Engineering (September): 16-22.
http://www.stickyminds.com/sitewide.asp?ObjectId=1802&ObjectType=ART&Function=edetail
Fewster, Mark and Dorothy Graham. 1999. Software Test Automation, Addison-Wesley.
Groder, Chip. “Building Maintainable GUI Tests” in [Fewster 1999].
Hendrickson, Elisabeth. 1999. “Making the Right Choice: The Features you Need in a GUI Test Automation Tool.” Software Testing and Quality Engineering Magazine (May): 21-25.
http://www.qualitytree.com/feature/mtrc.pdf
Iterative Automation Process using Rational Unified Process
Test automation is mantra of success in today’s IT world and is used as a strong qualifier to win new projects yet often doesn’t add much value to client. Although automation promises to save cost and effort dramatically but they rarely succeed. Project where RUP is followed, requirements are implemented in phased manner, automation can be very fruitful but it is only possible if RUP is understood and if the same is applied to automation too. Testing team should follow ATLM (Automation Testing Life-cycle Methodology) along with Rational Unified Process. The key is to start automation right from the inception phase and to continue till transition to get ROI. This paper presents the process and guidelines to be followed by Managers and Automation Test Leads. Follow these steps as you staff, tool, or schedule your test automation project, and you will be well on your way to success.
A Fable
I have seen lot of many test automation problems. I have worked in many automation projects, big and small. I have talked to many people from many other projects. I am presenting this paper to avoid these problems faced by them. But first we need to understand it. Let me illustrate with a fable.
Once upon a time, we have a pilot test automation project for a very important client. Client is very quality conscious and provides the team with Rational Testing Tools. Development team has decided to follow Rational Unified Process. Mr. Rakesh is a Test Manager who hires experienced testers with good programming knowledge. Everyone is very excited as they are going to work with best tools in the market and no more excel sheet to write test cases, track defects and to generate report.
Project starts and Rational RequisitePro is used for requirement management, development team uses Rational Rose to design and ClearQuest for configuration management. Testing team doesn’t have any clue what so ever what going on. Then Rakesh realized and instructed team to start writing test cases using Test Manager. Testers found TestManager as great tools and wrote some test cases based on the requirements. By the time development team releases the first build to test. Testing has started executing test cases manually and some how ended up.
Project has moved to next phase, elaboration and they are designing the data model along with coding for critical modules. Testing doesn’t know what to do it in this phase and they are waiting for the build to come. Testing team expands test cases in Test Manager and leave early to house.
Project moves to Construction phase and build is given again with most of the functionalities implemented and testing time is finding difficult to manage time efficiently. Defects are found and development team starts fixing the defects. Then Rakesh instructs all the testing members to automate the application and they do some quick record and playback. After spending lot of effort, testing team is ready is record and playback suite. Again, build comes and testing team fails to run even a single script properly, comes to the conclusion that slight maintenance is required but once again they fail and then again fail.
Client feels organization is incapable of doing automation; Rakesh feels that team is inefficient, team feels testing tool is not up to the mark. Rajesh asks for release, testing team members feel automation is of no use, manual testing is better. Project is a failure.
That's my fable. Perhaps parts of the story sound familiar to you. But I hope you haven't seen a similar ending. This paper will suggest some ways to avoid the same fate
The Problems
This fable illustrates several problems that plague test automation projects:
Automation at last moment: There is a general trend and people think that automation should come at the end of the project. Managers don’t focus on automation right from the initial phase.
Automating only regression candidates: This is a myth that only automation candidates should be automated and regression testing come late during the project. It pushes the automation to the last moment.
Short of time: Testers often feel that time given is insufficient for testing. It is because testing is manually all the time and automation comes into picture only in the end very near to deadline.
Automation takes less time: Managers think that automation takes less time and can be done in couple of days and then time can be saved. Reality is automation takes more time than manual testing and is only useful in long run.
Testing RUP project is no different: Managers think RUP project is just any other development model like Waterfall where product will be given as a whole.
Any tool will do: Managers don’t consider whether tool can be used for the AUT, they just go by the popularity and later realize that it is not good.
Follow the Rules of Iterative Automation
This paper will be organized by the normal steps that we all use for our test automation projects, making special notes of the considerations and challenges that are particular to test automation:
1. Test Tool Acquisition
2. Test Framework Selection
3. Test Planning and Development
4. Test Execution
5. Analysis & Assessment
Abstract
Step 1: Test Tool Acquisition
Role: Test Designer & Test Manager
Activity: Define Test Environment Configuration
Artifact: Test Environment Configuration
Phase: Inception
Test Tool Acquisition is an activity which should be done early in the Inception phase. During Inception phase, very few requirements are implemented to verify the feasibility. This is the phase when the automation tools should be evaluated and POC should be done. In this paper, out of other tools like QTP, Winrruner and Robot, Rational Robot came out as winner and was selected to automate the application-under-test.
Step 2: Test Framework Selection
Role: Test Designer
Activity: Select Automation Framework
Artifact: Test Automation Architecture
Phase: Elaboration
It is very important to select an appropriate automation framework. Different options available are Record/Playback, Module centric framework, Data-Driven framework and Keyword-Driven framework. It is preferable to start analyzing the option right from the inception phase. POC done in the inception phase will help Test Designer to select the framework which best suite the need. No. of iterations planned, size of the project and nature of the requirements are also considered as valuable inputs while selecting an automation framework.
Elaboration phase is the one when automation framework selected is frozen and making any changes after this can be risky and can involve lot of rework. In this feasibility of the framework is validated by automating the most critical requirements in Elaboration phase. A composition of various test automation design and implementation elements and their specifications that embody the fundamental characteristics of the test automation software system.
This artifact is particularly useful when the automated execution of software tests must be maintained and extended through multiple test cycles. This artifact is most useful as single artifact per project.
Step 3: Test Planning & Development
Role Test Analyst
Activity: Identify Test Ideas, Identify targets of test
Artifact: Test Idea List, Test Cases, Test Data
Phase: Elaboration
Test Analyst identifies the test cases to be automated and selects the automation candidates based on the following criteria:
Identifying and defining each Test Case, and approving all subsequent changes to it.
Ensuring that changes are communicated to affected downstream roles.
Ensuring that sufficient Test Cases have been identified to provide satisfactory evaluation of the Target Test Items.
Ensuring that sufficient detail has been provided to implement and conduct the test.
Managing and maintaining appropriate traceability relationships.
Managing the appropriate scope of the Test Cases in a given iteration.
Test Analyst is also responsible for Test Data:
Identifying potential data sources.
Gathering basic candidate Test Data.
Verifying the completeness, fitness for purpose and accuracy of the Test Data.
Test cases to be automated are present in Rational Test Manager and this activity is generally completed in Elaboration phase. Once all the test cases to be automated and test data is ready, automation can be done in full fledge manner in construction phase.
Role Tester
Activity: Implement Test, Implement Test Suite
Artifact: Test Script
Phase: Construction
This is most important activity which is carried out exclusively in Construction phase but you must have read in earlier sections that POC is done in inception phase and critical test cases are automated in Elaboration phase. Henceforth, automation is done for other automation candidate including the most important regression set of test cases.
Identifying and defining each Test Script, and managing all subsequent changes
Ensuring the Test Script accurately reflects the required test, identified by one or more a Test Ideas or defined in one or more Test Cases
Ensuring the Test Script is implemented according to defined standards to be compatible and maintainable with the other Test Scripts
Ensuring the Test Script makes reasonably efficient use of the available resources
Developing the Test Script with a focus on economy of effort and identifying opportunities for reuse and simplification
Developing the Test Script so that it can be used as part of a Test Suite
Step 4: Test Execution
Role Tester
Activity: Execute Test Suite, Analyze Test Failure
Artifact: Test log
Phase: Construction & Integration
Though script execution is something which even at Inception and Elaboration phase too but in construction phase all the scripts automated are executed as suites and overall functionality of the application is validated.
Different suites can be executed like BVT Suite which can be ran to do smoke testing and log can be used to take the decision to accept or reject the build. Regression suites can be ran to ensure that there is no regression impact of the fixes/changes made to the application.
Ensuring the accurate recording of the observed outcome of each test executed in the test cycle.
Ensuring the Test Logs are uniquely and accurately identified, and stored against the correct test cycle or test run.
Actively monitoring for anomalous and erroneous occurrences in the Test Log, and taking appropriate recovery and reporting actions.
Step 5: Analysis & Assessment
Role: Test Analyst
Activity: Determine Test Results
Artifact: Test Results
Phase: Construction, Integration & Transition
This activity is carried out exclusively in Construction phase and elaboration phase. The test logs are analyzed in Rational Test Manager and defects can be raised in ClearQuest.
If the defect raised is because of mistake or miscommunication, the same is fixed and the script is ran again in next iteration but if the failure was because of desirable change / new requirement / change in requirement, then script is modified and ran again in the same iteration and again test logs are analyzed.
Reviewing Test Logs and Change Requests
Actively monitoring for anomalous and erroneous occurrences in the Test Log, investigating and reporting a conclusion
Ensuring the accurate analysis of the observed outcome of each test conducted in the test execution cycle
Ensuring the Test Results are uniquely and accurately identified and recorded against the correct test execution cycle
Role: Test Manager
Activity: Assess and Advocate Quality, Assess and Improve Test Effort
Artifact: Test Evaluation Summary
Phase: Integration & Transition
In construction and transition phase, test manager performs the following activities:
Reviewing the Test Results, change request statistics, and coverage statistics.
Reviewing important Change Request and Issue details.
Presenting an accurate and fair assessment of the software based on the defined Evaluation Mission.
Overall Iterative Automation Process
An iterative process facilitates reuse of project elements because it is easier to identify common parts as they are partially designed or implemented instead of identifying all commonality in the beginning.
Iterative automation process includes the following steps:
Iterative automation has the following advantages:
1. The automated test suite should be executed for all the iterations. Doing this is very important to “Continuously Verify Quality” which is one of the best practice followed by RUP.
2. Doing automation in an iterative way allows flaws to be detected and resolved earlier in the product life cycle. It saves significant amount of time spent to perform manual testing for each and every build in each and every release.
3. Iterative automation provides better view of the quality and helps in correcting the major issues without jeopardizing target costs and schedules.
4. We cannot stop change from being introduced into our project. However we must control how and when changes are introduced into project artifacts, and who introduces the changes. Unified Change Management (UCM) is Rational Software’s approach to managing change in software system development, from requirement to release. “Manage Change” is one more important best practice followed by RUP.
5. Better Overall Quality: Automation scripts developed from an iterative process will be of better overall quality than are the products that result from a conventional sequential process. The script will have been tested several times, improving the quality of testing. At the time of delivery scripts will have been running longer.
6. Stage Containment: Iterative script development helps to contain the defect in the same phase and doesn’t let it go to the last phase. Design bugs can be identified in the elaboration phase and developers don’t need to wait for construction phase to find out design issues.
Conclusion
Automation should be iterative in nature and should be performed right from the beginning of the project. If the project is following RUP where requirements are delivered in multiple phases and multiple iterations are expected, automation can be very handy in reducing the testing effort and cost to a great extent. The effort saved can be better utilized in other areas. This paper can be considered as a reference document by automation testing teams to know what to do and how in different phases of RUP.
Reference
Rational Unified Process
Kaner, Cem. 1997. “Improving the Maintainability of Automated Test Suites.” http://www.kaner.com/lawst1.htm
Fewster, Mark and Dorothy Graham. 1999. Software Test Automation, Addison-Wesley.
A Fable
I have seen lot of many test automation problems. I have worked in many automation projects, big and small. I have talked to many people from many other projects. I am presenting this paper to avoid these problems faced by them. But first we need to understand it. Let me illustrate with a fable.
Once upon a time, we have a pilot test automation project for a very important client. Client is very quality conscious and provides the team with Rational Testing Tools. Development team has decided to follow Rational Unified Process. Mr. Rakesh is a Test Manager who hires experienced testers with good programming knowledge. Everyone is very excited as they are going to work with best tools in the market and no more excel sheet to write test cases, track defects and to generate report.
Project starts and Rational RequisitePro is used for requirement management, development team uses Rational Rose to design and ClearQuest for configuration management. Testing team doesn’t have any clue what so ever what going on. Then Rakesh realized and instructed team to start writing test cases using Test Manager. Testers found TestManager as great tools and wrote some test cases based on the requirements. By the time development team releases the first build to test. Testing has started executing test cases manually and some how ended up.
Project has moved to next phase, elaboration and they are designing the data model along with coding for critical modules. Testing doesn’t know what to do it in this phase and they are waiting for the build to come. Testing team expands test cases in Test Manager and leave early to house.
Project moves to Construction phase and build is given again with most of the functionalities implemented and testing time is finding difficult to manage time efficiently. Defects are found and development team starts fixing the defects. Then Rakesh instructs all the testing members to automate the application and they do some quick record and playback. After spending lot of effort, testing team is ready is record and playback suite. Again, build comes and testing team fails to run even a single script properly, comes to the conclusion that slight maintenance is required but once again they fail and then again fail.
Client feels organization is incapable of doing automation; Rakesh feels that team is inefficient, team feels testing tool is not up to the mark. Rajesh asks for release, testing team members feel automation is of no use, manual testing is better. Project is a failure.
That's my fable. Perhaps parts of the story sound familiar to you. But I hope you haven't seen a similar ending. This paper will suggest some ways to avoid the same fate
The Problems
This fable illustrates several problems that plague test automation projects:
Automation at last moment: There is a general trend and people think that automation should come at the end of the project. Managers don’t focus on automation right from the initial phase.
Automating only regression candidates: This is a myth that only automation candidates should be automated and regression testing come late during the project. It pushes the automation to the last moment.
Short of time: Testers often feel that time given is insufficient for testing. It is because testing is manually all the time and automation comes into picture only in the end very near to deadline.
Automation takes less time: Managers think that automation takes less time and can be done in couple of days and then time can be saved. Reality is automation takes more time than manual testing and is only useful in long run.
Testing RUP project is no different: Managers think RUP project is just any other development model like Waterfall where product will be given as a whole.
Any tool will do: Managers don’t consider whether tool can be used for the AUT, they just go by the popularity and later realize that it is not good.
Follow the Rules of Iterative Automation
This paper will be organized by the normal steps that we all use for our test automation projects, making special notes of the considerations and challenges that are particular to test automation:
1. Test Tool Acquisition
2. Test Framework Selection
3. Test Planning and Development
4. Test Execution
5. Analysis & Assessment
Abstract
Step 1: Test Tool Acquisition
Role: Test Designer & Test Manager
Activity: Define Test Environment Configuration
Artifact: Test Environment Configuration
Phase: Inception
Test Tool Acquisition is an activity which should be done early in the Inception phase. During Inception phase, very few requirements are implemented to verify the feasibility. This is the phase when the automation tools should be evaluated and POC should be done. In this paper, out of other tools like QTP, Winrruner and Robot, Rational Robot came out as winner and was selected to automate the application-under-test.
Step 2: Test Framework Selection
Role: Test Designer
Activity: Select Automation Framework
Artifact: Test Automation Architecture
Phase: Elaboration
It is very important to select an appropriate automation framework. Different options available are Record/Playback, Module centric framework, Data-Driven framework and Keyword-Driven framework. It is preferable to start analyzing the option right from the inception phase. POC done in the inception phase will help Test Designer to select the framework which best suite the need. No. of iterations planned, size of the project and nature of the requirements are also considered as valuable inputs while selecting an automation framework.
Elaboration phase is the one when automation framework selected is frozen and making any changes after this can be risky and can involve lot of rework. In this feasibility of the framework is validated by automating the most critical requirements in Elaboration phase. A composition of various test automation design and implementation elements and their specifications that embody the fundamental characteristics of the test automation software system.
This artifact is particularly useful when the automated execution of software tests must be maintained and extended through multiple test cycles. This artifact is most useful as single artifact per project.
Step 3: Test Planning & Development
Role Test Analyst
Activity: Identify Test Ideas, Identify targets of test
Artifact: Test Idea List, Test Cases, Test Data
Phase: Elaboration
Test Analyst identifies the test cases to be automated and selects the automation candidates based on the following criteria:
Identifying and defining each Test Case, and approving all subsequent changes to it.
Ensuring that changes are communicated to affected downstream roles.
Ensuring that sufficient Test Cases have been identified to provide satisfactory evaluation of the Target Test Items.
Ensuring that sufficient detail has been provided to implement and conduct the test.
Managing and maintaining appropriate traceability relationships.
Managing the appropriate scope of the Test Cases in a given iteration.
Test Analyst is also responsible for Test Data:
Identifying potential data sources.
Gathering basic candidate Test Data.
Verifying the completeness, fitness for purpose and accuracy of the Test Data.
Test cases to be automated are present in Rational Test Manager and this activity is generally completed in Elaboration phase. Once all the test cases to be automated and test data is ready, automation can be done in full fledge manner in construction phase.
Role Tester
Activity: Implement Test, Implement Test Suite
Artifact: Test Script
Phase: Construction
This is most important activity which is carried out exclusively in Construction phase but you must have read in earlier sections that POC is done in inception phase and critical test cases are automated in Elaboration phase. Henceforth, automation is done for other automation candidate including the most important regression set of test cases.
Identifying and defining each Test Script, and managing all subsequent changes
Ensuring the Test Script accurately reflects the required test, identified by one or more a Test Ideas or defined in one or more Test Cases
Ensuring the Test Script is implemented according to defined standards to be compatible and maintainable with the other Test Scripts
Ensuring the Test Script makes reasonably efficient use of the available resources
Developing the Test Script with a focus on economy of effort and identifying opportunities for reuse and simplification
Developing the Test Script so that it can be used as part of a Test Suite
Step 4: Test Execution
Role Tester
Activity: Execute Test Suite, Analyze Test Failure
Artifact: Test log
Phase: Construction & Integration
Though script execution is something which even at Inception and Elaboration phase too but in construction phase all the scripts automated are executed as suites and overall functionality of the application is validated.
Different suites can be executed like BVT Suite which can be ran to do smoke testing and log can be used to take the decision to accept or reject the build. Regression suites can be ran to ensure that there is no regression impact of the fixes/changes made to the application.
Ensuring the accurate recording of the observed outcome of each test executed in the test cycle.
Ensuring the Test Logs are uniquely and accurately identified, and stored against the correct test cycle or test run.
Actively monitoring for anomalous and erroneous occurrences in the Test Log, and taking appropriate recovery and reporting actions.
Step 5: Analysis & Assessment
Role: Test Analyst
Activity: Determine Test Results
Artifact: Test Results
Phase: Construction, Integration & Transition
This activity is carried out exclusively in Construction phase and elaboration phase. The test logs are analyzed in Rational Test Manager and defects can be raised in ClearQuest.
If the defect raised is because of mistake or miscommunication, the same is fixed and the script is ran again in next iteration but if the failure was because of desirable change / new requirement / change in requirement, then script is modified and ran again in the same iteration and again test logs are analyzed.
Reviewing Test Logs and Change Requests
Actively monitoring for anomalous and erroneous occurrences in the Test Log, investigating and reporting a conclusion
Ensuring the accurate analysis of the observed outcome of each test conducted in the test execution cycle
Ensuring the Test Results are uniquely and accurately identified and recorded against the correct test execution cycle
Role: Test Manager
Activity: Assess and Advocate Quality, Assess and Improve Test Effort
Artifact: Test Evaluation Summary
Phase: Integration & Transition
In construction and transition phase, test manager performs the following activities:
Reviewing the Test Results, change request statistics, and coverage statistics.
Reviewing important Change Request and Issue details.
Presenting an accurate and fair assessment of the software based on the defined Evaluation Mission.
Overall Iterative Automation Process
An iterative process facilitates reuse of project elements because it is easier to identify common parts as they are partially designed or implemented instead of identifying all commonality in the beginning.
Iterative automation process includes the following steps:
Iterative automation has the following advantages:
1. The automated test suite should be executed for all the iterations. Doing this is very important to “Continuously Verify Quality” which is one of the best practice followed by RUP.
2. Doing automation in an iterative way allows flaws to be detected and resolved earlier in the product life cycle. It saves significant amount of time spent to perform manual testing for each and every build in each and every release.
3. Iterative automation provides better view of the quality and helps in correcting the major issues without jeopardizing target costs and schedules.
4. We cannot stop change from being introduced into our project. However we must control how and when changes are introduced into project artifacts, and who introduces the changes. Unified Change Management (UCM) is Rational Software’s approach to managing change in software system development, from requirement to release. “Manage Change” is one more important best practice followed by RUP.
5. Better Overall Quality: Automation scripts developed from an iterative process will be of better overall quality than are the products that result from a conventional sequential process. The script will have been tested several times, improving the quality of testing. At the time of delivery scripts will have been running longer.
6. Stage Containment: Iterative script development helps to contain the defect in the same phase and doesn’t let it go to the last phase. Design bugs can be identified in the elaboration phase and developers don’t need to wait for construction phase to find out design issues.
Conclusion
Automation should be iterative in nature and should be performed right from the beginning of the project. If the project is following RUP where requirements are delivered in multiple phases and multiple iterations are expected, automation can be very handy in reducing the testing effort and cost to a great extent. The effort saved can be better utilized in other areas. This paper can be considered as a reference document by automation testing teams to know what to do and how in different phases of RUP.
Reference
Rational Unified Process
Kaner, Cem. 1997. “Improving the Maintainability of Automated Test Suites.” http://www.kaner.com/lawst1.htm
Fewster, Mark and Dorothy Graham. 1999. Software Test Automation, Addison-Wesley.
Application & Script Independent Framework: The Need For Data Normalization
Test Automation is one of the fastest growing areas and it has been very much successful in attracting the attention of clients, managers etc. Every company spends a considerable amount of money on acquiring the best automation tools and automation experts to make the project successful. On the other hand, world is becoming more and more dynamic and so the requirements. With the evolutions of development models like Agile, SCRUM and RUP, changes in requirements are possible almost at any phase but the do we think the automation frameworks available are good enough to cope up with all these problems. Answer is No. We have to always keep in mind “Requirement Changes are Inevitable”. Most of the automation project ends up with the “Maintenance phase” taking more effort than the automation effort spent initially. Often changes in requirements, in the form of Change Requests / through emails / over phone results into lot of automation maintenance effort. It generally takes 4-5 releases to get ROI on automation, which has a significant impact on the client’s decision to go for automation or not. This process presents the evolutionary automation framework ASIF (Application and Script Independent Framework), which can be followed to solve most of the problems stated above. Framework uses Data Normalization of the test data and objects details (object name, object type and object properties) to minimize the effort of rework whenever there is any change in the requirement or design or business flow. It leverages on various concepts of RDBMS to exploit maximum benefits. Use of database views provides an abstraction and flexibility to the overall architecture. Plug and Play automation library is provided with the framework which can be reused as it is for any other automation project. The key is to make the automation project independent of changes in application \ requirements by enabling automation scripts still continue to work efficiently with least possible maintenance effort. Paper provides the process and guidelines to be followed by Managers and Automation Test Leads. Follow these steps as you staff, tool, or schedule your test automation project, and you will be well on your way to success.
Introduction:
I have worked in many automation projects and have been discussing the automation frameworks used in different companies by different people. Everywhere people claim to use different framework but in the end complain that execution is not smooth, maintenance effort is too much, technical and experienced staff is required, test data changes too often, application is not stable, requirement changes in each releases etc. But don’t our frameworks should answer these questions? Why to use a framework which is inflexible and incapable of handling changes effectively?
The Problems
1. Redundancy: Typically, in the data-driven testing context, different automation consultants create / use test data for their respective functional modules where lot of it might be common with other modules and as a consequence of this maintaining multiple copies of the same data ultimately leads to redundancy. Particularly, in case of updation, all the data pools containing the redundant fields need to be updated.
Data normalization is not present in the data pools as they are stored as excel / CSV / datapool / datatable.
Scenario 1: Consider 500 Data pools being created for 1000 different test cases by different users. In this scenario the users might have unknowingly created many common fields in multiple data pools. If some common objects changes during the next release, the tester needs to revisit all the data pools to change the duplicate objects. This process is time consuming and error prone.
2. Inconsistency: It is generally very difficult to manage a big application that involves millions of test data. This often leads to inconsistency, which is one of the prime reasons for script failures.
Scenario 1:
If the object name, which is used for object identification, is changed or renamed then all the scripts using that object will be aborted as a consequence. Then the tester has to find all the occurrences of the changed object in all the scripts and replace them with the new value, which is a tedious job.
Scenario 2:
If the object type is changed, all the scripts using that object will be aborted as a consequence. Here again, the user has to find all the occurrences of the changed object type in all the scripts and replace them with the new statements, which is very time consuming.
3. Iterative software development: Given today's sophisticated software systems, it is not possible to sequentially define the entire problem at the beginning, design the entire solution, build the software and then test the product at the end. An iterative approach is required that allows for increased understanding of the problem through successive refinements enabling incremental growth resulting in an effective solution over multiple iterations. With every release of a project an executable is produced. As project’s progress and the application grow iteratively, the test suite grows as well.
4. Lack of experienced automation resources: There is a scarcity of automation experts who are technically sound with the testing attitude.
5. Change Requests: Requirements changes dynamically during the later stages of project which affects the scripts automated.
6. Maintenance effort: Generally maintenance effort exceeds the initial effort involved. In the end team members feel it is better to it from scratch rather than doing maintenance. I know many of you must have felt the same while maintaining someone else’s scripts or sometime even your own scripts.
7. Tedious: Automation becomes tedious and difficult to continue when complexity grows with along the functionality. When automation candidates are lengthy and contains too many verification points, it takes a day to automate whereas the same could be done in 30-45 minutes manually.
8. Too many failures: It becomes almost impossible to run the suites uninterrupted as script fails because of small reasons. Unattended playback becomes theoretical. I have often observed people leaving to their home after playing back the suites with the hope to see the test log next day morning but you are right, suite doesn’t continue for long and fails because of some silly mistakes in the code.
9. Difficult to debug: It is often difficult to debug the script written by someone else or even your own script if it done after few days or months of coding. To make a small change, you need to run through 300-400 lines of code if the documentation is not being done.
Solution: Application and script independent framework
ASIF is an automation framework which has been evolved by keeping into mind all the problems mentioned above. It uses the advantages the usage of RDBMS concepts and strongly argues that effort overall effort can be reduced drastically and ROI can be achieved in the first release itself. Another important aspect is that automation doesn’t really need people strong in programming. This method stores all the object properties and test data to be used in a database.
A database schema is designed before creating automation script. Database schema involves inter-dependencies among the data to avoid redundancy. Normalization enables this approach to be much more effective than the traditional approach.
This framework requires the development of base tables, which are independent of the test automation tool used to execute them. These tables are also independent of the common library functions that “drive” the application-under-test. In this case the script will just call the common functions, which will be used to retrieve the data from the database and in turn input the same to the application.
This paper will be organized by the normal steps that we all use for our automation frameworks, making special notes of the considerations and challenges that are particular to test automation:
1. Implementation
2. Advantages
3. Conclusion
4. References
Implementation: ASIF
1. Analysis: Perform size estimation, resource estimation, schedule planning, expected changes and development model. This approach yields maximum benefits especially when the iterative model is used.
2. Identify the automation candidates: Select automation candidates from the entire set of test cases. It is best to automate regression test suites, as they need to be executed for each build, in all the subsequent releases.
3. Identify all the possible screens in the AUT: These are the screens across to be navigated the application to execute the selected test cases.
4. Record object properties for the selected screens: Use tool to capture the object properties like object name, recognition method, and object name etc for the screens within scope.
5. Select an appropriate Database: Make a choice between the databases depending upon the maximum no. of users in team.
6. Design database schema & create base tables for all the selected screens: Design base table, select appropriate data types and field lengths and standard naming convention. Populates tables with the properties captured.
7. Create View by joining required base tables: Design views by joining Base tables, based on the test case flow.
8. Include common library: Common functions are to be used across all the test cases. These functions are independent of the projects and writing them is a one-time activity.
9. Write scripts for test cases: Include Library files and just give calls to common functions in the library. Scripts are restricted to just couple of lines to call reusable functions.
Implementation: Database Design
1. Choice of a database:
a) Less than 4 users
If less than 4 users are going to connect to Automation Database then MS Access is a good choice but it may crash if more than 4 users try to connect to the database at the same time.
b) 4 or More than 4 users:
SQL Server or Oracle can be used to handle multiple connections at the same time without degradation in the performance.
2. Schema Design:
The following activities comprise schema design:
a) Base Tables:
Tables contain Object Names/Object types/Test Data for each screen in the application. The first row of all the base tables always contains Object Names/Object Types.
b) Logical Views
We can design views by joining different tables according to the need of any particular Test Case.
Scenario 1: Change in Application-Under-Test:
If an object name or an object type is changed by a developer in any of the subsequent releases, then the tester has to open the corresponding base table and make the changes only at one place. The changes made to the base tables will automatically get reflected in all the views using the base table and all the scripts will work in exactly the same way they worked in previous releases.
Scenario 2: Change in the Test Cases or flow
A view corresponds to one particular test case and if the test cases are changed then the tester has to just modify the corresponding view and need not to worry about the scripts. Whatever is changed or modified in the view will get reflected in all the joined base tables. This reduces the effort in case a particular step in a test case is being modified or added.
c) DSN (DATA Source Names):
To connect to database, ‘User DSN’ is needed on the machine where script is going to be executed.
Creation of DSN requires three parameters:
Type of the Database
DSN Name
Database Path
Advantages of ASIF
1. Normalization: Designing the database schema with normalization will eliminate the chances of inconsistency and redundancy to a large extent. We can reduce the effort by using the advantages of a RDBMS and can reduce the number of connections in any script. It uses the concept of “Views”, which is nothing but logical connection of tables. In this approach, common sets of tables (database) are going to be used by all the users instead of creating their individual data pools.
2. Application & Script independent:
We must strive to develop a single framework that should grow and continuously improve with future projects that challenge us. This approach makes the test script completely independent of the application. Script creation becomes a one-time effort and reduces maintenance effort at the time when changes are being made.
Fortunately, this heavy initial investment is mostly a one-shot deal. Once in place, ASIF is arguably the easiest of the existing automation frameworks. It provides a very great potential for a long-term success.
Scenario 1: Let us assume that 1000 test cases have the same precondition to login to the application. If the object name for Login is changed then the users need not to touch 1000 scripts but using ASIF they just need to change the object name in the single base table containing information about Login screen and the same will get reflected in all the 1000 scripts.
3. Data storage and Retrieval efficiency
Database can handle large amount of data in an efficient way and retrieval of the data is much faster. In case of changes, data and object names need to change only at one place that makes this approach very attractive. It doesn’t require connecting to more than one data source for the same script.
4. Views & Queries
Views can be created from base tables and each view corresponds to one test case. This makes script very simple to understand as it is fetching data only from one source and that is customized as per the test case. View doesn’t consume space as well as it is just a logical table. In case of any change, views alone will be updated and the entire base tables associated with the views will automatically get updated and vice versa.
Scenario 1: Consider that an object in a screen is stored in a single base table and multiple views are accessing the same base table. Scripts refer to the corresponding logical views to get the required object name, object type and test data. In case if object type is changed then the tester needs to just change that the details of the object, in the corresponding base table and the same will get reflected in all the views and the scripts as well.
5. Logical view and flow of the application
ASIF provides the user with a logical view of the overall application, which is easier for new resources, reviewers and the most importantly the clients.
It requires the design of the database schema, which is a time consuming process and needs good understanding of the database concepts. This can be used for various builds with new change requests. Once it has been designed then the process of making new changes is not very difficult and time consuming thereby making it very useful for the long run.
6. 100 % Reusable Library: ASIF provides the ready-to-use reusable library which can be used in any automation project.
7.Modularity: Scripts are broken down into modules that are independent of each other so that the same module can be called for multiple scripts and also make the debugging of the scripts very easy.
This insulates the application from modifications in the component and provides modularity in the application design. The test script modularity applies the principle of abstraction or encapsulation in order to improve the maintainability and scalability of automated test suites.
Conclusion
ASIF requires relatively more effort for the first time but it will reduce the maintenance time greatly for subsequent releases. This is ideal especially when the development team follows Iterative / Agile / SCRUM / RUP and the regression test case needs to be automated and executed for each and every build to ensure that the functionality is working fine.
Reference
Rational Unified Process
Test Automation Frameworks
http://safsdev.sourceforge.net/DataDrivenTestAutomationFrameworks.htm#KeywordDrivenAutomationFrameworkModel
Introduction:
I have worked in many automation projects and have been discussing the automation frameworks used in different companies by different people. Everywhere people claim to use different framework but in the end complain that execution is not smooth, maintenance effort is too much, technical and experienced staff is required, test data changes too often, application is not stable, requirement changes in each releases etc. But don’t our frameworks should answer these questions? Why to use a framework which is inflexible and incapable of handling changes effectively?
The Problems
1. Redundancy: Typically, in the data-driven testing context, different automation consultants create / use test data for their respective functional modules where lot of it might be common with other modules and as a consequence of this maintaining multiple copies of the same data ultimately leads to redundancy. Particularly, in case of updation, all the data pools containing the redundant fields need to be updated.
Data normalization is not present in the data pools as they are stored as excel / CSV / datapool / datatable.
Scenario 1: Consider 500 Data pools being created for 1000 different test cases by different users. In this scenario the users might have unknowingly created many common fields in multiple data pools. If some common objects changes during the next release, the tester needs to revisit all the data pools to change the duplicate objects. This process is time consuming and error prone.
2. Inconsistency: It is generally very difficult to manage a big application that involves millions of test data. This often leads to inconsistency, which is one of the prime reasons for script failures.
Scenario 1:
If the object name, which is used for object identification, is changed or renamed then all the scripts using that object will be aborted as a consequence. Then the tester has to find all the occurrences of the changed object in all the scripts and replace them with the new value, which is a tedious job.
Scenario 2:
If the object type is changed, all the scripts using that object will be aborted as a consequence. Here again, the user has to find all the occurrences of the changed object type in all the scripts and replace them with the new statements, which is very time consuming.
3. Iterative software development: Given today's sophisticated software systems, it is not possible to sequentially define the entire problem at the beginning, design the entire solution, build the software and then test the product at the end. An iterative approach is required that allows for increased understanding of the problem through successive refinements enabling incremental growth resulting in an effective solution over multiple iterations. With every release of a project an executable is produced. As project’s progress and the application grow iteratively, the test suite grows as well.
4. Lack of experienced automation resources: There is a scarcity of automation experts who are technically sound with the testing attitude.
5. Change Requests: Requirements changes dynamically during the later stages of project which affects the scripts automated.
6. Maintenance effort: Generally maintenance effort exceeds the initial effort involved. In the end team members feel it is better to it from scratch rather than doing maintenance. I know many of you must have felt the same while maintaining someone else’s scripts or sometime even your own scripts.
7. Tedious: Automation becomes tedious and difficult to continue when complexity grows with along the functionality. When automation candidates are lengthy and contains too many verification points, it takes a day to automate whereas the same could be done in 30-45 minutes manually.
8. Too many failures: It becomes almost impossible to run the suites uninterrupted as script fails because of small reasons. Unattended playback becomes theoretical. I have often observed people leaving to their home after playing back the suites with the hope to see the test log next day morning but you are right, suite doesn’t continue for long and fails because of some silly mistakes in the code.
9. Difficult to debug: It is often difficult to debug the script written by someone else or even your own script if it done after few days or months of coding. To make a small change, you need to run through 300-400 lines of code if the documentation is not being done.
Solution: Application and script independent framework
ASIF is an automation framework which has been evolved by keeping into mind all the problems mentioned above. It uses the advantages the usage of RDBMS concepts and strongly argues that effort overall effort can be reduced drastically and ROI can be achieved in the first release itself. Another important aspect is that automation doesn’t really need people strong in programming. This method stores all the object properties and test data to be used in a database.
A database schema is designed before creating automation script. Database schema involves inter-dependencies among the data to avoid redundancy. Normalization enables this approach to be much more effective than the traditional approach.
This framework requires the development of base tables, which are independent of the test automation tool used to execute them. These tables are also independent of the common library functions that “drive” the application-under-test. In this case the script will just call the common functions, which will be used to retrieve the data from the database and in turn input the same to the application.
This paper will be organized by the normal steps that we all use for our automation frameworks, making special notes of the considerations and challenges that are particular to test automation:
1. Implementation
2. Advantages
3. Conclusion
4. References
Implementation: ASIF
1. Analysis: Perform size estimation, resource estimation, schedule planning, expected changes and development model. This approach yields maximum benefits especially when the iterative model is used.
2. Identify the automation candidates: Select automation candidates from the entire set of test cases. It is best to automate regression test suites, as they need to be executed for each build, in all the subsequent releases.
3. Identify all the possible screens in the AUT: These are the screens across to be navigated the application to execute the selected test cases.
4. Record object properties for the selected screens: Use tool to capture the object properties like object name, recognition method, and object name etc for the screens within scope.
5. Select an appropriate Database: Make a choice between the databases depending upon the maximum no. of users in team.
6. Design database schema & create base tables for all the selected screens: Design base table, select appropriate data types and field lengths and standard naming convention. Populates tables with the properties captured.
7. Create View by joining required base tables: Design views by joining Base tables, based on the test case flow.
8. Include common library: Common functions are to be used across all the test cases. These functions are independent of the projects and writing them is a one-time activity.
9. Write scripts for test cases: Include Library files and just give calls to common functions in the library. Scripts are restricted to just couple of lines to call reusable functions.
Implementation: Database Design
1. Choice of a database:
a) Less than 4 users
If less than 4 users are going to connect to Automation Database then MS Access is a good choice but it may crash if more than 4 users try to connect to the database at the same time.
b) 4 or More than 4 users:
SQL Server or Oracle can be used to handle multiple connections at the same time without degradation in the performance.
2. Schema Design:
The following activities comprise schema design:
a) Base Tables:
Tables contain Object Names/Object types/Test Data for each screen in the application. The first row of all the base tables always contains Object Names/Object Types.
b) Logical Views
We can design views by joining different tables according to the need of any particular Test Case.
Scenario 1: Change in Application-Under-Test:
If an object name or an object type is changed by a developer in any of the subsequent releases, then the tester has to open the corresponding base table and make the changes only at one place. The changes made to the base tables will automatically get reflected in all the views using the base table and all the scripts will work in exactly the same way they worked in previous releases.
Scenario 2: Change in the Test Cases or flow
A view corresponds to one particular test case and if the test cases are changed then the tester has to just modify the corresponding view and need not to worry about the scripts. Whatever is changed or modified in the view will get reflected in all the joined base tables. This reduces the effort in case a particular step in a test case is being modified or added.
c) DSN (DATA Source Names):
To connect to database, ‘User DSN’ is needed on the machine where script is going to be executed.
Creation of DSN requires three parameters:
Type of the Database
DSN Name
Database Path
Advantages of ASIF
1. Normalization: Designing the database schema with normalization will eliminate the chances of inconsistency and redundancy to a large extent. We can reduce the effort by using the advantages of a RDBMS and can reduce the number of connections in any script. It uses the concept of “Views”, which is nothing but logical connection of tables. In this approach, common sets of tables (database) are going to be used by all the users instead of creating their individual data pools.
2. Application & Script independent:
We must strive to develop a single framework that should grow and continuously improve with future projects that challenge us. This approach makes the test script completely independent of the application. Script creation becomes a one-time effort and reduces maintenance effort at the time when changes are being made.
Fortunately, this heavy initial investment is mostly a one-shot deal. Once in place, ASIF is arguably the easiest of the existing automation frameworks. It provides a very great potential for a long-term success.
Scenario 1: Let us assume that 1000 test cases have the same precondition to login to the application. If the object name for Login is changed then the users need not to touch 1000 scripts but using ASIF they just need to change the object name in the single base table containing information about Login screen and the same will get reflected in all the 1000 scripts.
3. Data storage and Retrieval efficiency
Database can handle large amount of data in an efficient way and retrieval of the data is much faster. In case of changes, data and object names need to change only at one place that makes this approach very attractive. It doesn’t require connecting to more than one data source for the same script.
4. Views & Queries
Views can be created from base tables and each view corresponds to one test case. This makes script very simple to understand as it is fetching data only from one source and that is customized as per the test case. View doesn’t consume space as well as it is just a logical table. In case of any change, views alone will be updated and the entire base tables associated with the views will automatically get updated and vice versa.
Scenario 1: Consider that an object in a screen is stored in a single base table and multiple views are accessing the same base table. Scripts refer to the corresponding logical views to get the required object name, object type and test data. In case if object type is changed then the tester needs to just change that the details of the object, in the corresponding base table and the same will get reflected in all the views and the scripts as well.
5. Logical view and flow of the application
ASIF provides the user with a logical view of the overall application, which is easier for new resources, reviewers and the most importantly the clients.
It requires the design of the database schema, which is a time consuming process and needs good understanding of the database concepts. This can be used for various builds with new change requests. Once it has been designed then the process of making new changes is not very difficult and time consuming thereby making it very useful for the long run.
6. 100 % Reusable Library: ASIF provides the ready-to-use reusable library which can be used in any automation project.
7.Modularity: Scripts are broken down into modules that are independent of each other so that the same module can be called for multiple scripts and also make the debugging of the scripts very easy.
This insulates the application from modifications in the component and provides modularity in the application design. The test script modularity applies the principle of abstraction or encapsulation in order to improve the maintainability and scalability of automated test suites.
Conclusion
ASIF requires relatively more effort for the first time but it will reduce the maintenance time greatly for subsequent releases. This is ideal especially when the development team follows Iterative / Agile / SCRUM / RUP and the regression test case needs to be automated and executed for each and every build to ensure that the functionality is working fine.
Reference
Rational Unified Process
Test Automation Frameworks
http://safsdev.sourceforge.net/DataDrivenTestAutomationFrameworks.htm#KeywordDrivenAutomationFrameworkModel
Subscribe to:
Posts (Atom)