Correlation between Cyclometric Complexity & Code Coverage while doing White box testing

If you are among those testers (or developers) who have often wondered how cyclometric complexity can practically help me achieve higher code coverage and uncovering max number of bugs by executing minimum no. of unit test then please read on. I have tried to take a very simple example and have focussed on explaining the very basics with data and images only to make sure you can visualize and understand the differences between statement coverage, decision coverage and path coverage and their relationship with cyclometric complexity for once and all.

Pseudocode                                

Read a;

Read b;

if(a>=b)

         print  a

else

          print b

Do x;

if (a ==0)

        print “Zero”

do nothing;

 

Flow chart of the above pseudocode:

 

 

Cyclometric Complexity:

= E-N + 2P  (where E is no. of edges, N is the no. of node and P is the no. of connected components in the above graph)

= 10 – 9 + 2 x 1

= 3

Minimum Test Required for  100 % Statement coverage:

Unit Test #1:

Test data: a = 0, b = –1  (True & True)

Unit Test #2:

Test data = a =0, b = 2 (False & True)

Total  test required for 100 % statement coverage = 2

With the above two test cases we are able to execute each line of code at least once.

Minimum Test Required for 100 % Branch Coverage or Decision Coverage

Unit Test #1:

Test data: a = 0, b = –1    (True & True)

Unit Test #2:

Test data: a =1, b = 2  (False & False)

Total  test required for 100 % branch coverage = 2

Minimum Test Required for 100 % Path Coverage (Basis Path Coverage)

Unit Test #1:

Test data: a = 0, b = –1  (True & True Input)

< same as scenario #1 of statement coverage>

Unit Test #2:

Test Data : a = 1, b =0  (True & False Input)

 

 

Unit Test #3

Test data:  a =0, b = 2 (False & True Input)

Unit Test #4:

Test data:  a =1, b = 2  (False and False Input)

<same as Scenario #2 of branch coverage>

Total  test required for 100 % path coverage = 4  (Which is also 2^2 choices where 2 is n decided by first IF-ELSE)

basically what we have done above can also be shown in the form the grid:

Input\Output	First IF	O/p of First IF	Second IF	O/P of Second IF
a = 0, b = –1	True	Print a	True	Print 0
a = 1, b =0	True	Print a	False	-
a =0, b = 2	False	Print b	True	print a
a =1, b = 2	False	Print b	False	-

Universal Formula:

Branch Coverage <=  Cyclometric Complexity <= Path Coverage

Replacing our values:

2 <= 3 <= 4 (holds good :))

Now what do we conclude by doing all this:

Does 100 % statement coverage ensures no bugs?

Answer is NO. 100% statement coverage just ensure every statement is executed at least once BUT it doesn’t guarantee that all the conditions are tested for different combination of data and hence it is not the most efficient method

Does 100 % branch coverage ensures no bugs?

Answer is again NO. 100 % branch coverage is certainly better than statement coverage and more efficient but it only ensures that each branch is executed at least once but again it doesn’t cover all the possible combinations  (like two Ifs can be tested with 2 branch test case for TT, FF where as total possible combination will be 4 (TT,TF,FT,FF) which are not getting covered here)

Which is the best way of ensuring 100 % code coverage.

Answer is Path Coverage (a.k.a. basis path testing). This is better than both statement and branch coverage. Though this is practically very impossible to cover all permutations and combinations (For ex when there are 2 IF statements, total paths are 2^2= 4 combination and as no. of Ifs in the program  goes to n, no of possible combination goes to 2^n)

Thats where cyclomertic complexity is used to find out the min test to be executed to achieve path coverage and the max test to be executed for branch coverage.

In above example, if the bug was in a condition when one first IF resulted into True (YES) and Second IF resulted into False (NO) then none statement coverage or branch coverage could find that bug and hence cyclometric complexity = 3 shows that adding one more scenario to branch coverage  as TF or FT (other than TT,FF) will help to uncover that bug so minimum 3 test are required to have high probability of uncovering bugs but max 4 test are required (max possible combinations) to ensure no bugs, which generally become impractical with the growing code containing multiple nested if-else and loops.

Here cyclometric complexity helps unit tester decides at least what are the min no. of test that should be executed to cover the scenarios which have high probability of finding bugs out of total possible scenarios of path coverage (Which is 2^n)

PS: I want to thank Renu (an enthusiastic tester) to bring this topic in our discussions which triggered this interest in me to link all these pieces and try giving an overall perspective to testers by taking this simple example.

Feel free to write your comments to have a discussion or to clarify a point if you disagree.

Do you have it in you? Technical skills are no more OPTIONAL skills for Software Testing

 

Words of Caution: When i say “Technical skills”  please don't confuse it with “Test Automation” knowledge only. It more and the less has become a standard in the industry to presume that knowing QTP or RFT is same as having technical skills.

To start with, for a tester to be successful following are the 3 most important components which you should never forget:

In this blog post, we will talk about the second most important component “Technical Skills” which is only next to “Testing Skills & Aptitude” for a tester to be successful in the next decade. 

In last decade or so, testing has been tagged and given the title of an easy profession where non-technical candidates (who aren’t capable of writing or understanding the code) can write some test cases and execute them to find issues. It was considered having domain knowledge to understand the business is sufficient to be a successful tester.

In last 5 yrs, automation has become the talk of the town as it has become a successful marketing tool to win more projects by showing huge ROIs benefits of million of dollars to the customers. We have also witnessed the same trend by the emergence of specialized commercial testing tools by big players like IBM, HP & Microsoft in this arena.

Lot of candidates in testing do have technical skills but they don't hone it or have stopped using them because of the popular perception that “its not test‘s job to write/read code”. The max they do is automation (where record/playback is most popular and only a few percentage (<20%) can truly write automation code to validate the application)

Because testing is considered a second fiddle to Development and the best minds get/shift into development, testing community is lagging seriously on  candidates with strong technical skills.

Test automation is a subset of a broader category “Technical Skills”  and now (more than ever), importance is being given to other technical skills  other than test automation which are becoming mandatory to become a successful  test professionals at organizations.

Following  10 are the skills / activities / tasks where technical skills of a tester is tested

 (you might have read many of these in your testing training / process tutorials but now time has come to start exercising these)

1. Requirement Feasibility:

Don’t get surprised, there is not typo error here. You read it all right that requirement analysis ALSO requires you to have technical skills to understand if the requirements expected by the customer are POSSIBLE or not. Lot of times there are “technology constraints” and unless you understand the underlying technology you run a risk of committing on a requirement which can’t be implemented and you must know that telling customer on the last day wont be such a great idea.

Its not only dev job to call out technical constraints. In fact test can call out if a particular requirement is too unrealistic or ambitious to be implemented if they understand the technology well.

For ex. I can tell the business upfront that a specific multi select tree control is not present in c# and we need to think of an alternative.

2. Scalability Study (Hardware/Software):

Your app used to work fine and then a new functionality was added and then in the next release all of a sudden the app started hanging or the batch job is taking infinite time. If you have had experience something similar to this, you are a victim of “hardware/software constraints” which was not thought in advance.

Lets say the users have increased to million from thousand and you hardware RAM and Processor is not enough to handle so much traffic

OR

Your RDBMS cant handle more than 10000 concurrent connections.

So it is must for you to understand your platform (hardware + software) and their boundary limits so that you can avoid surprised and downtimes in production leading to SLA misses and customer dissatisfaction.

3. Design consideration

When is the last time you were in a design meeting where you contributed to the translation of the requirements into design or more importantly, when you helped avoiding developers to commit a design blunder where let’s say they misunderstood the requirement and the flow was incorrect.

Unless you understand the development models and don't participate in high level and low level designs meetings you wont understand what’s happening “behind-the-scenes”.

It can also help you to write better test cases by understanding the flow of the system and how modules are integrated. Black box testing is not sufficient always when the architecture & the flow becomes complex as not all implementation details are captured in the functional specifications.

Technical design documents should also be evaluated by the testing team where say why Approach A is preferred over other options considered like Approach B or C.

4. Code walkthrough & Inspection

This might give nightmares to many test professions. They just hate looking at the code but believe me you can find lot of bugs by inspecting the code even before the code is dropped to testing. This static testing can save lot of time by avoiding a buggy code to be dropped to test.

You must understand the syntax and semantics of the language being used (C#, java etc) and ask the right questions to find out any memory leak or incorrect loop/condition clause being used in the code

5. Unit Testing & Integration Testing

Okay I wont start a debate by saying test should do unit testing but you will surely come across a situation in your long testing career when you will agree that knowing unit testing help a big deal.

For example, how do you do code coverage now? Unless you know all the inputs/outputs, all the main flows and branches, all the conditions and exception handling code, we cant write good scenarios to exercise critical code paths.

6. Performance tuning

With more and more focus now given on customer satisfaction, performance testing is something now which goes without saying and every customer expects it by default. It is extremely critical now for test team to ensure that the performance is well within customer’s expectations and with high traffic OR volume of data the application would respond within the SLA agreed at the time of signing-off requirements (as part of acceptance criteria).

Unless test teams are familiar with performance counters of the operating system and the the software / technology being used, bottlenecks and potential issues cant be determined.

Also when you want to simulate real-world scenario say to simulate 100000 concurrent users, the knowledge of session management and other technical concepts are must.

Many a times the queries / code are not optimized and it is also important to understand if the code written is efficient or not  and also if there is a better way of writing the code which can improve the product’s performance.

8. Security testing

With the increasing number of virus attacks and the risk involved with losing highly confidential customer data  or any possible frauds, tester’s contribution has become very crucial.

For example:

How can you test your application for SQL injection if you aren’t good at SQL?

or

How would you test your application for exposing any confidential data if you aren’t good at encryption/decryption algorithms or understanding  different masking techniques?

or

How do you ensure which port shouldn’t be open/closed unless you understand the networking concepts well.

These are just some high level examples trust me security testing is much more than that and now constitutes of many modern  & advanced techniques like fuzzing etc

9. Deployment & Prod Support

We are the gatekeepers of the application. It goes from our hand to Support folks for UAT & Production deployment. If we don't understand the configurations setting or environment pre-requisites to run our applications, chances are there that business would lose lot of their precious time waiting for your dev team to come and rescue you.

I am sure you don't want to end up there and its critical to understand the installation and configuration setting, troubleshooting tips and common FAQs

For example, there was a permission issue which was holding your user from using the application and you just don't know how to give the appropriate permission. Believe me it can be very embarrassing and frustrating at times.

That’s your lucky chance to build credibility with business and Support folks and help them get-going without wasting any amount of time.

10. Automation

And just because I haven’t kept at top of the list doesn't mean its less important than any of the above activities. Its the most important activity only next to manual testing but lot has been written about it and there is much more understanding and awareness about test automation in the industry now and hence I am keeping this short.

With this I am closing this post and i hope this will provide a quick checklist to the testers to become ideal testers-of-future.

Testing:: Chosen-By-Profession turned Chose-the-Profession.

It was morning of 12th Nov, 2004 when I found myself standing in a classroom and for the first time it was my back facing the black board and a new batch of  future software professionals sitting in front of me and most of them were as anxious as me (obviously for different reasons). These young professionals fresh out of top colleges in India were campus hired ELTs (Entry Level Trainees) of Cognizant Technology Solutions.

Since that day and the day when I am typing this I have come across many Software Testers (99.9% of them where chosen by the profession and not otherwise) and the questions I have been asked or I could read it on their face is:

• “What am I doing in Software Testing Batch? OR Why am I not present in .Net/Java Batch running in next classroom? Am I not good enough?”
• “Don’t I have required technical skills to be in Development? OR Based on my aptitude have I been put into Testing?”
• “Do people get lesser raise / bonus in Testing over Development?”
• “If I perform well and prove my skills, will they shift me into development?”
• “My elder brother/ cousin/parents/uncle etc told me that testing is not a good profession and its just about validating someone’s one. Is that true?”

Anyone who was present in one of those classrooms is reading this blog would know that raising above questions is definitely no sin and in fact I had the same questions myself as i was “chosen-by-the-profession” too. Please make a note here (I said “I was” and not “I am”).

I would have moved into development way back in 2004 but i CHOSE to stay in Software Testing and that was BY-CHOICE. Just to give you a brief background (no boasting here), I used to and still think that i am amazingly good in development (and yes i did all graduation lab & major projects on my own –be it C, C++,Computer Graphics Linux, Oracle, Java and got 3 scholarships for topping the college)

Times have changed since 2003-04 and now testing has turned out to be a lucrative and easy career option for candidates who find it difficult to crack into development but in next few paragraphs you will read “How I fell in love with Software Testing” and “Why Software testing is as good as Building software”

Software Testing is fun

If as a kid you enjoyed playing with a new electronic gadget more than building sand castles you will know what am i talking about?

or

If you had the curiosity and strong urge to see how watch or a transistor works, you will be able to correlate the itching that you get when you want to understand the new unexplored  & unseen thing

so guys, testing is like seeing a product and then letting yourself go with the flow and try to understand how it works and where it doesn’t?

Human mind always wants to find out what you CANT do with any object in hand. and let’s say getting paid for it. Isn’t it a bonus?

Software Testing is an ART

Software Testing is art and so is software development as well but think of it that specifications and requirements say “what needs to be done” and design done by the architects & modellers says “how is it to be done” so development is left with very LITTLE room for innovation. I have known developers whose jobs are so monotonous and life is limited to just convert the plain English into a programming language code.

Now in Software Testing nobody says “HOW IT NEEDS TO BE TESTED?” :)

To an extent test plan and test strategy covers the scope and approaches to be followed but as a tester you can think innovative and challenge the very requirement or a workflow or a specific behaviour. You have absolute power to think of a undocumented alternative path and ultimately discover a MISSING/EXTRA requirement.

Software Testing is Advocacy

You like fighting for a cause ? You like making sure that justice is done?

You are customer’s advocate and most of the time he might not be even knowing it. Its YOU who has to ensure that justice is served and your customer is getting service / experience for the amount paid. Usability / Accessibility / Security / Performance are few non-explicit expectations which often get ignored by Developers, Managers, Business Analyst and that’s something can make a huge difference in a great customer experience.

Software Testing is Progressive

You want to be an entrepreneur? You want to understand your business?

There are lot of customer-facing professionals / business analyst who have been testers in their early career. The obvious reason why they are so successful is that they GET TO SEE A BIGGER PICTURE. They understand business better than developers because:

1. They clarify requirements / bugs directly or indirectly with customers

2. They understand the whole system (multiple modules and their integration)

3. They don’t get distracted by getting too deep in technology

Software Testing is Respectable

You have the key responsibility. You got to to take the call here. GO / NO-GO?

Customer & Management relies on your work to decide if we are ready to go into the wild and compete with the world.

You are the one who drives the Quality. You are the one who ensure processes are adhered to. You are the gate keeper.  You are the important one here.

Software Testing is Challenge

If you are the one who don't like life to be boring or predictable, then join Testing.  One life is not enough to learn everything in testing. Sometimes reproducing a simple-yet-important bug becomes a nightmare and at times understanding the customer requirement becomes so tough when you are working in an offshore model or when the dev slips and you are the one who has to still deliver quality without compromising on the timelines.

Software Testing is Technical

My favourite. This is exactly opposite to the popular belief that testing don’t need good testing skills. In fact, if you are not technical:

1. You won’t be able to repro many bugs because unless you understand what's-happening-behind-the-scenes you wont understand the problem

2. You wont be able to find the root cause many times. You will just be able to say “there is a problem” but “don’t-know-why”

3. You wont be able to suggest a better solutions / alternative

In the end, software testing is one profession that requires great communication skills, strong technical skills and most importantly a testing mind (passion to learn and understand new things)

Remember, Testing will Test you  FOREVER.

Reduced / Selective Test Data Execution to SAVE Time & Effort - Introduction

Context: Data Warehouse / Business Intelligence (ETL Execution)

Background:

If you have ever involved in DW/BI Testing, waiting for ETL execution to complete won't be an alien subject to you. Its not ONLY frustating to discover ETL fails at Step X after running for say 8 long hours but its ALSO a show stopper for you to continue with your functional testing and directly impacts your efficiency and effectiveness.

If there are performance lags (like low configuration h/w, network bandwidth etc), the ETL job is only going to take longer than expected and often times this adds to higher WAIT Time (a.k.a NON Productive time).

Root Cause:

If dev / test team wants to run the ETL with Production data, which can be really huge (in GBs or sometimes even TBs), due to the volume its obviously takes a long time.
Trade Off is REAL vs SIMULATED data. Testing with REAL data comes with a cost.

Is testing team the only victim?

Answer is NO. Even Dev team has to wait for entire ETL to run to see the changes made in the code while doing integration testing.


Why don't we create Smaller test data ALWAYS?

-> Due to the data integration and transformational complexities, the team has to have very deep domain knowledge of the business rules to create test data which satisfy the ETL conditions (often times lack of this results in ETL failure or ZERO o/p after the run)

-> If the complexity is high (consider data coming from multiple disparate sources) then creating data while maintainig logical relationship can turn out to be quite tedious and time consuming.

Why dont we SELECT REDUCED / SUBSET of Test data from Production?

This is the heart of this blog and we are going to emphasize on this in detail. Though the above sounds simple and the obvious solution but it has been observed that teams dont find it very popular time because of following important reasons:

1. Random sampling of data doesnt work. Many times it will result in ORPHAN RECORDS (where you might end up pulling child records without parents records from the source)

2. ETL might break if the keys are not found.

3. ETL might succeed but can result in ZERO output records as none of the business conditions got satisfied for the the sample data.

4. Business / Domain knowledge is required

What do we recommend to select SUBSET test data with success?

1. Select a pattern:

a) Master Data Driven

b) Transaction Data Driven:

2. Identify Business Conditions

3. Select Sampling Method

4. Copy the SELECTED test data to a test db

5. Modify Test Data to cover Boundary Value conditions

6. Run the ETL

What are the advantages?

1. Save Time

2. Save Effort

3. Save Hardware Cost (Less Data, Less Space and computational resources required)

4. Better Test Coverage (High Requirement Coverage)

5. Better Code Coverage

The subsequent post will focus on the above methods in detail....Please share your feedback and suggestions.