Monday, July 19, 2010

Bhopal’s man-made disaster (Testing Tragedies #2: Learning from past)

Bhopal’s gad tragedy is one of the most sad example of ‘things-went-wrong’ due to human negligence. We primarily blame the company, the management (& rightly so) but one thing we often overlook is that other than human error it is the system (machines) that failed us too. If the quality was given its due importance, we could have possibly averted this. This blog is part of the series where we are trying to learn from our past error in various fields and apply those learning in software testing world. In Churchill’s words “Those that fail to learn from history, are doomed to repeat it.”

 

History

The Bhopal disaster or Bhopal Gas Tragedy is the world's worst industrial catastrophe. It occurred on the night of December 2-3, 1984 at the Union Carbide India Limited (UCIL) pesticide plant in Bhopal, Madhya Pradesh, India

Loss:

Government agencies estimate 15,000 deaths

Bad Quality

Attempts to reduce expenses affected the factory's employees and their conditions. Kurzman argues that "cuts ... meant less stringent quality control and thus looser safety rules. A pipe leaked? Don't replace it, employees said they were told ... MIC workers needed more training? They could do with less. Workers were forced to use English manuals, even though only a few had a grasp of the language”

Timeline, Summary

  • 21:00 Water cleaning of pipes starts.
  • 22:00 Water enters tank 610, reaction starts.
  • 22:30 Gases are emitted from the vent gas scrubber tower.
  • 22:30 First sensations due to the gases are felt—suffocation, cough, burning eyes and vomiting.
  • 1:00 Police are alerted. Residents of the area evacuate. Union Carbide director denies any leak.
  • 2:00 The first people reached hospital Symptoms include visual impairment and blindness, respiratory difficulties, frothing at the mouth, and vomiting.
  • 2:10 The alarm is heard outside the plant.
  • 4:00 The gases are brought under control.
  • 7:00 Immediate death toll raises to 2,259
  • 7:00 A police loudspeaker broadcasts: "Everything is normal".

 

BHOPAL-GAS-TRAGEDY

Flaw:

During the night of December 2–3, 1984, large amounts of water entered tank 610, containing 42 tons of methyl isocyanate(MIC). The resulting exothermic reaction increased the temperature inside the tank to over 200 °C (392 °F), raising the pressure to a level the tank was not designed to withstand. This forced the emergency venting of pressure from the MIC holding tank, releasing a large volume of toxic gases into the atmosphere. The gases flooded the city of Bhopal, causing great panic as people woke up with a burning sensation in their lungs. Thousands died immediately from the effects of the gas and many were trampled in the panic

Incident Logs & My Observations:

Factors leading to the gas leak include:

  • The use of hazardous chemicals (MIC) instead of less dangerous ones
  • Plant's location near a densely populated area

      [raj]: Its a perfect example of poor risk management. We keep saying that test plan should have an important component which should talk about risk identification, risk assessment, risk mitigation and risk contingency but do we really do our part with sincerity.

      When you have alternatives one should always evaluate the worst case impact of each risk and then decide the alternative which is least impactful to the end users as user’s impact in many cases can be much more important than the cost effective   solution which might look better deal to us. In software world we can say that if there is a trade-off then choose the option which is say slightly more time consuming or costly over cheapest solution if it can pose some risk to the end users which can ultimately become too expensive. 

  • Storing these chemicals in large tanks instead of over 200 steel drums. Large-scale storage of MIC before processing

      [raj]: Poor capacity planning. Over utilization of resources can lead to fatigues and  fatal outcomes as it happened in this case. It is crucial to understand the threshold of each resource (software, hardware or humans) and understanding how much you can stretch them before they will collapse or explode. Resources are like a rubber band in your hand which can be stretched to meet your needs as long as you are not going beyond its threshold but if you keep stretching it will break and will hurt your finger real bad because of its rebound speed)

  • Safety systems being switched off to save money—including the MIC tank refrigeration system which alone would have prevented the disaster
  • lack of skilled operators due to the staffing policy
  • there had been a reduction of safety management due to reducing the staff, there was insufficient maintenance of the plant and there were only very loose plans for the course of action in the event of an emergency

      [raj]:  Cost controlling by resource reduction (software, hardware or humans) should NOT be done blindly. This is more applicable now when the economies are struggling, as many times it is seen as the simplistic and the most obvious solution but it can have such adverse impacts as we have witnessed in this tragedy. We all should have moral responsibility toward our users and we should think twice before taking such measures. There can be other ways to save cost but leaving critical systems running in modes where they are not maintained or giving in the hands of people who are not trained to do is nothing less than playing with the life of our users.

  • The MIC tank alarms had not worked for four years
  • There was only one manual back-up system, compared to a four-stage system used in the US

      [raj]: This only emphasizes the need of practicing security measures constantly to ensure that there are enough preventive techniques in place to avoid such accidents. Example in software could be anti-viruses or the scans/tests to be performed on periodic basis to know the possible threats in time.

Learning

      #1 Risk management should not just be done as a formality. Its not something that should only concern management or a project manager.

           It should be one of the most important item which should be raised, managed, controlled and monitored by every team throughout project lifecycle.

     #2 Choices made by humans CHANGE the course of future

         When you do risk management and decide on a choice, always remember that each choice you make can decide the fate of millions of users and hence ALWAYS think of your users while taking a call.

         Risk management is not just about you and  your team, its also about the impact of that risk on your end users.

     #3 Over utilization beyond a limit can backfire. Know your boundaries and use them diligently. Cost reduction is important but constant overloading can instantly fail your system if not used judiciously. 

          for example, In software world, load testing is important but stress testing is even more to understand when your software is going to break so that we can be better prepared for the failures. “Hope for the best; Prepare for the worst :P”

    #4  Cost reduction when done without understanding the impact on the end users can be counter productive.

    #5. For any application which involves direct or indirect physical interaction with users, SAFETY testing is must and needs to be done constantly to avoid accidents. e.g. Healthcare industry, Aviation etc.

    #6  Security testing is extremely critical when there are chances of sabotage in the system. In software it can be a malicious user who is trying to harm the system for his interest or to hurt the users.

    #7  Testability should be built into every system to find out the exact root cause of the behavior when something goes wrong.

         Even today the interested parties haven’t been able to reproduce the exact conditions that resulted into this gas tragedy.

         Only if the software has logging mechanism which can be used to find out the exact sequence of events that occurred before the failure, the root cause can be determined unanimously without leaving any room for speculation.

Thoughts for you

[raj]: If you think we have really learnt from past then read the comments below. Forget about applying the learning into other industries or fields, the very same company’s toxic waste is still lying there waiting to explode anytime again. 

In software testing, I learn that if a product crashes or causes a major loss to the customer, then there will no guarantee that preventive or even corrective actions will taken for sure. History has proved that we humans are so good at repeating our mistakes and suffering from it again and again without really fixing it. If you want to be good tester, make sure making mistake once is acceptable, repeating it again and again is sheer stupidity.

Clean-up operations: Lack of political willpower has led to a stalemate on the issue of cleaning up the plant and its environs of hundreds of tonnes of toxic waste, which has been left untouched. Environmentalists have warned that the waste is a potential minefield in the heart of the city, and the resulting contamination may lead to decades of slow poisoning, and diseases affecting the nervous system, liver and kidneys in humans. According to activists, there are studies showing that the rates of cancer and other ailments are high in the region.

Tuesday, July 13, 2010

Life Saver or Life Taker ? (Therac-25) – Impact of poor testing (Testing Tragedies #1: Learning from past)

This blog is for everyone who wants to know how software testing job touches human lives and why defects in applications such as healthcare cant be ignored.

History:

The Therac-25 was a radiation therapy machine produced by Atomic Energy of Canada Limited (AECL) It was involved with at least six accidents between 1985 and 1987, in which patients were given massive overdoses of radiation, approximately 100 times the intended dose.

Loss: 

Four of the six patients died as a direct result of poor design, coding and testing

 

therac

Company’s Response

After careful consideration, we are of the opinion that this damage could not have been produced by any malfunction of the Therac-25 or by any operator error

[raj]: Only if the company had taken very first of these incidents seriously they could have saved 3 precious lives. Every critical issue which customer finds should be given utmost priority before it becomes much worse.  

Facts

Only One person did the programming for this system and he largely did all the testing.

Therac-25 was tested as a whole machine rather then in separate modules.

[raj]: Yes, that was my reaction too. We left the lives of so many hundreds in the hands of ‘One/ person only . We are humans and to err is humans. Plus we humans are not so good in finding errors in our own work. 

If System and Integration testing is important, so is Unit testing, we can undermine the importance of any of these. They are meant to complement each other. becomes much worse.  

Incident Log & My Observations

Severity 1 Production Defect #1: 
A 40 year old women was receiving her 24th Therac-25 treatment. The machine stopped 5 seconds into the treatment with an error. The technician seeing that "No Dose" had been administered (according to the computer) hit the 'P' key thus proceeding with the dose. This was done a total of 5 times giving the patient 13 000 - 17 000 rads. To give an idea of how much of an overdose this is; a regular treatment is around 200 rads and 1000 rads of radiation to the entire body can be fatal. The patient died 3 months after the overdose.
Severity 1 Production Defect #4: 
The patient required only a small dose and according to the machine that is all he received. Yet again when the treatment was underway and error paused the machine and the technician hit the 'P' key to proceed. A overdose was administered and the man died just 3 months later.
[raj]: Better testability could have warned the technician that the dose had already been delivered. Misleading information and lack of transparency through the system confused him and he went on repeating the procedure again and again which made it fatal.
 Severity 1 Production Defect #2: 
A male required radiation treatment on his back. The machine was set to X-ray mode instead of Electron mode so the technician just used the "cursor up" key up and quickly changed this mistake. However, this only made things worse as a software bug had been mistakenly stumbled upon. While administering the first treatment an error "Malfunction 54" flashed up telling the technician an underdose had been administered. The technician hit the 'P' key and a 2nd dose was delivered. The patient had been given an overdose after the first treatment, and he knew something was wrong, due to the burning sensation he felt in his back. As he attempted to get up the 2nd dose was administered. The technician would have known the man was in pain if the audio and visual equipment was working. his man within weeks, lost the use of both legs and his left arm. Five months later he became the first fatality directly related to the Therac-25 system.
Severity 1 Production Defect #3: 
A month later at the same hospital, with the same technician another fatal dosage was given. The technician made the same error of quickly changing the mode from X-ray mode to Electron mode using the 'cursor up' key. This again caused "Malfunction 54". The patient this time was receiving treatment on his face. When the overdose was administered he yelled and then began to moan. The audio equipment was working this time but the initial dose was too much for the man. He received severe neurological damage, fell into a coma and died only 3 weeks later.
[raj]: If system was designed considering that a simple wrong choice can have such adverse effects then a choice made by technician could have warned him and possibly stopped him from making that mistake.
 

Learning

Learning #1: Never dismiss any failure without reaching the bottom of it. Over confidence about your quality can take you and your customer down

Learning #2: Never depend on just one resource for the entire functionality. It’s dangerous. and it takes two to tango  (certain activities can’t be achieved singly like arguing, fighting, dancing, making love :))

Learning #3: Unit, Integration and System testing, they all are equally important and one shouldn’t undermine importance of any of these.

Learning #4: Poor testability is extremely fatal. Lacks of user’s ability to validate the completion of the software operation/task can take lives as we have seen above

Learning #5: Don’t repeat any important function/operation/task without confirming the behaviour of the previous operation. Many times we  think that running the software function again is case of failure is perfectly fine but that can be risky if the last operation resulted into corruption or left the machine in inconsistent state

Learning #6: For critical functions in your software, ensure there are provisions to handle silly human errors where we perform an action what we don't intend.  Design should consider that humans can make mistakes and for important tasks, there should be a warning/message confirming the change (that can possibly warn him and correct the action as he intended).

Example of such human mistakes.

we want to click on Checkbox Yes but because page scroll happens and we click on NO and we don't even notice it

or

we are not 100 % concentrating and our brain is lost in it thoughts and we humans are sometime unaware of the  the action performed.

e.g. I bet you would have felt the same more than once  “Have i left the tap open after the bath?” when you would have closed it.

 

Thoughts for you

If you are thinking,this was a rare scenarios and example of worst engineering and the machine would have got retired for ever then i want to leave you with this fact that this machine is still in use today and there might be someone you know who might be sitting in front of the machine as we speak and that’s why it is important to find defects before a life-saver turns life-taker

Thursday, December 17, 2009

Correlation between Cyclometric Complexity & Code Coverage while doing White box testing

If you are among those testers (or developers) who have often wondered how cyclometric complexity can practically help me achieve higher code coverage and uncovering max number of bugs by executing minimum no. of unit test then please read on. I have tried to take a very simple example and have focussed on explaining the very basics with data and images only to make sure you can visualize and understand the differences between statement coverage, decision coverage and path coverage and their relationship with cyclometric complexity for once and all.

Pseudocode                                

Read a;

Read b;

if(a>=b)

         print  a

else

          print b

Do x;

if (a ==0)

        print “Zero”

do nothing;

 

Flow chart of the above pseudocode:

 

image

 

Cyclometric Complexity:

= E-N + 2P  (where E is no. of edges, N is the no. of node and P is the no. of connected components in the above graph)

= 10 – 9 + 2 x 1

= 3

Minimum Test Required for  100 % Statement coverage:

Unit Test #1:

Test data: a = 0, b = –1  (True & True)

cyclometric complexity_SC_Test1

Unit Test #2:

Test data = a =0, b = 2 (False & True)

Total  test required for 100 % statement coverage = 2

cyclometric complexity_SC_Test2

With the above two test cases we are able to execute each line of code at least once.

Minimum Test Required for 100 % Branch Coverage or Decision Coverage

Unit Test #1:

Test data: a = 0, b = –1    (True & True)

cyclometric complexity_SC_Test1

Unit Test #2:

Test data: a =1, b = 2  (False & False)

cyclometric complexity_bc_test1

Total  test required for 100 % branch coverage = 2

Minimum Test Required for 100 % Path Coverage (Basis Path Coverage)

Unit Test #1:

Test data: a = 0, b = –1  (True & True Input)

< same as scenario #1 of statement coverage>

cyclometric complexity_SC_Test1

Unit Test #2:

Test Data : a = 1, b =0  (True & False Input)

 cyclometric complexity_pc_t2

 

Unit Test #3

Test data:  a =0, b = 2 (False & True Input)

cyclometric complexity_pc_t3

Unit Test #4:

Test data:  a =1, b = 2  (False and False Input)

<same as Scenario #2 of branch coverage>

cyclometric complexity_bc_test1

Total  test required for 100 % path coverage = 4  (Which is also 2^2 choices where 2 is n decided by first IF-ELSE)

basically what we have done above can also be shown in the form the grid:

Input\Output First IF O/p of First IF    Second IF O/P of Second IF
a = 0, b = –1    True Print a  True Print 0
a = 1, b =0  True Print a  False     -
a =0, b = 2  False Print b  True print a
a =1, b = 2  False Print b  False -

Universal Formula:

Branch Coverage <=  Cyclometric Complexity <= Path Coverage

Replacing our values:

2 <= 3 <= 4 (holds good :))

Now what do we conclude by doing all this:

Does 100 % statement coverage ensures no bugs?

Answer is NO. 100% statement coverage just ensure every statement is executed at least once BUT it doesn’t guarantee that all the conditions are tested for different combination of data and hence it is not the most efficient method

Does 100 % branch coverage ensures no bugs?

Answer is again NO. 100 % branch coverage is certainly better than statement coverage and more efficient but it only ensures that each branch is executed at least once but again it doesn’t cover all the possible combinations  (like two Ifs can be tested with 2 branch test case for TT, FF where as total possible combination will be 4 (TT,TF,FT,FF) which are not getting covered here)

Which is the best way of ensuring 100 % code coverage.

Answer is Path Coverage (a.k.a. basis path testing). This is better than both statement and branch coverage. Though this is practically very impossible to cover all permutations and combinations (For ex when there are 2 IF statements, total paths are 2^2= 4 combination and as no. of Ifs in the program  goes to n, no of possible combination goes to 2^n)

Thats where cyclomertic complexity is used to find out the min test to be executed to achieve path coverage and the max test to be executed for branch coverage.

In above example, if the bug was in a condition when one first IF resulted into True (YES) and Second IF resulted into False (NO) then none statement coverage or branch coverage could find that bug and hence cyclometric complexity = 3 shows that adding one more scenario to branch coverage  as TF or FT (other than TT,FF) will help to uncover that bug so minimum 3 test are required to have high probability of uncovering bugs but max 4 test are required (max possible combinations) to ensure no bugs, which generally become impractical with the growing code containing multiple nested if-else and loops.

Here cyclometric complexity helps unit tester decides at least what are the min no. of test that should be executed to cover the scenarios which have high probability of finding bugs out of total possible scenarios of path coverage (Which is 2^n)

PS: I want to thank Renu (an enthusiastic tester) to bring this topic in our discussions which triggered this interest in me to link all these pieces and try giving an overall perspective to testers by taking this simple example.

Feel free to write your comments to have a discussion or to clarify a point if you disagree.