Thursday, February 13, 2014

MTM ( Microsoft Test Manager) - Getting latest results as email notification


Author: Ranjit Gupta and Raj Kamal

Today, MTM (Microsoft Test Manager) doesn't provide an email report with the latest test results for a given test plan. Well, you can do this  by using the below snippets of code and configure it for your test team and other stakeholders. We hope you find this useful. Your comments are welcome.

1.      Query test plan

ITestPlanCollection mTestPlanCollection = testProject.TestPlans.Query((string.Format("Select * From TestPlan where PlanName = '{0}'", testPlan)));

2.      Getting the status of the desired Test Plan

 

ITestPointCollection teamtestPass = testplan.QueryTestPoints(string.Format("SELECT * FROM TestPoint where LastResultOutcome='Passed'"));

                ITestPointCollection teamtestFail = testplan.QueryTestPoints(string.Format("SELECT * FROM TestPoint where  LastResultOutcome='Failed'"));

                ITestPointCollection teamtestBlocked = testplan.QueryTestPoints(string.Format("SELECT * FROM TestPoint where  LastResultOutcome='Blocked'"));

                ITestPointCollection teamtot = testplan.QueryTestPoints(string.Format("SELECT * FROM TestPoint "));

                teampass = teampass + teamtestPass.Count;

                teamfail = teamfail + teamtestFail.Count;

                teamblock = teamblock + teamtestBlocked.Count;

                teamtotal = teamtotal + teamtot.Count;

 

teamtestPass contains all those testpoints which has Outcome as Passed. So above queries gives you the status of your test plan

 

3.      In your report you might want to list all test cases or all failed/blocked test cases and with some details

Below statement will give you the ID, title, configuration, assigned to, outcome and the duration for the test point

foreach (ITestPoint point in teamtot)

{

Console.WriteLine(point.Id + "-- " + point.TestCaseWorkItem.Title + "-- " + point.ConfigurationName + "-- " + point.MostRecentResult.Outcome.ToString() + "-- " + point.AssignedToName + "-- " + point.MostRecentResult.Duration);

}

PS: The testpoint which are in Active state “point.MostRecentResult.Outcome.ToString()” would throw null exception, you need to handle that in your code

4.      You can capture all this information into a html file for better reporting and send it as automated mailer

 

SmtpClient client = new SmtpClient("smtphost");

            client.UseDefaultCredentials = true;

 

 

            string fromAddress = Environment.GetEnvironmentVariable("USERNAME") + "@microsoft.com";

            MailAddress from = new MailAddress(

                fromAddress, ProjectSettings.Default.FromName, System.Text.Encoding.ASCII);

            List<MailAddress> to = new List<MailAddress>();

           

            string address = ProjectSettings.Default.toAddress;

            string[] toRecipent = address.Split(';');

 

            foreach (string add in toRecipent)

            {

                to.Add(new MailAddress(add));

            }

MailMessage message = new MailMessage();

       message.IsBodyHtml = true;

       message.From = from;

       to.ForEach(entry => message.To.Add(entry));

       message.Body = report.ToString();

            message.BodyEncoding = System.Text.Encoding.UTF8;

            message.Subject = ProjectSettings.Default.mailSubject;

            message.SubjectEncoding = System.Text.Encoding.UTF8;

         client.Send(message);

 

 

Distributing Coded UI Scripts (with Selenium add-in) for Cross Browser Automation using Visual Studio Lab Management & Environment variables

Author: Ranjit Gupta and Raj Kamal

Background

If you are not already aware, Coded UI now supports cross browser testing using Selenium components. The Visual Studio add-in can be found here, that works if you are running VS 2012 Update 2 or above. There is also an official blog that talk topic in detail, if you are interested.

Customer Story

A large utility service company in United Stated has retained us to implement its internet facing web presence   - USD 7 Million engagement. The website will provide timely, business driven information along with functionality to its customers to do their regular interactions, such as paying bills, checking historical usage data, turning on/off services, and others mentioned in the business requirements section. As it’s an external facing site, it needs to be supported on Firefox, Chrome as well as IE 8 & IE 9. With the size and the criticality of the application to customer’s core business, selective manual testing on non-IE browsers is not an acceptable approach.

The power of Coded UI + Selenium & Visual Studio Lab Management

We wrote Coded UI Tests used this Selenium add-in along with Visual Studio Lab feature to distribute our automated tests on multiple browsers. Our goal was to dedicate each OS + Browser combination a dedicated agent machine, so all our tests run in parallel on different browsers on pre-defined machines.

One challenge was that, there is no way in Visual Studio Lab settings to specify that a particular agent machine should be used for a specific browser e.g. Chrome, Firefox, IE 8, IE 9 etc.  We didn’t want to create multiple copies of coded UI test methods and hardcode them to run against specific configurations. We were looking for a solution that didn’t require us to write custom logic to solve this very problem and fortunately we could achieve this without any additional coding using the below proposed solution.

Solution – Simple yet elegant

To get past this issue, we made use of environment variables. The steps are explained below.

1.      Create environment variable on each agent machine and set its value as the desired browser name (IE8/IE9/Chrome/Firefox) against which you want to run your automated tests.

 

2.      Create a Test Initialize method and read the value of the environment variable before test starts on each agent machine. This will return the name of the browser that needs to be used for playing back automation.

 

 [TestInitialize]

        public  void Init()

        {

           App_Constants.browser= Environment.GetEnvironmentVariable("browser",

           EnvironmentVariableTarget.User);

           

 }

 

 

3.      Inside your test method, set BrowserWindow.CurrentBrowser value as App_Constants.browser which holds the browser name stored in environment variable

[TestMethod]

        public void CodedUITestMethod1()

        {

            BrowserWindow.CurrentBrowser = App_Constants.browser;

            this.UIMap.RecordedMethod1();

            this.UIMap.AssertMethod1();           

           

    }

4.      Depending on the value stored in the environment variable, Coded UI launches the browser and runs the test against launched browser.

 


 


 

You can set these configuration as default = Yes, if you want these to be used for the test cases to bed added to the test plan

 

6.      Associate your automated tests with test cases in MTM (Microsoft Test Manager)

 

7.      Setup test controller and test agents

                       

8.      Run your automated test against each agent

 

Now you can control which OS/Browser you would like to use to run your automated tests without making any changes in your Coded UI Tests or writing any extra code.

 

Thursday, September 12, 2013

Win 8/1/IE 11 - Cross Browser Testing, Device Testing and more cool features


I am sure many of you would have discovered these if you have installed Win 8.1 but I wanted to quickly share with others the cool features that IE 11 brings out the box as part of developer toolbar. Highlighting few of them below:

 

1.      Cross Browser/Platform testing: This is cool to see how you site renders on different browser (not just limited to different versions of IE) and  OS’s as well as different resolutions. This will be quite handy for testers.

 
           

 

 

 

2.      UI Responsiveness: Looks pretty nice for performance benchmarking and quick assessment of performance.

                                   



 

 

3.      Network:

 



 

4.      Memory:

               



 

5.      Profiler

 
              


 

 

Test Effectiveness, Test Efficiency, Defect Removal Efficiency (DRE) - Jargons :)



I was talking to a colleague today around difference among these test metrics and I am sharing it with you all if it helps :)
 

Test Case Effectiveness = Total defects found by test cases / Total defects**

e.g. if 80 defects were found by test cases and 10 were through ad hoc testing and 10 were leaked to UAT / Prod then TC effectiveness is 80 %

 

** This includes all defects found post build  phase (after test cases are designed) so if buddy testing was done then sure it should be added to total defects as we will use our test cases to do buddy testing.  Total defects will also include the defects found in UAT and post production as they are leakages.

 

Test Efficiency = No. of valid defects  / Total Defects found by test team (including invalid defects)

 

e.g. if 90 defects out of 100 were accepted then Test efficiency is 90 %

 

This gives us the rework and wastage due to invalid bugs that we rejected

 

Defect Removal Efficiency (DRE)  = Total defect found before delivery (through review, inspection, testing etc) / Total defect during project lifetime

 

Here we are saying doesn’t matter if we found them by test case or reviews, if we found them before UAT then its good and the defects that are found in UAT and Prod will be added to denominator and will show our DRE.

Saturday, August 17, 2013

Mob Testing (inspired by Mob Programming)


Testers & QA's,

There is something interesting that we might want to try. I came across an upcoming agile practice called Mob Programming. In gist, this takes pair programming to next level by coding as a team. The idea is simple that you can write the best code when one person is actually typing the code and rest of the team is providing inputs. This has helped teams deliver builds without bugs. If you think, you are developing, unit testing, reviewing and testing the code all-at-once. How cool is that?

Read about it @ http://mobprogramming.org/mob-programming-basics/

Driver/Navigators

We follow a “Driver/Navigator” style of work, which I originally learned from Llewellyn Falco as a pair programming technique. Llewellyn’s approach is different from any other I have been shown, have seen, or have read about (and I think he invented it or evolved into it.)
In this “Driver/Navigator” pattern, the Navigator is doing the thinking about the direction we want to go, and then verbally describes and discusses the next steps of what the code must do. The Driver is translating the spoken English into code. In other words, all code written goes from the brain and mouth of the Navigator through the ears and hands of the Driver into the computer. If that doesn’t make sense, we’ll probably try to do a video about that or a more complete description sometime soon.
In our use of this pattern, there is one Driver, and the rest of the team joins in as Navigators and Researchers. One important benefit is that we are communicating and discussing our design to everyone on the team. Everyone stays involved and informed.
The main work is Navigators “thinking , describing, discussing, and steering” what we are designing/developing. The coding done by the Driver is simply the mechanics of getting actual code into the computer. The Driver is also often involved in the discussions, but her main job is to translate the ideas into code. Of course, being great at writing code is important and useful – as well as knowing the languages, IDE and tools, etc. – but the real work of software development is the problem solving, not the typing.
If the Driver is not highly skilled, the rest of the team will help by guiding the Driver in how to create the code – we often suggest things like keyboard short-cuts, language features, Clean Code practices, etc.  This is a learning opportunity for the Driver, and we transfer knowledge quickly througout the team which quickly improves everyones coding skills.

I would suggest that if we apply the same to testing how productive it can be. I know exploratory testing is cool but what would be even better is that the entire test team gets into a room and one person takes the responsibility of "driver"- he will follow the instructions of the team and document/execute the test cases. Rest of the team will play "navigators" and suggest scenarios and different test condition. This will be a great demonstration of collective IQ and things like mails, meetings, triage calls can be eliminated to reduce unproductivity.

Let me know your thoughts and if you find this interesting, we can pilot this in one of your projects and evangelize it.