Tester Testifies: December 2011

For all deliverables & details: http://webservicestesting.codeplex.com

Authors

Microsoft Services - Global Delivery, India

Objective

This artifact will provide Web Services Automation guidance for Functional & Performance Testing leveraging VS 2010. The document will provide recommendation on using either Unit Test or Web Test for functional testing depending on your project and test objective as will be later covered in the document. Load Test will be used for Performance Testing in both cases.

The primary objective is to propose a viable web services test automation approach & reusable libraries to customers, partners and consultant. It will also be a step in standardizing web services automation using VS 2010 platform across the Industry and filling the gaps by extending our tools as suggested in the talk. The document can also be used as a quick jump start guide for beginners to get started on Web Services Automation Testing.

Other than the guidance included in this document, key deliverable will primarily consists of the following:

a) Unit Test & Web Test Template projects for Web services testing

b) Reusable libraries for Unit Test

c) Reusable Add-ins (Extraction & Validation Rules) for Web Test that can be reused to reduce automation development effort drastically.

d) Performance Testing using Load Test once the Web Tests/Unit Tests are in place.

e) Sample Web Services application for beginners to gets started

Pre-requisites:

Ensure you have downloaded and deployed WebServiceAutomation_SampleApp that contains sample Web services projects (WCF/ASMX/WF etc.) that you can use to getting started with Web Services automation.

Verify you have downloaded our sample Template projects WebServiceAutomation_SampleUnitTest and WebServiceAutomation_SampleWebTest that contain sample unit test/web test which you can reuse / customize for your needs

Ensure you have downloaded our Reusable Libraries for Unit Testing (WebServicesTesting.Automation.UnitTest.Helper.dll) & Web Test (WebServicesTesting.Automation.WebTest.Helper.dll)

Visual Studio 2010, SQL Server Express or higher version, IIS (Internet Information Server 6.0 or above

Importance of Web Service Automation

It’s important to first understand the importance of Web Services Automation and the ideal scenarios where it must be recommended.

Web Services (WCF/ASMX) due to their worldwide implementation and popularity but limited test tool support, often raises the question around testing them effectively. Testers are craving to find effective ways to Test Web Services. Even VS 2010 doesn’t have dedicated test types for verifying Web Services (like for UI, there is Coded UI and for DB there is Database Unit Test) which results in low test coverage and non-standardization of proven ways to test web services especially WCF 4.0 Web Services. Web Services Testing in VS 2010 so far has been not been given a first-class citizen treatment and this guidance will bridge that gap by making it much easier and effective.

· In this agile development era, we are moving to a model where UI is ever changing and consummation of web services is taking the center stage and hence this layer attention it truly deserves by shifting the onus from UI testing to Web Services based validation.

Traditionally teams have also used Fiddler and other 3^rd Party tools for the same purpose (many times for manual testing alone) but as Web services have different needs, the knowledge harvested doesn’t get reused to its potential in future engagements

Ideal Candidates

· If your application is consuming APIs exposed by other upstream systems or 3rd Parties

· If your application is exposing APIs to be consumed by the downstream applications

· When your UI is dynamic and constantly changing and cost/maintenance of UI automation is much more.

· When your business logic and rules are primarily implemented using services (ASMX, WCF & Workflow Services) and testing at this layer is much faster and efficient.

· When you need to come up with a common automation framework for web services that can be used to validate both functional and performance aspect of the web services with minimum effort.

Test Approach – Identify your goal (Functional Test or Performance Test or Both)

When we understand and realize the importance of Web Services Automation, the next challenge is to select the right Test approach and Test Type before jumping into automation design and implementation.

Test Objective

It is important to answer the basic question “Why do you want to do Web Services Testing?” and that will help you decide the right kind of Test you should choose to automate them. Let’s begin with few possible answers

To validate the functional behavior of your application / APIs

To validate the performance aspect of your application/ APIs

To first validate the functionality and then also test the performance aspect of it. (a) + (b)

Now when you know about your test objective, you should also look at the technologies being used for your Web Services as that can make a difference as well. If you use Microsoft platform, chances are your services are built on:

ASMX – Legacy aps

WCF – New apps

WF – Window Workflow Foundation

Choosing the appropriate Test Type (Unit Test or Web Test)

Please find below the recommendation on the type of test you should decide depending on your test objective defined in the section above

Unit Test – provides you with flexibility to directly access your Web services APIs programmatically using familiar c# and test their behaviour based on your requirements, business rules etc. We will be covering this in depth in the section below

Web Test – also lets you invoke your web services methods, though not through better known record and playback feature of Web Test but by directly adding services method call and parameters, properties. We will be covering this in depth in the section below

At a high level, you can say that unit test is more of glass box testing whereas Web Test gives you ‘black box’ view.

As shown in the image below, Unit Test is an excellent choice when:

You have complex web services involving lot of business rules and logic which you need to validate and primary focus is on detailed functional validation

You are using WCF based services that includes RESTFul API, services returning non-xml kind of data

Web Test is a better choice when:

You have relatively simpler web services where you objective is to test simpler / straightforward functionality and behaviour and primary focus is to test their performance down the line

You are working on legacy app that are based on ASMX kind of services which have a better support in Web Test

Test Types vs. Test Objective Recommendation

Sample Lab Application: Web Service Automation

As many of you might be new to Web Services automation, we have created and shared sample Web Services applications that will help you better understand our helper functions/ Add-ins and sample unit test/web test shown later in the document.

If you already have sample Web Services deployed in your machine, you can skip this section.

Sample Application: Student Administration

This is a sample Student Administration application that comprises of simple functionalities around enrolling students and displaying their details back to the user. The UI app is built using Win forms and services are implemented using both ASMX and WCF for demonstration purpose. There are other sample projects to demonstrate features like Authentication, Caching, Workflow, Asynchronous calling etc.

Sample Student Administration Application UI

Sample Application project structure in TFS

Please refer to Appendix: Deploying Sample Web Services Application to know more about our sample application and instructions around deployment.

At this point we assume you have deployed our sample web services application. In subsequent sections, we will present the various approaches and steps involved in Web Services Automation Design, Development & Execution that you should take to achieve your objective while following the automation standards and guidelines that are recommended.

Web Services Automation – Unit Test Walkthrough

If “Unit Test” was selected as the Test approach for Web service automation based on the analysis and your test objective as covered in above section: Test Approach then proceed with the walk through.

Creating a new Unit Test Project / Reusing Sample Template Unit Test Project

You can create a new project from scratch or using our existing sample template project but it’s important to add your project to source control for version management and change tracking.

SL No.	Topics	Description
1.	Have you identified & marked the automatable tests as PLANNED in Microsoft Test Manager (MTM)	Open your test cases in Test Manager that you want to automate and mark Automation Status = “Planned” for those Whereas the test which are not good candidates for automation are marked as “Not Automated” to show that they are out-of-scope for automation. This is an optional step if you don’t want to run your test from Microsoft Test Manager and map your automation to manual test cases. We however recommend this.
2. a	Have you created new Test project & Added it to Source Control (TFS)?	Every automation project must be source controlled. Add project to Source Control by clicking the check box as shown and give appropriate name.
2.b	Alternatively, Open existing Test project and using it as Template	Launch Visual Studio, Browse and open existing ‘WebServiceAutomation_SampleUnitTest’ template project and add it to Source control. You can change the properties as per your needs. For this sample template project we have SampleUnitTestCases.cs that contains all our sample reusable unit test cases. Please find below a short description on the folder structure of the template project which will be used for case study: 1. Config: All your application configuration details that you need to run your test will go to app.config Example: Also refer to: http://msdn.microsoft.com/en-us/library/a65txexh(v=VS.100).aspx 2. Common Libraries References: Add references to our reusable helper library “WebServicesTesting.Automation.UnitTest.Helper.dll” for Unit Test Web Services Testing (provided as part of the release and documented below) . 3. Service References: Add / update services reference to point to your web services and provide a logical name 4. Scripts: Your actual test are stored in these .cs files along with components like Test Initialize, Test Cleanup etc You can create a folder structure to organize your test methods by grouping them in multiple class files depending on the size of the project. It is advisable to group test belonging to same feature/functional module to be grouped together in a .CS file and given an appropriate name. 5. Solution Configurations & Compilation Symbols With our template solution you get additional solution configuration which will help you run your same test for both functional and load test seamlessly by controlling the sections of the code that you may want to execute for functional and load testing For example. When you do load test, you might not want to do certain validation which makes sense for functional testing alone and these configurations will control that by just selecting the appropriate solution configuration as shown below. a) Custom Solution configurations b) Configuration – Functional Testing As shown below it uses a compilation symbol #WEBSERVICES_FUNCTIONAL_TESTING that is used in our test methods to control the behavior. c) Inside your test method, you will have code like this to control the behavior based on the solution configuration file that is selected For more reference: http://www.codeproject.com/Articles/49448/Use-Visual-Studio-Conditional-Compilation-to-manag.aspx

Unit Test Automation Design & Development (WCF & ASMX based services)

Now that you have your test project in place and our sample reusable unit test cases for automating common web services test scenario, in this section we will be walking you through few sample reusable tests that you can customize it for your own needs or create new unit test for your project scope. The approach shown below would remain same for WCF as well as ASMX based services and we have shown examples of both.

Follow the below steps for Automation design guidance:

SL No.

Steps

Description

Global Settings

a) Set the test context which provides information about and functionality for the current test run.

b) Test Initialize: To perform steps before running each test like instantiating objects, defining global variables including connection strings etc.

c) Test Cleanup: To run code after running each steps

2a.

Add New Test Methods / Reuse Existing Template Test Methods

(Simple Lookup Verification)

Sample Test scenario: Verifying Web service call response value against expected results

Below is sample data driven test method for web services testing that comes with our template project for Unit Test. You can add your new test methods by modeling the below example.

Step 1.

Create a new Test method / Modify our existing sample test method

Step 2.

Instantiate and invoke your web service method (in this example: ASMX based service) as shown in the code snippet below and get the results in a variable of appropriate data type

In our sample example, the web service returns an integer datatype which we cast to String for comparison. Your services method can return any data type like int, double, decimal, bool, string etc and we can use the same helper method below to validate the response.

Step 3.

Get your expected results from data source of your choice as shown below.

Step 4.

Reuse our appropriate library / helper method for validating the web service response against expected results.

In this example CompareString is a reusable method in our helper / library that can be called from your test methods to save effort related to validating the results and marking the test as pass/fail.

Note: Depending on your test scenario, you need to choose different helper/library methods. The complete documentation about our reusable library method is given in sections below

Add New Test Methods / Reuse Existing Template Test Methods

(Simple Transactions Verification)

Sample Test scenario: Verifying Web service call results in database operation like Insert

Below is sample test method when tester needs to validate if the web service call results into a Database Insert by validating the database for ensuring the transaction was committed. Instead of Insert, you might have different operations like update, delete etc. for which we have reusable helper methods in our library.

Here the web service method Enroll_Student (in this case WCF based Service) is inserting a record into the database and the expected result is the query against the database to validate the operation to ensure transaction was committed.

Reuse our appropriate library / helper method for validating the web service response against expected results.

In this example Verify_DBInsert is a reusable method in our helper / library that can be called from your test methods to save effort related to validating the database insert and marking the test as pass/fail.

Note: Depending on your test scenario, you need to choose different helper/library methods. The complete documentation about our reusable library method is given in sections below

Add New Test Methods / Reuse Existing Template Test Methods

(Handling Complex Object Types)

Use this approach as a generic way of comparing / validating web services response of various data types including user defined/abstract data types like Struct, Classes as well as primitive data types like Int, String, Dictionary, List etc.

DTO (Data Transfer Objects) Having the results of a remote method available in an object instance makes it easy to pass this data to a test module or to compare it with the desired outcome

It works by converting the service response (whether abstract or primitive data types) and expected results into standard generic objects which then are finally converted into XML for comparison with the help of our reusable library methods as described below in the example

Sample Test scenario: Verifying Web service response which returns a Struct object against test data from database.

Step 1: Create Data structure for your object which your service returns as

shown below

(In this case it returns a Struct which contain elements like

First_Name, Last_Name etc.

Step 2. This is an optional step for better organizing your code which invoked your web services and returns the response to the main test method for verification.

Step 3: Add your test method

Step 4. You can read your data source (in this case database) and convert it into generic DTO object using reusable method in our library called DTO_PopulateDBResults which takes Dataset as input parameter. DTO Type should be same as the type of your complex object (in this case Student which is struct)

List<Student> ExpectedStudentDTOList = null;

ExpectedStudentDTOList = helper.DTO_PopulateDBResults<Student>(dt);

Step 5. In this step you call you Action code to get your web service response

Actions.DTO_Action complexObj = new Actions.DTO_Action();

Object myObj = complexObj.ComplexObjectSample();

Step 6. Convert web services response into generic DTO using our generic library method called DTO_PopulateObject which accepts Object and structure of the object defined above as parameters.

object ActualStudentDTOList = helper.DTO_PopulateObject<Student>(myObj, s1);

Step 7. Finally the generic DTO objects returned from expected results i.e. Test data source and Web services response are converted into XML and compared using our reusable library method called DTO_XMLComparison and accordingly marks the test method as pass/fail

bool result = helper.DTO_XmlComparison(ExpectedStudentDTOList, ActualStudentDTOList, typeof(List<Student>));

Assert.AreEqual(true, result, "Expected does not match actual");

Step 8. Expected and Actual XML files with the test method name are created under your folder c:\TestResults for your future reference. Please create this folder if it doesn’t exist

Make your scripts data-driven

Irrespective of kind of test methods you have created above

Data-Driven Test

Ensure every variable is parameterized by passing the test data to each test using data-driven capabilities in VS 2010.

Right-click on the test method and select Properties, to view the properties of the test

To start making this a data-driven test, we have to first create a data source for the test to draw from. On the Data Connection String property, click the ellipse button to start the New Test Data Source Wizard

There are three options for creating a data source:

Database -- which allows you to connect to a database or Microsoft Excel spreadsheet

CSV File -- which allows you to use a comma-delimited file

XML File -- which allows you to use an XML file

When you add a data source A Data Source attribute is added to the Test method.

[DataSource("System.Data.SqlClient", "Data Source=(local);Initial Catalog=Sample_DB;Integrated Security=True", "Sample_App", DataAccessMethod.Sequential), TestMethod]

Map your automation test to your test cases in MTM

This is an optional step if you don’t want to run your test from Microsoft Test Manager and map your automation unit test to manual test cases. We however recommend this.

Repeat the above steps for all your test scenarios against your web services to be tested. After you write and debug your scripts its important to run them in a batch or as part of a suite. There are various way to do that and factors to keep in mind as we will see in next section.

Web Services Automation – Web Test Walkthrough

If “Web Test” was selected as the Test approach for Web service automation based on the analysis and your test objective as covered in above section: Test Approach then proceed with the walk through.

Creating a new Web Test Project / Reusing Template Web Test Project

You can create a new project from scratch or using our existing sample template project but it’s important to add your project to source control for version management and change tracking.

SL No.

Topics

Description

Have you created new Test project & Added it to Source Control (TFS)?

Every automation project must be source controlled.

Add project to Source Control by clicking the check box as shown

Alternatively,

Open existing Test project and using it as Template

Launch Visual Studio, Browse and open existing “ WebServiceAutomation_SampleWebTest” template project and add it to Source control. You can change the properties as per your needs.

Please find below a short description on the folder structure of the template project which will be used for case study:

1. Common Libraries References:

a) Add references to our reusable helper library “WebServicesTesting.Automation.WebTest.Helper.dll” for WebTest based Web Services Testing (provided as part of the release and documented below)

b) Reusable Validation Rules

c) Reusable Extraction Rules

2. Service References: Add / update services reference to point to your web services and provide a logical name

3. Scripts: Your actual test are stored in these .webtest files.

You can create a folder structure to organize your test methods by grouping them in multiple class files depending on the size of the project. It is advisable to group test belonging to same feature/functional module to be grouped together in a .CS file and given an appropriate name.

For this sample template project we have many reusable webtest that can be reused / customized for your needs.

4. Test Data: You can choose to have you test data as CSV or XML or Database as support by Visual Studio Testing framework for making your test data driven.

WebTest Automation Design & Development (WCF & ASMX based services)

Now that you have your test project in place, in this section we will be walking you through our sample reusable Web test for automating common web services test scenario that you can customize it for your own needs or create new additional Web tests for your project scope.

Follow the below steps for Automation design guidance:

S No.	Steps	Description
1a.	Add New Web Test / Reuse Existing Template Web Test (Dataset Verification for WCF/ ASMX Service)	**Sample Test scenario: Verifying Web service method that returns a dataset against the expected test data from Database If you are adding a new web test from scratch then follow the steps: Step 1. Open your web service method request XML by launching WCFTestClient** utility or any other way you are comfortable with. Step 2. Add a new Web Test or Reuse our existing template Web Test and customize them Step 3. Right click the node and press “Add Web Service Request” and provide your web service URL Step 4. Right click the node and press “Add Header” and provide SOAPAction value as shown below by retrieving it from request XML of Step 1. Step 5. Add <string body> a) From your request XML as shown below, extract Envelope and Body (you shouldn’t copy Header part) <s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/"> <s:Body> <Display xmlns="http://tempuri.org/" /> </s:Body> </s:Envelope> b) Add this to to you string body now Step 6. Add our reusable Validation and Extraction Rule called DatasetValidation that reads the expected data from database using SQL query and validate it against the response from web services by using XPathExpression Note: Depending on your test scenario, you may need to choose different Extraction and Validation Rule complete documentation about our reusable library method is given in sections below
1b.	Add New Web Test / Reuse Existing Template Web Test (Method Prototype Verification for WCF/ ASMX Service)	**Sample Test scenario: Verifying Web service method parameters and response values against expected test data. Repeat Step 1 to 5 from 1 a) above as explained Step 6. Add our reusable Extraction Rule called ParameterExtraction and provide XPathExpression for example Enroll_Student and it will retrieve all the parameters name and their respective values and store it inside ContextParam Step 7. Step 6. Add our reusable Validation Rule called ParamValidation** and provide ParamValues that contain expected test data as shown below along with contextParam name as defined in Step 6. Step 8 & 9. As shown in this web test you can also add our reusable ServiceResponseExtraction and ServiceResponseValidation rules from our helper class for validating the response values against expected test data. Step 10. After you run this web test, the result will show Pass / Fail with context tab showing the discrepancies if any. Note: Depending on your test scenario, you may need to choose different helper/library methods. The complete documentation about our reusable library method is given in sections below
2.	Make your scripts data-driven	Data-Driven Test Ensure every variable is parameterized by passing the test data to each test using data-driven capabilities in VS 2010. To start making this a data-driven test, we have to first create a data source for the test to draw from. On the Data Connection String property, click the ellipse button to start the New Test Data Source Wizard There are three options for creating a data source: Database -- which allows you to connect to a database or Microsoft Excel spreadsheet CSV File -- which allows you to use a comma-delimited file XML File -- which allows you to use an XML file

Automation Execution & Deployment (Unit Test/Web Test)

Ensure you have configured test settings to capture logs, code coverage, test impact etc. and you also need to keep in mind that your script should be able to successfully run in multiple environment (for ex. Test to UAT)

After automation design and debugging is completed, you can now run and deploy your automation scripts using the below steps:

SL No.	Areas	Description
1.	Configuring Test Settings	Test Settings a) Data and Diagnostic: Enable appropriate diagnostic are selected as shown below (Code Coverage, Event Log, Test Impact etc.) b) Define other settings like Roles, Test Timeouts, Setup and Clean scripts
2.	Command Line Execution (MSTest.Exe)	*Grouping & Classifying Test Methods using various parameters* It lets you run groups of tests based on their assigned categories without the requirement to maintain test lists. Go to Test View and you can filter and view your tests by setting these parameters MSTest.EXE Options: MSTest.exe is the command-line command that is used to run tests. This command has several options that you can use to customize your test run. You can specify these options in any order on the MSTest.exe command line. Example: mstest /testcontainer:"C:\ WebServiceAutomation_SampleUnitTest" /test:testproject32\generic Refer to http://msdn.microsoft.com/en-us/library/ms182489.aspx
3.	TCM.exe or Test Execution from Microsoft Test Manager (Optional though recommended)	a) If you have mapped you automation unit test with test cases in Microsoft Test Manager then you can use TCM.exe as shown below to run your web service test and it will also update your test cases results in MTM without any custom code. You don’t need to use MSTest.exe in that case Example: tcm run /create /title:title /planid: id /collection:CollectionURL /teamproject:project (suiteid:id /configid:configid \| /querytext: query) [/settingsname:name] [/owner:owner] [/builddir:directory] [/testenvironment:name] [/login:username,[password]] [/include] Refer to : http://msdn.microsoft.com/en-us/library/dd465192.aspx & http://msdn.microsoft.com/en-us/library/ff942469.aspx b)You can also Run Automated Tests from a Test Plan Using Microsoft Test Manager. To run the latest version of your automated test, you must make sure that the correct build is selected for your test plan. You must also use a physical or virtual environment to run your automated tests. This is out of scope of this guidance however we recommend this the most. Refer to: http://msdn.microsoft.com/en-us/library/dd553270.aspx

Web Service Performance Testing using Unit Test/Web Test + Load Test

Now when you have your Web Services Test automated either using Unit Test or Web Test as suggested in this guidance above, you can go ahead and create you Load Test to performance test your web services.

SL No.	Areas	Description
1.	Configuration – Load Testing	a) Ensure you have selected our custom solution configuration for Load Test i.e. Debug – Load Test b) As shown below it uses a compilation symbol #WEBSERVICES_LOAD_TESTING that is used in our test methods to control the behavior. c) Inside your test methods, you will have code like this to control the behavior based on the solution configuration file that is selected
2.	Creating Load Test using Web Services Unit Test or Web Test (Designed in above sections)	Step 1. Add a new Load Test Step 2. Give a scenario name for your Load test Step 3. Specify your load pattern (Constant Load/ Step Load) Step 4. Choose an appropriate test mix Step 5. This is the most important step where you have an option to select Unit Test or Web Test to be added to your Load test depending on the kind of approach you had selected for your Web services automation above Press Add and select your “Web Services Unit Test” OR “Web Services Web Tests” created in the sections above that you want to performance test Press OK and adjust Distribution of tests. Step 6. Refer to below section for adding custom web services counters other than standard counters that you may want to monitor Step 7. Specify Load run settings Step 8. You load test is created Our sample Unit Test and WebTest template project has this reusable SampleLoadTest.loadtest sample load test for you to model your load tests around.
3.	Running Load Test and analyzing results	For Web services Load testing, we recommend you adding following custom counters against the web services server (where the web services are hosted): ServiceModelEndPoint 3 ServiceModelEndPoint 4 ServiceModelOperation ServiceModelServices Please follow Adding custom counters set using Load Test Editor for detailed steps. Step 1. Open you load test. Right click you counter set and add “Custom Counter Sets” Step 2. Right click and now click “Add Counters” Select Performance Category to include counters related to Web Services as shown below Step 3. Repeat it for all the categories as shown below. Tip: You can enable performance counters for a WCF service through the app.config configuration file of the WCF service as follows: <configuration> <system.serviceModel> <diagnostics performanceCounters="All" /> </system.serviceModel> </configuration> *Sample Load Test Summary Report*

Web Services Helper Documentation for Unit Test & Web Test Add-ins

This section provides documentation of the complete list of reusable helper functions (WebServicesTesting.Automation.WebTest.Helper.dll) and web add-ins (WebServicesTesting.Automation.UnitTest.Helper.dll) for your reference and usage. We have shown few of these in sample walkthrough above to give you a glimpse of the same.

Category	Title	Description	Unit Method Signature & Example invoke call	Web Custom Extension (Extraction & Validation Rules)
Lookup Verification	Verifies the Scalar value or multiple scalar values of even different data types returned by services against expected results	This method will let you compare the scalar values of different types like <INT>, <STRING> etc. returned by the service method call converting them to <string>against the expected values	public void CompareStringArray(int condition, string query, string[] ArrgetAgeActual) Example: obj.CompareStringArray(2, query, ArrgetAgeActual);	For scalar value Extraction Rule ServiceResponseExtraction Context Parameters 1.XpathExpression 2.ContextParam Validation Rule ServiceResponseValidation Context Parameters 1.Expected 2.ContextParam For multiple values Validation Rule DataSetValidation Context Parameters 1.ConnString 2.Query 3.XPathExpression 4.ContextParam The validation rule extracts the dictionary returned by web service from xml and compares with the expected dataset based on the query
Lookup Verification	Verifies the values of <DICTIONARY> data type returned by services against the expected results	This method will let you compare the <Dictionary> <key, value> values returned by service method call against the expected values <key, value>.	public void CompareDictionaryWithExpectedArray(Dictionary<int, string> ActualDic, string[,] ArrgetAgeExpected) Example : obj.CompareDictionaryWithExpectedArray(col_Symbols, ArrgetAgeExpected);	Validation Rule DataSetValidation Context Parameters 1.ConnString 2.Query 3.XPathExpression 4.ContextParam The validation rule extracts the dictionary returned by webservice from xml and compares with the expected dataset based on the query
Lookup Verification	Verifies the values of <LIST> data type returned by services against the expected results	This method will let you compare the values of data type <LIST> returned by service method call against the expected values	public string[] CompareList(string query) Example: string[] ArrgetAgeExpected = obj.CompareList(query);	Validation Rule DataSetValidation Context Parameters 1.ConnString 2.Query 3.XPathExpression 4.ContextParam The validation rule extracts the List returned by webservice from xml and compares with the expected dataset based on the query
Lookup Verification	Verifies the result set returned by services against expected results	This method will let you compare the result set returned by service method call against the expected result set	Method Prototype public void Verify_Dataset(string ConnString, string Query, DataSet dst) Method Invocation UnitTest help = new UnitTest(); help.Verify_Dataset(conn, query, set);	Validation Rule DataSetValidation Context Parameters 1.ConnString 2.Query 3.XPathExpression 4.ContextParam The validation rule extracts the dataset returned by webservice from xml and compares with the expected dataset based on the query
Lookup Verification	Verifies the row count of the values returned by Web Services against expected result	This method will let you validate the row count of the values/list/dataset returned by service method call against the expected row count	Method Prototype public void Verify_RowCountDataSet(int expected, int actual) Method Invocation UnitTest help = new UnitTest(); help.Verify_RowCountDataSet(expected, actual);	Extraction Rule RowCountExtraction Context Parameters 1.XpathExpression 2.ContextParam This method extracts the row count and stores the value to Context Param specified by the user. Validation Rule RowValidation Context Parameters 1.ConnString 2.Query 3.ContextParam 4.ContextParam This method validates the value set to the context param and the one returned by the specified query
Metadata Verification	Verifies the called web service method has schema as expected	This method validates the XML file to make sure it fits to your application's needs to prevent any exception in your code. To check whether the XML document conforms to an XSD Schema, the document must be validated against that XSD Schema	Method Prototype public bool Verify_Schema(string targetNameSpace, string xsdPath,string requestXml) Event Handler to raise error private static void ValidationEventHandler(object sender, System.Xml.Schema.ValidationEventArgs arg) Method Invocation UnitTest help = new UnitTest(); bool validation = help.Verify_Schema(targetNameSpace ,XsdPath,requestXml);	Validation Rule SchemaValidation Context Parameters 1.targetNameSpace 2. xsdPath
Metadata Verification	Verifies that web service methods exist and their parameters definition (like input parameters, return type) against expected result	This method will let you smoke test / BVT if the expected list of Web Service methods exists and their definition like input parameters and return type matches.	Method Prototype public void Verify_methodPrototype(object myObject, string Methodname, string paramlist, string returnTypelist) Method Invocation UnitTest help = new UnitTest(); help.Verify_methodPrototype(obj, methodlist, paramlist, returnTypelist);	Extraction Rule ParameterExtraction Context Parameters 1.XpathExpression 2.ContextParam This method extracts the parameter passed to the request and set to the contextparam Validation Rule ParamValidation Context Parameters 1.ParamValues 2.ContextParam This method validates the value set to the context param and the Values passed to the ParamValues The above extraction & validation rule is used to validate the parameter passed The below extraction & validation rule is used to validate value returned by the webservice Extraction Rule ServiceResponseExtraction Context Parameters 1.XpathExpression 2.ContextParam Validation Rule ServiceResponseValidation Context Parameters 1.Expected 2.ContextParam
Transactional Verification	Verifies that called web service method creates the output file as expected	This method verifies the output file (e.g. CSV, XML, TXT) has got created by the web service method call	Method Prototype public void Verify_FileCreation(string FilePath) Method Invocation UnitTest help = new UnitTest(); help.Verify_FileCreation("D:\\employees.xml");	Validation Rule FileCreationValidation Context Parameters 1.FilePath
Transactional Verification	Verifies that web service INSERTS/CREATES a new record into the database	This method verifies the web service call results into new record insertion in a given database	Method Prototype public void Verify_DbInsert(string ConnString, string Query, string expected) Method Invocation UnitTest help = new UnitTest(); help.Verify_DbInsert(conn, query, expected);	Extraction Rule ParameterExtraction Context Parameters 1.XpathExpression 2.ContextParam This method extracts the parameter passed to the request and set to the contextparam Validation Rule DbInsertValidation Context Parameters 1.ConnString 2.ContextParam 3.Query 4.ContextParam This method validates the contextparam and the values returned by the query
Transactional Verification	Verifies that web service UPDATES existing records into the database	This method verifies the web service call results into update of existing record(s) in the given database	Method Prototype public void Verify_DBUpdate(string ConnString, string Query, string expected) Method Invocation UnitTest help = new UnitTest(); help.Verify_DBUpdate(conn, query, expected);	Extraction Rule ParameterExtraction Context Parameters 1.XpathExpression 2.ContextParam This method extracts the parameter passed to the request and set to the contextparam Validation Rule UpdateDbValidation Context Parameters 1.ConnString 2.ContextParam 3.Query 4.ContextParam This method validates the contextparam and the values returned by the query
Transactional Verification	Verifies that web service DELETES records into a database	This method verifies the web service call results into deletion of existing record(s) from a given database	Method Prototype public void Verify_DBDelete(string ConnString, string Query) Method Invocation UnitTest help = new UnitTest(); help.Verify_DBDelete(conn, query);	Validation Rule DeleteDbValidation Context Parameters 1.ConnString 2.Query This method verifies the rows returned by query is 0
Workflow service	Verify that web service executed the workflow and end result is as expection	_________For this one we just need a test method in our test file instead of a separate helper method____	Sample code to verify the result returned by the workflow IDictionary<string, object> _d = new Dictionary<string, object>() { { "userArgument", arg } }; _d = WorkflowInvoker.Invoke(new Workflow1(), _d); Int32 outPut = (Int32)_d["OutputValue"]; Assert.AreEqual(expected, outPut); output can be validated with expected for its correctness	WorkFlow cannot be invoked. We can invoke the web service to verify the result returned by service method against the expected result. Extraction Rule ServiceResponseExtraction Context Parameters 1.XpathExpression 2.ContextParam Validation Rule ServiceResponseValidation Context Parameters 1.Expected 2.ContextParam
Authentication	Verifies that user credentials & certificate are validated against the server to authenticate the user trying to access the web service	This method verifies the user credentials and certificate to ensure that he is authenticated to get the results from the web service	var ServiceendPoint = new EndpointAddress(new Uri("http://localhost:91/Service1.svc"), EndpointIdentity.CreateDnsIdentity("MyWebSite")); var binding = new WSHttpBinding(); binding.Security.Mode = SecurityMode.Message; binding.Security.Message.ClientCredentialType = MessageCredentialType.UserName; var result = new Auth_Service.Service1Client(binding, ServiceendPoint); try { result.ClientCredentials.ServiceCertificate.Authentication.CertificateValidationMode = X509CertificateValidationMode.None; result.ClientCredentials.UserName.UserName = auth_username; result.ClientCredentials.UserName.Password = auth_passwd; time = result.GetServertime(); }
Encryption	Verifies that web service is encrypting the message/transport layer when returning data	This method will verify that web service is encrypting and decrypting the data as required by business.	Configuration appConfig = ConfigurationManager.OpenExeConfiguration(ConfigurationUserLevel.None); ServiceModelSectionGroup serviceModel = ServiceModelSectionGroup.GetSectionGroup(appConfig); Console.WriteLine("Configuration Name : Address : Binding : Contract"); foreach (ChannelEndpointElement endpoint in serviceModel.Client.Endpoints) { if (endpoint.Contract == "Auth_Service.IService1") { binding_used = endpoint.Binding; break; } else { } } Auth_Service.Service1Client result = new Auth_Service.Service1Client(); result.ClientCredentials.ServiceCertificate.Authentication.CertificateValidationMode = X509CertificateValidationMode.None; result.ClientCredentials.UserName.UserName = auth_username; result.ClientCredentials.UserName.Password = auth_passwd; string time = result.GetServertime(); TestContext.WriteLine("Bindings used by this contract is " + binding_used);
Caching	Verifies that data returned by called web service is returned from cache	_________For this one we just need a test method in our test file instead of a separate helper method____ Caching is frequently used as an optimization in distributed systems. It can be used to avoid re-doing computations or complex database access when the results remain valid for an extended period of time. In this case, subsequent requests for the same information can be served with the cached version rather than repeat the processing with the associated overheads.	string url = "http://localhost/Cache/Service1.svc/xml/12"; Uri uri = new Uri(url); WebRequest request = (HttpWebRequest)WebRequest.Create(uri); request.Method = "GET"; HttpWebResponse response = (HttpWebResponse)request.GetResponse(); Stream receiveStream = response.GetResponseStream(); StreamReader readStream = new StreamReader(receiveStream, Encoding.UTF8); string firstResponse = readStream.ReadLine(); Above code snippet shows to retrieve server response. To test caching two request should be made befor cache expires and should be validated.	Extraction Rule CacheExtraction Context Parameters 1.XpathExpression 2.ContextParam 3.Delay To test caching two request should be made before caching expire and one after caching expire. Parameter Delay has to be passed for the second request And plugin DelayPlugin should be associated with the second service CacheOutcomePlugin ContextParameters 1.ContextParam1 2.ContextParam2 3.ContextParam3
Caching	Verifies that data returned by called web service is returned from database when cache is expired	_________For this one we just need a test method in our test file instead of a separate helper method____	string url = "http://localhost/Cache/Service1.svc/xml/12"; Uri uri = new Uri(url); WebRequest request = (HttpWebRequest)WebRequest.Create(uri); request.Method = "GET"; HttpWebResponse response = (HttpWebResponse)request.GetResponse(); Stream receiveStream = response.GetResponseStream(); StreamReader readStream = new StreamReader(receiveStream, Encoding.UTF8); string firstResponse = readStream.ReadLine(); Above code snippet shows to retrieve server response. To test caching two request should be made before caching expire and one after caching expire and should be validated.
Non-XML Data	Verifies that called web service method sends/returns non-xml data (like blob, image) as expected	This method will verify that web service method is storing or retrieving the non XML data like blob from file system or database	Refer our reusable [TestMethod] public void VerifyNonXMLData()
Asynchronous messaging	Verifies that called web service method if implemented as asynchronous then returns the data as expected at a later point in time	This method verifies the asynchronous behavior of web service method as expected and implemented.	Refer our reusable [TestMethod] public void VerifyAsync_Response()
Exception and Error handling	Verify that service returns appropriate error message when there is a fault exception like underlying database connectivity issue to ensure exception and error handling	This method will verify that web service method that is called handles the exceptions like timeout, connectivity/networking issue as expected	<serviceDebug includeExceptionDetailInFaults="True" /> Set the IncludeExceptionDetailInFaults property to true to instruct to return managed exception information to the client in SOAP faults to ease debugging try { WCFServiceRef.Service1Client obj = new WCFServiceRef.Service1Client(); obj.Display(); } catch (FaultException e) { string str = e.Message.ToString(); TestContext.WriteLine(str); }	Extraction Rule ExceptionExtraction ContextParameter 1.contextParam Validation Rule ExceptionValidation ContextParameter 1.contextParam 2.ErrorMsg ExceptionPlugin ContextParameter 1.contextParam Plugin is used to pass/fail the test case contextParam name provided in extraction/validation/plugin should be same
Merge	Verifies the result obtained from multiple web services calls by merging them matches against consolidated expected output	This method verifies when similar result is obtained from multiple web services and then need to be validated against single expected result set Think of a scenario where you are getting similar result from various source - each a different web service call. The tester wants to validate the consolidated output from these services by comparing it against expected values	WCFServiceRef.Service1Client obj = new WCFServiceRef.Service1Client(); string expected = "5 Test"; string exmaple1 = obj.mergeExample().ToString(); string example2 = obj.mergeExample1(); string actual = exmaple1 + " "+ example2; Assert.AreEqual(expected, actual);	Extraction Rule ServiceResponseExtraction Context Parameters 1.XpathExpression 2.ContextParam The extraction rule is used with both the request to extract the response from the server Associate the validation rule with the 2^nd request Validation Rule MergeTwoServcResponseValidation ContextParameter Context Parameters 1. ContextParam1 2.ContextParam2 3.expected
DTO based Validations	DTO (Data Transfer Objects) based validation is a generic way to validate web services response of any data type against expected result by converting them into generics objects	It works by converting the service response (whether abstract or primitive data types) and expected results into standard generic objects which then are finally converted into XML for comparison with the help of our reusable library methods as described below in the example	public List<T> DTO_PopulateObject<T>(object DTOObject, object myObj) public List<T> DTO_PopulateDBResults<T>(DataTable dt) where T : class public bool DTO_XmlComparison(Object ExpectedDTOList, Object ActualDTOList, Type ObjectType) Examples: ExpectedStudentDTOList = helper.DTO_PopulateDBResults<Student>(dt); object ActualStudentDTOList = helper.DTO_PopulateObject<Student>(myObj, s1); bool result = helper.DTO_XmlComparison(ExpectedStudentDTOList, ActualStudentDTOList, typeof(List<Student>));

Tools to speed up Web Services Testing

WCF Load Test

This tool helps us in accelerating the generation of Skeleton unit test which can be used for load test of Application.

This tool takes the WCF or Fiddler2 message log file and a WCF client proxy, or a WCF interface contract, and generates a unit test that replays the same sequence of calls found in the message log file for WCF and ASMX services. The code generated is easily modifiable so that data variation can be introduced for the purpose of doing performance testing.

The tool will also integrate with Visual Studio 2005/2008 Team Edition for Software Testers and Visual Studio 2010 Ultimate and installs a wizard for creating the message log and processing it from within Visual Studio.

Tool can be downloaded from

http://wcfloadtest.codeplex.com/releases/view/51762

This tool is an add-in to VS and after running the .msi, open Visual studio and from the test menu, create a new test .It contains a template for the WCF Test.

1. Create a new Test Project and select the WCF test from the templates available for the test.

2. Click on Next

3. Give the path of the executable, it’s your application exe files, in this case it’s the sample App application which we are using for the generation of unit test.

4. Click on Run, will run the application and load the data as expected.

5. Test the application; change the values in the List of Students to fire the service with different set of Data. Once this is done, close the application.

And click on Next on the ‘New WCF Service Test Generator wizard’

6. There will be screen which depicts what all methods were invoked when we navigated the application with different set of inputs.

7. Click Next, Now we need to add the Dll/exe which define the contracts which are implemented for the WCF service which we have consumed.

8. Click on finish and the Unit tests are created for actions which we have performed on the Application.

Web Test Plug In for Silverlight + RIA based services

When testing Silver Light applications, we frequently run into dealing with non-Html content types. This is because most of the Silver Light applications use WCF RIA web services. Typically, this will be a binary format called MSBin1. Also, Silver Light applications may use other content types such as JSON, SOAP and REST to communicate with server. In general there is significantly greater use of Web Services and JSON in today's web applications.

The Web Test Plugins for Web Service Messages leverage these extensibility points to provide the following functionality for all these content types:

1. Select the Web Service plugin to enable Web Service message plugins before recorder shows up when we create a new web test project.

2. After Recording is completed it adds a reference to the runtime assembly to project, and adds the web service plugin to the Web Test

3. In recording result we can see a new tab named as “Web Service”

4. Right click on the selected node, and select “Add Extraction Rule” or “Add Validation Rule” from the context menu.

Reference- <http://blogs.msdn.com/b/mahipalkante/archive/2010/04/07/web-test-plugins-for-web-service-messages.aspx>

Web Services Automation Benefits

· Increase the quality of the application and find issues/bugs quicker and easily by validating web services for functionality without relying or testing the UI layer too much which is many times slower.

· Many times when UI is under development or for V1.0 projects where UI is changing a lot often, UI automation doesn’t give you’re the ROI in short term and Web Services automation can help you quickly validate the functionality and business layer.

· The effort and investment on automating Web Services for functionality will also pay off later to quickly build you performance test suite with lesser effort and reuse.

· This will also help to scale up when doing performance testing to ensure that web services behave as per SLA and as-expected when put under lot of stress and load.

· As a performance engineer, you will be able to do effective performance (load/stress) testing of web services along with validating their behaviour.

· As developer/consultant, you will be able to use web services automation as part of continuous Integration to improve the build and code quality.

· As a support engineer, you can also run automation tests quickly to find the culprit such an unintentional regression impact and that will help in debugging and troubleshooting critical failures.

Appendix

i. Getting Started with VS 2010 Unit Test & Web Test

Walkthrough: Creating and Running Unit Tests

Web Performance Test Walkthroughs

Introduction to WCF Testing

ii. Deploying Sample Web Services Application

Please find below the steps to get started with our sample web services

Step 1: Deploy sample DB

From our sample solution, run Create_Sample_DB.sql which will create the Database and Tables required for our sample app.

Step 2. Deploy our services

a) Sample WCF Service –

This project contains the WCF services will be consumed by Sample_APP_UI. It contains the contract as well as the implementation of the contracts. This project also uses the LinQ feature for integration with the database.

Below is sample step to deploy services on your machine

i) Launch solution and publish “WcfService” project to FileSystem

ii) Add an application to your IIS and provide the File system directory and App pool details

iii) Browse and test your services in browser or using WCFTestClient

Browser: From IIS, select your application and press Browse

WCFTestClient:

- Launch VS Command Prompt and type WCFTESTCLIENT

- Add URL of your web service

- Click on any test method, provide request parameter and invoke

b) Sample ASMX Web Services

This project contains the services which are built on ASMX types. These services are also consumed by the Sample_App_UI.

For deployment follow the steps given in above step (a).

Note: You can skip this as well if your scope doesn’t include testing ASMX based services

c) Sample Windows Workflow Foundation Services

This project contains the sample WCF service which is consumed by the workflow application.

For deployment follow the steps given in above step (a).

Note: You can skip this as well if your scope doesn’t include testing Workflow based services

d) Sample Authentication Project

This project contains the WCF service which entails the authenticate service. We have used the “UserNamePasswordValidator” validate method for the verification of the client credentials to authenticate the client.

The main intent is to check the authentication of the client, so this service has 1 contract implemented “GetServertime” which returns the current time at the server.

Note: You can skip this as well if your scope doesn’t include testing Authentication of your services

e) Sample Non-XML Data Project

This project contains the WCF service which is required for returning the non xml data from the service. For this case we are using the service to return the image file using the stream.

There are two response from the service implemented under this project.

- Buffered response

- Streamed response

Note: You can skip this as well if your scope doesn’t include testing of Non-XML based response from web services

f) Sample Web Services Caching Project

This project contains a WCF service which contains the service representing the cache behaviour of the WCF service. For this project we have leveraged the Asp.Net cache in the WCF. The response is cached for 60 seconds if the request URL doesn’t changes. On changing the parameter to the request URL or elapse of 60 seconds the call again to this service returns the fresh response from the service.

Note: You can skip this as well if your web services don’t use caching

g) Sample Asynchronous Web Services Project

This project shows another feature of the WCF asynchronous call where in client sends a request to the server and continues with the regular task and on completion of the service i.e. response comes from server the required action is performed using the response received.

For this example we have used client making the Asynchronous call using the call-back mechanism. This project contains the contract as well as the Windows client which uses the service return.

Note: You can skip this as well if your web services don’t use Asynchronous calls

Step 3. Run Sample Client apps

a) Sample UI application

This is the main project in the solution which consumes different services WCF and ASMX. This project is a sample app to demonstrate the services when consumed from UI and their return

Note: Please update the service reference to point to the URL of the our sample web services (WCF and ASMX) as shown below by using “Configure service reference”

You can now run sample UI app that consumes WCF and ASMX based services.

b) Sample Authentication Console Application

This project in the solution contains the client which will test service, and provide the valid credentials so that the service returns the valid return.

This is a console based application which hosts the Auth_Service and send the valid credentials to test it.

c) Sample Workflow Console Application

This is a Workflow project which uses the WCF services with in the workflow, and executes the rest of action in the workflow.

This project is just a sample to be demonstrated on how the services are integrated with the WF app.

iii. Sample Restful Services

iv. Request/Request XML for services not supported by Fiddler/ WCFTestClient

Working on a project where we create WCF Services there may be a need to capture the raw xml passed in to the service operation and also capture the reply xml sent back by the service. Our reusable code captures raw xml even for the scenarios (e.g. Non Xml data being returned by service) where WCF Test Client or fiddler are not able to trace the request\response xml.

Reusable Code snippet:

ImageServc.StreamingServiceClient ClientHandle = new ImageServc.StreamingServiceClient();

ResponseXmlInterceptingBehavior responseXmlInterceptingBehaviour = new ResponseXmlInterceptingBehavior();

ClientHandle.Endpoint.Behaviors.Add(responseXmlInterceptingBehaviour);

OutputFile = ClientHandle.GetData("Dryden");

string requestXml = responseXmlInterceptingBehaviour.ResponseXmlInterceptor.RequestXml;

v. Binding & Hosting configuration & consideration

This project one of the features of WCF to host the WCF service on the console. In this project we have used the basic test for the call of contract/method of WCF which is hosted on the console app.

Please refer the snapshot which will give you an idea on hosting the service on the console.

Another sample unit test

vi. Data driven testing

Use below link if you need more information about Data driven testing using Coded UI and access/store data in MS EXCEL from Coded UI

Data Driven Testing by Mathew

http://msdn.microsoft.com/en-us/library/ee624082(VS.100).aspx

http://msdn.microsoft.com/en-us/magazine/cc163536.aspx

Sunday, December 18, 2011

Friday, December 16, 2011

Sunday, December 11, 2011

Importance of Web Service Automation

Test Approach – Identify your goal (Functional Test or Performance Test or Both)

Web Services Automation – Unit Test Walkthrough

Creating a new Unit Test Project / Reusing Sample Template Unit Test Project

Web Services Automation – Web Test Walkthrough

Web Service Performance Testing using Unit Test/Web Test + Load Test

ii. Deploying Sample Web Services Application

iii. Sample Restful Services