PaaS testing

PaaS (Platform as a Service) is an application delivery model which provides services to design, develop, test, deploy and host applications. It plays a key role in cloud computing space similar to the other services, IaaS (infrastructure as a service) and SaaS (software as a service). Google App engine, vmForce and WSO2 Stratos are some of the popular PaaS offerings.
This post is not about detailed concepts of PaaS or cloud computing. Rather, I'm going to look at the testing and quality assurance aspects of PaaS. PaaS testing is not a widely discussed topic. There is no pre-defined model for PaaS testing. We need to explore the features of PaaS and derive an approach for testing PaaS apps. I will go through some basic components based on my experience of testing WSO2 Stratos opensource cloud platform.

Hosted services testing - Minimized deployment overhead

In simplest terms, PaaS testing is about testing a hosted platform. So, there is no deployment and configuration overhead for the test team. Testers are expected to access the hosted services remotely using web browser and carry out functional and non-functional testing.

Lesser test platform combinations

When you test a standalone product or a combination of products, you should try out the possible platform combinations such as different types of clustering setups, multiple DBMSs, Application Servers or various operating systems. However, when you test the same product suite or platform on cloud (PaaS), you are bound to one optimum configuration stack. You have one or two choices. There is no requirement to try all possible platform setups.

Multi-tenancy aspects

Multi-tenancy allows a single application to emulate multiple application instances. With multi-tenancy, a single application can be shared across many organizations. Therefore, series of tests should be carried out carefully to verify the multi-tenant aspects. For example, if company A logs in to a web application hosting service in your PaaS offering and deploy a web app, then the users of company B should not be able to locate it unless company A made it public. In other words, the configurations done by one tenant should not be exposed to other tenants.
These mission-critical aspects are some of the key requirements which should be taken into account in any PaaS testing model.

Performance and scalability

Any hosted service should conform with the SLAs. Similar to a website, services included in a PaaS can be accessed by multiple users concurrently. Therefore, sufficient amount of performance testing must be carried out. The usual usage pattern of standalone technology platform can be dramaticallty changed when it is hosted as a service in cloud. For example, the general use cases of WSO2 Carbon SOA middleware platform are different from the standalone version when the platform is exposed as a service. WSO2 SOA middleware platform consists of 12 different products and all of them includes a management UI for administration purposes. In non-cloud based deployment, the multiple, concurrent users access to the management UIs are minimum since it is very rare that hundred of administrators accessing management console at once.
However, if this platform is hosted as a service in cloud (PaaS), then the concurrent user access to the management UIs are a highly desirable scenario.

It is a requirement that the hosted platform should be able to handle load seamlessly without affecting consumers. Auto-scaling is a key feature of WSO2 Stratos cloud platform. When new resources are needed, WSO2 Stratos transparently adds services and when load goes down, WSO2 Stratos automatically brings services down. Testers should ensure, with the load, the new EC2 instances spawning up and down correctly. The tools like hybridfox are very useful in these situations.

Data

If your PaaS provides users with the ability to store their data in cloud and various database centric operations, then the testing will be much complicated. As a tester, you must ensure the availability of data sources, accessibility and provisioning aspects. You must also take extra care that, if the data are geographically distributed in cloud, then it adheres to the legal requirements of the users of your PaaS.

Security

Security is the utmost important aspect of any cloud offering. It is recommended to have a separate group of security testers who do penetration testing and ethical hacking in order to ensure secure infrastructure of your PaaS. WSO2 Stratos cloud platform allows users to deploy their webapps, web services and various custom code. Therefore we must ensure that no vulnerable code are deployed and a code deployed by one tenant does not affect the operations of others or whole platform.

Virtualization

Public PaaS offerings are usually based on virtualized infrastructure such as Amazon EC2. Therefore, a full functional test cycle should be carried out on EC2 VMs inorder to make sure that there are no regressions due to virtual servers.

In addition to the above key areas, PaaS testing model should consists of the general web application testing aspects such as cross browser testing, accessibility testing, localization testing etc..

Data driven testing with Jmeter user parameters

This is a follow up to one of my previous posts which explained data driven web service testing using CSV config element in Jmeter. There, we used CSV file to read input data for SOAP/XML-RPC sampler.

In this post, we will look in to using User Parameters pre-processor element as the data source instead of a CSV file.

Step 1

We are going to use the same web service which we used in my previous post, temperature conversion service. Please add the SOA/XML-RPC sampler, the SOAP request and the necessary thread group as described in step 1 and 2 of that post

Step 2

Lets parameterize the payload of SOAP message so that different requests will be sent to the service with each test run. Instead of reading data from a CSV file, we can add a User Parameter pre processor element in Jmeter test plan.

Right click on the thread group of your jmeter test plan and select Add --> Pre Processors ---> User Parameters

Click on Add Variable and specify celcius as the name of variable. Add few users and enter celcius values for each user as shown below.

Step 3

Now, parameterize the payload of SOAP as follows.

<tem:nCelcius>${celcius}</tem:nCelcius>

Step 4

Increase the thread count corresponding to the user count in your user parameters pre-processor element and run the test. You will notice that the Celcius figure will be varied in each request.

Based on your requirements, you can select either CSV config element or User Parameter pre-processor element for data driven testing. If you have large number of variables to be parameterized, CSV config is the best option.

WSO2 Carbon-3.2.0 - The latest with many new features and a release to remember

On 12th of June 2011, the latest version of WSO2 SOA middleware platform, WSO2 Carbon-3.2.0 has been released. Since 2009 January, WSO2 SOA middleware platform has been grown and matured and now it consists of 12 different products.

WSO2 Application Server

WSO2 Enterprise Service Bus

WSO2 Mashup Server

WSO2 Identity Server

WSO2 Governance Registry

WSO2 Data Services Server

WSO2 Business Process Server

WSO2 Business Activity Monitor

WSO2 Gadget Server

WSO2 Business Rules Server

WSO2 Complex Event Processing Server

WSO2 Message Broker

In this latest version, two new members joined into WSO2 Carbon product family. Those are WSO2 Complex Event Processing Server and WSO2 Message Broker.

In this post, I'm not going to explain the new features and enhancements included in each of these products. You can find them out in the respective product pages in wso2.com.

But, I would like to share how we evolve in terms of features, process and release methodology during the last couple of years.

In late 2008, we started the implementation of WSO2 Carbon platform. Before that, we had four isolated products (WSO2 ESB, WSO2 WSAS, WSO2 IS and WSO2 Registry) but those had limitations to work as a platform. The major objective of WSO2 Carbon platform was to build a suite of products which can be integrated easily and run together with minimum configuration overhead. At the end of the first WSO2 Carbon platform release (version 1.5.0), we were able to achieve that objective to some extent. The first release was a VERY different experience for us since we used to do releases with one product at a time so that the test team had sufficient time and resources to try out various scenarios. Because of that, we had to move in to a totally different testing paradigm.

In early 2009, we had to do a patch release of WSO2 Carbon platform (version 1.5.1) because we uncovered some critical issues after the first Carbon release. With this, we badly felt the need of adjusting our testing methodology to align with WSO2 platform vision. We spent hours in each build to test management console UIs manually and our first objective was to cut down that time and effort. In order to do that, we started developing a selenium based test automation framework. In 2009, we did 3 WSO2 Carbon platform releases (version 2.0.0, 2.0.1, 2.0.2) and we were able to make use of selenium tests heavily in all those release cycles.

In 2010, we went through a major UI refactoring process across the whole WSO2 Carbon platform. Unfortunately, we could not keep maintaining selenium tests with the pace of UI changes. Therefore, we wanted to have a test framework with minimum maintenance overhead. We wanted to call methods of various features bypassing UI elements so that the frequent UI changes do not break the tests. We started admin services based test automation framework, which used to call the operations of administration services associated with each of the features programmatically (Java/Junit).

2010 was an extremely busy year for most of us since we did a lot of frequent releases, but there were very limited automated tests. At the end of 2010, WSO2 Carbon became a fully componentized, modular SOA middleware platform which consisted of multi-tenancy and a lot of usability enhancements. As the testing team, we kept on observing the growth of WSO2 Carbon platform more than anyone else since we deal with almost all aspects of each and every product. As I said at the beginning, one of the objectives of WSO2 product platform was to provide users with ease of integration. By end of 2010, we realized how far we achieved that. Installing a new feature on any of the WSO2 Carbon based product became a simple click'n'click process. Most features worked out-of-the-box.

Now, after 2.5 years of the first Carbon release, our product platform has been improved in various aspects with 12 different products. Our development and testing methodology has been changing with each release to accommodate the pace of growth and objectives in WSO2 SOA platform. In the latest 3.2.0 release, our focus shifted towards more platform level testing, which required us to learn and familiar with each product. With the platform aspect, we cannot do individual product releases. We do all 12 products at once, which demands huge work load on both development and testing teams. At WSO2, we do not trust or rely on the number of testers in the team but their individual capabilities. We realize the need of having 80% of test coverage through some kind of automation in order to sustain with the pace. For all these years, we do not adopt or follow any specific process blindly because that works for some other organization so that we do. Instead, we are following a context-driven testing approach. We learn and enhance our processes with our own mistakes and experiences. While WSO2 is revolutionizing the world of enterprise middleware, testing and development methodology of middleware will also be redefined.

WSO2 SOA middleware Deployment Tips - 1 - validationQuery to avoid broken DB connections

I thought to put together some best practices, guidelines on deploying WSO2 SOA middleware platform. This post will remind you a well-known best practice used in most of the product deployments.

When you are maintaining DB connections, it is always recommended to use a validationQuery to check the health of the TCP connection of the connections stay in DB connection pool.

Because the connection opening is an expensive and time consuming operation, after a connection is created, it will be kept open for a specific time in the pool. When re-using these connections from the pool, there can be situations that the TCP connection to the DB is interrupted and the connection consumer gets errors such as communication link failures etc..

In order to avoid that, a validationQuery, an SQL statement specific to DBMS type, can be used which runs before using the connection.



In WSO2 middleware platform, you usually use central governance/configuration registry to store SOA metadata and various governance tasks. Governance registry supports multiple DBs as underlying data stores. You establish the connection to DB in CARBON_HOME/repository/conf/registry.xml as follows.

<dbConfig name="wso2registry">

<url>jdbc:mysql://localhost:3306/config_db</url>

<userName>>regadmin</userName>

<password>regadmin</password>

<driverName>com.mysql.jdbc.Driver</driverName>

<maxActive>50</maxActive>

<maxWait>60000</maxWait>

<minIdle>5</minIdle>

</dbConfig>

This is one of the places where you may experience DB connection issues as I explained above. Therefore, it is always a best practice to use a validationQuery.

If your DBMSs is MySQL or MSSQL, then use the following.

<validationQuery>SELECT 1</validationQuery>

In Oracle;

<validationQuery>SELECT 1 FROM DUAL</validationQuery>

In Postgres,

<validationQuery>SELECT version(); </validationQuery>

Similarly, when you make DB connections in WSO2 Data Services Server, make sure to use validationQuery in data source definition section as follows.

<property name="org.wso2.ws.dataservice.validationquery">SELECT 1</property>

Monitoring WSO2 ESB using JConsole

My latest tutorial on how to monitor WSO2 ESB using JConsole is now published on WSO2 Oxygen Tank

How to send JSON messages to web services deployed on WSO2 Application Server

JSON (JavaScript Object Notation) is an open and text-based data exchange format, that provides a standardized data exchange format better suited for Ajax style web applications. You can find more information about JSON from www.json.org

WSO2 SOA middleware platform supports sending and receiving JSON messages.

This post takes you through the steps to deploy a simple web service in WSO2 Application Server and write a client using Axis2 ServiceClient API to invoke the service by sending JSON message.

Pre-requisites:

Download and install WSO2 Application Server 4.0.0 or later

Step 1

We are using a simple web service, which echo's user input as follows.

public class EchoService {

 public OMElement echo(OMElement element) {
           return element;
    }
}

You can download the service archive (EchoService.aar) from here and deploy on WSO2 Application Server.

Step 2

Have a look at CARBON_HOME/repository/conf/axis2.xml (CARBON_HOME is the root directory of WSO2 Application Server). You will notice that the JSON specific message builders and formatters are enabled by default.

<!--JSON Message Formatters-->

<messageFormatter contentType="application/json"

class="org.apache.axis2.json.JSONMessageFormatter"/>

<messageFormatter contentType="application/json/badgerfish"

class="org.apache.axis2.json.JSONBadgerfishMessageFormatter"/>

<messageFormatter contentType="text/javascript"

class="org.apache.axis2.json.JSONMessageFormatter"/>

<!--JSON Message Builders-->

<messageBuilder contentType="application/json"

class="org.apache.axis2.json.JSONOMBuilder"/>

<messageBuilder contentType="application/json/badgerfish"

class="org.apache.axis2.json.JSONBadgerfishOMBuilder"/>

<messageBuilder contentType="text/javascript"

class="org.apache.axis2.json.JSONOMBuilder"/>

WSO2 Application Server accepts any JSON message with the content type application/json, application/json/badgerfish or text/javascript. In this example, we will use a message with the content-type, application/json

Step 3

Now, If you invoke the service using the Tryit utility associated with EchoService and trace the message, you will notice the payload of the SOAP message as follows.

<p:echo xmlns:p="http://service.carbon.wso2.org">

<echo xmlns="http://service.carbon.wso2.org" xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="xs:string">charitha</echo>

</p:echo>

Basically, the message payload will be similar to the following.

<echo>

<value>charitha</value>

</echo>

Step 4

Now, we are going to send the above payload as a JSON message. The JSON format of the above message payload will be as follows.

{"echo":{"value":"charitha"}}

Add the following class to your java project. Compile it using the libraries included in CARBON_HOME/repository/component/plugins directory. You may also need to add CARBON_HOME/lib/endorsed into your class path.

package org.wso2.carbon.service;

import org.apache.commons.httpclient.HttpClient; import org.apache.commons.httpclient.methods.PostMethod; import org.apache.commons.httpclient.methods.RequestEntity; import org.apache.commons.httpclient.methods.StringRequestEntity;

public class JSONClient {

    public static void main(String[] args) throws Exception {
          String strURL = "http://localhost:8281/services/JSONProxy/";                  PostMethod post = new PostMethod(strURL);                RequestEntity entity = new StringRequestEntity("{\"echo\":{\"value\":\"charitha\"}}","application/json", "UTF-8");         post.setRequestEntity(entity);         HttpClient httpclient = new HttpClient();         try {             int result = httpclient.executeMethod(post);             System.out.println("Response status code: " + result);             System.out.println("Response body: ");             System.out.println(post.getResponseBodyAsString());         } finally {             post.releaseConnection();        }    } 
}

You could use the same axis2.xml located at CARBON_HOME/repository/conf as the client axis2.xml when creating ConfigurationContext

If you forward the message to tcpmon and trace it, you will see the request as follows.

POST /services/EchoService HTTP/1.1

Content-Type: application/json; charset=UTF-8

User-Agent: WSO2 WSAS-3.2.0

Host: 127.0.0.1:9764

Transfer-Encoding: chunked

1d

{"echo":{"value":"charitha"}}

0

Data driven web service testing with Apache Jmeter

You will not get enough coverage in your web service testing, if you repeatedly send the same request to the webservice under testing. You must read data from a data source and parameterize each request. If you are dealing with SOAP messages, you should parameterize SOAP message payload.
This post guides you how to do data driven web service testing using Apache Jmeter.

Pre-requisites:
Install Apache Jmeter 2.3.4 or later

Step 1

We are going to invoke a publicly available web service, Temerature Conversion service . The WSDL of this web service can be accessible through http://webservices.daehosting.com/services/TemperatureConversions.wso?WSDL
You can invoke the CelciusToFahrenheit operation of this webservice using SOAPUI and capture the request message. It will be similar to the below.

<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:tem="http://webservices.daehosting.com/temperature">
<soapenv:Header/>
<soapenv:Body>
<tem:CelciusToFahrenheit>
<tem:nCelcius>37</tem:nCelcius>
</tem:CelciusToFahrenheit>
</soapenv:Body>
</soapenv:Envelope>
Step 2

Lets create a new JMeter test plan and add a thread group. Then, add SOAP/XML-RPC sampler into the thread group and copy and paste the above SOAP request into the SOAP/XML-RPC data section of the sampler.
Enter the URL of the service as http://webservices.daehosting.com/services/TemperatureConversions.wso (You can capture this from the location attribute of the address element in the WSDL)
Now add a listener to view the result.
Save your test plan. Your test plan will be similar to the below.



Run the test and check the results. You will get the Fahrenheit value of the provided Celsius figure.

Step 3

Now, you can increase the thread count and extend this test into a performance test. However, in that case you will be sending the same request again and again. It will not be a good simulation of a real-world scenario. You should be able to alter the payload (in our example, Celsius value) of the SOAP request with each thread.
In order to do so, you should read Celsius data from a data source. In Jmeter, you can easily read data from a csv file.
Lets create a csv file, temperature.csv and save it in the location where you saved the above JMeter test plan.
Enter a set of values row-by-row in the temperature.csv
eg:-
10
20
30
40

Step 4

Next, we will add CSV Data Set Config element which will read data from the csv file.
Right click on Thread Group and select Add --> Config Element ---> CSV Data Set Config
Now you can configure your CSV data source as follows.

Filename: <Give the full path of temperature.csv>
Variable Names: celcius

Keep the other values intact.

Step 5

Now access the SOAP/XML-RPC Data section of the request and replace the hard-coded celcius figure with the variable name we have configured in CSV Data Set Config element (i.e:- ${celcius})
After parameterizing, your SOAP request will be as follows.

<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:tem="http://webservices.daehosting.com/temperature">
<soapenv:Header/>
<soapenv:Body>
<tem:CelciusToFahrenheit>
<tem:nCelcius>${celcius}</tem:nCelcius>
</tem:CelciusToFahrenheit>
</soapenv:Body>
</soapenv:Envelope>

Increase the thread count corresponding to the row count in your csv file and run the test. You will notice that the Celcius figure will be varied in each request by reading data from CSV data source.

In this way, you can easily do data driven web service testing using Jmeter.