Effective Test Automation and WSO2 System Test Framework

The automated tests are an absolute necessity in agile software development. However,  the true benefits of test automation can only be achieved only through a systematic process which should consist of;

• Focused automation strategy

• Dedicated teams to maintain and manage test automation infrastructure

• Firm commitment by everyone in engineering NOT specific to a separate testing team

For the past few years, we tried to build various types of automation test frameworks as part of WSO2 Carbon middleware platform. Our first attempt was to build a Selenium based test framework back in 2009. We were able to use that for four release cycles in that year. Later on, the drastic architectural and UI changes made the framework unmanageable hence we could not continue use it.

WSO2 System Test Framework (A.K.A WSO2 Clarity) came in to picture during 2011. The new test framework development was handled by a separate test automation team headed by Krishantha. The team overcame many challenges and worked like crazy to build a test framework which would serve not only for one or two release cycles but as a long-lasting test automation solution.

As I mentioned in many of my previous blog posts, it takes time to achieve the real advantages of test automation. The one most important thing is, everyone needs to be patient. You cannot achieve greater test coverage in overnight. WSO2 Clarity test framework is primarily driven through admin web service APIs. There is a little chance to break tests in between releases when comparing to traditional UI oriented automated tests.
The future releases of WSO2 middleware products and cloud services will evaluate the success or failure of the new WSO2 Clarity test framework. But personally, I'm experiencing the true value given by the new framework in day-to-day work which are carried out at WSO2.
WSO2 Clarity framework is an integral part of WSO2 continuous integration system (CI). The tests are automatically executed as part of the build process. However, we usually have many requirements to execute tests externally instead of gluing into build system. For example, we patch the products, we do configuration changes such as change the underlying OS, JDK or DBMS etc.. In such cases, a test framework which is tightly coupled with build system does not help much. Because of that, we wanted to have a mechanism to launch tests externally from the maven build. On the other hand, we wanted to have a simple mechanism to run tests without building massive Carbon code base.
Thanks to Krishantha and test automation team,  binary deliverable of Clarity test framework has been provided as a solution for this long outstanding requirement. A preview version of it can be downloaded from https://svn.wso2.org/repos/wso2/people/krishantha/wso2clarity-1.0.5.zip

This will surely help anyone who uses WSO2 products/services not specific to internal testers and/or developers. How?
Suppose, you download the latest version of WSO2 ESB and want to customize default settings (e.g:- you simply want to deploy the product in IBM JDK and change the default H2 registry database to DB2). Now, you want to make sure there are no adverse effects due to those changes. You can download the Clarity binary and launch all ESB integration/system tests. The test runner is completely ANT based and you just need to have Apache ant. Update the clarity configuration file by modifying the host, IP etc (as mentioned in INSTALL.txt inside Clarity binary distribution) and run the tests you need.
Note that, we still have a preview version of Clarity binary. This will eventually be part of product downloads.

WSO2 QA Test Framework - Fundamentals

WSO2 QA Test framework has been developed to replace repetitive manual test procedures followed during the release cycles. We identified the tests which provide much ROI with automation and used them for phase1 of the test framework development project.
With the introduction of WSO2 Carbon product platform in late 2008, all java based products are implemented using the base carbon platform. All products (9 all together) are released at the same day which is a very different experience specially for a QA team. All products are supposed to go through a set of common tests and a set of product specific tests. All are supposed to work on multiple application servers, multiple JVMs, multiple browsers, multiple operating systems, multiple DBMSs etc.. hell a lot of test combinations! The product count exceeded the number of people in test team. More products are expected to be introduced in near future. Therefore, the only viable solution to manage the QA process is to automating as more tests as possible.
So, we started implementing our automated test framework in 2009 March.
As I discussed in some previous posts, we chose selenium for automation. We used Selenium Remote Control Java client driver to drive selenium tests with Junit. However our tests were not restricted to web based selenium scripts. We have written numerous tests which used Axis2 ServiceClient API and some other API methods to invoke web services, sending messages via secure channels, reliable messaging, message mediation etc.
We derived our project structure which adheres to maven as given below. You can get a SVN checkout of 2.0.3 branch of the test framework (which is the most stable version at the moment) from https://wso2.org/repos/wso2/branches/commons/qa/web-test-framework/2.0.3

common
bps
registry
wsas
esb
gs
mashup
is
ds

Each project is built using its pom.xml at the root of the project directory. We used selenium maven plugin and surefire plugin to start selenium RC server and launch Junit tests respectively.
commons project is used to maintain tests which are used in all products. For example, org.wso2.carbon.web.test.common.RegistryBrowser is common for all products since all products have a common registry browser. We used svn:externals to link the common classes to relevant projects.
You can find all common classes at https://wso2.org/repos/wso2/branches/commons/qa/web-test-framework/2.0.3/commons/src/test/java/org/wso2/carbon/web/test/common/

In addition to that, commons project is used to store the test artifacts which are shared among multiple products. For example JDBC connector jars and keystores are used in all products and those are stored in commons/lib directory.

The other most important resource included in commons project is framework.properties configuration file. It is used to configure the test framework according to the test environment. Under the "global properties" section of the framework.properties file, you could find the following properties.

host.name=172.16.37.1
http.port=9763
https.port=9443
carbon.home=/home/charitha/products/wsas/wso2wsas-3.1.3
context.root=/wsas
browser.version=firefox
admin.username=admin
admin.password=admin
module.version=2.03

You must change these properties as per your test environment settings. http.port and https.port are the embedded tomcat servlet transport ports used in WSO2 Carbon products. (full explanation of these properties will be included in a future post)

Lets look at a product specific test suite. Go to wsas directory in your test framework checkout. This is the project used to run WSO2 WSAS specific tests. You will find the following files and directories at the root of this project.

lib
src
pom.xml
runAll.sh
runAll.rb
wsas_test_suites.txt

lib directory is used to store the test artifacts specific to WSAS such as axis2 services, jaxws services etc.
You can invoke tests using two different ways.

• Direct maven invocation - you can run tests individually by passing a system property as follows

mvn clean install -Dtest.suite=usermanagement

• Run tests through a shell script - You can run tests individually or as a whole suite or few tests at once using this way. Here, it is required to uncomment the selected tests in wsas_test_suites.txt. The runAll.sh shell script read the test names from wsas_test_suites.txt and invoke each test.

sh runAll.sh

In any of the above mechanisms, we call a central test suite class which takes care of calling the individual tests. For each project we have a separate AllTests.java class which extends junit.framework.TestSuite parent. In our example, org.wso2.carbon.web.test.wsas.AllTests.java is the TestSuite.

See https://wso2.org/repos/wso2/branches/commons/qa/web-test-framework/2.0.3/wsas/src/test/java/org/wso2/carbon/web/test/wsas/AllTests.java for more details about this class.

You will also notice in AllTests class that initBrowser() method of the BrowserInitializer class is called to launch the browser instance for a particular test as follows.

public synchronized static void initBrowser()throws Exception{


if (browser == null) {
browser = new DefaultSelenium("localhost", 4444, "*"+property.getProperty("browser.version"), "https://" + property.getProperty("host.name") + ":" + property.getProperty("https.port"));
browser.start();
browser.setSpeed("200");

}
}

You can easily try out WSO2 QA test framework once you download and start using any of WSO2 Carbon based product. This is not a detailed explanation of the all features available in our test framework. I will guide you through more information in future posts. If you encounter any issues while using the test framework, please drop us a mail - architecture@wso2.org

Stay tuned.. will post more on automation soon!

Google Automation - Automated testing search engine

If you are looking for software test automation information, you will get the best results by using Google Automation search engine. It only searches the sites that matter most to automators.

Key factors for successful test automation

At WSO2, we have been making considerable progress in building a complete test automation solution. According to my experience on that effort, I think the followings are the key factors contributed to a successful test automation project.

• Dedication to automation (Instead of treating it as a spare-time activity)

• Commitment by the entire team (rather than just one or two testers),

• Commitment to automation from the start (rather than trying to automate a manual process later)

• Making use of correct tools/frameworks/technology

• Allocating sufficient time and resources
  

Automated testing of wso2 products using Selenium

I have never been a fan of open source UI test automation tools due to the annoying issues of majority of them. Most of the tools provides you with a good first user experience however fails when try to use in advanced use cases. At WSO2, we have been looking for a test automation tool during the last few years. Due to the tight release schedules, we were not able to focus much on researches.
At the end of 2008, WSO2 started to build its SOA product suite based on revolutionary Carbon platform and a new JSP based UI framework has been designed. When I got the initial builds of new WSO2 carbon products, I was curious about the possibility of automating user interface. I tried with one of the tools which we have tried once but never got a chance to continue working on. That is, Selenium! For me, it is the best open source tool I have used so far in my career.
After the releases of WSO2 carbon based products, we have started to automate UI and functional use cases using Selenium. We are making a rapid and steady progress so far with this great tool.

We use Selenium RC with Junit. All the test are written using Junit. WSO2 products are built using maven2 and eventually our test framework will be integrated to build system. We have already used maven selenium plugin and surefire plugin to integrate our tests with maven. With the power and high flexibility of Selenium we are in a good position of cross browser testing.

While automating the product UIs with selenium, we had to overcome some blocking issues but thanks for the materials available in openQA forums and some nice blogs, we were able to resolve most of them. I'd appreciate Selenium team for their great vision and innovativeness of developing such a helpful free web automation tool.

Web application testing in Ruby(Watir) - 2 minutes guide

Watir (pronounced as Water) is a free open source tool which can be used to automate web applications. It is an extension of Ruby programming language. Unlike most of the other testing tools it gains the advantage of powerful features of Ruby and simulate browser interactions in very simple manner.
Lets see how a simple google search is automated using Watir in few steps.

Pre-requisites
Install Ruby (1.8.5-24 or later)

Step 1

Install Watir. Open a command window or shell and issue the following commands
gem update --system
gem install watir

The above two commands update gem installer and then install watir in your system.

Step 2

Open SciTE ruby editor or notepad and start to create the following script.

require "watir"

ie = Watir::IE.new

ie.goto("http://www.google.com")

ie.text_field(:name, "q").set("WSO2 WSAS")

ie.button(:name, "btnG").click


Step 3

Save the above file as SimpleTest.rb and run it from the command line by typing SimpleTest.rb
You will see that an Internet Explorer browser instance will automatically be popped up, access google, type "WSO2 WSAS" text and click on search button as a user interacts with the web site.

Step 4

Lets see what each of the above statements of our test script do.

require "watir" - This is similar to an import statement. This tells ruby script to use Watir as an extension library

ie = Watir::IE.new - Instantiate a new IE browser instance and open it

ie.goto("http://www.google.com") - Instructs IE instance to access google.com

ie.text_field(:name, "q").set("WSO2 WSAS") - Set text, "WSO2 WSAS" as the search query

ie.button(:name, "btnG").click - Click the "search" button

If you need to simulate web interaction with Firefox, you can use FireWatir, which allows to write test scripts for Firefox browser.