I found this nice article at stickyminds.com, which discusses about identifying bugs in an application if you don't have any specs, reference programs, docs etc..
Excerpt from the article;
" A bug is something that bugs somebody who matters. Testers can't be sure that something will bug someone, and so we must apply heuristics--useful but fallible ways of solving a problem or making a decision. Oracles--heuristic principles or mechanisms by which we might recognize a bug--include comparable programs, documents, mathematical principles, personal experience, and people--"live oracles."
Emotional reactions are associated with a psychological and physiological phenomenon called "arousal," which is the state of becoming alert or awake. For good testers, emotional reactions are oracles--trigger heuristics that wake us up to the possibility of a problem. Although emotional reactions don't come with a guarantee, they can often aid our decisions about the existence or the significance of a problem.
We sometimes depend on logical and objective assessments of software at the expense of other ways of thinking--ways that include feeling. Most of the results that this system is returning are functionally correct, but that correctness doesn't matter when there are problems that interfere with the customers' goals. If we engaged emotions and empathy more often and more consciously, they would point us rapidly to things that are important to people. Try testing a program while simply pretending that we have some skin in the game.
Our customers constantly find bugs in our systems without any preparation, documentation, reference programs, or live oracles. Customers test the product by using it, and they recognize bugs and assess their significance and the associated value of our products. They can do that. So can we."
Have a look!!!!
Tuesday, January 15, 2008
Monday, January 14, 2008
Performance benchmarking of data services
This is a follow up post of my previous blog entry Call MySQL stored procedures using WSAS data services.
Hence, I'd recommend the steps given there should be exercised before going through this post.
I'm going to explain how I analyzed the performance of WSAS data service using ApacheBench, which is an embedded benchmarking utility comes with Apache httpd server. The procedure is very simple and can be enhanced with more scenarios as you wish.
Step 1
Assuming the Apache web server is installed in your system, go to Apache_home/bin and run AB.exe
You will see a set of options in the command prompt.
Step 2
Invoke the data service which has been created in the previous post using Tryit utility. Make sure to switch on the 'Soap Tracer' so that we can capture the request soap envelope. Copy the soap message and save in xml (Say it is request.xml) format. You may have a look at How to monitor messages sent/received through WSAS in order to see how WSAS Soap Tracer is used.
Also , make sure to switch off the soap tracer after capturing the message since it degrades the server performance significantly.
<?xml version='1.0' encoding='UTF-8'?>
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
<s:Body>
<p:operation1 xmlns:p="http://ws.wso2.org/dataservice">
<empid>1</empid>
</p:operation1>
</s:Body>
</soapenv:Envelope>
request SOAP envelope
Step 3
Now run the ApacheBench with the following parameters. Change the service url and request file location according to your system.
C:\Program Files\Apache Software Foundation\Apache2.2\bin>ab -p D:\testing-resources\ds-blog-request.xml -n 100 -c 20 -k -H "SOAPAction: urn:operation1" -T "text/xml; charset=UTF-8" "http://localhost:9762/services/blogds"
Lets look at the usage of the the above parameters. -p specifies the location of the request message. -n is used to specify the number of requests transmitted to the service. -c is the number of concurrent requests is being sent to the server. -k is used to switch on http keep alive settings. -H specifies the attribute headers. In our case it is Soap action of the opeartion.
-T denotes content-type header of the HTTP POST message. Finally, the service endpoint is specified.
When Apache Bench completes the transmission of the total number of requests, a comprehensive test results report will be displayed .
Hence, I'd recommend the steps given there should be exercised before going through this post.
I'm going to explain how I analyzed the performance of WSAS data service using ApacheBench, which is an embedded benchmarking utility comes with Apache httpd server. The procedure is very simple and can be enhanced with more scenarios as you wish.
Step 1
Assuming the Apache web server is installed in your system, go to Apache_home/bin and run AB.exe
You will see a set of options in the command prompt.
Step 2
Invoke the data service which has been created in the previous post using Tryit utility. Make sure to switch on the 'Soap Tracer' so that we can capture the request soap envelope. Copy the soap message and save in xml (Say it is request.xml) format. You may have a look at How to monitor messages sent/received through WSAS in order to see how WSAS Soap Tracer is used.
Also , make sure to switch off the soap tracer after capturing the message since it degrades the server performance significantly.
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
<s:Body>
<p:operation1 xmlns:p="http://ws.wso2.org/dataservice">
<empid>1</empid>
</p:operation1>
</s:Body>
</soapenv:Envelope>
request SOAP envelope
Step 3
Now run the ApacheBench with the following parameters. Change the service url and request file location according to your system.
C:\Program Files\Apache Software Foundation\Apache2.2\bin>ab -p D:\testing-resources\ds-blog-request.xml -n 100 -c 20 -k -H "SOAPAction: urn:operation1" -T "text/xml; charset=UTF-8" "http://localhost:9762/services/blogds"
Lets look at the usage of the the above parameters. -p specifies the location of the request message. -n is used to specify the number of requests transmitted to the service. -c is the number of concurrent requests is being sent to the server. -k is used to switch on http keep alive settings. -H specifies the attribute headers. In our case it is Soap action of the opeartion.
-T denotes content-type header of the HTTP POST message. Finally, the service endpoint is specified.
When Apache Bench completes the transmission of the total number of requests, a comprehensive test results report will be displayed .
Friday, January 4, 2008
Call MySQL stored procedures using WSAS data services
WSO2 Web services application server (WSAS) is shipped with a lot of useful features which can be used effectively in a production system. One of these cool functionalities is exposing relational data using WSAS data services.
Lets see how we can call an existing stored procedure using an WSAS data service. Here I'm going to use a MySQL DB instance for the demonstration purposes. You may use any DBMS as your preference.
Step 1
Create a data base
mysql>create database employeedb;
Step 2
Create a table and inseert values.
mysql>create table employee (id INTEGER, name varchar(20), address varchar(30));
mysql>insert into employee values (1, 'Charitha', 'Colombo');
Repeat the insert statement with more data.
Step 3
Create a stored procedure.
CREATE PROCEDURE getEmployee(empid INTEGER) select id, name, address from employee where id=empid;
Step 4
We haven't used WSO2 WSAS data service configuration yet. All of the above are basic MySQL database setting up steps. Now, we can create a data service configuration using WSAS management console. Lets start our WSAS instance by running WSAS_HOME/bin/wso2wsas.bat {sh}
Then log in to the management console using default administrator credentials (admin/admin). Next, select 'Services' from the left navigation and click on 'Define data service'
Step 5
Now you are in the first step of the data service configuration wizard. Provide a unique name for your data service. Then, select 'RDBMS' as the data source. You will get a pop-up window as given in the following screen.
Step 6
In the above pop-up window, select 'MySQL' as the database type. Then enter the following values for the remaining fields.
Driver class = com.mysql.jdbc.Driver
JDBC URL = jdbc:mysql://localhost:3306/employeedb
user = root
password = root
Next, click on 'Test Connection' to verify the communication between WSAS and our database.
If the connection is successful, click 'OK' in the pop-up window.
Step 7
Now we are done with the first step of the data service configuration. Click 'Next' to proceed to the step 2.
Click on 'New Query' button to add a query to our data service. You will get 'Add new Query' pop-up window. Fill the values as given in the screen shot.
Note that, the SQL statement field contains the stored procedure call (call getEmployee(?)).
Click 'OK' in the pop-up window. Then select 'Next' to continue through the wizard. You will be directed to the step 3 of the data service configuration.
Step 8
Click on 'Add new operation' button. 'Add new operation' pop-up window will be displayed. Enter a unique name for the operation. Select 'query1' from the 'Query' drop down menu.
Click 'Finish' to save our data service.
If everything is successful, you will see the deployed data service in the 'Services and service group management' page. Now, we need to verify whether the stored procedure call generate correct results. In order to do that, we may use 'Tryit', one of the very helpful tools in WSAS which can be used to verify simple service invocations.
Step 9
Select the newly created data service from the 'Services and service group management' page.
Select 'Tryit'.
Click on the operation button after entering empid.
As you can see above, our MySQL stored procedure has been invoked by data service. We can enhance the data service configuration to handle more complex stored procedures.
Lets see how we can call an existing stored procedure using an WSAS data service. Here I'm going to use a MySQL DB instance for the demonstration purposes. You may use any DBMS as your preference.
Step 1
Create a data base
mysql>create database employeedb;
Step 2
Create a table and inseert values.
mysql>create table employee (id INTEGER, name varchar(20), address varchar(30));
mysql>insert into employee values (1, 'Charitha', 'Colombo');
Repeat the insert statement with more data.
Step 3
Create a stored procedure.
CREATE PROCEDURE getEmployee(empid INTEGER) select id, name, address from employee where id=empid;
Step 4
We haven't used WSO2 WSAS data service configuration yet. All of the above are basic MySQL database setting up steps. Now, we can create a data service configuration using WSAS management console. Lets start our WSAS instance by running WSAS_HOME/bin/wso2wsas.bat {sh}
Then log in to the management console using default administrator credentials (admin/admin). Next, select 'Services' from the left navigation and click on 'Define data service'
Step 5
Now you are in the first step of the data service configuration wizard. Provide a unique name for your data service. Then, select 'RDBMS' as the data source. You will get a pop-up window as given in the following screen.
Step 6
In the above pop-up window, select 'MySQL' as the database type. Then enter the following values for the remaining fields.
Driver class = com.mysql.jdbc.Driver
JDBC URL = jdbc:mysql://localhost:3306/employeedb
user = root
password = root
Next, click on 'Test Connection' to verify the communication between WSAS and our database.
If the connection is successful, click 'OK' in the pop-up window.
Step 7
Now we are done with the first step of the data service configuration. Click 'Next' to proceed to the step 2.
Click on 'New Query' button to add a query to our data service. You will get 'Add new Query' pop-up window. Fill the values as given in the screen shot.
Note that, the SQL statement field contains the stored procedure call (call getEmployee(?)).
Click 'OK' in the pop-up window. Then select 'Next' to continue through the wizard. You will be directed to the step 3 of the data service configuration.
Step 8
Click on 'Add new operation' button. 'Add new operation' pop-up window will be displayed. Enter a unique name for the operation. Select 'query1' from the 'Query' drop down menu.
Click 'Finish' to save our data service.
If everything is successful, you will see the deployed data service in the 'Services and service group management' page. Now, we need to verify whether the stored procedure call generate correct results. In order to do that, we may use 'Tryit', one of the very helpful tools in WSAS which can be used to verify simple service invocations.
Step 9
Select the newly created data service from the 'Services and service group management' page.
Select 'Tryit'.
Click on the operation button after entering empid.
As you can see above, our MySQL stored procedure has been invoked by data service. We can enhance the data service configuration to handle more complex stored procedures.
Thursday, December 13, 2007
Top replies by programmers to testers when their program doesn't work
Found a set of funny replies from Quality Assurance and Testing blog
1. "It works on my machine"
2. "Why do you want to do it that way?"
3. "You can't use that version in your system"
4. "Even though it doesn't work, how does it feel?"
5. "Somebody must have changed my code"
6. "You must have the wrong version"
7. "I haven't touched that module in weeks!!"
8. "There is something funky in your data"
9. "What did you type in wrong to get it to crash?"
10."It worked yesterday"
11."That is not a bug, that was a requirement"
1. "It works on my machine"
2. "Why do you want to do it that way?"
3. "You can't use that version in your system"
4. "Even though it doesn't work, how does it feel?"
5. "Somebody must have changed my code"
6. "You must have the wrong version"
7. "I haven't touched that module in weeks!!"
8. "There is something funky in your data"
9. "What did you type in wrong to get it to crash?"
10."It worked yesterday"
11."That is not a bug, that was a requirement"
Wednesday, December 12, 2007
Deploying an axis2 service in application scope
If you deploy a service without specifying a scope, it will be considered as sessionless and will be deployed in request session scope. In other words, request scope is the default session scope of an axis2 web service.
When a service is deployed in request session scope, an instance of the service implementation class is created for every invocation. Suppose you want to get rid of that and maintain a single instance of service class throughout the server life time.
Then you should deploy the service in application session scope. Lets see how it can be achieved and tested the service invocation using WSO2 WSAS.
First, create a simple java bean as follows.
public class Employee {
private int Age;
private String name;
private String emp_id;
public int getAge() {
return Age;
}
public void setAge(int age) {
Age = age;
}
public String getName() {
return name;
}
public void setName(String name) {
this.name = name;
}
public String getEmp_id() {
return emp_id;
}
public void setEmp_id(String emp_id) {
this.emp_id = emp_id;
}
}
Now create the service descriptor (services.xml) for the above class.
<service name="EmployeeService" scope="application">
<messagereceivers>
<messagereceiver mep="http://www.w3.org/2006/01/wsdl/in-only" class="org.apache.axis2.rpc.receivers.RPCInOnlyMessageReceiver">
<messagereceiver mep="http://www.w3.org/2006/01/wsdl/in-out" class="org.apache.axis2.rpc.receivers.RPCMessageReceiver">
</messagereceiver>
<parameter name="ServiceClass" locked="false">
org.wso2test.Employee
</parameter>
</messagereceiver>
Here, we define the session scope of the service as 'application', which will create a single instance when server is started.
Now, include these in a deployable service archive. (Create a META-INF directory and add the services.xml in there and save the archive as an aar)
Next, deploy the service archive using WSO2 WSAS management console. In the WSAS management console, go to "Services and Service Group Management' page and select 'EmployeeService'. Then click on 'Tryit' to test the service invocation.
The all getter and setter methods of the service class are displayed there. Click on setName and enter a string value in the given text box. You will get a NuLL response since it is an request only invocation. Next, select getName and click on getName button. You will get the name you have set in the previous step.
The above scenario demonstrates the creation of a single service class instance using application scope.
When a service is deployed in request session scope, an instance of the service implementation class is created for every invocation. Suppose you want to get rid of that and maintain a single instance of service class throughout the server life time.
Then you should deploy the service in application session scope. Lets see how it can be achieved and tested the service invocation using WSO2 WSAS.
First, create a simple java bean as follows.
public class Employee {
private int Age;
private String name;
private String emp_id;
public int getAge() {
return Age;
}
public void setAge(int age) {
Age = age;
}
public String getName() {
return name;
}
public void setName(String name) {
this.name = name;
}
public String getEmp_id() {
return emp_id;
}
public void setEmp_id(String emp_id) {
this.emp_id = emp_id;
}
}
Now create the service descriptor (services.xml) for the above class.
<service name="EmployeeService" scope="application">
<messagereceivers>
<messagereceiver mep="http://www.w3.org/2006/01/wsdl/in-only" class="org.apache.axis2.rpc.receivers.RPCInOnlyMessageReceiver">
<messagereceiver mep="http://www.w3.org/2006/01/wsdl/in-out" class="org.apache.axis2.rpc.receivers.RPCMessageReceiver">
</messagereceiver>
<parameter name="ServiceClass" locked="false">
org.wso2test.Employee
</parameter>
</messagereceiver>
Here, we define the session scope of the service as 'application', which will create a single instance when server is started.
Now, include these in a deployable service archive. (Create a META-INF directory and add the services.xml in there and save the archive as an aar)
Next, deploy the service archive using WSO2 WSAS management console. In the WSAS management console, go to "Services and Service Group Management' page and select 'EmployeeService'. Then click on 'Tryit' to test the service invocation.
The all getter and setter methods of the service class are displayed there. Click on setName and enter a string value in the given text box. You will get a NuLL response since it is an request only invocation. Next, select getName and click on getName button. You will get the name you have set in the previous step.
The above scenario demonstrates the creation of a single service class instance using application scope.
Saturday, November 17, 2007
Balanced QA team
Balanced team.. may be a traditional management term. However, I am trying to understand the term in quality assurance perspective. In my experience most of the QA teams do not consist of the right set of skill combinations required to get the maximum out of available resources. First I believe the following members should be included in a balanced QA team.
My personal belief is, having a great QA process will do nothing if the right set of skills are not available in your team. A particular QA team may have best automation engineers, but that will not produce good results if their knowledge on the domain or subject matter of the AUT (Application Under Test) is minimum. Saying that, we cannot expect the same automation gurus are trained in application domain will resolve the issue.
Therefore, we should make sure to employ the experts in different QA technologies. During the test planning and design, domain expert could identify the potential issues in the application according to his/her vast knowledge in the subject and include those problems in the QA test plan.
Automation expert will analyze the architecture of the product and plan for UI or some other system level test automation.
A team member with exceptional deployment skills can provide the relevant inputs to identify the installation/deployment specific problems as early as possible. Also, the same person could prepare the QA environment while the other members of the QA team is busy with the test plaiing and designing tasks.
The DB expert is expected to play an important character in a balanced QA team. He should be able to derive test scenarios and test cases by analyzing the data base design of the AUT.
A Security testing expert will verify how secure the AUT by generating test scenarios with his/her knowledge in system/web security.
Finally a person with vast experience and knowledge in various performance tools and benchmarks will facilitate performance/load and stress testing of the AUT in effective and efficient manner.
By discussing the above I should say that the different experts should not work individually in isolate manner. I don't say that the 'TEAM WORK' concept is no longer applicable in a balanced QA team. Each member should communicate frequently by giving their opinions to have a solid QA plan/test cases which will eventually become the one and only one reference in test execution. Everybody must follow the same guidelines paying attention to the assigned area. Each member should take care of the respective technology they are having the expertise knowledge with.
In other words, application domain expert must be responsible for the functional aspects of the product. He or she should derive functional test scenarios by getting the inputs from the other technical experts in the team and execute them in functional test cycles.
After all, I hope you should raise questions like, 'How do we form such a team with so called experts?' 'Are these type of resources available in the SQA world?'
Yes, these questions are absolutely valid! Most QA teams do not even have a subject matter expert rather having various tech gurus. How do we find them? I strictly believe we should create the experts from our own existing teams. We should identify what are they capable of , which testing methodologies they re handling in ease in their daily works and how well the particular skill can be improved. We shouldn't overload each QA team member with all QA methods and technologies. We should identify individuals and provide necessary trainings to improve their knowledge in ONE particular area. So that the non-hypothetical balanced QA team can be realized.
- Domain/subject matter expert
- Test automation expert
- Deployment/build expert
- Security testing expert
- Performance testing expert
- Database technology expert
My personal belief is, having a great QA process will do nothing if the right set of skills are not available in your team. A particular QA team may have best automation engineers, but that will not produce good results if their knowledge on the domain or subject matter of the AUT (Application Under Test) is minimum. Saying that, we cannot expect the same automation gurus are trained in application domain will resolve the issue.
Therefore, we should make sure to employ the experts in different QA technologies. During the test planning and design, domain expert could identify the potential issues in the application according to his/her vast knowledge in the subject and include those problems in the QA test plan.
Automation expert will analyze the architecture of the product and plan for UI or some other system level test automation.
A team member with exceptional deployment skills can provide the relevant inputs to identify the installation/deployment specific problems as early as possible. Also, the same person could prepare the QA environment while the other members of the QA team is busy with the test plaiing and designing tasks.
The DB expert is expected to play an important character in a balanced QA team. He should be able to derive test scenarios and test cases by analyzing the data base design of the AUT.
A Security testing expert will verify how secure the AUT by generating test scenarios with his/her knowledge in system/web security.
Finally a person with vast experience and knowledge in various performance tools and benchmarks will facilitate performance/load and stress testing of the AUT in effective and efficient manner.
By discussing the above I should say that the different experts should not work individually in isolate manner. I don't say that the 'TEAM WORK' concept is no longer applicable in a balanced QA team. Each member should communicate frequently by giving their opinions to have a solid QA plan/test cases which will eventually become the one and only one reference in test execution. Everybody must follow the same guidelines paying attention to the assigned area. Each member should take care of the respective technology they are having the expertise knowledge with.
In other words, application domain expert must be responsible for the functional aspects of the product. He or she should derive functional test scenarios by getting the inputs from the other technical experts in the team and execute them in functional test cycles.
After all, I hope you should raise questions like, 'How do we form such a team with so called experts?' 'Are these type of resources available in the SQA world?'
Yes, these questions are absolutely valid! Most QA teams do not even have a subject matter expert rather having various tech gurus. How do we find them? I strictly believe we should create the experts from our own existing teams. We should identify what are they capable of , which testing methodologies they re handling in ease in their daily works and how well the particular skill can be improved. We shouldn't overload each QA team member with all QA methods and technologies. We should identify individuals and provide necessary trainings to improve their knowledge in ONE particular area. So that the non-hypothetical balanced QA team can be realized.
Friday, September 14, 2007
Web application testing with Httpunit
Recently I wrote a tutorial on using httpunit to test Axis2 web module, which was published in wso2 oxygen tank. Httpunit provides a set of APIs to test a web application even without opening a browser, so that httpunit tests can be easily integrated in to your nightly build system. Have a look!!!
Subscribe to:
Posts (Atom)