Why is testing taking so long?

When you are involved in software testing, you may have heard the following 3 common questions at the end, beginning or middle of any testing cycle.

1. Why did not you find that bug during testing cycle?

2. Why is testing taking so long?

3. Why did not you automate the damn thing?

I have answers for all three questions but Michael Boloton, the testing genius in our era, explained the answer for question 2 with some great examples. I would recommend this to be a must read for anyone work in software engineering.

[1]http://www.developsense.com/blog/2009/11/why-is-testing-taking-so-long-part-1/

[2]http://www.developsense.com/blog/2009/11/what-does-testing-take-so-long-part-2/

Study the facts given by Michael. You will realize the truth of test estimation and traditional way of managing test processes.

Test Reviews - An efficient way to validate test coverage

When the complexity of testing increases, deriving test scenarios becomes extremely hard. Individual testers are not always capable of thinking all aspects of the products and finding out test cases. In agile processes which demands frequent releases, this becomes much harder since individual testers do not get sufficient time to learn and explore products.

At WSO2, code reviews have been an integral part of engineering process. Code review groups get together and go through different code segments in weekly basis.

Similar to code reviews, WSO2 QA team conducts weekly test reviews. The primary objective of reviewing tests is to finding out gaps in a particular product's testing methodology.

Usually there are 2 main actors take part in a test review meeting.

- Tester(s) who test a particular feature

- Developer (s) who implemented the feature

Tester goes through the existing test scenarios and developer provides his or her feedback on the tests. Specially when a particular use case is not clear enough, tester can clarify it during reviews very effective manner.

It is the testers responsibility to try out the new test scenarios which have been captured during the review in next release cycles.

Test review will not take more than 1 hour and testers should be prepared properly to get the maximum out of this short time and capture more and more test scenarios.

Similar to the manual test scenarios, all automated tests can also be reviewed during test reviews.

In agile processes where you do not have detailed system specs to derive tests, test reviews are one of the most useful mechanisms to validate your test coverage.

Are you doing software testing or quality assurance?

While going through some of the software QA related stuff, I found a very nice presentation done by Michael Bolton. There, he challenged the traditional way of defining software quality assurance teams. While agreeing with his insight on QA, I thought to highlight the contents included in one slide of his presentation.

If you are involved in software quality assurance (or if you think that you are a software QA person) then the following will be able to help you to get yourself categorized into the correct role that you are playing in your organization.

Do you

• design the product?
• hire programmers?
• decide which bugs to fix?
• allocate staff?
• set the schedule?
• fix problems in code?
• decide on raises?
• allocate training budgets?
• produce manuals?
• choose the development model?
• fire some programmers?
• control the budget?
• set the company's strategic direction?

If the answer for all of the above questions is , YES, then you assure the quality of the products produced by your organization. So you are doing software QA!

I have never found a QA/testing team who performs all of the above activities in their job. But most of us test the products. We explore the products in order to reveal the issues and provide feedback to the stakeholders of the product under development. Therefore our role is Software testing NOT quality assurance.

James Bach defines software testing as questioning the product in order to evaluate it.

Cem Kaner says software testing is a technical, empirical investigation of a product, done on behalf of stakeholders, with the intention of revealing quality-related information of the kind that they seek.

So, there is no assurance!

Are you testing or checking?

Just found a nice post from Micheal Bolton who expressed his opinions about the difference between Testing and Checking.

Extracted from the post;

Checking Is Confirmation

Checking is something that we do with the motivation of confirming existing beliefs. Checking is a process ofconfirmation, verification, and validation. When we already believe something to be true, we verify our belief bychecking. We check when we’ve made a change to the code and we want to make sure that everything that worked before still works. When we have an assumption that’s important, we check to make sure the assumption holds. Excellent programmers do a lot of checking as they write and modify their code, creating automated routines that they run frequently to check to make sure that the code hasn’t broken. Checking is focused on making sure that the program doesn’t fail.

Testing Is Exploration and Learning

Testing is something that we do with the motivation of finding new information. Testing is a process of exploration, discovery, investigation, and learning. When we configure, operate, and observe a product with the intention of evaluating it, or with the intention of recognizing a problem that we hadn’t anticipated, we’re testing. We’re testing when we’re trying to find out about the extents and limitations of the product and its design, and when we’re largely driven by questions that haven’t been answered or even asked before. As James Bach and I say in our Rapid Software Testing classes, testing is focused on “learning sufficiently everything that matters about how the program works and about how it might not work.”

QA testing in agile world - Attitude matters most!

Few short tips about a viable approach for testing in agile world. May be valid for most organizations.

• Understand your development team first. Study each engineer and adjust your way of acting (communication, management etc..) according to the team you are working on

• Do not act as a quality police. Everyone is responsible for quality. It is your task to guide the others NOT finding faults and complain
  

• Be as flexible as you can in all situations

• Be quick on everything - learn, design, implement, configure, deploy quickly and test fast
  

• Be innovative

• Identify highly risked features first and start hunting bugs

Finally;

• CREATE A VALUE as a tester for your organization and BUILD the CONFIDENCE among the team!
  

Creating value with testing

Jonathan Kohl discusses about creating value with software testing. You will find very important set of points which we must follow in daily QA/testing activities.

Is my testing work defensible? (Cem Kaner talks a lot about this.) Think of a court case. What would a jury think if you testified and described what you did as a tester and why. How did you determine priority? Why did you test some things and not test others? (100% complete testing is impossible, so you have to make decisions to optimize your work. Are those decisions well thought out, or more subconscious? What sorts of things might you be missing that you haven't thought of?)

Read the full article from here

Process vs Tools and Technologies - What should Sri lankan QA community be concerned with?

I have been thinking about discussing the matters related to local QA community in Sri lanka but never got a chance. Recently I was able to meet a lot of folks who are engaged in software quality assurance in various sri lankan organizations in one place at a quality summit. By listening to the presentations and talking with people, I came out with a few basic questions.

Some of them were;

what the biggest concern of software quality assurance in our country? are those processes or tools/technologies? Do we have the necessary trainings or knowledge sharing mechanism to overcome the issues which we face during daily QA tasks?

In my view, the biggest concern of QA in our country is, not having people with enough technical skills. By interviewing a lot of QA folks for the past few years, I personally have a good experience about the way people are approaching QA. Most fresh graduates believe that QA as a first step towards entering in to software industry! Some people joins QA merely to get some understanding about the product/project then move in to business analysis or sales.

Why is this? IMO, it is totally due to the perception of software QA in sri lanka. We, sri lankan QA community, must be responsible for drawing that image on people's minds about QA.

For the past few years, I never noticed any training (were there any?) for educating QA community about the usage of tools in daily QA tasks to be more productive or technical aspects such as performance/automation testing tools. Instead, whenever there is something about QA, it is about CMMI or process frameworks.

I'm not going to say that those are not important. BUT those are not what our teams need at the moment.

People struggle with configuring application server X on operating system Z. QA folks face in to difficulties when automating AJAX based UIs. How QA should be dealt with the frequent UI changes during UI based test automation? How can we be more productive using linux? Do we use any scripting language for automate repetitive configuration tasks? Are we doing continuous integration? Are QA people familiar with build tools such as Maven or Ant? Do we know about exploratory testing? Do we know how to use test coverage tools? Do we report bugs with the adequate logs and find the root cause of them?

I think these are the questions that most of the QA teams have. We should try to be more productive and be experts as a community. We should try to change the perceptive about QA by empowering everyone with the right set of skills.

If people are comfortable with the tools and technology they are handling in daily work life, educating them about processes and process improvements is not a big thing!

Importance of logging in QA testing

Recently James Bach posted a nice blog entry in which he identified logging as a good friend of an exploratory test team. I believe the comprehensive logging of AUT (application under test) is an extremely valuable asset to any tester regardless of the context of exploratory testing.
A detailed log will surely help to minimize the effort required to reproduction of issues. There are intermittent failures, random application crashes which are extermely hard to regenerate in successive attempts. If the AUT is written in a way in which all the major user events are logged, then the root causes of such random failures can be traced easily.
A comprehensive log is a useful test report. After a test run, you can store the log somewhere safely and use it as the test report. Specially, in agile test processes, test teams do not find sufficient time to record test results and maintain detailed test reports due to the short release cycles. If the AUT provides good set of logs, it will help testers to use them instead of maintaining separate test logs.

IMHO, comprehensive logging is a must have feature of any of the enterprise-scale application. At WSO2, this has been identified as a extremely important tool and the necessary modifications are in-progress to improve logging in WSO2 Carbon SOA middleware suite.

Key factors for successful test automation

At WSO2, we have been making considerable progress in building a complete test automation solution. According to my experience on that effort, I think the followings are the key factors contributed to a successful test automation project.

• Dedication to automation (Instead of treating it as a spare-time activity)

• Commitment by the entire team (rather than just one or two testers),

• Commitment to automation from the start (rather than trying to automate a manual process later)

• Making use of correct tools/frameworks/technology

• Allocating sufficient time and resources
  

Complexity of middleware testing

I have been testing SOA middleware applications for nearly 3 years. I also tested traditional CRM, Telco billing applications for more than 4 years. Quality assurance of middle ware is completely different from the traditional business application testing. It requires a lot of effort. I have observed following facts/concerns which related to middleware testing.

1. Target audience/end users of middleware apps
Most middleware products are developed for the use of highly technical tasks and they are used by programmers. When it comes to SOA middleware, product end-users are technically savy engineers. Therefore when testing them, QA/test teams should understand the purpose of it, target audience and act accordingly.

2. Complexity of applications
The developers understand the use cases of middleware applications when implementing them. Obviously without knowing the requirements, they can't be implemented. Therefore the developers of middleware applications gets the knowledge of the application under development while it is being developed regardless of the complexity of it.
This is not applicable for QA/testers. In agile like processes, due to the release crunch, QA gets no time to learn the use of middleware. They do not even get time to derive test cases/scenarios. So, QA teams are forced to test applications with these limitations.

3. Test Automation
Majority of the middle ware application tests can be automated. However, it cannot be done by overnight. Without having a better understanding of the applications, no body can automate them. Automation is highly effective for middleware testing.
Having said that, I must emphasize that, automation is not the solution for everything. Specially in middleware testing, human intervention is critical. If you are a middleware tester, you must live with the application and its surrounding technologies. You must learn the use cases of middle ware app and apply them to derive more and more test scenarios. In my career, 80% of highly severe bugs were uncovered by exploratory testing.

It takes times to learn the technologies behind middleware apps. It takes time to understand the use of them. A content management system or a telco billing application can be learned and understand with considerably less time and effort. However, middleware applications are different and deriving use cases are not simple.

If you have any doubts about the complexity of middleware testing, the best way to clarify them is to involve in middleware testing for few days. Work with middleware test teams. Then you can understand the real situation and issues behind it.

Importance of defect isolation

A problem well stated is half solved

True! A good bug report should provide with the information to reproduce the issue. It is QA tester's responsibility to specify the exact steps of bug recreation. Isolating the environment and conditions at which the issue occurred are extremely important components of a defect report.

Sadly I have seen some people ignore these important practices due to tight schedules and rush testing processes. No! In any case, even with management push towards quicker feedback of the products under test, ignoring these vital practices are totally unacceptable. Your bug report mirrors the professionalism of software QA testing job. Developers do not waste time or jump in to assumptions by reading your bug report. It provides all the necessary details. It consists of logs, screen shots when they are required.

A good QA tester must isolate the issue first. He never reports bugs just by seeing an exception in server console. He attempts multiple instances to recreate and isolate the issue. Some failures can be repeated easily but some requires more effort. Then prepare a detailed defect report.

As Cem Karner said, "there are no intermittent software errors. The problem may appear rarely but each time the exact conditions occur, the behavior will repeat".

More to follow....

QA role in Hackathons and Risk based testing

A few days back, we had a hackathon at WSO2 in order to resolve a large number of open bugs and boost the schedule of Data Services 1.0 release. A set of developers and QA team sat together and went through the open issues and eventually got more than 100 defects fixed and verified.
If you are familiar with hackathons or if you are a tester, you may certainly raise the following questions.
- What is the role of QA in a Hackathon
- How to deal with the regression issues and frequent builds

It certainly is a challenge to involve in testing and quality assurance during a hackathon. There is a high risk of breaking the features in between builds. However if you manage it properly, Hackathons are good opportunities for QA and testing teams to get more familiar with the AUT (Application under test) hence uncovering more and more bugs, learn and share the knowledge of practical use cases of products etc..

Lets try to find out answers for the above questions from QA user point of view.

What is the role of QA in a hackathon?

As defined in the wikipedia, Hackathon is an event when programmers meet to do collaborative software development. Testers, BAs, PMs and other astake holders may also take part in it.

The primary role of QA/test teams in a hackaton is reproduce the issues which were reported by them. This involves functional issues, cross browser issues as well as cross platform related bugs. The reporter of the bug knows the way to recreate it than any other.
This will greatly reduce the time to reproduce bugs and help developers to get them fixed effectively.
In most occasions, the AUT is started in debug mode in testers machines and someone familiar with the code, debug the source remotely to get the issues fixed quickly.

Whenever a bug is marked as resolved, testers get an update from source repository (in our case SVN), build the source and verify and close or reopen the bug.

Hackathons are ideal places to discuss the usage scenarios/patterns of the product being developed. QA/testers can identify more and more scenarios and apply them in testing. This help to uncover a lot of hidden bugs.

It is an essential for QA/testers to have and set the source code up in their IDEs. If you are familiar with the code and know the root cause of the issue, you are free to go ahead and fix the bug by your self. We did it and worked well :)
One of the most important requirement to have source with QA during a hackathon is, QA can go through a module with some developer and try to find out the places where bugs can be introduced. This glass box testing approach is one of the most important feature in a hackathon.

During a hackathon, QA find bugs and report it immediately to the relevant developer and get it fixed. QA should make sure to track all issues using the bug tracking system though some of the issues are reported verbally.

Dealing with regression issues and frequent builds

While fixing more and more bugs it is essential to ensure that the regression issue are minimum. Regression testing is the most difficult task for QA in a hackathon. Whenever a set of bugs are fixed, you get an update from source repository and build in your machine. Then you verify the bug and close it.
When you do this, there is a high chance to missing more and more regression issues. So, as a QA tester, how do you deal with it?

Before starting a hackathon, you should have a proper idea about the most critical features of the AUT. You should identify the areas which can cause most damages to the recognition of the product as well as for your customers. Then you should keep in mind that these critical features are not regressed in any new build use for testing. In other wards, though you test 4-5 new builds per day, you must ensure that the highly risked features are verified and tested properly.
Risk based testing is the most suitable approach to ensure the critical features are not broken during a hackathon.

It is important to get a few labeled builds for testing during a hackathon. We could not do this in Data service 1.0 hackathon though. We cannot guarantee the quality of the builds taken from users local repositories. Therefore, to make sure the AUT is in a proper state, it will be essential to have a labeled QA release built from a pure build environment and release for testing.

After all, Data Service 1.0 hackathon went really well and helped to deliver the product on time with acceptable quality!

Why is quick learning ability extremely important for QA?

Quick Understanding of software product requirements and technologies around them are important qualities expected from any engineer. However, I believe
it is THE most important and must-have skill for a QA engineer. Why is that?

In theory, a QA cycle is supposed to start with test analysis and design. Requirements specifications/design specs are the ultimate references to derive test scenarios and
test cases in test analysis/design phase. But those reference materials are not always available in early stages of the development and in some projects you don't even see such docs.
Even if you have the documented requirements and design docs, you don't get the feel of actual business flow and scenarios until you play around with the
working instance of the product. There were thousands of test cases written by referring to functional specs, however the real business
scenarios may have missed due to the lack of understanding of the domain and nature of the product.

When the initial builds are released for QA testing, you will get a better knowledge of the product and its domain while you play around the features of product.
You will start to realize the functional flows documented in test scenarios (which were derived using requirement specs) and sudden requirements to aware of new technologies will arise.
However you don't get much time to learn every bits of business domain and related technologies with the frequent test build releases.

This is where your quick learning ability comes in to play. If QA engineer possesses;
-grasp things with minimum time
-eager to find out how it works
-Love to read
then I always believe such QA engineer is capable to move forward in a challenging environment.

Remember, QA is expected to act as user's advocates and should know what and how the product functions. Therefore catching the things as quickly as possible
and apply them in testing are key to deliver a successful QA outcome.