Better testing, better software
Testing is the final hurdle before you launch you product and earn money for all the sweat, blood and tears you invested. This results in a lot of pressure to shorten the test period and may result in bad quality products entering the market. This article shows you how to systematically test your software, which will save you money and time in the end.
Testing can be separated in white-box and black-box testing. White-box tests are performed by a programmer to check if the code is correct and conform specifications. The programmer uses his knowledge of the internal software structure and operations to carry out the tests. Black-box testing doesn’t use this knowledge (therefore black-box) and is based on the functional specifications and quality requirements. This article focuses on black-box testing, because this looks at your software from a customer perspective and doesn’t depend on the programming language you use.
Start early
You get the best test results if you don’t wait with testing till the last moment. As soon as you have written down the user requirements and the tasks and goals a customer must be able to perform, you can start testing. I once worked on a program that helped managers to choose a strategy for their company. We decided to verify the early requirements with a number of potential users to see if we were on the right track. A simple online survey with about ten questions checked if our requirements fulfilled the needs and problems of our clients. Fortunately they did and we were able to proceed the project with a lot of confidence.
Another early test is to ask a fellow programmer or potential customer to have a look at the requirements themselves. Ask him or her to explain to you what the software will do and if features are missing. Interesting discussions and product improvements will be the result. A test task that you should do early in the project is to write down the test criteria. What criteria must your software fulfill before it is good enough to enter the market? Make them SMART, Specific, Measurable, Agreed upon, Realistic and Time-bound. Vague and ambiguous test criteria are simply impossible to test.
Not all functionality of your program is equally important. Therefore you should assign weights to the different test criteria you have written down. This helps to ensure that you test crucial parts of your program in depth and others quickly (and dirty).
Create solid tests
Test criteria and the software requirements specification are the foundation of black-box testing. However, test scenarios are needed before you can do any solid testing. A test scenario basically describes the actions and checks a tester needs to carry out. It focuses on how a test should be performed.
Simple scenarios can be very effective already. I remember a test that asked users to enter certain location data, a key task of the software. Most users had great difficulty to do this and one user, an IT-veteran with 20+ years of experience failed. If that happens, you know it is time to start reprogramming!
If you have use cases for your software, use these to create your test scenarios. Make sure to incorporate all exceptions and alternative routes. It is possible to create a test scenario directly from the test criteria and use cases. A useful intermediate step is to write down the type of test that is needed. Test methods like decision tables and data cycle tests each have their own strong points and weaknesses *. Choosing the right method for the job makes life easier.
Example
A user needs to enter a new contact person in the customer database. The test scenario to verify if the person you enter doesn’t exist already may look like this:
ID Part Check Expected result OK?
Syn 1 Add person dialog -Enter name John Wolf
-Enter e-mail john@wolf.com John Wolf is added in customer
database
Syn 2 -Enter name John Wolf
-Enter e-mail John@hotmail.com Error message Person already
exists
Syn 3 -Enter name Alice Johnson
-Enter e-mail john@wolf.com Error message Person already
exists
The test scenarios must reflect closely how you want the software to work. The program in the example doesn’t allow a new person with the same name or the same e-mail. If you want to be absolutely certain that the program works correctly you should also test that a new person with the same name and the same e-mail is blocked. Other test scenarios may be needed to verify that the same person can not be entered elsewhere in the program.
Test and retest
Once you have prepared your test criteria and scenarios you have done most of the test work. The testing itself can go rather quickly. A few points of attention:
— Provide the proper test infrastructure: if you state your program runs on a Mac, make sure you have one available to test this.
— Find good testers: people that will actually use your software are the best testers. They bring in their real life experience. Computer illiterates -like mothers- are second-best. They do things you don’t expect and work at a speed that makes it easy to observe their actions.
— Instruct testers: structured testing is not error guessing, seeking randomly for errors. Make sure users stick to the test scenarios your have provided and tell them how they should write down the errors they find.
— Cheer up testers: testing can be a frustrating and time consuming task, especially if you find a lot of errors. Make sure your testers stay motivated.
Tests usually result in a great number of bugs, but also in remarks for improvement and requests for new features. Make sure you write them all down or save them in a bug-tracking system like bugzilla. Next get rid of the bugs, test, test and retest. Your product introduction is in reach.
Success with your test efforts and may it lead to beautiful programs and lots of happy customers!
* Recommended reading: TMAP (Test Management Approach) http://eng.tmap.net/Home/
TMAP is a structured approach to testing and the default methodology used in the Netherlands.
Bart Jutte is managing partner at Concilio a software company specialized in project risk management. www.concilio.nl
published in ASPECTS, a journal aimed at IT-entrepreneurs worldwide. Reprinted with permission
