Improving Software Quality

Since 1968 end users have come to depend more on software, and their expectations for product quality have risen dramatically. Moreover, the pace of development has accelerated in the new millennium, thanks to the Internet, competition and the tools developers use. It is easier to write Java code and port it than it ever was to write C code and port it. The crop of rapid prototyping languages—scripting languages—like Python, Perl and Ruby makes it easy to build Web sites quickly. Databases have become commodities and don’t need to be reinvented each time.

“QA is still a challenge, still generally left to the end, and the staff is treated as second-class citizens,” said Ed Hirgelt, manager of services development for Quest Software. However, because of the speed of development and time-to-market requirements, QA is becoming more visible. Test-driven development moves testing to earlier in the life cycle. Tools like JUnit and Ant make it easier to run tests as part of the nightly build process. The concept of a continuous build is helping produce reliable software.

Hirgelt characterizes a continuous build process as one in which a build is initiated when a developer commits code back to the source repository. The product is built and tests run automatically. Problems are caught sooner rather than later.

QA also has been changing as the result of such factors as the wind-down following Y2K and the subsequent business decline. As software companies faced hard times, one solution was to improve efficiency by hiring the most skilled testers available and automating as much testing as possible, according to Elfriede Dustin, internal SQA consultant for global security services at Symantec.

The loss of jobs following the dot-com implosion meant software companies went from having to hire practically anyone with a pulse in the late 1990s to the luxury of choosing from only the most highly qualified candidates. That change has affected who is being hired for QA positions. In some large companies, “coding skills are what you are judged and hired by, with testing skills coming in a distant second,” said Duri Price of Exceed Training, who has worked in the software QA field since 1992. Jeff Feldstein, who manages a team of 35 test engineers at Cisco Systems, concurred. He hires software engineers exclusively, and then sells them on test engineering.

“Testers need to be involved from the beginning of the development life cycle,” said Symantec’s Dustin. More important, however, is that so much depends on the developers’ skills. The most efficient and knowledgeable testers cannot succeed if developers implement bad software or ineffective software development life cycles and processes are in place. If testing is the only quality phase implemented as part of the QA process, it can at most be considered a Band-Aid often too late in the development life cycle to make much of a quality difference. Testing is only one piece of the quality puzzle.
Agile Methods

The growing influence of agile processes has had direct and indirect consequences on quality. With the advent of Extreme Programming (XP) and the agile movement, testing has become more of a developer activity, said Dustin. Agile methodologies have provided a model for moving testing forward and putting more of the responsibility in the hands of developers.

“With the onset of agile methodologies comes the concept of test-driven development, which was introduced with Extreme Programming,” said Bob Galen, a senior QA manager at Thomson-Dialog. The principle is to design tests before designing code, usually using a unit-testing framework such as JUnit or xUnit to help support the practice. Test-driven development has gone beyond XP to become a mainstream development practice, he noted. “It has also sensitized a new generation of software developers on the importance of and skills required to properly test software.”

Testers are getting better code and can’t simply repeat the same tests as the development team. Galen said they must find other value areas, such as getting involved at the front end with requirement definition and acceptance test development, working in parallel with the development teams, and at the back end working with the customer to run the acceptance testing and doing performance and load testing or usability testing.


Not everyone is convinced that agile processes are the answer. Dustin said she doubts that Extreme Programming can work in a development effort larger than 10 or so developers. Feldstein’s group hasn’t embraced agile methodologies (though he acknowledged that they have been used successfully elsewhere in Cisco) because he doesn’t see them as a way to get high-quality software, but as a way to get software fast. “It’s not clear that agile puts quality at the forefront,” he said. “Getting it out quickly isn’t a priority for us. Getting it right is.”

At the Cisco facility where Feldstein works, testers become involved during the development of the product requirements document. “Marketing, development and test are all equal in the team, and they all get involved early on,” he explained. The whole team owns the quality, and the whole team decides when to ship. The process is requirements-driven, he said, and they don’t need test-driven development. He also noted that the processes are constantly being refined.

“Once developers have committed to a schedule, which occurs when the functionality spec is complete,” he said, “they tell you what the API looks like. We can start coding simultaneously. We do unit testing on the test software.” When a stand-alone component is complete, it’s handed off for functional and performance testing. A stand-alone component is complete after developers have done unit testing, and integration testing between components, and have established baseline performance.
Automation

Another agile community influence has been to drive testers toward automation, according to Thomson-Dialog’s Galen. “The notion of complete automated unit-testing capabilities is carrying over into an expectation of general automated testing. Why have automated unit testing and the increased flexibility and safety of executing them, when you have to manually test the application from a QA point of view? It simply doesn’t make sense.” He said that there is pressure for testers to create high degrees of automation leveraging agile development practices.

The tools are evolving from capture/playback methods toward alternative methods of driving tests that are longer lived and require less maintenance, said Galen. Concepts like “keyword driven,” “model driven,” “coverage driven” and “database driven” are coming into play.

The advantage of manual testing is that it’s not done the same way every time. In classic automation, you execute the same set of test cases every time. Model-based testing, which is a recent development, is a form of automated testing that adds randomness by inserting random behavior into test automation software. You run through tests in different order so you exercise different areas of code.

Automation is not altogether a panacea. Many companies are spending lots of time and effort on automation that doesn’t return the investment, said Exceed Training’s Price. “I’ve seen huge batches of automated tests that merely informed us that what we thought should work, did work. They verified. But they didn’t usually do a great job at testing.”

Price said that a competent tester with or without coding skills could often break the system manually. Coding skills aren’t the only skills needed, or even the most important ones, he insisted. The most important thing a tester needs to know how to do is figure out what to test, why to test it and how to test it. The implementation of the test will vary, but first you have to figure out your target. That skill set is getting less and less attention.

Exploratory Testing

Test automation, by executing a large number of planned tests, frees up resources to do more free-form exploratory testing, according to Price. Exploratory, or context-based testing, is a movement spearheaded by James Bach and articulated in the book he co-authored with Cem Kaner and Bret Pettichord, “Lessons Learned in Software Testing” (Wiley, 2001). Dion Johnson, an independent consultant who focuses on QA, QC, requirements analysis and process improvement, said that exploratory testing supports less initial test planning and documentation and more execution of spur-of the-moment testing concepts based on a tester’s intuition about what is happening with the application in real time.

Galen characterized it as testing within the context presented for a given effort or project. For example, in a schedule-driven context, management might give a team two days in which to test a release of the product. That’s the context. If the team is operating in a plan and scripted model, it picks which test cases to run that will fit within the two days. The team might try some prioritization, but it stays within the bounds of plan and then test.

Context-based and exploratory testing leverage the background, skills and experience of the testers in order to make better decisions under real conditions, rather than trying vainly to create a never-changing set of tests that anyone can run.

In an exploratory model, the team might first look at the intent of the release. If it is a beta test for a mortgage broker customer, the team might choose to test customer-centric and known problem areas in the allotted two days, using an intimate knowledge of the customer and of the product, to define the focus of testing. The team would spend little time planning but much more time reacting to the specific need.
Security

The Internet explosion has led to a new focus on security, usability and performance testing, said Johnson. Security testing offers a new set of challenges to testers since security flaws don’t necessarily affect how a system works from an application perspective. Functional testing will typically ferret out functional problems, but security vulnerabilities have nothing to do with application functionality.

Roger Thornton, founder and CTO of Fortify Software, said that application expertise doesn’t translate into security expertise. Security expertise comes from the operations side of the company. Even recognizing that a security flaw exists can be difficult. There is a tendency to blame hackers rather than accept the fact that the software is vulnerable and needs to be fixed. Security flaws may be features of the software. “The hacker’s best friend used to be the sys admin. Now it’s the programmer,” he said, citing SQL insertion as a way that theoretically inaccessible information can be retrieved.

Security testing involves testing for conditions that could lead to security breaches, and that means you have to know where to look. “Security bugs are hard to tease out,” said John Viega, co-author with Gary McGraw of “Building Secure Software: How to Avoid Security Problems the Right Way” (Addison-Wesley Professional, 2001). “You need to find vulnerabilities early.” He advocates using a static approach and examining source code to find potential vulnerabilities, such as buffer overflows. But he said that right now static testing suffers from too many false positives.
Performance Testing

Usability design and testing also have gained in importance as an approach for ensuring not only that applications meet customer expectations, but also that customers can navigate through them. And performance testing is important to ensure that the system can support the expected load on the system, given the potentially high traffic on Internet applications. Feldstein runs a performance test bed in parallel with functional testing, and he monitors resources all the time.

When performance testing specialist Scott Barber, chief technology officer at PerfTestPlus, started doing performance testing five years ago, it was seen as a minor add-on service after the completion of functional testing to validate the maximum supported load on the way to production. Today, performance testing is typically not considered an optional add-on service tacked on at the end, although Barber said it is still five years behind functional testing in terms of industry acceptance and maturity. While performance testing is considered early in the process and is thought to be important, it is still generally outside the overall development process and doesn’t start until a beta release.
Open-Source Tools

Not only have processes and methods evolved, but the tools landscape has changed rapidly over the past few years as well, particularly with the availability of open-source tools. “Open source is invading the test space as aggressively as it is within mainstream development,” said Galen, “and we need to be adapting towards the evolution.”

Open-source tools range from development tools to new scripting languages to competitive frameworks to off-the-shelf automation development tools. Besides the xUnit tools, there are tools for acceptance testing, such as FitNesse. Scripting languages such as Python, Jython and Ruby are gaining ground on Perl. “As a tester, it’s not good enough any longer to know a little Perl,” he said. “You must understand a variety of scripting languages to leverage in your day-to-day tasks.”

Testing frameworks that are becoming available in open source rival the traditional automation tool vendor offerings in breadth and capabilities. Barber said that vendors of commercial test tools are “about to be in for a shock, as they find that the new tools are significantly cheaper, and as they learn that the open-source tools are competitive.”
A Glance Forward

Beyond the impact of open-source tools, the test-tool market is on the verge of a major overhaul, according to Barber. Testing tools are being integrated into development environments, a confirmation that the industry is beginning to acknowledge that developers need to be the first to test and that testers need to work hand-in-hand with developers.

Parallel Testing

Usage:


• To ensure that the processing of new application (new version) is consistent with respect to the processing of previous application version.
  

Objective:


• Conducting redundant processing to ensure that the new version or application performs correctly.
• Demonstrating consistency and inconsistency between 2 versions of the application.

How to Use


• Same input data should be run through 2 versions of same application system.
• Parallel testing can be done with whole system or part of system (segment).

When to Use


• When there is uncertainty regarding correctness of processing of new application where the new and old version are similar.
• In financial applications like banking where there are many similar applications the processing can be verified for old and new version through parallel testing

Example


• Operating new and old version of a payroll system to determine that the paychecks from both systems are reconcilable.
• Running old version of application to ensure that the functions of old system are working fine with respect to the problems encountered in the new system.

Manual Testing

Usage:


• It involves testing of all the functions performed by the people while preparing the data and using these data from automated system.
  

Objective:


• Verify manual support documents and procedures are correct.
• Determine Manual support responsibility is correct
• Determine Manual support people are adequately trained.
• Determine Manual support and automated segment are properly interfaced.

How to Use


• Process evaluated in all segments of SDLC.
• Execution of the can be done in conjunction with normal system testing.
• Instead of preparing, execution and entering actual test transactions the clerical and supervisory personnel can use the results of processing from application system.
• To test people it requires testing the interface between the people and application system.

When to Use


• Verification that manual systems function properly should be conducted throughout the SDLC.
• Should not be done at later stages of SDLC.
• Best done at installation stage so that the clerical people do not get used to the actual system just before system goes to production.

Example


• Provide input personnel with the type of information they would normally receive from their customers and then have them transcribe that information and enter it in the computer.
• Users can be provided a series of test conditions and then asked to respond to those conditions. Conducted in this manner, manual support testing is like an examination in which the users are asked to obtain the answer from the procedures and manuals available to them.

Error Handling Testing

Usage:


• It determines the ability of applications system to process the incorrect transactions properly
  
• Errors encompass all unexpected conditions.


• In some system approx. 50% of programming effort will be devoted to handling error condition.

Objective:


• Determine Application system recognizes all expected error conditions
• Determine Accountability of processing errors has been assigned and procedures provide a high probability that errors will be properly corrected
• Determine During correction process reasonable control is maintained over errors.

How to Use


• A group of knowledgeable people is required to anticipate what can go wrong in the application system.
• It is needed that all the application knowledgeable people assemble to integrate their knowledge of user area, auditing and error tracking.
• Then logical test error conditions should be created based on this assimilated information.

When to Use


• Throughout SDLC.
• Impact from errors should be identified and should be corrected to reduce the errors to acceptable level.
• Used to assist in error management process of system development and maintenance.

Example


• Create a set of erroneous transactions and enter them into the application system then find out whether the system is able to identify the problems..
• Using iterative testing enters transactions and trap errors. Correct them. Then enter transactions with errors, which were not present in the system earlier.

Regression Testing

Usage:


• All aspects of system remain functional after testing.
  
• Change in one segment does not change the functionality of other segment.

Objective:


• Determine System documents remain current
• Determine System test data and test conditions remain current
• Determine Previously tested system functions properly without getting effected though changes are made in some other segment of application system.

How to Use


• Test cases, which were used previously for the already tested segment is, re-run to ensure that the results of the segment tested currently and the results of same segment tested earlier are same.
• Test automation is needed to carry out the test transactions (test condition execution) else the process is very time consuming and tedious.
• In this case of testing cost/benefit should be carefully evaluated else the efforts spend on testing would be more and payback would be minimum.

When to Use


• When there is high risk that the new changes may effect the unchanged areas of application system.
• In development process: Regression testing should be carried out after the pre-determined changes are incorporated in the application system.
• In Maintenance phase : regression testing should be carried out if there is a high risk that loss may occur when the changes are made to the system

Example


• Re-running of previously conducted tests to ensure that the unchanged portion of system functions properly.
• Reviewing previously prepared system documents (manuals) to ensure that they do not get effected after changes are made to the application system.

Disadvantage


• Time consuming and tedious if test automation not done

Requirements Testing

Usage:


• To ensure that system performs correctly
  
• To ensure that correctness can be sustained for a considerable period of time.


• System can be tested for correctness through all phases of SDLC but incase of reliability the programs should be in place to make system operational.
  

Objective:


• Successfully implementation of user requirements,/li>
• Correctness maintained over considerable period of time Processing of the application complies with the organization’s policies and procedures.

Secondary users needs are fulfilled:
• Security officer
• DBA
• Internal auditors
• Record retention
• Comptroller

How to Use

Test conditions created
• These test conditions are generalized ones, which becomes test cases as the SDLC progresses until system is fully operational.
• Test conditions are more effective when created from user’s requirements.
• Test conditions if created from documents then if there are any error in the documents those will get incorporated in Test conditions and testing would not be able to find those errors.
• Test conditions if created from other sources (other than documents) error trapping is effective.
• Functional Checklist created.

When to Use


• Every application should be Requirement tested
• Should start at Requirements phase and should progress till operations and maintenance phase.
• The method used to carry requirement testing and the extent of it is important.

Example


• Creating test matrix to prove that system requirements as documented are the requirements desired by the user.
• Creating checklist to verify that application complies to the organizational policies and procedures.

Unit Testing

In computer programming, a unit test is a method of testing the correctness of a particular module of source code.

The idea is to write test cases for every non-trivial function or method in the module so that each test case is separate from the others if possible. This type of testing is mostly done by the developers.

Benefits

The goal of unit testing is to isolate each part of the program and show that the individual parts are correct. It provides a written contract that the piece must satisfy. This isolated testing provides four main benefits:

Encourages change

Unit testing allows the programmer to refactor code at a later date, and make sure the module still works correctly (regression testing). This provides the benefit of encouraging programmers to make changes to the code since it is easy for the programmer to check if the piece is still working properly.

Simplifies Integration

Unit testing helps eliminate uncertainty in the pieces themselves and can be used in a bottom-up testing style approach. By testing the parts of a program first and then testing the sum of its parts will make integration testing easier.

Documents the code

Unit testing provides a sort of "living document" for the class being tested. Clients looking to learn how to use the class can look at the unit tests to determine how to use the class to fit their needs.

Separation of Interface from Implementation

Because some classes may have references to other classes, testing a class can frequently spill over into testing another class. A common example of this is classes that depend on a database; in order to test the class, the tester finds herself writing code that interacts with the database. This is a mistake, because a unit test should never go outside of its own class boundary. As a result, the software developer abstracts an interface around the database connection, and then implements that interface with their own Mock Object. This results in loosely coupled code, thus minimizing dependencies in the system.

Limitations

It is important to realize that unit-testing will not catch every error in the program. By definition, it only tests the functionality of the units themselves. Therefore, it will not catch integration errors, performance problems and any other system-wide issues. In addition, it may not be trivial to anticipate all special cases of input the program unit under study may receive in reality. Unit testing is only effective if it is used in conjunction with other software testing activities.

Unit Testing - Software Unit Testing, Tools, Research Topics, Toolkits, Extreme Programming Unit Testing

White box testing

White box testing is a test case design method that uses the control structure of the procedural design to derive test cases. Test cases can be derived that

1. guarantee that all independent paths within a module have been exercised at least once,
2. exercise all logical decisions on their true and false sides,
3. execute all loops at their boundaries and within their operational bounds, and
4. exercise internal data structures to ensure their validity.

The Nature of Software Defects

Logic errors and incorrect assumptions are inversely proportional to the probability that a program path will be executed. General processing tends to be well understood while special case processing tends to be prone to errors.

We often believe that a logical path is not likely to be executed when it may be executed on a regular basis. Our unconscious assumptions about control flow and data lead to design errors that can only be detected by path testing.

Typographical errors are random.

Basis Path Testing

This method enables the designer to derive a logical complexity measure of a procedural design and use it as a guide for defining a basis set of execution paths. Test cases that exercise the basis set are guaranteed to execute every statement in the program at least once during testing.

Flow Graphs

Flow graphs can be used to represent control flow in a program and can help in the derivation of the basis set. Each flow graph node represents one or more procedural statements. The edges between nodes represent flow of control. An edge must terminate at a node, even if the node does not represent any useful procedural statements. A region in a flow graph is an area bounded by edges and nodes. Each node that contains a condition is called a predicate node. Cyclomatic complexity is a metric that provides a quantitative measure of the logical complexity of a program. It defines the number of independent paths in the basis set and thus provides an upper bound for the number of tests that must be performed.

The Basis Set

An independent path is any path through a program that introduces at least one new set of processing statements (must move along at least one new edge in the path). The basis set is not unique. Any number of different basis sets can be derived for a given procedural design. Cyclomatic complexity, V(G), for a flow graph G is equal to

1. The number of regions in the flow graph.
2. V(G) = E - N + 2 where E is the number of edges and N is the number of nodes.
3. V(G) = P + 1 where P is the number of predicate nodes.

Deriving Test Cases
1. From the design or source code, derive a flow graph.
2. Determine the cyclomatic complexity of this flow graph.
   Even without a flow graph, V(G) can be determined by counting
the number of conditional statements in the code.
3. Determine a basis set of linearly independent paths.
    Predicate nodes are useful for determining the necessary paths.
4. Prepare test cases that will force execution of each path in the basis set.
   Each test case is executed and compared to the expected results.

Automating Basis Set Derivation
The derivation of the flow graph and the set of basis paths is amenable to automation. A software tool to do this can be developed using a data structure called a graph matrix. A graph matrix is a square matrix whose size is equivalent to the number of nodes in the flow graph. Each row and column correspond to a particular node and the matrix corresponds to the connections (edges) between nodes. By adding a link weight to each matrix entry, more information about the control flow can be captured. In its simplest form, the link weight is 1 if an edge exists and 0 if it does not. But other types of link weights can be represented:

� the probability that an edge will be executed,
� the processing time expended during link traversal,
� the memory required during link traversal, or
� the resources required during link traversal.

Graph theory algorithms can be applied to these graph matrices to help in the analysis necessary to produce the basis set.

Loop Testing

This white box technique focuses exclusively on the validity of loop constructs. Four different classes of loops can be defined:

1. simple loops,
2. nested loops,
3. concatenated loops, and
4. unstructured loops.

Simple Loops

The following tests should be applied to simple loops where n is the maximum number of allowable passes through the loop:

1. skip the loop entirely,
2. only pass once through the loop,
3. m passes through the loop where m < n,
4. n - 1, n, n + 1 passes through the loop.

Nested Loops

The testing of nested loops cannot simply extend the technique of simple loops since this would result in a geometrically increasing number of test cases. One approach for nested loops:

1. Start at the innermost loop. Set all other loops to minimum values.
2. Conduct simple loop tests for the innermost loop while holding the outer loops at their minimums. Add tests for out-of-range or excluded values.
3. Work outward, conducting tests for the next loop while keeping all other outer loops at minimums and other nested loops to typical values.
4. Continue until all loops have been tested.

Concatenated Loops

Concatenated loops can be tested as simple loops if each loop is independent of the others. If they are not independent (e.g. the loop counter for one is the loop counter for the other), then the nested approach can be used.

Unstructured Loops

This type of loop should be redesigned not tested!!!
Other White Box Techniques
Other white box testing techniques include:

1. Condition testing
exercises the logical conditions in a program.
2. Data flow testing
selects test paths according to the locations of definitions and uses of variables in the program.

Black box testing

Black box testing attempts to derive sets of inputs that will fully exercise all the functional requirements of a system. It is not an alternative to white box testing. This type of testing attempts to find errors in the following categories:

1. incorrect or missing functions,
2. interface errors,
3. errors in data structures or external database access,
4. performance errors, and 5. initialization and termination errors.

Tests are designed to answer the following questions:

1. How is the function's validity tested?
2. What classes of input will make good test cases?
3. Is the system particularly sensitive to certain input values?
4. How are the boundaries of a data class isolated?
5. What data rates and data volume can the system tolerate?
6. What effect will specific combinations of data have on system operation?

White box testing should be performed early in the testing process, while black box testing tends to be applied during later stages. Test cases should be derived which

1. reduce the number of additional test cases that must be designed to achieve reasonable testing, and
2. tell us something about the presence or absence of classes of errors, rather than an error associated only with the specific test at hand.

Equivalence Partitioning

This method divides the input domain of a program into classes of data from which test cases can be derived. Equivalence partitioning strives to define a test case that uncovers classes of errors and thereby reduces the number of test cases needed. It is based on an evaluation of equivalence classes for an input condition. An equivalence class represents a set of valid or invalid states for input conditions.

Equivalence classes may be defined according to the following guidelines:

1. If an input condition specifies a range, one valid and two invalid equivalence classes are defined.
2. If an input condition requires a specific value, then one valid and two invalid equivalence classes are defined.
3. If an input condition specifies a member of a set, then one valid and one invalid equivalence class are defined.
4. If an input condition is boolean, then one valid and one invalid equivalence class are defined.

Boundary Value Analysis

This method leads to a selection of test cases that exercise boundary values. It complements equivalence partitioning since it selects test cases at the edges of a class. Rather than focusing on input conditions solely, BVA derives test cases from the output domain also. BVA guidelines include:

1. For input ranges bounded by a and b, test cases should include values a and b and just above and just below a and b respectively.
2. If an input condition specifies a number of values, test cases should be developed to exercise the minimum and maximum numbers and values just above and below these limits.
3. Apply guidelines 1 and 2 to the output.
4. If internal data structures have prescribed boundaries, a test case should be designed to exercise the data structure at its boundary.

Cause-Effect Graphing Techniques

Cause-effect graphing is a technique that provides a concise representation of logical conditions and corresponding actions. There are four steps:

1. Causes (input conditions) and effects (actions) are listed for a module and an identifier is assigned to each.
2. A cause-effect graph is developed.
3. The graph is converted to a decision table.
4. Decision table rules are converted to test cases.
What is blackbox testing, difference between blackbox testing and whitebox testing, Blackbox Testing plans, unbiased blackbox testing