Table of Contents

Introduction 3

Objective 3

Focuses 3

Strategy 4

Unit Test Bed 4

Unit Testing 4

White Box Testing Basis Path Testing of Tree Repository Module 4

Black Box Testing - Equivalence Partitioning 20

System Testing 22

System-Level Testing - Equivalence Partitioning 22

System-Level Testing - Boundary Value Testing 24

System-Level Testing - Performance Testing 27

Problem Reporting 28

Tool Usage 28

Junit 28

Custom Automation Tool 28

eValid 28

Element Tool 29

Experience and Lessons Learned 29

Overall Experience 29

Junit 30

Custom Automation Tool 30

eValid 30

Element Tool 30

Comparison of Manual vs. Automated Testing 31

References 32

Introduction

This document includes the objective, focuses, and strategy used in our test automation. In addition, it includes the test bed structure and tools used for unit white box testing and the test environment and supporting tools for the system level testing. The results and planning involved in of our automation testing will be compared with the manual testing. The actual code written for the automation and the results themselves will be included in accompanying documentation.

Objective

The primary objective for our test automation was to become familiar with the process of automating test cases and generating test data. Through this process, we were able to compare the automation with manual testing. In determining how much automation would be accomplished, we took into account the given period of time as well as the availability of tools to assist in the automation. Since all of the testing had previously been performed manually, there was no need to develop test cases as we were able to reuse the test cases used in manual testing. We kept track of the number of hours spent developing the automated tests for the selected features and in our conclusion, a comparison to the number of hours that we spent to test the same features manually is presented. These time studies will provide two measures that can be used for setting future goals and predicting the percentage of coverage from test automation:

Time to automate per feature

Time to automate as compared to time to manually test

We anticipate however, that subsequent projects would take less time because of experience.

Focuses

Because none of our team had any experience in developing automation tools, determining a focus for our testing was difficult. We spent some time just deciding what to automate and agreed to perform the following automation:

Perform test data generation

Automate test case execution (using both an existing tool and a using our own routine)

System-level testing using eValid

Problem report generation using an existing tool (Element Tool)

We then had to plan the test automation and determine how we would automate the test cases and generate test case data. The ability to control test data in the test environment is important in both manual and automated testing to ensure the validity of expected results. When tests are automated, it is important to have the ability to reset the data prior to test execution and to simulate feeding the data. We decided to prepare a separate set of data for each test case to ensure that the result of one test case did not impact the result of another. In addition, we decided to use random inputs that were generated using a specific set of rules for each required input. In generating the test case functions, we developed and enforced a set coding standards and naming conventions to promote efficiency so the results of each test case could be easily evaluated.

Strategy

Our strategy was to provide automated coverage for as many areas as possible where manual testing had already taken place. This allowed us to perform a direct comparison of the two.

Unit Test Bed

This section addresses the tools used to automate the white box testing for the repository portion of the program.

Unit Testing

Unit Testing is done at the source or code level for language-specific programming errors such as bad syntax, logic errors, or to test particular functions or code modules. The unit test cases were designed to test the validity of the programs correctness.

White Box Testing Basis Path Testing of Tree Repository Module

In white box testing, the user interface is bypassed. Inputs and outputs are tested directly at the code level and the results are compared against specifications. This form of testing ignores the function of the program under test and will focus only on its code and the structure of that code. The test cases that have been generated shall cause each condition to be executed at least once.

Each function of the binary tree repository is executed independently; therefore, a program flow for each function was derived from the code. Using the program flow graph for each function in our tree repository module, we were be able to determine all of the paths that needed to tested and have developed the corresponding test cases. In order to test the success of each path, we used Junit to create a test suite, which included all of our white box test cases. Any preconditions needed to exercise a path were created upon execution of each test case. The test cases were all executed independently, so the test data was reloaded with each test case.

Upon initialization of the complete suite being run, a random number generator is used to set the input data. It sets the variables according to the rules associated with the respective variable (i.e., root, right child, left child, etc.) We also created a version of the Junit test suite that utilizes the specific values called out in the test cases. Both sets of results are shown in the Test Automation Results. To make the internal test visible in the program, assertion checking was used. If the path executed correctly, the test was set up to return true. The test cases included in the Junit test suite are shown in the following tables.

Insert

Test Cases

Delete

Test Cases

Search

Test Cases

List - Ascending

Test Cases

List - Descending

Test Cases

Read

Test Cases

Store

Test Cases

Write

Test Cases

Black Box Testing - Equivalence Partitioning

The following table represents the automated equivalence classes, both valid and invalid, for the repository. There are many other equivalence classes at the system level that were not within the scope of this automation.

In addition to testing official test cases, we added some other miscellaneous function testing into this tool. Since a number of values were loaded (25 integers), we decided to test the functionality of the ascending and descending lists functions as well as search, delete, and clear.

Test Cases

System Testing

The goals of system testing are to detect faults that can only be exposed by testing the entire integrated system or some major part of it. Generally, system testing is mainly concerned with areas such as performance, security, validation, load/stress, and configuration sensitivity. We will perform the system-level testing allowed by eValid.

System-Level Testing - Equivalence Partitioning

The valid and invalid classes are shown below along with the corresponding valid and invalid test values. By using eValid to test the equivalence classes, we can guarantee that using invalid equivalence class values will not cause any unexpected problems within the applet.

Test Cases

System-Level Testing - Boundary Value Testing

The acceptable range of values for this application was set by the development team. Due to the limitations of the GUI, the developers also limited the size of the input values to three digit integers. The valid and invalid ranges are shown below along with the corresponding valid and invalid boundary test values. By using eValid to test the boundary values, we can guarantee that using values outside of the allowed boundaries will not cause any unexpected problems within the applet.

Acceptable Range: -999 x 999

Invalid Range: - < x < -999 and 999 < x < +

Valid Boundary Tests:

Boundary₁: x = -999

Boundary₂: x = 0

Boundary₃: x = 999

Invalid Boundary Tests:

Boundary₄: x = 1000

Boundary₅: x = -1000

Boundary₆: x = 999999

Boundary₇: x = -999999

Test Cases

System-Level Testing - Performance Testing

This test will be conducted to evaluate the fulfillment of a system with specified performance requirements. It will be done using the eValid test tool and will be performed on every button/text field in the GUI. eValid will help guarantee that there are not missing links and that all buttons are functioning properly. It will also display the results which will allow us to test the functionality of our system. (All functional specifications are included in the Software Specification).

• Load a file
• Insert an integer
• Text field
• Insert button
• Ascending list
• Ascending button
• Descending list
• Descending button
• Delete an integer in the tree and try to delete an integer not in the tree
• Text field
• Delete button
• Text field
• Delete button
• Search an integer in the tree and try to search an integer not in the tree
• Text field
• Search button
• Text field
• Search button
• Store
• Store button
• Clear
• Clear button

Problem Reporting

The online freeware tool, Element Tool, was used for problem reporting. Since all of our bugs had been fixed by the time we did the automated testing, we entered our bug from the manual testing.

Tool Usage

We were able to apply tools to all aspect of testing white box, black box, and system-level testing.

Junit

Junit was used to perform white-box testing on the Binary Search Tree repository code. We were able to create a test suite that would test every path identified using the Basis-Path Testing approach. We tested a total of 31 of our test cases. We were also able to implement automated test data generation within this tool.

Custom Automation Tool

We used our custom automation tool to perform black-box, equivalence partitioning for the Binary Search Tree repository code. In addition, we were able to do some unit-level functional testing to see how each of the separate pieces worked together. We included detailed comments that were both written to a file and displayed to the screen to verify every step in the automation routine.

eValid

The eValid tool was used at the system-level to test the functionality of the applet. It can confirm that there are no broken links and that each button is working properly. The advantage to using eValid is that it can record a string of operations that can be performed over and over using the playback feature. This is useful if changes are made to the code since it records the operations.

Element Tool

Element Tool was used for bug tracking. All of the bugs from the manual testing phase were entered into Element Tool for tracking.

Experience and Lessons Learned

Overall Experience

Upon completion of our automation project, there were many lessons learned about the automation process as well as some different pros and cons while working within an automation paradigm. One of the biggest challenges of automation is setting up a specific application to integrate within an automation infrastructure. Because all software is different, third party automation software vendors can only make their best predictions on how an application might be structured in order to successfully automate. This can cause problems within any automation project because most software is created differently and will not always use standard objects. In fact, software engineers will sometimes use rather clever methodologies in creating certain modules that a third party vendor never even thought of designing their automation tool around. This problem affected our own binary search application while trying to automate within eValid. It was apparent that eValid could not find a certain class within Java and so it would not allow any automation scripts to run. It took weeks of communication with the vendor before we were given a solution to the problem.

Other problems with automation can sometimes occur while writing the scripts themselves. If a script is not written properly or not robust enough, although it may work correctly, many a time it is only temporary. As a software product evolves over time, scripts that have once worked in the past will cease to work and may either require modification or abortion. This system will create a constant maintenance requirement for all automation scripts produced past, present, and future.

Although automation can be difficult to setup and maintain, there are definitely strong benefits to it. During the manual testing phase of our project, it took many hours to determine requirements, write each black box test cases, and write each white box test case. Furthermore, after the test cases were completed, it took even longer to go through each one and manually test each part of the product. After each test case was modified, this process quickly became effortless. What initially took hours to complete, was done in a matter of seconds. On account of this, these tests could be repeated over and over again. This would be of a great benefit as when the product changes over time, the entire test conglomeration could easily be run as many times as management would require. Furthermore, test generation could be truly random and tested more thoroughly. Overall, difference in time is so significant that testing the legacy areas of a product become faster and easier.

Junit

Junit was our favorite tool as it was easy to use and provided just the right features to perform white box testing in an organized, simple manner. Since white box testing can sometimes be difficult to perform, this tool is especially useful. It only took a few minutes to figure out how to apply and the coding of the test cases was very straightforward. It would also be a useful tool if changes were to be made to the code. The complete suite of white box testing can be done in one step. In addition, be implementing the random number generation, all of the test cases could be run using completely different sets of values in a very short time. This provided even more assurance in the correctness of the code.

Custom Automation Tool

The custom generated test tool was more of an experiment to see how difficult it would be to develop our own tool. Since the white-box testing was already covered by the Junit tool, we decided to implement this tool for black-box testing. With the idea of how to develop a tool achieved using Junit, the custom tool was not too difficult to create. It turned out to be a great way to not only perform two of the black-box test cases, but also test how all of the functions in the tree repository function together.

eValid

The eValid tool turned out to be by far the most frustrating. Not only did we have to convert our application into an applet just to use the tool, we also spent two weeks emailing back and forth with eValid to get the tool to recognize our applet. Then, even after we got the tool to recognize the applet, it was difficult to get the system properly configured so it would operate properly. It would initially not recognize our mouseclicks or text entry and we finally received word from eValid that we had to use the absolute mode for it to record. Because it took so long to get everything worked out with eValid, we ended up using another tool that would allow our application be recognized by eValid. We found a tool called WebCream that converted the applet/application into JSP/HTML pages. Once we did this, we were finally able to actually create and play back a script. Automation tools are only useful if they make testing easier or provide better coverage. This tool really did neither. The documentation was not adequate to solve the problems on our own and relying on the support team resulted in numerous delays in our testing. Now that we have the tool working, future testing would be simplified as we could just playback our recorded series of mouseclicks to retest the whole set of system tests. Unfortunately, were not sure if it was worth all the trouble.

Element Tool

Our team did not feel that there was any value added by using this tool. Since we were only using the freeware version, it did not provide enough detail or flexibility to effectively manage our problem reports. It was however very easy to use and would probably be more worth while if were able to use one of the more robust versions.

Comparison of Manual vs. Automated Testing

Since this was our first experience with test automation, we werent as efficient as we could have been. It took us a while to understand you to use Junit, we had numerous problems with eValid that increased testing time, and we also needed to spend time figuring our where automation could be used. All of these tasks would have taken less time if we had some previous experience. With the automation in place, however, subsequent testing will be much quicker and easier. We could make changes to the software and easily rerun all of the unit tests. We can also change the input data and rerun the test suite. Based on our best estimation, the following timetable was produced to provide a direct comparison. The times below do not reflect the time spent in developing the test cases which took up much more time than the actual testing.

* does not include the 20-30 hours spent working out the eValid problems

References

Pressman, Roger S. Software Engineering - A Practitioner's Approach. Fifth edition. The McGraw-Hill companies, Inc.

Kaner, C., Falk, J., Nguyen, H.-Q. Testing Computer Software. Wiley Computer Publishing, 1999.