Software Development: Contrasting Beta Testing with Traditional Testing
By Donna Bridgham, Division of Aging and Rehabilitative Services, Commonwealth of Virginia
During the early years of software development, few companies had formalized test processes. Many software packages were released to the public with bugs, errors, and configuration errors. As the computer industry grew and the types of operating systems escalated, many companies saw the need for software testing.
During the turn of the century, Y2K (Year 2000) testing became an important part of technology and this determined how programs were written and tested. Some companies realized the importance of testing during this initiative, and they started testing and added quality assurance departments as a part of their business practice. Other companies saw that it was expensive to test and test correctly, so they released beta versions of their software for professionals such as developers and programmers to use as a means of testing.
("Computer Software" is an image by Nick Youngson. It is released through a CC BY-SA license through Alpha Stock Images.)
This form of testing brings about the concern to whether beta (β) testing or traditional testing is better.
Defining Beta Testing vs. Traditional Testing
Beta testing is the process in which users, developers, and programmers are allowed to use new software before it is released for sale to the public. Many leaders in large companies feel that too much money is spent testing software, so they feel that if professionals use their product and record issues, errors, and bugs, they will be able to produce a quality product.
Beta software is an application that is a full version of a product up to date but is not ready for full implementation. This means the application is ready to use but not ready to sell because there may be issues with the software. Companies such as Microsoft Corporation believe that their developers have taken the software as far as it will go, and since they do not have the time to test the products themselves, they are going to release beta versions to professionals for testing.
Traditional software testing follows a process that a tester or team of testers will take the software through to see if and when the software will be available for implementation. Many companies feel that if the time is taken to do a formal test process, they will produce a software application that is ready for use in the business world. A traditional test process usually consists of the following parts: a test plan, business requirements, test scripts, test scenarios, maintenance, and documentation.
Traditional testing follows a formalized test plan. Organizations such as the IEEE (Institute of Electrical and Electronics Engineers) have publications that introduce a standard approach for designing a test plan. The test plan should follow a standardized process in order to have a strong test presence. The test plan includes sections of documentation which are invaluable to software professionals.
Even though many companies such as Microsoft use beta testing as part of their software testing process, traditional testing should not be replaced by this form of testing. Since companies are replacing traditional testing with beta testing, an evaluation of both forms of testing will be conducted. From this, a determination will be presented as to the form of testing that is the most cost efficient and which form of testing releases quality software.
A Public Case: Microsoft and Beta Testing
When computer programs are written, literally thousands of lines of code are written. Testing is performed on some of the code, but it is too expensive and time-consuming to test each line of code. Many companies such as Microsoft believe that it is impossible to test large applications so they create beta-version software. They allow users to pay for the privilege of testing their software.
“Microsoft will get users to pay for the privilege of testing the product to find those bugs before the product's final release. On the surface, paying for a new version of software that comes with bugs but without formal support seems foolish, but Microsoft discovered that the prestige of having beta software was enough to get people to fork over cash for the right to tell their friends that they have the beta code." (Chernicoff, 1999)
As a developer, using beta products can have an advantage for developers. If there is a computer that can run a beta version of the software, the developers will be able to look at a product and evaluate it before it is released to the public. It will help the organization decide if the product is something that would be useful to them, and it would help them anticipate the operating and hardware needs for running the software.
When .NET was introduced, many developers thought that they would have to have a powerful processor, ample memory, and an operating system that would support all the files associated with the .NET environment. For the developers who used .NET in its beta form, they found a product that had the potential to be powerful and replace several existing technologies. Also for companies that were looking for Window Form solutions as well as Web solutions, .NET was the perfect software for satisfying both needs. As .NET went from beta to actual proprietary software, many people were ready for the technology and glad they had an opportunity to test the beta version.
Beta testing does have disadvantages. Developers using the beta version of particular software look at the software from a different perspective than a Software Test Engineer. They know what they want to do with the software from a developer’s viewpoint and only test that portion of the software. Another disadvantage of using beta software is that in many cases, the released software is much different than the beta version. As a developer, if you liked a particular feature of the software, there is no guarantee that these features will be part of the released software.
Microsoft does make money on selling beta software but with products such as Office 98, there were many problems. Some businesses lost their confidence in Microsoft Corporation because of issues with the software but as newer versions such as 2000, XP and 2003 were released this confidence was regained. If Microsoft had used a formalized test initiative, there may not have been as many problems. [Over the years, Microsoft has developed a reputation for being more mindful of software security and addressing bugs.]
Beta Software Test Process
When a beta test is performed, it is usually by users outside of an organization. The organization expects the user to give them feedback as they use the product. One form of testing that is based on beta products is threshold testing. This type of test measures the performance characteristics of a system against "predetermined goals". For example, could system users complete a particular task in a particular time with only a certain number of mistakes? If it does, it passes; if it doesn't, it does not meet the "release criteria" (U.S. Bureau of Labor Statistics, n.d.).
Beta releases are good for trying out a particular product. It allows users to get an idea of what is going to be available in a new or existing product. Allowing people to use beta products and saying it is your testing initiative is not a good way to rely on finding bugs and errors in your products. If there are systematic processes and documentation for each problem, bug, or error, the software should be reliable.
Traditional Software Testing
Traditional Software Testing is when a company develops software and runs a formal test process on the software before it is released to the public. The testing is usually done with a test team that may include a Certified Software Test Engineer, End Users, Programmers, Developers and a Project Manager. Each person in the team is assigned a task and responsible for their area of the test process. For one researcher, a test plan for software testing usually includes the following components (verbatim):
- "Objectives
- Testing Strategy
- Scope
- Reference Material
- Definitions and Acronyms
- Test Items
- Features to be tested,
- Features not to be tested
- Approach
- Pass / Fail criteria
- Documentation
- Environmental needs
- Responsibilities
- Schedule
- Risks and Contingencies
- Checklist” (Stottlemyer, Spring 2004, p. 35)
The test plan can be long, tedious and expensive but by following a formalized plan, your organization will be able to find bugs and errors that exist in the present version of the application.
The biggest advantage of having a formalized test plan is the documentation that is created and stored during and after the process. As each step of the test plan is gone through, there is associated documentation. The documentation will address what happened, what went wrong, and how each error or bug’s correction was handled. This documentation can be published into a manual for users or can be placed on a web site as a white paper. This allows the users of the software to read about the software and potential problems and to also buy additional hardware/software as needed.
The biggest disadvantage that many people see in a formalized test process is that it is costly and time consuming. This may be a disadvantage up front but if your company releases software that has several bugs and it needs to be rewritten, the company will lose the confidence from your customers and will have to pay for the cost of rewriting and releasing the product.
Forms of Software Testing Documentation
When testing software, applications and databases all errors and bugs should be tracked. This should be done no matter what type of testing you are doing. A database can be created using a tool like MS Access, SQL Server, or MySQL. The database should have fields that match the testing process. A dashboard can be created that will allow different actions to take place.
Figure 1. Simple Problem Tracking Database Switchboard
When the user or one of test team creates a script to use to test the application the data can be stored in the database if actual results are found. By keeping track of the problems, they can be placed in the database and assigned to a developer or engineer to fix depending on the issue. The test script should be used and labeled so there can be a cross reference in the database.
Figure 2. Test Script from Automated Web Testing Toolkit (Stottlemyer, 2001, used with permission)
Once the scripts have been collected, then the data is logged into the database according to the information your organization needs to proceed. A form can be created that will input the data into the database. The information should include who will fix the problem, what is the resolution for the problem, the priority of the problem, and other data needed for creating reports that can be used throughout the test process.
Figure 3. Form for Recording Problems When Testing
Once you have collected the data, reports can be created to help mitigate the process. The reports can prioritize the problems, can list problems that have been resolved and other reports that are needed for your organization.
Figure 4. Making a Report in MS Access
Traditional Software Test Process
The traditional test process includes the following components. The test process begins with an understanding of the background of the software and its purpose. If a card sorting process is available it may be beneficial to look at all the cards (index cards or notes) that were compiled for the process and see how they fit into a test initiative. Then a Heuristic Evaluation is done to make sure there is a project overview that describes the objectives, target audiences and expected usage patterns of the software that needs to be tested. The process proceeds to Scenario-Based testing which allows a test for different scenarios that will come to light during the use of the software. Then a Questionnaire for User Interaction Satisfaction can be implemented to make sure that the product meets the need of the user. This is one of the most important steps in any software test process since the user is going to be the ultimate end user of the product.
It is also important for someone to maintain the logs. The system will create a log that will allow testers to see what is happening when the software is being used. It can track transactions, errors or problems with the software. This will allow a tester to see what is happening to the application when it is being used in real time.
At this point, it is important to bring a conclusion to the test process. Once your usability testing is complete, all the items on the test plan are checked and completed your company is ready for a full release. The test process then moves into maintenance testing. Even though maintenance testing is part of the process and should continue throughout the software life cycle, the software is ready for release and a new project will start.
A Beta-testing Testimonial
Some companies believe that using beta products and then becoming a partner with that company will save them time and money. From an article in TechRepublic, the following testimonial was asserted by Mindbridge Software:
“The tech leader advises that companies and the vendor spell out specifics—including expectations for response times, uptime, installation support, and expected outcomes…An important aspect of the negotiations involves incentives—such as the free application Mindbridge received for participating—and the incentives will differ from one beta contract to another. For engineering betas, it’s not unreasonable to expect a free copy of the software. For sales and marketing betas, some kind of discount or preferential service contract could be negotiated…Keep in mind that not all beta programs are created equal. Nor do they always run as smoothly as Mindbridge's, even if a tech leader is familiar with the process. To avoid possible pitfalls, it’s critical to know exactly what kind of beta is being offered and to ask as many questions as needed.
“For Mindbridge and the vendor, the recent beta experience was a mutually beneficial exercise. Not only did Mindbridge get free CRM software, but it also purchased one of the vendor’s other products. “We got some free software, and now we’ve become a paying customer,” Testa said. That’s just the kind of win-win outcome you can expect from a well-negotiated and well-executed beta” (CNET Networks, 2004).
For Mindbridge, this was a good solution to their software problems. One reason that this was a successful venture was they had the resources to do testing within their company on the beta products. If you are a small company and need to rely only on beta products you may end up releasing inferior products which you could not afford to fix.
Traditional Software Testing Costs
Though many people believe that traditional testing is a waste of money, professionals agree that the quality achieved during the test process will save money in the long run. Kaner (1996) points out the reasons for traditional testing:
“Quality costs are the costs associated with preventing, finding, and correcting defective work. These costs are huge, running at 20% - 40% of sales. Many of these costs can be significantly reduced or completely avoided. One of the key functions of a Quality Engineer is the reduction of the total cost of quality associated with a product.
Below are six costs that can be reduced with a good test initiative:1. Prevention Costs2. Appraisal Costs3. Failure Costs4. Internal Failure Costs5. External Failure Costs6. Total Cost of Quality“ (Kaner, 1996, p. 1)
Correcting faults and failures before software is released saves money. You will also gain the confidence of customers so they will continue to purchase software from your company.
Example of Deliverable for Testing
When testing, you need the final deliverable to be an application that meets the requirements given during the discovery phase. Requirements may need to be added as the project progresses but risk, time and cost will be evaluated for each. For built applications, the deliverables occur in four stages:
1. Unit Testing2. User Testing3. Acceptance Testing4. Beta Release for one month
You will also include testing with assistive technology. Accessibility testing is part of the testing stages. If training is needed for the testing it will be provided as a preliminary stage to end user and acceptance testing. Usability and accessibility are included in the test phase since applications should be built with a universal design in mind.
All requirements from the I/O screen will be tested against standardized organizational scripts. If you are converting an Access Database to a SQL Server Web-based application the following would be the steps for testing. During the analysis phase, the activities listed below were conducted to gain the necessary information and knowledge of how the application performs business procedures.
- An analysis of comparing the data from the MS Access Database to the SQL Database Tables, queries were run with test data to ensure to make sure that the tables did not leave any orphans.
- Inner joins were tested to make sure that duplicated data was not displayed during the running of multiple queries.
- Tables were checked to make sure that the names, data types and size complied with agency standards.
- Interviews were held with the users of the present application after the database was designed to ensure that none of the concepts or requirements was left out of the design.
Some tools that can be used are MS Office 2015 Suite, WebAIM for Accessibility Testing, Six Sigma Control Testing, Organizational Test Scripts and MS Visio for screen shots.
This will all be done on a developer’s server, once verified the data will be moved to a test server and then during implementation, the data will be moved to a production server.
Since the front end that has the web forms for I/O, it will also be built on a developer’s server and verified, moved to a test server and once approved will be moved to production for full implementation.
The product will have four levels of sign-offs:
- Discovery phase sign off
- Developer’s sign-off to move to test
- User acceptance testing and approval
- Managerial approval for implementation
There are three platforms that are set up that have a web server and SQL Server for development. The Projects all start on Sandbox, so developers can build the database and web interface. Unit Testing is done on this platform. Once the initial phase of each piece is done, the users meet with the project lead and developer to review and approve. If the user approves, the application will be moved to the Development Platform for User Testing and approval.
The next platform is the development platform for testing. All user acceptance testing is done on this platform. The environment of this platform is identical to the production server so the users will have the real experience of using the application. On this platform, the following occurs:
- A script is given to the user with the requirements for each screen they will be using, they check off what is working correctly and note any issues. These issues are given to the Project Lead to tabulate and send to the Project Manager.
- Any changes that need to be made are done and reviewed on this platform. Once again it is reported to the Project Lead who reports to the Project Manager.
- Any additional risks that are uncovered are documented and sent to the Project Lead and Project Manager.
- Once all issues are resolved, the users sign off and the managers come for a formal walk-through of the application and sign off.
The next platform is the production platform. Once all the sign-offs have been received, the Project Manager works with the Project Lead to schedule a move to production. When the implementation has been approved, the application is moved to production and tested for one month before there is a full sign-off. Any issues, bugs or fixes are tracked and remediated. If there are any showstoppers or high risks documented during this time, a decision will be made whether the application should stay on production or return for more testing.
Agile Testing
Agile software development is a set of processes for software development. Agile is known as a programming form that through collaboration offers solutions. It promotes adaptive planning, evolutionary development, early delivery, and continuous improvement, and encourages rapid and flexible response to change.
Agile testing, unlike other forms of testing, tries to identify and fix faults/errors immediately instead of waiting until the end of the project. When developing a project, if all testing is done at the end of the project, it can increase the time and cost of the project. If development coincides with testing, it will help bring the cost down and allows for faster deployment of applications. The other advantage of testing during the agile process is that when testing is an ongoing process, bugs will take less time to repair and evaluate. This also ensures that the code, if reused, will already be tested, and developers will be ready to go onto the next project.
Different types of software development require different team members. Pressman (2010) points out that Agile Development requires a team that can complete different tasks. The members must have generalized skills. This helps the team moves from one point to another and keeps the team structure in place.
In other software development projects, a project manager assigns resources to the project, and the resources include the development team. When Specifying Non-functional Requirements (NFR), they need to be defined so they are part of the support and future of the application ("Scaled Agile Framework," 2013). An NFR should be bounded. Each NFR should be independent of each other so it can be tested independently. The NFR should be negotiable. Understanding the context mandates that there be some room for understanding a measurable negotiation. Each NFR should be testable ("Scaled Agile Framework," 2013). If it can’t be tested, it can’t be used because you will not know if it meets the criterion.
Conclusion
It is clear that different groups of professionals view testing differently. Professional and Certified Testers believe that traditional software testing is the only way to ensure the quality of the software. Many companies such as Microsoft feel that beta testing allows them to release their software quicker and also allows developers to do their own testing. It seems that any company who wants to be successful would follow a traditional test process. Traditional testing will allow for the company to create a test process that would ensure that they would release quality software to their customers and users.
Beta testing is designed for professionals within the field of software. Programmers and developers who do beta testing understand the product and how it works. They will use the beta product as they would use it in their environment. They do not care or do not test the product to work in ways other than how they will use it. As developers find bugs in beta testing it is difficult to contact the company and tell them exactly what the problem is in the beta product before it is pushed into production.
From my view point, I believe that Traditional Testing is a good way to see if your software works as designed, follows the business requirements for the product and has been through a thorough Software Testing Life Cycle. If the software can pass through the test process, it is ready for release and provide customers a quality product. Traditional testing will save you time and money in the long run, if it is done correctly.
References
Chernicoff, D. (1999). Microsoft and Beta Testing. Retrieved Sept. 14, 2004, from http://www.winnetmag.com/Windows/Article/ArticleID/4793/4793.html
CNET Networks, TechRepublic. (2004). Participating in beta testing offers big cost savings Retrieved September 18, 2004 from http://techrepublic.com.com/5100-6296-1044364.html
Electrical and Electronics Engineers, Inc. (2004). Home Page. Retrieved Sept. 12, 2004 from http://www.ieee.org/portal/index.jsp
Kaner, K (1996). Quality Cost Analysis: Benefits and Risks Retrieved September 20, 2004 from https://www.kaner.com/pdfs/Quality_Cost_Analysis.pdf.
Microsoft, Inc. (2004) Security. Retrieved Sept. 15, 2004 from http://www.microsoft.com/security/bulletins/default.mspx
Mohanty, H. Mohanty, J. R., & Balakrishnan, A. (2017). Trends in Software Testing. Switzerland: SpringerLink. doi:10.1007/978-981-10-1415-4.
Pressman, R. (2010). Software engineering: A practitioner's approach (7th Ed). New York, NY: McGraw-Hill Education.
Scaled Agile Framework. (2013) Nonfunctional requirements abstract. Retrieved from http://scaledagileframework.com/nonfunctional-requirements/
Southgate, D. (2004). Participating in beta testing offers big cost savings. Retrieved May 2019 from https://www.techrepublic.com/article/participating-in-beta-testing-offers-big-cost-savings/
Stottlemyer, D. (2004) Partnership with People with Disabilities Web Accessibility Test Plan. Masters Thesis, California State University - Dominguez Hills (CSUDH).
Stottlemyer, D. (2001) Automated Web testing toolkit. New York: Wiley, 2001.
U.S. Bureau of Labor Statistics. (n.d.) Usability Testing of World Wide Web Sites. Retrieved Sept. 18, 2004 from http://stats.bls.gov/ore/htm_papers/st960150.htm
About the Author
Donna Bridgham is a Systems Analyst with the Commonwealth of Virginia with the Division of Aging and Rehabilitative Services. She provides customer service to the Disability Service Agencies. Bridgham is proficient in breaking down high level information into details for designing databases and prototypes for application design . She also has a technical expertise in data analysis. She has a B.S. from Lyndon State College, an MBA from Northcentral University (NCU), and an MS in Information Technology from American Public University System (APUS). She may be reached at donnabridgham@comcast.net.
Note: The author has been studying software beta-testing for years. She worked to update the research but did not find additional sources beyond these. Some of the older referenced resources are no longer available on the Web and Internet, but they were retained in the References list to indicate original sourcing. Some of the best practices for testing software continue through the present, even though there are newer tools that enable some automation in the testing.
| Previous page on path | Cover, page 15 of 23 | Next page on path |
Discussion of "Software Development: Contrasting Beta Testing with Traditional Testing"
Add your voice to this discussion.
Checking your signed in status ...