QA Plan

Release Information

Project: PIMS
Internal Release Number: 0.1
Release Audience:
Developer release (Internal usage only)
Early access release (Controlled external access)
Attached Worksheets:
QA plan > SAMPLE System test case suite
QA plan > SAMPLE System test runs
Related Documents:
Process impact: This document specifies quality goals, selects strategies for assuring that those goals have been met, and details a plan of action to carry out those strategies.

Introduction

Why is this QA plan needed?
"Quality" refers to all the good things that we would like to see in our product. We build a quality product and assure its quality by keeping quality in mind all the time and performing the selected activities below. Testing is one QA activity, but it is not the best or only one, other QA activities include the use of style guides and checklists, review meetings, use of analysis tools, and careful quality measurements and estimates. A plan is needed to select and coordinate all the QA activities.
What QA lessons were learned in previous releases?
None yet. This is the first release.
What is the scope of this QA plan?
All components and aspects of the system will be evaluated in this release.
There are many quality goals and approaches to assuring them. Since we have limited time and resources for this release, we will focus on the following components and aspects:
  • SQL Table Definitions, review to reduce future changes
  • Data access layer, basic functionality only
  • Data Model API, review to reduce future changes
  • Generic UI demo quality only, since UI designer is not appointed yet
  • Specific UI and Lab API, proof of concept only
What is the summary of this plan?
The goals for release 0.1 are to:
  • provide a context for discussion with scientists about features required
  • validate the software architecture
  • provide a basis for independent development of different features in future releases
  • validate the groupware and QA techniques
We will do that with the following major activities:
  • review the API offered by each layer
  • performing automated unit and regression testing with JUnit
  • using assert statements to test pre-conditions, invariants and postconditions
  • beginning automated testing through the UI with jWebUnit
  • keeping all defect reports and to-do items up-to-date in an issue tracking database

QA Strategy

Activity Coverage or Frequency Description
Preconditions
All methods
 We will use assertions at the beginning of all methods to validate each argument value. This helps to document assumptions and catch invalid values before they can cause faults. It also helps to make random test case generation more effective, by providing some semantic checks on the processing that it causes.
Class invariants and postconditions
Some methods
 Assertions will also be used to test class invariants and some postconditions, at the developer's discretion.
Unit testing
All methods other than simple getters and setters
75% of basic blocks
 We will develop and maintain a unit test suite using the JUnit framework, for all classes below the UI layer. Tests must be run and passed before each commit. The test suite will exercise at least 75% of all basic blocks in each class. We will measure test coverage with EMMA. The QA team will prioritise for testing and review those classes that have the least unit test coverage. The number of unit tests that have been passed will be our primary measurement of code size.
Static analysis
Strict compiler warnings
Automated style checking
XML validation
Lint tools
 Compilation will be done with the maximum level of warnings requested. Our Java compiler is Jikes.
We will use checkstyle to check for errors and inconsistency with coding standards. We will use jlint to check for common programming errors. XML validation ensures that each XML document conforms to its DTD. Resources permitting, we will try other lint tools e.g.: PMD, Jcsc, PyLint, PyChecker, Tidy
Automated system testing
100% of HTML screens and fields
100% of scenarios from the requirements
Applets TBD
 The QA team will use jWebUnit to author and maintain a suite of test scripts to test the entire system through the user interface. These will check error paths as well as correct operation, and in particular will test for CSS and SQL injection vulnerabilities. Resources permitting, we will also write a test script that takes a random walk through the system, after loading the database with test data. We will evaluate the use of the java Robot class to test applets.
Regression testing
Run all unit tests and UI tests nightly
Add new test when verifying fixes
 We will adopt a policy of frequently re-running all automated tests, including those that have previously been successful. This will help catch regressions (bugs that we thought were fixed, but that appear again).
Field failure reports
Prompt users to report failures
 We want to understand each post-deployment system failure and actively take steps to correct the defect. We will establish a mechanism for users to report problems.
Review meetings
Every team meeting
 We will hold review meetings where developers will perform formal inspections of selected code or documents. We choose to spend a small, predetermined amount of time. These are intended to find defects in maintainability and security of the code, or defects in requirements, design documents, and project plans.
Manual system testing
TBD
 Resources permitting, the QA team will find published experimental protocols and use them as test scripts for the system. Before major releases, the whole development team will spend a week on testing and defect correction.
Load, stress, and capacity testing later
TBD
 We use a load testing tool and/or custom scripts to simulate heavy usage of the system. Load will be defined by scalability parameters such as number of concurrent users, number of transactions per second, or number/size of data items stored/processed. We will verify that the system can handle loads within its capacity without crashing, producing incorrect results, mixing up results for distinct users, or corrupting the data. We will verify that when capacity limits are exceeded, the system safely rejects, ignores, or defers requests that it cannot handle.
Instrumentation and monitoring
TBD
 We will consider what logging facilities are appropriate.
Buddy review, later
TBD
 Whenever changes must be made to code on a release branch (e.g., to prepare a maintenance release) the change may be reviewed by another developer before it is committed. The goal is to make sure that fixes do not introduce new defects.

Plan of Action

TODO: Tasks should be assigned to individuals and tracked to completion.
  1. Agree a coding style standard, at first team meeting. Developers whose preferred coding style is different from the project standard are welcome to use Jalopy after checking out and before checking in.
  2. Preconditions and Assertions
    • Write preconditions and assertions in code
    • Create validation functions for use by preconditions and assertions, as needed
  3. Unit tests and static checks
    • Set up infrastructure for easy execution of JUnit tests (this is just an Ant target)
    • Create unit tests for each class when the class is created
    • Execute unit tests and checkstyle before each commit. All tests must pass before developer can commit, otherwise open new issue(s) for failed tests. These "smoke tests" will be executed in each developer's normal development environment.
    • Execute unit tests (with EMMA to check coverage) completely each night, and also checkstyle and jlint. Execute unit tests completely each weekend, with Jester to check coverage. These tests will be carried out on the development server.
  4. System tests
    • Execute system tests daily. These system tests will be carried out on the development server.
    • Review the system test suite to make sure that every UI screen and element is covered
    • Investigate the possibility of using EMMA with system tests.
  5. Review meetings
    • Select an at-risk document or section of code for review at team meetings. The selection criteria will be
      1. user defect reports
      2. low test coverage
      3. metrics that indicate a risk of defects, notably import coupling
      4. warnings from static checkers
    • Reviewers study the material individually in advance
    • Reviewers meet to inspect the material for 1 hours. We may divide into several groups to review several classes at once.
    • Notes of issues found will be given to the author of the code, and also used to accumulate a checklist for future reviews.
    • We will consider changes to the coding standard or checking rules in the light of issues found.
  6. QA Management
    • Update this test plan whenever requirements change
    • Document test results and communicate them to the entire development team
    • Later estimate remaining (not yet detected) defects based on current issue tracking data, defect rates, and metrics on code size and the impact of changes.
    • Keep all issues up-to-date in an issue tracking database. The issue tracker is available to all project members here.

QA-Plan Checklist

Do the selected activities in the QA Strategy provide enough assurance that the project will meet it's quality goals?
No, we do not yet have sufficient plans to verify usability.
Have human resources been allocated to carry out the QA activities?
In progress. See the Resource Needs document.
Have machine and software resources been allocated as needed for the QA activities?
Yes, the QA team will use desktop machines and servers that are already allocated to them.
Has this QA Plan been communicated to the development team and other stakeholders?
Yes, this document is being posted to the project website. It will also be discussed at the first team meeting. Feedback is welcome.
Company Proprietary
Copyright © 2003-2004 Jason Robbins. All rights reserved. License terms. Retain this copyright statement whenever this file is used as a template.