Open Quality™ Dashboards (Beta Version):

On this page we list quality dashboards for our own software products and for selected other projects. We do this as part of our evolving 'Open Quality Initiative' (see sidebar on right). During the beta period we'll expand the list of projects, validate the dashboard reports, and enhance the supporting documentation. We welcome feedback! Let us know what you think at openquality@agitar.com.
 

Disclaimer

This is a beta project at Agitar Software. All dashboards in the list are now created by an automated process which uses CruiseControl to check for updates to the source code, and Ant to build the product, run the unit tests, and build the dashboard. We do review the dashboards briefly before we publish them to the Open Quality table.

Since we are not experts for most of the products in the table, the results may at times be impacted by our ignorance. For instance, on one product the maintainer pointed out that we had measured test coverage on his tutorial files, which came without unit tests. We welcome feedback and cooperation with the development teams for the open source projects in our list.

What do the Dashboard metrics mean?

The projects: The table shows some key metrics about each of the projects. If you click on the project name, you'll be able to review the supporting data in the linked dashboards in greater detail.

What these reports are, and what they are not: These dashboard reports show the results of unit testing. They are not an attempt to show the products' requirements, results of functional/acceptance tests, or the results of system, load, and/or performance tests.

What these reports are, and what they are not (2): These dashboard reports show observable data about unit test results. They cannot interpret the data. However, as we use these tools ourselves and for our customers, we see a clear correlation between increasing coverage, increasing numbers of test points (assertions), and lowered numbers of assertion failures on one hand and improving overall software quality on the other.

Number of methods: Java projects consist of packages, which consist of classes, which contain methods. Even if there are as many ways to organize a piece of code in units as there are developers, the number of methods is some prediction of the size and scope of a project. The full dashboard report also lists the number of classes, the number of executable lines of code, as well as the number of test classes, test methods, and executable lines of test code in the associated JUnit code.

Number of assertions: The dashboard reports recognize assertions created in both JUnit and Agitator. Assertions define expected behavior of the tested software and pass or fail during test execution. Assertions correspond to 'Test Points'. You can analyze which of the assertions failed, in what method, and how.

Coverage: Before you can make assertions about the tested code, you must be able to execute it. The dashboards report coverage as a combination of both line and condition coverage. Line Coverage is a measurement of code execution that shows what percentage of lines of code are executed at least once during test execution. Condition Coverage is a measurement of code execution that shows what percentage of all Boolean conditions in a unit of code have occurred at least once during test execution.

Classes with Test Points: This number gives the percentage of the total number of classes that have assertions associated with them. The full dashboard report also lists the more granular metrics of percentage of methods with test points and outcomes with test points.

Outcome coverage can only be measured for Agitator assertions, not for JUnit assertions. Outcomes are the results of method calls. For example, a method that can return a value or throw 2 different exceptions has 3 possible outcomes. The best coverage result is when all three outcomes occur during testing.

Untested code: Although not listed here, the summary page of the dashboard report for each project also lists a set of key indicators about the code that has not been tested.

Targets: For each software project you can specify the minimum acceptable values for certain variables. These variables will be printed on a green background if they are equal to or larger than the threshhold, and they'll be on a red background if they fail to meet the threshhold.

Overrides: Overrides are a mechanism in Agitator to give developers the ability to exempt certain test results from being measured by the dashboard. Specifically:

Override 1 (Bugs): If an assertion failure is the result of an issue that is known to the developer, but will not be immediately fixed, the developer can override the assertion failure and mark it as a bug. This way it does not interfere with the progress of the project.

Override 2 (Suppressions): If the developer wants to exclude certain classes or methods from unit testing, (s)he can suppress it, providing a note stating the reason why.

Override 3 (Coverage): In some cases target coverage levels for a class or method prove to be unrealistic. In those cases the developer can override them with better coverage targets. The dashboard makes it possible to easily inspect the coverage overrides.

Override 4 (Exclusions): Some classes and/or methods may not be easily unit tested. For example, GUI code does not lend itself to unit testing. By marking them as exclusions, the developer avoids having these classes and methods negatively impact the dashboard report.

Testing trends: Charts on the project summary page show trends in four key metrics (Test Failures, Coverage Percentage, Percentage of Classes with Test Points, and Total Test Points) between consecutive updates to the dashboard. Early in the open quality initiative these charts may not yet show many data points.

Zooming in: The dashboard reports have tabs that let you review the data per package, per class (with details per method), per test, and per developer. In this Open Quality page, the detailed information is only provided for the open source projects. For the other projects developer names and specific class and method names are not provided (they are of course available to the development teams on these projects).

Analysis: The Analysis tab lists the highest risk classes (based on complexity and usage) with assertion failures and/or no coverage. It also shows the distribution of complexity and coverage across classes and methods.