Open Quality Data

Where This Data Comes From

All dashboards in the list are now created by an automated process which uses CruiseControl to check for updates to the source code, and Ant to build the product, run the unit tests, and build the dashboard. The table is created through an automated process, and it is not thoroughly reviewed.

Since we are not experts for most of the products in the table, the results may at times be impacted by our ignorance. For instance, on one product the maintainer pointed out that we had measured test coverage on his tutorial files, which came without unit tests. This was quickly corrected once we were notified of the issue.

What Do the Dashboard Overview Metrics Mean?

These dashboard reports show observable data about unit test results. They cannot interpret the data. However, having used these tools ourselves and for our customers, we see a clear correlation between increased coverage, increased numbers of test points (assertions), and improved overall software quality.

The projects: The table shows some key metrics about each of the projects. If you click on the project name, you'll be able to review the supporting data in the linked dashboards in greater detail.

Number of methods: Java projects consist of packages, which consist of classes, which contain methods. The number of methods is a good indication of the size, scope, and complexity of a project. The full dashboard report also lists the number of classes, the number of executable lines of code, as well as the number of test classes, test methods, and executable lines of test code in the associated JUnit code.

Coverage: Before you can make assertions that will test code, you must be able to execute it. The dashboards report coverage as a combination of both line and condition coverage. Line coverage is a measurement of what percentage of lines of code are executed at least once during test execution. Condition coverage is a measurement of code execution that shows what percentage of all Boolean conditions in a unit of code have occurred at least once during test execution. Condition coverage provides a better measurement of the logical "testing space."

In the detailed dashboards, we present a new statistic called Outcome Coverage. Outcome coverage can only be measured for Agitator assertions, not for JUnit assertions. Outcomes are the results of method calls. For example, a method that can return a value or throw two different exceptions has three possible outcomes. The best coverage result is when all three outcomes occur during testing.

Number of assertions: This is the number of assertions created in either JUnit or Agitator. Assertions define expected behavior of the tested software and pass or fail during test execution. Assertions correspond to test points. You can analyze which of the assertions failed, in what method, and how.

Classes with test points: This number gives the percentage of the total number of classes that have assertions associated with them. The full dashboard report also lists the more granular metrics of percentage of methods with test points and outcomes with test points.

What Other Details Are Available?

In the supporting pages for each project (get there by clicking on the project’s name in the left-hand column of the main table), we present detailed data for each of the items above, plus extra information about the tests and results.

Untested code: The summary page of the dashboard report for each project also lists a set of key indicators about the code that has not been tested.

Targets: For each software project you can specify the minimum acceptable values for certain variables. These variables will be printed on a green background if they are equal to or larger than the threshold, and they'll be on a red background if they fail to meet the threshold.

Overrides: Overrides are a mechanism in Agitator to give developers the ability to exempt certain test results from being measured by the dashboard. Specifically:

• Override 1 (Bugs): If an assertion failure is the result of an issue that is known to the developer, but will not be immediately fixed, the developer can override the assertion failure and mark it as a bug. This way it does not interfere with the progress of the project.
• Override 2 (Suppressions): If the developer wants to exclude certain classes or methods from unit testing, (s)he can suppress it, providing a note stating the reason why.
• Override 3 (Coverage): In some cases, blanket target coverage levels prove to be unrealistic for a particular class or method. In those cases the developer can override them with better coverage targets. The dashboard makes it possible to easily inspect the coverage overrides.
• Override 4 (Exclusions): Some classes and/or methods may not be easily unit tested. For example, GUI code does not lend itself to unit testing. By marking them as exclusions, the developer avoids having these classes and methods negatively impact the project-level dashboard.
• Testing trends: Charts on the project summary page show trends in four key metrics (Test Failures, Coverage Percentage, Percentage of Classes with Test Points, and Total Test Points) between consecutive updates to the dashboard. Early in the open quality initiative, these charts may not yet show many data points.
• Zooming in: The dashboard reports have tabs that let you review the data per package, per class (with details per method), per test, and per developer. In this Open Quality page, the detailed information is only provided for the open source projects. For the other projects, developer names and specific class and method names are not provided (they are of course available to the development teams on these projects).
• Analysis: The analysis tab lists the highest risk classes (based on complexity and usage) with assertion failures and/or no coverage. It also shows the distribution of complexity and coverage across classes and methods.

Disclaimer

This data is provided by Agitar Software and other contributors on an AS IS basis. The data is believed to be correct at the time of publication, but no warranties or other representations are made regarding the fitness of this data for a particular purpose, merchantability, or non-infringement.