JUnit, Assertions & Management
I've recently been having a discussion about the quality of JUnit tests and how manager types might be able to judge the quality of the tests.
In my mind this is a not so easy to solve. I know managers (esp on very big projects) need a way to judge the quality of what is being delivered but often times they take their statistics far to seriously. As an example take code coverage calculated by Clover or the like. As a developer I find code coverage results to be a fantastic tool to determine if my developer tests are covering all the code under test that they should. And to a certain extent the code coverage numbers are good to give to management so they can see the progress of the tests. If the tests are good then as this number goes up you should be able to assume better quality in the code base. All too often though I see tests that have 100 lines of code and a single println at the end, not a single assert. Tests like this will generate a lot of coverage but don't really tell you anything about the quality of your code.
In my experience developer tests are too often on the poorer end of quality. So on to the point of my post. How do we measure what good is? There are the simple statements link 'each test should be tightly focused and have one assertion'. The problem is that this statement is too broad to measure the quality of your tests. This statement could possibly be refined with a statement like this 'the ratio of non-comment lines of code to assertions should be low'. In other words you should have a few lines of code in a test and an assertion to make sure that what was expected actually happens.
What are your thoughts on measuring test quality? Should management even be concerned with this measurement?
I think unit test quality is a bit below the purview of management. It's more the concern of technical leads and/or senior programmers (unless your definition of management includes these folks) who teach it and then monitor it the old fashioned way: by looking at the tests themselves. If quality were of high enough concern on the project, unit test would be examined as part of the tested code?s code review. It seems to me that management being concerned with unit test quality would be a sign of micromanagement.
At the minimum, I think I agree with the unspoken premise to the comment about statistics: if they are overly relied upon by management, those managed will find a way to manipulate them to the point they loose their usefulness.
Posted by Steve on August 30, 2004 at 01:42 PM MDT #
I think that code coverage is a key metric in driving up the value of the test suite. But you can still fall way short if it's your only guide. What unit testing should really be about is "results coverage". I have seen plenty of tests that exercise a bunch of code, but assert nothing. The only result coverage then is that it didn't throw any exceptions.
I do think that management needs be involved, because in any real project, unfortunately, some kind of stick/carrot motivator is generally required. Code coverage reports help. Result coverage reports would be even better. I'm just not sure how to measure it directly. For now, it helps to at least charaterize coverage in those terms.
Posted by Chris Noe on September 01, 2004 at 06:25 PM MDT #
[Trackback] daddy earnest diminution.twigs twinkle,transforming modesto mortgage loan http://www.enter-mortgage.com/modesto-mortgage-loan.html
Posted by mortgage broker tools on June 19, 2006 at 05:07 PM MDT #