A Gentle Reminder about Test Coverage

| 7 Comments

Devel::Cover is wonderful, and so is Dist::Zilla::App::Command::cover. Measuring your test coverage can help you increase your confidence in the correctness and reliability of your software.

The numbers in the coverage report are only tools, however. Your job as an intelligent and capable human being is to interpret those numbers and to understand what they mean.

For example, my new book formatting software (which needs more documentation before I release it publicly) has a handful of hard-coded escape codes for the LaTeX emitter to produce the right code. Part of that code is:

my %characters = (
    acute    => sub { qq|\\'| . shift },
    grave    => sub { qq|\\`| . shift },
    uml      => sub { qq|\\"| . shift },
    cedilla  => sub { '\c' },              # ccedilla
    opy      => sub { '\copyright' },      # copy
    dash     => sub { '---' },             # mdash
    lusmn    => sub { '\pm' },             # plusmn
    mp       => sub { '\&' },              # amp
    rademark => sub { '\texttrademark' }
);

sub emit_character
{
    my $self    = shift;

    my $content = eval { $self->emit_kids( @_ ) };
    return unless defined $content;

    if (my ($char, $class) = $content =~ /(\w)(\w+)/)
    {
        return $characters{$class}->($char) if exists $characters{$class};
    }

    return Pod::Escapes::e2char( $content );
}

While emit_character() is interesting on its own and worthy of testing, the important code is the %characters data structure. Devel::Cover can't tell me if every entry in that hash gets accessed appropriately (though I suppose it could in theory track the use of the anonymous functions). Only my knowledge of the tests and the code can satisfy me that I've tested this important code thoroughly.

Again, Devel::Cover is a great and useful tool, and acknowledging limitations like this in no way diminishes its efficacy. (It's particularly informative about branch and condition coverage.) Yet as with most tools, it exists to enhance human knowledge and judgment, not to replace them.

7 Comments

Maybe that's a sign that that hash wants to be an actual object. :)

I believe that your "in theory" should actually be "in practice" since that's exactly what should happen. In fact, you should get missing subroutine coverage and statement coverage for those anonymous subs you don't call. If this isn't happening please let me know.

However, this doesn't diminish any of your main points.

I tend to think of this particular problem as "Data Coverage" and use the example of:

my @squares = (0, 1, 5, 9);

I'm sure there are ways to gather that sort of coverage, and I wish I had the time to implement them.

When I hear people saying that they have got 100% coverage as if that means they're done, a little voice inside my head says "hold on a moment while I implement a new criterion".

I tested before I posted, but didn't see any coverage in the report for those anonymous functions. I'm using the latest D::C with Perl 5.14.1.

I know I've seen anonymous functions covered before, but those were closures stuffed into the symbol table at BEGIN time.

Sorry, i didn't realize before what you were talking about. We've actually been doing some work recently to fix subtle coverage gaps with regards to closures. Can you please check out the branch dzil from the github repo ( https://github.com/pjcj/Devel--Cover/tree/dzil ) and run your code through that?

If it still doesn't show coverage for those lines, please either publish enough code so i can cut it down to a minimal test case, or otherwise, please do cut it down yourself. :)

I'm not sure anyone is maintaining Dist::Zilla::App::Command::cover now that Marcel has quit... just an FYI, though the Test modules were all pretty well taken over.

Mithaldu has done some work recently in this area, but I just ran a test on a brand new 5.14.1 with just Devel::Cover 0.79 installed and got coverage on this construct, so I don't think it's a fix that hasn't been released.

The test file and the output can be seen at https://gist.github.com/1338915

And, as Mithaldu mentioned, if you have found a bug we'd love to have a test case.

That reminded me of something, there was an unresolved issue still where intermediate libraries excluded by D::C caused coverage gaps. For example, in order to test one of my Web::Simple projects i have to run:

perl -MDevel::Cover=-select,Plack/Request.pm,Sub/Quote.pm t/test.t

Modern Perl: The Book

cover image for Modern Perl: the book

The best Perl Programmers read Modern Perl: The Book.

affiliated with ModernPerl.net

Categories

Pages

About this Entry

This page contains a single entry by chromatic published on November 3, 2011 10:48 AM.

What's Difficult in Teaching Programming was the previous entry in this blog.

On Technical Friction is the next entry in this blog.

Find recent content on the main index or look in the archives to find all content.


Sponsored by Blender Recipe Reviews and the Trendshare how to invest guide

Powered by the Perl programming language

what is programming?