=pod

=head0 Testing and Debugging

Z<CHP-7>

Programming languages and the virtual machines on which they run are just
one small part of the total programming ecosystem. Programmers require not
just development tools, but also maintenance and management tools. Critical
to development and maintenance of code are comprehensive testing tools and
code debugging tools. Luckily, Parrot supports both. This chapter is going
to discuss the testing frameworks available for HLLs written on Parrot,
and the Parrot debugger which can be used for live debugging of software
written in any of the HLLs that Parrot supports.

=head1 Writing Tests

Z<CHP-7-SECT-1>

X<testing Parrot>
Parrot is volunteer-driven, and contributions from new users are always
welcome. Contributing tests is a good place for a new developer to start.
You don't have to understand the code behind a PASM opcodeN<Or PIR
instruction, or whatever.> to test it, you only have to understand
the desired behavior. If you're working on some code and it doesn't do
what the documentation claims, you can isolate the problem in a test or
series of tests and send them to the bug tracking system. There's a
good chance the problem will be fixed before the next release. Writing
tests makes it a lot easier for the developer to know when they've solved
your problem--it's solved when your tests pass. It also prevents that
problem from appearing again, because it's checked every time anyone
runs C<make> C<test>. As you move along, you'll want to write tests
for every bug you fix or new feature you add.

X<Test::Builder>
X<Parrot::Test module>
The Perl 5 testing framework is at the core of Parrot tests,
particularly F<Test::Builder>. Parrot's F<Parrot::Test> module is an
interface to F<Test::Builder> and implements the extra features needed
for testing Parrot, like the fact that PASM and PIR code has to be
compiled to bytecode before it runs. F<Parrot::Test> handles the
compilation and running of the test code, and collects the output for
verification.

The main Parrot tests are in the top-level F<t/> directory of the
Parrot source tree. F<t/op> contains tests for basic opcodes and
F<t/pmc> has tests for PMCs. The names of the test files indicate the
functionality tested, like F<integer.t>, F<number.t>, and F<string.t>.
Part of the F<make test> target is the command F<perl t/harness>,
which runs all the F<.t> files in the subdirectories under F</t>. You
can run individual test files by passing their names to the F<harness>
script:

  $ perl t/harness t/op/string.t t/op/integer.t

X<set opcode (PASM);test example>
Here's a simple example that tests the C<set> opcode with integer
registers, taken from F<t/op/integer.t>:

  output_is(E<lt>E<lt>CODE, E<lt>E<lt>OUTPUT, "set_i");
      set     I0, 42
      set     I1, I0
      print   I1
      print   "\\n"
      end
  CODE
  42
  OUTPUT

The code here sets integer register C<I0> to the value 42, sets C<I1>
to the value of C<I0>, and then prints the value in C<I1>. The test
passes if the value printed was 42, and fails otherwise.

The C<output_is> subroutine takes three strings: the code to run, the
expected output, and a description of the test. The first two strings
can be quite long, so the convention is to use Perl 5 here-documents.
If you look into the code section, you'll see that the literal C<\n>
has to be escaped as C<\\n>. Many tests use the non-interpolating
(C<E<lt>E<lt>'CODE>') form of here-document to avoid that problem. The
description can be any text.  In this case, it's the fully qualified
name of the C<set> opcode for integer registers, but it could have
been "set a native integer register."

If you look up at the top of F<integer.t>, you'll see the line:

  use Parrot::Test tests => 38;

(although the actual number may be larger if more tests have been added
since this book went to press).

The C<use> line for the F<Parrot::Test> module imports a set of
subroutines into the test file, including C<output_is>. The end of the
line gives the number of tests contained in the file.

The C<output_is> subroutine looks for an exact match between the
expected result and the actual output of the code. When the test
result can't be compared exactly, you want C<output_like> instead. It
takes a Perl 5 regular expression for the expected output:

  output_like(<<'CODE', <<'OUTPUT', "testing for text match");
  ...
  CODE
  /^Output is some \d+ number\n$/
  OUTPUT

F<Parrot::Test> also exports C<output_isnt>, which tests that the
actual output of the code I<doesn't> match a particular value.

There are a few guidelines to follow when you're writing a test for a
new opcode or checking that an existing opcode has full test coverage.
Tests should cover the opcode's standard operation, corner cases, and
illegal input. The first tests should always cover the basic
functionality of an opcode. Further tests can cover more complex
operations and the interactions between opcodes. If the test program
is complex or obscure, it helps to add comments. Tests should be
self-contained to make it easy to identify where and why a test is
failing.

=head1 The Parrot Debugger

Z<CHP-7-SECT-2>

=cut

# Local variables:
#   c-file-style: "parrot"
# End:
# vim: expandtab shiftwidth=4: