TestingFramework

Inkscape Testing Framework

In order to detect and identify bugs quickly and reliably, we should set up a testing framework for Inkscape.

There are several different ways to test a desktop Linux application. Each method provides different kinds of information about the codebase and are each valuable in different ways. A multi-pronged approach that uses more than one kind of testing is expected to gain the best results.

Compile Testing

In compile testing, the codebase is repetitively checked out and compiled in one or more different ways. For example, with different compilers or for different platforms (Linux, Windows, Mac OSX, ...) Errors in configuration, compilation, linking, or basic executation are detected, collected, and reported on a periodic basis (such as daily compile reports).

It may be possible to set up compile testing using SourceForge's compile farm. Another option to investigate is the Mozilla project's "Tinderbox".

Unit Testing

Unit testing is implemented at the file level. Each *.c or *.cpp file is provided with a test file containing calls for each of the functions, and that exercises the API, including a wide variety of error situations such as NULL pointers, buffer overflows, very large or very small floating point numbers, zero or negative int's, uninitialized objects, etc.

The intention of unit tests is to provide sanity checks that the developer can use as they implement the code. Ideally, when implementing a new code file, the programmer could first implement the unit test from a spec, and then work on implementing the code to pass the unit test; this is the development style advocated in the Extreme Programming philosophy.

Regression Testing

In regression testing, the philosophy is to implement test cases for each bug found, and to run these test cases against the codebase to determine that bug fixes actually fix the bug, and to ensure in the future that the bugs do not re-occur.

This can be implemented by assembling a repository of "problem" SVG files that have caused Inkscape to error in programmatically detectable manners (such as crashes or emission of invalid SVG or PNG's).

GUI Testing

To eneable a11y in Inkscape for testing with dogtail, start Inkscape with the following command.

• GTK_MODULES=gail:atk-bridge inkscape

Fault Response Testing

In fault response testing, the application is put into various critical situations and the behavior detected and compared against expectations. Examples include out-of-disk-space, out-of-memory, response to different signals, loading and saving very large files, starting up with invalid or unreadible config files, and so forth.

Specification Compliance Testing

In spec testing, the application is run against various input cases and caused to generate output that can be tested against "known" correct data. For instance, the W3C produces an SVG compliance test suite with SVG files and the expected rendered output as PNG's. Inkscape can be run on each of the SVG's from the commandline and caused to save the rendering as a PNG. An image-difference tool can then be used to calculate a diff-score between the emitted PNG and the W3C image.

Similarly, this process can be used against other applications such as Batik or librsvg, to check inter-application conformance on the same SVG files. Ideally, all SVG applications should render a given SVG file identically.

Performance Testing

In performance testing the application is measured as to how well it works against various criteria such as start-up time, output quality, processing speed, etc.

Valgrind Memory Leak Testing

What I did here is simply ran inkscape through valgrind with the following command:

   valgrind -v --leak-check=yes --leak-resolution=high --num-callers=15 --show-reachable=yes --log-file=inkscape.log src/inkscape

I then drew a few objects, edited some points and exit.

What I would like to see is people investigate the memory problems listed here and as they are resolved, we remove them from the list until we don't have any more. Then we can try to find more! :)