API Sanity Autotest
From ISP_RAS
API Sanity Autotest is an automatic generator of basic unit tests for shared (dynamic) C/C++ libraries. It helps to quickly generate simple ("sanity" or "shallow"-quality) tests for all functions using their signatures and data type definitions straight from the library header files ("Header-Driven Generation"). The quality of generated tests allows to check absence of critical errors in simple use cases and can be improved by involving of highly reusable specialized types for the library. API Sanity Autotest can execute generated tests and detect all kinds of emitted signals, early program exits, program hanging and specified requirement failures. API Sanity Autotest can be considered as a tool for out-of-box low-cost sanity checking of library API or as a test development framework for initial generation of templates for advanced tests. Also it supports universal Template2Code format of tests, random test generation mode and other useful features. This tool is free software: you can redistribute it and/or modify it under the terms of the dual license: GNU GPL or LGPL.
Contents |
Downloads
The latest release can be downloaded from this page.
System Requirements
The tool requires GCC (>=3.0.0), binutils (c++filt, readelf, objdump) and Perl (base).
Supported Platforms
GNU/Linux, FreeBSD, Haiku (BeOS)
Examples of Generated Tests
| Library | Version | Number of Tests | Problems Found |
|---|---|---|---|
| freetype2 | 2.3.11 | 178 | 13 |
| libQtCore | 4.6.0 | 2936 | 17 |
| ALSA | 1.0.22 | 717 | 27 |
| libxml2 | 2.7.6 | 1633 | 56 |
| libssh | 0.4.0 | 224 | 83 |
| libX11 | 1.3.4 | 778 | 286 |
See test results for more than 160 libraries in the Upstream Tracker system.
Usage
For generating, building and running tests you should provide the descriptor for your library version. It is a simple file with an XML-like structure that specifies version number, paths to header files and shared objects and optionally some other information. Generating, building and running stages are described below. An example of the descriptor is the following (version.xml):
<version>
0.3.4
</version>
<headers>
/usr/local/libssh/0.3.4/include/
</headers>
<libs>
/usr/local/libssh/0.3.4/lib/
</libs>
Generating Tests
Command for generating the test suite:
perl api-sanity-autotest.pl -l <library_name> -d <version.xml> -gen
Viewing Tests
You can view generated tests using the index file:
tests/<library_name>/<version>/view_tests.html
or manually in the directory:
tests/<library_name>/<version>/groups/
Building Tests
Command for building tests:
perl api-sanity-autotest.pl -l <library_name> -d <version.xml> -build
Running Tests
Command for running tests:
perl api-sanity-autotest.pl -l <library_name> -d <version.xml> -run
After a time will be generated test report:
test_results/<library_name>/<version>/test_results.html
Improving Tests
To improve generated tests quality, you can provide the collection of specialized types for the library.
Examples
Generate, build and execute the tests for the libssh library:
perl api-sanity-autotest.pl -l libssh -d 0.3.4.xml -gen -build -run
Tests will be generated to:
tests/libssh/0.3.4/groups/
tests/libssh/0.3.4/view_tests.html
The report will be placed to:
test_results/libssh/0.3.4/test_results.html
Generate tests in the Template2Code (T2C) format:
perl api-sanity-autotest.pl -l libssh -d 0.3.4.xml -gen -t2c
The package with T2C tests will be generated to:
tests_t2c/libssh/0.3.4/t2c-libssh-tests-0.3.4.tar.gz
Execute the tests using the Xvfb server:
perl api-sanity-autotest.pl -l allegro -d 4.9.21.xml -run -xvfb
Generate tests using the Splint specifications (annotations) in the headers:
perl api-sanity-autotest.pl -l libssh -d 0.3.4.xml -gen -splint-specs
Options
The tool has many useful options for manipulating with the test generation and execution processes. See the list of all options on this page.
Bugs
Please send bug reports, feature requests and questions to api-sanity-autotest@linuxtesting.org
Future Plans
- Extending support of Splint and ACSL annotations.
- More appropriate distinguishing of pointer arguments (*x) and arrays (x[]).
- New kinds of test data files (images, fonts, ...).
- Crash test generation on the bound values for parameters.
- Improving tests quality.
- Continuing testing on the different libraries.
Maintainers
The tool was developed by the Russian Linux Verification Center at ISPRAS. Andrey Ponomarenko is the leader of this project.