API Sanity Autotest
From ISP_RAS
API Sanity Autotest is a unit test generator for shared C/C++ libraries in Linux. It helps to quickly generate simple ("sanity" or "shallow"-quality) tests for all functions from the library API using their signatures and data type definitions straight from the library header files. The quality of generated tests allows to check absence of critical errors in simple use cases and can be improved by involving of highly reusable specialized types for the library. API Sanity Autotest can execute generated tests and detect all kinds of emitted signals, early program exits, program hanging and specified requirement failures. API Sanity Autotest can be considered as a tool for low-cost sanity checking of library API or as a powerful test development framework. Also it supports universal Template2Code format of tests, random test generation mode and other useful features. This tool is free software: you can redistribute it and/or modify it under the terms of the GNU GPL.
Contents |
Downloads
The latest release of API Sanity Autotest can be downloaded from this page.
System requirements
API Sanity Autotest requires gcc (>=3.0.0), binutils and perl.
Examples of generated tests
| Library | Version | Number of Tests | Problems Found |
|---|---|---|---|
| freetype2 | 2.3.11 | 175 | 9 |
| libxml2 | 2.7.6 | 1633 | 63 |
| libssh | 0.4.0 | 224 | 78 |
| libX11 | 1.3.2 | 777 | 167 |
| libQtCore | 4.6.0 | 2936 | 17 |
| ALSA | 1.0.22 | 717 | 27 |
Usage
For generating, building and running tests you must provide the so-called descriptor for your library version. It is a simple file with an XML-like structure that specifies version number, paths to header files and shared objects and optionally some other information. Generating, building and running stages are described below.
Generating tests
Command for generating the test suite:
./api-sanity-autotest.pl -l <library_name> -d <descriptor_path> -gen
Viewing tests
You can view generated tests using the index file:
tests/<library_name>/<version>/view_tests.html
or manually in the directory:
tests/<library_name>/<version>/
Building tests
Command for building tests:
./api-sanity-autotest.pl -l <library_name> -d <descriptor_path> -build
Running tests
Command for running tests:
./api-sanity-autotest.pl -l <library_name> -d <descriptor_path> -run
After a time will be generated test report:
test_results/<library_name>/<version>/test_results.html
Improving tests
To improve generated tests quality you can provide the collection of specialized types for the library.
Options
API Sanity Autotest has many useful options for manipulating with the test generation and execution processes. See the list of all options on this page.
Bugs
Bug reports, feature requests and questions please send to api-sanity-autotest@linuxtesting.org
Future plans
- Support of Splint and ACSL annotations.
- Distinguishing pointer arguments (*x) and arrays (x[]).
- More appropriate test data generation.
- Crash test generation (on the bound values).
- Improving tests quality.
- Continuing testing on the different libraries in Linux.
Author
Andrey Ponomarenko, Institute for System Programming of Russian Academy of Sciences (ISP RAS)