procyberian/gcc
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

[Ada] Remove gnattest section from gnat user guide

Browse source 
gcc/ada/

	* doc/gnat_ugn/gnat_utility_programs.rst: Remove gnattest
	section.
This commit is contained in:
Fedor Rybin 2022-01-11 12:04:37 +03:00 • committed by Pierre-Marie de Rodat
parent 9a9d06f711
commit 0e9e014cf7
1 changed files with 0 additions and 917 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

917
gcc/ada/doc/gnat_ugn/gnat_utility_programs.rst
View file

@ -18,7 +18,6 @@ This chapter describes a number of utility programs:
* :ref:`The_GNAT_Metrics_Tool_gnatmetric`
* :ref:`The_GNAT_Pretty_Printer_gnatpp`
* :ref:`The_Body_Stub_Generator_gnatstub`
* :ref:`The_Unit_Test_Generator_gnattest`
* :ref:`The_Backtrace_Symbolizer_gnatsymbolize`
It also describes how several of these tools can be used in conjunction
@ -3264,922 +3263,6 @@ building specialized scripts.
:switch:`--files={filename}`
.. only:: PRO or GPL
.. _The_Unit_Test_Generator_gnattest:
The Unit Test Generator ``gnattest``
====================================
.. index:: ! gnattest
``gnattest`` is an ASIS-based utility that creates unit-test skeletons
as well as a test driver infrastructure (harness). ``gnattest`` creates
a skeleton for each visible subprogram in the packages under consideration when
they do not exist already.
``gnattest`` is a project-aware tool.
(See :ref:`Using_Project_Files_with_GNAT_Tools` for a description of
the project-related switches but note that ``gnattest`` does not support
the :switch:`-U`, :switch:`-eL`, :switch:`--subdirs={dir}`, or
:switch:`--no-objects-dir` switches.)
The project file package that can specify
``gnattest`` switches is named ``gnattest``.
The user can choose to generate a single test driver
that will run all individual tests, or separate test drivers for each test. The
second option allows much greater flexibility in test execution environment,
allows to benefit from parallel tests execution to increase performance, and
provides stubbing support.
``gnattest`` also has a mode of operation where it acts as the test
aggregator when multiple test executables must be run, in particular when
the separate test drivers were generated. In this mode it handles individual
tests execution and upon completion reports the summary results of the test
run.
In order to process source files from a project, ``gnattest`` has to
semantically analyze the sources. Therefore, test skeletons can only be
generated for legal Ada units. If a unit is dependent on other units,
those units should be among the source files of the project or of other projects
imported by this one.
Generated skeletons and harnesses are based on the AUnit testing framework.
AUnit is an Ada adaptation of the xxxUnit testing frameworks, similar to JUnit
for Java or CppUnit for C++. While it is advised that gnattest users read
the AUnit manual, deep knowledge of AUnit is not necessary for using ``gnattest``.
For correct operation of ``gnattest``, AUnit should be installed and
aunit.gpr must be on the project path. Except for some special circumstances
(e.g. a custom run-time is used), this should normally be the case out of the box.
.. _Running_gnattest:
Running ``gnattest``
--------------------
There are two ways of running ``gnattest``.
.. _Framework_Generation_Mode:
Framework Generation Mode
^^^^^^^^^^^^^^^^^^^^^^^^^
In this mode ``gnattest`` has the following command-line interface:
::
$ gnattest -Pprojname [ switches ] [ filename ] [ -cargs gcc_switches ]
where
* :switch:`-P{projname}`
specifies the project defining the location of source files. When no
file names are provided on the command line, all sources in the project
are used as input. This switch is required.
* :switch:`{filename}`
is the name of the source file containing the library unit package *declaration*
(the package "spec") for which a test package will be created. The file name
may be given with a path.
* :samp:`{switches}`
is an optional sequence of switches as described below.
* :samp:`{gcc_switches}`
is a list of additional switches for
``gcc`` that will be passed to all compiler invocations
made by ``gnattest`` to generate a set of ASIS trees.
``gnattest`` results can be found in two different places.
* *automatic harness*:
This is the harness code, which is located by default in
"gnattest/harness" directory created in the object directory of
the main project file. All of this code is generated completely
automatically and can be destroyed and regenerated at will, with the
exception of the file *gnattest_common.gpr*, which is created if absent,
but never overwritten. It is not recommended to modify other files
manually, since these modifications will be lost if ``gnattest`` is re-run.
The entry point in the harness code is
the project file named *test_driver.gpr*. Tests can be compiled and run
using a command such as:
::
$ gprbuild -P<harness-dir>/test_driver
Note that if you need to adjust any options used to compile the harness,
you can do so by editing the file *gnattest_common.gpr*.
* *actual unit test skeletons*:
A test skeleton for each visible subprogram is created in a separate file, if it
doesn't exist already. By default, those separate test files are located in a
"gnattest/tests" directory that is created in the object directory of
corresponding project file. For example, if a source file my_unit.ads in
directory src contains a visible subprogram Proc, then the corresponding unit
test will be found in file src/tests/my_unit-test_data-tests.adb and will be
called Test_Proc_<code>. <code> is a signature encoding used to differentiate
test names in case of overloading.
Note that if the project already has both my_unit.ads and my_unit-test_data.ads,
this will cause a name conflict with the generated test package.
.. _Test_Execution_Mode:
Test Execution Mode
^^^^^^^^^^^^^^^^^^^
In this mode ``gnattest`` has a the following command-line interface:
::
$ gnattest test_drivers.list [ switches ]
where
* :samp:`{test_drivers.list}`
is the name of the text file containing the list of executables to treat as
test drivers. This file is automatically generated by gnattest, but can be
hand-edited to add or remove tests. This switch is required.
* :samp:`{switches}`
is an optional sequence of switches as described below.
.. _Switches_for_gnattest_in_framework_generation_mode:
Switches for ``gnattest`` in framework generation mode
------------------------------------------------------
.. index:: --strict (gnattest)
:switch:`--strict`
Return error exit code if there are any compilation errors.
.. index:: -q (gnattest)
:switch:`-q`
Quiet mode: suppresses noncritical output messages.
.. index:: -v (gnattest)
:switch:`-v`
Verbose mode: produces additional output about the execution of the tool.
When specified alone on the command line, prints tool version and exits.
.. index:: -r (gnattest)
:switch:`-r`
Recursively considers all sources from all projects.
.. index:: -files (gnattest)
:switch:`-files={filename}`
Take as arguments the files listed in text file ``file``.
Text file ``file`` may contain empty lines that are ignored.
Each nonempty line should contain the name of an existing file.
Several such switches may be specified simultaneously.
.. index:: --ignore (gnattest)
:switch:`--ignore={filename}`
Do not process the sources listed in a specified file.
.. index:: --RTS (gnattest)
:switch:`--RTS={rts-path}`
Specifies the default location of the runtime library. Same meaning as the
equivalent ``gnatmake`` flag (:ref:`Switches_for_gnatmake`). For restricted
profiles, ``gnattest`` takes into account the run-time limitations when
generating the harness.
.. index:: --additional-tests (gnattest)
:switch:`--additional-tests={projname}`
Sources described in ``projname`` are considered potential additional
manual tests to be added to the test suite.
.. index:: --harness-only (gnattest)
:switch:`--harness-only`
When this option is given, ``gnattest`` creates a harness for all
sources, treating them as test packages. This option is not compatible with
closure computation done by -U main.
.. index:: --separate-drivers (gnattest)
:switch:`--separate-drivers[={val}]`
Generates a separate test driver for each test or unit under test, rather
than a single executable incorporating all tests. ``val`` can be "unit" or
"test", or may be omitted, which defaults to "unit".
.. index:: --stub (gnattest)
:switch:`--stub`
Generates the testing framework that uses subsystem stubbing to isolate the
code under test.
.. index:: --harness-dir (gnattest)
:switch:`--harness-dir={dirname}`
Specifies the directory that will hold the harness packages and project file
for the test driver. If the ``dirname`` is a relative path, it is considered
relative to the object directory of the project file.
.. index:: --tests-dir (gnattest)
:switch:`--tests-dir={dirname}`
All test packages are placed in the ``dirname`` directory.
If the ``dirname`` is a relative path, it is considered relative to the object
directory of the project file. When all sources from all projects are taken
recursively from all projects, ``dirname`` directories are created for each
project in their object directories and test packages are placed accordingly.
.. index:: --subdir (gnattest)
:switch:`--subdir={dirname}`
Test packages are placed in a subdirectory of the corresponding source
directory, with the name ``dirname``. Thus, each set of unit tests is located
in a subdirectory of the code under test. If the sources are in separate
directories, each source directory has a test subdirectory named ``dirname``.
.. index:: --tests-root (gnattest)
:switch:`--tests-root={dirname}`
The hierarchy of source directories, if any, is recreated in the ``dirname``
directory, with test packages placed in directories corresponding to those
of the sources.
If the ``dirname`` is a relative path, it is considered relative to the object
directory of the project file. When projects are considered recursively,
directory hierarchies of tested sources are
recreated for each project in their object directories and test packages are
placed accordingly.
.. index:: --stubs-dir (gnattest)
:switch:`--stubs-dir={dirname}`
The hierarchy of directories containing stubbed units is recreated in
the ``dirname`` directory, with stubs placed in directories corresponding to
projects they are derived from.
If the ``dirname`` is a relative path, it is considered relative to the object
directory of the project file. When projects are considered recursively,
directory hierarchies of stubs are
recreated for each project in their object directories and test packages are
placed accordingly.
.. index:: --exclude-from-stubbing (gnattest)
:switch:`--exclude-from-stubbing={filename}`
Disables stubbing of units listed in ``filename``. The file should contain
corresponding spec files, one per line.
:switch:`--exclude-from-stubbing:{unit}={filename}`
Same as above, but corresponding units will not be stubbed only when testing
specified ``unit``.
.. index:: --validate-type-extensions (gnattest)
:switch:`--validate-type-extensions`
Enables substitution check: run all tests from all parents in order
to check substitutability in accordance with the Liskov substitution principle (LSP).
.. index:: --inheritance-check (gnattest)
:switch:`--inheritance-check`
Enables inheritance check: run inherited tests against descendants.
.. index:: --no-inheritance-check (gnattest)
:switch:`--no-inheritance-check`
Disables inheritance check.
.. index:: --no-inheritance-check (gnattest)
:switch:`--test-case-only`
Generates test skeletons only for subprograms that have at least one
associated pragma or aspect Test_Case.
.. index:: --skeleton-default (gnattest)
:switch:`--skeleton-default={val}`
Specifies the default behavior of generated skeletons. ``val`` can be either
"fail" or "pass", "fail" being the default.
.. index:: --passed-tests (gnattest)
:switch:`--passed-tests={val}`
Specifies whether or not passed tests should be shown. ``val`` can be either
"show" or "hide", "show" being the default.
.. index:: --exit-status (gnattest)
:switch:`--exit-status={val}`
Specifies whether or not generated test driver should return failure exit
status if at least one test fails or crashes. ``val`` can be either
"on" or "off", "off" being the default.
.. index:: --omit-sloc (gnattest)
:switch:`--omit-sloc`
Suppresses comment line containing file name and line number of corresponding
subprograms in test skeletons.
.. index:: --no-command-line (gnattest)
:switch:`--no-command-line`
Don't add command line support to test driver. Note that regardless of this
switch, ``gnattest`` will automatically refrain from adding command
line support if it detects that the selected run-time doesn't provide
this capability.
.. index:: --separates (gnattest)
:switch:`--separates`
Bodies of all test routines are generated as separates. Note that this mode is
kept for compatibility reasons only and it is not advised to use it due to
possible problems with hash in names of test skeletons when using an
inconsistent casing. Separate test skeletons can be incorporated to monolith
test package with improved hash being used by using ``--transition``
switch.
.. index:: --transition (gnattest)
:switch:`--transition`
This allows transition from separate test routines to monolith test packages.
All matching test routines are overwritten with contents of corresponding
separates. Note that if separate test routines had any manually added with
clauses they will be moved to the test package body as is and have to be moved
by hand.
.. index:: --test-duration (gnattest)
:switch:`--test-duration`
Adds time measurements for each test in generated test driver.
:switch:`--tests_root`, :switch:`--subdir` and :switch:`--tests-dir` switches are mutually exclusive.
.. _Switches_for_gnattest_in_test_execution_mode:
Switches for ``gnattest`` in test execution mode
------------------------------------------------
.. index:: --passed-tests (gnattest)
:switch:`--passed-tests={val}`
Specifies whether or not passed tests should be shown. ``val`` can be either
"show" or "hide", "show" being the default.
.. index:: --queues (gnattest)
.. index:: -j (gnattest)
:switch:`--queues={n}`, :switch:`-j{n}`
Runs ``n`` tests in parallel (default is 1).
.. index:: --copy-environment (gnattest)
:switch:`--copy-environment={dir}`
Contents of ``dir`` directory will be copied to temporary directories
created by gnattest in which individual test drivers are spawned.
.. _Project_Attributes_for_gnattest:
Project Attributes for ``gnattest``
-----------------------------------
Most of the command-line options can also be passed to the tool by adding
special attributes to the project file. Those attributes should be put in
package ``Gnattest``. Here is the list of attributes:
* ``Tests_Root``
is used to select the same output mode as with the ``--tests-root`` option.
This attribute cannot be used together with ``Subdir`` or ``Tests_Dir``.
* ``Subdir``
is used to select the same output mode as with the ``--subdir`` option.
This attribute cannot be used together with ``Tests_Root`` or ``Tests_Dir``.
* ``Tests_Dir``
is used to select the same output mode as with the ``--tests-dir`` option.
This attribute cannot be used together with ``Subdir`` or ``Tests_Root``.
* ``Stubs_Dir``
is used to select the same output mode as with the ``--stubs-dir`` option.
* ``Harness_Dir``
is used to specify the directory in which to place harness packages and project
file for the test driver, otherwise specified by ``--harness-dir``.
* ``Additional_Tests``
is used to specify the project file, otherwise given by
``--additional-tests`` switch.
* ``Skeletons_Default``
is used to specify the default behaviour of test skeletons, otherwise
specified by ``--skeleton-default`` option. The value of this attribute
should be either ``pass`` or ``fail``.
* ``Default_Stub_Exclusion_List``
is used to specify the file with list of units whose bodies should not
be stubbed, otherwise specified by ``--exclude-from-stubbing=filename``.
* ``Stub_Exclusion_List ("unit")``
is used to specify the file with list of units whose bodies should not
be stubbed when testing "unit", otherwise specified by
``--exclude-from-stubbing:unit=filename``.
Each of those attributes can be overridden from the command line if needed.
Other ``gnattest`` switches can also be passed via the project
file as an attribute list called ``Gnattest_Switches``.
.. _Simple_gnattest_Example:
Simple Example
--------------
Let's take a very simple example using the first ``gnattest`` example
located in:
::
<install_prefix>/share/examples/gnattest/simple
This project contains a simple package containing one subprogram. By running ``gnattest``:
::
$ gnattest --harness-dir=driver -Psimple.gpr
a test driver is created in directory ``driver``. It can be compiled and run:
::
$ cd obj/driver
$ gprbuild -Ptest_driver
$ test_runner
One failed test with the diagnosis "test not implemented" is reported.
Since no special output option was specified, the test package ``Simple.Tests``
is located in:
::
<install_prefix>/share/examples/gnattest/simple/obj/gnattest/tests
For each package containing visible subprograms, a child test package is
generated. It contains one test routine per tested subprogram. Each
declaration of a test subprogram has a comment specifying which tested
subprogram it corresponds to. Bodies of test routines are placed in test package
bodies and are surrounded by special comment sections. Those comment sections
should not be removed or modified in order for gnattest to be able to regenerate
test packages and keep already written tests in place.
The test routine ``Test_Inc_5eaee3`` located at :file:`simple-test_data-tests.adb` contains
a single statement: a call to procedure ``Assert``. It has two arguments:
the Boolean expression we want to check and the diagnosis message to display if
the condition is false.
That is where actual testing code should be written after a proper setup.
An actual check can be performed by replacing the ``Assert`` call with:
::
Assert (Inc (1) = 2, "wrong incrementation");
After recompiling and running the test driver, one successfully passed test
is reported.
.. _Setting_Up_and_Tearing_Down_the_Testing_Environment:
Setting Up and Tearing Down the Testing Environment
---------------------------------------------------
Besides test routines themselves, each test package has a parent package
``Test_Data`` that has two procedures: ``Set_Up`` and ``Tear_Down``. This package is never
overwritten by the tool. ``Set_Up`` is called before each test routine of the
package, and ``Tear_Down`` is called after each test routine. Those two procedures
can be used to perform necessary initialization and finalization,
memory allocation, etc. Test type declared in ``Test_Data`` package is parent type
for the test type of test package and can have user-defined components whose
values can be set by ``Set_Up`` routine and used in test routines afterwards.
.. _Regenerating_Tests:
Regenerating Tests
------------------
Bodies of test routines and ``Test_Data`` packages are never overridden after they
have been created once. As long as the name of the subprogram, full expanded Ada
names and order of its parameters are the same, and comment sections are
intact, the old test routine will fit in its place and no test skeleton will be
generated for the subprogram.
This can be demonstrated with the previous example. By uncommenting declaration
and body of function Dec in ``simple.ads`` and ``simple.adb``, running
``gnattest`` on the project, and then running the test driver:
::
$ gnattest --harness-dir=driver -Psimple.gpr
$ cd obj/driver
$ gprbuild -Ptest_driver
$ test_runner
The old test is not replaced with a stub, nor is it lost, but a new test
skeleton is created for function ``Dec``.
The only way of regenerating tests skeletons is to remove the previously created
tests together with corresponding comment sections.
.. _Default_Test_Behavior:
Default Test Behavior
---------------------
The generated test driver can treat unimplemented tests in two ways:
either count them all as failed (this is useful to see which tests are still
left to implement) or as passed (to sort out unimplemented ones from those
actually failing).
The test driver accepts a switch to specify this behavior:
:switch:`--skeleton-default={val}`, where ``val`` is either ``pass`` or ``fail`` (exactly as for
``gnattest``).
The default behavior of the test driver is set with the same switch
as passed to ``gnattest`` when generating the test driver.
Passing it to the driver generated on the first example:
::
$ test_runner --skeleton-default=pass
makes both tests pass, even the unimplemented one.
.. _Testing_Primitive_Operations_of_Tagged_Types:
Testing Primitive Operations of Tagged Types
--------------------------------------------
Creation of test skeletons for primitive operations of tagged types entails
a number of features. Test routines for all primitives of a given tagged type
are placed in a separate child package named according to the tagged type. For
example, if you have tagged type ``T`` in package ``P``, all tests for primitives
of ``T`` will be in ``P.T_Test_Data.T_Tests``.
Consider running ``gnattest`` on the second example (note: actual tests for this
example already exist, so there's no need to worry if the tool reports that
no new stubs were generated):
::
$ cd <install_prefix>/share/examples/gnattest/tagged_rec
$ gnattest --harness-dir=driver -Ptagged_rec.gpr
Taking a closer look at the test type declared in the test package
*Speed1.Controller_Test_Data* is necessary. It is declared in:
::
<install_prefix>/share/examples/gnattest/tagged_rec/obj/gnattest/tests
Test types are direct or indirect descendants of
*AUnit.Test_Fixtures.Test_Fixture* type. In the case of non-primitive tested
subprograms, the user doesn't need to be concerned with them. However,
when generating test packages for primitive operations, there are some things
the user needs to know.
Type ``Test_Controller`` has components that allow assignment of various
derivations of type ``Controller``. And if you look at the specification of
package *Speed2.Auto_Controller*, you will see that ``Test_Auto_Controller``
actually derives from ``Test_Controller`` rather than AUnit type ``Test_Fixture``.
Thus, test types mirror the hierarchy of tested types.
The ``Set_Up`` procedure of ``Test_Data`` package corresponding to a test package
of primitive operations of type ``T`` assigns to ``Fixture`` a reference to an
object of that exact type ``T``. Note, however, that if the tagged type has
discriminants, the ``Set_Up`` only has a commented template for setting
up the fixture, since filling the discriminant with actual value is up
to the user.
The knowledge of the structure of test types allows additional testing
without additional effort. Those possibilities are described below.
.. _Testing_Inheritance:
Testing Inheritance
-------------------
Since the test type hierarchy mimics the hierarchy of tested types, the
inheritance of tests takes place. An example of such inheritance can be
seen by running the test driver generated for the second example. As previously
mentioned, actual tests are already written for this example.
::
$ cd obj/driver
$ gprbuild -Ptest_driver
$ test_runner
There are 6 passed tests while there are only 5 testable subprograms. The test
routine for function Speed has been inherited and run against objects of the
derived type.
.. _Tagged_Type_Substitutability_Testing:
Tagged Type Substitutability Testing
------------------------------------
*Tagged Type Substitutability Testing* is a way of verifying the global type
consistency by testing. Global type consistency is a principle stating that if
``S`` is a subtype of ``T`` (in Ada, ``S`` is a derived type of tagged type ``T``),
then objects of type ``T`` may be replaced with objects of type ``S`` (that is,
objects of type ``S`` may be substituted for objects of type ``T``), without
altering any of the desirable properties of the program. When the properties
of the program are expressed in the form of subprogram preconditions and
postconditions (let's call them pre and post), the principle is formulated as
relations between the pre and post of primitive operations and the pre and post
of their derived operations. The pre of a derived operation should not be
stronger than the original pre, and the post of the derived operation should
not be weaker than the original post. Those relations ensure that verifying if
a dispatching call is safe can be done just by using the pre and post of the
root operation.
Verifying global type consistency by testing consists of running all the unit
tests associated with the primitives of a given tagged type with objects of its
derived types.
In the example used in the previous section, there was clearly a violation of
type consistency. The overriding primitive ``Adjust_Speed`` in package ``Speed2``
removes the functionality of the overridden primitive and thus doesn't respect
the consistency principle.
``gnattest`` has a special option to run overridden parent tests against objects
of the type which have overriding primitives:
::
$ gnattest --harness-dir=driver --validate-type-extensions -Ptagged_rec.gpr
$ cd obj/driver
$ gprbuild -Ptest_driver
$ test_runner
While all the tests pass by themselves, the parent test for ``Adjust_Speed`` fails
against objects of the derived type.
Non-overridden tests are already inherited for derived test types, so the
``--validate-type-extensions`` enables the application of overridden tests
to objects of derived types.
.. _Testing_with_Contracts:
Testing with Contracts
----------------------
``gnattest`` supports pragmas ``Pre``, ``Post``, and ``Test_Case``,
as well as the corresponding Ada 2012 aspects.
Test routines are generated, one per each ``Test_Case`` associated with a tested
subprogram. Those test routines have special wrappers for tested functions
that have composition of pre- and postcondition of the subprogram with
"requires" and "ensures" of the ``Test_Case`` (depending on the mode, pre and post
either count for ``Nominal`` mode or do *not* count for ``Robustness`` mode).
The third example demonstrates how this works:
::
$ cd <install_prefix>/share/examples/gnattest/contracts
$ gnattest --harness-dir=driver -Pcontracts.gpr
Putting actual checks within the range of the contract does not cause any
error reports. For example, for the test routine which corresponds to
test case 1:
::
Assert (Sqrt (9.0) = 3.0, "wrong sqrt");
and for the test routine corresponding to test case 2:
::
Assert (Sqrt (-5.0) = -1.0, "wrong error indication");
are acceptable:
::
$ cd obj/driver
$ gprbuild -Ptest_driver
$ test_runner
However, by changing 9.0 to 25.0 and 3.0 to 5.0, for example, you can get
a precondition violation for test case one. Also, by using any otherwise
correct but positive pair of numbers in the second test routine, you can also
get a precondition violation. Postconditions are checked and reported
the same way.
.. _Additional_Tests:
Additional Tests
----------------
``gnattest`` can add user-written tests to the main suite of the test
driver. ``gnattest`` traverses the given packages and searches for test
routines. All procedures with a single in out parameter of a type which is
derived from *AUnit.Test_Fixtures.Test_Fixture* and that are declared in package
specifications are added to the suites and are then executed by the test driver.
(``Set_Up`` and ``Tear_Down`` are filtered out.)
An example illustrates two ways of creating test harnesses for user-written
tests. Directory ``additional_tests`` contains an AUnit-based test driver written
by hand.
::
<install_prefix>/share/examples/gnattest/additional_tests/
To create a test driver for already-written tests, use the ``--harness-only``
option:
::
gnattest -Padditional/harness/harness.gpr --harness-dir=harness_only \\
--harness-only
gprbuild -Pharness_only/test_driver.gpr
harness_only/test_runner
Additional tests can also be executed together with generated tests:
::
gnattest -Psimple.gpr --additional-tests=additional/harness/harness.gpr \\
--harness-dir=mixing
gprbuild -Pmixing/test_driver.gpr
mixing/test_runner
.. _Individual_Test_Drivers:
Individual Test Drivers
-----------------------
By default, ``gnattest`` generates a monolithic test driver that
aggregates the individual tests into a single executable. It is also possible
to generate separate executables for each test or each unit under test, by
passing the switch ``--separate-drivers`` with corresponding parameter. This
approach scales better for large testing campaigns, especially involving target
architectures with limited resources typical for embedded development. It can
also provide a major performance benefit on multi-core systems by allowing
simultaneous execution of multiple tests.
``gnattest`` can take charge of executing the individual tests; for this,
instead of passing a project file, a text file containing the list of
executables can be passed. Such a file is automatically generated by gnattest
under the name :file:`test_drivers.list`, but it can be
hand-edited to add or remove tests, or replaced. The individual tests can
also be executed standalone, or from any user-defined scripted framework.
.. _Stubbing:
Stubbing
--------
Depending on the testing campaign, it is sometimes necessary to isolate the
part of the algorithm under test from its dependencies. This is accomplished
via *stubbing*, i.e. replacing the subprograms that are called from the
subprogram under test by stand-in subprograms that match the profiles of the
original ones, but simply return predetermined values required by the test
scenario.
This mode of test harness generation is activated by the switch ``--stub``.
The implementation approach chosen by ``gnattest`` is as follows.
For each package under consideration all the packages it is directly depending
on are stubbed, excluding the generic packages and package instantiations.
The stubs are shared for each package under test. The specs of packages to stub
remain intact, while their bodies are replaced, and hide the original bodies by
means of extending projects. Also, for each stubbed
package, a child package with setter routines for each subprogram declaration
is created. These setters are meant to be used to set the behavior of
stubbed subprograms from within test cases.
Note that subprograms belonging to the same package as the subprogram under
test are not stubbed. This guarantees that the sources being tested are
exactly the sources used for production, which is an important property for
establishing the traceability between the testing campaign and production code.
Due to the nature of stubbing process, this mode implies the switch
``--separate-drivers``, i.e. an individual test driver (with the
corresponding hierarchy of extending projects) is generated for each unit under
test.
.. note::
Developing a stubs-based testing campaign requires
good understanding of the infrastructure created by ``gnattest`` for
this purpose. We recommend following the two stubbing tutorials
``simple_stubbing`` and ``advanced_stubbing`` provided
under :file:`<install_prefix>/share/examples/gnattest` before
attempting to use this powerful feature.
.. _Gnatcov_Integration:
Integration with GNATcoverage
-----------------------------
In addition to the harness, ``gnattest`` generates a Makefile. This Makefile
provides targets for building the test drivers and also the targets for
computing the coverage information using GNATcoverage framework when this
coverage analysis tool is available. The target ``coverage`` fully automates
the process: it will first build all test drivers, then run them under
GNATcoverage, analyze individual trace files, and finally aggregate them:
::
make coverage
GNATcoverage options, such as coverage criteria and generated report format,
can be adjusted using Makefile variables provided for this purpose.
Note that coverage targets are not generated in the Makefile when
--separate-drivers=test is passed to gnattest.
.. _Putting_Tests_under_Version_Control:
Putting Tests under Version Control
-----------------------------------
As has been stated earlier, ``gnattest`` generates two different types
of code, test skeletons and harness. The harness is generated completely
automatically each time, does not require manual changes and therefore should
not be put under version control.
It makes sense to put under version control files containing test data packages,
both specs and bodies, and files containing bodies of test packages. Note that
test package specs are also generated automatically each time and should not be
put under version control.
Option ``--omit-sloc`` may be useful when putting test packages under version control.
.. _Current_Limitations:
Current Limitations
-------------------
The tool currently has the following limitations:
* generic tests for nested generic packages and their instantiations are
not supported;
* tests for protected subprograms and entries are not supported;
* pragma ``No_Run_Time`` is not supported;
* pragma ``No_Secondary_Stack`` is not supported;
* if pragmas for interfacing with foreign languages are used, manual
adjustments might be necessary to make the test harness compilable;
* use of some constructs, such as elaboration-control pragmas, Type_Invariant
aspects, and complex variable initializations that use Subprogram'Access,
may result in elaboration circularities in the generated harness.
.. only:: PRO or GPL
.. _The_Backtrace_Symbolizer_gnatsymbolize: