diff --git a/gcc/ada/doc/gnat_ugn/gnat_utility_programs.rst b/gcc/ada/doc/gnat_ugn/gnat_utility_programs.rst index d030cd40839..def11a6b120 100644 --- a/gcc/ada/doc/gnat_ugn/gnat_utility_programs.rst +++ b/gcc/ada/doc/gnat_ugn/gnat_utility_programs.rst @@ -18,7 +18,6 @@ This chapter describes a number of utility programs: * :ref:`The_GNAT_Metrics_Tool_gnatmetric` * :ref:`The_GNAT_Pretty_Printer_gnatpp` * :ref:`The_Body_Stub_Generator_gnatstub` - * :ref:`The_Unit_Test_Generator_gnattest` * :ref:`The_Backtrace_Symbolizer_gnatsymbolize` It also describes how several of these tools can be used in conjunction @@ -3264,922 +3263,6 @@ building specialized scripts. :switch:`--files={filename}` -.. only:: PRO or GPL - - .. _The_Unit_Test_Generator_gnattest: - - The Unit Test Generator ``gnattest`` - ==================================== - - .. index:: ! gnattest - - ``gnattest`` is an ASIS-based utility that creates unit-test skeletons - as well as a test driver infrastructure (harness). ``gnattest`` creates - a skeleton for each visible subprogram in the packages under consideration when - they do not exist already. - - ``gnattest`` is a project-aware tool. - (See :ref:`Using_Project_Files_with_GNAT_Tools` for a description of - the project-related switches but note that ``gnattest`` does not support - the :switch:`-U`, :switch:`-eL`, :switch:`--subdirs={dir}`, or - :switch:`--no-objects-dir` switches.) - The project file package that can specify - ``gnattest`` switches is named ``gnattest``. - - The user can choose to generate a single test driver - that will run all individual tests, or separate test drivers for each test. The - second option allows much greater flexibility in test execution environment, - allows to benefit from parallel tests execution to increase performance, and - provides stubbing support. - - ``gnattest`` also has a mode of operation where it acts as the test - aggregator when multiple test executables must be run, in particular when - the separate test drivers were generated. In this mode it handles individual - tests execution and upon completion reports the summary results of the test - run. - - In order to process source files from a project, ``gnattest`` has to - semantically analyze the sources. Therefore, test skeletons can only be - generated for legal Ada units. If a unit is dependent on other units, - those units should be among the source files of the project or of other projects - imported by this one. - - Generated skeletons and harnesses are based on the AUnit testing framework. - AUnit is an Ada adaptation of the xxxUnit testing frameworks, similar to JUnit - for Java or CppUnit for C++. While it is advised that gnattest users read - the AUnit manual, deep knowledge of AUnit is not necessary for using ``gnattest``. - For correct operation of ``gnattest``, AUnit should be installed and - aunit.gpr must be on the project path. Except for some special circumstances - (e.g. a custom run-time is used), this should normally be the case out of the box. - - - .. _Running_gnattest: - - Running ``gnattest`` - -------------------- - - There are two ways of running ``gnattest``. - - .. _Framework_Generation_Mode: - - Framework Generation Mode - ^^^^^^^^^^^^^^^^^^^^^^^^^ - - In this mode ``gnattest`` has the following command-line interface: - - :: - - $ gnattest -Pprojname [ switches ] [ filename ] [ -cargs gcc_switches ] - - where - - * :switch:`-P{projname}` - specifies the project defining the location of source files. When no - file names are provided on the command line, all sources in the project - are used as input. This switch is required. - - * :switch:`{filename}` - is the name of the source file containing the library unit package *declaration* - (the package "spec") for which a test package will be created. The file name - may be given with a path. - - * :samp:`{switches}` - is an optional sequence of switches as described below. - - * :samp:`{gcc_switches}` - is a list of additional switches for - ``gcc`` that will be passed to all compiler invocations - made by ``gnattest`` to generate a set of ASIS trees. - - - ``gnattest`` results can be found in two different places. - - * *automatic harness*: - This is the harness code, which is located by default in - "gnattest/harness" directory created in the object directory of - the main project file. All of this code is generated completely - automatically and can be destroyed and regenerated at will, with the - exception of the file *gnattest_common.gpr*, which is created if absent, - but never overwritten. It is not recommended to modify other files - manually, since these modifications will be lost if ``gnattest`` is re-run. - The entry point in the harness code is - the project file named *test_driver.gpr*. Tests can be compiled and run - using a command such as: - - :: - - $ gprbuild -P/test_driver - - Note that if you need to adjust any options used to compile the harness, - you can do so by editing the file *gnattest_common.gpr*. - - * *actual unit test skeletons*: - A test skeleton for each visible subprogram is created in a separate file, if it - doesn't exist already. By default, those separate test files are located in a - "gnattest/tests" directory that is created in the object directory of - corresponding project file. For example, if a source file my_unit.ads in - directory src contains a visible subprogram Proc, then the corresponding unit - test will be found in file src/tests/my_unit-test_data-tests.adb and will be - called Test_Proc_. is a signature encoding used to differentiate - test names in case of overloading. - - Note that if the project already has both my_unit.ads and my_unit-test_data.ads, - this will cause a name conflict with the generated test package. - - - .. _Test_Execution_Mode: - - Test Execution Mode - ^^^^^^^^^^^^^^^^^^^ - - In this mode ``gnattest`` has a the following command-line interface: - - :: - - $ gnattest test_drivers.list [ switches ] - - where - - * :samp:`{test_drivers.list}` - is the name of the text file containing the list of executables to treat as - test drivers. This file is automatically generated by gnattest, but can be - hand-edited to add or remove tests. This switch is required. - - - * :samp:`{switches}` - is an optional sequence of switches as described below. - - - .. _Switches_for_gnattest_in_framework_generation_mode: - - Switches for ``gnattest`` in framework generation mode - ------------------------------------------------------ - - .. index:: --strict (gnattest) - - :switch:`--strict` - Return error exit code if there are any compilation errors. - - .. index:: -q (gnattest) - - :switch:`-q` - Quiet mode: suppresses noncritical output messages. - - - .. index:: -v (gnattest) - - :switch:`-v` - Verbose mode: produces additional output about the execution of the tool. - When specified alone on the command line, prints tool version and exits. - - - .. index:: -r (gnattest) - - :switch:`-r` - Recursively considers all sources from all projects. - - .. index:: -files (gnattest) - - :switch:`-files={filename}` - Take as arguments the files listed in text file ``file``. - Text file ``file`` may contain empty lines that are ignored. - Each nonempty line should contain the name of an existing file. - Several such switches may be specified simultaneously. - - .. index:: --ignore (gnattest) - - :switch:`--ignore={filename}` - Do not process the sources listed in a specified file. - - .. index:: --RTS (gnattest) - - :switch:`--RTS={rts-path}` - Specifies the default location of the runtime library. Same meaning as the - equivalent ``gnatmake`` flag (:ref:`Switches_for_gnatmake`). For restricted - profiles, ``gnattest`` takes into account the run-time limitations when - generating the harness. - - - .. index:: --additional-tests (gnattest) - - :switch:`--additional-tests={projname}` - Sources described in ``projname`` are considered potential additional - manual tests to be added to the test suite. - - - .. index:: --harness-only (gnattest) - - :switch:`--harness-only` - When this option is given, ``gnattest`` creates a harness for all - sources, treating them as test packages. This option is not compatible with - closure computation done by -U main. - - - .. index:: --separate-drivers (gnattest) - - :switch:`--separate-drivers[={val}]` - Generates a separate test driver for each test or unit under test, rather - than a single executable incorporating all tests. ``val`` can be "unit" or - "test", or may be omitted, which defaults to "unit". - - - .. index:: --stub (gnattest) - - :switch:`--stub` - Generates the testing framework that uses subsystem stubbing to isolate the - code under test. - - - .. index:: --harness-dir (gnattest) - - :switch:`--harness-dir={dirname}` - Specifies the directory that will hold the harness packages and project file - for the test driver. If the ``dirname`` is a relative path, it is considered - relative to the object directory of the project file. - - - .. index:: --tests-dir (gnattest) - - :switch:`--tests-dir={dirname}` - All test packages are placed in the ``dirname`` directory. - If the ``dirname`` is a relative path, it is considered relative to the object - directory of the project file. When all sources from all projects are taken - recursively from all projects, ``dirname`` directories are created for each - project in their object directories and test packages are placed accordingly. - - - .. index:: --subdir (gnattest) - - :switch:`--subdir={dirname}` - Test packages are placed in a subdirectory of the corresponding source - directory, with the name ``dirname``. Thus, each set of unit tests is located - in a subdirectory of the code under test. If the sources are in separate - directories, each source directory has a test subdirectory named ``dirname``. - - - .. index:: --tests-root (gnattest) - - :switch:`--tests-root={dirname}` - The hierarchy of source directories, if any, is recreated in the ``dirname`` - directory, with test packages placed in directories corresponding to those - of the sources. - If the ``dirname`` is a relative path, it is considered relative to the object - directory of the project file. When projects are considered recursively, - directory hierarchies of tested sources are - recreated for each project in their object directories and test packages are - placed accordingly. - - - .. index:: --stubs-dir (gnattest) - - :switch:`--stubs-dir={dirname}` - The hierarchy of directories containing stubbed units is recreated in - the ``dirname`` directory, with stubs placed in directories corresponding to - projects they are derived from. - If the ``dirname`` is a relative path, it is considered relative to the object - directory of the project file. When projects are considered recursively, - directory hierarchies of stubs are - recreated for each project in their object directories and test packages are - placed accordingly. - - - .. index:: --exclude-from-stubbing (gnattest) - - :switch:`--exclude-from-stubbing={filename}` - Disables stubbing of units listed in ``filename``. The file should contain - corresponding spec files, one per line. - - :switch:`--exclude-from-stubbing:{unit}={filename}` - Same as above, but corresponding units will not be stubbed only when testing - specified ``unit``. - - .. index:: --validate-type-extensions (gnattest) - - :switch:`--validate-type-extensions` - Enables substitution check: run all tests from all parents in order - to check substitutability in accordance with the Liskov substitution principle (LSP). - - .. index:: --inheritance-check (gnattest) - - :switch:`--inheritance-check` - Enables inheritance check: run inherited tests against descendants. - - .. index:: --no-inheritance-check (gnattest) - - :switch:`--no-inheritance-check` - Disables inheritance check. - - .. index:: --no-inheritance-check (gnattest) - - :switch:`--test-case-only` - Generates test skeletons only for subprograms that have at least one - associated pragma or aspect Test_Case. - - .. index:: --skeleton-default (gnattest) - - :switch:`--skeleton-default={val}` - Specifies the default behavior of generated skeletons. ``val`` can be either - "fail" or "pass", "fail" being the default. - - - .. index:: --passed-tests (gnattest) - - :switch:`--passed-tests={val}` - Specifies whether or not passed tests should be shown. ``val`` can be either - "show" or "hide", "show" being the default. - - - .. index:: --exit-status (gnattest) - - :switch:`--exit-status={val}` - Specifies whether or not generated test driver should return failure exit - status if at least one test fails or crashes. ``val`` can be either - "on" or "off", "off" being the default. - - - .. index:: --omit-sloc (gnattest) - - :switch:`--omit-sloc` - Suppresses comment line containing file name and line number of corresponding - subprograms in test skeletons. - - - .. index:: --no-command-line (gnattest) - - :switch:`--no-command-line` - Don't add command line support to test driver. Note that regardless of this - switch, ``gnattest`` will automatically refrain from adding command - line support if it detects that the selected run-time doesn't provide - this capability. - - - .. index:: --separates (gnattest) - - :switch:`--separates` - Bodies of all test routines are generated as separates. Note that this mode is - kept for compatibility reasons only and it is not advised to use it due to - possible problems with hash in names of test skeletons when using an - inconsistent casing. Separate test skeletons can be incorporated to monolith - test package with improved hash being used by using ``--transition`` - switch. - - - .. index:: --transition (gnattest) - - :switch:`--transition` - This allows transition from separate test routines to monolith test packages. - All matching test routines are overwritten with contents of corresponding - separates. Note that if separate test routines had any manually added with - clauses they will be moved to the test package body as is and have to be moved - by hand. - - - .. index:: --test-duration (gnattest) - - :switch:`--test-duration` - Adds time measurements for each test in generated test driver. - - - :switch:`--tests_root`, :switch:`--subdir` and :switch:`--tests-dir` switches are mutually exclusive. - - - .. _Switches_for_gnattest_in_test_execution_mode: - - Switches for ``gnattest`` in test execution mode - ------------------------------------------------ - - - .. index:: --passed-tests (gnattest) - - :switch:`--passed-tests={val}` - Specifies whether or not passed tests should be shown. ``val`` can be either - "show" or "hide", "show" being the default. - - - .. index:: --queues (gnattest) - .. index:: -j (gnattest) - - :switch:`--queues={n}`, :switch:`-j{n}` - Runs ``n`` tests in parallel (default is 1). - - - .. index:: --copy-environment (gnattest) - - :switch:`--copy-environment={dir}` - Contents of ``dir`` directory will be copied to temporary directories - created by gnattest in which individual test drivers are spawned. - - - .. _Project_Attributes_for_gnattest: - - Project Attributes for ``gnattest`` - ----------------------------------- - - Most of the command-line options can also be passed to the tool by adding - special attributes to the project file. Those attributes should be put in - package ``Gnattest``. Here is the list of attributes: - - - * ``Tests_Root`` - is used to select the same output mode as with the ``--tests-root`` option. - This attribute cannot be used together with ``Subdir`` or ``Tests_Dir``. - - * ``Subdir`` - is used to select the same output mode as with the ``--subdir`` option. - This attribute cannot be used together with ``Tests_Root`` or ``Tests_Dir``. - - * ``Tests_Dir`` - is used to select the same output mode as with the ``--tests-dir`` option. - This attribute cannot be used together with ``Subdir`` or ``Tests_Root``. - - * ``Stubs_Dir`` - is used to select the same output mode as with the ``--stubs-dir`` option. - - * ``Harness_Dir`` - is used to specify the directory in which to place harness packages and project - file for the test driver, otherwise specified by ``--harness-dir``. - - * ``Additional_Tests`` - is used to specify the project file, otherwise given by - ``--additional-tests`` switch. - - * ``Skeletons_Default`` - is used to specify the default behaviour of test skeletons, otherwise - specified by ``--skeleton-default`` option. The value of this attribute - should be either ``pass`` or ``fail``. - - * ``Default_Stub_Exclusion_List`` - is used to specify the file with list of units whose bodies should not - be stubbed, otherwise specified by ``--exclude-from-stubbing=filename``. - - * ``Stub_Exclusion_List ("unit")`` - is used to specify the file with list of units whose bodies should not - be stubbed when testing "unit", otherwise specified by - ``--exclude-from-stubbing:unit=filename``. - - Each of those attributes can be overridden from the command line if needed. - Other ``gnattest`` switches can also be passed via the project - file as an attribute list called ``Gnattest_Switches``. - - - .. _Simple_gnattest_Example: - - Simple Example - -------------- - - Let's take a very simple example using the first ``gnattest`` example - located in: - - :: - - /share/examples/gnattest/simple - - This project contains a simple package containing one subprogram. By running ``gnattest``: - - :: - - $ gnattest --harness-dir=driver -Psimple.gpr - - a test driver is created in directory ``driver``. It can be compiled and run: - - :: - - $ cd obj/driver - $ gprbuild -Ptest_driver - $ test_runner - - One failed test with the diagnosis "test not implemented" is reported. - Since no special output option was specified, the test package ``Simple.Tests`` - is located in: - - :: - - /share/examples/gnattest/simple/obj/gnattest/tests - - - For each package containing visible subprograms, a child test package is - generated. It contains one test routine per tested subprogram. Each - declaration of a test subprogram has a comment specifying which tested - subprogram it corresponds to. Bodies of test routines are placed in test package - bodies and are surrounded by special comment sections. Those comment sections - should not be removed or modified in order for gnattest to be able to regenerate - test packages and keep already written tests in place. - The test routine ``Test_Inc_5eaee3`` located at :file:`simple-test_data-tests.adb` contains - a single statement: a call to procedure ``Assert``. It has two arguments: - the Boolean expression we want to check and the diagnosis message to display if - the condition is false. - - That is where actual testing code should be written after a proper setup. - An actual check can be performed by replacing the ``Assert`` call with: - - :: - - Assert (Inc (1) = 2, "wrong incrementation"); - - After recompiling and running the test driver, one successfully passed test - is reported. - - - .. _Setting_Up_and_Tearing_Down_the_Testing_Environment: - - Setting Up and Tearing Down the Testing Environment - --------------------------------------------------- - - Besides test routines themselves, each test package has a parent package - ``Test_Data`` that has two procedures: ``Set_Up`` and ``Tear_Down``. This package is never - overwritten by the tool. ``Set_Up`` is called before each test routine of the - package, and ``Tear_Down`` is called after each test routine. Those two procedures - can be used to perform necessary initialization and finalization, - memory allocation, etc. Test type declared in ``Test_Data`` package is parent type - for the test type of test package and can have user-defined components whose - values can be set by ``Set_Up`` routine and used in test routines afterwards. - - - .. _Regenerating_Tests: - - Regenerating Tests - ------------------ - - Bodies of test routines and ``Test_Data`` packages are never overridden after they - have been created once. As long as the name of the subprogram, full expanded Ada - names and order of its parameters are the same, and comment sections are - intact, the old test routine will fit in its place and no test skeleton will be - generated for the subprogram. - - This can be demonstrated with the previous example. By uncommenting declaration - and body of function Dec in ``simple.ads`` and ``simple.adb``, running - ``gnattest`` on the project, and then running the test driver: - - :: - - $ gnattest --harness-dir=driver -Psimple.gpr - $ cd obj/driver - $ gprbuild -Ptest_driver - $ test_runner - - The old test is not replaced with a stub, nor is it lost, but a new test - skeleton is created for function ``Dec``. - - The only way of regenerating tests skeletons is to remove the previously created - tests together with corresponding comment sections. - - - .. _Default_Test_Behavior: - - Default Test Behavior - --------------------- - - The generated test driver can treat unimplemented tests in two ways: - either count them all as failed (this is useful to see which tests are still - left to implement) or as passed (to sort out unimplemented ones from those - actually failing). - - The test driver accepts a switch to specify this behavior: - :switch:`--skeleton-default={val}`, where ``val`` is either ``pass`` or ``fail`` (exactly as for - ``gnattest``). - - The default behavior of the test driver is set with the same switch - as passed to ``gnattest`` when generating the test driver. - - Passing it to the driver generated on the first example: - - :: - - $ test_runner --skeleton-default=pass - - makes both tests pass, even the unimplemented one. - - - .. _Testing_Primitive_Operations_of_Tagged_Types: - - Testing Primitive Operations of Tagged Types - -------------------------------------------- - - Creation of test skeletons for primitive operations of tagged types entails - a number of features. Test routines for all primitives of a given tagged type - are placed in a separate child package named according to the tagged type. For - example, if you have tagged type ``T`` in package ``P``, all tests for primitives - of ``T`` will be in ``P.T_Test_Data.T_Tests``. - - Consider running ``gnattest`` on the second example (note: actual tests for this - example already exist, so there's no need to worry if the tool reports that - no new stubs were generated): - - :: - - $ cd /share/examples/gnattest/tagged_rec - $ gnattest --harness-dir=driver -Ptagged_rec.gpr - - Taking a closer look at the test type declared in the test package - *Speed1.Controller_Test_Data* is necessary. It is declared in: - - :: - - /share/examples/gnattest/tagged_rec/obj/gnattest/tests - - Test types are direct or indirect descendants of - *AUnit.Test_Fixtures.Test_Fixture* type. In the case of non-primitive tested - subprograms, the user doesn't need to be concerned with them. However, - when generating test packages for primitive operations, there are some things - the user needs to know. - - Type ``Test_Controller`` has components that allow assignment of various - derivations of type ``Controller``. And if you look at the specification of - package *Speed2.Auto_Controller*, you will see that ``Test_Auto_Controller`` - actually derives from ``Test_Controller`` rather than AUnit type ``Test_Fixture``. - Thus, test types mirror the hierarchy of tested types. - - The ``Set_Up`` procedure of ``Test_Data`` package corresponding to a test package - of primitive operations of type ``T`` assigns to ``Fixture`` a reference to an - object of that exact type ``T``. Note, however, that if the tagged type has - discriminants, the ``Set_Up`` only has a commented template for setting - up the fixture, since filling the discriminant with actual value is up - to the user. - - The knowledge of the structure of test types allows additional testing - without additional effort. Those possibilities are described below. - - - .. _Testing_Inheritance: - - Testing Inheritance - ------------------- - - Since the test type hierarchy mimics the hierarchy of tested types, the - inheritance of tests takes place. An example of such inheritance can be - seen by running the test driver generated for the second example. As previously - mentioned, actual tests are already written for this example. - - :: - - $ cd obj/driver - $ gprbuild -Ptest_driver - $ test_runner - - There are 6 passed tests while there are only 5 testable subprograms. The test - routine for function Speed has been inherited and run against objects of the - derived type. - - - .. _Tagged_Type_Substitutability_Testing: - - Tagged Type Substitutability Testing - ------------------------------------ - - *Tagged Type Substitutability Testing* is a way of verifying the global type - consistency by testing. Global type consistency is a principle stating that if - ``S`` is a subtype of ``T`` (in Ada, ``S`` is a derived type of tagged type ``T``), - then objects of type ``T`` may be replaced with objects of type ``S`` (that is, - objects of type ``S`` may be substituted for objects of type ``T``), without - altering any of the desirable properties of the program. When the properties - of the program are expressed in the form of subprogram preconditions and - postconditions (let's call them pre and post), the principle is formulated as - relations between the pre and post of primitive operations and the pre and post - of their derived operations. The pre of a derived operation should not be - stronger than the original pre, and the post of the derived operation should - not be weaker than the original post. Those relations ensure that verifying if - a dispatching call is safe can be done just by using the pre and post of the - root operation. - - Verifying global type consistency by testing consists of running all the unit - tests associated with the primitives of a given tagged type with objects of its - derived types. - - In the example used in the previous section, there was clearly a violation of - type consistency. The overriding primitive ``Adjust_Speed`` in package ``Speed2`` - removes the functionality of the overridden primitive and thus doesn't respect - the consistency principle. - ``gnattest`` has a special option to run overridden parent tests against objects - of the type which have overriding primitives: - - :: - - $ gnattest --harness-dir=driver --validate-type-extensions -Ptagged_rec.gpr - $ cd obj/driver - $ gprbuild -Ptest_driver - $ test_runner - - While all the tests pass by themselves, the parent test for ``Adjust_Speed`` fails - against objects of the derived type. - - Non-overridden tests are already inherited for derived test types, so the - ``--validate-type-extensions`` enables the application of overridden tests - to objects of derived types. - - - .. _Testing_with_Contracts: - - Testing with Contracts - ---------------------- - - ``gnattest`` supports pragmas ``Pre``, ``Post``, and ``Test_Case``, - as well as the corresponding Ada 2012 aspects. - Test routines are generated, one per each ``Test_Case`` associated with a tested - subprogram. Those test routines have special wrappers for tested functions - that have composition of pre- and postcondition of the subprogram with - "requires" and "ensures" of the ``Test_Case`` (depending on the mode, pre and post - either count for ``Nominal`` mode or do *not* count for ``Robustness`` mode). - - The third example demonstrates how this works: - - :: - - $ cd /share/examples/gnattest/contracts - $ gnattest --harness-dir=driver -Pcontracts.gpr - - Putting actual checks within the range of the contract does not cause any - error reports. For example, for the test routine which corresponds to - test case 1: - - :: - - Assert (Sqrt (9.0) = 3.0, "wrong sqrt"); - - and for the test routine corresponding to test case 2: - - :: - - Assert (Sqrt (-5.0) = -1.0, "wrong error indication"); - - are acceptable: - - :: - - $ cd obj/driver - $ gprbuild -Ptest_driver - $ test_runner - - However, by changing 9.0 to 25.0 and 3.0 to 5.0, for example, you can get - a precondition violation for test case one. Also, by using any otherwise - correct but positive pair of numbers in the second test routine, you can also - get a precondition violation. Postconditions are checked and reported - the same way. - - - .. _Additional_Tests: - - Additional Tests - ---------------- - - ``gnattest`` can add user-written tests to the main suite of the test - driver. ``gnattest`` traverses the given packages and searches for test - routines. All procedures with a single in out parameter of a type which is - derived from *AUnit.Test_Fixtures.Test_Fixture* and that are declared in package - specifications are added to the suites and are then executed by the test driver. - (``Set_Up`` and ``Tear_Down`` are filtered out.) - - An example illustrates two ways of creating test harnesses for user-written - tests. Directory ``additional_tests`` contains an AUnit-based test driver written - by hand. - - :: - - /share/examples/gnattest/additional_tests/ - - To create a test driver for already-written tests, use the ``--harness-only`` - option: - - :: - - gnattest -Padditional/harness/harness.gpr --harness-dir=harness_only \\ - --harness-only - gprbuild -Pharness_only/test_driver.gpr - harness_only/test_runner - - Additional tests can also be executed together with generated tests: - - :: - - gnattest -Psimple.gpr --additional-tests=additional/harness/harness.gpr \\ - --harness-dir=mixing - gprbuild -Pmixing/test_driver.gpr - mixing/test_runner - - - .. _Individual_Test_Drivers: - - Individual Test Drivers - ----------------------- - - By default, ``gnattest`` generates a monolithic test driver that - aggregates the individual tests into a single executable. It is also possible - to generate separate executables for each test or each unit under test, by - passing the switch ``--separate-drivers`` with corresponding parameter. This - approach scales better for large testing campaigns, especially involving target - architectures with limited resources typical for embedded development. It can - also provide a major performance benefit on multi-core systems by allowing - simultaneous execution of multiple tests. - - ``gnattest`` can take charge of executing the individual tests; for this, - instead of passing a project file, a text file containing the list of - executables can be passed. Such a file is automatically generated by gnattest - under the name :file:`test_drivers.list`, but it can be - hand-edited to add or remove tests, or replaced. The individual tests can - also be executed standalone, or from any user-defined scripted framework. - - - .. _Stubbing: - - Stubbing - -------- - - Depending on the testing campaign, it is sometimes necessary to isolate the - part of the algorithm under test from its dependencies. This is accomplished - via *stubbing*, i.e. replacing the subprograms that are called from the - subprogram under test by stand-in subprograms that match the profiles of the - original ones, but simply return predetermined values required by the test - scenario. - - This mode of test harness generation is activated by the switch ``--stub``. - - The implementation approach chosen by ``gnattest`` is as follows. - For each package under consideration all the packages it is directly depending - on are stubbed, excluding the generic packages and package instantiations. - The stubs are shared for each package under test. The specs of packages to stub - remain intact, while their bodies are replaced, and hide the original bodies by - means of extending projects. Also, for each stubbed - package, a child package with setter routines for each subprogram declaration - is created. These setters are meant to be used to set the behavior of - stubbed subprograms from within test cases. - - Note that subprograms belonging to the same package as the subprogram under - test are not stubbed. This guarantees that the sources being tested are - exactly the sources used for production, which is an important property for - establishing the traceability between the testing campaign and production code. - - Due to the nature of stubbing process, this mode implies the switch - ``--separate-drivers``, i.e. an individual test driver (with the - corresponding hierarchy of extending projects) is generated for each unit under - test. - - .. note:: - - Developing a stubs-based testing campaign requires - good understanding of the infrastructure created by ``gnattest`` for - this purpose. We recommend following the two stubbing tutorials - ``simple_stubbing`` and ``advanced_stubbing`` provided - under :file:`/share/examples/gnattest` before - attempting to use this powerful feature. - - - .. _Gnatcov_Integration: - - Integration with GNATcoverage - ----------------------------- - - In addition to the harness, ``gnattest`` generates a Makefile. This Makefile - provides targets for building the test drivers and also the targets for - computing the coverage information using GNATcoverage framework when this - coverage analysis tool is available. The target ``coverage`` fully automates - the process: it will first build all test drivers, then run them under - GNATcoverage, analyze individual trace files, and finally aggregate them: - - :: - - make coverage - - GNATcoverage options, such as coverage criteria and generated report format, - can be adjusted using Makefile variables provided for this purpose. - - Note that coverage targets are not generated in the Makefile when - --separate-drivers=test is passed to gnattest. - - - .. _Putting_Tests_under_Version_Control: - - Putting Tests under Version Control - ----------------------------------- - - As has been stated earlier, ``gnattest`` generates two different types - of code, test skeletons and harness. The harness is generated completely - automatically each time, does not require manual changes and therefore should - not be put under version control. - It makes sense to put under version control files containing test data packages, - both specs and bodies, and files containing bodies of test packages. Note that - test package specs are also generated automatically each time and should not be - put under version control. - Option ``--omit-sloc`` may be useful when putting test packages under version control. - - - .. _Current_Limitations: - - Current Limitations - ------------------- - - The tool currently has the following limitations: - - * generic tests for nested generic packages and their instantiations are - not supported; - * tests for protected subprograms and entries are not supported; - * pragma ``No_Run_Time`` is not supported; - * pragma ``No_Secondary_Stack`` is not supported; - * if pragmas for interfacing with foreign languages are used, manual - adjustments might be necessary to make the test harness compilable; - * use of some constructs, such as elaboration-control pragmas, Type_Invariant - aspects, and complex variable initializations that use Subprogram'Access, - may result in elaboration circularities in the generated harness. - - .. only:: PRO or GPL .. _The_Backtrace_Symbolizer_gnatsymbolize: