Aries Agent Test Harness: Advanced Test Reporting¶
Contents¶
- Aries Agent Test Harness: Advanced Test Reporting
- Default Reporting \& Output
- Allure Integration
- Other Reporting Options
- References
Default Reporting & Output¶
The Aries Agent Test Harness(AATH), in utilizing the Behave test engine, has default test output that specifically shows passed/failed steps along with a test/feature summary after an execution run. This output can of course be piped to a file as the user sees fit.
The need will arise to give more formal and graphic reporting along with keeping historical trends of test executions under a certain configuration. The AATH integrates with the Allure Reporting framework to fulfill this requirement.
Allure Integration¶
The AATH utilizes Allure in as much as Behave and Allure integrate. See Behave with Allure for details.
Using Allure with manage Script¶
The test execution container that is ramped up with the manage script gets the Allure framework installed for use inside the continer. So to execute the tests and have Allure generated report files, use the -r allure
option on the manage
script.
To generate the html report and start an allure report server. Use any IP or port in the open
command.
allure generate
command after another test run.
Allure reports with the Aries Agent Test Harness will resemble the following,

Using Allure at the Command Line¶
For debugging or developing purposes you may not want to always be running the test containers with the manage script, but you still may wish to maintain the reporting locally. To do that, just follow the standard command line options used with behave with custom formatters and reporters. To run this command you will need to have Allure installed locally as above.
behave -f allure_behave.formatter:AllureFormatter -o ./reports -t @AcceptanceTest -t ~@wip --no-skipped -D Acme=http://0.0.0.0:8020 -D Bob=http://0.0.0.0:8030 -D Faber=http://0.0.0.0:8050
Using Allure with CI¶
The AATH is executed with varying configurations and Aries Agent types at pre-determined intervals to make find issues and track deltas between builds of these agents. You can find the Allure reports for these test runs at the following links.
- Acapy to Acapy Agent Interop Testing
- Full Acapy to Acapy Agent Interop Testing
- Acapy to Dotnet Agent Interop Testing
- Acapy to JavaScript Agent Interop Testing
- Dotnet to Dotnet Agent Interop Testing
- JavaScript to JavaScript Agent Interop Testing
- JavaScript to Dotnet Agent Interop Testing
- Acapy to AFGO Agent Interop Testing
- AFGO to AFGO Agent Interop Testing
Other Reporting Options¶
junit¶
If your build pipeline is using junit style test results for reporting purposes, the AATH supports this as behave does. To use junit style report data, add the following to the behave.ini file or create your own ini file to use with behave that includes the following,
The above junit reports cannot be used in conjunction with Allure.