The set of tests run (the scope) per runset vary by the combined state of the agents involved in a test. For example:
- Some agents implement Aries Interop Profile (AIP) 1.0, others 2.0 and others, both.
- Some agents have implemented more features than others, and therefore support more tests,
- Some agents are unable to support particular tests because of practical limitations of the device they run on (e.g. mobile) or the ease with which that test can be conducted.
For these reasons it's not possible to say that, for example, an 80% pass result is "good" or 50% is "bad". The numbers need to be understood in context.
The scope and exceptions columns in the summary and the summary statement found on each runset detail page on this website, document the scope and expectations of runset.
Each runset detail page also provides narrative on the current status of the runset — for example, why some tests of a runset are failing, what issues have been opened and where to address the issue.
Tests can fail for many reasons, and much of the work of maintaining the tests and runsets is staying on top of the failures. The following are some notes about failing tests and what to do about them:
- Most failures happen when new runsets are added. For example, a new Test Agent is added to the repository and it doesn't play well with the existing Test Agents.
- Since most runsets pull the latest main branch code for each Test Agent for each run of the set, errors may be introduced as a result of code added to a Test Agent. Thus, the AATH is an example of a new form of "CI", Continuous Interoperability.
- With each failure, an evaluation is needed about the cause of the error:
- Is the test that is incorrectly interpreting the RFC?
- Is the RFC itself wrong?
- Is one of the agents/frameworks involved in the testing implementing the RFC incorrectly?
- Is the driver (called the Backchannel) that is controlling the agent/framework have a bug?
- Each failure should result in a GitHub issue being opened in the appropriate repo and the runset narrative updated to note the failure and the issue. Once solved, the runset narrative should be updated to remove the reference to the issue.
Investigating Failing Tests
The Allure reports accessible from this site provide a lot of information about failing tests and are a good place to start in figuring out what is happening. Here's how to follow the links to get to the test failure details:
- On the main page, pick a Test Agent whose tests you want to drill into by clicking the name of the Test Agent from the main page summary table (first column), taking you to the Test Agent page.
- On the Test Agent page, find a runset with at least some failures, and click on the runset name from the Test Agent summary table (first column), taking you to the section of the Test Agent page about that runset.
- Within the runset details section, check the "Current Runset Status" summary to see if there is any reason for the failures there, and then drill into the test results by clicking the link entitled "Results by executed Aries RFCs." Clicking that link takes you from the Aries Interop Info site into the Allure test results, and lots (and lots) of details about test runs.
- The page you will arrive upon will show the handful of RFCs scenarios executed during the test runs (with titles like "RFC 0036 Aries agent issue credential"), and for each a count of the status for tests cases within each scenario (e.g. passed, failed, broken and so on).
- Expand a scenario with one or more failing tests and then click on the one of the failed tests (with an ugly red "X" beside the test case) to see the details (stack trace) of the error on the right part of the page.
- From here you can look at a variety of things about the failing test:
- Review the test failure "Overview" to see the specific test step that failed and related failed assertion stack trace.
- Scroll down to the sequence of steps in the test, to see how far along the test got before the failure.
- Click on the "History" tab to see if it has always failed, or if this is a new issue.
- If it has passed in earlier runs, drill into the passing test to see if you can learn anything from that.
- Dive into the weeds…look at the stack trace, find the associated code, and see if you can figure out what happened. Find anything? Report it via an issue, or even better, submit a Pull Request to fix the issue. Remember to consider all of the possible places an error could occur (above).
In addition to drilling into a specific test scenario (aka "stories")/case (aka "behavior")/step, you can look at the recent runset history (last 20 runs). On the left side menu, click on "Overview", and then take a look at the big "history" graph in the top right, showing how the runset execution has varied over time. Ideally, it's all green, but since you started from a runset that had failures, it won't be. Pretty much every part of the overview page is a drill down link into more and more detailed information about the runset, a specific run of the runset, a specific test case and so on. Lots to look at!
What is Aries Interop Profile?
Aries Interop Profile (AIP) is a set of concepts and protocols that every Aries agent that wants to be interoperable should implement. Specific Aries agents may implement additional capabilities and protocols, but for interoperability, they must implement those defined in an AIP.
AIP currently has two versions:
- AIP 1.0, finalized in January 2020, defines how Aries agents communicate with each other using a single type of verifiable credential
- AIP 2.0, is expected to be finalized in March 2021, builds on AIP 1.0 including how Aries agents can exchange several types of verifiable credentials, including W3C standard verifiable credentials.
AIP versions go through a rigorous community process of discussion and refinement before being agreed upon. During that process, the RFCs that go into each AIP are debated and the specific version of each included RFC is locked down. AIPs are available for anyone to review (and potentially contribute to) in the Aries RFC repo.
How can I contribute?
For developers improving an Aries agent or framework, each runset's page has a link to a detailed report in Allure. This allows the specific tests and results to be explored in detail.
If you are a stakeholder interested in improving the results for an agent, this website (and the Allure links, described above) should have enough material for your teams to take action.
Finally, if you want your Aries agent to be added to this website, or wish to expand the tests covered for your agent, your developers can reference the extensive information in the Aries Agent Test Harness repo on GitHub.
In addition an API reference for backchannels can be found here
Aries Interoperability Test Results