The set of tests run (the scope) per runset vary by the combined state of the agents involved in a test. For example:
For these reasons it's not possible to say that, for example, an 80% pass result is "good" or 50% is "bad". The numbers need to be understood in context.
The scope and exceptions columns in the summary and the summary statement found on each runset's page on this website, document the scope and expectations of runset.
Each runset's page also provides narrative on the current status of the runset — for example, why some tests of a runset are failing, what issues have been opened and where to address the issue.
Tests can fail for many reasons, and much of the work of maintaining the tests and runsets is staying on top of the failures. The following are some notes about failing tests and what to do about them:
Aries Interop Profile (AIP) is a set of concepts and protocols that every Aries agent that wants to be interoperable should implement. Specific Aries agents may implement additional capabilities and protocols, but for interoperability, they must implement those defined in an AIP.
AIP currently has two versions:
AIP versions go through a rigorous community process of discussion and refinement before being agreed upon. During that process, the RFCs that go into each AIP are debated and the specific version of each included RFC is locked down. AIPs are available for anyone to review (and potentially contribute to) in the Aries RFC repo.
For developers improving an Aries agent or framework, each runset's page has a link to a detailed report in Allure. This allows the specific tests and results to be explored in detail.
If you are a stakeholder interested in improving the results for an agent, this website (and the Allure links, described above) should have enough material for your teams to take action.
Finally, if you want your Aries agent to be added to this website, or wish to expand the tests covered for your agent, your developers can reference the extensive information in the Aries Agent Test Harness repo on GitHub.
In addition an API reference for backchannels can be found here