How can text and data mining activity be audited?

The COUNTER Code of Practice includes an Access Method called TDM, which is used to report on text and data mining activity, for example through a specific API. The Code has scripts for auditors to follow, but doesn’t specify whether that’s for Regular (human) usage or for TDM. Traditionally audits have run on Access Method Regular, but we’ve recently had a question about how TDM activity can be included.

Which reports?

Let’s start with the basics. TDM only shows up in the Platform, Database, Title and Item Reports, not any of the standard views. Which of the reports is relevant is defined by the platform’s Host Type.

Which audit tests?

Most of the audit tests listed in Appendix E can be applied to TDM. There are some exceptions. Audit tests for double-click filtering aren’t relevant in a machine-first environment like TDM. Similarly, the audit tests for search metrics are unlikely to be relevant. That leaves the tests for denials (No License and/or Limit Exceeded), and the tests for usage metrics (Investigations and Requests).

How should we run the tests?

Let’s take the example of a test scripts in Appendix E for No License denials:

“The content for which the auditor has no license MUST be declared by the report provider prior to audit testing.

The auditor MUST force 50 No_License turnaways by logging into the site and requesting an item. Each time access is refused, the auditor will record this as 1 No_License.

The test MUST result in 50 No_License.”

That test could be run just as well through an API, resulting in 50 No License denials with Access Method TDM, instead of 50 No License denials with Access Method Regular. The same applies to our other denial and usage tests.

Do we have to run the audit twice – once for Regular and once for TDM?

That isn’t necessary. We have some guidelines about how to tackle the issue of TDM audits:

1. At least once of the usage metric tests must be run on both Regular and TDM, and they must return identical results. If there is any discrepancy in the results between Regular and TDM on this test, the auditor must issue an interim report and require fixes before the audit can be passed.
2. At least once of the denial metric tests must be run on both Regular and TDM, and they must return identical results. Again, any discrepancies mean the auditor must issue an interim report.
3. The remaining tests should be split between Regular and TDM. For example, the Access Type tests could be run on Access Method Regular, and the Investigations Independent of Requests tests on Access Method TDM.

Next steps

If you’re interested in a TDM audit, speak to your auditor. Just make sure the information about Regular and TDM is very clear in the audit report that’s returned to COUNTER!