Key Takeaways from COUNTER’s AI Usage Metrics Consultation

Thank you to everyone in the community who participated in COUNTER’s Generative and Agentic AI usage metrics consultation. Your responses to the survey, by email, and in-person at NISO Plus provided invaluable guidance. We’re bringing those insights into developing the best practice, and into the next phase of our work.

New Access_Method: Agent

An overwhelming 97.5% of respondents agreed on the value of adding Access_Method: Agent to our reports, alongside Regular and TDM. This new method will be available in Platform, Database, Title, and Item Reports. It is a crucial first step in distinguishing machine usage from human usage.

You were clear in your responses that we’ll need a very precise definition of what constitutes ‘Agent’ usage. We’re also looking at how platforms can reliably classify agents, especially as AI functionality becomes more deeply embedded in user-facing tools.

Distinct AI metrics

While Access_Method: Agent provides separation, a strong majority of responses wanted separate, AI-specific Metric_Types. The consensus is that AI-mediated use represents a fundamentally different interaction pattern. You want us to report AI activities like summarisation, retrieving content that is never displayed, and zero-click usage, separately from traditional human activity. 

Based on the consultation feedback and discussions at NISO Plus, we are now working on these potential new Metric_Types:

• Generated_Responses, to track instances where an AI system generates a response to a user prompt. Generated_Responses acts as an AI equivalent to our existing Searches_Platform. Each response to a user prompt will count as 1 Generated_Responses, regardless of the underlying processes the AI tool used to generate it.
• Total_AI_Investigations and Unique_AI_Investigations. These paired metrics will deliver information about when a content item or information related to a content item was used by an agent as part of a user session. We know that AI agents might assess hundreds or thousands of pieces of content. Only the content items selected by the agent for final synthesis will count as investigations. By excluding the items that are just checked during initial processing, we will avoid counting system noise.
• Total_AI_Requests and Unique_AI_Requests. Traditional COUNTER Requests reflect usage of full text content. In the same way, these paired metrics will indicate where an AI is able to access the full text of a content item. As with the AI_Investigation metrics, only items selected for final synthesis should be counted as requests.

Using AI metrics 

The feedback highlighted several vital uses for the new metrics:

• Demonstrating the continued value of scholarly content.
• Ensuring human usage metrics are not uncontrollably distorted by AI activity.
• Assessing library investments in AI features.
• Guide information literacy programs.
• Informing licensing policy.

Our AI working group is refining the definitions and technical specifications for these new COUNTER metrics, ensuring they are consistent with the existing Code of Practice while supporting the critical infrastructure shift into an AI-mediated world.