AI best practice phase two

We’ve published our updated Best Practice on Generative and Agentic AI usage metrics!

AI tools are an increasingly regular part of how people search for and read scholarly content. As that’s happened, tracking agentic usage in a clear, consistent way has become a top priority for our community. This new update extends our original best practice to cover not just agentic activity on publisher platforms, but in third-party tools too.

As part of this work, we also updated our Best Practice on Syndicated Usage. That means everything should fit together smoothly no matter where users are when they access content – whether that’s a publisher platform, an aggregator site, or a third-party AI tool.

What’s New in Phase Two?

Phase two builds on our initial AI guidelines by introducing Retrieval Metrics. These give us a clearer look at what happens behind the scenes during the early stages of an AI response:

• Total Retrieved Investigations track every time an AI system pulls a snippet or excerpt of content during its initial search. Retrievals are counted even if the excerpt isn’t used in an agent’s response.
• Total Retrieved Requests identifies when those retrievals are by agents with access to the full text of an item.

We’ve also aligned our guidelines with the SPUR telemetry framework. By mapping from SPUR to COUNTER, it’s more likely that publishers and librarians will get the usage data they need. We’ve also mapped Google Scholar’s new MERU tool to our metrics.

A Quick Recap of the Core AI Metrics

What we established in phase one is still in place:

• AI traffic is tagged with Access Method: Agent, and will only shows up in reports when specifically requested.
• The established AI metrics like Unique AI Requests still only count excerpts that are actually picked by the AI to synthesize a final answer.
• And AI Responses Generated still keeps track of the responses delivered back to users, in an agentic counterpart to our traditional search metrics.

Syncing Up with Syndicated Usage

Because so much AI activity happens across different services, we also updated our Best Practice on Syndicated Usage. We’ve been a lot clearer that where third parties like AI developers are sharing unprocessed usage data, the publisher can process and report that data just as they do all other usage. That is, the AI developers don’t need to be COUNTER compliant (though they can be!).

For those who have forgotten, these are the highlights of our syndication guidelines:

• Third-party tools sharing usage data back to original publishers can either send raw data for the publisher to process or share audited COUNTER metrics directly.
• All syndicated usage added to a publisher’s reports has to be labelled with SYNDICATED:{platform name} so everyone knows where the activity occurred.
• Institutional reporting relyies on precise matches (like Shibboleth/OpenAthens, shared IP lists, or GetFTR) rather than guesswork.

Find out more

We’ll be running member-only webinars about our AI work in January (for publishers and technology providers) and February (for libraries and consorita). We’ll open registration in the November newsletter, so sign up if you haven’t already – and make sure you’re a COUNTER member!

What’s Next?

With phase two published, our attention is turning to updating the COUNTER API and JSON Schema to accommodate AI usage metrics. As mentioned earlier this month, that development work is part-funded by TRACE – a cross-stakeholder initiative involving STM, NISO, COUNTER, and other partners examining the future of trusted retrieval and attribution in AI-mediated scholarly communication, for the benefit of the entire scholarly communications ecosystem. Contributors to the development work were received from American Chemical Society, American Psychological Association, Cambridge University Press, Elsevier, IEEE, IOP Publishing, Sage, Springer Nature, and Wiley.

We’ll also be keeping a close eye on other TRACE projects, including work on agent gateways, that will influence future COUNTER work (e.g. on bot management).

Thanks to the AI working group!

This would not have been possible without the invaluable input of members of our AI working group, listed below, and the technical input of our Code Team.

• Chaired by Michael Sisolak, Sage Publishing
• Adam Pulford, Oxford University Press
• Athena Hoeppner, University of Central Florida
• Ed Hallett, Springer Nature
• Heather Staines, Delta Think & STM
• Jon Blackburn, OCLC
• Katie Arabie, SPIE
• Kirsten Fuoti, Silverchair
• Lisa Walton, BMJ Publishing Group
• Lisa Wylie, Wiley
• Michelle Urberg, Liblynx
• Monica Westin, Cambridge University Press & Assessment
• Nico Zazworka, Elsevier
• Peter White, Cambridge University Press & Assessment
• Rikki Gracia, IEEE
• Sam Wotring, Clarivate
• Sara Crowley-Vigneau, Zendy
• Senol Akay, American Chemical Society
• Stuart Maxwell, Scholarly iQ
• Sue Hodgson, Cashmere