Redefining value in the age of AI

Tasha recently had a fantastic chat with Becky Moakes of Maverick Publishing Consultants, as part of their work on “Redefining Value in the Age of AI”. They covered a ton of ground, and we thought you might like to read Tasha’s answers to some of Becky’s questions!

It’s more than just metrics

The landscape is changing quickly, but the need for reliable, community-defined measurements is more important than ever. While our Best Practice on Generative and Agentic AI Usage Metrics is a start, there’s still plenty of challenges and opportunities we’re working on right now. When we talk about how to measure scholarly content usage in the age of AI, however, it’s not just a question of metrics. Standards for AI attribution and provenance don’t exist yet, but we need them to feed back into our metrics conversation. That’s why we’re actively collaborating with NISO on these crucial questions.

AI is not the first change that’s meant we don’t have a comprehensive picture of usage. Aggregation databases, which are heavily used by libraries, haven’t historically shared COUNTER-processed usage metrics back with source publishers. That lack of visibility was one of my big issues back when I was a publisher: I simply didn’t have a full picture of where and how content from our society journals was being used. That meant I couldn’t decide where it was worth investing time to work with aggregators. The problem is even bigger for highly distributed open access content! That’s why we put out our Best Practice on Syndicated Usage. Learning from that syndication process is part of the ongoing phase two of our AI working group project, as well as for the OA working group.

The nuance of the ‘bot’ problem

One of the most practical challenges for measuring AI usage is figuring out which “users” should count and which shouldn’t. COUNTER has always said that bots aren’t users, so usage reports must exclude crawler and bot activity. But the new wave of AI bots is different:

• Some are still just crawling, and need to be excluded.
• Others represent valid, user-initiated activity, such as AI agents accessing publisher content through an MCP, and these should be reported.

The main barrier now isn’t a lack of willingness from publishers, who are prioritisng development time for AI metrics. Instead, the challenge is developing a more nuanced understanding of bot behavior. We are updating the old COUNTER bots repository and collaborating with the community to work out whether it’s possible to define behavioral patterns for these new generation of bots and agents.

Usage metrics in a zero-click world

Becky’s original question posited that usage metrics are less reliable in an AI world. I argue that usage metrics have been and remain more reliable than citations or altmetrics, because those measurements aren’t backed by a standard. They are both subject to the whims of commercial entities.

However, I do agree that the changing nature of knowledge consumption, whether that’s through widespread syndication of OA content or the replacement of traditional search with AI-generated summaries, means that all of the ways we measure research probably need to change. Just as we are asking what usage means when it’s AI doing the using, we absolutely need to ask what citations mean when it’s AI doing the citing!

Libraries and publishers are telling me that traditional human usage is decreasing while AI usage is increasing (what we call zero-click usage). Much of the change appears to be happening as AI-generated summaries start to replace traditional search. Our AI working group has been developing ways to measure and report that shift in a way that fits with the trusted, community-defined metrics that the community has relied on for over two decades. As I said, we really need some standards around attribution and provenance to really make the new AI metrics work, and we’re collaborating with NISO and other partners to make that happen.

By the way, why don’t we have a standard for citations? Answers on a postcard, please!

Better metadata is better for AI – and for accessibility

AI doesn’t consume an article or chapter (an Item, in COUNTER terms) the way a person does. Instead, most AI use is of smaller parts of Items such as tokens or chunks. Luckily COUNTER already has a term for sub-units of an Item: we call them Components. Part of the measurement challenge is that we don’t know how different AI developers are defining their Components. The other part is that we don’t yet have a standardised metadata package at the Component level. We’re going to have to come up with agreements about what level of content chunking and what metadata needs to be exchanged in order to solve the attribution and provenance question, which will in turn help answer some of our open questions about usage.

Better metadata can help link an AI’s usage of Components (like a table in an article) back to the original Item for reporting. Even more exciting is their potential to ensure AI tools deliver better quality information – for instance, communicating retraction status clearly to help keep retracted research flagged or out of AI summaries. One of the great things about improving metadata, by the way, is that it also improves accessibility!

Standards aren’t cutting edge

The risk that AI development outstrips standards is real. We’ve faced similar paradigm shifts before, whether it was open access or the semantic web. Standards aren’t meant to work at the cutting edge. Instead, they bring the community together to agree on technically viable solutions that work for everyone, even if (when!) they’re imperfect.

The goal isn’t perfection, but a shared understanding. Throwing out the standards baby with the AI bathwater would be a mistake. Imagine buying a new kitchen: you wouldn’t be happy if suppliers used cubits and ells in their quotes, rather than millimetres (or inches, for our US friends). Standards, like COUNTER, solve that by ensuring everyone is measuring the same thing and can easily aggregate data in a simple, shared format.

My takeaway? If anything, the community needs a standard for usage metrics more than ever, and COUNTER is committed to evolving to meet the challenge of AI.