Generative and Agentic AI: updates from our working group

What is COUNTER doing about generative and agentic AI and bots? This is a task that one of our new working groups has begun to look at.

At the first meeting of the new COUNTER Advisory Committee earlier this year, we spun out two working groups. One of these is on generative and agentic AI, and the other on pathways to compliance. In this post, we’ll outline progress in the AI working group.

The working group has agreed that it’s too early to make breaking changes to the Code of Practice in response to AI. Partly because the technology is changing so fast. Also because we committed to limiting the frequency of breaking changes to no more than once every four years – so not before 2030. Our goal is therefore to develop best practice guidance on AI usage that fits within the existing framework for extensions to the Code. As with other best practices, we’ll be consulting the whole community on the guidance before it comes into effect. Keep your eyes open for that towards the end of this year.

Where we started

As much as we’d like to get straight into defining how to measure AI usage, we first needed to re-define what constitutes an AI automated agent for COUNTER reporting purposes. That meant picking up Section 7.8 of the Code of Practice, which is our existing (outdated) description of bots.

COUNTER created both the bots list in Section 7.8 and the text and data mining (TDM) rules in Section 7.10 before the advent of AI. When we put those rules together, it was not easy for users to generate and activate agents or initiate large-scale data scraping from within institutional subscriptions. In the new age of AI, it’s increasingly difficult to distinguish between human and bot usage. There’s a much fuzzier line between the two. Beyond the challenge of distinguishing humans from human-initiated agents from traditional crawlers and bots, there’s also the question of separating legitimate, licensed, permission-based automated activities from unlicensed AI usage by users within an institution’s account. 

The first decision of the AI working group was therefore that COUNTER needs a more nuanced approach to reporting and exclusion for bots and automated agents. As part of that work, we’ll be bringing the old bots list up to date in a feature release (R5.1.1). We will also move it to a location that is under COUNTER’s control instead of its current third-party repository.

What about generative and agentic AI?

One straightforward scenario is publishers that offer AI-augmented search tools. These publishers return generated text but also deliver lists of relevant content in response to users’ prompts. As we’ve already said in the FAQs, this activity should be counted as a search metric. And, of course, if the user then clicks through to look at the original content, that would be an investigation and/or a request.

Our next steps are to understand the capabilities of current AI tools beyond that simple search scenario. We want to highlight reporting needs that the Code doesn’t capture. For example, what value and metrics can or should be placed on agentic usage? We are particularly interested in where that agent reduces human traffic to content? Ben Kaube and Steve Smith wrote an interesting article for Scholarly Kitchen on just that subject in July.

We have scheduled product demonstrations for our next meeting, at the end of September. These will help us start creating proposals around those usage metrics. For example, could we extend the Code to allow publishers to clearly pull out and report AI usage separately from human usage, potentially via a new optional Access_Method: Agent? 

Down the line there is also the issue of third-party tools that are using publisher materials to generate results, not to mention browser changes that may reduce traffic to publisher sites.

Get involved

We’ve already referenced the forthcoming consultation on the AI working group’s best practice guidance. If you’ve got a specific tool or scenario that you’d like us to consider in our development process, please email ai@countermetrics.org to let us know. And of course, keep an eye on our newsletters for updates.

The working group

The COUNTER team is very grateful for the enthusiasm and engagement of the AI working group members. Thank you to: Katie Arabie, Kirsten Fuoti, Rkki Gracia, Athena Hoeppner, Stuart Maxwell, Christopher Rennie, Michael Sisolak, Michelle Urberg, Monica Westin, Peter White, and Nico Zazworka.