COUNTER's bot repository

For many years, COUNTER’s list of bots and spiders was hosted in a GitHub repository that we didn’t control. As part of the fix-and-feature Release 5.1.1 of the Code of Practice, we’ve moved and expanded the repository. COUNTER Bots is now part of our own GitHub account.

Some things haven’t changed

• We’re still requiring crawler and bot activity to be stripped out so that only genuine, user-driven usage is included in COUNTER reports.
• Our repository is still a non-exclusive list that is open for extension. WYou can add a bot or crawler to the list by making a new pull request with your additions.
• And we still recommend case-insensitive matching, so that ‘bot’ will match ‘BOT’, ‘Bot’, ‘BoT’ etc.

So what’s new?

We’ve pulled in a much more comprehensive list of bots from the ai.robots.txt repository. The repo includes AI-related crawlers of all types, regardless of purpose. The longer list can be used to exclude AI usage from COUNTER reports, and to help identify AI usage in line with the best practice on Generative and Agentic AI usage metrics.