Harvesting multiple reports

We know that some report consumers (libraries) want to collect or harvest lots of reports at one time. That can cause some difficulties where report providers (publishers, aggregators) have rate limitations in place to prevent their COUNTER API (sushi) servers from being overwhelmed. In this post we hear from Bernd Oberknapp, one of COUNTER’s Executive Committee members and the brains behind our Validation Tool, about how he sets up report harvests to avoid this problem

“We request one report at a time, per platform, with a new report requested or queued every 3 seconds. To speed things up we also set up harvests to run on 25 to 30 platforms in parallel. That means a usage data host like Atypon or Scholarly iQ with lots of platforms may see a lot of requests from us at the same time, but so far this hasn’t caused any problems.”

Bernd and his team collect COUNTER usage reports monthly. This approach allows them to harvest approximately 20,000 reports within a few hours, even from usage data hosts offering thousands of reports covering different publisher platforms.

We asked Bernd how he manages Exception codes like 1010 or 1011, both of which recommend that the requestor tries again later.

“We retry after 10 minutes for Exception 1011 and some other Exceptions. Just as the CELUS team said in their post about COUNTER API Exceptions, it’s more of a problem when there is no Exception, for example when a usage data host takes a long time to return reports and blocks the request instead of returning Exception 1011.”

So now you know: request reports for different platforms in parallel, but only one at a time for each platform, and wait ten minutes before retrying if you receive an Exception. This is, of course, just a recommendation, but it should help you streamline your report harvesting and avoid common problems.