At a minimum, publishers and technology providers have to use the Validator as the first stage of the audit process. We recommend making validation a regular part of your workflow, either by integrating it into development quality assurance processes or by scheduling regular tests. We recommend testing reports and the COUNTER API once every four months at a minimum, and monthly where possible.
An Investigation might be an interaction with metadata (e.g. a book blurb or a video thumbnail), or use of a link such as one to inter-library loan.

We encourage librarians to pass reports through the Validator if they think something looks peculiar, or when they are onboarding a new publisher’s reports.

If you find a problem, please share the Validator result with the publisher in the first instance. If the problem persists, please copy tasha@countermetrics.org into future emails.

The most serious of the Validator messages, fatal errors indicate that the testing module had to stop testing because it found an unrecoverable error. Fatal errors have to be fixed before the report can be validated, and they indicate an automatic audit fail. These are structural issues with files, or logical inconsistencies – for example, where requested values are missing from the report. For example, if Data_Type Book is requested, but the Report_Filters field in the header does not include Data_Type=Book.

The three most common fatal errors in our sampling set were:

• Mis-matches between Institution_ID in the report header, and the Customer_ID calling the report.An Investigation might be an interaction with metadata (e.g. a book blurb or a video thumbnail), or use of a link such as one to inter-library loan.
• Unsupported file extensions. This happens where someone tries to validate a file that isn’t appropriate for COUNTER, such as a PDF or Word file.
• “Buffer is not a JSON object or array”, which means that the Validator is expecting to see a JSON array or object, but something else was returned (e.g. an HTML page).

Critical errors suggest serious problems that would usually result in an audit fail. They are often formatting issues, such as when columns in a tabular report are in the wrong order, or values are left empty. The other type of critical error are data inconsistencies, for example missing data or a Book with more Unique_Title_Requests than Unique_Title_Investigations.

The three most common critical errors in our sampling set were:

• Improperly formatted JSON files:
  • Missing Attribute_Performance property.
  • Item_ID being reported as a JSON array instead of an object.
  • Missing Data_Type property.
• Invalid Metric_Types, where someone has tried to include non-COUNTER metrics without properly namespacing them per the Extensions protocol.
• A Reporting_Period_Total that is either empty or does not match the sum of the month-by-month reporting.

Errors are unlikely to prevent a library from ingesting the report into their internal systems, but they do still cause problems and must be fixed to pass an audit. Examples of errors include silly mistakes like a report name not matching a report identifier in the header, or failing to add a namespace to proprietary identifiers.

The three most common errors in our sampling set were:

• Properties being incorrectly applied or displayed:
  • Data_Type being included in the Attributes_To_Show field in the Platform Report header. This was valid in R5, but because Data_Type is included by default in R5.1, it should no longer be included.
  • Access_Method, Access_Type, or YOP being included in the report body, but not listed in the Attributes_To_Show in the report header.
• Missing namespaces. Some fields must be shown as {namespace}:{value}, such as Publisher_ID.
• Invalid identifiers. For example, a non-DOI being included in the DOI field of a Title Report.

We’re now at the point where the testing module is picking up things that should be checked, but which are not immediately concerning. Warnings cover everything from the presence of Data_Type Unspecified, which the Code says should only be used  where there is not sufficient information to apply a more precise Data_Type, to missing Parent details in an Item Report. These are things that may be challenged in an audit, but are unlikely to result in failure.

The single most common warning is for incorrectly formatted ISBNs. Per our glossary:

“ISBN-13 MUST be formatted as follows: the string MUST include four hyphens, the first of which appears after the first three digits and the last of which appears before the last digit. There must be at least one digit between each of the four hyphens in the ISBN-13. ISBN-10 MUST NOT be used.”

Notices present additional information but will not affect the result of an audit. Notices commonly draw attention to Metric_Types which are valid for a report, but are not present. For example, a Title Report with no No_License counts will result in a notice from the Validator.

You can upload a COUNTER report in any of CSV, TSV, Excel, or JSON file formats. As noted above, both R5.1 and R5 reports are supported. You can upload several files at the same time, with the option to add a plain-text note to each file. The Validator will test and present results for each file separately.

As well as files, you can validate a COUNTER API response by providing the URL of the API endpoint and the corresponding credentials. The Validator is hooked up to the COUNTER Registry, so when you select Validate API you’re able to pre-populate key fields before testing. The Validator will guide you through the process of selecting the endpoint, potential filters, etc. Once the validation is complete, the results will be displayed in the same way as when validating a file. 

For the purposes of this guide, I validated 

• The sample reports from the Code in tabular and JSON format.
• The open /status endpoints for a variety of publishers.

After running these validations, my dashboard is populated with a list of the tests I’ve run, as well as the validation results. The results remain available for a week from the date of validation.

By clicking on the status icon (magnifying glass), I can jump through to see basic information about the validation, statistics, and the details of messages associated with the validation results. 

The Information Page

This is the landing page for a specific file or COUNTER API response validation. It starts at the top with basic metadata about the validation, including when it was run, the status of the validation task, what the results were, whether the results are visible, and any notes you have applied. Below that is the information the testing module extracted, such as which Release (CoP) is being tested. Lastly comes a rendering of the report header.

If you test a COUNTER API response instead of a file, you’ll see the same basic information, and then the details of the COUNTER API parameters that you requested. You’ll also see a Repeat Validation button. That will allow you to run another validation against the same server. It will set up a new validation with the same settings as the original one, which you can then modify as needed and run again.

The Statistics Page

The second tab of the validation results is a summary of the various errors found during testing. This is an example of a live test run by a publisher getting ready for audit. It’s been included here because it’s anonymous! As you can see, there are six different types of critical error, four different types of errors, one warning and a notice. As well as the severity and summary for each issue, the statistics page also shows the number of each type of issue as well as the option to jump to the relevant validation messages.

Validation Messages

Clicking through from the statistics page to just one of the critical errors, we can see the specific location of each issue. In this instance, the file included Unique_Title_Requests without showing Total_Item_Investigations, which is inconsistent with the Code of Practice.

You have two choices for sharing your validation results from the browser: you can export them as a tabular file, or hit the Share button from the validation information page. By using the Share feature, you’ll create a unique read-only link to the results, excluding any private information like COUNTER API credentials. Shared results remain available for 30 days, giving you plenty of time to collaborate with colleagues or with COUNTER. You can also choose to Hide the results once you’ve shared them.

Just as with browser-based validation, you can validate files and your COUNTER API setup. 

• For file validation, submit your report files via a POST request to https://validator.countermetrics.org/api/v1/validations/validation/file/.
• Alternatively, validate reports directly from a COUNTER API endpoint by sending a POST request to https://validator.countermetrics.org/api/v1/validations/validation/counter-api/ with the necessary URL and credential parameters.

You can retrieve validation results using a GET request to https://validator.countermetrics.org/api/v1/validations/validation/{id}/. The results will include all the validation messages for that validation id, including all of the information that you’d see in the browser-based tool.