GitHub API error `Validation Failed`

I was unaware of the check_run_annotations feature until I merged a PR that added this action, and I saw my job fail with this error:

2022-01-18 16:09:27 +0000 - publish-unit-test-results -  INFO - reading artifacts/**/*.xml
2022-01-18 16:09:29 +0000 - publish -  INFO - publishing success results for commit 3216c158b21d36f62d209672014e94160a428780
2022-01-18 16:09:31 +0000 - publish -  INFO - creating check
2022-01-18 16:09:33 +0000 - publish -  INFO - creating check
Traceback (most recent call last):
  File "/action/publish_unit_test_results.py", line 301, in <module>
    main(settings, gha)
  File "/action/publish_unit_test_results.py", line 102, in main
    Publisher(settings, gh, gha).publish(stats, results.case_results, conclusion)
  File "/action/publish/publisher.py", line 66, in publish
    check_run = self.publish_check(stats, cases, conclusion)
  File "/action/publish/publisher.py", line 242, in publish_check
    output=output)
  File "/usr/local/lib/python3.6/site-packages/github/Repository.py", line 3645, in create_check_run
    input=post_parameters,
  File "/usr/local/lib/python3.6/site-packages/github/Requester.py", line 355, in requestJsonAndCheck
    verb, url, parameters, headers, input, self.__customConnection(url)
  File "/usr/local/lib/python3.6/site-packages/github/Requester.py", line 378, in __check
    raise self.__createException(status, responseHeaders, output)
github.GithubException.GithubException: 422 {"message": "Validation Failed", "errors": [{"resource": "CheckRun", "code": "invalid", "field": "annotations"}], "documentation_url": "https://docs.github.com/rest/reference/checks#create-a-check-run"}

The repo I was working on has way more than 2000 tests, so I think it hit https://github.com/EnricoMi/publish-unit-test-result-action/blob/043296c976c53f4536194ebe3841ed720e04d496/python/publish/publisher.py#L238 more than once and tried to create a duplicate check_run, which the GitHub API didn’t allow.

I can see 50 annotations on the Unit Test Results check just fine, which has the first 2157 passing tests and 4 skipped tests listed.

[dgholz]

You must have very long test names to require that many annotations for those 2k tests

You got it, our test names are extremely long. It’s a private repo so I can’t share the list with you…

I’ll try with debug logging, and also try out the updated action.

[EnricoMi]

The limit of 50 annotations did not change, I found it: https://docs.github.com/en/rest/reference/checks#create-a-check-run

It is hidden under Properties of the output object.

The problem might be that I am creating all these checks, not creating the check with the first 50 annotations and then updating it with more annotations. Will test and revise this.