Identify subpar error messages

Chore summary

Although the overall usefulness of errors reported by Spectral has increased over the course of last months, there is still plenty of room for improvement.

Certain errors are still inconsistent, some of them are not particularly human-friendly, to say at least. Error messages that are not constant are primarily prone to be troublesome since there is a variety of cases to cover, therefore let’s focus on them - looking at you, schema function.

To help us identify flaws, we can leverage some existing repositories containing plenty of public APIs, such as APIs guru, and similar.

Then, ideally, in the context of Studio, let’s play around with specs using form or code view and watch errors produced in the diagnostics panel - see whether they make sense, they are easy to locate, and ideally, easy to resolve by a potentially non-technical person.

While testing the openapi documents and playing around making edits in studio forms/code view, document each subpar spectral error that you come across, along with as basic a spec document example as possible to reproduce the bad error.

Chore Scope

Let’s accumulate these bad error messages and an example reproduction spec for each one in this issue (we’ll use these example specs as test cases when we improve the errors). Fixing the issues is out of scope, we’ll create a separate issue once this is done. I suggest time boxing this exercise to a couple of hours.

[philsturgeon]

Here’s a not entirely helpful issue:

We need to override this to try and talk about the specific extra item that should not be there.

openapi: 3.0.0
info:
  title: Should Not Have What API
  version: '1.0'
  contact:
    email: [e-mail]
  description: >
    We get a "should NOT have addition properties" which points to `paths`, with no further information about the actual problem.

servers:
  - url: 'http://localhost/api/v1'

paths:
  regions/{id}':
    get:
      summary: Get region
      responses:
        '200':
          description: OK

Back in the day we used to have message and details, one of which was a short one-liner and the other was “more information”. We were inconsistent with which one was going to be the useful “show this in the listing” and which was going to contain specific information about the problem so we threw it away, but I would love to see a “More Information” come back.

It would be pretty useful to explain background information for rules, like we do in the ruleset docs over here https://stoplight.io/p/docs/gh/stoplightio/spectral/docs/reference/openapi-rules.md

But it would also be good to see the full error if people wanted to try and figure out what was going on. I’d rather see JSON-blobs of raw AJV output than be given nothing to work with (in Studio or anywhere else).

[jonsgold]

I don’t know if this has been identified yet. Using "required": true gives error oas3-schema 'requestBody' property should have required property '$ref':

{
  "components": {
    "schemas": {
      "association_by_id": {
        "oneOf": [{ "type": "string" }, { "type": "integer" }]
      },
      "association_by_name": {
        "type": "object",
        "oneOf": [{
          "type": "object",
          "required": ["name"],
          "properties": {
            "name": { "type": "string", "required": true }
          }
        },
        { "enum": ["nil", "", null] }]
      }
    }
  }
}