RFC: Ignore extra columns

Summary

Allow additional fields in REST inbound payloads.

Motivation

.NET developers serialize and deserialize JSON payloads into classes. Before this enhancement, DAB would require two different Request and Response payloads. The reason for this is that REST payloads to DAB could not include key and read-only fields. To solve this, .NET developers either 1) maintain two sets of POCO/DTO models - which is far from idea - or 2) manipulate the outbound payload dynamically - which is even further from ideal.

Old behavior

This demonstrates DAB behavior we want to change.

SQL Schema:

CREATE TABLE Customer
(
    Id INT PRIMARY KEY IDENTITY(1, 1),
    First VARCHAR(50),
    Last VARCHAR(50),
    Name AS CONCAT(First, ' ', 'Last)
)

POST (insert) {server}/API/customer

Request body:

{
    "Id": 1,
    "First": "Jerry",
    "Last": "Nixon",
}

Response body:

{
    "Id": 1,
    "First": "Jerry",
    "Last": "Nixon",
    "Name": "Jerry Nixon",
}

PUT (update) {server}/API/customer/Id/1

Request body:

{
    "First": "Jerry",
    "Last": "Smith"
}

Note: If either Id or Name were included in the Request body, DAB would return an error.

Response body:

{
    "Id": 1,
    "First": "Jerry",
    "Last": "Smith",
    "Name": "Jerry Smith",
}

New behavior

This demonstrates DAB behavior we want as a result of this enhancement.

SQL Schema:

CREATE TABLE Customer
(
    Id INT PRIMARY KEY IDENTITY(1, 1),
    First VARCHAR(50),
    Last VARCHAR(50),
    Name AS CONCAT(First, ' ', 'Last)
)

POST (insert) {server}/API/customer

Request body:

{
    "Id": 1,
    "First": "Jerry",
    "Last": "Nixon",
}

Response body:

{
    "Id": 1,
    "First": "Jerry",
    "Last": "Nixon",
    "Name": "Jerry Nixon",
}

PUT (update) {server}/API/customer/Id/1

Request body:

{
    "Id": 1,
    "First": "Jerry",
    "Last": "Smith",
    "Name": "Jerry Smith",
}

Note: Both Id and Name are included in the Request body. As they are primary key and read-only fields, DAB ignores them. No error is returned (please see configuration setting necessary for this behavior below).

Response body:

{
    "Id": 1,
    "First": "Jerry",
    "Last": "Smith",
    "Name": "Jerry Smith",
}

Functional Specifications

This enhancement only applies to the REST side of DAB.

The basics

1. This behavior is part of the .9 theme to remove adoption friction for enterprise developers.
2. DAB behavior does not change from current behavior without configuration flag (see below).

Desired behavior

1. Any JSON property that does not map to a database field is ignored. (PUT, POST, DELETE)
2. Any JSON property mapping to a read-only primary key is ignored.
3. Any JSON property referring to a read-only, computed column is ignored.

Ignoring read-only fields requires knowledge of the database object metadata.

Configuration change

Introduce request-body-strict to the dab-config.json file.

{
  "runtime": {
    "rest": {
      "enabled": true,
      "path": "/api",
      "request-body-strict":  false 👈
    },
  }
}

Value	Meaning
true	Extra fields are not allowed, results in error
false	Extra fields are allowed, ignored

Note: when the value is not present in the config file, it is the same as being present as true; that is, strict mode is on.

Normal behavior does not change

• If a key is read-write and I include it in the request payload, is the DB be updated during a PUT? Yes.
• If a key is read-write and I include it in the request payload, is the DB be updated during a POST? Yes.

Enhancement Q&A (when configured to not-strict)

• Can properties mapping to read-only keys (db-generated) be included in the Insert payloads? Yes.
• Can properties mapping to read-only keys (db-generated) be included in the Update payloads? Yes.
• What happens when properties mapping to read-only keys are included? They are ignored.
• Can properties mapping to read-only computed fields be included in the Insert payloads? Yes.
• Can properties mapping to read-only computed fields be included in the Update payloads? Yes.
• What happens when properties mapping to read-only computed fields are included? They are ignored.
• Can properties not mapping to any database field be included in the Insert payloads? Yes.
• Can properties not mapping to any database field be included in the Update payloads? Yes.
• What happens when properties not mapping to any database field are included? They are ignored.
• Does property data type change (e.g.: scalar, array, object) this behavior? No. Regular REST limitations apply.
• How many extra fields are supported? Is there a max? No. Regular REST limitations apply.
• Are ignored properties forwarded to the response payload? Never. They are ignored.
• Are ignored properties populated in the GET response payload? Never. They are ignored.
• Are ignored properties compared to the URL key value? Never. They are ignored.
• Can I bypass the need for metadata by indicating which fields are read-only somehow? No.

OpenAPI Updated

OpenAPI will need to be updated to reflect this change in payload requirements.

References

Discussion around this originally started here: #1453, #158

[seantleonard]

due to the limitations of .NET cli

Specifically, the package commandlineparser https://github.com/commandlineparser/commandline

[JerryNixon]

When strict mode = false, we will allow extraneous fields in the request body. So to speak, we won’t include them in the actual update/insert query that we generate because we cannot mutate a read-only field. As I mentioned, this can be implemented only for pg/mssql right now as for MySql we don’t have a solution yet.

Understood. While this is true, should we validate strict:false is incompatible with “mysql” in data-source.database-type?

"data-source" {
  "database-type": "mysql",
  "connection-string": ""
}

A question for PUT/PATCH requests @JerryNixon : In PUT/PATCH requests, the primary keys are allowed to be present only in the request URL. If we are operating with strict mode=false, do we want to allow the primary key fields to be present in the request body as well (and just ignore them)?

That’s right. Yes.

Also @JerryNixon, I am a bit confused with the examples PUT(insert) and POST(update) that you have provided. For PUT(insert), I don’t see PK being included in the url. Did you intend to include it but maybe missed it? For POST(update). - First, POST operation being used with phrase UPDATE seems a bit confusing. Also, the url contains the value of PK, however POST requests don’t allow any PKs in url.

Yep, I updated the original comment to correctly be post and put respectively. Good catch.