Define ECS set of fields for SQL information

Different beats modules exist that are collecting SQL information (MySQL, PostgreSQL). APM also collects SQL information when available. Aligning the information by defining a common field set in ECS would help retrieving information from the data in a unified way.

Following fields are currently defined in APM and beats and might help as a starting point for discussions:

packetbeat

transaction_event fields: transaction_event event.dataset type method query

postgresql: pgsql.error_code pgsql.error_message pgsql.error_severity pgsql.num_fields pgsql.num_rows

mysql: mysql.affected_rows mysql.insert_id mysql.num_fields mysql.num_rows mysql.query mysql.error_code mysql.error_message

filebeat ecs compliant service.type event.module event.dataset

postgresql postgresql.log.timestamp postgresql.log.core_id postgresql.log.database postgresql.log.query postgresql.log.timezone postgresql.log.thread_id postgresql.log.user postgresql.log.level postgresql.log.message

mysql mysql.thread_id mysql.error.thread_id mysql.error.level mysql.error.message mysql.slowlog.lock_time.sec mysql.slowlog.rows_sent mysql.slowlog.rows_examined mysql.slowlog.rows_affected mysql.slowlog.bytes_sent mysql.slowlog.bytes_received mysql.slowlog.query mysql.slowlog.id mysql.slowlog.schema mysql.slowlog.current_user mysql.slowlog.last_errno mysql.slowlog.killed mysql.slowlog.query_cache_hit mysql.slowlog.tmp_table mysql.slowlog.tmp_table_on_disk mysql.slowlog.tmp_tables mysql.slowlog.tmp_disk_tables mysql.slowlog.tmp_table_sizes mysql.slowlog.filesort mysql.slowlog.filesort_on_disk mysql.slowlog.priority_queue mysql.slowlog.full_scan mysql.slowlog.full_join mysql.slowlog.merge_passes mysql.slowlog.sort_merge_passes mysql.slowlog.sort_range_count mysql.slowlog.sort_rows mysql.slowlog.sort_scan_count mysql.slowlog.log_slow_rate_type mysql.slowlog.log_slow_rate_limit mysql.slowlog.read_first mysql.slowlog.read_last mysql.slowlog.read_key mysql.slowlog.read_next mysql.slowlog.read_prev mysql.slowlog.read_rnd mysql.slowlog.read_rnd_next mysql.slowlog.innodb.trx_id mysql.slowlog.innodb.io_r_ops mysql.slowlog.innodb.io_r_bytes mysql.slowlog.innodb.io_r_wait.sec mysql.slowlog.innodb.rec_lock_wait.sec mysql.slowlog.innodb.queue_wait.sec mysql.slowlog.innodb.pages_distinct mysql.slowlog.user mysql.slowlog.host mysql.slowlog.ip

APM span.type span.subtype span.name (query summary) span.action span.db.statement span.db.type span.db.instance span.db.link span.db.user

[mbrancato]

Does an RFC exist for this request? I think this is important when you have applications which log entries that are effectively reporting out changes from a backend database of some type.

I like something flexible for multi-term DB concepts like table (most SQL) / collections (Mongo / Firestore) / index (Elasticsearch). For that I’d like something like these where certain child fields should be populated based on the type of DB:

{
  "db": {
    "type": "sql",
    "operation": "SELECT",
    "table": ["table_one", "table_two"]
  }
}

{
  "db": {
    "type": "elasticsearch",
    "operation": "_bulk",
    "index": ["my_index"]
  }
}

{
  "db": {
    "type": "mongo",
    "collection": ["my_collection"]
  }
}

[webmat]

Looking at OpenTelemetry for ideas and confirmation is a great idea 👍

Quick comment on the JSON sample above: I would avoid nesting the DB details inside span in the schema. From the APM context this database information is related to a span, sure. But in the general context of logging (including parsing DB access logs, slow logs or even replication logs), there’s no concept of a span.

So I’d rather have an APM event fill both field sets:

{
  "span": { 
    "id": "foo",
    "type": "db",
    ...
  },
  "db": {
    "type": "sql",
    "name": "SELECT",
    "statement": "",
    ...
  }
}

This will ensure users can correlate with other kinds of DB logs I’ve mentioned:

{
  "db": {
    "type": "sql",
    "name": "SELECT",
    "statement": "",
    ...
  }
}