Lightrun
question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Different Elasticsearch errors end up with identical `error.grouping_key`

See original GitHub issue

Errors from Elasticsearch are grouped too aggressively. As an example I’ve attached two error documents which imo are separate errors but they end up with the same error.grouping_key.

Can the Node agent take other attributes like custom.reason or custom.type into account when calculating grouping_key for Elasticsearch errors?

Error #1

{
  "exception": [
    {
      "stacktrace": [...],
      "handled": true,
      "module": "@elastic/transport",
      "message": "resource_not_found_exception: [resource_not_found_exception] Reason: No known job with id 'security'",
      "type": "ResponseError"
    }
  ],
  "culprit": "request (/usr/share/kibana/node_modules/@elastic/transport/lib/Transport.js)",
  "custom": {
    "reason": "No known job with id 'security'",
    "type": "resource_not_found_exception",
    "status": 404
  },
  "id": "e99ca8f4ac40ea782c6b938b6b5aad62",
  "grouping_key": "21a1dd7bb6773c29a5f1820cccf05884",
  "grouping_name": "resource_not_found_exception: [resource_not_found_exception] Reason: No known job with id 'security'"
}
(full sample) 
{
  "_index": "metrics-gcp-europe-west1:.ds-logs-apm.error-default-2022.06.06-000004",
  "_id": "8uHzXIEBt4Z7xqdIl3UV",
  "_version": 1,
  "_score": 1,
  "_source": {
    "container": {
      "id": "aa03c576e12645394cccca57e167a5534d377086ff6dab5d04506087ef2516c0"
    },
    "parent": {
      "id": "88dcdd4381d46a8c"
    },
    "agent": {
      "name": "nodejs",
      "version": "3.31.0"
    },
    "error": {
      "exception": [
        {
          "stacktrace": [
            {
              "exclude_from_grouping": false,
              "library_frame": true,
              "abs_path": "/usr/share/kibana/node_modules/@elastic/transport/lib/Transport.js",
              "filename": "/usr/share/kibana/node_modules/@elastic/transport/lib/Transport.js",
              "line": {
                "number": 476,
                "context": "                    throw new errors_1.ResponseError(result);"
              },
              "function": "request",
              "context": {
                "pre": [
                  "                }",
                  "                if (!ignoreStatusCode && statusCode >= 400) {"
                ],
                "post": [
                  "                }",
                  "                else {"
                ]
              }
            },
            {
              "exclude_from_grouping": false,
              "library_frame": true,
              "line": {
                "number": 0
              },
              "function": "runMicrotasks"
            },
            {
              "exclude_from_grouping": false,
              "library_frame": true,
              "abs_path": "node:internal/process/task_queues",
              "filename": "node:internal/process/task_queues",
              "line": {
                "number": 96
              },
              "function": "processTicksAndRejections"
            }
          ],
          "handled": true,
          "module": "@elastic/transport",
          "message": "resource_not_found_exception: [resource_not_found_exception] Reason: No known job with id 'security'",
          "type": "ResponseError"
        }
      ],
      "culprit": "request (/usr/share/kibana/node_modules/@elastic/transport/lib/Transport.js)",
      "custom": {
        "reason": "No known job with id 'security'",
        "type": "resource_not_found_exception",
        "status": 404
      },
      "id": "e99ca8f4ac40ea782c6b938b6b5aad62",
      "grouping_key": "21a1dd7bb6773c29a5f1820cccf05884",
      "grouping_name": "resource_not_found_exception: [resource_not_found_exception] Reason: No known job with id 'security'"
    },
    "observer": {
      "hostname": "5d90685ff710",
      "name": "instance-0000000076",
      "id": "80c36a58-c8a0-4634-83ba-85533c2a0957",
      "type": "apm-server",
      "ephemeral_id": "e678b686-5c24-4b58-8f0b-398f82c9e78c",
      "version": "8.2.0"
    },
    "trace": {
      "id": "8fa23b158189083d940e278ae06d0ad5"
    },
    "ecs": {
      "version": "1.12.0"
    },
    "host": {
      "hostname": "aa03c576e126",
      "os": {
        "platform": "linux"
      },
      "name": "aa03c576e126",
      "architecture": "x64"
    },
    "client": {
      "ip": "10.43.0.188"
    },
    "event": {
      "agent_id_status": "missing",
      "ingested": "2022-06-13T12:04:36Z"
    },
    "user_agent": {
      "original": "Faraday v1.8.0",
      "name": "Other",
      "device": {
        "name": "Other"
      }
    },
    "timestamp": {
      "us": 1655121870692000
    },
    "process": {
      "args": [
        "/usr/share/kibana/node/bin/node",
        "/usr/share/kibana/src/cli/dist",
        "-c",
        "/app/config/kibana.yml"
      ],
      "pid": 17,
      "title": "/usr/share/kibana/bin/../node/bin/node",
      "ppid": 6
    },
    "data_stream.namespace": "default",
    "message": "resource_not_found_exception: [resource_not_found_exception] Reason: No known job with id 'security'",
    "data_stream.type": "logs",
    "processor": {
      "name": "error",
      "event": "error"
    },
    "url": {
      "path": "/api/stats",
      "original": "/api/stats?extended",
      "scheme": "https",
      "port": 9244,
      "query": "?extended",
      "domain": "c0fcde436702405394ed916bb8df0531.containerhost",
      "full": "https://c0fcde436702405394ed916bb8df0531.containerhost:9244/api/stats?extended"
    },
    "labels": {
      "organizationId": "2351180961",
      "subscriptionLevel": "standard",
      "deploymentId": "593cf9535b8e4ea58edcb1423119cd74",
      "git_rev": "9a5003d8cf0062bf24ef64d6712b44823888cc03",
      "deploymentName": "app-search-publishing-1655078343",
      "isElasticUser": "true"
    },
    "@timestamp": "2022-06-13T12:04:30.692Z",
    "service": {
      "node": {
        "name": "aa03c576e12645394cccca57e167a5534d377086ff6dab5d04506087ef2516c0"
      },
      "environment": "production",
      "framework": {
        "name": "hapi",
        "version": "20.2.1"
      },
      "name": "kibana",
      "runtime": {
        "name": "node",
        "version": "16.14.2"
      },
      "language": {
        "name": "javascript"
      },
      "version": "8.2.0"
    },
    "data_stream.dataset": "apm.error",
    "http": {
      "request": {
        "headers": {
          "X-Elastic-App-Auth": [
            "[REDACTED]"
          ],
          "Authorization": [
            "[REDACTED]"
          ],
          "X-Found-Kibana-Cluster": [
            "c0fcde436702405394ed916bb8df0531"
          ],
          "X-Forwarded-Host": [
            "c0fcde436702405394ed916bb8df0531.containerhost:9244"
          ],
          "X-Forwarded-Proto": [
            "https"
          ],
          "User-Agent": [
            "Faraday v1.8.0"
          ],
          "X-Forwarded-For": [
            "10.43.0.188"
          ],
          "Host": [
            "10.43.1.234:18239"
          ],
          "Accept-Encoding": [
            "gzip,deflate"
          ],
          "X-Cloud-Request-Id": [
            "heukrnyXR8Ko-h_eykUIdA"
          ],
          "Content-Type": [
            "application/json"
          ]
        },
        "method": "GET"
      },
      "response": {
        "status_code": 200,
        "finished": false,
        "headers_sent": false
      },
      "version": "1.1"
    },
    "user": {
      "name": "[REDACTED]"
    },
    "transaction": {
      "name": "GET /api/stats",
      "id": "8aacd98501f3b617",
      "type": "request",
      "sampled": true
    }
  }
}

Error #2

{
  "exception": [
    {
      "stacktrace": [...],
      "handled": true,
      "module": "@elastic/transport",
      "message": "index_not_found_exception: [index_not_found_exception] Reason: no such index [[.alerts-observability.metrics.alerts-default]]",
      "type": "ResponseError"
    }
  ],
  "culprit": "request (/usr/share/kibana/node_modules/@elastic/transport/lib/Transport.js)",
  "custom": {
    "reason": "no such index [[.alerts-observability.metrics.alerts-default]]",
    "type": "index_not_found_exception",
    "status": 404
  },
  "id": "35d46897e00e7d6405e341e1efa9f23e",
  "grouping_key": "21a1dd7bb6773c29a5f1820cccf05884",
  "grouping_name": "index_not_found_exception: [index_not_found_exception] Reason: no such index [[.alerts-observability.metrics.alerts-default]]"
}
(full sample) 
{
  "_index": "metrics-gcp-europe-west1:.ds-logs-apm.error-default-2022.06.06-000004",
  "_id": "x7nrXIEBh-x1qD5I2jU0",
  "_version": 1,
  "_score": 1,
  "_source": {
    "container": {
      "id": "a452d04443687d86c858e415476faf160696a20404798e346764e6e37408b577"
    },
    "parent": {
      "id": "fda0f5bc60de02d6"
    },
    "agent": {
      "name": "nodejs",
      "version": "3.31.0"
    },
    "error": {
      "exception": [
        {
          "stacktrace": [
            {
              "exclude_from_grouping": false,
              "library_frame": true,
              "abs_path": "/usr/share/kibana/node_modules/@elastic/transport/lib/Transport.js",
              "filename": "/usr/share/kibana/node_modules/@elastic/transport/lib/Transport.js",
              "line": {
                "number": 476,
                "context": "                    throw new errors_1.ResponseError(result);"
              },
              "function": "request",
              "context": {
                "pre": [
                  "                }",
                  "                if (!ignoreStatusCode && statusCode >= 400) {"
                ],
                "post": [
                  "                }",
                  "                else {"
                ]
              }
            },
            {
              "exclude_from_grouping": false,
              "library_frame": true,
              "line": {
                "number": 0
              },
              "function": "runMicrotasks"
            },
            {
              "exclude_from_grouping": false,
              "library_frame": true,
              "abs_path": "node:internal/process/task_queues",
              "filename": "node:internal/process/task_queues",
              "line": {
                "number": 96
              },
              "function": "processTicksAndRejections"
            }
          ],
          "handled": true,
          "module": "@elastic/transport",
          "message": "index_not_found_exception: [index_not_found_exception] Reason: no such index [[.alerts-observability.metrics.alerts-default]]",
          "type": "ResponseError"
        }
      ],
      "culprit": "request (/usr/share/kibana/node_modules/@elastic/transport/lib/Transport.js)",
      "custom": {
        "reason": "no such index [[.alerts-observability.metrics.alerts-default]]",
        "type": "index_not_found_exception",
        "status": 404
      },
      "id": "35d46897e00e7d6405e341e1efa9f23e",
      "grouping_key": "21a1dd7bb6773c29a5f1820cccf05884",
      "grouping_name": "index_not_found_exception: [index_not_found_exception] Reason: no such index [[.alerts-observability.metrics.alerts-default]]"
    },
    "observer": {
      "hostname": "db0eaab0ad15",
      "name": "instance-0000000074",
      "id": "2eb6b60d-92e7-41c6-9189-83cd3386e7ad",
      "type": "apm-server",
      "ephemeral_id": "0ad2df97-3df1-4db4-bbc8-e2d8132a63da",
      "version": "8.2.0"
    },
    "trace": {
      "id": "695b594c39c79da67fe1e2653e5f1078"
    },
    "ecs": {
      "version": "1.12.0"
    },
    "host": {
      "hostname": "a452d0444368",
      "os": {
        "platform": "linux"
      },
      "name": "a452d0444368",
      "architecture": "x64"
    },
    "client": {
      "geo": {
        "continent_name": "Europe",
        "region_iso_code": "NL-ZH",
        "city_name": "Oostvoorne",
        "country_iso_code": "NL",
        "country_name": "Netherlands",
        "region_name": "South Holland",
        "location": {
          "lon": 4.1008,
          "lat": 51.9094
        }
      },
      "ip": "157.97.192.70"
    },
    "event": {
      "agent_id_status": "missing",
      "ingested": "2022-06-13T11:56:09Z"
    },
    "user_agent": {
      "original": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.0.0 Safari/537.36",
      "os": {
        "name": "Mac OS X",
        "version": "10.15.7",
        "full": "Mac OS X 10.15.7"
      },
      "name": "Chrome",
      "device": {
        "name": "Mac"
      },
      "version": "102.0.0.0"
    },
    "timestamp": {
      "us": 1655121360526000
    },
    "process": {
      "args": [
        "/usr/share/kibana/node/bin/node",
        "/usr/share/kibana/src/cli/dist",
        "-c",
        "/app/config/kibana.yml"
      ],
      "pid": 17,
      "title": "/usr/share/kibana/bin/../node/bin/node",
      "ppid": 6
    },
    "data_stream.namespace": "default",
    "message": "index_not_found_exception: [index_not_found_exception] Reason: no such index [[.alerts-observability.metrics.alerts-default]]",
    "data_stream.type": "logs",
    "processor": {
      "name": "error",
      "event": "error"
    },
    "url": {
      "path": "/internal/bsearch",
      "original": "/internal/bsearch?compress=true",
      "scheme": "https",
      "port": 9243,
      "query": "?compress=true",
      "domain": "demo-pb.kb.europe-west1.gcp.cloud.es.io",
      "full": "https://demo-pb.kb.europe-west1.gcp.cloud.es.io:9243/internal/bsearch?compress=true"
    },
    "labels": {
      "organizationId": "4057428689",
      "subscriptionLevel": "standard",
      "plugin": "apm",
      "deploymentId": "d5b1babac4cc4dd4bedf24199dbaf425",
      "name": "observability-overview",
      "git_rev": "22fe96e5ac37d9f3bc633036d4173915877314a1",
      "deploymentName": "Demo_PB",
      "entityId": "reports:monitor",
      "isElasticUser": "true"
    },
    "@timestamp": "2022-06-13T11:56:00.526Z",
    "service": {
      "node": {
        "name": "a452d04443687d86c858e415476faf160696a20404798e346764e6e37408b577"
      },
      "environment": "production",
      "framework": {
        "name": "hapi",
        "version": "20.2.2"
      },
      "name": "kibana",
      "runtime": {
        "name": "node",
        "version": "16.14.2"
      },
      "language": {
        "name": "javascript"
      },
      "version": "8.2.2"
    },
    "data_stream.dataset": "apm.error",
    "http": {
      "request": {
        "headers": {
          "Cookie": [
            "fs_uid=rs.fullstory.com#G3PDG#6654877577777152:5783167507238912#a0620e58#/1686657205; sid=Fe26.2**f9f85de66b8839c7245119809ecaf0dba466fc91ffa807812b17cfc0d4ea2be2*sGJtaFHmMk491klODbe5XA*LfzUQQv_lz5BnQubxlFP7JkmG_Xta3U2dYW7DrIpNoQ-W_czeHkxwH0xe1uMOLWVHxLyF480NbrZxV4TL2zX5u0QLY4Ieu3015yyzqMEfONpfPejUS2QeGPQtS7T1ErAONngi5TOi5CJ0fnDSiDN6QIToB9FVqaaCrJsnjmnBpND1k0O1AFDOEUUD5ETb7wmH3AKOMxfB1KiHq_nKVkVkBji2CE8UY8kuqiiZz4MhYbhFLDpOZf_DUlwoO8wIu2Z**08f791d297fa454b9fbd09212ae03d62baa975db45c7e574ee1a62ee190c03ac*zGLMf5fy901OW81D4yYs_bh09mcko3PTFGqT5BvqlkY"
          ],
          "Origin": [
            "https://demo-pb.kb.europe-west1.gcp.cloud.es.io:9243"
          ],
          "Sec-Ch-Ua": [
            "\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"102\", \"Google Chrome\";v=\"102\""
          ],
          "Accept": [
            "*/*"
          ],
          "Kbn-Version": [
            "8.2.2"
          ],
          "X-Found-Kibana-Cluster": [
            "10ad64d7633844da9aa8f41c050de44d"
          ],
          "Sec-Ch-Ua-Platform": [
            "\"macOS\""
          ],
          "User-Agent": [
            "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.0.0 Safari/537.36"
          ],
          "X-Forwarded-Host": [
            "demo-pb.kb.europe-west1.gcp.cloud.es.io:9243"
          ],
          "Referer": [
            "https://demo-pb.kb.europe-west1.gcp.cloud.es.io:9243/app/observability/overview?rangeFrom=now-7d&rangeTo=now"
          ],
          "X-Forwarded-Proto": [
            "https"
          ],
          "Sec-Fetch-Dest": [
            "empty"
          ],
          "Sec-Fetch-Site": [
            "same-origin"
          ],
          "Host": [
            "10.43.255.177:18434"
          ],
          "Accept-Encoding": [
            "gzip, deflate, br"
          ],
          "X-Cloud-Request-Id": [
            "R_HsSfb2RO2jaD-vrI8Tdg"
          ],
          "Traceparent": [
            "00-695b594c39c79da67fe1e2653e5f1078-0a0af9e48773a72a-01"
          ],
          "Sec-Fetch-Mode": [
            "cors"
          ],
          "Authorization": [
            "[REDACTED]"
          ],
          "X-Forwarded-For": [
            "157.97.192.70"
          ],
          "Accept-Language": [
            "en-GB,en-US;q=0.9,en;q=0.8"
          ],
          "Content-Length": [
            "375"
          ],
          "Sec-Ch-Ua-Mobile": [
            "?0"
          ],
          "Content-Type": [
            "application/json"
          ]
        },
        "method": "POST"
      },
      "response": {
        "headers": {
          "Referrer-Policy": [
            "no-referrer-when-downgrade"
          ],
          "Transfer-Encoding": [
            "chunked"
          ],
          "Cache-Control": [
            "private, no-cache, no-store, must-revalidate"
          ],
          "X-Content-Type-Options": [
            "nosniff"
          ],
          "Kbn-License-Sig": [
            "8fe48478c8212ef9ec33bc2c8333d48b9e269eb278bfb3e40b4698ecff22314b"
          ],
          "Connection": [
            "keep-alive"
          ],
          "Set-Cookie": [
            "[REDACTED]"
          ],
          "Date": [
            "Mon, 13 Jun 2022 11:56:00 GMT"
          ],
          "Content-Type": [
            "application/x-ndjson"
          ],
          "Kbn-Name": [
            "a452d0444368"
          ]
        },
        "status_code": 200,
        "finished": false,
        "headers_sent": true
      },
      "version": "1.1"
    },
    "transaction": {
      "name": "POST /internal/bsearch",
      "id": "e37d9a4d8bd1f0fa",
      "type": "request",
      "sampled": true
    }
  }
}

Issue Analytics

  • State:closed
  • Created a year ago
  • Comments:12 (8 by maintainers)

github_iconTop GitHub Comments

1reaction
felixbarnycommented, Jun 17, 2022

I’d gravitate towards the second checklist for this. Whenever we want to track the implementation progress cross-agent, we should use the second checklist. Every agent can still prioritize the implementation as they see fit or close the issue if they don’t plan on implementing it at all.

1reaction
felixbarnycommented, Jun 13, 2022

Can the Node agent take other attributes like custom.reason or custom.type into account when calculating grouping_key for Elasticsearch errors?

The issue is that the grouping key is not calculated within the agent but in APM Server.

However, what might be possible is to set a custom type in the agent if it detects the error to be an Elasticsearch ResponseError. So instead of

"type": "ResponseError"

the Node.js agent might be able to set

"type": "ResponseError (resource_not_found_exception)"
Read more comments on GitHub >

github_iconTop Results From Across the Web

Incorrect Error Grouping - APM - Discuss the Elastic Stack
It seems the errors that get grouped together have the same error class ... Some of the errors are being grouped correctly (different...
Read more >
How to solve 8 common Elasticsearch errors and exceptions
Common Elasticsearch errors and exceptions and how to avoid them! Including best practices to help identify, minimize, and handle ES issues.
Read more >
Elasticsearch integration - GitLab Docs
class of Gitlab::Elastic::Indexer::Error . These errors may occur when indexing Git repository data. Enable Advanced Search. For GitLab instances with more than ...
Read more >
Finding a tricky bug in Elasticsearch 8.4.2 - alexwlchan
Every bug is different, but the same techniques appear again and again. ... As the errors stacked up in Slack, the meaning was...
Read more >
Elasticsearch 6.7 _reindex operation inconsistently reports ...
value field of a particular incoming document should be float , even though for other documents coming in the same batch, it has...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found
Previous page

support @koa/router@11 and @koa/router@12

Next page

External requests are not always tracked by APM in Lambdas