Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Get an warning: ResponseError: too_long_frame_exception

See original GitHub issue

Hi all, I encountered with a “too_long_frame_exception” warning when using scroll to fetch data from Elastic.

Issue

Here is the query to get data in 1 month:

                "gte": "2019-11-17T17:00:00.000Z",
                "lte": "2019-12-17T17:00:00.000Z"

GET band-monitor-*/_search
{
    "query": {
      "bool": {
        "must": [
          {
            "bool": {
              "should": [
                {
                  "match": {
                    "band_id": 118217
                  }
                },
                {
                  "match": {
                    "band_id": 118220
                  }
                },
                {
                  "match": {
                    "band_id": 118221
                  }
                }
              ]
            }
          },
          {
            "range": {
              "createdDate": {
                "gte": "2019-11-17T17:00:00.000Z",
                "lte": "2019-12-17T17:00:00.000Z"
              }
            }
          }
        ],
        "filter": {
          "script": {
            "script": {
              "source": "return doc['scan_id'].value % 10 == 0"
            }
          }
        }
      }
    },
    "sort": [
      {
        "scan_id": {
          "order": "asc"
        }
      }
    ],
    "aggs": {},
    "size": 9999
}

Run in Kibana and It returns 13384 records. I think this amount is not big at all. However when using npm @elastic/elasticsearch in nodejs, it shows an error:

{ ResponseError: too_long_frame_exception
    at IncomingMessage.response.on (/home/admin1/Documents/backend_v1/node_modules/@elastic/elasticsearch/lib/Transport.js:287:25)
    at IncomingMessage.emit (events.js:203:15)
    at IncomingMessage.EventEmitter.emit (domain.js:448:20)
    at endReadableNT (_stream_readable.js:1143:12)
    at process._tickCallback (internal/process/next_tick.js:63:19)
  name: 'ResponseError',
  meta:
   { body: { error: [Object], status: 400 },
     statusCode: 400,
     headers:
      { 'content-type': 'application/json; charset=UTF-8',
        'content-length': '211' },
     warnings: null,
     meta:
      { context: null,
        request: [Object],
        name: 'elasticsearch-js',
        connection: [Object],
        attempts: 0,
        aborted: false } } }

With a smaller range, it works fine. E.g: “gte”: “2019-11-17T17:00:00.000Z”, “lte”: “2019-11-27T17:00:00.000Z”

Expected behaviour

Just 20k data, this package should be working probably. I even try to use scroll with scroll time: 30s, size: 100. Please help to find any solution for this issue. Thank all.

Your Environment

Node version: 8.10
@elastic/elasticsearch version: ^7.5.0
OS: Ubuntu 16.04
ES cluster version: 6.8

Issue Analytics

State:
Created 4 years ago
Comments:5 (1 by maintainers)

Top GitHub Comments

5reactions

jdelriopscommented, Feb 8, 2021

In my case, that error happened because the API was trying to set the scroll_id in the queryString as part of the URL, and that has limit.

I solved the problem changing from:

response = await client.scroll({
  scrollId: my_scroll_id,
  scroll: scroll_params
})

To:

response = await client.scroll({
  method: 'POST',
  body: {
    scroll_id: my_scroll_id,
    scroll: scroll_params
  }
})

That way you can prevent the API code to try to set the scroll_id in the queryString, using the POST body instead.

0reactions

lijianlcommented, Jun 10, 2021

In my case, that error happened because the API was trying to set the scroll_id in the queryString as part of the URL, and that has limit.

I solved the problem changing from:
response = await client.scroll({
  scrollId: my_scroll_id,
  scroll: scroll_params
})
To:
response = await client.scroll({
  method: 'POST',
  body: {
    scroll_id: my_scroll_id,
    scroll: scroll_params
  }
})
That way you can prevent the API code to try to set the scroll_id in the queryString, using the POST body instead.