question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Protocol error in kinesis putRecords in aws sdk > 3.81.0

See original GitHub issue

Describe the bug

node:internal/http2/util in new NghttpError at line 556:5

Your environment

SDK version number

        "@aws-sdk/client-dynamodb": "3.87.0",
        "@aws-sdk/client-kinesis": "3.88.0",
        "@aws-sdk/client-s3": "^3.88.0",
        "@aws-sdk/client-sqs": "^3.87.0",

Is the issue in the browser/Node.js/ReactNative?

Node.js

Details of the browser/Node.js/ReactNative version

Node.js v16.15.0

Steps to reproduce

Reproduces only in production 😦 Code:

    const usRequests: StatRequest[] = [];
    const euRequests: StatRequest[] = [];
    async function sendToKinesis(requests: StatRequest[], kinesis: KinesisClient, streamName: string) {
        if (requests.length >= flushAfter) {
            const requestsToSend = requests.splice(0);
            try {
                await kinesis.send(
                    new PutRecordsCommand({
                        StreamName: streamName,
                        Records: requestsToSend.map((data, idx) => ({
                            Data: textEncoder.encode(JSON.stringify(data)),
                            PartitionKey: idx.toString(),
                        })),
                    }),
                );
            } catch (err) {
                logger.error(err as Error);
                Sentry.captureException(err);
            }
        }
    }

await sendToKinesis(usRequests, usKinesis, "some-us-stream");
await sendToKinesis(euRequests, euKinesis, "some-eu-stream");

Observed behavior

Getting this error in 90% cases:

Error: Protocol error
  File "node:internal/http2/util", line 556, col 5, in new NghttpError
  File "node:internal/http2/core", line 785, col 26, in Http2Session.onSessionInternalError
  File "node:internal/async_hooks", line 130, col 17, in Http2Session.callbackTrampoline

Expected behavior

No error

Additional context

3.81 worked fine

Issue Analytics

  • State:closed
  • Created a year ago
  • Comments:16 (3 by maintainers)

github_iconTop GitHub Comments

1reaction
cesarfdcommented, May 23, 2022
  • Node 16.15.0.
  • Official Docker 16.15.0-alpine3.15 image.
  • x86-64
  • EKS AMI 1.21.5-20220406
  • aws-sdk 3.95.0
  • Writing to Kinesis Data Streams.

More or less how we write to Kinesis to give you guys an idea:

import { Kinesis } from '@aws-sdk/client-kinesis';

const kinesisClient = new Kinesis({ region: 'us-east-1' }); // Permissions, etc. inherited from instance

let retriedRecords = 0;

const kinesisRecords = [{
  PartitionKey: Math.floor(Math.random() * 100000000).toString(),
  Data: Buffer.from('some gzipped stuff')
}];

try {
  const response = await kinesisClient.putRecords({
    Records: kinesisRecords,
    StreamName: 'StreamName'
  });

  // Logging

  if (response.Records) {
    const recordsLength = kinesisRecords.length - 1;
    // Some of the records may be marked as failed, check if they can be retried
    for (let i = recordsLength; i >= 0; i -= 1) {
      const errorCode = response.Records[i].ErrorCode;

      if (errorCode) {
        // Put all the throttled/failed records back in the front of the list so they are retried later
        // Internal stuff
      }
    }
  }

  retriedRecords = response.FailedRecordCount ?? 0;
} catch (error) {
  if (error instanceof Error && error.message) {
    // Logging
  }

  // Put all the records back in the front of the list so they are retried later
  // Internal stuff
  retriedRecords = kinesisRecords.length;
}
1reaction
curtdeptcommented, May 19, 2022

@AllanZhengYP @up73k @L1fescape my issue appears to be stable and resolved. LGTM.

@gugu how about yours?

Read more comments on GitHub >

github_iconTop Results From Across the Web

Protocol error in kinesis putRecords in aws sdk > 3.81.0 #3780
Expected Behavior. putRecords() should work correctly. Current Behavior. Connection fails, error msg gives "Protocol error" . Reproduction Steps.
Read more >
Common Errors - Amazon Kinesis Data Streams Service
This section lists the errors common to the API actions of all AWS services. For errors specific to an API action for this...
Read more >
PutRecords - Amazon Kinesis Data Streams Service
Writes multiple data records into a Kinesis data stream in a single call (also referred to as a PutRecords request). Use this operation...
Read more >
kinesis - Amazon Web Services - Go SDK
Amazon Kinesis Data Streams is a managed service that scales elastically for real-time processing of streaming big data. See https://docs.aws.amazon.com/goto/ ...
Read more >
Troubleshooting Amazon Kinesis Data Streams Producers
If it's a mobile app, we recommend using the PutRecords operation or the Kinesis Recorder in the AWS Mobile SDKs. For more information,...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found