Clients read config from file resulting in EMFILE (too many open files) errors

Describe the bug

This applies to every auto-generated client, but I will use client-sqs as an example.

This is related to: #2271, #2027, #2993

---

Note that in the above snippet, loadNodeConfig is called multiple times to get the defaults for the client configuration:

https://github.com/aws/aws-sdk-js-v3/blob/bda3b40dd773511b8d5c84a07e9e158a70073523/clients/client-sqs/src/runtimeConfig.ts#L31-L53

Which is the exported fn loadConfig from @aws-sdk/node-config-provider package:

https://github.com/aws/aws-sdk-js-v3/blob/73923416ef2beaefbe7feda0473bc39c734be4b8/packages/node-config-provider/src/configLoader.ts#L27-L37

Which then uses fromSharedConfigFiles to load config values from the disk.

This is a very undesirable “feature”, especially in the serverless environments.

Because under heavy load this results in the following errors:

    error: {
      "type": "NodeError",
      "message": "A system error occurred: uv_os_homedir returned EMFILE (too many open files)",
      "stack":
          SystemError [ERR_SYSTEM_ERROR]: A system error occurred: uv_os_homedir returned EMFILE (too many open files)
              at Object.getHomeDir (/node_modules/@aws-sdk/shared-ini-file-loader/dist-cjs/index.js:82:17)
              at Object.loadSharedConfigFiles (/node_modules/@aws-sdk/shared-ini-file-loader/dist-cjs/index.js:11:89)
              at null.<anonymous> (/node_modules/@aws-sdk/node-config-provider/dist-cjs/fromSharedConfigFiles.js:9:53)
              at null.<anonymous> (/node_modules/@aws-sdk/property-provider/dist-cjs/chain.js:11:28)
              at runMicrotasks (<anonymous>)
              at processTicksAndRejections (internal/process/task_queues.js:95:5)
              at null.coalesceProvider (/node_modules/@aws-sdk/property-provider/dist-cjs/memoize.js:13:24)
              at Object.isConstant (/node_modules/@aws-sdk/property-provider/dist-cjs/memoize.js:24:28)
              at Object.getEndpointFromRegion (/node_modules/@aws-sdk/config-resolver/dist-cjs/endpointsConfig/utils/getEndpointFromRegion.js:12:34)
              at null.buildHttpRpcRequest (/node_modules/@aws-sdk/client-sqs/dist-cjs/protocols/Aws_query.js:2540:68)
      "code": "ERR_SYSTEM_ERROR",
      "info": {
        "errno": -24,
        "code": "EMFILE",
        "message": "too many open files",
        "syscall": "uv_os_homedir"
      },
      "errno": -24,
      "syscall": "uv_os_homedir"
    }

---

Yes the call is memoized, but if your Lambda is getting executed heavily, and installation happens within the handler, then this call happens multiple times.

Your environment

SDK version number

@aws-sdk/client-sqs@3.40.0

Is the issue in the browser/Node.js/ReactNative?

Node.js

Details of the browser/Node.js/ReactNative version

node -v v14.17.5

Steps to reproduce

import { SQS } from '@aws-sdk/client-sqs'

const sqs = new SQS({})

Observed behavior

    error: {
      "type": "NodeError",
      "message": "A system error occurred: uv_os_homedir returned EMFILE (too many open files)",
      "stack":
          SystemError [ERR_SYSTEM_ERROR]: A system error occurred: uv_os_homedir returned EMFILE (too many open files)
              at Object.getHomeDir (/node_modules/@aws-sdk/shared-ini-file-loader/dist-cjs/index.js:82:17)
              at Object.loadSharedConfigFiles (/node_modules/@aws-sdk/shared-ini-file-loader/dist-cjs/index.js:11:89)
              at null.<anonymous> (/node_modules/@aws-sdk/node-config-provider/dist-cjs/fromSharedConfigFiles.js:9:53)
              at null.<anonymous> (/node_modules/@aws-sdk/property-provider/dist-cjs/chain.js:11:28)
              at runMicrotasks (<anonymous>)
              at processTicksAndRejections (internal/process/task_queues.js:95:5)
              at null.coalesceProvider (/node_modules/@aws-sdk/property-provider/dist-cjs/memoize.js:13:24)
              at Object.isConstant (/node_modules/@aws-sdk/property-provider/dist-cjs/memoize.js:24:28)
              at Object.getEndpointFromRegion (/node_modules/@aws-sdk/config-resolver/dist-cjs/endpointsConfig/utils/getEndpointFromRegion.js:12:34)
              at null.buildHttpRpcRequest (/node_modules/@aws-sdk/client-sqs/dist-cjs/protocols/Aws_query.js:2540:68)
      "code": "ERR_SYSTEM_ERROR",
      "info": {
        "errno": -24,
        "code": "EMFILE",
        "message": "too many open files",
        "syscall": "uv_os_homedir"
      },
      "errno": -24,
      "syscall": "uv_os_homedir"
    }

Expected behavior

1. Do not error out
2. Do not read config from disk by default, or allow overriding this behaviour.

Screenshots

N/A

Additional context

N/A

[moltar]

If anyone stumbles on this, the only way I found to shut the file reading, is to do the following hack.

import * as sharedIniFileLoader from '@aws-sdk/shared-ini-file-loader'

Object.assign(sharedIniFileLoader, {
  loadSharedConfigFiles: async (): Promise<sharedIniFileLoader.SharedConfigFiles> => ({
    configFile: {},
    credentialsFile: {},
  }),
})

[adrai]

fyi: found an interesting package: https://github.com/samswen/lambda-emfiles

made some tests:

Some insights (same workload for all tests)

1) v3.46.0 without any workarounds:

Details

=> some leaks (around 230 emfiles)

2) v3.49.0 without any workarounds:

Details

=> some leaks and more emfiles (up to more than 600 emfiles)

3) v3.49.0 with loadSharedConfigFiles workaround:

Details

=> much less leaks, probably also less emfiles (up to more than 400 emfiles)

4) v3.46.0 with loadSharedConfigFiles workaround:

Details

=> much less leaks, much less emfiles (up to 180 emfiles)

So seems “4) v3.46.0 with loadSharedConfigFiles workaround” is the best!