Regression: Event is not UTF-8 encoded when testing lambda locally

Description:

Passing a non-Latin body to sam local invoke produces a UnicodeEncodeError.

My guess is the changes from #2281 might be involved (cc @awood45).

Steps to reproduce:

Unfortunately I haven’t yet had time to produce a minimal repo but the command line I am using is:

sam local invoke "FunctionName" -e <path to below event json file>.json -n <path to test env>.json

Event JSON file:

{
  "Records": [
    {
      "messageId": "19dd0b57-b21e-4ac1-bd88-01bbb068cb78",
      "receiptHandle": "MessageReceiptHandle",
      "body": "[{\"type\":\"kanji\",\"deleted\":false,\"value\":{\"r\":{\"py\":[\"ya4\"],\"on\":[\"ア\"],\"kun\":[\"つ.ぐ\"],\"na\":[\"や\",\"つぎ\",\"つぐ\"]},\"m\":{\"en\":[\"Asia\",\"rank next\",\"come after\",\"-ous\"],\"fr\":[\"Asie\",\"suivant\",\"sub-\",\"sous-\"],\"es\":[\"pref. para indicar\",\"venir después de\",\"Asia\"],\"pt\":[\"Ásia\",\"próxima\",\"o que vem depois\",\"-ous\"]},\"rad\":{\"x\":7,\"nelson\":1},\"refs\":{\"nelson_c\":43,\"nelson_n\":81,\"halpern_njecd\":3540,\"halpern_kkd\":4354,\"halpern_kkld\":2204,\"halpern_kkld_2ed\":2966,\"heisig\":1809,\"heisig6\":1950,\"gakken\":1331,\"oneill_names\":525,\"oneill_kk\":1788,\"moro\":{\"vol\":1,\"page\":525,\"ref\":\"272\"},\"henshall\":997,\"sh_kk\":1616,\"sh_kk2\":1724,\"jf_cards\":1032,\"tutt_cards\":1092,\"kanji_in_context\":1818,\"kodansha_compact\":35,\"maniette\":1827,\"skip\":\"4-7-1\",\"sh_desc\":\"0a7.14\",\"four_corner\":\"1010.6\",\"deroo\":3273,\"conning\":545},\"misc\":{\"sc\":7,\"gr\":8,\"freq\":1509,\"jlpt\":1,\"kk\":25,\"jlptn\":1},\"c\":\"亜\",\"comp\":\"⼀⼁⼝\"}},{\"type\":\"kanji\",\"deleted\":false,\"value\":{\"r\":{\"py\":[\"ya1\"],\"on\":[\"ア\",\"アク\"],\"kun\":[\"おし\"]},\"m\":{\"en\":[\"mute\",\"dumb\"]},\"rad\":{\"x\":30},\"refs\":{\"nelson_c\":939,\"nelson_n\":795,\"heisig\":2958,\"heisig6\":2964,\"moro\":{\"vol\":2,\"page\":1066,\"ref\":\"3743\"},\"skip\":\"1-3-7\",\"sh_desc\":\"3d8.3\",\"four_corner\":\"6101.7\"},\"misc\":{\"sc\":10,\"kk\":15},\"c\":\"唖\",\"comp\":\"⼀⼁⼝⺉\"}},{\"type\":\"kanji\",\"deleted\":false,\"value\":{\"r\":{\"py\":[\"wa2\"],\"on\":[\"ア\",\"アイ\",\"ワ\"],\"kun\":[\"うつく.しい\"],\"na\":[\"い\"]},\"m\":{\"en\":[\"beautiful\"]},\"rad\":{\"x\":38},\"refs\":{\"nelson_n\":1213,\"halpern_kkd\":419,\"halpern_kkld_2ed\":313,\"heisig\":2200,\"heisig6\":2313,\"moro\":{\"vol\":3,\"page\":707,\"ref\":\"6262\"},\"skip\":\"1-3-6\",\"sh_desc\":\"3e6.5\",\"four_corner\":\"4441.4\"},\"misc\":{\"sc\":9,\"gr\":9,\"kk\":15},\"c\":\"娃\",\"comp\":\"⼟⼥\"}},{\"type\":\"kanji\",\"deleted\":false,\"value\":{\"r\":{\"py\":[\"a1\",\"e1\",\"a5\",\"a2\",\"a4\"],\"on\":[\"ア\",\"オ\"],\"kun\":[\"おもね.る\",\"くま\"],\"na\":[\"ほとり\",\"あず\",\"あわ\",\"おか\",\"きた\",\"な\"]},\"m\":{\"en\":[\"Africa\",\"flatter\",\"fawn upon\",\"corner\",\"nook\",\"recess\"],\"fr\":[\"flatter\",\"flagorner\",\"Afrique\",\"coin\",\"recoin\"],\"es\":[\"A fonética\",\"África\",\"meandro\",\"adular\",\"halagar\"]},\"rad\":{\"x\":170},\"refs\":{\"nelson_c\":4985,\"nelson_n\":6435,\"halpern_njecd\":346,\"halpern_kkd\":408,\"halpern_kkld\":256,\"halpern_kkld_2ed\":305,\"heisig\":1295,\"heisig6\":1391,\"gakken\":1515,\"oneill_names\":569,\"moro\":{\"vol\":11,\"page\":798,\"ref\":\"41599\"},\"sh_kk\":2258,\"maniette\":1304,\"skip\":\"1-3-5\",\"sh_desc\":\"2d5.6\",\"four_corner\":\"7122.0\",\"conning\":819},\"misc\":{\"sc\":8,\"gr\":9,\"freq\":1126,\"jlpt\":1,\"kk\":15,\"jlptn\":1},\"c\":\"阿\",\"comp\":\"⼅⼝⻖\"}},{\"type\":\"kanji\",\"deleted\":false,\"value\":{\"r\":{\"py\":[\"ai1\"],\"on\":[\"アイ\"],\"kun\":[\"あわ.れ\",\"あわ.れむ\",\"かな.しい\"]},\"m\":{\"en\":[\"pathetic\",\"grief\",\"sorrow\",\"pathos\",\"pity\",\"sympathize\"],\"fr\":[\"pitoyable\",\"peine\",\"chagrin\",\"pitié\",\"pathétique\",\"compatir\"],\"es\":[\"compasión\",\"lástima\",\"miseria\",\"piedad\",\"pena\",\"compadecerse de\"],\"pt\":[\"patético\",\"pesar\",\"pena\",\"emoção\",\"compaixão\",\"solidariesar\"]},\"rad\":{\"x\":30,\"nelson\":8},\"refs\":{\"nelson_c\":304,\"nelson_n\":791,\"halpern_njecd\":2068,\"halpern_kkd\":2568,\"halpern_kkld\":1310,\"halpern_kkld_2ed\":1781,\"heisig\":401,\"heisig6\":428,\"gakken\":1670,\"oneill_kk\":1249,\"moro\":{\"vol\":2,\"page\":997,\"ref\":\"3580\"},\"henshall\":998,\"sh_kk\":1675,\"sh_kk2\":1790,\"jf_cards\":1131,\"tutt_cards\":1239,\"kanji_in_context\":1804,\"kodansha_compact\":327,\"maniette\":408,\"skip\":\"2-2-7\",\"sh_desc\":\"2j7.4\",\"four_corner\":\"0073.2\",\"deroo\":465,\"conning\":1860},\"misc\":{\"sc\":9,\"gr\":8,\"freq\":1715,\"jlpt\":1,\"kk\":3,\"jlptn\":1},\"c\":\"哀\",\"comp\":\"⼇⼝⾐\"}},{\"type\":\"kanji\",\"deleted\":false,\"value\":{\"r\":{\"py\":[\"ai4\"],\"on\":[\"アイ\"],\"kun\":[\"いと.しい\",\"かな.しい\",\"め.でる\",\"お.しむ\",\"まな\"],\"na\":[\"あ\",\"あし\",\"え\",\"かな\",\"なる\",\"めぐ\",\"めぐみ\",\"よし\",\"ちか\"]},\"m\":{\"en\":[\"love\",\"affection\",\"favourite\"],\"fr\":[\"amour\",\"affection\",\"favori\"],\"es\":[\"amor\",\"afecto\",\"favorito\"],\"pt\":[\"amor\",\"afeição\",\"favorito\"]},\"rad\":{\"x\":61,\"nelson\":87},\"refs\":{\"nelson_c\":2829,\"nelson_n\":1927,\"halpern_njecd\":2492,\"halpern_kkd\":3133,\"halpern_kkld\":1606,\"halpern_kkld_2ed\":2191,\"heisig\":737,\"heisig6\":796,\"gakken\":436,\"oneill_names\":2018,\"oneill_kk\":456,\"moro\":{\"vol\":4,\"page\":1123,\"ref\":\"10947\"},\"henshall\":417,\"sh_kk\":259,\"sh_kk2\":259,\"sakade\":339,\"jf_cards\":545,\"henshall3\":441,\"tutt_cards\":602,\"crowley\":268,\"kanji_in_context\":1079,\"kodansha_compact\":790,\"maniette\":745,\"skip\":\"2-4-9\",\"sh_desc\":\"4i10.1\",\"four_corner\":\"2024.7\",\"deroo\":2067,\"skip_miss\":[{\"misclass\":\"posn\",\"ref\":\"2-1-12\"},{\"misclass\":\"posn\",\"ref\":\"2-6-7\"}],\"conning\":778},\"misc\":{\"sc\":13,\"gr\":4,\"freq\":640,\"jlpt\":2,\"kk\":7,\"jlptn\":3},\"c\":\"愛\",\"comp\":\"受⽖⼍⼡⼼\"}},{\"type\":\"kanji\",\"deleted\":false,\"value\":{\"r\":{\"py\":[\"ai1\",\"ai2\"],\"on\":[\"アイ\"],\"kun\":[\"ひら.く\"]},\"m\":{\"en\":[\"approach\",\"draw near\",\"push open\"],\"es\":[\"empujar\",\"acercarse\"]},\"rad\":{\"x\":64},\"refs\":{\"nelson_c\":1910,\"nelson_n\":2160,\"halpern_kkd\":510,\"halpern_kkld_2ed\":383,\"heisig\":2248,\"heisig6\":1310,\"moro\":{\"vol\":5,\"page\":229,\"ref\":\"12082\"},\"sh_kk2\":1101,\"skip\":\"1-3-7\",\"sh_desc\":\"3c7.12\",\"four_corner\":\"5303.4\",\"deroo\":1363,\"conning\":2247},\"misc\":{\"sc\":10,\"gr\":8,\"freq\":2258,\"kk\":2},\"c\":\"挨\",\"comp\":\"⼛⽮⺘\"}},{\"type\":\"kanji\",\"deleted\":false,\"value\":{\"r\":{\"py\":[\"yan3\"],\"on\":[\"オウ\",\"アイ\"],\"kun\":[\"あい\"]},\"m\":{\"en\":[\"good-looking\",\"quiet\"]},\"rad\":{\"x\":38},\"refs\":{\"nelson_n\":1208,\"oneill_names\":1091,\"moro\":{\"vol\":3,\"page\":696,\"ref\":\"6242\"},\"skip\":\"1-3-6\",\"sh_desc\":\"3e6.7\",\"four_corner\":\"4846.1\"},\"misc\":{\"sc\":9,\"kk\":15},\"c\":\"姶\",\"comp\":\"⼀⼝⼥𠆢\"}},{\"type\":\"kanji\",\"deleted\":false,\"value\":{\"r\":{\"py\":[\"feng2\"],\"on\":[\"ホウ\"],\"kun\":[\"あ.う\",\"むか.える\"],\"na\":[\"あい\",\"おう\"]},\"m\":{\"en\":[\"meeting\",\"tryst\",\"date\",\"rendezvous\"],\"es\":[\"encuentro\",\"cita\",\"encuentro casual\",\"encontrarse\",\"reunirse\",\"citarse\",\"verse por casualidad\"]},\"rad\":{\"x\":162},\"refs\":{\"nelson_c\":4694,\"nelson_n\":6054,\"halpern_kkd\":4002,\"halpern_kkld_2ed\":2774,\"heisig\":2417,\"heisig6\":2497,\"oneill_names\":1516,\"moro\":{\"vol\":11,\"page\":75,\"ref\":\"38901X\"},\"skip\":\"3-3-7\",\"sh_desc\":\"2q7.15\",\"four_corner\":\"3730.4\",\"deroo\":2555,\"skip_miss\":[{\"misclass\":\"stroke_diff\",\"ref\":\"3-4-7\"}],\"conning\":1379},\"misc\":{\"sc\":11,\"gr\":9,\"freq\":2116,\"kk\":15},\"c\":\"逢\",\"comp\":\"⼁⼆⼡⻍\",\"var\":[\"162-nyou-2\"]}},{\"type\":\"kanji\",\"deleted\":false,\"value\":{\"r\":{\"py\":[\"kui2\"],\"on\":[\"キ\"],\"kun\":[\"あおい\"],\"na\":[\"まもる\",\"け\"]},\"m\":{\"en\":[\"hollyhock\"],\"es\":[\"malva real\",\"malva loca\"]},\"rad\":{\"x\":140},\"refs\":{\"nelson_c\":3990,\"nelson_n\":5120,\"halpern_njecd\":2317,\"halpern_kkd\":2906,\"halpern_kkld\":1493,\"halpern_kkld_2ed\":2018,\"heisig\":2391,\"heisig6\":2472,\"oneill_names\":1723,\"moro\":{\"vol\":9,\"page\":806,\"ref\":\"31458X\"},\"sh_kk\":2222,\"skip\":\"2-3-9\",\"sh_desc\":\"3k9.17\",\"four_corner\":\"4443.0\"},\"misc\":{\"sc\":12,\"gr\":9,\"freq\":2122,\"jlpt\":1,\"kk\":15,\"jlptn\":1},\"c\":\"葵\",\"comp\":\"⼃⼆⼈⼤⽨⺾\"}}]",
      "attributes": {
        "ApproximateReceiveCount": "1",
        "SentTimestamp": "1523232000000",
        "SenderId": "123456789012",
        "ApproximateFirstReceiveTimestamp": "1523232000001"
      },
      "messageAttributes": {
        "Type": {
          "dataType": "String",
          "stringValue": "kanji"
        },
        "Timestamp": {
          "dataType": "Number",
          "stringValue": "1590969632292"
        }
      },
      "md5OfBody": "7b270e59b47ff90a553787216d55d91d",
      "eventSource": "aws:sqs",
      "eventSourceARN": "arn:aws:sqs:us-west-2:xxxxx",
      "awsRegion": "us-west-2"
    },
  ]
}

Observed result:

Stack trace

Mounting /home/runner/work/jpdict-sync/jpdict-sync/.aws-sam/build/UpdateDbFunction as /var/task:ro,delegated inside runtime container
Traceback (most recent call last):
  File "/opt/pipx_bin/sam", line 8, in <module>
    sys.exit(cli())
  File "/opt/pipx/venvs/aws-sam-cli/lib/python3.6/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/opt/pipx/venvs/aws-sam-cli/lib/python3.6/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/opt/pipx/venvs/aws-sam-cli/lib/python3.6/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/opt/pipx/venvs/aws-sam-cli/lib/python3.6/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/opt/pipx/venvs/aws-sam-cli/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/pipx/venvs/aws-sam-cli/lib/python3.6/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/opt/pipx/venvs/aws-sam-cli/lib/python3.6/site-packages/click/decorators.py", line 73, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
  File "/opt/pipx/venvs/aws-sam-cli/lib/python3.6/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/opt/pipx/venvs/aws-sam-cli/lib/python3.6/site-packages/samcli/lib/telemetry/metrics.py", line 100, in wrapped
    return func(*args, **kwargs)
  File "/opt/pipx/venvs/aws-sam-cli/lib/python3.6/site-packages/samcli/commands/local/invoke/cli.py", line 91, in cli
    parameter_overrides,
  File "/opt/pipx/venvs/aws-sam-cli/lib/python3.6/site-packages/samcli/commands/local/invoke/cli.py", line 158, in do_cli
    context.function_name, event=event_data, stdout=context.stdout, stderr=context.stderr
  File "/opt/pipx/venvs/aws-sam-cli/lib/python3.6/site-packages/samcli/commands/local/lib/local_lambda.py", line 111, in invoke
    self.local_runtime.invoke(config, event, debug_context=self.debug_context, stdout=stdout, stderr=stderr)
  File "/opt/pipx/venvs/aws-sam-cli/lib/python3.6/site-packages/samcli/local/lambdafn/runtime.py", line 101, in invoke
    container.wait_for_result(name=function_config.name, event=event, stdout=stdout, stderr=stderr)
  File "/opt/pipx/venvs/aws-sam-cli/lib/python3.6/site-packages/samcli/local/docker/container.py", line 241, in wait_for_result
    self.wait_for_http_response(name, event, stdout)
  File "/opt/pipx/venvs/aws-sam-cli/lib/python3.6/site-packages/samcli/lib/utils/retry.py", line 30, in wrapper
    return func(*args, **kwargs)
  File "/opt/pipx/venvs/aws-sam-cli/lib/python3.6/site-packages/samcli/local/docker/container.py", line 230, in wait_for_http_response
    timeout=(self.RAPID_CONNECTION_TIMEOUT, None),
  File "/opt/pipx/venvs/aws-sam-cli/lib/python3.6/site-packages/requests/api.py", line 119, in post
    return request('post', url, data=data, json=json, **kwargs)
  File "/opt/pipx/venvs/aws-sam-cli/lib/python3.6/site-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/opt/pipx/venvs/aws-sam-cli/lib/python3.6/site-packages/requests/sessions.py", line 530, in request
    resp = self.send(prep, **send_kwargs)
  File "/opt/pipx/venvs/aws-sam-cli/lib/python3.6/site-packages/requests/sessions.py", line 643, in send
    r = adapter.send(request, **kwargs)
  File "/opt/pipx/venvs/aws-sam-cli/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/opt/pipx/venvs/aws-sam-cli/lib/python3.6/site-packages/urllib3/connectionpool.py", line 677, in urlopen
    chunked=chunked,
  File "/opt/pipx/venvs/aws-sam-cli/lib/python3.6/site-packages/urllib3/connectionpool.py", line 392, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/lib/python3.6/http/client.py", line 1281, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib/python3.6/http/client.py", line 1326, in _send_request
    body = _encode(body, 'body')
  File "/usr/lib/python3.6/http/client.py", line 175, in _encode
    (name.title(), data[err.start:err.end], name)) from None
UnicodeEncodeError: 'latin-1' codec can't encode character '\u30a2' in position 228: Body ('ア') is not valid Latin-1. Use body.encode('utf-8') if you want to send it encoded in UTF-8.

As best I can tell, the issue appears to be the event is passed to requests.post (as introduced by 57ef28401ec58f7d79dca1835c4da27653c79bec) where it is assumed to be encoded as Latin-1.

Expected result:

This used to work without error (until presumably GitHub updated the version of sam-cli deployed on their Actions image).

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

1. OS: 18.04.5
2. sam --version: Whatever is currently deployed on GitHub actions

[hoffa]

This has been fixed in 1.17.0:

% sam --version
SAM CLI, version 1.17.0
% sam local invoke HelloWorldFunction -e events/event.json
Invoking app.lambda_handler (python3.8)
Skip pulling image and use local one: amazon/aws-sam-cli-emulation-image-python3.8:rapid-1.17.0.

Mounting /Users/rehnc/Desktop/tmp/sam-app/.aws-sam/build/HelloWorldFunction as /var/task:ro,delegated inside runtime container
START RequestId: c6276e1b-60ec-46a2-9e6c-ddc361ba2fc1 Version: $LATEST
END RequestId: c6276e1b-60ec-46a2-9e6c-ddc361ba2fc1
REPORT RequestId: c6276e1b-60ec-46a2-9e6c-ddc361ba2fc1	Init Duration: 0.13 ms	Duration: 86.39 ms	Billed Duration: 100 ms	Memory Size: 128 MB	Max Memory Used: 128 MB	
{"statusCode": 200, "body": "{\"message\": \"hello world\"}"}

[hoffa]

Thanks for reporting this!

As @eduardomourar mentioned, the issue is event being a str, which requests silently tries to encode to fallback latin-1, which in turn fails on the unsupported Unicode code points. Explicitly encoding to UTF-8 bytes fixes it.