Lightrun
question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Timestream client does not accept TIMESTAMP as MeasureValueType

See original GitHub issue

Describe the bug A clear and concise description of what the bug is.

Timestream accepts TIMESTAMP as MeasureValueType according to the docs: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/timestream-write.html#TimestreamWrite.Client.write_records

When writing a record with this value type, the write fails with

Traceback (most recent call last):

  File "/var/folders/j5/ym3yhfx510d4v_cnz9j6rmd80000gn/T/ipykernel_75705/577096492.py", line 1, in <module>
    _write_to_timestream(df)

  File "/Users/user/GitHub/folder/software/file.py", line 213, in _write_to_timestream
    result = client.write_records(DatabaseName="name",

  File "/opt/homebrew/Caskroom/mambaforge/base/envs/name/lib/python3.9/site-packages/botocore/client.py", line 391, in _api_call
    return self._make_api_call(operation_name, kwargs)

  File "/opt/homebrew/Caskroom/mambaforge/base/envs/name/lib/python3.9/site-packages/botocore/client.py", line 719, in _make_api_call
    raise error_class(parsed_response, operation_name)

ValidationException: An error occurred (ValidationException) when calling the WriteRecords operation: TIMESTAMP is an invalid measure value type.

Steps to reproduce If you have a runnable example, please include it as a snippet or link to a repository/gist for larger code examples.

Sample record:

[{'MeasureName': 'Test date', 'MeasureValueType': 'TIMESTAMP', 'MeasureValue': '1643987700000000000'}, {'MeasureName': 'Test value', 'MeasureValueType': 'DOUBLE', 'MeasureValue': '184532.6961294596385'}]

Sample Common Attributes:

{'Dimensions': [{'Name': 'name', 'Value': '0928.9274'}, {'Name': 'currency', 'Value': 'USD'}, {'Name': 'type', 'Value': 'cash'}], 'Time': '1643932800', 'TimeUnit': 'SECONDS'}

Sample call:

result = t_client.write_records(DatabaseName="db",
                                        TableName="table",
                                        Records=records_list,
                                        CommonAttributes=common_attributes)

(records_list is Sample record from above and common_attributes is Sample Common Attributes.)

Error:

Traceback (most recent call last):

  File "/var/folders/j5/ym3yhfx510d4v_cnz9j6rmd80000gn/T/ipykernel_75705/577096492.py", line 1, in <module>
    _write_to_timestream(df)

  File "/Users/user/GitHub/folder/software/file.py", line 213, in _write_to_timestream
    result = client.write_records(DatabaseName="name",

  File "/opt/homebrew/Caskroom/mambaforge/base/envs/name/lib/python3.9/site-packages/botocore/client.py", line 391, in _api_call
    return self._make_api_call(operation_name, kwargs)

  File "/opt/homebrew/Caskroom/mambaforge/base/envs/name/lib/python3.9/site-packages/botocore/client.py", line 719, in _make_api_call
    raise error_class(parsed_response, operation_name)

ValidationException: An error occurred (ValidationException) when calling the WriteRecords operation: TIMESTAMP is an invalid measure value type.

Expected behavior A clear and concise description of what you expected to happen.

The record should be successfully written to Timestream.

Debug logs Full stack trace by adding boto3.set_stream_logger('') to your code.

2022-02-08 17:12:18,836 botocore.hooks [DEBUG] Event before-parameter-build.timestream-write.WriteRecords: calling handler <bound method EndpointDiscoveryHandler.gather_identifiers of <botocore.discovery.EndpointDiscoveryHandler object at 0x1ba46f250>>
2022-02-08 17:12:18,837 botocore.hooks [DEBUG] Event before-parameter-build.timestream-write.WriteRecords: calling handler <function generate_idempotent_uuid at 0x10cb66dc0>
2022-02-08 17:12:18,838 botocore.hooks [DEBUG] Event before-call.timestream-write.WriteRecords: calling handler <function add_recursion_detection_header at 0x10cb66a60>
2022-02-08 17:12:18,838 botocore.hooks [DEBUG] Event before-call.timestream-write.WriteRecords: calling handler <function inject_api_version_header_if_needed at 0x10cb6d670>
2022-02-08 17:12:18,839 botocore.endpoint [DEBUG] Making request for OperationModel(name=WriteRecords) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'X-Amz-Target': 'Timestream_20181101.WriteRecords', 'Content-Type': 'application/x-amz-json-1.0', 'User-Agent': 'Boto3/1.20.50 Python/3.9.10 Darwin/21.3.0 Botocore/1.23.50'}, 'body': b'{"DatabaseName": "db", "TableName": "table", "Records": [{"MeasureName": "Test date", "MeasureValueType": "TIMESTAMP", "MeasureValue": "1643987700000000000"}, {"MeasureName": "Test value", "MeasureValueType": "DOUBLE", "MeasureValue": "184532.6961294596385"}], "CommonAttributes": {"Dimensions": [{"Name": "name", "Value": "0928.9274"}, {"Name": "currency", "Value": "USD"}, {"Name": "type", "Value": "cash"}], "Time": "1643932800", "TimeUnit": "SECONDS"}}', 'url': 'https://ingest.timestream.eu-central-1.amazonaws.com/', 'context': {'client_region': 'eu-central-1', 'client_config': <botocore.config.Config object at 0x1ba455790>, 'has_streaming_input': False, 'auth_type': None, 'discovery': {'identifiers': {}}}}
2022-02-08 17:12:18,840 botocore.hooks [DEBUG] Event request-created.timestream-write.WriteRecords: calling handler <bound method EndpointDiscoveryHandler.discover_endpoint of <botocore.discovery.EndpointDiscoveryHandler object at 0x1ba46f250>>
2022-02-08 17:12:18,840 botocore.discovery [DEBUG] Discovering endpoints with kwargs: {}
2022-02-08 17:12:18,841 botocore.hooks [DEBUG] Event before-parameter-build.timestream-write.DescribeEndpoints: calling handler <bound method EndpointDiscoveryHandler.gather_identifiers of <botocore.discovery.EndpointDiscoveryHandler object at 0x1ba46f250>>
2022-02-08 17:12:18,841 botocore.hooks [DEBUG] Event before-parameter-build.timestream-write.DescribeEndpoints: calling handler <function generate_idempotent_uuid at 0x10cb66dc0>
2022-02-08 17:12:18,841 botocore.hooks [DEBUG] Event before-call.timestream-write.DescribeEndpoints: calling handler <function add_recursion_detection_header at 0x10cb66a60>
2022-02-08 17:12:18,842 botocore.hooks [DEBUG] Event before-call.timestream-write.DescribeEndpoints: calling handler <function inject_api_version_header_if_needed at 0x10cb6d670>
2022-02-08 17:12:18,842 botocore.endpoint [DEBUG] Making request for OperationModel(name=DescribeEndpoints) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'X-Amz-Target': 'Timestream_20181101.DescribeEndpoints', 'Content-Type': 'application/x-amz-json-1.0', 'User-Agent': 'Boto3/1.20.50 Python/3.9.10 Darwin/21.3.0 Botocore/1.23.50', 'x-amz-api-version': '2018-11-01'}, 'body': b'{}', 'url': 'https://ingest.timestream.eu-central-1.amazonaws.com/', 'context': {'client_region': 'eu-central-1', 'client_config': <botocore.config.Config object at 0x1ba455790>, 'has_streaming_input': False, 'auth_type': None}}
2022-02-08 17:12:18,843 botocore.hooks [DEBUG] Event request-created.timestream-write.DescribeEndpoints: calling handler <bound method EndpointDiscoveryHandler.discover_endpoint of <botocore.discovery.EndpointDiscoveryHandler object at 0x1ba46f250>>
2022-02-08 17:12:18,843 botocore.hooks [DEBUG] Event request-created.timestream-write.DescribeEndpoints: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x1ba455520>>
2022-02-08 17:12:18,844 botocore.hooks [DEBUG] Event choose-signer.timestream-write.DescribeEndpoints: calling handler <function set_operation_specific_signer at 0x10cb66ca0>
2022-02-08 17:12:18,845 botocore.auth [DEBUG] Calculating signature using v4 auth.
2022-02-08 17:12:18,845 botocore.auth [DEBUG] CanonicalRequest:
POST
/

content-type:application/x-amz-json-1.0
host:ingest.timestream.eu-central-1.amazonaws.com
x-amz-api-version:2018-11-01
x-amz-date:20220208T161218Z
x-amz-target:Timestream_20181101.DescribeEndpoints

content-type;host;x-amz-api-version;x-amz-date;x-amz-target

2022-02-08 17:12:18,846 botocore.auth [DEBUG] StringToSign:
AWS4-HMAC-SHA256
20220208T161218Z
20220208/eu-central-1/timestream/aws4_request
2022-02-08 17:12:18,846 botocore.auth [DEBUG] Signature:
2022-02-08 17:12:18,847 botocore.hooks [DEBUG] Event request-created.timestream-write.DescribeEndpoints: calling handler <function add_retry_headers at 0x10cb6dd30>
2022-02-08 17:12:18,847 botocore.endpoint [DEBUG] Sending http request: <AWSPreparedRequest stream_output=False, method=POST, url=https://ingest.timestream.eu-central-1.amazonaws.com/, headers={'X-Amz-Target': b'Timestream_20181101.DescribeEndpoints', 'Content-Type': b'application/x-amz-json-1.0', 'User-Agent': b'Boto3/1.20.50 Python/3.9.10 Darwin/21.3.0 Botocore/1.23.50', 'x-amz-api-version': b'2018-11-01', 'X-Amz-Date': b'20220208T161218Z', 'Authorization': b'AWS4-HMAC-SHA256 Credential=//eu-central-1/timestream/aws4_request, SignedHeaders=content-type;host;x-amz-api-version;x-amz-date;x-amz-target, Signature=', 'amz-sdk-invocation-id': b'', 'amz-sdk-request': b'attempt=1', 'Content-Length': '2'}>
2022-02-08 17:12:18,848 botocore.httpsession [DEBUG] Certificate path: /usr/local/Caskroom/miniforge/base/envs/env_name/lib/python3.9/site-packages/certifi/cacert.pem
2022-02-08 17:12:18,849 urllib3.connectionpool [DEBUG] Starting new HTTPS connection (1): ingest.timestream.eu-central-1.amazonaws.com:443
2022-02-08 17:12:19,031 urllib3.connectionpool [DEBUG] https://ingest.timestream.eu-central-1.amazonaws.com:443 "POST / HTTP/1.1" 200 108
2022-02-08 17:12:19,033 botocore.parsers [DEBUG] Response headers: {'x-amzn-RequestId': '', 'Content-Type': 'application/x-amz-json-1.0', 'Content-Length': '108', 'Date': 'Tue, 08 Feb 2022 16:12:18 GMT'}
2022-02-08 17:12:19,033 botocore.parsers [DEBUG] Response body:
b'{"Endpoints":[{"Address":"ingest-cell1.timestream.eu-central-1.amazonaws.com","CachePeriodInMinutes":1440}]}'
2022-02-08 17:12:19,034 botocore.hooks [DEBUG] Event needs-retry.timestream-write.DescribeEndpoints: calling handler <botocore.retryhandler.RetryHandler object at 0x1ba46f160>
2022-02-08 17:12:19,034 botocore.retryhandler [DEBUG] No retry needed.
2022-02-08 17:12:19,035 botocore.hooks [DEBUG] Event needs-retry.timestream-write.DescribeEndpoints: calling handler <bound method EndpointDiscoveryHandler.handle_retries of <botocore.discovery.EndpointDiscoveryHandler object at 0x1ba46f250>>
2022-02-08 17:12:19,035 botocore.discovery [DEBUG] Injecting discovered endpoint: https://ingest-cell1.timestream.eu-central-1.amazonaws.com
2022-02-08 17:12:19,035 botocore.hooks [DEBUG] Event request-created.timestream-write.WriteRecords: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x1ba455520>>
2022-02-08 17:12:19,036 botocore.hooks [DEBUG] Event choose-signer.timestream-write.WriteRecords: calling handler <function set_operation_specific_signer at 0x10cb66ca0>
2022-02-08 17:12:19,036 botocore.auth [DEBUG] Calculating signature using v4 auth.
2022-02-08 17:12:19,037 botocore.auth [DEBUG] CanonicalRequest:
POST
/

content-type:application/x-amz-json-1.0
host:ingest-cell1.timestream.eu-central-1.amazonaws.com
x-amz-date:20220208T161219Z
x-amz-target:Timestream_20181101.WriteRecords

content-type;host;x-amz-date;x-amz-target
be666034f1f56562acb0c69508f237f5556c8cf93771d6a13cabefee515fba3d
2022-02-08 17:12:19,037 botocore.auth [DEBUG] StringToSign:
AWS4-HMAC-SHA256
20220208T161219Z
20220208/eu-central-1/timestream/aws4_request
2022-02-08 17:12:19,038 botocore.auth [DEBUG] Signature:
2022-02-08 17:12:19,038 botocore.hooks [DEBUG] Event request-created.timestream-write.WriteRecords: calling handler <function add_retry_headers at 0x10cb6dd30>
2022-02-08 17:12:19,038 botocore.endpoint [DEBUG] Sending http request: <AWSPreparedRequest stream_output=False, method=POST, url=https://ingest-cell1.timestream.eu-central-1.amazonaws.com, headers={'X-Amz-Target': b'Timestream_20181101.WriteRecords', 'Content-Type': b'application/x-amz-json-1.0', 'User-Agent': b'Boto3/1.20.50 Python/3.9.10 Darwin/21.3.0 Botocore/1.23.50', 'X-Amz-Date': b'20220208T161219Z', 'Authorization': b'AWS4-HMAC-SHA256 Credential=/eu-central-1/timestream/aws4_request, SignedHeaders=content-type;host;x-amz-date;x-amz-target, Signature=', 'amz-sdk-invocation-id': b', 'amz-sdk-request': b'attempt=1', 'Content-Length': '478'}>
2022-02-08 17:12:19,039 botocore.httpsession [DEBUG] Certificate path: /usr/local/Caskroom/miniforge/base/envs/env_name/lib/python3.9/site-packages/certifi/cacert.pem
2022-02-08 17:12:19,040 urllib3.connectionpool [DEBUG] Starting new HTTPS connection (1): ingest-cell1.timestream.eu-central-1.amazonaws.com:443
2022-02-08 17:12:19,201 urllib3.connectionpool [DEBUG] https://ingest-cell1.timestream.eu-central-1.amazonaws.com:443 "POST / HTTP/1.1" 400 123
2022-02-08 17:12:19,203 botocore.parsers [DEBUG] Response headers: {'x-amzn-RequestId': 'OF7ZHL7QMX4UL6RNBEOHDJGL6U', 'Content-Type': 'application/x-amz-json-1.0', 'Content-Length': '123', 'Date': 'Tue, 08 Feb 2022 16:12:19 GMT', 'Connection': 'close'}
2022-02-08 17:12:19,203 botocore.parsers [DEBUG] Response body:
b'{"__type":"com.amazonaws.timestream.v20181101#ValidationException","Message":"TIMESTAMP is an invalid measure value type."}'
2022-02-08 17:12:19,204 botocore.parsers [DEBUG] Response headers: {'x-amzn-RequestId': 'OF7ZHL7QMX4UL6RNBEOHDJGL6U', 'Content-Type': 'application/x-amz-json-1.0', 'Content-Length': '123', 'Date': 'Tue, 08 Feb 2022 16:12:19 GMT', 'Connection': 'close'}
2022-02-08 17:12:19,204 botocore.parsers [DEBUG] Response body:
b'{"__type":"com.amazonaws.timestream.v20181101#ValidationException","Message":"TIMESTAMP is an invalid measure value type."}'
2022-02-08 17:12:19,205 botocore.hooks [DEBUG] Event needs-retry.timestream-write.WriteRecords: calling handler <botocore.retryhandler.RetryHandler object at 0x1ba46f160>
2022-02-08 17:12:19,206 botocore.retryhandler [DEBUG] No retry needed.
2022-02-08 17:12:19,206 botocore.hooks [DEBUG] Event needs-retry.timestream-write.WriteRecords: calling handler <bound method EndpointDiscoveryHandler.handle_retries of <botocore.discovery.EndpointDiscoveryHandler object at 0x1ba46f250>>
Traceback (most recent call last):

  File "/var/folders/bh/vqvnzrw12913pc_4ny0bxyfh0000gn/T/ipykernel_35022/3734318343.py", line 1, in <module>
    t_client.write_records(DatabaseName="db",

  File "/usr/local/Caskroom/miniforge/base/envs/env_name/lib/python3.9/site-packages/botocore/client.py", line 391, in _api_call
    return self._make_api_call(operation_name, kwargs)

  File "/usr/local/Caskroom/miniforge/base/envs/env_name/lib/python3.9/site-packages/botocore/client.py", line 719, in _make_api_call
    raise error_class(parsed_response, operation_name)

ValidationException: An error occurred (ValidationException) when calling the WriteRecords operation: TIMESTAMP is an invalid measure value type.

Issue Analytics

  • State:closed
  • Created 2 years ago
  • Comments:10 (8 by maintainers)

github_iconTop GitHub Comments

1reaction
kdailycommented, Feb 8, 2022

Thanks @BastianZim that’s what I needed, I can see the full request and response. I might need to check in with the Timestream team to go further, will let you know what I learn.

0reactions
BastianZimcommented, Nov 22, 2022

Oh sorry, completely forgot this. Yes, all works and I’m closing.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Timestream client does not accept TIMESTAMP as ...
Can you try converting the timestamp value from a Date string to milliseconds, similar as you see on the time attribute? e.g.:
Read more >
Record - Amazon Timestream
Represents a time-series data point being written into Timestream. Each record contains an array of dimensions. Dimensions represent the metadata attributes ...
Read more >
TimestreamWrite — Boto3 Docs 1.26.34 documentation - AWS
Amazon Timestream is a fast, scalable, fully managed time series database service that ... uses a programming language that does not yet have...
Read more >
Time Series Databases - Bill, the Vest Guy
In a time-series database, however, the timestamp is the most important piece of information. ... Timestream does not take that approach.
Read more >
Amazon Timestream — 101
Timestream became generally available on AWS from Sep 30, 2020. ... This may simply happen because your timestamp is not of data type: ......
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found
Previous page

Add concept of "assume role credentials provider" from other AWS SDK implementations

Next page

ProxyConnectionError when specify username and password