AWS : The security token included in the request is expired

Hi,

Having a AWS Managed 5.1 ElasticSearch and running ElastAlert on an EC2 instances that have IAM role allowing ES access. The service starts and monitor correctly for a certain amount of time, then starts gettings 403 error with expired token.

Can you please help me with this issue ?

Here is my config file :

rules_folder: rules
run_every:
  minutes: 1
buffer_time:
  minutes: 15
es_host: search-servicename-environment-elk-51-uniqueid.us-east-1.es.amazonaws.com
es_port: 80
aws_region: us-east-1
use_ssl: False
writeback_index: elastalert_status
alert_time_limit:
  days: 2

Here are the logs :

Creating Elastalert index in Elasticsearch...
Index elastalert_status already exists. Skipping index creation.
INFO:elastalert:Starting up
INFO:elastalert:Sleeping for 59 seconds
... repeated 395 times ...
INFO:elastalert:Sleeping for 59 seconds
WARNING:elasticsearch:GET http://search-prd00000280-nonprod1-elk-51-uppu56qlo57axx4ncafuozjk3e.us-east-1.es.amazonaws.com:80/elastalert_status/elastalert/_search?size=1000 [status:403 request:0.043s]
ERROR:root:Error finding recent pending alerts: TransportError(403, u'{"message":"The security token included in the request is expired"}') {'sort': {'alert_time': {'order': 'asc'}}, 'query': {'bool': {'filter': {'range': {'alert_time': {'to': '2017-04-04T19:37:35.417433Z', 'from': '2017-04-02T19:37:35.417402Z'}}}, 'must': {'query_string': {'query': '!_exists_:aggregate_id AND alert_sent:false'}}}}}
Traceback (most recent call last):
  File "elastalert/elastalert.py", line 1092, in find_recent_pending_alerts
    size=1000)
  File "build/bdist.linux-x86_64/egg/elasticsearch/client/utils.py", line 69, in _wrapped
    return func(*args, params=params, **kwargs)
  File "build/bdist.linux-x86_64/egg/elasticsearch/client/__init__.py", line 539, in search
    doc_type, '_search'), params=params, body=body)
  File "build/bdist.linux-x86_64/egg/elasticsearch/transport.py", line 327, in perform_request
    status, headers, data = connection.perform_request(method, url, params, body, ignore=ignore, timeout=timeout)
  File "build/bdist.linux-x86_64/egg/elasticsearch/connection/http_requests.py", line 84, in perform_request
    self._raise_error(response.status_code, raw_data)
  File "build/bdist.linux-x86_64/egg/elasticsearch/connection/base.py", line 114, in _raise_error
    raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, error_message, additional_info)
AuthorizationException: TransportError(403, u'{"message":"The security token included in the request is expired"}')

Thanks Guillaume

[micahhausler]

OK, I can verify that my PR does not fix this. I’m getting the same message for a cross-account role. I believe this is due to the fact that the Session is calling session.get_credentials().get_frozen_credentials(), and never issuing a refresh. I’ll make a patch for this

[micahhausler]

@BardiaAfshin this has been released. I don’t have any sample code, the AWS library should just pick up that you’re using an IAM role.