AWS Elasticsearch keeps giving 403 unathorized

Here is my config.yaml

# This is the folder that contains the rule yaml files
# Any .yaml file will be loaded as a rule
rules_folder: example_rules

# How often ElastAlert will query elasticsearch
# The unit can be anything from weeks to seconds
run_every:
  minutes: 1

# ElastAlert will buffer results from the most recent
# period of time, in case some log sources are not in real time
buffer_time:
  minutes: 15

# The elasticsearch hostname for metadata writeback
# Note that every rule can have it's own elasticsearch host
es_host: search-itmcc-elasticsearch-cluster-47ux565rtlkza7gewdodwdh7b4.us-west-2.es.amazonaws.com

# The elasticsearch port
es_port: 80

aws_region: us-west-2

#boto_profile: default

# Optional URL prefix for elasticsearch
es_url_prefix: elasticsearch

# Connect with SSL to elasticsearch
#use_ssl: True

# Option basic-auth username and password for elasticsearch
#es_username: someusername
#es_password: somepassword

# The index on es_host which is used for metadata storage
# This can be a unmapped index, but it is recommended that you run
# elastalert-create-index to set a mapping
writeback_index: elastalert_status

# If an alert fails for some reason, ElastAlert will retry
# sending the alert until this time period has elapsed
alert_time_limit:
  days: 2

And the error that I get:

WARNING:elasticsearch:GET /elasticsearch/elastalert_status/elastalert/_search?size=1000 [status:403 request:0.015s]
WARNING:elasticsearch:GET /elasticsearch/elastalert_status/elastalert_status/_search?_source_include=endtime%2Crule_name&size=1 [status:403 request:0.005s]
ERROR:root:Error querying for last run: TransportError(403, u'{"Message":"User: anonymous is not authorized to perform: es:ESHttpGet on resource: itmcc-elasticsearch-cluster"}')
WARNING:elasticsearch:POST /elasticsearch/elastalert_status/elastalert_error?op_type=create [status:403 request:0.003s]
ERROR:root:Error writing alert info to elasticsearch: TransportError(403, u'{"Message":"User: anonymous is not authorized to perform: es:ESHttpPost on resource: itmcc-elasticsearch-cluster"}')
Traceback (most recent call last):
  File "elastalert/elastalert.py", line 998, in writeback
    doc_type=doc_type, body=body)
  File "build/bdist.linux-x86_64/egg/elasticsearch/client/utils.py", line 69, in _wrapped
    return func(*args, params=params, **kwargs)
  File "build/bdist.linux-x86_64/egg/elasticsearch/client/__init__.py", line 248, in create
    return self.index(index, doc_type, body, id=id, params=params, op_type='create')
  File "build/bdist.linux-x86_64/egg/elasticsearch/client/utils.py", line 69, in _wrapped
    return func(*args, params=params, **kwargs)
  File "build/bdist.linux-x86_64/egg/elasticsearch/client/__init__.py", line 279, in index
    _make_path(index, doc_type, id), params=params, body=body)
  File "build/bdist.linux-x86_64/egg/elasticsearch/transport.py", line 329, in perform_request
    status, headers, data = connection.perform_request(method, url, params, body, ignore=ignore, timeout=timeout)
  File "build/bdist.linux-x86_64/egg/elasticsearch/connection/http_urllib3.py", line 109, in perform_request
    self._raise_error(response.status, raw_data)
  File "build/bdist.linux-x86_64/egg/elasticsearch/connection/base.py", line 108, in _raise_error
    raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, error_message, additional_info)
AuthorizationException: TransportError(403, u'{"Message":"User: anonymous is not authorized to perform: es:ESHttpPost on resource: itmcc-elasticsearch-cluster"}')
WARNING:elasticsearch:GET /logstash-*/_search?_source_include=%40timestamp%2C%2A&ignore_unavailable=true&size=10000 [status:403 request:0.006s]
ERROR:root:Error running query: TransportError(403, u'{"Message":"User: anonymous is not authorized to perform: es:ESHttpGet on resource: itmcc-elasticsearch-cluster"}')
WARNING:elasticsearch:POST /elasticsearch/elastalert_status/elastalert_error?op_type=create [status:403 request:0.004s]
ERROR:root:Error writing alert info to elasticsearch: TransportError(403, u'{"Message":"User: anonymous is not authorized to perform: es:ESHttpPost on resource: itmcc-elasticsearch-cluster"}')
Traceback (most recent call last):
  File "elastalert/elastalert.py", line 998, in writeback
    doc_type=doc_type, body=body)
  File "build/bdist.linux-x86_64/egg/elasticsearch/client/utils.py", line 69, in _wrapped
    return func(*args, params=params, **kwargs)
  File "build/bdist.linux-x86_64/egg/elasticsearch/client/__init__.py", line 248, in create
    return self.index(index, doc_type, body, id=id, params=params, op_type='create')
  File "build/bdist.linux-x86_64/egg/elasticsearch/client/utils.py", line 69, in _wrapped
    return func(*args, params=params, **kwargs)
  File "build/bdist.linux-x86_64/egg/elasticsearch/client/__init__.py", line 279, in index
    _make_path(index, doc_type, id), params=params, body=body)
  File "build/bdist.linux-x86_64/egg/elasticsearch/transport.py", line 329, in perform_request
    status, headers, data = connection.perform_request(method, url, params, body, ignore=ignore, timeout=timeout)
  File "build/bdist.linux-x86_64/egg/elasticsearch/connection/http_urllib3.py", line 109, in perform_request
    self._raise_error(response.status, raw_data)
  File "build/bdist.linux-x86_64/egg/elasticsearch/connection/base.py", line 108, in _raise_error
    raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, error_message, additional_info)
AuthorizationException: TransportError(403, u'{"Message":"User: anonymous is not authorized to perform: es:ESHttpPost on resource: itmcc-elasticsearch-cluster"}')

My IAM role attached to instance with elastalert gives full access to es (elasticsearch):

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Stmt1454694581000",
            "Effect": "Allow",
            "Action": [
                "cloudwatch:*"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Sid": "Stmt1454694609000",
            "Effect": "Allow",
            "Action": [
                "events:*"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Sid": "Stmt1454694630000",
            "Effect": "Allow",
            "Action": [
                "logs:*"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Sid": "Stmt1454694649000",
            "Effect": "Allow",
            "Action": [
                "es:*"
            ],
            "Resource": [
                "*"
            ]
        }
    ]
}

I confirmed with IAM policy simulator as well as other cloudformation instances that use other apps to access elasticsearch in AWS:

[abcondas]

I ran into this problem when attempting to run a bucketing search in es (also despite having the “es:*” action on my es domain resource). I was able to resolve it by adding the following to my lambda role. Hope it gives you some ideas…

Effect: Allow
Action:
	- es:ESHttpGet
Resource:
	- !Join [":", ["arn:aws:es", !Ref "AWS::Region", !Ref "AWS::AccountId", "domain/<domain name>/<index name>*/_search"]]

[alexanderfichel]

Finally got it working, thanks a lot Qmando! I haven’t used the docker, but had to update some libraries when I built it on the amazon linux instace:

pip install -U setuptools
pip install -U mock
pip install -U pbr
pip install -U requests-oauthlib

As well as update in requirements.txt

configparser==3.5.0
elasticsearch==1.7.0