Elasticsearch TLS connection problem

I’m having trouble configuring and debugging a TLS connection between APIMAN and Elasticsearch for metrics. In my apiman.properties file I have:

apiman.es.protocol=https
apiman.es.host=elasticsearch
apiman.es.port=9200
apiman.es.username=elastic
apiman.es.password=changeme
apiman.es.timeout=10000
apiman.es.keystore=/home/apiman/jboss/standalone/configuration/apiman.jks
apiman.es.keystore.password=secret
apiman.es.truststore=/home/apiman/jboss/standalone/configuration/truststore.jks
apiman.es.truststore.password=secret

All of the other settings use these properties.

I can connect to elasticsearch from the APIMAN system using the certificates in the truststore with no problems:

curl -u elastic:changeme --cacert cacert.cer  https://elasticsearch:9200
{
  "name" : "hAVci1Q",
  "cluster_name" : "docker-cluster",
  "cluster_uuid" : "Pc_Nnr6SStuRgddEeLk_rw",
  "version" : {
    "number" : "5.6.16",
    "build_hash" : "3a740d1",
    "build_date" : "2019-03-13T15:33:36.565Z",
    "build_snapshot" : false,
    "lucene_version" : "6.6.1"
  },
  "tagline" : "You Know, for Search"
}

java -Djavax.net.ssl.trustStore=./jboss/standalone/configuration/truststore.jks SSLPoke elasticsearch 9200 Successfully connected

But when an API is called, the APIMAN log has:

2019-11-19 19:23:38,099 INFO  [io.searchbox.client.AbstractJestClient] (default task-416) Setting server pool to a list of 1 servers: [https://elasticsearch:9200]
2019-11-19 19:23:38,099 INFO  [io.searchbox.client.JestClientFactory] (default task-416) Using multi thread/connection supporting pooling connection manager
2019-11-19 19:23:38,146 INFO  [io.searchbox.client.JestClientFactory] (default task-416) Using default GSON instance
2019-11-19 19:23:38,146 INFO  [io.searchbox.client.JestClientFactory] (default task-416) Node Discovery disabled...
2019-11-19 19:23:38,146 INFO  [io.searchbox.client.JestClientFactory] (default task-416) Idle connection reaping disabled...
2019-11-19 19:23:38,300 INFO  [io.searchbox.client.AbstractJestClient] (ESMetricsConsumer) Setting server pool to a list of 1 servers: [https://elasticsearch:9200]
2019-11-19 19:23:38,301 INFO  [io.searchbox.client.JestClientFactory] (ESMetricsConsumer) Using multi thread/connection supporting pooling connection manager
2019-11-19 19:23:38,303 INFO  [io.searchbox.client.JestClientFactory] (ESMetricsConsumer) Using default GSON instance
2019-11-19 19:23:38,303 INFO  [io.searchbox.client.JestClientFactory] (ESMetricsConsumer) Node Discovery disabled...
2019-11-19 19:23:38,303 INFO  [io.searchbox.client.JestClientFactory] (ESMetricsConsumer) Idle connection reaping disabled...
2019-11-19 19:23:38,350 INFO  [stdout] (ESMetricsConsumer) WARN: Error adding metric to ES

And Elasticsearch has:

elasticsearch | [2019-11-19T18:55:50,063][WARN ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [hAVci1Q] http client did not trust this server’s certificate, closing connection [id: 0x9487bc47, L:0.0.0.0/0.0.0.0:9200 ! R:/172.23.0.3:33010]

Ideally I’d like to be able to use self-signed certificates fort testing, but for this I used signed certificates.

Any idea what else to check?

[msavy]

Have you turned on ssl debugging in Java? That will let you know whether it’s trying to use your certificates or not.

On Wed, 20 Nov 2019, 13:15 Florian Volk, [e-mail] wrote:

I’m not sure if this is really a bug, because we are running the setup with vertx and we have no problems with self-signed certificates. But I have also no further idea what is wrong here.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/apiman/apiman/issues/775?email_source=notifications&email_token=AADHMWK42G2R5DEVY7UQEOTQUUZ7DA5CNFSM4JPQZYK2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEER53BA#issuecomment-555998596, or unsubscribe https://github.com/notifications/unsubscribe-auth/AADHMWPTOYTNWOAVCKHJOWDQUUZ7DANCNFSM4JPQZYKQ .

[EricWittmann]

I think an option to trust self-signed certs seems like a good update. I’m not much of an SSL expert, so would defer to @msavy for review. 😃