Lightrun
question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

SSL check is not working when a private CA is configured

See original GitHub issue

Note: If you have a feature request, you should contact support so the request can be properly tracked.

Output of the info page

Getting the status from the agent.

===============
Agent (v6.14.0)
===============

  Status date: 2019-10-17 16:56:47.686227 UTC
  Agent start: 2019-10-17 16:56:05.183719 UTC
  Pid: 22918
  Go Version: go1.12.9
  Python Version: 2.7.16
  Check Runners: 4
  Log Level: debug

  Paths
  =====
    Config File: /etc/datadog-agent/datadog.yaml
    conf.d: /etc/datadog-agent/conf.d
    checks.d: /etc/datadog-agent/checks.d

  Clocks
  ======
    NTP offset: -993µs
    System UTC time: 2019-10-17 16:56:47.686227 UTC

  Host Info
  =========
    bootTime: 2019-10-16 17:12:52.000000 UTC
    kernelVersion: 4.15.0-1027-aws
    os: linux
    platform: debian
    platformFamily: debian
    platformVersion: 10.1
    procs: 178
    uptime: 23h43m14s
    virtualizationRole: guest
    virtualizationSystem: docker

  Hostnames
  =========
    ec2-hostname: ip-10-1-1-147.eu-west-1.compute.internal
    hostname: consul-server-1-i-04dad273ab3198d97
    instance-id: i-04dad273ab3198d97
    socket-fqdn: 6a943b3d3603
    socket-hostname: 6a943b3d3603
    host tags:
      location:eu-west-1
      group_role:consul-server
      provider:aws
    hostname provider: configuration

=========
Collector
=========



  Running Checks
  ==============

    consul (1.9.1)
    --------------
      Instance ID: consul:29cecf8b342b908c [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/consul.yaml
      Total Runs: 3
      Metric Samples: Last Run: 1, Total: 3
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 2, Total: 7
      Average Execution Time : 13ms

      Instance ID: consul:fd3ee6b1b01a81b4 [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/consul.d/auto_conf.yaml
      Total Runs: 3
      Metric Samples: Last Run: 1, Total: 3
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 2, Total: 7
      Average Execution Time : 17ms


    cpu
    ---
      Instance ID: cpu [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/cpu.d/conf.yaml.default
      Total Runs: 3
      Metric Samples: Last Run: 6, Total: 12
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 0s


    disk (2.5.0)
    ------------
      Instance ID: disk:1a1171fc8f9456e3 [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/disk.d/conf.yaml
      Total Runs: 2
      Metric Samples: Last Run: 134, Total: 268
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 101ms


    docker
    ------
      Instance ID: docker [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/docker.d/conf.yaml.default
      Total Runs: 2
      Metric Samples: Last Run: 36, Total: 72
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 1, Total: 2
      Average Execution Time : 11ms


    file_handle
    -----------
      Instance ID: file_handle [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/file_handle.d/conf.yaml.default
      Total Runs: 3
      Metric Samples: Last Run: 5, Total: 15
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 0s


    http_check (4.2.0)
    ------------------
      Instance ID: http_check:Consul:928f17239ebab106 [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/http_check.d/conf.yaml
      Total Runs: 3
      Metric Samples: Last Run: 5, Total: 15
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 2, Total: 6
      Average Execution Time : 33ms


    io
    --
      Instance ID: io [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/io.d/conf.yaml.default
      Total Runs: 2
      Metric Samples: Last Run: 39, Total: 51
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 0s


    load
    ----
      Instance ID: load [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/load.d/conf.yaml.default
      Total Runs: 3
      Metric Samples: Last Run: 6, Total: 18
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 0s


    memory
    ------
      Instance ID: memory [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/memory.d/conf.yaml.default
      Total Runs: 3
      Metric Samples: Last Run: 17, Total: 51
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 0s


    network (1.11.4)
    ----------------
      Instance ID: network:e0204ad63d43c949 [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/network.d/conf.yaml.default
      Total Runs: 3
      Metric Samples: Last Run: 49, Total: 147
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 4ms


    ntp
    ---
      Instance ID: ntp:133ed7da27793e16 [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/ntp.yaml
      Total Runs: 2
      Metric Samples: Last Run: 1, Total: 2
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 1, Total: 2
      Average Execution Time : 0s


    uptime
    ------
      Instance ID: uptime [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/uptime.d/conf.yaml.default
      Total Runs: 3
      Metric Samples: Last Run: 1, Total: 3
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 0s

========
JMXFetch
========

  Initialized checks
  ==================
    no checks

  Failed checks
  =============
    no checks

=========
Forwarder
=========

  Transactions
  ============
    CheckRunsV1: 2
    Dropped: 0
    DroppedOnInput: 0
    Events: 0
    HostMetadata: 0
    IntakeV1: 2
    Metadata: 0
    Requeued: 0
    Retried: 0
    RetryQueueSize: 0
    Series: 0
    ServiceChecks: 0
    SketchSeries: 0
    Success: 6
    TimeseriesV1: 2

  API Keys status
  ===============
    API key ending with <REDACTED>: API Key valid

==========
Endpoints
==========
  https://app.datadoghq.eu - API Key ending with:
      - <REDACTED>

==========
Logs Agent
==========
    LogsProcessed: 728
    LogsSent: 728

  journald
  --------
    Type: journald
    ExcludeUnits: proc-sys-fs-binfmt_misc.automount
    Status: OK
    Inputs: default

=========
Aggregator
=========
  Checks Metric Sample: 727
  Dogstatsd Metric Sample: 1,929
  Event: 1
  Events Flushed: 1
  Number Of Flushes: 2
  Series Flushed: 430
  Service Check: 58
  Service Checks Flushed: 44

=========
DogStatsD
=========
  Event Packets: 0
  Event Parse Errors: 0
  Metric Packets: 1,928
  Metric Parse Errors: 0
  Service Check Packets: 0
  Service Check Parse Errors: 0
  Udp Bytes: 76,498
  Udp Packet Reading Errors: 0
  Udp Packets: 1,936
  Uds Bytes: 0
  Uds Origin Detection Errors: 0
  Uds Packet Reading Errors: 0
  Uds Packets: 0

Additional environment details (Operating System, Cloud provider, etc):

Steps to reproduce the issue:

  1. Enable tls_verify
  2. Configure a private with tls_ca_cert

Describe the results you received:

2019-10-17 16:53:40 UTC | CORE | DEBUG | (pkg/collector/python/check.go:69 in runCheck) | Running python check http_check http_check:Consul:928f17239ebab106
2019-10-17 16:53:40 UTC | CORE | DEBUG | (pkg/collector/python/datadog_agent.go:120 in LogMessage) | http_check:Consul:928f17239ebab106 | (http_check.py:111) | Connecting to https://consul-server-1.eu-west-1.dev.<REDACTED>:8501
2019-10-17 16:53:40 UTC | CORE | DEBUG | (pkg/collector/python/datadog_agent.go:120 in LogMessage) | - | (connectionpool.py:815) | Starting new HTTPS connection (1): consul-server-1.eu-west-1.dev.<REDACTED>:8501
2019-10-17 16:53:40 UTC | CORE | DEBUG | (pkg/collector/python/datadog_agent.go:120 in LogMessage) | - | (connectionpool.py:396) | https://consul-server-1.eu-west-1.dev.<REDACTED>:8501 "GET / HTTP/1.1" 301 39
2019-10-17 16:53:40 UTC | CORE | DEBUG | (pkg/collector/python/datadog_agent.go:120 in LogMessage) | - | (connectionpool.py:396) | https://consul-server-1.eu-west-1.dev.<REDACTED>:8501 "GET /ui/ HTTP/1.1" 200 5729
2019-10-17 16:53:40 UTC | CORE | DEBUG | (pkg/collector/python/datadog_agent.go:120 in LogMessage) | http_check:Consul:928f17239ebab106 | (http_check.py:92) | https://consul-server-1.eu-west-1.dev.<REDACTED>:8501 is UP
2019-10-17 16:53:40 UTC | CORE | DEBUG | (pkg/collector/python/datadog_agent.go:120 in LogMessage) | http_check:Consul:928f17239ebab106 | (http_check.py:318) | Site is down, unable to connect to get cert expiration: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:727)
2019-10-17 16:53:40 UTC | CORE | INFO | (pkg/collector/runner/runner.go:327 in work) | Done running check http_check

Describe the results you expected:

I expect the SSL verification to work since the CA has been configured.

Additional information you deem important (e.g. issue happens only occasionally): After an investigation I think I got the bottom of it. Here instance_ca_certs should be the value configured in the yaml. For some reason it’s not, it the default CA of the Agent as you can see here:

2019-10-17 16:53:40 UTC | CORE | DEBUG | (pkg/collector/python/check.go:69 in runCheck) | Running python check http_check http_check:Consul:928f17239ebab106
2019-10-17 16:53:40 UTC | CORE | DEBUG | (pkg/collector/python/datadog_agent.go:120 in LogMessage) | http_check:Consul:928f17239ebab106 | (http_check.py:111) | Connecting to https://consul-server-1.eu-west-1.dev.<REDACTED>:8501
2019-10-17 16:53:40 UTC | CORE | DEBUG | (pkg/collector/python/datadog_agent.go:120 in LogMessage) | - | (connectionpool.py:815) | Starting new HTTPS connection (1): consul-server-1.eu-west-1.dev.<REDACTED>:8501
2019-10-17 16:53:40 UTC | CORE | DEBUG | (pkg/collector/python/datadog_agent.go:120 in LogMessage) | - | (connectionpool.py:396) | https://consul-server-1.eu-west-1.dev.<REDACTED>:8501 "GET / HTTP/1.1" 301 39
2019-10-17 16:53:40 UTC | CORE | DEBUG | (pkg/collector/python/datadog_agent.go:120 in LogMessage) | - | (connectionpool.py:396) | https://consul-server-1.eu-west-1.dev.<REDACTED>:8501 "GET /ui/ HTTP/1.1" 200 5729
2019-10-17 16:53:40 UTC | CORE | DEBUG | (pkg/collector/python/datadog_agent.go:120 in LogMessage) | http_check:Consul:928f17239ebab106 | (http_check.py:92) | https://consul-server-1.eu-west-1.dev.<REDACTED>:8501 is UP
2019-10-17 16:53:40 UTC | CORE | DEBUG | (pkg/collector/python/datadog_agent.go:120 in LogMessage) | http_check:Consul:928f17239ebab106 | (http_check.py:292) | ca file /opt/datadog-agent/embedded/ssl/certs/cacert.pem
2019-10-17 16:53:40 UTC | CORE | DEBUG | (pkg/collector/python/datadog_agent.go:120 in LogMessage) | http_check:Consul:928f17239ebab106 | (http_check.py:318) | Site is down, unable to connect to get cert expiration: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:727)
2019-10-17 16:53:40 UTC | CORE | INFO | (pkg/collector/runner/runner.go:327 in work) | Done running check http_check

I fixed the issue by doing something like this:

url = instance.get('url')
ca_cert= instance.get('tls_ca_cert')

o = urlparse(url)
host = o.hostname
server_name = instance.get('ssl_server_name', o.hostname)
port = o.port or 443

try:
    sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    sock.settimeout(float(timeout))
    sock.connect((host, port))
    context = ssl.SSLContext(ssl.PROTOCOL_SSLv23)
    context.verify_mode = ssl.CERT_REQUIRED
    context.check_hostname = check_hostname
    context.load_verify_locations(ca_cert)

Here the output

2019-10-17 17:08:48 UTC | CORE | INFO | (pkg/collector/runner/runner.go:261 in work) | Running check http_check
2019-10-17 17:08:48 UTC | CORE | DEBUG | (pkg/collector/python/check.go:69 in runCheck) | Running python check http_check http_check:Consul:928f17239ebab106
2019-10-17 17:08:48 UTC | CORE | DEBUG | (pkg/collector/python/datadog_agent.go:120 in LogMessage) | http_check:Consul:928f17239ebab106 | (http_check.py:111) | Connecting to https://consul-server-1.eu-west-1.dev.<REDACTED>:8501
2019-10-17 17:08:48 UTC | CORE | DEBUG | (pkg/collector/python/datadog_agent.go:120 in LogMessage) | - | (connectionpool.py:815) | Starting new HTTPS connection (1): consul-server-1.eu-west-1.dev.<REDACTED>:8501
2019-10-17 17:08:48 UTC | CORE | DEBUG | (pkg/collector/python/datadog_agent.go:120 in LogMessage) | - | (connectionpool.py:396) | https://consul-server-1.eu-west-1.dev.<REDACTED>:8501 "GET / HTTP/1.1" 301 39
2019-10-17 17:08:48 UTC | CORE | DEBUG | (pkg/collector/python/datadog_agent.go:120 in LogMessage) | - | (connectionpool.py:396) | https://consul-server-1.eu-west-1.dev.<REDACTED>:8501 "GET /ui/ HTTP/1.1" 200 5729
2019-10-17 17:08:48 UTC | CORE | DEBUG | (pkg/collector/python/datadog_agent.go:120 in LogMessage) | http_check:Consul:928f17239ebab106 | (http_check.py:92) | https://consul-server-1.eu-west-1.dev.<REDACTED>:8501 is UP
2019-10-17 17:08:48 UTC | CORE | DEBUG | (pkg/collector/python/datadog_agent.go:120 in LogMessage) | http_check:Consul:928f17239ebab106 | (http_check.py:292) | ca file /opt/datadog-agent/embedded/ssl/certs/cacert.pem
2019-10-17 17:08:48 UTC | CORE | DEBUG | (pkg/collector/python/datadog_agent.go:120 in LogMessage) | http_check:Consul:928f17239ebab106 | (http_check.py:307) | cert {'subjectAltName': (('DNS', 'consul.service.consul'), ('DNS', 'consul.eu-west-1.dev.<REDACTED>'), ('DNS', 'consul-server-1.eu-west-1.dev.<REDACTED>'), ('DNS', 'consul-server-2.eu-west-1.dev.<REDACTED>'), ('DNS', 'consul-server-3.eu-west-1.dev.<REDACTED>'), ('IP Address', '127.0.0.1'), ('IP Address', '172.17.0.1')), 'notBefore': u'Feb 16 19:43:15 2019 GMT', 'serialNumber': u'***************************CB8F5', 'notAfter': 'Feb 16 19:43:15 2020 GMT', 'version': 3L, 'subject': ((('organizationName', u'<REDACTED>'),), (('commonName', u'<REDACTED> cert'),)), 'issuer': ((('organizationName', u'<REDACTED>'),), (('commonName', u'<REDACTED> cert'),))}
2019-10-17 17:08:48 UTC | CORE | DEBUG | (pkg/collector/python/datadog_agent.go:120 in LogMessage) | http_check:Consul:928f17239ebab106 | (http_check.py:326) | Exp_date: 2020-02-16 19:43:15
2019-10-17 17:08:48 UTC | CORE | DEBUG | (pkg/collector/python/datadog_agent.go:120 in LogMessage) | http_check:Consul:928f17239ebab106 | (http_check.py:327) | seconds_left: 10550066.7529
2019-10-17 17:08:48 UTC | CORE | INFO | (pkg/collector/runner/runner.go:327 in work) | Done running check http_check

Issue Analytics

  • State:closed
  • Created 4 years ago
  • Reactions:1
  • Comments:5 (3 by maintainers)

github_iconTop GitHub Comments

1reaction
ofekcommented, Oct 18, 2019
0reactions
manan728commented, Oct 25, 2020

@renaudhager Would you mind sharing your tls.d/conf.yaml that worked for you? I’d like to see whether you used “cert” or the “ca_cert” parameter under instances. Asking coz none are working for me although there are no errors reported in the datadog logs either. The check seem to be working ok, yet no metric is getting passed back to the Datadog UI. Thanks.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Common SSL Certificate Errors and How to Fix Them
Your private key matching your certificate is usually located in the same directory the CSR was created. If the private key is no...
Read more >
What is an SSL certificate and How to fix SSL Errors?
This SSL error occurs when the Certificate Authority has revoked/canceled a particular domain's SSL certificate. The CA may revoke the certificate if its ......
Read more >
Troubleshooting SSL related issues (Server Certificate)
This document will help you in troubleshooting SSL issues related to ... Check if the server certificate has the private key corresponding ...
Read more >
How to Fix "Your Connection is Not Private" Error (18 Tips)
Check Your Computer's Clock; Try in Incognito Mode; Clear Browser Cache and Cookies; Try Clearing the SSL State on Your Computer; Change DNS ......
Read more >
Certificate Not Trusted | View Security Certificate Errors
To resolve this problem, install the intermediate certificate (or chain certificate) file to the server that hosts your website. To do that, log...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found
Previous page

Agent 7.16.0 cannot parse non-semantic nginx version

Next page

Stop write warning in redisb if no keys configuration