question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

[BUG] `filebeat` communication error for AKS

See original GitHub issue

Describe the bug During installation filebeat get host as hostname, this name isn’t added into /etc/hosts so after installation filebeat cannot resolve it, so cannot communicate to AKS

How to reproduce Steps to reproduce the behavior:

  1. execute epicli init ... (with params)
  2. edit config file
  3. execute epicli apply ...

Expected behavior filebeat communicate properly

Config files n/a

Environment

  • Cloud provider: Azure
  • OS: Ubuntu 18.04.5 LTS

epicli version: 0.10.0

Additional context n/a


DoD checklist

  • Changelog updated (if affected version was released)
  • COMPONENTS.md updated / doesn’t need to be updated
  • Automated tests passed (QA pipelines)
    • apply
    • upgrade
  • Case covered by automated test (if possible)
  • Idempotency tested
  • Documentation updated / doesn’t need to be updated
  • All conversations in PR resolved
  • Backport tasks created / doesn’t need to be backported

Issue Analytics

  • State:closed
  • Created 2 years ago
  • Comments:10 (5 by maintainers)

github_iconTop GitHub Comments

1reaction
wkalbarcommented, Apr 23, 2021

@mkyc little update (and possible solution): We’ve dig deeper into this, and problem isn’t on epiphany cluster side, but on AKS pods configuration. There are filebeat pods in namespace epi-logging , named filebeat-filebeat-[5 alphanumeric id here], for example filebeat-filebeat-mnl8p This pods during epicli apply get hostname of our logging vm, but this hostname is unresolvable for them. We can manually add it to /etc/hosts on this pod, but if this pod/node next creation this entry will be deleted.

Solution working for us now is to edit daemonsets object (kubectl -n epi-logging edit daemonset filebeat-filebeat) and add block:

      - hostnames:
        - prefix-clustername-logging-vm-0
        ip: ip.address.of.logging.vm

After this, every new pod created will have in his /etc/hosts entry like below:

# Entries added by HostAliases.
ip.address.of.logging.vm     prefix-clustername-logging-vm-0

Now for this issue it should be added during epicli apply in filebeat part, or loging vm ip address should be added instead of hostname.

0reactions
mkyccommented, May 24, 2021

@atsikham please create backport tasks to 0.9.x and 1.0.x.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Troubleshooting methods | Elastic Cloud on Kubernetes [2.5]
Configure Elasticsearch timeoutsedit. The operator needs to communicate with each Elasticsearch cluster in order to perform orchestration tasks. The default ...
Read more >
Troubleshoot connectivity issues - Azure Event Hubs
This article provides information on troubleshooting connectivity issues with Azure Event Hubs.
Read more >
Kubernetes autodiscover provider fails silently if it can't ...
The reason seems to be that the k8s API fails to respond in time (this is an Azure AKS cluster and the underlying...
Read more >
Breaking Change to Beats May Impact your Elasticsearch
At Logz.io, we've started moving away from Logstash and Metricbeat already, but have kept recommending Filebeat as a good log shipper for many ......
Read more >
Kubernetes Observability: Log Aggregation Using ELK Stack
Logging, when used in the earliest design stages, helps diagnose bugs, gain insight into system behavior and spots potential issues before ...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found