Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Reflector crashing when cert-manager is not installed

See original GitHub issue

I just installed:

- name: reflector
  namespace: reflector
  chart: emberstack/reflector
  version: 5.3.16

with

resources:
  limits:
    cpu: 200m
    memory: 256Mi
  requests:
    cpu: 100m
    memory: 256Mi

the pod now always crahes with:

2020-09-18 08:35:40.883 +00:00 [INF] (ES.Kubernetes.Reflector.Host.Program) Starting host
2020-09-18 08:37:28.791 +00:00 [ERR] (ES.Kubernetes.Reflector.CertManager.CertManagerMonitor) V1CustomResourceDefinition watcher Faulted
System.Threading.Tasks.TaskCanceledException: The operation was canceled.
 ---> System.IO.IOException: Unable to read data from the transport connection: Operation canceled.
 ---> System.Net.Sockets.SocketException (125): Operation canceled
   --- End of inner exception stack trace ---
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.GetResult(Int16 token)
   at System.Net.Security.SslStream.<FillBufferAsync>g__InternalFillBufferAsync|215_0[TReadAdapter](TReadAdapter adap, ValueTask`1 task, Int32 min, Int32 initial)
   at System.Net.Security.SslStream.ReadAsyncInternal[TReadAdapter](TReadAdapter adapter, Memory`1 buffer)
   at System.Net.Http.HttpConnection.FillAsync()
   at System.Net.Http.HttpConnection.ChunkedEncodingReadStream.ReadAsyncCore(Memory`1 buffer, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.HttpConnection.ChunkedEncodingReadStream.ReadAsyncCore(Memory`1 buffer, CancellationToken cancellationToken)
   at k8s.WatcherDelegatingHandler.CancelableStream.ReadAsync(Byte[] buffer, Int32 offset, Int32 count, CancellationToken cancellationToken)
   at System.IO.StreamReader.ReadBufferAsync(CancellationToken cancellationToken)
   at System.IO.StreamReader.ReadLineAsyncInternal()
   at k8s.WatcherDelegatingHandler.PeekableStreamReader.PeekLineAsync()
   at k8s.WatcherDelegatingHandler.LineSeparatedHttpContent.SerializeToStreamAsync(Stream stream, TransportContext context)
   at System.Net.Http.HttpContent.LoadIntoBufferAsyncCore(Task serializeToStreamTask, MemoryStream tempBuffer)
   at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at k8s.Kubernetes.ListCustomResourceDefinitionWithHttpMessagesAsync(Nullable`1 allowWatchBookmarks, String continueParameter, String fieldSelector, String labelSelector, Nullable`1 limit, String resourceVersion, Nullable`1 timeoutSeconds, Nullable`1 watch, String pretty, Dictionary`2 customHeaders, CancellationToken cancellationToken)
   at ES.Kubernetes.Reflector.CertManager.CertManagerMonitor.<>c.<<-ctor>b__10_2>d.MoveNext() in /src/ES.Kubernetes.Reflector.CertManager/CertManagerMonitor.cs:line 77
--- End of stack trace from previous location where exception was thrown ---
   at ES.Kubernetes.Reflector.Core.Monitoring.ManagedWatcher`3.Start() in /src/ES.Kubernetes.Reflector.Core/Monitoring/ManagedWatcher.cs:line 100
2020-09-18 08:37:29.385 +00:00 [FTL] (ES.Kubernetes.Reflector.Host.Program) Host terminated unexpectedly
System.Threading.Tasks.TaskCanceledException: The operation was canceled.
 ---> System.IO.IOException: Unable to read data from the transport connection: Operation canceled.
 ---> System.Net.Sockets.SocketException (125): Operation canceled
   --- End of inner exception stack trace ---
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.GetResult(Int16 token)
   at System.Net.Security.SslStream.<FillBufferAsync>g__InternalFillBufferAsync|215_0[TReadAdapter](TReadAdapter adap, ValueTask`1 task, Int32 min, Int32 initial)
   at System.Net.Security.SslStream.ReadAsyncInternal[TReadAdapter](TReadAdapter adapter, Memory`1 buffer)
   at System.Net.Http.HttpConnection.FillAsync()
   at System.Net.Http.HttpConnection.ChunkedEncodingReadStream.ReadAsyncCore(Memory`1 buffer, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.HttpConnection.ChunkedEncodingReadStream.ReadAsyncCore(Memory`1 buffer, CancellationToken cancellationToken)
   at k8s.WatcherDelegatingHandler.CancelableStream.ReadAsync(Byte[] buffer, Int32 offset, Int32 count, CancellationToken cancellationToken)
   at System.IO.StreamReader.ReadBufferAsync(CancellationToken cancellationToken)
   at System.IO.StreamReader.ReadLineAsyncInternal()
   at k8s.WatcherDelegatingHandler.PeekableStreamReader.PeekLineAsync()
   at k8s.WatcherDelegatingHandler.LineSeparatedHttpContent.SerializeToStreamAsync(Stream stream, TransportContext context)
   at System.Net.Http.HttpContent.LoadIntoBufferAsyncCore(Task serializeToStreamTask, MemoryStream tempBuffer)
   at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at k8s.Kubernetes.ListCustomResourceDefinitionWithHttpMessagesAsync(Nullable`1 allowWatchBookmarks, String continueParameter, String fieldSelector, String labelSelector, Nullable`1 limit, String resourceVersion, Nullable`1 timeoutSeconds, Nullable`1 watch, String pretty, Dictionary`2 customHeaders, CancellationToken cancellationToken)
   at ES.Kubernetes.Reflector.CertManager.CertManagerMonitor.<>c.<<-ctor>b__10_2>d.MoveNext() in /src/ES.Kubernetes.Reflector.CertManager/CertManagerMonitor.cs:line 77
--- End of stack trace from previous location where exception was thrown ---
   at ES.Kubernetes.Reflector.Core.Monitoring.ManagedWatcher`3.Start() in /src/ES.Kubernetes.Reflector.Core/Monitoring/ManagedWatcher.cs:line 100
   at ES.Kubernetes.Reflector.CertManager.CertManagerMonitor.StartAsync(CancellationToken cancellationToken) in /src/ES.Kubernetes.Reflector.CertManager/CertManagerMonitor.cs:line 102
   at Microsoft.Extensions.Hosting.Internal.Host.StartAsync(CancellationToken cancellationToken)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
   at ES.Kubernetes.Reflector.Host.Program.Main(String[] args) in /src/ES.Kubernetes.Reflector.Host/Program.cs:line 33

I use the reflector to copy manually added tls-secrets by auto-mirroring. Is this a config error or some bug?

The exakt same deployment works on another cluster which uses Certificate just perfect.

Issue Analytics

State:
Created 3 years ago
Comments:10 (4 by maintainers)

Top GitHub Comments

1reaction

winromuluscommented, Sep 18, 2020

@aeimer got it. I’ll test this tomorrow with no cert-manager. Looks like a bug not covered when no cert-manager is installed.

0reactions

sdandamudi5ccommented, Feb 17, 2021

@aeimer - Currently I am using v5.4.17 version of reflector but I am still facing this issue and we are not using cert-manager also. Can you please look at the below error suggest me a solution.

Error: 2021-02-09 01:16:42.453 +00:00 [ERR] (ES.Kubernetes.Reflector.CertManager.CertManagerMonitor) Exception occurred while processing 'cert-manager' discovery Microsoft.Rest.HttpOperationException: Operation returned an invalid status code 'InternalServerError' at k8s.Kubernetes.ListCustomResourceDefinitionWithHttpMessagesAsync(Nullable'1 allowWatchBookmarks, String continueParameter, String fieldSelector, String labelSelector, Nullable'1 limit, String resourceVersion, Nullable'1 timeoutSeconds, Nullable'1 watch, String pretty, Dictionary'2 customHeaders, CancellationToken cancellationToken) at k8s.KubernetesExtensions.ListCustomResourceDefinitionAsync(IKubernetes operations, Nullable'1 allowWatchBookmarks, String continueParameter, String fieldSelector, String labelSelector, Nullable'1 limit, String resourceVersion, Nullable'1 timeoutSeconds, Nullable'1 watch, String pretty, CancellationToken cancellationToken) at ES.Kubernetes.Reflector.CertManager.CertManagerMonitor.OnCertManagerDiscovery() in /src/ES.Kubernetes.Reflector.CertManager/CertManagerMonitor.cs:line 207 at ES.Kubernetes.Reflector.CertManager.CertManagerMonitor.<>c__DisplayClass11_0.<<StartAsync>g__ReadChannel|0>d.MoveNext() in /src/ES.Kubernetes.Reflector.CertManager/CertManagerMonitor.cs:line

Top Results From Across the Web

cert-manager Crash Loop - kubernetes

You have to install CustomResourceDefinition resources separately kubectl apply --validate=false -f ...

Troubleshooting Problems with ACME / Let's Encrypt ...

When requesting ACME certificates, cert-manager will create Order and Challenges to ... If your challenge self-check fails with a 404 not found error....

Ingress-nginx-controller crash

Rancher running in AK Cluster all from scratch, followed the instructions below All looks good even got an external IP from the ingress, ......

Anthos clusters on VMware known issues

However, the current system does not properly restart the pods of vsphere-csi-controller , causing vsphere-csi-controller to crash after the rotation.

Kubeadm init: Everything crash after several ...

CrashLoopBackOff right after kubeadm init rpc error: code = Unknown desc = malformed header: missing HTTP content-type ...