question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Controller is not being called after options.WatcherHttpTimeout has elapsed.

See original GitHub issue

Describe the bug After the watcher http timeout has elapsed, the watcher is not being recreated. This looks to be related to the fix in https://github.com/buehler/dotnet-operator-sdk/issues/460 https://github.com/buehler/dotnet-operator-sdk/commit/a2290ff9edc41bbbb3a7b47a13a00c99e469f8f9?diff=split

To Reproduce I’m not sure if the original issue was reliably reproduced, but I was getting the original Microsoft.Rest.SerializationException. If you were getting this I expect you should be able to reproduce this issue as well.

Expected behavior Microsoft.Rest.SerializationException should not break the watcher.

Screenshots I’d attach logs but I haven’t been able to get any logs out of the watcher class. I tried setting Logging__LogLevel__Microsoft=Trace on the pod.

Additional context Prior to a2290ff9edc41bbbb3a7b47a13a00c99e469f8f9, the SerializationException would result in a call to restart the watcher WatchResource().ConfigureAwait(false);. We now catch this exception and do not call restart on the watcher. Eventually we hit our WatcherHttpTimeout and enter the OnClose method where the watcher is not restarted because the cancellation token was cancelled in the exception handling earlier.

Unsure how we should handle this, calling WatchResource().ConfigureAwait(false);, we really don’t want to be calling WatchResource().ConfigureAwait(false); every time that SerializationException is thrown.

Issue Analytics

  • State:closed
  • Created a year ago
  • Comments:6 (2 by maintainers)

github_iconTop GitHub Comments

2reactions
anekdoticommented, Feb 23, 2023

I think your latest proposal should be fine, @tomasfabian . Thank you for your thoughts! What do you think @buehler ?

1reaction
anekdoticommented, Feb 17, 2023

Hello @buehler,

in https://github.com/buehler/dotnet-operator-sdk/commit/a2290ff9edc41bbbb3a7b47a13a00c99e469f8f9 you wanted to make the ResourceWatcher ignore exceptions due to empty responses (In https://github.com/buehler/dotnet-operator-sdk/issues/460#issuecomment-1210262006). In https://github.com/buehler/dotnet-operator-sdk/blob/master/src/KubeOps/Operator/Kubernetes/ResourceWatcher{TEntity}.cs#L163 the watcher is disposed of in any case, independently of the type of exception.

In the case of a SerializationException, the method returns immediately without recreating the watcher - neither directly like in the case of a TaskCancelledException, nor with an exponential backoff like for all other exceptions.

In order to ensure that the watcher is restarted even after a SerializationException, I would suggest to either remove the return statement in https://github.com/buehler/dotnet-operator-sdk/blob/master/src/KubeOps/Operator/Kubernetes/ResourceWatcher{TEntity}.cs#L183 or to remove the special case for this type of exception completely (if a specific log entry for it is not needed) and handle it together with all other exceptions below.

Do I miss something?

Read more comments on GitHub >

github_iconTop Results From Across the Web

Watcher HTTP input | Elasticsearch Guide [8.9]
The timeout for reading data from http connection. If no response was received within this time, the input will timeout and fail. request.body....
Read more >
AngularJS : Prevent error $digest already in progress when ...
The problem arises when you have a call to $apply that is sometimes run asynchronously outside of Angular code (when $apply should be...
Read more >
Watcher HTTP input | Elasticsearch Guide [7.17]
The timeout for reading data from http connection. If no response was received within this time, the input will timeout and fail. request.body....
Read more >
Retry and timeout client plugins don't work together - YouTrack
I have both HttpRequestRetry and HttpTimeout installed in my client, using the Apache engine. However, when I make a request that times out,...
Read more >
Top 18 Most Common AngularJS Developer Mistakes
Common Mistake #5: Not Using Named Views With UI-ROUTER For Power”. The de facto routing solution for AngularJS has been, until now, the...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found