[bug]: ResourceWatcher stops working after leadership changes

Describe the bug

When the leadership changes the ResourceWatcher throws an unobserved exception:

System.AggregateException: A Task's exception(s) were not observed either by Waiting on the Task or accessing its Exception property. As a result, the unobserved exception was rethrown by the finalizer thread. (Cannot access a disposed object.)
 ---> System.ObjectDisposedException: Cannot access a disposed object.
   at System.Reactive.Subjects.Subject`1.ThrowDisposed() in /_/Rx.NET/Source/src/System.Reactive/Subjects/Subject.cs:line 53
   at System.Reactive.Subjects.Subject`1.OnNext(T value) in /_/Rx.NET/Source/src/System.Reactive/Subjects/Subject.cs:line 141
   at KubeOps.Operator.Kubernetes.ResourceWatcher`1.OnException(Exception e)
   at k8s.Watcher`1.WatcherLoop(CancellationToken cancellationToken)
   at k8s.Watcher`1.<.ctor>b__9_0()

My suspicion is that this happens after calling the StartAsync again on the same instance after StopAsync.

The problem could be fixed by removing this line where the _reconnectHandler is disposed again outside of the previous if (!fromStop) clause.

To reproduce

Change the leadership from LeaderState.Leader to LeaderState.Candidate and back to LeaderState.Leader.

Expected behavior

The resource watcher should not stop watching after leadership changed.

Screenshots

No response

Additional Context

I can prepare a PR if you will confirm that the issue is legitimate. Could be related to #482.

[tomasfabian]

Ok no problem if it is a hassle, we will upgrade our operator to v7.

[buehler]

Hey @tomasfabian Hmm… are you not able to upgrade to 7? backporting is kind of a burden and I need more time to set up release/support branches for semantic release