`Hotswap` exposes problems with resource leaks on concurrent access

Firstly, thanks for the amazing work on CE3, it really is a joy to use.

While working on Trace4Cats, I had the need to swap resources at runtime, but split the functionality of swapping and access of Hotswap where the resource may be swapped concurrently to access. Hotswap seemed perfect for this, but quickly threw up a problem: what if something is using the resource R, while it is being swapped? Secondly, what if two fibers try to swap the resource simultaneously?

Admittedly my use case is not the same as in the documentation, but I believe it exposes a problem that others may encounter without realising. I also believe that the example is not completely safe, as you could theoretically have two fibers hitting the character limit in two separate operations and swap the resource almost simultaneously, with no guard around whether it is still in use in another fiber.

To this end @catostrophe and I created HotswapRef to separate the actions of swapping and access safer for concurrent calls, using a semaphore to protect calls to swap and reference counting when accessing the resource R, which is returned in a Resource to model acquistion/release of a handle. While this does build on Hotswap using other cats-effect primitives and might not belong in the standard library itself, I do believe that at least some of this functionality should belong in Hotswap, like the semaphore around calls to swap.

Edit: just to be clear we deliberately designed HotswapRef so that calls to access would never block, only calls to swap. This does present its own challenges whereby calling swap on the HotswapRef within use on the Resource returned by access will cause a deadlock, this is highlighted in ConditionalHotswapRefConstructor.

[janstenpickle]

No problem @vasilmkd, I hadn’t fully figured this out when I messaged in Discord, this is probably a better place to raise it anyway.

For the record, I’m more than happy to raise a PR with any agreed changes, but I think we probably need to agree if there are any to be made.

[bastewart]

I’m copy/pasting this from the Hotswap scaladoc.

  /**
   * [[swap]] finalizes the previous resource immediately, so users must ensure
   * that the old `R` is not used thereafter. Failure to do so may result in an
   * error on the _consumer_ side. In any case, no resources will be leaked.
   */

My personal opinion is that Hotswap has already documented this deficiency (or the documentation could be further improved), and I personally see the current Hotswap as a good basic building block for resource safety, upon which other data structures can build, much like HotswapRef. I honestly wouldn’t push for changes to Hotswap, except improvements to the documentation. However, I cannot speak on behalf of other maintainers, who would have to approve such a change to Hotswap, like the ones discussed in this thread.

FWIW I don’t personally see it an an exceptional useful building block, or at least not one that can be picked up and used easily without some careful thought beforehand. Beyond some extremely simple situations I don’t think it’s possible to use without also creating your own Ref to hold the current active resource*. This then introduces a whole host of pitfalls for a new (or even experienced!) user to fall into around concurrency. The current example is unsafe as concurrent access to log could lead to a resource which has been finalised being accessed from another fiber! I think the fact no-one picked up that the docs have an unsafe example is telling that the problems are pretty subtle and even solid documentation may not cover them.

*The only situation I can think of where it would be safe is how it’s used in fs2. In fs2 the whole lifespan of the Hotswap is inside a single method call to write data to disk. The method is a simple loop (in a stream) which cycles resources as it writes, ultimately closing the Hotswap once all data has been written.

Definitely not my call (and the code is here now, which makes it really hard to move away from) but I would argue that Hotswap should take a safety-first approach. Especially when it comes to concurrency which is so hard to get right; leaving it up to end-users to handle safely is a recipe for loads of subtle bugs out in the wild. Having a safe interface would also mean there’s no need for the Ref (it becomes internal to the implementation) so it’s a lot easier for people to pick up and use as a building block in all situations.