Operation "element in set" never returns

Bug: After a series of add and remove operations on a set in nopython mode the operation “in set” does not return and gets stuck at 100% CPU load.

Isolated code example:

import numba as nb

@ nb.njit()
def problem():

    # init
    openSet = set()

    openSet.add((0, 0))
    openSet.add((1, 0))
    openSet.add((2, 0))
    openSet.add((3, 0))
    openSet.add((4, 0))
    openSet.add((5, 0))
    openSet.add((6, 0))
    openSet.add((7, 0))
    openSet.add((8, 0))

    openSet.remove((2, 0))
    openSet.add((2, 1))
    openSet.remove((2, 1))
    openSet.remove((4, 0))
    openSet.add((5, 1))
    openSet.add((4, 1))
    openSet.remove((7, 0))
    openSet.add((8, 1))
    openSet.add((7, 1))
    openSet.add((6, 1))
    openSet.remove((8, 0))
    openSet.remove((1, 0))
    openSet.remove((0, 0))
    openSet.remove((5, 0))
    openSet.remove((6, 0))
    openSet.remove((3, 0))
    openSet.remove((7, 1))
    openSet.add((8, 2))
    openSet.add((7, 2))
    openSet.add((6, 2))
    openSet.remove((8, 1))
    openSet.remove((6, 1))
    openSet.add((5, 2))
    openSet.remove((4, 1))
    openSet.add((4, 2))
    openSet.remove((5, 1))
    openSet.remove((6, 2))
    openSet.add((7, 3))
    openSet.add((6, 3))
    openSet.add((5, 3))
    openSet.remove((7, 2))
    openSet.add((8, 3))
    openSet.remove((8, 2))
    openSet.remove((4, 2))
    openSet.add((4, 3))
    openSet.add((3, 3))
    openSet.remove((5, 2))
    openSet.remove((7, 3))
    openSet.add((8, 4))
    openSet.add((7, 4))
    print('About to check (6,4) in openSet')
    return (6, 4) in openSet


if __name__ == "__main__":
    print(problem())
    print('DONE')

The code gets stuck with numba 0.52.0 with Python 3.8.5 on Ubuntu 20.04.1 LTS x86_64.

However, @esc on gitter.im also confirmed that the issue persists on OSX.

Isolated code example: https://github.com/icezyclon/astart-pathfinding/blob/main/isolated.py Original code example (from A* search): https://github.com/icezyclon/astart-pathfinding/blob/main/problem.py

[esc]

@stuartarchibald and I never got to the bottom of this fully. But we are certain there is a bug present, so I will re-label thhis accordingly.

[esc]

We can’t simply terminate on seeing “DELETED” because the lookup chain needs to remain intact. Otherwise you get issues like the one with About to remove (8, 2) from openSet – where there is one (or maybe more) “DELETED” slots within the lookup chain. Only when we have circled around once, visited all the slots and determined they are either “IN-USE” or “DELTED”, can we safely determine, that the element being looked for isn’t in the set. Incidentally, this also means, the runtime will be exactly O(n) since we need to visit every slot to determine the element isn’t in there.

Should the lookup time for sets no be an average of O(1)? I thought they are similar to dicts? If it is always O(n) then that would mean losing a big advantage in comparison to lists.

See https://wiki.python.org/moin/TimeComplexity for CPython TimeComplexity.

The worst case, only happens under the special circumstances, if the set contains only in-use or deleted slots. If the load on the set is low, the runtime will be O(1) yes, since only a hash is needed to lookup an element.