RuntimeError: Reader to writer privilege escalation not allowed

https://github.com/harlowja/fasteners/commit/467ed75ee1e9465ebff8b5edf452770befb93913

Hi, I think this commit caused an issue with nova (at the very least). Had an oslo guy (bnemec) look at it and he pointed to the commit.

Here’s the log while it’s still up.

http://logs.openstack.org/54/660254/2/check/cross-nova-functional/a0617a9/testr_results.html.gz

Backtraces follow.

2019-05-21 21:29:32,148 ERROR [nova.api.openstack.wsgi] Unexpected exception in API method
Traceback (most recent call last):
  File "nova/api/openstack/wsgi.py", line 671, in wrapped
    return f(*args, **kwargs)
  File "nova/api/validation/__init__.py", line 110, in wrapper
    return func(*args, **kwargs)
  File "nova/api/validation/__init__.py", line 110, in wrapper
    return func(*args, **kwargs)
  File "nova/api/validation/__init__.py", line 110, in wrapper
    return func(*args, **kwargs)
  File "nova/api/validation/__init__.py", line 110, in wrapper
    return func(*args, **kwargs)
  File "nova/api/validation/__init__.py", line 110, in wrapper
    return func(*args, **kwargs)
  File "nova/api/validation/__init__.py", line 110, in wrapper
    return func(*args, **kwargs)
  File "nova/api/validation/__init__.py", line 110, in wrapper
    return func(*args, **kwargs)
  File "nova/api/validation/__init__.py", line 110, in wrapper
    return func(*args, **kwargs)
  File "nova/api/validation/__init__.py", line 110, in wrapper
    return func(*args, **kwargs)
  File "nova/api/validation/__init__.py", line 110, in wrapper
    return func(*args, **kwargs)
  File "nova/api/validation/__init__.py", line 110, in wrapper
    return func(*args, **kwargs)
  File "nova/api/openstack/compute/servers.py", line 661, in create
    nova_context.get_admin_context(), ['nova-compute'])
  File "nova/objects/service.py", line 515, in get_minimum_version_all_cells
    binaries)
  File "nova/context.py", line 564, in scatter_gather_all_cells
    fn, *args, **kwargs)
  File "nova/context.py", line 452, in scatter_gather_cells
    with target_cell(context, cell_mapping) as cctxt:
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "nova/tests/fixtures.py", line 492, in _wrap_target_cell
    with self._cell_lock.write_lock():
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional/local/lib/python2.7/site-packages/fasteners/lock.py", line 219, in write_lock
    " escalation not allowed" % me)
RuntimeError: Reader <_MainThread(MainThread, started 140521207375680)> to writer privilege escalation not allowed

2019-05-21 21:32:12,672 ERROR [nova.api.openstack.wsgi] Unexpected exception in API method
Traceback (most recent call last):
  File "nova/api/openstack/wsgi.py", line 671, in wrapped
    return f(*args, **kwargs)
  File "nova/api/validation/__init__.py", line 110, in wrapper
    return func(*args, **kwargs)
  File "nova/api/validation/__init__.py", line 110, in wrapper
    return func(*args, **kwargs)
  File "nova/api/validation/__init__.py", line 110, in wrapper
    return func(*args, **kwargs)
  File "nova/api/validation/__init__.py", line 110, in wrapper
    return func(*args, **kwargs)
  File "nova/api/validation/__init__.py", line 110, in wrapper
    return func(*args, **kwargs)
  File "nova/api/validation/__init__.py", line 110, in wrapper
    return func(*args, **kwargs)
  File "nova/api/validation/__init__.py", line 110, in wrapper
    return func(*args, **kwargs)
  File "nova/api/validation/__init__.py", line 110, in wrapper
    return func(*args, **kwargs)
  File "nova/api/validation/__init__.py", line 110, in wrapper
    return func(*args, **kwargs)
  File "nova/api/validation/__init__.py", line 110, in wrapper
    return func(*args, **kwargs)
  File "nova/api/validation/__init__.py", line 110, in wrapper
    return func(*args, **kwargs)
  File "nova/api/openstack/compute/servers.py", line 661, in create
    nova_context.get_admin_context(), ['nova-compute'])
  File "nova/objects/service.py", line 515, in get_minimum_version_all_cells
    binaries)
  File "nova/context.py", line 564, in scatter_gather_all_cells
    fn, *args, **kwargs)
  File "nova/context.py", line 452, in scatter_gather_cells
    with target_cell(context, cell_mapping) as cctxt:
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "nova/tests/fixtures.py", line 492, in _wrap_target_cell
    with self._cell_lock.write_lock():
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional/local/lib/python2.7/site-packages/fasteners/lock.py", line 219, in write_lock
    " escalation not allowed" % me)
RuntimeError: Reader <_MainThread(MainThread, started 140534644954944)> to writer privilege escalation not allowed

2019-05-21 21:30:53,495 ERROR [nova.api.openstack.wsgi] Unexpected exception in API method
Traceback (most recent call last):
  File "nova/api/openstack/wsgi.py", line 671, in wrapped
    return f(*args, **kwargs)
  File "nova/api/openstack/compute/hypervisors.py", line 202, in index
    return self._index(req, limit=limit, marker=marker, links=True)
  File "nova/api/openstack/compute/hypervisors.py", line 218, in _index
    marker=marker, links=links)
  File "nova/api/openstack/compute/hypervisors.py", line 151, in _get_hypervisors
    context, hypervisor_match)
  File "nova/api/openstack/compute/hypervisors.py", line 101, in _get_compute_nodes_by_name_pattern
    context, hostname_match)
  File "nova/compute/api.py", line 5286, in compute_node_search_by_hypervisor
    with nova_context.target_cell(context, cell) as cctxt:
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "nova/tests/fixtures.py", line 492, in _wrap_target_cell
    with self._cell_lock.write_lock():
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional/local/lib/python2.7/site-packages/fasteners/lock.py", line 219, in write_lock
    " escalation not allowed" % me)
RuntimeError: Reader <_MainThread(MainThread, started 140089940105024)> to writer privilege escalation not allowed

The errors look more or less the same to me. If I could get indication if I should mask 0.15 in global-requirements.txt (as a known bad release) that’d be nice, right now it’s just constraints that are keeping openstack safe.

Thanks

[cybertron]

Okay, luckily I was able to dredge up some memories of this. 😃

It does still reproduce with the nova functional tests. I created a fresh Fedora VM and installed the necessary bits, then tweaked the package versions to use the latest fasteners, and my first test run failed on this error. Here are the highlights of what I did:

git clone https://opendev.org/openstack/nova
cd nova
sudo dnf install tox
sudo dnf install gcc-c++
tox -e bindep
# Install all of the missing packages reported by that, e.g.
sudo dnf install -y graphviz libffi-devel libxslt-devel lsscsi mysql mysql-devel mysql-server openssl pcre-devel postgresql postgresql-server sqlite-devel
tox -e functional # This should succeed, it will take a while
.tox/functional/bin/pip install -U fasteners
tox -e functional # This should eventually fail, it might take more than one run

You don’t necessarily have to use Fedora, bindep is distro-aware so it will give you the right package names for your distro. You’ll need to figure out the tox and gcc package names if you don’t already have them, of course.

[psarka]

I’m closing this, as the issue here most likely was related to spawn_n, which is solved in fasteners 0.18, and oslo.concurency fixed it independently as well.

I don’t have any guidance on how nova should migrate to fixed versions, the project is very complicated 😃

Let me know if you ever try the fixed version and it does not work.