Exception raised trying to collect facts for file
See original GitHub issueDescribe the bug
When attempting to execute the files.file
or files.directory
operations defined here an exception is raised while attempting to execute the file fact.
This error is new as of v1.4.5 and v1.4.6. Previous versions work without error.
To Reproduce
Clone the repository at github.com/mitodl/ol-infrastructure and create a deploy file that contains:
from bilder.components.hashicorp.steps import install_hashicorp_products
from bilder.components.hashicorp.vault.models import Vault
install_hashicorp_products(Vault())
Expected behavior
The expected behavior is to download and unpack the Vault binary and then set the permissions and create a configuration directory. Instead it generates a traceback.
Meta
- Include output of
pyinfra --support
.
--> Support information:
If you are having issues with pyinfra or wish to make feature requests, please
check out the GitHub issues at https://github.com/Fizzadar/pyinfra/issues .
When adding an issue, be sure to include the following:
System: Linux
Platform: Linux-5.12.13-1-MANJARO-x86_64-with-glibc2.32
Release: 5.12.13-1-MANJARO
Machine: x86_64
pyinfra: v1.4.6
Executable: /home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/bin/pyinfra
Python: 3.8.6 (CPython, GCC 10.2.0)
- How was pyinfra installed (source/pip)? pip
- Include pyinfra-debug.log (if one was created)
File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra_cli/main.py", line 218, in cli
_main(*args, **kwargs)
File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra_cli/main.py", line 604, in _main
run_ops(state, serial=serial, no_wait=no_wait)
File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/operations.py", line 375, in run_ops
_run_single_op(state, op_hash)
File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/operations.py", line 339, in _run_single_op
if not greenlet.get():
File "src/gevent/greenlet.py", line 803, in gevent._gevent_cgreenlet.Greenlet.get
File "src/gevent/greenlet.py", line 371, in gevent._gevent_cgreenlet.Greenlet._raise_exception
File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/gevent/_compat.py", line 65, in reraise
raise value.with_traceback(tb)
File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run
File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/operations.py", line 134, in _run_server_op
status = _run_shell_command(state, host, command, global_kwargs, executor_kwargs)
File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/operations.py", line 48, in _run_shell_command
status, combined_output_lines = command.execute(state, host, executor_kwargs)
File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/command.py", line 104, in execute
return host.run_shell_command(
File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/host.py", line 200, in run_shell_command
return self.executor.run_shell_command(self.state, self, *args, **kwargs)
File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/connectors/ssh.py", line 277, in run_shell_command
actual_command = command.get_raw_value()
File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/command.py", line 91, in get_raw_value
return self.separator.join(self._get_all_bits(
File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/command.py", line 81, in _get_all_bits
bit = bit_accessor(bit)
File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/command.py", line 92, in <lambda>
lambda bit: bit.get_raw_value(),
File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/command.py", line 91, in get_raw_value
return self.separator.join(self._get_all_bits(
File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/command.py", line 84, in _get_all_bits
bit = shlex_quote(bit)
File "/home/tmacey/.pyenv/versions/3.8.6/lib/python3.8/shlex.py", line 325, in quote
if _find_unsafe(s) is None:
TypeError: expected string or bytes-like object
- Consider including output with
-vv
and--debug
.
Loaded fact sha256_file (path=/tmp/vault.zip)
[3.88.169.89] noop: file /tmp/vault.zip has already been downloaded
[pyinfra.api.operation] Adding operation, {'Install Hashicorp Products | Unzip vault'}, opOrder=(124, 65), opHash=7b609db36deb56f0c3e4ce53d1a2453fdaa7b6de
[pyinfra.api.operation] Adding operation, {'Install Hashicorp Products | Ensure vault binary is executable'}, opOrder=(124, 71), opHash=3293f92ef79521f2b72f9af8c9dffa6116146892
[pyinfra.api.facts] Getting fact: file (path=/usr/local/bin/vault) (ensure_hosts: (Host(3.88.169.89),))
Traceback (most recent call last):
File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run
File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/host.py", line 200, in run_shell_command
return self.executor.run_shell_command(self.state, self, *args, **kwargs)
File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/connectors/ssh.py", line 277, in run_shell_command
actual_command = command.get_raw_value()
File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/command.py", line 102, in get_raw_value
return self.separator.join(self._get_all_bits(
File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/command.py", line 92, in _get_all_bits
bit = bit_accessor(bit)
File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/command.py", line 103, in <lambda>
lambda bit: bit.get_raw_value(),
File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/command.py", line 102, in get_raw_value
return self.separator.join(self._get_all_bits(
File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/command.py", line 95, in _get_all_bits
bit = shlex_quote(bit)
File "/home/tmacey/.pyenv/versions/3.8.6/lib/python3.8/shlex.py", line 325, in quote
if _find_unsafe(s) is None:
TypeError: expected string or bytes-like object
2021-06-28T21:04:59Z Traceback (most recent call last):
File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run
File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/host.py", line 200, in run_shell_command
return self.executor.run_shell_command(self.state, self, *args, **kwargs)
File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/connectors/ssh.py", line 277, in run_shell_command
actual_command = command.get_raw_value()
File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/command.py", line 102, in get_raw_value
return self.separator.join(self._get_all_bits(
File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/command.py", line 92, in _get_all_bits
bit = bit_accessor(bit)
File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/command.py", line 103, in <lambda>
lambda bit: bit.get_raw_value(),
File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/command.py", line 102, in get_raw_value
return self.separator.join(self._get_all_bits(
File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/command.py", line 95, in _get_all_bits
bit = shlex_quote(bit)
File "/home/tmacey/.pyenv/versions/3.8.6/lib/python3.8/shlex.py", line 325, in quote
if _find_unsafe(s) is None:
TypeError: expected string or bytes-like object
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/gevent/hub.py", line 624, in print_exception
context = self.format_context(context)
File "/home/tmacey/.pyenv/versions/3.8.6/lib/python3.8/pprint.py", line 67, in saferepr
return _safe_repr(object, {}, None, 0, True)[0]
File "/home/tmacey/.pyenv/versions/3.8.6/lib/python3.8/pprint.py", line 569, in _safe_repr
rep = repr(object)
File "src/gevent/greenlet.py", line 533, in gevent._gevent_cgreenlet.Greenlet.__repr__
File "src/gevent/greenlet.py", line 539, in gevent._gevent_cgreenlet.Greenlet._formatinfo
File "src/gevent/greenlet.py", line 539, in gevent._gevent_cgreenlet.Greenlet._formatinfo
File "src/gevent/greenlet.py", line 558, in gevent._gevent_cgreenlet.Greenlet._formatinfo
File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/command.py", line 80, in __repr__
return 'StringCommand({0})'.format(self.get_masked_value())
File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/command.py", line 109, in get_masked_value
for bit in self._get_all_bits(lambda bit: bit.get_masked_value())
File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/command.py", line 95, in _get_all_bits
bit = shlex_quote(bit)
File "/home/tmacey/.pyenv/versions/3.8.6/lib/python3.8/shlex.py", line 325, in quote
if _find_unsafe(s) is None:
TypeError: expected string or bytes-like object
Traceback (most recent call last):
File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run
File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/host.py", line 200, in run_shell_command
return self.executor.run_shell_command(self.state, self, *args, **kwargs)
File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/connectors/ssh.py", line 277, in run_shell_command
actual_command = command.get_raw_value()
File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/command.py", line 102, in get_raw_value
return self.separator.join(self._get_all_bits(
File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/command.py", line 92, in _get_all_bits
bit = bit_accessor(bit)
File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/command.py", line 103, in <lambda>
lambda bit: bit.get_raw_value(),
File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/command.py", line 102, in get_raw_value
return self.separator.join(self._get_all_bits(
File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/command.py", line 95, in _get_all_bits
bit = shlex_quote(bit)
File "/home/tmacey/.pyenv/versions/3.8.6/lib/python3.8/shlex.py", line 325, in quote
if _find_unsafe(s) is None:
TypeError: expected string or bytes-like object
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/gevent/hub.py", line 624, in print_exception
context = self.format_context(context)
File "/home/tmacey/.pyenv/versions/3.8.6/lib/python3.8/pprint.py", line 67, in saferepr
return _safe_repr(object, {}, None, 0, True)[0]
File "/home/tmacey/.pyenv/versions/3.8.6/lib/python3.8/pprint.py", line 569, in _safe_repr
rep = repr(object)
File "src/gevent/greenlet.py", line 533, in gevent._gevent_cgreenlet.Greenlet.__repr__
File "src/gevent/greenlet.py", line 539, in gevent._gevent_cgreenlet.Greenlet._formatinfo
File "src/gevent/greenlet.py", line 539, in gevent._gevent_cgreenlet.Greenlet._formatinfo
File "src/gevent/greenlet.py", line 558, in gevent._gevent_cgreenlet.Greenlet._formatinfo
File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/command.py", line 80, in __repr__
return 'StringCommand({0})'.format(self.get_masked_value())
File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/command.py", line 109, in get_masked_value
for bit in self._get_all_bits(lambda bit: bit.get_masked_value())
File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/command.py", line 95, in _get_all_bits
bit = shlex_quote(bit)
File "/home/tmacey/.pyenv/versions/3.8.6/lib/python3.8/shlex.py", line 325, in quote
if _find_unsafe(s) is None:
TypeError: expected string or bytes-like object
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run
File "src/gevent/greenlet.py", line 896, in gevent._gevent_cgreenlet.Greenlet._Greenlet__report_error
File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/gevent/hub.py", line 541, in handle_error
self.print_exception(context, type, value, tb)
File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/gevent/hub.py", line 627, in print_exception
context = repr(context)
File "src/gevent/greenlet.py", line 533, in gevent._gevent_cgreenlet.Greenlet.__repr__
File "src/gevent/greenlet.py", line 539, in gevent._gevent_cgreenlet.Greenlet._formatinfo
File "src/gevent/greenlet.py", line 539, in gevent._gevent_cgreenlet.Greenlet._formatinfo
File "src/gevent/greenlet.py", line 558, in gevent._gevent_cgreenlet.Greenlet._formatinfo
File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/command.py", line 80, in __repr__
return 'StringCommand({0})'.format(self.get_masked_value())
File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/command.py", line 109, in get_masked_value
for bit in self._get_all_bits(lambda bit: bit.get_masked_value())
File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/command.py", line 95, in _get_all_bits
bit = shlex_quote(bit)
File "/home/tmacey/.pyenv/versions/3.8.6/lib/python3.8/shlex.py", line 325, in quote
if _find_unsafe(s) is None:
TypeError: expected string or bytes-like object
2021-06-28T21:04:59Z <callback at 0x7fcfd9ddd5c0 stopped> failed with TypeError
--> An unexpected exception occurred in: src/bilder/images/edxapp/deploy.py:
File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra_cli/util.py", line 79, in exec_file
exec(PYTHON_CODES[filename], data)
File "src/bilder/images/edxapp/deploy.py", line 124, in <module>
install_hashicorp_products(hashicorp_products)
File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/deploy.py", line 142, in decorated_func
func(*args, **kwargs)
File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/src/bilder/components/hashicorp/steps.py", line 71, in install_hashicorp_products
files.file(
File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/operation.py", line 365, in decorated_func
commands = unroll_generators(func(*actual_args, **actual_kwargs))
File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/util.py", line 192, in unroll_generators
for item in generator:
File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/operations/files.py", line 1075, in file
info = host.get_fact(File, path=path)
File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/host.py", line 138, in get_fact
return get_host_fact(self.state, self, name_or_cls, args=args, kwargs=kwargs)
File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/facts.py", line 336, in get_host_fact
fact_data = get_facts(state, name, args=args, kwargs=kwargs, ensure_hosts=(host,))
File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/facts.py", line 276, in get_facts
status, combined_output_lines = greenlet.get()
File "src/gevent/greenlet.py", line 803, in gevent._gevent_cgreenlet.Greenlet.get
File "src/gevent/greenlet.py", line 371, in gevent._gevent_cgreenlet.Greenlet._raise_exception
File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/gevent/_compat.py", line 65, in reraise
raise value.with_traceback(tb)
File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run
File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/host.py", line 200, in run_shell_command
return self.executor.run_shell_command(self.state, self, *args, **kwargs)
File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/connectors/ssh.py", line 277, in run_shell_command
actual_command = command.get_raw_value()
File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/command.py", line 102, in get_raw_value
return self.separator.join(self._get_all_bits(
File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/command.py", line 92, in _get_all_bits
bit = bit_accessor(bit)
File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/command.py", line 103, in <lambda>
lambda bit: bit.get_raw_value(),
File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/command.py", line 102, in get_raw_value
return self.separator.join(self._get_all_bits(
File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/command.py", line 95, in _get_all_bits
bit = shlex_quote(bit)
File "/home/tmacey/.pyenv/versions/3.8.6/lib/python3.8/shlex.py", line 325, in quote
if _find_unsafe(s) is None:
TypeError: expected string or bytes-like object
Issue Analytics
- State:
- Created 2 years ago
- Comments:6 (6 by maintainers)
Top Results From Across the Web
Exception handling - Wikipedia
In computing and computer programming, exception handling is the process of responding to the occurrence of exceptions – anomalous or exceptional conditions ...
Read more >Clean Code and the Art of Exception Handling - Toptal
In most languages, raising a new exception from within a handler will cause the original exception to be lost forever, so it's better...
Read more >What to Except When You're Excepting: Practice Pointers for ...
A practitioner will find it far more successful to focus on his or her strongest points and fully articulate the legal and factual...
Read more >How can I solve "java.lang.NoClassDefFoundError"?
Class file located, Exception raised while initializing static blocks. In the original question, it was the first case which can be corrected by...
Read more >Rule 103. Rulings on Evidence | LII / Legal Information Institute
It is designed to resolve doubts as to what testimony the witness would have in fact given, and, in nonjury cases, to provide...
Read more >Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start FreeTop Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
Top GitHub Comments
After testing it is unfortunately not fixed yet. For now I can coerce the
Path
objects tostr
in the function call. Longer term it’s a design question of whether you want to supportPath
objects as inputs to the function. If that is the case then it might be beneficial to add a helper that does that coercion early in the function calls for operations that take paths as input to make the internal representation uniform rather than having to do the coalescing at multiple locations throughout the code.I had tried out moving the change that you made in line 99 of https://github.com/Fizzadar/pyinfra/commit/b91281e9e00106e4c1860b0209aa3bd8abf0ee19 above line 94 where the current error is happening. That resolved the traceback noted here, but resulted in a different traceback at a separate code location because of the parameter still being a
Path
object. That is why I think this merits a more thorough design consideration as opposed to attempting to patch the immediate issue.@blarghmatey agreed - the commit fixes unrelated issues but I definitely jumped the gun a bit here! Would be great to chat real-time about this and understand the issue better before identifying a proper fix (I’ve emailed you!).