File operations fail when folder or file name contains parentheses

Describe the bug

File operations (sync, put, get) fail when a folder or file name contains parentheses, although these are valid *nix file and folder names.

To Reproduce

I discovered this whilst attempting to sync a folder containing a Calibre ebooks library. Calibre uses the following on-disk structure throughout the library: “a_calibre_library/author name/book title (1)” where the number in parenthesis is an integer that is auto-incremented for each new book.

A little testing showed that more generally, any folder or file containing parenthesis causes file operations to fail.

Create example directory tree:

mkdir -p "./testdir1/folder_with_parens_(foo)"
touch "./testdir1/folder_with_parens_(foo)/file_without_parens.txt"

Write a deploy containing the following operation:

files.sync(src="testdir1/", dest="/tmp/testdir1")

Result (excerpt from pyinfra output with -v):

--> Preparing operations...
Loading: deploy.py
Loaded fact link (path=/tmp/testdir1)
Loaded fact directory (path=/tmp/testdir1)
[testhost] sh: 1: Syntax error: "(" unexpected
[testhost] Error: could not load fact: directory (path=/tmp/testdir1/folder_with_parens_(foo))
Loaded fact directory (path=/tmp/testdir1/folder_with_parens_(foo))
--> An unexpected exception occurred in: deploy.py

The same occurs for files.put() - (tested). Presumably files.get() would fail similarly, though not tested.

Expected behavior

Files and/or folders sync/put/get without error

Meta

• Include output of pyinfra --support.

  –> Support information:

  If you are having issues with pyinfra or wish to make feature requests, please check out the GitHub issues at https://github.com/Fizzadar/pyinfra/issues . When adding an issue, be sure to include the following:

  System: Linux Platform: Linux-5.8.0-53-generic-x86_64-with-glibc2.31 Release: 5.8.0-53-generic Machine: x86_64 pyinfra: v1.4.2 Executable: /path/to/bin/pyinfra Python: 3.9.2 (CPython, GCC 9.3.0)

• How was pyinfra installed (source/pip)?

pip, into a virtualenv

• Include pyinfra-debug.log (if one was created)

No log

• Consider including output with -vv and --debug.

Didn’t do this.

Workarounds

files.rsync() succeeds as a workaround, though doesn’t work as a solution for single file operations.

More generally, I can fix the issue entirely for all the operations by applying the following hacks to “pyinfra/api/connectors/util.py”:

Add this to imports:

import shlex

Wrap the unix path with shlex.quote(path):

def escape_unix_path(path):
'''
Escape unescaped spaces in a (unix) path.
'''

if PurePath and isinstance(path, PurePath):
    path = str(path)

# return UNIX_PATH_SPACE_REGEX.sub(r'\1\\ ', path)
return shlex.quote(UNIX_PATH_SPACE_REGEX.sub(r'\1\\ ', path))

I think in Python 2.7 “pipes.quote(path)” does the same thing, for what it’s worth.

I thought about just escaping the parens by modifying the existing regex to escape parens, but that didn’t feel like a general enough solution. shlex did the trick, and maybe would catch any special shell characters in a path string.

That might not be the best general solution, but hopefully that’s enough of a hint to point you in the right direction.

[Pimmelton]

Confirmed fixed using both my contrived test cases and real deploy.

Nicely done, and thanks.

[Fizzadar]

Excellent news, thanks for confirming that!