"Unable to open fd 8 for writing, quitting" on Linux

Relevant older issue: https://github.com/kislyuk/argcomplete/issues/142 where I believe users also reported it happening on Linux but it was closed without any resolution (e.g. via suggested use of temporary files).

In my case I have a stock Debian (testing/unstable mix), python 3.8.3rc1 and I observe the same error with our DataLad which recently was fixed up to work (again) with argcomplete and it works for everyone else on the team but (un)lucky me. In the session you below can see that I am getting the exception, and the process has no 8 fd among its fds:

$> _ARC_DEBUG=1 _ARGCOMPLETE=1 datalad --pdb -l 1 --help
[DEBUG  ] Command line args 1st pass. Parsed: Namespace() Unparsed: ['--pdb', '--help'] 
[DEBUG  ] Discovering plugins 
[DEBUG  ] Loading entrypoints 
[DEBUG  ] Generating detailed description for the parser 
[Level 5] Finished setup_parser 
> /home/yoh/proj/misc/argcomplete/argcomplete/__init__.py(188)__call__()
-> try:
(Pdb) l
183  	        except:
184  	            debug_stream = sys.stderr
185  	
186  	        if output_stream is None:
187  	            import pdb; pdb.set_trace()
188  ->	            try:
189  	                output_stream = os.fdopen(8, "wb")
190  	            except:
191  	                debug("Unable to open fd 8 for writing, quitting")
192  	                exit_method(1)
193  	
(Pdb) os.getpid()
1092474
(Pdb) 
[3]  + 1092474 suspended  _ARC_DEBUG=1 _ARGCOMPLETE=1 datalad --pdb -l 1 --help

$> ls -l /proc/1092474/fd 
total 0
lrwx------ 1 yoh yoh 64 May 19 13:48 0 -> /dev/pts/16
lrwx------ 1 yoh yoh 64 May 19 13:48 1 -> /dev/pts/16
lrwx------ 1 yoh yoh 64 May 19 13:48 2 -> /dev/pts/16
lr-x------ 1 yoh yoh 64 May 19 13:48 3 -> pipe:[41232747]
lr-x------ 1 yoh yoh 64 May 19 13:48 5 -> pipe:[41232748]

$> fg
[3]    1092474 continued  _ARC_DEBUG=1 _ARGCOMPLETE=1 datalad --pdb -l 1 --help
l
l
194  	        # print("", stream=debug_stream)
195  	        # for v in "COMP_CWORD COMP_LINE COMP_POINT COMP_TYPE COMP_KEY _ARGCOMPLETE_COMP_WORDBREAKS COMP_WORDS".split():
196  	        #     print(v, os.environ[v], stream=debug_stream)
197  	
198  	        ifs = os.environ.get("_ARGCOMPLETE_IFS", "\013")
199  	        if len(ifs) != 1:
200  	            debug("Invalid value for IFS, quitting [{v}]".format(v=ifs))
201  	            exit_method(1)
202  	
203  	        comp_line = os.environ["COMP_LINE"]
204  	        comp_point = int(os.environ["COMP_POINT"])
(Pdb) n
> /home/yoh/proj/misc/argcomplete/argcomplete/__init__.py(189)__call__()
-> output_stream = os.fdopen(8, "wb")
(Pdb) 
OSError: [Errno 9] Bad file descriptor
> /home/yoh/proj/misc/argcomplete/argcomplete/__init__.py(189)__call__()
-> output_stream = os.fdopen(8, "wb")
(Pdb) 
> /home/yoh/proj/misc/argcomplete/argcomplete/__init__.py(190)__call__()
-> except:
(Pdb) fg
*** NameError: name 'fg' is not defined
(Pdb) 
Traceback (most recent call last):
  File "/home/yoh/proj/misc/argcomplete/argcomplete/__init__.py", line 189, in __call__
    output_stream = os.fdopen(8, "wb")
  File "/usr/lib/python3.8/os.py", line 1023, in fdopen
    return io.open(fd, *args, **kwargs)
OSError: [Errno 9] Bad file descriptor

in the shell I do get calls which do not explicitly create fd 8 failing:

$> python -c 'import os; os.fdopen(8)'     
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/lib/python3.8/os.py", line 1023, in fdopen
    return io.open(fd, *args, **kwargs)
OSError: [Errno 9] Bad file descriptor
$> python -c 'import os; os.fdopen(8)' 8>&1
$>

[yarikoptic]

THANK YOU @evanunderscore for the guidance – I am ready to declare “success” in troubleshooting !

What version of argcomplete are you running?

since it is argcomplete which generates it, may be it should embed argcomplete version which generated it somewhere in the comment

TL;DR summary: it is interaction between

• argcomplete setting IFS=$'\013' which is later ‘exported’ into the underlying process:

_python_argcomplete() {
    local IFS=$'\013'
    local SUPPRESS_SPACE=0
    if compopt +o nospace 2> /dev/null; then
        SUPPRESS_SPACE=1
    fi
    COMPREPLY=( $(IFS="$IFS" \
                  COMP_LINE="$COMP_LINE" \
                  COMP_POINT="$COMP_POINT" \
                  COMP_TYPE="$COMP_TYPE" \
                  _ARGCOMPLETE_COMP_WORDBREAKS="$COMP_WORDBREAKS" \
                  _ARGCOMPLETE=1 \
                  _ARGCOMPLETE_SUPPRESS_SPACE=$SUPPRESS_SPACE \
                  __python_argcomplete_run "$1") )

• git-annex standalone (filed fresh bug report) to @joeyh build shim showing intolerance to that IFS causing it to segfault.

$> IFS=$'\013' /usr/lib/git-annex.linux/git version
[1]    1039928 segmentation fault (core dumped)  IFS=$'\013' /usr/lib/git-annex.linux/git version

That also explains why other @datalad/developers didn’t experience this – since they typically do not use standalone git-annex build.

I guess there is nothing wrong done on argcomplete side, so feel free to close this issue, unless you see that you could avoid setting IFS and set some other “argcomplete specific” variable so it might not cause disturbance in some weak tools underneath?

Some additional notes which are no longer pertinent probably

How did you generate it?

re-doing everything now after pip install --upgrade argcomplete to 1.12.0:

$> venvs/dev3/bin/register-python-argcomplete --shell bash `which datalad` >| /tmp/datalad-argcomplete
$> bash  # since I was in zsh
$> which datalad
$> source /tmp/datalad-argcomplete

and verified that it still doesn’t work by trying to complete datalad inst<TAB>

You can try manually removing 1>/dev/null 2>/dev/null from the line …

*$> sed -e 's, 1>/dev/null 2>&1,,g' /tmp/datalad-argcomplete > /tmp/datalad-argcomplete-noswallow
$> diff /tmp/datalad-argcomplete /tmp/datalad-argcomplete-noswallow
20c20
<         "$@" 8>&1 9>&2 1>/dev/null 2>&1
---
>         "$@" 8>&1 9>&2

[evanunderscore]

Thanks for the detailed notes @yarikoptic. Glad you managed to find the source of your problem!

Embedding the argcomplete version in the generated wrapper is a good idea, though currently argcomplete doesn’t have a __version__ and keeping one in sync with the package version number is a workflow problem that would need to be left to @kislyuk.