[BUG] Installing a salt-minion v3001 from pypi tarball (via pip) versus installing using setup.py from git repository

Description Sorry for the length of this description, I literally am trying to drop all of the facts I know about it because the symptoms are confusing and I really want this thing to work. There are sub-topics, so you can just skip to the question and skim backwards for what’s relevant.

Comprehensive Details So, I’ve been chasing a vaporware (sorta?) bug the past week. I mean it’s definitely there, but I had a heck of a time trying to track it down because I realized earlier today that it’s because I’m apparently using two different environments to perform tests and was mistakenly thinking that they were the same due to selecting them both by the same version. These are both being done on the Windows 8.1 platform.

So the initial environment where the symptoms are occuring was simply using chocolatey to install Python 3.7. Once Python has been installed, and pip is updated. I’d install saltstack v3001 via pip by doing pip install salt==3001. At this point I’d use \Scripts\salt-minion.exe to start the minion with my configuration as LocalSystem (using salt’s nssm fork as a service).

It was during usage of this salt environment that I’d encounter an issue when using salt.modules.cmd that had the same symptoms as #57752. It would specifically occur when I was using “SoftPerfect RAM Disk” 3.4.7 (the freeware version over at https://www.majorgeeks.com/files/details/softperfect_ram_disk.html). Whenever I’d try to execute the ramdiskws.exe executable, the process would be created but never started. This hits the CreateProcessWithTokenW case that is under salt.utils.win_runas. What was happening is that an exception would get raised (OSError: ERROR_SUCCESS) during the advapi32 call to CreateProcessWithTokenW. This exception would exit the try-except block which results in the process never being resumed. If you guys consider this relevant, then the logs of troubleshooting this are at #57840. I closed that ticket because I had come to a conclusion that the issue was something completely different which has resulted in this current ticket.

At this point I made a snapshot of the VM that I was testing this in, cloned salt’s repository, and began a git-bisect to narrow down which commits were resulting in the ERROR_SUCCESS exception being raised when trying to create the process. To test each commit, I removed the prior installed commit and then used python setup.py build && python setup.py install to install each one that was being tested. After installation of each individual version, I’d use salt $M cmd.run 'ramdiskws -mount:"all"' 'cwd=C:\Program Files\SoftPerfect RAM Disk' hide_output=false output_loglevel=debug to reproduce the error.

Unfortunately during the whole git-bisect, only negatives were returned which I assumed were false-negatives since I could still reproduce the error in the first environment. Doing a sanity check, I immediately checked out and tested both v3001 and v3000.3 and was actually surprised that I couldn’t reproduce the original issues anymore. Instead the only issue that I was having was when using runas to execute as a different user (this was happening without the onlyif parameter which I resolved to being different than #57752).

Question So the question is pretty much, is there some sort of difference on the windows platform between the tarball that you guys deploy to pypi versus the package that’s built when running setup.py install? The wheel package (0.33.4) is installed within both environments using pkg/windows/req.txt, and although I’m not 100% sure… I think this is directly responsible for creating the salt-minion binary as Scripts/salt-minion.exe. As I mentioned in my comprehensive description, running Scripts/salt-minion.exe installed by pip is seeming to have differing effects than from running the Scripts/salt-minion script produced by setup.py through the Python interpreter.

I’ve currently modified my pip.installed state to install salt==3001 with use_wheel: false and no_binary: ':all:' as that seems to result in the original error going away, and then hopefully I can do a better job of narrowing down the other issues. Ideally, I’d like to avoid cloning the whole git repository as a workaround, but due to the compiler requirements for setproctitle and timelib (issue #57740) when trying to run the generated Scripts/salt-minion script an exception is raised due to the dependency checks of pkg_resource. The reason why I’m using pip to install salt is due to issue #51719 and some other non-extension-modules that I need to patch until you guys can make some more dents on Project 5.

Setup Not sure if you guys want to actually reproduce this, but since the section is here. I’ll include things.

Both are windows 8.1 environments, you can use packer (http://packer.io) and the eval-win81x86-enterprise.json template from github.com/boxcutter/windows if you’re okay with downloading the iso.

In both Windows 8.1 environments, chocolatey was used to install Python37. After upgrading pip, all of the requirements as listed under pkg/windows (other than timelib and setproctitle) are then installed (Make sure you get wheel == 0.33.4). In all situations, saltstack is running as LocalSystem using a service. An archived version of SoftPerfect RAM Disk (https://www.majorgeeks.com/files/details/softperfect_ram_disk.html) is used as the executable to run. After installing it, you cacn find the ramdiskws.exe binary under /Program Files/SoftPerfect RAM Disk. Snapshot a VM after installing if you’re using one.

I’m currently running these in a master+minion setup, but you should be able to use it in masterless mode with salt-call.

Environment 1 After setting up the base environment. Use pip to install salt with the requirement specifier salt == 3001. You can do this with the pip.installed state. If you don’t have a windows compiler as per #57740, you can either install with no-deps or you can comment them out of pkg/windows/req.txt anyways since they’re no-ops on windows.

Now you can run /Python/Scripts/salt-minion.exe with your configuration to get it started. Give it a good loglevel (info or debug)

Environment 2 Install git (you can use chocolatey here). Do a git clone https://github.com/saltstack/salt.git to get the repo. Then checkout tag v3001 by using git checkout -b release-3001 v3001. Once you have a working branch, you can patch out the no-ops on pkg/windows/req.txt. Then use python setup.py build, followed by python setup.py install.

To run salt, it should’ve produced a salt-minion script under /Python/Scripts/salt-minion. Use python to run that with your configuration and an info/debug loglevel.

Steps to Reproduce the behavior I’m doing this remotely, but you can use salt-call in either.

When running the following, it’ll block for a little bit before giving up.

salt $M cmd.run 'ramdiskws.exe -unmount:"all"' 'cwd=C:\Program Files\SoftPerfect RAM Disk' hide_output=false output_loglevel=debug

At this point, check the logs. You should see the following types of errors in the logs on the “pip”-installed environment 1. On environment 2 where setup.py was used to install it, you’ll only see the CreateProcessWithTokenW exception if you use the runas= parameter to cmd.run.

Log of OSError exception (ERROR_SUCCESS)

2020-06-30 03:16:32,887 [salt.state       :328 ][ERROR   ][3656] The operation completed successfully.

2020-06-30 03:24:15,157 [salt.loaded.int.module.cmdmod:1223][ERROR   ][1644] output: 
2020-06-30 03:25:06,859 [salt.loaded.int.module.cmdmod:1223][ERROR   ][3712] output: 
2020-06-30 03:25:31,884 [salt.loaded.int.module.cmdmod:1223][ERROR   ][3616] output: 
2020-06-30 03:28:51,827 [salt.minion      :1949][WARNING ][540] The minion function caused an exception
Traceback (most recent call last):
  File "c:\python37\lib\site-packages\salt\minion.py", line 1870, in _thread_return
    opts, data, func, args, kwargs
  File "c:\python37\lib\site-packages\salt\executors\direct_call.py", line 12, in execute
    return func(*args, **kwargs)
  File "c:\python37\lib\site-packages\salt\modules\cmdmod.py", line 1205, in run
    **kwargs
  File "c:\python37\lib\site-packages\salt\modules\cmdmod.py", line 424, in _run
    return win_runas(cmd, runas, password, cwd)
  File "c:\python37\lib\site-packages\salt\utils\win_runas.py", line 207, in runas
    environment=env,
  File "c:\python37\lib\site-packages\salt\platform\win.py", line 1161, in CreateProcessWithTokenW
    raise exc
OSError: The operation completed successfully.

Log of WSASelect exceptions originating from subprocess via vendored tornado

2020-06-30 03:28:52,002 [salt.utils.process:786 ][ERROR   ][540] An un-handled exception from the multiprocessing process 'ProcessPayload' was caught:
Traceback (most recent call last):
  File "c:\python37\lib\site-packages\salt\minion.py", line 1550, in _send_req_sync
    return channel.send(load, timeout=timeout)
  File "c:\python37\lib\site-packages\salt\utils\asynchronous.py", line 126, in wrap
    six.reraise(*results[1])
  File "c:\python37\lib\site-packages\salt\ext\six.py", line 693, in reraise
    raise value
  File "c:\python37\lib\site-packages\salt\utils\asynchronous.py", line 132, in _target
    result = io_loop.run_sync(lambda: getattr(self.obj, key)(*args, **kwargs))
  File "c:\python37\lib\site-packages\salt\ext\tornado\ioloop.py", line 454, in run_sync
    self.start()
  File "c:\python37\lib\site-packages\salt\ext\tornado\ioloop.py", line 865, in start
    event_pairs = self._impl.poll(poll_timeout)
  File "c:\python37\lib\site-packages\salt\ext\tornado\platform\select.py", line 64, in poll
    self.read_fds, self.write_fds, self.error_fds, timeout)
OSError: [WinError 10038] An operation was attempted on something that is not a socket

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "c:\python37\lib\site-packages\salt\utils\process.py", line 777, in wrapped_run_func
    return run_func()
  File "c:\python37\lib\multiprocessing\process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "c:\python37\lib\site-packages\salt\minion.py", line 1774, in _target
    run_func(minion_instance, opts, data)
  File "c:\python37\lib\site-packages\salt\minion.py", line 1768, in run_func
    return Minion._thread_return(minion_instance, opts, data)
  File "c:\python37\lib\site-packages\salt\minion.py", line 1988, in _thread_return
    ret, timeout=minion_instance._return_retry_timer()
  File "c:\python37\lib\site-packages\salt\minion.py", line 2213, in _return_pub
    ret_val = self._send_req_sync(load, timeout=timeout)
  File "c:\python37\lib\site-packages\salt\minion.py", line 1550, in _send_req_sync
    return channel.send(load, timeout=timeout)
  File "c:\python37\lib\site-packages\salt\utils\asynchronous.py", line 143, in __exit__
    self.close()
  File "c:\python37\lib\site-packages\salt\utils\asynchronous.py", line 108, in close
    io_loop.close(all_fds=True)
  File "c:\python37\lib\site-packages\salt\ext\tornado\ioloop.py", line 722, in close
    self._waker.close()
  File "c:\python37\lib\site-packages\salt\ext\tornado\platform\common.py", line 113, in close
    self.reader.close()
  File "c:\python37\lib\socket.py", line 420, in close
    self._real_close()
  File "c:\python37\lib\socket.py", line 414, in _real_close
    _ss.close(self)
OSError: [WinError 10038] An operation was attempted on something that is not a socket

Expected behavior This issue is a question on the potential differences between pip vs git installations. An expectation could be about what might cause these differing symptoms despite the same version, v3001, being installed (albeit differently).

Versions Report

salt --versions-report

Salt Version:
           Salt: 3001
 
Dependency Versions:
           cffi: 1.12.2
       cherrypy: 17.4.1
       dateutil: 2.8.0
      docker-py: Not Installed
          gitdb: 2.0.5
      gitpython: Not Installed
         Jinja2: 2.10.1
        libgit2: Not Installed
       M2Crypto: Not Installed
           Mako: 1.0.7
   msgpack-pure: Not Installed
 msgpack-python: 0.5.6
   mysql-python: Not Installed
      pycparser: 2.19
       pycrypto: 3.9.8
   pycryptodome: 3.9.7
         pygit2: Not Installed
         Python: 3.7.7 (tags/v3.7.7:d7c567b08f, Mar 10 2020, 09:44:33) [MSC v.1900 32 bit (Intel)]
   python-gnupg: 0.4.4
         PyYAML: 5.1.2
          PyZMQ: 18.0.1
          smmap: 2.0.5
        timelib: Not Installed
        Tornado: 4.5.3
            ZMQ: 4.3.1
 
System Versions:
           dist:   
         locale: cp1252
        machine: x86
        release: 8.1
         system: Windows
        version: 8.1 6.3.9600 SP0 

[OrangeDog]

I believe the current Python best-practice is to always install with pip, whether it’s a tarball, wheel, or source.

[arizvisa]

No worries on the delay. Thanks for looking into such a strange discrepancy.

I also still think that there’s some weird situation related to what setuptools is doing on the frozen .exe files that it generates in scripts/*. It’s been a while since I’ve done research into frozen modules, but there might be some different state when modules are unfrozen when ssm runs the executable as opposed to ssm running Python to interpret the script generated by the same pip install. Especially because if I use the python3 interpreter to run the scripts/salt-minion.py file directly, the token problems seems to change for some odd reason. So the other strangeness that I also included in the issue might not be related to you guys.

After I finish the current thing I’m working on (which might be a few weeks), I’ll rebuild the env and try to diff the tokens that the Salt-Minion-3001-Py3-x86.exe installed service, the pip-installed salt-minion.exe, and the pip-installed python scripts/salt-minion.py variations are running with…and then do the same for the token that’s being used to spawn the ramdisk process with.