Remote Python execution hooks

Description

Just a placeholder that eventually, if people need the feature and it’s easy enough to integrate / makes sense to integrate instead of just saying “do it yourself, it’s just Python” – we should consider making it easy to leverage tools for remote Python execution/RPC. Modern-ly useful candidates include:

• execnet
  • actively developed but may only support Python up to 3.4?
  • transport is popen (local), ssh (via binary 😒) or direct socket (with bootstrappy remote script/server in-repo)
  • serialization is completely handrolled, using struct
  • used for distributed pytest so is at least somewhat battle tested
  • API relatively sane (gateway and channel objects, send/receive, light async available) if lacking explicit modern-style API docs (seems to be ONLY example/prose driven?)
• Pyro(4)
  • actively developed, seems to support at least up to 3.6 if not also 3.7
  • serializes with an AST-based standalone library called serpent (which explicitly states in its README why it exists and why Pyro is not using XML, JSON, Pickle etc)
  • transports with TCP/IP, Unix sockets, and can optionally use SSL/TLS
  • extremely featureful, including things like raising remote exceptions on the caller’s end, handling reconnects automatically, etc
  • nice clean API (understandably, at a high level, similar to execnet’s re: objects representing the remote interpreter, if slightly more modern feeling) including option to use decorators to declare RPC calls
• Mitogen
  • actively developed (maintainer is in comments below), supports up to Python 3.6 at least
  • serializes w/ pickle, unfortunately, though there is rationale
  • transport: actually not 100% sure but feels like direct socket connections combined with each Mitogen child being capable of routing requests onwards in a topology?
  • Relatively featureful and the features it does have are all very powerful-sounding
    • can run even if remote end has no free disk space, etc - neat!
    • Fancy caller-side module forwarding; not sure if/how the other options even handle non-stdlib imports.
    • remote-end logging module calls automatically forwarded back to caller
    • less immediately relevant to the average Fab single-node/list-of-sibling-nodes use case: some very neat emphases on trees of child processes, headless trees, and other topology strategies
  • Bootstraps automatically as long as remote end has SSH and Python, and does not require transferring a file first (compared to e.g. execnet requiring use of a remote script)
  • API somewhat more complex than eg Pyro/execnet (about 1 more layer of objects) but not necessarily in a bad way given its additional features/layout
• IPython’s kernel side
  • actively maintained (is it ever!)
  • uses ZMQ for transport between some client and a jupyter/ipython kernel process
  • serializes with JSON, alternately msgpack/pickle

Options that may not be feasible but are worth tracking:

• Pushy (link goes to blog entry specifically discussing integrating Fabric with it) - unsupported/archived since 2016, though per below, the author earlier expressed collaboration interest
• Salt - remote (pun not intended) chance one could try reusing salt’s ZMQ-driven remote execution bits, but at a very quick glance they do not appear to be factored out in a reusable fashion or anything
• Celery - similar to salt re: high level pub/sub/message-passing arch and need to try extracting a lower level flow, except requires a broker/queue system (vs ZMQ which can do direct socket to socket style connections) so even less feasible?
• etc (Python wiki list of projects) - last scanned early 2019, all obvious useful candidates already pulled out

---

Offhand I think the main draw would be to hook up your Fabric host/user/etc settings into such a tool. So probably something more along the lines of our Django contrib, instead of a core integration (especially since many users may not have Python remotely, so we absolutely would not want to require use of such a tool).

---

The author of Pushy also mentions the possibility of using his tool in Fabric as a potential layer on top of Paramiko; I guess the same approach could be applied to the other tools as well.

Not entirely sure if that would be a good way to go; plus is an eventual decoupling from SSH specifically; minus is that, well, we’re pretty SSH specific in a lot of ways (and use a lot of core Paramiko stuff) and I’m not positive it would be worth the loss of focus.

---

Originally submitted by Jeff Forcier (bitprophet) on 2010-09-20 at 11:43am EDT

Relations

• Duplicated by #246: Ability to easily remotely execute small Python snippets (rather than shell commands)

[anarcat]

i just found out about mitogen, and i think it would be an excellent backend for fabric.

in fact, I think it could advantageously replace a lot of fabric code (and possibly most of pyinvoke as well). it removes the “special case” distinction between fabric/invoke (ssh/local) and merges all of this in a magic “context” that also works across backends like “sudo” more transparently than fabric (IMHO).

right now I really feel the pain of this feature missing in fabric. i’m looking at implementing more and more complex business logic on the remote end. right now this involves transfering files back and forth with sftp to make sure things are correctly setup, and it’s kind of clunky. it would be so much better to just do that remotely: just open the file, look for content, search/replace and so on becomes much more intuitive.

it also makes it possible to write Python code that is reusable outside of fabric. right now I can’t really do that, unless i squint really hard, because fundamentally, the python code only runs locally. so i need to copy files (or processes?!) back and forth between hosts to do what i need. it’s far from ideal…

right now I’m at a junction: either I keep going the fabric route, or I just port all my code to mitogen and make a small executor wrapper around it… i don’t use the fabfile feature enough to warrant the complexity of fabric, while mitogen gives me much more powerful features for my main use case (which is bootstrapping and recovering servers from heterogeneous environments)…

but I understand that there are other use cases for fabric and invoke as well, of course. I just feel it would be much more natural to write native python code, even for existing functionality in Fabric (like code execution), because it would make error handling, pipe control, and existing reflexes work more or less normally…

[dw]

Hi @bitprophet, another note to say Python 3 has at long (loooong) last reached master, and the result is going to be on PyPI as the first stable series release by tomorrow evening. The ‘fakessh’ support is presently broken and not in scope, but I’m dying to get stuck to fixing it, and generalizing it for TCP/UDP/pipe forwarding – hoping after getting this Python 3 doldrums behind me, energy will return and lots of new stuff will happen real quick now 😃