Migrate STDOUT/STDIN exchanges from asynchio pipes to queues

Migrated from GitLab: https://gitlab.com/meltano/meltano/-/issues/2900

Originally created by @aaronsteers on 2021-08-18 21:44:57

---

As discussed in #2743 and this comment (https://gitlab.com/meltano/meltano/-/issues/2743#note_569087851), asyncio queues can be used in place of pipes to send data between processes.

cc @pandemicsyn

---

From our code:

https://gitlab.com/meltano/meltano/-/blob/c58cb8e56be1b8bae0eefcb6ba906e3c2010852e/src/meltano/core/logging/output_logger.py#L201

   async def _read_from_fd(self, read_fd):
        # Since we're redirecting our own stdout and stderr output,
        # the line length limit can be arbitrarily large.
        line_length_limit = 1024 * 1024 * 1024  # 1 GiB

        reader = asyncio.StreamReader(limit=line_length_limit)
        read_protocol = asyncio.StreamReaderProtocol(reader)

        loop = asyncio.get_event_loop()
        read_transport, _ = await loop.connect_read_pipe(  # <<<<
            lambda: read_protocol, os.fdopen(read_fd)
        )

        await capture_subprocess_output(reader, self)

From https://docs.python.org/3/library/asyncio-platforms.html#windows:

SelectorEventLoop has the following limitations:

• …
• Pipes are not supported, so the loop.connect_read_pipe() and loop.connect_write_pipe() methods are not implemented.

---

Alternative implementation using queues from here:

async def first_pipe_cmd(command, queue, cwd="."):
   proc = await asyncio.create_subprocess_shell(
           command,
           stdout=asyncio.subprocess.PIPE,
           #stderr=asyncio.subprocess.PIPE, 
           cwd=cwd)
  #await asyncio.wait(_outstream_handler(proc.stderr, "stderr", "first_pipe_cmd")) #Broken at the moment
   data ="first"
   while data:
       data = await proc.stdout.readline()
       line = data.decode()
       if data: await queue.put(line)
       logging.info(f"Queue data for processes, data is {line}")
   logging.info("First piped process has completed")

[aaronsteers]

@tayloramurphy - thanks for the ping.

On reviewing this, I don’t think it is a high priority as of now, given that:

1. BATCH messages mitigate the performance side of the issue. (Mitigated in part although admittedly not in full.)
2. Windows support is working now natively for both meltano run and elt. (Interop was one of the drivers here, if I remember correctly.)

There are still potential benefits, but there are risks also in terms of compatibility and performance. There’s another discussion (I think I the SDK repo) about preformance benchmarking and I think we’d want that process be in place before making a big change like this.

It’s also arguable that investing in adding BATCH support to stream maps (along with good docs) may be a better investment, since that would bring similar benefits of reducing memory pressure to legacy taps and targets as well as sdk-based ones.

@BuzzCutNorman, @visch - do you see strong value on this as of now or would you agree it’s ok to deprioritize?

[tayloramurphy]

Windows does not work with elt

Thankfully we’ve documented it as such in https://docs.meltano.com/guide/installation-guide#windows 😅