shellTask logs all stdout when `ShellTask(return_all=True)`

Context Currently, when prefect.tasks.shell.ShellTask is invoked with return_all=True, the shell task will not log the stdout through logger.debug.

This is defined in doc https://docs.prefect.io/api/latest/tasks/shell.html#shelltask and implement in the code. https://github.com/PrefectHQ/prefect/blob/73fe1eb57945fc13bdd36b361a6a17c3f324b1ba/src/prefect/tasks/shell.py#L107-L111

Problem

Sometimes, we execute a long-running shell process, for example, we are running dbt, which might take 20 to 40 minutes to run. the task would need to return the entire stdout log as its output, for the next task to parse and interpret the result. Meanwhile, We would also like to see the shell stdout logging out in real-time, which makes it easy for us to inspect issues and monitoring / debug shell runs on-the-go, instead of waiting for 20 minutes not knowing what is going on.

proposed change

                for raw_line in iter(sub_process.stdout.readline, b""):
                    line = raw_line.decode("utf-8").rstrip()
                    if self.return_all:
                        lines.append(line)
                    # we still log every line
                    self.logger.debug(line)

[dcalde]

Hi, should the log level be configurable or set to INFO? Anyone setting stream_output=True is likely to actually want to see the shell outputs in their logs without having to configure their agent to DEBUG. I was able to show the debug logs by configuring the ShellTask logger and adding a new StreamHandler when running locally, but not when deployed and running on the agent.

[dcalde]

@madkinsz thanks for the feedback. I have created a separate issue and referenced this discussion