[BUG] Running python step fails when it tries to install the wrong dependency

Describe the bug When running a python step that depends on py-august package that exports august package (and adding a dependency via magic comment). The executor still tries to install the august package which fails overall step because it depends on Rust (https://pypi.org/project/august/)

To Reproduce Steps to reproduce the behavior:

Run the following Python step

# pipedream add-package py-august
from august.api import Api
from august.authenticator import Authenticator, AuthenticationState


def handler(pd: "pipedream"):
  return {"test": True}

It’ll fail with the error like:

Command failed: python3 -m pip install --target=/tmp/__pdg__/dist/python -r requirements.txt --upgrade WARNING: The directory '/home/sbx_user1051/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you should use sudo's -H flag. error: subprocess-exited-with-error × Preparing metadata (pyproject.toml) did not run successfully. │ exit code: 1 ╰─> [6 lines of output] Cargo, the Rust package manager, is not installed or is not on PATH. This package requires Rust and Cargo to compile extensions. Install it through the system's package manager or via https://rustup.rs/ Checking for Rust toolchain.... [end of output] note: This error originates from a subprocess, and is likely not a problem with pip. error: metadata-generation-failed × Encountered error while generating package metadata. ╰─> See above for output. note: This is an issue with the package mentioned above, not pip. hint: See above for details. WARNING: You are using pip version 22.0.4; however, version 22.2.2 is available. You should consider upgrading via the '/var/lang/bin/python3 -m pip install --upgrade pip' command. 

Expected behavior The environment does not try to install august package and only installs py-august (and doesn’t crash as a consequence 😛)

[Hassan-Abbas-SQA]

@Stvad thanks for reporting the issue iam able to reproduce the issue

[dylburger]

@stefanbohacek We just corrected the issue. You should be able to from mastodon import Mastodon successfully now.

@elibenton We also addressed the issue with newspaper3k. Try from newspaper import Article again.

The root issue is the same in each case. The system we use to magically map import statements to PyPI packages works like this:

1. Scan code and collect imports
2. Check if a package with the import name exists in PyPI. If yes, write to the requirements.txt file.
3. (handle other edge cases)

In these two cases, there are technically packages for mastodon and newspaper. mastodon is unmaintained and doesn’t properly package Python code, so fails on install. newspaper points to the Python 2 version of the package. So these packages were written to the requirements.txt file, but failed to install.

We added custom mappings to point these import names to the correct PyPI package names:

newspaper:newspaper3k
mastodon:Mastodon.py

If anyone else finds this issue and is having the same problem, please open up a new GitHub issue with the build error you’re having and we’ll take a look. We also plan to improve the mapping system, and will likely provide users the ability to provide a custom requirements file so you can override the magic imports if necessary.

@Stvad as you noted, your issue results from Rust not being part of our execution environment. I’m keeping this issue open so we can address that later.