Problems installing mitmproxy (requires cryptography>=3.3,<3.5 and zstandard>=0.11,<0.16)

Chaquopy version

11.0.0

Relevant parts of your build.gradle file

        ndk {
            abiFilters "armeabi-v7a", "arm64-v8a", "x86", "x86_64"
        }
        python {
            pip {
                install "mitmproxy"
            }
            pyc {
                src true
            }
        }

Describe your issue

I’m trying to install mitmproxy with your plugin but it fails when trying to install cryptography>=36,<37 and zstandard>=0.11,<0.17 .

Log for cryptography:

Collecting cryptography<37,>=36 (from mitmproxy==8.0.0.dev0)
  Using cached https://files.pythonhosted.org/packages/f9/4b/1cf8e281f7ae4046a59e5e39dd7471d46db9f61bb564fddbff9084c4334f/cryptography-36.0.1.tar.gz
    ERROR: Command errored out with exit status 1:
     command: /usr/bin/python3.8 -S -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-9we8ppnw/cryptography/setup.py'"'"'; __file__='"'"'/tmp/pip-install-9we8ppnw/cryptography/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' --no-user-cfg egg_info --egg-base pip-egg-info
         cwd: /tmp/pip-install-9we8ppnw/cryptography/
    Complete output (14 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-9we8ppnw/cryptography/setup.py", line 14, in <module>
        from setuptools_rust import RustExtension
    ModuleNotFoundError: No module named 'setuptools_rust'
    
    ERROR: Command errored out with exit status 1:

            =============================DEBUG ASSISTANCE==========================
            If you are seeing an error here please try the following to
            successfully install cryptography:
    
            Upgrade to the latest pip and try again. This will fix errors for most
            users. See: https://pip.pypa.io/en/stable/installing/#upgrading-pip
            =============================DEBUG ASSISTANCE==========================
    
    ----------------------------------------
    ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
ERROR: Failed to install cryptography<37,>=36 from https://files.pythonhosted.org/packages/f9/4b/1cf8e281f7ae4046a59e5e39dd7471d46db9f61bb564fddbff9084c4334f/cryptography-36.0.1.tar.gz (from mitmproxy==8.0.0.dev0).
For assistance, please raise an issue at https://github.com/chaquo/chaquopy/issues.
Or try using one of the following versions, which are available as pre-built wheels: ['2.8', '3.3.2'].
    ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

Failed to install cryptography<37,>=36 from https://files.pythonhosted.org/packages/f9/4b/1cf8e281f7ae4046a59e5e39dd7471d46db9f61bb564fddbff9084c4334f/cryptography-36.0.1.tar.gz (from mitmproxy==8.0.0.dev0).

Chaquopy: Exit status 1

> Task :app:generateDebugPythonRequirements FAILED
Execution failed for task ':app:generateDebugPythonRequirements'.
> Process 'command 'python3.8'' finished with non-zero exit value 1

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

Log for zstandard:

Collecting zstandard<0.17,>=0.11 (from mitmproxy==8.0.0.dev0)
  Downloading https://files.pythonhosted.org/packages/7d/74/50521e915f23ea5fecee190e933527174a34e8623317369d3b9a13ba109c/zstandard-0.16.0.tar.gz (563kB)
    ERROR: Command errored out with exit status 1:
    ERROR: Command errored out with exit status 1:

     command: /usr/bin/python3.8 -S -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-u4_zlr4a/zstandard/setup.py'"'"'; __file__='"'"'/tmp/pip-install-u4_zlr4a/zstandard/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' --no-user-cfg egg_info --egg-base pip-egg-info
         cwd: /tmp/pip-install-u4_zlr4a/zstandard/
    Complete output (7 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-u4_zlr4a/zstandard/setup.py", line 90, in <module>
        setup_zstd.get_c_extension(
      File "/tmp/pip-install-u4_zlr4a/zstandard/setup_zstd.py", line 75, in get_c_extension
        raise Exception("unhandled compiler type: %s" % compiler.compiler_type)
    Exception: unhandled compiler type: disabled
    ----------------------------------------
    ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
    ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

ERROR: Failed to install zstandard<0.17,>=0.11 from https://files.pythonhosted.org/packages/7d/74/50521e915f23ea5fecee190e933527174a34e8623317369d3b9a13ba109c/zstandard-0.16.0.tar.gz (from mitmproxy==8.0.0.dev0).
Failed to install zstandard<0.17,>=0.11 from https://files.pythonhosted.org/packages/7d/74/50521e915f23ea5fecee190e933527174a34e8623317369d3b9a13ba109c/zstandard-0.16.0.tar.gz (from mitmproxy==8.0.0.dev0).

For assistance, please raise an issue at https://github.com/chaquo/chaquopy/issues.
Chaquopy: Exit status 1

> Task :app:generateDebugPythonRequirements FAILED
Execution failed for task ':app:generateDebugPythonRequirements'.
> Process 'command 'python3.8'' finished with non-zero exit value 1

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

[mhsmith]

I can post/send you the recipes if they can be useful (e.g. integrating them in your repo), just let me know where.

That would be great, you can make a PR in the other repository, or email them to me if you prefer.

lack of ability to run a python script directly. In particular, I had to inspect how the mitmdump program was implemented and created a python script to import it and run it via functions calls.

In this case the script is auto-generated by the entry_points section in setup.py, which tells you exactly what function you need to call. I agree it might be useful to provide an API to call entry-point scripts in a more command-line-like way. But as you discovered, these scripts often assume they’re the only thing that will ever be run in the lifetime of the process, so calling them multiple times doesn’t work and you end up needing to use the function call interface anyway.

lack of ability to reset the python environment

Because many native modules store state in static variables, the only reliable way to reset the Python environment is to exit the whole process. This is generally discouraged on Android, but it’s still possible with the standard Java or Python APIs. Then you can restart the process with a new intent, either from a different app, or a different process in the same app.

[emanuele-f]

That’s great!

I’ve just published an initial version of the PCAPdroid mitm addon. There are a lot of things to refine but I’m satisfied with the result. Thank you for providing this great framework!