Buildozer silently ignores local recipes – and this is bad

tl;dr

Relative dirnames in p4a.local_recipes and --local_recipes have been broken for a year or two. Until this issue is resolved, please replace the ./ prefixing these dirnames with ../../../../ instead: e.g.,

# Instead of this...
p4a.local_recipes = ./p4a-recipes/

# ...just do this!
p4a.local_recipes = ../../../../p4a-recipes/

Rejoice as local recipes suddenly work. If you too are now facepalming, this emoji’s for you. 🤦

Versions or It Didn’t Happen

• Python: Any
• OS: Gentoo Linux, of course!
• Buildozer: 1.3.0

Exegesis or It Didn’t Happen

Buildozer silently ignores relative dirnames for both the p4a.local_recipes setting in buildozer.spec files and the --local_recipes option accepted by most p4a subcommands. Since the only sane means of specifying a local recipe directory is with a relative dirname, local recipes have effectively been broken for at least a year or two… which is bad.

Clearly, local recipes used to work. As this issue will show, they no longer do. The breakage probably happened when Buildozer was refactored to change the current working directory (CWD) to the deeply nested .buildozer/android/platform/python-for-android/ subdirectory before running p4a. Previously, Buildozer must not have done that; it must have preserved the CWD as the current top-level project directory containing the root buildozer.spec file. Now, Buildozer changes the CWD to a deeply nested subdirectory breaking relative dirnames in buildozer.spec files.

Let’s dissect this. ✂️

buildozer.spec

Let’s use this spec file to build the Blap app, Buildozer!

[app]
p4a.local_recipes = ./p4a-recipes/
title = Blap
package.name = blap
source.dir = .
version = 0.0.1
requirements = python3>=3.9.0, kivy>=2.1.0, bleak>=0.14.0
android.accept_sdk_license = True
android.archs = arm64-v8a, armeabi-v7a

[buildozer]
log_level = 2

Let’s define a trivial recipe for the bleak requirement that mostly does nothing except exist!

$ mkdir -p p4a-recipes/bleak
$ cat << EOF > p4a-recipes/bleak/__init__.py
from pythonforandroid.recipe import PythonRecipe

class BleakRecipe(PythonRecipe):
    version = '0.14.0'
    url = 'https://pypi.python.org/packages/source/b/bleak/bleak-{version}.tar.gz'
    name = 'bleak'

recipe = BleakRecipe()
EOF

Lastly, let’s build the Android Blap app, Buildozer!

$ buildozer android debug

We are now ready to break you, Buildozer.

Logs

There’s little point in pasting the full output, so let’s reduce this to the only line that matters:

$ buildozer android debug
...
[INFO]:    	blap: min API 21, includes recipes (hostpython3, libffi, openssl, sdl2_image, sdl2_mixer, sdl2_ttf, sqlite3, python3, sdl2, setuptools, six, pyjnius, android), built for archs (arm64-v8a, armeabi-v7a)

Note the conspicuous absence of bleak above. So, Buildozer has failed to find our local p4a-recipes/bleak recipe. B-b-but whatever could the matter be…? Just kidding! I figured everything out already. It’s the pythonforandroid.recipe.Recipe.recipe_dirs() classmethod, which is painfully broken with respect to Buildozer’s CWD behaviour. </sigh>

It’s Raining Solutions

Ah-ha. Bet you weren’t expecting that one, were you, @misl6? Nobody ever submits solutions. They only complain endlessly. Right? I know. I’m usually one of those people, because I am lazy. But complaining mostly doesn’t work, as evidenced by the 199 currently open issues on this tracker.

So… I did a deep dive on this for the community. There are multiple intersecting issues here, each of which should be resolved before this issue is eventually closed. They all involve the aforementioned pythonforandroid.recipe.Recipe.recipe_dirs() classmethod, whose implementation is so naive it makes me just want to stop doing anything and play Persona 5 Royal for the 10th time. Specifically, that classmethod:

1. Isn’t tested by anything. Tests shouldn’t pass, because p4a.local_recipes and --local_recipes are both utterly broken – and have been that way for a few years now. But tests pass. Ergo, someone who is not me really needs to write working unit and integration tests ensuring this never happens again. Of course, test coverage is at a shocking low of 39%, so… what you gonna do? It doesn’t help that both Buildozer and python-for-android need additional tests:
   • Buildozer needs integration tests ensuring that python-for-android is actually respecting the Buildozer-specific p4a.local_recipes setting. It currently isn’t.
   • python-for-android needs unit tests ensuring that the Recipe.recipe_dirs() classmethod:
     • Raises exceptions when the local recipe directory doesn’t actually exist.
     • Registers the local recipe directory when that directory does exist.
2. Fails to log anything. Seriously. This classmethod needs to start logging something.
3. Fails to raise any exceptions. If the user explicitly specifies a local recipes directory (via either p4a.local_recipes or --local_recipes) that does not exist, this classmethod needs to raise a human-readable exception. Currently, this classmethod silently ignores this edge case – which is actually all cases, because this classmethod erroneously canonicalizes most relative dirnames to non-existent absolute dirnames. Cue more facepalm emojis. 🤦‍♂️ 🤦‍♀️
4. Fails to correctly canonicalize relative dirnames. When the user specifies a local recipes directory like p4a.local_recipes = ./p4a-recipes/ or --local_recipes=./p4a-recipes/, they expect the ./ prefixing that dirname to refer to the top-level project directory containing the root buildozer.spec file. Instead, Buildozer unexpectedly:
   1. Changes the current working directory (CWD) to a private Buildozer-isolated python-for-android subdirectory:

      # Cwd /home/itsmeleycec/blap/.buildozer/android/platform/python-for-android
      

   2. Eventually attempts to canonicalize that local recipes directory against that private Buildozer-isolated python-for-android subdirectory rather than against the top-level project directory. Why does this classmethod do that? Because this classmethod is naive:

          @classmethod
          def recipe_dirs(cls, ctx):
              recipe_dirs = []
              if ctx.local_recipes is not None:
                  recipe_dirs.append(realpath(ctx.local_recipes))  # <-- THIS IS BALLS
              ...
      

See the line conspicuously marked # <-- THIS IS BALLS? Yeah. That’s the problem. When that line is interpreted, ctx.local_recipes == './p4a-recipes/'. Since Buildozer changes the CWD beforehand to the .buildozer/android/platform/python-for-android/ subdirectory, passing that relative dirname to the os.path.realpath() function incorrectly expands that dirname to .buildozer/android/platform/python-for-android/p4a-recipes/.

Of course, that directory doesn’t actually exist – but recipe_dirs() doesn’t care. recipe_dirs() is naive. It crosses its fingers and just silently appends that non-existent directory to the returned list of recipe dirnames. Cue badness. 🤞

Here’s what the Recipe.recipe_dirs() classmethod needs to look like instead:

```python
from os.path import (
    basename, dirname, exists, isabs, isdir, isfile, join, realpath, split)
...

class Recipe(with_metaclass(RecipeMeta)):
    ...

    @classmethod
    def recipe_dirs(cls, ctx):
        recipe_dirs = []
        if ctx.local_recipes is not None:
            # If the local recipes directory is absolute, accept this directory
            # as is; else, canonicalize this relative directory against this
            # project's directory (rather than the current working directory).
            local_recipes_dir = realpath(
                ctx.local_recipes
                if isabs(ctx.local_recipes) else
                join(ctx.project_dir, ctx.local_recipes)
            )

            if not isdir(local_recipes_dir):
                raise ValueError(
                    f'Local recipes directory "{local_recipes_dir}" not found.')

            recipe_dirs.append(local_recipes_dir)
        ...

Crucially, we now:

1. Correctly canonicalize that directory against the project directory.
2. Raise an exception when that directory is missing.

Wunderbar. So, why haven’t I already submitted a PR conveniently solving this for everyone? Because there is no ctx.project_dir. I just made that up, you see. The pythonforandroid.build.Context class doesn’t seem to provide the project directory, which leaves that prospective PR with nowhere good to go.

Internal documentation in the python-for-android codebase is so scarce that I can’t be sure, though. The project directory must be accessibly stored somewhere, right? That’s the most important directory. Everything else is just a footnote to the project directory. It’d be kinda sad (but ultimately unsurprising) if the build context fails to capture the most important build context. The feeling of vertigo is now overwhelming. :face_with_spiral_eyes:

I now beseech someone with greater wisdom than me (…so, basically anyone) to do this for me.

Everything, It Now Makes Sense

There’s actually a related open issue. Ironically, that issue was submitted by @dlech – the active maintainer of Bleak, whose recipe prompted both of these issues. Double-ironically, @dlech also identified the wacky short-term workaround for this insane long-standing issue: just replace the ./ substring prefixing a p4a.local_recipes dirname with ../../../../. Since the .buildozer/android/platform/python-for-android subdirectory is nested four levels deep, four levels of “de-nesting” are required to break out of the Buildozer-isolated p4a directory and return back to the top-level project directory. Note the judicious use of adjectives like “wacky” and “insane.”

There’s also a fascinating Reddit thread on the /r/kivy subreddit debating this very topic. Numerous Reddit users self-report the same issue. Of course, no one knew how to resolve the issue or that there even was an issue. Instead, everyone uselessly flailed around and (presumably) eventually abandoned Kivy because nothing was working. I’ll admit I would have done the same in their position.

The fact that p4a.local_recipes and --local_recipes broken also breaks Buildozer’s official documentation on the subject as well as downstream projects previously leveraging these things.

Sad cats unite. 😹 😿 🙀

[misl6]

Even if I’m not too concerned about the security issue found during the discussion, I’m taking this occasion to refresh some code.

The weekend plan was to work on a python-for-android WIP PR, but his one has the priority ATM, so something will land (hopefully) soon.

Why I’m not concerned:

There are plenty of ways (even simpler ones than the one discussed here) of injecting malicious shell commands into the developer shell (which I’m not going to discuss here, for obvious reasons) when using buildozer (and is not Kivy’s fault). A developer should trust the author of e.g. a buildozer.spec example, before using it, or at least should read the most essential parts, unless the code is running on a sandboxed environment.

Why even if I’m not concerned I’m working on it:

Also, it’s a stupid and dangerous bug no matter how you see it, and it should be solved, no argument.

Agree, and that’s an excuse to refresh some code.

What we should do in the future:

Set up a Security Policy so we can talk about this kind of issues privately (and later share info with the community about security bugs when are already fixed (or at least triaged), and not exploitable anymore). [ Was on my low-priority task list ]

[tshirtman]

First, thanks a lot for the investigation. Indeed security was certainly not a major concern during the design of buildozer, as it’s usually ran manually by the user wishing to build their application, while that’s certainly no excuse, as seemingly unexploitable bugs can always find their use in chains later as people rely on software and building more complex solutions on it without understanding their security characteristics.

Also, it’s a stupid and dangerous bug no matter how you see it, and it should be solved, no argument.

I don’t remember from the top of my head why we pass shell=True, but if possible at all, we should indeed close that gap first. Other than that, yeah, quoting every parameters would be the way, although it’s indeed harder to make sure you didn’t let any slip through.

This should get priority in solving.