Interactions between --reload-dir, --reload-include, and --reload-exclude are not very intuitive or well explained (and may even be wrong)

Checklist

• The bug is reproducible against the latest release or master.
• There are no similar issues or pull requests to fix it yet.

Describe the bug

I am trying to get uvicorn to automatically reload in a dev environment based on source file changes and config file changes. I have two Python src directories (/path/to/first_dir and /path/to/second_dir) and a config directory (/path/to/env_dir) which contains *.env files that my application loads.

It is really hard to understand what uvicorn intends to do, much less what it actually does with regard to tuned reload behavior. Consider:

% uvicorn \
     --reload \
     --reload-exclude '*_flymake.py' \
     --reload-exclude 'flycheck_*.py' \
     --reload-dir /path/to/first_dir \
     --reload-dir /path/to/second_dir \
     mm.txns.main:app

The above will only ignore files matching *_flymake.py and flycheck_*.py in /path/to/first_dir, not /path/to/second_dir. So it looks like one needs to repeat the exclude patterns prior to each --reload-dir: UPDATE: This wasn’t correct. It turns out nothing was being excluded in this configuration because watchgod was not installed. This confusion is part of the bug.

% uvicorn \
     --reload \
     --reload-exclude '*_flymake.py' \
     --reload-exclude 'flycheck_*.py' \
     --reload-dir /path/to/first_dir \
     --reload-exclude '*_flymake.py' \
     --reload-exclude 'flycheck_*.py' \
     --reload-dir /path/to/second_dir \
     mm.txns.main:app

There is no mention of the need for repeated uses of --reload-exclude in advance of any --reload-dir options that I can find.

Further, (in what I am perceiving as a contradiction of the docs), this won’t work:

% uvicorn \
     --reload \
     --reload-exclude '*_flymake.py' \
     --reload-exclude 'flycheck_*.py' \
     --reload-dir /path/to/first_dir \
     --reload-exclude '*_flymake.py' \
     --reload-exclude 'flycheck_*.py' \
     --reload-dir /path/to/second_dir \
     --reload-include '*.env' \
     --reload-dir /path/to/env_dir \
     mm.txns.main:app

Adding, deleting, or modifying /path/to/env_dir/foo.env will not trigger a reload.

The documentation for --reload-exclude makes it clear (or at least strongly suggests) I should be able to expand the patterns matched for reload detection (emphasis mine):

--reload-exclude <glob-pattern> - Specify a glob pattern to match files or directories which will excluded from watching. May be used multiple times. By default the following patterns are excluded: .*, .py[cod], .sw.*, ~*. These defaults can be overwritten by including them in --reload-include.

That’s not what I’m seeing by including *.env files. Then there’s this, which seems to contradict the inclusion docs (emphasis mine):

By default Uvicorn uses simple changes detection strategy that compares python files modification times few times a second. If … or you need watching of non python files you can install watchgod or install uvicorn with uvicorn[standard], which will include watchgod.

Does that mean there’s some “upper bound” of patterns uvicorn is capable of considering? Is it *.py? That can’t be correct, because the counterexample explicitly states that *.py[cod] is excluded by default, but could be included via --reload-include. It’s not clear where the limitations are, even after spending dozens of minutes reading, re-reading, re-re-reading, experimenting, etc.

If the answer is, “Use watchgod,” I can’t find any documentation of how to do that or what additional configuration steps I would need to meet my use case.

Environment

[root@localhost vagrant]# uname -a Linux localhost.localdomain 5.4.17-2136.300.7.el8uek.x86_64 #2 SMP Fri Oct 8 16:23:01 PDT 2021 x86_64 x86_64 x86_64 GNU/Linux [root@localhost vagrant]# /opt/local/app/bin/uvicorn --version Running uvicorn 0.16.0 with CPython 3.9.6 on Linux

Additional context

I’m not the only person to be confused by this. This StackOverflow question poses something very similar, but remains unanswered:

Could you advise on how to exclude [*.json] using this method while preserving the reload functionality? I tried: [uvicorn main:app --reload-exclude ‘*.json’] however it generated this warning: WARNING: Current configuration will not reload as not all conditions are met, please refer to documentation.

When I misfiled this against gunicorn in a fit of blindess, I noted that this comment from @Andrew-Chen-Wang suggests there is some magic around merely making watchgod available to uvicorn, but fails to go into additional detail, so I have no way of verifying it (at least not without digging into a lot of code), much less figuring out how I can take advantage:

… [I]f you have watchgod installed, uvicorn should automatically pick that up to make the reloading process better. …

He responded with:

… In terms of resolving the issue, I believe you’re not supposed to reuse reload-include several times. Instead, wrap all your values in double quotes and make them csv

If that’s true, it’s also not clear from the docs. It also doesn’t help me, since I’m only using --reload-include once. (The above use of --reload-exclude multiple times does appear to get the behavior I want with respect to excluding various files/directories.)

UPDATE: I’m pretty sure the single, comma-separated argument advice is wrong. This strongly suggests one should be able to provide --reload-include and --reload-exclude multiple times via the command line.

[posita]

I’ll try to put together a PR that proposes more helpful/accessible language.

UPDATE: PR #1331 is up.

[Andrew-Chen-Wang]

Awesome investigation and apologies for the wrong advice. I believe the reason this was only included for watchgod was because only watchgod had CLI flags for reloading? IIRC, in cookiecutter django, I think I had to manually create a custom watcher to include and exclude what I needed before watchgod implemented the flags. Perhaps take a look at watchgod docs?