Fix packager.py name normalization

After deployment we noticed that our application didn’t work anymore: [ERROR] Runtime.ImportModuleError: Unable to import module 'app': No module named 'sqlalchemy'

Our requirements.txt file has the following, so no problem there:

$ grep -i sqlalchemy requirements.txt 
SQLAlchemy==1.3.9

While debugging we noticed that the resulting deployment has sqlalchemy files in the wrong location:

$ unzip -l 5a76ade4f39e704f0925422520b454f2-python3.7.zip | grep -i 'sqlalchemy[^/]*/[^/]*$'
     4621  2019-10-29 17:49   SQLAlchemy-1.3.9.data/purelib/sqlalchemy/__init__.py
     5648  2019-10-29 17:49   SQLAlchemy-1.3.9.data/purelib/sqlalchemy/processors.py
     2377  2019-10-29 17:49   SQLAlchemy-1.3.9.data/purelib/sqlalchemy/schema.py
     3377  2019-10-29 17:49   SQLAlchemy-1.3.9.data/purelib/sqlalchemy/types.py
     2990  2019-10-29 17:49   SQLAlchemy-1.3.9.data/purelib/sqlalchemy/inspection.py
     6693  2019-10-29 17:49   SQLAlchemy-1.3.9.data/purelib/sqlalchemy/log.py
    16938  2019-10-29 17:49   SQLAlchemy-1.3.9.data/purelib/sqlalchemy/exc.py
    12721  2019-10-29 17:49   SQLAlchemy-1.3.9.data/purelib/sqlalchemy/interfaces.py
    51669  2019-10-29 17:49   SQLAlchemy-1.3.9.data/purelib/sqlalchemy/events.py
       11  2019-10-29 17:49   SQLAlchemy-1.3.9.dist-info/top_level.txt
      104  2019-10-29 17:49   SQLAlchemy-1.3.9.dist-info/WHEEL
     1100  2019-10-29 17:49   SQLAlchemy-1.3.9.dist-info/LICENSE
    23329  2019-10-29 17:49   SQLAlchemy-1.3.9.dist-info/RECORD
     7240  2019-10-29 17:49   SQLAlchemy-1.3.9.dist-info/METADATA

It looks like packager.py does in fact have the code to move purelib back to the root directory – function _install_purelib_and_platlib – however, it is not working as intended.

While debugging further with print()s we noticed that wheel.data_dir variable is just wrong:

    @property
    def data_dir(self):
        # type: () -> str
        # The directory format is {distribution}-{version}.data
        return '%s-%s.data' % (self._name, self._version)

This yields sqlalchemy-1.3.9.data rather than SQLAlchemy-1.3.9.data and therefore the code in _install_purelib_and_platlib doesn’t execute. The reason is the normalization function:

    def _normalize_name(self, name):
        # type: (str) -> str
        # Taken directly from PEP 503
        return re.sub(r"[-_.]+", "-", name).lower()

I don’t think package names should be forced lower-case like that – even though in theory they should be lower-case, in practice our requirements.txt does have quite a few upper-cased package names and identifiers:

$ grep '^[A-Z]' requirements.txt 
PyMySQL==0.9.3
SQLAlchemy==1.3.10
SQLAlchemy-FullText-Search==0.2.5
SQLAlchemy-Utils==0.34.2
WeasyPrint==50

I am creating a pull request to fix this issue, and after using it our deployment is able to successfully use SQLAlchemy package again.

$ unzip -l bf895adfe5c60f297701196b732057ca-python3.7.zip | grep -i 'sqlalchemy[^/]*/[^/]*$'
     4621  2019-10-29 22:03   sqlalchemy/__init__.py
     5648  2019-10-29 22:03   sqlalchemy/processors.py
     2377  2019-10-29 22:03   sqlalchemy/schema.py
     3377  2019-10-29 22:03   sqlalchemy/types.py
     2990  2019-10-29 22:03   sqlalchemy/inspection.py
     6693  2019-10-29 22:03   sqlalchemy/log.py
    16938  2019-10-29 22:03   sqlalchemy/exc.py
    12721  2019-10-29 22:03   sqlalchemy/interfaces.py
    51669  2019-10-29 22:03   sqlalchemy/events.py
       11  2019-10-29 22:03   SQLAlchemy-1.3.8.dist-info/top_level.txt
      104  2019-10-29 22:03   SQLAlchemy-1.3.8.dist-info/WHEEL
     1100  2019-10-29 22:03   SQLAlchemy-1.3.8.dist-info/LICENSE
    23329  2019-10-29 22:03   SQLAlchemy-1.3.8.dist-info/RECORD
     7240  2019-10-29 22:03   SQLAlchemy-1.3.8.dist-info/METADATA

Thanks! John Vencky

[johnvencky]

@stealthycoin did you have a chance to take a look at it? Thanks.

[jamesls]

Trying to track down this issue. Here’s the set of steps I tried. Are people still running into this? I’d like to repro this myself, but it seems to work as I expect when I ran on ubuntu. https://github.com/aws/chalice/issues/1356#issuecomment-590981069