installer fails because of "new" pip dependency resolver

Impacted versions

• Distribution: Ubuntu
• Codename: Focal (20.04.3)
• Arch: 64 Bits
• Database: PostgreSQL

Also tried with the following OSes, but stopped the installer after about 4~6 hours without any result:

• CentOS 7
• Debian 10
• Ubuntu 18.04 (latest version)

Steps to reproduce

Basic Ubuntu 20.04.3 server setup (or any of the above listes OSes), with SSH enabled. After setup is completed (XXXXX is a real existing domain):

$ time sudo python3 run.py --debug XXXXX
[...]
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtracking
  Downloading modoboa-webmail-1.1.4.tar.gz (87 kB)
     |████████████████████████████████| 87 kB 3.0 MB/s
  Downloading modoboa-webmail-1.1.3.tar.gz (87 kB)
     |████████████████████████████████| 87 kB 4.5 MB/s
  Downloading modoboa-webmail-1.1.2.tar.gz (85 kB)
     |████████████████████████████████| 85 kB 4.3 MB/s
  Downloading modoboa-webmail-1.1.1.tar.gz (84 kB)
     |████████████████████████████████| 84 kB 3.8 MB/s
ERROR: Exception:
Traceback (most recent call last):
  File "/srv/modoboa/env/lib/python3.8/site-packages/pip/_internal/cli/base_command.py", line 173, in _main
    status = self.run(options, args)
  File "/srv/modoboa/env/lib/python3.8/site-packages/pip/_internal/cli/req_command.py", line 203, in wrapper
    return func(self, options, args)
  File "/srv/modoboa/env/lib/python3.8/site-packages/pip/_internal/commands/install.py", line 315, in run
    requirement_set = resolver.resolve(
  File "/srv/modoboa/env/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 94, in resolve
    result = self._result = resolver.resolve(
  File "/srv/modoboa/env/lib/python3.8/site-packages/pip/_vendor/resolvelib/resolvers.py", line 472, in resolve
    state = resolution.resolve(requirements, max_rounds=max_rounds)
  File "/srv/modoboa/env/lib/python3.8/site-packages/pip/_vendor/resolvelib/resolvers.py", line 383, in resolve
    raise ResolutionTooDeep(max_rounds)
pip._vendor.resolvelib.resolvers.ResolutionTooDeep: 2000000
Traceback (most recent call last):
  File "/srv/modoboa/env/bin/modoboa-admin.py", line 4, in <module>
    from modoboa.core.commands import handle_command_line
  File "/srv/modoboa/env/lib/python3.8/site-packages/modoboa/core/__init__.py", line 3, in <module>
    from django.core.urlresolvers import reverse
  File "/srv/modoboa/env/lib/python3.8/site-packages/django/core/urlresolvers.py", line 16, in <module>
    from django.http import Http404
  File "/srv/modoboa/env/lib/python3.8/site-packages/django/http/__init__.py", line 4, in <module>
    from django.http.response import (
  File "/srv/modoboa/env/lib/python3.8/site-packages/django/http/response.py", line 13, in <module>
    from django.core.serializers.json import DjangoJSONEncoder
  File "/srv/modoboa/env/lib/python3.8/site-packages/django/core/serializers/__init__.py", line 23, in <module>
    from django.core.serializers.base import SerializerDoesNotExist
  File "/srv/modoboa/env/lib/python3.8/site-packages/django/core/serializers/base.py", line 6, in <module>
    from django.db import models
  File "/srv/modoboa/env/lib/python3.8/site-packages/django/db/models/__init__.py", line 6, in <module>
    from django.db.models.query import Q, QuerySet, Prefetch  # NOQA
  File "/srv/modoboa/env/lib/python3.8/site-packages/django/db/models/query.py", line 13, in <module>
    from django.db.models.fields import AutoField, Empty, FieldDoesNotExist
  File "/srv/modoboa/env/lib/python3.8/site-packages/django/db/models/fields/__init__.py", line 18, in <module>
    from django import forms
  File "/srv/modoboa/env/lib/python3.8/site-packages/django/forms/__init__.py", line 6, in <module>
    from django.forms.fields import *  # NOQA
  File "/srv/modoboa/env/lib/python3.8/site-packages/django/forms/fields.py", line 18, in <module>
    from django.forms.utils import from_current_timezone, to_current_timezone
  File "/srv/modoboa/env/lib/python3.8/site-packages/django/forms/utils.py", line 15, in <module>
    from django.utils.html import format_html, format_html_join, escape
  File "/srv/modoboa/env/lib/python3.8/site-packages/django/utils/html.py", line 16, in <module>
    from .html_parser import HTMLParser, HTMLParseError
  File "/srv/modoboa/env/lib/python3.8/site-packages/django/utils/html_parser.py", line 12, in <module>
    HTMLParseError = _html_parser.HTMLParseError
AttributeError: module 'html.parser' has no attribute 'HTMLParseError'
None

real    1562m4.217s
user    1557m54.697s
sys     1m13.077s

Especially this line:

pip._vendor.resolvelib.resolvers.ResolutionTooDeep: 2000000

Expected behavior

Installation finishes.

Notes

• I did this installation on a Debian 10 about 5 months ago, and it finished pretty fast.
• Currently this does not work on any of the tested OS versions, because pip is installed in the modoboa environment at the latest version which uses the new dependency resolver.
• I tried setting ~modoboa/env/pip.conf (aka $VIRTUAL_ENV/pip.conf) with different values, but currently I didn’t find a working fix/hack to make the installer work. This idea was based on several “stackoverflow suggestions”. Especially use-deprecated=legacy-resolver did not work, actually it fails pretty fast.

[tonioo]

Just to let you know, I’ve pushed a fix for this issue. Problem is latest stable version of Modoboa (1.17) depends on old requirement versions which are not compatible with latest pip version anymore. If you update your installer copy, you can now use the --beta flag to install latest beta release instead, which has updated requirements.

[chipincode]

Here are the steps I took to get it installed on Ubuntu 20.04:

Change to root:

~$ sudo -i

Download Modoboa Installer:

~# git clone https://github.com/modoboa/modoboa-installer

Install build dependencies: (The installer will keep crashing without: rustc, python3-feedparser & python3-lxml)

~# apt install build-essential python3-dev python3-feedparser python3-lxml libxml2-dev libxslt-dev libjpeg-dev librrd-dev rrdtool libffi-dev libssl-dev rustc

Installer will keep crashing if /tmp is noexec:

~# mount -t tmpfs -o remount,exec tmpfs /tmp

Run the Modoboa Installer to generate config file:

~# cd modoboa-installer ~# ./run.py --stop-after-configfile-check <your domain>

Edit config file:

~# nano installer.cfg

Change the following sections:

[certificate] type = letsencrypt

[letsencrypt] email = <your email>@<your domain>

[database] engine = <postgres or mysql>

^S ^X

Install Modoboa: (it’s going to fail)

~# ./run.py --interactive --debug <your domain>

Once you start seeing errors hit ^C, then fix the issues keeping it from installing:

Install django-braces:

~# su - modoboa $ bash ~$ source ./env/bin/activate ~$ pip install django-braces ~$ exit $ exit

Copy feedparser & lxml to modoboa env:

~# cp -r /usr/lib/python3/dist-packages/feedparser* /srv/modoboa/env/lib/python3.8/site-packages/ ~# cp -r /usr/lib/python3/dist-packages/lxml* /srv/modoboa/env/lib/python3.8/site-packages/

Re-install Modoboa:

~# ./run.py --interactive --debug <your domain>

You should see: Success!

Remount /tmp noexec:

~# mount -t tmpfs -o remount,noexec tmpfs /tmp

Going to webmail site will generate server error, so let’s fix it:

~# nano /srv/modoboa/instance/instance/settings.py

Change the following line:

USE_TZ = false

^S ^X

Restart uWSGI:

~# /etc/init.d/uwsgi restart

Fix Postfix greylisting:

~# nano /etc/postfix/main.cf

Comment out the following:

postscreen_pipelining_enable = yes postscreen_pipelining_action = enforce

postscreen_non_smtp_command_enable = yes postscreen_non_smtp_command_action = enforce

postscreen_bare_newline_enable = yes postscreen_bare_newline_action = enforce

^S ^X

Restart Postfix:

~# systemctl restart postfix

~# exit

Success!