Pip ignores extras_require when description in setup.py contains \n

Environment

• pip version: 19.2.3
• Python version: 3.6.8
• Poetry version 1.0.0b1

Description This issue was discovered while publishing docs from undp/MsGraphExporter on ReadTheDocs. When pyproject.toml option description has a multi-line string enclosed in """ (which is perfectly legit for a string in TOML), it causes poetry to build the package with the description parameter in the setup.py file containing string like 'this is\na short description' which apparently causes pip install pkg.whl[docs] to fail picking up extras_require option.

Expected behavior Having \n in long_description of setup_kwargs in setup.py does not cause such issue. So, I would expect having it in description should not affect pip either.

How to re-create the issue

1. Have the description in the pyproject.toml as a multi-line value enclosed in """.

   # ...snip...
   [tool.poetry]
   name = "ms-graph-exporter"
   version = "0.1.0-rc.1"
   description = """A distributed Celery application to export time-domain data
   periodically from Microsoft Graph API into a buffer key-value store."""
   # ...snip...
   

2. Check pyproject.toml syntax with poery.

   $ poetry check
   All set!
   

3. Build the package with poetry

   $ poetry build
   Building ms-graph-exporter (0.1.0-rc.1)
   - Building sdist
   - Built ms-graph-exporter-0.1.0rc1.tar.gz
   
   - Building wheel
   - Built ms_graph_exporter-0.1.0rc1-py3-none-any.whl
   

4. Check setup.py from the package tar.gz to ensure it has all extras_require defined and also has \n in the description.

   # -*- coding: utf-8 -*-
   from distutils.core import setup
   
   packages = \
   ['ms_graph_exporter', 'ms_graph_exporter.celery', 'ms_graph_exporter.ms_graph']
   
   package_data = \
   {'': ['*']}
   
   install_requires = \
   ['adal>=1.2,<2.0',
   # ...snip...
   'typing>=3.7,<4.0']
   
   extras_require = \
   {'code-format': ['black>=19.3b0,<20.0', 'blacken-docs>=0.5.0,<0.6.0'],
   'code-lint': ['flake8>=3.7,<4.0',
   # ...snip...
                 'pygments>=2.4,<3.0'],
   'docs': ['recommonmark>=0.5.0,<0.6.0',
             'sphinx>=2.0,<3.0',
             'sphinx_rtd_theme>=0.4.3,<0.5.0',
             'sphinx-autodoc-typehints>=1.7,<2.0'],
   'test': ['pytest>=5.0.1,<6.0.0',
   # ...snip...
             'pytest-variables[yaml]>=1.7,<2.0']}
   
   setup_kwargs = {
       'name': 'ms-graph-exporter',
       'version': '0.1.0rc1',
       'description': 'A distributed Celery application to export time-domain data\nperiodically from Microsoft Graph API into a buffer key-value store.',
       # ...snip...
       'extras_require': extras_require,
       'python_requires': '>=3.6,<4.0',
   }
   
   
   setup(**setup_kwargs)
   

5. Try to install docs extra requirements with the package.

   $ pip install ./dist/ms_graph_exporter-0.1.0rc1-py3-none-any.whl[docs]
   Processing ./dist/ms_graph_exporter-0.1.0rc1-py3-none-any.whl
   WARNING: ms-graph-exporter 0.1.0rc1 does not provide the extra 'docs'
   Installing collected packages: ms-graph-exporter
   Successfully installed ms-graph-exporter-0.1.0rc1
   

How to resolve the issue

1. Ensure pyproject.toml has a single-line description.

   # ...snip...
   [tool.poetry]
   name = "ms-graph-exporter"
   version = "0.1.0-rc.1"
   description = "A distributed Celery application to export time-domain data periodically from Microsoft Graph API into a buffer key-value store."
   # ...snip...
   

2. Check pyproject.toml syntax with poery.

   $ poetry check
   All set!
   

3. Build the package with poetry

   $ poetry build
   Building ms-graph-exporter (0.1.0-rc.1)
   - Building sdist
   - Built ms-graph-exporter-0.1.0rc1.tar.gz
   
   - Building wheel
   - Built ms_graph_exporter-0.1.0rc1-py3-none-any.whl
   

4. Check setup.py from the package tar.gz to ensure it has all extras_require defined and description without \n.

   # -*- coding: utf-8 -*-
   from distutils.core import setup
   
   packages = \
   ['ms_graph_exporter', 'ms_graph_exporter.celery', 'ms_graph_exporter.ms_graph']
   
   package_data = \
   {'': ['*']}
   
   install_requires = \
   ['adal>=1.2,<2.0',
   # ...snip...
   'typing>=3.7,<4.0']
   
   extras_require = \
   {'code-format': ['black>=19.3b0,<20.0', 'blacken-docs>=0.5.0,<0.6.0'],
   'code-lint': ['flake8>=3.7,<4.0',
   # ...snip...
                 'pygments>=2.4,<3.0'],
   'docs': ['recommonmark>=0.5.0,<0.6.0',
             'sphinx>=2.0,<3.0',
             'sphinx_rtd_theme>=0.4.3,<0.5.0',
             'sphinx-autodoc-typehints>=1.7,<2.0'],
   'test': ['pytest>=5.0.1,<6.0.0',
   # ...snip...
             'pytest-variables[yaml]>=1.7,<2.0']}
   
   setup_kwargs = {
       'name': 'ms-graph-exporter',
       'version': '0.1.0rc1',
       'description': 'A distributed Celery application to export time-domain data periodically from Microsoft Graph API into a buffer key-value store.',
       # ...snip...
       'extras_require': extras_require,
       'python_requires': '>=3.6,<4.0',
   }
   
   
   setup(**setup_kwargs)
   

5. Install docs extra requirements with the package without any issue.

   $ pip install ./dist/ms_graph_exporter-0.1.0rc1-py3-none-any.whl[docs]
   Processing ./dist/ms_graph_exporter-0.1.0rc1-py3-none-any.whl
   [... snip ...]
   Successfully installed Jinja2-2.10.1 MarkupSafe-1.1.1 PyJWT-1.7.1 Pygments-2.4.2
   adal-1.2.2 alabaster-0.7.12 amqp-2.5.1 asn1crypto-0.24.0 attrs-19.1.0 babel-2.7.0
   billiard-3.6.1.0 celery-4.3.0 celery-redbeat-0.13.0 certifi-2019.9.11 cffi-1.12.3
   chardet-3.0.4 commonmark-0.9.0 cryptography-2.7 docutils-0.15.2 future-0.17.1 gevent-1.4.0
   greenlet-0.4.15 idna-2.8 imagesize-1.1.0 importlib-metadata-0.22 kombu-4.6.4
   more-itertools-7.2.0 ms-graph-exporter-0.1.0rc1 packaging-19.1 pycparser-2.19
   pyparsing-2.4.2 python-dateutil-2.8.0 pytz-2019.2 pyyaml-3.13 recommonmark-0.5.0
   redis-3.3.8 requests-2.22.0 six-1.12.0 snowballstemmer-1.9.1 sphinx-2.2.0
   sphinx-autodoc-typehints-1.8.0 sphinx-rtd-theme-0.4.3 sphinxcontrib-applehelp-1.0.1
   sphinxcontrib-devhelp-1.0.1 sphinxcontrib-htmlhelp-1.0.2 sphinxcontrib-jsmath-1.0.1
   sphinxcontrib-qthelp-1.0.2 sphinxcontrib-serializinghtml-1.1.3 tenacity-5.1.1
   typing-3.7.4.1 urllib3-1.25.3 vine-1.3.0 zipp-0.6.0
   

_Originally posted by @OK-UNDP in https://github.com/readthedocs/readthedocs.org/issues/6151#issuecomment-531048562_

[chrahunt]

I will close this now since the issue has been reported over in poetry. Thanks again for such a thorough report!

[chrahunt]

You did most of the work - it was very clear what to check. 😃 Thank you for that.

I’d better file an issue with Poetry to truncate multi-line descriptions, or flagging it by running poetry check.

Yup, that’s what I would do.