Lightrun
question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

#30214 (Parallelized Build / CI) caused a build failure for me

See original GitHub issue

Currently I can’t get pandas build.

I get the message when running python setup.py build_ext --inplace -j 4:

Compiling pandas\_libs/parsers.pyx because it changed.
Compiling pandas\_libs/tslibs/timestamps.pyx because it changed.
Compiling pandas\_libs/window/aggregations.pyx because it changed.
Compiling pandas\io/msgpack/_packer.pyx because it changed.
Compiling pandas\io/msgpack/_unpacker.pyx because it changed.
Compiling pandas\_libs/groupby.pyx because it changed.
Compiling pandas\_libs/index.pyx because it changed.
Compiling pandas\_libs/internals.pyx because it changed.
Compiling pandas\_libs/lib.pyx because it changed.
Compiling pandas\_libs/parsers.pyx because it changed.
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\TP\Miniconda3\envs\pandas-dev\lib\multiprocessing\spawn.py", line 105, in spawn_main
Compiling pandas\_libs/tslibs/timestamps.pyx because it changed.
Compiling pandas\_libs/window/aggregations.pyx because it changed.
Compiling pandas\io/msgpack/_packer.pyx because it changed.
Compiling pandas\io/msgpack/_unpacker.pyx because it changed.
Compiling pandas\_libs/groupby.pyx because it changed.
Compiling pandas\_libs/index.pyx because it changed.
Compiling pandas\_libs/internals.pyx because it changed.
Compiling pandas\_libs/lib.pyx because it changed.
Compiling pandas\_libs/parsers.pyx because it changed.
    exitcode = _main(fd)
  File "C:\Users\TP\Miniconda3\envs\pandas-dev\lib\multiprocessing\spawn.py", line 114, in _main
    prepare(preparation_data)
  File "C:\Users\TP\Miniconda3\envs\pandas-dev\lib\multiprocessing\spawn.py", line 225, in prepare
    _fixup_main_from_path(data['init_main_from_path'])
  File "C:\Users\TP\Miniconda3\envs\pandas-dev\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path
    run_name="__mp_main__")
  File "C:\Users\TP\Miniconda3\envs\pandas-dev\lib\runpy.py", line 263, in run_path
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\TP\Miniconda3\envs\pandas-dev\lib\multiprocessing\spawn.py", line 105, in spawn_main
    exitcode = _main(fd)
  File "C:\Users\TP\Miniconda3\envs\pandas-dev\lib\multiprocessing\spawn.py", line 114, in _main
    prepare(preparation_data)
  File "C:\Users\TP\Miniconda3\envs\pandas-dev\lib\multiprocessing\spawn.py", line 225, in prepare
    _fixup_main_from_path(data['init_main_from_path'])
  File "C:\Users\TP\Miniconda3\envs\pandas-dev\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path
    run_name="__mp_main__")
  File "C:\Users\TP\Miniconda3\envs\pandas-dev\lib\runpy.py", line 263, in run_path
Compiling pandas\_libs/tslibs/timestamps.pyx because it changed.
Compiling pandas\_libs/window/aggregations.pyx because it changed.
Compiling pandas\io/msgpack/_packer.pyx because it changed.
Compiling pandas\io/msgpack/_unpacker.pyx because it changed.
    pkg_name=pkg_name, script_name=fname)
    pkg_name=pkg_name, script_name=fname)
  File "C:\Users\TP\Miniconda3\envs\pandas-dev\lib\runpy.py", line 96, in _run_module_code
    mod_name, mod_spec, pkg_name, script_name)
  File "C:\Users\TP\Miniconda3\envs\pandas-dev\lib\runpy.py", line 85, in _run_code
  File "C:\Users\TP\Miniconda3\envs\pandas-dev\lib\runpy.py", line 96, in _run_module_code
    exec(code, run_globals)
  File "C:\Users\TP\Documents\Python\pandasdev\pandasdev\setup.py", line 815, in <module>
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\TP\Miniconda3\envs\pandas-dev\lib\multiprocessing\spawn.py", line 105, in spawn_main
    mod_name, mod_spec, pkg_name, script_name)
  File "C:\Users\TP\Miniconda3\envs\pandas-dev\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Users\TP\Documents\Python\pandasdev\pandasdev\setup.py", line 815, in <module>
    exitcode = _main(fd)
  File "C:\Users\TP\Miniconda3\envs\pandas-dev\lib\multiprocessing\spawn.py", line 114, in _main
    ext_modules=maybe_cythonize(extensions, compiler_directives=directives),
  File "C:\Users\TP\Documents\Python\pandasdev\pandasdev\setup.py", line 543, in maybe_cythonize
    prepare(preparation_data)
  File "C:\Users\TP\Miniconda3\envs\pandas-dev\lib\multiprocessing\spawn.py", line 225, in prepare
    _fixup_main_from_path(data['init_main_from_path'])
  File "C:\Users\TP\Miniconda3\envs\pandas-dev\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path
    run_name="__mp_main__")
  File "C:\Users\TP\Miniconda3\envs\pandas-dev\lib\runpy.py", line 263, in run_path
    return cythonize(extensions, *args, **kwargs)
  File "C:\Users\TP\Miniconda3\envs\pandas-dev\lib\site-packages\Cython\Build\Dependencies.py", line 1073, in cythonize
    pkg_name=pkg_name, script_name=fname)
  File "C:\Users\TP\Miniconda3\envs\pandas-dev\lib\runpy.py", line 96, in _run_module_code
    ext_modules=maybe_cythonize(extensions, compiler_directives=directives),
  File "C:\Users\TP\Documents\Python\pandasdev\pandasdev\setup.py", line 543, in maybe_cythonize
    mod_name, mod_spec, pkg_name, script_name)
  File "C:\Users\TP\Miniconda3\envs\pandas-dev\lib\runpy.py", line 85, in _run_code
    return cythonize(extensions, *args, **kwargs)
  File "C:\Users\TP\Miniconda3\envs\pandas-dev\lib\site-packages\Cython\Build\Dependencies.py", line 1073, in cythonize
    exec(code, run_globals)
  File "C:\Users\TP\Documents\Python\pandasdev\pandasdev\setup.py", line 815, in <module>
    ext_modules=maybe_cythonize(extensions, compiler_directives=directives),
    nthreads, initializer=_init_multiprocessing_helper)
  File "C:\Users\TP\Miniconda3\envs\pandas-dev\lib\multiprocessing\context.py", line 119, in Pool
  File "C:\Users\TP\Documents\Python\pandasdev\pandasdev\setup.py", line 543, in maybe_cythonize
    nthreads, initializer=_init_multiprocessing_helper)
    return cythonize(extensions, *args, **kwargs)
  File "C:\Users\TP\Miniconda3\envs\pandas-dev\lib\site-packages\Cython\Build\Dependencies.py", line 1073, in cythonize
  File "C:\Users\TP\Miniconda3\envs\pandas-dev\lib\multiprocessing\context.py", line 119, in Pool
    nthreads, initializer=_init_multiprocessing_helper)
  File "C:\Users\TP\Miniconda3\envs\pandas-dev\lib\multiprocessing\context.py", line 119, in Pool
Compiling pandas\_libs/groupby.pyx because it changed.
Compiling pandas\_libs/index.pyx because it changed.
    context=self.get_context())
    context=self.get_context())
    context=self.get_context())
Compiling pandas\_libs/internals.pyx because it changed.
  File "C:\Users\TP\Miniconda3\envs\pandas-dev\lib\multiprocessing\pool.py", line 176, in __init__
  File "C:\Users\TP\Miniconda3\envs\pandas-dev\lib\multiprocessing\pool.py", line 176, in __init__
  File "C:\Users\TP\Miniconda3\envs\pandas-dev\lib\multiprocessing\pool.py", line 176, in __init__
Compiling pandas\_libs/lib.pyx because it changed.
Compiling pandas\_libs/parsers.pyx because it changed.
Compiling pandas\_libs/tslibs/timestamps.pyx because it changed.
Compiling pandas\_libs/window/aggregations.pyx because it changed.
Compiling pandas\io/msgpack/_packer.pyx because it changed.
Compiling pandas\io/msgpack/_unpacker.pyx because it changed.
    self._repopulate_pool()
    self._repopulate_pool()
    self._repopulate_pool()
  File "C:\Users\TP\Miniconda3\envs\pandas-dev\lib\multiprocessing\pool.py", line 241, in _repopulate_pool
  File "C:\Users\TP\Miniconda3\envs\pandas-dev\lib\multiprocessing\pool.py", line 241, in _repopulate_pool
  File "C:\Users\TP\Miniconda3\envs\pandas-dev\lib\multiprocessing\pool.py", line 241, in _repopulate_pool
Traceback (most recent call last):
    w.start()
    w.start()
    w.start()
  File "<string>", line 1, in <module>
  File "C:\Users\TP\Miniconda3\envs\pandas-dev\lib\multiprocessing\process.py", line 112, in start
  File "C:\Users\TP\Miniconda3\envs\pandas-dev\lib\multiprocessing\process.py", line 112, in start
  File "C:\Users\TP\Miniconda3\envs\pandas-dev\lib\multiprocessing\process.py", line 112, in start
  File "C:\Users\TP\Miniconda3\envs\pandas-dev\lib\multiprocessing\spawn.py", line 105, in spawn_main
    exitcode = _main(fd)
  File "C:\Users\TP\Miniconda3\envs\pandas-dev\lib\multiprocessing\spawn.py", line 114, in _main
    prepare(preparation_data)
  File "C:\Users\TP\Miniconda3\envs\pandas-dev\lib\multiprocessing\spawn.py", line 225, in prepare
    _fixup_main_from_path(data['init_main_from_path'])
  File "C:\Users\TP\Miniconda3\envs\pandas-dev\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path
    run_name="__mp_main__")
  File "C:\Users\TP\Miniconda3\envs\pandas-dev\lib\runpy.py", line 263, in run_path
    pkg_name=pkg_name, script_name=fname)
  File "C:\Users\TP\Miniconda3\envs\pandas-dev\lib\runpy.py", line 96, in _run_module_code
    mod_name, mod_spec, pkg_name, script_name)
  File "C:\Users\TP\Miniconda3\envs\pandas-dev\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Users\TP\Documents\Python\pandasdev\pandasdev\setup.py", line 815, in <module>
    ext_modules=maybe_cythonize(extensions, compiler_directives=directives),
  File "C:\Users\TP\Documents\Python\pandasdev\pandasdev\setup.py", line 543, in maybe_cythonize
    return cythonize(extensions, *args, **kwargs)
  File "C:\Users\TP\Miniconda3\envs\pandas-dev\lib\site-packages\Cython\Build\Dependencies.py", line 1073, in cythonize
    nthreads, initializer=_init_multiprocessing_helper)
  File "C:\Users\TP\Miniconda3\envs\pandas-dev\lib\multiprocessing\context.py", line 119, in Pool
    context=self.get_context())
  File "C:\Users\TP\Miniconda3\envs\pandas-dev\lib\multiprocessing\pool.py", line 176, in __init__
    self._repopulate_pool()
  File "C:\Users\TP\Miniconda3\envs\pandas-dev\lib\multiprocessing\pool.py", line 241, in _repopulate_pool
    w.start()
  File "C:\Users\TP\Miniconda3\envs\pandas-dev\lib\multiprocessing\process.py", line 112, in start
    self._popen = self._Popen(self)
    self._popen = self._Popen(self)
    self._popen = self._Popen(self)
    self._popen = self._Popen(self)
  File "C:\Users\TP\Miniconda3\envs\pandas-dev\lib\multiprocessing\context.py", line 322, in _Popen
  File "C:\Users\TP\Miniconda3\envs\pandas-dev\lib\multiprocessing\context.py", line 322, in _Popen
  File "C:\Users\TP\Miniconda3\envs\pandas-dev\lib\multiprocessing\context.py", line 322, in _Popen
  File "C:\Users\TP\Miniconda3\envs\pandas-dev\lib\multiprocessing\context.py", line 322, in _Popen
    return Popen(process_obj)
    return Popen(process_obj)
    return Popen(process_obj)
    return Popen(process_obj)
  File "C:\Users\TP\Miniconda3\envs\pandas-dev\lib\multiprocessing\popen_spawn_win32.py", line 46, in __init__
  File "C:\Users\TP\Miniconda3\envs\pandas-dev\lib\multiprocessing\popen_spawn_win32.py", line 46, in __init__
  File "C:\Users\TP\Miniconda3\envs\pandas-dev\lib\multiprocessing\popen_spawn_win32.py", line 46, in __init__
  File "C:\Users\TP\Miniconda3\envs\pandas-dev\lib\multiprocessing\popen_spawn_win32.py", line 46, in __init__
    prep_data = spawn.get_preparation_data(process_obj._name)
    prep_data = spawn.get_preparation_data(process_obj._name)
    prep_data = spawn.get_preparation_data(process_obj._name)
    prep_data = spawn.get_preparation_data(process_obj._name)
  File "C:\Users\TP\Miniconda3\envs\pandas-dev\lib\multiprocessing\spawn.py", line 143, in get_preparation_data
  File "C:\Users\TP\Miniconda3\envs\pandas-dev\lib\multiprocessing\spawn.py", line 143, in get_preparation_data
  File "C:\Users\TP\Miniconda3\envs\pandas-dev\lib\multiprocessing\spawn.py", line 143, in get_preparation_data
  File "C:\Users\TP\Miniconda3\envs\pandas-dev\lib\multiprocessing\spawn.py", line 143, in get_preparation_data
    _check_not_importing_main()
    _check_not_importing_main()
    _check_not_importing_main()
  File "C:\Users\TP\Miniconda3\envs\pandas-dev\lib\multiprocessing\spawn.py", line 136, in _check_not_importing_main
  File "C:\Users\TP\Miniconda3\envs\pandas-dev\lib\multiprocessing\spawn.py", line 136, in _check_not_importing_main
  File "C:\Users\TP\Miniconda3\envs\pandas-dev\lib\multiprocessing\spawn.py", line 136, in _check_not_importing_main
    _check_not_importing_main()
    is not going to be frozen to produce an executable.''')
RuntimeError:
        An attempt has been made to start a new process before the
        current process has finished its bootstrapping phase.

        This probably means that you are not using fork to start your
        child processes and you have forgotten to use the proper idiom
        in the main module:

            if __name__ == '__main__':
                freeze_support()
                ...

        The "freeze_support()" line can be omitted if the program
        is not going to be frozen to produce an executable.    is not going to be frozen to produce an executable.''')
  File "C:\Users\TP\Miniconda3\envs\pandas-dev\lib\multiprocessing\spawn.py", line 136, in _check_not_importing_main
    is not going to be frozen to produce an executable.''')

RuntimeError:
        An attempt has been made to start a new process before the
        current process has finished its bootstrapping phase.

        This probably means that you are not using fork to start your
        child processes and you have forgotten to use the proper idiom
        in the main module:

            if __name__ == '__main__':
                freeze_support()
                ...

        The "freeze_support()" line can be omitted if the program
        is not going to be frozen to produce an executable.    is not going to be frozen to produce an executable.''')
RuntimeError:
        An attempt has been made to start a new process before the
        current process has finished its bootstrapping phase.

        This probably means that you are not using fork to start your
        child processes and you have forgotten to use the proper idiom
        in the main module:

            if __name__ == '__main__':
                freeze_support()
                ...

I can’t really parse what’s going on, but the build seems to go back to the same RuntimeError repeatedly.

I’ve triaged the issue to stem from #30214. Any idea what’s happening, @WillAyd ?

Workaround

I can work around the issue by building with parallelization, i.e. run python setup.py build_ext --inplace -j 0 instead of python setup.py build_ext --inplace -j 4.

Output of pd.show_versions()

INSTALLED VERSIONS

commit : 95e1a63dd3382db6663bc8a2b334b422b93dd7fe python : 3.7.5.final.0 python-bits : 64 OS : Windows OS-release : 10 machine : AMD64 processor : Intel64 Family 6 Model 78 Stepping 3, GenuineIntel byteorder : little LC_ALL : None LANG : None LOCALE : None.None

pandas : 0.26.0.dev0+1358.g95e1a63dd numpy : 1.17.4 pytz : 2019.1 dateutil : 2.8.0 pip : 19.3.1 setuptools : 42.0.2.post20191203 Cython : 0.29.13 pytest : 5.2.2 hypothesis : 4.28.2 sphinx : 2.2.0 blosc : None feather : None xlsxwriter : None lxml.etree : None html5lib : None pymysql : None psycopg2 : None jinja2 : 2.10.3 IPython : 7.6.1 pandas_datareader: None bs4 : None bottleneck : None fastparquet : None gcsfs : None lxml.etree : None matplotlib : 3.1.1 numexpr : 2.6.9 odfpy : None openpyxl : 3.0.0 pandas_gbq : None pyarrow : None pytables : None pytest : 5.2.2 s3fs : None scipy : 1.3.1 sqlalchemy : None tables : None xarray : None xlrd : 1.2.0 xlwt : None xlsxwriter : None

Issue Analytics

  • State:closed
  • Created 4 years ago
  • Comments:14 (13 by maintainers)

github_iconTop GitHub Comments

1reaction
simonjayhawkinscommented, Dec 19, 2019

Yeah, I have the same problem in Windows and am now using just python setup.py build_ext --inplace. maybe should have a doc update in the meantime.

0reactions
Dr-Irvcommented, Dec 31, 2019

Here’s more. The way multiprocessing works in python you have to use if __name__ == '__main__':, which would be a big change for us. See https://docs.python.org/3.7/library/multiprocessing.html?highlight=multiprocessing#multiprocessing-programming where if you scroll down it says “Make sure that the main module can be safely imported by a new Python interpreter without causing unintended side effects (such a starting a new process).” followed by “Instead one should protect the “entry point” of the program by using if __name__ == '__main__':”

I’ll create a patch for Windows.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Handle cythonize(..., nthreads=1) for "spawn" start method
Build.cythonize(..., nthreads=1) is called ... #30214 (Parallelized Build / CI) caused a build failure for me pandas-dev/pandas#30356.
Read more >
Troubleshooting Unexplained Build Failures - CircleCI
This is the most common cause of builds failing when “nothing changed”. If your language of choice uses the DependencyList.lock model, then you ......
Read more >
jenkins - Failing a build in Jenkinsfile - Stack Overflow
If a user has a Declarative Pipeline script with parallel stages, and they set failFast true for those stages, if one of the...
Read more >
Bitbucket Pipelines: failing parallel builds fast
Hi everyone, Support for fast failing parallel builds has been a long awaiting feature . We are happy to announce that this feature...
Read more >
Build Failure Analyzer | Jenkins plugin
This plugin analyzes the causes of failed builds and presents the causes on the build page. It does this by using a knowledge...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found
Previous page

DataFrame.loc multiple columns replace

Next page

json_normalize skips an entry in a pymongo cursor