[KED-2518] kedro new --starter argument is broken on Windows for python<3.8

Description

I tried to create a kedro project by running kedro new on Windows in a conda environment with python 3.7.10. The command fails with a Error: Failed to generate project. error.

The command should work, since the badge in the README claims support for python=3.6, 3.7, 3.8.

After investigation, the error comes from tempfile library and the context manager used to create a temporary file in _prompt_user_for_config

https://github.com/quantumblacklabs/kedro/blob/6d9ffaba46ed1e984500a86ff5177327857c74f9/kedro/framework/cli/starters.py#L259-L268

This context manager __exit__ method calls _rmtree_unsafe on a read only file. This discussion in the official python issue tracker (msg262584 has the exact same stack trace than me, and msg344037 reference the commit which fixes the issue) show that this is a known bug which has been resolved when python==3.8 was released.

I checked and everything works fine with python=3.8 on the same computer. The compatibility with different os and python version seems important to me, especially for enterprise support. For my personal kedro use I can easily upgrade my python version, but in a professional setup I often have to deal with the constraints of the team I work with, and I barely choose if I work on windows/linux and if I can upgrade to a newer python version (a lot of teams are conservative and do not want to upgrade their python version by fear of breaking something in production).

Context

I can’t create a new kedro project with kedro new --starter=pandas-iris command on Windows with python=3.7.

Steps to Reproduce

On a Windows 7 or Windows 10 computer, create a conda environment with python=3.7 and call kedro new:

conda create -n ked171_py37 python=3.7 -y
pip install kedro==0.17.1
kedro new --starter==pandas-iris

Expected Result

The kedro project should be created.

Actual Result

The usual questions are asked, then the message Error: Failed to generate project. is displayed and no project is created.

Running the command with --verbose flag return the following stacktrace:

Traceback (most recent call last):
  File "c:\users\xxx\anaconda3\envs\kp_171\lib\site-packages\kedro\framework\cli\cli.py", line 300, in _create_project
    config = _prompt_user_for_config(template_path, checkout, directory)
  File "c:\users\xxx\anaconda3\envs\kp_171\lib\site-packages\kedro\framework\cli\cli.py", line 385, in _prompt_user_for_config
    return config
  File "c:\users\xxx\anaconda3\envs\kp_171\lib\tempfile.py", line 807, in __exit__
    self.cleanup()
  File "c:\users\xxx\anaconda3\envs\kp_171\lib\tempfile.py", line 811, in cleanup
    _shutil.rmtree(self.name)
  File "c:\users\xxx\anaconda3\envs\kp_171\lib\shutil.py", line 516, in rmtree
    return _rmtree_unsafe(path, onerror)
  File "c:\users\xxx\anaconda3\envs\kp_171\lib\shutil.py", line 395, in _rmtree_unsafe
    _rmtree_unsafe(fullname, onerror)
  File "c:\users\xxx\anaconda3\envs\kp_171\lib\shutil.py", line 395, in _rmtree_unsafe
    _rmtree_unsafe(fullname, onerror)
  File "c:\users\xxx\anaconda3\envs\kp_171\lib\shutil.py", line 395, in _rmtree_unsafe
    _rmtree_unsafe(fullname, onerror)
  [Previous line repeated 1 more time]
  File "c:\users\xxx\anaconda3\envs\kp_171\lib\shutil.py", line 400, in _rmtree_unsafe
    onerror(os.unlink, fullname, sys.exc_info())
  File "c:\users\xxx\anaconda3\envs\kp_171\lib\shutil.py", line 398, in _rmtree_unsafe
    os.unlink(fullname)
PermissionError: [WinError 5] Accès refusé: 'C:\\Users\\xxx\\AppData\\Local\\Temp\\tmpntjghogt\\kedro-starters\\.git\\objects\\pack\\pack-26ae52b934aecc262c73b726202a49ce0bb01487.idx'
The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "c:\users\xxx\anaconda3\envs\kp_171\lib\site-packages\click\core.py", line 782, in main
    rv = self.invoke(ctx)
  File "c:\users\xxx\anaconda3\envs\kp_171\lib\site-packages\click\core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "c:\users\xxx\anaconda3\envs\kp_171\lib\site-packages\click\core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "c:\users\xxx\anaconda3\envs\kp_171\lib\site-packages\click\core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "c:\users\xxx\anaconda3\envs\kp_171\lib\site-packages\kedro\framework\cli\cli.py", line 229, in new
    directory=directory,
  File "c:\users\xxx\anaconda3\envs\kp_171\lib\site-packages\kedro\framework\cli\cli.py", line 332, in _create_project
    raise KedroCliError("Failed to generate project.") from exc
kedro.framework.cli.utils.KedroCliError: Failed to generate project.
Error: Failed to generate project.

This is the very same stacktrace that the one in msg262584 in above link on python issue tracker.

Your Environment

Include as many relevant details about the environment in which you experienced the bug:

• Kedro version used (pip show kedro or kedro -V): 0.17.1
• Python version used (python -V): 3.7.10
• Operating system and version: Windows 10

[noklam]

Don’t worry! Thanks for open sourcing kedro, it definitely helps to solve some of our problems. 😃 It would be nice to learn more about where is kedro heading to.

[AntonyMilneQB]

Hi @noklam, thanks for digging into this and sharing. You make a very good point about not just hiding the ModuleNotFoundError but raising some sort of log. I’ll see if we can easily get this added (though in due course, when 0.18 is released, the whole except block should disappear anyway).