Suggested changes to the Wiki

I am part of a group of CS masters students who are currently pursuing a course on Software Architecture and selected electricitymap as our open source project of choice to study and hopefully contribute to. However, while we were attempting to setup the project in our local environments, we ran into a number of problems even though we followed the Wiki on the master repository.

Technical specifications of one of the machines that was used to setup locally:

• Operating System: Windows 10 Home (x64) Version 2009 (build 19042.867)
• CPU: AMD Ryzen 5 3500U with Radeon Vega Mobile Graphics
• RAM: 6 GB Usable Installed Memory
• IDE: Pycharm Community Edition 2020.3.3

Note: While the people in our group used different machines, the fixes/workarounds listed below worked for all of us. The IDE and OS were the same - Pycharm Community Edition 2020.3.3 running on Windows 10

What follows is a list of problems we faced and how we were able to solve them and setup the project successfully on our local machines.

When we ran the poetry install -E parsers command, the following errors popped up for opencv-python (Note: we had already installed Cython at this point):

After a lot of Google-fu and Stack Over Flow searches we finally figured out that cytoolz was the issue. But when we tried to install cytoolz, we ran into this:

Essentially, both wheels and setup.py were failing with the same error - and yes we had already installed MicroSoft C++!

Upon reaching out to the slack channel, it was recommended that we use python 3.6.x instead of 3.9.x (which was the latest version of python as of the time of opening this issue). We performed the following steps and the poetry issues were finally fixed!

1. Delete the local repository permanently (on Windows this is Shift + Delete).
2. Install python 3.6.8 from python.org - this was the last feature release for python 3.6.x. Every release of 3.6.x after this was a security update.
3. Clone the electricitymap repo from github into pycharm again (do a fresh clone basically).
4. Ensure that Python 3.6 is the interpreter - this step is crucial. Click here for a quick tutorial.
5. Make sure the clone is inside a virtual environment. Pycharm IDE lets you do this via GUI when you configure it to use python 3.6 as the interpreter - click here for a quick tutorial.
6. Check the version of poetry that you have installed. The latest version as of March 14 2021 was 1.1.5 and that is the recommended version for electricitymap.
7. If you do not have the latest version of poetry, run the poetry self update command. You can do this in the IDE itself.
8. Reinstall the basic python packages for version 3.6.x (wheel, numpy, pandas).
9. Run the poetry install -E parsers command again - it should work now!

Clarification on step 8 - these packages were already installed but on python 3.9.x, not 3.6.x which is the recommended version on the GitHub page for electricitymap. Pycharm shows a list of all packages installed when you select a specific version of python as the interpreter, which is how we realized we would have to reinstall the packages for python 3.6.x

Once this saga was over, we ran into some minor trouble while trying to run the parsers themselves. When we attempted to run the python test_parser.py FR price command as stated in the Wiki, this popped up:

Here is the offending code from the file test_parser.py: from electricitymap.contrib.parsers.lib.parsers import PARSER_KEY_TO_DICT

Here is what was recommended by the developers as a workaround: from parsers.lib.parsers import PARSER_KEY_TO_DICT

Apparently this problem is caused by the way Windows handles symlinks. The developers are currently working on a fix, but no timeline was provided at the time of opening this issue.

[srinathjayaraman]

Ah correct, sorry I miscalculated. Since the permanent fix for the test_parsers.py file is to get rid of symlinks, is it alright to close this issue, or would you prefer that it be kept open for tracking purposes?

[srinathjayaraman]

I believe this is outside the scope of our wiki/project to document how to set up various IDEs.

Yes I understand that, it’s why I included information about what IDE we used so if people need to perform a similar function (like setting up virtual env) in their own IDE’s they can google it. It’s really in there to provide as much clarity as possible on what caused our issues.

As we do support Python 3.7 internally, we should change this to python = ‘>= 3.6, < 3.8’

Should I change the pyproject.toml file locally and open a PR to commit to main?

Does poetry run python test_parser.py FR price work?

I will try this and provide an update!