Cannot use shared library storage with lib_deps

---

Configuration

Operating system: Linux

PlatformIO Version (platformio --version): PlatformIO, version 4.0.0b2

Description of problem

Having a project with multiple environments that re-use same lib_deps variable, I encountered a confusing behaviour with global library storage:

1. Global library storage is ignored with specific versions from lib_deps variable (library @ version or github links)
2. It is seemingly impossible to specify shared library storage for multiple environments. libdeps_dir is following new rules for workspace_dir, forcing each environment to create a subdirectory {libdeps_dir}/{env}/…library…

Steps to Reproduce

The content of platformio.ini:

[env:a]
platform = espressif32
board = lolin32
framework = arduino
lib_deps =
    ArduinoJson@5.13.4

[env:b]
platform = espressif32
board = lolin32
framework = arduino
lib_deps =
    ArduinoJson

The content of src/dummy.cpp

#include <Arduino.h>
void setup() {}
void loop() {}

$ pio lib -g install ArduinoJson@5.13.4
Library Storage: /home/builder/.platformio/lib
Looking for ArduinoJson library in registry
Found: https://platformio.org/lib/show/64/ArduinoJson
LibraryManager: Installing id=64 @ 5.13.4
Using cache: /home/builder/.platformio/.cache/7e/ba586a56b4024a92728654afd0862c7e
Unpacking  [####################################]  100%
ArduinoJson @ 5.13.4 has been successfully installed!

$ pio run -e a
Processing a (platform: espressif32; board: lolin32; framework: arduino)
------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Verbose mode can be enabled via `-v, --verbose` option
CONFIGURATION: https://docs.platformio.org/page/boards/espressif32/lolin32.html
PLATFORM: Espressif 32 1.8.0 > WEMOS LOLIN32
HARDWARE: ESP32 240MHz, 320KB RAM, 4MB Flash
DEBUG: Current (esp-prog) External (esp-prog, iot-bus-jtag, jlink, minimodule, olimex-arm-usb-ocd, olimex-arm-usb-ocd-h, olimex-arm-usb-tiny-h, olimex-jtag-tiny, tumpa)
PACKAGES: toolchain-xtensa32 2.50200.80 (5.2.0), framework-arduinoespressif32 2.10002.190416 (1.0.2), tool-esptoolpy 1.20600.0 (2.6.0)
LDF: Library Dependency Finder -> http://bit.ly/configure-pio-ldf
LDF Modes: Finder ~ chain, Compatibility ~ soft
Looking for ArduinoJson library in registry
Found: https://platformio.org/lib/show/64/ArduinoJson
LibraryManager: Installing id=64 @ 5.13.4
Using cache: /home/builder/.platformio/.cache/7e/ba586a56b4024a92728654afd0862c7e
Unpacking  [####################################]  100%
ArduinoJson @ 5.13.4 has been successfully installed!
Found 58 compatible libraries
Scanning dependencies...
^CBuild interrupted.

$ pio run -e b
Processing b (platform: espressif32; board: lolin32; framework: arduino)
------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Verbose mode can be enabled via `-v, --verbose` option
CONFIGURATION: https://docs.platformio.org/page/boards/espressif32/lolin32.html
PLATFORM: Espressif 32 1.8.0 > WEMOS LOLIN32
HARDWARE: ESP32 240MHz, 320KB RAM, 4MB Flash
DEBUG: Current (esp-prog) External (esp-prog, iot-bus-jtag, jlink, minimodule, olimex-arm-usb-ocd, olimex-arm-usb-ocd-h, olimex-arm-usb-tiny-h, olimex-jtag-tiny, tumpa)
PACKAGES: toolchain-xtensa32 2.50200.80 (5.2.0), framework-arduinoespressif32 2.10002.190416 (1.0.2), tool-esptoolpy 1.20600.0 (2.6.0)
LDF: Library Dependency Finder -> http://bit.ly/configure-pio-ldf
LDF Modes: Finder ~ chain, Compatibility ~ soft
Found 57 compatible libraries
Scanning dependencies...
Dependency Graph
|-- <ArduinoJson> 5.13.4
^CBuild interrupted.

Examining libdeps contents:

$ ls .pio/libdeps/*
.pio/libdeps/a:
ArduinoJson_ID64

.pio/libdeps/b:

I have also tried adding [env:...] lib_extra_dirs = ./my_shared_libs and installing the library there.

Actual Results

lib_deps = … library gets installed in the .pio/libdeps/env/… despite being in the global storage

Expected Results

Library presence is checked in global or lib_extra storage dir before installing it in the {workspace_dir}/libdeps storage. Which I think I solved locally for me via mcspr@c231535e

It would also be extremely useful to still be able to use pio run -e ... with old-style .piolibdeps storage when there is no practical reason of having multiple copies of the same library in .pio/libdeps/…/library For example, when environments only differ in src_build_flags values or without internet connection present. I wonder if this can be solved by making default libdeps_dir actually reference that it uses current environment as a part of the path (e.g {workspace_dir}/libdeps/{pioenv})?

[ivankravets]

I like your implementation! Let keep this in mind when someone will ask. We don’t want to support different workflows in official docs. It will help us to save resources when developers will have any issues.

Extra scripting was specially created for these cases when you don’t like default PlatformIO Core behavior. So, everyone can tune it for own needs.

[mcspr]

BTW, I did try extra script just now. Mini template project with subprocess and LibraryManager (probably OK too?) https://gist.github.com/mcspr/80829350513faf37d3154ab67c40ae8b Do you think this can be added to the extra_scripts examples? Being both an example of the GetProjectOption (which I did not notice before!) and general library management.