question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Replace custom 'activate' method and symlink injection with a `MELTANO_PATH` env var - or a modified `PATH` - for child processes

See original GitHub issue

In #6468, we are improving our interop for how Meltano is able to create symlinks which ensure it will be made available to child processes and processes like Airflow which will want to callback or invoke Meltano directly.

Now that we have the EDK and a growing set of best practice paradigms, this seems like a problem we could solve in a better way than the symlink approach.

Two possible options for working around the symlink approach:

  1. Pass to child processes a MELTANO_PATH env var which points to the Meltano executable.
  2. Before invoking a child process, prefix the child process’s PATH env var with the path to the Meltano executable.

These options also have the nice benefits of:

  1. Working well within containers, where Meltano is expected to be installed but may not be in an obvious location.
  2. Working well in contexts where multiple versions of Meltano might be installed.

Issue Analytics

  • State:open
  • Created a year ago
  • Reactions:2
  • Comments:10 (6 by maintainers)

github_iconTop GitHub Comments

1reaction
DouweMcommented, Oct 31, 2022

@aaronsteers The symlink existed only for the benefit of https://gitlab.com/meltano/files-airflow/-/blob/master/bundle/orchestrate/dags/meltano.py#L38, so indeed adding it to the PATH or exposing it through MELTANO_PATH/EXECUTABLE should be sufficient.

1reaction
aaronsteerscommented, Oct 19, 2022

(Since I/we don’t have a lot of background info here, perhaps good to phone a friend with more historical context 📞 😄.)

Seems pretty niche upon reflection, so I suppose there isn’t much reason for the symlink save for its legacy usage.

@DouweM - We postulate above that the symlink’s primary benefit is for subprocess/sideprocesses which need to find the meltano executable - and that similar functionality could be replaced by simply ensuring meltano is on the PATH or else explicitly pointed to in a future MELTANO_PATH env var which would be leveraged by meltano-integrated tools when it is found in context.

By chance do you have any additional historic or technical context for the need for the symlink which we haven’t covered here?

Read more comments on GitHub >

github_iconTop Results From Across the Web

How to set child process' environment variable in Makefile
The best way to prove that an environment variable has been set, is to query this environment variable inside another program that make...
Read more >
hashicorp/envconsul: Launch a subprocess with ... - GitHub
Launch a subprocess with environment variables using data from @HashiCorp ... are given to the child process. custom = ["PATH=$PATH:/etc/myapp/bin"] # This ...
Read more >
change environment of a running process - Unix & Linux Stack ...
Is there a way to change another process's environment variables? ... unique directory and pass the same path to the child shell script....
Read more >
A process inherits its environment from its parent, and the ...
Of course, once the process is running, it is free to change its environment variables by calling SetEnvironmentVariable , and those modified ......
Read more >
Env Variables and Modes - Vite
During production, these env variables are statically replaced. It is therefore necessary to always reference them using the full static string.
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found