question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

google provider requires leveldb installed in system

See original GitHub issue

google provider now requires plyvel, which is used by leveldb hook.

to install plyvel, the user needs leveldb headers installed in the system, otherwise it will fail:

  plyvel/_plyvel.cpp:632:10: fatal error: 'leveldb/db.h' file not found
  #include "leveldb/db.h"
           ^~~~~~~~~~~~~~
  1 error generated.
  error: command 'gcc' failed with exit status 1
  ----------------------------------------
  ERROR: Failed building wheel for plyvel

the average GCP user may not use leveldb and it seems a little burdernsome to require its installation just to use GCP hooks, when most of them don’t need this library.

perhaps there is a way to make this dependency optional?

or perhaps is there a way to install only certain components of a provider?

update

after looking more closely, i see that levelDB isn’t a GCP service

it seems to me levelDB might belong as a distinct provider, e.g. google-gcp vs google-level like there is currently with apache and microsoft? @potiuk any thoughts?

cc: @mik-laj related: https://github.com/apache/airflow/pull/14105

Issue Analytics

  • State:closed
  • Created 2 years ago
  • Comments:18 (17 by maintainers)

github_iconTop GitHub Comments

1reaction
potiukcommented, May 26, 2022

Yeah. As mentioned in #15933 the right solution to your problem is to upgrade to Airflow 2.1+.

1reaction
potiukcommented, Oct 7, 2021

Why not - will you make PR for that @javatarz ? I kept on using devel anyway.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Google LevelDB Operator - Apache Airflow
LevelDB is a fast key-value storage library written at Google that provides an ordered mapping from string keys to string values. To use...
Read more >
LevelDB - Wikipedia
LevelDB is an open-source on-disk key-value store written by Google fellows Jeffrey Dean and ... ported to a variety of Unix-based systems, macOS,...
Read more >
apache-airflow-providers-google 2.2.0 - PyPI
Provider package. This is a provider package for google provider. All classes for this provider package are in airflow.providers.google python package.
Read more >
INSTALL - airflow - Git at Google
Depending on your system you might need different prerequisites, but the following ... By default `pip install` in Airflow 2.0 installs only the...
Read more >
Creating a content provider - Android Developers
You also need to increase the version number for your app that implements the new content provider. Making this change prevents system ......
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found