question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Raise a warning for required, but uninstalled packages

See original GitHub issue

Describe the feature

The error message for an uninstalled package comes straight out of the Jinja library:

Compilation Error in analysis scratch (analysis/scratch.sql)
    'dbt_utils' is undefined

This error message is strictly true, but I think it might be worth taking the time to write a more explicit warning message for uninstalled packages

Warning: The following packages are listed as required in `packages.yml` but have not been installed
- dbt_utils

Run `dbt deps` to install these packages

Since we have the expected packages in packages.yml and the installed packages in dbt_modules, this should be possible, right? And not that hard?

Note: I’ve only got it as a warning so that the places where the package is referenced still causes an error, but I’m open to having this as an error instead

Describe alternatives you’ve considered

Keep as is

Who will this benefit?

Confused users

Are you interested in contributing this feature?

No ty 😃

Issue Analytics

  • State:closed
  • Created 3 years ago
  • Reactions:1
  • Comments:6 (5 by maintainers)

github_iconTop GitHub Comments

1reaction
jtcohen6commented, May 13, 2021

@jaypeedevlin I don’t know why I thought this would be so tricky! At the prompting of your PR comment, I took a look down this code path, and I think it could be as simple as sticking a check right in load_dependencies: https://github.com/fishtown-analytics/dbt/blob/1f33b6a74a614aa346824f16c5629e4d910d2106/core/dbt/config/runtime.py#L327-L334

    def load_dependencies(self) -> Mapping[str, 'RuntimeConfig']:
        if self.dependencies is None:
            all_projects = {self.project_name: self}
            internal_packages = get_include_paths(self.credentials.type)
            # raise exception if fewer installed packages than specified packages
            package_dirs = self._get_project_directories()
            count_packages_specified = len(self.packages.packages)
            count_packages_installed = len(tuple(package_dirs))
            if count_packages_specified > count_packages_installed:
                raise_compiler_error(
                    f'dbt found {count_packages_specified} package(s) specified in packages.yml, '
                    f'but only {count_packages_installed} package(s) installed in {self.modules_path}. '
                    f'Run "dbt deps" to install package dependencies.'
                )
            project_paths = itertools.chain(internal_packages, package_dirs)

We can’t match on names, since self.packages.packages is just the info from packages.yml (which may or may not match the package name, if it’s a git repo name or local directory). We also can’t check lengths for perfect equality, since it’s possible that count_packages_installed will be greater than count_packages_installed, if the specified packages have package dependencies of their own. But I’d say this is a lot better than nothing, and I’d be excited about pairing it with the kinder UndefinedMacroException you’ve got over in #3343.

1reaction
jaypeedevlincommented, May 10, 2021

@jtcohen6 I’d love to jump onto this one in the next day or so.

Just to clarify, regarding:

I’m going to mark this a good first issue! And a nice chance to add a new dbt exception 😃

Will catching and reraising the exception be enough, or do I also need to modify exceptions.py?

Read more comments on GitHub >

github_iconTop Results From Across the Web

Raise a warning for required, but uninstalled packages #2999
Describe the feature The error message for an uninstalled package comes straight out of the Jinja library: Compilation Error in analysis ...
Read more >
How to Fix Pip “Yanked Version” Warnings - Adam Johnson
To fix a yanked version warning, we need to find an un-yanked version of the package that will work for us. This might...
Read more >
warnings — Warning control — Python 3.11.1 documentation
Commonly used warning filters apply to either all warnings, warnings in a particular category, or warnings raised by particular modules or packages.
Read more >
pip cannot uninstall <package>: "It is a distutils installed project"
This error means that this package's metadata doesn't include a list of files that belong to it. Most probably, you have installed this ......
Read more >
How to capture warnings — pytest documentation
Although not recommended, you can use the --disable-warnings command-line option to suppress the warning summary entirely from the test run output. Disabling ...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found