Unit tests on BigQuery fail without GCP setup

The BigQuery tests fail when running Make test in amundsendatabuilder, if GCP is not set up (figured this out b/c I don’t have GCP set up on this comp).

Not urgent, but I just figured this is not the intended behavior, so I’d flag it. A partial traceback for one of the tests is shown below (all the other TestBigqueryWatermarkExtractor tests show a similar message):

_____________________________________ TestBigQueryWatermarkExtractor.test_can_handle_no_datasets ______________________________________

self = <tests.unit.extractor.test_bigquery_watermark_extractor.TestBigQueryWatermarkExtractor testMethod=test_can_handle_no_datasets>
mock_build = <MagicMock name='build' id='4618166800'>

    @patch('databuilder.extractor.base_bigquery_extractor.build')
    def test_can_handle_no_datasets(self, mock_build):
        mock_build.return_value = MockBigQueryClient(NO_DATASETS, None, None)
        extractor = BigQueryWatermarkExtractor()
        extractor.init(Scoped.get_scoped_conf(conf=self.conf,
>                                             scope=extractor.get_scope()))

tests/unit/extractor/test_bigquery_watermark_extractor.py:96:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
databuilder/extractor/bigquery_watermark_extractor.py:24: in init
    BaseBigQueryExtractor.init(self, conf)
databuilder/extractor/base_bigquery_extractor.py:55: in init
    credentials, _ = google.auth.default(scopes=self._DEFAULT_SCOPES)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

scopes = ['https://www.googleapis.com/auth/bigquery.readonly'], request = None

    def default(scopes=None, request=None):
        """Gets the default credentials for the current environment.

        `Application Default Credentials`_ provides an easy way to obtain
        credentials to call Google APIs for server-to-server or local applications.
        This function acquires credentials from the environment in the following
        order:

        1. If the environment variable ``GOOGLE_APPLICATION_CREDENTIALS`` is set
           to the path of a valid service account JSON private key file, then it is
           loaded and returned. The project ID returned is the project ID defined
           in the service account file if available (some older files do not
           contain project ID information).
        2. If the `Google Cloud SDK`_ is installed and has application default
           credentials set they are loaded and returned.

           To enable application default credentials with the Cloud SDK run::

                gcloud auth application-default login

           If the Cloud SDK has an active project, the project ID is returned. The
           active project can be set using::

                gcloud config set project

        3. If the application is running in the `App Engine standard environment`_
           then the credentials and project ID from the `App Identity Service`_
           are used.
        4. If the application is running in `Compute Engine`_ or the
           `App Engine flexible environment`_ then the credentials and project ID
           are obtained from the `Metadata Service`_.
        5. If no credentials are found,
           :class:`~google.auth.exceptions.DefaultCredentialsError` will be raised.

        .. _Application Default Credentials: https://developers.google.com\
                /identity/protocols/application-default-credentials
        .. _Google Cloud SDK: https://cloud.google.com/sdk
        .. _App Engine standard environment: https://cloud.google.com/appengine
        .. _App Identity Service: https://cloud.google.com/appengine/docs/python\
                /appidentity/
        .. _Compute Engine: https://cloud.google.com/compute
        .. _App Engine flexible environment: https://cloud.google.com\
                /appengine/flexible
        .. _Metadata Service: https://cloud.google.com/compute/docs\
                /storing-retrieving-metadata

        Example::

            import google.auth

            credentials, project_id = google.auth.default()

        Args:
            scopes (Sequence[str]): The list of scopes for the credentials. If
                specified, the credentials will automatically be scoped if
                necessary.
            request (google.auth.transport.Request): An object used to make
                HTTP requests. This is used to detect whether the application
                is running on Compute Engine. If not specified, then it will
                use the standard library http client to make requests.

        Returns:
            Tuple[~google.auth.credentials.Credentials, Optional[str]]:
                the current environment's credentials and project ID. Project ID
                may be None, which indicates that the Project ID could not be
                ascertained from the environment.

        Raises:
            ~google.auth.exceptions.DefaultCredentialsError:
                If no credentials were found, or if the credentials found were
                invalid.
        """
        from google.auth.credentials import with_scopes_if_required

        explicit_project_id = os.environ.get(
            environment_vars.PROJECT, os.environ.get(environment_vars.LEGACY_PROJECT)
        )

        checkers = (
            _get_explicit_environ_credentials,
            _get_gcloud_sdk_credentials,
            _get_gae_credentials,
            lambda: _get_gce_credentials(request),
        )

        for checker in checkers:
            credentials, project_id = checker()
            if credentials is not None:
                credentials = with_scopes_if_required(credentials, scopes)
                effective_project_id = explicit_project_id or project_id
                if not effective_project_id:
                    _LOGGER.warning(
                        "No project ID could be determined. Consider running "
                        "`gcloud config set project` or setting the %s "
                        "environment variable",
                        environment_vars.PROJECT,
                    )
                return credentials, effective_project_id

>       raise exceptions.DefaultCredentialsError(_HELP_MESSAGE)
E       google.auth.exceptions.DefaultCredentialsError: Could not automatically determine credentials. Please set GOOGLE_APPLICATION_CREDENTIALS or explicitly create credentials and re-run the application. For more information, please see https://cloud.google.com/docs/authentication/getting-started

../../../../envs/default/lib/python3.7/site-packages/google/auth/_default.py:321: DefaultCredentialsError
-------------------------------------------------------- Captured stderr call ---------------------------------------------------------
INFO:google.auth.compute_engine._metadata:Compute Engine Metadata server unavailable onattempt 1 of 3
INFO:google.auth.compute_engine._metadata:Compute Engine Metadata server unavailable onattempt 2 of 3
INFO:google.auth.compute_engine._metadata:Compute Engine Metadata server unavailable onattempt 3 of 3
---------------------------------------------------------- Captured log call ----------------------------------------------------------
_metadata.py                95 INFO     Compute Engine Metadata server unavailable onattempt 1 of 3
_metadata.py                95 INFO     Compute Engine Metadata server unavailable onattempt 2 of 3
_metadata.py                95 INFO     Compute Engine Metadata server unavailable onattempt 3 of 3

[feng-tao]

i c, then it may be good to mock the server code in test instead of calling directly…

[jornh]

@feng-tao mybad on disabling those tests when I originally made some tests run on Travis in https://github.com/lyft/amundsendatabuilder/pull/140 Prior to that we had no Travis testing at all on databuilder - and I figured back then some Travis was better than nothing.