Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Dagit should not crash if one of the repository locations throw an error

See original GitHub issue

If Dagit is unable to load one of several repository locations, then it would be nice for it to still load but display an error message in Dagit itself instead of at initialization time.

If we imagine a set up where each team manages their own gRPC server (or python environment) but all the servers are managed through a single Dagit server, then one team can bring down Dagit by breaking their gRPC server.

At the very least, we should have better error messages. Currently, when Dagit is unable to connect to a gRPC repository location, the following stack trace below is shown. Ideally, we should say something along the lines of:

Unable to connect to gRPC repository location "xyz"

Current trace:

Traceback (most recent call last):
  File "/Users/sashankthupukari/.pyenv/versions/dagster-3.8.1/bin/dagit", line 11, in <module>
    load_entry_point('dagit', 'console_scripts', 'dagit')()
  File "/Users/sashankthupukari/projects/dagster/python_modules/dagit/dagit/cli.py", line 198, in main
    cli(auto_envvar_prefix="DAGIT")  # pylint:disable=E1120
  File "/Users/sashankthupukari/.pyenv/versions/3.8.1/envs/dagster-3.8.1/lib/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/Users/sashankthupukari/.pyenv/versions/3.8.1/envs/dagster-3.8.1/lib/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/Users/sashankthupukari/.pyenv/versions/3.8.1/envs/dagster-3.8.1/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/sashankthupukari/.pyenv/versions/3.8.1/envs/dagster-3.8.1/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/Users/sashankthupukari/projects/dagster/python_modules/dagit/dagit/cli.py", line 91, in ui
    host_dagit_ui(host, port, path_prefix, storage_fallback, port_lookup, **kwargs)
  File "/Users/sashankthupukari/projects/dagster/python_modules/dagit/dagit/cli.py", line 98, in host_dagit_ui
    with get_workspace_from_kwargs(kwargs, instance) as workspace:
  File "/Users/sashankthupukari/projects/dagster/python_modules/dagster/dagster/cli/workspace/cli_target.py", line 217, in get_workspace_from_kwargs
    return workspace_from_load_target(created_workspace_load_target(kwargs), instance)
  File "/Users/sashankthupukari/projects/dagster/python_modules/dagster/dagster/cli/workspace/cli_target.py", line 179, in workspace_from_load_target
    return load_workspace_from_yaml_paths(load_target.paths, python_user_process_api)
  File "/Users/sashankthupukari/projects/dagster/python_modules/dagster/dagster/cli/workspace/load.py", line 22, in load_workspace_from_yaml_paths
    workspaces = [
  File "/Users/sashankthupukari/projects/dagster/python_modules/dagster/dagster/cli/workspace/load.py", line 23, in <listcomp>
    load_workspace_from_config(workspace_config, yaml_path, python_user_process_api)
  File "/Users/sashankthupukari/projects/dagster/python_modules/dagster/dagster/cli/workspace/load.py", line 60, in load_workspace_from_config
    _location_handle_from_location_config(
  File "/Users/sashankthupukari/projects/dagster/python_modules/dagster/dagster/cli/workspace/load.py", line 287, in _location_handle_from_location_config
    return _location_handle_from_grpc_server_config(location_config["grpc_server"], yaml_path)
  File "/Users/sashankthupukari/projects/dagster/python_modules/dagster/dagster/cli/workspace/load.py", line 235, in _location_handle_from_grpc_server_config
    return RepositoryLocationHandle.create_grpc_server_location(
  File "/Users/sashankthupukari/projects/dagster/python_modules/dagster/dagster/core/host_representation/handle.py", line 179, in create_grpc_server_location
    list_repositories_response = sync_list_repositories_grpc(client)
  File "/Users/sashankthupukari/projects/dagster/python_modules/dagster/dagster/api/list_repositories.py", line 39, in sync_list_repositories_grpc
    api_client.list_repositories(), (ListRepositoriesResponse, SerializableErrorInfo)
  File "/Users/sashankthupukari/projects/dagster/python_modules/dagster/dagster/grpc/client.py", line 123, in list_repositories
    res = self._query("ListRepositories", api_pb2.ListRepositoriesRequest)
  File "/Users/sashankthupukari/projects/dagster/python_modules/dagster/dagster/grpc/client.py", line 72, in _query
    response = getattr(stub, method)(request_type(**kwargs))
  File "/Users/sashankthupukari/.pyenv/versions/3.8.1/envs/dagster-3.8.1/lib/python3.8/site-packages/grpc/_channel.py", line 826, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/Users/sashankthupukari/.pyenv/versions/3.8.1/envs/dagster-3.8.1/lib/python3.8/site-packages/grpc/_channel.py", line 729, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "DNS resolution failed"
	debug_error_string = "{"created":"@1600178764.985842000","description":"Failed to pick subchannel","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":3948,"referenced_errors":[{"created":"@1600178764.985816000","description":"Resolver transient failure","file":"src/core/ext/filters/client_channel/resolving_lb_policy.cc","file_line":216,"referenced_errors":[{"created":"@1600178764.985815000","description":"DNS resolution failed","file":"src/core/ext/filters/client_channel/resolver/dns/c_ares/dns_resolver_ares.cc","file_line":375,"grpc_status":14,"referenced_errors":[{"created":"@1600178764.985727000","description":"C-ares status is not ARES_SUCCESS qtype=AAAA name=grpc is_balancer=0: Domain name not found","file":"src/core/ext/filters/client_channel/resolver/dns/c_ares/grpc_ares_wrapper.cc","file_line":287,"referenced_errors":[{"created":"@1600178764.978788000","description":"C-ares status is not ARES_SUCCESS qtype=A name=grpc is_balancer=0: Domain name not found","file":"src/core/ext/filters/client_channel/resolver/dns/c_ares/grpc_ares_wrapper.cc","file_line":287}]}]}]}]}"