Support parameterized SQL in bigquery integration (bq_solid_for_queries)

@solid
def get_foo_id_list(_) -> List[str]:
    return ['a', 'b', 'c']

bq_solid_for_queries([
   f""" select * from my_table where id in {foo_id_list}"
]

I’d like to construct a SQL query using the output of an upstream solid. How would I do that?

[sryza]

One way of addressing this would be with a bigquery_solid decorator. Thoughts?

@solid
def get_foo_id_list(_) -> List[str]:
    return ['a', 'b', 'c']

@bigquery_solid(input_defs=[InputDefinition(List[str])])
def my_bq_solid(foo_id_list):
    return [f""" select * from my_table where id in {foo_id_list}"]

[natekupp]

hey @zzztimbo thanks for checking!

Right now, bq_solid_for_queries is a solid factory, and expects the queries to be available at pipeline compilation time, vs. fed in as a Dagster input. If it’s possible to feed ['a', 'b', 'c'] in non-Dagster Python code at pipeline construction time, you can try that.

If you definitely need SQL to be an input to the solid, the below should work:

from dagster_pandas import DataFrame
from dagster_gcp.bigquery.configs import define_bigquery_query_config
from dagster_gcp.bigquery.solids import _preprocess_config
from google.cloud.bigquery.job import QueryJobConfig

@solid(
    config=define_bigquery_query_config(), required_resource_keys={'bigquery'},
)
def bq_query_input_solid(context, sql_queries: List[str]) -> List[DataFrame]:
    query_job_config = _preprocess_config(context.solid_config.get('query_job_config', {}))

    results = []
    for sql_query in sql_queries:
        cfg = QueryJobConfig(**query_job_config) if query_job_config else None
        context.log.info(
            'executing query %s with config: %s'
            % (sql_query, cfg.to_api_repr() if cfg else '(no config provided)')
        )
        results.append(context.resources.bigquery.query(sql_query, job_config=cfg).to_dataframe())

    return results

I’ll keep this issue open to track adding the above to the dagster_gcp library, since this seems like it should be functionality we provide out of the box.