Driver args and parameters example

Apologies if this is a stupid question, but I’m having difficulty finding information / examples in the documentation about how to use args and parameters together in a custom driver. I need to pass to the driver the username and password environment variables, which I know is possible, but I couldn’t find an example of custom driver where this is implemented.

Any pointers to examples would be greatly appreciated! Looking forward to using intake at my organization.

[martindurant]

This is a fine place for this kind of question and, again, if you have advice on how the documentation and examples can be made better, it would be much appreciated.

It is anticipated that a data source may refer to a whole data-set, which is too big for memory, and can then only be processed either by iterating with read_chunked or to_dask. However, you could easily have several data sources based on the same dataset, or use parameters to select from a larger data-set to end up with manageable pieces, or even use the alias mechanism to pass data sources to another driver for further filtering. Which option is best is up to the catalog author, and doubtless depends on the specifics of the problem. It sounds like in your situation, you would prefer to push filtering to the server, so parameters or several “views” onto the same data-set sound best.

Is there any way to pass all of the parameters into the arguments section of the catalog as a dictionary?

This is not how things are currently implemented, parameters are only used at the time of data source instantiation, in the method LocalCatalogEntry._create_open_args, which fills in argument templated with the parameter values (user-supplied, or defaults). However, the expand_templates function that is called allows for optionally passing back those names that do not get used, and I don’t see why these should not be passed to the source as a dictionary as you describe.

should be datetime.today() - datetime.timedelta(days=5*365). Is something like that currently supported

No, the only functions currently supported for expansion of the defaults are for environment variables and external shell. Others could be possible, but it does not seem right to try to put executable code directly into the catalog spec like this. However, a custom driver is free to have whatever internal defaults it likes, or interpret strings to mean specific things, like any python code. Suggestions on a comprehensive grammar of parameters that would be able to express your requirements welcome here.

I regularly just want to see a preview of it with actual values. If I were querying the database directly,

This is actually what intake_sql does already, although it may be buried in the Dask code, and may involve reading the whole of the first partition. Other drivers like ElasticSearch are more explicit about reading N rows, to figure out dtypes. However, a head() or preview() method does sounds like something that users would want. It would require implementation on a driver-by-driver basis, would not be useful for some unpartionionable data types, and not make any sense for some data that is not obviously sequential (like N-D arrays). You could make a separate issue to request the feature to get discussion.

df = cat.orders.get(start_dt=…, end_dt=…, order_type=…, amount=…)

Yes, that’s exactly how you specify parameter values as a user. The get verb is actually optional and assumed, you have the same behaviour with cat.orders(start_dt=...). Any parameter not given will take its default value, if there is one.

[youngsaj]

Simple enough, using .get() and some further logic works great! Once I get this up public I’ll point you to it, but that may take a while yet.