Data observatory frontend integration

In order to revamp the Data Observatory analysis form (design here https://invis.io/5G9YKGISB), the current SQL call is not enough, and we need to use all the helpers already available in the API.

@talos we need some guidance, let me do some questions about the Data Observatory:

• Could you point us to the proper documentation? I know we have all signatures listed here https://github.com/CartoDB/dataservices-api/blob/development/client/renderer/interface.yaml but it should be great to know what method to use when.
• How do we need to manage the geometry param? I can see some methods accept a bounds param of type geometry, but not sure how to deal with it. Can we pass a country name? Do you have an example of this param using the current bounding box?
• Related to above, how do we get the regions available? If you see the design, there is a number next to the region’s name, I guess is the number of measurements available in that region. Does the API provide this number?
• We can’t see the new column name to augment the dataset, how does it work?

A measurement has a structure like this, according to the API response:

{
    numer_id: "xxx",
    numer_name: "...",
    numer_description: null,
    numer_weight: "3",
    numer_license: null,
    numer_source: null,
    numer_type: "Numeric",
    numer_aggregate: "sum",
    numer_extra: null,
    numer_tags: {
         section/tags.ca: "Canada",
         unit/tags.people: "People",
         subsection/tags.language: "Language",
         license/ca.statcan.license.statcan-license: "Statistics Canada Open Licence Agreement",
         source/ca.statcan.license.statcan-census-2011: "Statistics Canada Census of Population 2011"
    },
    valid_denom: false,
    valid_geom: false,
    valid_timespan: false
},

• In order to implement the filters, should we use tags?
• I see the description is null in almost every result, how we get the description of a measurement? and for filters?
• In order to run the analysis, what parameters are mandatory? region and measurement? all but normalize?

@rafatower do you see any troubles in order to get the quota info? Take in mind we can have multiple measurements embedded in the same form.

[xavijam]

Closing because it is already working but we are polishing it 🎉 .

[talos]

Thanks for all the comments! Very much appreciated.

There are a few columns in OBS_GetAvailableNumerators that you are correct are not especially needed. Some, like weight and license could simply be eliminated.

While I generally prefer SELECT *, if you wanted to reduce the weight of the response the simplest approach would be to SELECT only the columns you needed – excluding the license, weight, valid, etc.

@saleiva mentioned the responses are coming in heavier than you’d like. Do you have an example request so I could think of ways to trim it down?

I’m not sure I see how we could get rid of requests subsequent to the main measurements one (geometry, timespan, normalization). Geometries, timespans, and normalizations are all shared between measurements, so to include them nested would explode the size of the response. And each time a different measurement is selected, the exact set that works changes, so I don’t see any way to avoid subsequent requests on selection change.

For example, if someone chooses “Total Population” in the USA, then “2011 - 2015” for timespan, they will have county, census tract, zip code, and many other geometry options.

However, if they switch their timespan to “2006 to 2010”, they will no longer have zip code as an option, as that’s not a geometry that was published by the Census in that release. I don’t see any practical way to capture that information without either very large, complex nested objects (which will excacerbate existing response sizes) or making several requests (which is what we’re doing now).

There may be some reasoning behind combining the initial region selector and measurements options into one request. Indeed, it should be possible to rewrite that SQL into a nested version that could achieve that. However, it would likely again somewhat bloat response sizes.

That would look something like this:

• obs_measurements_by_region
• obs_getavailabledenominators
• obs_getavailablegeometries
• obs_getavailabletimespans

One could argue for having that first one be absorbed into the extension API, but to me it seems very specific to the Builder implementation and thus better left as custom SQL outside the extension.