DOv2 catalog discovery (gather feedback)

The objective of this issue is to describe the DO catalog as we have defined in the DO doc

Objective

The objective is to allow the user to explore what

• Datasets
• Geographies
• Variables / Variables groups

we have available on CARTO in a similar way as we do on the web, but programmatically.

It is public and the catalog will be accessible without an API key.

How

From the main class of DO, you can get all the countries or all the categories:

DO.countries => [Country, Country, ...] #Static
DO.categories => [Category, Category, ...] #Static

From every Country or Category, you can access to datasets, geographies and categories/countries:

Country.categories => [Category, Category, ...] 
Country.datasets => [Dataset, Dataset, ...]
Country.geographies => [Geography, Geography, ...]

Category.datasets => [Dataset, Dataset, ...]
Category.countries => [Country, Country, ...]
Category.geographies => [Geography, Geography, ...]

A from a Dataset you can access to the variables:

Dataset.variables = [Variable, Variable, ...]

This is the main idea. Get more info checking the doc.

Where

Inside the data namespace, we are going to create a new one called catalog

Classes structure

First of all, in the doc, check the backend classes in the DO metadata to start creating the big picture.

In CARTOframes, (of course, it could be changed in the implementation) we expect to have the following structure:

• data
  • Catalog
    • Dataset
    • Country
    • Variable
    • Category
    • Geography

And probably, for each class, we will need a list class

List classes

We want to avoid methods usage, making the discovery cleaner, and at the same time, we want to support the following case:

Dataset.variables => [Variable, Variable, ...]
Dataset.variables.starred => [Variable, Variable, ...]

The idea here is to create a new list class called, for example, Variables extending the Python list class to allow returning the list (Dataset.variables) and also adding methods (Dataset.variables.starred)

Variable and Variables group

A Dataset has variables. The variables are the types/groups of information of the dataset, for example: total population, temperature, …

In the same way, a dataset can have a group of variables, for example: by age, by gender, … A varibles group has several variables, for the case of by age for example: 0-20, 20-40, 40-60, 60+

Also, we are going to have starred variables.

How it works in the catalog

Dataset.variables => returns the first level of variables in the dataset

Variable.children => returns array of variables (if it is a group) or None (if it is a variable)

Getting a element from a list

For example, to get a Country from a countries list:

DO.countries => [Country, Country, ...] #Static

we can do it from several ways (from one or from every one):

• using the list index:

country = DO.countries[0]

• static method:

country = Country.get(<country_id>) #Static

• magic method from the list:

country = DO.countries.get(<country_id>)

PS: Is Dataset from DO the same as the current Dataset from CARTOframes? Good question!!

cc @alasarr @alrocar @andy-esch

[alasarr]

As a quick view, I will remove the download part of this task, since it will involve all the workflow with service accounts and it’s not a discovery method it’s more an access method.

[alrocar]

We are moving this to in review let’s close it or freeze it if we don’t gather more feedback during this week.