Automatic loading at Data Library from API keys

Use case:

1. Let’s assume that there’s a “data-provider-company” CARTO user account, managed by us, with several datasets for paid customers.
2. “alice”, a customer, gets in touch with us and buys 3 of those datasets.
3. A bizops person opens that account and creates a “user:alice:…” api key for that user with read access to the datasets that the customer paid.
4. When the user goes to the Dashboard, the paid datasets are at the Data Library.
5. When the contract ends, the API key gets renewed, so the user can’t import it anymore.

Implementation details:

• API key pattern. It must be index-friendly, unambiguous and informative. In addition, as normal keys can’t be edited, keep in mind that one user might have several of them.
  • It’d be great if the pattern for read only datasets couldn’t be used to grant write access, making human failure harder.
  • Creating this key should trigger a regeneration of target user Data Library (maybe in a lazy way, marking it as “to regenerate in the next load”).
• This needs a premium_datasets_users table that contains the usernames of those accounts with premium data. That way, at step 4, after adding all “common-data” datasets, system will look for api keys for current user at those accounts.
• Nice to have / Must have: on API key renewal or removal, the Data Library entry should dissappear (same behaviour than #13667).

This will be tackled after implementing the basics (#13666, #13667), but it’s a nice approach from a management point of view.

cc @javitonino @ethervoid @antoniocarlon let me know any additional suggestion or things that should be taken into account. In addition, I’d like to come up with a design that doesn’t couple API key management to Data Library management.

[ethervoid]

I’ve just read this, sorry 🙏

I like the idea to auto-discover premium datasets, it’ll great for the biz operations people.

The search capability sounds great indeed!

And as all said, I prefer to use a table for multiple reasons:

• As was said is data even if we have a few members by now
• Every config change needs infrastructure people involved, we don’t have merge permissions, and we have to wait 30 minutes in the worst case to have the change spread
• The config change is not made on all servers at once
• I could be wrong with this one but if we change app_config.yml we have to reboot the servers to load the new configuration

That said, great feature 😃

[juanignaciosl]

After rereading this some weeks later, I think that coupling Auth API to Data Library is a mistake, because as you’ve seen there are some issues, from autodiscovery at user level to multi cloud environments. Instead, building on top of it is better and simpler.

New proposal: a new rake similar to cartodb:remotes:load_in_data_library but receiving an API key token instead of a source dataset. That rake will load all granted datasets into user account:

rake cartodb:remotes:load_in_data_library_from_api_key[https,carto.com,80,s_user,s_user_key,t_user,g_key]

As we get a few requests to do this, we can create an endpoint and add this to superadmin.

This push approach makes updating trivial, and doesn’t add the harm of polling api keys in every common data update.