Rethink Limits model rather than add a new field in User model

TL;DR: users table keeps growing with every single small modification.

users table currently has 92 columns (!), and it keeps growing with every single configuration parameter. Following projects (check inbound links) are going to need more configuration parameters, and we should stop this mess before it explodes.

I’d like to open the debate about the need and the approach.

Possible approaches (not all of them are excluding): I) Remove unused columns. II) Create some user_XXX tables, grouping existing columns into meaningful sets. III) Group meaningful sets into JSONB/HSTORE columns. IV) Address new needs with any of the previous approaches, postpone migrating existing columns.

Any proposal should also take into account the impact on queries, especially at presenters. Currently, presenters present most of the columns because the cost is low. Nevertheless, we might want to change that to take advantage of the refactor.

Thoughts, @CartoDB/builder-backend ? Not urgent at all.

cc @matallo

[javitonino]

So, after spending some more time thinking/talking about this, a proposal:

• Create a rate_limits table, with an id and a column per limit.
• Create an account_types table, with fields account_type, rate_limits_id (will probably grow)
• Add a rate_limits_id field to users, NULL by default

We use a common table for all limits, so we can share the same columns for account_types and users, instead of having two separate tables with almost the same schema.

Users will get the limits from rate_limits_id if set, or from the relation through the account type otherwise. As most users don’t overwrite the limits, that table will probably be very small (one row per plan, plus a few users which need specializations). We minimize data duplication while still alowing overrides. As the table will be small, migrations should not pose any issue, so I feel confident using postgresql columns, so we can benefit from psql’s not null/default.

Syncing the data will be a bit trickier, since we still have to duplicate the data in redis, so a single change of a limit associated to a plan will imply many users invalidations, which are unavoidable anyway. But it should not be too hard (just loop over affected users), and the design is nicer.

[oleurud]

Redis convention proposal

Database As we have talked, it is a good idea to use another Redis db (not in use). I am using the 8 db in the rate limit development. Any problem?

Key limits:rate:store:{user}:{app}:{endpointGroup}

Being: user: the CARTO username endpointGroup: the endpoint group name. app: sql or maps

Maps API endpoints anonymous static static_named dataview dataview_search analysis tile --> it has 6 values attributes named_list named_create named_get named named_update named_delete named_tiles

SQL API endpoints query query_format job_create job_get job_delete

Data to save and persistence First of all, we must think about the best way in performance terms. For each API request, we will exec a Lua script that makes 2 tasks:

• get the limits values by user and endpoint
• get the rate limit state

As we talked, we need to save 3 values (each limit needs: maxBurst, countPerPeriod and period) by user and endpoint, less in the tile case that we need to save 6 values (we will save 2 limits). And maybe, we will need to add/remove a limit in a special case.

Thinking about the best way to save the data with @javitonino: use a Redis list, with 3 values per limit. The order must be: maxBurst, countPerPeriod and period (this is important, we have an app writing the limits and another different consuming them)

So in case, we have to save 2 limits: 5 req/second and 20 req/minute:

[ 0, 5, 1, 0, 20, 60]