Entities ids cache update workflow

This is not a bug report or a new feature idea, it is a question about cache update workflow for an offline app using Offix, so I apologize if I added this issue using the wrong label

I am building an offline-first application that lets users switch between offline and online modes, and Offix looked like a very good solution for what I needed, until I ran into a problem:

Users can create items in a nested list like so:

The GraphQL mutation I use to create items has the following signature: CreateItem(itemTitle, parentId) , parentId being the id of the item where the newly created item will be placed under. So, for example, to create the “sub-item 1” in the image above, I’d call CreateItem(“sub-item 1”, <item 4 id>) Entities Ids must be generated by the database on the back-end instead of using client-generated ones.

I’m using Offix’s OptimisticResponse helper, which generates an unique Id for each item using the client:UUID format. If a users creates many nested items while offline, Offix will queue up many mutations with the second parameters (parentId) being populated with the Offix-generated ids. As soon as the user switches to online-mode, I assume those mutations will be sent to the server in parallel (with limitations imposed by the browser). But all of the parentIds for the mutations will still be the Offix-generated ids, which will result in the API returning an error for all of them, because entities with those parentIds do not exist in the database.

With that in mind, and assuming it is correct, here is the workflow I want to implement when a user creates items:

1. User switches to Offline Mode.

2. User creates many new items.

3. optimisticResponses with client-generated ids are created for all of them.

4. Inside the CreateItem mutation update() function: Add all optimistic responses to the cache using writeFragment() and writeQuery() so that components using useQuery() will be reactively updated

5. User can close/refresh the page many times. Offline data won’t be lost since the optimistic responses were all cached in the persistent store.

6. User switches to Online Mode.

7. Many mutations will be queued up by Offix. Send them to the server asynchronously in series. After each mutation call returns, and before sending the next one, do the following: Replace all references to the client-generated ids from the call that just returned, with the real id returned from the server.

   Example: Using the image above as reference, say there are 2 mutations queued up:

   • CreateItem(“sub item 3”, <real item 4 id>), having a client-generated id of client:subitem3.
   • CreateItem(“sub sub item 1”, “client:subitem3”), having a client-generated id of client:subsub1.

   The first mutation will work since <real item 4 id> exists in the database. The second will fail since “client:subitem3” id was client-generated and it doesn’t exist in the database. So, as soon as the first mutation returns, and before any other are sent, all references to “client:subitem3” in the queue will be replaced with the real id contained in the first mutation’s API response. So the second mutation now becomes:

   • CreateItem(“sub item 3”, < real id returned from first mutation >), having a client-generated id of client:subitem3.

   And the same process will happen for every CreateItem mutation.

8. The CreateItem mutation update() function will be called again, this time with the actual responses from the server. Inside it, do: Replace client-generated id for the entity *everywhere* it appears in the cache with the real entity id returned from the API. (this could be a performance bottleneck)

Does this workflow makes sense for an app that has online and offline modes where users may be offline for months before synching their changes? Is it possible to acheive it using Offix? Can I control the mutation queue behavior to achieve that? Is it recommended to update the cache using optimisticResponses?

Thank you!

[lucasmez]

@wtrocki Just a suggestion regarding the problem of nested updates, where “parentIds” in the queue should be updated once a mutation returns from the server (in addition to the regular id field), maybe it’d be worthwile having Offix accept an array of id names to replace in the queue. For example:

We would pass this option to Offix:

const idsToReplace = ["parentId", "anotherId"];

Then the replaceClientGeneratedIDsInQueue() method, in addition to replacing the “idField” variables for the mutations in the queue, would also replace fields with the names added above.

Not sure if this is a common enough problem to warrant the change, but I’ll try doing this for my case.

[wtrocki]

As for hacks - All listeners in queue will be called but nothing stopping us to flush existing ones and attach new one. It happens that they are public and can be accessed from client

https://github.com/aerogear/offix/blob/master/packages/offix-scheduler/src/queue/OfflineQueue.ts#L55

Obviously this is just us trying hard to get this to work in really convoluted way. I would love to have more time and figure out how we can fix this right so no such hacks are needed. Obviously if you want to propose something that will work we are open for any suggestions.

We are really glad you put so much effort to detail this problem. We know we need to figure some solution out. It was @kingsleyzissou who also found this problem and came with fix just today