RFC: Improved Reader/Writer API

RFC: Improved Reader/Writer API

Because of WatermelonDB’s primarily asynchronous API, we need a built-in primitive to block concurrent database writes. Otherwise, multiple concurrent asynchronous chains of operations would break consistency within a chain. A more thorough explanation.

This is currently achieved by the await database.action(() => ...) API and the @action decorator shortcut for Model methods. A complication occurs when we have an action method/block and want to call it from another action method/block. JavaScript/WatermelonDB does not have sufficient magic to understand that the intention is to call the contents of the action as part of another action - therefore, by definition, the “sub-action” is blocked and queued - and our action queue deadlocks. So we explicitly allow this by subAction(() => ...)

All of this complicates Watermelon’s API, makes the syntax uglier, and DX worse, especially for beginners, but I don’t know of a better way to achieve mutal exclusion / get rid of the need for mutual exclusion without getting rid of Watermelon’s asynchronicity.

Still, I think one of the problems of action/@action/subAction API is just how confusing the naming is. What’s even an “Action”? It’s so generic as to mean anything, or it might be confused to mean precisely “database transaction” – and actions are not transactions (despite some similarities). I also dislike “subAction”, but couldn’t think of anything better.

Proposed changes

“Actions” are now “Writers” and “Readers”.

We explicitly tell database we want to write to it. In the writer block, we get a writer object, which we can use to explicitly call other readers/writers. It also has a batch method, which is short for database.batch(...):

await database.write(async writer => {
  // performing write (and read) operations
  await writer.batch(...)
  await model.update(...)

  // calling other writers (and readers) -- ideas for the API name?
  await writer.andWrite(() => someAction())
  await writer.andRead(() => someAction())
  await writer.call(() => someAction())
  await writer.also(() => someAction())
  await writer.withWriter(() => someAction())
  await writer.useWriter(() => someAction())
  await writer.callWriter(() => someAction())
  await writer.with(() => someAction())
})

“Readers” also exist. Why? If you want to compute some information that requires multiple fetches, you also need mutual exclusion to ensure no write happens in between your fetches. But if we change “Actions” to “Writers”, then we’d wrongly suggest to the user that you should write to the database in the block. In fact, we should teach users, that they need to wrap not only related reads+writes in a writer block, but also a read+read. In addition to being dx-friendly, the separation also allows us to prevent unintended mistakes and improve performance. In a reader block, we’d only enforce queueing, but forbid database writes and “sub-action” calling of another writers. Also, we could allow multiple reader blocks to operate concurrently, since we know that’s safe as long as no writes occur.

await database.read(async reader => {
  // performing read-only operations
  await table.fetch(...)

  // calling other readers:
  await reader.andRead(() => someAction())
  await reader.call(() => someAction())
  await reader.also(() => someAction())
  await reader.with(() => someAction())
})

We keep the decorator-based shortcut API for defining Model methods. Yeah, it pollutes the Model namespace, which I’m not a great fan of, but it makes Model methods (which should make up the vast majority of writers/readers in a idiomatic WatermelonDB codebase) so much cleaner, I think it’s worth it.

@writer async deleteComment() {
  await this.update(...)
  await this.batch(...)

  // calling other writers/readers
  await this.andWrite(() => this.togglePin())
  await this.callWriter(() => this.togglePin())
  await this.writeWith(() => this.togglePin())
}

@reader async getAllMembers() {
  // perform reads
  await collection.query(...)

  // calling other readers
  await this.andRead(() => this.getAllAdmins())
  await this.callReader(() => this.getAllAdmins())
  await this.readWith(() => this.getAllAdmins())
}

As you’ve noticed, the part I struggle with is coming up with a nice, reasonably self-explanatory, non-confusing, unambiguous name for calling readers/writers from readers/writers. All the alternatives considered:

  await writer.andThen(() => someOtherWriter()) // sounds like it's related to Promises...
  await writer.thenWrite(() => someOtherWriter() // ditto + sounds like it would be executed *after* current action
  await writer.andWrite(() => someOtherWriter())
  await writer.withWriter(() => someOtherWriter()) // withX, useX can create confusion for React users (HOCs/Hooks convention)
  await writer.useWriter(() => someOtherWriter())
  await writer.callWriter(() => someOtherWriter()) // seems most unambiguous?
  await writer.writeWith(() => someOtherWriter()) // also seems quite nice
  await writer.then(() => someOtherWriter()) // one-word methods might be confusing when defined on Model?
  await writer.call(() => someOtherWriter())
  await writer.also(() => someOtherWriter())
  await writer.with(() => someOtherWriter())
  await writer.use(() => someOtherWriter())
  await writer.sub(() => someOtherWriter())
  await writer.subAction(() => someOtherWriter()) // would be good to get rid of the "action" word once and for all
  await writer.subWriter(() => someOtherWriter())

I’d appreciate feedback especially on this part.

Deprecations & Removals

1. Legacy new Database({ actionsEnabled: false }) is no longer supported and an error
2. new Database({ actionsEnabled: true }) raises a deprecation warning (no longer necessary)
3. @action, database.action, action.subAction, Model.subAction are deprecated and will be removed later
4. (This may be controversial!) database.batch() is now deprecated, it’s expected that all writes happen in writers via writer.batch() (or Model.batch() when using @writer). This improves safety - it’s less likely that an illegal write triggered outside a writer is accepted (because a writer happens to be active), since you need a handle to call writer.batch() on an active writer.

[michalpopek]

@radex The proposal seems like a big improvement, I like it. As for the naming, how about writer.writeMore and (writer || reader).readMore?

[michalpopek]

I decided to stick to callWriter - it’s a bland, technical, but hopefully unambiguous name. I wasn’t sure how writeMore would be interpreted by a new user.

Sure, sounds like a reasonable decision. We could also be even more explicit and extend the name to callChildWriter/callChildReader or something in that spirit, but I’m not sure, if it is necessary.

The downside is that there’s another concept and the diagnostic warnings and documentation that have to keep repeating “writer or reader”.

IMO the reader API makes the whole read/write concept complete and rules out any potential doubts that could arise, if we just introduced the writer API alone. From what I saw in your implementation PR, the reader API is already marked as an advanced topic, so I don’t think it will bother newcomers that much. And the maintainance cost of being aware of the API when editing docs or writing warnings sounds relatively low for me.

We could just have Writers, and explain to the (very) advanced users that have a use case that requires consistent view of the data to wrap the reads in a Writer.

We could do that, but TBH it would feel like we’re serving our users workarounds instead of providing them with a proper solution.

OTOH, I already implemented it 😛

Yeah, and from what I saw it looks like the reader implementation isn’t extensive, so it shouldn’t create any serious debt