Event firehose

Related to #292 / #293.

Context

The end goal is to be able to send emails when one of the following conditions are true:

• A particular time has passed (eg; “an event is happening in 24hrs”)
• A database mutation occurred (eg; “an event has been created”)

This problem can be broken down into 2 halves:

1. Triggering an email send
2. Actually sending / queueing up emails

The act of sending the emails (2) can be handled by something like Zapier or ifttt for my usecase.

This issues is mostly about (1): Triggering events.

Triggers

The two triggers present different challenges, with different solutions

Mutation based

This one is mostly solved with pre/post hooks.

I think we need to expose those hooks at a list-config level, but the logic is already there for triggering them on various CRUD operations.

It then becomes a matter of making the correct call to Zapier / ifttt within a hook:

keystone.createList('Meetup', {
  fields: {
    when: { type: DateTime },
    name: { type: Text },
    info: { type: Text },
  },
  hooks: {
    postCreate: async meetup => {
      const whoToNotify = await getGroupSubscribersFromMeetup(meetup.id);
      sendToZapier('send-email', whoToNotify, `New meetup ${event.name}!!1!1one`);
    }
  }
});

Time based

This one is a bit trickier - time based triggers require some kind of state.

There’s a possibility that ifttt / Zapier support handling that state (ie; we tell Zapier when to trigger something we send it). However, that adds complexity around updates / cancellations of events.

Another option is to ditch the idea of time-based and say “Hey user; if you want to send emails to your members, you’ll have to set an alarm on your smart phone, then trigger an event” (See Bike Shedding for a method of triggering events via a write to an Events list, which can be done in the admin UI)

Bike Shedding

There may be a need to keep track of what events have been sent, so instead of sending to Zapier in the Event’s postCreate hook, we might setup a separate list like so:

keystone.createList('Meetup', {
  fields: {
    when: { type: DateTime },
    name: { type: Text },
    info: { type: Text },
  },
  hooks: {
    postCreate: async meetup => {
      const whoToNotify = await getGroupSubscribersFromEvent(meetup.id);
      keystone.mutation(`
        createEvent(data: { whoToNotify: ${whoToNotify}, title: "New event ${event.name}!!1!1one" }, type: 'send-email') {
          id
        }
      `);
    }
  }
});

keystone.createList('Event', {
  fields: {
    triggerTime: { type: DateTime },
    type: { type: Select, options: ['send-email'] },
    data: { type: JSON },
    triggered: { type: Checkbox },
  },
  hooks: {
    preCreate: async event => {
      // Only trigger it if it's in the past, or there's no event time set (ie; immediate)
      if (!event.triggerTime || event.triggerTime < now()) {
        sendToZapier(event.type, event.data);
        event.triggered = true;
      }
      return event;
    }
  }
});

Then for time-based ones, we could run an independent cron job every 1s, that looks for unsent events:

cron.on('trigger', () => {
  const pendingEvents = await keystone.query(`
    allEvents(where: { triggered_is: false, triggerTime_lt: "${Date.now().toUTCString()}" }) {
      id
      type
      data
    }
  `);
  pendingEvents.forEach(event => {
    keystone.mutation(`
      updateEvent(where: { id: "${event.id}" }, data: { triggered: true }) {
        id
      }
    `);
    sendToZapier(event.type, event.data);
  });
});

[jesstelford]

Now that our list & field hooks are much more fleshed out, that gives me all the extension points I need to trigger events.

As for time-based events, that’s still on my todo list. However, I no long expect Keystone to handle this for me. Instead, I might look to something like BullMQ or Bee-Queue instead.

[MadeByMike]

@jess when you get a chance could you put some eyes over this issue: https://github.com/keystonejs/keystone/issues/419 is this still relevant as something we want to add to Keystone? I feel like maybe it’s better people BYO their own library for this. Perhaps you have patterns now from Cete that ‘solve’ this.