Future of the DataStore API

Description

The dataStore API in this SDK is great at aggregating a model that applications can count on for state, as if the application was connected as any Slack client would be. It’s super convenient.

But it has some problems.

1. The SlackDataStore interface is cumbersome to implement. There’s very little to suggest that any developer use anything besides the default SlackMemoryDataStore implementation that ships with the SDK (an exception: #246). Potentially, very simple CRUD operations are all that should be needed.

2. It’s only useful for apps that use an RTM client. In general (but not 100% of the time) apps would work better, faster, and more secure if they used a combination of the Events API and the Web API. So essentially the dataStore is built for a very small and specific population of this SDK’s users, but all apps using this SDK are paying the cost in terms of memory and performance.

3. The SlackDataStore interface is not asynchronous. This means either the data must be in memory or you must block the node event loop to use it. Both bad options for apps that need scale. In #246 a developer asks for a Promise-based async interface.

4. On Enterprise workspaces and workspaces with shared channels, you cannot depend on the dataStore to know about all users. This is because the client is connected to one workspace at a time, and it might see events in shared channels or on files that do not originate from users in that workspace (see: https://github.com/slackapi/hubot-slack/issues/404).

5. On Enterprise workspaces and workspaces with shared channels, its difficult (maybe impossible) to de-duplicate events to consistently update the state in the data store. Slack has added an event_id property (when they are delivered using the Events API) that could be used to determine duplication, but as stated in point 2, the data store is mostly used with RTM. Besides, the Events API is already deduplicated.

This ticket is meant to be a home to discussion around what should be done about this next. I’ve got a few options I’d like to lay out now:

Plan A Deprecate the current dataStore API, and replace it with something better in the next major version of this SDK. We could get an easier to implement interface, with async operations out of this. At a minimum we should provide a migration guide for current users to help them move to the next major version.

Plan B Make some larger sweeping architectural changes that break apart the Web client, RTM client, and the Incoming Webhook parts of this SDK into more modular packages. Leave this package around as-is. Provide a new umbrella package that helps you avoid all the wiring up of the packages as an 80/20 solution with sane defaults. Try to get the community to migrate to the new umbrella package and/or the modular packages its composed from.

IMHO, I’d like to see Plan B happen, because I think it’s more aligned with the Node Way™. We already have a decent start with the Events API module. I do want to acknowledge that it is more development work and it’s no easy feat to move the community from this package to a new alternative.

• I’ve read and understood the Contributing guidelines and have done my best effort to follow them.
• I’ve read and agree to the Code of Conduct.
• I’ve searched for any related issues and avoided creating a duplicate issue.

[aoberoi]

@Everesty thanks so much for your thorough and well structured thoughts. i tend to agree with what you’ve said.

my biggest fear with Plan B is that we will build it and they won’t come. but i’m optimistic because users get so many advantages beyond getting a better Data Store API from Plan B. also, since Slack is involved here, there’s a decent chance that we can get the community to migrate: we can add links to the official documentation site (api.slack.com) to drive new users to the preferred package, we can hopefully get more people to write tutorials and blog posts that use those modules, and maybe since the lower level modules will mostly be utilities and opinion-less, we can get 3rd party tools to build on top and depend on them.

this discussion is definitely still open to more opinions and i can be persuaded otherwise.

---

my thoughts on some of the points you raised:

• migrating to ES6: the @slack/events-api module is already written in ES6. i already live in an ES6 world, so i’m not going to start new packages that don’t hit that minimum bar. that being said, until node core transitions the ES Modules EPS to ACCEPTED status, and even for some time after, we’ll need to continue to transpile down to ES5. i’d like to adopt the minimum target of node 4.x across all newer modules, and if your app has to run on something less, then this module might be the answer for you. there’s actually one more build change i’m considering and that’s authoring in TypeScript. its become apparent to me that the community wants types and doing them out of band of the authoring of the package is a pain from a maintenance point of view, but we might need to save that for another discussion.
• what do we call it? how about @slack/sdk? also, @brianleroux owns the slack package on npm and has previously offered for us to use it for another purpose, so that could be possible. if we use a new name, we’ll start the version number over.
• what’s being split? here’s a hypothetical list:
  • RTM client (potentially use slack implementation, which i think is great)
  • Web client (potentially use slack implementation, which i think is great)
  • Events API
  • Interactive Messages (TBD, but similar to Events API)
  • Incoming Webhook (for migratability, could prob just use code from here)
  • Slash Command + Outgoing Webhook (prob a ~30 line helper, plugs in like Events API)
  • OAuth helpers (@aoberoi/passport-slack can be cleaned up and reused)
  • Slack Store (inspired by express-session with pluggable backends and options to turn on/off certain models, behave as a team cache versus an org cache)
  • Message helpers (similar to a templating library specific to outputting JSON suitable for Slack messages)
• where do docs live? i think each module needs reference docs and a good README at a minimum. the umbrella module should get a little more in depth treatment with guides for different kinds of usage (similar to the gh-pages site for this module). the umbrella module could also have a listing of its individual deps and a short description about what they are used for.

[aoberoi]

it seems appropriate to close this issue and redirect any conversation about a new API into #410.