Proposal: Remove ORM/Collection, introduce this.serialize

There have been several discussions in the past few weeks pointing to some shortcomings in the current 0.2 API for models and serializers. In summary,

• ORM/Collection carries type information used by the Serializer layer, but because Collections are array-like, users reasonably expect to be able to use methods from JS, Lodash and Ramda to manipulate them. However, doing so escapes the Collection API completely, dropping type information and breaking the serialization process.
• Users need a way to munge data on serialized JSON since Mirage doesn’t have first-class APIs for things like links, sorting, filtering, and adding metadata.

I’m proposing the following changes as a remedy.

1. Add this.serialize to route handlers

Currently, data that’s returned from a route handler passes through the serializer layer. If you return an ORM Model or Collection, it will get serialized:

this.get('/users', (schema) => {
  return schema.user.all();  
});

In this example, Mirage will recognize a collection is being returned and attempt to serialize each user. To do this, it will first look for a model-specific serializer (which it can infer from the model’s type) in /mirage/serializers/user.js, then fall back to the application serializer in /mirage/serializers/application.js, then finally fall back to the default serializer.

If a POJO or POJA is returned

this.get('/users', () => {
  return [
    {id: 1, name: 'Person 1'},
    {id: 2, name: 'Person 2'}
  ];
});

this data still technically passes through the serializer layer, but it is ignored and returned as is.

The proposal is to add an explicit this.serialize method to route handlers. The signature would be

this.serialize(data, type, options)

This allows for a few things.

First, users can explicitly choose to serialize POJOs/POJAs. This is nice because for a variety of reasons it’s sometimes easier to just return raw JS in a route handler, instead of a Model or Collection. Now, when doing so, users can take advantage of the Serializer class and its various formatting hooks:

this.get('/users/current', () => {
  let session = {id: 'current'};

  return this.serialize(session, 'session');
});

In this example, the POJO {id: 'current'} would pass through the SessionSerializer, perhaps an extension of JSONAPISerializer, and would end up looking something like

{
  data: {
    type: 'sessions',
    id: 'current',
    attributes: {}
  }
}

Note that the type information was explicitly declared, since it could not be inferred from the plain object.

Second, this.serialize provides a way for users to do some final data munging in their route handlers. For example, there is no current API for dealing with sorting, filtering or metadata. I initially told users to use the this.serializerOrRegistry.serialize API

this.get('/users', (schema, request) => {
  let users = schema.user.all(); 

  let json = this.serializerOrRegistry.serialize(users, request);

  // add meta, sort, filter...
  json.meta = {};

  return json;
});

but that’s private.

Note: this.serializerOrRegistry.serialize(users, request), the current implementation of serializing a route handler response, involves passing through the request object. As this never changes, I don’t want users to have to do this in the new API. I will set the request as an instance variable on the handler and pass it through automatically.

Then, I recommended users override serialize in their Serializer, call super, and do munging there:

// serializers/user.js
export default Serialize.extend({

  serialize(object, request) {
    let json = Serializer.prototype.serialize.apply(this, arguments);

    // additional munging

    return json;
  }

});

which is okay, but sometimes users have logic already available in route handlers that they now need to pass to the serializer, perhaps by attaching arbitrary data to the request object. So, adding this.serialize to route handlers gives a single place for this kind of logic:

this.get('/users', (schema, request) => {
  let users = schema.user.all(); 
  let json = this.serialize(users);

  // sort, add metadata...
  json.meta = {};

  return json;
});

Note that because users is a Collection, type can still be inferred. So the type param to this.serialize is optional.

Finally, adding this.serialize to route handlers allows us to…

2. Remove the ORM/Collection class and change the default serializer behavior

One issue that keeps coming up is, users escape the Collection API.

Initially, ORM/Collection was modeled after Rails’ CollectionProxy. The idea was, we needed a class for arrays of models, so that we could pass along type information that could be accessed by the serializer, or other pieces in Mirage (e.g. factories, eventually). This is why various APIs like schema.blogPost.all() or user.blogPosts return a Collection with type blog-post.

The problem is, users often need to perform array operations in route handlers, and this makes it very easy to escape the Collection API and lose type information. Collection is array-like, so users often want to do things like

let users = schema.user.all().filter(u => u.isAdmin);

Initiailly, filter returned a plain array rather than a Collection, since it was just the native JS filter. We then tried to combat this by adding a filter method (along with other Array methods) directly to Collection, where those methods would operate on the underlying array but return a new Collection that still carried the type information. But this turns out to be quite difficult. For example, given the following code

let authors = schema.blogPosts.map(b => b.author);

Mirage has no way of knowing what the type of the new Collection should be without inspecting the contents of the new array returned by #map.

Additionally, JavaScript developers often use libraries like Lodash or Ramda to manipulate arrays. Since Collection is Array-like, but not a true subclass of Array (which is very difficult/impossible to do in JS), often these library methods would not work on the Collection, which lead to inconsistent, confusing and frustrating experiences for users.

We tried adding new APIs like Collection.where in an attempt to keep users within the Collection API, but for all the previously mentioned reasons users needed to be able to use a variety of array methods. I think one main reason this is a problem for us but not for Rails is, JS developers are just much more used to manipulating arrays in their own way, and if they’re working with something that looks like an Array, they (reasonably) expect to be able to manipulate it using methods that are familiar to them.

For these reasons, I’m proposing that we remove the ORM/Collection class altogether. This has a few implications.

First, it means that any other code throughout the system that works with Collection will now be working with plain arrays of Model objects instead. This means that any calling code will no logner be able to get type information via Collection.type.

I think the general solution here is for calling code to inspect the contents of returned arrays. If an array is a homogenous array of models, e.g. if each element in the array is an instanceof Model, and each model has the same model.type, the calling code can assume the array represents a Collection with type type.

As a specific example, serializers currently use Collection.type to look up model-specific serializers. If we drop the Collection class, the serializer layer could instead inspect any array that was returned, and if it’s a homogenous array of models, it could treat that array exactly as it treats a typed Collection today.

One big caveat here is empty Collections. Currently,

return schema.user.all();

returns a typed Collection, even if there are no users in the database. This allows the serializer layer to still end up with something like

{
  users: []
}

which is important.

The workaround is for users to use the new this.serialize API to explictly specify type:

this.get('/users', (schema) => {
  let users = schema.user.all();

  return this.serialize(users, 'user');
});

This would allow empty arrays to still be serialized with the correct type.

The fact that arrays, especially after manipulation via map, filter and the like, could always turn out to be empty arrays, means that technically speaking, users would always need to call this.serialize and pass in a type to guard against the empty array case. In practice, since it probably doesn’t happen for many route handlers, we could simply throw a warning or even an error whenever a route handler actually returns an empty array. At that point, we’d tell the user that a particular handler returned an empty array and that they need to go in and use this.serialize to make sure Mirage knows the type.

This is simply an ergonomic decision, meaning that we’d still encourage users to

this.get('/users', (schema) => {
  return schema.user.all();
});

and only after they run into an empty array case would we tell them to go back and add the explicit call to this.serialize.

Note that we can make the shorthands smart enough so that they do this every time. So shorthands will still work, even if no models exist for a given type.

The final implication of dropping Collection is that, like the serializer layer, any future code within Mirage that will need to work with Collections must replicate this behavior of inspecting arrays, as well as provide APIs for users to explicitly declare type. The only area I can currently think of that is planning to use Collections is the upcoming factory relationships layer, but I think the type information will always be known, since it will be taken from the keys of the factory.

We should try to think ahead and see if there are any situations where not having a Collection class is going to cause us serious problems.

Finally, given that Collection would be replaced with plain arrays, this leads to some updates to the schema methods that currently return Collections…

3. Updates to schema.modelClass

Currently, given a blogPost model, schema.blogPost is how users interact with all the blogPost models in the ORM. Users get all blog posts via

schema.blogPost.all()

which returns a typed Collection. With the removal of Collection, this method would now return a plain array of blog-post models.

The singular name (schema.blogPost) and .all() method were inspired by Rails. It may not make sense to keep these conventions anymore, given the removal of Collection.

Instead, I propose making a few changes to the ORM model class.

• Rename schema.modelClass to schema.modelClasses (pluralize).

• Drop #all. We can mimic the database’s API here, and get rid of modelClass.all() completely. Along with pluralizing the model class, this means

  let posts = schema.blogPost.all();
  

  becomes

  let posts = schema.blogPosts;
  

  which again is an array of blogPost models.

• Remove #where from schema.modelClasses. Now that schema.modelClasses returns a true array, we can drop #where from the API, since we no longer are passing around type information, and we already have plenty of ways to filter arrays in JS.

• Keep a few additional helper methods, but shy away from adding any array-like helper methods. Even though schema.modelClasses returns a true JS array, we can still attach helper methods to that array, just like we do with db today.

  For example, today users can do db.blogPosts.insert(...) but db.blogPosts itself is still a true array. Similarly, schema.blogPosts will be a true array (of model objects), but it will have helpers like

  • schema.blogPosts.new
  • schema.blogPosts.create
  • schema.blogPosts.find
  • schema.blogPosts.first

  In general, we will shy away from adding future helpers to this API that aid in array manipulation, instead encouraging users to use JS and libraries like Lodash and Ramda.

  Again this is all possible because there’s no longer a Collection API to “escape.” And, I think this will be clear to users, just as it is today with the db api. I believe it’s clear that once users start manipulating db collections

  db.blogPosts.filter(post => post.publishedAt > '2015-01-01')
  

  they don’t expect to be able to then call the collection methods insert, remove etc. on the resulting array. They’re just left with a plain array that they created, and they understand that.

  I think the find API (e.g. schema.blogPosts.find(1)) still makes sense to include in Mirage since it’s so common, whereas where doesn’t (we have Array.filter).

Questions/concerns

• Can anyone think of problems with the this.serialize(data, type, options) API? One alternative I can think of is making this a class instead, perhaps expanding Mirage.Respnose to something like

  return new Response(data, headers, code, {
    serializer: 'type',
    meta: {}
  });
  

  but I think let json = this.serialize() feels a bit more natural.

• In general I don’t want people to be doing a lot of let json = this.serialize and then munging and returning, unless they have to. This is about separation of concerns and the serializer layer should really contain all serialization logic. Would a different API that nudges users towards serializer layer be better, perhaps for things like sorting/filtering? Or is this fine as an escape hatch for now?

• One kind of response that these changes don’t address is a response like the one @ef4 mentioned here:

  this.post('/login', (schema, request) => {
    return {
      user: schema.user.create(),
      access_token: '123123'
    };
  });
  

  Given my propsal above the serialize layer would recognize the top-level object as a POJO and ignore it, by default. One option would be to tell users to write this like

  this.post('/login', (schema, request) => {
    return {
      user: this.serialize(schema.user.create()),
      access_token: '123123'
    };
  });
  

  The other is to make the serializer layer smart enough to crawl POJOs and look for model instances, but this seems hairy. Thoughts?

---

Would love to hear any and all feedback on this! I think once we nail these changes down, we’ll knock out most of the outstanding issues for 0.2 and will be really close to releasing.

/cc @mitchlloyd @ef4

Affected issues

• #623
• #596
• #595
• #591
• #486
• #367
• #588

[samselikoff]

Excellent. I’m going to move forward with

• adding this.serialize
• adding toJSON to Mirage.Model
• removing ORM/Collection
• updating schema with the originally proposed API updates (schema.posts, etc.)

[samselikoff]

Donezo: https://github.com/samselikoff/ember-cli-mirage/pull/705

Those were the last planned breaking changes for the 0.2 api - it’s about time we release!