Proposal: frint-experiments for A/B testing and more...

(Intended to be done in a separate repositiory)

What

Guidelines and API spec on how we can run experiments in the UI with FrintJS.

Extracted our usage out of the internal PoC we did recently.

/cc @AlexDudar, @juliamaksimchik, @discosultan, @viacheslaff

Proposed packages

frint-experiments

Will export:

• ExperimentsService
  • will be set as a provider in Apps
  • will provide access to the collection
• ExperimentsCollection: will contain the experiments data
• Experiment: Individual experiment item as a model
• createExperiments: Generates a new configured ExperimentsService class

Example usage:

import { createApp } from 'frint';
import { createExperiments } from 'frint-experiments';

const App = createApp({
  name: 'MyApp',
  providers: [
    {
      name: 'experiments',
      useFactory() {
        const ExperimentsService = createExperiments({
          activate(model, attributes = {}) {
            // implement activation logic
          },

          track(model, eventTags = {}) {
            // implement tracking logic
          },
        });

        return new ExperimentsService([]); // feed it with Experiments collection
      }
    }
  ]
});

The model can have this schema:

// Experiment.js
import { createModel, Types } from 'frint-data';

export default createModel({
  schema: {
    name: Types.string,
    variant: Types.string,
  },
});

The service:

class ExperimentsService {
  constructor(experimentsArray) {
    this.experiments = new ExperimentsCollection(experimentsArray);
  }

  findByName(name) {
    return Experiment;
  }

  findByName$(name) {
    return Observable;
  }

  activate(model, attributes = {}) {
  
  }
  
  track(model, eventTags = {}) {

  }
}

frint-experiments-react

This package will be exporting React components, for implementing our experiments at UI-level.

import React from 'react';
import { Experiment, Variant } from 'frint-experiments-react';

export default function MyComponent(props) {
  return (
    <div>
      <Experiment name="my-experiment-name">
        <Variant when="b" render={() => 'foo'} />
        <Variant when="c" render={() => 'bar'} />
        <Variant render={() => 'default'} />
      </Experiment>
    </div>
  );
}

Experiment component will receive either a name (String) or model (Experiment) as prop.

Benefits

• Declarative experiments
• Always defined at code-level
• No external manipulation with JS code
• Can go for aggressive code-splitting at Component-level, so that users only download the code they need for their own experiments only

[viacheslaff]

One small addition. When talking of A/B testing, as far as I know, variant A represents what we already have. So not using <Variant when="a" render={() => 'foo'} /> will make more sense:

<Experiment model={experiment}>
  <Variant when="b" render={() => 'bar'} />
  <Variant render={() => 'default'} />
</Experiment>

Then from b on, more variants can be added if needed (c, d, etc.)

[fahad19]

Thanks for your questions, @asci! I am responding to each one below:

---

Is it possible to use the model without React components? For example, you need to make a request or to do some calculations for some case, not just render?

Yes. In FrintJS, rendering is an optional thing to do. You first define Apps, that can later be optionally rendered on demand.

The instance of ExperimentsService class will be set as a provider in a FrintJS App, which means you can access it, along with the Collection/Models by doing:

app.get('experiments'); // the ExperimentsService instance

---

Any thought about styles and ab\testing? Usually, the test requires some style changes as well. Would be nice to have everything for AB testing under one project with some guides.

Could you please give me an example scenario? Would help me understand and answer better.

If using CSS Modules, why not just render different Components that can have their own variations in styling?

---

Not clear about activation — when we get data for tests (model content) we can perform activation at that moment (or even do at server time). OR there is no way to split users by experiments\pages at Google Analytics? Also, how it would work during SSR?

Activation/Tracking would require more thoughts. In this proposal, I am limiting the activation/tracking part by defining them in the ExperimentsService interface only.

Based on a particular application’s requirements in a given environment (client OR server), we can always decide what to do with the incoming experiment Model.

AFAIU it is against Frint to use context types and top-level providers (thinking about Apollo)? If I’m wrong, you can omit the whole part about getting experiment prop.

FrintJS Apps have Providers, and when the Apps are rendered with frint-react, the App instance is made available in the React’s context.

The higher-order component then deals with React’s context API and gives us access to the app instance:

import { observe } from 'frint-react';

function MyComponent(props) { 

}

export default observe(function (app) {
  // make props compatible object here
})(MyComponent);

If we really want to enable developers to write less code, we can additionally support a name prop in the <Experiment> Component:

import { Experiment, Variant } from 'frint-experiments-react';

export default function MyComponent(props) {
  return (
    <div>
      <Experiment name="my-experiment-name">
        <Variant when="b" render={() => 'b'} />
        <Variant when="c" render={() => 'c'} />
        <Variant render={() => 'default'} />
      </Experiment>
    </div>
  );
}

The Experiment component’s behaviour would then be:

• If model prop is available, use that Experiment model
• If name prop is available, get the model from ExperimentService, and then use that