AB Testing on Node

How I approached solving AB testing on a node server; starting small and making it flexible.

The problem

We wanted to test 2 different appearances of a button, let’s call it the Filter Button.

How this was setup was the page which rendered the Filter Button was updated to switch between 2 different appearances based on a boolean prop passed down from the server.

Now all we needed was the server to decide whether or not to pass true or false into the page's prop, and thus trigger the switch between the 2 buttons.

The challenge

• Distribute the visitors evenly between the 2 choices.
• Ensure that when a visitor returns, they always see the same outcome.
• We get a true or false value to pass to our component.
• Be able to force the test to return a specific bucket’s outcome. Primarily so we can test the component without having it switch on us.
• Make this reusable.

The solution

I started by looking at existing solutions on NPM and after some initial searching, I found this package: ab-testing; which seemed simple enough in it’s approach to cover most of what I needed.

Although it wasn’t very well maintained, it achieved the basics of what I wanted, provide a configuration that allowed me to define a weight and place the user into a group. What it didn’t quite cover was how I wanted to use the outcome of the test, what this package did was allow you to define a function to be executed for either bucket A or bucket B; I needed this to return a value however.

After some deliberation, I decided to fork, update and re-publish it as a new package: red-blue-testing. The primary update I made was what the function returned. Previously it simply executed the function it was given depending on the bucket, but now the config requires a “outcome” which lets you define what gets returned from executing the test.

The usage is quite straight forward, first import the package:

const ABTesting = require('blue-red-testing');

Setup the config as a simple array of objects which provides 3 pieces of information for each test; name, weight, outcome:

const testConfig = [
  {
    name: 'filterAppearanceA',
    weight: 0.2,
    outcome: false,
  },
  {
    name: 'filterAppearanceB',
    weight: 0.8,
    outcome: { foo: 'bar' },
  },
]

Now we have some control over what comes back from the test, the world is our oyster! I can now safely receive a true or false value from my test to pass to my component check.

Next we initiate the test package with a name and the above config:

const testObject = ABTesting.createTest('MyFirstTest', testConfig)

Then provide the test package with a unique user ID.

const testGroup = testObject.getGroup('uniqueUserID')

This is where the magic happens, this getGroup function takes the uniqueUserID and places it in either bucket A or B, and returns the name of the bucket; in our case, it might return filterAppearanceA; As long as the same UserID is passed in, you will get the same outcome.

The final step is to pair the result of the group with the testConfig to get the outcome:

const result = testObject.test(testGroup, testConfig)

And you’re done! The result is one of your defined outcomes for that UserID you defined.

Node implementation

Now we switch to the node side implementation. To make this re-usable it needs to accept generic parameters and abstract away the setup for each individual visitor, automating the placement of each visitor, persisting the visitor placement and lastly letting a tester force a specific bucket.

Which bring me to this function I wrote for our Node Express server: red-blue-node-testing.

It starts by accepting a req and res (from the node server), and from there you can define as many tests as you want, passing in a name for the test and the config.

const createTest = aBTest(res, req);

const filterABTest = createTest(
    'filterMorebuttonTest',
    [
      {
            name: 'filterAppearanceA',
            weight: 0.2,
            outcome: true,
      },
      {
            name: 'filterAppearanceB',
            weight: 0.8,
            outcome: false,
      },
    ]
)

The rest is taken care of for you; the name of your test is prepended with AB_TEST_ to help with identifying your test cookie, it generates a random userID per visitor, stores it as a cookie for a repeat visit and you can set the browser cookie value to a bucket name to force the test to give you a specific bucket.

What get’s returned is an object with three keys:

• name: the full name of the test (ie. AB_TEST_filterMorebuttonTest)
• result: is an object with the name of the group the user was placed in (ie. filterAppearanceB), and the outcome of your test (ie. true)
• digitalData: contains the full name of your test as well as the bucket. (ie. AB_TEST_filterMorebuttonTest_filterAppearanceB). Primarily used for things like Google Analytics.

Last but not least, simply changing the cookie value to filterAppearanceB will force the testGroup to always return the outcome for filterAppearanceB.

The result is something I think is simple to setup on any component and the config is clear as to what you will receive back from the test.

Let me know what you think, I’d love to expand this to cover more use cases.