When considering how to setup a dynamic experiment we propose 5 levels of testing to ensure you are deploying optimal recommenders to production. These are:
More of a classical measure, than dynamic, this is where you would look at engagement recall and accuracy.
This is where you can configure different parameters, input your data, or link to a feature store and see if the prediction results make sense to you. If not, this phase gives you the opportunity to delve deeper and figure out what’s going on, or if there’s a mistake somewhere that is impacting results.
Simulation and Dynamic Engagement
Simulations are immensely useful, they’re the first step where you can validate that you’re doing the right thing. They’re not only useful for you to interpret results, they’re also essential to figure if the performance, security, and downstream execution is all in place.
New to Production
This phase is where you launch your experiment, whether that’s a campaign, new recommender, intervention, or a product offer into production. The real time capability lets you track so many different criteria using dashboards and other tracking tools.
Now that the simulation and move to production phases are complete, this is where you continuously track the recommender you’re deploying or alter the behavior of the interaction you’re testing. This approach happens in production, without ever having to go back into your pipeline.