Seasonality is the thought that users’ behavior may differ based on the day of the week or the time of the year. It is a critical part of A/B testing initiatives to account for seasonality when defining the duration of your experiment and to implement it into the design and planning. For example, when A/B testing an e-commerce platform, it would be wise to run the experiment for a number of weeks as consumers are more likely to do online shopping on the weekend.
Using A/B Testing to Run a Tight Ship
At the moment, DevOps teams may be looking into changing code or infrastructure to increase efficiency in an effort to either handle increased demand or to reduce infrastructure spend. Choosing to place changes behind a feature flag split while adding events and metrics to measure performance effects means that you can gain clear data on whether or not the changes negatively affect customer experience. This also allows you to roll out new features to a small, select amount of users and monitor the effects with the ability to roll the feature back without negatively affecting the entire customer base.
With the way the world is reacting to the COVID-19 pandemic, constant dramatic changes for businesses are the new normal, and because of this, other tests (aside from A/B) should be considered, as well as altering the measurement of tests. These could help you get a better grasp on customer behavior. Some suggestions include:
- New Customers – Some industries have seen a major spike due the boom in remote work, while others have seen drastic declines due to people staying at home. If your business is one of the lucky ones that have seen an increase of traffic during this time, then you should maintain the date on which a user registered. You can then use that value either as a targeting attribute to target users who registered after a given date or as an event property.
- A/A Test – It is a good guess that past metric baselines no longer represent the current state of things. Therefore you should consider running A/A testing. This is the process of using A/B testing to test two identical versions against each other. Running one for a week before pulling new metric baselines should be sufficient.
- Existing Metrics – Enhancing existing metrics would greatly help to understand the implications of COVID-19 on the usage of your application. For example, if you currently use a simple ‘Items Added to Cart Per User’ metric, you may want to add a product category to see which products are experiencing an uptake or a decrease during this time. Enhancing existing metrics can also bring to light any outliers that could be skewing metrics in unexpected ways. Creating a capped version of the metric and comparing it to the uncapped version can help you understand the effect these outliers are having.
It is best to continue running A/B testing during this unpredictable time, but ensure that you are fully aware that the environment in which that test is running is not representative of what you might usually see or have seen a few weeks before. However, this doesn’t mean you can’t still learn from these results. It can teach you how your company will be affected in a time of crisis. While this will pass, the current climate we are in proves that change is constant and therefore we should always prepare for it.