As an industry professional, you're eligible to receive a printed copy of the journal.

Fill out your address below.

Please reset your password to access the new
Reset my password
Welcome aboard, !
You're all set. We've send you an email confirmation to
just to confirm you're you.

Welcome to DRJ

Already registered user? Please login here

Existing Users Log In

Create new account
(it's completely free). Subscribe

COVID-19 is causing a larger shift in many businesses that typical high seasons such as Christmas. While we can learn a lot of the data recorded during this time, seasonality should especially be taken into account during these unprecedented times as a result of COVID-19. However, it is difficult to get clear results due to the unpredictable actions of everyone who is spending more time in their homes, and therefore more time online.

Seasonality is the thought that users’ behavior may differ based on the day of the week or the time of the year. It is a critical part of A/B testing initiatives to account for seasonality when defining the duration of your experiment and to implement it into the design and planning. For example, when A/B testing an e-commerce platform, it would be wise to run the experiment for a number of weeks as consumers are more likely to do online shopping on the weekend.

Using A/B Testing to Run a Tight Ship

At the moment, DevOps teams may be looking into changing code or infrastructure to increase efficiency in an effort to either handle increased demand or to reduce infrastructure spend. Choosing to place changes behind a feature flag split while adding events and metrics to measure performance effects means that you can gain clear data on whether or not the changes negatively affect customer experience. This also allows you to roll out new features to a small, select amount of users and monitor the effects with the ability to roll the feature back without negatively affecting the entire customer base.

With the way the world is reacting to the COVID-19 pandemic, constant dramatic changes for businesses are the new normal, and because of this, other tests (aside from A/B) should be considered, as well as altering the measurement of tests. These could help you get a better grasp on customer behavior. Some suggestions include:

  • New Customers – Some industries have seen a major spike due the boom in remote work, while others have seen drastic declines due to people staying at home. If your business is one of the lucky ones that have seen an increase of traffic during this time, then you should maintain the date on which a user registered. You can then use that value either as a targeting attribute to target users who registered after a given date or as an event property.
  • A/A Test – It is a good guess that past metric baselines no longer represent the current state of things. Therefore you should consider running A/A testing. This is the process of using A/B testing to test two identical versions against each other. Running one for a week before pulling new metric baselines should be sufficient.
  • Existing Metrics – Enhancing existing metrics would greatly help to understand the implications of COVID-19 on the usage of your application. For example, if you currently use a simple ‘Items Added to Cart Per User’ metric, you may want to add a product category to see which products are experiencing an uptake or a decrease during this time. Enhancing existing metrics can also bring to light any outliers that could be skewing metrics in unexpected ways. Creating a capped version of the metric and comparing it to the uncapped version can help you understand the effect these outliers are having.
See the Opportunity

It is best to continue running A/B testing during this unpredictable time, but ensure that you are fully aware that the environment in which that test is running is not representative of what you might usually see or have seen a few weeks before. However, this doesn’t mean you can’t still learn from these results. It can teach you how your company will be affected in a time of crisis. While this will pass, the current climate we are in proves that change is constant and therefore we should always prepare for it.


Channing Benson

Channing Benson is a experimentation advisor for Split where he works with our customers’ product managers and developers to develop and refine their experimentation efforts in order to iterate quickly and safely. Benson is also a senior technical consultant for AppDynamics.

Backup Software’s New Countermeasures to Detect, Protect, and Recover from Ransomware
Scarcely a month goes by without some organization publicly reporting its need to recover from a ransomware attack. This continuing...
Jumpstart your Disaster Recovery and Remote Work Strategy: 6 Considerations for your Virtual Desktops
White paper sponsored by teradici Virtualized desktop infrastructure (VDI) has become a strategic choice for many IT organizations because it...
Object Storage and Continuous Data Protection – An Integrated/ Decoupled Approach
DR Object Stores Evolve Beyond the “Cold Data Tier” The cost-efficient scalability of object storage makes it an ideal resource...
Stadiums Making a Play for Better Digital Infrastructure
There is no doubt that modern and emerging technologies have drastically changed how fans experience live events. Fans want to...