A/B Testing Known Issues

We've compiled a list of known issues and troubleshooting to help you configure your A/B tests as desired.

"Visitor" Metric Incremented Twice Upon Logout

Sometimes a visitor metric can be counted twice. For example, an A/B test has Everyone as the assigned customer group. If a registered customer logs out after placing an order, and then places another order as a guest, the visitor metric is counted twice. This customer is counted twice even though it's the same customer using the same PC in the same session. When a customer performs a guest checkout only, the customer is counted once, no matter how many orders the customer makes. They’re only counted once because when a registered customer logs out, B2C Commerce increments the visits and visitors metrics as if a new, anonymous customer has come to the site. This anonymous customer is considered another visitor and is again evaluated for A/B test participation. If the customer qualifies for an A/B test, B2C Commerce generates relevant data as part of the customer's participation.

This could cause a problem with the metrics because the registered customer is counted twice and a guest checkout is counted once. When collecting A/B testing metrics, B2C Commerce doesn’t know how many customers are registered or how many of users could result in a guest checkout.

Download of A/B Test Results Displays Security Warning in IE8

If you download A/B tests into a .csv file for the first time using an IE8 browser, a security message appears. The message informs you that the download has been blocked. After you confirm the message, the A/B test list page appears and the file isn't downloaded. If you try a second time, the download begins. If you leave the A/B test module and return, the security warning reappears.

This error only occurs with Microsoft® IE8.

A/B Test Limitations Enforced by Quotas

Quotas enforce the following limitations:

• B2C Commerce allows a flexible distribution of test segments across a maximum of 12 A/B tests.
• Maximum of five test segments within a test, including the control group segment, so four user-defined segments total.

Anonymous Customer Participates in A/B Test for Registered Customers

An anonymous customer can participate in an A/B test configured for registered customers only in the following scenario. A registered customer participates in an A/B test that has expiration set to Never and is configured for registered customers. The registered customer leaves or logs out of the site and the personalization cookie still exists in the respective system. Now, an anonymous customer can participate in the test.

This is to help produce a consistent storefront experience for the A/B test at the expense of potential false positives. When the cookie was set in a previous visit for someone using the same browser, we assume it was also set for the current browser user.

Participation in A/B Test for Registered Customers Might Happen Rarely

B2C Commerce uses a round-robin assignment of participants to tests. For example, an A/B test is configured for Everyone. A second A/B test configured for Registered Customers only has participants if those customers are remembered and the round-robin mechanism has chosen that test.

For example:

Potential scenarios:

• A customer starts a new session. The customer typically starts as Anonymous. so A/B test B doesn't apply. A/B test A is selected every time because all allowed A/B tests are looped until one matches, and B never matches.
• A customer logs in later. Participation is always in progress. Therefore, A/B test B isn't assigned.
• A customer logs in with the Remember Me flag set to true. They return later to the store and are remembered. A/B test B is next in the round-robin sequence. Only then is the customer selected for participation in A/B test B.

We recommend that you run one test at a time for most participants. Running one test at a time can shorten the time to statistical significance; and no overlap, to prevent skewing the results.

Cannot Add a Campaign or A/B Test Schedule to a Content Slot

Users occasionally get confused about why they can't configure a campaign within the Business Manager Content Slot module. This is by design because users who have permission to access the Content Slot module don’t always have permission to access the Campaigns module. You can only create campaigns and A/B tests if you have permission to access the Business Manager Campaigns module.