A/B Testing
Using Business Manager, you can assign promotions or slot content to an A/B test with common qualifiers such as source codes or customer groups. A/B test statistics use a combination of two-sided t-tests and chi-square tests to weigh test segments against each other and compute the statistical significance.
You can create and manage A/B tests, view test results, and change your site based on those results. You can configure changes to your sites that affect only a subset of visitors so that you can determine from collected data the experience that produces the best results.
You create test segment, where each segment configures a unique experience for a set of participants, against which you can test other segments and a control segment. You can define triggers that a customer must access to be considered for participation in a particular test.
When creating A/B tests, understand that, relative to other promotions available in Salesforce B2C Commerce, A/B tests are considered to have high precedence.
You can also customize A/B tests using the B2C Commerce API, and use the A/B test import/export functionality to manage A/B tests across your company. You can extract A/B test results in .csv output files, and use those files in your favorite spreadsheet application for further analysis.
A/B Testing vs. Personalization
Although the effect is similar, A/B testing segmentation is different from personalization segmentation. The regular personalization of a site, for example, done using customer groups, is segmented by criteria that you can define. Meanwhile, A/B test segmentation is executed randomly. With both, the user is presented with an experience that is potentially different than other users. A/B testing, however, goes beyond personalization. A/B testing collects metrics to help determine which experience, among two or more, performs the best according to one or more segmentation criteria.
A/B Testing Eligibility
A shopper’s eligibility for A/B tests is determined by the Tracking site preference. The Tracking site preference is session-specific and only shoppers for whom tracking is allowed are eligible for A/B tests.
To set this preference select Merchant Tools > Site Preferences > Privacy.
If a shopper opts in or out of tracking mid session, the A/B Test Metrics show a slight inconsistency. Changing tracking mid session can also cause a shopper's page to appear as though it's still tracking. Refreshing the page fixes this issue.- Eligible Registered–The mapping between the A/B test group assigned to the shopper's customer ID are stored in the database. As a result, the shopper's A/B test assignments are recalled, and therefore the shopping experience is consistent regardless of which device the shopper uses.
- Eligible Guest–The A/B test group that the shopper is assigned to is written to a cookie on the shopper's browser. As a result, the shopper's A/B test assignments are recalled. The shopping experience is consistent only when the shopper is on the same device and using the same browser.
Eligibility and A/B Test Sample Size
The consent requirement for A/B test eligibility and the possibility of a varied A/B test experience for unregistered shoppers can impact your test sample size. Consider planning longer test durations or altering your test criteria to capture statistically significant A/B test date.
A/B Test Data Deletion
Shopper data deletion requests cascade to A/B test participant records. When a shopper requests to have their records deleted from the system, all the shopper's A/B test participant records are deleted.