How do you get the conversation rolling at a party filled with web optimization specialists? (And who wouldn’t want to attend such a party?) Naturally, you throw this question to the crowd: Google Optimize or Optimizely?
When comparing A/B testing tools, focus on the features that are most important to your experimentation team and how they may differ between each tool. Optimize and Optimizely are both solid choices for experimentation and personalization tools, though each has unique capabilities and best use cases. This post will cover how each tool handles important elements of building tests, with a focus on:
- Implementation
- Creating variants
- Test objectives
- Methodology and results
- Integrations
Implementation
Google Optimize
Adding Google Optimize to your website requires a snippet of code in order to display experiments and personalizations. Here is an example of what that code looks like if you are using the Global Site Tag with Google Analytics.
Alternatively, you can implement Optimize through Google Tag Manager utilizing the built-in template tag.
Google also recommends implementing an optional anti-flicker snippet to avoid the infamous A/B testing flicker that sometimes occurs with many tools. This flickering occurs when there is a moment when the original content is visible while the variant loads. To prevent this from happening (or you notice it happening after installing Optimize), simply add the additional script to temporarily hide the page while the Optimize container fully loads.
Optimizely
Optimizely also requires a snippet of code added to the source for installation. This code should be added as high up in the head tag as possible on all pages of your site. The snippet may look something like this:
Although it is possible to load Optimizely with a tag management system, it is not recommended; using a TMS can cause flicker issues. There is no additional script provided by Optimizely to mitigate possible flicker issues.
Creating Variants
Google Optimize
Creating variants in Optimize is pretty simple with the built-in WYSIWYG visual editor. This tool allows you to simply click the elements you would like to change and make any changes you want. In the event that you need to make a more advanced change, you also have the option to utilize HTML, CSS, and custom Javascript to fine-tune your variant.
Optimizely
The Optimizely visual editor also allows you to make changes to the layout and appearance of your site. This includes editing an element’s visibility or position, modification of text, images, background styles, and borders. In addition to the WYSIWYG editor, you can insert HTML, CSS, JavaScript, and use jQuery selectors.
Targeting Options
Google Optimize
Google Optimize allows you to choose who is eligible to be in your experiment with user targeting rules. And since Optimize natively integrates with both Google Analytics and Google Ads, you can easily target users in audiences you’ve created in Google Analytics, or users who have been exposed to specific Google Ads campaigns, ad groups, and keywords.
In addition to basic targeting rules like device category, technology, and geography, Optimize allows you to target more advanced onsite attributes using javaScript or the dataLayer. If you are already utilizing the dataLayer in your Google Analytics implementation, this is a huge time-saver since no additional coding is required to target those attributes.
Any of these targeting options can be combined to create very specific rules for your experiments.
Optimizely
Similarly, Optimizely audiences allow you to decide who will see your experiment. When you create an experiment the default audience is all visitors but you can combine any number of attributes to create a more targeted audience. Optimizely is pre-loaded with some standard built-in audience attributes, as shown below.
Custom attributes can also be used to target users within specific audiences, but requires adding some custom JavaScript code for each attribute to be collected. However, there is no option to target elements that are setup in the dataLayer directly.
Test Objectives
Google Optimize
Google Optimize offers system, Google Analytics goal, and custom objective types. Goals that have been created in the linked Google Analytics view are automatically imported into Optimize for use as experiment objectives.
Custom objectives can be used to create goals based on a user visiting a specific page or any event tracking that is set up on your linked Google Analytics property. No additional coding is required to create custom objectives as long as they are being tracked in Google Analytics already as events.
Optimizely
Similar to Google Optimize, Optimizely has the ability to set a primary metric as well as secondary metrics to holistically measure the effects of the changes made during the experiment. Optimizely’s metric builder allows you to quickly define experiment objectives based on events such as clicks, pageviews, form submissions, purchases, and scroll depth.
In Optimizely, metrics are created from click, pageview, or custom events. Basic click events can be created using the visual editor to select an element’s CSS selector. This could apply to elements like “add to cart” or “buy now” buttons. Pageview events track whether a visitor views a certain page.
Custom events require additional coding on your site using the Optimizely JavaScript API. These events don’t need to be set up every time you create a test. For example, if you set up a custom event to track submission of an email subscription form on a specific page, you could reuse that event for other experiments at a later time. Other applications for custom events might include video plays, transactions, or chatbot interactions.
Methodology and Results
Google Optimize
Optimize uses Bayesian Inference to generate predicted outcomes (as opposed to frequentist-based analysis used by other testing tools). This methodology expresses how a degree of belief, expressed as a probability, should rationally change to account for availability of related evidence.
Optimize uses existing data to make better assumptions about new data. As it collects new data, the “model” is refined, producing more accurate results. Some of the models used in combination with Bayesian Inference include:
- Hierarchical models: Consistency of the variant’s conversion rates over time
- Contextual models: Accounts for experiment or user context
- Restless models: Neutralizes overall performance trends that could affect all variants
The Google Optimize reporting interface contains three reports to allow you to quickly understand your experiment’s status.
- The summary header gives you a quick message about the experiment’s objective, like “At least one variant is better than the original” or “Keep your experiment running.”
- The objective card displays both the observed data (from the linked Google Analytics view) and the predicted outcomes for the selected objective. The Optimize analysis shows the probability to be best (PBB), probability to beat the original (PBO), modeled conversion rate, and modeled improvement.
- The modeled conversion rate chart shows each variant’s modeled conversion rate over time for the selected objective.
Optimizely
Optimizely utilizes a proprietary model called Stats Engine. This model combines sequential hypothesis testing and false discovery rate controls to determine predicted outcomes. Stats Engine corrects for common pitfalls in experimentation, such as adding too many metrics or too many variations. Optimizely does not require that you set a sample size or duration in advance. But although you can check your results at any time, statistical significance increases over time as more data is collected.
The Optimizely experiment results page also provides a high-level summary, as well as a module with details about each metric by variant. The summary shows how each variant performs for the primary metric as compared to the original version. The metrics section of the results page provides more detailed information for each variant such as unique or total conversions, conversion rates, relative improvement over the baseline, confidence interval, and statistical significance.
The Optimizely interface also gives you the option to segment your results to understand how results change for specific groups of users versus overall visitors. However, it is much more likely that false positives could occur when applying this segmentation.
Integrations
Google Optimize
One of the biggest selling points of Google Optimize is that it natively integrates with a number of Google products. If you already use Google Analytics and have taken the time to customize your implementation, very little additional setup is required to get up and running with A/B testing. Your Google Analytics events and goals are automatically available, as well as commonly used metrics such as transactions, revenue, pageviews, and bounce rate.
If you use Google Optimize 360 (the enterprise-level version of Optimize), you can also use your Google Analytics audiences. The audience integration gives you the ability to target users who have exhibited specific behaviors. For example, you could use audiences to target users who have purchased from a specific product category or who have made a purchase higher than a specific threshold. This targeting is great for experimentation as well as the personalization capabilities of Google Optimize.
Optimize also has a native integration with Google Ads. This integration empowers you to test the effect of customized content and landing pages on objectives. You can use this integration to target accounts, campaigns, ad groups or even specific keywords.
Experiment data can be exported to BigQuery for more advanced or large-scale analysis and visualization.
In addition to native integrations, many tools offer third-party integrations to enhance your understanding and activation of the available data in Google Optimize. Session replay and heatmapping tools like SessionCam and customer data platforms (CDPs) like BlueConic are just a few examples of tools that can further enrich your experimentation process.
Optimizely
Optimizely offers a number of third-party integrations as well. All Optimizely plans include integrations with Google Analytics, Adobe Analytics and Clicktale. The Google Analytics integration allows you to use a custom dimension to segment data within Google Analytics based on the user’s inclusion in a specific experiment. The Adobe Analytics integration functions similarly, giving you the option to segment your traffic based on eVars.
The built-in integration with Clicktale is really easy to set up. This integration helps you to find recordings for specific experiments and variations. And, if you are using an Enterprise Clicktale account, you can also use Optimizely to create customized aggregated heatmap reports for mouse movement and clicks, attention, and scroll reach.
Implementation requirements, variant creation, test objectives, methodology, reporting and integrations are all important factors to consider when choosing an experimentation platform. There are of course other factors to consider such as cost, product support offerings and your own internal development resources. But no matter what experimentation platform you choose, the key to success is an experimentation strategy that meets the needs of your organization.
Still Can’t Decide Between A/B Testing Platforms?
Feel free to contact the team at InfoTrust for help determining which platform is right for your organization’s goals and long-term A/B testing needs. InfoTrust is a certified Google Marketing Platform Sales Partner (Google Optimize is part of the GMP stack) and works with many clients who use Optimizely for their experimentation strategy.
