The ROI Revolution Blog

Viewing A/B Experiments in Google Analytics

July 24, 2009

testtubes.jpgI've written a few articles before about the power of integrating Google Website Optimizer with Google Analytics, but the landscape has changed considerably in the past few months. I thought it would be a good idea to provide an update as to how you can view Google Website Optimizer and other kinds of experiment data within Google Analytics as it has become much easier with the addition of a magnificent feature called Advanced Segments.

As the title implies, this article will focus on finding data quickly and easily for A/B (or A/B/C/D, etc.) experiments, whether they be Google Website Optimizer experiments, or just some testing that you're doing on your own.

First, why would you want to do this? Doesn't Google Website Optimizer (or AdWords, etc.) provide you with Conversion Rate already? Sure it does! But I maintain that an aggregated Conversion Rate (for a single goal no less) is simply not enough. I demand more from my experiments, and by looking at my experiment data within Google Analytics, I can get a great idea of how the experiment is working for different segments of visits and for every goal on my site. What's even better is that by using Google Analytics, I'll also have access to transaction and revenue data, along with any powerful customizations that I'm already using to get the most out of my tracking.

Here are some of the benefits to using Google Analytics to measure your A/B experiment results:

  • Segmenting the data
  • Seeing statistics for more than one goal
  • Looking at transaction and revenue data
  • Excluding internal traffic
  • Flexibility and the ability to customize data
  • Custom reports, emailing and exporting reports, all of the other fun features in Google Analytics

I hope you'll agree there these are some pretty massive benefits. So enough of my gabbing, here's how to do all of this for an A/B test in two simple steps:

1. Create your Advanced Segments

This is the step that has made examining A/B tests within Google Analytics a joy instead of the awkward wobbling toddler of a process that it used to be.

Fortunately, now that Google Analytics has put on its big boy pants, creating these Advanced Segments is a relatively simple task. You can access Advanced Segments from one of two places. The most common place is in the top-right corner of the Google Analytics interface:

advanced-seg1.jpg

You can also access Advanced Segments from the link on the left-hand side of the interface:

advanced-seg2.jpg

Once you choose to create a new Advanced Segment, you'll be taken to a screen that looks like this:

new-segment1.jpg

Since we're looking at the efforts of landing pages here, you'll want to select either Page or Landing Page from the list of dimensions on the left. You can simply type in 'Page' into the search bar to find them a little more easily.

dimension.jpg

Once you've done than, you can start typing the name of the page into the input bar and a list of the pages that match your string will pop up, allowing you to easily select the page you are looking for. If your site uses query parameters or URL variations, you may want to change the match type to 'Contains'. You can even use Regular Expressions if you have more advanced needs.

page-list.jpg

Once you've created both segments, you're ready for step 2.

2. Apply your Advanced Segments

The easiest way to apply the segments that you created is to use the dropdown menu in the top-right of the Google Analytics UI. You'll see something like this:

choosing-segments.jpg

Simply check the new segments that you've made and click on the Apply button. The report that you are looking at within Google Analytics will change to feature the new segments:

abreport1-small.jpg

As you can see, there is some pretty powerful information here. The above report is an example of the AdWords Campaigns report that's been enhanced with our Google Analytics Report Enhancer tool. As you can see, sometimes a page may be winning in one campaign and losing in another.

Also, you can see how conversions are faring for multiple steps in a single funnel, and you can segment the data even further by Ad Group, keyword, ad version, geographic location, and anything else you may need. You can even set the report up to show up in your inbox every morning!

The other great thing about looking at this data within Google Analytics is that you'll have a record of the entire test's performance over time. You can see how overall fluctuations in the marketplace, or snags with your server's load times have impacted your test over time, and can exclude irrelevant data. I really can't stress enough how powerful this kind of analysis can be.

That's all there is to it! Please feel free to try this method out and leave comments - I'd love to hear your feedback or if you've found a better way to do this.

Google Analytics for Online Advertisers
Here at ROI Revolution, we consider Google Analytics tracking essential for paid search, so it's included in our PPC Campaign Management service.

Comments

Mehdi said:

As usual, great insights from you Shawn ;)

Am I crazy or you can compare 2 segments without displaying the "all visits" segment?

Next change from Google?

July 26, 2009 5:41 PM

steve said:

Any way to see multivariate tests?

July 27, 2009 11:33 AM

Shawn Purtell, Senior Web Analytics Engineer Author Profile Page said:

@Mehdi:

I've noticed that sometimes I can deselect the 'All Visits' segment and sometimes I can't. I haven't figured why exactly that happens yet, but if I unlock the mystery I'll be sure to blog about it.

July 27, 2009 1:12 PM

Shawn Purtell, Senior Web Analytics Engineer Author Profile Page said:

@Steve:

There are actually a few ways out there right now to view multivariate test data within Google Analytics. One involves virtual pageviews with Site Search, and another version involves the User Defined variable. Either way, you need to be able to get the combination data from the Google Website Optimizer cookie and send it over to Google Analytics. I'm planning on spelling out these options in a future post, but you can find out more about how it's done by visiting this post

July 27, 2009 1:16 PM

James Stratford @JRStratford on Twitter said:

That's a great use of Google Analytics and Website Analyzer. I'd never thought of the combination or knew that it would be so useful if coupled. It beats out my A/B testing script on my website. Though I use Google Analytics it's under utilized ATM, but thanks to you that will be changing.

I'll do a comparison between the tools and now that I think of it the big benefit is it won't utilize my server and that certainly won't hurt a thing.

Will be doing some more bookmarking since this is a great post and others need to know about it for sure.

July 27, 2009 8:42 PM

Trevor Claiborne said:

Nice write up, Shawn.

The only thing I've done differently is I don't base the segment on landing page. I make my segments to be anyone who visited the A page and anyone who visited the B page. Since the two pages should be mutually exclusive, this works fine, and I don't always run tests on landing pages.

July 27, 2009 11:13 PM

Shawn Purtell, Senior Web Analytics Engineer Author Profile Page said:

@James:

Thanks for reading and I hope you find this as useful as I have!

July 28, 2009 7:45 AM

Shawn Purtell, Senior Web Analytics Engineer Author Profile Page said:

@Trevor:

Thanks for the compliment. I tend to use the Page Dimension instead of Landing Page as well, and you'll see I actually reference them both in the article and in the screenshots. I think they both can be useful depending on the situation and how the pages interact with the rest of a given website.

July 28, 2009 7:48 AM

Nick said:

Hi Shawn,

Interesting post and a great way to review more detailed info on your experiments than WO itself provides. Seems like you could learn a lot using this technique that would help you design subsequent experiments.

However, I'm still wondering exactly how this kind of data is actionable in the immediate term. If you're mid-experiment, you'll most likely want to wait for WO to determine which of your experimental variations is the overall 'winner,' so reviewing the data along the way would satisfy the curiosity of a data geek such as myself, but I'm not sure I could do a lot with it immediately. Could you say a bit more about how this is actionable in the immediate term?

Thanks!

July 28, 2009 3:49 PM

Shawn Purtell, Senior Web Analytics Engineer Author Profile Page said:

@Nick:

The most immediate benefit of all is the ability to view multiple goals or in the case of ecommerce, transactions and revenue. This is information you simply just don't have within the Google Website Optimizer interface, and can be absolutely vital to Ecommerce sites. Ignoring revenue in a Google Website Optimizer test is a HUGE mistake.

Also, why would you always wait for WO to determine a winner? Google Website Optimizer uses some general statistical significance formulas, but by no means should you always wait for an experiment to be decided by the interface. Time is money, and often you can tell if an experiment has a successful iteration long before the interface will tell you.

That being said, it is also wise to not be too hasty. If you are testing a page across a wide variety of marketing sources and keywords, the overall conversion data within Google Website Optimizer can be misleading in the same way that site averages can be misleading. To give you an example, we've seen experiments where the overall conversion rate for our new page was much better than our old page, except in a specific campaign or ad group. By swapping to the new page without drilling down even a little, we could be actually hurting performance for a specific campaign and not even realizing it.

On the flip side, you may find that while Google Website Optimizer has not yet declared a winner, that for a particular campaign there is a clear winner. If that's the case, you can take action for that campaign while leaving the remaining traffic in a test.

Finally, I'm not necessarily suggesting that you should examine the data within Google Analytics every day. I'm simply saying that when it's time to look at the data, it's worth looking at the data within Google Analytics as well since it takes so little time to set up and you may find an insight or two. The insights you find will be determined by the nature of your website, your business, and your test.

July 28, 2009 4:31 PM

suhan said:

this article is awesome and it really helped me a lot thanx

July 29, 2009 8:31 AM

Shawn Purtell, Senior Web Analytics Engineer Author Profile Page said:

As an update, the mystery concerning the deselection of the 'All Visits' segment has been solved. It turns out Jeremy here at ROI has been doing some work on the Google Analytics Report Enhancer, and this was one of the things he was working on. Look for an update from him in the coming weeks.

August 5, 2009 8:58 AM

Kathryn said:

Thanks for the great post. This seems like a *much* easier way of tracking test results than setting up two different profiles -- but I read elsewhere that the Advanced Segments functionality doesn't work with the Funnel Visualization report, has that been your experience?

August 26, 2009 4:17 PM

Shawn Purtell, Senior Web Analytics Engineer Author Profile Page said:

@Kathryn,

Unfortunately, as you pointed out, Advanced Segments do not work with the Funnel Visualization report. There are a few other reports that also fall into this category. Most of the time, you can still get the data by using the Content reports (Navigation Summary, etc.) creatively, but currently there is no great way to reproduce the Funnel Navigation reports exactly without the separate profiles you mentioned.

September 1, 2009 10:01 AM

Geoff Jackson said:

Great tutorial. Rarely do I find that Google's Website Optimizer records data so most of my reporting for A/B testing has to be produced via GA.

August 19, 2010 6:43 AM

Mehdi said:

As usual, great insights from you Shawn ;)

Am I crazy or you can compare 2 segments without displaying the "all visits" segment?

Next change from Google?

July 26, 2009 5:41 PM

steve said:

Any way to see multivariate tests?

July 27, 2009 11:33 AM

Shawn Purtell, Senior Web Analytics Engineer Author Profile Page said:

@Mehdi:

I've noticed that sometimes I can deselect the 'All Visits' segment and sometimes I can't. I haven't figured why exactly that happens yet, but if I unlock the mystery I'll be sure to blog about it.

July 27, 2009 1:12 PM

Shawn Purtell, Senior Web Analytics Engineer Author Profile Page said:

@Steve:

There are actually a few ways out there right now to view multivariate test data within Google Analytics. One involves virtual pageviews with Site Search, and another version involves the User Defined variable. Either way, you need to be able to get the combination data from the Google Website Optimizer cookie and send it over to Google Analytics. I'm planning on spelling out these options in a future post, but you can find out more about how it's done by visiting this post

July 27, 2009 1:16 PM

James Stratford @JRStratford on Twitter said:

That's a great use of Google Analytics and Website Analyzer. I'd never thought of the combination or knew that it would be so useful if coupled. It beats out my A/B testing script on my website. Though I use Google Analytics it's under utilized ATM, but thanks to you that will be changing.

I'll do a comparison between the tools and now that I think of it the big benefit is it won't utilize my server and that certainly won't hurt a thing.

Will be doing some more bookmarking since this is a great post and others need to know about it for sure.

July 27, 2009 8:42 PM

Trevor Claiborne said:

Nice write up, Shawn.

The only thing I've done differently is I don't base the segment on landing page. I make my segments to be anyone who visited the A page and anyone who visited the B page. Since the two pages should be mutually exclusive, this works fine, and I don't always run tests on landing pages.

July 27, 2009 11:13 PM

Shawn Purtell, Senior Web Analytics Engineer Author Profile Page said:

@James:

Thanks for reading and I hope you find this as useful as I have!

July 28, 2009 7:45 AM

Shawn Purtell, Senior Web Analytics Engineer Author Profile Page said:

@Trevor:

Thanks for the compliment. I tend to use the Page Dimension instead of Landing Page as well, and you'll see I actually reference them both in the article and in the screenshots. I think they both can be useful depending on the situation and how the pages interact with the rest of a given website.

July 28, 2009 7:48 AM

Nick said:

Hi Shawn,

Interesting post and a great way to review more detailed info on your experiments than WO itself provides. Seems like you could learn a lot using this technique that would help you design subsequent experiments.

However, I'm still wondering exactly how this kind of data is actionable in the immediate term. If you're mid-experiment, you'll most likely want to wait for WO to determine which of your experimental variations is the overall 'winner,' so reviewing the data along the way would satisfy the curiosity of a data geek such as myself, but I'm not sure I could do a lot with it immediately. Could you say a bit more about how this is actionable in the immediate term?

Thanks!

July 28, 2009 3:49 PM

Shawn Purtell, Senior Web Analytics Engineer Author Profile Page said:

@Nick:

The most immediate benefit of all is the ability to view multiple goals or in the case of ecommerce, transactions and revenue. This is information you simply just don't have within the Google Website Optimizer interface, and can be absolutely vital to Ecommerce sites. Ignoring revenue in a Google Website Optimizer test is a HUGE mistake.

Also, why would you always wait for WO to determine a winner? Google Website Optimizer uses some general statistical significance formulas, but by no means should you always wait for an experiment to be decided by the interface. Time is money, and often you can tell if an experiment has a successful iteration long before the interface will tell you.

That being said, it is also wise to not be too hasty. If you are testing a page across a wide variety of marketing sources and keywords, the overall conversion data within Google Website Optimizer can be misleading in the same way that site averages can be misleading. To give you an example, we've seen experiments where the overall conversion rate for our new page was much better than our old page, except in a specific campaign or ad group. By swapping to the new page without drilling down even a little, we could be actually hurting performance for a specific campaign and not even realizing it.

On the flip side, you may find that while Google Website Optimizer has not yet declared a winner, that for a particular campaign there is a clear winner. If that's the case, you can take action for that campaign while leaving the remaining traffic in a test.

Finally, I'm not necessarily suggesting that you should examine the data within Google Analytics every day. I'm simply saying that when it's time to look at the data, it's worth looking at the data within Google Analytics as well since it takes so little time to set up and you may find an insight or two. The insights you find will be determined by the nature of your website, your business, and your test.

July 28, 2009 4:31 PM

suhan said:

this article is awesome and it really helped me a lot thanx

July 29, 2009 8:31 AM

Shawn Purtell, Senior Web Analytics Engineer Author Profile Page said:

As an update, the mystery concerning the deselection of the 'All Visits' segment has been solved. It turns out Jeremy here at ROI has been doing some work on the Google Analytics Report Enhancer, and this was one of the things he was working on. Look for an update from him in the coming weeks.

August 5, 2009 8:58 AM

Kathryn said:

Thanks for the great post. This seems like a *much* easier way of tracking test results than setting up two different profiles -- but I read elsewhere that the Advanced Segments functionality doesn't work with the Funnel Visualization report, has that been your experience?

August 26, 2009 4:17 PM

Shawn Purtell, Senior Web Analytics Engineer Author Profile Page said:

@Kathryn,

Unfortunately, as you pointed out, Advanced Segments do not work with the Funnel Visualization report. There are a few other reports that also fall into this category. Most of the time, you can still get the data by using the Content reports (Navigation Summary, etc.) creatively, but currently there is no great way to reproduce the Funnel Navigation reports exactly without the separate profiles you mentioned.

September 1, 2009 10:01 AM

Geoff Jackson said:

Great tutorial. Rarely do I find that Google's Website Optimizer records data so most of my reporting for A/B testing has to be produced via GA.

August 19, 2010 6:43 AM

Mehdi said:

As usual, great insights from you Shawn ;)

Am I crazy or you can compare 2 segments without displaying the "all visits" segment?

Next change from Google?

July 26, 2009 5:41 PM

steve said:

Any way to see multivariate tests?

July 27, 2009 11:33 AM

Shawn Purtell, Senior Web Analytics Engineer Author Profile Page said:

@Mehdi:

I've noticed that sometimes I can deselect the 'All Visits' segment and sometimes I can't. I haven't figured why exactly that happens yet, but if I unlock the mystery I'll be sure to blog about it.

July 27, 2009 1:12 PM

Shawn Purtell, Senior Web Analytics Engineer Author Profile Page said:

@Steve:

There are actually a few ways out there right now to view multivariate test data within Google Analytics. One involves virtual pageviews with Site Search, and another version involves the User Defined variable. Either way, you need to be able to get the combination data from the Google Website Optimizer cookie and send it over to Google Analytics. I'm planning on spelling out these options in a future post, but you can find out more about how it's done by visiting this post

July 27, 2009 1:16 PM

James Stratford @JRStratford on Twitter said:

That's a great use of Google Analytics and Website Analyzer. I'd never thought of the combination or knew that it would be so useful if coupled. It beats out my A/B testing script on my website. Though I use Google Analytics it's under utilized ATM, but thanks to you that will be changing.

I'll do a comparison between the tools and now that I think of it the big benefit is it won't utilize my server and that certainly won't hurt a thing.

Will be doing some more bookmarking since this is a great post and others need to know about it for sure.

July 27, 2009 8:42 PM

Trevor Claiborne said:

Nice write up, Shawn.

The only thing I've done differently is I don't base the segment on landing page. I make my segments to be anyone who visited the A page and anyone who visited the B page. Since the two pages should be mutually exclusive, this works fine, and I don't always run tests on landing pages.

July 27, 2009 11:13 PM

Shawn Purtell, Senior Web Analytics Engineer Author Profile Page said:

@James:

Thanks for reading and I hope you find this as useful as I have!

July 28, 2009 7:45 AM

Shawn Purtell, Senior Web Analytics Engineer Author Profile Page said:

@Trevor:

Thanks for the compliment. I tend to use the Page Dimension instead of Landing Page as well, and you'll see I actually reference them both in the article and in the screenshots. I think they both can be useful depending on the situation and how the pages interact with the rest of a given website.

July 28, 2009 7:48 AM

Nick said:

Hi Shawn,

Interesting post and a great way to review more detailed info on your experiments than WO itself provides. Seems like you could learn a lot using this technique that would help you design subsequent experiments.

However, I'm still wondering exactly how this kind of data is actionable in the immediate term. If you're mid-experiment, you'll most likely want to wait for WO to determine which of your experimental variations is the overall 'winner,' so reviewing the data along the way would satisfy the curiosity of a data geek such as myself, but I'm not sure I could do a lot with it immediately. Could you say a bit more about how this is actionable in the immediate term?

Thanks!

July 28, 2009 3:49 PM

Shawn Purtell, Senior Web Analytics Engineer Author Profile Page said:

@Nick:

The most immediate benefit of all is the ability to view multiple goals or in the case of ecommerce, transactions and revenue. This is information you simply just don't have within the Google Website Optimizer interface, and can be absolutely vital to Ecommerce sites. Ignoring revenue in a Google Website Optimizer test is a HUGE mistake.

Also, why would you always wait for WO to determine a winner? Google Website Optimizer uses some general statistical significance formulas, but by no means should you always wait for an experiment to be decided by the interface. Time is money, and often you can tell if an experiment has a successful iteration long before the interface will tell you.

That being said, it is also wise to not be too hasty. If you are testing a page across a wide variety of marketing sources and keywords, the overall conversion data within Google Website Optimizer can be misleading in the same way that site averages can be misleading. To give you an example, we've seen experiments where the overall conversion rate for our new page was much better than our old page, except in a specific campaign or ad group. By swapping to the new page without drilling down even a little, we could be actually hurting performance for a specific campaign and not even realizing it.

On the flip side, you may find that while Google Website Optimizer has not yet declared a winner, that for a particular campaign there is a clear winner. If that's the case, you can take action for that campaign while leaving the remaining traffic in a test.

Finally, I'm not necessarily suggesting that you should examine the data within Google Analytics every day. I'm simply saying that when it's time to look at the data, it's worth looking at the data within Google Analytics as well since it takes so little time to set up and you may find an insight or two. The insights you find will be determined by the nature of your website, your business, and your test.

July 28, 2009 4:31 PM

suhan said:

this article is awesome and it really helped me a lot thanx

July 29, 2009 8:31 AM

Shawn Purtell, Senior Web Analytics Engineer Author Profile Page said:

As an update, the mystery concerning the deselection of the 'All Visits' segment has been solved. It turns out Jeremy here at ROI has been doing some work on the Google Analytics Report Enhancer, and this was one of the things he was working on. Look for an update from him in the coming weeks.

August 5, 2009 8:58 AM

Kathryn said:

Thanks for the great post. This seems like a *much* easier way of tracking test results than setting up two different profiles -- but I read elsewhere that the Advanced Segments functionality doesn't work with the Funnel Visualization report, has that been your experience?

August 26, 2009 4:17 PM

Shawn Purtell, Senior Web Analytics Engineer Author Profile Page said:

@Kathryn,

Unfortunately, as you pointed out, Advanced Segments do not work with the Funnel Visualization report. There are a few other reports that also fall into this category. Most of the time, you can still get the data by using the Content reports (Navigation Summary, etc.) creatively, but currently there is no great way to reproduce the Funnel Navigation reports exactly without the separate profiles you mentioned.

September 1, 2009 10:01 AM

Geoff Jackson said:

Great tutorial. Rarely do I find that Google's Website Optimizer records data so most of my reporting for A/B testing has to be produced via GA.

August 19, 2010 6:43 AM

Mehdi said:

As usual, great insights from you Shawn ;)

Am I crazy or you can compare 2 segments without displaying the "all visits" segment?

Next change from Google?

July 26, 2009 5:41 PM

steve said:

Any way to see multivariate tests?

July 27, 2009 11:33 AM

Shawn Purtell, Senior Web Analytics Engineer Author Profile Page said:

@Mehdi:

I've noticed that sometimes I can deselect the 'All Visits' segment and sometimes I can't. I haven't figured why exactly that happens yet, but if I unlock the mystery I'll be sure to blog about it.

July 27, 2009 1:12 PM

Shawn Purtell, Senior Web Analytics Engineer Author Profile Page said:

@Steve:

There are actually a few ways out there right now to view multivariate test data within Google Analytics. One involves virtual pageviews with Site Search, and another version involves the User Defined variable. Either way, you need to be able to get the combination data from the Google Website Optimizer cookie and send it over to Google Analytics. I'm planning on spelling out these options in a future post, but you can find out more about how it's done by visiting this post

July 27, 2009 1:16 PM

James Stratford @JRStratford on Twitter said:

That's a great use of Google Analytics and Website Analyzer. I'd never thought of the combination or knew that it would be so useful if coupled. It beats out my A/B testing script on my website. Though I use Google Analytics it's under utilized ATM, but thanks to you that will be changing.

I'll do a comparison between the tools and now that I think of it the big benefit is it won't utilize my server and that certainly won't hurt a thing.

Will be doing some more bookmarking since this is a great post and others need to know about it for sure.

July 27, 2009 8:42 PM

Trevor Claiborne said:

Nice write up, Shawn.

The only thing I've done differently is I don't base the segment on landing page. I make my segments to be anyone who visited the A page and anyone who visited the B page. Since the two pages should be mutually exclusive, this works fine, and I don't always run tests on landing pages.

July 27, 2009 11:13 PM

Shawn Purtell, Senior Web Analytics Engineer Author Profile Page said:

@James:

Thanks for reading and I hope you find this as useful as I have!

July 28, 2009 7:45 AM

Shawn Purtell, Senior Web Analytics Engineer Author Profile Page said:

@Trevor:

Thanks for the compliment. I tend to use the Page Dimension instead of Landing Page as well, and you'll see I actually reference them both in the article and in the screenshots. I think they both can be useful depending on the situation and how the pages interact with the rest of a given website.

July 28, 2009 7:48 AM

Nick said:

Hi Shawn,

Interesting post and a great way to review more detailed info on your experiments than WO itself provides. Seems like you could learn a lot using this technique that would help you design subsequent experiments.

However, I'm still wondering exactly how this kind of data is actionable in the immediate term. If you're mid-experiment, you'll most likely want to wait for WO to determine which of your experimental variations is the overall 'winner,' so reviewing the data along the way would satisfy the curiosity of a data geek such as myself, but I'm not sure I could do a lot with it immediately. Could you say a bit more about how this is actionable in the immediate term?

Thanks!

July 28, 2009 3:49 PM

Shawn Purtell, Senior Web Analytics Engineer Author Profile Page said:

@Nick:

The most immediate benefit of all is the ability to view multiple goals or in the case of ecommerce, transactions and revenue. This is information you simply just don't have within the Google Website Optimizer interface, and can be absolutely vital to Ecommerce sites. Ignoring revenue in a Google Website Optimizer test is a HUGE mistake.

Also, why would you always wait for WO to determine a winner? Google Website Optimizer uses some general statistical significance formulas, but by no means should you always wait for an experiment to be decided by the interface. Time is money, and often you can tell if an experiment has a successful iteration long before the interface will tell you.

That being said, it is also wise to not be too hasty. If you are testing a page across a wide variety of marketing sources and keywords, the overall conversion data within Google Website Optimizer can be misleading in the same way that site averages can be misleading. To give you an example, we've seen experiments where the overall conversion rate for our new page was much better than our old page, except in a specific campaign or ad group. By swapping to the new page without drilling down even a little, we could be actually hurting performance for a specific campaign and not even realizing it.

On the flip side, you may find that while Google Website Optimizer has not yet declared a winner, that for a particular campaign there is a clear winner. If that's the case, you can take action for that campaign while leaving the remaining traffic in a test.

Finally, I'm not necessarily suggesting that you should examine the data within Google Analytics every day. I'm simply saying that when it's time to look at the data, it's worth looking at the data within Google Analytics as well since it takes so little time to set up and you may find an insight or two. The insights you find will be determined by the nature of your website, your business, and your test.

July 28, 2009 4:31 PM

suhan said:

this article is awesome and it really helped me a lot thanx

July 29, 2009 8:31 AM

Shawn Purtell, Senior Web Analytics Engineer Author Profile Page said:

As an update, the mystery concerning the deselection of the 'All Visits' segment has been solved. It turns out Jeremy here at ROI has been doing some work on the Google Analytics Report Enhancer, and this was one of the things he was working on. Look for an update from him in the coming weeks.

August 5, 2009 8:58 AM

Kathryn said:

Thanks for the great post. This seems like a *much* easier way of tracking test results than setting up two different profiles -- but I read elsewhere that the Advanced Segments functionality doesn't work with the Funnel Visualization report, has that been your experience?

August 26, 2009 4:17 PM

Shawn Purtell, Senior Web Analytics Engineer Author Profile Page said:

@Kathryn,

Unfortunately, as you pointed out, Advanced Segments do not work with the Funnel Visualization report. There are a few other reports that also fall into this category. Most of the time, you can still get the data by using the Content reports (Navigation Summary, etc.) creatively, but currently there is no great way to reproduce the Funnel Navigation reports exactly without the separate profiles you mentioned.

September 1, 2009 10:01 AM

Geoff Jackson said:

Great tutorial. Rarely do I find that Google's Website Optimizer records data so most of my reporting for A/B testing has to be produced via GA.

August 19, 2010 6:43 AM

Mehdi said:

As usual, great insights from you Shawn ;)

Am I crazy or you can compare 2 segments without displaying the "all visits" segment?

Next change from Google?

July 26, 2009 5:41 PM

steve said:

Any way to see multivariate tests?

July 27, 2009 11:33 AM

Shawn Purtell, Senior Web Analytics Engineer Author Profile Page said:

@Mehdi:

I've noticed that sometimes I can deselect the 'All Visits' segment and sometimes I can't. I haven't figured why exactly that happens yet, but if I unlock the mystery I'll be sure to blog about it.

July 27, 2009 1:12 PM

Shawn Purtell, Senior Web Analytics Engineer Author Profile Page said:

@Steve:

There are actually a few ways out there right now to view multivariate test data within Google Analytics. One involves virtual pageviews with Site Search, and another version involves the User Defined variable. Either way, you need to be able to get the combination data from the Google Website Optimizer cookie and send it over to Google Analytics. I'm planning on spelling out these options in a future post, but you can find out more about how it's done by visiting this post

July 27, 2009 1:16 PM

James Stratford @JRStratford on Twitter said:

That's a great use of Google Analytics and Website Analyzer. I'd never thought of the combination or knew that it would be so useful if coupled. It beats out my A/B testing script on my website. Though I use Google Analytics it's under utilized ATM, but thanks to you that will be changing.

I'll do a comparison between the tools and now that I think of it the big benefit is it won't utilize my server and that certainly won't hurt a thing.

Will be doing some more bookmarking since this is a great post and others need to know about it for sure.

July 27, 2009 8:42 PM

Trevor Claiborne said:

Nice write up, Shawn.

The only thing I've done differently is I don't base the segment on landing page. I make my segments to be anyone who visited the A page and anyone who visited the B page. Since the two pages should be mutually exclusive, this works fine, and I don't always run tests on landing pages.

July 27, 2009 11:13 PM

Shawn Purtell, Senior Web Analytics Engineer Author Profile Page said:

@James:

Thanks for reading and I hope you find this as useful as I have!

July 28, 2009 7:45 AM

Shawn Purtell, Senior Web Analytics Engineer Author Profile Page said:

@Trevor:

Thanks for the compliment. I tend to use the Page Dimension instead of Landing Page as well, and you'll see I actually reference them both in the article and in the screenshots. I think they both can be useful depending on the situation and how the pages interact with the rest of a given website.

July 28, 2009 7:48 AM

Nick said:

Hi Shawn,

Interesting post and a great way to review more detailed info on your experiments than WO itself provides. Seems like you could learn a lot using this technique that would help you design subsequent experiments.

However, I'm still wondering exactly how this kind of data is actionable in the immediate term. If you're mid-experiment, you'll most likely want to wait for WO to determine which of your experimental variations is the overall 'winner,' so reviewing the data along the way would satisfy the curiosity of a data geek such as myself, but I'm not sure I could do a lot with it immediately. Could you say a bit more about how this is actionable in the immediate term?

Thanks!

July 28, 2009 3:49 PM

Shawn Purtell, Senior Web Analytics Engineer Author Profile Page said:

@Nick:

The most immediate benefit of all is the ability to view multiple goals or in the case of ecommerce, transactions and revenue. This is information you simply just don't have within the Google Website Optimizer interface, and can be absolutely vital to Ecommerce sites. Ignoring revenue in a Google Website Optimizer test is a HUGE mistake.

Also, why would you always wait for WO to determine a winner? Google Website Optimizer uses some general statistical significance formulas, but by no means should you always wait for an experiment to be decided by the interface. Time is money, and often you can tell if an experiment has a successful iteration long before the interface will tell you.

That being said, it is also wise to not be too hasty. If you are testing a page across a wide variety of marketing sources and keywords, the overall conversion data within Google Website Optimizer can be misleading in the same way that site averages can be misleading. To give you an example, we've seen experiments where the overall conversion rate for our new page was much better than our old page, except in a specific campaign or ad group. By swapping to the new page without drilling down even a little, we could be actually hurting performance for a specific campaign and not even realizing it.

On the flip side, you may find that while Google Website Optimizer has not yet declared a winner, that for a particular campaign there is a clear winner. If that's the case, you can take action for that campaign while leaving the remaining traffic in a test.

Finally, I'm not necessarily suggesting that you should examine the data within Google Analytics every day. I'm simply saying that when it's time to look at the data, it's worth looking at the data within Google Analytics as well since it takes so little time to set up and you may find an insight or two. The insights you find will be determined by the nature of your website, your business, and your test.

July 28, 2009 4:31 PM

suhan said:

this article is awesome and it really helped me a lot thanx

July 29, 2009 8:31 AM

Shawn Purtell, Senior Web Analytics Engineer Author Profile Page said:

As an update, the mystery concerning the deselection of the 'All Visits' segment has been solved. It turns out Jeremy here at ROI has been doing some work on the Google Analytics Report Enhancer, and this was one of the things he was working on. Look for an update from him in the coming weeks.

August 5, 2009 8:58 AM

Kathryn said:

Thanks for the great post. This seems like a *much* easier way of tracking test results than setting up two different profiles -- but I read elsewhere that the Advanced Segments functionality doesn't work with the Funnel Visualization report, has that been your experience?

August 26, 2009 4:17 PM

Shawn Purtell, Senior Web Analytics Engineer Author Profile Page said:

@Kathryn,

Unfortunately, as you pointed out, Advanced Segments do not work with the Funnel Visualization report. There are a few other reports that also fall into this category. Most of the time, you can still get the data by using the Content reports (Navigation Summary, etc.) creatively, but currently there is no great way to reproduce the Funnel Navigation reports exactly without the separate profiles you mentioned.

September 1, 2009 10:01 AM

Geoff Jackson said:

Great tutorial. Rarely do I find that Google's Website Optimizer records data so most of my reporting for A/B testing has to be produced via GA.

August 19, 2010 6:43 AM

Mehdi said:

As usual, great insights from you Shawn ;)

Am I crazy or you can compare 2 segments without displaying the "all visits" segment?

Next change from Google?

July 26, 2009 5:41 PM

steve said:

Any way to see multivariate tests?

July 27, 2009 11:33 AM

Shawn Purtell, Senior Web Analytics Engineer Author Profile Page said:

@Mehdi:

I've noticed that sometimes I can deselect the 'All Visits' segment and sometimes I can't. I haven't figured why exactly that happens yet, but if I unlock the mystery I'll be sure to blog about it.

July 27, 2009 1:12 PM

Shawn Purtell, Senior Web Analytics Engineer Author Profile Page said:

@Steve:

There are actually a few ways out there right now to view multivariate test data within Google Analytics. One involves virtual pageviews with Site Search, and another version involves the User Defined variable. Either way, you need to be able to get the combination data from the Google Website Optimizer cookie and send it over to Google Analytics. I'm planning on spelling out these options in a future post, but you can find out more about how it's done by visiting this post

July 27, 2009 1:16 PM

James Stratford @JRStratford on Twitter said:

That's a great use of Google Analytics and Website Analyzer. I'd never thought of the combination or knew that it would be so useful if coupled. It beats out my A/B testing script on my website. Though I use Google Analytics it's under utilized ATM, but thanks to you that will be changing.

I'll do a comparison between the tools and now that I think of it the big benefit is it won't utilize my server and that certainly won't hurt a thing.

Will be doing some more bookmarking since this is a great post and others need to know about it for sure.

July 27, 2009 8:42 PM

Trevor Claiborne said:

Nice write up, Shawn.

The only thing I've done differently is I don't base the segment on landing page. I make my segments to be anyone who visited the A page and anyone who visited the B page. Since the two pages should be mutually exclusive, this works fine, and I don't always run tests on landing pages.

July 27, 2009 11:13 PM

Shawn Purtell, Senior Web Analytics Engineer Author Profile Page said:

@James:

Thanks for reading and I hope you find this as useful as I have!

July 28, 2009 7:45 AM

Shawn Purtell, Senior Web Analytics Engineer Author Profile Page said:

@Trevor:

Thanks for the compliment. I tend to use the Page Dimension instead of Landing Page as well, and you'll see I actually reference them both in the article and in the screenshots. I think they both can be useful depending on the situation and how the pages interact with the rest of a given website.

July 28, 2009 7:48 AM

Nick said:

Hi Shawn,

Interesting post and a great way to review more detailed info on your experiments than WO itself provides. Seems like you could learn a lot using this technique that would help you design subsequent experiments.

However, I'm still wondering exactly how this kind of data is actionable in the immediate term. If you're mid-experiment, you'll most likely want to wait for WO to determine which of your experimental variations is the overall 'winner,' so reviewing the data along the way would satisfy the curiosity of a data geek such as myself, but I'm not sure I could do a lot with it immediately. Could you say a bit more about how this is actionable in the immediate term?

Thanks!

July 28, 2009 3:49 PM

Shawn Purtell, Senior Web Analytics Engineer Author Profile Page said:

@Nick:

The most immediate benefit of all is the ability to view multiple goals or in the case of ecommerce, transactions and revenue. This is information you simply just don't have within the Google Website Optimizer interface, and can be absolutely vital to Ecommerce sites. Ignoring revenue in a Google Website Optimizer test is a HUGE mistake.

Also, why would you always wait for WO to determine a winner? Google Website Optimizer uses some general statistical significance formulas, but by no means should you always wait for an experiment to be decided by the interface. Time is money, and often you can tell if an experiment has a successful iteration long before the interface will tell you.

That being said, it is also wise to not be too hasty. If you are testing a page across a wide variety of marketing sources and keywords, the overall conversion data within Google Website Optimizer can be misleading in the same way that site averages can be misleading. To give you an example, we've seen experiments where the overall conversion rate for our new page was much better than our old page, except in a specific campaign or ad group. By swapping to the new page without drilling down even a little, we could be actually hurting performance for a specific campaign and not even realizing it.

On the flip side, you may find that while Google Website Optimizer has not yet declared a winner, that for a particular campaign there is a clear winner. If that's the case, you can take action for that campaign while leaving the remaining traffic in a test.

Finally, I'm not necessarily suggesting that you should examine the data within Google Analytics every day. I'm simply saying that when it's time to look at the data, it's worth looking at the data within Google Analytics as well since it takes so little time to set up and you may find an insight or two. The insights you find will be determined by the nature of your website, your business, and your test.

July 28, 2009 4:31 PM

suhan said:

this article is awesome and it really helped me a lot thanx

July 29, 2009 8:31 AM

Shawn Purtell, Senior Web Analytics Engineer Author Profile Page said:

As an update, the mystery concerning the deselection of the 'All Visits' segment has been solved. It turns out Jeremy here at ROI has been doing some work on the Google Analytics Report Enhancer, and this was one of the things he was working on. Look for an update from him in the coming weeks.

August 5, 2009 8:58 AM

Kathryn said:

Thanks for the great post. This seems like a *much* easier way of tracking test results than setting up two different profiles -- but I read elsewhere that the Advanced Segments functionality doesn't work with the Funnel Visualization report, has that been your experience?

August 26, 2009 4:17 PM

Shawn Purtell, Senior Web Analytics Engineer Author Profile Page said:

@Kathryn,

Unfortunately, as you pointed out, Advanced Segments do not work with the Funnel Visualization report. There are a few other reports that also fall into this category. Most of the time, you can still get the data by using the Content reports (Navigation Summary, etc.) creatively, but currently there is no great way to reproduce the Funnel Navigation reports exactly without the separate profiles you mentioned.

September 1, 2009 10:01 AM

Geoff Jackson said:

Great tutorial. Rarely do I find that Google's Website Optimizer records data so most of my reporting for A/B testing has to be produced via GA.

August 19, 2010 6:43 AM

Mehdi said:

As usual, great insights from you Shawn ;)

Am I crazy or you can compare 2 segments without displaying the "all visits" segment?

Next change from Google?

July 26, 2009 5:41 PM

steve said:

Any way to see multivariate tests?

July 27, 2009 11:33 AM

Shawn Purtell, Senior Web Analytics Engineer Author Profile Page said:

@Mehdi:

I've noticed that sometimes I can deselect the 'All Visits' segment and sometimes I can't. I haven't figured why exactly that happens yet, but if I unlock the mystery I'll be sure to blog about it.

July 27, 2009 1:12 PM

Shawn Purtell, Senior Web Analytics Engineer Author Profile Page said:

@Steve:

There are actually a few ways out there right now to view multivariate test data within Google Analytics. One involves virtual pageviews with Site Search, and another version involves the User Defined variable. Either way, you need to be able to get the combination data from the Google Website Optimizer cookie and send it over to Google Analytics. I'm planning on spelling out these options in a future post, but you can find out more about how it's done by visiting this post

July 27, 2009 1:16 PM

James Stratford @JRStratford on Twitter said:

That's a great use of Google Analytics and Website Analyzer. I'd never thought of the combination or knew that it would be so useful if coupled. It beats out my A/B testing script on my website. Though I use Google Analytics it's under utilized ATM, but thanks to you that will be changing.

I'll do a comparison between the tools and now that I think of it the big benefit is it won't utilize my server and that certainly won't hurt a thing.

Will be doing some more bookmarking since this is a great post and others need to know about it for sure.

July 27, 2009 8:42 PM

Trevor Claiborne said:

Nice write up, Shawn.

The only thing I've done differently is I don't base the segment on landing page. I make my segments to be anyone who visited the A page and anyone who visited the B page. Since the two pages should be mutually exclusive, this works fine, and I don't always run tests on landing pages.

July 27, 2009 11:13 PM

Shawn Purtell, Senior Web Analytics Engineer Author Profile Page said:

@James:

Thanks for reading and I hope you find this as useful as I have!

July 28, 2009 7:45 AM

Shawn Purtell, Senior Web Analytics Engineer Author Profile Page said:

@Trevor:

Thanks for the compliment. I tend to use the Page Dimension instead of Landing Page as well, and you'll see I actually reference them both in the article and in the screenshots. I think they both can be useful depending on the situation and how the pages interact with the rest of a given website.

July 28, 2009 7:48 AM

Nick said:

Hi Shawn,

Interesting post and a great way to review more detailed info on your experiments than WO itself provides. Seems like you could learn a lot using this technique that would help you design subsequent experiments.

However, I'm still wondering exactly how this kind of data is actionable in the immediate term. If you're mid-experiment, you'll most likely want to wait for WO to determine which of your experimental variations is the overall 'winner,' so reviewing the data along the way would satisfy the curiosity of a data geek such as myself, but I'm not sure I could do a lot with it immediately. Could you say a bit more about how this is actionable in the immediate term?

Thanks!

July 28, 2009 3:49 PM

Shawn Purtell, Senior Web Analytics Engineer Author Profile Page said:

@Nick:

The most immediate benefit of all is the ability to view multiple goals or in the case of ecommerce, transactions and revenue. This is information you simply just don't have within the Google Website Optimizer interface, and can be absolutely vital to Ecommerce sites. Ignoring revenue in a Google Website Optimizer test is a HUGE mistake.

Also, why would you always wait for WO to determine a winner? Google Website Optimizer uses some general statistical significance formulas, but by no means should you always wait for an experiment to be decided by the interface. Time is money, and often you can tell if an experiment has a successful iteration long before the interface will tell you.

That being said, it is also wise to not be too hasty. If you are testing a page across a wide variety of marketing sources and keywords, the overall conversion data within Google Website Optimizer can be misleading in the same way that site averages can be misleading. To give you an example, we've seen experiments where the overall conversion rate for our new page was much better than our old page, except in a specific campaign or ad group. By swapping to the new page without drilling down even a little, we could be actually hurting performance for a specific campaign and not even realizing it.

On the flip side, you may find that while Google Website Optimizer has not yet declared a winner, that for a particular campaign there is a clear winner. If that's the case, you can take action for that campaign while leaving the remaining traffic in a test.

Finally, I'm not necessarily suggesting that you should examine the data within Google Analytics every day. I'm simply saying that when it's time to look at the data, it's worth looking at the data within Google Analytics as well since it takes so little time to set up and you may find an insight or two. The insights you find will be determined by the nature of your website, your business, and your test.

July 28, 2009 4:31 PM

suhan said:

this article is awesome and it really helped me a lot thanx

July 29, 2009 8:31 AM

Shawn Purtell, Senior Web Analytics Engineer Author Profile Page said:

As an update, the mystery concerning the deselection of the 'All Visits' segment has been solved. It turns out Jeremy here at ROI has been doing some work on the Google Analytics Report Enhancer, and this was one of the things he was working on. Look for an update from him in the coming weeks.

August 5, 2009 8:58 AM

Kathryn said:

Thanks for the great post. This seems like a *much* easier way of tracking test results than setting up two different profiles -- but I read elsewhere that the Advanced Segments functionality doesn't work with the Funnel Visualization report, has that been your experience?

August 26, 2009 4:17 PM

Shawn Purtell, Senior Web Analytics Engineer Author Profile Page said:

@Kathryn,

Unfortunately, as you pointed out, Advanced Segments do not work with the Funnel Visualization report. There are a few other reports that also fall into this category. Most of the time, you can still get the data by using the Content reports (Navigation Summary, etc.) creatively, but currently there is no great way to reproduce the Funnel Navigation reports exactly without the separate profiles you mentioned.

September 1, 2009 10:01 AM

Geoff Jackson said:

Great tutorial. Rarely do I find that Google's Website Optimizer records data so most of my reporting for A/B testing has to be produced via GA.

August 19, 2010 6:43 AM

Mehdi said:

As usual, great insights from you Shawn ;)

Am I crazy or you can compare 2 segments without displaying the "all visits" segment?

Next change from Google?

July 26, 2009 5:41 PM

steve said:

Any way to see multivariate tests?

July 27, 2009 11:33 AM

Shawn Purtell, Senior Web Analytics Engineer Author Profile Page said:

@Mehdi:

I've noticed that sometimes I can deselect the 'All Visits' segment and sometimes I can't. I haven't figured why exactly that happens yet, but if I unlock the mystery I'll be sure to blog about it.

July 27, 2009 1:12 PM

Shawn Purtell, Senior Web Analytics Engineer Author Profile Page said:

@Steve:

There are actually a few ways out there right now to view multivariate test data within Google Analytics. One involves virtual pageviews with Site Search, and another version involves the User Defined variable. Either way, you need to be able to get the combination data from the Google Website Optimizer cookie and send it over to Google Analytics. I'm planning on spelling out these options in a future post, but you can find out more about how it's done by visiting this post

July 27, 2009 1:16 PM

James Stratford @JRStratford on Twitter said:

That's a great use of Google Analytics and Website Analyzer. I'd never thought of the combination or knew that it would be so useful if coupled. It beats out my A/B testing script on my website. Though I use Google Analytics it's under utilized ATM, but thanks to you that will be changing.

I'll do a comparison between the tools and now that I think of it the big benefit is it won't utilize my server and that certainly won't hurt a thing.

Will be doing some more bookmarking since this is a great post and others need to know about it for sure.

July 27, 2009 8:42 PM

Trevor Claiborne said:

Nice write up, Shawn.

The only thing I've done differently is I don't base the segment on landing page. I make my segments to be anyone who visited the A page and anyone who visited the B page. Since the two pages should be mutually exclusive, this works fine, and I don't always run tests on landing pages.

July 27, 2009 11:13 PM

Shawn Purtell, Senior Web Analytics Engineer Author Profile Page said:

@James:

Thanks for reading and I hope you find this as useful as I have!

July 28, 2009 7:45 AM

Shawn Purtell, Senior Web Analytics Engineer Author Profile Page said:

@Trevor:

Thanks for the compliment. I tend to use the Page Dimension instead of Landing Page as well, and you'll see I actually reference them both in the article and in the screenshots. I think they both can be useful depending on the situation and how the pages interact with the rest of a given website.

July 28, 2009 7:48 AM

Nick said:

Hi Shawn,

Interesting post and a great way to review more detailed info on your experiments than WO itself provides. Seems like you could learn a lot using this technique that would help you design subsequent experiments.

However, I'm still wondering exactly how this kind of data is actionable in the immediate term. If you're mid-experiment, you'll most likely want to wait for WO to determine which of your experimental variations is the overall 'winner,' so reviewing the data along the way would satisfy the curiosity of a data geek such as myself, but I'm not sure I could do a lot with it immediately. Could you say a bit more about how this is actionable in the immediate term?

Thanks!

July 28, 2009 3:49 PM

Shawn Purtell, Senior Web Analytics Engineer Author Profile Page said:

@Nick:

The most immediate benefit of all is the ability to view multiple goals or in the case of ecommerce, transactions and revenue. This is information you simply just don't have within the Google Website Optimizer interface, and can be absolutely vital to Ecommerce sites. Ignoring revenue in a Google Website Optimizer test is a HUGE mistake.

Also, why would you always wait for WO to determine a winner? Google Website Optimizer uses some general statistical significance formulas, but by no means should you always wait for an experiment to be decided by the interface. Time is money, and often you can tell if an experiment has a successful iteration long before the interface will tell you.

That being said, it is also wise to not be too hasty. If you are testing a page across a wide variety of marketing sources and keywords, the overall conversion data within Google Website Optimizer can be misleading in the same way that site averages can be misleading. To give you an example, we've seen experiments where the overall conversion rate for our new page was much better than our old page, except in a specific campaign or ad group. By swapping to the new page without drilling down even a little, we could be actually hurting performance for a specific campaign and not even realizing it.

On the flip side, you may find that while Google Website Optimizer has not yet declared a winner, that for a particular campaign there is a clear winner. If that's the case, you can take action for that campaign while leaving the remaining traffic in a test.

Finally, I'm not necessarily suggesting that you should examine the data within Google Analytics every day. I'm simply saying that when it's time to look at the data, it's worth looking at the data within Google Analytics as well since it takes so little time to set up and you may find an insight or two. The insights you find will be determined by the nature of your website, your business, and your test.

July 28, 2009 4:31 PM

suhan said:

this article is awesome and it really helped me a lot thanx

July 29, 2009 8:31 AM

Shawn Purtell, Senior Web Analytics Engineer Author Profile Page said:

As an update, the mystery concerning the deselection of the 'All Visits' segment has been solved. It turns out Jeremy here at ROI has been doing some work on the Google Analytics Report Enhancer, and this was one of the things he was working on. Look for an update from him in the coming weeks.

August 5, 2009 8:58 AM

Kathryn said:

Thanks for the great post. This seems like a *much* easier way of tracking test results than setting up two different profiles -- but I read elsewhere that the Advanced Segments functionality doesn't work with the Funnel Visualization report, has that been your experience?

August 26, 2009 4:17 PM

Shawn Purtell, Senior Web Analytics Engineer Author Profile Page said:

@Kathryn,

Unfortunately, as you pointed out, Advanced Segments do not work with the Funnel Visualization report. There are a few other reports that also fall into this category. Most of the time, you can still get the data by using the Content reports (Navigation Summary, etc.) creatively, but currently there is no great way to reproduce the Funnel Navigation reports exactly without the separate profiles you mentioned.

September 1, 2009 10:01 AM

Geoff Jackson said:

Great tutorial. Rarely do I find that Google's Website Optimizer records data so most of my reporting for A/B testing has to be produced via GA.

August 19, 2010 6:43 AM

Mehdi said:

As usual, great insights from you Shawn ;)

Am I crazy or you can compare 2 segments without displaying the "all visits" segment?

Next change from Google?

July 26, 2009 5:41 PM

steve said:

Any way to see multivariate tests?

July 27, 2009 11:33 AM

Shawn Purtell, Senior Web Analytics Engineer Author Profile Page said:

@Mehdi:

I've noticed that sometimes I can deselect the 'All Visits' segment and sometimes I can't. I haven't figured why exactly that happens yet, but if I unlock the mystery I'll be sure to blog about it.

July 27, 2009 1:12 PM

Shawn Purtell, Senior Web Analytics Engineer Author Profile Page said:

@Steve:

There are actually a few ways out there right now to view multivariate test data within Google Analytics. One involves virtual pageviews with Site Search, and another version involves the User Defined variable. Either way, you need to be able to get the combination data from the Google Website Optimizer cookie and send it over to Google Analytics. I'm planning on spelling out these options in a future post, but you can find out more about how it's done by visiting this post

July 27, 2009 1:16 PM

James Stratford @JRStratford on Twitter said:

That's a great use of Google Analytics and Website Analyzer. I'd never thought of the combination or knew that it would be so useful if coupled. It beats out my A/B testing script on my website. Though I use Google Analytics it's under utilized ATM, but thanks to you that will be changing.

I'll do a comparison between the tools and now that I think of it the big benefit is it won't utilize my server and that certainly won't hurt a thing.

Will be doing some more bookmarking since this is a great post and others need to know about it for sure.

July 27, 2009 8:42 PM

Trevor Claiborne said:

Nice write up, Shawn.

The only thing I've done differently is I don't base the segment on landing page. I make my segments to be anyone who visited the A page and anyone who visited the B page. Since the two pages should be mutually exclusive, this works fine, and I don't always run tests on landing pages.

July 27, 2009 11:13 PM

Shawn Purtell, Senior Web Analytics Engineer Author Profile Page said:

@James:

Thanks for reading and I hope you find this as useful as I have!

July 28, 2009 7:45 AM

Shawn Purtell, Senior Web Analytics Engineer Author Profile Page said:

@Trevor:

Thanks for the compliment. I tend to use the Page Dimension instead of Landing Page as well, and you'll see I actually reference them both in the article and in the screenshots. I think they both can be useful depending on the situation and how the pages interact with the rest of a given website.

July 28, 2009 7:48 AM

Nick said:

Hi Shawn,

Interesting post and a great way to review more detailed info on your experiments than WO itself provides. Seems like you could learn a lot using this technique that would help you design subsequent experiments.

However, I'm still wondering exactly how this kind of data is actionable in the immediate term. If you're mid-experiment, you'll most likely want to wait for WO to determine which of your experimental variations is the overall 'winner,' so reviewing the data along the way would satisfy the curiosity of a data geek such as myself, but I'm not sure I could do a lot with it immediately. Could you say a bit more about how this is actionable in the immediate term?

Thanks!

July 28, 2009 3:49 PM

Shawn Purtell, Senior Web Analytics Engineer Author Profile Page said:

@Nick:

The most immediate benefit of all is the ability to view multiple goals or in the case of ecommerce, transactions and revenue. This is information you simply just don't have within the Google Website Optimizer interface, and can be absolutely vital to Ecommerce sites. Ignoring revenue in a Google Website Optimizer test is a HUGE mistake.

Also, why would you always wait for WO to determine a winner? Google Website Optimizer uses some general statistical significance formulas, but by no means should you always wait for an experiment to be decided by the interface. Time is money, and often you can tell if an experiment has a successful iteration long before the interface will tell you.

That being said, it is also wise to not be too hasty. If you are testing a page across a wide variety of marketing sources and keywords, the overall conversion data within Google Website Optimizer can be misleading in the same way that site averages can be misleading. To give you an example, we've seen experiments where the overall conversion rate for our new page was much better than our old page, except in a specific campaign or ad group. By swapping to the new page without drilling down even a little, we could be actually hurting performance for a specific campaign and not even realizing it.

On the flip side, you may find that while Google Website Optimizer has not yet declared a winner, that for a particular campaign there is a clear winner. If that's the case, you can take action for that campaign while leaving the remaining traffic in a test.

Finally, I'm not necessarily suggesting that you should examine the data within Google Analytics every day. I'm simply saying that when it's time to look at the data, it's worth looking at the data within Google Analytics as well since it takes so little time to set up and you may find an insight or two. The insights you find will be determined by the nature of your website, your business, and your test.

July 28, 2009 4:31 PM

suhan said:

this article is awesome and it really helped me a lot thanx

July 29, 2009 8:31 AM

Shawn Purtell, Senior Web Analytics Engineer Author Profile Page said:

As an update, the mystery concerning the deselection of the 'All Visits' segment has been solved. It turns out Jeremy here at ROI has been doing some work on the Google Analytics Report Enhancer, and this was one of the things he was working on. Look for an update from him in the coming weeks.

August 5, 2009 8:58 AM

Kathryn said:

Thanks for the great post. This seems like a *much* easier way of tracking test results than setting up two different profiles -- but I read elsewhere that the Advanced Segments functionality doesn't work with the Funnel Visualization report, has that been your experience?

August 26, 2009 4:17 PM

Shawn Purtell, Senior Web Analytics Engineer Author Profile Page said:

@Kathryn,

Unfortunately, as you pointed out, Advanced Segments do not work with the Funnel Visualization report. There are a few other reports that also fall into this category. Most of the time, you can still get the data by using the Content reports (Navigation Summary, etc.) creatively, but currently there is no great way to reproduce the Funnel Navigation reports exactly without the separate profiles you mentioned.

September 1, 2009 10:01 AM

Geoff Jackson said:

Great tutorial. Rarely do I find that Google's Website Optimizer records data so most of my reporting for A/B testing has to be produced via GA.

August 19, 2010 6:43 AM

Mehdi said:

As usual, great insights from you Shawn ;)

Am I crazy or you can compare 2 segments without displaying the "all visits" segment?

Next change from Google?

July 26, 2009 5:41 PM

steve said:

Any way to see multivariate tests?

July 27, 2009 11:33 AM

Shawn Purtell, Senior Web Analytics Engineer Author Profile Page said:

@Mehdi:

I've noticed that sometimes I can deselect the 'All Visits' segment and sometimes I can't. I haven't figured why exactly that happens yet, but if I unlock the mystery I'll be sure to blog about it.

July 27, 2009 1:12 PM

Shawn Purtell, Senior Web Analytics Engineer Author Profile Page said:

@Steve:

There are actually a few ways out there right now to view multivariate test data within Google Analytics. One involves virtual pageviews with Site Search, and another version involves the User Defined variable. Either way, you need to be able to get the combination data from the Google Website Optimizer cookie and send it over to Google Analytics. I'm planning on spelling out these options in a future post, but you can find out more about how it's done by visiting this post

July 27, 2009 1:16 PM

James Stratford @JRStratford on Twitter said:

That's a great use of Google Analytics and Website Analyzer. I'd never thought of the combination or knew that it would be so useful if coupled. It beats out my A/B testing script on my website. Though I use Google Analytics it's under utilized ATM, but thanks to you that will be changing.

I'll do a comparison between the tools and now that I think of it the big benefit is it won't utilize my server and that certainly won't hurt a thing.

Will be doing some more bookmarking since this is a great post and others need to know about it for sure.

July 27, 2009 8:42 PM

Trevor Claiborne said:

Nice write up, Shawn.

The only thing I've done differently is I don't base the segment on landing page. I make my segments to be anyone who visited the A page and anyone who visited the B page. Since the two pages should be mutually exclusive, this works fine, and I don't always run tests on landing pages.

July 27, 2009 11:13 PM

Shawn Purtell, Senior Web Analytics Engineer Author Profile Page said:

@James:

Thanks for reading and I hope you find this as useful as I have!

July 28, 2009 7:45 AM

Shawn Purtell, Senior Web Analytics Engineer Author Profile Page said:

@Trevor:

Thanks for the compliment. I tend to use the Page Dimension instead of Landing Page as well, and you'll see I actually reference them both in the article and in the screenshots. I think they both can be useful depending on the situation and how the pages interact with the rest of a given website.

July 28, 2009 7:48 AM

Nick said:

Hi Shawn,

Interesting post and a great way to review more detailed info on your experiments than WO itself provides. Seems like you could learn a lot using this technique that would help you design subsequent experiments.

However, I'm still wondering exactly how this kind of data is actionable in the immediate term. If you're mid-experiment, you'll most likely want to wait for WO to determine which of your experimental variations is the overall 'winner,' so reviewing the data along the way would satisfy the curiosity of a data geek such as myself, but I'm not sure I could do a lot with it immediately. Could you say a bit more about how this is actionable in the immediate term?

Thanks!

July 28, 2009 3:49 PM

Shawn Purtell, Senior Web Analytics Engineer Author Profile Page said:

@Nick:

The most immediate benefit of all is the ability to view multiple goals or in the case of ecommerce, transactions and revenue. This is information you simply just don't have within the Google Website Optimizer interface, and can be absolutely vital to Ecommerce sites. Ignoring revenue in a Google Website Optimizer test is a HUGE mistake.

Also, why would you always wait for WO to determine a winner? Google Website Optimizer uses some general statistical significance formulas, but by no means should you always wait for an experiment to be decided by the interface. Time is money, and often you can tell if an experiment has a successful iteration long before the interface will tell you.

That being said, it is also wise to not be too hasty. If you are testing a page across a wide variety of marketing sources and keywords, the overall conversion data within Google Website Optimizer can be misleading in the same way that site averages can be misleading. To give you an example, we've seen experiments where the overall conversion rate for our new page was much better than our old page, except in a specific campaign or ad group. By swapping to the new page without drilling down even a little, we could be actually hurting performance for a specific campaign and not even realizing it.

On the flip side, you may find that while Google Website Optimizer has not yet declared a winner, that for a particular campaign there is a clear winner. If that's the case, you can take action for that campaign while leaving the remaining traffic in a test.

Finally, I'm not necessarily suggesting that you should examine the data within Google Analytics every day. I'm simply saying that when it's time to look at the data, it's worth looking at the data within Google Analytics as well since it takes so little time to set up and you may find an insight or two. The insights you find will be determined by the nature of your website, your business, and your test.

July 28, 2009 4:31 PM

suhan said:

this article is awesome and it really helped me a lot thanx

July 29, 2009 8:31 AM

Shawn Purtell, Senior Web Analytics Engineer Author Profile Page said:

As an update, the mystery concerning the deselection of the 'All Visits' segment has been solved. It turns out Jeremy here at ROI has been doing some work on the Google Analytics Report Enhancer, and this was one of the things he was working on. Look for an update from him in the coming weeks.

August 5, 2009 8:58 AM

Kathryn said:

Thanks for the great post. This seems like a *much* easier way of tracking test results than setting up two different profiles -- but I read elsewhere that the Advanced Segments functionality doesn't work with the Funnel Visualization report, has that been your experience?

August 26, 2009 4:17 PM

Shawn Purtell, Senior Web Analytics Engineer Author Profile Page said:

@Kathryn,

Unfortunately, as you pointed out, Advanced Segments do not work with the Funnel Visualization report. There are a few other reports that also fall into this category. Most of the time, you can still get the data by using the Content reports (Navigation Summary, etc.) creatively, but currently there is no great way to reproduce the Funnel Navigation reports exactly without the separate profiles you mentioned.

September 1, 2009 10:01 AM

Geoff Jackson said:

Great tutorial. Rarely do I find that Google's Website Optimizer records data so most of my reporting for A/B testing has to be produced via GA.

August 19, 2010 6:43 AM

Mehdi said:

As usual, great insights from you Shawn ;)

Am I crazy or you can compare 2 segments without displaying the "all visits" segment?

Next change from Google?

July 26, 2009 5:41 PM

steve said:

Any way to see multivariate tests?

July 27, 2009 11:33 AM

Shawn Purtell, Senior Web Analytics Engineer Author Profile Page said:

@Mehdi:

I've noticed that sometimes I can deselect the 'All Visits' segment and sometimes I can't. I haven't figured why exactly that happens yet, but if I unlock the mystery I'll be sure to blog about it.

July 27, 2009 1:12 PM

Shawn Purtell, Senior Web Analytics Engineer Author Profile Page said:

@Steve:

There are actually a few ways out there right now to view multivariate test data within Google Analytics. One involves virtual pageviews with Site Search, and another version involves the User Defined variable. Either way, you need to be able to get the combination data from the Google Website Optimizer cookie and send it over to Google Analytics. I'm planning on spelling out these options in a future post, but you can find out more about how it's done by visiting this post

July 27, 2009 1:16 PM

James Stratford @JRStratford on Twitter said:

That's a great use of Google Analytics and Website Analyzer. I'd never thought of the combination or knew that it would be so useful if coupled. It beats out my A/B testing script on my website. Though I use Google Analytics it's under utilized ATM, but thanks to you that will be changing.

I'll do a comparison between the tools and now that I think of it the big benefit is it won't utilize my server and that certainly won't hurt a thing.

Will be doing some more bookmarking since this is a great post and others need to know about it for sure.

July 27, 2009 8:42 PM

Trevor Claiborne said:

Nice write up, Shawn.

The only thing I've done differently is I don't base the segment on landing page. I make my segments to be anyone who visited the A page and anyone who visited the B page. Since the two pages should be mutually exclusive, this works fine, and I don't always run tests on landing pages.

July 27, 2009 11:13 PM

Shawn Purtell, Senior Web Analytics Engineer Author Profile Page said:

@James:

Thanks for reading and I hope you find this as useful as I have!

July 28, 2009 7:45 AM

Shawn Purtell, Senior Web Analytics Engineer Author Profile Page said:

@Trevor:

Thanks for the compliment. I tend to use the Page Dimension instead of Landing Page as well, and you'll see I actually reference them both in the article and in the screenshots. I think they both can be useful depending on the situation and how the pages interact with the rest of a given website.

July 28, 2009 7:48 AM

Nick said:

Hi Shawn,

Interesting post and a great way to review more detailed info on your experiments than WO itself provides. Seems like you could learn a lot using this technique that would help you design subsequent experiments.

However, I'm still wondering exactly how this kind of data is actionable in the immediate term. If you're mid-experiment, you'll most likely want to wait for WO to determine which of your experimental variations is the overall 'winner,' so reviewing the data along the way would satisfy the curiosity of a data geek such as myself, but I'm not sure I could do a lot with it immediately. Could you say a bit more about how this is actionable in the immediate term?

Thanks!

July 28, 2009 3:49 PM

Shawn Purtell, Senior Web Analytics Engineer Author Profile Page said:

@Nick:

The most immediate benefit of all is the ability to view multiple goals or in the case of ecommerce, transactions and revenue. This is information you simply just don't have within the Google Website Optimizer interface, and can be absolutely vital to Ecommerce sites. Ignoring revenue in a Google Website Optimizer test is a HUGE mistake.

Also, why would you always wait for WO to determine a winner? Google Website Optimizer uses some general statistical significance formulas, but by no means should you always wait for an experiment to be decided by the interface. Time is money, and often you can tell if an experiment has a successful iteration long before the interface will tell you.

That being said, it is also wise to not be too hasty. If you are testing a page across a wide variety of marketing sources and keywords, the overall conversion data within Google Website Optimizer can be misleading in the same way that site averages can be misleading. To give you an example, we've seen experiments where the overall conversion rate for our new page was much better than our old page, except in a specific campaign or ad group. By swapping to the new page without drilling down even a little, we could be actually hurting performance for a specific campaign and not even realizing it.

On the flip side, you may find that while Google Website Optimizer has not yet declared a winner, that for a particular campaign there is a clear winner. If that's the case, you can take action for that campaign while leaving the remaining traffic in a test.

Finally, I'm not necessarily suggesting that you should examine the data within Google Analytics every day. I'm simply saying that when it's time to look at the data, it's worth looking at the data within Google Analytics as well since it takes so little time to set up and you may find an insight or two. The insights you find will be determined by the nature of your website, your business, and your test.

July 28, 2009 4:31 PM

suhan said:

this article is awesome and it really helped me a lot thanx

July 29, 2009 8:31 AM

Shawn Purtell, Senior Web Analytics Engineer Author Profile Page said:

As an update, the mystery concerning the deselection of the 'All Visits' segment has been solved. It turns out Jeremy here at ROI has been doing some work on the Google Analytics Report Enhancer, and this was one of the things he was working on. Look for an update from him in the coming weeks.

August 5, 2009 8:58 AM

Kathryn said:

Thanks for the great post. This seems like a *much* easier way of tracking test results than setting up two different profiles -- but I read elsewhere that the Advanced Segments functionality doesn't work with the Funnel Visualization report, has that been your experience?

August 26, 2009 4:17 PM

Shawn Purtell, Senior Web Analytics Engineer Author Profile Page said:

@Kathryn,

Unfortunately, as you pointed out, Advanced Segments do not work with the Funnel Visualization report. There are a few other reports that also fall into this category. Most of the time, you can still get the data by using the Content reports (Navigation Summary, etc.) creatively, but currently there is no great way to reproduce the Funnel Navigation reports exactly without the separate profiles you mentioned.

September 1, 2009 10:01 AM

Geoff Jackson said:

Great tutorial. Rarely do I find that Google's Website Optimizer records data so most of my reporting for A/B testing has to be produced via GA.

August 19, 2010 6:43 AM

Mehdi said:

As usual, great insights from you Shawn ;)

Am I crazy or you can compare 2 segments without displaying the "all visits" segment?

Next change from Google?

July 26, 2009 5:41 PM

steve said:

Any way to see multivariate tests?

July 27, 2009 11:33 AM

Shawn Purtell, Senior Web Analytics Engineer Author Profile Page said:

@Mehdi:

I've noticed that sometimes I can deselect the 'All Visits' segment and sometimes I can't. I haven't figured why exactly that happens yet, but if I unlock the mystery I'll be sure to blog about it.

July 27, 2009 1:12 PM

Shawn Purtell, Senior Web Analytics Engineer Author Profile Page said:

@Steve:

There are actually a few ways out there right now to view multivariate test data within Google Analytics. One involves virtual pageviews with Site Search, and another version involves the User Defined variable. Either way, you need to be able to get the combination data from the Google Website Optimizer cookie and send it over to Google Analytics. I'm planning on spelling out these options in a future post, but you can find out more about how it's done by visiting this post

July 27, 2009 1:16 PM

James Stratford @JRStratford on Twitter said:

That's a great use of Google Analytics and Website Analyzer. I'd never thought of the combination or knew that it would be so useful if coupled. It beats out my A/B testing script on my website. Though I use Google Analytics it's under utilized ATM, but thanks to you that will be changing.

I'll do a comparison between the tools and now that I think of it the big benefit is it won't utilize my server and that certainly won't hurt a thing.

Will be doing some more bookmarking since this is a great post and others need to know about it for sure.

July 27, 2009 8:42 PM

Trevor Claiborne said:

Nice write up, Shawn.

The only thing I've done differently is I don't base the segment on landing page. I make my segments to be anyone who visited the A page and anyone who visited the B page. Since the two pages should be mutually exclusive, this works fine, and I don't always run tests on landing pages.

July 27, 2009 11:13 PM

Shawn Purtell, Senior Web Analytics Engineer Author Profile Page said:

@James:

Thanks for reading and I hope you find this as useful as I have!

July 28, 2009 7:45 AM

Shawn Purtell, Senior Web Analytics Engineer Author Profile Page said:

@Trevor:

Thanks for the compliment. I tend to use the Page Dimension instead of Landing Page as well, and you'll see I actually reference them both in the article and in the screenshots. I think they both can be useful depending on the situation and how the pages interact with the rest of a given website.

July 28, 2009 7:48 AM

Nick said:

Hi Shawn,

Interesting post and a great way to review more detailed info on your experiments than WO itself provides. Seems like you could learn a lot using this technique that would help you design subsequent experiments.

However, I'm still wondering exactly how this kind of data is actionable in the immediate term. If you're mid-experiment, you'll most likely want to wait for WO to determine which of your experimental variations is the overall 'winner,' so reviewing the data along the way would satisfy the curiosity of a data geek such as myself, but I'm not sure I could do a lot with it immediately. Could you say a bit more about how this is actionable in the immediate term?

Thanks!

July 28, 2009 3:49 PM

Shawn Purtell, Senior Web Analytics Engineer Author Profile Page said:

@Nick:

The most immediate benefit of all is the ability to view multiple goals or in the case of ecommerce, transactions and revenue. This is information you simply just don't have within the Google Website Optimizer interface, and can be absolutely vital to Ecommerce sites. Ignoring revenue in a Google Website Optimizer test is a HUGE mistake.

Also, why would you always wait for WO to determine a winner? Google Website Optimizer uses some general statistical significance formulas, but by no means should you always wait for an experiment to be decided by the interface. Time is money, and often you can tell if an experiment has a successful iteration long before the interface will tell you.

That being said, it is also wise to not be too hasty. If you are testing a page across a wide variety of marketing sources and keywords, the overall conversion data within Google Website Optimizer can be misleading in the same way that site averages can be misleading. To give you an example, we've seen experiments where the overall conversion rate for our new page was much better than our old page, except in a specific campaign or ad group. By swapping to the new page without drilling down even a little, we could be actually hurting performance for a specific campaign and not even realizing it.

On the flip side, you may find that while Google Website Optimizer has not yet declared a winner, that for a particular campaign there is a clear winner. If that's the case, you can take action for that campaign while leaving the remaining traffic in a test.

Finally, I'm not necessarily suggesting that you should examine the data within Google Analytics every day. I'm simply saying that when it's time to look at the data, it's worth looking at the data within Google Analytics as well since it takes so little time to set up and you may find an insight or two. The insights you find will be determined by the nature of your website, your business, and your test.

July 28, 2009 4:31 PM

suhan said:

this article is awesome and it really helped me a lot thanx

July 29, 2009 8:31 AM

Shawn Purtell, Senior Web Analytics Engineer Author Profile Page said:

As an update, the mystery concerning the deselection of the 'All Visits' segment has been solved. It turns out Jeremy here at ROI has been doing some work on the Google Analytics Report Enhancer, and this was one of the things he was working on. Look for an update from him in the coming weeks.

August 5, 2009 8:58 AM

Kathryn said:

Thanks for the great post. This seems like a *much* easier way of tracking test results than setting up two different profiles -- but I read elsewhere that the Advanced Segments functionality doesn't work with the Funnel Visualization report, has that been your experience?

August 26, 2009 4:17 PM

Shawn Purtell, Senior Web Analytics Engineer Author Profile Page said:

@Kathryn,

Unfortunately, as you pointed out, Advanced Segments do not work with the Funnel Visualization report. There are a few other reports that also fall into this category. Most of the time, you can still get the data by using the Content reports (Navigation Summary, etc.) creatively, but currently there is no great way to reproduce the Funnel Navigation reports exactly without the separate profiles you mentioned.

September 1, 2009 10:01 AM

Geoff Jackson said:

Great tutorial. Rarely do I find that Google's Website Optimizer records data so most of my reporting for A/B testing has to be produced via GA.

August 19, 2010 6:43 AM

Mehdi said:

As usual, great insights from you Shawn ;)

Am I crazy or you can compare 2 segments without displaying the "all visits" segment?

Next change from Google?

July 26, 2009 5:41 PM

steve said:

Any way to see multivariate tests?

July 27, 2009 11:33 AM

Shawn Purtell, Senior Web Analytics Engineer Author Profile Page said:

@Mehdi:

I've noticed that sometimes I can deselect the 'All Visits' segment and sometimes I can't. I haven't figured why exactly that happens yet, but if I unlock the mystery I'll be sure to blog about it.

July 27, 2009 1:12 PM

Shawn Purtell, Senior Web Analytics Engineer Author Profile Page said:

@Steve:

There are actually a few ways out there right now to view multivariate test data within Google Analytics. One involves virtual pageviews with Site Search, and another version involves the User Defined variable. Either way, you need to be able to get the combination data from the Google Website Optimizer cookie and send it over to Google Analytics. I'm planning on spelling out these options in a future post, but you can find out more about how it's done by visiting this post

July 27, 2009 1:16 PM

James Stratford @JRStratford on Twitter said:

That's a great use of Google Analytics and Website Analyzer. I'd never thought of the combination or knew that it would be so useful if coupled. It beats out my A/B testing script on my website. Though I use Google Analytics it's under utilized ATM, but thanks to you that will be changing.

I'll do a comparison between the tools and now that I think of it the big benefit is it won't utilize my server and that certainly won't hurt a thing.

Will be doing some more bookmarking since this is a great post and others need to know about it for sure.

July 27, 2009 8:42 PM

Trevor Claiborne said:

Nice write up, Shawn.

The only thing I've done differently is I don't base the segment on landing page. I make my segments to be anyone who visited the A page and anyone who visited the B page. Since the two pages should be mutually exclusive, this works fine, and I don't always run tests on landing pages.

July 27, 2009 11:13 PM

Shawn Purtell, Senior Web Analytics Engineer Author Profile Page said:

@James:

Thanks for reading and I hope you find this as useful as I have!

July 28, 2009 7:45 AM

Shawn Purtell, Senior Web Analytics Engineer Author Profile Page said:

@Trevor:

Thanks for the compliment. I tend to use the Page Dimension instead of Landing Page as well, and you'll see I actually reference them both in the article and in the screenshots. I think they both can be useful depending on the situation and how the pages interact with the rest of a given website.

July 28, 2009 7:48 AM

Nick said:

Hi Shawn,

Interesting post and a great way to review more detailed info on your experiments than WO itself provides. Seems like you could learn a lot using this technique that would help you design subsequent experiments.

However, I'm still wondering exactly how this kind of data is actionable in the immediate term. If you're mid-experiment, you'll most likely want to wait for WO to determine which of your experimental variations is the overall 'winner,' so reviewing the data along the way would satisfy the curiosity of a data geek such as myself, but I'm not sure I could do a lot with it immediately. Could you say a bit more about how this is actionable in the immediate term?

Thanks!

July 28, 2009 3:49 PM

Shawn Purtell, Senior Web Analytics Engineer Author Profile Page said:

@Nick:

The most immediate benefit of all is the ability to view multiple goals or in the case of ecommerce, transactions and revenue. This is information you simply just don't have within the Google Website Optimizer interface, and can be absolutely vital to Ecommerce sites. Ignoring revenue in a Google Website Optimizer test is a HUGE mistake.

Also, why would you always wait for WO to determine a winner? Google Website Optimizer uses some general statistical significance formulas, but by no means should you always wait for an experiment to be decided by the interface. Time is money, and often you can tell if an experiment has a successful iteration long before the interface will tell you.

That being said, it is also wise to not be too hasty. If you are testing a page across a wide variety of marketing sources and keywords, the overall conversion data within Google Website Optimizer can be misleading in the same way that site averages can be misleading. To give you an example, we've seen experiments where the overall conversion rate for our new page was much better than our old page, except in a specific campaign or ad group. By swapping to the new page without drilling down even a little, we could be actually hurting performance for a specific campaign and not even realizing it.

On the flip side, you may find that while Google Website Optimizer has not yet declared a winner, that for a particular campaign there is a clear winner. If that's the case, you can take action for that campaign while leaving the remaining traffic in a test.

Finally, I'm not necessarily suggesting that you should examine the data within Google Analytics every day. I'm simply saying that when it's time to look at the data, it's worth looking at the data within Google Analytics as well since it takes so little time to set up and you may find an insight or two. The insights you find will be determined by the nature of your website, your business, and your test.

July 28, 2009 4:31 PM

suhan said:

this article is awesome and it really helped me a lot thanx

July 29, 2009 8:31 AM

Shawn Purtell, Senior Web Analytics Engineer Author Profile Page said:

As an update, the mystery concerning the deselection of the 'All Visits' segment has been solved. It turns out Jeremy here at ROI has been doing some work on the Google Analytics Report Enhancer, and this was one of the things he was working on. Look for an update from him in the coming weeks.

August 5, 2009 8:58 AM

Kathryn said:

Thanks for the great post. This seems like a *much* easier way of tracking test results than setting up two different profiles -- but I read elsewhere that the Advanced Segments functionality doesn't work with the Funnel Visualization report, has that been your experience?

August 26, 2009 4:17 PM

Shawn Purtell, Senior Web Analytics Engineer Author Profile Page said:

@Kathryn,

Unfortunately, as you pointed out, Advanced Segments do not work with the Funnel Visualization report. There are a few other reports that also fall into this category. Most of the time, you can still get the data by using the Content reports (Navigation Summary, etc.) creatively, but currently there is no great way to reproduce the Funnel Navigation reports exactly without the separate profiles you mentioned.

September 1, 2009 10:01 AM

Geoff Jackson said:

Great tutorial. Rarely do I find that Google's Website Optimizer records data so most of my reporting for A/B testing has to be produced via GA.

August 19, 2010 6:43 AM

Mehdi said:

As usual, great insights from you Shawn ;)

Am I crazy or you can compare 2 segments without displaying the "all visits" segment?

Next change from Google?

July 26, 2009 5:41 PM

steve said:

Any way to see multivariate tests?

July 27, 2009 11:33 AM

Shawn Purtell, Senior Web Analytics Engineer Author Profile Page said:

@Mehdi:

I've noticed that sometimes I can deselect the 'All Visits' segment and sometimes I can't. I haven't figured why exactly that happens yet, but if I unlock the mystery I'll be sure to blog about it.

July 27, 2009 1:12 PM

Shawn Purtell, Senior Web Analytics Engineer Author Profile Page said:

@Steve:

There are actually a few ways out there right now to view multivariate test data within Google Analytics. One involves virtual pageviews with Site Search, and another version involves the User Defined variable. Either way, you need to be able to get the combination data from the Google Website Optimizer cookie and send it over to Google Analytics. I'm planning on spelling out these options in a future post, but you can find out more about how it's done by visiting this post

July 27, 2009 1:16 PM

James Stratford @JRStratford on Twitter said:

That's a great use of Google Analytics and Website Analyzer. I'd never thought of the combination or knew that it would be so useful if coupled. It beats out my A/B testing script on my website. Though I use Google Analytics it's under utilized ATM, but thanks to you that will be changing.

I'll do a comparison between the tools and now that I think of it the big benefit is it won't utilize my server and that certainly won't hurt a thing.

Will be doing some more bookmarking since this is a great post and others need to know about it for sure.

July 27, 2009 8:42 PM

Trevor Claiborne said:

Nice write up, Shawn.

The only thing I've done differently is I don't base the segment on landing page. I make my segments to be anyone who visited the A page and anyone who visited the B page. Since the two pages should be mutually exclusive, this works fine, and I don't always run tests on landing pages.

July 27, 2009 11:13 PM

Shawn Purtell, Senior Web Analytics Engineer Author Profile Page said:

@James:

Thanks for reading and I hope you find this as useful as I have!

July 28, 2009 7:45 AM

Shawn Purtell, Senior Web Analytics Engineer Author Profile Page said:

@Trevor:

Thanks for the compliment. I tend to use the Page Dimension instead of Landing Page as well, and you'll see I actually reference them both in the article and in the screenshots. I think they both can be useful depending on the situation and how the pages interact with the rest of a given website.

July 28, 2009 7:48 AM

Nick said:

Hi Shawn,

Interesting post and a great way to review more detailed info on your experiments than WO itself provides. Seems like you could learn a lot using this technique that would help you design subsequent experiments.

However, I'm still wondering exactly how this kind of data is actionable in the immediate term. If you're mid-experiment, you'll most likely want to wait for WO to determine which of your experimental variations is the overall 'winner,' so reviewing the data along the way would satisfy the curiosity of a data geek such as myself, but I'm not sure I could do a lot with it immediately. Could you say a bit more about how this is actionable in the immediate term?

Thanks!

July 28, 2009 3:49 PM

Shawn Purtell, Senior Web Analytics Engineer Author Profile Page said:

@Nick:

The most immediate benefit of all is the ability to view multiple goals or in the case of ecommerce, transactions and revenue. This is information you simply just don't have within the Google Website Optimizer interface, and can be absolutely vital to Ecommerce sites. Ignoring revenue in a Google Website Optimizer test is a HUGE mistake.

Also, why would you always wait for WO to determine a winner? Google Website Optimizer uses some general statistical significance formulas, but by no means should you always wait for an experiment to be decided by the interface. Time is money, and often you can tell if an experiment has a successful iteration long before the interface will tell you.

That being said, it is also wise to not be too hasty. If you are testing a page across a wide variety of marketing sources and keywords, the overall conversion data within Google Website Optimizer can be misleading in the same way that site averages can be misleading. To give you an example, we've seen experiments where the overall conversion rate for our new page was much better than our old page, except in a specific campaign or ad group. By swapping to the new page without drilling down even a little, we could be actually hurting performance for a specific campaign and not even realizing it.

On the flip side, you may find that while Google Website Optimizer has not yet declared a winner, that for a particular campaign there is a clear winner. If that's the case, you can take action for that campaign while leaving the remaining traffic in a test.

Finally, I'm not necessarily suggesting that you should examine the data within Google Analytics every day. I'm simply saying that when it's time to look at the data, it's worth looking at the data within Google Analytics as well since it takes so little time to set up and you may find an insight or two. The insights you find will be determined by the nature of your website, your business, and your test.

July 28, 2009 4:31 PM

suhan said:

this article is awesome and it really helped me a lot thanx

July 29, 2009 8:31 AM

Shawn Purtell, Senior Web Analytics Engineer Author Profile Page said:

As an update, the mystery concerning the deselection of the 'All Visits' segment has been solved. It turns out Jeremy here at ROI has been doing some work on the Google Analytics Report Enhancer, and this was one of the things he was working on. Look for an update from him in the coming weeks.

August 5, 2009 8:58 AM

Kathryn said:

Thanks for the great post. This seems like a *much* easier way of tracking test results than setting up two different profiles -- but I read elsewhere that the Advanced Segments functionality doesn't work with the Funnel Visualization report, has that been your experience?

August 26, 2009 4:17 PM

Shawn Purtell, Senior Web Analytics Engineer Author Profile Page said:

@Kathryn,

Unfortunately, as you pointed out, Advanced Segments do not work with the Funnel Visualization report. There are a few other reports that also fall into this category. Most of the time, you can still get the data by using the Content reports (Navigation Summary, etc.) creatively, but currently there is no great way to reproduce the Funnel Navigation reports exactly without the separate profiles you mentioned.

September 1, 2009 10:01 AM

Geoff Jackson said:

Great tutorial. Rarely do I find that Google's Website Optimizer records data so most of my reporting for A/B testing has to be produced via GA.

August 19, 2010 6:43 AM

Mehdi said:

As usual, great insights from you Shawn ;)

Am I crazy or you can compare 2 segments without displaying the "all visits" segment?

Next change from Google?

July 26, 2009 5:41 PM

steve said:

Any way to see multivariate tests?

July 27, 2009 11:33 AM

Shawn Purtell, Senior Web Analytics Engineer Author Profile Page said:

@Mehdi:

I've noticed that sometimes I can deselect the 'All Visits' segment and sometimes I can't. I haven't figured why exactly that happens yet, but if I unlock the mystery I'll be sure to blog about it.

July 27, 2009 1:12 PM

Shawn Purtell, Senior Web Analytics Engineer Author Profile Page said:

@Steve:

There are actually a few ways out there right now to view multivariate test data within Google Analytics. One involves virtual pageviews with Site Search, and another version involves the User Defined variable. Either way, you need to be able to get the combination data from the Google Website Optimizer cookie and send it over to Google Analytics. I'm planning on spelling out these options in a future post, but you can find out more about how it's done by visiting this post

July 27, 2009 1:16 PM

James Stratford @JRStratford on Twitter said:

That's a great use of Google Analytics and Website Analyzer. I'd never thought of the combination or knew that it would be so useful if coupled. It beats out my A/B testing script on my website. Though I use Google Analytics it's under utilized ATM, but thanks to you that will be changing.

I'll do a comparison between the tools and now that I think of it the big benefit is it won't utilize my server and that certainly won't hurt a thing.

Will be doing some more bookmarking since this is a great post and others need to know about it for sure.

July 27, 2009 8:42 PM

Trevor Claiborne said:

Nice write up, Shawn.

The only thing I've done differently is I don't base the segment on landing page. I make my segments to be anyone who visited the A page and anyone who visited the B page. Since the two pages should be mutually exclusive, this works fine, and I don't always run tests on landing pages.

July 27, 2009 11:13 PM

Shawn Purtell, Senior Web Analytics Engineer Author Profile Page said:

@James:

Thanks for reading and I hope you find this as useful as I have!

July 28, 2009 7:45 AM

Shawn Purtell, Senior Web Analytics Engineer Author Profile Page said:

@Trevor:

Thanks for the compliment. I tend to use the Page Dimension instead of Landing Page as well, and you'll see I actually reference them both in the article and in the screenshots. I think they both can be useful depending on the situation and how the pages interact with the rest of a given website.

July 28, 2009 7:48 AM

Nick said:

Hi Shawn,

Interesting post and a great way to review more detailed info on your experiments than WO itself provides. Seems like you could learn a lot using this technique that would help you design subsequent experiments.

However, I'm still wondering exactly how this kind of data is actionable in the immediate term. If you're mid-experiment, you'll most likely want to wait for WO to determine which of your experimental variations is the overall 'winner,' so reviewing the data along the way would satisfy the curiosity of a data geek such as myself, but I'm not sure I could do a lot with it immediately. Could you say a bit more about how this is actionable in the immediate term?

Thanks!

July 28, 2009 3:49 PM

Shawn Purtell, Senior Web Analytics Engineer Author Profile Page said:

@Nick:

The most immediate benefit of all is the ability to view multiple goals or in the case of ecommerce, transactions and revenue. This is information you simply just don't have within the Google Website Optimizer interface, and can be absolutely vital to Ecommerce sites. Ignoring revenue in a Google Website Optimizer test is a HUGE mistake.

Also, why would you always wait for WO to determine a winner? Google Website Optimizer uses some general statistical significance formulas, but by no means should you always wait for an experiment to be decided by the interface. Time is money, and often you can tell if an experiment has a successful iteration long before the interface will tell you.

That being said, it is also wise to not be too hasty. If you are testing a page across a wide variety of marketing sources and keywords, the overall conversion data within Google Website Optimizer can be misleading in the same way that site averages can be misleading. To give you an example, we've seen experiments where the overall conversion rate for our new page was much better than our old page, except in a specific campaign or ad group. By swapping to the new page without drilling down even a little, we could be actually hurting performance for a specific campaign and not even realizing it.

On the flip side, you may find that while Google Website Optimizer has not yet declared a winner, that for a particular campaign there is a clear winner. If that's the case, you can take action for that campaign while leaving the remaining traffic in a test.

Finally, I'm not necessarily suggesting that you should examine the data within Google Analytics every day. I'm simply saying that when it's time to look at the data, it's worth looking at the data within Google Analytics as well since it takes so little time to set up and you may find an insight or two. The insights you find will be determined by the nature of your website, your business, and your test.

July 28, 2009 4:31 PM

suhan said:

this article is awesome and it really helped me a lot thanx

July 29, 2009 8:31 AM

Shawn Purtell, Senior Web Analytics Engineer Author Profile Page said:

As an update, the mystery concerning the deselection of the 'All Visits' segment has been solved. It turns out Jeremy here at ROI has been doing some work on the Google Analytics Report Enhancer, and this was one of the things he was working on. Look for an update from him in the coming weeks.

August 5, 2009 8:58 AM

Kathryn said:

Thanks for the great post. This seems like a *much* easier way of tracking test results than setting up two different profiles -- but I read elsewhere that the Advanced Segments functionality doesn't work with the Funnel Visualization report, has that been your experience?

August 26, 2009 4:17 PM

Shawn Purtell, Senior Web Analytics Engineer Author Profile Page said:

@Kathryn,

Unfortunately, as you pointed out, Advanced Segments do not work with the Funnel Visualization report. There are a few other reports that also fall into this category. Most of the time, you can still get the data by using the Content reports (Navigation Summary, etc.) creatively, but currently there is no great way to reproduce the Funnel Navigation reports exactly without the separate profiles you mentioned.

September 1, 2009 10:01 AM

Geoff Jackson said:

Great tutorial. Rarely do I find that Google's Website Optimizer records data so most of my reporting for A/B testing has to be produced via GA.

August 19, 2010 6:43 AM

Post Your Comments

© 2002-2014 ROI Revolution, Inc. All rights reserved.