This is a great help, Michael. I've been using Jim's filters for several months now, and while they are helpful, there are definitely some shortcomings as you mentioned (not the least of which is getting conversions attributed to the proper term). I noticed in your screen shot that your solution attributed transactions and revenue to the terms, but I didn't see any goals. Is it safe to assume that this works with goals as well?

@Tyson: Yep. Definitely works with goals. The profiles in the screenshots just didn't have any goals configured (and they're Photoshopped to protect the innocent ;).

Let me know if you implement the script yourself. I'd love to hear how it goes for you. We've had it in place for a few weeks and we're already narrowing down our broad match phrases. It's awesome.

Google analytics should pay you for this script and integrate it. Knowing exact terms for every broad match is a great incentive to use Google Analytics.

Would it be possible to use this script with other analytics packages? I assume the technique would be similar regardless of the tool.

@Ben: Thanks for the kind words. :) It might be possible to use something similar to this script, but because it uses the proprietary __utmSetVar function to keep track of the exact search queries, you'd need to change it around a little bit. Feel free to use it as you see fit, but if you do change it around a bit, let me know so I can share in the wealth. ;)

Hi Michael,

Great stuff, an elegant solution to the inconsistencies of the filters. On that note, I can't take all the credit for the filter-based solution, it was actually Brian Clifton who developed it, I just blogged off the back of it :)

Also I'm no longer with GA Experts and haven't been writing the blog for a while. Good to see ROI Revolution going from strength to strength though (No. 7 Analytics blog!). Keep up the great work.

Jim

Hey Jim,
The things I do aren't often called elegant, so this must be a good one. Thanks for the nice comment, and I've updated the post to reflect Brian's involvement with your filter set.

Hope all is well with you and yours since your move from Omega. If fatherhood has been anything for you like it has for me, you've been just about as busy at home as you have in the office, eh?

I really liked the IP address filter and the jury is still out on whether IP addresses constitute private data, but I do understand why Google took the stance they did.

Personally, all is well here. The boy is sucking the life out of me - in the nicest possible way, I'm sure you know what I mean :)

Hello Michael, I've also been using the keyword filter from your blog. This new idea of getting the real PPC keywords into the user defined segmentation variable is great! It will be really helpful for analyzing the data. Since I have no easy access to the code of the site I'm testing with (and I wanted to test this quickly :)), I used a filter to get the real keyword data into the user defined segmentation variable. I've been playing around with filters in Google Analytics lately and I'm really enthusiastic about them. I was wondering what the most important shortcomings are which you are experiencing with filters? And does this apply for filters in general or just the keyword filter? Kind regards.

@Simone: Theoretically, you could use a filter to grab the search query from the Google Analytics filter field. Essentially, that's what the guys at Omega did. You could then store that data right into the user defined variable.

There's one main reason why I don't recommend doing this. Using a filter in Google Analytics does not actively set a cookie in the visitor's browser, so if they happen to come back via another method--one that does not have a q or a p query parameter and a search query--then the visitor will not be tagged as such.

Just to follow up on that question, Michael, you are saying that using a filter-based method would cause problems for latent conversion tracking? So the cookie would not get set on the initial visit, to mark the user as a PPC visitor? I'm not sure I understand.

@Tyson: Well, the visitor's standard GA cookies would be set. These tag them as a CPC visitor. This is actually mandatory, because my script will only work if the visitor comes in from a manually tagged CPC ad, or from an auto-tagged AdWords ad. That's not the problem.

The issue is that the exact query that the visitor used to arrive at your site is available in document.referrer only on their initial visit to the site. So, if you use a filter to grab that data from Analytics' Referral field, it will only last for their first session and until they come back to your site. At that point, the query no longer exists in the Referral field.

Therefore, it seemed necessary to set a cookie that would persist across multiple sessions. Thankfully, Google Analytics offers the usage of the User Defined cookie, and the nice __utmSetVar function to set it.

Hopefully that sets it straight. :)

Perfectly, thank you much!

Thank you very much for the explanation!

Michael, I am finally ready to use this, but -- once the sleuth() query parses the URI and find cpc or ppc, it has to know to execute utmSetVar. Enter your script.

Where does it go? On every page, above the GA code? On a separate page? If on a separate page, does that page have to have a special name?

BTW, I too have been using Jim's filters, and they are wonderful for negative keywords, but this will be truly awesome. Now, show the non-tech world how to do it. Or as you once said to me on the phone, "I just love Google Analytics, and I want everyone to be able to use them."

Hey Robbin! Your wish is my command. Let's see if I can pull up the old "expert blinders" for half a second.

Personally, I think it's easier to save the JavaScript to its own file and then call it via a JavaScript include (as in my example). Then just replace your standard Google Analytics code with the one I've provided above, either in your common GA include or on every single page of the site. sleuth() will run every single time someone hits the page, but will only set a new User Defined cookie if and only if the following circumstances are met:

• the visitor has arrived with a search engine query in tow, and


• the visitor has arrived via an auto-tagged AdWords or manually tagged cpc ad

Sounds like too much work to me, though. I'd just put it on every page.

Man, this is good info. Im a relatively new blogger still on the learning curve with this stuff. Sites like this are a goldmine to me. Thanks for putting this information out!

Eric

Glad to be of service, Eric. Add us to your blog reader of choice, because there's lots more to mine in the Google Analytics quarry. :)

When going to the "Visitors" > "User Defined" Screen of Google Analytics...

What does the "(not set)" amount in the "User Defined Value" column represent?

I'm guessing that it's equal to:

"All visitors that did not arrive by clicking on an AdWords ad"

PLUS

"All repeat visitors who originally arrived through an AdWords Ad".

@Ed: (not set) is going to include visits from:

"All visitors that did not arrive by clicking on an AdWords ad"

PLUS

"All visitors that arrived from an AdWords ad, but did not have the exact keyword information within the q or p query parameter of their internet search (this includes AOL visits, for example)"

PLUS

"All visitors that arrived from an AdWords ad, but whose keyword information did not make it through to the DOM's document.referrer"

Your second item, "All repeat visitors who originally arrived through an AdWords Ad" should still show up with the exact keyword in the user-defined field, even if they come back to the site through some other means.

It's normal, even when segmenting user defined by AdWords visits, to see (not set). Hope this helps explain why.

We're still working on trying to include as many exact queries as possible.

Hi,

After implementing this, our bounce rate for the 3 paid engines appears abnormally low. Has anyone else noticed this or do we have something coded incorrectly?

Thanks in advance.

-Matt

@Matt,

We haven't seen any dips in bounce rate due to the script. The script up on www.signaturehardware.com seems correct. Has the change affected any other metrics aside from bounce rate?

Michael,

Thanks for checking. Our average time on site for the 3 paid engines has also dipped at the same time. Bounce rate went from around 30% to 10% and average time on site went from about 5:30 to below 4 minutes. All other metrics appear the same. Note that this is only for Google Adwords, Yahoo PPC, and MSN PPC. All other metrics appear fine for our natural traffic and shopping comparison engines.

-Matt

Thanks for the script, but may I ask you if it is safe to use it ?? The script reads the visitors cookies and I am wondering if Google might deal with it as a malware and mark the site "This website may harm your computer"

@Wisam,
By your definition, then, Google Analytics itself is malware. :) As far as I know, as long as you're accessing only cookies that are written your your domain (first party cookies), there shouldn't be an issue.

This article is very helpful in understanding. The filter I was using has some Loopholes. I noticed in your screen shot that your solution attributed transactions and revenue to the terms. I hope it will be very beneficial.

Hello,

Does your script works the same way with the New GA? does it tracks sales by the exact keyword?

Where would I go to see the sales made by the exact keyword on the new GA interface?

Thank you very much, I appreciate your help and time.

Manny

@Deepak: Thanks! I'm glad you're finding the tool helpful.

@Manny: You can certainly find the same exact keyword information in the new interface. Just go to Visitors > User Defined, and then switch to the Ecommerce tab.

If you're interested in matching up exact keyword data to each individual transaction, you may want to check out Shawn's article on matching specific transactions to specific keywords. You can include the user-defined field in one of Shawn's filters, and you can match up the bid term AND the search term with your transactions.

Is it possible to read the bid term from a cookie and save it to a hidden field in the order form?

This would allow me to display the bid term directly in our order processing system. I know I could use the transaction matching technique above to get a similar result but I'd like to have the information in our own system if possible?

Thank you for the great script.

Will this track yahoo ppc referrers as well? Or, does another step need to be taken?

Much tks

@Darwin: The bid term is stored in a cookie, and cookies are accessible via Javascript. It sounds do-able, but be aware of Google Analytics' Terms of Service, specifically Article 7. I'll leave it at that. :)

@Bob: Yep, Yahoo! Search Marketing should be tracked, too, so long as it's tagged properly.

Hi Michael,
We implemented the Exact Keyword Tracking Script on a client site and it disrupted the GA code. Essentially we lost 2 days worth of data. How can we trouble shoot this because I would love to be able to implement this tracking script.

Thanks much!

@Nicole: Can you elaborate a bit on exactly how the script disrupted the standard Google Analytics code? It seems very odd that you would lose all data for the site.

Hi Michael,
After we adjusted the code with the Keyword Script we did not collect data for two days. Not sure why and I'm not the technical person so I can't really elaborate. Do you need to put anything in the "sleuth()" script? Any suggestions for trouble shooting?

Hi,
Relatively new to AdWords and having great fun trying to analyse the mass of data.
My Anayltics does not have CPC Vs. Organic Conversion report, found in Marketing Optimization / Search Engine Marketing - but if I search for a report it says the same thing is available under Traffic Sources - but I do not have the drill down options you show.
I apologsie if this question is a little basic - but does this "other" version of Analytics work with your ammended tracking code?

Thank you for such a nice post!

I have one question though. I am using both methods and have two days of data with me. However I found that the number of queries captured by above method is less as compared to queries found in "Keywords" section in "Traffic Sources". Shouldn't the script capturing all queries?

Once again thank you for great post!

@Nicole: This is very strange. You shouldn't have to put anything in the the sleuth function to get it to work.

We've encountered some issues with the Google Analytics __utmSetVar function acting strangely on landing pages and reducing bounce rates, but never total data loss.

@Phil: Yep, definitely does work. There are two primary ways to use the data gathered by the script in the new version of Google Analytics.

If you want to get a list of all collected search terms, go to Visitors > User Define. Here you'll see a line-by-line list of all visits to the site, broken down by the visitors' exact search terms. It's normal to see a large segment of (not set); these visits are from non-CPC sources.

Another way to find the exact keyword information is to look in Traffic Sources > Search Engines report. Drill down into a specific search engine and then view your paid traffic. You can use the Segment dropdown just above the data table to segment this report by "User Defined". This will show you the exact search terms used on just that search engine.

@Merwan: This depends on whether you are looking at just the paid search keywords in Traffic Sources. The function outlined in this article will only gather exact search queries for paid search (i.e., sources with either a "ppc" or "cpc" utm_medium tag).

Another thing to bear in mind: certain Google AdWords visits that originate at sites other than Google.com (e.g., AOL and other search network partners) will supply a keyword via their utm tags, but do not provide a referral string that can be parsed by the sleuth script. This means that some of your paid links' exact queries will not be gathered.

Hi Michael,

I'm starting to work with the negative words for my campaigns and I found your articles, so I want to try your script, but I have a question, I read in some blog that the Search Query Performance report of Adwords works the same of this sleuth() function.

What do you think? I just ran the report, but I'm not an expert in Analytics!

Thanks,

@Veronica: Good question! The Search Query Performance report in AdWords is very similar to the function I've outlined in my post. It provides a list of the search queries that triggered your AdWords ads, and supplies data on their quality.

That being said, I still think that it's valuable to have this query data in Google Analytics, simply because it provides you with a more complete picture of the query performance. You can segment most reports in Google Analytics with this information, which gives a lot of intelligence on how specific keyword query combinations are interacting with your site.

Will the Sleuth code work with the new GA javascript which just came out? If so can you show the exact scripting/
Thanks

Hi Larry,

The sleuth code as it stands will not work with ga.js. It uses the urchin.js method of setting the User Defined variable. We'll definitely work on creating a compatible one soon, so stay tuned! Thanks for reading.

For ga.js compatibility, how about this within the getRef() function?
** Note, not yet tested.**

// for urchin.js code
if(typeof __utmSetVar=='function'){__utmSetVar(searchq[3])}

// new ga.js migration
if (typeof pageTracker=='object'){pageTracker._setVar(searchq[3]);}

@Al: Seems like it would work, but I'd recommend that anyone test this out thoroughly before implementing it.

That being said, we'll be posting an update in the next week about new versions of the script. Keep an eye on the blog, and add the RSS feed if you haven't already.

Hello Michael,

I implemented the script recommeded here, but I have one problem.

85% of my keywords show up on the (not set) section :( the worst part is that most of my conversions are coming from that section and I would really want to be able to find out the exact keywords that are converting.

Why does this happends and more important, how do I fix it?

my idea is to use a broad keyword to find out all the negatives keywords I can use on my PPC campaigns and all the keywords I can use for SEO that have traffic and convert well.

Also, does google analytics using this script only tracks the PPC traffic or also the natural, SEO traffic?

Manny

@Manny:

Sometimes, certain sources don't include a query parameter in the referring URL. This means that the script is unable to grab the exact search query, and thus you will get visits that are attributed to a (not set) User Defined.

If you are segmenting your AdWords traffic by User Defined, and 85% of the campaigns or keywords are showing up (not set), then you may want to double check the landing pages and ensure that the referring URL contains the exact search query. Currently, the script grabs query parameter content for a "q" or a "p" parameter.

This script only affects CPC traffic. Organic traffic will automatically include the search query as the visitor typed it into the search engine.

Stay tuned... we have some improvements to the script that should be up by the end of the week.

I'm curious to know if you looked into using the script for the new ga.js version? I can't find it anywhere.

Thanks!

@Danel: Definitely. We've been testing a new version of the script, internally, and it should be up on the blog by the end of the week.

@Michael

Any update on the ga.js version of this script? I'm very keen to try it out.

@Dave: It should be ready by the end of the week. You'll hear about it here first.

@Michael: That's great news - thanks for keeping us updated.

I only wish I'd found this site sooner - keep up the great work!

Is this still working for others. This script appears to have stopped working on my urchin sites. I want to make sure this isn't a wide spread problem before I continue to narrow it down.

@Brandon: When you say "urchin sites," do you mean sites that are still using urchin.js. Or do you mean sites that use Urchin Software?

It's still working for our clients. Could you elaborate on how it's not working?

Thanks for your response Michael-

I'm referring to using urchin.js. To elaborate further I had pulled the code off our sites for a little while and just re-uploaded the same code. I've double checked the placement of the ga-keywords script and everything appears as before, except in analytics my info is still coming through as 'not set' inside of user-defined.

Granted I only placed the scripts back up yesterday morning, however, last time I put this up I remember viewing the results (of the partial data) the next day.

@Brandon: Just wondering, since you're posting your comments on the old version of the script... have you seen the updates posted here? Be sure to grab the urchin.js version of the script.

Otherwise, put up your site's URL and I'll see if there's anything wonky with the script.

Michael,

It looks like I got it figured out. It was indeed a bad setup on my part. (2 people working on the same problem at the same time). Thanks for looking into with me though.

(All of the data is coming through perfectly!)

@Brandon: Glad to hear it. Let us know via the comments if you notice anything else out of order.

Love this script and just added it to 3 sites last week. Works great on 2, but I'm not getting data on the third. We're tracking conversions across domains on that one and are using this code (not all of which will display, but I think you can get the idea):

script src="http://www.google-analytics.com/urchin.js" type="text/javascript">
script src="kwtrack.js" type="text/javascript">

_uacct = "UA-number";
_udn="none";
_ulink=1;
urchinTracker();

kwtrack is my copy of your script. Would udn or ulink interfere with your script?

@Vicki,

You may want to try adding this line:

_udn="none";

Right above the call to sleuth(); in the kwtrack.js file. It would look like:

_uacct = "UA-1";
_udn = "none";
sleuth();

Try that and see what happens.