Google Analytics, on the other hand, is the most popular tool in the digital analytics world. It is used to keep track of marketing efforts, user behavior, traffic sources and more. The most common way to access data from Google Analytics is the web user interface. Alternative options include the Reporting API or external tools like Looker or Data Studio.

What’s interesting, though, is that there is no way to use the most popular query language to query data from the most popular analytics platform. That’s right, you cannot query Google Analytics data using SQL.

This is no big deal for the more basic users just checking the built-in reports in the Google Analytics UI. It is, for sure, a limitation for power users working with tools like Python and use data in custom models or even feed it into machine learning algorithms and recommendation engines.

In this article, we’re going to cover the solutions that will enable you to query your Google Analytics data using SQL.

Step 1 – Getting data into a database/data warehouse

As mentioned above, there is, unfortunately, no way to query data directly from Google Analytics using SQL. This means that the first step is to get data into some sort of a relational database or a data warehouse that support SQL queries.

There are three options for sending Google Analytics data into some external data storage.

1. Google Analytics 360 to BigQuery export

If your company has the 360 (premium) version of Google Analytics then you can use its native BigQuery export feature. Here are the details for setting this up. Keep in mind, though, that this solution will not work with the standard (free) version of Google Analytics.

2. Google Analytics Parallel Tracking

Google Analytics Parallel Tracking is a third party service that sends all of the raw hits into a data warehouse of your choice (i.e. BigQuery). These hits are then processed into sessions to provide a dataset similar to Google Analytics 360 export.

While this solution is not free, it will cost you only a small fraction of the cost of Google Analytics 360.

Getting started with Google Analytics Parallel Tracking.

3. Export data using the API

This is the most technical solution of the three. It leverages the Google Analytics Reporting API to pull data from Google Analytics and into your database or data warehouse.

One way to get started with this solution is to use the gago library in Go language to communicate with the Reporting API. Then you need the code for writing this data into your database.

The limitations with data export solution are that you can’t query all of the metrics and dimensions at the same time, you’re still affected by the data collection and sampling limits of Google Analytics and a modification in your tracking system is required (custom dimensions for hit timestamp, hit type, sessions ID, client ID and more).

Step 2 – Query Google Analytics data using SQL

This depends a bit on the setup you used to send data into a database/data warehouse and the type of data warehouse being used. Since the most common option is Parallel Tracking and Google BigQuery, we are going to use these in our examples as well.

The simplest way you can run your first SQL query against your Google Analytics data stored in BigQuery is to go into the BigQuery user interface and choose the right dataset containing your Google Analytics data.

With parallel tracking, you will have your Google Analytics data stored in three separate tables.

• raw_hits
• processed_hits
• processed_sessions

Depending on the type of query you want to run, choose the right table. Keep in mind that some of the information (bounce rate, geo-location etc.) isn’t available before hits are processed into sessions.

For example, we could write a simple SQL query like this and see the top 10 countries by the number of Google Analytics sessions.

Now, to make this query a bit more interesting, we might add in metrics like users and bounce rate. All doable with SQL, of course.

Notice how you can define your own rules for things like “bounce” and “bounce rate”.

Pretty neat, right?

Using the BigQuery UI is good for testing your setup and quick ad-hoc queries. In most cases, though, you’ll probably use some sort of a BI tool. One of the popular options these days is Google Data Studio. So, let’s use this in our examples a well.

Google Data Studio is free and probably the easiest to get started with. That being said, it is still packed with useful features and makes creating interactive dashboards a fun and enjoyable process.

Connecting BigQuery with Google Data Studio

Google Data Studio, like many other BI tools, has a native integration for BigQuery as a data source.

While you can automatically pull all data in a certain table, in most cases it’s a better idea to use a custom SQL query.

Data Studio has this feature built-in as well.

In our example, we’re going to analyze the results of an A/B test run using VWO. We’re using VWO’s custom-event-based integration to send experiment data into Google Analytics.

It’s always a good idea to try your query in the BigQuery’s query editor before implementing it in Data Studio.

Once we’re happy with the query and the results it returned, it’s time to put the query in Data Studio.

In Data Studio, we can start building all sorts of cool visualizations. For example this one for visualizing the funnels for Control and Variant 1 of our A/B test.

Data Studio is really flexible when it comes to building custom reports, interactive dashboards and quick visualizations. If you can’t find the right chart from the default chart types, check out the community visualizations or go ahead and build one yourself.

Conclusion

Working with your Google Analytics data using SQL opens up a whole new world of analysis opportunities for your digital analytics data. You no longer have to use the metrics and concepts defined and pre-calculated by Google. Feel free to come up with your own rules for things like calculating bounce rate or defining a session.

To get started, you first need to transfer your Google Analytics data into a data warehouse that supports SQL queries. For example, Google BigQuery. There are a few ways you can send Google Analytics data into BigQuery, our recommended method is Parallel Tracking. This will provide you with the most complete dataset, delivered in near-real-time and for a fraction of the cost of GA 360.

Once you have data in your database or data warehouse, you can start writing ad-hoc SQL queries right away. To take your productivity to a next level, we recommend running your queries and reports through a BI platform (i.e. Data Studio).

If you have any questions about analyzing your Google Analytics data using SQL or sending data into a data warehouse, feel free to ask them in the comments below.

The post How to Query Google Analytics Data Using SQL appeared first on Reflective Data.

Working with Google Analytics data in BigQuery has mostly been a privilege of those having a 360 version of Google Analytics. Its hefty price tag, though, has made that list quite short.

With Google Analytics Parallel Tracking Service from Reflective Data, having access to Google Analytics data in BigQuery has become a lot more affordable. (you can get a quote from the previous link)

Now, regardless of whether you have Google Analytics 360, a Parallel Tracking system from Reflective Data or some other integration in place, the structure of the data in BigQuery is quite similar.

In this article, I’m giving you a tour of the features in BigQuery and some of the ways how you can leverage them when working with your Google Analytics data.

Take a look at this article if you want to learn more about sending your Google Analytics data into BigQuery first.

In case you don’t have access to any BigQuery dataset containing Google Analytics data, you can check out the Google Analytics sample dataset for BigQuery. The sample dataset provides an obfuscated Google Analytics 360 dataset that can be accessed via BigQuery. It’s a great way to look at business data and experiment and learn the benefits of analyzing Google Analytics 360 data in BigQuery.

In this blog post, though, we are going to work with the demo dataset generated by Reflective Data’s Parallel Tracking System. As mentioned before, the structure of the data is pretty much the same.

Creating and running your first query

With Reflective Data’s Parallel Tracking System, you will have three main tables in your BigQuery dataset. Let’s take a quick look at each one of them with a simple query.

Table: raw_hits

This is the most up to date table and contains raw data coming in from the website, app or via the measurement protocol. Data is available withing ~5 seconds after it was sent.

raw_hits table contains only four columns.

• ua – the user agent string of the client
• ip – the ip address of the client
• q – the actual Google Analytics hit payload
• timestamp – when data hit the processing engine (~50ms after it was sent)

For example, we could query the number of hits for every hour in a given day.

Table: processed_hits

This is a processed version of the raw_hits table. Data in this table becomes available ~30 seconds after the hit was sent. This is still really-really fast and considered near-real-time data.

When it comes to columns, this table is much richer compared to the raw_hits table. It has all the time-related columns (date, hour, minute, timestamp etc.) converted into the correct timezone. Some of the data is in a special format known as a record in BigQuery. This includes columns like traffic_source, page, event, device, transaction etc. In fact, this table has a column for every parameter you can send with a Google Analytics hit. Added are some data points normally not available in Google Analytics (ip address, user agent, client ID, user ID etc.)

List of paramaters/columns as seen in Measurement Protocol Reference.

As an example, we are querying top 10 visitors (by client ID) that created the most hits in a given day.

Isn’t that easy? and cool, too!

I certainly like working with my Google Analytics data using SQL queries. Okay, let’s move on.

Table: processed_sessions

This table is, for sure, the richest of the three. It has all the columns from processed_rows but also includes quite a few new ones. These are the columns that require more processing (ip address to geolocation etc.) and/or the session to be ended (total hits in a session etc.). For these reasons, this table is usually updated a few times a day and a final version is generated at the beginning of the next day.

If you aren’t completely sure how a web session is defined in Google Analytics, take a look at this article.

A good thing about raw data is that whether you like the fact that Google Analytics is mostly session-based or not, you have the freedom to calculate your metrics however you wish. You could also mix and match different techniques.

For example, we might query top country/browser combinations by the number of sessions and pageviews in a given month. Let’s see how that would look like with our demo dataset.

Creating and running more complex queries

The best thing about raw hit-level data is that you’re not limited by things like sampling or pre-defined metric/dimension combinations. Both of which are common in Google Analytics, both the UI and the Reporting API.

What’s more, with BigQuery you can load in data from almost anywhere and mix it with your analytics data in any way you like. This may include data from your CRM, CMS, Ads platform or other analytics tools.

Google Analytics’ data model is structured so that session-based dimensions (like source/medium) don’t play well when combined with user-level or page-level dimensions and metrics. And there’s a limit to the number of dimensions we can see side-by-side: two dimensions is usually our limit in the interface, five in custom reports, and the API allows for seven dimensions. With BigQuery, there are no such limitations.

For example – ecommerce customers may have trouble pulling exact stats of the number of users from Social Media that saw a product page and then subsequently purchased the same product. BigQuery users can handle that with a single query.

By Alex Moore, source

One of the queries that almost always ends up being inaccurate (due to sampling) in Google Analytics is when you want to analyze all hits triggered by a single user. With BigQuery, this query is fast and simple, and there is no sampling. Never.

First, let’s go ahead and query some totals.

Now, let’s take a look at all of the hits in chronological order. This will give us a detailed overview of their user journey.

I believe you can see how analyzing a user journey like this can be very useful. For example, when a new lead comes in, before contacting them you could check what content they’ve already interacted with.

***

For someone that has worked mostly with the Google Analytics UI and not with the raw data before, there are some things that can be a bit confusing in the beginning. For example, not all of your common metrics are available as a column in BigQuery. Even the really basic ones, like bounce rate. That means you have to define them in your query.

Let’s take a look at the bounce rate for the top 10 landing pages in our sample dataset.

Calculating your own metrics like this can be a bit frustrating in the beginning but you actually only have to write most queries once and can later save them as views or saved queries in BigQuery. A benefit of defining your own metrics is that you are not limited by pre-defined metrics in Google Analytics. As you can see, in the query above we defined bounce as a session that has less than 2 pageviews but we could also include a rule that there can’t be any interaction hits in the session. We could also play with session duration or other data available in BigQuery.

A true value of having your analytics data in BigQuery, though, is that you can easily join it with other data that you’ve also sent to BigQuery.

For example, you could use Google Ads Data Transfer tool for BigQuery to automatically send all of your Google Ads data into a BigQuery dataset. This is especially useful if you’re using Google Ads auto-tagging solution because that means the only connection between your Google Analytics and Ads data is the gclid ID which is unique for every click.

With a simple query, you can join your processed_sessions table with ClickStats table.

After joining two datasets, you can combine any data points that are available in either dataset. The most important one, perhaps, is the cost data from the Ads dataset. This would allow you to calculate the amount of money you spent on acquiring each user.

Analyzing BigQuery data in a BI tool

Not everyone can write complex SQL queries to access their analytics data in BigQuery. Luckily, BigQuery has a native connector with almost all of the major BI and data visualization platforms.

My default recommendation is Data Studio. It’s free, full of powerful features and since it’s also from Google, you get a smart caching layer between Data Studio and BigQuery which can significantly lower your BigQuery costs.

Connecting BigQuery and Data Studio is really simple. All you have to do is create a new source in Data Studio and select BigQuery. It will automatically show you the list of Google Cloud projects you have access to and from there you can easily navigate to your datasets and tables.

Once you found the correct table, you can either choose the fields you want to include in your report (if not sure, choose all) or write your own query.

In our example, we connected two tables (processed_hits and processed_sessions) and selected all fields available in these tables.

Now, let’s create a quick time-series chart to see the number of hits of each hit type per day in our selected time period. Settings for a chart like this are really simple.

And this is how the chart itself looks like. Nothing special but allows you to have a quick overview of your traffic and spot any anomalies in tracking.

Another useful chart I recommend setting up is a time-series comparison between data in Google Analytics and BigQuery. This means you have to blend two data sources.

As you can see, the numbers are pretty close.

For more advanced dashboards you might want to write a custom query on a Data Studio data source level. This way you can pre-join different tables and/or datasets.

***

While being more difficult than using the Google Analytics UI, the freedom you get from having access to raw hit-level data in BigQuery is definitely worth it. As mentioned at the beginning of this article, you don’t necessarily have to spend a fortune on Google Analytics 360 because with Reflective Data’s Google Analytics Parallel Tracking System you can send your analytics data into BigQuery for a fraction of the money.

Should you have any questions about working with Google Analytics data in BigQuery, post them in the comments below and someone from our team will get back to you.

If you want to learn more about the Parallel Tracking System, click here.

The post How to Query and Analyze Google Analytics Data with BigQuery appeared first on Reflective Data.

Two of the most common shortcomings of Google Analytics that most of the advanced users experience, though, are the lack of hit-level granularity and data sampling. In this article, we are taking a look at some of the ways you can overcome these shortcomings without spending a fortune on Google Analytics 360.

PS! If your company already has 360, these techniques can give you an even more robust and complete dataset.

What is sampling and who’s affected

At first, let’s take a look at how Google Analytics describes sampling in their official documentation.

In data analysis, sampling is the practice of analyzing a subset of all data in order to uncover meaningful information in the larger data set. For example, if you wanted to estimate the number of trees in a 100-acre area where the distribution of trees was fairly uniform, you could count the number of trees in 1 acre and multiply by 100, or count the trees in a half acre and multiply by 200 to get an accurate representation of the entire 100 acres.

So, it simply means that some of the reports you see in Google Analytics (or any other tool that pulls data from it via the Reporting API) may not represent 100% of the relevant hits.

This is especially true when you build more advanced custom reports with detailed custom segments. Google Analytics samples complex ad-hoc queries even below 500k sessions. Unfortunately, this is exactly when you normally care about the data accuracy the most.

You can check if your report is affected from sampling by hovering over the shield icon at the top left of your report.

Normally, the reporting-level sampling starts when your selected date range has more than 500k sessions in total. Be vary, though, because it sometimes happens with less traffic as well.

In some circumstances, you may see fewer than 500k sessions sampled. This can result from the complexity of your Analytics implementation, the use of view filters, query complexity for segmentation, or some combination of those factors. Although we make a best effort to sample up to 500k sessions, it’s normal to sometimes see slightly fewer than 500k sessions returned for an ad-hoc query.

And that is not the only kind of sampling that can haunt you in the free version of Google Analytics. The second type of sampling takes place when data is being collected and the limits you need to know are as follows.

• 500 hits per session [1]
• 200,000 hits per user per day [1]
• 10 million hits per month per account [2]

source 1, source 2

It is important to understand that hits aren’t users, sessions or page views – hits are all data sent to Google Analytics including events, timing and data coming from the Measurement Protocol.

One way to check if you’re getting close to those limits is to go to Admin –> Property Settings in Google Analytics.

It is worth mentioning, though, that Google won’t automatically ignore all hits past the 10M mark. It will notify you in the UI and will likely contact you via email and suggest considering the GA 360 version. In the terms, though, it says “there is no assurance that the excess hits will be processed” and also that the warning message is not guaranteed to appear.

You should take action if…

• your site receives >10M hits a month
• you see the yellow shield next to your reports regularly
• your visitors generate more than 500 hits per session

How to avoid sampling in Google Analytics

If you hit any of the sampling limits but need reliable data in your work, you need to find a solution rather sooner than later.

Let’s take a look at the options you have.

Google Analytics 360 – This is the solution Google itself recommends. And no wonder why, it costs around $150k a year. GA 360 is a great tool and we recommend it to all companies with huge traffic and money to spend. Keep in mind, though, that if you only need the more generous data limits or BigQuery access, there are cheaper solutions (described in this article).

Collect less data – Well, who would want less data, right? We can’t recommend to skip tracking of some important user action like page view or file download but there may some automatic events that you don’t really care about. This could be some timing event that your systems sends every 10 seconds, a scroll depth event every 5% or something similar. Take a look at your events and you may find something. Just don’t remove something useful!

Unsample your Google Analytics data – If you don’t hit the data collection limits and the only worry is the sampling happening on reporting-level, you may be interested in solutions that let you unsample your existing data. How it works is that your query (run via the API) is divided into many sub-queries that are small enough that no sampling is applied. You can do so by writing your own small program in your favorite language or use one that others have built. For example this one written in GO.

There are also some paid tools available but we haven’t used them. Some of them run the API requests periodically to build an unsampled database based on your Google Analytics data. Keep in mind, though, that this doesn’t save you from data collection limits (i.e. 10M sessions/month) or some data aggregation that is inevitable in GA.

Use a parallel tracker – This is the most reliable solution against all data collection, processing and reporting limits you come across in Google Analytics. How it functions is that it duplicates all hits going from your site to Google Analytics, processes them separately and stores everything in your favorite data warehouse – BigQuery, for example.

With a parallel tracking setup, you are always free from any sampling and data collection limits. Furthermore, data processing incidents are rare but do occasionally happen in Google Analytics – having your own raw dataset lets you reprocess your data whenever needed. As a bonus, since you own the data, you may include PII, mix it with any other data or delete the records you don’t want. More on this solution later in the article.

What is hit-level data and why do I need it

Most data you see in Google Analytics is aggregated, and without custom configuration, you can’t get much of the raw hit-level (also known as event-level) data that you may need in more detailed analysis.

Hit-level data means that you can access the underlying hits that were sent to Google Analytics, allowing you to do your own aggregation as you wish, based on any criteria or dimension.

A good example of using hit-level data is analyzing the journey of a single user. On what page did they land on, what was the traffic source, which pages did they visit, how many interactions before converting etc. Having access to hit-level data lets you analyze the journey of each visitor in near-real-time. Furthermore, this data is perfect for machine learning algorithms that could, for example, detect and analyze users that are most likely to convert – you could then target them with ads and other campaigns.

Here’s the simplest query in BigQuery that shows you each hit from a single user in chronological order.

With hit-level data, one can access every single hit that was collected from the site along with all data-points that each hit included.

Unfortunately, there is no way you can get the true raw hit-level data out of Google Analytics. Not even using the API. Using custom dimension for the Client ID, Hit Type, Timestamp etc. can get you closer but it’s still far from perfect.

How to get access to raw, unsampled hit-level data

As mentioned before, you can’t get the raw hits out from Google Analytics. The premium version of Google Analytics (360) and its BigQuery export feature will get you closer (for 150k a year) but even that is not ideal.

The only way to gain access to the real underlying hits, with zero sampling and aggregation, is to leverage technology known as parallel tracking.

Parallel tracking means that all hits sent to Google Analytics are duplicated and sent to another endpoint. Depending on the solution, the data may be stored in Amazon S3, Google BigQuery or some other data warehouse.

Tools like Snowplow offer parallel tracking solution and accessing raw data in your data warehouse without any further processing (by default). This means there are no sessions, channel attribution or other really useful features that you do get in Google Analytics.

With Reflective Data’s Parallel Tracking (RDPT) solution, not only will you get all the raw hits but also a data processing engine that works very similarly to the one in Google Analytics itself. This means you will get sessions, attribution and features like referral exclusion out of the box. More advanced users can build (or request) their own rules for defining sessions, attribution and other features.

In order to break data silos, RDPT can integrate with any tool that has an API. Including your CRM and ad platforms.

RDPT’s default data warehouse is BigQuery but storing data elsewhere (Amazon S3 etc.) is also possible. BigQuery’s native integration to Google Data Studio makes it easy and cost-effective to build all sorts of reports and dashboards. Integrations with most other BI and data visualization tools are widely available.

How much does it cost

Google Analytics 360 is a really good tool for enterprises that want to gain more detailed access to their marketing data. Hefty price tag, limited access to raw data and occasional sampling should make you think twice before upgrading, though.

For companies that aren’t fully sold on Google Analytics 360 or companies that already have 360 but need access to even more complete dataset, the Reflective Data’s Parallel Tracker (RDPT) may be the perfect solution.

Pricing for RDPT depends on the amount of traffic, the complexity of the setup and the number of integrations. Compared to 150k, though, it will always be a bargain. The initial setup usually costs somewhere between $1k and $5k, and the monthly plans start at around $350. This includes a generous quota for BigQuery usage.

So, whatever your current analytics stack looks like, you should consider adding RDPT for the most robust, unsampled raw hit-level digital analytics data you can get. Learn more here.

The post Unsampled Hit-Level Google Analytics Data Without 360 appeared first on Reflective Data.

  If you need help with planning or building a data pipeline or a data warehouse then get in touch for a free consultation session with one of our top data engineers!

These days, data is at the core of almost all successful companies. The system that feeds those companies with data is known as a data pipeline.

This blog post is for anyone eager to learn more about data pipelines, planning to build a new pipeline or upgrade their current setup.

An end-to-end analytics data pipeline is a secure and reliable mechanism that is responsible for feeding your business with valuable data that can be used for reporting, analysis, machine learning or any other activity that requires accurate data about your business.

Every business is different and so are the analytics data pipelines that best suits their needs. Our model of an enterprise-grade, fully customizable data pipelines are divided into six logical steps. Let’s take a closer look at each one of them.

Events → Enrichment → Process → Manage & Monitor → Storage → Report & Visualize

Events

Events are the basis of most data pipelines, they trigger other actions and take the largest part (storage) of your data lake/warehouse.

There is a wide variety of events that may act as a source for your data pipeline. Here are the six main categories of sources that generate an ongoing flow of events.

• Website users
  • Click stream
  • Google Analytics parallel tracking
  • Form analytics
• Server events
  • Orders
  • Payments
• Mobile Apps
  • Real-time
  • Batch
• Ads
  • Google Ads
  • Facebook Ads
  • Twitter Ads
  • Others
• Feedback tool
  • Surveys
  • On-site polls
• 3rd parties
  • Testing tools
  • Personalization tools
  • Email providers
  • Call tracking

Events in data pipelines are usually invoked by webhooks that put all events into the processing queue where they are validated, enriched and batched.

For example, here’s a webhook for Google Analytics parallel tracking.

https://analytics-6785943.appspot.com/collect?v=1&_v=j79&a=1359602631&t=pageview&_s=1&dl=https%3A%2F%2Freflectivedata.com%2F&ul=en-us&de=UTF-8&dt=Homepage&sd=24-bit&sr=1920x1080&vp=1336x977&je=0&_u=SCCACEAjR~&jid=&gjid=&cid=1177657878.1573135112&tid=UA-3696947-1&_gid=775207114.1575821022&z=1779653917

The above webhook is quite similar to Google Analytics hit payload and is sent at the same time. It is then processed, enriched and streamed into the data warehouse.

All events are generally stored in three locations:

1. Log files – raw hits
2. Data Lake – Processed data (sometimes also enriched)
3. BigQuery – Processed & enriched data

This setup ensures us that we can re-process everything should there be something wrong with the enrichment process or should the data somehow get lost from the BigQuery instance.

Enrichment

To give more context to the raw event-level data, our systems are pulling data from various sources, including:

• CRM & CMS
• Google Analytics
• 3rd party APIs

For sources that don’t allow real-time access, we are using periodic batch load processes.

Data from various sources is either combined with the hit level data or sent to the data warehouse separately for query-level joining or processing after collection.

Process

Processing makes sure your data is secure, reliable, accurate and free from duplicates. To securely process your data, we are using the following tools from Google Cloud:

• Cloud Dataflow
• Cloud Pub/Sub
• Cloud Functions

Similar tools are also available in Amazon Web Services. There are several open-source solutions (often used by providers like Google and Amazon) but we prefer and recommend using managed services that can scale automatically. This allows us (and you) to focus on the data pipeline itself instead of maintaining the infrastructure.

We build the data processing queues to handle late-arriving data, duplicates and storage system outages in a way that best fits every use case. For example, people using your app on a 10-hour flight can produce a lot of events that are arriving and getting processed hours after they actually happened.

Manage & Monitor

To make sure everything is working as expected and to alert us and our clients if something is not, we are setting up a set of tools built in-house and from Google Cloud. Such as Stackdriver Monitoring.

Our goal is to provide our clients with the most accurate data with the least amount of latency.

In the pipeline management tools, you can see the pipeline’s structure, sources, processors, storage options etc. You would also see how data is moving between different elements and if there have been any errors or other incidents.

  If you need help with planning or building a data pipeline or a data warehouse then get in touch for a free consultation session with one of our top data engineers!

Storage

We’ve built the data pipeline so that it can send and store data in almost any database, data lake or data warehouse available. Here’s our standard recommendation.

• Google BigQuery
• Digital Analytics Platforms
  • Google/Adobe Analytics
• Long-term storage
  • Cloud Storage 
  • AWS S3

We make sure that your data stays safe and meets all the security requirements while giving you the full ownership and control over your data.

As mentioned earlier, we store raw hits as well to ensure that we can re-process everything should there be something wrong with the enrichment process or should the data somehow get lost from the BigQuery (or other reporting) instance.

Report & Visualize

Collecting data is pointless if it’s never being used to benefit your business. We think about the value our clients’ data has to provide in the earliest phase of planning and building the data pipeline. Our team works closely with our clients in order to figure out the KPIs, reports and dashboards that are needed for their companies growth.

Tools we are using for reporting, analysis and visualization include:

• Google Data Studio
• Jupyter Notebooks
• Redash
• Other BI tools

We build all analytics data pipelines and storage systems so that they could be connected with almost any BI tool available on the market. Wherever possible, we make data available for reporting in near real-time.

It’s no news that data is running most of the successful businesses these days and more and more companies as implementing modern data pipelines.

When thinking of a modern data pipeline, keep these keywords on mind:

• Serverless
• Scalable
• Real-time access
• Security
• Flexibility

Our team is happy to answer any analytics data pipeline related questions in the comments below.

Need help with planning or upgrading your data pipeline? Contact us and get a free consultation with one of our top data engineers.

The post Six Key Components of an Analytics Data Pipeline appeared first on Reflective Data.

In this article, we are covering everything you need to know about the problem of duplicate transactions, a root cause of skewed data in many Google Analytics instances.

Detecting Duplicate Transactions

Duplicate transactions in Google Analytics means simply that a single transaction was counted more than once. This can dramatically skew your data because along with the transaction count, this also inflates the revenue, quantity and other metrics directly related to transactions.

To check if your Google Analytics E-commerce setup is affected by duplicate transactions, let’s start by creating a custom report (Customisation –> Custom Reports –> New Custom Report) with the following settings.

Metrics: Transactions

Dimensions: Transaction ID

Hit “Save” and you should see your Duplicate Transactions custom report right away.

Firstly, choose a time frame. I’d recommend starting with 30, 60 or 90 days depending on how many transactions your site generates and how often your analytics setup changes.

Secondly, sort your table by most Transactions first.

If every transaction ID shows only one transaction then congratulations, in the selected time frame there were no duplicate transactions.

Every number greater than one is a sign of duplicate transaction. Now, to get an overview of the magnitude of the problem, compare the number of transactions against the number of rows in the table. The latter is the actual count of unique transactions.

In the above screenshot, there are some duplicate transactions but considering the total amount of transactions, the problem is minor.

If your duplicate transaction ratio is >10% you have a serious problem with your e-commerce numbers and before fixing it, you should take your numbers with a grain of salt.

PS! We should be thankful that Google has kept Transaction ID available as a dimension in almost all reports instead of hiding them as they do with the Client ID, User ID, Session ID and other really useful dimensions (that you should track using custom dimensions).

Fixing Duplicate Transactions

By far the most common reason duplicate transactions are happening is that a transaction hit is sent to Google Analytics every time the “thank you” page is loaded. There are people who bookmark this page and go back to check the order id for tracking or whatnot. Also, some companies send the “thank-you” URL in the order confirmation email.

To fix this, there are two possible solutions.

Option 1

Make the event trigger based on something coming from the back end, something that only triggers when the actual transaction is happening. For example, after the user has entered their credit card information and pressed “Complete payment” and your system has validated the transaction. I.e. just before redirecting the user onto the “thank you” page.

Option 2

Make your “thank you” pages to load only once. If you keep your transaction hits on the “thank you” page, make sure a user can see and visit the page only once. Every subsequent attempt to access this page should redirect them to some other page. This could be an order overview page or something similar (should still be order-related and contain order details).

…

Now, to take it one step further, you can make your e-commerce tracking even more reliable by sending transaction hits straight from the back end using Measurement Protocol. This way you can make sure that no transaction goes untracked or that someone is sending fake transactions towards your Google Analytics instance.

Another option is to use localStorage to keep track of transaction IDs that have already been sent to Google Analytics. This method is described in detail by Simo Ahava in this blog post.

Final thoughts

Google Analytics is an excellent tool for keeping track of your e-commerce business. What you need to keep in mind, though, is that skewed numbers can do more harm than having no numbers at all.

This is why you should audit (or have someone else do it for you) your Google Analytics and Enhanced Ecommerce setups regularly and compare the numbers in GA against the numbers in your back end and accounting.

If you see that the numbers don’t add up, see if you have any duplicate transactions by creating a simple custom report seen in the beginning of this blog post.

Have any questions or thoughts about detecting and fixing duplicate transactions in Google Analytics? Post them in the comments below.

***

To make sure you have access to reliable data even after problems like duplicate transactions have skewed your data, we recommend sending all of your Google Analytics data into a data warehouse like BigQuery. This makes sure you can filter out duplicated data and reprocess your reports to be 100% accurate. One solution to achieve this is to use a Google Analytics Parallel Tracker from Reflective Data.

The post How to Detect and Fix Duplicate Transactions in Google Analytics appeared first on Reflective Data.

Python is a programming language with virtually limitless functionalities and one of the best languages for working with data. Jupyter Notebooks, on the other hand, is the most popular tool for running and sharing both your Python code and data analysis.

Putting Python and Notebooks together with Google Analytics, the most popular and a really powerful tool for tracking websites, gives you almost like a superpower for doing your analysis.

This is exactly what this post is about. Pulling in and analysing your Google Analytics data using Python and Notebooks.

Oh, and by the way, this blog post itself is a Jupyter Notebook created in Google’s Colab, a version of Jupyter Notebooks that makes it super easy to collaboratively work on and share your Notebooks.

We wrote more on how we managed to turn a Notebook into a WordPress blog post in this article.

Without furder ado, let’s start by loading in some dependencies we are going to need for superfsadfsadfsdfsafasdfsadfpulling data from Google Analytics.

import pandas as pd # Pandas is an open source library providing high-performance, easy-to-use data structures and data analysis tools for Python
import json # JSON encoder and decoder for Python
import requests # Library for sending HTTP requests

There are many ways you can pull data from Google Analytics but one of the simplest is by using the API Query URI that you can grab from the Step 1 – Go to Query Explorer

If you are not familiar with the Query Explorer, this is an official tool from Google that lets

you play with the Core Reporting API by building queries to get data from your Google Analytics views (profiles). You can use these queries in any of the client libraries to build your own tools. In this case, the tool is a Notebook.

Step 2 – Authenticate

Choose the Google Account that has access to the Google Analytics View from which you want to get data.

Step 3 – Build the query

If you know how Google Analytics works, building a query is rather straightforward.

First, choose the right Account, Property and View you want to access.

Then choose the date range. You can use the calendar picker or write dynamic ranges like from 90daysAgo to yesterday.

Next, pick the metrics and dimensions. For example, you might pick metrics ga:sessions and ga:users and a dimension ga:date

.

Finally, you can apply sorting, filtering and segmenting rules as you wish. I’d recommend using them after you feel comfortable with the basics.

Perfect. Now test your query by pressing the “Run Query” button. If everything went as expected, you should see a table with results appear in a few seconds.

Step 4 – Get the API Query URI

Now, copy the API Query URI – we are going to use it for making requests straight from the Notebook.

Other options to pull data from Google Analytics using Python include:

• Using the Google Analytics Reporting API library for Python in your Notebook – based on Google API Client for Python
• Building a custom endpoint for handling your queries using the abovementioned API
• Exporting data from Google Analytics UI into CSV or XLSX and importing using pd.read_csv()

Making the request

Now that we have the API Query URI, we can make our first request using Python in the Notebook.

In this example, we are loading the number of Sessions and Users per day for the last 30 days.

from getpass import getpass # required only for hiding your variables

api_query_uri = getpass('Enter API Query URI here') # get the URI
r = requests.get(api_query_uri) # make the request
data= r.json() # read data from a JSON format
df = pd.DataFrame(data['rows']) # turn data into a Pandas data frame
df = df.rename(columns={0: 'Date', 1: 'Sessions', 2: 'Users'}) # giving the columns some proper titles
df['Sessions'] = df['Sessions'].astype(int) # formatting sessions as ints
df['Users'] = df['Users'].astype(int) # formatting users as ints
df['Date'] = pd.to_datetime(df['Date'])

df.head() # printing the first five rows

Enter API Query URI here··········

	Date	Sessions	Users
0	2019-09-25	64076	83335
1	2019-09-26	74569	103784
2	2019-09-27	59752	77642
3	2019-09-28	57743	76690
4	2019-09-29	63712	82980

Great, so our data is now loaded and formatted for our needs.

Let’s go ahead and try visualizing it. For timeseries data, a line chart should work well.

df.plot.line(x='Date', y=['Sessions', 'Users'], ylim=[0,None], figsize=[10, 6])

<matplotlib.axes._subplots.AxesSubplot at 0x7f3cb89692b0>

That was easy, right?

Now, let’s take a look at a bit more advanced visualization by drawing a population pyramid based on data we are now pulling from Google Analytics.

This is how the query looks like:

dimensions: ga:userAgeBracket, ga:userGender
metrics: ga:sessions

api_query_uri = getpass('Enter API Query URI here') # get the URI
r = requests.get(api_query_uri) # make the request
data= r.json() # read data from a JSON format
df = pd.DataFrame(data['rows']) # turn data into a Pandas data frame
df = df.rename(columns={0: 'Age', 1: 'Gender', 2: 'Sessions'}) # giving the columns some proper titles
df['Sessions'] = df['Sessions'].astype(int) # formatting sessions as ints
df.head()

Enter API Query URI here··········

	Age	Gender	Sessions
0	18-24	female	56244
1	18-24	male	9362
2	25-34	female	228866
3	25-34	male	33987
4	35-44	female	109795

import matplotlib.pyplot as plt # Matplotlib is a popular library for drawing plots in Python
import seaborn as sns # Seaborn is a Python data visualization library based on matplotlib

# Making copy of the dataframe and modifying session values to be negative for females (see chart below to see why)
df['Sessions'] = df.apply(lambda row: row['Sessions'] * -1 if row['Gender'] == 'female' else row['Sessions'], axis=1)

# Draw Plot
plt.figure(figsize=(8,5), dpi= 80)
group_col = 'Gender'
order_of_bars = df.Age.unique()[::-1]
colors = [plt.cm.Spectral(i/float(len(df[group_col].unique())-1)) for i in range(len(df[group_col].unique()))]

for c, group in zip(colors, df[group_col].unique()):
    sns.barplot(x='Sessions', y='Age', data=df.loc[df[group_col]==group, :], order=order_of_bars, color=c, label=group)

# Decorations    
plt.xlabel("$Sessions$")
plt.ylabel("Age")
plt.yticks(fontsize=12)
plt.title("Population pyramid of the website visitors", fontsize=16)
plt.legend()
plt.show()

Beautiful, now we know that majority of our audience is younger females.

More custom visualizations

With the wide variety of Python packages available for data visualization, the number of different charts you can use is virtually limitless.

In the next example, let’s try out joy plot. If you don’t know what a joy plot is then you are about to learn it now.

First, let’s install the required Python package.

!pip install joypy # install joypy
import joypy # JoyPy is a one-function Python package based on matplotlib + pandas with a single purpose: drawing joyplots.

Requirement already satisfied: joypy in /usr/local/lib/python3.6/dist-packages (0.2.1)

Now, let’s make the request to load required data. For this example we are using the number of transactions per age bracket and hour of the day for the past 90 days.

api_query_uri = getpass('Enter API Query URI here') # get the URI
r = requests.get(api_query_uri) # make the request
data= r.json() # read data from a JSON format
df = pd.DataFrame(data['rows']) # turn data into a Pandas data frame
df = df.rename(columns={0: 'Hour', 1: 'Age', 2: 'Transactions'}) # giving the columns some proper titles
df['Hour'] = df['Hour'].astype(int) # formatting hours as ints
df['Transactions'] = df['Transactions'].astype(int) # formatting transactions as ints
df.head()

Enter API Query URI here··········

	Hour	Age	Transactions
0	0	18-24	47
1	0	25-34	195
2	0	35-44	32
3	0	45-54	26
4	0	55-64	26

To make our dataframe more suitable for the joy plot, we need to pivot it.

pivot = pd.pivot_table(df, values='Transactions', index=['Hour'],
    columns=['Age'])

pivot = pivot.fillna(0)

pivot.head()

Age	18-24	25-34	35-44	45-54	55-64	65+
Hour
0	47	195	32	26	26	0
1	37	68	37	16	42	0
2	21	16	5	11	5	0
3	0	16	11	0	5	0
4	5	5	5	0	0	0

Now, let’s see how our joy plot looks like by running the following code.

x_range = list(range(24))
fig, axes = joypy.joyplot(pivot, kind="values", x_range=x_range, figsize=(10,8))
axes[-1].set_xticks(x_range);

Looks quite good already. but since most of our customers were in the age bracket 25-34 and we want to visualize the difference in their behavior rather than absolute numbers, let’s normalize the data.

df_norm = (pivot - pivot.min()) / (pivot.max() - pivot.min())
df_norm.head()

Age	18-24	25-34	35-44	45-54	55-64	65+
Hour
0	0.356061	0.313531	0.096429	0.130	0.206349	0.0
1	0.280303	0.103960	0.114286	0.080	0.333333	0.0
2	0.159091	0.018152	0.000000	0.055	0.039683	0.0
3	0.000000	0.018152	0.021429	0.000	0.039683	0.0
4	0.037879	0.000000	0.000000	0.000	0.000000	0.0

Now that the data is normalized, let’s see how a joy plot would look like.

fig, axes = joypy.joyplot(df_norm, kind="values", x_range=x_range, ylim='own', overlap=1.5, figsize=(10,8))
axes[-1].set_xticks(x_range);

Much better in my opinion.

So, what does this joy plot tell us?

• Most transactions are made between 8 AM and 10 PM, almost no one buys between 2 AM and 6 AM.
• Older (65+) people don’t buy after 11 PM while the youngest bracket (18-24) continues buying until 3 AM.

While I agree that this plot didn’t teach us something extremely interesting or new, it is a good example of what’s possible with a few lines of code and Jupyter Notebook (or any other similar tool).

Working with raw hit-level Google Analytics data

While connecting Jupyter Notebooks directly to your Google Analytics view is enough for many use cases, having access to raw hit-level Google Analytics data opens completely new opportunities for more advanced analysis.

To gain access to raw hit-level Google Analytics, one can use a tool like Marketing Data Pipeline to send data into BigQuery. Python has a library for connecting to BigQuery which makes it easy to explore and visualize based on raw data and metrics calculated based on your own criteria.

Besides other use cases, having access to raw hit-level data enables you to get a complete overview of the user journey, calculate metrics like LTV and use your own models for marketing channel attribution (Markov, ML-based, FPA etc.).

Learn more about the benefits of having a marketing data warehouse.

Conclusion

Combining powerful tools like Google Analytics, Python and Jupyter Notebooks gives you the flexibility and options you probably didn’t even dream about. The first time I got to play around with these three together made me feel like a little child, like I had discovered something really-really cool (and it is).

I hope this was enough to get you excited and that you will now go and try doing it on your own. And as always, your comments, questions and suggestions are welcome in the comments below.

The post Working with Google Analytics Data Using Python and Jupyter Notebooks appeared first on Reflective Data.

While it might look like a normal WordPress blog post (like all the previous posts on our blog), you are actually looking at a Jupyter Notebook.

If you are not familiar with Jupyter Notebooks then here’s what you need to know about them:

The Jupyter Notebook is an open-source web application that allows you to create and share documents that contain live code, equations, visualizations and narrative text. Uses include: data cleaning and transformation, numerical simulation, statistical modelling, data visualization, machine learning, and much more.

In this first proof-of-concept blog post/Notebook, we are showing you how Notebooks work and a few cool things you can accomplish with them.

Why are we doing this?

The reason we are testing with “blogging using a Jupyter Notebook” is that it allows us to share our code example and analysis done in Python more easily and in a more repeatable way.

At Reflective Data, we are using a lot of Python in all steps of data analysis. Jupyter Notebooks is a tool we use to run and share our code both internally and with our clients. Why not leverage the same workflow in our blogging efforts as well, we thought.

Our plan is to create a series of useful tutorial-type blog posts where we show you how to use Python (and Notebooks) in your everyday work as a digital analyst. We are going to pull data from Google Analytics, testing tools and databases, then run all sorts of analysis and visualizations using this data (and Notebooks).

How are we doing this?

Since sharing Notebooks is all about repeatability, we thought it will be wise to also share how we managed to turn Notebooks into a WordPress blog post.

While you can host and run Jupyter Notebooks both locally or on the cloud, we’ve found Colaboratory by Google to be the best solution when it comes to (as the name suggests) collaborative coding. That’s why we are mostly using Colab when working with Notebooks.

Turning a Notebook into HTML

The first step in the process of sharing your Notebook on WordPress would be turning your Notebook from ipynb format into HTML.

The hosted version of Jupyter Notebooks comes with a handy “Download as HTML” option built-in. Colab, unfortunately, doesn’t have this feature. So, we found a workaround which also allowed us to semi-automate the whole process.

We are using Google Drive’s API to load the ipynb file from Drive (Colab hosts Notebooks on your Drive) and then using nbpreview to turn it to HTML. We haven’t yet done so, but the plan is to build a WordPress plugin that would take a Colab Notebook’s ID and then automatically turn it into HTML.

Teaching WordPress how to render a Notebook

Currently, we simply paste the HTML version of the Notebook into WordPress blog post using a text-edit-mode. This will soon be handled by the plugin we mentioned in the previous step.

Everey blog post that contains a Notebook will receive a tag “notebook”. Using this tag we can then tell WordPress to load some extra CSS to properly render the Notebook.

We are currently doing this in a few lines of code in our functions.php file.

add_action('wp_head', 'load_notebook_styles');
function load_notebook_styles() {
    if ( has_tag( 'notebook' ) ) {
        wp_enqueue_style( 'notebook-css', get_template_directory_uri() . '/notebooks/notebook.css' );
        wp_enqueue_style( 'notebook-prism-css', get_template_directory_uri() . '/notebooks/prism.css' );
    }
}

Here are the CSS files we load:

• notebooks.css
• prism.css

What’s possible with Notebooks?

So, now that you have at least a basic knowledge of what a Notebook is and how we managed to turn them into WordPress blog posts, let’s see what you could actually accomplish with them.

Them main thing, of course, is that you can run Python code and immediately see the output.

Here’s a very basic example:

x = 5
for i in range(x):
  print(i * '*')

*
**
***
****

Awesome, right?

Now, the only limits you really have are your creativity and Python skills. In the upcoming blog posts, our goal is to get you better at both by showing some cool and useful things we are doing using Python and Jupyter Notebooks.

While the first Python script was quite useless, let’s try and load in some real data from Google Analytics.

First, let’s import our required dependencies.

import pandas as pd
import json
import requests

There are many ways you can pull data from Google Analytics but one of the simplest is by using the API Query URI that you can grab from the Query Explorer.

In this example, we are loading the number of Sessions and Users per day for the last 30 days.

from getpass import getpass # required only for hiding your variables

api_query_uri = getpass('Enter API Query URI here') # get the URI
r = requests.get(api_query_uri) # make the request
data= r.json() # read data from a JSON format
df = pd.DataFrame(data['rows']) # turn data into a Pandas data frame
df = df.rename(columns={0: 'Date', 1: 'Sessions', 2: 'Users'}) # giving the columns some proper titles
df['Sessions'] = df['Sessions'].astype(int) # formatting sessions as ints
df['Users'] = df['Users'].astype(int) # formatting users as ints
df['Date'] = pd.to_datetime(df['Date'])

df.head() # printing the first five rows

Enter API Query URI here··········

	Date	Sessions	Users
0	2019-09-12	50526	58667
1	2019-09-13	53919	63156
2	2019-09-14	58735	68200
3	2019-09-15	67371	77597
4	2019-09-16	65780	77305

Great, so our data is now loaded and formatted for our needs.

Let’s go ahead and try visualizing it. For timeseries data, a line chart should work well.

df.plot.line(x='Date', y=['Sessions', 'Users'], ylim=[0,None], figsize=[10, 6])

<matplotlib.axes._subplots.AxesSubplot at 0x7fc6a53c3278>

That was simple, right?

There is so much more you can do with Python and Jupyter Notebooks and we are going to cover more advanced stuff in our upcoming blog posts.

I’d like to end this post with a few thoughts I got while writing this one.

1. Writing a blog post using Markdown is fun (and effective). I know some people are doing it but this is the first time for me. Markdown is the way you write text in Notebooks.
2. While I thought so before, now I know for sure that Notebooks + WordPress is a really good solution for sharing Python code examples and data visualizations online.
3. I’d love to hear your thoughts, ideas, questions and topic suggestions in the comments below! We are happy to respond/help.

PS! More Notebooks-related content is coming soon! Consider subscribing to our newsletter.

The post Jupyter Notebooks in WordPress appeared first on Reflective Data.

Custom Events provide a perfect solution for adding context and tracking more specific user actions. In this article, we are giving you a good amount of ideas for custom events you should implement on your own and/or your clients’ websites.

This post is divided into sections by different event categories so you could skip the ones that are not relevant for you.

Clicks

For almost all websites, the most common user interaction is a click. By default, Google Analytics isn’t very good at tracking them. Let’s see how we could improve this situation.

Navigation clicks

Tracking usage of the main menu, sidebar and other navigation elements is critical. This is how your visitors explore your business and move in the funnel.

Detecting navigation interactions is easy with Google Tag Manager and sending data to Google Analytics works as with any other event.

Since there may be several navigation elements containing links to the same page, it is important to also pass the information about which navigation was used. For example, you might a have link to your blog both in the header and in the footer.

Recommended event parameters:

category: click
action: top-navigation, sidebar etc.
label: link text
non-interaction: false

CTA clicks

Tracking clicks on all of your CTAs (call to actions) is extremely important if you have more than one CTA on a single page pointing to the same location (i.e. long-form sales pages).

Don’t forget that buttons like “Get a Quote”, “Add to Cart” and “Checkout” are also CTAs.

Setup is the same as with generic click tracking using GTM.

Recommended event parameters:

category: click
action: CTA
label: text (and index the same CTA appears multiple times)
non-interaction: false

Outbound link clicks

Also known as tracking website exits, this custom event tells you exactly to which external pages people navigate from your site.

Again, setting this up is quite easy with GTM.

Recommended event parameters:

category: click
action: outbound link
label: link URL - link text
non-interaction: false

File download clicks

Doesn’t matter if you provide whitepapers, ebooks, PDFs, or other types of downloadable files, you should be tracking how often they get downloaded.

There are some really good tutorials online but we recommend setting it up using Google Tag Manager. A simple “just links” click trigger with a regex that matches desired filetypes should work just fine.

Regex Example:

\.(pdf|docx)$

Recommended event parameters:

category: click
action: download
label: filename / link text
non-interaction: false

Rage clicks

Visitors rage clicking on certain elements on your website is a good indicator of a UX error. For example, people may click on a blue text that is not a link or on an image that has no click functionality.

We have a separate blog post about tracking rage clicks using Google Tag Manager and Google Analytics.

Recommended event parameters:

category: click
action: rage click
label: element selector
non-interaction: false

Button clicks

Your website is likely to have all sorts of clickable buttons. There might be some other ways for you to get an overview of the usage of the features related to those buttons but knowing how many clicks each button received definitely helps.

Some ideas for which buttons should be tracked:

• Add to wishlist/favorites
• Add/remove item from cart
• Update cart
• Continue shopping
• Log in/out
• Register
• etc.

Recommended event parameters:

category: click
action: button
label: button text/description
non-interaction: false

Form Interactions

When it comes to tracking forms, I would like to recommend a few good articles by Simo Ahava that cover everything you need for getting started.

• Track Form Engagement With Google Tag Manager
• Track Form Abandonment With Google Tag Manager
• Form Field Timing With Google Tag Manager

Since forms are the most important part of many websites (i.e checkout, lead form etc.) you should really nail tracking them correctly.

A/B Testing

Experiment and Variation

Most A/B testing tools provide some sort of integration with Google Analytics. Most of them, though, are using custom dimensions. A problem with custom dimension based integration is that you need to make sure that the dimensions aren’t overlapping between experiments because otherwise, you might end up overwriting the dimension value for some experiments.

What we recommend having, besides the default integration, is a secondary event-based integration. That is, sending a custom event for every running experiment in a following format:

category: ab testing
action: experiment ID
label: variation number
non-interaction: true

PS! One event per experiment-variation combination.

This gives you raw information about each experiment and variation that the visitor saw. You can easily use this information in your custom segments or reports.

Every testing has a bit different solution for exposing experiment data to the front-end but here are a few links that should be helpful.

• VWO – instead of custom dimension, send this data using custom event action and label.
• Google Optimize – this allows you to get experiment data to the datalayer, now send it using a custom event.
• Optimizely X – access data about live experiments using Javascript.

Google Optimize Anti-Flicker Snippet timeout

If you have any experience with JavaScript-based A/B testing and/or personalization tools you know that flicker can be a real headache.

In case your tool of choice is Google Optimize, you should be using their official anti-flicker snippet to minimize the flicker effect.

We wrote a full separate article on Tracking Optimize Anti-Flicker Snippet Timeout in Google Analytics.

Recommended event parameters:

category: optimize
action: snippet timeout
label: true/false
non-interaction: true

Scroll depth

Whether you are working with a blog, some landing pages or any other page that is longer than a fold, you want to know how far down do the visitors scroll.

It used to required some custom Javascript but now, tracking scroll depth is a built-in feature of Google Tag Manager.

Recommended event parameters:

category: scroll
action: scroll depth threshold
non-interaction: false

Social interactions

While Google Analytics has a dedicated feature for tracking social interactions, having this data in custom events gives you more flexibility in reporting.

There is a really thorough blog post about tracking social interactions using Google Tag Manager – although, if you want something simpler, tracking clicks on your social icons is a good starting point.

Recommended event parameters:

category: social
action: share/like/follow
label: social platform
non-interaction: false

Video events

Doesn’t matter if it’s a sales video, a video that explains how your product works or a tutorial – you do care about how many people start the video and how far through will they make.

Google Tag Manager provides a really easy solution for tracking YouTube videos but tracking others such as Vimeo is completely doable as well.

Recommended event parameters:

category: videos
action: start/pause/percentage
label: video id
non-interaction: false

Errors

No matter how good your site is, there will always be errors. Since errors are directly related to bad user experience, tracking them is absolutely crucial.

The first step here should be sending your Javascript errors to Google Analytics. Next, consider sending other errors like popup alerts and system errors as well.

Recommended event parameters:

category: error
action: error message
label: feature or other relevant information
non-interaction: true

reCAPTCHA bot score

Google Analytics does have a simple bot-filtering system in place but it comes nowhere near catching all the bots that visit your site. Even tools like reCAPTCHA by Google can’t usually say for sure whether a visitor is a bot or a human but they can give you a score (from 0.1 to 1). Simo Ahava has put together a great tutorial for tracking the reCAPTCHA both score using Google Analytics custom dimensions.

Recommended event parameters:

category: reCAPTCHA
action: reCAPTCHA answer
label: reCAPTCHA score
non-interaction: true

Don’t forget to test whatever new custom event you add to your setup.

Should you have any questions related to custom events in Google Analytics, shoot them in the comments below.

PS! If you need help with implementing custom events, you can hire us to help you out!

Photo by Annie Spratt on Unsplash

The post Data You Should Be Tracking Using Google Analytics Custom Events appeared first on Reflective Data.

Detecting and fixing such usability issues is crucial, and luckily the detection part is doable using Google Tag Manager and Google Analytics.

Rage click is a situation where, in a short period of time, a website user clicks multiple times on a specific element. I.e. in two seconds, a visitor clicks three times on an image.

Without further ado, let’s see how tracking rage clicks works in Google Tag Manager.

Step 1 – Create a new Tag that detects rage clicks

This tag is responsible for detecting rage clicks and pushing a custom event rage_click into the data layer.

There are a few variables that you can play with to make it best fit your use case:

no_of_clicks – after how many clicks do we consider it a rage click. Default is 3.

time – in what time period should the clicks happen. Default is 2 seconds.

radius – in what radius should the clicks happen. Default is 100px.

The following variables are pushed into the data layer when a rage click is detected:

event – rage_click

rc_element – selector of the element on which the clicks happened

rc_count – the number of clicks

rc_max_distance – maximum distance between the clicks

rc_time_diff – period from first to last click

PS! You don’t have to send all of them to GA, pick the ones you think make the most sense.

Full code snippet. – Requires jQuery to be loaded on the page.

if ( typeof(jQuery) === 'function' ) {
    jQuery(document).ready(function($) {
        jQuery.fn.extend({
            getPath: function() {
                var path, node = this;
                while (node.length) {
                    var realNode = node[0],
                        name = realNode.localName;
                    if (!name) break;
                    name = name.toLowerCase();
                    var parent = node.parent();
                    var sameTagSiblings = parent.children(name);
                    if (sameTagSiblings.length > 1) {
                        var allSiblings = parent.children();
                        var index = allSiblings.index(realNode) + 1;
                        if (index > 1) {
                            name += ':nth-child(' + index + ')';
                        }
                    }
                    path = name + (path ? '>' + path : '');
                    node = parent;
                }
                return path;
            }
        });
        // Number of rage clicks
        var no_of_clicks = 3;
        //Time interval - 3 for 3 secs, 4 for secs and likewise
        var time = 2;
        var click_events = [];
        //internal variables
        var possible_click = 3;
        var radius = 100;
        function detectXClicks(count, interval) {
            var last = click_events.length - 1;
            var time_diff = (click_events[last].time.getTime() - click_events[last - count + 1].time.getTime()) / 1000;
            //returns false if it event period is longer than 5 sec
            if (time_diff > interval) return null;
            //check click distance
            var max_distance = 0;
            for (i = last - count + 1; i < last; i++) {
                for (j = i + 1; j <= last; j++) {
                    var distance = Math.round(Math.sqrt(Math.pow(click_events[i].event.clientX - click_events[j].event.clientX, 2) + Math.pow(click_events[i].event.clientY - click_events[j].event.clientY, 2)));
                    if (distance > max_distance) max_distance = distance;
                    if (distance > radius) return null;
                }
            }
            var result = {
                count: count,
                max_distance: max_distance,
                time_diff: time_diff
            }
            return result;
        }
        function removeUsedClickPoints(count) {
            click_events.splice(click_events.length - count, count);
        }
        $("body").click(function(event) {
            click_events.push({
                event: event,
                time: new Date()
            });
            //remain only required number of click events and remove left of them.
            if (click_events.length > possible_click) {
                click_events.splice(0, click_events.length - possible_click);
            }
            //detect 3 click in 5 sec
            if (click_events.length >= 3) {
                var result = detectXClicks(no_of_clicks, time);
                if (result != null) {
                    var path = $(event.target).getPath();
                    //console.log('Rage Click: ' + JSON.stringify(result));
                    dataLayer.push({
                        'event': 'rage_click',
                        'rc_element': path,
                        'rc_count': result.count,
                        'rc_max_distance': result.max_distance,
                        'rc_time_diff': result.time_diff
                    });
                    removeUsedClickPoints(3);
                }
            }
        });
    });
}

Step 2 – Create a new Trigger

The snippet from the previous step is pushing the rage_click event into the data layer every time a rage click is detected. Now we need to create a trigger that listens to that event.

Step 3 – Send an event to Google Analytics

Awesome, now let’s create a new tag that is responsible for sending the custom event to Google Analytics.

You can play around with the event parameters, especially with the event label to best match your needs but the recommended values are as seen in the screenshot above.

Before publishing your changes, don’t forget to enter the preview mode in Google Tag Manager and test the setup yourself. Go ahead and do a few rage clicks and see if an event is getting sent to Google Analytics. Also, check the event parameters and see if they match your expectations.

Step 4 – Using this data

Okay, so you have configured everything correctly, tested it and it works fine. Now, how do you put this data into use?

First, go into Google Analytics and find your rage click events. If you did as described in this post, you find them by clicking on Overview under Behavior –> Events, then find the “Rage Click” event category.

Click on the “Rage Click” event category and choose Event Action to see the elements on which the rage clicks happened.

Since elements with the same selectors might be on different pages, it might be helpful to apply a secondary dimension “Page”.

Now, all you need to do is to pick one element, copy its selector and go to the page seen in the secondary dimension.

Next, open the Javascript console of your browser (press F12) and paste your selector in the following format:

jQuery('YOUR>SELECTOR>HERE');

Hover over the line that was returned in the console and it should highlight the element in the DOM.

If you prefer element class over the selector, you can change the event action in GTM from the rc_element to GTM’s built-in Click Class variable.

Great! That’s it, should you need any help with tracking rage clicks using Google Tag Manager and Google Analytics, post your questions in the comments below.

Should you need further help with your analytics setup, take a look at our services.

Cover photo by Emmanuel on Unsplash

The post Tracking Rage Clicks Using Google Tag Manager and Google Analytics appeared first on Reflective Data.

In case your tool of choice is Google Optimize, you should be using their official anti-flicker snippet to minimize the flicker effect.

<!-- Anti-flicker snippet (recommended) -->
<style>.async-hide { opacity: 0 !important} </style>
<script>(function(a,s,y,n,c,h,i,d,e){s.className+=' '+y;h.start=1*new Date; h.end=i=function(){s.className=s.className.replace(RegExp(' ?'+y),'')}; (a[n]=a[n]||[]).hide=h;setTimeout(function(){i();h.end=null},c);h.timeout=c; })(window,document.documentElement,'async-hide','dataLayer',4000, {'OPT_CONTAINER_ID':true});</script>

Looking at the anti-flicker snippet (previously known as page-hiding snippet), you see that the default timeout is set to 4000ms. That means that, potentially, some of your visitors may see a blank white page for 4 seconds.

Since four seconds is a long time, many companies reduce it. A common timeout I’ve seen is 2 seconds. Changing the maximum timeout to a shorter period, although, means that there can be a good amount of people to whom the snippet occasionally times out.

When the anti-flicker snippet times out, no experiments will be loaded on that page load (people will see the control for all experiments and personalizations) which can lead to a user experience where a single user is seeing different variations in single session.

What you probably want to know is the optimal timeout period for your website. To figure that out, you’d need to know how many timeouts occur with your current setup. Let’s take a look how to set that up using Google Tag Manager and Google Analytics.

Before moving forward, you should read about how the anti-flicker snippet works.

Catching the timeout

To make this a little bit easier for us, Google has added a variable in the data layer that indicates whether the snipped has timed out or not.

You can access this variable like so:

window.dataLayer.hide["GTM-xxxxxx"] where GTM-xxxxxx stands for your Optimize container ID.

To make sure it’s not too easy, though, they don’t push this variable into the data layer as you normally would. This means you can’t access it using GTM Data Layer variable nor the Custom Javascript variable.

What you need is a Custom Javascript function variable in GTM to check the value.

Here’s the function you can use

function () { 
    if ( window.dataLayer.hide ) { 
        return window.dataLayer.hide["GTM-xxxxxxx"]; 
    } 
}

And here is the same function implemented in Google Tag Manager.

PS! In the beginning, the value is always true, it turns to false if the timeout period is over and it managed to load Optimize before this happened (the snippet did not time out).

Our recommendation is to check the value on GTM’s default DOM Ready event.

Sending data to Google Analytics

This part is quite straightforward for anyone that has used GTM to send custom events to Google Analytics so let’s get right to it.

1. Create the DOM Ready trigger

In case you haven’t configured this default trigger in GTM, go ahead do it now.

2. Create a new Tag for sending the event to Google Analytics

Create a new Tag with a tag type of Google Analytics: Universal Analytics and choose track type of Event.

These are the recommended event parameters but you can use whatever works for you.

Category: optimize

Action: timeout

Label: true/false (using the Custom Javascript function variable you created before)

Non-Interaction Hit: True (because it’s not related to a user action)

That’s it. This is all you need to start tracking Optimize anti-flicker snippet timeouts using Google Tag Manager and Google Analytics.

How to use this data?

Your goal should be to find the optimal timeout duration for your website and audience. The percentage of users who experience the timeout should be kept under 10% and you might consider excluding those with timeouts from your post-analysis process using a custom segment in Google Analytics.

Any questions or ideas? As always, feel free to post them in the comments below.

The post Tracking Optimize Anti-Flicker Snippet Timeout in Google Analytics appeared first on Reflective Data.