In this article, originally posted on medium.com, we’re going to walk through the reasons you might want to access raw Google Analytics data in BigQuery and a few solutions that will get you there.

***

Google Analytics is, without a doubt, the most popular tool in the digital analytics segment. Besides being the best tool for tracking e-commerce, content and lead-gen websites, with some modification you can use Google Analytics for tracking SaaS and other web apps as well.

While being a really powerful and flexible tool, Google Analytics, too, has a few shortcomings. Perhaps the biggest problem every advanced user faces is the fact that there is no way to access raw hit-level data. The second major issue with Google Analytics is related to sampling and data collection limits.

Now, let’s take a look at some of the ways you can overcome these shortcomings without spending a ton on Google Analytics 360 — a premium version of Google Analytics that, to some extent, also eliminates these issues.

PS! If you already have GA 360, these techniques will give you an even more robust and flexible tracking solution.

Table of Contents

– Sampling in Google Analytics
– Ways to avoid sampling in Google Analytics
– Why do you even need the raw data
– How to get access to unsampled hit-level raw data
– What about the price

***

Read the full post on medium.com

The post [Medium] Hit-Level Unsampled Google Analytics to BigQuery Without 360 appeared first on Reflective Data.

The main limitations of Google Analytics are related to sampling and data collection limits. Most affected are companies that can’t afford the premium 360 version of Google Analytics (~150k/year) but still have a good amount of traffic visiting their websites. In general, Google Analytics properties with >1M sessions/month or >10M hits/month are being affected by some heavy sampling and data collection limits.

In this article, we’re going to cover the different types of limitations present in the free version of Google Analytics and provide solutions/workarounds to all of them. Oh, and the solution, in most cases, does not include buying the 360 version.

Types of limitations in Google Analytics

Here’s a quick overview of the different types of limitations that come with the free version of Google Analytics.

Sampling in reports

Sampling on the reporting level means that, even if Google Analytics collected 100% of the hits, the reports you are seeing in the user interface are not based on 100% of those hits/sessions. In general, sampling starts if the period you are looking at contains more than 500k sessions in total. It can start a lot sooner, though, when more advanced custom segments are in use.

Data collection limits

From the official documentation.

If a property sends more hits per month to Analytics than allowed by the Analytics Terms of Service, there is no assurance that the excess hits will be processed. If the property’s hit volume exceeds this limit, a warning may be displayed in the user interface and you may be prevented from accessing reports.

Data collection limits that apply to all free Google Analytics accounts.

• up to 10M hits total per month
• 200,000 hits per user per day
• 500 hits per session

Data processing latency

Processing latency in Google Analytics is up to 24-48 hours. Standard accounts that send more than 200,000 sessions per day to Analytics will result in the reports being refreshed only once a day. This can delay updates to reports and metrics for up to two days.

This makes Google Analytics quite useless for things like monitoring real-time performance, detecting usability issues, automated reports on an hourly basis, real-time product recommendations, and other use cases that require fresh data (i.e. machine learning, marketing automation).

Custom dimensions and metrics

There are 20 indices available for different custom dimensions and 20 indices for custom metrics in each free Google Analytics property. 360 accounts have 200 indices available for custom dimensions and 200 for custom metrics.

While 20 is plenty for a small business with a simple website, this number can become quite limiting if you need custom dimensions for things like A/B testing data (one for each live experiment), e-commerce variables, user-specific information, and more.

Aggregated metrics and no access to raw data

Unfortunately, in the free version of Google Analytics, there is no way to access the raw hit-level data. This means that if the way it calculates certain metrics or aggregates hits into sessions doesn’t suit your business requirements then you’re out of luck – without access to raw data, you can’t define your own calculations or aggregations.

Also, in case you would like to analyze the user journey of a specific visitor, there is no way to easily query and analyze all hits from one visitor in a timely order.

Google Analytics 360 does have a BigQuery export but, by default, the metrics and sessions are still defined and pre-aggregated just like in the reports.

Hit payload limited to 8000 bytes

Depending on your analytics setup, some of your hits may include a lot data – ecommerce product impressions, several custom dimensions etc. The Google Analytics hit payload size is limited to 8000 bytes, this means that all hits greater than this will be ignored by Google Analytics’ data processing engine and your dataset ends up being incomplete. What’s worse, Google Analytics doesn’t alert you when this is happening!

This affects both the free and 360 versions of Google Analytics.

Workarounds to sampling and other limitations

Now, let’s take a look at the same list of limitations and provide a solution/workaround to each of them.

Sampling in reports

Option 1 – Divide your queries into smaller chunks and join data using another tool (Excel, Pandas, R). This can be done manually in the UI, using the Reporting API, or by using a library for your favorite programming language (i.e. gago for Go) that has an “anti sampler” feature built-in.

Option 2 – Collect raw hit-level Google Analytics data in your data warehouse (i.e. BigQuery). This process is known as parallel tracking and functions by duplicating all hits that are sent to your Google Analytics property into your data warehouse. This allows you to write ad-hoc queries in standard SQL without any sampling.

Data collection limits

Option 1 – To get past the 10M hits/mo, 200,000 hits per user per day and 500 hits per session limits, the same Google Analytics parallel tracking solution can be used. This means that even if Google Analytics starts to skip some hits, they’re still available in your data warehouse and ready to be queried.

Option 2 – Set up multiple Google Analytics properties. This can be a little tricky but it is possible to divide your Google Analytics integration into multiple properties. Depending on how much traffic your site gets, you might divide all hits from the first week of the month into one property, hits from the second week into another property etc. Alternatively, you could divide hits based on the Client ID into as many groups/properties as needed. Then, you can use the Reporting API to pull data from all properties and join them using Excel, Pandas or some other tool.

Data processing latency

If you need to monitor the site’s performance in real-time or need data for product recommendations or other machine-learning efforts, the 24-48 hour delay is probably way too long. Your only option to get access to near-real-time Google Analytics data is to use the parallel tracking technology. With parallel tracking, processed hit data will be available in ~5 seconds after it was collected from your site. This allows you to build alert systems, monitor performance in real-time or feed your machine learning algorithms with fresh data at all times.

Custom dimensions and metrics

As mentioned earlier, Google Analytics free version limits you to 20 indices of custom dimensions and metrics. If you hit the limit and need to add more, your first action should be to check whether there’s a dimension or metric that you no longer need. If that’s not the case, your only solution is to either upgrade to Google Analytics 360 (to get 200 metrics and dimensions) or to use parallel tracking and get access to unlimited custom metrics and dimensions. When using parallel tracking, dimensions and metrics with indices 20+ will not show up in Google Analytics reports but are available in your data warehouse.

Aggregated metrics and no access to raw data

If you’re not happy with how Google Analytics aggregates data into sessions or calculates certain metrics, you need access to the raw hit-level data that allows you to define your own rules for almost any kind of aggregation or calculation. There are two options you can sue to get access to raw hit-level Google Analytics data.

Option 1 – You guessed it! Parallel tracking sends the raw hit-level Google Analytics data into your data warehouse (with no sampling) and you can use standard SQL to query your data and tools like Excel, R or Pandas to make all sorts of calculations and aggregations. BigQuery has a native connection with most BI and data visualisation tools so reporting is easy and flexible.

Option 2 – If your Google Analytics property is not affected by the data collection limits (it gets <10M hits/mo), you could configure your custom dimensions to collect data like Client ID, Session ID, Hit Timestamp and Hit Type. With all these dimensions available in your dataset, you can use the Reporting API to pull the raw hit-level data into your local machine or data warehouse. If your property receives more than 10M hits a month, this option is only available if you split your data between multiple properties.

Hit payload limited to 8000 bytes

If your enhanced ecommerce transactions include many products or you have lots of custom dimensions or for some other reason some of your hits exceed the 8000-byte limit then you are losing data. Google Analytics will, without any warnings, simply skip those hits. You have three options for solving this issue.

Option 1 – Send less data by removing some data points. Simo Ahava has written a great blog post on automatically reducing the Google Analytics payload length by removing some of the unnecessary data points and creating an order for removing other data points until the required payload size is reached.

Option 2 – Use the data import feature in Google Analytics to import extra information after the data has been sent to Google Analytics. This way you can send only the most important information (Client ID, transaction ID etc.) with the main hit and import other details (product information, custom dimensions etc.) later using the import feature. Check out this post by Dan Wilkerson for a detailed guide for doing this with Google Tag Manager.

Option 3 – With parallel tracking your hits are limited to 16 000 bytes (double from Google Analytics) and this can be raised even more on request.

Key takeaways

Now that you have a pretty good overview of the limitations that come with the free version of Google Analytics and some that come with the 360 (premium) version, it’s a good time to assess your own Google Analytics implementation(s). Does any of the limitations affect your data and its accuracy? May some of the limitations become an issue in the future?

Knowing the limitations of a tool is extremely important. You may be investing a lot of time and money into a setup that may not be sufficient anymore as your business grows.

With all its limitations, Google Analytics is still an extremely powerful tool that has an awesome community. If sampling or some of the other limitations are an issue for you, I believe you can fix them with some of the solutions suggested in this blog post.

Most of the sampling and data collection limitations of Google Analytics are solvable with parallel tracking, a system that sends raw hit-level Google Analytics data into BigQuery (or any other data warehouse) in real-time.

***

Should you have any other issues with Google Analytics or have an alternative solution to any of the problems mentioned in this post, please share your thoughts in the comments below.

The post How to Avoid Google Analytics Sampling and Data Limits? appeared first on Reflective Data.

Cloud databases (i.e. BigQuery) and various data management tools have made it possible for marketers to build data pipelines without a big investment in hardware, software, and custom development.

In this article, we will walk through the steps of gathering data from Google Analytics, Google Ads, Google Search Console, Facebook, CRM and several other sources into Google BigQuery data warehouse and making it the single source of truth for all marketing data.

Takeaways

• Learn why most businesses need a marketing data warehouse
• Learn how to plan the best marketing data warehouse for your business
• Learn the basics of working with marketing data stored in BigQuery

Why (almost) every business needs a marketing data warehouse?

In the past few years, the number of marketing tools an average marketing person uses has grown rapidly. Doesn’t matter whether it’s a SaaS, lead gen or an e-commerce business, besides one or two analytics platforms there usually are a few ads platforms in play, a CRM, CMS, several social media platforms, an email automation system and probably a few more tools and platforms.

All of those tools and platforms are there to make our work as marketers, business owners or data analysts easier and more effective. In reality, though, you will end up having a bunch of silos – systems that don’t really communicate with each other and rarely agree on any of the important KPIs or metrics.

Data silos are the source of confusion and disagreement between teams within a business, leading to a situation where, at the end of the day, no-one knows which tool or numbers they can trust.

The solution? A marketing data warehouse.

Below are a few reasons that should explain quite vividly that your business needs a marketing data warehouse.

– Single source of truth: the main benefit you get from having all of your marketing data in one central data warehouse is that every team in your business will look at the same metrics, calculated the same way. This benefit alone will save you and your colleagues from lots of misunderstanding and possible conflicts between teams and functions.

– Access to raw data: most marketing tools show you aggregated data and calculated metrics. While it’s okay for a more basic user, it is definitely limiting your ability to do your own calculations. It is also the main reason why data from different tools never match – they simply measure and calculate the core metrics like users, sessions, conversion rate, revenue etc. in a different way. When working with raw data, you define your own aggregation and calculation rules – this should be agreed on company level and leads to everyone looking at the same numbers.

– Visualization and BI tools: sadly, not all marketing tools have a connector with most major data visualization and BI tools (Tableau, Looker, Google Data Studio etc.). Working with CSV or Excel files isn’t automatic or very scalable. Data warehouses like BigQuery have a native integration with nearly all better-known BI and data visualization tools.

– More accurate attribution: tools like Google/Adobe Analytics are great when you’re just getting started with marketing channel attribution. What most tools lack, though, are access to all user touch-points (online and offline) and the ability to use custom attribution models (FBA, ML-based, Markov etc.).

– Machine learning: with tools like BigQuery ML, Auto ML and others being widely available, getting started with machine learning has never been easier or more affordable. The only part that’s missing, though, is access to high-quality raw hit-level data that is required for training your models. With all your marketing data in your data warehouse, you can create models from product recommendations to smarter remarketing campaigns.

– Manual data joining doesn’t scale: exporting data from various sources and joining them in Excel/Sheets can give you interesting insights you wouldn’t have seen in any single tool but let’s be honest, this system isn’t very scalable. Working with a data warehouse, you can pre-join commonly combined data into views or pull everything together in a dashboard.

– A complete overview of the user journey: no one tool knows everything about your users thus looking at them doesn’t show you the complete user journey. Combining data from all tools and sources, both online and offline gives you a much more detailed picture of the entire user journey. Furthermore, this gives you much more accurate long term metrics like users’ lifetime value.

***

I could go on for long writing about the benefits that explain having a proper marketing data warehouse. What about you? Let us know of the reasons your business needs (or already has) a marketing data warehouse in the comments below.

How to plan and get the best marketing data warehouse for your business?

Having a marketing data warehouse is becoming the new norm. Setting one up, though, is not a trivial task.

There are some tools that offer sort of a plug-and-play solution but just like every business is different, so are their requirements for a marketing data warehouse. There is no good solution that would meet the requirements of all (or even most) businesses.

Mapping data sources

Before dealing with anything else, start by mapping all the data sources your business has. This may include, but isn’t limited to, Google Analytics, CRM, CMS, offline data, Ads platforms, email automation tools, SMS and push notification platforms etc.

After knowing your sources, try to group all of them into a document and sort them in a logical order. For example, (1) analytics platforms, (2) social media platforms, (3) email etc.

Next, think more deeply about each of the listed data sources. Take a look at their documentation and data structure. In your document, take note of what kind of data and in what format you’d need in your marketing data warehouse. Start by thinking of how your team is using this data today, what kind of dashboards you’ve built etc.

Furthermore, take note of how fresh each dataset should be in the data warehouse. Do you need it in real-time, hourly, or maybe a daily update is fine? Probably you’d want your click-stream data to be as fresh as possible while having your ad cost data update once a day may be fine.

As an extra, I recommend you check if each data source has a public API or if they allow automated data export. If that’s not the case, it’s highly likely you can’t easily export data from that tool and I’d look for an alternative for that tool. Every decent marketing tool should have a public API for exporting data these days.

Mapping use cases

Based on the list of data sources you just listed, it is now time to think about all the great things you could do with a marketing data warehouse that includes all the data.

Once again, I recommend you start a document and structure it logically based on some category. For example, (1) user behavior analysis, (2) attribution, (3) alerts etc.

Of course, you don’t have to think about everything beforehand but it helps you later on deciding the right schema, load frequency (batch load vs stream), pre-calculated views, metrics etc.

Start by thinking of all the ways you are currently using the data available within each marketing tool your company has. Next, think of ways you’ve joined the data in the past using tools like Excel or Google Sheets. Then, I recommend, you go completely creative. Think of things you were never able to get from your data but would really like to. For example, analyzing user journey across all touchpoints, online and offline.

Build or buy?

The truth is, that building a data warehouse has never been easier nor cheaper. Every cloud platform has its own offering and several solutions have been built on top of major cloud platforms by other companies.

To be honest, setting up a data warehouse is actually the easy part. What makes it difficult though, is getting data in the data warehouse – automatically, reliably, fast and free of duplicate records. This part is known as a data pipeline and this is the system that feeds your data warehouse and, ultimately, your business with trustworthy data.

Now, where do you get a solid data pipeline? You have three options.

1. Build: if your company has enough resources (money, time, developers, data scientists, data engineers) then building everything in-house might be a good idea. There are some useful building blocks available in most cloud platforms (i.e. Cloud Dataflow, Pub/Sub, Cloud Functions in GCP). What I would recommend, though, is to make sure you have at least one person in the team that has led a data pipeline build before or at least someone you could consult with. Otherwise, there are things that will probably go wrong in your first attempt.

2. Ready-made solutions: depending on the level of customization you require, some of the ready-made data pipeline solutions might fit most of your needs. Some tools to check out are Stitch, Fivetran, Funnel and Segment. All of them do a great job at what they’re built for – offering fixed or semi-flexible data pipeline to feed your data warehouse with data. Unfortunately, though, each of them has their shortcomings. For example, none of them is able to send raw hit-level Google Analytics data into your data warehouse and process it into sessions based on your rules. Oh, and setting such a tool up can still be a considerable amount of work.

3. Data Pipeline as a Service: to gain full control of your marketing data, you need a system that’s tailored to your business’ needs. A way to get there is to hire a company that has all the necessary infrastructure (connectors, integrations, batch loaders, monitoring systems etc.) and is willing to take a personal approach with every client. This means getting to know your business, interviewing your team and building a solid data pipeline using the building blocks they already have – and code everything they don’t.

***

Start by mapping your data sources and use cases as described above. Then see if any of the ready-made solutions could fit your needs. If you need a more customized/advanced solution, think about your internal resources. If you need help with deciding which way to go or want to learn more about data pipeline as a service, talk to experts at RD.

The basics of working with marketing data stored in BigQuery

As BigQuery is one of the most popular data warehousing solutions for marketing data, let’s use it in our examples here as well.

Having all of your marketing data available in BigQuery is great but quite useless if don’t use it or if you only use it for things that are easily doable within Google Analytics or some other tool. E.g. for checking how many sessions your website received, the bounce rate etc.

You have to use SQL

Coming from a tool that lets you build reports by dragging and dropping various metrics, dimensions and filters in the editor, getting started with BigQuery can look like a bit of a learning curve.

When going to BigQuery, this is literally all your welcomed with.

No fancy data explorer features or a drag and drop report builder. Just a blank white text input area for your SQL queries.

Yes, there are tools like Looker and Google Data Studio that connect with BigQuery and automatically understand your schema. Using these tools with little or no SQL skills, though, you’re likely better off using the reporting features in your data sources (Google Analytics, CRM, Ads) instead.

The power of BigQuery becomes apparent only through (quite advanced) SQL queries. This is what allows you to join various sources, calculate your own metrics, define your own attribution models and so much more.

As a marketer, you really have two options: (1) learn SQL (here’s a super good getting started guide from Mode) or (2) work closely with a developer/data scientist/analyst that knows SQL really well.

No matter which route you take, know that SQL is no rocket science but there is no way around it when getting into more advanced data analysis.

Now, let’s take a look at a few use cases for a marketing data pipeline a.k.a your single source of truth.

Analyzing the journey of a single user

In this example, we are using raw hit-level Google Analytics data sent to BigQuery using RD Marketing Data Pipeline.

What’s good about BigQuery is that, unlike Google Analytics, you can use it to work with personally identifiable information (PII) – for example, our dataset includes users’ email addresses. This makes it easy to connect incoming leads with their activity on the site.

Let’s start by looking at how many sessions our lead has had on our website in the selected time frame. For this example, we’re connecting all sessions using email address, IP address and the client ID.

Seems like this user has visited the site a whopping 151 times (the number of sessions)!

Next, you might be interested in where all those sessions initiated. You could do so with a query like this.

While this gives you decent overview of how they found your site, you should also look into the landing pages (especially for organic, to understand what they’re looking for) and all the pages they’ve visited prior to submitting the contact form.

At Reflective Data, we do a similar analysis with all leads before we contact them – just to understand how much they already know and what they might be looking for.

Custom attribution models

Most tools out there default to one specific attribution model. For example, Google Analytics uses the last non-direct click model with a 6-month look-back window. Some tools let you modify the look-back window and/or compare different models but usually, the options are quite limited.

An attribution model is a really important part of marketing analysis. Different models can yield completely different results. Picking or building an attribution model should be a well-reasoned decision that’s based on the actual behavior of your customers, buying cycle, types of campaigns etc.

In the following example, we are using the same dataset as in the previous example. The default attribution model used there is “last interaction” but we’d like to compare it against the more commonly used “last non-direct model”.

First, let’s see the numbers using the “last interaction” attribution model.

And now let’s see the numbers using the “last non-direct click” attribution model.

The query here is a bit bulkier but it’s actually quite simple and logical when you take a closer look.

While the difference isn’t massive for this dataset, we can see that there’s a lot less direct/none traffic using the “last non-direct click” model – which makes total sense, because all direct clicks that came after a non-direct click were attributed to that other source.

***

We aren’t going in too much detail with all the attribution models you could build using the raw data in BigQuery but let me say that lately, many of our clients have requested us to build them all sorts of custom attribution models like linear and funnel-based or Markov, and even some machine-learning-based models.

Various data sources

While Google Analytics raw hit-level data (from GA 360 or using the parallel tracker) tends to be the most commonly used dataset in BigQuery or any other marketing data warehouse, you can (and should) include data from all marketing tools and platforms. This allows you to join data from various sources and spot insights and connections that weren’t possible before.

For example, one might capture a daily snapshot of their Twitter account in BigQuery and run a quick query to see the data for a current month.

The data above would fit perfectly in a social media marketing dashboard.

In another example, let’s query a table containing Mailchimp data to see how many new subscribers were acquired per day.

As you’ve probably realized, you can (and should) send data from all marketing tools into one unified data warehouse. As BigQuery and most other solutions are available at a rather reasonable pricing point, I would recommend collecting data from all sources, even ones you don’t plan on using in the nearest future because it’s always good to have access to historical data for a more advanced ad-hoc analysis.

Moving forward

If your company already has a marketing data warehouse that’s built on a modern cloud infrastructure and contains data from all the marketing tools that your business is using – good job! You’re working at a good, forward-thinking company. Just make sure this data is actually put in use as well.

If that’s not the case and your company doesn’t have a proper marketing data warehouse in place, start by thinking of all the marketing (and other relevant) tools that are producing data. You’ll probably realize that what you have is a bunch of silos, limiting the efficiency of your teams and holding your business back from being a truly data-driven one.

Whether you’re planning on building the data pipeline that feeds your data warehouse with high-quality data yourself or to buy it in as a service, it never hurts to consult with the experts that have been building data management systems for years.

If you have any questions, feel free to shoot them in the comments below.

The post Making BigQuery a Single Source of Truth for Marketing Data Analysis appeared first on Reflective Data.

All of those tools are supposed to make our work as marketers, business owners or data analysts easier and more effective. In reality, though, you will end up with a bunch of silos – systems that don’t really communicate well with each other and almost never agree on any of the important metrics.

Data silos create confusion and disagreement between teams, leading to a situation where, at the end of the day, no-one knows which tool or numbers to trust.

You need a marketing data warehouse.

As time goes on and your business grows, the number of marketing tools and platforms involved is likely to grow as well. Without one central location for all data, this means more and more data silos and less efficiency.

A marketing data warehouse is the only real solution to break these silos. To put it very simply, you need a system that sends all of your marketing data, from all tools, into one central location.

Below are some of the reasons that should make it clear that your business needs a marketing data warehouse.

– Single source of truth: the main benefit you receive from having a marketing data warehouse is that everyone from your business will look at the same metrics, calculated the same way and coming from one location. This alone will save you from a lot of confusion and possible conflicts between teams and functions.

– A complete overview of the user journey: no one tool knows everything about your users thus looking at them doesn’t show you the complete user journey. Combining data from all tools and sources, both online and offline gives you a much more detailed picture of the entire user journey. Furthermore, this gives you much more accurate long term metrics like users’ lifetime value.

– Access to raw data: most marketing tools’ user interfaces show you aggregated data and calculated metrics. Besides limiting your ability to do your own calculations, this is also the main reason why data from different tools never match – they simply measure and calculate metrics like users, sessions, conversion rate, revenue etc. in a different way. When working with raw data, you define your own aggregation and calculation rules – this should be agreed on company level.

– More accurate attribution: tools like Google Analytics are great when you’re just getting started with marketing channel attribution. What most tools lack, though, are access to all touch-points (online and offline) and the ability to use custom attribution models (Markov, FBA, ML-based etc.).

– Manual data joining doesn’t scale: exporting data from various sources and joining them in Excel/Sheets can give you interesting insights you wouldn’t have seen in any single tool but let’s be honest, this system isn’t very scalable. Working with a data warehouse, you can pre-join commonly combined data into views or pull everything together in a dashboard.

– Visualization and BI tools: unfortunately, not all marketing tools have a connector with all major data visualization and BI tools. Working with CSV files and Sheets isn’t automatic nor scalable. Data warehouses like BigQuery, have a native integration with nearly all decent BI and visualization tools.

– Machine learning: with tools like BigQuery ML, Auto ML and others, machine learning has become available to the masses. The only part that’s missing, though, is access to high-quality raw hit-level data that is required for training your models. With all your marketing data in your data warehouse, you can create models from product recommendations to smarter remarketing campaigns.

***

I could go on for hours writing about the benefits that favor having a proper data warehouse. What about you? Let us know of the reasons your business needs (or already has) a data warehouse in the comments below.

How to set up a data warehouse?

Having a data warehouse is kind of a no-brainer these days. That doesn’t mean, though, that setting one up is simple as 1-2-3.

Yes, there are some tools that are kind of plug-and-play solutions. But just like every business is different so are their requirements for a data warehouse. There is no real one size fits all solution here.

Mapping data sources

Before thinking of anything else, map all the data sources you have. This can include but isn’t limited to: Google Analytics, CRM, CMS, offline data, Ads platforms, email tools, SMS and push notification platforms etc.

Next, try on group all data sources into a document and sort them somehow in a logical order. For example, (1) analytics platforms, (2) ads platforms etc.

After writing down and sorting your data sources, think about each one. Take a look at their documentation and data structure. In your document, take note of what kind of data and in what format you’d need in your data warehouse. Start by thinking of how you’re using this data today.

Furthermore, take note of how fresh each dataset has to be. Do you need in real-time, hourly, or maybe a daily update is fine?

As an extra, I recommend you check if your data source has a public API or if they allow automated data export. If that’s not the case, it’s highly likely you can’t easily export data from the tool and I’d look for an alternative for that tool. Every decent marketing tool should have a public API for exporting data.

Mapping use cases

Based on the list of data sources you have available, now it’s time to think about all the use cases for a data warehouse that includes all of them.

Again, I recommend you start a document and structure it logically based on some category. For example, (1) attribution, (2) user behavior analysis, (3) alerts etc.

Of course, you don’t have to think about everything beforehand but it helps you later on deciding the right schema, load frequency (batch load vs stream), pre-calculated views and metrics etc.

Start by thinking of all the ways you are currently using the data available within each marketing tool. Next, think of ways you’ve joined the data in the past using tools like Excel or Google Sheets. Then, I recommend, you go completely creative. Think of things you were never able to get from your data but would really like to. For example, analyzing user journey across all touchpoints, online and offline.

Build or buy?

Building a data warehouse has never been easier. Every cloud platform has their own offering and several solutions have been built on top of major cloud platforms.

To be honest, setting up a data warehouse is the easy part. What makes it difficult though, is getting data in the data warehouse – automatically, reliably, fast and without duplicates. This part is known as a data pipeline and this is the system that feeds your data warehouse and, ultimately, your business with trustworthy data.

Now, where do you get a solid data pipeline? You have three options.

1. Build: if your company has enough resources (money, time, developers, data scientists, data engineers) then building everything in-house might be a good idea. There are some useful building blocks available in most cloud platforms (i.e. Cloud Dataflow, Pub/Sub, Cloud Functions in GCP). What I would recommend, though, is to make sure you have at least one person in the team that has led a data pipeline build before or at least someone you could consult with. Otherwise, there are things that will probably go wrong in your first attempt.

2. Ready-made solutions: depending on the level of customization you require, some of the ready-made data pipeline solutions might fit most of your needs. Some tools to check out are Stitch, Fivetran, Funnel and Segment. All of them do a great job at what they’re built for – offering fixed or semi-flexible data pipeline to feed your data warehouse with data. Unfortunately, though, each of them has their shortcomings. For example, none of them is able to send raw hit-level Google Analytics data into your data warehouse and process it into sessions based on your rules. Oh, and setting such a tool up can still be a considerable amount of work.

3. Data Pipeline as a Service: to gain full control of your marketing data, you need a system that’s tailored to your business’ needs. A way to get there is to hire a company that has all the necessary infrastructure (connectors, integrations, batch loaders, monitoring systems etc.) and is willing to take a personal approach with every client. This means getting to know your business, interviewing your team and building a solid data pipeline using the building blocks they already have – and code everything they don’t.

***

Start by mapping your data sources and use cases as described above. Then see if any of the ready-made solutions could fit your needs. If you need a more customized/advanced solution, think about your internal resources. If you need help with deciding which way to go or want to learn more about data pipeline as a service, talk to experts at RD.

Costs

No matter what path you choose, your costs will mostly depend on the number of different data sources and the amount of data you’re going to collect.

1. Build: the process of planning, building and testing a data pipeline is likely to take 1-3 months. Pricing for the cloud platform depends on usage but expect to pay at least $250/month. Most cloud platforms offer calculators to help you estimate the cost (i.e. Google Cloud Platform). You are probably not going to build all connectors yourself, therefore expect to pay $500 – $1,000 for tools like Stitch and SuperMetrics.

2. Ready-made solutions: some solutions start as low as $120/mo but the pricing grows rapidly into thousands based on your usage. Another thing to keep in mind is that some providers have free connectors and premium connectors, meaning you might have to pay extra for some data sources.

3. Data Pipeline as a Service: when hiring a company to build you a custom data pipeline, there will almost always be an initial setup cost involved. This depends heavily on the number of connectors and customization required. A very light setup usually starts around $500 and a decent pipeline is usually a few thousand $ (one-time fee). In most cases, the monthly cost depends on usage only. In fact, we’ve seen this to be the most cost-effective solution of the three in the long run. This is because there is no premium to pay for certain connectors and data streams are better optimized than in most in-house solutions.

How to move forward

As mentioned before, start by mapping your data sources and use cases. Then, based on your needs and resources available try to pick one of the paths listed in the previous section. If you’re not sure if a solution is right for you, I’d recommend contacting some service providers or independent data warehousing experts. You can start by posting your questions or ideas in the comments below this article.

Learn more about the Data Pipeline as a Service offered by Reflective Data and schedule a call with one of the experts today.

The post Why Every Business Needs a Marketing Data Warehouse and How to Set One Up? appeared first on Reflective Data.

Working with Google Analytics data in BigQuery has mostly been a privilege of those having a 360 version of Google Analytics. Its hefty price tag, though, has made that list quite short.

With Google Analytics Parallel Tracking Service from Reflective Data, having access to Google Analytics data in BigQuery has become a lot more affordable. (you can get a quote from the previous link)

Now, regardless of whether you have Google Analytics 360, a Parallel Tracking system from Reflective Data or some other integration in place, the structure of the data in BigQuery is quite similar.

In this article, I’m giving you a tour of the features in BigQuery and some of the ways how you can leverage them when working with your Google Analytics data.

Take a look at this article if you want to learn more about sending your Google Analytics data into BigQuery first.

In case you don’t have access to any BigQuery dataset containing Google Analytics data, you can check out the Google Analytics sample dataset for BigQuery. The sample dataset provides an obfuscated Google Analytics 360 dataset that can be accessed via BigQuery. It’s a great way to look at business data and experiment and learn the benefits of analyzing Google Analytics 360 data in BigQuery.

In this blog post, though, we are going to work with the demo dataset generated by Reflective Data’s Parallel Tracking System. As mentioned before, the structure of the data is pretty much the same.

Creating and running your first query

With Reflective Data’s Parallel Tracking System, you will have three main tables in your BigQuery dataset. Let’s take a quick look at each one of them with a simple query.

Table: raw_hits

This is the most up to date table and contains raw data coming in from the website, app or via the measurement protocol. Data is available withing ~5 seconds after it was sent.

raw_hits table contains only four columns.

• ua – the user agent string of the client
• ip – the ip address of the client
• q – the actual Google Analytics hit payload
• timestamp – when data hit the processing engine (~50ms after it was sent)

For example, we could query the number of hits for every hour in a given day.

Table: processed_hits

This is a processed version of the raw_hits table. Data in this table becomes available ~30 seconds after the hit was sent. This is still really-really fast and considered near-real-time data.

When it comes to columns, this table is much richer compared to the raw_hits table. It has all the time-related columns (date, hour, minute, timestamp etc.) converted into the correct timezone. Some of the data is in a special format known as a record in BigQuery. This includes columns like traffic_source, page, event, device, transaction etc. In fact, this table has a column for every parameter you can send with a Google Analytics hit. Added are some data points normally not available in Google Analytics (ip address, user agent, client ID, user ID etc.)

List of paramaters/columns as seen in Measurement Protocol Reference.

As an example, we are querying top 10 visitors (by client ID) that created the most hits in a given day.

Isn’t that easy? and cool, too!

I certainly like working with my Google Analytics data using SQL queries. Okay, let’s move on.

Table: processed_sessions

This table is, for sure, the richest of the three. It has all the columns from processed_rows but also includes quite a few new ones. These are the columns that require more processing (ip address to geolocation etc.) and/or the session to be ended (total hits in a session etc.). For these reasons, this table is usually updated a few times a day and a final version is generated at the beginning of the next day.

If you aren’t completely sure how a web session is defined in Google Analytics, take a look at this article.

A good thing about raw data is that whether you like the fact that Google Analytics is mostly session-based or not, you have the freedom to calculate your metrics however you wish. You could also mix and match different techniques.

For example, we might query top country/browser combinations by the number of sessions and pageviews in a given month. Let’s see how that would look like with our demo dataset.

Creating and running more complex queries

The best thing about raw hit-level data is that you’re not limited by things like sampling or pre-defined metric/dimension combinations. Both of which are common in Google Analytics, both the UI and the Reporting API.

What’s more, with BigQuery you can load in data from almost anywhere and mix it with your analytics data in any way you like. This may include data from your CRM, CMS, Ads platform or other analytics tools.

Google Analytics’ data model is structured so that session-based dimensions (like source/medium) don’t play well when combined with user-level or page-level dimensions and metrics. And there’s a limit to the number of dimensions we can see side-by-side: two dimensions is usually our limit in the interface, five in custom reports, and the API allows for seven dimensions. With BigQuery, there are no such limitations.

For example – ecommerce customers may have trouble pulling exact stats of the number of users from Social Media that saw a product page and then subsequently purchased the same product. BigQuery users can handle that with a single query.

By Alex Moore, source

One of the queries that almost always ends up being inaccurate (due to sampling) in Google Analytics is when you want to analyze all hits triggered by a single user. With BigQuery, this query is fast and simple, and there is no sampling. Never.

First, let’s go ahead and query some totals.

Now, let’s take a look at all of the hits in chronological order. This will give us a detailed overview of their user journey.

I believe you can see how analyzing a user journey like this can be very useful. For example, when a new lead comes in, before contacting them you could check what content they’ve already interacted with.

***

For someone that has worked mostly with the Google Analytics UI and not with the raw data before, there are some things that can be a bit confusing in the beginning. For example, not all of your common metrics are available as a column in BigQuery. Even the really basic ones, like bounce rate. That means you have to define them in your query.

Let’s take a look at the bounce rate for the top 10 landing pages in our sample dataset.

Calculating your own metrics like this can be a bit frustrating in the beginning but you actually only have to write most queries once and can later save them as views or saved queries in BigQuery. A benefit of defining your own metrics is that you are not limited by pre-defined metrics in Google Analytics. As you can see, in the query above we defined bounce as a session that has less than 2 pageviews but we could also include a rule that there can’t be any interaction hits in the session. We could also play with session duration or other data available in BigQuery.

A true value of having your analytics data in BigQuery, though, is that you can easily join it with other data that you’ve also sent to BigQuery.

For example, you could use Google Ads Data Transfer tool for BigQuery to automatically send all of your Google Ads data into a BigQuery dataset. This is especially useful if you’re using Google Ads auto-tagging solution because that means the only connection between your Google Analytics and Ads data is the gclid ID which is unique for every click.

With a simple query, you can join your processed_sessions table with ClickStats table.

After joining two datasets, you can combine any data points that are available in either dataset. The most important one, perhaps, is the cost data from the Ads dataset. This would allow you to calculate the amount of money you spent on acquiring each user.

Analyzing BigQuery data in a BI tool

Not everyone can write complex SQL queries to access their analytics data in BigQuery. Luckily, BigQuery has a native connector with almost all of the major BI and data visualization platforms.

My default recommendation is Data Studio. It’s free, full of powerful features and since it’s also from Google, you get a smart caching layer between Data Studio and BigQuery which can significantly lower your BigQuery costs.

Connecting BigQuery and Data Studio is really simple. All you have to do is create a new source in Data Studio and select BigQuery. It will automatically show you the list of Google Cloud projects you have access to and from there you can easily navigate to your datasets and tables.

Once you found the correct table, you can either choose the fields you want to include in your report (if not sure, choose all) or write your own query.

In our example, we connected two tables (processed_hits and processed_sessions) and selected all fields available in these tables.

Now, let’s create a quick time-series chart to see the number of hits of each hit type per day in our selected time period. Settings for a chart like this are really simple.

And this is how the chart itself looks like. Nothing special but allows you to have a quick overview of your traffic and spot any anomalies in tracking.

Another useful chart I recommend setting up is a time-series comparison between data in Google Analytics and BigQuery. This means you have to blend two data sources.

As you can see, the numbers are pretty close.

For more advanced dashboards you might want to write a custom query on a Data Studio data source level. This way you can pre-join different tables and/or datasets.

***

While being more difficult than using the Google Analytics UI, the freedom you get from having access to raw hit-level data in BigQuery is definitely worth it. As mentioned at the beginning of this article, you don’t necessarily have to spend a fortune on Google Analytics 360 because with Reflective Data’s Google Analytics Parallel Tracking System you can send your analytics data into BigQuery for a fraction of the money.

Should you have any questions about working with Google Analytics data in BigQuery, post them in the comments below and someone from our team will get back to you.

If you want to learn more about the Parallel Tracking System, click here.

The post How to Query and Analyze Google Analytics Data with BigQuery appeared first on Reflective Data.

  If you need help with planning or building a data pipeline or a data warehouse then get in touch for a free consultation session with one of our top data engineers!

These days, data is at the core of almost all successful companies. The system that feeds those companies with data is known as a data pipeline.

This blog post is for anyone eager to learn more about data pipelines, planning to build a new pipeline or upgrade their current setup.

An end-to-end analytics data pipeline is a secure and reliable mechanism that is responsible for feeding your business with valuable data that can be used for reporting, analysis, machine learning or any other activity that requires accurate data about your business.

Every business is different and so are the analytics data pipelines that best suits their needs. Our model of an enterprise-grade, fully customizable data pipelines are divided into six logical steps. Let’s take a closer look at each one of them.

Events → Enrichment → Process → Manage & Monitor → Storage → Report & Visualize

Events

Events are the basis of most data pipelines, they trigger other actions and take the largest part (storage) of your data lake/warehouse.

There is a wide variety of events that may act as a source for your data pipeline. Here are the six main categories of sources that generate an ongoing flow of events.

• Website users
  • Click stream
  • Google Analytics parallel tracking
  • Form analytics
• Server events
  • Orders
  • Payments
• Mobile Apps
  • Real-time
  • Batch
• Ads
  • Google Ads
  • Facebook Ads
  • Twitter Ads
  • Others
• Feedback tool
  • Surveys
  • On-site polls
• 3rd parties
  • Testing tools
  • Personalization tools
  • Email providers
  • Call tracking

Events in data pipelines are usually invoked by webhooks that put all events into the processing queue where they are validated, enriched and batched.

For example, here’s a webhook for Google Analytics parallel tracking.

https://analytics-6785943.appspot.com/collect?v=1&_v=j79&a=1359602631&t=pageview&_s=1&dl=https%3A%2F%2Freflectivedata.com%2F&ul=en-us&de=UTF-8&dt=Homepage&sd=24-bit&sr=1920x1080&vp=1336x977&je=0&_u=SCCACEAjR~&jid=&gjid=&cid=1177657878.1573135112&tid=UA-3696947-1&_gid=775207114.1575821022&z=1779653917

The above webhook is quite similar to Google Analytics hit payload and is sent at the same time. It is then processed, enriched and streamed into the data warehouse.

All events are generally stored in three locations:

1. Log files – raw hits
2. Data Lake – Processed data (sometimes also enriched)
3. BigQuery – Processed & enriched data

This setup ensures us that we can re-process everything should there be something wrong with the enrichment process or should the data somehow get lost from the BigQuery instance.

Enrichment

To give more context to the raw event-level data, our systems are pulling data from various sources, including:

• CRM & CMS
• Google Analytics
• 3rd party APIs

For sources that don’t allow real-time access, we are using periodic batch load processes.

Data from various sources is either combined with the hit level data or sent to the data warehouse separately for query-level joining or processing after collection.

Process

Processing makes sure your data is secure, reliable, accurate and free from duplicates. To securely process your data, we are using the following tools from Google Cloud:

• Cloud Dataflow
• Cloud Pub/Sub
• Cloud Functions

Similar tools are also available in Amazon Web Services. There are several open-source solutions (often used by providers like Google and Amazon) but we prefer and recommend using managed services that can scale automatically. This allows us (and you) to focus on the data pipeline itself instead of maintaining the infrastructure.

We build the data processing queues to handle late-arriving data, duplicates and storage system outages in a way that best fits every use case. For example, people using your app on a 10-hour flight can produce a lot of events that are arriving and getting processed hours after they actually happened.

Manage & Monitor

To make sure everything is working as expected and to alert us and our clients if something is not, we are setting up a set of tools built in-house and from Google Cloud. Such as Stackdriver Monitoring.

Our goal is to provide our clients with the most accurate data with the least amount of latency.

In the pipeline management tools, you can see the pipeline’s structure, sources, processors, storage options etc. You would also see how data is moving between different elements and if there have been any errors or other incidents.

  If you need help with planning or building a data pipeline or a data warehouse then get in touch for a free consultation session with one of our top data engineers!

Storage

We’ve built the data pipeline so that it can send and store data in almost any database, data lake or data warehouse available. Here’s our standard recommendation.

• Google BigQuery
• Digital Analytics Platforms
  • Google/Adobe Analytics
• Long-term storage
  • Cloud Storage 
  • AWS S3

We make sure that your data stays safe and meets all the security requirements while giving you the full ownership and control over your data.

As mentioned earlier, we store raw hits as well to ensure that we can re-process everything should there be something wrong with the enrichment process or should the data somehow get lost from the BigQuery (or other reporting) instance.

Report & Visualize

Collecting data is pointless if it’s never being used to benefit your business. We think about the value our clients’ data has to provide in the earliest phase of planning and building the data pipeline. Our team works closely with our clients in order to figure out the KPIs, reports and dashboards that are needed for their companies growth.

Tools we are using for reporting, analysis and visualization include:

• Google Data Studio
• Jupyter Notebooks
• Redash
• Other BI tools

We build all analytics data pipelines and storage systems so that they could be connected with almost any BI tool available on the market. Wherever possible, we make data available for reporting in near real-time.

It’s no news that data is running most of the successful businesses these days and more and more companies as implementing modern data pipelines.

When thinking of a modern data pipeline, keep these keywords on mind:

• Serverless
• Scalable
• Real-time access
• Security
• Flexibility

Our team is happy to answer any analytics data pipeline related questions in the comments below.

Need help with planning or upgrading your data pipeline? Contact us and get a free consultation with one of our top data engineers.

The post Six Key Components of an Analytics Data Pipeline appeared first on Reflective Data.

In this article, we are covering everything you need to know about the problem of duplicate transactions, a root cause of skewed data in many Google Analytics instances.

Detecting Duplicate Transactions

Duplicate transactions in Google Analytics means simply that a single transaction was counted more than once. This can dramatically skew your data because along with the transaction count, this also inflates the revenue, quantity and other metrics directly related to transactions.

To check if your Google Analytics E-commerce setup is affected by duplicate transactions, let’s start by creating a custom report (Customisation –> Custom Reports –> New Custom Report) with the following settings.

Metrics: Transactions

Dimensions: Transaction ID

Hit “Save” and you should see your Duplicate Transactions custom report right away.

Firstly, choose a time frame. I’d recommend starting with 30, 60 or 90 days depending on how many transactions your site generates and how often your analytics setup changes.

Secondly, sort your table by most Transactions first.

If every transaction ID shows only one transaction then congratulations, in the selected time frame there were no duplicate transactions.

Every number greater than one is a sign of duplicate transaction. Now, to get an overview of the magnitude of the problem, compare the number of transactions against the number of rows in the table. The latter is the actual count of unique transactions.

In the above screenshot, there are some duplicate transactions but considering the total amount of transactions, the problem is minor.

If your duplicate transaction ratio is >10% you have a serious problem with your e-commerce numbers and before fixing it, you should take your numbers with a grain of salt.

PS! We should be thankful that Google has kept Transaction ID available as a dimension in almost all reports instead of hiding them as they do with the Client ID, User ID, Session ID and other really useful dimensions (that you should track using custom dimensions).

Fixing Duplicate Transactions

By far the most common reason duplicate transactions are happening is that a transaction hit is sent to Google Analytics every time the “thank you” page is loaded. There are people who bookmark this page and go back to check the order id for tracking or whatnot. Also, some companies send the “thank-you” URL in the order confirmation email.

To fix this, there are two possible solutions.

Option 1

Make the event trigger based on something coming from the back end, something that only triggers when the actual transaction is happening. For example, after the user has entered their credit card information and pressed “Complete payment” and your system has validated the transaction. I.e. just before redirecting the user onto the “thank you” page.

Option 2

Make your “thank you” pages to load only once. If you keep your transaction hits on the “thank you” page, make sure a user can see and visit the page only once. Every subsequent attempt to access this page should redirect them to some other page. This could be an order overview page or something similar (should still be order-related and contain order details).

…

Now, to take it one step further, you can make your e-commerce tracking even more reliable by sending transaction hits straight from the back end using Measurement Protocol. This way you can make sure that no transaction goes untracked or that someone is sending fake transactions towards your Google Analytics instance.

Another option is to use localStorage to keep track of transaction IDs that have already been sent to Google Analytics. This method is described in detail by Simo Ahava in this blog post.

Final thoughts

Google Analytics is an excellent tool for keeping track of your e-commerce business. What you need to keep in mind, though, is that skewed numbers can do more harm than having no numbers at all.

This is why you should audit (or have someone else do it for you) your Google Analytics and Enhanced Ecommerce setups regularly and compare the numbers in GA against the numbers in your back end and accounting.

If you see that the numbers don’t add up, see if you have any duplicate transactions by creating a simple custom report seen in the beginning of this blog post.

Have any questions or thoughts about detecting and fixing duplicate transactions in Google Analytics? Post them in the comments below.

***

To make sure you have access to reliable data even after problems like duplicate transactions have skewed your data, we recommend sending all of your Google Analytics data into a data warehouse like BigQuery. This makes sure you can filter out duplicated data and reprocess your reports to be 100% accurate. One solution to achieve this is to use a Google Analytics Parallel Tracker from Reflective Data.

The post How to Detect and Fix Duplicate Transactions in Google Analytics appeared first on Reflective Data.

Python is a programming language with virtually limitless functionalities and one of the best languages for working with data. Jupyter Notebooks, on the other hand, is the most popular tool for running and sharing both your Python code and data analysis.

Putting Python and Notebooks together with Google Analytics, the most popular and a really powerful tool for tracking websites, gives you almost like a superpower for doing your analysis.

This is exactly what this post is about. Pulling in and analysing your Google Analytics data using Python and Notebooks.

Oh, and by the way, this blog post itself is a Jupyter Notebook created in Google’s Colab, a version of Jupyter Notebooks that makes it super easy to collaboratively work on and share your Notebooks.

We wrote more on how we managed to turn a Notebook into a WordPress blog post in this article.

Without furder ado, let’s start by loading in some dependencies we are going to need for superfsadfsadfsdfsafasdfsadfpulling data from Google Analytics.

import pandas as pd # Pandas is an open source library providing high-performance, easy-to-use data structures and data analysis tools for Python
import json # JSON encoder and decoder for Python
import requests # Library for sending HTTP requests

There are many ways you can pull data from Google Analytics but one of the simplest is by using the API Query URI that you can grab from the Step 1 – Go to Query Explorer

If you are not familiar with the Query Explorer, this is an official tool from Google that lets

you play with the Core Reporting API by building queries to get data from your Google Analytics views (profiles). You can use these queries in any of the client libraries to build your own tools. In this case, the tool is a Notebook.

Step 2 – Authenticate

Choose the Google Account that has access to the Google Analytics View from which you want to get data.

Step 3 – Build the query

If you know how Google Analytics works, building a query is rather straightforward.

First, choose the right Account, Property and View you want to access.

Then choose the date range. You can use the calendar picker or write dynamic ranges like from 90daysAgo to yesterday.

Next, pick the metrics and dimensions. For example, you might pick metrics ga:sessions and ga:users and a dimension ga:date

.

Finally, you can apply sorting, filtering and segmenting rules as you wish. I’d recommend using them after you feel comfortable with the basics.

Perfect. Now test your query by pressing the “Run Query” button. If everything went as expected, you should see a table with results appear in a few seconds.

Step 4 – Get the API Query URI

Now, copy the API Query URI – we are going to use it for making requests straight from the Notebook.

Other options to pull data from Google Analytics using Python include:

• Using the Google Analytics Reporting API library for Python in your Notebook – based on Google API Client for Python
• Building a custom endpoint for handling your queries using the abovementioned API
• Exporting data from Google Analytics UI into CSV or XLSX and importing using pd.read_csv()

Making the request

Now that we have the API Query URI, we can make our first request using Python in the Notebook.

In this example, we are loading the number of Sessions and Users per day for the last 30 days.

from getpass import getpass # required only for hiding your variables

api_query_uri = getpass('Enter API Query URI here') # get the URI
r = requests.get(api_query_uri) # make the request
data= r.json() # read data from a JSON format
df = pd.DataFrame(data['rows']) # turn data into a Pandas data frame
df = df.rename(columns={0: 'Date', 1: 'Sessions', 2: 'Users'}) # giving the columns some proper titles
df['Sessions'] = df['Sessions'].astype(int) # formatting sessions as ints
df['Users'] = df['Users'].astype(int) # formatting users as ints
df['Date'] = pd.to_datetime(df['Date'])

df.head() # printing the first five rows

Enter API Query URI here··········

	Date	Sessions	Users
0	2019-09-25	64076	83335
1	2019-09-26	74569	103784
2	2019-09-27	59752	77642
3	2019-09-28	57743	76690
4	2019-09-29	63712	82980

Great, so our data is now loaded and formatted for our needs.

Let’s go ahead and try visualizing it. For timeseries data, a line chart should work well.

df.plot.line(x='Date', y=['Sessions', 'Users'], ylim=[0,None], figsize=[10, 6])

<matplotlib.axes._subplots.AxesSubplot at 0x7f3cb89692b0>

That was easy, right?

Now, let’s take a look at a bit more advanced visualization by drawing a population pyramid based on data we are now pulling from Google Analytics.

This is how the query looks like:

dimensions: ga:userAgeBracket, ga:userGender
metrics: ga:sessions

api_query_uri = getpass('Enter API Query URI here') # get the URI
r = requests.get(api_query_uri) # make the request
data= r.json() # read data from a JSON format
df = pd.DataFrame(data['rows']) # turn data into a Pandas data frame
df = df.rename(columns={0: 'Age', 1: 'Gender', 2: 'Sessions'}) # giving the columns some proper titles
df['Sessions'] = df['Sessions'].astype(int) # formatting sessions as ints
df.head()

Enter API Query URI here··········

	Age	Gender	Sessions
0	18-24	female	56244
1	18-24	male	9362
2	25-34	female	228866
3	25-34	male	33987
4	35-44	female	109795

import matplotlib.pyplot as plt # Matplotlib is a popular library for drawing plots in Python
import seaborn as sns # Seaborn is a Python data visualization library based on matplotlib

# Making copy of the dataframe and modifying session values to be negative for females (see chart below to see why)
df['Sessions'] = df.apply(lambda row: row['Sessions'] * -1 if row['Gender'] == 'female' else row['Sessions'], axis=1)

# Draw Plot
plt.figure(figsize=(8,5), dpi= 80)
group_col = 'Gender'
order_of_bars = df.Age.unique()[::-1]
colors = [plt.cm.Spectral(i/float(len(df[group_col].unique())-1)) for i in range(len(df[group_col].unique()))]

for c, group in zip(colors, df[group_col].unique()):
    sns.barplot(x='Sessions', y='Age', data=df.loc[df[group_col]==group, :], order=order_of_bars, color=c, label=group)

# Decorations    
plt.xlabel("$Sessions$")
plt.ylabel("Age")
plt.yticks(fontsize=12)
plt.title("Population pyramid of the website visitors", fontsize=16)
plt.legend()
plt.show()

Beautiful, now we know that majority of our audience is younger females.

More custom visualizations

With the wide variety of Python packages available for data visualization, the number of different charts you can use is virtually limitless.

In the next example, let’s try out joy plot. If you don’t know what a joy plot is then you are about to learn it now.

First, let’s install the required Python package.

!pip install joypy # install joypy
import joypy # JoyPy is a one-function Python package based on matplotlib + pandas with a single purpose: drawing joyplots.

Requirement already satisfied: joypy in /usr/local/lib/python3.6/dist-packages (0.2.1)

Now, let’s make the request to load required data. For this example we are using the number of transactions per age bracket and hour of the day for the past 90 days.

api_query_uri = getpass('Enter API Query URI here') # get the URI
r = requests.get(api_query_uri) # make the request
data= r.json() # read data from a JSON format
df = pd.DataFrame(data['rows']) # turn data into a Pandas data frame
df = df.rename(columns={0: 'Hour', 1: 'Age', 2: 'Transactions'}) # giving the columns some proper titles
df['Hour'] = df['Hour'].astype(int) # formatting hours as ints
df['Transactions'] = df['Transactions'].astype(int) # formatting transactions as ints
df.head()

Enter API Query URI here··········

	Hour	Age	Transactions
0	0	18-24	47
1	0	25-34	195
2	0	35-44	32
3	0	45-54	26
4	0	55-64	26

To make our dataframe more suitable for the joy plot, we need to pivot it.

pivot = pd.pivot_table(df, values='Transactions', index=['Hour'],
    columns=['Age'])

pivot = pivot.fillna(0)

pivot.head()

Age	18-24	25-34	35-44	45-54	55-64	65+
Hour
0	47	195	32	26	26	0
1	37	68	37	16	42	0
2	21	16	5	11	5	0
3	0	16	11	0	5	0
4	5	5	5	0	0	0

Now, let’s see how our joy plot looks like by running the following code.

x_range = list(range(24))
fig, axes = joypy.joyplot(pivot, kind="values", x_range=x_range, figsize=(10,8))
axes[-1].set_xticks(x_range);

Looks quite good already. but since most of our customers were in the age bracket 25-34 and we want to visualize the difference in their behavior rather than absolute numbers, let’s normalize the data.

df_norm = (pivot - pivot.min()) / (pivot.max() - pivot.min())
df_norm.head()

Age	18-24	25-34	35-44	45-54	55-64	65+
Hour
0	0.356061	0.313531	0.096429	0.130	0.206349	0.0
1	0.280303	0.103960	0.114286	0.080	0.333333	0.0
2	0.159091	0.018152	0.000000	0.055	0.039683	0.0
3	0.000000	0.018152	0.021429	0.000	0.039683	0.0
4	0.037879	0.000000	0.000000	0.000	0.000000	0.0

Now that the data is normalized, let’s see how a joy plot would look like.

fig, axes = joypy.joyplot(df_norm, kind="values", x_range=x_range, ylim='own', overlap=1.5, figsize=(10,8))
axes[-1].set_xticks(x_range);

Much better in my opinion.

So, what does this joy plot tell us?

• Most transactions are made between 8 AM and 10 PM, almost no one buys between 2 AM and 6 AM.
• Older (65+) people don’t buy after 11 PM while the youngest bracket (18-24) continues buying until 3 AM.

While I agree that this plot didn’t teach us something extremely interesting or new, it is a good example of what’s possible with a few lines of code and Jupyter Notebook (or any other similar tool).

Working with raw hit-level Google Analytics data

While connecting Jupyter Notebooks directly to your Google Analytics view is enough for many use cases, having access to raw hit-level Google Analytics data opens completely new opportunities for more advanced analysis.

To gain access to raw hit-level Google Analytics, one can use a tool like Marketing Data Pipeline to send data into BigQuery. Python has a library for connecting to BigQuery which makes it easy to explore and visualize based on raw data and metrics calculated based on your own criteria.

Besides other use cases, having access to raw hit-level data enables you to get a complete overview of the user journey, calculate metrics like LTV and use your own models for marketing channel attribution (Markov, ML-based, FPA etc.).

Learn more about the benefits of having a marketing data warehouse.

Conclusion

Combining powerful tools like Google Analytics, Python and Jupyter Notebooks gives you the flexibility and options you probably didn’t even dream about. The first time I got to play around with these three together made me feel like a little child, like I had discovered something really-really cool (and it is).

I hope this was enough to get you excited and that you will now go and try doing it on your own. And as always, your comments, questions and suggestions are welcome in the comments below.

The post Working with Google Analytics Data Using Python and Jupyter Notebooks appeared first on Reflective Data.

While it might look like a normal WordPress blog post (like all the previous posts on our blog), you are actually looking at a Jupyter Notebook.

If you are not familiar with Jupyter Notebooks then here’s what you need to know about them:

The Jupyter Notebook is an open-source web application that allows you to create and share documents that contain live code, equations, visualizations and narrative text. Uses include: data cleaning and transformation, numerical simulation, statistical modelling, data visualization, machine learning, and much more.

In this first proof-of-concept blog post/Notebook, we are showing you how Notebooks work and a few cool things you can accomplish with them.

Why are we doing this?

The reason we are testing with “blogging using a Jupyter Notebook” is that it allows us to share our code example and analysis done in Python more easily and in a more repeatable way.

At Reflective Data, we are using a lot of Python in all steps of data analysis. Jupyter Notebooks is a tool we use to run and share our code both internally and with our clients. Why not leverage the same workflow in our blogging efforts as well, we thought.

Our plan is to create a series of useful tutorial-type blog posts where we show you how to use Python (and Notebooks) in your everyday work as a digital analyst. We are going to pull data from Google Analytics, testing tools and databases, then run all sorts of analysis and visualizations using this data (and Notebooks).

How are we doing this?

Since sharing Notebooks is all about repeatability, we thought it will be wise to also share how we managed to turn Notebooks into a WordPress blog post.

While you can host and run Jupyter Notebooks both locally or on the cloud, we’ve found Colaboratory by Google to be the best solution when it comes to (as the name suggests) collaborative coding. That’s why we are mostly using Colab when working with Notebooks.

Turning a Notebook into HTML

The first step in the process of sharing your Notebook on WordPress would be turning your Notebook from ipynb format into HTML.

The hosted version of Jupyter Notebooks comes with a handy “Download as HTML” option built-in. Colab, unfortunately, doesn’t have this feature. So, we found a workaround which also allowed us to semi-automate the whole process.

We are using Google Drive’s API to load the ipynb file from Drive (Colab hosts Notebooks on your Drive) and then using nbpreview to turn it to HTML. We haven’t yet done so, but the plan is to build a WordPress plugin that would take a Colab Notebook’s ID and then automatically turn it into HTML.

Teaching WordPress how to render a Notebook

Currently, we simply paste the HTML version of the Notebook into WordPress blog post using a text-edit-mode. This will soon be handled by the plugin we mentioned in the previous step.

Everey blog post that contains a Notebook will receive a tag “notebook”. Using this tag we can then tell WordPress to load some extra CSS to properly render the Notebook.

We are currently doing this in a few lines of code in our functions.php file.

add_action('wp_head', 'load_notebook_styles');
function load_notebook_styles() {
    if ( has_tag( 'notebook' ) ) {
        wp_enqueue_style( 'notebook-css', get_template_directory_uri() . '/notebooks/notebook.css' );
        wp_enqueue_style( 'notebook-prism-css', get_template_directory_uri() . '/notebooks/prism.css' );
    }
}

Here are the CSS files we load:

• notebooks.css
• prism.css

What’s possible with Notebooks?

So, now that you have at least a basic knowledge of what a Notebook is and how we managed to turn them into WordPress blog posts, let’s see what you could actually accomplish with them.

Them main thing, of course, is that you can run Python code and immediately see the output.

Here’s a very basic example:

x = 5
for i in range(x):
  print(i * '*')

*
**
***
****

Awesome, right?

Now, the only limits you really have are your creativity and Python skills. In the upcoming blog posts, our goal is to get you better at both by showing some cool and useful things we are doing using Python and Jupyter Notebooks.

While the first Python script was quite useless, let’s try and load in some real data from Google Analytics.

First, let’s import our required dependencies.

import pandas as pd
import json
import requests

There are many ways you can pull data from Google Analytics but one of the simplest is by using the API Query URI that you can grab from the Query Explorer.

In this example, we are loading the number of Sessions and Users per day for the last 30 days.

from getpass import getpass # required only for hiding your variables

api_query_uri = getpass('Enter API Query URI here') # get the URI
r = requests.get(api_query_uri) # make the request
data= r.json() # read data from a JSON format
df = pd.DataFrame(data['rows']) # turn data into a Pandas data frame
df = df.rename(columns={0: 'Date', 1: 'Sessions', 2: 'Users'}) # giving the columns some proper titles
df['Sessions'] = df['Sessions'].astype(int) # formatting sessions as ints
df['Users'] = df['Users'].astype(int) # formatting users as ints
df['Date'] = pd.to_datetime(df['Date'])

df.head() # printing the first five rows

Enter API Query URI here··········

	Date	Sessions	Users
0	2019-09-12	50526	58667
1	2019-09-13	53919	63156
2	2019-09-14	58735	68200
3	2019-09-15	67371	77597
4	2019-09-16	65780	77305

Great, so our data is now loaded and formatted for our needs.

Let’s go ahead and try visualizing it. For timeseries data, a line chart should work well.

df.plot.line(x='Date', y=['Sessions', 'Users'], ylim=[0,None], figsize=[10, 6])

<matplotlib.axes._subplots.AxesSubplot at 0x7fc6a53c3278>

That was simple, right?

There is so much more you can do with Python and Jupyter Notebooks and we are going to cover more advanced stuff in our upcoming blog posts.

I’d like to end this post with a few thoughts I got while writing this one.

1. Writing a blog post using Markdown is fun (and effective). I know some people are doing it but this is the first time for me. Markdown is the way you write text in Notebooks.
2. While I thought so before, now I know for sure that Notebooks + WordPress is a really good solution for sharing Python code examples and data visualizations online.
3. I’d love to hear your thoughts, ideas, questions and topic suggestions in the comments below! We are happy to respond/help.

PS! More Notebooks-related content is coming soon! Consider subscribing to our newsletter.

The post Jupyter Notebooks in WordPress appeared first on Reflective Data.

Custom Events provide a perfect solution for adding context and tracking more specific user actions. In this article, we are giving you a good amount of ideas for custom events you should implement on your own and/or your clients’ websites.

This post is divided into sections by different event categories so you could skip the ones that are not relevant for you.

Clicks

For almost all websites, the most common user interaction is a click. By default, Google Analytics isn’t very good at tracking them. Let’s see how we could improve this situation.

Navigation clicks

Tracking usage of the main menu, sidebar and other navigation elements is critical. This is how your visitors explore your business and move in the funnel.

Detecting navigation interactions is easy with Google Tag Manager and sending data to Google Analytics works as with any other event.

Since there may be several navigation elements containing links to the same page, it is important to also pass the information about which navigation was used. For example, you might a have link to your blog both in the header and in the footer.

Recommended event parameters:

category: click
action: top-navigation, sidebar etc.
label: link text
non-interaction: false

CTA clicks

Tracking clicks on all of your CTAs (call to actions) is extremely important if you have more than one CTA on a single page pointing to the same location (i.e. long-form sales pages).

Don’t forget that buttons like “Get a Quote”, “Add to Cart” and “Checkout” are also CTAs.

Setup is the same as with generic click tracking using GTM.

Recommended event parameters:

category: click
action: CTA
label: text (and index the same CTA appears multiple times)
non-interaction: false

Outbound link clicks

Also known as tracking website exits, this custom event tells you exactly to which external pages people navigate from your site.

Again, setting this up is quite easy with GTM.

Recommended event parameters:

category: click
action: outbound link
label: link URL - link text
non-interaction: false

File download clicks

Doesn’t matter if you provide whitepapers, ebooks, PDFs, or other types of downloadable files, you should be tracking how often they get downloaded.

There are some really good tutorials online but we recommend setting it up using Google Tag Manager. A simple “just links” click trigger with a regex that matches desired filetypes should work just fine.

Regex Example:

\.(pdf|docx)$

Recommended event parameters:

category: click
action: download
label: filename / link text
non-interaction: false

Rage clicks

Visitors rage clicking on certain elements on your website is a good indicator of a UX error. For example, people may click on a blue text that is not a link or on an image that has no click functionality.

We have a separate blog post about tracking rage clicks using Google Tag Manager and Google Analytics.

Recommended event parameters:

category: click
action: rage click
label: element selector
non-interaction: false

Button clicks

Your website is likely to have all sorts of clickable buttons. There might be some other ways for you to get an overview of the usage of the features related to those buttons but knowing how many clicks each button received definitely helps.

Some ideas for which buttons should be tracked:

• Add to wishlist/favorites
• Add/remove item from cart
• Update cart
• Continue shopping
• Log in/out
• Register
• etc.

Recommended event parameters:

category: click
action: button
label: button text/description
non-interaction: false

Form Interactions

When it comes to tracking forms, I would like to recommend a few good articles by Simo Ahava that cover everything you need for getting started.

• Track Form Engagement With Google Tag Manager
• Track Form Abandonment With Google Tag Manager
• Form Field Timing With Google Tag Manager

Since forms are the most important part of many websites (i.e checkout, lead form etc.) you should really nail tracking them correctly.

A/B Testing

Experiment and Variation

Most A/B testing tools provide some sort of integration with Google Analytics. Most of them, though, are using custom dimensions. A problem with custom dimension based integration is that you need to make sure that the dimensions aren’t overlapping between experiments because otherwise, you might end up overwriting the dimension value for some experiments.

What we recommend having, besides the default integration, is a secondary event-based integration. That is, sending a custom event for every running experiment in a following format:

category: ab testing
action: experiment ID
label: variation number
non-interaction: true

PS! One event per experiment-variation combination.

This gives you raw information about each experiment and variation that the visitor saw. You can easily use this information in your custom segments or reports.

Every testing has a bit different solution for exposing experiment data to the front-end but here are a few links that should be helpful.

• VWO – instead of custom dimension, send this data using custom event action and label.
• Google Optimize – this allows you to get experiment data to the datalayer, now send it using a custom event.
• Optimizely X – access data about live experiments using Javascript.

Google Optimize Anti-Flicker Snippet timeout

If you have any experience with JavaScript-based A/B testing and/or personalization tools you know that flicker can be a real headache.

In case your tool of choice is Google Optimize, you should be using their official anti-flicker snippet to minimize the flicker effect.

We wrote a full separate article on Tracking Optimize Anti-Flicker Snippet Timeout in Google Analytics.

Recommended event parameters:

category: optimize
action: snippet timeout
label: true/false
non-interaction: true

Scroll depth

Whether you are working with a blog, some landing pages or any other page that is longer than a fold, you want to know how far down do the visitors scroll.

It used to required some custom Javascript but now, tracking scroll depth is a built-in feature of Google Tag Manager.

Recommended event parameters:

category: scroll
action: scroll depth threshold
non-interaction: false

Social interactions

While Google Analytics has a dedicated feature for tracking social interactions, having this data in custom events gives you more flexibility in reporting.

There is a really thorough blog post about tracking social interactions using Google Tag Manager – although, if you want something simpler, tracking clicks on your social icons is a good starting point.

Recommended event parameters:

category: social
action: share/like/follow
label: social platform
non-interaction: false

Video events

Doesn’t matter if it’s a sales video, a video that explains how your product works or a tutorial – you do care about how many people start the video and how far through will they make.

Google Tag Manager provides a really easy solution for tracking YouTube videos but tracking others such as Vimeo is completely doable as well.

Recommended event parameters:

category: videos
action: start/pause/percentage
label: video id
non-interaction: false

Errors

No matter how good your site is, there will always be errors. Since errors are directly related to bad user experience, tracking them is absolutely crucial.

The first step here should be sending your Javascript errors to Google Analytics. Next, consider sending other errors like popup alerts and system errors as well.

Recommended event parameters:

category: error
action: error message
label: feature or other relevant information
non-interaction: true

reCAPTCHA bot score

Google Analytics does have a simple bot-filtering system in place but it comes nowhere near catching all the bots that visit your site. Even tools like reCAPTCHA by Google can’t usually say for sure whether a visitor is a bot or a human but they can give you a score (from 0.1 to 1). Simo Ahava has put together a great tutorial for tracking the reCAPTCHA both score using Google Analytics custom dimensions.

Recommended event parameters:

category: reCAPTCHA
action: reCAPTCHA answer
label: reCAPTCHA score
non-interaction: true

Don’t forget to test whatever new custom event you add to your setup.

Should you have any questions related to custom events in Google Analytics, shoot them in the comments below.

PS! If you need help with implementing custom events, you can hire us to help you out!

Photo by Annie Spratt on Unsplash

The post Data You Should Be Tracking Using Google Analytics Custom Events appeared first on Reflective Data.