GA360 is expensive and we felt like, for our use case, it would be too much. I was delighted to learn that Reflective Data can help us collect over 1M events per day in our BigQuery dataset for a fraction of the cost.

Jurgita, Digital Data Analyst, Vilnius

GA360 is a great tool, and for the enterprises that can afford and justify the cost, probably the best analytics tool they can invest in. At the same time, many companies don’t have the budget to pay upwards of $150k for an analytics tool.

The challenge

One of the best features of GA4 over the previous version is that even the free version has the BigQuery export pipeline available.

The free version of GA4, though, has a limit on how many events can be exported each day. More specifically, this limit is set at 1M events per day. Barbora, like many companies with large traffic numbers, was reaching this limit in GA4 but at the same time wasn’t able to justify the cost of GA360.

The solution

At Reflective Data, we built a solution called Parallel Tracking for GA4.

Here’s how it works.

Parallel Tracking duplicates all HTTP requests going from your site or app to GA4 and sends them to Reflective Data’s processing endpoint.

Data is then processed just like GA4 does it and inserted into your BigQuery dataset. There is no limit on the daily event count with Parallel Tracking.

Getting started with Parallel Tracking is easy as 1, 2, 3.

Step 1: Tracking code modification

After signing up for Parallel Tracking, you’ll have to enable it by modifying your GA4 implementation. We have plug-and-play solutions for Google Tag Manager, gtag.js, Measurement Protocol and most other implementation methods that GA4 has.

Step 2: Pipeline activation

In order to activate the pipeline, all you need to do is provide Reflective Data with a Google Cloud Platform Service Account key file. After providing the key file, GA4 data will start flowing into your BigQuery dataset.

Step 3: Querying the data

Once the pipeline is active, you can start querying your GA4 data just like with the default GA4 to BigQuery data export.

Conclusion

If your daily event volume in GA4 is close to or exceeds 1 million then you have two options. Upgrade to GA360 or implement Parallel Tracking. If the event volume exceeding 1 million is the only reason you’re looking to upgrade, Parallel Tracking is likely a better option as it is several times more affordable compared to GA360.

Ending with another quote from the customer here.

I guess you could say we were your average enterprise with LOTS of legacy data infrastructure that had been built over many years. This system was extremely complex and very expensive to maintain. When Reflective Data came in, they acted as true professionals, worked very closely with our IT and came up with the plan that pleased everyone.

Today, being on the new cloud-based data infrastructure for almost a year now, I can say with all certainty that this project was a success. Not only are we saving tens of thousands of dollars every month on the infrastructure alone, the amount of hours it takes to maintain the system has gone from hundreds down to a ten or so. This has huge impact on our business. Implementing anything new would’ve taken at least 6 months with the old system, now it’s a matter of week or two to get everything up and running.

Last but not the least, Reflective Data saved us another $150k year as we no longer needed to use GA360 and instead implemented their Parallel Tracking solution for GA4.

Daria, VP of Marketing, Warsaw

It’s feedback like this that makes us love the work we do even more! Get in touch and learn how we can help you, too.

For more case studies, see here.

The post Case Study: Overcoming The 1 Million Event Limit in GA4 Without Upgrading to GA360 appeared first on Reflective Data.

This came as an unwelcome surprise for anyone working in the experimentation industry. Even if you didn’t use the tool itself, it was the first tool most newcomers used to get themselves into experimenting on their websites.

For a while, Google Optimize going away was everything people talked about on Twitter and LinkedIn.

Exporting Experiment Data From Google Optimize

Since several of our existing clients asked for it, we built Google Optimize Data Exporter to store your experiment data for as long as you need it. It supports almost any data destination, including popular ones like Google BigQuery, Amazon S3 and Snowflake.

Google Optimize Data Exporter runs on the same robust and scalable Reflective Data Infrastructure that you hopefully already know and love.

We’ve made the process of exporting your Google Optimize data as simple as possible. Here’s a quick overview.

1. Planning and scoping

Get in touch with one of our data analysts to plan your Google Optimize Data export. The main questions to answer are the list of experiments, dimensions, metrics, time frames and the data destination you wish to use for your Optimize data export.

2. Data export and storage

Executing the plan. Our data analyst will configure Reflective Data Export System to pull the requested data from your Google Optimize instance and store at your chosen data storage destination.

Most exports use Google BigQuery as a data destination.

3. Reporting and consultation

The post Export Experiment Data From Google Optimize – While You Still Can appeared first on Reflective Data.

Possible use cases are virtually limitless and depend on what kind of business you run. In this article, I’m providing some of the more common ways together with examples of how to benefit from having a data warehouse.

In no specific order, here we go.

Better attribution leads to more optimized marketing and ad spend

It’s no secret that some analytics tools and most ad platforms tend to default to attribution models that work best for them. I mean, they use models that show as if they’re bringing you the most traffic and conversions. Some tools let you modify the model or attribution windows to some extent but generally, it’s quite limited.

Let’s say someone scrolled past your ad on Facebook without clicking on it. Later that day they receive an email from you with a nice offer – they end up visiting your suite and making a purchase. Guess what Facebook will tell where this purchase came from? You guessed it, they will attribute it to seeing the ad on Facebook.

Now imagine having all this attribution data available in your data warehouse in a raw format. This will give you full flexibility over which attribution model and window to use. No longer “apples to oranges” comparisons when it comes to your traffic sources.

Combine this with data from other sources like your CRM or backend and you’ll have the ultimate attribution machine in your hands. Use attribution models like Markov Chain or even AI/ML-based models and use metrics like LTV or churn for long-term analysis.

Proper attribution with an “apples to apples” comparison is the key to optimizing your marketing and ad spend.

Perhaps the best way to start collecting this kind of data in your data warehouse would be Parallel Tracking.

More accurate audiences lead to better targeting and savings in ad spend

What if I told you there was a way to evaluate visitors or groups of visitors based on their likelihood of becoming a customer or even a high-value long-term loyal customer? All before they even land on your website or make their first purchase. Imagine building your advertising audiences based on those predictions while constantly improving the model. Well, the big players have been doing this for years but today the barriers have lowered and most businesses could leverage this kind of technology. Trust me, it’s easier than you might think.

There are three main components to getting started with AI/ML-based audience future value predictions.

1. Set up a data warehouse – build yourself or hire Reflective Data to consult and help with the technical setup
2. Configure data pipelines (the systems that transfer data from various sources into your data warehouse) – We’ve got you covered if you need help
3. Leverage a system similar to BigQuery ML for the predictive models – Yes, we can help with the ML part, too

Instead of spending your precious marketing budget on people that’ll never buy from you, focus on those that have the highest potential of becoming high LTV customers.

$$$

Data-driven product recommendations lead to higher AOV

Product recommendations, if done right, can provide enormous value for both the customers and merchants. Now that we have tools like Recommendations AI from Google, getting started with your own recommendations engine has never been easier. Or cheaper.

All you need is data, preferably in a data warehouse like BigQuery. For this use case, I’d recommend using Parallel Tracking to get your data into your data warehouse.

Case study: How IKEA managed to increase their Click Through Rates by 30% and AOV by 2% by leveraging the Recommendations AI. LINK

Data-driven marketing automation drives more sales

Still sending generic emails to all customers or manually building audiences? Welcome to the modern days where we can leverage the rich customer data available in our data warehouses. Use it to automate many critical parts of our marketing. Including emails.

A few scenarios to consider.

• Predictive model described above detects people with the highest future customer lifetime value (LTV). Nudge those people a bit by sending them an offer that’s a bit more generous than what you’d send to everyone. Believe me, it’s worth it.
• Include personal product recommendations in your marketing emails.

To get high-quality data into your data warehouse, you need a data pipeline that connects with your existing tools.

Data-driven personalization drives more sales and higher AOV

There are many ways how you can introduce personalization to your customers but one aspect is true for all of them – they all need good quality data as input.

Below are a few common personalization solutions our clients have implemented.

• Customize the front page of your e-commerce (or any other) website based on each customer’s previous browsing and shopping behavior. This can include banners, offers, discounts, product recommendations and more.
• Customize the website experience based on the type of customer that has landed on your site. Depends on your business type and target audience but generally individuals, people representing small businesses and those shopping for a large enterprise all expect a somewhat different experience.
• Personalize overall messaging based on visitors’ behavior. Let’s say you’re an insurance company and someone landed on your blog article comparing different car insurance options, then they move on and read a few more related articles. Now, when they land on your main website, it’d be good to welcome them with relevant car insurance options as opposed to generic content.

To get high-quality data into your data warehouse, you need a data pipeline that connects with your existing tools.

Analyzing user behavior can lead to savings in customer support costs

Maintaining a proper support team can be costly. Many companies that we’ve worked with could’ve (and eventually have) saved anywhere from 5% to 60% in support-related costs by having relevant data available in their data warehouse, analyzing it and, of course, making the changes based on insights found.

Data points to collect here include.

• Live chat messages
• Call transcripts
• Site search behavior
• Help center behavior
• Social media, Reddit and other platforms

To collect this kind of data into your data warehouse, you need a data pipeline that connects with the tools necessary.

Potential ways to execute those insights include.

• A/B testing
• Chat bot
• Re-arranging help center
• Train support personnel
• Improve your product UI/UX
• Better onboarding guides

Running an experimentation program on top of a data warehouse will save $ on tool vendor costs

Most conventional experimentation (A/B testing) tools like Optimizely, Google Optimize and others operate in silos and keep their own copy of all the data they need to operate. In reality, though, this dataset is very similar to many other tools that you may already use, including Google Analytics. Having another tool to operate in a silo isn’t (cost) effective and can often lead to data discrepancies.

Modern experimentation tools like Eppo, Growthbook and others don’t create another siloed copy of your data. Instead, they will sit on top of your data warehouse and use data you already have – nicely integrated with various sources. Not only is this much more cost-effective but it also helps you avoid creating yet another source for data discrepancies. Not to mention that you don’t have to define all the goals beforehand as you can simply use SQL to define those after the fact. Or while the test is already running.

A simple data pipeline and an open-source tool like Growthbook is all you need to get started. Save tons compared to tools like Optimizely and get a more robust solution that enables you to test across all parts of your tech stack. Front end, back end, APIs, databases and more.

Using a data warehouse in favor of a conventional analytics tool will save $ on tool vendor costs

There are free analytics tools out there but they all come with quite serious limitations which leads most bigger companies to look for a paid solution. What most tools really do is provide you with a tracker, data processing endpoint and a few shiny reports – oh, and ask a fortune for doing so.

In reality, all you really need is a tool like Parallel Tracking that connects with your existing Google Analytics (both UA and GA4) trackers while duplicating hits to another processing endpoint and then sending them straight into your data warehouse. Free from data collection, cardinality or other limitations which are often seen even in the paid analytics platforms. From there, connect your BI tool and build all the reports you might ever need. All while saving like 10x compared to something like GA360.

As a bonus, you’ll likely have data from other sources in your data warehouse, too. This unlocks the potential to do even more advanced analysis that’s not possible with any one analytics tool alone.

Conclusion

It’s not a question of whether you need a data warehouse or even whether the ROI is there. It’s more about the specific tools and tech you’re going to implement from the get-go – switching a vendor later is a lot of effort and costly, too.

Our recommendation is to avoid CDPs and other fancy buzzword tools and instead get something more robust – BigQuery as your data warehouse, a set of data pipelines, a BI tool for reporting and then go from there.

Data engineers at Reflective Data would be more than happy to answer your questions in the comments below or hop on a quick free consultation session.

The post Increase Your Company’s Profits by Getting and Leveraging a Data Warehouse appeared first on Reflective Data.

The amount of load that Reflective Data was able to lift from the shoulders of my team was unbeliavable. Without their help, we’d still be building out the pipelines and never gotten to the level of advanced analysis and ML that we’re able to do now. They let us focus on what brings the most value to our business.

Melanie, Director of Data Analytics, Frankfurt

At Reflective Data, we’ve worked with companies big and small. This means we have seen all levels of maturity when it comes to the infrastructure and knowledge around data pipelines and data warehouses.

Some of the most challenging projects have been enterprises with quite some infrastructure, legacy pipelines, and of course, opinions. Smaller businesses are just starting to adopt the concept of having all of their data stored in a data warehouse but many enterprises have been doing this for a decade!

The challenge

When many of the enterprises that we’ve worked with started building their data pipelines, they didn’t have tools like Airflow, BigQuery etc. that we use and love today. This means the bulk of it was built in-house. Even the concept of cloud computing was in its early days and most operations were kept on-premise.

The challenge with this kind of setup starts by understanding the existing setup. In some cases, the documentation is close to none and the people that built it are no longer with the company. This alone can take a month or so – mapping everything out, understanding the structure, creating the plan for moving forward.

Another challenge is getting everyone on the team on board. More often than not there are people who value the work that has been put into the old system over the years so much that it blinds them from seeing the obvious benefits of moving to a much more modern infrastructure.

The solution

When working with the enterprise and legacy infrastructure, nothing happens overnight. Below are the phases of a typical project of getting an enterprise client onto a modern cloud-based data infrastructure.

Phase 1: understanding and mapping the existing situation

With most enterprises, it’s not just one team or system that depends on the data infrastructure. More often than not, this is the backbone of the entire business. This means we need to make sure we understand every aspect of the current system, where it gets the data, how it’s being processed and what processes depend on this data.

Phase 2: planning the infrastructure

We do our best to work closely with all teams involved to make sure their needs are taken into account. This means a series of on-hands meetings where we learn about their use cases and problems they’re having with the existing setup. The output of this phase is a clear plan for moving forward, including the tool stack, reporting mechanisms and several feedback rounds to make sure everyones’ needs are taken into account.

Phase 3: implementation

Depending on the in-house knowledge, resources and other aspects, a company can decide to implement the plan themselves and continue using Reflective Data as a consultant or hire us to handle the technical execution as well. By far, the most effective arrangement in our experience has been where we do the bulk of the work while including a few technical people from the client’s side in every step of the process. In some cases, those people is hired specifically for this purpose.

Phase 4: monitoring, reporting and integrations

The whole point of having high-quality data is to make it actionable. Of course, we handle core integrations within the implementation phase but in a sense, data infrastructure is a growing organism that needs constant attention. Reflective Data is here to build long-term relationships with its clients, ready to help whenever there’s a new data source to be added, a report to be built or if a new team member needs training.

Conclusion

Moving away from a legacy data infrastructure is one of the best actions an enterprise can take towards being more data-driven, more effective in managing the infrastructure and, in all reality, keeping up with the competition.

Ending with another quote from the customer here.

I guess you could say we were your average enterprise with LOTS of legacy data infrastructure that had been built over many years. This system was extremely complex and very expensive to maintain. When Reflective Data came in, they acted as true professionals, worked very closely with our IT and came up with the plan that pleased everyone.

Today, being on the new cloud-based data infrastructure for almost a year now, I can say with all certainty that this project was a success. Not only are we saving tens of thousands of dollars every month on the infrastructure alone, the amount of hours it takes to maintain the system has gone from hundreds down to a ten or so. This has huge impact on our business. Implementing anything new would’ve taken at least 6 months with the old system, now it’s a matter of week or two to get everything up and running.

Julien, VP of Marketing, Austin

It’s feedback like this that makes us love the work we do even more! Get in touch and learn how we can help you, too.

For more case studies, see here.

The post Case Study: Building and maintaining a Data Pipeline and Data Warehouse for the Enterprise appeared first on Reflective Data.

In this article, we are going to explore some of the ways we can leverage Google Analytics to track churn, LTV and other really useful metrics.

Depending on the software you’re using, there may be some off-the-shelf solutions that you can install. For example, if you’re on Shopify then you can use something like Littledata to send a more accurate LTV value into a custom dimension in Google Analytics. More often than not, though, there is no good solution available or you just need more control over the setup.

One common misconception is that such long-term retention metrics are relevant for a few specific business types only. Yes, metrics like churn are vital for SaaS and subscription products but any company that gets return business should have their long-term KPIs in place. And I don’t mean simply tracking them but actually analyzing them and optimizing the business with those metrics in mind.

Acquiring a new customer is anywhere from five to 25 times more expensive than retaining an existing one. It makes sense: you don’t have to spend time and resources going out and finding a new client — you just have to keep the one you have happy.

Harvard Business Review

So, if you’ve been focusing on getting new customers and metrics like revenue or transactions, this article is just for you!

How to measure retention metrics like LTV and churn

The long-term retention metrics most relevant to you depend on the type of business you’re working with but the most common ones are customer lifetime value (LTV) and churn. Below is a list of other popular retention KPIs. Think about the ones that would be relevant for your business.

Common Customer Retention Metrics

1. Customer Churn
2. Revenue Churn
3. Existing Customer Growth Rate
4. Repeat Purchase Ratio
5. Product Return Rate
6. Days Sales Outstanding
7. Net Promoter Score
8. Time Between Purchases
9. Loyal Customer Rate
10. Customer Lifetime Value

Source

Almost all retention metrics require a proper User ID implementation. This means you’d have to identify the user over time and even if they’re using multiple devices or browsers. Luckily, in most cases, actions like completing a purchase or signing up for a subscription do involve some kind of authentication.

While it’s possible to track retention metrics with Google Analytics alone, in most cases you’d get much better (more accurate) results when combining it with some other technology. Let’s explore two of the more popular options.

Sending retention data into Google Analytics

This solution involves sending retention data into a custom dimension or a custom metric in Google Analytics.

The exact workflow depends on the software (CRM, CMS, database etc.) your site uses but the general process would look something like this.

1. Create a custom dimension in Google Analytics (should be user-scoped)
   
2. For logged-in/identified users, pull/calculate the values for the relevant retention metrics from a database or other system (CRM, CMS etc.)
   Something like this if your order data is stored in BigQuery.
   
3. Make the retention metrics available in the data layer
   
4. Use Google Tag Manager to send your retention metrics to Google Analytics, using the custom dimension or metrics slots/indices according to how you configured them in step #1

Now, having this data available in Google Analytics, you do whatever you want with it. Here are a few examples.

Using LTV in a Google Analytics custom report

LTV in the Google Analytics user explorer report

Notice the difference between LTV that Google Analytics is reporting by default ($439) and the value we see in our custom dimension ($2,016). This is because Google Analytics can’t keep a track of the user as accurately as your backend system or an e-commerce platform you’re using. The same goes with other retention metrics, getting accurate metrics requires some custom work.

The list of possible use cases for this kind of data is unlimited. For example, think about creating custom segments in Google Analytics for customers that are in the top 10% in terms of LTV and see what differentiates them from the rest of the visitors. Besides making more/larger purchases, of course. Things like their traffic source, what pages they landed on, what A/B test variants they saw etc. can be quite insightful.

Talking about ways you can analyze your data and the insights it will give you. Let’s take measuring retention metrics to a whole new level by sending data into a data warehouse.

Storing data in a data warehouse

If you’re just getting started with retention metrics and you still mostly optimize for generic metrics like leads, total transactions and total revenue, then you’ll still be better off with having them in Google Analytics. Compared to not having them at all, that is. If you’re serious about analyzing and optimizing for retention and customer lifetime value then you need a data warehouse.

Here’s a quick step-by-step guide that will lead you in the right direction.

1. Send all Google Analytics data into a data warehouse (i.e. BigQuery). Tools using the Reporting API (most of them) can get you started but for true unsampled hit-level data you need something like Parallel Tracking.
2. Send, pull, push data from other relevant sources into your data warehouse. This should include your database, CRM, CMS, marketing tools, ads platforms, customer support, live chat and every other tool that has data about your customers and their interactions with your brand. Self-service tools like Stitch will get you started but we’d recommend more flexible managed solutions.
3. Solution for accessing data stored in your data warehouse. You’d need something (could be separate tools) that can handle ad-hoc queries, dashboarding, automated reports, and building data models. Tools like Google Data Studio will get you started. Something like Looker or Tableau would be better. Our recommendation is to go with a managed service that will put together the best set of tools for you and configure the rest as well.

If having retention metrics in Google Analytics enabled you to do all sorts of useful new reports and analysis then your options with the setup above are truly limitless.

Having a proper data warehouse will be your competitive advantage. Not only does it give you an option to get a very good overview of the current status of your business and your customers, but it will also allow for truly optimizing the user experience and user journey. Leading to a better user experience and improved retention metrics. Remember, acquiring a new customer is anywhere from five to 25 times more expensive than retaining an existing one!

One way to describe the usefulness of a data warehouse is by giving you a few sample questions. Questions that would be very difficult to answer without a data warehouse.

• Purchases from which traffic channels are most likely to be refunded at some point in the future? Might lead to revising your marketing budget.
• Which traffic sources have the highest retention/LTV?
• What is the correlation between subscription value ($) and churn rate?
• What is the long-term impact of your campaigns or A/B experiments? Do quick wins lead to higher churn or lower LTV?
• Does data from different sources add up? Maybe Google Analytics is missing some transactions that are in Shopify or perhaps some of them are duplicates?

Here’s an example of the last one on the list above.

As you can see, Google Analytics is missing a good amount of transactions and this requires further investigation. Definitely something you should include in your Google Analytics dashbaord.

This was just a short list of ideas to get you thinking about what is possible with a proper data warehouse. You can trust me when I say that I’ve seen companies become truly data-driven after they’ve implemented a tailor-made data warehouse and started digging for insights they couldn’t before.

Working with automatically recurring events

It is important to keep in mind that some retention metrics can change without the user themselves taking any action. You need to make sure that those cases are being tracked and taken into account. Here are a few examples.

• Recurring orders/payments
• Subscription expirations
• Payment method expirations
• Orders being changed/cancelled (i.e. due to missing item)

If your data warehouse was configured properly, you should have this data already available. Just make sure to include it in your analysis and reports.

In case you don’t have a data warehouse and you’re trying to solve this with Google Analytics alone then you need to use the Measurement Protocol. Some of the more common subscription platforms like ReCharge for Shopify have this built-in or solvable with some 3-rd party solutions but quite often custom development is required. In which case, you should think about implementing a data warehouse instead.

Conclusion

If you’re in a business where customers are expected to generate value more than once (repeat purchase, subscription etc.) then you need to start focusing on your retention metrics.

Google Analytics can get you started with the basic metrics and limited accuracy. A much better setup would be Google Analytics combined with Parallel Tracking and if you’re serious about optimizing for those metrics then you need a custom-built data warehouse where all marketing data is pulled together.

Feel free to post your ideas and questions in the comments below. If you’d like to get some consultation and discuss your ideas further, get in touch with us.

The post Measure Long-Term Metrics Like Customer Lifetime Value (LTV) Using Google Analytics appeared first on Reflective Data.

With the help from Reflective Data and using their custom data pipeline for Google Analytics, we were able to avoid sending data to Google Analytics, store everything within the EU and still benefit from the rich and actionable dataset you get with Google Analytics.

Aaron, Chief Data Officer

Everything mentioned in this article applies to both Google Analytics Universal Analytics and GA4, including the built solutions.

Data protection and privacy rules are getting tougher all over the world. This is especially true for the European Union and even more so for some specific industries. Including finance, medical and others that handle sensitive information about their users.

While Google Analytics has been making some improvements in the privacy area and is GDPR compliant, this is not enough for many businesses and industries.

At Reflective Data, we’re often working with companies that are under close monitoring of their regulators. To help them out, we’ve built custom solutions that allow storing Google Analytics data within the European Union or sometimes even completely locally, on the servers of our client.

The challenge

Do we want it or not, the regulations for collecting, processing and storing data are getting stiffer each year. European Union, with rules like GDPR, is a trendsetter in data protection.

While it’s good from a customers’ perspective, following all those rules and regulations can be quite a challenge for businesses that have to follow them. Especially, as new rules are introduced and the penalties for not following them can reach millions.

More often than not, the regulations are being launched faster than companies can adopt. Even if the company themselves are doing everything right, it’s likely that some third-party software they’re using is lacking behind.

At Reflective Data, we have been contacted by numerous companies that, for a reason or another, aren’t able to use Google Analytics but would still require a similar dataset and comparable functionality. Let’s be honest, Google Analytics and its Universal Analytics data model is still one of the best ones available for marketing attribution and website user behavior analysis.

Yes, Google Analytics 4 will be the future of tracking but it’s nowhere near being ready for production for most businesses due to lacking several critical features.

The solution

One of our most popular tools is the Parallel Tracking for Google Analytics. It’s a system that allows sending raw hit-level Google Analytics data into a data warehouse. Most commonly, our clients choose this solution for the following reasons.

• They hit the 10M/hits/mo limit
• Problems with data sampling
• Looking for alternatives for Google Analytics 360
• Need access to raw hit-level data

These are all nice features that our clients love but there’s one more – Parallel Tracking can work without sending data to Google Analytics at all. This is thanks to how our technology was built. Instead of pulling data from Google Analytics using the Reporting API (as many solutions do), we are duplicating all hits that are sent to Google’s data processing endpoint and sending them to our engine as well. This means, with a simple tracking code modification, we can disable hits being sent to Google Analytics.

Our data processing engine works very similarly to what’s happening in Google Analytics. Hits are processed and enriched using the same methodology and sessions are processed just like in Google Analytics.

Ultimately, this means you’ll get the best from both worlds. The rich and actionable dataset of Google Analytics and full ownership of your data. So, whatever regulations apply to how you can store this kind of data, now you have a solution to comply with them.

Everything mentioned in this article applies to both Google Analytics Universal Analytics and GA4, including the built solutions.

Conclusion

There are many reasons why you’d want to use Google Analytics as your analytics platform. Including the rich data model that works for both marketing attribution and user behavior analysis, (almost) unlimited amount of resources, great community, ease of finding people that can work with this model and more.

If, for some reason, you can’t use Google Analytics. Could be because they’re storing data outside of the EU or that you need to store everything locally. Then we have a solution for you!

For details and contact information, check out the product page for Parallel Tracking.

Ending with another quote from a customer here.

Before joining the financial institution that I’m currently working for, I had been a marketing director for various e-commerce companies. This meant that I was really used to tools like Google Analytics. Now, working in finance, I learned that using Google Analytics was not an option due to several regulations and we had to store all data locally on our own servers.

We had some kind of an in-house solution but that was nowhere near the capabilites and functionality I was used to. I was really lucky to learn about Reflective Data and their Parallel Tracking solution! They were really helpful, worked closely with our team to learn about our needs and finally built a solution that exceeded our expectations. 10/10 would recommend to anyone that for some reason isn’t allowed to use Google Analytics.

Michelle, Director of Online Marketing, Frankfurt

It’s feedback like this that makes us love the work we do even more! Get in touch and learn how we can help you, too.

For more case studies, see here.

The post Case Study: Storing Google Analytics Data Within The European Union or Locally appeared first on Reflective Data.

About the client

Our client is a successful e-commerce business, headquartered in the UK.

Their focus is on food supplements, nutritional products, and diet plans.

Depending on a season, their monthly online revenue is 1-1.5 million GBP (1.3 – 2 million USD). This comes from around 14k orders every month.

Due to NDAs, we are not allowed to disclose the name of the client. Everything else in this case study remains unchanged.

The challenge

When the company in the focus of this case study first contacted us, their main concerns were about properly tracking customer lifetime value (LTV) and churn. Being in the business of food supplements where, naturally, a big chunk of revenue is coming through subscriptions (automatically recurring orders), keeping track of these metrics is extremely important.

Tech stack

The main tech tool stack for this client was a fairly standard one for an e-commerce business of this size. Shopify Plus for website and order processing, ReCharge for handling subscriptions, Google Analytics for general reporting, and ShipStation for managing shipments.

***

While Shopify can provide you with reports for both LTV and churn (via ReCharge analytics), it is rather limited when it comes to attributing those numbers to your marketing and other efforts like CRO or A/B testing.

The client pointed out three main areas where they need a more detailed look into LTV and churn-related metrics:

• Ad Spend (mostly Google Ads and Facebook)
• E-mail campaigns
• CRO and A/B testing

The solution

After having several meetings with various teams from the client’s side, we had a quite good understanding of what kind of reports they’re trying to build and what kind of data it requires.

In order to accommodate all of their needs, we had to involve data from the following sources:

• Shopify
• ReCharge
• Google Analytics
• Google Ads
• Facebook Ads
• ShipStation
• Google Optimize

As our solution involved data from more than three sources, it was quite obvious that setting up a data warehouse is required. The client decided to go with our default recommendation for data warehouse software — BigQuery.

Getting data in BigQuery

As LTV and churn had always been such essential metrics for the client, they had already tried solving this in-house. Their existing tool stack consisted of Stitch, Supermetrics, MySQL database and Google Sheets. While this could’ve worked in theory, the reality was a really slow system that constantly failed and produced inaccurate numbers.

Our solution involved using Parallel Tracking technology for Google Analytics and a custom data pipeline for the rest of the sources. This enabled us to modify the ETL for maximum efficiency while meeting all the data requirements.

Building the reports

Since our client was already familiar with the rest of Google’s Marketing Suite and it works really well with BigQuery, using Data Studio for building the reports was an easy choice. Besides being a free (!) tool, by leveraging the BigQuery BI Engine, Data Studio can provide you with the best performance as well!

After understanding the reports that would be used most frequently, we built a set of 2 main dashboards:

• LTV per source/medium/country/product/campaign/age
• Churn per source/medium/country/product/campaign/age

These dashboards provide the client with a good overview of their key metrics and how they correlate with some of the more common dimensions/attributes. This is a good stepping stone for a deeper analysis and digging for insights.

Luckily, the client’s marketing team already had a few people that were comfortable working with SQL, Jupyter Notebooks etc. This meant that by pointing them in the right direction and providing a few guidelines, they had everything needed for digging deeper into data stored in BigQuery.

Besides dashboards and enabling ad-hoc analysis, a few teams also wished for a periodic automated report in their inbox. This was also something we were happy to set up using some custom software on Google Cloud.

Conclusion

While this project started as a rather standard one for us, it turned out to include several custom parts that made it fun for us and enabled maximum effectiveness for the client.

What better way to take it all together than a few words from the team we worked with on the client’s side.

We turned to Reflective Data with a rather concrete goal in mind — to fix our LTV and churn tracking and reporting. Being in the business of  food supplements, a large portion of our revenues come from recurring orders. This makes keeping track of those metrics extremely important for us. Not only was Reflective Data able to set up a proper tracking system, they helped us figure out the dashboards, email reports and helped our analysts on SQL analysis! Couldn’t be happier about this!

Nothing’s better than such warm words from a happy client!

If you, too, need help with your analytics & data setup, get in touch with us for a free consultation session. I’m sure we can find a solution that suits both of us.

The post Case Study: Sending Data From Shopify and Google Analytics to BigQuery for More Advanced Analysis appeared first on Reflective Data.

In this article, originally posted on medium.com, we’re going to walk through the reasons you might want to access raw Google Analytics data in BigQuery and a few solutions that will get you there.

***

Google Analytics is, without a doubt, the most popular tool in the digital analytics segment. Besides being the best tool for tracking e-commerce, content and lead-gen websites, with some modification you can use Google Analytics for tracking SaaS and other web apps as well.

While being a really powerful and flexible tool, Google Analytics, too, has a few shortcomings. Perhaps the biggest problem every advanced user faces is the fact that there is no way to access raw hit-level data. The second major issue with Google Analytics is related to sampling and data collection limits.

Now, let’s take a look at some of the ways you can overcome these shortcomings without spending a ton on Google Analytics 360 — a premium version of Google Analytics that, to some extent, also eliminates these issues.

PS! If you already have GA 360, these techniques will give you an even more robust and flexible tracking solution.

Table of Contents

– Sampling in Google Analytics
– Ways to avoid sampling in Google Analytics
– Why do you even need the raw data
– How to get access to unsampled hit-level raw data
– What about the price

***

Read the full post on medium.com

The post [Medium] Hit-Level Unsampled Google Analytics to BigQuery Without 360 appeared first on Reflective Data.

Google Analytics, on the other hand, is the most popular tool in the digital analytics world. It is used to keep track of marketing efforts, user behavior, traffic sources and more. The most common way to access data from Google Analytics is the web user interface. Alternative options include the Reporting API or external tools like Looker or Data Studio.

What’s interesting, though, is that there is no way to use the most popular query language to query data from the most popular analytics platform. That’s right, you cannot query Google Analytics data using SQL.

This is no big deal for the more basic users just checking the built-in reports in the Google Analytics UI. It is, for sure, a limitation for power users working with tools like Python and use data in custom models or even feed it into machine learning algorithms and recommendation engines.

In this article, we’re going to cover the solutions that will enable you to query your Google Analytics data using SQL.

Step 1 – Getting data into a database/data warehouse

As mentioned above, there is, unfortunately, no way to query data directly from Google Analytics using SQL. This means that the first step is to get data into some sort of a relational database or a data warehouse that support SQL queries.

There are three options for sending Google Analytics data into some external data storage.

1. Google Analytics 360 to BigQuery export

If your company has the 360 (premium) version of Google Analytics then you can use its native BigQuery export feature. Here are the details for setting this up. Keep in mind, though, that this solution will not work with the standard (free) version of Google Analytics.

2. Google Analytics Parallel Tracking

Google Analytics Parallel Tracking is a third party service that sends all of the raw hits into a data warehouse of your choice (i.e. BigQuery). These hits are then processed into sessions to provide a dataset similar to Google Analytics 360 export.

While this solution is not free, it will cost you only a small fraction of the cost of Google Analytics 360.

Getting started with Google Analytics Parallel Tracking.

3. Export data using the API

This is the most technical solution of the three. It leverages the Google Analytics Reporting API to pull data from Google Analytics and into your database or data warehouse.

One way to get started with this solution is to use the gago library in Go language to communicate with the Reporting API. Then you need the code for writing this data into your database.

The limitations with data export solution are that you can’t query all of the metrics and dimensions at the same time, you’re still affected by the data collection and sampling limits of Google Analytics and a modification in your tracking system is required (custom dimensions for hit timestamp, hit type, sessions ID, client ID and more).

Step 2 – Query Google Analytics data using SQL

This depends a bit on the setup you used to send data into a database/data warehouse and the type of data warehouse being used. Since the most common option is Parallel Tracking and Google BigQuery, we are going to use these in our examples as well.

The simplest way you can run your first SQL query against your Google Analytics data stored in BigQuery is to go into the BigQuery user interface and choose the right dataset containing your Google Analytics data.

With parallel tracking, you will have your Google Analytics data stored in three separate tables.

• raw_hits
• processed_hits
• processed_sessions

Depending on the type of query you want to run, choose the right table. Keep in mind that some of the information (bounce rate, geo-location etc.) isn’t available before hits are processed into sessions.

For example, we could write a simple SQL query like this and see the top 10 countries by the number of Google Analytics sessions.

Now, to make this query a bit more interesting, we might add in metrics like users and bounce rate. All doable with SQL, of course.

Notice how you can define your own rules for things like “bounce” and “bounce rate”.

Pretty neat, right?

Using the BigQuery UI is good for testing your setup and quick ad-hoc queries. In most cases, though, you’ll probably use some sort of a BI tool. One of the popular options these days is Google Data Studio. So, let’s use this in our examples a well.

Google Data Studio is free and probably the easiest to get started with. That being said, it is still packed with useful features and makes creating interactive dashboards a fun and enjoyable process.

Connecting BigQuery with Google Data Studio

Google Data Studio, like many other BI tools, has a native integration for BigQuery as a data source.

While you can automatically pull all data in a certain table, in most cases it’s a better idea to use a custom SQL query.

Data Studio has this feature built-in as well.

In our example, we’re going to analyze the results of an A/B test run using VWO. We’re using VWO’s custom-event-based integration to send experiment data into Google Analytics.

It’s always a good idea to try your query in the BigQuery’s query editor before implementing it in Data Studio.

Once we’re happy with the query and the results it returned, it’s time to put the query in Data Studio.

In Data Studio, we can start building all sorts of cool visualizations. For example this one for visualizing the funnels for Control and Variant 1 of our A/B test.

Data Studio is really flexible when it comes to building custom reports, interactive dashboards and quick visualizations. If you can’t find the right chart from the default chart types, check out the community visualizations or go ahead and build one yourself.

Conclusion

Working with your Google Analytics data using SQL opens up a whole new world of analysis opportunities for your digital analytics data. You no longer have to use the metrics and concepts defined and pre-calculated by Google. Feel free to come up with your own rules for things like calculating bounce rate or defining a session.

To get started, you first need to transfer your Google Analytics data into a data warehouse that supports SQL queries. For example, Google BigQuery. There are a few ways you can send Google Analytics data into BigQuery, our recommended method is Parallel Tracking. This will provide you with the most complete dataset, delivered in near-real-time and for a fraction of the cost of GA 360.

Once you have data in your database or data warehouse, you can start writing ad-hoc SQL queries right away. To take your productivity to a next level, we recommend running your queries and reports through a BI platform (i.e. Data Studio).

If you have any questions about analyzing your Google Analytics data using SQL or sending data into a data warehouse, feel free to ask them in the comments below.

The post How to Query Google Analytics Data Using SQL appeared first on Reflective Data.

The main limitations of Google Analytics are related to sampling and data collection limits. Most affected are companies that can’t afford the premium 360 version of Google Analytics (~150k/year) but still have a good amount of traffic visiting their websites. In general, Google Analytics properties with >1M sessions/month or >10M hits/month are being affected by some heavy sampling and data collection limits.

In this article, we’re going to cover the different types of limitations present in the free version of Google Analytics and provide solutions/workarounds to all of them. Oh, and the solution, in most cases, does not include buying the 360 version.

Types of limitations in Google Analytics

Here’s a quick overview of the different types of limitations that come with the free version of Google Analytics.

Sampling in reports

Sampling on the reporting level means that, even if Google Analytics collected 100% of the hits, the reports you are seeing in the user interface are not based on 100% of those hits/sessions. In general, sampling starts if the period you are looking at contains more than 500k sessions in total. It can start a lot sooner, though, when more advanced custom segments are in use.

Data collection limits

From the official documentation.

If a property sends more hits per month to Analytics than allowed by the Analytics Terms of Service, there is no assurance that the excess hits will be processed. If the property’s hit volume exceeds this limit, a warning may be displayed in the user interface and you may be prevented from accessing reports.

Data collection limits that apply to all free Google Analytics accounts.

• up to 10M hits total per month
• 200,000 hits per user per day
• 500 hits per session

Data processing latency

Processing latency in Google Analytics is up to 24-48 hours. Standard accounts that send more than 200,000 sessions per day to Analytics will result in the reports being refreshed only once a day. This can delay updates to reports and metrics for up to two days.

This makes Google Analytics quite useless for things like monitoring real-time performance, detecting usability issues, automated reports on an hourly basis, real-time product recommendations, and other use cases that require fresh data (i.e. machine learning, marketing automation).

Custom dimensions and metrics

There are 20 indices available for different custom dimensions and 20 indices for custom metrics in each free Google Analytics property. 360 accounts have 200 indices available for custom dimensions and 200 for custom metrics.

While 20 is plenty for a small business with a simple website, this number can become quite limiting if you need custom dimensions for things like A/B testing data (one for each live experiment), e-commerce variables, user-specific information, and more.

Aggregated metrics and no access to raw data

Unfortunately, in the free version of Google Analytics, there is no way to access the raw hit-level data. This means that if the way it calculates certain metrics or aggregates hits into sessions doesn’t suit your business requirements then you’re out of luck – without access to raw data, you can’t define your own calculations or aggregations.

Also, in case you would like to analyze the user journey of a specific visitor, there is no way to easily query and analyze all hits from one visitor in a timely order.

Google Analytics 360 does have a BigQuery export but, by default, the metrics and sessions are still defined and pre-aggregated just like in the reports.

Hit payload limited to 8000 bytes

Depending on your analytics setup, some of your hits may include a lot data – ecommerce product impressions, several custom dimensions etc. The Google Analytics hit payload size is limited to 8000 bytes, this means that all hits greater than this will be ignored by Google Analytics’ data processing engine and your dataset ends up being incomplete. What’s worse, Google Analytics doesn’t alert you when this is happening!

This affects both the free and 360 versions of Google Analytics.

Workarounds to sampling and other limitations

Now, let’s take a look at the same list of limitations and provide a solution/workaround to each of them.

Sampling in reports

Option 1 – Divide your queries into smaller chunks and join data using another tool (Excel, Pandas, R). This can be done manually in the UI, using the Reporting API, or by using a library for your favorite programming language (i.e. gago for Go) that has an “anti sampler” feature built-in.

Option 2 – Collect raw hit-level Google Analytics data in your data warehouse (i.e. BigQuery). This process is known as parallel tracking and functions by duplicating all hits that are sent to your Google Analytics property into your data warehouse. This allows you to write ad-hoc queries in standard SQL without any sampling.

Data collection limits

Option 1 – To get past the 10M hits/mo, 200,000 hits per user per day and 500 hits per session limits, the same Google Analytics parallel tracking solution can be used. This means that even if Google Analytics starts to skip some hits, they’re still available in your data warehouse and ready to be queried.

Option 2 – Set up multiple Google Analytics properties. This can be a little tricky but it is possible to divide your Google Analytics integration into multiple properties. Depending on how much traffic your site gets, you might divide all hits from the first week of the month into one property, hits from the second week into another property etc. Alternatively, you could divide hits based on the Client ID into as many groups/properties as needed. Then, you can use the Reporting API to pull data from all properties and join them using Excel, Pandas or some other tool.

Data processing latency

If you need to monitor the site’s performance in real-time or need data for product recommendations or other machine-learning efforts, the 24-48 hour delay is probably way too long. Your only option to get access to near-real-time Google Analytics data is to use the parallel tracking technology. With parallel tracking, processed hit data will be available in ~5 seconds after it was collected from your site. This allows you to build alert systems, monitor performance in real-time or feed your machine learning algorithms with fresh data at all times.

Custom dimensions and metrics

As mentioned earlier, Google Analytics free version limits you to 20 indices of custom dimensions and metrics. If you hit the limit and need to add more, your first action should be to check whether there’s a dimension or metric that you no longer need. If that’s not the case, your only solution is to either upgrade to Google Analytics 360 (to get 200 metrics and dimensions) or to use parallel tracking and get access to unlimited custom metrics and dimensions. When using parallel tracking, dimensions and metrics with indices 20+ will not show up in Google Analytics reports but are available in your data warehouse.

Aggregated metrics and no access to raw data

If you’re not happy with how Google Analytics aggregates data into sessions or calculates certain metrics, you need access to the raw hit-level data that allows you to define your own rules for almost any kind of aggregation or calculation. There are two options you can sue to get access to raw hit-level Google Analytics data.

Option 1 – You guessed it! Parallel tracking sends the raw hit-level Google Analytics data into your data warehouse (with no sampling) and you can use standard SQL to query your data and tools like Excel, R or Pandas to make all sorts of calculations and aggregations. BigQuery has a native connection with most BI and data visualisation tools so reporting is easy and flexible.

Option 2 – If your Google Analytics property is not affected by the data collection limits (it gets <10M hits/mo), you could configure your custom dimensions to collect data like Client ID, Session ID, Hit Timestamp and Hit Type. With all these dimensions available in your dataset, you can use the Reporting API to pull the raw hit-level data into your local machine or data warehouse. If your property receives more than 10M hits a month, this option is only available if you split your data between multiple properties.

Hit payload limited to 8000 bytes

If your enhanced ecommerce transactions include many products or you have lots of custom dimensions or for some other reason some of your hits exceed the 8000-byte limit then you are losing data. Google Analytics will, without any warnings, simply skip those hits. You have three options for solving this issue.

Option 1 – Send less data by removing some data points. Simo Ahava has written a great blog post on automatically reducing the Google Analytics payload length by removing some of the unnecessary data points and creating an order for removing other data points until the required payload size is reached.

Option 2 – Use the data import feature in Google Analytics to import extra information after the data has been sent to Google Analytics. This way you can send only the most important information (Client ID, transaction ID etc.) with the main hit and import other details (product information, custom dimensions etc.) later using the import feature. Check out this post by Dan Wilkerson for a detailed guide for doing this with Google Tag Manager.

Option 3 – With parallel tracking your hits are limited to 16 000 bytes (double from Google Analytics) and this can be raised even more on request.

Key takeaways

Now that you have a pretty good overview of the limitations that come with the free version of Google Analytics and some that come with the 360 (premium) version, it’s a good time to assess your own Google Analytics implementation(s). Does any of the limitations affect your data and its accuracy? May some of the limitations become an issue in the future?

Knowing the limitations of a tool is extremely important. You may be investing a lot of time and money into a setup that may not be sufficient anymore as your business grows.

With all its limitations, Google Analytics is still an extremely powerful tool that has an awesome community. If sampling or some of the other limitations are an issue for you, I believe you can fix them with some of the solutions suggested in this blog post.

Most of the sampling and data collection limitations of Google Analytics are solvable with parallel tracking, a system that sends raw hit-level Google Analytics data into BigQuery (or any other data warehouse) in real-time.

***

Should you have any other issues with Google Analytics or have an alternative solution to any of the problems mentioned in this post, please share your thoughts in the comments below.

The post How to Avoid Google Analytics Sampling and Data Limits? appeared first on Reflective Data.