Welcome to the first of potentially many posts in a newly-found yearly tradition. At some point during the year, I noticed that I don’t have a continuously updated list of the things I’ve been up to outside of this blog. And even besides that, it is sometimes hard to keep up with all the content that eventually makes it onto the blog. To help with both, I’m planning to release a post like this every year, potentially in the last week of the year. I’m going to try my best to remember all the conferences, events, webinars, and foreign posts (or even update a draft post throughout the year) and put them in some sort of order. Thank you for another great year on this blog and I hope you enjoy all the different pieces of content! I have a YouTube channel now! Writing blog posts definitely is my go-to […]
Tag: Tutorial
Using Adobe Analytics as a free Data Warehouse
Adobe Analytics is, without a doubt, the most mature Digital Analytics solution we can buy today. We know and love it for the flexible data collection, superior data model and processing, as well as the phenomenal Analysis Workspace interface. I have yet to find a use case I can’t cover with the tool or a platform where it can’t be implemented. While Adobe Analytics works perfectly for the data we collect from digital experiences, companies often use many more digital tools to manage advertisements, analyze search performance, post to social media platforms, etc. Most of those tools have some form of integrated reporting feature, which can hold crucial information to monitor a company’s offsite presence. With growing online activity, this can create a situation where the people working with many tools have to switch between tools many times per day. To help manage this complexity, companies often consider using either […]
Understanding and getting the most out of Activity Map in Adobe Analytics
Adobe Analytics offers a metric ton of useful features which make website tracking super easy. Next to builtin dimensions like Referrer Type or Entry and Exit Page, there are many clever metrics like Entries and Exits for any given dimension. Today, I want to show you one of my favorite features and explain why I love it so much: Activity Map! Activity Map, also known as Click Map, helps Adobe Analytics users understand where website visitors, you guessed it, click on a given page. It comes with a bunch of dedicated dimensions, its very own user interface, and even is super easy to implement through Adobe Launch. In this post I’m going to cover some basics of Activity Map, explain why you should use the dedicated interface more often, and how you can customize it for your very own use cases. And once you’re done reading this post, watch Jenn […]
Understanding and utilizing the Event Object in Adobe Launch
Adobe Launch provides a lot of comfort and clever features to make tag management super easy. Quite a few of those features, like the amazing Constant Data Element, are provided directly in the Launch UI. However, there are a bunch of nice surprises waiting for us once we start exploring the more backend-y technical capabilities. One place where we can catch a glimpse of what Launch has in store for us is in the configuration of Custom Code Actions from the Core Extension. All we have to do is hover our mouse over the Execute globally tool tip: Adobe points us towards three technical variables which we can use in our implementations. One of them is, in my experience, especially useful: The event object. But what is it and why should you use it? Let’s start exploring! What is the Event Object in Adobe Launch? As you are probably aware, […]
Creating Marketing Channel Stacking and Pathing Reports from Adobe Analytics with Power BI
Ever since my very first attendance, Adobe Summit is my number-one source for inspiration for new things to try out in Adobe Analytics. When the world’s leading practitioners and product experts from Adobe come together to share their knowledge, there is a lot to learn for everyone. This year, Eric Matisoff invited me to share a visualization I created in Analysis Workspace as part of the Analytics Rockstars session. However, the true Rockstar content in that session was the Tips & Tricks shared by Jenn Kunz using Excel with the Flow viz in Workspace. A followup conversation on Measure Slack then unveiled some improvements using Data Warehouse and reminded me of an approach of my own that I want to share today. Some years back I used Adobe Analytics’ Data Feeds with Elasticsearch and Grafana to analyze marketing performance beyond what Adobe Analytics has to offer. While that was a […]
Cookie-less Server Side Tracking with Adobe Customer Journey Analytics
If there is one big hot topic in digital analytics right now (besides the unfortunate sunset of Google Analytics 3 and GDPR news) it quite possibly is the recent trend of what many call server side tracking. Currently, server side tracking is an obligatory agenda item at every analytics conference and virtually every vendor of analytics or tag management systems is working on a way to serve the rising demand. However, while there is a lot of talk around the topic, there is no shared definition in our industry of what server side tracking actually is. Jim Gordon has assembled a nice overview of what people might mean when they talk about any of the underlying concepts. In my personal experience, people usually refer to a form of server side tag management, often using Google’s server side tag manager, that still uses some logic in the client’s browser. Adobe has […]
Why you should use the Constant Data Element more often in Adobe Launch
Adobe Experience Platform Dynamic Data Collection Tags, by Adobe Launch is the most prevalent Tag Management System for Adobe Experience Cloud customers. It is nicely integrated with the available Adobe Solutions and is offered completely for free to those customers. But while it is best enjoyed to fuel other solutions, it is a beast of its own with some unique features and mechanics. I’ve already written a few posts about Adobe Launch’s particular features, like the ability to use call traces between rules or how rules can be executed exactly once. Today I want to focus on a specific Data Element, called the Constant Data Element Type. It has first been created by Jan Exner as a dedicated Launch Extension but has (thankfully!) later been built into the Core Extension. I’m going to show you three of my favorite use cases for this, in my opinion, shamefully overlooked gem. You […]
Exploring Adobe Launch Server Side (aka Adobe Experience Platform Data Collection Event Forwarding)
In digital analytics, there has been a trend lately to move data collection away from the client towards a server side implementation. In most cases, companies try to circumvent technical restrictions like Apple’s “Intelligent” Tracking Protection, make collected data more consistent across analytics tools and marketing platforms, or hide their non-GDPR-compliant setups from their users. This trend also brings some (but not all) elements of tag management to the server side, as Jim Gordon described well. In most scenarios, data and events are collected from the client (like a website or app) using a tag manager. Instead of sending events directly to, for example, Adobe Analytics, Google Ads, Facebook, etc., from the browser, they are first sent to a common endpoint that collects, enriches, and forwards data to the desired tools. This common endpoint is usually referred to as a server side tag manager and is implemented in a first-party […]
5 Awesome Adobe Analytics Classification Use Cases
Adobe Analytics has a rich set of features that help businesses collect and report on data in most efficient ways. I’ve written a lot of posts already on how it can be configured to collect data in better ways or how the interfaces can be used to its full extent. Today I want to show you some use cases for one of my favorite features, as it can help you in every stage of an analytics project: Dimension Classifications. Using Classifications, at least in my opinion, is a big step towards making the best use of Adobe Analytics. It can help simplify implementations, manage resources (like the number of used dimensions), surprise users with great new features, or even help you correct errors in data collection retroactively. But before I start with the use cases, let’s make sure we are on the same page by asking… What are Classifications in […]
Keep track of goals using the Linearity Indicator in Adobe’s Analysis Workspace
There is an universal truth in life: Inspiration always strikes when and where you least expect it. The same happened to me the other day, when I was reading High Output Management by former Intel CEO Andrew Grove. While the book is definitely worth reading for anyone interested in management, analysts can benefit just as much from reading it to get inspiration for valuable performance indicators and visualizations. Quite early in the book Grove presents one of his favorite visualizations to track progress towards specific goals: The linearity indicator. This chart shows the current progress towards a set target and where the performance might be heading. Here is his example for a hiring target from the book: My initial reaction was “wow, this is super cool and simple to understand”. If the current progress is above the linear progress, we’re in good shape to reach our goals. If it is […]
Monitor Adobe Analytics usage for free with Power BI and Python
Adobe Analytics is, without a doubt, the most complete and feature-rich product for Web Analytics and Reporting on the market. I won’t even try to list all its features, since I would definitely forget some or would have to update the list in a few months as new functionality is released. And while I, as an analyst and power user, love to have all those great tools available, they create a challenge for me in my other role as an analytics admin. All of those features bring complexity to the every-day work of our business users. For example, when Analysis Workspace was released in 2016, it meant that users had to learn a new interface to get the most value out of Adobe Analytics. But as an admin who knows their users, I have a strong feeling that some people still use the old Reports & Analytics interface in 2021. […]
Web Analytics with Adobe’s Customer Journey Analytics, Part 8: A new home
This post is the eight and last post of the eight-part-series Web Analytics with Adobe’s Customer Journey Analytics, showing how web sites can be analyzed better using Adobe’s next evolution of Adobe Analytics. In the previous post, we were creating the connection from Experience Platform to Customer Journey Analytics. In this post, we are going to take a look at our web analytics data and explore some use cases. Believe it or not, but this series of posts is almost finished! Starting with nothing, we have created a sophisticated schema for our data in Experience Platform, created a tracking implementation using the new Web SDK, enriched our data in Query Service, and pulled all that data into Customer Journey Analytics. If you have been following since the start of the series, I want to say: Thank you, hope you enjoyed the ride! Now it is time for the finale, where […]
Web Analytics with Adobe’s Customer Journey Analytics, Part 7: Customer Journey Analytics Backend Configuration
This post is the seventh post of the eight-part-series Web Analytics with Adobe’s Customer Journey Analytics, showing how web sites can be analyzed better using Adobe’s next evolution of Adobe Analytics. In the previous post, we were enriching our basic web analytics data with some advanced fields in Query Service. In this post, we are creating the connection from Customer Journey Analytics to Experience Platform. At this point in this series, we have a world-class dataset of web analytics data in Experience Platform, ready to be analyzed. I’m personally very proud of the things we were able to achieve in Query Service, especially with the pathing dimensions. With all of that, we have even more than what normal Adobe Analytics would give us! With all the data enriched, we now have only one step left before we can start analyzing our digital user’s behavior. First, we need to pull data […]
Visualizing Adobe Analytics Report Suites for free with Python and Power BI
Adobe Analytics is super flexible in the way it can be set up to exactly match all requirements of business users, analysts, and developers. A crucial part of any implementation is the creation and configuration of the Report Suites, which can be seen as the backend database of Adobe Analytics, that will hold the events sent to Adobe Analytics. In theory (and practice in some setups), each and every Report Suite can have a completely individual set of variables and metrics. However, having the option to create an individual configuration of dimensions and events for each Report Suite comes with a hefty long-term cost. For example, each and every setup needs to be implemented in Adobe Launch, where the on-page data layer needs to be matched to the dimensions and metrics of the Report Suite. If every Report Suite is configured differently, a lot of work needs to be put […]
Web Analytics with Adobe’s Customer Journey Analytics, Part 6: Advanced Data Processing in Query Service
This post is the sixth post of the eight-part-series Web Analytics with Adobe’s Customer Journey Analytics, showing how web sites can be analyzed better using Adobe’s next evolution of Adobe Analytics. In the previous post, we took a look at processing some basic data we need for our web analytics use case utilizing Query Service in Experience Platform. In this post, we are creating some advanced fields to our data in Query Service. I think it’s fair to say that even with just the information from the previous part, we could have a very useful web analytics tool already. But if you know me, you know that I like to take things to the next level wherever I can, especially if it involves writing code. And is SQL not some sort of code too? Entry and exit page were a nice start last time, but we have some fields still […]
Web Analytics with Adobe’s Customer Journey Analytics, Part 5: Basic Data Processing in Query Service
This post is the fifth post of the eight-part-series Web Analytics with Adobe’s Customer Journey Analytics, showing how web sites can be analyzed better using Adobe’s next evolution of Adobe Analytics. In the previous post, we took a look at doing the implementation using Adobe Launch, the Adobe Web SDK, and Client Data Layer. In this post, we are going to processing some basic data we need for our web analytics use case utilizing Query Service in Experience Platform. This series of posts is coming along quite nicely. If you followed all the previous posts until now, you will now have a functioning Web SDK implementation that tracks your data into Experience Platform following the Experience Data Schema we have tailor-made for our use case. Nice! Now we are ready to feed our data into Customer Journey Analytics, right? Well, we could. If we are just interested in the plain […]
Implementing Adobe Analytics in a First-Party context
One of the things I love most about Adobe Analytics is how flexible it is. That is not only true for its interface, Analysis Workspace, and the numerous integrations, but also for everything that happens on the actual website where it is implemented. Almost every functional detail can be configured and tweaked, including the destination where the data is actually sent to. For years Adobe has offered Analytics customers the Managed Certificate program, where Adobe would allow us to send data to a server that looks like it belongs to our own company (or, more specifically, our company’s domain). But upon closer inspection, those requests are only disguised as first-party and are actually still sent to Adobe’s servers directly instead of our own. And in addition to that, Adobe Launch and the Experience Cloud ID Service will still load their data from different domains that don’t belong to your company […]
Web Analytics with Adobe’s Customer Journey Analytics, Part 4: Capturing Data with Web SDK (Alloy)
This post is the fourth post of the eight-part-series Web Analytics with Adobe’s Customer Journey Analytics, showing how web sites can be analyzed better using Adobe’s next evolution of Adobe Analytics. In the previous post, we took a look at our business questions and how we can structure our data most effectively. In this post, we are doing the actual implementation using Adobe Launch, the Adobe Web SDK, and Client Data Layer. On our way to creating a full-scope, front-to-back implementation of Customer Journey Analytics to track a web site, we are now ready to think about our actual implementation. Since we have the data structure in place and already have an awesome Experience Event Schema, we just need some actual data. The logical choice to feed data to the Adobe stack is, of course, by utilizing their client-side tools as well. Specifically, we are going to use Adobe Launch […]
Web Analytics with Adobe’s Customer Journey Analytics, Part 3: Data Structure in Experience Platform
This post is the third post of the eight-part-series Web Analytics with Adobe’s Customer Journey Analytics, showing how web sites can be analyzed better using Adobe’s next evolution of Adobe Analytics. In the previous post, we took a look at the different possible solution architectures we can use to bring data into Customer Journey Analytics and decided on the best one. In this post, we will take a look at our actual business questions and how we can structure our data most effectively. From the last post we already know that we want to track data using only the new Adobe Web SDK going forward. To make that work, we need to create a schema in Experience Platform first, which defines the structure of the data that we want to capture. While some people (sometimes me included) see schema management as one of the more tedious tasks in Platform, I […]
Web Analytics with Adobe’s Customer Journey Analytics, Part 2: System Architecture in Experience Platform
This post is the second post of the eight-part-series Web Analytics with Adobe’s Customer Journey Analytics, showing how web sites can be analyzed better using Adobe’s next evolution of Adobe Analytics. In the previous post, we discussed the motivation and scope of this project and why, eventually, existing Adobe Analytics customers will start moving to Adobe’s Customer Journey Analytics. In this post, we will take a look at the different possible solution designs we can use to bring data into Customer Journey Analytics and decide on the best one. Adobe’s Customer Journey Analytics is built on Adobe’s brand new Experience Platform. With that, it is very flexible in terms of how data can be brought into the tool. Depending on the setup it may seem very easy to bring data in quickly. However, all that flexibility also means we have many ways to deviate from the ideal path, so we […]
Execute a rule only once in Adobe Launch
When you are managing an implementation of Adobe tools like Analytics, chances are you are using Adobe Experience Platform Data Collection, formerly known as Adobe Launch, Adobe Dynamic Tag Management, or the Adobe Tag Manager (and no, this is not a SEO text, it’s the actual list of names. I will still call it Launch for the near future) to implement other tags as well. While Launch is great for making implementation of Adobe’s own tools very fast and easy, managing other tools is not always so straight forward. A common requirement for those 3rd Party tags is to fire a certain tag or pixel only once. What once really means (once per session, user, day, year?) might differ from tag to tag, so as a result it can be surprisingly difficult to fulfill those requirements reliably and consistently. On top of those varying definitions of once, Adobe Launch has […]
Web Analytics with Adobe’s Customer Journey Analytics, Part 1: Goodbye Adobe Analytics, my Old Friend
This post is the first post of the eight-part-series Web Analytics with Adobe’s Customer Journey Analytics, showing how web sites can be analyzed better using Adobe’s next evolution of Adobe Analytics. In this part, we discuss the motivation and scope of this project and why, eventually, existing Adobe Analytics and new customers will start moving to Adobe’s Customer Journey Analytics. If you found this article, chances are high you work in or adjacent to the field of digital analytics or web analytics. It doesn’t really matter if you are an existing Adobe Analytics user, on the Google stack, or just looking for your very first web analytics tool. If you have been following the trends and discussions in our industry in the recent time, you will likely already have caught on the massive changes that both our industry and Adobe’s products go through. With changes to privacy requirements and cookie […]
Tracking Apps with Adobe’s new Experience Platform SDKs and the Edge Network
Tracking mobile apps has always been fun for me. Compared to measuring websites, apps are a more controlled environment, which is great for data consistency. Sure, there are some less-fun cases (like those pesky hybrid apps) but the general experience on major platforms like Android and iOS has been quite great! Changes even happen a bit slower compared to browsers and there are way less moving parts to keep track of. Just like when tracking websites, Adobe has been most helpful in providing us the tools we need to collect data in the most effective and consistent way. If you have been tracking mobile apps with Adobe for a while, you will know about the similarities between websites and apps (like having a trackState function that increments Page Views and trackAction for Custom Links) and nice comfort features (like automatically tracking App ID, Launches, Upgrades, etc.) that make it so […]
A basic AEP Web SDK (Codename Alloy) implementation for Adobe Analytics
If you are taking care of an Adobe Experience Cloud implementation, there is practically no way for you to not hear about the new kid on the block: The Adobe Experience Platform Web SDK (also know as Alloy). It is the next big thing for how to implement solutions like Adobe Analytics, Target, Audience Manager, or the Experience Cloud ID Service. Instead of calling a plethora of different libraries and scripts, Web SDK is a single script that handles all functionality and calls trough a shared endpoint. On the other hand, it’s not available to the public yet. If you want to try it out, you need to contact your Adobe Customer Success Manager and request to be included in the beta. That will give you access to the Edge Configuration in Launch and a stripped-down version of Adobe’s Experience Platform to manage the XDM. This already hints that there […]
Announcing the open collection of Adobe Analytics best practices
Imagine a situation like this: You are facing a new challenge when using or implementing Adobe Analytics. What do you do? If you are like me, you first check out the documentation to make sure you’ve understood the available features correctly. Then, you start researching blog posts and articles around your topic to see if and how anyone has solved this before. If you are still unsure, you might ask some people on Twitter, LinkedIn, or the Measure Chat. As a last resort, you might even reach out to Client Care and ask for help. It’s easy to see why this approach is not ideal. First, it’s not easy to know if the way you approach a task is still the best way or if new solutions exist. Depending on which pages you found when researching, you might end up with an outdated solution or contradicting approaches by different authors […]
(Time-)Normalize Performance over time in Adobe Analytics’s Analysis Workspace
In Digital Analytics, one of the most common requests from business stakeholders is to compare the performance of two or more items on our websites, like marketing campaigns or content pages. While it is immediately obvious why this comparison is important to the business, it quite often leads to graphs like this, where the analyst tries to visualize performance over time: This solution is technically correct but makes it hard to really compare how both pages perform in direct comparison with each other. They went public on different dates and while Page A is rather stable in regards to traffic, Page B got a boost at around the middle of its time online. So, how do we make this simpler? When enjoying my free time between jobs, I caught up on some older videos from the Superweek Analytics Summit’s Youtube Channel. In 2019, Tim Wilson demonstrated how to align dates […]
Cool Approximate Count Distinct Use Cases – Adobe Analytics Tips
One of the things that really sets Adobe Analytics apart from other solutions is the ability to create sophisticated Calculated Metrics and Segments on the fly. You don’t need to be a highly trained Analyst or Data Scientist to create your very own set of Measures and Dimensions unique to your business question. The best thing for me personally is that we can create those metrics from the same interface where we do our day-to-day analysis and reporting. It doesn’t matter if we want to quickly create an average or build advanced time series analysis dashboards, it’s all right there at our fingertips. Today I want to tell you about one of my personal-favorite functions called Approximate Count Distinct. This functionality allows us to count how many different values from a dimension we tracked and use that number in both Calculated Metrics and Segments (making this function the closest we […]
Privacy-centered Analytics with Matomo and Adobe’s Customer Journey Analytics
Legal Disclaimer: Data Privacy is a diverse and ever-changing topic. This makes it nearly impossible to give reliable recommendations to a broad audience. Please consult your company’s legal department on whether those ideas described here are feasible under your jurisdiction. If there has been one predominant topic in the web analytics space for the last couple of years, it surely is data privacy. GDPR is a thing in Europa, COPPA in the US, ITP on planet Apple, and cookie consent banners on every website. Conducting a safe data collection practice as a global business has become more and more challenging, pushing businesses to be more and more careful. Because of this landscape, a lot of businesses are looking for a “bullet-proof” way to analyze website users’s behavior. While Google Analytics is a data privacy nightmare, tools like Piwik Matomo try to justify their existence by claiming to be more privacy […]
Even more Query Service tips to level up your Customer Journey Analytics game
It has been more than two weeks since my last post. Where have I been? Easy answer: Fully and blissfully engulfed in the blessing that is Adobe’s Customer Journey Analytics. I’ve imported my Adobe Analytics data, done some custom user stitching, and created some fun things that I didn’t even knew I’ve always wanted from Adobe Analytics. This experience (pun intended) has completely shifted the way I look at my own data. Following up on my last post, I would like to share some new and cool things I discovered using some more Adobe Defined Functions, window functions, and standard SQL. I wont give a long introduction into Query Service again, but feel free to revisit the previous post for some information on how Query Service interacts with Adobe Experience Platform datasets and how to write data back to those. Getting creative with Sessionization Recreating Adobe Analytics sessions in Query […]
My favorite Query Service tricks in Adobe Experience Platform for Customer Journey Analytics
Wow, that’s a long title for a post. As you might have heard, I recently got the chance to dive into Adobe’s Experience Platform since my company got provisioned for Customer Journey Analytics. That means that I now have all my Adobe Analytics data in Platform, ready for Query Service to enrich it. I think I will do another post on how I amp up my Analytics game by using both Query Service and Customer Journey Analytics together. But this post will give some examples on how to use Query Service in general and how it interacts with Experience Platform. It will be a quick tutorial and provide some use cases for the custom Adobe Defined Functions (ADFs). At the end, you might end up with long SQL queries like I do: How Query Service works and interacts with Platform So what actually is Query Service? It provides a SQL […]
Retention Analysis in Adobe Analytics – Part 2: Custom Segments and Metrics
User Retention is crucial to any digital offering. If you optimize your offering to a point where users come back on their own, you can not only save on marketing cost but also engage your existing users more. This makes retention analysis a prime example for how digital analytics can provide tangible business value. In the previous post, we used Cohort Tables and some builtin features of Adobe Analytics to analyze User Retention. But there is a lot more Adobe Analytics has to offer once we start using Segments and Calculated Metrics. In this post we are going to build our very own Segments to see how many of our Users we are able to retain. Based on those Segments we will then define some Calculated Metrics to make our lives even easier. I’ve also put the results on the Open Adobe Analytics Components Repository. Let’s start building! Simple User […]
Retention Analysis in Adobe Analytics – Part 1: Cohort Tables and Builtin Functionality
User Retention is crucial to any digital offering. If you optimize your offering to a point where users come back on their own, you can not only save on marketing cost but also engage your existing users more. This makes retention analysis a prime example for how digital analytics can provide tangible business value. In this post, we are going to take a look at how we can analyze user retention with the most advanced digital analytics tool, Adobe Analytics. We are going to start in this post with the builtin analytics dimensions and metrics, then take a look at cohort tables, and in the next post even build our own Segments and Calculated Metrics to help us understand retention. Let’s get started! Builtin Retention Metrics and Dimension To start things off, we will take a quick look at what Adobe Analytics has to offer out-of-the-box to help us understand […]
Adobe Analytics Introduction: Terms and Concepts
This is one of several post aiming to give an introduction into Adobe Analytics. They are intended as both tutorials and references for future use. While there already are a lot of good sources for this, some are quite dated and miss connections to recently released features and enhancements. In this post, I will explain some general things that are helpful to know when starting with Adobe Analytics. We will go over different interfaces to analyze data, explain Dimensions, Metrics, and Events and name some common integrations. Know what you are looking at: Dimensions One of the most important building blocks of Adobe Analytics are Dimensions. With Dimensions, we capture descriptive values on our websites or in our apps. Many people call them variables when explaining the general concept. On a website we might record the name of a certain page in a dimension. This would allow us to report […]
Integrating Adobe Target and Adobe Analytics into Voice Assistants
With digital experiences on the rise, interactive Voice Assistants like Amazon Alexa, Google Home or Apple’s Siri are still gaining popularity. Companies now need to meet their customers expectations and allow them to interact with their brand however they like. Those new possibilities require a clear strategy to avoid wasting time and resources on products which nobody actually uses. Adobe Analytics can help understand digital experiences better and drive value through customer feedback. With Adobe Target for personalization and experimentation, nothing can stop you from delivering the right experience at the right time. This article describes how both Analytics and Target can be integrated in Voice Assistants’ backend systems to track and test how users are interacting with your App. We will be using the a direct integration with the Experience Cloud ID Service, sync identifiers and use them for Analytics and Target. Target will then be used to personalize […]
Building an Enterprise Grade OpenSource Web Analytics System – Part 7: Analytics Dashboard
This is the seventh part of a seven-part-series explaining how to build an Enterprise Grade OpenSource Web Analytics System. In this post we are building an Analytics Dashboard in Kibana for our data in Elasticsearch. In the previous post we build the connection from Kafka to Elasticsearch and Clickhouse to store the data. If you are new to this series it might help to start with the first post. We have come a long way in this series. We built everything from the client implementation with Snowplow to the processing and enrichment pipelines with Kafka and Python and stored all the data in Elasticsearch. Now it is time to make that data accessible in an appealing way to analysts and business users. The obvious solution for Elasticsearch is Kibana, which is developed by the same company and is designed to work perfectly with Elasticsearch! Webanalytics Dashboard in Kibana In Kibana, […]
Building an Enterprise Grade OpenSource Web Analytics System – Part 6: Data Storage
This is the sixth part of a seven-part-series explaining how to build an Enterprise Grade OpenSource Web Analytics System. In this post we are taking a brief look on what we can do with the data we collected and processed with Clickhouse. In the previous post we built a persisted visitor profile for our visitors with Python and Redis. If you are new to this series it might help to start with the first post. During this series we defined multiple topics within Kafka. Now we have different levels of processing and persistence available. If we want to keep any of it, we should put it in a persistent storage like a Data Lake with Hadoop or a Database. For this project, we are using Elasticsearch and dipping our toes in a database called Clickhouse for fun! Feeding Data into Elasticsearch From the previous part, we have a nice Kafka […]
Building an Enterprise Grade OpenSource Web Analytics System – Part 5: Visitor Profile
This is the fifth part of a seven-part-series explaining how to build an Enterprise Grade OpenSource Web Analytics System. In this post we are going to build a visitor profile to persist some of the data we track with Python and Redis. In the last post we processed the raw data using Python and wrote it back to Kafka. If you are new to this series it might help to start with the first post. Now that we have a nice processed version of our events, we want to remember certain things about our users. To do this, we are going to create a Visitor Profile in Redis as high performance storage. The process for persisting values will look like this: Building our Visitor Profile First things in this part, we are setting up a little helper script that will take our processed tracking events and flatten them. It looks […]
Building an Enterprise Grade OpenSource Web Analytics System – Part 4: Data Processing
This is the fourth part of a seven-part-series explaining how to build an Enterprise Grade OpenSource Web Analytics System. In this post we are building the processing layer to work with our raw log lines. In the last post we used Nginx and Filebeat to write our tracking events to Kafka. If you are new to this series it might help to start with the first post. At this part of the series, we have a lot of raw tracking events in our Kafka topic. We could already use this topic to store the raw loglines to our Hadoop cluster or a database. But it would be much easier later on to do some additional processing to make our life a litte easier. Since Python is the data science language today we will be using that language. The result will then be written to another Kafka topic for further processing […]
Building an Enterprise Grade OpenSource Web Analytics System – Part 3: Data Collection
This is the third part of a seven-part-series explaining how to build an Enterprise Grade OpenSource Web Analytics System. In this post we are setting up the tracking backend with Nginx and Filebeat. In the last post we took care of the client side implementation of Snowplow Analytics. If you are new to this series it might help to start with the first post. Now that we have a lot of data that is being sent from our clients, we need to build a backend to take care of all the events we want. Since we are sending our requests unencoded via GET, we can just configure our web server to write all requests to a logfile and send them off to the processing layer. Configuring Nginx with Filebeat In our last project we used a configuration just like the one we need. As web server, we used and will […]
Building an Enterprise Grade OpenSource Web Analytics System – Part 2: Client Tracking
This is the second part of a seven-part-series explaining how to build an Enterprise Grade OpenSource Web Analytics System. In this post we are setting up the Client Tracking using the Javascript tracker from Snowplow Analytics. In the last post we took a look at the system architecture that we are going to build. If you are new to this series it might help to start with the first post. When building a mature Web Analytics system yourself, the first step is to build some function into your app or website to enable sending events to the backend analytics system. This is called client side tracking, since we rely on the application to send us events instead of looking at logfiles alone. For this series we are going to look at website tracking specifically, but the same principles apply to mobile apps or even server side tracking. Almost every mature […]
Building an Enterprise Grade OpenSource Web Analytics System – Part 1: Architecture
Some time ago I wrote a litte series on how to amp up your log analytics activities. Ever since then I wanted to start another project building a fully fledged Analytics system with client side tracking and unlimited scalability out of OpenSource components. This is what this series is about, since I had some time to kill during Easter in isolation ? This time, we will be using a tracker on the browser or mobile app of our users instead of logfiles alone, which is called client side tracking. That will give us a lot more information about our visitors and allow for some cool new use cases. It also is similar to how tools like Adobe Analytics or Google Analytics work. The data we collect has then to be processed and stored for analysis and future use. As a client side tracker, we will be using the Snowplow tracker. […]
Tutorial: Real time dynamic Personalization with Adobe Target
Adobe Target is part of the Adobe Experience Cloud. It can be used for A/B Testing and Personalization of Websites, Apps and Server Side Applications. This article describes how it can be used to create dynamic Experiences based on the user profile and mbox parameters. Personalization is not a feature, it’s obligatory Some years ago, the internet was not more than a collection of static pages. It was perceived as the worlds largest library, so websites were created the same way as a library: Books don’t change their content over time, or react to the reader. Much like in a library, search indices like Google helped to navigate the web and find the best content. Over time, websites started to make better use of what computers are capable of. Animated images and blinking text rivaled for the user’s attention. If you got an email a human voice would tell you […]
Building your own Web Analytics from Log Files – Part 6: Conclusion
This is the sixth part of the six-part-series “Building your own Web Analytics from Log Files”. In this series we built a rather sophisticated logging and tracking functionality for our website. We used OpenResty to identify and fingerprint our users via cookies, stored that information to log files which were shipped to Elasticsearch and visualized with Kibana. Web Analytics democratized By using those techniques, we are able to use what we already have (log file processing) to answer questions about our users. Under best conditions this doesn’t even lead to a bigger technical footprint. This way we can have deep insights into our user behavior without external tools. Even as a startup or hobby developer you are now able to put the user first on your digital platforms. Next steps While this series is done for now we have a starting point to further build our platform. With some frontend […]
Building your own Web Analytics from Log Files – Part 5: Building our first Dashboard
This is the fifth part of the six-part-series “Building your own Web Analytics from Log Files”. At this part of the series we have our log files in Elasticsearch with indices like “custom-filebeat-tracking-logs-7.4.0-2020.01.03”. First thing is to set up a Kibana index pattern for this. Kibana Configuration In Kibana we go to Management -> Index Patterns -> Create index pattern. As Index pattern we use “custom-filebeat-tracking-logs-*”, which gives us all the indices with our daily index pattern. In the next step, we set the Time Filter field name to “@timestamp”. This is the timestamp that marks the point where Filebeat indexed the document. This is fine for now, we click “Create index pattern” and are done with this part! Checking our Data Now, let’s head to the Discover section in Kibana and look at our index pattern. And there it is: Our log entries show up like we wanted: This […]
Building your own Web Analytics from Log Files – Part 4: Data Collection and Processing
This is the fourth part of the six-part-series “Building your own Web Analytics from Log Files”. Legal Disclaimer: This post describes how to identify and track the users on your website using cookies, IP adresses and browser fingerprinting. The information and process described here may be subject to data privacy regulations under your legislation. It is your responsibility to comply with all regulations. Please educate yourself if things like GDPR apply to your use case (which is very likely), and act responsibly. In the last part we have built a configuration for OpenResty to generate user and session IDs and store them in browser cookies. Now we need a way to actually log and collect those IDs together with the requests our web server handles. OpenResty Configuration To be able to log our custom variables we need to announce them to Nginx. This is done right in the server-part of […]
Building your own Web Analytics from Log Files – Part 3: Setting up Nginx with OpenResty
This is the third part of the six-part-series “Building your own Web Analytics from Log Files”. Legal Disclaimer: This post describes how to identify and track the users on your website using cookies and browser fingerprinting. The information and process described here may be subject to data privacy regulations under your legislation. It is your responsibility to comply with all regulations. Please educate yourself if things like GDPR apply to your use case (which is very likely), and act responsibly. Identifying Users and Sessions One of our goals for this project is to be able to tell how many people are using our site. This means we need a way to differentiate between the users on our site. One approach would be to look at the IP addresses of our users. This is not very precise since all devices with the same internet connection share an IP address. Especially for […]
Building your own Web Analytics from Log Files – Part 2: Architecture
This is the second part of the six-part-series “Building your own Web Analytics from Log Files”. Architecture Overview To start of this series, let’s remember what we want to achieve: We want to enable a deeper understanding of our website users by enriching and processing the log files we already collect. This article looks at the components we need for this and how to make our life as easy as possible. To achieve our goal, we need to teach our web server to identify our users, store information about the activity in the log files, ship those files to storage and make it actionable with a way of visualizing it. Because I believe in Open Source Software, we will look at our options among that category. Another requirement is to introduce as less components as possible and keep scalability in mind. Choosing our Web Server The first part of our […]
Building your own Web Analytics from Log Files – Part 1: Motivation
This is the first part of the six-part-series “Building your own Web Analytics from Log Files”. What is Web Analytics As the owner or administrator of a website, you will go through different phases of maturity. When you are just starting with a hobby or web project, you will most likely care about the technical setup and gaining traction. Once everything is up and running, you will start asking yourself questions like How many People are using my website? How many of those are new Visitors? Which page on my website attracts the most (new) Visitors? Those questions are Web Analytics questions. It is what Web Analysts spent their time on to deliver value to the business behind it. To achieve that, we most commonly use tools like Piwik (Matomo), Google Analytics, or Adobe Analytics. Those tools rely on some Javascript code that needs to be integrated on a website […]