Web Analytics with Adobe’s Customer Journey Analytics, Part 2: System Architecture in Experience Platform
This post is the second post of the eight-part-series Web Analytics with Adobe’s Customer Journey Analytics, showing how web sites can be analyzed better using Adobe’s next evolution of Adobe Analytics. In the previous post, we discussed the motivation and scope of this project and why, eventually, existing Adobe Analytics customers will start moving to Adobe’s Customer Journey Analytics. In this post, we will take a look at the different possible solution designs we can use to bring data into Customer Journey Analytics and decide on the best one.
Adobe’s Customer Journey Analytics is built on Adobe’s brand new Experience Platform. With that, it is very flexible in terms of how data can be brought into the tool. Depending on the setup it may seem very easy to bring data in quickly. However, all that flexibility also means we have many ways to deviate from the ideal path, so we must at least think through the long-term implications our solution design might have for all parties involved, from the developer to the analyst to the business user.
Experience Platform already offers plenty of integrations out-of-the-box, like pulling data from Adobe Analytics, polling FTP servers, cloud storage, and even generic APIs. For this post, we are going to explore possible integrations with and without Adobe Analytics, so that both existing and new Adobe customers can read along. What the right solution for you is may depend on more than the factors outlined in this post, so you should of course always keep your own requirements in mind.
For now, we are going to explore three different ways of bringing our data into Customer Journey Analytics:
- Using the Adobe Analytics connector (the “easy” solution for existing customers)
- Using Adobe Analytics Data Feeds and a custom data processing platform (the “complete” solution for existing customers)
- Using only the Web SDK (the best way to go for new customers and those looking to fully migrate)
For each way, we will look at both the architecture as well as the pros and cons for and against it. So, without further ado, let’s start exploring the first option!
Approach 1: Bringing data into Experience Platform using the Adobe Analytics connector
When you are an existing Adobe Analytics customer, this is probably the most easy way for you to bring data into Experience Platform (and from there into Customer Journey Analytics). With the connector, you can use a nice visual interface and don’t have to worry about any complicated setup. In a diagram, this approach would look like this:
I think it’s pretty obvious why this approach may seem preferable: It’s super easy. You can literally spend only 10 minutes on the setup using this approach. Adobe takes care of all the heavy lifting and will pull your existing data into Platform. However, this approach will not pull your full Analytics data, but only a subset that Adobe calls “Mid-values”. If you are familiar with data feeds, you may know about pre- and post-values already. You guessed it: Mid values sit right in the middle, as they hold processed data before the Visitor profile. But that means that things like the Visit Number or persisted Evars are not in the data! Also, the connector will only pull data from the last 13 months, but not more.
Here are the pros and cons of this approach:
Pros:
- Fast and easy setup
- Maintained by Adobe
- Works for both historical and new data
- Can also include classifications
Cons:
- Does not include full data but only Mid values
- Works only for 13 months of historical data
- Still takes weeks to import full amount of data
Approach 2: Using Analytics data feeds and custom processing
To circumvent some of the shortcomings of the first approach, we could also use Adobe Analytics’ data feeds. With those, we can export the full Analytics data set for as long as we have data for. Contrary to the Analytics connector, we can even get things like Visitor Profile data (Visit Number, etc.) without any issues. This is the flow chart:
You can already see that this solution is more involved than the first one. We need the custom data processing step because Experience Platform requires the data to be in normal CSV files, while data feeds provide data in files without header and with additional lookup files. I’ve had some great experiences using Apache NiFi for this use case. From my own experience, it takes Adobe about 30 minutes to deliver a full day of historical data, so you can estimate how long it would take to export your own data.
Here are the pros and cons of this approach:
Pros:
- Full Analytics data set, including Visitor Profile
- Complete historical data
- Also works for new data
- Adobe can provide FTP accounts for transit
- Allows custom data processing and enrichment
Cons:
- Way more complex setup
- Manual processing of data needed
- Maintenance of platform and integration with Adobe needed
- Changes to data feeds or AEP might break your setup
Approach 3: Utilizing Web SDK directly
If you follow this blog, you will have seen my previous posts about the new way to track data with the Adobe stack. Web SDK (formerly know as Alloy) uses Adobe’s new Experience Edge Network to distribute data to different Experience Cloud solutions, one of which is Experience Platform. Since that alone will not give us all the data we would want from a web analytics solution, we can then further process the data with Query Service. The architecture is quite simple, since we can stick entirely with Adobe’s solutions:
This approach will definitely be the way to go for new Adobe customers and those who want to migrate completely. Since processing data with Adobe Analytics incurs a cost per event, you will effectively pay twice for your data with approaches 1 and 2, but only once with this third method. Of course this only works for new data.
Here are the pros and cons of this approach:
Pros:
- Most future-proof approach
- Uses only Adobe solutions
- Allows to use structured data
Cons:
- More complex than approach 1
- Requires to change implementation for existing customers
- Works only for new data
Conclusion
As you might already have guessed, it pretty much depends on your situation which approach you should take. If you use Adobe Analytics today and want to bring your historical data into CJA, my personal recommendation would be going with Approach 2 for historical data while simultaneously adopting Approach 3 for new data. That way, you can use all systems to their maximum potential. If however you should need a quicker solution or don’t have the resources for custom data processing, Approach 1 might be best while you figure out a way to adopt Approach 3. The additional cost is literally what buys you time to plan your migration.
When you are a new Adobe customer, there is no better solution than approach 3. This will get you the most value from the start and be pretty much as future-proof as possible. This is also the approach that we are going to follow for our little series here, so make sure you check out the next part on this blog to learn how to do it!
German Analyst and Data Scientist working in and writing about (Web) Analytics and Online Marketing Tech.