Building an Enterprise Grade OpenSource Web Analytics System – Part 7: Analytics Dashboard
This is the seventh part of a seven-part-series explaining how to build an Enterprise Grade OpenSource Web Analytics System. In this post we are building an Analytics Dashboard in Kibana for our data in Elasticsearch. In the previous post we build the connection from Kafka to Elasticsearch and Clickhouse to store the data. If you are new to this series it might help to start with the first post.
We have come a long way in this series. We built everything from the client implementation with Snowplow to the processing and enrichment pipelines with Kafka and Python and stored all the data in Elasticsearch. Now it is time to make that data accessible in an appealing way to analysts and business users. The obvious solution for Elasticsearch is Kibana, which is developed by the same company and is designed to work perfectly with Elasticsearch!
Webanalytics Dashboard in Kibana
In Kibana, first head to the Index Patterns to tell Kibana about our tracking index. Set up the pattern to macht your configuration in Filebeat. Make sure to have some data in there before starting! Next, choose the tracking.custom_webserver-timestamp field as timestamp like this and click create:
Now we head over to the Discover tab and can look at our final data for the first time and in Realtime:
Expand one of the documents to see the structure of our documents within Elasticsearch. We see that our Visitor Profile is working just like we want it to and all our variables are at the expected place:
To start building a Dashboard in Kibana we need to first build the visualizations we want to put on that dashboard. The most asked questions for our website is most likely “how much traffic do we have?”. To know this, we need to count how many different Users and Sessions we had for a given timeframe. Since Snowplow gives us some nice IDs for both we can use the Unique Count function in Kibana. Build a table like below, where we count both Users and Sessions and the amount of interactions those had on our site:
If we want this to be broken down by our website’s pages, we just add the page variable as a bucket like below:
To make use of our Visitor Profile, we add the Entry Page as the first page of a visit as bucket:
Another interesting thing is which Browsers our Users are using. Let’s add this as a pie chart like this:
To round things off, add another Vis showing the Traffic trend like this:
That’s plenty for the first Dashboard! Let’s go to the Dashboard section in Kibana and add all our beautiful visualizations, maybe like this:
Now we have a very nice realtime dashboard to give some overview to our website metrics! Because of the awesome Kibana capabilities, we can click on the items in our tables to filter the whole dashboard or drag a timeframe on the graph to zoom in. Very fancy!
This is the end for this series. We’ve gone from tracking events on our page to processing and enriching them to a nice interactive dashboard, completely with Open Source components! I hope you find some value in this project, let me know if you build something cool yourself ☺
German Analyst and Data Scientist working in and writing about (Web) Analytics and Online Marketing Tech.