Building your own Web Analytics from Log Files – Part 5: Building our first Dashboard
This is the fifth part of the six-part-series “Building your own Web Analytics from Log Files”.
At this part of the series we have our log files in Elasticsearch with indices like
“custom-filebeat-tracking-logs-7.4.0-2020.01.03”. First thing is to set up a Kibana index pattern for this.
Kibana Configuration
In Kibana we go to Management -> Index Patterns -> Create index pattern. As Index pattern we use “custom-filebeat-tracking-logs-*”, which gives us all the indices with our daily index pattern.
In the next step, we set the Time Filter field name to “@timestamp”. This is the timestamp that marks the point where Filebeat indexed the document. This is fine for now, we click “Create index pattern” and are done with this part!
Checking our Data
Now, let’s head to the Discover section in Kibana and look at our index pattern. And there it is: Our log entries show up like we wanted:
This is great already! We can look at the documents and see our fields just like we want them:
Building our Dashboard
With all that great data we can finally build a dashboard in Kibana. To get started, head over to the Visualize section and create a new Vertical Bar chart for our index pattern.
First, lets look at how many users we have on our site. Put the timestamp on the X-axis. On the Y-axis, select the Unique Count Aggregation for the json.visitor_id field. This gives us the count of visitor IDs, which is equivalent to the number of users on our site for a given time:
Let’s save that visualization. Next, we want to know how many new and returning users we have. We can use the previous visualization and modify it a bit. To do this, we split the series by the Terms in the json.new_visitor field. In the search bar, we need to filter for the entry requests, since we only then know about the user status:
Now we see how many of our users are new or returning! Let’s save that as well and create one more visualization for our first dashboard. We would like to know which entry page is popular for our new users. To do this, we create a new Data Table for our index pattern. Now we need to filter by user status and session entry, so we put “json.new_visitor : true and json.session_entry : true” in the search bar. Next we create Buckets for the json.uri field, which gives us the filename of the request:
Now we can see which pages are popular for our new users. With those three visualizations we can now build our first dashboard!
Head over to the Dashboard section in Kibana and create a new dashboard. On the top menu, click “Add” and add our three visualizations. Arrange them by dragging and dropping and resizing them. This gives us something like this, our first dashboard!
Next steps
Now it is up to you to further extend that dashboard. Why not do the same thing we did to get our users with our session ID to get the number of session? Or build a visualization to look at the top pages for all users? Or include referrers or URL parameters to know where your users are coming from and via which marketing campaigns? Or which browsers are most popular? Or specifically in- or exclude Google’s Bot from our dashboard?
The possibilities are endless. In the next part of this series we will step back and take a look at what we accomplished.
German Analyst and Data Scientist working in and writing about (Web) Analytics and Online Marketing Tech.