The content on this blog is provided to you completely for free. If you find any value in my content please consider giving back by helping the people affected by the current Ukraine crisis.
You can donate to a charity of your choice or join me in donating to the UNHCR by clicking Donate Here!
Thank you!

Building your own Web Analytics from Log Files – Part 1: Motivation

| Posted on |

This is the first part of the six-part-series “Building your own Web Analytics from Log Files”.

What is Web Analytics

As the owner or administrator of a website, you will go through different phases of maturity. When you are just starting with a hobby or web project, you will most likely care about the technical setup and gaining traction. Once everything is up and running, you will start asking yourself questions like

  • How many People are using my website?
  • How many of those are new Visitors?
  • Which page on my website attracts the most (new) Visitors?

Those questions are Web Analytics questions. It is what Web Analysts spent their time on to deliver value to the business behind it. To achieve that, we most commonly use tools like Piwik (Matomo), Google Analytics, or Adobe Analytics. Those tools rely on some Javascript code that needs to be integrated on a website to collect data about the Visitors. This is the mature phase, where time and money is spent on Analysts and their tools.

Delivering value fast, precise

While it is definitely helpful to have some experts with you, I don’t think not having those should stop you from being able to answer questions like above. This can be done with some hints on how to amp up the log file processing that is already happening.

This series is intended for those between the first phase, where log files are used for initial technical diagnosis, and the mature phase, where dedicated tools are built and used by specialized experts. It aims to keep the cost and technical footprint as low as possible by using tools commonly found in that phases.

The expectation is that this approach will enable more people to dive into user behavior analysis and therefore help them put the customer first and iterate with more precision based on meaningful data instead of log files alone.

While progressing through this series we will also look at the concepts behind big Web Analytics tools, since we need to build something similar. To round things off, we then take a peek at common questions relevant in mature Analytics practice.

In the next part of the series, we will look at the architectural considerations when starting with log file based analytics. After that, we actually teach our web server how to identify users and sessions to finally collect data and build some dashboards with it.

Other cool posts:

  1. Building your own Web Analytics from Log Files – Part 6: Conclusion
    This is the sixth part of the six-part-series “Building your own Web Analytics from Log Files”. In this series we built a rather sophisticated logging and tracking functionality for our website. We used OpenResty to identify and fingerprint our users via cookies, stored that information to log files which were...
  2. Building your own Web Analytics from Log Files – Part 2: Architecture
    This is the second part of the six-part-series “Building your own Web Analytics from Log Files”. Architecture Overview To start of this series, let’s remember what we want to achieve: We want to enable a deeper understanding of our website users by enriching and processing the log files we already...
  3. Building an Enterprise Grade OpenSource Web Analytics System – Part 1: Architecture
    Some time ago I wrote a litte series on how to amp up your log analytics activities. Ever since then I wanted to start another project building a fully fledged Analytics system with client side tracking and unlimited scalability out of OpenSource components. This is what this series is about,...