Command line tips to analyse log files

I was reading Ivan's post in the forum about visitor activity and thought it might be a good opportunity to mention a few CLI (Command Line Interface) tips for analysing Apache Log files. I actually helped Ivan by using these same commands a few months ago....

I will be using the following two unix commands:
1. grep Global Regular Expression Parser
2. wc Word Count

First you will need to locate your Apache access log file. On OS X it is located here:

/var/log/httpd/access_log


A simple scenario we covered was what percentage of the hits were for the main RSS feed ?

We need to calculate how many hits the log file was showing:

wc -l /var/log/httpd/access_log


The -l switch makes wc count lines, not words.

On my personal development machine I have 858 entries.

Next we can see the number of RSS hits by using the RSS url:
http://creativebits.org/node/feed

grep 'node/feed' /var/log/httpd/access_log | wc -l


On my personal screenshot I am using 'index' instead of 'node/feed' since i do not run Drupal locally.

The symbol between access_log and wc is the vertical bar called 'pipe'.

What we are doing is using grep to search the access_log file, and find all lines that contain 'node/feed' and 'pipe' them to the wc command.

On my example I have 67 hits for index.

If Ivan wanted to see how many people clicked through to the member's forum he would type:

grep 'membersforum' /var/log/httpd/access_log | wc -l


We could also calculate how many people viewed this website were not using a Mac.

grep -v 'PPC Mac' /var/log/httpd/access_log | wc -l


This would parse the access log for any line that did NOT mention 'PPC Mac', which both Safari and Firefox declare in their headers by default.

These are handy tips to know since they use tools which are installed on probably every unix machine, and only require SSH access.

There are many, many features of grep which I have not covered here, mainly Regular Expressions. There is enough there to fill a book.