Ignoring Googlebot When Tailing Apache Logs

Album Cover: Icky Thump

"You can't be a pimp and a prostitute, too."
White Stripes / Icky Thump

Posted on February 09, 2011 8:38 PM in Web Development
Warning: This blog entry was written two or more years ago. Therefore, it may contain broken links, out-dated or misleading content, or information that is just plain wrong. Please read on with caution.

Every now and then I like to have a live view of my server logs to see what kinds of requests are being made of my various web servers. Anyone with even limited exposure to the Apache web server and Linux hosting environments knows that tail is the weapon of choice for viewing web server requests as they happen:

tail -f /var/log/apache2/access_log

On one of my web servers, Googlebot is a bit like a moth to a flame, constantly making requests that far out-measure requests coming from any other consumer of my web server's files. Because of this, I wanted to figure out how I could run tail but filter out all Googlebot-related requests in an attempt to increase the likelihood of me spotting real requests coming from real users.

In a post titled Using grep to find and not find strings, I learned that the feat can be accomplished simply by combining the tail call with a call to grep, being sure to include the latter's -v option. That option tells grep to match any line that doesn't match the given string or regular expression.

Therefore, to watch my web server log while ignoring all Googlebot-related requests, all I needed to do was use a command like the following:

tail -f /var/log/apache2/access_log | grep -v 'Googlebot'

By replacing "Googlebot" with other strings like "bingbot" or "Firefox," one could easily change the output of the live view to meet his or her needs.

Comments

Dofollow Niche Relevant blog comments on September 05, 2016 at 2:51 AM:

You're the best to share us about this overhaul. Trust you won't get tired on making posts as educational as this.
You're the best to share us about this overhaul. Trust you won't get tired on making posts as educational as this.

Permalink

netflix on October 10, 2016 at 1:26 AM:

it was a brilliant opportunity to visit this sort of site and I am cheerful to know. much thanks to you such a great amount for allowing us to have this open door.. minecraft free free netflix account

Permalink

jxcdnvse on May 15, 2017 at 5:23 AM:

Post Comments

If you feel like commenting on the above item, use the form below. Your email address will be used for personal contact reasons only, and will not be shown on this website.

Name:

Email Address:

Website:

Comments:

Check this box if you hate spam.