Using AWStats Perl script for HTTP log abalysis

• Introduction
• Configuration file
• On demand update of statistics
• Generation of specific sections from the command line
• Queries in Web form

Introduction

AWStats is Perl script licensed under the GNU General Public License that can work in standalone (command line) and CGI modes and generates web site usage report as well as several other reports (can be used for proxy log processing and several other protocols like ftp and mail).

It uses a partial information file to be able to process large log files, often and quickly. Supported are all major servers that produce Apache-style log files (NCSA combined/XLF/ELF log format or common/CLF log format), as well as WebStar, IIS (W3C log format) and several others. Proxy servers are also supported.

AWStats works from the command line as CGI. It is widely used by web hosting providers (provided via Cpanel).

You can install AWstats from RPM. See AWStats Documentation - Setup page

Configuration file

After install the first thing to do is to edit /etc/httpd/conf.d/awstats.conf

Note: conf file in the /etc/httpd/conf.d/ folder are automatically loaded as part of the Apache configuration. There is no need to add them into httpd.conf. This is a cleaner setup separating different applications in their own sections of configuration files; also in a hosted environment you typically can't edit httpd.conf:

Alias /awstats/icon/ /var/www/awstats/icon/

ScriptAlias /awstats/ /var/www/awstats/
<Directory /var/www/awstats/>
        DirectoryIndex awstats.pl
        Options ExecCGI
        order deny,allow
        allow from all
</Directory>

Alias /awstatsclasses "/var/www/awstats/lib/"
Alias /awstats-icon/ "/var/www/awstats/icon/"
Alias /awstatscss "/var/www/awstats/examples/css"

Note: the mod_cgi module of Apache must be pre-loaded into Apache otherwise Apache will not try to view the file, it will try to execute it. This can be done in two ways, either enable for the entire web server, or utilizing VirtualHosts, enable for AWStats.

Edit the following lines in the default AWStats configuration file /etc/awstats/awstats.localhost.localdomain.conf:

SiteDomain="<server name>.<domain>"
HostAliases="<any aliases for the server>"

Rename config file:

mv /etc/awstats/awstats.localhost.localdomain.conf /etc/awstats/awstats.<server 
	name>.<domain>.conf

Start Apache:

/etc/init.d/httpd start

To automate startup of Apache on boot up, type

chkconfig --add httpd

Verify Install

Go to

http://<server name>.<domain>/awstats/awstats.pl?config=<server name>.<domain>

Report consists of sections that can be generated also from the command line:

Note: you must have created a statistics data base for the analysis period by processing your log files before you try to create reports.

To build the main reports, in a static HTML page, from the command line you can use the following syntax (skip to the second option if you only have CGI access):

perl awstats.pl -config=mysite -output -staticlinks > awstats.mysite.html 

On demand update of statistics

You can manually update Statistics (Note: By default, statistics will be updated every hour.):

/usr/bin/awstats_updateall.pl now -confdir="/etc" -awstatsprog="/var/www/awstats/awstats.pl"

Generation of specific sections from the command line

You can also generate specific sections from the command line :

perl awstats.pl -config=mysite -output=alldomains -staticlinks > awstats.mysite.alldomains.html
perl awstats.pl -config=mysite -output=allhosts -staticlinks > 	awstats.mysite.allhosts.html
perl awstats.pl -config=mysite -output=lasthosts -staticlinks > awstats.mysite.lasthosts.html
perl awstats.pl -config=mysite -output=unknownip -staticlinks > awstats.mysite.unknownip.html
perl awstats.pl -config=mysite -output=alllogins -staticlinks > awstats.mysite.alllogins.html
perl awstats.pl -config=mysite -output=lastlogins -staticlinks > awstats.mysite.lastlogins.html
perl awstats.pl -config=mysite -output=allrobots -staticlinks > awstats.mysite.allrobots.html
perl awstats.pl -config=mysite -output=lastrobots -staticlinks > awstats.mysite.lastrobots.html
perl awstats.pl -config=mysite -output=urldetail -staticlinks > awstats.mysite.urldetail.html
perl awstats.pl -config=mysite -output=urlentry -staticlinks > awstats.mysite.urlentry.html
perl awstats.pl -config=mysite -output=urlexit -staticlinks > awstats.mysite.urlexit.html
perl awstats.pl -config=mysite -output=browserdetail -staticlinks > awstats.mysite.browserdetail.html
perl awstats.pl -config=mysite -output=osdetail -staticlinks > awstats.mysite.osdetail.html
perl awstats.pl -config=mysite -output=unknownbrowser -staticlinks > awstats.mysite.unknownbrowser.html
perl awstats.pl -config=mysite -output=unknownos -staticlinks > awstats.mysite.unknownos.html
perl awstats.pl -config=mysite -output=refererse -staticlinks > awstats.mysite.refererse.html
perl awstats.pl -config=mysite -output=refererpages -staticlinks > awstats.mysite.refererpages.html
perl awstats.pl -config=mysite -output=keyphrases -staticlinks > awstats.mysite.keyphrases.html
perl awstats.pl -config=mysite -output=keywords -staticlinks > awstats.mysite.keywords.html
perl awstats.pl -config=mysite -output=errors404 -staticlinks > awstats.mysite.errors404.html

Queries in Web form

When you access AWstats in CGI mode you can enter queries against each section of the report, for example Navigation/Viewed/full list which contain important statistics and number of visits for each page. You can use Perl regular expression in such queries. For example

Lang|Scripting

NEWS CONTENTS

• 20180129 : Installing Awstat for analyzing Apache logs - LinuxTechLab ( Jan 29, 2018 , linuxtechlab.com )
• 20180129 : How To Install, Secure, And Automate AWStats (CentOS/RHEL) by rasha ( howtoforge.com )
• 20180129 : A good and free log analysis tool - awstats ( petefreitag.com )
• 20180129 : AWStats - Free log file analyzer for advanced statistics (GNU GPL) ( AWStats - Free log file analyzer for advanced statistics (GNU GPL), )

Old News ;-)

[Jan 29, 2018] Installing Awstat for analyzing Apache logs - LinuxTechLab

Jan 29, 2018 | linuxtechlab.com

Once the epel-repository has been enabled on the system, awstat can be installed by running,

$ yum install awstat

When awstat is installed, it creates a file for apache at '/etc/httpd/conf.d/awstat.conf' with some configurations. These configurations are good to be used incase web server &awstat are configured on the same machine but if awstat is on different machine than the webserver, then some changes are to be made to the file.

Configuring Apache for Awstat

To configure awstat for a remote web server, open /etc/httpd/conf.d/awstat.conf, & update the parameter 'Allow from' with the IP address of the web server

$ vi /etc/httpd/conf.d/awstat.conf <Directory "/usr/share/awstats/wwwroot"> Options None AllowOverride None <IfModulemod_authz_core.c> # Apache 2.4 Require local </IfModule> <IfModule !mod_authz_core.c> # Apache 2.2 Order allow,deny Allow from 127.0.0.1 Allow from 192.168.1.100 </IfModule> </Directory>

Save the file & restart the apache services to implement the changes,

$ systemctl restart httpd

Configuring AWSTAT

For every website that we add to awstat, a different configuration file needs to be created with the website information . An example file is created in folder '/etc/awstats' by the name 'awstats.localhost.localdomain.conf', we can make copies of it & configure our website with this,

$ cd /etc/awstats $ cp awstats.localhost.localdomain.conf awstats.linuxtechlab.com.conf

Now open the file & edit the following three parameters to match your website,

$ vi awstats.linuxtechlab.com.conf LogFile="/var/log/httpd/access.log" SiteDomain="linuxtechlab.com" HostAliases=www.linuxtechlab.com localhost 127.0.0.1

Last step is to update the configuration file, which can be done executing the command below,

/usr/share/awstats/wwwroot/cgi-bin/awstats.pl -config=linuxtechlab.com–update

Checking the awstat page

To test/check the awstat page, open web-browser & enter the following URL in the address bar,
https://linuxtechlab.com/awstats/awstats.pl?config=linuxtechlab.com

... ... ...

Note- we can also schedule a cron job to update the awstat on regular basis. An example for the crontab

$ crontab –e 0 1 * * * /usr/share/awstats/wwwroot/cgi-bin/awstats.pl -config=linuxtechlab.com–update

How To Install, Secure, And Automate AWStats (CentOS/RHEL) by rasha

howtoforge.com

AWStats is a free and very powerful tool that generates advanced web, streaming, ftp or mail server statistics, graphically. It can analyze log files from all major server tools and convert them into nice graphical display. There are a lot of articles out there for AWStats, the reason I decided to write this one was to consolidate all the different tips and tricks I've learned through my journey into one comprehensive article associating the fantastic efforts of so many out there. There are a few methods of installing AWStats, this article will describe the simpler method of the two utilizing YUM.

This article assumes that you have root/sudo access to achieve all the goals required for a functional setup. This article also assumes you have SELinux set to permissive or disabled and that IPTables configuration is either disabled or customized for AWStats. Let's begin.

Install

Add RPMForge Yum Repository:

For x86 (32-bit) systems:

rpm -Uhv http://apt.sw.be/redhat/el5/en/i386/rpmforge/RPMS/rpmforge-release-0.3.6-1.el5.rf.i386.rpm

from Dag Wieers.

For x64 (64-bit) systems:

rpm -Uhv http://apt.sw.be/redhat/el5/en/x86_64/rpmforge/RPMS//rpmforge-release-0.3.6-1.el5.rf.x86_64.rpm

from Dag Wieers.

Install & Configure Prerequisites

Install Apache:

yum install httpd

Configure Apache to start on boot:

/sbin/chkconfig --levels 345 httpd on

Configure iptables to allow Apache traffic:

/sbin/iptables -I INPUT -p tcp --dport 80 -j ACCEPT 
/etc/init.d/iptables save >
/etc/init.d/iptables restart

Install & Configure AWStats

Now that YUM has its additional repository we are ready to install. From the commandline type:

yum install awstats

Modify AWStats Apache Configuration:

Edit /etc/httpd/conf.d/awstats.conf

(Note: When putting your conf file in the /etc/httpd/conf.d/ folder it's automatically loaded as part of the Apache configuration. There is no need to add it again into httpd.conf. This setup is usually for one of two reasons; A cleaner approach and separating of different applications in their own configuration files, or you are in a hosted environment that does not allow for direct editing of httpd.conf):

Alias /awstats/icon/ /var/www/awstats/icon/

ScriptAlias /awstats/ /var/www/awstats/
<Directory /var/www/awstats/>
        DirectoryIndex awstats.pl
        Options ExecCGI
        order deny,allow
        allow from all
</Directory>

Alias /awstatsclasses "/var/www/awstats/lib/"
Alias /awstats-icon/ "/var/www/awstats/icon/"
Alias /awstatscss "/var/www/awstats/examples/css"

Note: the mod_cgi module of Apache must be pre-loaded into Apache otherwise Apache will not try to view the file, it will try to execute it. This can be done in two ways, either enable for the entire web server, or utilizing VirtualHosts, enable for AWStats.

Edit the following lines in the default awstats configuration file /etc/awstats/awstats.localhost.localdomain.conf:

SiteDomain="<server name>.<domain>"
HostAliases="<any aliases for the server>"

Rename config file:

mv /etc/awstats/awstats.localhost.localdomain.conf /etc/awstats/awstats.<server 
	name>.<domain>.conf

Update Statistics (Note: By default, statistics will be updated every hour.):

/usr/bin/awstats_updateall.pl now -confdir="/etc" -awstatsprog="/var/www/awstats/awstats.pl"

Start Apache:

/etc/init.d/httpd start

To automate startup of Apache on boot up, type

chkconfig --add httpd

Verify Install

Go to

http://<server name>.<domain>/awstats/awstats.pl?config=<server name>.<domain>

Setting File System Permissions

The webserver needs only read-access to your files in order for you to be able to access AWStats from the browser. Limiting your own permissions will keep you from accidentally messing with files. Just remember that with this setup you will have to run perl to execute scripts rather than executing the scripts themselves.

$ find ./awstats -type d -exec chmod 701 '{}' \; 
$ find ./awstats -not -type d -exec chmod 404 '{}' \;

Apache doesn't need direct access to AWStats configuration files therefore we can secure them tightly and not affect the relationship between them. To ensure that your .htaccess files are not readable via browser:

chmod 400 /etc/awstats/*.conf

Protecting The AWStats Directory With And Adding .htaccess

To secure the Awstats folder(s), is a measured process. Ensuring ownership of the awstats folder is owned by the user that needs access to it, creating an htpasswd.users file and adding the corresponding .htaccess file to authenticate against it. Let's first secure the awstats folder by typing the below from the command-line:

find ./awstats -type d -exec chmod 701 '{}' \; 
find ./awstats -not -type d -exec chmod 404 '{}' \;

Now that our folders have been secured, we'll need to create the .htpasswd.users file. Go to the /etc/awstats folder and execute the following command:

htpasswd -c /etc/awstats/htpasswd.users user 

(Select whatever username you'd like.)

It'll ask you to add a password for the user you've selected, add it and re-type it for confirmation and then save. The final step is to create an .htaccess file pointing to the .htpasswd file for authentication. Go to /var/www/awstats/ and create a new file called .htaccess using your favorite editor, typically nano or vi tend to be the more popular ones. In this example we'll use vi. From the command line type

vi .htaccess

An alternate method of creating an .htaccess file is using the Htaccess Password Generator. Add the following content to your newly created .htaccess file:

AuthName "STOP - Do not continue unless you are authorized to view this site! - Server Access"
AuthType Basic
AuthUserFile /etc/awstats/htpasswd.users
Require valid-user
htpasswd -c /etc/awstat/htpasswd.users awstats_online

Once done, secure the .htaccess file by typing:

chmod 404 awstats/.htaccess

How To Install, Secure, And Automate AWStats (CentOS/RHEL) - Page 2

A good and free log analysis tool - awstats

petefreitag.com

There are a lot of free http log file analysis tools out there that haven't been updated since the mid 90's, awstats however is both free, and up to date. It looks a bit like web trends (though I haven't used web trends in several years). Here's an online demo. awstats can be used on several web servers including IIS, and Apache. You can either have generate static html files, or run with a perl script in the cgi-bin.

Here's a quick rundown of setting it up on unix/apache

Each virtual web site you want to track stats for should have a file /etc/awstats.sitename.conf the directives for the configuration file can be found here: http://awstats.sourceforge.net/docs/awstats_config.html they also provide a default conf file in cgi-bin/awstats.model.conf you can use this as a base.

Make sure your log files are using NCSA combined format, this is usually done in apache by saying CustomLog /logs/access.log combined you can use other formats but you have to customize the conf file.

You will probably want to edit the LogFile directive to point to where your logfile is stored, SiteDomain this is the main domain for the site, HostAliases lets you put in other domains for the site, and the DirData directive lets you specify where the awstats databases will be stored (each site will have its own file in the directory).

Once that is setup you will want to update the database this is done from the command line by running

perl awstats.pl -config=sitename -update

Now copy everything in the wwwroot folder to a web root, and visit http://sitename.com/cgi-bin/awstats.pl if you want to view other domains use /cgi-bin/awstats.pl?config=othersitename

Where sitename would be the name of your config file awstats.sitename.conf

If you want to generate static html files run the awstats_buildstaticpages.pl script found in the tools folder. You have to give it the path to the awstats.pl perl script, and a directory to put the static html files in.

perl awstats_buildstaticpages.pl -config=sitename -awstatsprog=/web/cgi-bin/awstats.pl -dir=/web/stats/sitename/ 

More setup info can be found here: http://awstats.sourceforge.net/docs/index.html

AWStats - Free log file analyzer for advanced statistics (GNU GPL)

awstats.sourceforge.net

AWStats is a free powerful and featureful tool that generates advanced web, ftp or mail server statistics, graphically. This log analyzer works as a CGI or from command line and shows you all possible information your log contains, in few graphical web pages. It uses a partial information file to be able to process large log files, often and quickly. It can analyze log files from IIS (W3C log format), Apache log files (NCSA combined/XLF/ELF log format or common/CLF log format), WebStar and most of all web, proxy, wap, streaming servers, mail servers (and some ftp).

Take a look at this comparison table for an idea on differences between most famous statistics tools (AWStats, Analog, Webalizer,...).

AWStats is a free software distributed under the GNU General Public License. You can have a look at this license chart to know what you can/can't do.

As AWStats works from the command line but also as a CGI, it can work with major web hosting provider that allows CGI and log access.

You can browse AWStats demo (Real-time feature to update stats from web has been disabled on demos) to see a sample of most important information AWStats shows you...

Recommended Links

Softpanorama Recommended

Top articles

Sites

• awstats.sourceforge.net/ -- official page
• AWStats Documentation - Setup page
• How To Install, Secure, And Automate AWStats (CentOS/RHEL) - Page 2
• AWStats - Wikipedia, the free encyclopedia
• AWStats Documentation - Configuration directives and parameters
• AWStats Installation - DreamHost

Etc

Society

Groupthink : Two Party System as Polyarchy : Corruption of Regulators : Bureaucracies : Understanding Micromanagers and Control Freaks : Toxic Managers :   Harvard Mafia : Diplomatic Communication : Surviving a Bad Performance Review : Insufficient Retirement Funds as Immanent Problem of Neoliberal Regime : PseudoScience : Who Rules America : Neoliberalism  : The Iron Law of Oligarchy : Libertarian Philosophy

Quotes

War and Peace : Skeptical Finance : John Kenneth Galbraith :Talleyrand : Oscar Wilde : Otto Von Bismarck : Keynes : George Carlin : Skeptics : Propaganda  : SE quotes : Language Design and Programming Quotes : Random IT-related quotes :  Somerset Maugham : Marcus Aurelius : Kurt Vonnegut : Eric Hoffer : Winston Churchill : Napoleon Bonaparte : Ambrose Bierce :  Bernard Shaw : Mark Twain Quotes

Bulletin:

Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks The efficient markets hypothesis : Political Skeptic Bulletin, 2013 : Unemployment Bulletin, 2010 :  Vol 23, No.10 (October, 2011) An observation about corporate security departments : Slightly Skeptical Euromaydan Chronicles, June 2014 : Greenspan legacy bulletin, 2008 : Vol 25, No.10 (October, 2013) Cryptolocker Trojan (Win32/Crilock.A) : Vol 25, No.08 (August, 2013) Cloud providers as intelligence collection hubs : Financial Humor Bulletin, 2010 : Inequality Bulletin, 2009 : Financial Humor Bulletin, 2008 : Copyleft Problems Bulletin, 2004 : Financial Humor Bulletin, 2011 : Energy Bulletin, 2010 : Malware Protection Bulletin, 2010 : Vol 26, No.1 (January, 2013) Object-Oriented Cult : Political Skeptic Bulletin, 2011 : Vol 23, No.11 (November, 2011) Softpanorama classification of sysadmin horror stories : Vol 25, No.05 (May, 2013) Corporate bullshit as a communication method  : Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law

History:

Fifty glorious years (1950-2000): the triumph of the US computer engineering : Donald Knuth : TAoCP and its Influence of Computer Science : Richard Stallman : Linus Torvalds  : Larry Wall  : John K. Ousterhout : CTSS : Multix OS Unix History : Unix shell history : VI editor : History of pipes concept : Solaris : MS DOS :  Programming Languages History : PL/1 : Simula 67 : C : History of GCC development :  Scripting Languages : Perl history   : OS History : Mail : DNS : SSH : CPU Instruction Sets : SPARC systems 1987-2006 : Norton Commander : Norton Utilities : Norton Ghost : Frontpage history : Malware Defense History : GNU Screen : OSS early history

Classic books:

The Peter Principle : Parkinson Law : 1984 : The Mythical Man-Month :  How to Solve It by George Polya : The Art of Computer Programming : The Elements of Programming Style : The Unix Hater’s Handbook : The Jargon file : The True Believer : Programming Pearls : The Good Soldier Svejk : The Power Elite

Most popular humor pages:

Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : IT Slang : C++ Humor : ARE YOU A BBS ADDICT? : The Perl Purity Test : Object oriented programmers of all nations : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor : C Humor : Scripting Humor : Real Programmers Humor : Web Humor : GPL-related Humor : OFM Humor : Politically Incorrect Humor : IDS Humor : "Linux Sucks" Humor : Russian Musical Humor : Best Russian Programmer Humor : Microsoft plans to buy Catholic Church : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor : Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor : Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor

The Last but not Least Technology is dominated by two types of people: those who understand what they do not manage and those who manage what they do not understand ~Archibald Putt. Ph.D

The statements, views and opinions presented on this web page are those of the author (or referenced source) and are not endorsed by, nor do they necessarily reflect, the opinions of the Softpanorama society. We do not warrant the correctness of the information provided or its fitness for any purpose. The site uses AdSense so you need to be aware of Google privacy policy. You you do not want to be tracked by Google please disable Javascript for this site. This site is perfectly usable without Javascript.

Last modified: January, 11, 2020