AWStats - Open source log file analyzer for advanced statistics

AWStats

Global setup / initial conditions :

  • a Debian host running the Lighttpd web server
  • several sites are served via distinct virtualhosts
  • all sites (virtualhosts, actually) log their hits and errors in shared logfiles (splitting those will be our first task )
The goal is to analyze the logs of several of these virtualhosts with AWStats. This will imply creating an extra virtualhost for AWStats itself.
A single AWStats instance is enough for several analyzed websites : we'll just have to generate a distinct AWStats configuration for each virtualhost to be analyzed. Which website data is displayed in AWStats web interface is specified as a URL parameter : http://awstats.myDomain.tld/cgi-bin/awstats.pl?config=www.example.com

Prerequisite : Configure your webserver to split logs per virtualhost

AWStats (source) :

Install :
My first idea was to :
apt install awstats
But it turned out that some files / directories mentioned in the docs (Perl scripts, wwwroot, classes, ...) were missing (). So I tried a different solution.
  1. cd /var/www/awstats/ && wget https://prdownloads.sourceforge.net/awstats/awstats-7.7.tar.gz
  2. tar zxf awstats-7.7.tar.gz
  3. chown -R www-data. *
Configuration :
  1. cd /var/www/awstats/tools && ./awstats_configure.pl
    ... and answer questions...
  2. You may see :
    Sorry, configure.pl does not support automatic add to cron yet.
    You can do it manually by adding the following command to your cron:
    /var/www/awstats/wwwroot/cgi-bin/awstats.pl -update -config=www.example.com
    Or if you have several config files and prefer having only one command:
    /var/www/awstats/tools/awstats_updateall.pl now
    We'll see this later
  3. Then :
    You can then manually update your statistics for 'www.example.com' with command:
    > perl awstats.pl -update -config=www.example.com
    You can also read your statistics for 'www.example.com' with URL:
    > http://localhost/awstats/awstats.pl?config=www.example.com
  4. At this step :
    • AWStats has analyzed /var/log/yourWebserver/www.example.com.log and created + populated its internal database
    • As new hits arrive continuously, this database has to be updated periodically with the provided command
    • After updating your webserver (the next step), you'll also be able to have a look at the web interface at the given URL
  5. Have a look at the generated configuration file for fine tuning : /etc/awstats/awstats.www.example.com.conf

Configure Lighttpd :

  1. Modules :
    server.modules = (
    	...,
    	"mod_cgi",
    	...
    	)
  2. Virtualhost :
    $HTTP["host"] =~ "^awstats" + domainNameRegExp {
    	server.document-root	= "/var/www/awstats/wwwroot"
    	accesslog.filename	= logRoot + "awstats.access.log"
    	cgi.assign = (
    		".pl"   => "/usr/bin/perl",
    		".cgi"  => "/usr/bin/perl"
    		)
    	alias.url  = ( "/awstatsclasses"    => "/var/www/awstats/wwwroot/classes/" )
    	alias.url += ( "/awstatscss"        => "/var/www/awstats/wwwroot/css/" )
    	alias.url += ( "/awstatsicons"      => "/var/www/awstats/wwwroot/icon/" )
    	}
  3. Disable cache :
    $HTTP["host"] =~ "^awstats" + domainNameRegExp {
    	setenv.add-response-header = ( "Cache-Control" => "no-cache" )
    	}
  4. Check Lighttpd configuration
  5. Restart Lighttpd :
    systemctl restart lighttpd.service
  6. No need to let robots in : put in /var/www/awstats/wwwroot/robots.txt
    User-agent: *
    Disallow: /
  7. If you use a reverse web proxy / load balancer, update its settings accordingly (cache, ...). If you've been too impatient and already had a look at the AWStats web interface, consider purging its cache to see updates next time you'll press F5.

Periodically update the database with crontab :

*/15 * * * * perl /var/www/awstats/wwwroot/cgi-bin/awstats.pl -update -config=www.example.com

And finally : logrotate (source) :

  1. Create the configuration file :
    cat << EOF > /etc/logrotate.d/www.example.com
    /var/log/varnish/www.example.com.log
    	{
    	missingok
    	notifempty
    	daily
    	rotate 30
    	compress
    	sharedscripts
    
    	prerotate
    		/var/www/awstats/wwwroot/cgi-bin/awstats.pl -update -config=www.example.com
    	endscript
    
    	create
    
    	postrotate
    		kill -HUP \$(cat /run/varnishncsa/varnishncsa_www.example.com.pid)
    	endscript
    	}
    EOF
  2. check it :
    /usr/sbin/logrotate -f /etc/logrotate.d/www.example.com; echo $?