                      _         _     _
                     |_) _ |_  /  |_   \ |   _  _
                     | \(_)(_, \_ (_, _/ (_,(_)(_)
                     By Stefano Falsetto       ,_)

This is Rot[t]Log

archive, rotates, compresses, and mails system logs.

This is a replacement to Red Hat's logrotate. It have similar
syntax, but more powerful features to cut and store logs. 
This README is heavily bases on Red Hat's logrotate man page.
It's all written in BASH (2.x compatible).

This script require GNU fileutils, GNU grep, GNU awk , GNU shell-utils and 
util-linux.

USE WITH CARE
At this moment this software is still in alpha stage, so... handle with care 
your logs...

Main configuration file	  : /etc/rottlog/rc
Other configuration files : /etc/rottlog/[daily|weekly|monthly|custom]

rottlog first read main configuration file, then read (in order)
daily, weekly and monthly configuration files.

  ::::: RC FILE SYNTAX :::::

  This is the main configuration file. It defines default behavoir of
  Rot[t]Log. 
  Following keywords are valid:
  
  compress=<options>
  createdir
  default_storefile=<storefile>
  dir_grp=<group>
  dir_own=<owner>
  dir_perm=<permissions>
  extension=<ext>
  follow_symlinks
  fromuser=<sender address>
  ifempty
  mail=<mailer prg>
  maxdepth=<depth>
  missingok
  nocompress
  nocreate
  nomail
  nomissingok
  notifempty
  packdir=<path-to-archive-dir>
  packer=<absolute path> 
  pager=<pager-program>
  remove_missing
  SunMon=<sun|mon>
  touser=<receiver address>
  uncompress=<options>
  unpacker=<complete path>

  compress=<options>
  options to be used with packer program to compress data
  IMPORTANT: Packer program must output data to stdout.
  (i.e. for gzip/bzip2 use compress="bzip2 -c")
  
  createdir
  Create dirs specified in storedir if needed. It's useful to use
  with meta-vars. See section USE OF META-VARIABLES for more info.

  default_storefile=<storefile>
  Default filename to use with archived/rotated logfiles. It's useful 
  to use with meta-vars. See section USE OF META-VARIABLES for more info.
  WARNING: 
    >>> Time-related meta-vars are NOT compatibles with rotate parameter
  
  dir_grp=<group>
  Default group of newly created directory. Must exists
  in /etc/group file.
  
  dir_own=<owner>
  Default owner of newly created directory. Must exists 
  in /etc/passwd file.
  
  dir_perm=<permissions>
  Default permissions of newly created directory. Only numerical
  notation is permitted.
  
  fil_grp=<group>
  Default group of newly created logfile. Must exists
  in /etc/group file.
  
  fil_own=<owner>
  Default owner of newly created logfile. Must exists 
  in /etc/passwd file.
  
  fil_perm=<permissions>
  Default permissions of newly created logfile. Only numerical
  notation is permitted.

  extension=<ext>
  Extension normally used when compressing data with packer.
  (i.e. for gzip is gz, for bzip2 is bz2, ...)
  This variable is used to expand meta-variable @COMP_EXT.

  follow_symlinks
  Follow symlinks to logfiles. Default behavoir is to not to follow
  symlinks for security reasons.
  
  fromuser=<sender address>
  Sender address used in mail messages.

  ifempty
  Rotate the log file even if it is empty.
  
  mail=<mailer prg>
  Mail program to send messages. Must be used option to read header
  recipient addresses.  
  Example:
   - for sendmail use: mail="sendmail -t"
   - for qmail use: mail="qmail-inject -h"

  maxdepth=<depth>
  Default maximum depth to search for files when used * metachar in log
  filename definition.
  
  missingok
  If the log file is missing, go on to the next one without issuing an
  error message. Lastaction (if defined) will be performed.  Postrotate
  script (if defined) will NOT be performed.

  nocompress
  Don't compress logfiles to be archived/rotated.
  
  nocreate
  Don't create a new filename. Can't be used with logpart option.
  
  nomail
  Don't mail rottlog's activity report.
  
  nomissingok
  rottlog will return an error message and stop execution 
  if the log file is missing.

  notifempty
  Do not rotate a logfile if it's empty.

  packdir=<path-to-archive-dir>
  default basedir to store rotated/archived files.
  This variable is used to expand @DEF_PATH meta-variable.

  packer=<absolute path> 
  complete path to packer program to be used to compress logs.

  pager=<pager-program>
  Path to program to use with --showlog parameter. It's used in a pipe,
  so must be a program able to read input from stdin.
  
  remove_missing
  rottlog will stop execution if a delayed file is no longer existant
  or readable.

  SunMon=<sun|mon>
  Use sunday or monday as first day of week.
  This option is useful only in conjunction with @WEEK meta-variable.

  touser=<receiver address>
  Receiver of mail messages.
 
  uncompress=<options>
  options to be used with unpacker program to uncompress data
  IMPORTANT: Unpacker program must output data to stdout.
  (i.e. for gunzip/bunzip2 use -c)

  unpacker=<complete path>
  complete path to program to be used to uncompress logs.
  (i.e. gunzip, bunzip2...)


  ::::: DAILY|WEEKLY|MONTHLY|CUSTOM FILE SYNTAX :::::

  Period config files are checked once per month, week and day, 
  and all logfiles are handled at the same time. Custom config
  file is read every time rottlog is called.
  Example: 
   - rottlog is run on 1st of a month 
     1. Logfiles in 'monthly' will all be handled on 1st of 
        each following month
     2. Logfiles in 'weekly' will all be handled on 1st, 7th, 14th,
        21th, 28th day and so on
     3. Logfiles in 'daily' will all be handled each day in the 
        month
     4. Logfiles in 'custom' will all be handled every time
        rottlog is called.

  So if a logfile must be rotated/archived only one day of each
  month it must be inserted in *custom* config file and must be 
  used "period" parameter.

  Global structure of a period-config file is:

  out-of-block-definitions

  configuration-block-1-start {
    block-contents
  }

  configuration-block-2-start {
    block-contents
  }

  ....

  Comments can be inserted in rows beginning with '#'.
  Comments on same row of valid items are not allowed.
  Example:

  # This is a valid comment
  logpart "#1 day" # This is an INVALID comment

  
  There is only one out-of-block-definition (at the moment):

  include <path-to-conffiles>[,<path-to-conffiles>,...]

  This option permits to include other files at the end of configuration
  file. Nested includes are not supported: only one level of inclusion
  is supported. 
  This keyword must be inserted at beginning of configuration file before
  and out of all configuration blocks. If used in other positions will
  be treated as error.
  <path-to-conffiles> must be an absolute path to a file or a group of
  files (using * wildcard). To include all files in a directory must be
  used path-to-dir/* (i.e. /etc/rottlog/conf-monthly/*)

  A configuration block for logfiles has following structure:

  <logfile1>,<logfile2>,... {
    <append-only>
    <collate> <logfile>
    <create> <mode> <owner> <group>
    <createdir> [<perms>] [<own> <grp>]
    <delaycompress>
    <dateoffset>
    <firstaction> ... <endaction>
    <ifempty>
    <lastaction> ... <endaction>
    <logpart> <qdldl>
    <maxdepth> <depth>
    <missingok>
    <nocompress>
    <nocreate>
    <nomail>
    <nostoredir>
    <notifempty>
    <period> <period-definitions>
    <postrotate> ... <endscript>
    <prerotate> ... <endscript>
    <rotate> <number>
    <sharedscripts>
    <size> <num[b,k,M]>
    <storedir> <path-to-dir>
    <storefile> <filename>
    <tarcollate> <logfile>
    <touser> <email address>
  }

  Where:
  
  logfile1, logfile2, ... 
  Is a comma separated list of complete absolute path to logfiles to
  archive/rotate. From 0.10alpha+ version it's possible to use ONLY
  metachar asterisk. (i.e. /var/adm/log.* means that definitions between
  curly brackets are applied to all files in /var/adm dir beginning by
  'log.').  If logfile is a filename containing special chars like
  ']' or '{' you must protect it from expansion with DOUBLE slash
  (i.e. apache\\[1\\].log). If same logfile is used more than one time
  in same configuration file, will be used only the first.

  append-only
  To be used only on ext[23] fs. 
  Set append-only attribute to a log (More precisely: remove this
  attribute before handle log, and reset it after performed all
  operations).

  collate
  Store more than one logfile in only one file compressed.
  This option makes possible to weekly rotate a logfile (i.e. in a little
  volume) and monthly store a compressed file containing previously rotated
  files (i.e. in a bigger backup volume). 
  If used in monthly configuration file, <logfile> will be searched in weekly
  configuration file.
  If used in weekly configuration file, <logfile> will be searched in daily
  configuration file.
  Can't be used in daily and custom configuration files.
  Logfiles referenced in collate option must use rotate option with correct
  periods:
  - collate monthly a weekly rotated file : rotate 4
  - collate weekly a daily rotated file   : rotate 7
  Example:
  Collect monthly a weekly rotated log.daemon
 
  In weekly configuration file:
  /var/adm/log.daemon {
    storedir /mnt/tiny/@YEAR/@BASENAME
    createdir 4700 stefano root
    create 600 stefano root 
    prerotate
        /sbin/killall -HUP syslogd
    endscript
    # Important!
    rotate 4
    delaycompress
  }

  In monthly configuration file:
  /var/adm/log.daemon {
    storedir /mnt/BigVolume/Collections/@YEAR/@BASENAME
    createdir
    collate
  }

  In this example in /mnt/BigVolume/Collections/2002/log.daemon will
  be stored a file called log.daemon.<number> containing log.daemon.1,
  log.daemon.2, log.daemon.3 and log.daemon.4 weekly created in
  /mnt/tiny/2002/log.daemon dir.

  This option is compatible only with following options:
    - [no]storedir
    - storefile
    - createdir
    - [no]missingok
    - firstaction/lastaction
    - prerotate/postrotate
    - nocompress
    - touser
    - nomail
  Options ifempty/notifempty are silently ignored.

  create [[<mode>] [<owner> <group>]]
  Immediately  after  rotation (before the postrotate script is run)
  the log file is  created  (with  the same  name  as  the  log  file
  just rotated).  mode specifies the mode for the log file in  octal (the
  same  as  chmod(2)),  owner specifies the user name who will own the
  log file, and group specifies  the group  the  log file will belong to.
  If none of three parameters is specified new file will be created
  with permissions and owner defined in default variables fil_perm,
  fil_own, and fil_grp. You can specify mode only, or mode and owner,
  but you can't to specify mode and group (second parameter is always
  interpreted as owner).

  createdir [[<mode>] [<owner> <group>]] 
  Create (if needed) dir to store archived/rotated logs with specified
  permissions and owner:group.  Directory will be created with permissions
  and owner defined in default variables dir_perm, dir_own, and dir_grp if
  none of three parameters is used. You can specify mode only, or mode
  and owner, but you can't to specify owner and group (second parameter
  is always interpreted as owner).  perms specifies the mode for the
  log file in octal (the same as chmod(2)).

  IMPORTANT:
  If path to be created is deeper than one dir, only last dir will have
  owner, group and mode specified. 
  Example:
  
      storedir 2002/04/apache_logs
      createdir 4300 rottlog

  If dir 2002 (and following) are not already present, on your HD after
  rottlog run you will have (rottlog is run by user stefano):
  $ ls -ld 2002 2002/04 2002/04/apache_logs
  drwxr-xr-x   2 stefano  users        1024 Apr 11 18:35 2002/
  drwxr-xr-x   2 stefano  users        1024 Apr 11 18:35 2002/04
  d-ws------   2 rottlog  users        1024 Apr 11 18:35 2002/04/apache_logs

  delaycompress
  Postpone compression of the previous log file to the next rotation
  cycle.
  It can be used when some program can not be told to close its
  logfile and thus might continue writing to the previous log file for
  some time.

  dateoffset <+/-day>
  Force time related metavariables to be calculed with specified days of offset.
  Example:
  if today is Aug 5 and is used "dateoffset -1", @DAY metavariable will be 04 
  and not 05.

  firstaction <EXIT_CODE>... endaction
  Unique line between firstaction and endaction (both of which must
  appear on lines by themselves) are executed before ANY operation. It
  will be performed during parameter parsing, before filename and metavars
  expansion.  Optional parameter EXIT_CODE is default exit code expected
  by action. If not specified will expect 0.
  If used in multiple files definition action will be run only one 
  time, before processing first file.

  ifempty
  Rotate the log file even if it is empty.
  
  lastaction <EXIT_CODE>... endaction
  Unique line between lastaction and endaction (both of which must
  appear on lines by themselves) are executed after ALL operations. It
  will be performed just before handle next logfile.  Optional parameter
  EXIT_CODE is default exit code expected by action.  If not specified
  will expect 0.
  If used in multiple files definition action will be run only one
  time, after processed all files.

  logpart <qdldl>
  Leave a part of log to archive/rotate in new logfile.
  qdldl can be:
  <number>l
  <number>b
  "<regexp>"
  "<#1 day>"
  Where l and b means lines and bytes respectively.
  "regexp" is a regular expression used by grep to find first row to
  leave in log.
  "#1 day" is a shortcut to leave in log all events recorded from first
  day of current month.

  maxdepth <depth>
  Default maximum depth to search for files when used * metachar in log
  filename definition.

  missingok
  If the log file is missing, go on to the next one without issuing an
  error message. Default behavoir is to stop executing and return an
  error exit code.

  nocompress
  Old versions of log files are not compressed with packer defined in
  rc file.

  nocreate
  Don't create a new filename. Can't be used with logpart.

  nomail
  Don't send notification e-mail message.

  nostoredir
  Logs are rotated/archived in the same directory the log normally
  resides in.

  notifempty
  Do not rotate the log if it is empty.

  period <period-definitions>
  This is a very powerful option usable only in custom config file.
  It force a rotation only in a specified period or specific day of 
  month.
  <period-definitions> is a comma-separated list of items defining
  a day, or a set of days, a month and so on. Usable items can be:
  D              -> Each Dth day of the month
  DDd            -> Each DD days
  WWw            -> Each WW weeks
  mmM            -> Each mm months
  <weekday>      -> Each <weekday> (abbreviated or not) of the week
  <monthname>    -> Each <monthname> (abbreviated or not) of the year
  HH:MM          -> At specified hour and minutes (*)
  0              -> Each time rottlog is called (*)

  (*) It depends on rottlog's entries in crontab

  In definition of period parameters following characters have particular 
  signification:
  
  , (comma)		-> OR
  " " (white space)	-> AND
  ! (exclamation mark)	-> NOT
  - (minus)		-> Week or month range

  Minus carachter can be used to define a list of days or months. 
  Comma separed tokens are analized in given order. When one of it
  defines a true condition, the check is stopped without continue to 
  parse following conditions.
  WARNING: Correct behavoir of rottlog and period parameter strictly
  depends from correct configuration of rottlog's crontab.

  Some examples:

  1) period mon-fri 21:00,sat 10:00

  Is equivalent to:
     period mon 21:00,tue 21:00,wed 21:00,thu 21:00,fri 21:00,sat 10:00

  And means that rotation must be performed if rottlog is run and:
  today is monday AND it is 21:00 hour OR today is tuesday AND at it is 
  21:00 hour, and so on just to friday, OR today is saturday and at is
  10:00 hour.

  2) period jan-jun 23:00,jul-sep 10:00,oct-dec 23:00

  Is equivalent to:
     period !jul-sep 23:00,jul-sep 10:00
  
  And means that rotation must be performet if rottlog is run and:
  today is NOT in a month between (including) july and september AND it is 23:00
  hour, OR it is 10:00. 

  postrotate ... endscript
  Lines between postrotate and endscript (both of which must
  appear on lines by themselves) are executed after the log file is
  archived/rotated.  Can be used metavariables. See section USE OF
  META-VARIABLES for more info.

  prerotate ... endscript
  Lines between prerotate and endscript (both of which must appear
  on lines by themselves) are executed before the log file is
  archived/rotated.  Can be used metavariables. See section USE OF
  META-VARIABLES for more info.
  
  rotate <number>
  Log files are rotated <number> times before being overwrited.

  sharedscripts
  Let postrotate script to be run only one time, after handled last file in 
  a multiple file definition block.
  Example:
  /var/adm/log.* {
    sharedscripts
    postrotate
      killall -HUP syslogd
    endscript
  }
  The killall command will be run only after handled all log files beginning 
  by "log" in /var/adm dir.

  size <num[b,k,M]>
  Rotate a logfile only if it grows specified dimension.
  Parameter must be a number followed by one of following chars:
  b - means bytes
  k - means kbytes (num * 1024)
  M - means megabytes (num * 1000000)

  If none of these three chars is used, b is default.

  storedir <path-to-dir>
  If it's a relative path it will be appended to packdir defined in rc
  file, if it's an absolute path it will be used instead of packdir. Can
  be used metavariables. See section USE OF META-VARIABLES for more info.

  storefile <filename>
  Name of compressed logfile to be used (whitout any path: filename
  only) Can be used metavariables. See section USE OF META-VARIABLES
  for more info.

  tarcollate
  Same of collect, but store files in a compressed tar file. In addition this
  option is compatible with ifempty option too. If set, a void file will be
  archived in tar file. If not set it will be skipped.
  Tar archive will contain relative path if possible (if archived files are in
  subdirs of packdir defined in main rc file) or absolute path.
  
  touser <email address>
  Override global value of touser variable defined in rc file.

  ::::: USE OF META-VARIABLES :::::

  In monthly/weekly/daily logs can be used metavariables.
  Metavariables are identifiers of a runtime defined value.
  Handled metavariables:

  @DIRNAME  : Dir containing logfile to be archived/rotated
  @FILENAME : filename with numerical extension (BASENAME+NEXT_EXT)
  @BASENAME : filename without extension
  @DEF_PATH : default path to archive dir
  @COMP_EXT : extension of compressed format file (bz2, gz, ...)
  @NEXT_EXT : next available numbered extension 
  @YEAR     : actual year(*)
  @DAY      : actual day(*)
  @MONTH    : actual month(*)
  @WEEK     : number of week in year (0-53) accordingly to SunMon variable (*)
  @TEMPDIR  : a temporary directory 
  @1..@n    : string corresponding to first, second and so on directory
              in path of file to archive/rotate

 (*) Can't be used in storefile if using rotate parameter too

  Parameters handling metavariables:
  storedir
  storefile
  prerotate...endscript
  postrotate...endscript

  Values of metavariables are defined while script is running, so a definition
  like this:

  /var/adm/log.* {
    storedir /var/adm/archive-log/@BASENAME
    rotate 5
    ...
  }

  Means that I wants to rotate 5 times all files in /var/adm beginning by 
  "log." and store each file in a dir named with logname.
  (i.e. /var/adm/archive-log/log.kern, /var/adm/archive-log/log.daemon, ...)

  WARNING:
  Metavariables substitution is merely textual, so in a definition like this:

  /var/http/www.mysite.com/log/access.log {
    storefile @3-@13-@FILENAME
    rotate 4
    ....
  }
  Will create logfiles called somethink like:
  www.mysite.com-var3-access.log.1
  
  @13 is not defined (path in first line of block definition contains only 5 tokens)
  so @1 will be substituted and following text will not be affected.

  ::::: COMMAND LINE PARAMETERS :::::

  Valid command line parameters allowed are:
  
    --help, -h
    Display help page and exit. Does nothing else.
    
    --version, -V
    Display version, disclamier and exit. Does nothing else.
    
    --defaults
    Display defaults values and exit. Useful to know default behavoir of
    rottlog
    
    --showlog <logfile>, -s <logfile>
    Show contents of archived <logfile> with default pager. <logfile> can be
    only a filename without any path or an absolute or relative path. If
    <logfile> is a relative path to a file, will be searched in $packdir
    dir.
    
    --pager <program>
    Use <program> for paging output of showlog. Useful only with --showlog
    
    --summary <logfile>
    Display a summary of some useful information about <logfile>. Does not
    exec any rotate/archive action. It's something like a --checkrc 
    but more verbose.
    
    --checkrc <what>,<what>,...
    Check for syntax of specified configuration file(s)
    <what> must be one of: daily, weekly, monthly, custom, all

    --force [<daily>] [<weekly>] [<monthly>] [<custom>]
    --forceonly [<daily>] [<weekly>] [<monthly>] [<custom>]

    Force rotation/archiviation of logfiles , even  if  it doesn't think
    this is necessary.  Sometimes this is useful after adding new entries
    to some config file, or if old log files have been removed by hand,
    as the new files will be created, and  logging  will  continue
    correctly.  If none of daily,weekly and monthly is used, will be
    forced rotation/archiviation of all entries in all config files;
    If used --force option will be forced rotation/archiviation of specified 
    config file and will be checked for rotation/archiviation remaining config
    files.
    If used --forceonly option will be forced rotation/archiviation of specified
    config file without check for rotation/archiviation remaining config files.
    (For custom config file this option is useful only if used forceonly 
    parameter.)


