Bash Backup Script

This bash script is designed to backup the filesystems here at PyeNet. This script was developed to surround and manipulate the DAR (Disk ARchive) utility as needed for the filesystems. The version 4, shown here, now has the ability to copy the backup files using the openSSH scp utility. Thus avoiding the use of NFS mounted file systems.

DAR is designed to backup filesystems to disk or CD/DVD media. For more details, and downloading, visit the home page of DAR. However check your distribution's installable software to see if they have included a version of DAR.
Features and Overview

Features of this bash script are:

    Uses a configuration file to specify options, so the script itself does not need to change* between machines.
    Script checks for availability of all the utilities, and commands needed and errors out if any are missing.
    Backup files are created on a local destination directory then copied to a remote distination**.
    Can use the openSSH scp utility to copy the files, this needs the setup of authentication keys to allow for automatic login to occur.
    Rotation system based on a daily, weekly and monthly backup schedule.
    Can create seperate new backup files each time the script runs, or overwrite the previous backup file.
    Creates a log file that indicates what happened during the backup, and what didn't happen if things didn't work.
    The script can email the logfile. This feature uses a PERL script to do this, details below.
    Uses a feature to add the latest log entries to the beginning of the log file and only emails the latest information.

    * There maybe a requirement to change some variables that point to the location of executables, these are commented in the script.
    ** For best fault tolerance the local destination should be a separate hard disk in the machine, but does not need to be.

Basic overview of what the script does:

    Reads the configuration file to setup the variables required for the rest of the script.
    Works out current day to determine which backup to perform within the rotation system.
    Confirms that the filesystems mentioned in the configuration are mounted or are accessable using the openSSH utilities.
    Checks the free disk space and the size of the data to be backed up and compares them.
    Starts the backup of the data to the local destination.
    Copies the backup files from the local destination to the remote destination.
    Emails the logfile to the account specified in the configuration.
    If there is an error in any of the above steps the logfile is updated with a possible cause and then emailed.

Requirements:

    bash v2 or greater. I have not tested this on an earlier version so cannot say if I am using v2 only features.
    Disk ARchive (DAR) available from http://dar.linux.free.fr/
    sendEmail perl script available from http://caspian.dotconf.net/menu/Software/SendEmail/
    If wishing to use the openSSH scp utility to copy the files this needs to be installed.

Installation

Install the DAR utiltiy as per the current instructions. Note that this script is using the compression feature of DAR so if you do compile from the sources make sure that this option is compiled in.

Install the sendEmail script as per it's README file. Basically as root copy the file to the /usr/local/bin directory. Set the file as executable and it is ready to go (assuming you have a recent version of PERL installed).

Logged back in as a non-root user and running the sendEmail command should produce a listing of the options required to correctly run sendEmail. This confirms it is installed correctly.

Copy the example configuration file to a filename that indicates the data being backed up. As explained in the comments within the configuration file, the name given to the file is also the base name used for the backup files once things get underway.

Go through each configuration line changing it as needed by your backup requirements. Each option is documented in the configuration file itself.

To run the backup from the command line the following syntax is used ./do_backup_v4  <config_filename>. It is suggested to set up a configuration file that backs up a test directory first. Once this is working change it to backup a real one.

Once this is all completed adding an entry to the root users CRONTAB will make sure that the backup is performed on a regular basis. Note that the /usr/local/bin directory may have to be added to the crontab's PATH environment variable so the sendEmail script is run correctly.

Things I would like to add:

    More comprehensive reporting from the DAR utility so these could be included in the logfile.

Cheers Jim

Disclaimer - Please note that this system is working well on my servers and configuration. I can not be responsible for it working, or not working, on your configuration as I do not know what your configuration is. However if you do find any issues please let me know so I can find a solution to them.

Note that even though the lines below flow off the edge of the page they will copy and paste into an editor AOK.

#!/bin/bash
#########################################################################################
#       Backup routine using DAR
#
# This will backup the directories specified to a local disk and then
# to a remote location over scp
#
# Invocation: do_backup_v4  configuration_file
#
# Version History:
# Version 1, Feb 2005
#
# Version 2, Dec 2005
# - Modified to make more variables to make modifications of script easier
# - Also added the loop thru the SRC_DIRS array to make the script more compact
# - Added the Directory space checks
# - Added the ability to control whether the backup creates subdirs for each backup
#   or not
# - Moved the variables to seperate configuration file to allow for an even more generic
#   script
#
# - Feb 2006
# - Changed the email subject line from host name to backup name to make it a bit easier to
#   see which backup the message is for when there are more than one coming from a
#   host.
#
# Version 3, Aug 2006
# - Backup rotation system added
# - Changed the disk free calculation from using df to stat as an issue was seen if space
#   in directory path
# - Tidied up format of log file to make little easier to read
#
# Version 3.1, Sept 2006
# - Modified log file so new log is appended to beginning of file and only current log
#   is emailed.
#
# Version 4, Dec 2007
# - changed the way the utilities are checked for so if they are unavailable script
#   stops.
# - changed the file copying to use scp etc. so no nfs drive mappings required.
#
# TODOs
#
# - Need to output the error message returned by each command to the logfile
# to aid in troubleshooting
#
#
#########################################################################################

# Script Begins...
#
# Setup variables to use later on
#
#
# This VERSION number must match the CFG_VERSION if it is defined in the configuration file
#
VERSION="4.0"

# Utilities used in script
# The full path is required to the utility commands as this script is intended to be run
# under a CRON job
#
UTILNOTFOUND="false"

if [ `which date` ]; then DATE=`which date`; else UTILNOTFOUND="date"; fi
if [ `which tail` ]; then TAIL=`which tail`; else UTILNOTFOUND="tail"; fi
if [ `which cut` ]; then CUT=`which cut`; else UTILNOTFOUND="cut"; fi
if [ `which gawk` ]; then GAWK=`which gawk`; else UTILNOTFOUND="gawk"; fi
if [ `which mount` ]; then MOUNT=`which mount`; else UTILNOTFOUND="mount"; fi
if [ `which grep` ]; then GREP=`which grep`; else UTILNOTFOUND="grep"; fi
if [ `which du` ]; then DU=`which du`; else UTILNOTFOUND="du"; fi
if [ `which df` ]; then DF=`which df`; else UTILNOTFOUND="df"; fi
if [ `which stat` ]; then STAT=`which stat`; else UTILNOTFOUND="stat"; fi
if [ `which mkdir` ]; then MKDIR=`which mkdir`; else UTILNOTFOUND="mkdir"; fi
if [ `which cp` ]; then CP=`which cp`; else UTILNOTFOUND="cp"; fi
if [ `which cat` ]; then CAT=`which cat`; else UTILNOTFOUND="cat"; fi
if [ `which rm` ]; then RM=`which rm`; else UTILNOTFOUND="rm"; fi
if [ `which mv` ]; then MV=`which mv`; else UTILNOTFOUND="mv"; fi
if [ `which ssh` ]; then SSH=`which ssh`; else UTILNOTFOUND="ssh"; fi
if [ `which scp` ]; then SCP=`which scp`; else UTILNOTFOUND="scp"; fi
if [ `which touch` ]; then TOUCH=`which touch`; else UTILNOTFOUND="touch"; fi
if [ `which dar` ]; then DAR=`which dar`; else UTILNOTFOUND="dar"; fi
if [ `which sendEmail` ]; then SENDEMAIL=`which sendEmail`; else UTILNOTFOUND="sendEmail"; fi

if [ ${UTILNOTFOUND} != "false" ]; then
  echo ""
  echo "Error: Required utility missing"
  echo ""
  echo "  This backup script requires access to several utilities to"
  echo "  perform a successful backup."
  echo "  It seems that - ${UTILNOTFOUND} - is missing."
  echo ""
  echo "  Exiting..."
  echo ""
  exit 1
fi

#
# Check to see if configuration file is specified on the command line.
# If so, use it to populate variables needed for script, if not error out...
#
# The name used for the configuration file is also used to specify the name of the backup
# being performed. This name is used as the base for the directories and filename for the
# backup
#
if [ -z "$1" ]; then
  echo "do_backup: Oops, you forgot to specify the configuration file I am to use"
  exit 1
else
  if [ ! -f "$1" ] || [ ! -r "$1" ]; then
    echo "do_backup: There is a problem accessing the file: $1"
    echo "           Please make sure this file exists and is readable"
    exit 1
  else
    BACKUP_FULL_NAME="$1"
    . "$1"
  fi
fi

#
# Check to see if the configuration file has the CFG_VERSION variable
# If not assume configuration file is from earlier version and
# does not have v4 onwards features. Like USESSH etc.
#
if [ -n "${CFG_VERSION}" ]; then
  #
  # confirm that the configuration file is the correct version
  #
  if [ ${CFG_VERSION} != ${VERSION} ]; then
    echo ""
    echo "Error: Configuration file in not correct version"
    echo ""
    echo "  The installation configuration file is required to be"
    echo "  the correct version."
    echo ""
    echo "  Script is version: ${VERSION}. Config file is version: ${CFG_VERSION}"
    echo ""
    echo "  Exiting..."
    echo ""
    exit 1
  fi
else
  #
  # We need to set variables here that will not exist because the configuration
  # is an older version.
  #
  USESSH="false"
fi

#
# Setup the thousands separator function to convert numbers into their
# thousands separated equals
#
function thou_sep
{
# THOU_SEP defines the character to be used as the seperator
THOU_SEP=","
## Declare Display Number as a string equal to number entered
declare +i DISP_NUM=$1
# Find the length of the Display Number variable
declare -i LGTH_DISP_NUM=${#DISP_NUM}
#Set up a variable to contain the part of the string being worked on and the thousands separator
SEP_NUM=""
# Loop thru cutting the number string into 3 character chunks and add the thousands separator
until [ $(($LGTH_DISP_NUM < 3)) = 1 ] ; do
CURRENT_PART=${DISP_NUM:$LGTH_DISP_NUM-3}
DISP_NUM=${DISP_NUM%$CURRENT_PART}
SEP_NUM="$THOU_SEP$CURRENT_PART$SEP_NUM"
LGTH_DISP_NUM=$(($LGTH_DISP_NUM - 3))
done
# Add the last chunk to the beginning of the number to be displayed
DISP_NUM=$DISP_NUM$SEP_NUM
# However if the number of digits in the number entered is divisible by 3 it will have a leading
# thousands separator, this will remove it
DISP_NUM=${DISP_NUM#$THOU_SEP}
# Display the separated number
echo "$DISP_NUM"
}

#
# log rotation function basically makes sure latest information in log is at top
#
function log_rotate
{
# Rotate the log file by appending the archive file to the current .out.log file then renaming
# the .out.log file to the .log archive file. Effectively putting the newer log information at the
# beginning.
$CAT $LOGFILE_ARCHIVE >> $LOGFILE_LOC
$MV $LOGFILE_LOC $LOGFILE_ARCHIVE
}

# Strip off the path to get the name of this backup from the filename of the configuration file used.
BACKUP_NAME=${BACKUP_FULL_NAME##/*/}

# Logfile Name - This is the name of log file based on the combination of the directory specified
# in the configuration file, the name of the backup and the string -backup.log
LOGFILE_LOC="$LOGFILE_DIR/$BACKUP_NAME-backup.out.log"
LOGFILE_ARCHIVE="$LOGFILE_DIR/$BACKUP_NAME-backup.log"

# Make sure the logfile exists. If not create it.
if [ ! -f ${LOGFILE_ARCHIVE} ]; then
  ${TOUCH} ${LOGFILE_ARCHIVE}
fi

# CURDATE_EMAIL is used for the time stamp in emails and logfiles
CURDATE_EMAIL="$($DATE +%Y%m%d-%H%M)"

# CURDATE is set to FullBackup here. This will be the name of the subdirectory created if we
# are not using rotation. If rotation is being used this will be overridden.
CURDATE="FullBackup"

# Backup Rotation System
# For this to work correctly the backup must be scheduled to happen in the morning
# The directory name used is indicating the date of data the backup contains
# not the day the backup was performed. To change this to the date the backup was
# performed remove the -d yesterday from the DATE commands below.
# If the ROTATE is not empty then the backup is to be in the rotation system
# If the ROTATE is empty then the backup will be not be rotated and each backup will
# overwrite previous backup. The CURDATE is left as FullBackup in this case.
if [ -n "$ROTATE" ]; then
    # Set CURDATE to the day of the week. We use yesterday as the backup will be performed
    # in the morning of the next day. The CURDATE variable will be overidden if the day
    # happens to be either the first of the month (monthly) or end of the working week
    # ie. Saturday (weekly)
    CURDATE="$($DATE -d yesterday +%A)"
    CURDAYOFMONTHNUM=$($DATE +%_d)
    CURDAYOFWEEKNUM=$($DATE +%u)

    # Check for first of Month if it is then we need to make this the
    # Full monthly backup for the previous month
    if [ $CURDAYOFMONTHNUM -eq 1 ]; then
        echo "Doing Monthly Backup for: $($DATE -d yesterday +%B)"
        CURDATE="$($DATE -d yesterday +%B-%Y)-Monthly"
    else
        # If today is Saturday then we are doing the Weekly backup and
        # need to find which week number we are in.
        if [ $CURDAYOFWEEKNUM -eq 6 ]; then
            if [ $CURDAYOFMONTHNUM -lt 8 ]; then
                CURDATE="Friday-Week1"
            elif [ $CURDAYOFMONTHNUM -lt 15 ]; then
                CURDATE="Friday-Week2"
            elif [ $CURDAYOFMONTHNUM -lt 22 ]; then
                CURDATE="Friday-Week3"
            elif [ $CURDAYOFMONTHNUM -lt 29 ]; then
                CURDATE="Friday-Week4"
            else
                CURDATE="Friday-Week5"
            fi
        fi
    fi
fi

###########################################
#
# Actual backup procedure from here on...
#
###########################################

# Create a header for current backup job in the logfile, we append current job to log
echo " " >> $LOGFILE_LOC
echo "=*=*=*=  $CURDATE backup of $BACKUP_NAME from $LOC_SYSTEM_NAME  =*=*=*=" >> $LOGFILE_LOC
echo " " >> $LOGFILE_LOC
echo -n "Started backup job at:  " >> $LOGFILE_LOC
$DATE >> $LOGFILE_LOC
echo " " >>  $LOGFILE_LOC


echo "do_backup: Checking backup destinations are accessable"
echo -n "Checking backup destinations are accessable at:  " >> $LOGFILE_LOC
$DATE >> $LOGFILE_LOC

# Check for and mount local mount point
echo "do_backup: Checking that local filesystem $LOC_MOUNT_POINT is mounted"
echo -n "  Checking that local filesystem $LOC_MOUNT_POINT is mounted at:  " >> $LOGFILE_LOC
$DATE >> $LOGFILE_LOC
if ! $GREP -E [[:space:]]$LOC_MOUNT_POINT[[:space:]] /etc/mtab
then
        echo "do_backup: $LOC_MOUNT_POINT not mounted, trying to mount"
        if ! $MOUNT $LOC_MOUNT_POINT
        then
                echo "do_backup: Failed to mount $LOC_MOUNT_POINT"
                echo -n "!!!*** WARNING ***!!! Failed to mount $LOC_MOUNT_POINT at:  " >> $LOGFILE_LOC
                $DATE >> $LOGFILE_LOC
                $SENDEMAIL      -f $EMAIL_FROM \
                                -t $EMAIL_TO \
                                -u "*** FAILED!!! *** $BACKUP_NAME Backup of $LOC_SYSTEM_NAME" \
                                -m "$CURDATE backup of $BACKUP_NAME from $LOC_SYSTEM_NAME *** FAILED *** : $CURDATE_EMAIL.\n\nReason: Failed to mount $LOC_MOUNT_POINT" \
                                -s $EMAIL_SERVER \
                                -a $LOGFILE_LOC
                `log_rotate`
                exit 1
        fi
fi
echo "do_backup: $LOC_MOUNT_POINT is mounted AOK"
echo -n "    $LOC_MOUNT_POINT is mounted AOK at:  " >> $LOGFILE_LOC
$DATE >> $LOGFILE_LOC


if [ ${USESSH} = "false" ]; then
  # Check for, and mount. remote mount point for the copy later on. Doing this here
  # to make sure operator sees login prompt. We test at end of backup to make
  # sure that the remote mount point is still available before trying to copy the files
  echo "do_backup: Checking that remote filesystem $REM_MOUNT_POINT is mounted"
  echo -n "  Checking that remote filesystem $REM_MOUNT_POINT is mounted at:  " >> $LOGFILE_LOC
  $DATE >> $LOGFILE_LOC
  if ! $GREP -E [[:space:]]$REM_MOUNT_POINT[[:space:]] /etc/mtab
  then
          echo "do_backup: $REM_MOUNT_POINT not mounted, trying to mount"
          echo " "
          echo "Please enter login password for user account on $REM_SYSTEM_NAME machine"
          if ! $MOUNT $REM_MOUNT_POINT
          then
                  echo "do_backup: Failed to mount $REM_MOUNT_POINT"
                  echo -n "!!!*** WARNING ***!!! Failed to mount $REM_MOUNT_POINT at:  " >> $LOGFILE_LOC
                  $DATE >> $LOGFILE_LOC
                  $SENDEMAIL    -f $EMAIL_FROM \
                                  -t $EMAIL_TO \
                                  -u "*** FAILED!!! *** $BACKUP_NAME Backup of $LOC_SYSTEM_NAME" \
                                  -m "$CURDATE backup of $BACKUP_NAME from $LOC_SYSTEM_NAME *** FAILED *** : $CURDATE_EMAIL.\n\nReason: Failed to mount $REM_MOUNT_POINT" \
                                  -s $EMAIL_SERVER \
                                  -a $LOGFILE_LOC
                  `log_rotate`
                  exit 1
          fi
  fi
  echo "do_backup: $REM_MOUNT_POINT is mounted AOK"
  echo -n "    $REM_MOUNT_POINT is mounted AOK at:  " >> $LOGFILE_LOC
  $DATE >> $LOGFILE_LOC
else
  # Check the a ssh connection is using the key authentication. If not stop the
  # script as a password prompt is going to be asked for each remote command. This
  # will not work successfully for an automated script
  echo "do_backup: Checking that remote commands can be run using authentication keys on $REM_SYSTEM_NAME"
  echo -n "  Checking that remote commands can be run using authentication keys on $REM_SYSTEM_NAME at:  " >> $LOGFILE_LOC
  $DATE >> $LOGFILE_LOC
  if ! echo "pwd" | $SSH root@${REM_SYSTEM_NAME} /bin/bash
  then
    echo "do_backup: Failed to run remote command on $REM_SYSTEM_NAME"
    echo -n "!!!*** WARNING ***!!! Failed to run remote command on $REM_SYSTEM_NAME at:  " >> $LOGFILE_LOC
    $DATE >> $LOGFILE_LOC
    $SENDEMAIL  -f $EMAIL_FROM \
        -t $EMAIL_TO \
        -u "*** FAILED!!! *** $BACKUP_NAME Backup of $LOC_SYSTEM_NAME" \
        -m "$CURDATE backup of $BACKUP_NAME from $LOC_SYSTEM_NAME *** FAILED *** : $CURDATE_EMAIL.\n\nReason: Failed to run remote command on $REM_SYSTEM_NAME\n\nAre authentication keys setup correctly?" \
        -s $EMAIL_SERVER \
        -a $LOGFILE_LOC
    `log_rotate`
    exit 1
  fi
fi
echo -n "Finished checking access to remote filesystems at:  " >> $LOGFILE_LOC
$DATE >> $LOGFILE_LOC
echo " " >>  $LOGFILE_LOC

# Check disk space available at the destination directories
# Using the stat utility for this
# The %f format is the free blocks
# The %s is the block size in bytes
# The -f means stat reports on the filesystem itself instead of the filename specified
echo "do_backup: Checking backup size and free space available"
echo -n "Checking estimated backup size and free space available at:  " >> $LOGFILE_LOC
$DATE >> $LOGFILE_LOC

# Local Destination
echo "Local Dest Dir is: $LOC_DEST_DIR"
declare -i REAL_LOC_DISK_FREE=$(($($STAT -f -c%f "$LOC_DEST_DIR")*$(($($STAT -f -c%s "$LOC_DEST_DIR")))))

# Adjust REAL_LOC_DISK_FREE if the local destination directory exists as this backup is
# going to overwrite the previous one.
if [ -d "$LOC_DEST_DIR/$BACKUP_NAME/$CURDATE" ]; then
    REAL_LOC_DISK_FREE=$REAL_LOC_DISK_FREE+$($DU "$LOC_DEST_DIR/$BACKUP_NAME/$CURDATE" -h -c -b | $TAIL -n1 | $CUT -f1)
fi

# Convert raw number into thousands separated number for display
DISPLAY_REAL_LOC_DISK_FREE=`thou_sep $REAL_LOC_DISK_FREE`

echo "do_backup: $DISPLAY_REAL_LOC_DISK_FREE bytes available at the local destination"
echo -n "  $DISPLAY_REAL_LOC_DISK_FREE bytes available at the local destination at:  " >> $LOGFILE_LOC
$DATE >> $LOGFILE_LOC

# Remote Destination
if [ ${USESSH} = "false" ]; then
  declare -i REAL_REM_DISK_FREE=$(($($STAT -f -c%f "$REM_DEST_DIR")*$(($($STAT -f -c%s "$REM_DEST_DIR")))))
else
  declare -i REAL_REM_DISK_FREE=$(echo "${DF} "\"${REM_DEST_DIR}\"" | ${TAIL} -n1 | ${GAWK} -F\" \" '{ print \$4 }'" | ${SSH} root@${REM_SYSTEM_NAME} /bin/bash)
  REAL_REM_DISK_FREE=$((${REAL_REM_DISK_FREE}*1024))
fi

# Adjust REAL_REM_DISK_FREE if the remote destination directory exists as this backup is
# going to overwrite the previous one
declare -i CURRENT_REMOTE_USAGE=0
if [ ${USESSH}="false" ]; then
  if [ -d "$REM_DEST_DIR/$BACKUP_NAME/$CURDATE" ]; then
    CURRENT_REMOTE_USAGE=$($DU "$REM_DEST_DIR/$BACKUP_NAME/$CURDATE" -h -c -b | $TAIL -n1 | $CUT -f1)
  fi
else
  if ${SSH} root@${REM_SYSTEM_NAME} "ls "\'${REM_DEST_DIR}/${BACKUP_NAME}/${CURDATE}\'"" > /dev/null 2>&1
  then
    CURRENT_REMOTE_USAGE=$(echo "${DU} "\"${REM_DEST_DIR}/${BACKUP_NAME}/${CURDATE}\"" -h -c -b | ${TAIL} -n1 | ${CUT} -f1" | ${SSH} root@${REM_SYSTEM_NAME} /bin/bash)
  fi
fi
REAL_REM_DISK_FREE=${REAL_REM_DISK_FREE}+${CURRENT_REMOTE_USAGE}

# Convert raw number into thousands separated number for display
DISPLAY_REAL_REM_DISK_FREE=`thou_sep $REAL_REM_DISK_FREE`

echo "do_backup: $DISPLAY_REAL_REM_DISK_FREE bytes available at the remote destination"
echo -n "  $DISPLAY_REAL_REM_DISK_FREE bytes available at the remote destination at:  " >> $LOGFILE_LOC
$DATE >> $LOGFILE_LOC

# Check the possible size of the backup dirs
declare -i TOTAL_DIR_SIZE=0
for EACH_SRC_DIR in "${SRC_DIRS[@]}"; do
        declare -i DIR_SIZE=$($DU "$EACH_SRC_DIR" -h -c -b | $TAIL -n1 | $CUT -f1)
#       echo "do_backup: $EACH_SRC_DIR = $DIR_SIZE"
        TOTAL_DIR_SIZE=$TOTAL_DIR_SIZE+$DIR_SIZE
done

# Convert raw number into thousands separated number for display
DISPLAY_TOTAL_DIR_SIZE=`thou_sep $TOTAL_DIR_SIZE`

echo "do_backup: $DISPLAY_TOTAL_DIR_SIZE bytes is the estimated size of the backup"
echo -n "  $DISPLAY_TOTAL_DIR_SIZE bytes is the estimated size of the backup at:  " >> $LOGFILE_LOC
$DATE >> $LOGFILE_LOC

# Calculate if the backup is going to fit on the destination
# Note that the calculated size of the data to be backed up is going to be bigger than the actual
# data as it does not take into account the DAR utility compression or the files it will not
# backup. But this means there should be plenty of spare space after backup has completed
if  (( ($REAL_LOC_DISK_FREE < $TOTAL_DIR_SIZE) || ($REAL_REM_DISK_FREE < $TOTAL_DIR_SIZE) )) ; then
        echo "do_backup: BooHoooo not enough space left to backup files..."
        echo " " >> $LOGFILE_LOC
        echo -n "!!!*** WARNING ***!!! Not enough disk space to complete backup at:  " >> $LOGFILE_LOC
        $DATE >> $LOGFILE_LOC
        $SENDEMAIL      -f $EMAIL_FROM \
                        -t $EMAIL_TO \
                        -u "*** FAILED!!! *** $BACKUP_NAME Backup of $LOC_SYSTEM_NAME" \
                        -m "$CURDATE backup of $BACKUP_NAME from $LOC_SYSTEM_NAME *** FAILED *** : $CURDATE_EMAIL.\n\nReason: No space for backup file.\n\nEstimated backup size: $DISPLAY_TOTAL_DIR_SIZE.\nLocal space available: $DISPLAY_REAL_LOC_DISK_FREE.\nRemote space available: $DISPLAY_REAL_REM_DISK_FREE." \
                        -s $EMAIL_SERVER \
                        -a $LOGFILE_LOC
        `log_rotate`
        exit 1
else
        echo "do_backup: Yahooo we have enough space to backup files..."
        echo -n "  Looks like we are good to go...  at:   " >> $LOGFILE_LOC
        $DATE >> $LOGFILE_LOC
fi

echo "do_backup: Done checking size of backup data and free disk space"
echo -n "Done checking size of data and free disk space at:  " >> $LOGFILE_LOC
$DATE >> $LOGFILE_LOC
echo " " >> $LOGFILE_LOC

# Checking to see if the local destination directory exists, if not creates it
echo "do_backup: Checking if local backup destination directory $CURDATE exists"
echo -n "Checking if local backup destination directory $CURDATE exists at:  " >> $LOGFILE_LOC
$DATE >> $LOGFILE_LOC
if [ ! -d $LOC_DEST_DIR/$BACKUP_NAME/$CURDATE ]; then
    echo "do_backup: Creating local destination directory $LOC_DEST_DIR/$BACKUP_NAME/$CURDATE"
    echo -n "  Creating local destination directory $LOC_DEST_DIR/$BACKUP_NAME/$CURDATE at:  " >> $LOGFILE_LOC
    $DATE >> $LOGFILE_LOC
    # Create Local destination directory
    if ! $MKDIR -p $LOC_DEST_DIR/$BACKUP_NAME/$CURDATE
    then
        echo "do_backup: Failed to create $LOC_DEST_DIR/$BACKUP_NAME/$CURDATE on local $LOC_MOUNT_POINT"
        echo -n "!!!*** WARNING ***!!! Failed to create  $LOC_DEST_DIR/$BACKUP_NAME/$CURDATE on local $LOC_MOUNT_POINT at:  " >> $LOGFILE_LOC
        $DATE >> $LOGFILE_LOC
            $SENDEMAIL  -f $EMAIL_FROM \
                        -t $EMAIL_TO \
                        -u "*** FAILED!!! *** $BACKUP_NAME Backup of $LOC_SYSTEM_NAME" \
                        -m "$CURDATE backup of $BACKUP_NAME from $LOC_SYSTEM_NAME *** FAILED *** : $CURDATE_EMAIL.\n\nReason: Failed to create $LOC_DEST_DIR/$BACKUP_NAME/$CURDATE on local $LOC_MOUNT_POINT" \
                        -s $EMAIL_SERVER \
                        -a $LOGFILE_LOC
        `log_rotate`
        exit 1
    fi
    echo "do_backup: Created local destination dir $LOC_DEST_DIR/$BACKUP_NAME/$CURDATE"
    echo -n "  Created local destination dir $LOC_DEST_DIR/$BACKUP_NAME/$CURDATE at:  " >> $LOGFILE_LOC
    $DATE >> $LOGFILE_LOC
fi
echo "do_backup: Done checking if local backup destination directory exists"
echo -n "Done checking if local backup destination directory exists at:  " >> $LOGFILE_LOC
$DATE >> $LOGFILE_LOC
echo " " >> $LOGFILE_LOC

# Loop thru each of the source directories performing the backup
######
# DAR Options used
#
# -v = verbose
# -w = no warn on overwriting slice files
# -s = slice file size 2147483640 =/= 2G
# -z = use gzip compression
# -m = do not compress files smaller than 1024 bytes
# -X = exclude ~ backup files
# -Z = exclude from compression
# -R = root directory of files to be backed up
# -c = base filename for the archive that is to be created. Files will be <base>.<number>.dar
#
######

declare -i CURRENT_DUPLICATE=0

for EACH_SRC_DIR in "${SRC_DIRS[@]}"; do
  echo "do_backup: Starting ${EACH_SRC_DIR} dir backup"
  echo -n "Starting ${EACH_SRC_DIR} dir backup at:  " >> ${LOGFILE_LOC}
  ${DATE} >> ${LOGFILE_LOC}

  # We need to modify DAR's archive name if it starts with a "." as this causes issues
  # with the copy of the files to the remote destination directory.
  # This snippet adds a "_" to the beginning of the filename.
  BU_BASE_NAME=${EACH_SRC_DIR##*/}
  if [[ ${BU_BASE_NAME::1} == "." ]]; then
    BU_BASE_NAME="_${BU_BASE_NAME}"
  fi

  # We need to make sure that the BU_BASE_NAME is unique for this backup. We have a problem if
  # 2 source directories end with the same name eg. /home/data/test and /home/jim/test. In this
  # case the BU_BASE_NAME will be the same. We use the -w switch of DAR so it will overwrite
  # files from previous backups but this also means that the second file within the current
  # backup will overwrite the first.
  #
  declare -i DUPLICATE_COUNTER=0
  # Compare the BU_BASE_NAME with the entries in the array. If we find the name more than once
  # then add an extra number to the name of the backup filename to make in unique
  for EACH_DIR_NAME in "${SRC_DIRS[@]}"; do
    if [ "${BU_BASE_NAME}" = "${EACH_DIR_NAME##*/}" ] || [ "${BU_BASE_NAME}" = "_${EACH_DIR_NAME##*/}" ]; then
      DUPLICATE_COUNTER=$((DUPLICATE_COUNTER + 1))
      if [ ${DUPLICATE_COUNTER} -gt 1 ]; then
        CURRENT_DUPLICATE=$((CURRENT_DUPLICATE + 1 ))
        BU_BASE_NAME="${BU_BASE_NAME}_dir_${CURRENT_DUPLICATE}"
        echo "do_backup: Found duplicate name for the ${EACH_SRC_DIR} dir backup. Using ${BU_BASE_NAME}-bu as base name."
        echo "  Found duplicate name for the ${EACH_SRC_DIR} dir backup. Using ${BU_BASE_NAME}-bu as base name. " >> ${LOGFILE_LOC}
      fi
    fi
  done
  DUPLICATE_COUNTER=0

  if ! ${DAR}   -v \
                -w \
                -s 2147483640 \
                -z \
                -m 1024 \
                -X "*~" \
                -Z "*.png" \
                -Z "*.PNG" \
                -Z "*.pdf" \
                -Z "*.gif" \
                -Z "*.GIF" \
                -Z "*.zip" \
                -Z "*.ZIP" \
                -Z "*.jpg" \
                -Z "*.jpeg" \
                -Z "*.JPG" \
                -Z "*.tgz" \
                -Z "*.gz" \
    -Z "*.bz2" \
                -R "${EACH_SRC_DIR}" \
                -c ${LOC_DEST_DIR}/${BACKUP_NAME}/${CURDATE}/"${BU_BASE_NAME}-bu"
  then
        echo "do_backup: $EACH_SRC_DIR dir backup failed"
        echo -n "!!!*** WARNING ***!!! $EACH_SRC_DIR dir backup failed at:  " >> $LOGFILE_LOC
        $DATE >> $LOGFILE_LOC
        $SENDEMAIL      -f $EMAIL_FROM \
                        -t $EMAIL_TO \
                        -u "*** FAILED!!! *** $BACKUP_NAME Backup of $LOC_SYSTEM_NAME" \
                        -m "$CURDATE backup of $BACKUP_NAME from $LOC_SYSTEM_NAME *** FAILED *** : $CURDATE_EMAIL.\n\nReason: DAR failed backing up files to $LOC_DEST_DIR/$BACKUP_NAME/$CURDATE/" \
                        -s $EMAIL_SERVER \
                        -a $LOGFILE_LOC
        `log_rotate`
        exit 1
  fi

  echo "do_backup: Finished $EACH_SRC_DIR directory backup"
  echo -n "Finished $EACH_SRC_DIR directory backup at:  " >> $LOGFILE_LOC
  $DATE >> $LOGFILE_LOC
  echo " "  >> $LOGFILE_LOC
done

if [ ${USESSH} = "false" ]; then
  # Check again to make sure the remote point is still mounted
  echo "do_backup: Checking $REM_MOUNT_POINT still mounted"
  echo -n "Checking $REM_MOUNT_POINT is still mounted at:  " >> $LOGFILE_LOC
  $DATE >> $LOGFILE_LOC
  if ! $GREP $REM_MOUNT_POINT /etc/mtab
  then
          echo "do_backup: $REM_MOUNT_POINT not mounted, trying to mount"
          if ! $MOUNT $REM_MOUNT_POINT
          then
                  echo "do_backup: Failed to mount $REM_MOUNT_POINT"
                  echo -n "!!!*** WARNING ***!!! Failed attempting to mount $REM_MOUNT_POINT at:  " >> $LOGFILE_LOC
                  $DATE >> $LOGFILE_LOC
                  $SENDEMAIL    -f $EMAIL_FROM \
                                  -t $EMAIL_TO \
                                  -u "*** FAILED!!! *** $BACKUP_NAME Backup of $LOC_SYSTEM_NAME" \
                                  -m "$CURDATE backup of $BACKUP_NAME from $LOC_SYSTEM_NAME *** FAILED *** : $CURDATE_EMAIL.\n\nReason: Failed to mount $REM_MOUNT_POINT before copying files." \
                                  -s $EMAIL_SERVER \
                                  -a $LOGFILE_LOC
                  `log_rotate`
                  exit 1
          fi
  fi
  echo "do_backup: $REM_MOUNT_POINT is still mounted AOK"
  echo -n "  $REM_MOUNT_POINT is still mounted AOK at:  " >> $LOGFILE_LOC
  $DATE >> $LOGFILE_LOC
  echo " " >> $LOGFILE_LOC
fi

# Checking to see if the remote destination directory exists, if not creates it
echo "do_backup: Checking if remote backup destination directory $CURDATE exists"
echo -n "Checking if remote backup destination directory $CURDATE exists at:  " >> $LOGFILE_LOC
$DATE >> $LOGFILE_LOC
if [ ${USESSH} = "false" ]; then
  if [ ! -d $REM_DEST_DIR/$BACKUP_NAME/$CURDATE ]; then
    echo "do_backup: Creating remote destination directory $REM_DEST_DIR/$BACKUP_NAME/$CURDATE"
    echo -n "  Creating remote destination directory $REM_DEST_DIR/$BACKUP_NAME/$CURDATE at:  " >> $LOGFILE_LOC
    $DATE >> $LOGFILE_LOC
    if ! $MKDIR -p $REM_DEST_DIR/$BACKUP_NAME/$CURDATE
    then
      echo "do_backup: Failed to create destination directory $REM_DEST_DIR/$BACKUP_NAME/$CURDATE"
      echo -n "!!!*** WARNING ***!!! Failed to create destination directory $REM_DEST_DIR/$BACKUP_NAME/$CURDATE"
      $DATE >> $LOGFILE_LOC
      $SENDMAIL       -f $EMAIL_FROM \
                      -t $EMAIL_TO \
                      -u "*** FAILED!!! *** $BACKUP_NAME Backup of $LOC_SYSTEM_NAME" \
                      -m "$CURDATE backup of $BACKUP_NAME from $LOC_SYSTEM_NAME *** FAILED *** : $CURDATE_EMAIL.\n\nReason: Failed to create destination directory $REM_DEST_DIR/$BACKUP_NAME/$CURDATE" \
                      -s $EMAIL_SERVER \
                      -a $LOGFILE_LOC
      `log_rotate`
      exit 1
    fi
    echo "do_backup: Created remote destination directory $REM_DEST_DIR/$BACKUP_NAME/$CURDATE"
    echo -n "  Created remote destination directory $REM_DEST_DIR/$BACKUP_NAME/$CURDATE at:  " >> $LOGFILE_LOC
    $DATE >> $LOGFILE_LOC
  fi
else
  if ! echo "ls "\'${REM_DEST_DIR}/${BACKUP_NAME}/${CURDATE}\'"" | ${SSH} root@${REM_SYSTEM_NAME} /bin/bash #> /dev/null 2>&1
  then
    echo "do_backup: Creating remote destination directory $REM_DEST_DIR/$BACKUP_NAME/$CURDATE"
    echo -n "  Creating remote destination directory $REM_DEST_DIR/$BACKUP_NAME/$CURDATE at:  " >> $LOGFILE_LOC
    $DATE >> $LOGFILE_LOC
    if ! echo "$MKDIR -p "\"${REM_DEST_DIR}/${BACKUP_NAME}/${CURDATE}\""" | ${SSH} root@${REM_SYSTEM_NAME} /bin/bash
    then
        echo "do_backup: Failed to create destination directory $REM_DEST_DIR/$BACKUP_NAME/$CURDATE"
        echo -n "!!!*** WARNING ***!!! Failed to create destination directory $REM_DEST_DIR/$BACKUP_NAME/$CURDATE"
        $DATE >> $LOGFILE_LOC
        $SENDMAIL       -f $EMAIL_FROM \
                        -t $EMAIL_TO \
                        -u "*** FAILED!!! *** $BACKUP_NAME Backup of $LOC_SYSTEM_NAME" \
                        -m "$CURDATE backup of $BACKUP_NAME from $LOC_SYSTEM_NAME *** FAILED *** : $CURDATE_EMAIL.\n\nReason: Failed to create destination directory $REM_DEST_DIR/$BACKUP_NAME/$CURDATE" \
                        -s $EMAIL_SERVER \
                        -a $LOGFILE_LOC
        `log_rotate`
        exit 1
    fi
    echo "do_backup: Created remote destination directory $REM_DEST_DIR/$BACKUP_NAME/$CURDATE"
    echo -n "  Created remote destination directory $REM_DEST_DIR/$BACKUP_NAME/$CURDATE at:  " >> $LOGFILE_LOC
    $DATE >> $LOGFILE_LOC
  fi
fi
echo "do_backup: Done checking if remote backup destination directory exists"
echo -n "Done checking if remote backup destination directory exists at:  " >> $LOGFILE_LOC
$DATE >> $LOGFILE_LOC
echo " " >> $LOGFILE_LOC

# Copy the backup files to remote system.
# NOTE: the // /\ in the varables escapes any spaces in the path with a \. The scp command seems more sensitive
#       to the escaping than the ssh command
echo "do_backup: Copying backup files to ${REM_SYSTEM_NAME}"
echo -n "Begin copy of backup files to ${REM_SYSTEM_NAME} at:  " >> ${LOGFILE_LOC}
${DATE} >> ${LOGFILE_LOC}
REMOTE_BACKUP_DESTINATION="${REM_DEST_DIR// /\ }/${BACKUP_NAME// /\ }/${CURDATE// /\ }/"
if [ ${USESSH} = "false" ]; then
  ${CP} --reply=yes -r ${LOC_DEST_DIR// /\ }/${BACKUP_NAME// /\ }/${CURDATE// /\ }/* "${REMOTE_BACKUP_DESTINATION}"
  RETURN_FROM_COPY_COMMAND=$?
else
  ${SCP} -r ${LOC_DEST_DIR// /\ }/${BACKUP_NAME// /\ }/${CURDATE// /\ }/* root@${REM_SYSTEM_NAME}:"${REMOTE_BACKUP_DESTINATION}"
  RETURN_FROM_COPY_COMMAND=$?
fi

if [ ${RETURN_FROM_COPY_COMMAND} != 0 ];then
        echo "do_backup: Failed to copy backups to ${REM_SYSTEM_NAME}"
        echo -n "!!!*** WARNING ***!!! Failed to copy backups to ${REM_SYSTEM_NAME} at:  " >> ${LOGFILE_LOC}
        ${DATE} >> ${LOGFILE_LOC}
        ${SENDEMAIL}    -f ${EMAIL_FROM} \
                        -t ${EMAIL_TO} \
                        -u "*** FAILED!!! *** ${BACKUP_NAME} Backup of ${LOC_SYSTEM_NAME}" \
                        -m "${CURDATE} backup of ${BACKUP_NAME} from ${LOC_SYSTEM_NAME} *** FAILED *** : ${CURDATE_EMAIL}.\n\nReason: Failed to copy files to ${REM_SYSTEM_NAME}" \
                        -s ${EMAIL_SERVER} \
                        -a ${LOGFILE_LOC}
  `log_rotate`
  exit 1
else
        echo "do_backup: Copied backups to ${REM_SYSTEM_NAME} successfully"
        echo -n "Copied backups to ${REM_SYSTEM_NAME} successfully at:  " >> ${LOGFILE_LOC}
        ${DATE} >> ${LOGFILE_LOC}
        echo " "
        echo "do_backup: Backup completed successfully"
        echo -n "Backup completed successfully at:  " >> ${LOGFILE_LOC}
        ${DATE} >> ${LOGFILE_LOC}
fi

echo " " >> ${LOGFILE_LOC}
echo "=*=*=*=  Finished ${CURDATE} backup of ${BACKUP_NAME} from ${LOC_SYSTEM_NAME}  =*=*=*=" >> ${LOGFILE_LOC}
echo " " >> ${LOGFILE_LOC}

####################################
# Email log files
#
# The sendEmail is a perl script downloaded from:
#                http://caspian.dotconf.net/menu/Software/SendEmail/
#
# It is located in the /usr/local/bin directory and uses the following switches
#
# -f  From address
# -t  To address
# -u  Subject line
# -m  Message body
# -s  Server to send SMTP
# -a  Logfile to send as an attachment, note we are only sending the log for
#     the current backup. Below we rotate this into an archive file.
#
####################################
echo -n "do_backup: "
$SENDEMAIL      -f $EMAIL_FROM \
                -t $EMAIL_TO \
                -u "$BACKUP_NAME Backup Log for $CURDATE performed $CURDATE_EMAIL" \
                -m "$CURDATE backup of $BACKUP_NAME from $LOC_SYSTEM_NAME for $CURDATE_EMAIL Completed.\n\nUpdated Logfile attached :" \
                -s $EMAIL_SERVER \
                -a $LOGFILE_LOC

`log_rotate`
exit 0

 

===

Configuration File

Below is the configuration file required for the do_backup_v4 script above.
This file needs to be modified to match your system requirements and saved with the filename that will be used to identify the backup.
===

 

# Configuration file for the do_backup script
#
# dated 28 Dec 2007
#
# (c) 2007, Jim Pye - PyeNet Universal
#

#
# Configuration version
#
# This is the version of the configuration file. It must match the
# version defined in the do_backup_vX script
CFG_VERSION="4.0"

#
# Use SSH
#
# Use SSH and SCP to communicate with remote host instead of expecting
# the remote destination to be a mounted filesystem via the fstab
# Value here is either "true" or "false"
# Default is false for backward compatibility
USESSH="false"

#
# Email Details
#
# Email address to receive logfiles and notification of backup failures
# Separate addresses with spaces within the quotes for more than one recipient
# eg. EMAIL_TO="backup.reports@pyenet.co.nz"
#     or
#     EMAIL_TO="backup.reports@pyenet.co.nz  another.user@pyenet.co.nz"
EMAIL_TO="backup.reports@pyenet.co.nz"

# Email address that logfiles and notifications are sent from
# This must be a valid email address for the email server specified
# eg. EMAIL_FROM="backup.process@pyenet.co.nz"
EMAIL_FROM="backup.process@pyenet.co.nz"

# The email server used for the sending of email notifications
# eg. EMAIL_SERVER="mailserver.pyenet.co.nz"
EMAIL_SERVER="mailserver.pyenet.co.nz"

#
# Backup Locations details
#
#  The hostname of the local system
# This is the system that the backup script is running on
# eg. LOC_SYSTEM_NAME=linux-srv1
LOC_SYSTEM_NAME=local-srv

#  The hostname of the remote system
# This is the system that the backup will be copied to
# eg. REM_SYSTEM_NAME=linux-backup-srv
REM_SYSTEM_NAME=remote-srv

#  Backup Rotation
# Backups can either be made to overwrite the previous backup or to be in a rotation system
# of daily, weekly and monthly backups.
# If we are using the rotation system then the ROTATE variable needs
# to be true, therefore:
# To _USE_ROTATION_ place a character (space, text) in the string eg. ROTATE=" "
# To _NOT_USE_ROTATION_ make the variable an empty string eg. ROTATE=""
ROTATE="yes"

#  Log File Directory
# This variable defines the directory for the log files.
# The log file will placed in this directory with the name: $BACKUP_NAME-backup.log
# Note the trailing / is _NOT_ needed
# eg. LOGFILE_DIR=/var/log/backups
LOGFILE_DIR=/var/log/testbackups

#  Source Directories
# This is an array of the directories to be backed up with this configuration.
# Place each directory in quotes. This allows for directories with spaces.
# eg. SRC_DIRS=("/data/test dir with space" "/data/test-dir-wo-space")
# NOTE: If adding new directories to an already established backup, add them to the end.
#       This is so the backup order of established directories is not changed.
SRC_DIRS=("/home/data/temp-downloaded")

#  Local mount point
# This specifies the mount point for the local file system where the local copy of the backup
# is to be created. The DAR utility makes its backup directly to a sub directory of this mount point.
# Specifying this local mount point allows for an additional hard disk to be used for the local backup.
# If the local destination directory (defined next) is a directory under the root partition then
# use a / as the local mount point.
# eg. LOC_MOUNT_POINT=/mnt/data  or  LOC_MOUNT_POINT=/
LOC_MOUNT_POINT=/

#  Local Destination Directory
# The DAR utility creates the archive directly here first
# A subdirectory, or subdirectories, will be created below this directory using the $BACKUP_NAME and
# $CURDATE variables.
# The directory specified here must already be created and must be writable by the user account used
# to mount the filesystem.
# eg. LOC_DEST_DIR=/mnt/data/backups/linux-srv1
LOC_DEST_DIR=/home/data/testbackups

#  Remote mount point
# This specifies the mount point for the remote file system this is where the backup archives are to be
# copied.
# NOTE: If using SSH then this has no effect and can be left empty
# eg. REM_MOUNT_POINT=/mnt/linux-backup-srv or REM_MOUNT_POINT=""
REM_MOUNT_POINT=/mnt/remote-srv/backups

#  Remote Destination Directory
# This is where the backup archive files are copied to once they are created by DAR.
# The directory specified here must already be created and must be writable by the user account
# used to mount the filesystem.
# If this directory has spaces then enclose the path in quotes.
# eg. REM_DEST_DIR=/mnt/linux-backup-srv/backups/linux-srv1
#     or
#     REM_DEST_DIR="/mnt/linux-backup-srv/Jims Stuff/backups/linux-srv1"
# NOTE: for SSH the directory specified here is the directory portion of the scp syntax:
#    <user>@<host>:/home/data/backups  ie.  "/home/data/backups"
REM_DEST_DIR=/mnt/remote-srv/backups/test-backups