My posts about issues concerning Linux OS and network administration

Advanced Bacula management – getting rid of unused Bacula volumes

I’ve been using Bacula software for couple of years in different companies and in different configurations.
Bacula is a great and complex backup software which can compete with big commercial backup solutions in some ways but it also have some cons you have to deal with when implementing it in production environment.

One of the issues is how Bacula is handling volumes once you store them as a files on the hard disk. Because Bacula was written to primarily support tapes as a media to store the data to, it still treads the volumes stored on hard disk this way which means Bacula NEVER deletes the physical volumes from the disk. This can under certain circumstances lead to a situation where there are some obsolete volumes on the disk which Bacula is not aware of anymore and these would never be deleted. There are actually two situations when you would want to free the disk space:

  • You delete some volumes from Bacula catalog manually (detele volume=xxx) but you forget to delete the volume from the disk
  • You suddenly backup more data than normally(eg. due to one-time full-level manual backup) within the Pool which can force Bacula to create new volumes(files) on the disk. These volumes can of course be recycled once the retention times for them expire but as Bacula marks the volume purged (all retentions are expired and can be reused if Recycle flag is set to yes) only when it needs new volume to write data to, there still can, and after some time surely will, be many volumes marked as “Full” and event thought its data could be deleted because all retention times are already expired this will never happen. All these volumes will be reused (recycled) one time but as there are many of these “obsolete” volumes it can take long time to recycle them all, much longer than retention period of data stored in these volumes

Anyway, if you use Bacula for some longer time to store the data to hard disk, you for sure will have some dead/obsolete data on disk which Bacula itself will never delete. I was searching on Internet for the best approach of how to handle these obsolete volumes and remove them. I’ve found couple of articles written by people who needed to handle similar issues but none of them I liked fully.

I’ve decided for an approach of using as much as possible features already implemented in Bacula (such as Truncating volume feature presented in Bacula 5.0.1) and only perform the operations Bacula can not do.

This is a technical background of an implementation ensuring automatic wiping of old volumes I’ve decided for:

  1. Reconfigure Bacula director to use “Action On Purge = Truncate” in all Pools
  2. Set “actiononpurge=Truncate” for every already created volume. New volumes will inherit this setting automatically from Pool
  3. Prune all volumes. This will force Bacula to mark volumes as Purged if all the retention periods are expired for this volume. Bacula doesn’t change the volume state if these retention periods aren’t met. This means all volumes marked as Purged can be further deleted
  4. Delete all volumes marked as Purged from Bacula catalog as Bacula doesn’t store any relevant data in them.
  5. Wipe all obsolete volumes from disk. These are volumes Bacula isn’t aware of anymore as we let Bacula delete (purge) these frim its catalog(database)
#!/bin/bash

print_usage() {
        echo "  -p";
        echo "     prune all volumes ";
        echo "  -t";
        echo "     update all volumes to \"actiononpurge=Truncate\"";
        echo "  -n";
        echo "     update all volumes to \"actiononpurge=None\"";
        echo "  -Do";
        echo "     delete obsolete volume files from the disk (those not listed in catalog)"
        echo "       NOTE: this operation can take several minutes to complete"
        echo "  -Dp";
        echo "     delete all purged volumes from Bacula catalog"
        echo "  -h";
        echo "     print this screen";
        echo ""
        exit 0

}

if [ $# -lt 1 ]; then
        print_usage
        exit 3
fi

update_volume() {

        BACULA_BATCH="$(mktemp)"

        echo ""
        echo "updating all volumes to \"actiononpurge=$1\"..."
        echo ""

        cd /srv/backup/Pools/
        ls -1 | while read pool; do
                cd /srv/backup/Pools/$pool
                if [ `find . -maxdepth 1 -type f | wc -l` -gt 0 ]; then
                        for i in $pool-*; do
                                echo "update volume=$i ActionOnPurge=$1" >> $BACULA_BATCH;
                        done
                fi
        done
        bconsole < $BACULA_BATCH | grep $1
        rm -f $BACULA_BATCH

}

prune_all() {

        BACULA_BATCH="$(mktemp)"

        echo ""
        echo "pruning all volumes and let Bacula mark them as purged once the retention periods are expired..."
        echo ""

        cd /srv/backup/Pools/
        ls -1 | while read pool; do
                cd /srv/backup/Pools/$pool
                if [ `find . -maxdepth 1 -type f | wc -l` -gt 0 ]; then
                        for i in $pool-*; do
                                echo "prune volume=$i" >> $BACULA_BATCH;
                                echo "yes" >> $BACULA_BATCH;
                        done
                fi
        done
        bconsole < $BACULA_BATCH | grep -i "marking it purged"
        rm -f $BACULA_BATCH
}

delete_obsolete_volumes() {

        VOLUMES_TO_DELETE="$(mktemp)"

        echo ""
        echo "searching for obsolete files on disk. These could be some old volumes deleted from catalog manually..."
        echo ""

        cd /srv/backup/Pools/
        ls -1 | while read pool; do
                cd /srv/backup/Pools/$pool
                if [ `find . -maxdepth 1 -type f | wc -l` -gt 0 ]; then
                        for i in $pool-*; do
                                echo "list volume=$i" | bconsole | if grep --quiet "No results to list"; then
                                        echo "$i is ready to be deleted"  
                                        echo "/srv/backup/Pools/$pool/$i" >> $VOLUMES_TO_DELETE
                                fi
                        done
                fi
        done

        # Check whether we have found some volumes(files) to delete
        if [ `wc -l $VOLUMES_TO_DELETE | awk {'print $1'}` -gt 0 ]; then

                echo ""
                echo -n "Are you sure you want to delete these volumes(files) from the disk ? (yes|no): "
                read response

                if [ $response = "yes" ]; then
                        cat $VOLUMES_TO_DELETE | while read I; do
                                rm -f "$I"
                        done

                        echo ""
                        echo "DONE: following files were deleted: "
                        cat $VOLUMES_TO_DELETE
                fi

        else
                echo "No volumes to delete found on disk"

        fi

        rm -f $VOLUMES_TO_DELETE

}

delete_purged_volumes() {

        echo ""
        echo "searching for all purged volumes to be deleted..."
        echo ""

        PURGED_VOLUMES=`echo "list volumes" | bconsole | grep "Purged" | awk {'print $4'}`
        echo "$PURGED_VOLUMES"

        echo ""
        echo -n "Are you sure you want to delete all these purged volumes from Bacula catalog ? (yes|no): "
        read response

        if [ $response = "yes" ]; then
                BACULA_BATCH="$(mktemp)"
                rm -f /tmp/bacula.log
                echo "@output /tmp/bacula.log" > $BACULA_BATCH
                echo "$PURGED_VOLUMES" | while read I; do
                        echo "delete volume=$I" >> $BACULA_BATCH
                        echo "yes" >> $BACULA_BATCH
                done

                bconsole < $BACULA_BATCH
                rm -f $BACULA_BATCH
                echo "Done. See the log file /tmp/bacula.log"
        fi

}

# Parse parameters
while [ $# -gt 0 ]; do
    case "$1" in
        -h | --help)
                print_usage
                exit 0
                ;;
        -p)
                shift
                prune_all
                exit 0
                ;;
        -t)
                shift
                update_volume Truncate
                exit 0
                ;;
        -n)
                shift
                update_volume None
                exit 0
                ;;
        -Do)
                shift
                delete_obsolete_volumes
                exit 0
                ;;
        -Dp)
                shift
                delete_purged_volumes
                exit 0
                ;;
        *)  echo "Unknown argument: $1"
            print_usage
            exit 3
            ;;
        esac
done
exit 1

Posted in Backups


Leave a Reply

Your email address will not be published. Required fields are marked *

Bad Behavior has blocked 83 access attempts in the last 7 days.