md5sum bash script to create check file for a directory tree

Just a quick script that will run through the current directory and all descendant directories, creating a single "verify-tree.md5" file (using md5sum). If the check file already exists, then the existing one gets moved out of the way to "verify-tree.yyyymmdd-hhmmss.md5" for safekeeping.

Check files are useful for whenever you have a set of files that will not (or should not) change over time.  Such as files written off to an archive tape / disk / optical media / flash drive.  While the media and file system might also do some checking, it's good to have a second layer that is under your control and which can be queried.

(If you want file recovery features, you should look into MultiPar or par2j.)

The script's output looks like:

The file (verify-tree.md5) that is created is a standard md5sum file and can be read by just about any compatible software.  Some software cannot handle sub-folders, however, so you may have to use the md5sum program to do the verification.

----------------------------------------------------------------
#!/bin/bash

# stop script on errors
set -e

PROG=md5sum
FILENAME=verify-tree

if [[ -e "${FILENAME}.md5" ]]; then
    mv "${FILENAME}.md5" \
    "${FILENAME}.$(date --reference=${FILENAME}.md5 '+%Y%m%d-%H%M%S').md5"
fi

echo ""
echo "Output Filename: ${FILENAME}.md5"
echo "Files Found: $(find . -type f -not -name "${FILENAME}*.md5" | wc -l)"
echo "Size: $(du -chs . | grep 'total')"

time find . -type f -not -name "${FILENAME}*.md5" \
    -exec ${PROG} "{}" \; >> "${FILENAME}.md5"

echo ""
echo "Files Processed: $(wc -l ${FILENAME}.md5)"

echo ""
echo "Checking..."
time ${PROG} -c --quiet "${FILENAME}.md5"
echo ""
echo "All files verified."

----------------------------------------------------------------

Note: There are multiple ways to write the md5sum line.  It's probably better to use the xargs method, but I have not tested it out.

find . -type f -not -name "${FILENAME}*.md5" -exec ${PROG} "{}" \; >> "${FILENAME}.md5

find . -type f -not -name "${FILENAME}*.md5" -print0 | xargs -0 ${PROG} >> "${FILENAME}.md5"

find . -type f -not -name "${FILENAME}*.md5" | xargs ${PROG} >> "${FILENAME}.md5"

It's also easy to adapt this script to work with sha256sum or sha1sum.

The verification script is a trimmed down version of the original script:

----------------------------------------------------------------

#!/bin/bash

# stop script on errors
set -e

PROG=md5sum
FILENAME=verify-tree

echo ""
echo "Checking: ${FILENAME}.md5"
echo "File Hashes: $(wc -l ${FILENAME}.md5)"
time ${PROG} -c --quiet "${FILENAME}.md5"
echo ""
echo "All files verified."

----------------------------------------------------------------

I have tested this in CygWin 64-bit on Windows 7 64-bit Professional, but it should also work fine on Linux/Unix/OSX systems as long as the md5sum command is available.  The script is conservative in design with very few "tricks" so it should be portable.

bash script to verify ZIP or RAR archives

Bash script that will check all ZIP files in the current directory and all sub-directories. Valid files are left alone. Files with problems (encrypted, password protected, CRC errors, not a ZIP file) get renamed with various extensions.

#!/bin/bash

find . -type f -name '*.zip' -print0 | while read -d $'\0' file
do
    # skip loop iteration if file no longer exists
    if [[ ! -f "$file" ]] ; then continue; fi
    #echo "$file"
    
    unzip -t -qq "$file"
    RETVAL=$?

    # return values (error codes) are generally unique to each archive tool
    case $RETVAL in
        # 0=success, archive file is okay
        0) 
            echo "OK: $file"
            ;;
        
        # codes that indicate a broken archive
        3) 
            echo "BROKEN($RETVAL): $file"
            mv "$file" "$file.broken"
            ;;
            
        # codes that indicate file is not a supported archive
        9) 
            echo "WRONGFORMAT($RETVAL): $file"
            mv "$file" "$file.wrongformat"
            ;;
            
        # user pressed ctrl-C or break or killed the process    
        80) 
            echo "USER ABORT: $file"
            ;;
            
        # codes that indicate an encrypted archive
        81) 
            echo "ENCRYPTED($RETVAL): $file"
            mv "$file" "$file.encrypted"
            ;;
            
        # codes that indicate a password-protected archive
        82) 
            echo "PASSWORD($RETVAL): $file"
            mv "$file" "$file.password"
            ;;
            
        # other error codes
        *) 
            echo "ERROR($RETVAL): $file"
            ;;
    esac
done

Same script, but changed to handle RAR files. The error codes are defined as an enum in "errhnd.hpp". An approximate list is:

SUCCESS=0, WARNING=1, FATAL_ERROR=2, CRC_ERROR=3, LOCK_ERROR=4, WRITE_ERROR=5, OPEN_ERROR=6, USER_ERROR=7, MEMORY_ERROR=8, CREATE_ERROR=9, NO_FILES_ERROR=10, USER_BREAK=255

Code 3 is commonly returned if files within the RAR archive have errors. Code 10 may indicate that the RAR archive is not actually a RAR file (it might be a misnamed ZIP or 7Z archive).

#!/bin/bash

find . -type f -name '*.rar' -print0 | while read -d $'\0' file
do
    # skip loop iteration if file no longer exists
    if [[ ! -f "$file" ]] ; then continue; fi
    #echo "$file"
    
    unrar t -idq "$file"
    RETVAL=$?
    
    case $RETVAL in
        # 0=success, archive file is okay
        0) 
            echo "OK: $file"
            ;;
        
        # codes that indicate a broken archive
        3) 
            echo "BROKEN($RETVAL): $file"
            mv "$file" "$file.broken"
            ;;
            
        # probably indicates wrong format (ZIP or 7Z file as RAR)
        10) 
            echo "WRONGFORMAT($RETVAL): $file"
            mv "$file" "$file.wrongformat"
            ;;
            
        # user pressed ctrl-C or break or killed the process    
        255) 
            echo "USER ABORT: $file"
            ;;
            
        # other errors
        *) 
            echo "ERROR($RETVAL): $file"
            ;;
    esac
done

Both scripts work fine under Cygwin, but in order to get "unrar" for Cygwin you will have to download the UnRar portable source from RARLabs.

As a final note, here the script to undo the name change, for cases where the script doesn't work as expected. It searches for any files ending in '.broken' and strips that portion of the file name back off. This can be used for other mass-renaming activities with minor edits. It may be useful to change sed to use ':' as the delimiter instead of '/' (i.e. sed 's:.broken$::g') to make things easier to write.

#!/bin/bash

find . -type f -name '*.broken' -print0 | while read -d $'\0' file
do
    # skip loop if file no longer exists
    if [[ ! -f "$file" ]] ; then continue; fi
    #echo "$file"
    
    NEWF=$(echo "$file" | sed 's/.broken$//g')
    mv "$file" "$NEWF"
done

Subversion Backups - Finding SVN repositories

The first trick when backing up SVN repositories is finding them so that you can run the svnadmin hotcopy command. Well, you *could* just setup a list of export DIRS in your backup script - but as you add new SVN repositories, you have to constantly edit that script.

Caveat #1: This setup probably only works for FSFS repositories. I don't use BerkleyDB repositories (a.k.a. BDB) so I can't guarantee that it correctly locates them. I've chosen to look for folders that contain the "db/uuid" file as our "marker" file. Which should result in zero false-positives or mis-identified repositories.

Caveat #2: I'm making the assumption that all of your repositories are stored in a central location (/var/svn), but they do *not* all have to be at the same depth.

Step #1 - Find the uuid files

This is pretty simple, we're just going to use find and grep.

# find /var/svn -name uuid | grep "db/uuid"

/var/svn/tgh-photo/db/uuid
/var/svn/tgh-dev/db/uuid
/var/svn/tgh-web/db/uuid

Step #2 - Clean up the pathnames

Even better, we can tack a sed command onto the end to trim off the "/db/uuid" portion, which gets us exactly what we need for passing to the "svnadmin hotcopy" command.

# find /var/svn -name uuid | grep "db/uuid" | sed 's/\/db\/uuid//'

/var/svn/tgh-photo
/var/svn/tgh-dev
/var/svn/tgh-web

(Make sure that you get all of the "\" and "/" in the right places.)

Step #3 - Strip off the base directory

Since I'm going to create a script variable called "BASE" that equals "/var/svn/", I'm also going to strip that off of the front of the paths.

However, we'll need to convert the BASE variable into something that sed can properly deal with. Otherwise the slashes won't be escaped properly for the sed replacement.

# echo "/var/svn/" | sed 's/\//\\\//g'
\/var\/svn\/

# find /var/svn -name uuid | grep "db/uuid" | sed 's/\/db\/uuid//' | sed 's/^\/var\/svn\///'
tgh-photo
tgh-dev
tgh-web

Or, even better, we can use a different delimiter for sed. That gives us a search line of:

DIRS=`find ${BASE} -name uuid | grep 'db/uuid$' | sed 's:/db/uuid$::' | sed 's:^/var/svn/::'`

Which puts a list of directory names into the DIRS varaible in our bash script.

Step #4 - Putting it all together

Here's our basic script.

#!/bin/bash
BASE="/var/svn/"
HOTCOPY="/var/svn-hotcopy/"
DIRS=`find ${BASE} -name uuid | grep 'db/uuid$' | sed 's:/db/uuid$::' | sed 's:^/var/svn/::'`

for DIR in ${DIRS}
do
    echo "svnadmin hotcopy ${BASE}${DIR} to ${HOTCOPY}${DIR}"
    rm -r ${HOTCOPY}${DIR}
    svnadmin hotcopy ${BASE}${DIR} ${HOTCOPY}${DIR}
done

However, we're not quite done yet because the svn hotcopy command doesn't like it when the destination folder does not exist. So let's add the following scrap of code into the loop.

if ! test -d ${HOTCOPY}${DIR}
then
    mkdir -p ${HOTCOPY}${DIR}
fi

The final script

#!/bin/bash

BASE="/var/svn/"
HOTCOPY="/var/svn-hotcopy/"

FIND=/usr/bin/find
GREP=/bin/grep
RM=/bin/rm
SED=/bin/sed
SVNADMIN=/usr/bin/svnadmin

DIRS=`find ${BASE} -name uuid | $GREP 'db/uuid$' | $SED 's:/db/uuid$::' | $SED 's:^/var/svn/::'`

for DIR in ${DIRS}
do  
    echo "svnadmin hotcopy ${BASE}${DIR} to ${HOTCOPY}${DIR}"

    if ! test -d ${HOTCOPY}${DIR}
    then
        mkdir -p ${HOTCOPY}${DIR}
    fi

    $RM -r ${HOTCOPY}${DIR}
    $SVNADMIN hotcopy ${BASE}${DIR} ${HOTCOPY}${DIR}
done

# insert rdiff-backup line here

Hopefully that works out. Note the use of "rm -r", which could cause data loss if there are errors in the script. You will want to be very careful while working on the script.