bash script to verify ZIP or RAR archives

Bash script that will check all ZIP files in the current directory and all sub-directories. Valid files are left alone. Files with problems (encrypted, password protected, CRC errors, not a ZIP file) get renamed with various extensions.

#!/bin/bash

find . -type f -name '*.zip' -print0 | while read -d $'\0' file
do
    # skip loop iteration if file no longer exists
    if [[ ! -f "$file" ]] ; then continue; fi
    #echo "$file"
    
    unzip -t -qq "$file"
    RETVAL=$?

    # return values (error codes) are generally unique to each archive tool
    case $RETVAL in
        # 0=success, archive file is okay
        0) 
            echo "OK: $file"
            ;;
        
        # codes that indicate a broken archive
        3) 
            echo "BROKEN($RETVAL): $file"
            mv "$file" "$file.broken"
            ;;
            
        # codes that indicate file is not a supported archive
        9) 
            echo "WRONGFORMAT($RETVAL): $file"
            mv "$file" "$file.wrongformat"
            ;;
            
        # user pressed ctrl-C or break or killed the process    
        80) 
            echo "USER ABORT: $file"
            ;;
            
        # codes that indicate an encrypted archive
        81) 
            echo "ENCRYPTED($RETVAL): $file"
            mv "$file" "$file.encrypted"
            ;;
            
        # codes that indicate a password-protected archive
        82) 
            echo "PASSWORD($RETVAL): $file"
            mv "$file" "$file.password"
            ;;
            
        # other error codes
        *) 
            echo "ERROR($RETVAL): $file"
            ;;
    esac
done

Same script, but changed to handle RAR files. The error codes are defined as an enum in "errhnd.hpp". An approximate list is:

SUCCESS=0, WARNING=1, FATAL_ERROR=2, CRC_ERROR=3, LOCK_ERROR=4, WRITE_ERROR=5, OPEN_ERROR=6, USER_ERROR=7, MEMORY_ERROR=8, CREATE_ERROR=9, NO_FILES_ERROR=10, USER_BREAK=255

Code 3 is commonly returned if files within the RAR archive have errors. Code 10 may indicate that the RAR archive is not actually a RAR file (it might be a misnamed ZIP or 7Z archive).

#!/bin/bash

find . -type f -name '*.rar' -print0 | while read -d $'\0' file
do
    # skip loop iteration if file no longer exists
    if [[ ! -f "$file" ]] ; then continue; fi
    #echo "$file"
    
    unrar t -idq "$file"
    RETVAL=$?
    
    case $RETVAL in
        # 0=success, archive file is okay
        0) 
            echo "OK: $file"
            ;;
        
        # codes that indicate a broken archive
        3) 
            echo "BROKEN($RETVAL): $file"
            mv "$file" "$file.broken"
            ;;
            
        # probably indicates wrong format (ZIP or 7Z file as RAR)
        10) 
            echo "WRONGFORMAT($RETVAL): $file"
            mv "$file" "$file.wrongformat"
            ;;
            
        # user pressed ctrl-C or break or killed the process    
        255) 
            echo "USER ABORT: $file"
            ;;
            
        # other errors
        *) 
            echo "ERROR($RETVAL): $file"
            ;;
    esac
done

Both scripts work fine under Cygwin, but in order to get "unrar" for Cygwin you will have to download the UnRar portable source from RARLabs.

As a final note, here the script to undo the name change, for cases where the script doesn't work as expected. It searches for any files ending in '.broken' and strips that portion of the file name back off. This can be used for other mass-renaming activities with minor edits. It may be useful to change sed to use ':' as the delimiter instead of '/' (i.e. sed 's:.broken$::g') to make things easier to write.

#!/bin/bash

find . -type f -name '*.broken' -print0 | while read -d $'\0' file
do
    # skip loop if file no longer exists
    if [[ ! -f "$file" ]] ; then continue; fi
    #echo "$file"
    
    NEWF=$(echo "$file" | sed 's/.broken$//g')
    mv "$file" "$NEWF"
done

3ware 9650SE failing RAID6 array in Linux

I'm using a 3ware 9650SE 16-port controller in my Linux server, setup as a big single-disk RAID6 array. However, it's currently operating in degraded mode due to at least one disk failure. See 3ware RAID maintenance with tw_cli for a link to the 3ware documentation PDF. See also Fixing a degraded disk on the 3ware raid (blur).

# tw_cli help

Copyright (c) 2009 AMCC
AMCC/3ware CLI (version 2.00.09.012)

Commands  Description
-------------------------------------------------------------------
show      Displays information about controller(s), unit(s) and port(s).
flush     Flush write cache data to units in the system.
rescan    Rescan all empty ports for new unit(s) and disk(s).
update    Update controller firmware from an image file.
commit    Commit dirty DCB to storage on controller(s).     (Windows only)
/cx       Controller specific commands.
/cx/ux    Unit specific commands.
/cx/px    Port specific commands.
/cx/phyx  Phy specific commands.
/cx/bbu   BBU specific commands.                               (9000 only)
/cx/ex    Enclosure specific commands.                       (9690SA only)
/ex       Enclosure specific commands.                      (9KSX/SE only)

Certain commands are qualified with constraints of controller type/model
support.  Please consult the tw_cli documentation for explanation of the
controller-qualifiers.

Type help  to get more details about a particular command.
For more detail information see tw_cli's documentation. 

So if we take a look at the drives installed:

# tw_cli show

Ctl   Model        (V)Ports  Drives   Units   NotOpt  RRate   VRate  BBU
------------------------------------------------------------------------
c6    9650SE-16ML  16        7        1       1       4       4      1  

The columns are as follows:

Ports - # of drive ports on the card
Drives - # of drives connected
Units - # of RAID units created on the card
NotOpt - "not optimal"
RRate - "rebuild rate"
VRate - ???
BBU - Battery backup

# tw_cli /c6 show

Unit  UnitType  Status         %RCmpl  %V/I/M  Stripe  Size(GB)  Cache  AVrfy
------------------------------------------------------------------------------
u0    RAID-6    DEGRADED       -       -       64K     4889.37   OFF    OFF    

Port   Status           Unit   Size        Blocks        Serial
---------------------------------------------------------------
p0     NOT-PRESENT      -      -           -             -
p1     NOT-PRESENT      -      -           -             -
p2     NOT-PRESENT      -      -           -             -
p3     NOT-PRESENT      -      -           -             -
p4     NOT-PRESENT      -      -           -             -
p5     NOT-PRESENT      -      -           -             -
p6     OK               u0     698.63 GB   1465149168    5QD4####            
p7     OK               u0     698.63 GB   1465149168    3QD0####            
p8     NOT-PRESENT      -      -           -             -
p9     OK               u0     698.63 GB   1465149168    3QD0####            
p10    NOT-PRESENT      -      -           -             -
p11    OK               u0     698.63 GB   1465149168    5QD3####            
p12    OK               u0     698.63 GB   1465149168    3QD0####            
p13    OK               u0     698.63 GB   1465149168    5QD4####            
p14    OK               u0     698.63 GB   1465149168    3QD0####            
p15    NOT-PRESENT      -      -           -             -

As you can see, port 8 and port 10 have failed. Which means our RAID6 array is in dire shape. After testing, one of the units had failed completely, the other is merely suspect and was put back into the array. I did the rebuild in the BIOS, but when rebuilding, you will see the following:

# tw_cli /c6/u0 show all
/c6/u0 status = DEGRADED-RBLD
/c6/u0 is rebuilding with percent completion = 13%(A)
/c6/u0 is not verifying, its current state is DEGRADED-RBLD
/c6/u0 is initialized.
/c6/u0 Write Cache = off
/c6/u0 volume(s) = 1
/c6/u0 name = vg6                  
/c6/u0 serial number = 5QD40L3K00005F00#### 
/c6/u0 Ignore ECC policy = off       
/c6/u0 Auto Verify Policy = off       
/c6/u0 Storsave Policy = protection  
/c6/u0 Command Queuing Policy = on        
/c6/u0 Parity Number = 2         

Unit     UnitType  Status         %RCmpl  %V/I/M  Port  Stripe  Size(GB)
------------------------------------------------------------------------
u0       RAID-6    DEGRADED-RBLD  13%(A)  -       -     64K     4889.37   
u0-0     DISK      OK             -       -       p14   -       698.481   
u0-1     DISK      OK             -       -       p13   -       698.481   
u0-2     DISK      OK             -       -       p12   -       698.481   
u0-3     DISK      OK             -       -       p11   -       698.481   
u0-4     DISK      DEGRADED       -       -       p10   -       698.481   
u0-5     DISK      OK             -       -       p9    -       698.481   
u0-6     DISK      DEGRADED       -       -       -     -       698.481   
u0-7     DISK      OK             -       -       p7    -       698.481   
u0-8     DISK      OK             -       -       p6    -       698.481   
u0/v0    Volume    -              -       -       -     -       4889.37

Specifically, we can see that the array is 13% through with the rebuild after only 32 minutes. I have not yet replaced port-8 as I'm going to wait for the array to finish rebuilding before I jostle it again.

Notes:

I strongly recommend that you feed the output of "tw_cli /c# show" and "tw_cli /c#/u# show all" into text files daily and parse them for issues. Or mail them to a monitoring email address. Being able to tell the technician to pull drive XYZ with a specific serial # helps eliminate errors. But that's hard to do if you don't keep track of your serial numbers.

On the systems I administer, we have a /reports/configuration folder where we consolidate all those types of reports. Things like the output of pvscan, lvscan, df, ntpq, /proc/mdstat, etc. all get dumped into text files daily and then committed to the central SVN repository for the server with FSVS. When things go bad later, we can step back through the SVN repository and look at the various reports at previous points in time.

Example of bad capacitors from a few years ago

A few years back, we bought a bunch of GeForce 6200LE PCIe video cards for various uses in servers/desktops.  I liked them at the time because they are fanless (one less moving part to break).  However, in the last 2 years, we've had a lot of them fail due to bad capacitors. 

And when one of these capacitors "pops", it sounds like a small firecracker going off in the room.  Very noticeable at the time. Reminds me of the old miniature pop-caps or the small paper bags of a single grain of gunpowder (or flash powder?) mixed in with a few rocks (the size of a pea) that you could throw at hard surfaces and it would make a popping noise.

FSVS: Updated install on CentOS 5.5

(Also see my older post on this: FSVS - Install on CentOS 5 or FSVS - Install on CentOS 5.4. Or the original post where I explained the power of FSVS for sysadmins.)

Once again, I'm starting with the assumption that this is a pretty bare-bones CentOS 5.5 server install, with only the "server" package group (or no package groups at all) being selected during the initial install.  The basic steps remain the same for situations where you merely want to use FSVS and SVN to keep track of changes to your system:

1. Setup the RPMForge repository
2. Install the packages needed for FSVS
3. Download and compile FSVS
4. Configure ignore patterns
5. Do the base check-ins

For the most part, I try to do this process as early in the lifespan of the server as possible.  But there's always a few minor things that get done before I get this far (creating an initial user, doing a "yum update" to patch the system up, etc).  Even a mature server can benefit from adding FSVS, but you'll find it much more useful the longer that you use it as it gives you a quick index to "what changed and why did you change it".

Setting up RPMForge

In order to get the latest Subversion packages for your system, you'll have to add RPMForge as a source repository. The CentOS base repository only has Subversion 1.4.2 and the latest is currently 1.6.12. I recommend doing this in conjunction with the yum-priorities package.

# yum install yum-priorities

After installing the yum-priorities package, you should edit the CentOS-Base.repo file found under /etc/yum.repos.d/. For the base repositories, I recommend setting them to priority values of 1 through 3  For example, in the "[base]" section, you would add (2) lines to the end of that section:

[base]
...
priority=1
exclude=subversion-*

The "priority=" tells yum-priorities that if it finds a package in multiple repositories, that [base] should take precedence.  The "exclude=" line keeps us from pulling Subversion from the [base] repository (and we'll instead pull it from RPMForge).

Now we can install the RPMForge repository (see Using RPMForge).  You'll need to look at the release folder in order to get the RPM name.

# cd /root/

# mkdir software

# cd software
# wget http://packages.sw.be/rpmforge-release/rpmforge-release-0.5.1-1.el5.rf.x86_64.rpm
# rpm -Uhv rpmforge-release-0.5.1-1.el5.rf.x86_64.rpm
# cd /etc/yum.repos.d/

Now you should edit the rpmforge.repo file and insert a priority= line. I recommend a value of 10 or 25. In addition, I suggest also telling the yum package manager to only pull in specific packages from RPMForge (or any 3rd party repository).  This is a bit of overkill, but yum-priorities is a bit of overkill in itself and I've run into issues in the past where priorities weren't enough and I wish I had gone with using "includepkgs" instead.  (Note that while "exclude" is used to exclude packages, the opposite term is "includepkgs" and not "include".)  This will make the end of your rpmforge.repo file look like:

[rpmforge]

...
priority=10
includepkgs=subversion-*

You can verify that you'll pull in the latest Subversion package by the following command:

# yum info subversion
Available Packages
Name       : subversion
Arch       : x86_64
Version    : 1.6.12
Release    : 0.1.el5.rf
Size       : 6.8 M
Repo       : rpmforge
Summary    : Modern Version Control System designed to replace CVS
URL        : http://subversion.tigris.org/
License    : BSD

Install the packages needed for FSVS

# yum install subversion subversion-devel ctags apr apr-devel gcc gdbm gdbm-devel pcre pcre-devel apr-util-devel

Download and compile FSVS

As always, you shouldn't compile code as the root user.

# su username
$ mkdir -p ~/software/fsvs
$ cd ~/software/fsvs
$ wget http://download.fsvs-software.org/fsvs-1.2.2.tar.bz2
$ tar xjf fsvs-1.2.2.tar.bz2
$ cd fsvs-1.2.2
$ ./configure
$ make
$ exit
# cp /home/username/fsvs/fsvs-1.2.2/src/fsvs /usr/local/bin/
# chmod 755 /usr/local/bin/fsvs

Creating the repository on the SVN server

This is how we setup users on our SVN server. Machine accounts are prefixed as "sys-" in front of the machine name. The SVN repository name matches the name of the machine. In general, only the machine account should have write access to the repository, although you may wish to add other users to the group so that they can gain read-only access.

# useradd -m sys-www-test
# passwd sys-www-test
# svnadmin create /var/svn/sys-www-test
# chmod -R 750 sys-www-test
# chmod -R g+s sys-www-test/db
# chown -R sys-www-test:sys-www-test sys-www-test

Back on the source machine (our test machine), we'll need to create an SSH key that can be used on our SVN server. You may wish to use a slightly larger RSA key (3200 bits or 4096 bits) if you're working on an extra sensitive server. But a key size of 2048 bits should be secure for another decade for this purpose.

# cd /root/
# mkdir .ssh
# chmod .ssh 700
# cd .ssh
# /usr/bin/ssh-keygen -N '' -C 'svn key for root@hostname' -t rsa -b 2048 -f root@hostname
# chmod 600 *
# cat root@hostname.pub

Copy this key into the clipboard or send it to the SVN server or the SVN server administrator. Then we'll need to create a ~/.ssh/config file to tell the user what account name, port and key file to use when talking to the SVN server.

# vi /root/.ssh/config
Host svn.tgharold.com
Port 22
User sys-www-test
IdentityFile /root/.ssh/root@hostname
# chmod 600 *

Back on the SVN server, you'll need to finish configuration of the user that will add files to the SVN repository.

# su username
$ cd ~/
$ mkdir .ssh
$ chmod 700 .ssh
$ cd .ssh
$ cat >> authorized_keys
(paste in the SSH key from the other server)
$ chmod 600 *

Now you'll want to prepend the following in front of the key line in the authorized_keys file.

command="/usr/bin/svnserve -t -r /var/svn",no-agent-forwarding,no-pty,no-port-forwarding,no-X11-forwarding

That ensures (mostly) that the key can only be used to run the svnserve command and that it can't be used to access a command shell on the SVN server. Test the configuration back on the original server by issuing the "svn info" command. Alternately, you can try to ssh to the SVN repository server. Errors will usually either be logged in /var/log/secure on the source server or in the same log file on the SVN repository server. Here's an example of a successful connection:

# ssh svn.tgharold.com
( success ( 2 2 ( ) ( edit-pipeline svndiff1 absent-entries commit-revprops depth log-revprops partial-replay ) ) )

This shows that they key is running the "svnserve" command automatically.

Connect the system to the SVN repository

The very first command that you'll need to issue for FSVS is the "urls" (or "initialize") command. This tells FSVS what repository will be used to store the files.

# cd /
# mkdir /var/spool/fsvs
# mkdir /etc/fsvs/
# fsvs urls svn+ssh://svn.tgharold.com/sys-www-test/

You may see the following error, which means you need to create the /var/spool/fsvs folder, then reissue the fsvs urls command.

stat() of waa-path "/var/spool/fsvs/" failed. Does your local WAA storage area exist?

The following error means that you forgot to create the /etc/fsvs/ folder.

Cannot write to the FSVS_CONF path "/etc/fsvs/".

Configure ignore patterns and doing the base check-in

When constructing ignore patterns, generally work on adding a few directories at a time to the SVN repository. Everyone has different directories that they won't want to version, so you'll need to tailor the following to match your configuration. However, I generally recommend starting with the following (this is the output from "fsvs ignore dump", which you can pipe into a file, edit, then pipe back into "fsvs ignore load"):

group:ignore,./backup/
group:ignore,./bin/
group:ignore,./dev/
group:ignore,./etc/fsvs/
group:ignore,./etc/gconf/
group:ignore,./etc/gdm/
group:ignore,./home/
group:ignore,./lib/
group:ignore,./lib64/
group:ignore,./lost+found
group:ignore,./media/
group:ignore,./mnt/
group:ignore,./proc/
group:ignore,./root/
group:ignore,./sbin/
group:ignore,./selinux/
group:ignore,./srv/
group:ignore,./sys/
group:ignore,./tmp/
group:ignore,./usr/bin/
group:ignore,./usr/include/
group:ignore,./usr/kerberos/
group:ignore,./usr/lib/
group:ignore,./usr/lib64
group:ignore,./usr/libexec/
group:ignore,./usr/sbin/
group:ignore,./usr/share/
group:ignore,./usr/src/
group:ignore,./usr/tmp/
group:ignore,./usr/X11R6/
group:ignore,./var/cache/
group:ignore,./var/gdm/
group:ignore,./var/lib/
group:ignore,./var/lock/
group:ignore,./var/log/
group:ignore,./var/mail/
group:ignore,./var/run/
group:ignore,./var/spool/
group:ignore,./var/tmp/

Then you'll either want to ignore (or encrypt) the SSH key files.

# cd /
# fsvs ignore group:ignore,./root/.ssh
# fsvs ignore group:ignore,./etc/ssh/shadow*
# fsvs ignore group:ignore,./etc/ssh/ssh_host_key
# fsvs ignore group:ignore,./etc/ssh/ssh_host_dsa_key
# fsvs ignore group:ignore,./etc/ssh/ssh_host_rsa_key

You can check what FSVS is going to version by using the "fsvs status pathname" command (such as "fsvs status /etc"). Once you are happy with the selection in a particular path, you can do the following command:

# fsvs ci -m "base check-in" /etc

Repeat this for the various top level trees until you have checked everything in. Then you should do one last check-in at the root level that catches anything you might have missed.

GRUB manual install into the boot sector

One of the useful things to know is how to install GRUB onto a disk when you replace one in the software RAID.

Installing GRUB natively

Caution: Installing GRUB's stage1 in this manner will erase the normal boot-sector used by an OS. 

# grub
grub> device (hd0) /dev/sdf
grub> root (hd0,0)
grub> find /grub/stage1
grub> setup (hd0)
grub> quit

Error: task md#_resync:### blocked for more than ### seconds

While building a new machine using CentOS 5.5, found this in the log files.  It also gets spammed to the console every 2-3 minutes.

kernel: INFO: task md1_resync:434 blocked for more than 120 seconds.
kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
kernel: md1_resync    D ffff8100bebbde30     0   434     35           437   433 (L-TLB)
kernel:  ffff8100bebbdd70 0000000000000046 0000000000000000 ffff8100bed6300c
kernel:  ffff8100bed6340c 0000000000000009 ffff8100beb9d100 ffff8100bff937a0
kernel:  00000011b743e63b 000000000005ad3b ffff8100beb9d2e8 0000000000000000
kernel: Call Trace:
kernel:  [] keventd_create_kthread+0x0/0xc4
kernel:  [] md_do_sync+0x1d8/0x833
kernel:  [] enqueue_task+0x41/0x56
kernel:  [] __activate_task+0x56/0x6d
kernel:  [] dequeue_task+0x18/0x37
kernel:  [] thread_return+0x62/0xfe
kernel:  [] autoremove_wake_function+0x0/0x2e
kernel:  [] keventd_create_kthread+0x0/0xc4
kernel:  [] md_thread+0xf8/0x10e
kernel:  [] keventd_create_kthread+0x0/0xc4
kernel:  [] md_thread+0x0/0x10e
kernel:  [] kthread+0xfe/0x132
kernel:  [] child_rip+0xa/0x11
kernel:  [] keventd_create_kthread+0x0/0xc4
kernel:  [] kthread+0x0/0x132
kernel:  [] child_rip+0x0/0x11

There's a Red Hat Bugzilla entry (#18061) on the topic.  Basically, it's because I have Linux software RAID in the process of synchronizing the freshly created arrays.  On the disks in the machine, I have each one divided up into (3) partitions, and each partition slice belongs to a different mdadm RAID.  Since mdadm is smart enough not to thrash the disks, any arrays which have not yet been synchronized will be put into a "DELAYED" mode

Personalities : [raid1]
md0 : active raid1 sdc1[2] sdb1[1] sda1[0]
      256896 blocks [3/3] [UUU]
     
md2 : active raid1 sdc3[2] sdb3[1] sda3[0]
      455892480 blocks [3/3] [UUU]
      [=>...................]  resync =  6.1% (27871296/455892480) finish=323.7min speed=22033K/sec
     
md1 : active raid1 sdc2[2] sdb2[1] sda2[0]
      32234304 blocks [3/3] [UUU]
        resync=DELAYED
     
unused devices:

The error is basically a false positive and no harm is being done except for the message spam on the console and in the system log files.  It will happen every time that this system does the weekly array sync, however (until it gets fixed).

Server migration to LogicWeb

Due to JTLNetworks being clueless about how to reliably deliver e-mail, I've severed my use of their services and moved to LogicWeb. But, Blogger.com has also changed how their service works, so all of my old blog URLs are going to result in 404 errors for a bit.

The new URL for this blog is now:

http://techblog.tgharold.com/

(So far, I'm a lot happier with LogicWeb then with JTL. Mostly because LogicWeb gives me secure IMAP storage of e-mail over SSL.)

Migrate from Direct Partitions to LVM Volumes

Weekend Project: Migrate from Direct Partitions to LVM Volumes by Nathan Willis

This tutorial covers the basics of migrating from regular disk partitions to LVM logical volumes (LVs). It displays some of the strengths of LVM, where you can resize logical volumes, or move logical volumes to another physical device in the same volume group.

Setting up NX Client access to Linux servers running FreeNX/NX

In order to talk to a Linux box running FreeNX/NX server...

1) Grab a copy of the NX client from NoMachine's download page. For Windows users, look for "NX Client for Windows".

2) Install the client software.

3) Start up "NX Client for Windows".

4) The "Login" field will be your username on the Linux server (i.e. "thomas"). The "Password" field is obviously, your login password on the Linux server. In the "Session" field, enter a descriptive name of the server that you are trying to connect to. (This name does not have to reflect the server's DNS name, IP address and can be any name or nickname.)

5) As you press [Tab] to leave the "Session" field, you will be prompted whether you want to "Create" a new session or "Rename" the existing session. In most cases, you'll want to pick "Create" so that you can setup a new session.

6) You will now be taken to the advanced configuration screen.

a) Server Host: Enter the fully qualified DNS name, or the IP address.
b) Server Port: Most servers run SSH on port 22.
c) Desktop: I usually use "Unix / Gnome"
d) Speed: ISDN or ADSL is a good first choice
e) Display: Choose full-screen or setup a window of any size that you want.

7) Before you click "Save" or "OK", check with your server administrator and find out whether you need to provide the NX client with a custom key. This key is located on the Linux server and is usually at the following location:

/etc/nxserver/client.id_dsa.key

If you need this key, be sure to obtain it, then use the "Server, Key" button on the advanced configuration screen. Just paste the key in to the provided box and you'll be able to talk to the server.

8) Click "Save" then "OK".

9) If everything is configured properly, you can now enter your login password and press the "Login" button to access the remote server.

...

When you are finished with your session on the remote server, you can either "Terminate" (logout) or "Suspend" it if you want to leave things running for later.

Why I don't trust JTL Networks

Here's an example of the spurious error messages I get from the ever-so-helpful people over at JTL Networks.

For the record - they've been ignoring my requests to have this error looked into and addressed for over a year now. Very very poor customer service and technical support over there.

Return-Path: <>
Received: (qmail 26238 invoked for bounce); 13 Jun 2010 23:45:18 -0000
Date: 13 Jun 2010 23:45:18 -0000
From: MAILER-DAEMON@apache.org
To: announce-return-133-@httpd.apache.org
Subject: failure notice

Hi. This is the qmail-send program at apache.org.
I'm afraid I wasn't able to deliver your message to the following addresses.
This is a permanent error; I've given up. Sorry it didn't work out.

<tgh@tgharold.com>:
69.36.9.168 does not like recipient.
Remote host said: 556 Address unavailable.
Giving up on 69.36.9.168.