RHEL: yum update - rpmforge causes segmentation fault

So, rpmforge is throwing errors today that look like:

# yum info rpmforge-release
Loaded plugins: rhnplugin, security
rpmforge   | 1.1 kB     00:00     
rpmforge: [####    ] 471/10722Segmentation fault

This happens any time that you do a 'yum check-update', 'yum update', 'yum search packagename', etc. The issue also happens with rpmforge-extras on this machine (if I disable the rpmforge repository). Disabling the rpmforge and rpmforge-extras repositories does temporarily fix the issue, but it means you cannot get updates from rpmforge / rpmforge-extras until it is fixed.

The following will not fix the issue (nor will most of the suggestions at the end):

# yum clean all
# yum makecache

Unfortunately, all of the search results are turning up either link-farms or issues not related to the yum software. But I'm still looking for answers.  (See the end of the post for the latest information.)

Suggestion #1: /var/lib/rpm/__db.* (from Jan 2007)

# cd /var/lib/rpm/
# rm __db.*
(that's two underscores)
# rpm --rebuilddb

Unfortunately, it didn't fix this particular issue.

Suggestion #2: /var/cache/yum/

Remove the rpmforge and rpmforge-extras directory, then have yum rebuild its cache (using either "yum update" or "yum makecache"). No luck here either for this particular issue.

Suggestion #3: uninstall / reinstall rpmforge-release RPM

# rpm -e rpmforge-release
# rpm -i rpmforge-release-0.5.2-2.el6.rf.*.rpm

No real luck there either.

Suggestion #4: Related to a recent "yum update"

As others here have noted, there was a large "yum update" (for the 5.7 release) this week on our RHEL5 server. An update which has not yet reached our CentOS 5 boxes, which are not misbehaving. Looking at the list of what was updated, there were 142 packages in this week's update. Some of the possibly interesting ones to this issue might be:

yum-security-1.1.16-16.el5.noarch.rpm
rhnsd-4.7.0-10.el5.x86_64.rpm
rhn-check-0.4.20-56.el5.noarch.rpm
authconfig-gtk-5.3.21-7.el5.x86_64.rpm
zlib-1.2.3-4.el5.i386.rpm
zlib-1.2.3-4.el5.x86_64.rpm
redhat-release-5Server-5.7.0.3.x86_64.rpm
yum-rhn-plugin-0.5.4-22.el5.noarch.rpm
rhnlib-2.5.22-6.el5.noarch.rpm
zlib-devel-1.2.3-4.el5.i386.rpm
zlib-devel-1.2.3-4.el5.x86_64.rpm
rhn-setup-0.4.20-56.el5.noarch.rpm
yum.noarch 0:3.2.22-37.el5

The list of packages that rpmforge-release depends on:

# rpm -qR rpmforge-release
/bin/sh  
config(rpmforge-release) = 0.5.1-1.el5.rf
rpmlib(CompressedFileNames) <= 3.0.4-1
rpmlib(PayloadFilesHavePrefix) <= 4.0-1

The packages that yum depends on:

# rpm -qR yum
/usr/bin/python  
config(yum) = 3.2.22-33.el5.centos
python >= 2.4
python(abi) = 2.4
python-elementtree  
python-iniparse  
python-sqlite  
rpm >= 0:4.4.2
rpm-python  
rpmlib(CompressedFileNames) <= 3.0.4-1
rpmlib(PayloadFilesHavePrefix) <= 4.0-1
rpmlib(VersionedDependencies) <= 3.0.3-1
urlgrabber >= 3.1.0
yum-fastestmirror  
yum-metadata-parser >= 1.1.0

Off-hand, and since the compression library has caused issues in the past, I'm thinking maybe an issue with zlib. Update: Someone has posted a stack trace below in the comments which points to an issue other then zlib. A bunch of gcc / glibc files were also updated on my server today:

gcc.x86_64 0:4.1.2-51.el5
gcc-c++.x86_64 0:4.1.2-51.el5
gcc-gfortran.x86_64 0:4.1.2-51.el5
gdb.x86_64 0:7.0.1-37.el5
glibc.i686 0:2.5-65
glibc.x86_64 0:2.5-65
glibc-common.x86_64 0:2.5-65
glibc-devel.i386 0:2.5-65
glibc-devel.x86_64 0:2.5-65
glibc-headers.x86_64 0:2.5-65

Suggestion #5 or Update #5:

Looks like a bug.  Reports on the CentOS-Users and RPMForge user's mailing lists are indicating that they are starting to get to the bottom of the issue. There's an issue with a package in the RPMForge repository with strange/incorrect metadata and that's causing bad things to happen in yum.

See Bug 725798 - yum segmentation fault on rpmforge repository over at Red Hat's bug tracker.

Latest posts from the mailing lists indicate that both issues are being addressed today.  So you can temporarily disable rpmforge / rpmforge-extras and do your other updates, or delay all updates for a day or two until it is fixed.

Update #6:

Everything now works correctly again. You may want to run a "yum clean all" and "yum makecache" to refresh your metadata cache files.


.

Postfix: Installing from source on RHEL 5.6

Right now, RHEL 5.6 ships with Postfix 2.3. Which is okay, but its an extremely old version (even with Red Hat's backporting of security fixes) so we decided to go with Postfix 2.8. The major downside of installing from source is that it is a manual process that will have to be repeated every time we see a new Postfix release. Which leads me to the first warning.

Warning #1: If you're going to install a package as complex as Postfix from source code, you really need to take good notes, keep log files from the console and make sure that you understand the process. Some sort of version control for your configuration files should be considered mandatory and you may want to put all of the Postfix binaries and the source / build tree into version control. I strongly suggest the use of FSVS and the repository size on a modern RHEL5 install will generally be in the 500MB to 2GB range (depending on what you include/exclude).

Reference links:

Postfix Announcements

Building Postfix 2.8 on RHEL5 / CentOS 5 from Source

Postfix Installation From Source Code

Getting the source:

Before installing from source, I recommend installing the 2.3 version that ships with RHEL5. This will make sure that the Postfix users and groups are already created and that other directories exist. And it can make configuration easier later.

Since there are no binary RPMs available for Postfix 2.8 combined with the way Postfix includes/excludes major features means that you will almost always build from source. Head over to the postfix.org downloads section and find a mirror close to you.

1. Create a folder where you will keep packages that are built from source. The standard location for files like this is "/usr/local/src" on RHEL5. Different Linux distributions may have alternate locations for source code.
   
2. Download the postfix-2.8.2.tar.gz (cd /usr/local/src ; wget URL)
   
3. Extract the tar.gz file with "tar zxvf filename", this will create a postfix-2.8.xyz directory.
   
4. Warning #2: Never build source as "root". Create an administrator / software install group, assign users to it, then use one of those users to do the compile.
   
5. Change the group ownership of the postfix folder (chgrp -R) to a group of administrators for the system. Change the permissions (chmod -R g+w) so that members of that group can read/write the files.
   

Optional Modules:
 
In order to compile in optional modules like SASL, PostgreSQL support, PCRE (Perl Compatible Regular Expressions), or TLS, you have to configure the Makefile so that the compiler and linker tools know where to look and so that certain constants are set prior to compiling. This is done using the "make makefiles options" command where you supply compiler/linker options and the Makefile is changed to reflect those options.

Note: If you need to include multiple optional modules, then you must create a monolithic "make makefiles" command instead of trying to turn on SASL, then turn on PCRE separately. However, it actually makes a lot of sense for debugging purposes to do test compiles after turning on a single optional module to make sure that you specified the "make makefiles" command properly. After testing the individual module inclusion, you will have to rewrite the commands so that all of the "CCARGS" options are grouped together, then all of the "AUXLIBS" need to be grouped together.

For example, the following command turns on PostgreSQL support in the Postfix source code. The command starting with 'make' should be all one line.

$ cd /usr/local/src/postfix-2.8-xyz
$ make -f Makefile.init makefiles 'CCARGS=-DHAS_PGSQL \
-I/usr/pgsql-9.0/include' \
'AUXLIBS=-L/usr/pgsql-9.0/lib -lpq'

The "-DHAS_PGSQL" is a constant that gets set during compile time that tells the compiler to include any code blocks in the postfix source code tagged with "HAS_PGSQL". (In C code this is done with #ifdef directives.) If this constant is not defined, then postfix will not know how to talk to a PostgreSQL database server.

The "-I/usr/pgsql-9.0/include" section tells the compiler where to find the header files for postfix such as "pg_config.h". If you don't have the PostgreSQL header files installed, then you probably need to install the "postgresql-devel" or "postgresql90-devel" package. If you don't know where the header files are located, try searching the filesystem (using "locate") for "pg_config.h" or "postgres_ext.h".

In the AUXLIBS section, "-L/usr/pgsql-9.0/lib" tells the linker where to find the compiled PostgreSQL libraries such as "libpq.so". These are installed as part of the "-devel" package for PostgreSQL. And their location can be easily found by searching for "libpg.so".
After figuring out where the include modules and libraries are for the optional module, you should do a "make makefiles options" step and then do a test compile by running "make" without any arguments. If this works, make a careful note of the command line options.
Once you have tested out all of the optional modules that you plan on turning on, then you can assemble the final "make makefiles options" command before doing the final compile of Postfix.

The following works on RHEL 5.6 64bit and turns on SASL, PCRE, PostgreSQL and TLS.

$ cd /usr/local/src/postfix-2.8.xyz 

$ make makefiles \
CCARGS='-fPIC -DUSE_SASL_AUTH -DDEF_SERVER_SASL_TYPE=\"dovecot\" \
-DUSE_TLS -DHAS_PCRE -I/usr/local/include -DHAS_PGSQL \
-I/usr/pgsql-9.0/include' \
AUXLIBS='-lssl -lcrypto -L/usr/local/lib -lpcre \
-L/usr/pgsql-9.0/lib -lpq'

$ make

Switch away from Sendmail:

In Unix/Linux, you do not want to have more then one local MTA (mail transfer agent) active at the same time. RHEL5 and CentOS5 both implement the "alternatives" command which lets you switch MTAs. Use "alternatives --display mta" to display the current MTA setting and "alternatives --config mta" to change which MTA takes precedence.

Installation:

First off, make sure that yum will not auto-upgrade postfix. This is done by adding a "exclude=postfix*" line to the /etc/yum.conf file. Any packages that match the space-delimited patterns listed in the "exclude=" will never be touched by yum, even if updated packages are rolled out in one of your RPM repositories.

(See section 6.2 in the Postfix install documents.) If you had previously installed Postfix on the system, then the "postfix" user will have already been created. I suggest doing a quick grep of the /etc/passwd and /etc/group files to verify that the user/group already exists.

Now you will want to either "make install" (new installation) or "make upgrade" (existing installation). You will probably (certainly) have to do this as root. Most of the defaults during the interactive install will be fine. The only default I tend to change is to exclude the README_FILES.

# cd /usr/local/src/postfix-2.8.xyz 
# make install 
install_root: [/] (enter) 
tempdir: [/usr/local/src/postfix-2.8.2] (enter) 
config_directory: [/etc/postfix] (enter) 
command_directory: [/usr/sbin] (enter) 
daemon_directory: [/usr/libexec/postfix] (enter) 
data_directory: [/var/lib/postfix] (enter) 
html_directory: [no] (enter) 
mail_owner: [postfix] (enter) 
mailq_path: [/usr/bin/mailq.postfix] (enter) 
manpage_directory: [/usr/share/man] (enter) 
newaliases_path: [/usr/bin/newaliases.postfix] (enter) 
queue_directory: [/var/spool/postfix] (enter) 
readme_directory: [/usr/share/doc/postfix-2.3.3/README_FILES] no (enter) 
sendmail_path: [/usr/sbin/sendmail.postfix] (enter) 
setgid_group: [postdrop] (enter)

After answering all of the questions, the interactive installer will copy files and create the directories. Now you can move on to configuring Postfix and setting the service to automatically start when the server starts.

Changing "ls" color so that directories don't show up in dark blue

One of the annoyances with Linux if you're using a color terminal (or a color terminal emulator like SecureCRT or PuTTY) is that the "ls" (list directory contents) command has a default of dark blue for directory entries.  Which is fine if you're using a white background, but doesn't work well if you prefer an old-school black background for your terminal.

Fortunately, this is configurable with the LSCOLORS environment variable.  The default string is "exfxcxdxbxegedabagacad" for OS X.  However, on Linux, it looks more like you may be able to configure this using /etc/DIR_COLORS instead.

On my CentOS 5.4 machines, DIR_COLORS lists the entry for directories as "DIR 01;34", which means that it uses a bold blue on the default background.  One option to ensure a bit more contrast would be to change this to always print on a white background.  So you would change this entry to look like:

DIR 01;34;47 # NEW default is Bold blue with White background

References:

change ls colour (color) in bash - GlowingApple  explains back in 2005 how to change colors on the MacOSXHints forum

COLORS Lscolors - another page that explains the LSCOLORS variable