Tuesday, July 26, 2011

RHEL: yum update - rpmforge causes segmentation fault

So, rpmforge is throwing errors today that look like:

# yum info rpmforge-release
Loaded plugins: rhnplugin, security
rpmforge   | 1.1 kB     00:00     
rpmforge: [####    ] 471/10722Segmentation fault

This happens any time that you do a 'yum check-update', 'yum update', 'yum search packagename', etc. The issue also happens with rpmforge-extras on this machine (if I disable the rpmforge repository). Disabling the rpmforge and rpmforge-extras repositories does temporarily fix the issue, but it means you cannot get updates from rpmforge / rpmforge-extras until it is fixed.

The following will not fix the issue (nor will most of the suggestions at the end):

# yum clean all
# yum makecache

Unfortunately, all of the search results are turning up either link-farms or issues not related to the yum software. But I'm still looking for answers.  (See the end of the post for the latest information.)

Suggestion #1: /var/lib/rpm/__db.* (from Jan 2007)

# cd /var/lib/rpm/
# rm __db.*
(that's two underscores)
# rpm --rebuilddb

Unfortunately, it didn't fix this particular issue.

Suggestion #2: /var/cache/yum/

Remove the rpmforge and rpmforge-extras directory, then have yum rebuild its cache (using either "yum update" or "yum makecache"). No luck here either for this particular issue.

Suggestion #3: uninstall / reinstall rpmforge-release RPM

# rpm -e rpmforge-release
# rpm -i rpmforge-release-0.5.2-2.el6.rf.*.rpm

No real luck there either.

Suggestion #4: Related to a recent "yum update"

As others here have noted, there was a large "yum update" (for the 5.7 release) this week on our RHEL5 server. An update which has not yet reached our CentOS 5 boxes, which are not misbehaving. Looking at the list of what was updated, there were 142 packages in this week's update. Some of the possibly interesting ones to this issue might be:

yum-security-1.1.16-16.el5.noarch.rpm
rhnsd-4.7.0-10.el5.x86_64.rpm
rhn-check-0.4.20-56.el5.noarch.rpm
authconfig-gtk-5.3.21-7.el5.x86_64.rpm
zlib-1.2.3-4.el5.i386.rpm
zlib-1.2.3-4.el5.x86_64.rpm
redhat-release-5Server-5.7.0.3.x86_64.rpm
yum-rhn-plugin-0.5.4-22.el5.noarch.rpm
rhnlib-2.5.22-6.el5.noarch.rpm
zlib-devel-1.2.3-4.el5.i386.rpm
zlib-devel-1.2.3-4.el5.x86_64.rpm
rhn-setup-0.4.20-56.el5.noarch.rpm
yum.noarch 0:3.2.22-37.el5

The list of packages that rpmforge-release depends on:

# rpm -qR rpmforge-release
/bin/sh  
config(rpmforge-release) = 0.5.1-1.el5.rf
rpmlib(CompressedFileNames) <= 3.0.4-1
rpmlib(PayloadFilesHavePrefix) <= 4.0-1
The packages that yum depends on:
# rpm -qR yum
/usr/bin/python  
config(yum) = 3.2.22-33.el5.centos
python >= 2.4
python(abi) = 2.4
python-elementtree  
python-iniparse  
python-sqlite  
rpm >= 0:4.4.2
rpm-python  
rpmlib(CompressedFileNames) <= 3.0.4-1
rpmlib(PayloadFilesHavePrefix) <= 4.0-1
rpmlib(VersionedDependencies) <= 3.0.3-1
urlgrabber >= 3.1.0
yum-fastestmirror  
yum-metadata-parser >= 1.1.0
Off-hand, and since the compression library has caused issues in the past, I'm thinking maybe an issue with zlib. Update: Someone has posted a stack trace below in the comments which points to an issue other then zlib. A bunch of gcc / glibc files were also updated on my server today:
gcc.x86_64 0:4.1.2-51.el5
gcc-c++.x86_64 0:4.1.2-51.el5
gcc-gfortran.x86_64 0:4.1.2-51.el5
gdb.x86_64 0:7.0.1-37.el5
glibc.i686 0:2.5-65
glibc.x86_64 0:2.5-65
glibc-common.x86_64 0:2.5-65
glibc-devel.i386 0:2.5-65
glibc-devel.x86_64 0:2.5-65
glibc-headers.x86_64 0:2.5-65

Suggestion #5 or Update #5:

Looks like a bug.  Reports on the CentOS-Users and RPMForge user's mailing lists are indicating that they are starting to get to the bottom of the issue. There's an issue with a package in the RPMForge repository with strange/incorrect metadata and that's causing bad things to happen in yum.

See Bug 725798 - yum segmentation fault on rpmforge repository over at Red Hat's bug tracker.

Latest posts from the mailing lists indicate that both issues are being addressed today.  So you can temporarily disable rpmforge / rpmforge-extras and do your other updates, or delay all updates for a day or two until it is fixed.

Update #6:

Everything now works correctly again. You may want to run a "yum clean all" and "yum makecache" to refresh your metadata cache files.


.

23 comments:

Anonymous said...

Same here. It didn't happen last week, but does today.

Anonymous said...

This happens for me on CentOS 5.6 64bit that I installed this morning but it also happens on a machine running CentOS 5.6 64bit that I installed last week.

I posted an email to the rpmforge user list.

Anonymous said...

I am having the same issue.

rpmforge: [######] 471/10722Segmentation fault

I tried all the above solutions, Nothing works.

Help Please!

grantc said...

I don't think it was zlib:

Core was generated by `/usr/bin/python /usr/bin/yum update'.
Program terminated with signal 11, Segmentation fault.
#0 0xb7507453 in strlen () from /lib/libc.so.6
(gdb) bt
#0 0xb7507453 in strlen () from /lib/libc.so.6
#1 0xb68f09b1 in g_string_chunk_insert_len () from /lib/libglib-2.0.so.0
#2 0xb68f0a20 in g_string_chunk_insert () from /lib/libglib-2.0.so.0

Anonymous said...

Same thing for me. Only happens with rpmforge though. As soon as i remove that repo, no issues. I wonder if they maybe have some bad metadata posted.

ejant said...

same problem here... CentOS 5.6 64 bits (xen). Nothing works... only removing rpmforge

Anonymous said...

Sam thing here, yesterday the yum update worked without any problems.

Today just seg faults all the time.

ejant said...

Same problem here.... CentOS 5.6 64 bits. None of those solutions works... still looking por a fix.

Ed Stafford said...

Having the exact same issue on RHEL 5.7 and I'm no closer to figuring out the issue right now.

Darn shame since I rely on RPMForge pretty heavily.

Graham said...

The problem must be with a very recent update on the rpmforge mirrors. I haven't run "yum update" at all recently - last time was in May to update mercurial only, and I'm still on Centos 5.4 (32-bit PAE). However, by chance I was using "yum list" and "yum deplist" yesterday to check for updates and dependencies for some packages and all was fine. I did the same thing today and first of all the rpmforge mirror on apt.sw.be failed the metadata checksum, then the segfault happened on the next mirror. --disablerepo=rpmforge allows me to continue with core packages

Normando said...

Me too. The same as others in pristine Centos 5.6 x64

realtechtalk said...

This is happening with at least two different servers the exact same way.

I suspect it is only related to rpmforge and disabling it would fix it, however Centos is nothing without rpmforge.

Unknown said...

I am also getting it

[hoshi@batman ~]$ yum repolist
Loaded plugins: fastestmirror
rpmforge: [#### ] 891/10953Segmentation fault

One thing i noticed that the number before the '/' is constant (891 or 471 i guess depending on what rpmforge it is connecting to). I dont know if this is significant or not

Son Nguyen said...

It happened to me today, on about 15 servers. Hope they have a fix soon.

Anonymous said...

Same problem...

Jindřich Šaur said...

It's because http://rpmrepo.net/RPMforge is down, IMHO...

Leandro Silva said...

Hi folks

I'm using Extra Packages for Enterprise Linux (EPEL) cause when I will install openvpn with yum, rpmforge stop work. Check http://wiki.centos.org/AdditionalResources/Repositories to other repos.

Elton Andrade said...

same thing as Leandro, the EPEL repo saved the day! :)

Anonymous said...

Same here :(

Jindřich Šaur said...

It's OK now :)

Spicysomtam said...

rpmforge is back now. Irony: this what you get when you don't pay for support from redhat; service is not guarenteed. Personally I am ok with this.

Thomas said...

It really doesn't have anything to do with using RHEL or CentOS or Scientific Linux and whether or not you pay for support. The fact of the matter is, if you use any of the distros based off of RHEL you will eventually run into situations where the software that you want is not available in the official repositories or you need a newer major revision then what is in there.

Which is where the 3rd party repositories like RPMForge, IUS, ATRPMs, PGDG, etc. come in. You can pull RPMs from those or you can choose to download source tarballs and build from scratch (like you have to do for Postfix). The repository method, assuming that whoever maintains the package is quick about pushing updates is ideal because of less work in keeping up to date.

But that's really an argument for one of the mailing lists and not for here.

My preference is to stick with the official RHEL stuff where possible, and I only pulling select packages from 3rd party repositories using "includepkgs=" in the /etc/yum.repos.d/ configuration files.

Anonymous said...

This one help me

https://helpdesk.cloudlinux.com/index.php?/Knowledgebase/Article/View/56/3/yum-doesnt-work-returns-segmentation-fault