MD5 vs SHA-1 vs SHA-256 performance

I was curious this weekend about how MD5 vs SHA-1 vs SHA-256 performance stacks up.  If you have the OpenSSL libraries installed, you can run a short test to calculate performance on your CPU.  It gives ballpark estimates, which may or may not carry over to real-world performance on actual file data.

$ openssl speed md5 sha1 sha256

Estimates/Summary:

SHA-1 is about 55-75% the speed of MD5
SHA-256 is about 25-40% the speed of MD5
SHA-256 is about 50-60% the speed of SHA-1

Whether or not you will be CPU-bound in computing the file hashes will depend on where you are reading the files from.  Over gigabit LAN or from older mechanical hard drives, or external USB2/USB3 drives, most modern CPUs can keep up even with SHA-256.  But if you are reading the files off of a local SSD or really fast mechanical drive (or RAID) then you are likely to be bottlenecked by the CPU.

Details (openssl test):

AMD Opteron 2210 HE @ 1.8GHz
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
md5              25350.32k    80336.17k   191208.72k   295806.29k   353239.04k
sha1             27497.51k    77933.74k   168168.28k   235850.41k   268205.53k
sha256           22543.94k    51445.66k    88818.90k   108275.37k   116416.81k

AMD Opteron 6212 @ 2.6GHz
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
md5              45318.86k   134308.27k   324486.83k   493194.58k   582216.36k
sha1             47027.79k   128927.42k   278593.02k   410244.44k   471425.02k
sha256           28690.18k    62662.12k   106333.61k   127998.98k   136325.80k

AMD Opteron 1214 @ 2.2GHz
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
md5              30417.85k    96951.65k   235868.24k   361845.33k   433094.16k
sha1             34126.39k    98072.04k   208456.70k   291041.35k   328654.85k
sha256           27362.92k    63127.50k   108537.35k   132564.38k   142727.86k

AMD Opteron 4180 @ 2.6GHz
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
md5              43180.73k   129437.27k   298339.15k   442248.19k   519602.18k
sha1             45699.96k   123160.41k   255322.20k   347329.88k   390250.50k
sha256           33757.18k    75191.85k   128929.37k   157659.82k   168790.70k

Intel Xeon E5520 @ 2.27GHz
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
md5              25945.50k    88897.43k   220292.01k   354356.91k   430339.41k
sha1             26568.69k    78716.78k   174506.35k   251495.08k   288858.11k
sha256           20495.77k    49206.73k    90735.59k   114904.75k   124605.78k

AMD Phenom II X4 810 @ 2.6GHz
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
md5              42430.27k   130101.07k   301091.11k   444996.84k   522918.05k
sha1             46015.86k   125298.40k   256391.07k   349968.95k   393227.03k
sha256           34432.84k    76378.03k   130828.67k   158859.76k   168782.78k

Additional tests/experiments:

Tests using "(sha256|sha1|md5)sum" programs on Opteron 4180 w/ SSDs, using 133GB of large files.  The CPU core assigned to the task was pegged at 100% utilization according to "atop".  The SSD used has an estimated top speed of 490 MB/s.

MD5: ~318 MiB/s (428 sec)
SHA-1: ~187 MiB/s (729 sec) - 1.7x slower then MD5
SHA-256: ~113 MiB/s (1204 sec) - 2.8x slower then MD5, 1.65x slower then SHA-1

Benchmarking - hdparm (follow-up)

If you recall, the last time I tested hdparm on my Celeron 566MHz system, I was getting very poor read performance on /dev/hda.

So I added in a HTP302 (HighPoint Rocket133) 2-port PCI card and moved hda over to the new card.

I now get 20MB/s buffered reads from both the primary and secondary disk in the RAID1 array and 30MB/s buffered reads from the mirror set. Which is a lot better then the 3.1MB/s for hda and 3.5MB/s for the overall RAID1 array.

That should make the system feel a lot more snappy. Now the bottleneck will probably be the CPU or the 100Mbit ethernet card.

Benchmarking - bonnie

The first test is done on /dev/md3 (composed of partitions on /dev/hda and /dev/sda).

nogitsune tmp # bonnie -s 16384 -m nogitsune-vgmirror
File './Bonnie.9315', size: 17179869184
Writing with putc()...done
Rewriting...done
Writing intelligently...done
Reading with getc()...done
Reading intelligently...done
Seeker 1...Seeker 2...Seeker 3...start 'em...done...done...done...
              -------Sequential Output-------- ---Sequential Input-- --Random--
              -Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --Seeks---
Machine    MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU  /sec %CPU
nogitsun 16384 16714 39.6 21765  8.4 12889  4.3 42827 80.7 55645  5.0 155.6  0.4

Test results for a 3-disk RAID5 volume on Nogitsune:

nogitsune backup # bonnie -s 16384 -m raid5-{3}
File './Bonnie.9472', size: 17179869184
Writing with putc()...done
Rewriting...done
Writing intelligently...done
Reading with getc()...done
Reading intelligently...done
Seeker 1...Seeker 2...Seeker 3...start 'em...done...done...done...
              -------Sequential Output-------- ---Sequential Input-- --Random--
              -Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --Seeks---
Machine    MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU  /sec %CPU
raid5-{3 16384 12250 30.9 15062  8.3 10926  6.3 34752 65.0 84890 15.4 134.5  0.7
nogitsune backup # 

These are test results for the RAID1 set on the VIA C3 unit.

nezumi / # bonnie -s 2047 -m nezumi-raid1
File './Bonnie.22025', size: 2146435072
Writing with putc()...done
Rewriting...done
Writing intelligently...done
Reading with getc()...done
Reading intelligently...done
Seeker 1...Seeker 2...Seeker 3...start 'em...done...done...done...
              -------Sequential Output-------- ---Sequential Input-- --Random--
              -Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --Seeks---
Machine    MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU  /sec %CPU
nezumi-r 2047  2637 94.2 27421 57.8 12256 18.9  2665 93.7 29279 25.7 231.7  5.9
nezumi / # 

Here's is the Celeron 566MHz unit showing performance for the RAID1 array:

coppermine svn # bonnie -s 2047 -m coppermine-raid1
File './Bonnie.8564', size: 2146435072
Writing with putc()...done
Rewriting...done
Writing intelligently...done
Reading with getc()...done
Reading intelligently...done
Seeker 1...Seeker 2...Seeker 3...start 'em...done...done...done...
              -------Sequential Output-------- ---Sequential Input-- --Random--
              -Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --Seeks---
Machine    MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU  /sec %CPU
coppermi 2047  1087 19.7  1272  6.6  1020 19.5  3505 95.4  6763 78.9 143.8  5.5
coppermine svn #

The following is the Celeron 566MHz talking to a single 120GB 5400rpm drive:

coppermine backup # bonnie -s 2047 -m direct          
File './Bonnie.8837', size: 2146435072
Writing with putc()...done
Rewriting...done
Writing intelligently...done
Reading with getc()...done
Reading intelligently...done
Seeker 1...Seeker 2...Seeker 3...start 'em...done...done...done...
              -------Sequential Output-------- ---Sequential Input-- --Random--
              -Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --Seeks---
Machine    MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU  /sec %CPU
direct   2047  4517 83.2 11374 58.1  9073 54.7  6932 98.6 36154 52.7  99.8  3.2
coppermine backup #

Benchmarking - hdparm

So as I prepare for the iSCSI build, I need to start gathering tools to help me find bottlenecks. As well as establishing baseline performance estimates.

hdparm -tT {blockdevice name} - This tool ships standard with most (all?) versions of Linux. It's a read-only, non-destructive (if used properly), command that can be used to test raw read performance from any block device. So you can check individual drives in a RAID array as well as doing a quick check of the over all RAID array performance. Run time is generally around 10 seconds.

Here are some sample runs from my firewall box (VIA C3 600MHz, 1GB RAM, 2x60GB 5400rpm notebook drives):

nezumi backup1 # hdparm -tT /dev/md2

/dev/md2:
Timing cached reads: 276 MB in 2.02 seconds = 136.88 MB/sec
Timing buffered disk reads: 106 MB in 3.06 seconds = 34.61 MB/sec

nezumi backup1 # hdparm -tT /dev/hda2

/dev/hda2:
Timing cached reads: 276 MB in 2.02 seconds = 136.78 MB/sec
Timing buffered disk reads: 90 MB in 3.04 seconds = 29.58 MB/sec

nezumi backup1 # hdparm -tT /dev/hdc3

/dev/hdc3:
Timing cached reads: 276 MB in 2.02 seconds = 136.87 MB/sec
Timing buffered disk reads: 88 MB in 3.04 seconds = 28.91 MB/sec

nezumi backup1 # hdparm -tT /dev/md2

/dev/md2:
Timing cached reads: 276 MB in 2.02 seconds = 136.49 MB/sec
Timing buffered disk reads: 108 MB in 3.03 seconds = 35.65 MB/sec

nezumi backup1 # hdparm -tT /dev/md2

/dev/md2:
Timing cached reads: 276 MB in 2.02 seconds = 136.72 MB/sec
Timing buffered disk reads: 108 MB in 3.00 seconds = 35.99 MB/sec

What you will notice is that the "cached" reads value has more to do with the memory speed then the speed of the disk. The buffered disk read values are closer to real-world performance values. There are no surprises in the performance of the VIA C3 system.

However, on my older Gigabyte Celeron 566MHz system, there's a big surprise:

coppermine thomas # hdparm -tT /dev/hda4

/dev/hda4:
Timing cached reads: 336 MB in 2.00 seconds = 167.99 MB/sec
Timing buffered disk reads: 10 MB in 3.18 seconds = 3.15 MB/sec

coppermine thomas # hdparm -tT /dev/hde4

/dev/hde4:
Timing cached reads: 336 MB in 2.01 seconds = 167.32 MB/sec
Timing buffered disk reads: 86 MB in 3.07 seconds = 27.99 MB/sec

coppermine thomas # hdparm -tT /dev/hdg1

/dev/hdg1:
Timing cached reads: 336 MB in 2.00 seconds = 167.65 MB/sec
Timing buffered disk reads: 58 MB in 3.02 seconds = 19.20 MB/sec

coppermine thomas # hdparm -tT /dev/md2

/dev/md2:
Timing cached reads: 336 MB in 2.01 seconds = 166.99 MB/sec
Timing buffered disk reads: 12 MB in 3.28 seconds = 3.66 MB/sec

coppermine thomas # hdparm -tT /dev/md3

/dev/md3:
Timing cached reads: 336 MB in 2.00 seconds = 167.65 MB/sec
Timing buffered disk reads: 10 MB in 3.18 seconds = 3.15 MB/sec

This shows that there are severe performance issues with any block devices using /dev/hda (such as /dev/md2 and /dev/md3). The motherboard chipset is either extremely slow, or there are performance bottlenecks that need to be investigated.

The test results also show that the old Celeron 566MHz is slightly faster then the VIA C3 (167MB/s vs 137MB/s). So if I can find and fix the bottleneck for /dev/hda, I should see a significant increase in performance from this particular unit.

nogitsune etc # hdparm -tT /dev/hda

/dev/hda:
Timing cached reads: 3220 MB in 2.00 seconds = 1609.90 MB/sec
Timing buffered disk reads: 172 MB in 3.02 seconds = 56.95 MB/sec

nogitsune etc # hdparm -tT /dev/sda

/dev/sda:
Timing cached reads: 3236 MB in 2.00 seconds = 1617.90 MB/sec
Timing buffered disk reads: 184 MB in 3.00 seconds = 61.25 MB/sec

nogitsune etc # hdparm -tT /dev/md3

/dev/md3:
Timing cached reads: 3208 MB in 2.00 seconds = 1603.90 MB/sec
Timing buffered disk reads: 188 MB in 3.03 seconds = 62.08 MB/sec

nogitsune etc # hdparm -tT /dev/hde1

/dev/hde1:
Timing cached reads: 3228 MB in 2.00 seconds = 1613.90 MB/sec
Timing buffered disk reads: 136 MB in 3.01 seconds = 45.15 MB/sec

nogitsune etc # hdparm -tT /dev/hdg1

/dev/hdg1:
Timing cached reads: 3232 MB in 2.00 seconds = 1615.90 MB/sec
Timing buffered disk reads: 136 MB in 3.00 seconds = 45.27 MB/sec

nogitsune etc # hdparm -tT /dev/md4

/dev/md4:
Timing cached reads: 3244 MB in 2.00 seconds = 1621.90 MB/sec
Timing buffered disk reads: 136 MB in 3.00 seconds = 45.33 MB/sec

nogitsune etc # hdparm -tT /dev/hdk1

/dev/hdk1:
Timing cached reads: 3232 MB in 2.00 seconds = 1615.90 MB/sec
Timing buffered disk reads: 136 MB in 3.01 seconds = 45.21 MB/sec

nogitsune etc # hdparm -tT /dev/hdo1

/dev/hdo1:
Timing cached reads: 3224 MB in 2.00 seconds = 1611.90 MB/sec
Timing buffered disk reads: 136 MB in 3.00 seconds = 45.33 MB/sec

nogitsune etc # hdparm -tT /dev/hds1

/dev/hds1:
Timing cached reads: 3224 MB in 2.00 seconds = 1611.90 MB/sec
Timing buffered disk reads: 134 MB in 3.01 seconds = 44.49 MB/sec

nogitsune etc # hdparm -tT /dev/md5

/dev/md5:
Timing cached reads: 3224 MB in 2.00 seconds = 1611.90 MB/sec
Timing buffered disk reads: 266 MB in 3.01 seconds = 88.43 MB/sec

nogitsune etc # hdparm -tT /dev/sdb1

/dev/sdb1:
Timing cached reads: 3232 MB in 2.00 seconds = 1615.90 MB/sec
Timing buffered disk reads: 162 MB in 3.01 seconds = 53.85 MB/sec

What we see here is that for the RAID1 arrays, speed is equivalent to the disks within the array. But for the RAID5 array with (3) disks, speed is 2x that of the single disks within the array.

3DMark06 Scores

FutureMark's 3DMark06

Note that 3DMark06 is heavily weighted towards 3D card power rather then CPU power.

1443 (SM2: 546 HDR/SM3: 549 CPU: 834)
AMD Opteron 148
PC3200 2GB (2x1GB) ECC DDR RAM
Asus SK8N motherboard
NVIDIA GeForce 6800 (128MB AGP)
NVIDIA 81.95

My GeForce 6800 shows some signs of aging. Definitely no longer the fastest thing on the block (not that it was when I bought it). Some of the dual-core configurations with newer video cards are in the 3500-4500 score range (with CPU scores of 1500-2000). Some of the single-core folks are scoring 3000+ 3DMark06 with CPU scores of only 900, so a newer video card could make a significant difference in this score.

Rather then buying a new PC this year, I may purchase one more AGP video card (NVIDIA GeForce 7800GS) for $300 and stretch this PC's lifespan for another 2 years. I'll have to do some research and see what kinds of 3DMark and AquaMark scores people can get with a similar CPU and a 7800GS.

172 (SM2: 79 HDR/SM3: n/a CPU: 711)
AMD Athlon64 3000+
PC3200 (2x512MB) DDR RAM
Asus A8N-VM CSM motherboard
NVIDIA GeForce 6150 integrated graphics (64MB)

This score reinforces that integrated graphics solutions are really not designed at all for gaming. Again, putting a $50-$75 video card in this would bring the scores up a lot and make it quite a bit faster. (Such as the $70 GeForce 6600 PCIe cards.)

(All systems are stock settings (no overclocking), usually with 7200rpm hard drives. If I get a chance this week, I'll run some tests on some older Dell systems at the main office.)

Update #1: Looking at other 7800GS scores, I could expect a pretty decent boost up to around 1500 for the two GPU score portions. Whether that translates into 3x faster performance is unlikely, but I'd probably see a 2x performance gain. I'll probably pickup an eVGA version of it within the next few weeks.

More scores:

172 (SM2: 80 HDR/SM3: n/a CPU: 1632)
AMD Athlon64 X2 4200+
PC3200 (2x1GB) DDR RAM
Asus A8N5X motherboard
NVIDIA GeForce 6200 PCIe (64MB)

194 (SM2: 89 HDR/SM3: n/a CPU: 1493)
AMD Athlon64 X2 3800+
DDR2 533 (2x1GB) RAM
Asus M2NPV-VM motherboard
NVIDIA GeForce 6150 integrated graphics (32MB)

3003 (SM2: 1324 HDR/SM3: 1279 CPU: 834)
AMD Opteron 148
PC3200 2GB (2x1GB) ECC DDR RAM
Asus SK8N motherboard
NVIDIA GeForce 7800GS (256MB AGP)
NVIDIA 91.47

(The 7800GS is a nice upgrade over my older 6800.)

PCMark05 Basic scores

FutureMark's PCMark05

More of a general office benchmark (even though it won't run on my Tecra 9100). The integrated video on the Asus A8N-VM CSM motherboard really doesn't slow the system down much.

3652 PCMarks
AMD Athlon64 X2 4200+
PC3200 (2x1GB) DDR RAM
Asus A8N5X motherboard
NVIDIA GeForce 6200 PCIe (64MB)

3329 PCMarks
AMD Opteron 148
PC3200 2GB (2x1GB) ECC DDR RAM
Asus SK8N motherboard
NVIDIA GeForce 6800 (128MB AGP)
NVIDIA 81.95

2340 PCMarks
AMD Athlon64 3000+
PC3200 (2x512MB) DDR RAM
Asus A8N-VM CSM motherboard
NVIDIA GeForce 6150 integrated graphics (64MB)

3341 PCMarks
AMD Athlon64 X2 3800+
DDR 533 (2x1GB) RAM
Asus M2NPV-VM CSM motherboard
NVIDIA GeForce 6150 integrated graphics (32MB)

(All systems are stock settings (no overclocking), usually with 7200rpm hard drives. If I get a chance this week, I'll run some tests on some older Dell systems at the main office.)

AquaMark3 scores

One benchmark that I used to use is AquaMark3 (a DirectX 9 based benchmark). Unfortunately, the company behind AquaMark3 is no longer in business. Which is a shame, because they allowed you to submit your scores to a central database which had some decent search parameters. (For instance, you could search for equivalent systems that were not overclocked to get an idea of what performance a particular upgrade would give you.)

Still, since I have it around, I threw it at the new Athlon64 3000+ system that I just built. I expect that the video scores will be pretty poor since it's using an integrated NVIDIA GeForce 6150 video card that uses 64MB of system RAM.

10100 (CPU: 7470, GFX: 1080)
AMD Athlon64 3000+
PC3200 (2x512MB) DDR RAM
Asus A8N-VM CSM motherboard
NVIDIA GeForce 6150 integrated graphics (64MB)

Indeed, the graphics scores are pretty much bottom of the barrel. Worse then my old Ti4600 video card from a few years ago. OTOH, the CPU score is very good for a low-end system and a $50 or $75 PCIe video card would spice things up quite a bit.

Older scores from past systems:

11879 (CPU: 8453, GFX: 1278)
AMD Athlon64 X2 4200+
PC3200 (2x1GB) DDR RAM
Asus A8N5X motherboard
NVIDIA GeForce 6200 PCIe (64MB)

11095 (CPU: 7069, GFX: 1299)
AMD Athlon64 X2 3800+
DDR2 533 (2x1GB) RAM
Asus M2NPV-VM motherboard
NVIDIA GeForce 6150 integrated graphics (32MB)

19500 (CPU: 6300, GFX: 2300)
AMD AthlonXP 2600+
PC2700 (2x512MB) DDR RAM
A7N8X-Deluxe motherboard
NVIDIA GeForce4 Ti4600 128MB (AGP)
NVIDIA 61.77

36500 (CPU: 7250, GFX: 4875)
AMD Opteron 144
PC2100 1GB (2x512MB) ECC DDR RAM
Asus SK8V motherboard
NVIDIA GeForce FX 5900 XT
NVIDIA 61.77

47000 (CPU: 7000, GFX: 7125)
AMD Opteron 144
PC2100 1GB (2x512MB) ECC DDR RAM
Asus SK8V motherboard
NVIDIA GeForce 6800
NVIDIA 61.77

50475 (CPU: 7250, GFX: 8300)
AMD Opteron 148
PC3200 2GB (2x1GB) ECC DDR RAM
Asus SK8N motherboard
NVIDIA GeForce 6800 AGP 128MB
NVIDIA 81.95

62935 (CPU: 8972, GFX: 9693)
AMD Opteron 148
PC3200 2GB (2x1GB) ECC DDR RAM
Asus SK8N motherboard
NVIDIA GeForce 7800GS AGP 256MB
NVIDIA 91.47