Success in improving RAID speeds

After a long and hard trial process, I have increased the speed of my RAID5 array on an IBM ServeRAID-6M to a very acceptable level

Before I worked on this, guests on a 5 disk RAID5 were getting 5MB/s and after-wards I am getting the following Speeds

• Windows 2003, 12GB, Read tests with HD_speed are 92 MB/s
• Ubuntu 9.04 server, 8GB Read+Write tests, using dd are 32 MB/s

My second RAID was only 3 disks because I had a problem with 2 of the disks. Here are my suggestions for building a RAID5 from used parts

• Test the READ and WRITE speeds of each disk by itself. I had 1 U320 disk that was reading fine at 60 MB/s but crippled down and would only write at 5 MB/s
• Have the same interface; one disk was a U160, and it could only read at 50 MB/s
• Do not include any disks that are considerably slower than others, it will slow down the whole array.
• Not sure this is true with all Array controllers, but make sure that both the disks and the array are set to write back, not write through. Also the setting for the disks can only be change while they are not in an array, and cannot be change once they are built into an array

These are my preferences, and not sure they will affect anything.

• When building the array use the largest stripe size. The ServeRAID-6M goes up to 64K
• When formatting the VMFS Datastore choose a larger block size like 4MB or 8MB.
  

Improving SCSI performance on ESXi Whitebox

OK, the upgrade to SCSI went well, I converted from a SATA to SCSI system.

Added the following components:

• IBM ServeRAID 6M, PCI-X card
• 1 36GB drive to use as boot, and ISO storage
  
• 5 74GB drives in RAID5 to be used as main datastore (272 GB)
• 1 500GB SATA drive from old ESXi Server. Source Datastore to move data onto RAID5

Now everything was going well, but was noticing that the speed performance was slow. Now I expected it to be slower with writes on RAID5, but I was expecting the reads to scream.

That was not what was happening. So I started using HD_Speed and noticed some things.

• Read speed on SATA drives 115
• Read Speed on SCSI RAID5 4.5

That was a big difference. So I did some digging and did the following changes to make some improvements

• Moved the ServerRAID 6M to PCI-X slot 1 because it was 133 Mhz, Removed the other SCSI card because if there was anything in Slot 2, it downgrade to 100 Mhz and the card was only rated at 133.
• Made sure that the Cache was on
• Upped the queue limit from 64 to 96 into the BIOS

After I did that the Read speed moved up to 45. Not bad. One other thing that I noticed is that when you move to larger downloads like 4MB it would move up to the 100 to 200 range.

Also I could benchmark 2 servers simultaneously from the SCSI RAID5 and it did not impact each other.

External Expansion from IBM

Still stumbling. Looking for an external SCSI chassis for my whitebox. I know about the HP StoroageWorks, but looking for what IBM has to offer:

• Found this EXP400 on Ebay: Auction
• Lead me to an IBM web site for the EXP400 which is discontinued: EXP400 info at IBM
• Found the current Expansion Arrays, but they are all SATA or Fiber Channel: Current IBM Expansion Arrays

All in all, the EXP400 looks like a good product. Here are the drives that can go into it:

• 40K1023 73.4GB 10K rpm Ultra320 SCSI HDD
• 40K1024 146.8GB 10K rpm Ultra320 SCSI HDD
• 40K1025 300GB 10K rpm Ultra320 SCSI HDD
• 40K1026 36.4GB 15K rpm Ultra320 SCSI HDD
• 40K1027 73.4GB 15K rpm Ultra320 SCSI HDD
• 40K1028 146.8 15K rpm Ultra320 SCSI HDD

Here is the location to translate the P/N to FRU & Newest part numbers: Hard drive Accessories