SuperMico SuperServer giving slow RAID 10 performance

Posted on Oct 4, 2009 by Paul White

When you go out of your way to build an awesome server, you demand performance. So after buying a SuperMicro 6014-32 Server off ebay, then upgradeing the 4 HotSwap bays with Fujitsu 146GB 15K SAS drives, and adding the AOC-SOZCR1 RAID card I would expect some awesome performance numbers. Instead It performs more like a single SATA drive on my workstation. This is not the kind of performance I was hoping for. Considering that my workstation is running the same drives ( but only 2 ) in RAID 0. And was achieving near 200 MB/sec reading 1024KB blocks, I would expect something similar when 4 drives are setup in RAID 10. Keep in mind my workstation is not using any fancy PCI-X RAID card. Its just using the built in Marvel RAID controller that comes with the newer ASUS motherboards ( for the core i7 ). If anyone has any suggestions please let me know. Else I might sell this server back on ebay, and upgrade to another brand. Maybe this time I will go for a Dell since I know from experience their RAID setups are quick. Just a matter of finding a Dell Server that will rock with 4 x 3.5" SAS drives all in a 1U form factor.

AOC-SOZCR1 RAID card

Expansion RAID card is Zero Channel Slot

BackPane on SuperMicro Server

Here you can see the connection between the backpane and the motherboard

UPDATE 10/5/2009

Got an Email today from SuperMicro's support department.
They asked me if I had the latest bios, which I did, and included a driver for the Adaptec SAS Controller, but windows server 2008 didn't want to take the driver they supplied. I believe the driver was a 32bit driver, and that doesn't work on an 64 bit OS. I informed the tech and asked him if he had any other suggestions. Hopefully he comes back with a solution soon

UPDATE 10/5/2009 Night

The Tech from SuperMicro emailed me a copy of the RAID driver for x64 windows. I tried to install the driver but windows wouldn't allow it. Server 2008 R2 didn't like the Adaptec Driver ( which is newer ), and instead preferred to run the Microsoft driver that is dated 2006.

UPDATE 10/6/2009

The Tech from SuperMicro emailed me again and asked for the detailed specs of my configuration. So I emailed him the specs

SuperMicro 6014-32
Dual 3.6 Xeons
4 x 1 GB PC3200 RAM
4 x 146 GB 15K RPM Fujitsu SAS drives ( Model MAX3147RC)
Installed Additional RAID card ( AOC-SOZCR1 ), with jumper set to SAS
System configured into a single Logical Driver using RAID 10, with strip size of 64K
Upgraded CD-ROM slim drive to DVD-RW slime drive ( works great )
Running Windows Server 2008 R2 Web Edition ( Keep in mind R2 uses the Windows 7 Kernel, and is only available in x64 ) in Trial mode ( got 5 days til I need to provide a key )
System benchmarks linear read of 75 MB/sec when reading 1024 K blocks, slower for smaller sizes
Tried to use the driver you provided, but windows refuses to use it, and continues to run the default microsoft driver.

SuperMicro SuperServer 6014-32 layout

UPDATE 10/6/2009 30 minutes later

He got back with me again, and included a benchmark program. He wanted me to run the benchmark and to report back the results. The program was nice, only took up about 26K, and can be run from a single EXE. no need for any installation, which is nice for a change. Here are the results of that benchmark

RAID 10 SAS benchmark on SuperMicro Server

RAID 10 SAS benchmark on SuperMicro Server

I have the system setup with a 64K strip size. There is something wrong, as write speeds don't usually exceed read speeds, especially when you start reading later blocks.

UPDATE 10/7/2009

SuperMicro didn't have any solutions. At this point my theory is that the problem lies with the Driver. After reading a few forums where people were having similar problems this would seem to be the main cause. The system will not let me override the default Microsoft Driver which works but has terrible performance. I am pretty sure the problem lies in no available Adaptec Driver that supports Windows 7 or Server 2008 R2. To test this theory I am going to install Server 2003 R2 x64 and rerun the benchmarks. The driver provided by Adaptec was made just for this OS, so it should work. If I can get decent performance on Server 2003, this will tell me that the problem is not the hardware, but the driver. Then its just a matter of time before Adaptec releases a newer driver for Server 2008 R2, and Windows 7. This server won't be overloaded so I can get by with the less than perfect performance. I will post my results later tonight.

UPDATE 10/7/2009 5:30PM

I submitted a question through Adaptec's Website to see if they have an updated driver for this device. We will see if they come back with any answers.

UPDATE 10/7/2009 6:30PM

I got the install of Windows Server 2003 R2 x64 started. The windows install screen came up and I hit F6 to install my own RAID driver. I tried the driver that was for the built in RAID controller, but it didn't like that one, Then I tried the Driver that came with the AOC-SOZCR1 Add on card, and it worked. Now its going through the steps of copying the files. The interesting part is just by looking at the LEDs on the front of the Hot Swap Trays, it would appear the RAID array is moving much faster than before. It seems that the driver is not a windows verified driver. I had to accept this for the install to continue. If Windows Server 2008 R2 has some issues with installing non verified drivers this would explain why it refused to accept the driver. There is hope!

UPDATE 10/7/2009 7:15PM

The install of Windows Server 2003 R2 x64 went fine. However the benchmarks looked the same as they did on Server 2008 R2. But I found that the driver was dated 2005. I googled and found a 2006 driver, installed it, and reboot. Same Lousy Benchmarks as before. Now I am trying to use windows update to see if after updating with some patches will make a difference.

UPDATE 10/7/2009/ 9:15 PM
The Windows Updates made no difference. Bench Marks still sucked. I have decided that this is no longer worth my time. I have 25 days to return the AOC-SOZCR1 addon card to wiredzone.com where I bought it. Unfortunately they are going to hit me with a 15% restocking fee. Which isn't bad considering anyone who reads this blog will no longer want one. I am starting to believe that the problem was not with the drivers, but with the hardware itself. So as a warning to anyone considering the AOC-SOZCR1 to give your SuperMicro Server raid 5, 10, or 50. This card is bad. Don't buy it! I am quickly learning that if you can't find a product for sale on NewEgg.com, you might want to rethink the purchase.

Final Solution to get fast RAID on SuperMicro Server
The server does have a Riser card that provides an PCI-E x8 slot.
So instead of using the built in raid controller and some crappy add-on card, I am going to get a real raid card.

RAID card HIGHPOINT RocketRAID 4310

HighPoint RocketRAID 4310 PCI-Express x8 Four-Port SATA and SAS HARDWARE RAID Controller Card

Model
Brand	HighPoint
Model	RocketRAID 4310
Form Factor	Low Profile Ready
Specifications
Type	SATA / SAS (Serial Attached SCSI)
Internal Connectors	1 x SFF-8087 mini-SAS
Interface	PCI-Express x8
Cache Memory	256MB DDR2 533
RAID	RAID 0/1/3/5/6/10/ JBOD
Operating Systems Supported	Microsoft Windows 2008/2003/XP/ 2000/x64 Edition/Vista 32 and 64 bit Red Hat Enterprise Linux, Open SuSE, Fedora Core, Debian FreeBSD Mac OS X Linux Open Source Drivers Native Support in Linux kernel 2.6
Features
Features	1 internal mini-SAS/SATA Device connector Intel IOP 348( 800MHz) NCQ (Native Command Queuing) NVRAM Battery Backup Unit Ready (BBU-03) Low Profile

Just ordered this and the Overpriced mini-SAS to SAS cable.
Hopefully this solves my problems.
I will repost in a week once it arrives and I have done some benchmarks

Permalink

13979 Visitors

13979 Views

Categories associated with SuperMico SuperServer giving slow RAID 10 performance

Computer Hardware

Hard Drives

Discussion

Paul | Nov 13, 2009 2:52 AM
Larry,
I am glad my blog helped you. Just a tip if you buy a PCI-E RAID card you might not be able to run it from the right side riser port. Mine would beep and not boot from the right side. I had to get a left side PCI-E riser card so my RAID card would work. Not sure why, I even contacted SuperMicro about it and they didn't have a clue as to why. Fortunately used riser cards sell for pretty cheap. Even though my RAID card is just kind of floating in the slot because the chasis doesn't properly lock it in, it seems to be working well.

Lloyd | Jan 24, 2010 5:12 PM

It would be great to hear if the Rocketraid did the trick for you; I have just bought an LSI Megaraid 9211-4i controller, it?s gotten great reviews/feedback for SSDs in RAID0. RAID10 I thought should follow suit (or at least better my motherboards RAID10). Thus far its driving me round the bend as it is achieving useless single figure 6-7MB/s reads (poor writes too ) from 4 x SATA drives in RAID10. Each drive on its own is capable of plus 100MB/s (Samsung SpinPoint F3 500GB)

I also tested an old LSI 1068e based SAS controller and it has the same behaviour with RAID10 though marginally better; but still pretty much useless. It does do okay-ish in RAID0, though write speeds are always 50% less of read (why!!).

Granted these cards are being used/tested in AMD based desktop motherboards but as they are based on deferent chipsets (AMD and Nvidia) I can?t believe both are suffering from compatibility issues ... if this is the case do these RAID card manufactures do any testing (LSI is selling the 9211 as a gamer card among other things)?

Ironically the Nvidia motherboard?s RAID10 performance which I was trying to better manages 120MB/s Reads and 250MB/s Writes; but again why the huge disparity in read /write performance... is this typical? Would Nvidia (anybody) think this is good RAID10 performance?

Thus far it would seem that this RAID stuff is very hit and miss once you move away from ?mated? platforms i.e. HP and Dell server kit; which I work with daily.

No doubt i'll be going through a few more days until I figure out what the hell is going on.

End

Paul | Jan 24, 2010 5:49 PM
Lloyd,
The rocket raid worked out nicely. I feel your pain, trying to find the right raid card can be an issue. Here are the benchmarks I got for the Rocketraid 4310. Even my 15K SAS drives don't get much above 100MB/s stand alone, I certainly wouldn't expect SATA drives to come close, considering they spin at half the speed. What block size are you using on your Array?

I highly recommend you go with a RAID card that has some on board RAM. I noticed that the LSI Megaraid 9211-4i doesn't have any on board RAM neither does the LSI 1068e. That can make a huge difference. I would get a card with 1 GB of RAM if I could afford it. As it is the 4310 has 256 MB of ram. I know the cards are expensive, but if you check on ebay you can find used cards with on board RAM for a decent price.

The Built in SuperMicro RAID controller was a joke, and the add on zero channel RAID card sucked too ( even though it had 64 MB of RAM ).

I don't think the RAID stuff is hit or miss. But when you start asking for huge performance from your disk Array, you have to go with a pretty beefy RAID card. I recently overhauled my workstation with a dedicated raid card. I should be posting benchmarks soon. ( about 300 MB/s for 4 x 15K SAS drives in RAID 0 )

Lloyd | Jan 26, 2010 7:48 AM
Thanks for responding Paul, I'm still doing further research and like your self will leave a post of my findings (that's if I'm not too lazy).

you would be surprised at the throughput of the most resent SATA drives have a look at http://www.tomshardware.co.uk/charts/2009-3.5-desktop-hard-drive-charts/h2benchw-3.12-Max-Read-Throughput,1009.html

my stand alone tests show that these speeds are plausible; granted they can't match the seek and I/Os of proper 15K SAS drives.

I too have a couple 15K's (Fujitsu MBA3147RC) they manage about 250MB/s read in raid zero but writes are always 50% less on the crappy 1068E. So far what are your max R+Ws on the RocketRaid in RAID10 if you have done such tests ?

if your interested the following review shows how good or bad various raid controllers perform depending on the RAID type chosen.

http://www.xbitlabs.com/articles/storage/display/6-sas-raid-controllers-roundup.html

Lastly spoke to LSI Tech Support; basically in a round about way I'm told if I don't have SAS and Server 7MB/s could be expected (Great) so basically ?130 ($210) for a card that's worst than generic motherboard raid!!

Off to find a Server for testing ....

Lloyd | Jan 26, 2010 8:48 AM
Sorry my bad I didn't click on your benchmark link.. so in that case what was the max write on the "Rocket" RAID10 array (hopefully within 10-20% off reads) ?

Paul | Jan 26, 2010 1:51 PM
Lloyd,
with the current setup (RAID 10 using the rocketraid 4310) my max write speed was 130 MB/sec, and read was around 150 MB/sec. Granted I do have a few websites running on the box now, plus mail / web / stats / mysql servers, and I am sure this is affecting my benchmark. Before when the box was idle I was getting 800 MB/sec read and 130 MB/sec write. This is a result of the 256 MB cache. Google for a copy of ATTO benchmark and see how your card benchmarks. Thanks for the link to that review. I had been looking for something like that. 7 MB/s is terrible. Hope you get thinsg worked out

name
Email	Needed to confirm comment, but not made public.
Website