Faster Than a Speeding Gigabit

by Cutter Stevens

Let me start off by apologizing to all you loyal DV411 clients and fans for being such a stranger this year. I generally try and keep you up to date with the latest and greatest, but alas you all have kept me so bloody busy, having time to write an article has been kind of like having lunch--doesn't happen.  This, however is a special occasion!

As I sit here 45 minutes after closing in order to inform you of a milestone--Breaking the gigabyte barrier! I put this up there in the fond memory category where my 1971 GTO broke the 10 Second quarter mile. This is heavy duty rock and roll.  Here is the setup (and I doubt I will have enough time to duplicate this on any other setup, so don't ask).

Base system:

HP 9300 with dual 275 2.2 AMD Opterons, 4GB (3.25 actual address), Quadro FX 1400, 10K 74GB SATA system drive, Pioneer burner etc..

To begin, I do all my testing in real world situations and in circumstances that my clients live with on a daily/hourly/minute-by-horrible-minute basis.  I have installed an ADS Firewire 800 card in the 32 bit slot; the Avid Liquid Chrome HD boardset w/ mega breakout box in the PCIx-100 slot, an ATTO Celerity FC-42XS 4GB fiber card in the 133 slot (mostly to simulate a spare SCSI drive array for these benchmarks) and finally the Sorenson Squeeze HD XCEL encoding board in the final PCI-X 100 slot.

On the onboard SATA I also have two 500GB 16MB cache drives as "whatever" drives. You get the idea.  Now lets sit down at the table and dig in!

The Meat

In the second PCIe slot (16 lane as well ) we have the Atto Celerity FC-44ES which retails for a whopping $2995 (we can get that price down a bit). This card has been out at least since NAB 2006.  Although it's worth drooling over, there aren't a lot of "real world" scenarios I could imagine it servicing. Possibly  SAN solutions (really really big ones) but nothing remotely cost-efficient.  If anyone wants to put up the cash for me to experiment however, no problem! 

Just to put this card in perspective 4G/b fiber is, on a bad day, almost twice as fast as U320 SCSI as far as total bandwidth. Eight lanes of PCIe should theoretically be able to clear 2 GIGABYTES a second. So, again, theoretically this card should be able to push those limits.  That, of course, provided I have enough drives and the right controllers.  As you read (above), I got halfway there--pretty f%$#@#=ing great in my opinion.

So that was the meat, now for the potatoes.

The Potatoes

Hooked up to this monster of a card we have four 3TB G-SPEED arrays (unfortunately, I was using XP 32 bit so I was only able to stripe 2TB, but most of you are still in this situation anyway. These guys are 6x500 GB Hitachi 16MB cache drive, available in RAID 0,1,3,5 and RAID 6. I did my tests using RAID 0 and RAID 5. They also feature hot swap drives and redundant power.  I like them and at about $2.25 per Gig, it's a bargain.

The ATTO card used the same drivers as the 2 channel card that was already in there and the G-SPEED took about 10 minutes to acclimate with the web interface. RAID 0 formatted in about 10 seconds.  All in all, pretty painless.

Single RAID configuration - 4x 2TB RAID 0 array formatted in windows as one 8TB disk

Nitty Gritty Results:

• Random reads of a 1GB files size were about 875MB/s spiking up and down around 50 to 70 MB/s depending on where on the drive I was reading from. The fast read I got was 975MB/s. The spikiness did have me a bit worried, but I never went below 850Mb/s so I'm ok with it for now.
• Writing a sequential 16MB files never went below 990MB/s for the entire array. For the first 60% of the disk I was writing a 1.024 GB/s.  Yes that's Gigabytes!!

Now, you may ask yourself why am I getting faster writes than reads.  That doesn't make sense!?!  Reason being: Caching a 16 MB file is a lot easier than a reading a 1GB file.  But what the hell do you care???  You're still editing multiple 2k resolution frames...That is, if you found software to do it.

The interesting thing, I disconnected one of the drive arrays (3x 2TB raid 0) and my benchmarks were almost the same. Now what does that mean???  Where's the bottleneck! That my friends is a very good question. I have to assume it's in the PCIe bridge.  I do have a 16 Lane graphics card running. Or it could be the DDR 400 RAM is just not fast enough. I wish I had enough time to really do some system wide bench but alas the arrays I'm testing are on their way out the door.  So to answer the question, I am not sure but....once again, 875 to 900 mega freaking bytes a second!   

Now to bring balance to all this information...

Dual RAID configuration - RAID 5 with 4x2TB drive arrays in windows RAID 0 (RAID 5+0)

Most of you HD editors will opt for this.  Both the 2 channel PCIX card and PCIe are around 525 to 550 MB/s read and 700 MB/s write.  This is more than enough for dual stream HD editing.  I will add that these are 3TB array so there is some extra room on these drives even after RAID 5ing.  However I left them blank in this config.

Nitty Gritty Results:

• Random reads of a 1GB files size where about 900MB/s and the spikiness was not very apparent.  My data graph was pretty constant only spiking 10 to 20 MB/s once I got past 60% drive capacity. The fast read I got was 1.016GB/s in the first 5% of the arrays. The lesson is with this machine (XW9300) and this amount of drives, stay in RAID 5. Things will be more stable. One day I'll try this on a few other machines or I'll figure out where the bottleneck is.
• My write speeds were of course much slower around 700 MB/s with the same 16MB sequence file.  That, unfortunately, is the nature of the beast.  Sill, it's fast enough to write a 4k file at 14 fps.

That's the fast and furious.  Yes, I am doing much more exhaustive testing using IOmeter and many others that take days to complete. I figured you want the non-geek numbers right away. My mind is blowing, and so should yours. So in conclusion I will answer the one question that really matters on this subject.

Why the hell do I need anything that reads or writes that fast?  Well I have two ideas on that subject. First and most obvious, in a SAN situation in conjunction with a 4Gb switch I believe we can serve HD/SD content to a number of editing workstations.  Second, and most geeky, to be used for Uncompressed 2K and 4K playback machines with footage from such cameras as the Panavision Genesis or the Red cameras.  In conjunction with the Bluefish 444 Catalyst card we can playback these unruly large DPX and Cineon files in full resolution!!!

After digesting all of this....you might need a tums!  Faster than anything I've ever seen.  You'll love it. 

Till next time,

Cutter Stevens
Technical Director
Contact