If you want to know how to optimize your Xsan settings to get the best performance possible, you might have a hard time to find any information on the net.
The combination of block size and stripe breadth as well as the Xsan topology decides how much bandwidth you will get in the end.
Depending on your other settings, the right block size/ stripe breadth combination can make a difference of more than 50 MB/s.
Although many companies have already been working with Xsan and got the knowledge which settings bring the best results, there’s amazingly few information publically avaialble.
This article tries to give you some numbers I collected in a test system.
During the german MacEXPO Björn Adamski and me built up an Xsan-based video environment (http://www.andre-aulich.de/en/perm/xsan-video-server-pro-lab), where we had the chance to try several settings. Also, our business partner Mediatec let us test the Xsan system that had just come back from the Tour de France. Thanks to them for the great support!
I have also set up several Xsan systems for my clients, and these information also made it into this article.
Because the information mentioned here only cover few aspects of Xsan, feel free to get in touch with me, if you have more questions.
Please use my feedback formular to do so.
This is what I tried out at Mediatec:
Hard- and software used for the test
- 1 x Xserve RAID 14 x 180 GB
static IPs, 1 controller 6 HDs RAID level 5, 1 x spare,
1 controller 2 HDs RAID level 1 - 1 x Xserve RAID 14 x 400 GB
static IPs, both controllers: 6 HDs RAID level 5 each, 1 x spare each - Emulex Fiber Channel Switch 355
- GigaBit Ethernet
- Power Mac G5 as Xsan client
Dual 2,7 GHz G5
3,5 GB RAM
Mac OS X 10.4.2
Xsan 1.1 Build M28
250 GB HD
Apple Fiber Channel card
Blackmagic DeckLink HD card - Xserve G5 Cluster Node as Metadata controller
Dual 2,3 GHz G5
3,5 GB RAM
Mac OS X Server 10.4.2
Xsan 1.1 Build M28
80 GB HD
Apple Fiber Channel card - Fibre Channel completely copper wired (the ones that come with the Apple card)
- switchted GigaBit Ethernet dedicated to Metadata
- Xsan Volume mounted on client and MDC
- Test tool: Blackmagic Disk Speed Test, version 5.0
Topology
As already mentioned above, I had two Xserve RAIDs at hand. Because Apple says you should use a dedicated Xserve RAID controller for metadata, in my case the recommended configuration would include two mirrored hard drives on one RAID controller dedicated to metadata storage. The other three RAID controllers can be put together to one storage pool dedicated to store user data.
Because many people with one or two Xserve RAIDs don’t want to give up a full RAID controller to store metadata only, I also tested, if striping metadata across all RAID controllers in the same storage pool as the user data really means a decrease in performance.
My tests suggest, that with two Xserve RAIDs, the performance seems to be the same, no matter if you use a dedicated metadata storage pool or not. Yet, As soon as you work with more than two Xserve RAIDs, a dedicated metadata storage pool really DOES mean better performance.
As I never used more than five Xsan clients in my tests, it might also turn out in your environment, that even with only two Xserve RAIDs you get better results with a dedicated metadata storage pool if you have many files or clients, meaning a lot of file requests at the same time.
The advantage of putting metadata and user data in one storage pool is, that you get more disk space for the same price. Try out, what better fits your needs.
ATTENTION: I heard that some people use the internal hard disks of their Xserves to store metadata. This means, that there is no metadata failover possible, as in case of MDC failure the backup server can not read nor write the metadata. It is strongly recommended to always have a MDC failover server at hand, to be able to upgrade your Xsan while it is still in use. It’s also nice to have it available in case your primary controller goes down anyway…
Back to our metadata. If you have two or more RAIDs, consider dedicating one RAID controller to store your metadata only.
On the Xserve RAIDs, activate read and write cache, but make sure to have the backup battery modules installed. This prevents data loss in case you might experience a power failure. You don’t need to fiddle around with the other cache settings of the Xserve RAIDs. Ususally the standard settings deliver the best results.
How I tested
After setting up the system with the settings described in the table, I mounted the Xsan volume on the client, copied the test program to the Xsan and started the test.
After this I unmounted and stopped the volume, changed the combination of block size and stripe breadth (and sometimes changed the topology of the Xsan, too) and did the next test.
Be aware, that changing the block size and/or stripe breadth of an Xsan volume means that you have to initialize the volume right after. All the data stored on the Xsan will be lost. So before you use your Xsan volume in a production environment, better make sure to choose the best settings before you start to work with it.
Test results
Here are the test results (click on the image to see a larger pdf version of the table):
Here are some graphics that visualize the setups described in the table above.
Setup 1
The two red hard disks are dedicated to store the metadata and journaling data of the Xsan volume. Controllers 2, 3, and 4 are configured as RAID level 5 LUNs and are put together to build a data-only storage pool.
Because the upper Xserve RAID came with smaller hard disks than the lower Xserve RAID, in this scenario we won’t have the full size of the second RAID system available. In the real world, you’d choose a different setup, yet, this test is all about performance, in addition, in a real-world-scenario you would probably use two identical Xserve RAID systems, so this setup makes sense.
The topology used here is very efficient, yet, the blocksize of 4 KB isn’t suited at all for either read or write performance.
Setup 2
In addition to setup 1, in this scenario we added four hard disks located on controller 1 to build a RAID 5 LUN and added it to the data storage pool, that now consists of four LUNs.
Again, you won’t see this in the real world, as the LUNs on controllers 2, 3, and 4 would only use the same disk space as the LUN based on controller 1. Yet, again, this is all about performance, and many people also like to use the storage space left over on controller 1.
What you do in the real world is to use these four disks to build a RAID 5 LUN and use it as a second data storage pool for performance-uncritical data, preferably for data with affinities only.
TIP: In the Peachpit Press book “Xsan Quick-Reference Guide” the authors suggest in a couple of example setups, that you should consider having dedicated storage pools for video, audio, and render data. This would make sense if you could choose optimized blocksize/stripebreadth combinations per storage pool, but this only works on a per volume basis. As Xsan increases its overall performance by striping data across as many Xserve RAID controllers as possible, to me it even seems to make more sense, NOT to choose dedicated storage pools for diverse purposes. If you want to optimize your settings for different purposes, you have to create a second volume with settings for audio, e.g.
I think that choosing more than one storage pool for different kinds of files is based on historical reasons. There was a time where you absolutely needed to have a second hard disk for audio data, as your internal disk in your Mac couldn’t play two SD streams AND a couple of audio streams at the same time.
In times of Xsan this behaviour is completely different. Having a second storage pool is no problem in situations where each of the storage pools is at least one Xserve RAID in size. Because then each storage pool is fast enough to deliver all the speed a client can access. In this situation it doesn’t matter if you have one or more storage pools, in any other case, it can slow down your file access.
If you know another reason why choosing multiple storage pools could make sense, please send me a mail to explain why – maybe I missed something here.
Setup 3
This setup is exactly the same as in setup 2 – with the only little difference, that in setup 2 there were about 81500 files on the root level of the Xsan volume.
As you can see, there is no performance difference between the two tests.
Setup 4
Like setup 3, yet another combination of blocksize/stripe breadth.
Setup 5
Like setup 4, yet there are no user data on the first controller.
As you can see, this results in a distinct performance decrease.
Setup 6
Like setup 5, yet another combination of blocksize/stripe breadth.
Setup 7
In this setup the metadata/journaling data as well as the user data were striped across the four controllers of the Xserve RAIDs using one big storage pool.
In the real world this setup results in a comparable performance as the topology described in setup 4 – at least if you work with two Xserve RAIDs.
If you plan to add more RAIDs later, you should consider starting with setup 4 from the beginning, as adding more Xserve RAIDs later on wouldn’t mean to change the whole topology later on.
Setup 8
Like setup 4, yet another combination of blocksize/stripe breadth.
A close look at the table might bring these things to your attention:
- First you’ll see that if you multiply block size and stripe breadth, it will always be one. This is, because Apple’s Xserve RAIDs are optimized to work with 1 MB chunks of files. If you change your settings always make sure, that
block size x stripe breadth = 1is always true.
- Another thing you might discover is, that there are no tests dealing with block sizes between 8 and 64 KB. This is a pity, as my experience in production environments is, that most systems get the best performance with block sizes between 16 and 64 KB – including for HD.
Unluckily, I haven’t found the time to add these data to my test table, yet. I’ll try to work on this later. - The best settings with the best performance are coloured in red. If you have a close look at the best results for each video format, you’ll see that for read and write performance there are different settings. It seems that block sizes between 32 and 128 Kb are better suited for writing while larger block sizes result in better reading performance.
As in a video production environment the writing performance is usually more important, you might consider this when you set up your Xsan.
Ususally you need real time performance for playing (i.e. reading) your files, while writing (i.e. saving) your files is not as time critical. - As mentioned above, I used six hard disks per Xserve RAID controller, while the seventh disk served as a spare disk. This is a more secure setting for production environments, yet, adding the seventh disk to your storage pool might slightly increase your overall Xsan performance.
I hope that these information are somewhat useful. I’d appreciate to get some feedback if you could use anything mentioned here.