I always wondered why there’s the command
/Library/Filesystems/Xsan/bin/snmetadump
available for Xsan metadata dumps, if there’s also ‘dd’ available on UNIX systems like Mac OS X to do things like this.
Yet, a while ago, when I analyzed the various Xsan volume extension strategies, I wondered, if the following approach could be a successful and efficient strategy:
1. Start small with a single Xserve RAID as your Xsan volume. Create two LUNs, each based on six hard disks with RAID level 5, stripe metadata, journaling data, and user data across these two LUNs.
Yes, I know, Apple doesn’t recommend striping metadata with your user data, but with one RAID, you don’t want to lose a whole side of an Xserve RAID for data management only.
2. If later you want to extend your Xsan volume, buy a second Xserve RAID, create a new RAID Level 1 set based on two hard drives on the new RAID and use snmetadump to move metadata from your mixed first RAID to this new LUN.
This way you could move from a shared user / metadata strategy (which makes sense for single RAIDs) to separated user / metadata without reformatting your first RAID.
Long story short: it doesn’t work.
This is how I tested this:
I created a working Xsan with a single (mini) volume.
This is the config file I used (named SANVolume.cfg):
# ****************************************************************************
# A global section for defining file system-wide parameters.
# ****************************************************************************
FileLocks Yes
GlobalSuperUser Yes
Quotas Yes
WindowsSecurity No
ForceStripeAlignment Yes
UnixIdFabricationOnWindows Yes
EnforceACLs No
Debug 0
AllocationStrategy Round
InodeExpandMin 8
InodeExpandInc 32
InodeExpandMax 2048
BufferCacheSize 32M
JournalSize 16M
FsBlockSize 8K
InodeCacheSize 8K
MaxConnections 75
MaxLogSize 10M
ThreadPoolSize 128
# ****************************************************************************
# A disktype section for defining disk hardware parameters.
# ****************************************************************************
[DiskType LUN1Type]
Sectors 490091377
SectorSize 512
# ****************************************************************************
# A disk section for defining disks in the hardware configuration.
# ****************************************************************************
[Disk LUN1]
Status Up
Type LUN1Type
# ****************************************************************************
# A stripe section for defining stripe groups.
# ****************************************************************************
[StripeGroup StoragePool1]
Status Up
Exclusive No
Metadata Yes
Journal Yes
Affinity StorageP
Read Enabled
Write Enabled
MultiPathMethod Rotate
StripeBreadth 128
Node LUN1 0
Then I dumped my metadata into a file:
sudo snmetadump -d SANVolume
This created a file with my metadata at /metadump.SANVolume.
The snmetadump man page tells you it would store this file somewhere at /Library/Filesystems/Xsan/database/metadumps/, but it turned out, that this file will find its place on the top level of your filesystem hierarchy.
Now I added a new LUN to my system and changed the volume config file to look like this (changes in bold):
# ****************************************************************************
# A global section for defining file system-wide parameters.
# ****************************************************************************
FileLocks Yes
GlobalSuperUser Yes
Quotas Yes
WindowsSecurity No
ForceStripeAlignment Yes
UnixIdFabricationOnWindows Yes
EnforceACLs No
Debug 0
AllocationStrategy Round
InodeExpandMin 8
InodeExpandInc 32
InodeExpandMax 2048
BufferCacheSize 32M
JournalSize 16M
FsBlockSize 8K
InodeCacheSize 8K
MaxConnections 75
MaxLogSize 10M
ThreadPoolSize 128
# ****************************************************************************
# A disktype section for defining disk hardware parameters.
# ****************************************************************************
[DiskType LUN1Type]
Sectors 490091377
SectorSize 512
[DiskType LUN2Type]
Sectors 117042952
SectorSize 512
# ****************************************************************************
# A disk section for defining disks in the hardware configuration.
# ****************************************************************************
[Disk LUN1]
Status Up
Type LUN1Type
[Disk LUN2]
Status Up
Type LUN2Type
# ****************************************************************************
# A stripe section for defining stripe groups.
# ****************************************************************************
[StripeGroup StoragePool2]
Status Up
Exclusive Yes
Metadata Yes
Journal Yes
Read Enabled
Write Enabled
MultiPathMethod Rotate
StripeBreadth 128
Node LUN2 0
[StripeGroup StoragePool1]
Status Up
Metadata No
Journal No
Affinity StorageP
Read Enabled
Write Enabled
MultiPathMethod Rotate
StripeBreadth 128
Node LUN1 0
The main things I added are the new LUN2 and the new StoragePool2.
To get the number of sectors for LUN2 you need to type
cvlabel -l
Then I restored the dumped metadata:
sudo snmetadump -r SANVolume
Now I tried to start my Xsan volume, but there was always an error message saying
Apr 19 18:45:34 odbackup fsm[560]: Xsan FSS 'SANVolume[0]': File system Icb check failed! ICB Check Error: [StripeGroup count mismatch]
I guess this means that somewhere in my metadata there’s something still pointing to my old config. cvfsck could not repair the filesystem / metadata mismatch, even not in interactive mode.
In the end it still means, if you want to switch from “metadata and user data together on one or more LUNs” to separated user- and metadata, you still need to reformat your Xsan volume.
Or –still the best to keep working from the beginning– buy at least two Xserve RAIDs to start with and separate your metadata from all your other data. This way you can easily expand your Xsan volume without the need of erasing it later.