In this article I will explain how Xsan 2.2.x’s fsevents feature in conjunction with Archiware PresSTORE’s 4.x Synchronize module can help you creating a secure copy of your data during normal business hours without interrupting your real-time I/O operations.

A. The challenge

Why is it that so many people don’t create copies of their online data?

1. To copy a large volume’s data to tape or disk, you need to spend money for the backup device, and nowadays most production environments need to be as cost-effective as possible. So it’s tempting not to spend money for a backup, which in an ideal world you are never going to use anyway.
2. If the primary online volume fails, most people need to be able to work on the backup as soon as possible, probably even without much or any interruption. That’s why you usually need to have a copy of your online data on a second disk set in exactly the same form like on the primary volume. Using a proprietary backup format on the secondary disk set wouldn’t help you much in disaster scenarios, because you would need to restore your data before you could actually use them. If you consider, that after some time you don’t need instant access to project data anymore, this leads to a typical three tier setup as a minimum environment: primary online volume, secondary online volume, tape archive (with data which can be removed from the disk sets). This setup can be complex and expensive.
3. If you are willing to spend the money for a primary and secondary disk volume, as well as for a tape library, you need to make sure that the copy on your secondary disk set is as recent as possible. With very large volumes, scanning the volume for file changes takes a reasonable amount of time (like a couple of hours) which you probably don’t have very often, probably even not at the weekend. In addition, this scanning process can put so much load on your storage system, that you can’t scan your volumes during normal business operations without interrupting your real-time I/O. So with large volumes and constantly changing files, in many environments you would simply not find a backup window to create regular copies of your data.

Regarding point 1 I’d like to say that yes, in an ideal world you would never need to restore data from your backup. In the real world I’d say, that you would only not need your backup if you have a very good backup strategy. As soon as you don’t set up your backup, you would probably need it. I haven’t seen a single system without some kind of backup, versioning, etc. where people haven’t lost data sooner or later, no matter if it was by accidentally deleting files, file system corruption, or hardware issues with or without power failures.
Ah, and did I mention that you absolutely need a UPS for your whole storage system?

So whatever you do, you need a copy of your data. Some people argue that they still have their footage on tape, Professional disk, P2 media, etc. But what happens with your Photoshop files, FCP project files, Word documents, etc? And how long would it take to re-ingest/-transfer your footage to your storage system?
And what about metadata? If you use an asset management system like Final Cut Server, you need to relink your reingested material to the metadata available in your asset management’s database only. Not so easy, especially if you are in a hurry.
I treat the copy of my XDCAM, P2, etc. files as the new originals, as in most environments the original media (I mean e.g. the Professional disk) would simply not be stored forever, or at least not in an environment, which would guarantee media availability after some years.

So I would simply never recommend to design a production system which doesn’t include any kind of backup.

How you deal with point 2 depends on your environment. If you never work on time-critical projects, you might backup your data from your online volume directly to tape. Depending on your tape library, this can be as fast as copying files over to a second storage system.
If in a disaster scenario you think you would still have enough time to restore your data before you continue working on them, that’s a good approach.
Yet, you still need to scan your online volume for file changes, and you would probably not do that during normal business operation as this might interrupt your real-time I/O.
So not having a second disk based volume would only work in environments which either have not many data on their primary volume (so scanning the filesystem for file changes would be superfast), or they don’t need recent backups and are fine if they only create a nightly backup. But even then you need to make sure that your backup window is large enough, so that you can scan your whole volume and copy the file changes to tape.

If your budget allows for it, you should try to get (at least) a setup like this one:

B. Concepts

Let’s have a look at some concepts, which are going to help us implement the 3 tier setup shown in the picture above.

1. Secondary Online Disk

In point A.2. I said, that you would probably need to design a 3 tier storage environment, with primary online storage, secondary online storage, and tape based archive.
As the secondary online storage is supposed to be a most recent copy of the primary online storage, this raises the question how to mirror the two volumes.

The first thing that comes to my mind is using a block level hardware solution like e.g. Cloverleaf’s iSN. I’ve used it in a 24/7 broadcast environment and was not unhappy with it, but what happens if the file system breaks? This would affect both the primary and the secondary volume. To avoid this you could use snapshots, which are supported by iSN, but the whole thing gets really complex at some stage, and I’d like to keep things as simple as possible.

So I’d prefer to have two independent file systems and make sure, that I only copy files from one file system to the other. If one system breaks, I could then simply repair/recreate it and copy my data onto it.

What happens if the primary volume really fails? How do I access the second volume?

Let’s say your volumes are called XsanVolume1 and XsanVolume2. They share the same size.
If XsanVolume1 is mounted on all your servers and your clients, XsanVolume2 needs to be mounted on your backup/synchronize machine at least, because at least one machine needs access to both volumes to copy data from volume 1 to volume 2.
In case XsanVolume1 fails, you would use Xsan Admin to unmount XsanVolume1 from all servers and clients, mount XsanVolume2 on all servers and clients, and then e.g. use Remote Desktop or something alike to create a symlink on all machines from /Volumes/XsanVolume2 to /Volumes/XsanVolume1.
Test this with your specific applications before you try this in a disaster scenario, but this will work with most apps.

You could of course fully automate this, but I would recommend to prepare the tools for a manual failover. This way you would be able to first investigate the reason for the failure.

2. Filesystem Scan vs. Filesystem Events

The functionality that we need to create a copy of our primary volume on the secondary volume is a synchronization function. If we use PresSTORE, we can still decide to support versioning like in a backup, and this would not prevent us from working on the secondary volume directly if we needed to. But right now we just need a way to synchronize our two volumes.

As I said, scanning our primary volume for file changes since the last synchronization puts a reasonable amount of load on our system and would probably interrupt real-time I/O like ingest and playout, so a full scan of our volume should only happen at night, at the weekend, or whenever we find the time to do so. A full scan can easily take a couple of hours, depending on the number of files on your file system.

The solution to this sounds very easy: let’s use filesystem events instead. This means, that each time a file in our filesystem gets created, deleted, or modified, this event will be written into a database. When my next synchronization starts, it just reads the database and copies the modified data to volume 2.
No scanning necessary, and instead of let’s say 5 hours of scanning, the process takes a couple of seconds only.
Wow!

Since version 2.2 Xsan supports filesystem events. Apple doesn’t really tell you about it directly, but you can extract it from their knowledge base articles if you understand, that Spotlight uses filesystem events as its foundation.
In Xsan environments, every file modification will trigger a filesystem event on the machine which writes the file (if it’s a Mac) as well as on the active metadata controller. The only machine which catches all filesystem events on your Xsan volume is the currently active metadata controller.

3. How PresSTORE Synchronize 4.x uses Xsan 2.2.x Filesystem Events

Even PresSTORE 3.x supports filesystem events, so why do I need PresSTORE4.x? In version 3, the filesystem event needed to happen on the machine on which PresSTORE was running. So with PresSTORE 3 you would need to run your synchronization software on the currently active metadata controller to copy data from volume 1 to volume 2.
The data copy processes would put so much load on your MDC, that its latency during metadata access would increase until your clients might loose frames during ingest or playout.
The idea in version 4 is to run the Synchronize module on a normal client system and have a little tool running on the MDC, which does nothing but collecting all the filesystem events of your active Xsan volumes.
Whenever your Sync process starts on the PresSTORE machine, it will simply copy the minimal information from the MDC, check if it needs to move some data, and then do the dirty work without impacting the load of the active MDC.
You can say, that PresSTORE 3′s fsevents implementation was supposed to support local HFS+ volumes only, while version 4 now supports fsevents in Xsan environments, too.

How does it look like?
You just install the PresSTORE client software on your MDCs (remember that you need a valid license for each of them), then you set up a synchronize plan which syncs XsanVolume1 to XsanVolume2.
In the “Synchronize Options” section of the sync plan you now find the option to set up an fsevents server, which in Xsan environments has to be the currently active MDC:

In case your primary MDC goes down, just select the secondary MDC as fsevents server for your sync plan.

4. Apple’s Implementation of Filesystem Events in Xsan 2.2 (in my understanding) and how to deal with it

There’s the File System Events Programming Guide available on the Apple website, which describes the general fsevents mechanisms Apple uses in HFS+ and Xsan 2.2.
At this stage you might think that with fsevents support, PresSTORE would simply get a list of files which have changed and would then work through it.
That would be great. If you work in the fsevents team at Apple and read this article, please add this feature – that would be so awesome.

Anyway. What really happens if the Apple fsevents programming guide applies to the Xsan world, too, is this:

• Instead of receiving an event, that a file has changed, you get an event, that something within a folder has changed. So your application (like PresSTORE), still needs to scan this folder for file changes. In video environments this could be a folder with an image sequence (150.000 files?, no problem, that’s just 40 minutes made of 60 frames per second), so scanning all these files would take some time. If you work with Final Cut Server, most of the assets will be stored on the root level of one folder, so you also end up with many files in one folder. Each time you upload a txt file into Final Cut Server, you get an fsevent, that you need to scan the whole FCSvr Library. You can structure your Final Cut Server environment to better work with this fsevents strategy, but you can’t rely on FCSvr’s default settings. Be aware of this when you design your Xsan’s folder structure.
• When you have too many fsevents at the same time or within a short time frame, the fsevents mechanism can merge these events and point your application to the common point of the events in the folder hierarchy. In the worst case this can mean that the fsevent says “Scan your whole Xsan volume”. Your app could then still ignore the whole event, but hey, you’d like to protect your data, wouldn’t you?
• All applications which register for fsevents share the same buffer. The first application to work on the fsevents list is Spotlight, which seems to deal with it pretty well. If you use other fsevents application, they might slow down the process so much, that new events don’t fit into the buffer anymore. This means, you will not catch all the events, which again means, that a part of your data would not be protected. Yet, you should not run any software on the MDC anyway, so chances are very, very high, that you never run into this problem.

The main message here is that in very rare situations it might happen that the fsevents mechanism doesn’t catch all the file operations on your Xsan volume, so at least from time to time you would still need a full scan of your volume.
I would recommend to use fsevents for your Synchronize module during the day and during normal business operation, and then use full scans as often as possible. If this is once a week, I would feel secure.

C. To Do

While Archiware’s implementation of fsevents support in Xsan environments is simply great, the current version doesn’t automatically deal with Xsan Metadata Controller failover. The perfect solution would be if for any Xsan volume you could enter multiple MDCs as fsevents source in a PresSTORE Synchronize plan, and then PresSTORE would automatically detect which one is the currently active server. If you don’t have a split-brain problem with one of your volumes (or run software on one of the MDCs which writes data to your Xsan volume), there will always be a single MDC serving your fsevents, so this is something which would greatly improve the software.
If you think that you really need automatic failover detection in PresSTORE 4, please go ahead and file a feature request at Archiware.

If you want to copy asset metadata from an external MAM into Final Cut Server, this basically includes the following steps:

1. copy media file into Final Cut Server
2. copy related metadata as an XML file into Final Cut Server

Both steps would work with or without watch folders, you can e.g. copy new assets into FCSvr as part of a Final Cut Pro project file and let Final Cut Server automatically grab any related XML files after analyzing a file. The real challenge lies in creating the proper XML file.

At http://developer.apple.com/mac/library/technotes/tn2009/tn2255.html you find a very good article, which describes how the XML file needs to be structured.
If your external MAM can create Final Cut Server import XML file structures directly, that’s great. If not, you need to restructure the XML file to prepare it for Final Cut Server usage.

While the latter is not part of this article, the main challenge lies in the fact, that Final Cut Server creates an asset ID for each new asset in its database, which needs to be contained in the related import XML file. This article mainly explains the process of finding this ID and writing it into the XML file.

Let’s start.

First we create a “Run an external script or command” response in the Final Cut Server administration panel. I call it “Add FCSvr Asset ID to VPMS XML”, as in my case I take an XML file as my input file, which has been generated by S4M’s VPMS.

Then I enter the path to the script I am going to show you in a minute, and then tell FCSvr to send the file name and the asset id as parameters:

At a later stage this would basically invoke my script like e.g.

/usr/local/vpms2fcsvr mytestfile.mxf 34

where “mytestfile.mxf” is the name of my file, and “34″ is the asset ID.

Now let’s have a look at the script, which is going to process everything:

#!/bin/bash
#
#######################################################
# #
# vpms2fcsvr reads an XML file exported by S4M's #
# VPMS and adds the Final Cut Server asset id of #
# the related media file to it. #
# copyright 2010 by Andre Aulich, www.andre-aulich.de #
# #
#######################################################
# First we define some variables:
VPMSFOLDER="/Users/cadmin/Desktop/Final Cut Server/VPMS_IN"
LOGFILE="/Users/cadmin/Desktop/Final Cut Server/vpms2fcsvr.log"
MEDIAFILENAME="$1"
ASSETID="$2"
TOBEREPLACED="STRING_TO_BE_REPLACED"
FCSvrXMLIN="/Users/cadmin/Desktop/Final Cut Server/Watchers/XML"
# Log some debugging information:
echo "Starting XML import at "$(date)"." >> "$LOGFILE"
echo "Media filename is "$MEDIAFILENAME"." >> "$LOGFILE"
echo "Asset ID is "$ASSETID"." >> "$LOGFILE"
# Let's generate the path to the XML file:
x="$MEDIAFILENAME"
XMLFILE="${x%.*}".xml
XMLFILEPATH="$VPMSFOLDER/$XMLFILE"
echo "The XML file is named \""$XMLFILEPATH"\"." >> "$LOGFILE"
# Now it's a good time to quit and log an error if the XML file doesn't exist.
if ! [[ -e "$XMLFILEPATH" ]]; then
echo "No XML file found at "$XMLFILEPATH". Can't import metadata from VPMS." >> "$LOGFILE"
exit 0
fi
# Now we replace STRING_TO_BE_REPLACED in our XML file with the FCSvr asset ID:
sed -i "" s/"$TOBEREPLACED"/"$ASSETID"/ "$XMLFILEPATH" >> "$LOGFILE"
# Copy the modified XML file into the FCSvr XML watch folder and remove the original XML file.
cp "$XMLFILEPATH" "$FCSvrXMLIN"/
rm "$XMLFILEPATH"
# We also remove the original media file.
rm "$VPMSFOLDER"/"$MEDIAFILENAME"

You see, there’s no magic involved.
But let’s go through it step by step:
First I tell my script where it finds all the media and XML files exported by your external MAM:
VPMSFOLDER="/Users/cadmin/Desktop/Final Cut Server/VPMS_IN"
This folder is not a Final Cut Server device.
Then I define a log path:
LOGFILE="/Users/cadmin/Desktop/Final Cut Server/vpms2fcsvr.log"
In the next step I map the FCSvr parameters filename and asset id to my scripts global variables:
MEDIAFILENAME="$1"
ASSETID="$2"
The incoming XML file uses a fixed text string instead of the asset ID which I am going to replace.
So I need to define which text string needs to be replaced:
TOBEREPLACED="STRING_TO_BE_REPLACED"
My final XML input file will be copied into a FCSvr watch folder, and here I define the path of that folder:
FCSvrXMLIN="/Users/cadmin/Desktop/Final Cut Server/Watchers/XML"
For debugging purposes, I log some information each time I invoke the script:
# Log some debugging information:
echo "Starting XML import at "$(date)"." >> "$LOGFILE"
echo "Media filename is "$MEDIAFILENAME"." >> "$LOGFILE"
echo "Asset ID is "$ASSETID"." >> "$LOGFILE"
Starting with the file name of the media file, I generate the path of the related XML file:
# Let's generate the path to the XML file:
x="$MEDIAFILENAME"
XMLFILE="${x%.*}".xml
XMLFILEPATH="$VPMSFOLDER/$XMLFILE"
echo "The XML file is named \""$XMLFILEPATH"\"." >> "$LOGFILE"
If this XML doesn’t exit, we quit the script:
# Now it's a good time to quit and log an error if the XML file doesn't exist.
if ! [[ -e "$XMLFILEPATH" ]]; then
echo "No XML file found at "$XMLFILEPATH". Can't import metadata from VPMS." >> "$LOGFILE"
exit 0
fi
Now we replace the place holder text string in the original XML file with the asset ID:
# Now we replace STRING_TO_BE_REPLACED in our XML file with the FCSvr asset ID:
sed -i "" s/"$TOBEREPLACED"/"$ASSETID"/ "$XMLFILEPATH" >> "$LOGFILE"
And then we copy the new XML file into our XML watch folder before deleting the original XML file:
# Copy the modified XML file into the FCSvr XML watch folder and remove the original XML file.
cp "$XMLFILEPATH" "$FCSvrXMLIN"/
rm "$XMLFILEPATH"
We also delete the original media file:
# We also remove the original media file.
rm "$VPMSFOLDER"/"$MEDIAFILENAME"
Removing the media and XML files is optional depending on your workflow, and instead of in-place-replacing the asset ID in the XML file you might also want to not touch the original XML file but redirect the “seded” XML file into the XML watch folder directly.

Take this as an example only and adapt it to your needs.

Now, as a next step, we need to tell Final Cut Server to trigger our script whenever a new asset gets created.
To do so, open the subscription panel in the FCSvr admin interface and create a new subscription which is attached to assets.
I call my subscription “Update VPMS metadata during asset creation”, enable it, and choose “Analyzed” as the only Event Type filter.
This means, that my subscription gets triggered whenever an asset gets analyzed. If you choose “Created” as your trigger, your script might be invoked before Final Cut Server analyzed technical metadata of your file, and then Final Cut Server might overwrite the metadata you just imported through the XML import. Feel free to choose the filter which best fits your needs.

In the Response List I select my response “Add FCSvr Asset ID To VPMS XML” and then I enable the subscription:

Now I need to define my XML watch folder.
I take Final Cut Server’s default Watchers device and create a subfolder inside called “XML”.
Now I create a “Read XML” response called “Read XML”.
After that I create a watcher called “Import translated VPMS XML” with these settings:


As you can see, I use the XML subfolder of my Watchers device as the Monitor Address, then I use “Read XML” and “Delete” as my responses, which means, that the XML will be read in and be deleted afterwards, if the read process worked well. This all gets triggered after a file has been created.
The process starts if the files has been listed twice without file size changes, and FCSvr pauses for 10 seconds before listing a file again, so the whole process takes at least a bit more than 10 seconds.

My watcher only scans for xml files and ignores any other file types.

If you enable this job, and copy both a media file and a related XML file (with the same name like the media file, but with xml as suffix) into our input folder, and from there drag the media file into FCSvr, Final Cut Server will now write the asset’s ID into the XML and import the XML into its DB.

Of course this assumes, that your incoming XML file is already structured in the way Final Cut Server likes it.
If not, you might want to adapt the script we use to restructure the incoming XML file.
For this you might want to use XMLStarlet within bash or better use Ruby, Python, or Perl instead.

If you need any help designing your XML workflow, please feel free to get in touch with me.

If you are looking for iPad printing solutions, you might have come across funny approaches like this one, but this article is about a serious solution.

Credits: to use Mac OS X Server’s mail services to provide printing features for the iPad was Armin Briegel’s idea. You can follow him on Twitter. I was just the guy to write the few lines of code.

The idea is very easy: iPad’s iWork applications as well as other iPad apps can export PDF files, which you can send to an email address of choice.

If now you use a Mac OS X Server machine (can be a Mac mini server, e.g.) which runs mail services to print all PDF files sent to a specific address, you have an easy solution to the iPad printing “challenge”.

I guess you know how to set up Mac OS X Server 10.6 to run basic mail services, if not, read my wonderful O’Reilly book on it :-D The book comes in german only, so see it as a chance to improve your german :-D , or get in touch with O’Reilly, that you want it translated into english.

So this is how it works:

1. List available printers and test them

First open the Terminal on your Mac OS X Server 10.6 and type the following line to see which printers are available for our mail printer.
mothership:iPadPrint root# lpstat -a
EPSON_Perfection_2480_2580 accepting requests since Mon Aug 31 18:33:07 2009
Fax accepting requests since Tue May 11 21:25:40 2010
hp_deskjet_5550 accepting requests since Tue May 11 21:55:46 2010
In this example you see that there are three printers available on my machine, where the EPSON device actually is a scanner, and there is a Fax available, too, but imagine that there are multiple printers you might want to use.

Now try if you can print PDF files to one of your printers:
lp -d "hp_deskjet_5550" 1.pdf
In the line above replace “hp_deskjet_5550″ with the name of your printer. Replace “1.pdf” with the absolute file path of a pdf file.
If this works fine, go to the next step.

2. Set up printing mail accounts on Mac OS X Server

I assume that your Mac OS X Server runs as an Open Directory Master. Now open the Workgroup Manager and create a new mail account for your printer. In my case the user will be called “hp_deskjet_5550″, the domain is “andre-aulich.de”, so the complete email address will be “hp_deskjet_5550@andre-aulich.de”.

Note: If you don’t want some Internet folks printing thousands of pages on your machines, make sure that this machine can not be reached from the internet. If this is not possible, create a dedicated internal domain which can not be reached from outside.

This is how the new account looks like in Workgroup Manager:


Save theses settings, send an email without any attachements, and a second one with a PDF file as attachment to this account, and then open the Terminal and change into your dovecot’s mail directory:
mothership:mail root# cd /var/spool/imap/dovecot/mail
mothership:mail root# ls -al
total 0
drwxrwxr-x 10 _dovecot mail 340 Mar 25 09:56 .
drwxr-xr-x 4 root wheel 136 Aug 31 2009 ..
drwx------ 15 valerie mail 510 Nov 10 2009 14167B4C-211D-4AA9-A2B4-4641AD5C508B
drwx------ 25 andre mail 850 May 12 11:38 22D9FA73-A1CA-4981-AD37-3E06D7E4F457
drwx------ 13 public mail 442 Nov 10 2009 2AF362D6-E665-48D7-98AF-E347B055135A
drwx------ 11 ab mail 374 Mar 12 10:48 2EF9F9D2-69FD-4670-B1FE-2C8FC55D3675
drwx------ 16 ical mail 544 Jan 15 16:32 A1BB97FC-5E6D-491C-AAED-F8B9212FE3D3
drwx------ 10 hp_deskjet_5550 mail 340 Mar 25 09:56 B806EEC7-9C41-49B7-8D7C-DF507DFE21DF
drwx------ 16 sadmin mail 544 Nov 18 12:15 E8043686-DCCC-432F-BD96-85C400288357
drwxrwxrwx 7 andre mail 238 Nov 10 2009 Public
mothership:mail root# cd B806EEC7-9C41-49B7-8D7C-DF507DFE21DF/
mothership:B806EEC7-9C41-49B7-8D7C-DF507DFE21DF root# ls -al
total 32
drwx------ 10 hp_deskjet_5550 mail 340 Mar 25 09:56 .
drwxrwxr-x 10 _dovecot mail 340 Mar 25 09:56 ..
drwx------ 2 hp_deskjet_5550 mail 68 Mar 25 09:56 cur
-rw------- 1 hp_deskjet_5550 mail 84 Mar 25 09:56 dovecot-uidlist
-rw------- 1 hp_deskjet_5550 mail 8 Mar 25 09:56 dovecot-uidvalidity
-rw------- 1 hp_deskjet_5550 mail 0 Mar 25 09:56 dovecot-uidvalidity.4bab2555
-rw------- 1 hp_deskjet_5550 mail 220 Mar 25 09:56 dovecot.index.log
-rw------- 1 hp_deskjet_5550 mail 11 Mar 25 09:56 maildirsize
drwx------ 3 hp_deskjet_5550 mail 102 Mar 25 09:56 new
drwx------ 2 hp_deskjet_5550 mail 68 Mar 25 09:56 tmp
mothership:B806EEC7-9C41-49B7-8D7C-DF507DFE21DF root# cd new
mothership:cur root# ls -al
total 0
drwx------ 2 hp_deskjet_5550 mail 68 Mar 25 09:56 .
drwx------ 10 hp_deskjet_5550 mail 340 Mar 25 09:56 ..
-rw------- 1 hp_deskjet_5550 mail 1.3K May 12 12:02 1273658576.M795669P68535.mothership.intern,S=1327,W=1364:2,
-rw------- 1 hp_deskjet_5550 mail 1.3K May 12 12:03 1273658602.M6558P68554.mothership.intern,S=1325,W=1362:2,
-rw------- 1 hp_deskjet_5550 mail 77K May 12 12:05 1273658717.M385494P68662.mothership.intern,S=78367,W=79416:2,
-rw------- 1 hp_deskjet_5550 mail 77K May 12 12:05 1273658742.M267533P68676.mothership.intern,S=78365,W=79414:2,
What have we done here? Dovecot stores emails in /var/spool/imap/dovecot/mail. We list the content of the folder and see that each user has its own folder, named like the UUID of the user. If you don’t know how to look up the UUID of a user, or if you just don’t want to spend the time doing it, just look for the folder which is owned by our new printer user.
If you need to look up the UUID of our newly created user, you can also type
dscl /LDAPv3/127.0.0.1 -read /Users/hp_deskjet_5550 | grep apple-generateduid
in Terminal to get an output like this:
dsAttrTypeNative:apple-generateduid: C784550D-B2A6-4881-8DAF-0FC96D4BE8ED
Now you can search for the mail folder of our freshly created mail user.
In the code above we see that the folder B806EEC7-9C41-49B7-8D7C-DF507DFE21DF is owned by hp_deskjet_5550, so we change into this directory and list its contents. New mails will always be copied into the “new” folder, so we change into this directory and list its content, which already holds 4 emails in my case.

3. Extract attachments from emails and print them

First, make sure you have MacPorts installed, then install ripmime:
bash-3.2# port install ripmime
---> Computing dependencies for ripmime
---> Fetching ripmime
---> Attempting to fetch ripmime-1.4.0.6.tar.gz from http://lil.fr.distfiles.macports.org/ripmime
---> Verifying checksum(s) for ripmime
---> Extracting ripmime
---> Configuring ripmime
---> Building ripmime
---> Staging ripmime into destroot
---> Installing ripmime @1.4.0.6_0
---> Activating ripmime @1.4.0.6_0
---> Cleaning ripmime

Now, create a new directory and a bash script which we are going to use for further automation:
mkdir -p /usr/local/iPadPrint/
cd /usr/local/iPadPrint/
touch printpdfs
chmod +x printpdfs
Open the script using nano and insert the following code:
#!/bin/bash
TMPDIR="/usr/local/iPadPrint/tmp/"
mkdir -p /usr/local/iPadPrint/tmp/
for i in /var/spool/imap/dovecot/mail/C784550D-B2A6-4881-8DAF-0FC96D4BE8ED/new/*; do
if [[ "$(grep "Content-Type: application/pdf;" "$i")" == "" ]]; then
echo "Deleting file $i as it holds no pdf files."
rm "$i"
else
echo "Printing pdf attachment(s) of file $i"
/opt/local/bin/ripmime -i "$i" -d "$TMPDIR"
rm "$i"
fi
done
for i in "$TMPDIR"*; do
if [[ "$i" != *.pdf ]]; then
echo "$i deleted. No pdf file"
rm -r "$i"
else
echo "$i will be printed."
lp -d "hp_deskjet_5550" "$i"
rm "$i"
fi
done

You can also download the script from here. Just extract it, copy it into /usr/local/iPadPrint and make it executable.
Change /var/spool/imap/dovecot/mail/C784550D-B2A6-4881-8DAF-0FC96D4BE8ED/new/* to match your environment.
Now invoke the script.

It will extract all attachments out of your emails. If an email doesn’t contain a pdf file, it will be deleted without printing.
The tool will print all contained pdf files and delete everything after printing.

4. Automate it

To run this whenever a new email arrives in the account of our printer mail account, download this file, copy it into /Library/LaunchDaeomons, and make sure, its permissions look like this:
-rw-r--r-- 1 root wheel 458 May 12 13:38 de.andre-aulich.ipadprinter.plist
Change the line /var/spool/imap/dovecot/mail/C784550D-B2A6-4881-8DAF-0FC96D4BE8ED/new/ to match your environment.
Then load the job:
launchctl load -w /Library/LaunchDaemons/de.andre-aulich.ipadprinter.plist

Now Mac OS X Server 10.6 automatically discovers new print jobs you send via email and prints all included pdf files to the selected printer.

You can use this solution to work with multiple printers by simply creating more scripts and launchd jobs.

Hope this helps!

P.S.: Most cli commands need to be invoked as root user or with sudo.

Now download my little middleware tool fiprecon to connect Final Cut Server to Archiware PresSTORE in a way which can look like this:

Make sure to install fiprecon on your Final Cut Server system. The PresSTORE server can be a separate machine which fiprecon controls over the network, but fiprecon supports running FCSvr and PresSTORE on the same machine, too.

After the installation, run the config tool using

sudo /usr/local/fiprecon/bin/config.command

and enter the paths /usr/local/fiprecon/bin/postarchive.command and /usr/local/fiprecon/bin/prerestore.command as postarchive and prerestore script paths of your archive volume in FCSvr.

I recommend to set the retry count in FCSvr to 10 and the timeout setting to 300 seconds.

I wrote a detailed documentation about “Backup and Archive setups with Final Cut Server, Xsan, and PresSTORE”, which can be downloaded here and describes fiprecon in detail.

It also explains these topics:

• Archiving FCSvr assets on tape.
• Automatic archiving after x days
• Automatic archiving right after asset creation (great for moving video tape based archives into the digital world)

Please feel free to use both the tools and the documentation – they come for free.
If you need any customization, or any other kind of Final Cut Server consulting, please get in touch with me.

Looking forward for your feedback!

Besides other topics the presentation deals with

• Rough Cut Editing
• FCSvr CLI
• XML and data exchange with other MAMs
• Automatic fetching of news feeds incl. XML metadata
• Working with growing clips
• Automatic XDCAM ingest
• FCSvr Archiving to tape

The presentation contains a great deal of demo parts, so not every topic will be explained in detail in the linked PDF file.

If you got any questions regarding the topics dealt with in the presentation, or if you’d like to hire me for one of these things, please use my feedback form to get in touch with me.

I will follow the setup guide described in my new book published by O’Reilly (in german). Everything will be done using a live demo system.

More information regarding entry fee, location, etc. can be found at http://www.lob.de/pics/BK04_Veranstaltung02032010.jpg.

I am looking forward to seeing you there!

It’s available in german and explains you how to set up your own mail, calendar, addressbook, and webserver based on Mac OS X Server 10.6 within no time.

You find more details about it at O’Reilly. The focus of the book is to use Mac OS X Server as the communication central for your Mac OS X, Windows, Linux, and iPhone users.

I’d appreciate your feedback!

You will find more details about the session and the event at the VideoVisions site. Please register for the free event.

The same event will take place in London on March 23rd.

One of the best ways to backup and/or archive your video data, is by using Quantum’s StorNext filesystem in conjunction with their StorNext Storage Manager.

StorNext is compatible with Apple’s Xsan filesystem, and it’s very easy to replace your existing Apple Xsan metadata controllers with Linux-based StorNext metadata controllers, which is a prerequisite if you want to implement Quantum’s StorNext Storage Manager.

What’s so special about the Storage Manager, though?

If you create files on your filesystem, there’s three ways for your backup and archive software to find out, that there are new files which need to be copied to a second storage device:

• The backup software can scan the filesystem for modified or freshly created files. This puts a reasonable amount of load on your storage, so this is an approach for environments with long backup windows. During real-time operations you usually don’t want to scan your central storage.
• The application, which writes a file, needs to inform the backup solution, that it has to back up a file. While this works between Final Cut Server and Archiware PressSTORE, e.g., this depends on the operability between your application and your backup/archive solution.
• The filesystem can send a filesystem event to your backup/archive app. This way your backup app doesn’t need to scan your whole filesystem and can write your data in a very efficient way.

Archiware PresSTORE supports filesystem events with HFS+ and Xsan 2.x volumes, so PresSTORE would be one way to back up your data without the need to scan your Xsan volumes for changed files.

Another way of doing this is the StorNext Storage Manager. The Storage Manager integrates with the StorNext filesystem, and can send freshly created or modified files to the backup and/or archive without scanning your central SAN volume.

In version 3.5.1 the Storage Manager runs on the active metadata controller, which under certain circumstances might cause so much load, that filesystem latency exceeds the critical limit for real-time operations. To deal with this, you can either schedule Storage Manager activity to run at uncritical times, or you use the brand-new StorNext Data Mover (part of StorNext 4.0), which means, that your primary MDC works as a MDC only, and a dedicated client machine actually moves files into the archive to keep your primary MDC responsive, even if there’s lots of Storage Manager activity going on.

Another great thing which comes with StorNext 4.0 is deduplication. Deduplication allows your filesystem to use less disk or tape space when it stores your data. If StorNext sees that a data block in a new file is the same like a data block in a file which is already part of the archive, then it doesn’t store that data block again, but instead creates a reference from the new file to the existing data block.
This is great for archive volumes, yet, on your primary (online) storage, this slows things down a bit, so you might want to turn on deduplication for your archive only, if you work in real-time environments.

If everything is set up properly using StorNext file system and Storage Manager, it allows you to work on your central storage without the necessity for backup/archive windows.

If you need help planning or implementing a StorNext based solution, please feel free to get in touch with me.

If you want to change this behavior, you just need to open the file /etc/postfix/main.cf and change the line

header_checks = pcre:/etc/postfix/custom_header_checks

to

# header_checks = pcre:/etc/postfix/custom_header_checks

Afterwards, type

sudo postfix reload

to activate the new setting.

You could also change the rule set in /etc/postfix/custom_header_checks to do the same thing, yet, as currently the only rule in this file is to reject incoming mails with an empty subject, we can easily ignore the whole file like described above.