Time Machine

I decided to set up Time Machine backups for my new Macbook Air using my FreeBSD based file server. This server was already configured to serve files over AFP and Samba and I had multicast DNS set up on it. The Time Machine setup was really an incremental improvement of the existing functionality.

There are a lot of guides around describing this kind of setup and a lot of them are out of date (not applicable to OSX 10.7). I found that using the latest version of Netatalk and Avahi along with a little bit of bait-and-switch for the disk image itself gave a satisfactory outcome.

Note: This guide assumes you are running OSX 10.7 and using a FreeBSD 8.2 server.

First thing to do was to update my FreeBSD ports collection and then do a full port upgrade. This took quite a long time (I am a bit lax about keeping ports up to date on this machine).

The relevant commands:

# portsnap fetch extract update
# portmanager -u

If you don’t have portmanager installed:

# cd /usr/ports/ports-mgmt/portmanager && make install clean

Wait a long time, occasionally selecting configuration options at random such that you can’t just leave it running. Oh and try and make sure you avoid anything which selects an optional dependency on gtk or X11 to avoid pulling in the entire universe…

At this point I found some conflicts and had to remove some of the conflicting ports manually using pkg_delete, these were:

# pkg_delete ghostscript8-nox11-8.71_8

# pkg_delete howl-1.0.0_1
# pkg_delete bash3-3.2.51

This is probably very specific to the state of my ports collection however!

When ports are up-to-date the versions of Netatalk and Avahi ought to be (or better):



Netatalk 2.2 in particular is needed to support some of the functionality Time Machine requires.

Configure /usr/local/etc/AppleVolumes.default:

/ "" options:noadouble,tm allow:,


/store "store" options:noadouble,tm allow:bob,luke

(The “tm” option indicates to OSX that this is a valid Time Machine backup share so it’ll appear in the Time Machine list without having to use the infamous “defaults write com.apple.systempreferences TMShowUnsupportedNetworkVolumes 1” hack.)

Configure /usr/local/etc/avahi/avahi-daemon.conf

-- use-ipv6=no

++ use-ipv6=yes

Create /usr/local/etc/avahi/services/afpd.service with contents:



(This will mean the file share shows up with a nice Xserve icon in the Finder and elsewhere. You don’t need to explicitly set up Avahi to advertise AFP since Netatalk 2.2 can automatically register with Avahi – Make sure you configure Netatalk with Avahi support (same for Samba) and if not switch to the port directory in question and do make config to change it).

You’re pretty much done for server configuration at this point. Restart all the applicable services:

# /usr/local/etc/rc.d/dbus restart && /usr/local/etc/rc.d/avahi-daemon restart && /usr/local/etc/rc.d/netatalk restart

On the client side, enable Time Machine and select your file store under “Select Disk”. By default it’ll create a sparsebundle as big as the containing volume (obviously being sparse it’ll only take up space as you start to back up to it). Obviously you will want some control over how big it gets. For example my Mac only has 120GB of disk space so I don’t want the backup set to grow to more than 300GB (to permit some storage of history of files etc.) There isn’t by default a way to easily specify how big the disk image should grow via the UI so this requires a bit of trickery. Also, if you want to enable encryption for your backup you also need to perform some magic.

Say your Mac is called neutrino, like mine, Time Machine will create a sparsebundle disk image at the root of your file share called “neutrino.sparsebundle”. This will by default be unencrypted and as big as the file store. After this is created Time Machine will start to backup to it. Interrupt the backup at this point (make sure the image has been created and that it’s on the “backing up” stage) and disable Time Machine.

Rename “neutrino.sparsebundle” to “neutrino-old.sparsebundle”. Open the disk image in Disk Utility. Create a new sparsebundle in Disk Utility with the settings:

Name: Time Machine Backups


Format: Mac OS Extended (Case-sensitive, Journaled)

Encryption: 128-bit AES encryption (optional, but why would you want an unencrypted backup?!)

Partitions: Single Partition – Apple Partition Map (default)

Image Format: sparse bundle disk image

Save it as “neutrino.sparsebundle” under the root of your Time Machine file share.

Next step is the clever bit. Make sure you have both old and new sparsebundles mounted in Disk Utility and then restore the “Time Machine Backups” volume from the old one onto the new one, replacing the existing (empty) volume.

When this process completes delete the old sparsebundle and unmount the new one. Time Machine can then be re-enabled and will use the new encrypted and size-limited sparsebundle to perform backups to in future.

You may find you need to use the MAC address trick to get Time Machine to recognise that your sparsebundle is the correct one. If after the steps above it still tries to create a new bundle when performing the backup rename the “neutrino.sparsebundle” file to “neutrinoYOURMACADDR.sparsebundle”. So for example if the MAC address of en0 on your machine happens to be: 00:11:22:33:44:55 then you’d rename the file to: “neutrino001122334455.sparsebundle”. Time Machine ought to rename this to just “neutrino.sparsebundle” when it runs the first time.

It would sure be nice if Apple allowed us to configure these kind of things in a less arcane way, but it works nicely once it’s set up.

Useful links:













Inconsistency and poor documentation

I was doing some further work on my Firefox extension SixOrNot last night, trying to improve the behavior and performance of the ctypes DNS resolution component. This uses JS-ctypes to access the native functionality of the underlying OS. Because of this it exposes me to the wonderful world of OS-specific network programming.

I’ve found some lovely inconsistencies which are made much worse by the poor documentation available. The worst offenders so far have been Microsoft and Apple.


On Windows there is support for the posix method getaddrinfo(), this is used to populate a linked list of addrinfo structures which can contain sockaddr structures (which themselves contain either IPv4 or IPv6 addresses – which is what I am after).

getaddrinfo() takes a hostname to lookup, a set of hints (which is an addrinfo struct defining the kinds of addresses you want to get back) and a pointer to the first addrinfo item of your output linked list (which it then populates). After you’re done reading the linked list you call freeaddrinfo() to free the dynamically allocated memory.

Calling this method on different platforms with null for the hints field has differing behavior when it comes to resolution of IPv6 addresses from DNS records. Below is a brief matrix.

Windows XP/Server 2003 – v4 and v6 returned always

Windows Vista/7 – v6 only when a local adapter has a global v6 address

OSX – v4 and v6 returned always

Linux (Ubuntu Natty) – v6 only when a local adapter has a global v6 address

There is a flag which enforces the behavior on Linux/Vista/Win7, AI_ADDRCONFIG, this is explained in the Microsoft documentation thusly:


If the AI_ADDRCONFIG bit is set, getaddrinfo will resolve only if a global address is configured. If AI_ADDRCONFIG flag is specified, IPv4 addresses shall be returned only if an IPv4 address is configured on the local system, and IPv6 addresses shall be returned only if an IPv6 address is configured on the local system. The IPv4 or IPv6 loopback address is not considered a valid global address.

The AI_ADDRCONFIG flag is defined on the Windows SDK for Windows Vista and later. The AI_ADDRCONFIG flag is supported on Windows Vista and later.

What they don’t tell you (and this is critical!) is that AI_ADDRCONFIG is on by default in Vista/Win7 whether you set the flag or not. I only found this out by chance from reading the comments in the code of the Chromium project on this page:



// The following comment in is the best documentation I found
// on AI_ADDRCONFIG for Windows:
// Flags used in "hints" argument to getaddrinfo()
// - AI_ADDRCONFIG is supported starting with Vista
// - default is AI_ADDRCONFIG ON whether the flag is set or not
// because the performance penalty in not having ADDRCONFIG in
// the multi-protocol stack environment is severe;
// this defaulting may be disabled by specifying the AI_ALL flag,
// in that case AI_ADDRCONFIG must be EXPLICITLY specified to
// enable ADDRCONFIG behavior
// Not only is AI_ADDRCONFIG unnecessary, but it can be harmful. If the
// computer is not connected to a network, AI_ADDRCONFIG causes getaddrinfo
// to fail with WSANO_DATA (11004) for "localhost", probably because of the
// following note on AI_ADDRCONFIG in the MSDN getaddrinfo page:
// The IPv4 or IPv6 loopback address is not considered a valid global
// address.
// See http://crbug.com/5234.

The AI_ALL flag is defined in the Microsoft documentation thusly:


If the AI_ALL bit is set, a request is made for IPv6 addresses and IPv4 addressses with AI_V4MAPPED.

The AI_ALL flag is defined on the Windows SDK for Windows Vista and later. The AI_ALL flag is supported on Windows Vista and later.

This is perfectly true, but it misses the key bit of information that if you want to override the default AI_ADDRCONFIG behavior you must use AI_ALL. Additionally it’s kind of misleading in that it hasn’t got a lot to do with the AI_V4MAPPED flag (and if you specify only AI_ALL you’ll get back normal v4 addresses – i.e. not mapped to IPv6 addresses…)

The same is true on Linux however their documentation actually specifies what the default flags are:


All the other fields in the structure pointed to by hints must contain either

0 or a null pointer, as appropriate. Specifying hints as NULL is equivalent

to setting ai_socktype and ai_protocol to 0; ai_family to AF_UNSPEC; and

ai_flags to (AI_V4MAPPED | AI_ADDRCONFIG).


This is a minor point but their developer documentation lists the fields of the addrinfo struct on this page incorrectly:


The ai_addr and ai_canonname fields are reversed compared to what they actually are in netdb.h, this is actually slightly bizarre to me, why would they change two fields of such a basic posix networking structure around? And then mis-document them?

Still, after all that I have a better understanding of at least one cross-platform issue, and my extension works perfectly. Just really frustrating that I wasted so much time because a few lines of documentation were missing/misleading/inaccurate.

Connecting to SQL Server Embedded Edition

Quick note, you can connect to the SQL Server Embedded edition instance using this connection string:


Useful if you need to change the backup model for internal system databases using this provider…

SQL 2008 R2 SP1 – Unattended/command line install

This was a pain to get right, thanks Microsoft for the comprehensive documentation you provide on this kind of thing(!)

The contents of the .exe for SP1 can be extracted using WinZip/7Zip. They contain a setup.exe, this takes the following command line flags:

Microsoft (R) SQL Server 2008 R2 Setup 10.50.2500.00

Copyright (c) Microsoft Corporation. All rights reserved.

setup.exe /[option]={value} /[option]={value} ...

ACTION Specifies a Setup work flow, like INSTALL,
UNINSTALL, or UPGRADE. This is a required
ALLINSTANCES Specifies that all instances are to be included
in the Setup operation. This parameter is only
supported when applying a patch.
CLUSTERPASSIVE Specifies that SQL Server Setup should not manage
the SQL Server services. This option should only
be used in a non-Microsoft cluster environment.
ENU Detailed help for command line argument ENU has
not been defined yet.
FARMACCOUNT User account for running the SharePoint Central
Administration service and connecting to the
SharePoint configuration database.
FARMADMINPORT A port number used to connect to the SharePoint
Central Administration web application.
FARMPASSWORD The password for the farm account.
HELP Displays the command line parameters usage
IACCEPTSQLSERVERLICENSETERMS By specifying this parameter and accepting the
SQL Server license terms, you acknowledge that
you have read and understood the terms of use.
INDICATEPROGRESS Specifies that the detailed Setup log should be
piped to the console.
INSTANCEID Specify the Instance ID for the SQL Server
features you have specified. SQL Server directory
structure, registry structure, and service names
will reflect the instance ID of the SQL Server
INSTANCENAME Specify a default or named instance. MSSQLSERVER
is the default instance for non-Express editions
and SQLExpress for Express editions. This
parameter is required when installing the SQL
Server Database Engine (SQL), Analysis Services
(AS), or Reporting Services (RS).
PASSPHRASE A pass phrase farm administrators will use to add
application servers or Web front end servers to
an existing farm. The pass phrase must be at
least 8 characters and contain a combination of
uppercase, lowercase, numeric, and non-alphabetic
characters (such as !, $, #, %).
QUIET Setup will not display any user interface.
QUIETSIMPLE Setup will display progress only without any user
UIMODE Valid values for the UIMode settings are Normal,
X86 Specifies that Setup should install into WOW64.
This command line argument is not supported on an
IA64 or a 32-bit system.

Press any key to exit...

In addition to the special secret hidden parameter:


Which must be set otherwise it will bomb out of the installation.

I went for the following command line:

\setup.exe /quietsimple /allinstances /indicateprogress /IAcceptSQLServerLicenseTerms

The /indicateprogress flag dumps the log data to stdout, which is useful since it pipes it back through our build system for analysis. The /quietsimple flag gives you in various places a progress bar.

OpenBSD file descriptor limits

Are you seeing messages like this in your logs?:

error: can't create socket: Too many open files

OpenBSD seems to come configured by default with a really low file descriptor limit (128) so increasing it is needed for daemons which are going to open a lot of network connections.

The limits are configured in two places; the system global limit (total files open by all processes) is set in a sysctl:

sysctl kern.maxfiles=7030

The per-login/process limits are set via /etc/login.conf via the openfiles directive. This is further split into openfiles-cur and openfiles-max (current, e.g. the initial value for a new login, and maximum, which is the largest value you can then increase the limit to using ulimit -n)




Would set a limit of file descriptors for daemons (started by root, or via /etc/rc) of 4096 initial, 4096 maximum and for all otherwise unclassified users to 1024 initial and 4096 maximum. Settings take effect on next login.

For performance you can compile /etc/login.conf into a database using cap_mkdb, e.g.:

cap_mkdb /etc/login.conf

Today’s the day!

Today is world IPv6 day, a global test for IPv6 connectivity on some of the biggest sites on the Internet. If you have already got IPv6 connectivity and want to see the change in action check out my Firefox extension SixOrNot which reports on the IPv6 status of the websites you’re visiting, then check out some of the sites in this list.

Zone Information and removing yet another annoying popup

Since XP SP2, Vista and Server 2003 SP2 Microsoft introduced a “security” “feature” into Windows which keeps track of the provenance of files with particular extensions. For example if you download a .exe file using Internet Explorer Windows will remember where the file came from (the big, bad Internet) and place a mark against the file so that attempts to open it result in an additional pop-up warning box.

This isn’t an entirely bad idea given how people seem to be unable to resist running executables they downloaded from the Internet. In an automation context it can be quite irritating. Many of the automation tasks I run involve copying files which will later be run either manually or automatically. Windows will record the origin of files copied from network locations and provide the same prompting behaviour.

As an interesting aside this feature makes use of a little known feature of the NTFS filesystem, that of alternative file streams. These work roughly the same way that resource forks do on HFS/HFS+ (the Mac OSX filesystem) permitting you to have more than one set of data associated with a particular file object. All files in an NTFS filesystem have a default stream which is accessed when no stream is explicitly specified, however you can also specify an unlimited number of alternative ones. These are referred to using the colon (:) operator, e.g.:


May be a regular file (with some content), but:


Would refer to an alternative stream labeled “secret” associated with th file blah.txt. You can create these streams using the command line tools built into Windows, e.g.:

echo Text data in secret > blah.txt:secret

Will put the text “Text data in secret” into the alternative file stream “secret” associated with the file “blah.txt”. You’ll notice that the file size of blah.txt is 0, streams don’t show up in cmd/explorer as contributing to file size. You can view the data using more, e.g.:

more < blah.txt:secret

Will show you the “secret” data. This is a nice little technique for “hiding” data on an NTFS filesystem since not very many people know about it. Some malware uses this technique for example. You can search for alternative file streams using a tool like Streams from Sysinternals.

So what does this have to do with the blocked file protection feature in Vista? Well that’s how it’s implemented, as an alternative file stream, specifically the “Zone.Identifier” stream. This is just like any other file stream so you can write to it and overwrite it if you wish. So if you want to set a file to be “blocked”, you create a file (e.g. zone.txt) containing:


And then set the Zone.Identifier stream of the file in question to the contents of that file:

more zone.txt > somefile.exe:Zone.Identifier

The file will then be “blocked”. To unblock it, set the ZoneId to 0 (or delete the Zone.Identifier stream entirely as the “Unblock” button on the General tab of the file’s properties dialog will do).

The ZoneId value relates to the Internet Explorer zones and there’s a listing of these here:


So one workaround to this problem would be to add a step to every single file copy that removes the Zone.Identifier data from each and every file… Tedious.

You can help to alleviate this issue by turning off the part of the feature which writes these alternative file streams to begin with. This is controlled via the following registry key command:

REG ADD "HKCU\Software\Microsoft\Windows\CurrentVersion\Policies\Attachments" /v "SaveZoneInformation" \
/t REG_DWORD /d 00000001 /f

Or via the local policy setting generally found under “User Configuration” -> “Windows (Components)” -> “Attachment Manager” -> “Do not preserve zone information in file attachments”. Set this to Enabled.

This only prevents Windows from adding the alternative file stream to freshly downloaded files. Any existing ones which have the alternative stream will still be blocked.

See this article for a description of how the Attachment Manager works:


The Sysinternals tool Streams has an option to delete streams from files. Combining this with universal use of the registry key to disable the Attachment Manager from setting zone information should rid you of this menace altogether.

Psexec AcceptEula via Sysprep

The latest versions of the PsTools suite require you to accept their EULA the first time they are run. This is on a per-user basis, so if you try and run something in the system context and forget to use the /accepteula or -accepteula flags it’ll fail. This is quite tedious.

Simply way to get around this is to set the registry key it uses to check if the EULA has been accepted manually. This is under:


With one key per product, e.g. for psexec:


The value pair to set under this is “EulaAccepted”, REG_DWORD, 0x00000001

Quick batch script which does this for all the pstools tools:

REG ADD "HKCU\Software\Sysinternals\PsExec" /v "EulaAccepted" /t REG_DWORD /d 00000001 /f
REG ADD "HKCU\Software\Sysinternals\psfile" /v "EulaAccepted" /t REG_DWORD /d 00000001 /f
REG ADD "HKCU\Software\Sysinternals\PsGetSid" /v "EulaAccepted" /t REG_DWORD /d 00000001 /f
REG ADD "HKCU\Software\Sysinternals\PsInfo" /v "EulaAccepted" /t REG_DWORD /d 00000001 /f
REG ADD "HKCU\Software\Sysinternals\PsKill" /v "EulaAccepted" /t REG_DWORD /d 00000001 /f
REG ADD "HKCU\Software\Sysinternals\PsList" /v "EulaAccepted" /t REG_DWORD /d 00000001 /f
REG ADD "HKCU\Software\Sysinternals\PsLoggedon" /v "EulaAccepted" /t REG_DWORD /d 00000001 /f
REG ADD "HKCU\Software\Sysinternals\PsLoglist" /v "EulaAccepted" /t REG_DWORD /d 00000001 /f
REG ADD "HKCU\Software\Sysinternals\PsPasswd" /v "EulaAccepted" /t REG_DWORD /d 00000001 /f
REG ADD "HKCU\Software\Sysinternals\PsService" /v "EulaAccepted" /t REG_DWORD /d 00000001 /f
REG ADD "HKCU\Software\Sysinternals\PsShutdown" /v "EulaAccepted" /t REG_DWORD /d 00000001 /f
REG ADD "HKCU\Software\Sysinternals\PsSuspend" /v "EulaAccepted" /t REG_DWORD /d 00000001 /f

Of course, this only works for the user you are currently logged in with. However if you’re preparing OS images which will be prepared with Sysprep you can run this as the user you are intending to run sysprep with and the sysprep process will then roll these keys (along with other modifications to HKCU) into the registry profile used for all users when the machine comes out of sysprep. All subsequent users created on that machine should thus have the EULA pre-accepted, neatly bypassing the possibility of it going wrong when running commands.

For the “well-known” SIDs (e.g. S-1-5-18/Local System, S-1-5-19/Local Service and S-1-5-20/Network Service) you can set these directly via HKEY_USERS:

REG ADD "HKU\S-1-5-18\Software\Sysinternals\PsExec" /v "EulaAccepted" /t REG_DWORD /d 00000001 /f
REG ADD "HKU\S-1-5-19\Software\Sysinternals\PsExec" /v "EulaAccepted" /t REG_DWORD /d 00000001 /f
REG ADD "HKU\S-1-5-20\Software\Sysinternals\PsExec" /v "EulaAccepted" /t REG_DWORD /d 00000001 /f

Server 2003, using sysocmgr to install windows components automatically

One important aspect of automating Windows deployment and setup is enabling particular roles on Server systems. For example, enabling the SMTP/POP service for email. On Server 2008/R2 this is really easy using servermanagercmd.exe and passing through a list of the features/roles you want to install. For 2003 this is a little more complex, requiring the use of the sysocmgr.exe tool.

This tool works on .inf files. Running the command:

sysocmgr /i:%windir%\inf\sysoc.inf

Will open up the usual “Windows Components Wizard” part of windows setup (as if you went to Add/Remove programs and clicked on Add/Remove windows components). You can automate which bits of Windows are added/removed by specifying an unattend file with the /u switch, e.g.:

sysocmgr /i:%windir%\inf\sysoc.inf /u:C:\ocm.txt

The answer file then specifies which components you want to add/remove, e.g. to add the POP3/SMTP service:

Pop3Srv = On
Pop3Service = On
Pop3Admin = On

The %windir%\inf\sysoc.inf file contains a roughly complete list of these components, and there’s another listing of them here.

Further flags can be used to refine the process, e.g.:

/r - suppress reboot
/x - suppress init banner
/q - run without UI (useful with /u)
/w - prompt before reboot if using /u
/c - disallow cancel

For a complete list do “sysocmgr /?” on a Server 2003 R2 system.

As an aside, configuring the POP3/SMTP server with a new domain requires the following command:

winpop add somedomain.com

The winpop command can be used to query and control the POP3/SMTP server service as well.

MediaTomb custom import script

I wrote about MediaTomb last year, but forgot to post a followup. MediaTomb builds its own database describing the layout of media it serves, this layout isn’t necessarily similar to the underlying layout of files in the filesystem. In addition various data about the files can be extracted and used to modify the way they display in the device retrieving them. One way this process can be controlled is using JavaScript to script the process.

This is controlled by default by the file import.js (on FreeBSD this is found at /usr/local/share/mediatomb/js/import.js). This is set in the config file via the virtual-layout setting:


I modified my import.js so that it displays the contents of the “TV Series” and “Movies” folders directly under the root of the media server. The filesystem layout is used from there since I am happy with that layout. Much more complicated arrangements are possible of course.

function addVideo(obj)
var chain, show, season;
var location = obj.location.split('/');
var rootindex = 0;
for (var i = 0; i < location.length; i++)
if (location[i] == "Media")
rootindex = i;
chain = new Array();

if (rootindex + 1 < location.length - 1)
for (i = rootindex + 1; i < location.length - 1; i++)
/* Ensure that title doesn't contain file extension */
obj.title = obj.title.substr(0, obj.title.lastIndexOf('.')) || obj.title;
addCdsObject(obj, createContainerChain(chain));

MediaTomb has various options to permit it to automatically rescan content, or even monitor the filesystem for changes. Since my media library very rarely changes I am not too interested in these methods (especially since the rebuilding process is slow and from my experimenting can often lead to duplicates if files are added in the wrong way).

I decided to simply rebuild the entire database whenever I make changes to the content. This takes around 15 minutes but as I only do this at most once a week that’s no great problem. The script is quite simple and consists of stopping the MediaTomb daemon, deleting the existing database (which will force it to rebuild it on next start) and then starting the daemon again with instructions on which parts of the filesystem it should scan and add to its database:


/usr/local/etc/rc.d/mediatomb stop

rm /var/mediatomb/mediatomb.db

/usr/local/bin/mediatomb -d -c /usr/local/etc/mediatomb/config.xml \
-l /var/mediatomb/mediatomb.log -u mediatomb -g mediatomb \
-P /var/mediatomb/mediatomb.pid -a /store/Media/TV\ Series/ \
-a /store/Media/Movies

-c gives the config file, -l the log file, -u/-g the uid/gid to run as, -P instructs it where to put its pidfile and then the two -a options add directories to scan on startup.