Zone Information and removing yet another annoying popup

Since XP SP2, Vista and Server 2003 SP2 Microsoft introduced a “security” “feature” into Windows which keeps track of the provenance of files with particular extensions. For example if you download a .exe file using Internet Explorer Windows will remember where the file came from (the big, bad Internet) and place a mark against the file so that attempts to open it result in an additional pop-up warning box.

This isn’t an entirely bad idea given how people seem to be unable to resist running executables they downloaded from the Internet. In an automation context it can be quite irritating. Many of the automation tasks I run involve copying files which will later be run either manually or automatically. Windows will record the origin of files copied from network locations and provide the same prompting behaviour.

As an interesting aside this feature makes use of a little known feature of the NTFS filesystem, that of alternative file streams. These work roughly the same way that resource forks do on HFS/HFS+ (the Mac OSX filesystem) permitting you to have more than one set of data associated with a particular file object. All files in an NTFS filesystem have a default stream which is accessed when no stream is explicitly specified, however you can also specify an unlimited number of alternative ones. These are referred to using the colon (:) operator, e.g.:

blah.txt

May be a regular file (with some content), but:

blah.txt:secret

Would refer to an alternative stream labeled “secret” associated with th file blah.txt. You can create these streams using the command line tools built into Windows, e.g.:

echo Text data in secret > blah.txt:secret

Will put the text “Text data in secret” into the alternative file stream “secret” associated with the file “blah.txt”. You’ll notice that the file size of blah.txt is 0, streams don’t show up in cmd/explorer as contributing to file size. You can view the data using more, e.g.:

more < blah.txt:secret

Will show you the “secret” data. This is a nice little technique for “hiding” data on an NTFS filesystem since not very many people know about it. Some malware uses this technique for example. You can search for alternative file streams using a tool like Streams from Sysinternals.

So what does this have to do with the blocked file protection feature in Vista? Well that’s how it’s implemented, as an alternative file stream, specifically the “Zone.Identifier” stream. This is just like any other file stream so you can write to it and overwrite it if you wish. So if you want to set a file to be “blocked”, you create a file (e.g. zone.txt) containing:

[ZoneTransfer]
ZoneId=3

And then set the Zone.Identifier stream of the file in question to the contents of that file:

more zone.txt > somefile.exe:Zone.Identifier

The file will then be “blocked”. To unblock it, set the ZoneId to 0 (or delete the Zone.Identifier stream entirely as the “Unblock” button on the General tab of the file’s properties dialog will do).

The ZoneId value relates to the Internet Explorer zones and there’s a listing of these here:

http://msdn.microsoft.com/en-us/library/ms537183.aspx

So one workaround to this problem would be to add a step to every single file copy that removes the Zone.Identifier data from each and every file… Tedious.

You can help to alleviate this issue by turning off the part of the feature which writes these alternative file streams to begin with. This is controlled via the following registry key command:

REG ADD "HKCU\Software\Microsoft\Windows\CurrentVersion\Policies\Attachments" /v "SaveZoneInformation" \
/t REG_DWORD /d 00000001 /f

Or via the local policy setting generally found under “User Configuration” -> “Windows (Components)” -> “Attachment Manager” -> “Do not preserve zone information in file attachments”. Set this to Enabled.

This only prevents Windows from adding the alternative file stream to freshly downloaded files. Any existing ones which have the alternative stream will still be blocked.

See this article for a description of how the Attachment Manager works:

http://support.microsoft.com/kb/883260

The Sysinternals tool Streams has an option to delete streams from files. Combining this with universal use of the registry key to disable the Attachment Manager from setting zone information should rid you of this menace altogether.

Advertisements

Psexec AcceptEula via Sysprep

The latest versions of the PsTools suite require you to accept their EULA the first time they are run. This is on a per-user basis, so if you try and run something in the system context and forget to use the /accepteula or -accepteula flags it’ll fail. This is quite tedious.

Simply way to get around this is to set the registry key it uses to check if the EULA has been accepted manually. This is under:

"HKCU\Software\Sysinternals\"

With one key per product, e.g. for psexec:

"HKCU\Software\Sysinternals\PsExec"

The value pair to set under this is “EulaAccepted”, REG_DWORD, 0x00000001

Quick batch script which does this for all the pstools tools:

REG ADD "HKCU\Software\Sysinternals\PsExec" /v "EulaAccepted" /t REG_DWORD /d 00000001 /f
REG ADD "HKCU\Software\Sysinternals\psfile" /v "EulaAccepted" /t REG_DWORD /d 00000001 /f
REG ADD "HKCU\Software\Sysinternals\PsGetSid" /v "EulaAccepted" /t REG_DWORD /d 00000001 /f
REG ADD "HKCU\Software\Sysinternals\PsInfo" /v "EulaAccepted" /t REG_DWORD /d 00000001 /f
REG ADD "HKCU\Software\Sysinternals\PsKill" /v "EulaAccepted" /t REG_DWORD /d 00000001 /f
REG ADD "HKCU\Software\Sysinternals\PsList" /v "EulaAccepted" /t REG_DWORD /d 00000001 /f
REG ADD "HKCU\Software\Sysinternals\PsLoggedon" /v "EulaAccepted" /t REG_DWORD /d 00000001 /f
REG ADD "HKCU\Software\Sysinternals\PsLoglist" /v "EulaAccepted" /t REG_DWORD /d 00000001 /f
REG ADD "HKCU\Software\Sysinternals\PsPasswd" /v "EulaAccepted" /t REG_DWORD /d 00000001 /f
REG ADD "HKCU\Software\Sysinternals\PsService" /v "EulaAccepted" /t REG_DWORD /d 00000001 /f
REG ADD "HKCU\Software\Sysinternals\PsShutdown" /v "EulaAccepted" /t REG_DWORD /d 00000001 /f
REG ADD "HKCU\Software\Sysinternals\PsSuspend" /v "EulaAccepted" /t REG_DWORD /d 00000001 /f

Of course, this only works for the user you are currently logged in with. However if you’re preparing OS images which will be prepared with Sysprep you can run this as the user you are intending to run sysprep with and the sysprep process will then roll these keys (along with other modifications to HKCU) into the registry profile used for all users when the machine comes out of sysprep. All subsequent users created on that machine should thus have the EULA pre-accepted, neatly bypassing the possibility of it going wrong when running commands.

For the “well-known” SIDs (e.g. S-1-5-18/Local System, S-1-5-19/Local Service and S-1-5-20/Network Service) you can set these directly via HKEY_USERS:

REG ADD "HKU\S-1-5-18\Software\Sysinternals\PsExec" /v "EulaAccepted" /t REG_DWORD /d 00000001 /f
REG ADD "HKU\S-1-5-19\Software\Sysinternals\PsExec" /v "EulaAccepted" /t REG_DWORD /d 00000001 /f
REG ADD "HKU\S-1-5-20\Software\Sysinternals\PsExec" /v "EulaAccepted" /t REG_DWORD /d 00000001 /f

Server 2003, using sysocmgr to install windows components automatically

One important aspect of automating Windows deployment and setup is enabling particular roles on Server systems. For example, enabling the SMTP/POP service for email. On Server 2008/R2 this is really easy using servermanagercmd.exe and passing through a list of the features/roles you want to install. For 2003 this is a little more complex, requiring the use of the sysocmgr.exe tool.

This tool works on .inf files. Running the command:

sysocmgr /i:%windir%\inf\sysoc.inf

Will open up the usual “Windows Components Wizard” part of windows setup (as if you went to Add/Remove programs and clicked on Add/Remove windows components). You can automate which bits of Windows are added/removed by specifying an unattend file with the /u switch, e.g.:

sysocmgr /i:%windir%\inf\sysoc.inf /u:C:\ocm.txt

The answer file then specifies which components you want to add/remove, e.g. to add the POP3/SMTP service:

[components]
Pop3Srv = On
Pop3Service = On
Pop3Admin = On

The %windir%\inf\sysoc.inf file contains a roughly complete list of these components, and there’s another listing of them here.

Further flags can be used to refine the process, e.g.:

/r - suppress reboot
/x - suppress init banner
/q - run without UI (useful with /u)
/w - prompt before reboot if using /u
/c - disallow cancel

For a complete list do “sysocmgr /?” on a Server 2003 R2 system.

As an aside, configuring the POP3/SMTP server with a new domain requires the following command:

winpop add somedomain.com

The winpop command can be used to query and control the POP3/SMTP server service as well.

MediaTomb custom import script

I wrote about MediaTomb last year, but forgot to post a followup. MediaTomb builds its own database describing the layout of media it serves, this layout isn’t necessarily similar to the underlying layout of files in the filesystem. In addition various data about the files can be extracted and used to modify the way they display in the device retrieving them. One way this process can be controlled is using JavaScript to script the process.

This is controlled by default by the file import.js (on FreeBSD this is found at /usr/local/share/mediatomb/js/import.js). This is set in the config file via the virtual-layout setting:

      
/usr/local/share/mediatomb/js/import.js

I modified my import.js so that it displays the contents of the “TV Series” and “Movies” folders directly under the root of the media server. The filesystem layout is used from there since I am happy with that layout. Much more complicated arrangements are possible of course.

function addVideo(obj)
{
var chain, show, season;
var location = obj.location.split('/');
var rootindex = 0;
for (var i = 0; i < location.length; i++)
{
if (location[i] == "Media")
{
rootindex = i;
break;
}
}
chain = new Array();

if (rootindex + 1 < location.length - 1)
{
for (i = rootindex + 1; i < location.length - 1; i++)
{
chain.push(location[i]);
}
/* Ensure that title doesn't contain file extension */
obj.title = obj.title.substr(0, obj.title.lastIndexOf('.')) || obj.title;
addCdsObject(obj, createContainerChain(chain));
}
}

MediaTomb has various options to permit it to automatically rescan content, or even monitor the filesystem for changes. Since my media library very rarely changes I am not too interested in these methods (especially since the rebuilding process is slow and from my experimenting can often lead to duplicates if files are added in the wrong way).

I decided to simply rebuild the entire database whenever I make changes to the content. This takes around 15 minutes but as I only do this at most once a week that’s no great problem. The script is quite simple and consists of stopping the MediaTomb daemon, deleting the existing database (which will force it to rebuild it on next start) and then starting the daemon again with instructions on which parts of the filesystem it should scan and add to its database:

#!/bin/sh

/usr/local/etc/rc.d/mediatomb stop

rm /var/mediatomb/mediatomb.db

/usr/local/bin/mediatomb -d -c /usr/local/etc/mediatomb/config.xml \
-l /var/mediatomb/mediatomb.log -u mediatomb -g mediatomb \
-P /var/mediatomb/mediatomb.pid -a /store/Media/TV\ Series/ \
-a /store/Media/Movies

-c gives the config file, -l the log file, -u/-g the uid/gid to run as, -P instructs it where to put its pidfile and then the two -a options add directories to scan on startup.

Further details of SixOrNot

I’m currently waiting for Mozilla to review the addon submitted via their addons directory. Due to the recent release of Firefox 4 there’s quite a backlog of extensions awaiting approval so this is looking like it’s going to take some time. This is especially frustrating given that if you post a new release to their website it’ll push the addon to the end of the queue again. I’ve actually got quite a lot of code cleanup and feature enhancements done since the last version I pushed to addons.mozilla.org – but I don’t want to risk waiting longer for the preliminary review.

So what does SixOrNot do? There are a number of IP address related extensions available. Some like ShowIP and Flagfox are quite sophisticated. ShowIP is quite out of date though, and really displays too much information by default. I don’t really feel the need to see the entire IP address all the time, that’s information I want to be easily accessible but not omnipresent. I do however want to see what the IPv4/IPv6 status of my connection to a website is. This is really useful as it provides a nice measure of v6 status.

Flagfox is a nice addon, but it too is somewhat old-fashioned in Firefox terms. It also doesn’t really expose much information about the IP address information of the local system with its primary purpose being to display the approximate location of the remote server. I learned a lot from the Flagfox source code but ultimately decided to start pretty much from scratch with SixOrNot rather than developing an addon using the Flagfox codebase.

The main reason for this was that I wanted to take advantage of the new bootstrapless addon functionality in Firefox 4. This permits you to have an extension which doesn’t require a restart of the browser to install. It’s actually fairly easy to achieve this once you figure it out and I’ll be diving more deeply into that side of the development in a later post. SixOrNot can be added to Firefox without restarting (removed and upgraded too of course).

Another reason was Flagfox’s rigid inflexibility concerning positioning of its icon. While I do to an extent agree that the address bar is a good location for such information I wanted to give users of my addon more of a choice as to where its icon ends up. To this end I’ve developed it primarily as a button which can be placed almost anywhere in the Firefox UI. This is all done via a standard Firefox interface which makes it a breeze to have both button and address bar icon. I really don’t understand why Flagfox doesn’t take this approach actually.

The other major innovation in SixOrNot is the use of the js-ctypes library. Ctypes is a way for high level languages to interact directly with low-level system C libraries. This lets you directly call methods like getaddrinfo and deal with data structures like sockaddr. This lets me do DNS lookups directly via the system C calls and bypassing the interference of Firefox (which can often limit the number of IP addresses returned or filter out IPv6 entries on some platforms). This overcomes one big limitation of extensions such as ShowIP, especially on the Mac OSX platform.

Use of js-ctypes also permits me to retrieve the local IP addresses on Mac OSX in a consistent way. This functionality will be extended to other Unix-based platforms later (e.g. Ubuntu). On Windows doing a DNS lookup for the local host name will always give you all the local IPv4/IPv6 addresses but this is not so on other platforms.

SixOrNot uses the local and remote address information to make an educated guess as to whether you are connecting to a remote site using IPv4 or IPv6. If the remote site has AAAA records and you have a global IPv6 address then it is assumed that the connection is being made via v6. If the remote site has AAAA records and you do not have a global v6 address then the icon is changed to indicate v6 availability – this should prove useful for anyone who hasn’t yet gotten IPv6 connectivity as it’ll show them they’re missing out.

This system isn’t infallible, and without actually connecting via both methods it isn’t ever going to 100% accurate. I felt however that the accuracy of this method is high enough for most situations. I’ll be improving this logic later to try and cope better with situations like proxies and IPv6 transition technologies.

The underlying DNS functionality will also be spun off as a JavaScript library which other extensions could potentially utilise. Support is there for doing asynchronous lookups and multi-platform support so this project has good re-use potential.

There’s more information about the functionality of the addon at the documentation website:
http://timothy.baldock.me/sixornot

And you can download the extension here (ignore all the Mozilla warnings of doom and gloom):

https://addons.mozilla.org/en-US/firefox/addon/sixornot/

SixOrNot – my foray into Firefox extension building

So I’ve been working on a new project in my spare time recently. This is a Firefox extension which presents IP address information about your machine and the website you are connecting to in a nice way via a button and/or address bar icon. The main purpose is to let you know whether the site you are connected to is using IPv6 or not.

I’ve tried to make it using the most modern methods (it’s a so-called “bootstrapped” or “restartless” addon – meaning you don’t have to restart the browser to install it).

I’ll post a more technical article going into how it works soon. For now you can find the beta releases here:

https://addons.mozilla.org/en-US/firefox/addon/sixornot/

Copy without .svn

So I keep my web documentation for projects in SVN, so I can easily track changes related to software revisions. I then copy over the correct version for each new release. Subversion working directories are littered with .svn directories which I don’t want to copy. There are quite a few ways to copy files and exclude these directories (or even delete them afterwards) but I find the simplest way is to use rsync. E.g.:

rsync -r --exclude=.svn workingcopy/trunk/docs/ www/project/docs/

Nice and simple!