OpenBSD 4.6 -> 4.7 on QEMU-based VPS

As a first step towards getting my DNS infrastructure DNSSEC-ready I needed to upgrade my secondary DNS server. This runs OpenBSD (and also serves as a secondary MX). I host it with a company in the US (for geographical diversity) called ARP Networks who are rather good (native IPv6 too!)

The upgrade process for OpenBSD is fairly well documented in a series of upgrade guides for each pair of releases, e.g.:

http://www.openbsd.org/faq/upgrade47.html

It’s recommended to perform the upgrade using the installer present in the ramdisk kernel of the next version, e.g. to go from 4.6 to 4.7 I used the bsd.rd from 4.7. It’s also recommended to only upgrade one version at a time.

My eventual aim is to have both this server and my router at home (which runs a stealth slave nameserver) upgraded to 4.9 (which is due for release in a little over a month). In the meantime I’m going to get them both up to 4.8 (the other box currently runs 4.7).

The upgrade went fairly well up to the first proper reboot. It’s a simple (if crude) process. I did need to enable redirection of console output to com0 (which is what the console provided by ARP Networks for out-of-band management uses). This can be done by adding:

set tty com0

To /etc/boot.conf

After rebooting the first time using the new kernel I found the following dmesg output:

OpenBSD 4.7 (GENERIC) #112: Wed Mar 17 20:43:49 MDT 2010
deraadt@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC
real mem = 267321344 (254MB)
avail mem = 247496704 (236MB)
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 2.4 @ 0xfbd3f (10 entries)
bios0: vendor QEMU version "QEMU" date 01/01/2007
acpi0 at bios0: rev 0
acpi0: tables DSDT FACP APIC
acpi0: wakeup devices
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
acpiprt0 at acpi0: bus 0 (PCI0)
acpicpu0 at acpi0
mpbios0 at bios0: Intel MP Specification 1.4
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: QEMU Virtual CPU version 0.9.1, 2667.15 MHz
cpu0:
FPU,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SSE3,NXE,LONG
cpu0: 64KB 64b/line 2-way I-cache, 64KB 64b/line 2-way D-cache, 512KB
64b/line 16-way L2 cache
cpu0: ITLB 255 4KB entries direct-mapped, 255 4MB entries direct-mapped
cpu0: DTLB 255 4KB entries direct-mapped, 255 4MB entries direct-mapped
cpu0: apic clock running at 1019MHz
mpbios0: bus 0 is type ISA
ioapic0 at mainbus0: apid 1 pa 0xfec00000, version 11, 24 pins
ioapic0: misconfigured as apic 0, remapped to apid 1
pci0 at mainbus0 bus 0
pchb0 at pci0 dev 0 function 0 "Intel 82441FX" rev 0x02
pcib0 at pci0 dev 1 function 0 "Intel 82371SB ISA" rev 0x00
pciide0 at pci0 dev 1 function 1 "Intel 82371SB IDE" rev 0x00: DMA,
channel 0 wired to compatibility, channel 1 wired to compatibility
wd0 at pciide0 channel 0 drive 0:
wd0: 16-sector PIO, LBA48, 5632MB, 11534336 sectors
atapiscsi0 at pciide0 channel 0 drive 1
scsibus0 at atapiscsi0: 2 targets
cd0 at scsibus0 targ 0 lun 0: ATAPI 5/cdrom
removable
wd0(pciide0:0:0): using PIO mode 0, DMA mode 2
cd0(pciide0:0:1): using PIO mode 0
atapiscsi1 at pciide0 channel 1 drive 0
scsibus1 at atapiscsi1: 2 targets
cd1 at scsibus1 targ 0 lun 0: ATAPI 5/cdrom
removable
cd1(pciide0:1:0): using PIO mode 0
uhci0 at pci0 dev 1 function 2 "Intel 82371SB USB" rev 0x01: apic 1 int
7 (irq 11)
piixpm0 at pci0 dev 1 function 3 "Intel 82371AB Power" rev 0x03: apic 1
int 4 (irq 10)
iic0 at piixpm0
iic0: addr 0x18 00=00 01=00 02=00 03=00 04=00 05=00 06=00 07=00 08=00
words 00=0000 01=0000 02=0000 03=0000 04=0000 05=0000 06=0000 07=0000
iic0: addr 0x1a 00=00 01=00 02=00 03=00 04=00 05=00 06=00 07=00 08=00
words 00=0000 01=0000 02=0000 03=0000 04=0000 05=0000 06=0000 07=0000
iic0: addr 0x29 00=00 01=00 02=00 03=00 04=00 05=00 06=00 07=00 08=00
words 00=0000 01=0000 02=0000 03=0000 04=0000 05=0000 06=0000 07=0000
iic0: addr 0x2b 00=00 01=00 02=00 03=00 04=00 05=00 06=00 07=00 08=00
words 00=0000 01=0000 02=0000 03=0000 04=0000 05=0000 06=0000 07=0000
iic0: addr 0x48 48=00 words 00=0000 01=0000 02=0000 03=0000 04=0000
05=0000 06=0000 07=0000
iic0: addr 0x49 48=00 words 00=0000 01=0000 02=0000 03=0000 04=0000
05=0000 06=0000 07=0000
iic0: addr 0x4a 48=00 words 00=0000 01=0000 02=0000 03=0000 04=0000
05=0000 06=0000 07=0000
iic0: addr 0x4b 48=00 words 00=0000 01=0000 02=0000 03=0000 04=0000
05=0000 06=0000 07=0000
iic0: addr 0x4c 00=00 01=00 02=00 03=00 04=00 05=00 06=00 07=00 08=00
48=00 words 00=0000 01=0000 02=0000 03=0000 04=0000 05=0000 06=0000 07=0000
iic0: addr 0x4d 48=00 words 00=0000 01=0000 02=0000 03=0000 04=0000
05=0000 06=0000 07=0000
iic0: addr 0x4e 00=00 01=00 02=00 03=00 04=00 05=00 06=00 07=00 08=00
48=00 words 00=0000 01=0000 02=0000 03=0000 04=0000 05=0000 06=0000 07=0000
vga1 at pci0 dev 2 function 0 "Cirrus Logic CL-GD5446" rev 0x00
wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
em0 at pci0 dev 3 function 0 "Intel PRO/1000MT (82540EM)" rev 0x03: apic
1 int 12 (irq 11), address 52:54:00:27:22:07
"Qumranet Virtio Memory" rev 0x00 at pci0 dev 4 function 0 not configured
"Qumranet Virtio Console" rev 0x00 at pci0 dev 5 function 0 not configured
isa0 at pcib0
isadma0 at isa0
com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
com0: console
intr_establish: pic ioapic0 pin 4: can't share type 3 with 2
pckbc0 at isa0 port 0x60/5
pckbd0 at pckbc0 (kbd slot)
pckbc0: using irq 1 for kbd slot
wskbd0 at pckbd0: console keyboard, using wsdisplay0
pmsi0 at pckbc0 (aux slot)
intr_establish: pic ioapic0 pin 12: can't share type 3 with 2
pckbc0: unable to establish interrupt for aux slot
wsmouse0 at pmsi0 mux 0
pcppi0 at isa0 port 0x61
midi0 at pcppi0:
spkr0 at pcppi0
fdc0 at isa0 port 0x3f0/6 irq 6 drq 2
fd0 at fdc0 drive 0: density unknown
fd1 at fdc0 drive 1: density unknown
usb0 at uhci0: USB revision 1.0
uhub0 at usb0 "Intel UHCI root hub" rev 1.00/1.00 addr 1
nvram: invalid checksum

The system then stuck at this point (for upwards of 10 minutes, I got bored of waiting after that…)

A Google search for the invalid checksum message with reference to QEMU revealed some recommendations to disable uhci, this didn’t help. I also tried disabling acpi, which also didn’t help. I found the correct bit of the kernel to turn off was actually ioapic. With this disabled the system was able to boot. I would imagine that this would break the multiprocessor kernel however and I am awaiting a response from ARP Networks to ask if there’s a better workaround for this.

Modifying the kernel can be done either at boot time by entering the kernel you wish to modify with the -c option at the boot> prompt, e.g.:

boot>bsd -c

In which case changes are temporary, or using config:

# config -e -f /bsd
ukc> disable uhci
ukc> quit

In which case changes are permanent and written into the kernel in question (you may want to make a backup of the kernel before doing this).

After I disabled ioapic in this manner the system booted up. All that was left to do then was to finish upgrading the configuration. It would’ve been quite a quick process if not for that little hiccup with the kernel configuration.

Next step is to upgrade both boxes to 4.8, install NSD and Unbound from ports, configure them to replace the existing BIND nameserver and then I can actually implement DNSSEC.

Firefox 4

So Mozilla finally got around to releasing Firefox 4. So far I am quite impressed, it’s a bit quicker and has a few new features.

One of the biggest changes seems to be a decision to remove the status bar. I initially thought this was a stupid idea but considering it and having done some work to not need it anymore it actually makes some sense. I’ve been using Vimperator for a while now and the 3.0 release (confusingly for Firefox 4) works really really well. They have moved to a single-line interface which takes up almost no screen-space when not in use.

Since there’s no longer a status bar I need somewhere for all my addons to live. I decided to see how hard it’d be to simply replace their icons with Vimperator commands. So far this has been fairly easy to do. I can just type “:firebug” or “:noscript” and access that functionality. I’ve even written a little javascript to set up proxies automatically so I can do away entirely with FoxyProxy…

I think it’d actually be fairly impossible to cut down the interface any more than it is already at this point.

Python on Windows – Unicode environment variables

Say you want to open a file picker dialog in the User’s profile root, or log to a file under AppData, or do anything else involving environment variables in Windows relating to file paths. You could use os.environ/os.getenv() for this however both these methods only return ASCII. If your user happens to have characters higher than codepoint 128 in their name (using some system codepage) then these methods will likely return a mangled approximation of the path. If the user has codepoints higher than 255 then it’ll just return question marks for the most part. Hence these paths:

"C:\Users\Rosnička"
"C:\Users\发涩"

Are returned as:

"C:\Users\Rosnicka"
"C:\Users\??"

Which is clearly unacceptable.

The function os.path.expanduser() suffers from the same problem since it uses environment variables internally.

Given that these paths have already been mangled in converting them to ASCII you can’t decode them using the system encoding (as you can do for some other file paths on windows using the sys.getsystemencoding() function as a second argument to unicode()).

The solution is to use ctypes to query the win32 API and get the actual unicode values of the environment variables. This function (cribbed from here) allows you to do this and returns a python-native unicode string.

def getEnvironmentVariable(name):
n= ctypes.windll.kernel32.GetEnvironmentVariableW(name, None, 0)
if n==0:
return None
buf= ctypes.create_unicode_buffer(u''*n)
ctypes.windll.kernel32.GetEnvironmentVariableW(name, buf, n)
return buf.value

Python 3.0 also solves this issue as it can query unicode environment variables as well.

ISATAP setup on Windows clients

Recently ran into the question of how to configure ISATAP adaptors on our Windows machines. ISATAP is an IPv6 transition mechanism which relies on an IPv4 infrastructure to automatically configure IPv6 addresses which can be used within a site. It doesn’t generally provide any external routeability to the wider world via IPv6 and acts as a workaround to avoid full IPv6 deployment, in this case for an implementation of Microsoft’s DirectAccess.

This assumes you have already configured your ISATAP router.

Configuration on clients is mostly automatic. It relies on two things being true, the first is that your hosts can resolve the DNS alias “isatap” on your local network (e.g. isatap.mycompany.local). This allows them to locate the ISATAP router. Second ISATAP itself needs to be enabled. This can be done using this command:

netsh interface isatap set state enabled

Alternatively you can enable this using Group Policy by editing the following policy:

Computer Configuration\Policies\Administrative Templates\Network\TCPIP Settings\IPv6 Transition Technologies\ISATAP State

The value of:

netsh interface isatap show router

Should generally show:

Router Name            : default
Use Relay : default
Resolution Interval : default

You can set the router name manually using either a netsh set command:

netsh interface isatap set router somerouter

Or via Group Policy.

After running these commands the output of ipconfig should change to show fe80:: link-local IPv6 addresses for the isatap adaptors and, all being well, a 2002:: isatap address as well.

If not check your DNS configuration, one thing you might need to do is to remove a block on the isatap name resolving:

http://technet.microsoft.com/en-us/library/ee649158(WS.10).aspx

And lots more troubleshooting for ISATAP/DirectAccess here:

http://technet.microsoft.com/en-us/library/ee844184(WS.10).aspx

Simutrans server administration guide – Part 1

Introduction

Simutrans (a transport sim game similar to Transport Tycoon etc.) recently gained support for online multiplayer games. This is based around a persistent-world client/server model (e.g. the server runs the game 24/7 and clients can connect to play for a while, with their transport companies continuing to exist when they quit).

I have set up a number of public game servers to allow people to play the game online and through doing so learnt quite a bit about the process of setting up and maintaining not just Simutrans but Unix server processes in general. I hope that this guide helps those that might follow me.

Compiling

Simutrans can be compiled using GCC 3 or 4 (or other means, not covered in this guide). For the server version it’s highly recommended to use the posix backend which completely disables the graphics engine, reducing the amount of processing that needs to be done by the server.

Currently there aren’t binaries compiled for the posix backend by default so generally you need to compile it yourself on the server it is going to be run on. This is fairly easy to do. I am going to assume for the purposes of this guide that you’re running Debian Linux, though it should be much the same for any Unix platform.

1. Check out the correct version of the Simutrans source code. You’ll need Subversion installed to do this. E.g. to check out version 110.0 (the first stable version with network support:

svn co --username anon -r 4303 svn://tron.homeunix.org/simutrans/trunk

When prompted for a password just hit enter (blank password).

2. Once Subversion finishes working copy the file config.template to config.default:

cp config.template config.default

Then use your favorite text editor to edit the new file. Uncomment the lines:

BACKEND = posix
COLOUR_DEPTH = 0
OSTYPE = linux
DEBUG = 3
OPTIMISE = 1
WITH_REVISION = 1

These should be the only non-commented lines in your default.config.

3. Save the file and then run these commands. You’ll need GCC installed as well as any other packages your OS requires to be able to compile C and C++ code.

make

When this completes, run:

strip sim

This will strip symbols from the binary and reduce its size dramatically.

4. You’re done, the binary “sim” will be used together with a set of game files to set up your server. You can use the same source tree to compile newer versions of the game later. To do this you’ll need to issue an svn update command with the revision of the version you wish to update to, e.g.:

svn up -r 4304

Then run:

make clean
make
strip sim

Again to rebuild the newer version.

User setup

To avoid running the Simutrans server processes as root (poor security) and to avoid running them as a real system user (also poor security) I set up a new user account called “simutrans”. This account doesn’t have a home directory or login shell since it is only going to be used to run the server. I would recommend (and will write here in this guide) that you use the username “simd” instead since “simutrans” is more than 8 characters so the name won’t appear in the output of the ps command.

The exact way you create a user is likely to be very OS specific so I will leave it to you.

File structure

Simutrans can run in one of two modes, “single user” and “multi-user”. This has no effect on network multiplayer and refers only to the scheme used to store the game’s configuration files. In multi-user mode the game will store its config per-user under your home directory. In single-user mode it’ll store everything under the installation’s config directory. I recommend using single-user mode for simplicity.

This setting is specified in the “/config/simuconf.tab” file under the simutrans directory for the revision you’re running, e.g.:

/var/simutrans/r4303/simutrans/config/simuconf.tab

The setting in question is:

singleuser_install = 1

See the Configuration File section for more details.

I created the following file structure for my server:

/var/simutrans/bin           (For control scripts, see below)
/var/simutrans/etc/ (For server instance configuration files)
/var/simutrans// (For simutrans game files)

Additionally I create a folder under /var/log:

/var/log/simutrans/

Which is where we’ll redirect the individual servers’ log output to using syslog.

“/var/simutrans” should be owned by root, group simd.
“/var/simutrans/bin” and “/var/simutrans/etc” should be owned by root, group root
“/var/simutrans//” and everything below it should be owned by simd, group simd
“/var/log/simutrans” should be owned by simd, group root (or adm on Debian)

Log Files

By default Simutrans logs to a file in the directory where the executable is located. The logging level is determined by the “-debug” command line switch. A value of 2 is reasonable, 3 will provide a lot more information. Unfortunately it isn’t possible to tell Simutrans where to log. It is also not possible to send it a signal to make it re-open its log file (so log rotation isn’t easily possible).

The solution to this is to use syslog. On Debian (and likely other platforms) there is a utility called logger which allows you to redirect the stdout/stderr of a process to syslog (and from there to log files). This is the best way to handle logging on Unix systems as it takes care of log rotation etc. for you.

Using logger is simply, you just pipe the output of your Simutrans instance to it. This will be covered in more detail later when I explain the use of the simctrl script but roughly you would execute a command like:

sim -server 13353 -server_id 123456789 -server_name testserver -debug 2 -lang en -objects pak/ 2>&1 | logger -i -t testtag &

This runs “sim” (the binary we compiled earlier) with various options. The “2>&1” tells the shell to redirect standard error to standard output (e.g. all messages go to the same place). Finally we pipe this output (the | is a pipe character) to the logger utility which we tell to tag each line with an arbitrary tag (in this case “testtag”) and the process id. The tag can be used later to filter the syslog messages into a distinct log file.

You will need to modify your syslog configuration. On Debian this is found at:

/etc/rsyslog.conf

Under the RULES section and before any other rules add the following:

# Simutrans log files
:syslogtag, contains, "simserver1" /var/log/simutrans/simserver1.log
& ~
:syslogtag, contains, "simserver2" /var/log/simutrans/simserver2.log
& ~

This assumes we have two instances of Simutrans set up, one known as “simserver1” and the other as “simserver2”. Any messages which syslog sees which are tagged with this string will go into the log file specified. The line with “& ~” just tells syslog that we can discard these messages at this point since we only want them to be logged to this place. If you don’t do this then the messages can end up in other log files as well.

The “tag” is a concept used extensively with the simctrl script and acts as an identity for the server instance in question. More on this later.

Save games + restarting server

Simutrans loads savegames from the “simutrans/save/” folder (when in single-user mode). You can place new games here and then load them on the command line. If you don’t specify the “-load xxxx.sve” command line switch then the game will automatically search for a file called “serverXXXXX-network.sve” (where XXXXX is the port number of the server instance, e.g. “server13353-network.sve”). If it finds this file it will load the game automatically. In most cases the last time this will have been saved is when a client enters the game, or if autosave is turned on. I believe the game also saves if the server is shut down cleanly.

To run a new game then specify “-load mygame.sve” on the command line. To resume a game after the server has been shut down or crashes you don’t need to specify this flag at all. IMPORTANT: The automatic savegame will be overwritten if you start another server instance on the same port using the “-load xxxx.sve” option. Remember that the savegame specified with the “-load” flag is found relative to the game’s savegame directory. In a single-user installation this will be “/save/”, e.g. if your binary is located here:

/var/simutrans/r4303/simutrans/

Then it will look for savegames here:

/var/simutrans/r4303/simutrans/save/

Simutrans Configuration File

Simutrans stores its configuration in many places. In single-user mode (I recommend using this!) the main locations are “/config/simuconf.tab” and the simuconf.tab files located under individual pakset folders. The only place you should need to make changes are the main simuconf.tab.

Settings important for network servers:

singleuser_install

Set this to 1.

autosave

If you wish you can enable this, though for an active server it shouldn’t be needed as people will join/leave so often.

server_frames_ahead

This setting determines the number of frames the server runs ahead of the client. Setting this to a higher value increases stability but also increases latency (e.g. takes longer to build things but less likely to get desyncs).

additional_client_frames_behind

This isn’t a server setting and is set per-client, but I will mention it here as it can help people who have laggy connections to stay connected to your server. It does roughly the same thing as server_frames_ahead but on a per-client basis.

server_frames_between_checks

This setting determines the number of frames between manually syncing clients (if nothing else requiring a sync has happened in the intervening period). Again, lower values increase stability but also increase bandwidth usage slightly.

The “server_announce”, “server_announce_interval”, “server_name” and “sever_comment” fields can all be set on the command line and for multiple servers running from the same source files this is a necessity so I will cover them in the simctrl config section.

Simctrl Utility Script

I created a shell script to automate the process of starting and stopping the Simutrans instances I have running on my server. This script relies on the concept of each instance having a “tag”, e.g. sim64 or sim128 which is unique to that instance. This tag is used to idenfity which instance we wish to control and to look up configuration parameters.

The script expects your Simutrans instances to be located at:

/var/simutrans/<revision>/simutrans/

E.g.:

/var/simutrans/r4303/simutrans/sim

You can change this by modifying the script if you need to. I recommend the directory structure explained earlier however as it seems fairly sensible (and lends itself to the possibility of using chroot).

The script must be run as root and it uses su to run the server as the user simd. The format of the command is:

simctrl  {status|start|restart|stop}

Replacing with the tag name of your simutrans instance and selecting one of the 4 following commands.

status – Basically runs “ps uww” filtered by the pid recorded in the pidfile created when simctrl is used to start a simutrans instance
start – Starts the specified simutrans instance. Configuration is read from the .config file located in the etc/ folder relative to the simutrans location defined in simctrl (by default /var/simutrans/etc/). See below for the config file format. The start command will load the savegame file defined in the configuration.
restart – Performs a stop followed by a start. The restart command does not specify a savegame path so use this for restarting the server if it has crashed as it will then pick up the last running game from the automatically saved file.
stop – Stops the specified simutrans instance.

The configuration file format is:

port="13353"
revision="r4303"
objects="pak/"
save="network_64.sve"
debug="2"
server_id="123456789"
server_name="your.dns.name"
server_comment="your.dns.name"

Hopefully this is fairly self-explanatory. Simctrl will complain if any of the fields are missing. This should be saved as .config in the /var/simutrans/etc/ directory. The tag should be a short string that uniquely identifies an instance of the game.

Note: The port number must be unique for each instance. You can obtain a server_id from the Server List website (see here: http://simutrans-germany.com/serverlist/). Use the “Add a new simutrans server” link and then fill out the details. This unique ID allows the game to automatically update its status for clients. Your server won’t show up in the client’s game listing until you do this.

The other fields:

revision – This should match whatever the subfolder under /var/simutrans/ is which contains your game, e.g. “r4303” would indicate: /var/simutrans/r4303
objects – The pakset to load, this is relative to the location simutrans is starting up from. E.g. “pak/” for pak64, “pak128.Britain.r376/” for pakBritain etc.
save – The savegame file to load when using the Start command, this is relative to the game’s save directory
debug – Debug level for logging, 2 is fairly useful, 3 is pretty verbose
server_name – IP address or DNS name, this is how clients know where to find your server to connect to you
server_comment – Currently doesn’t do much (this should update the comment field on the Server List website but this doesn’t seem to work)

You can download the simctrl script from here:

https://github.com/tbentropy/simctrl

Conclusion

This guide presents the basics for setting up a Simutrans server so I hope it proves a good starting point. There are some other concepts to cover (most important of these being monitoring since Simutrans can occasionally crash) but I will look at these in a future post. There is also a complementary utility to simctrl (known as simquery) which lets you talk to a running server and ask it questions. This script isn’t quite ready for release yet though.

I hope you found this guide useful. If you have any questions please ask!