Cygon’s Blog

After getting KDE and Samba to work, I started looking out for a good BitTorrent client. Those MythBusters episodes aren’t gonna download themselves and I was already showing the first signs of Top Gear Withdrawal Syndrome (TGWS)

What I was looking for was a fast downloading client that used little resources and that I could run in the background as a service or daemon. I had tried MLDonkey on Windows Server 2008, but it was quite a pain to get set up right and torrent download speeds weren’t all that great (plus some trackers have actually banned MLDonkey). Azureus has a textmode GUI that I could theoretically combine with a WebUI plugin, but that would still be a bit too heavyweight.

That’s when I discovered rTorrent. It runs in the console (and thus, you can run it with DTACH) and can be controlled via a simple XMLRPC interface. And then there’s wTorrent, a nice-looking, nifty Web 2.0 AJAX GUI written in PHP that you can run in Apache or lighthttpd. I’m now running a daemonized rTorrent with the wTorrent GUI and it’s working so well it’s almost too good to be true

BitTorrent Performance

Now I have tried µTorrent, Azureus, Halite, BitComet, MLDonkey und some more, double checked that I had opened the required ports (using nmap from a server on the internet), used random ports >50,000 to avoid throttling, tweaked my settings and what not, but download speeds were, at best, average.

I left rTorrent running overnight with a 7 GB download. One that had somehow caused my Windows Server 2008 system to commit suicide by paging in no time, or that would complete but still have missing chunks. After one night, the torrent was at 60%, the next day it was finished. And that’s no exception.

RAID / Samba Performance

I reported a stable 20 MB/s upload in my last post. Forget that, it’s a stable 40 MB/s now that I’ve got no compile or torrent rehash running in the background. And download speeds are at a stable 60 MB/s — I believe that’s pretty close to what the hard drive I’m downloading to can do.

This is just incredible. It’s still the same hardware, but the new system could run circles around my old setup.

Today, I continued setting up my Gentoo server. With internet access available to my windows machine through NAT and remote administration working through SSH, I could easily look up resources on the internet and copy & paste between the linux console and my web browser

RAID5 was a simple matter of activating the required kernel options (which I already did beforehand). Somehow, mdadm created a RAID5 array with one spare. I haven’t investigated this much further, but it seems this allows the array’s initial synch to work faster. Whatever, I didn’t want it, so after finding out how to take the RAID array down again, I used mdadm with –spare-devices=0 and –force to have all disks UP from the beginning.

Using –chunk to set a block size of 128 kb and mke2fs with the -E stride=n,stripe-width=n options allowed me to tailor the RAID array’s stripe size to the file system. Not that I expect any noticeable gain, but it’s easily done and can’t hurt.

Next was samba. Gentoo makes this surprisingly easy. emerge samba, edit /etc/samba/smb.conf and you’re done. It took some effort to figure out how to create a password protected share, so this is what I did:

I’ve got a linux user named ‘cygon‘ on this system. This was created with the good ol’ useradd script. I added this user to /etc/samba/smbusers as an alias for “administrator” and “cygon“, so the file now looks like this:

# Unix_name = SMB_name1 SMB_name2 ...
# $Header: /var/cvsroot/gentoo-x86/net-fs/samba/files/config/smbusers,v 1.1 2007/09/07 21:07:40 dev-zero Exp $
#root = administrator admin
nobody = guest pcguest smbguest
cygon = administrator cygon

Then I set up a password for this user with smbpasswd -a cygon.

Finally, I created a folder that I wanted to share, assigned it to the user “nobody” (this is what samba uses for all files creates by guests) with chown nobody:nobody /var/storage/raid -R. This I then added in my /etc/samba/smb.conf:

[Protected]
  comment = Administrative share for the entire RAID array
  path = /var/storage/raid
  public = yes
  guest ok = no
  writable = yes
  printable = no
  force user = nobody
  force group = nobody
  create mask = 744
  directory mask = 755
  valid users = cygon

‘Public‘ says the folder can be seen by other network users. ‘force user‘ and ‘force group‘ make samba assign all files and folders created from networked users to the ‘nobody‘ linux user account and ‘nobody‘ linux group. Likewise, ‘create mask‘ and ‘directory mask‘ are the attributes assigned to folders created by network users. Finally, ‘valid users‘ tells samba to only let the listed users access the share.

That’s all it took to get this working.

The funny thing, again, is that performance and reliability increased by an order or magnitude compared to Windows Server 2008. I configured Samba to always act as my domain master browser, so when I browse my network neighborhood in windows, one click and all PCs on the network appear — immediately. Instead of 20 seconds of searching and then maybe half of the local PCs showing up.

File copies to and from the array are easily 4 times faster than with windows. And, most notably, they run much smoother. Windows Server 2008 accepted an average 20 MB/s for several seconds, then blocked hard for a while (probably flushing its insane cache), then accepted data again. Download speed was kept up longer, but also wouldn’t go over around 25 MB/s (30 MB/s tops) during the whole transfer (at least until the cache grows to the size of the physical RAM and the OS starts paging out unimportant things, like its own kernel, the DHCP server, its TCP/IP stack, DNS database, the RDP server and any running foreground application the user is currently working with)

Now Samba, which is just an implementation of Microsoft’s SMB protocol, pieced together by logging network packets, combined with a (Soft-)RAID 5 partition manages a stable 20 MB/s upload and a stable 60 MB/s download. And the kernel just takes the load - no suicide by paging, swapping out of vital system components or anything noticeable happening at all! The copy beats the original - by far.

Now my enthusiasm has faded a bit. I worked under the presumption that anything linux would be tested by millions of users and nearly every obscure bug that might occur like if you are running your PSU near its limit and decide to attach an USB hairdryer to the USB hub in your keyboard would have been found eventually.

The install CD passes some unterminated string coming from GRUB to the linux kernel as a parameter. This string then creeps into your environment. When you chroot into your system as it is being installed (and maybe forget to env-update && source /etc/environment - might have happened to me because I chrooted quite often until the thing could boot itself), this unterminated string in the environment gets saved.

Linux version 2.6.24-gentoo-r7 (root@poseidon) (gcc version 4.1.2 (Gentoo 4.1.2 p1.0.2)) #1 SMP Sat Jun 21 06:59:43 UTC 2008
Command line: root=/dev/ram0 init=/linuxrc dokeymap looptype=squashfs
loop=/image.squashfs cdroot vga=791Y^ÛóØÐö^A^È´þ¸7Õ^Fð>å)Ë6B¿u
^Cª^×ÎV^Gé^K¿ú^Éþ¢Xm.^ÍdrRW½Y^×g^Ú^C¤^ÛQ’ñ÷^5÷Vb²Iuf2õ^Ê°ý^ÁÀ¶^U§^Ï7ãÞ÷;9^U^Òg^
Y^SÛKÈÅd^Ù_^Óê|äKºpN» ðK´¸¼ “NÇ@ܹ¤À×ÉTÊHõ^Rùc;¨øÁ²^Ñ=PcûmMùÒû^É^YH^Óò^É!

I think gentoo’s emerge stores the environment in which a package was installed so that, when the package is uninstalled, this can be done under the exact same environment. For example, if you installed PHP with the apache USE flag, it would have installed the apache extension for php. If you now remove the apache USE flag and uninstall PHP, it wouldn’t know that it has to uninstall the apache extension — unless you let it run in a sandbox with the exact environment at the time of installation.

/var/tmp/binpkgs/app-text/ghostscript-gpl-8.62/temp/environment

[…]
vga=$’791Y\233\363\330\320\366\001\001\001\001\001\001\001\001\001\001\001\001
\001\001\001\001\001\001\001\001\001\001\001\001\001\001\001\001\001\001\001
\001\001\001\001\001\001\001\001\001\001\001\001\001\001\001\001\001\001\001
\001\001\001\001\001\001\001\001\001\001\001\001\001\001\001\001\001\001…
[…]
vga=”791Y^ÛóØÐö^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A
^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A
^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A
^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A…

To make a long story short, this saved environment was close to 500 KB, maybe only limited by the maximum environment size being reached. Add just one more thing to the environment (something an uninstall sandbox would likely do) and restoring the environment causes it to overflow.

Whenever I tried to rebuild a package or update it, emerge would fail with an error message (and continue doing its stuff — ignoring that a required package just failed to install — wow!).

/usr/lib/portage/bin/ebuild.sh: line 1496: /bin/touch: Argument list too long

This sucks pretty hard. Not knowing how to solve this, I tried increasing my kernel’s environment size and then (after that didn’t work), tried to rob portage of its saved environments. For that, I used find|grep to make a list of all environment.bz2 files in /var/db/pkg. Then I tried to tar the files in the list with the –remove-files option and when couldn’t get tar to do what I wanted, I tried passing the list to rm.

Well, ultimately I managed to clean out my entire /var/db directory - with the exception of my list of files to delete.

I started a reinstall from scratch, this time explicitely unsetting the vga variable in my install CD’s environment before even so much as touching emerge. Hope it works out this time around.

Today, I finally got rid of my Windows Server 2008 trial server. No point in “trialing” this thing any more — Windows kept killing itself by paging out vital system services despite plenty of memory being available, created unkillable phantom processes and the firewall system in Windows Server 2008 is a class of itself in terms of unusability (<– I think I just created that word :))

My Gentoo Linux 2008.0 x64 install went extremely smooth. The install CD recognized my networking settings, PPPoE dial-in took just a few minutes to get working after launching the SSH daemon, I could conveniently install the system using PuTTY on my Vista box, allowing me to cross-reference the installation guide and playing some games during the longer tasks.

I still don’t have any idea how people get their linux kernels trimmed down — I simply don’t know what options I really need and which just sound like I might be needing them. Thus, I went ahead and selected any device drivers that sounded like my hardware, then added the most likely options for raid, vpn, ppp and routing.

After GRUB was installed (which was a lot easier this time around since I’ve only got one boot partition - the server hosting my blog uses two boot partitions on different hard drives as a fail-safe mechanism), the kernel booted, networking was working and basically everything just did what it should.

Amusing fact: my Corsair memory modules have LED indicators on them that display the current memory bandwidth similar to a volume indicator in a stereo. With Windows Server 2008 idling away, the LEDs were wildly flicking between 50% and 75% load. Now with linux idling, only one lonely LED (out of 20) is lit up.

It took me some time to get NAT (IP masquerading) working and I’m still not sure I got my iptables configuration right. The examples I could find on the net all had some confusing and from my limited knowledge erroneous rules in them, so I decided to try it myself. This is what I’ve come up with:

# Generated by iptables-save v1.3.8 on Sat Jul 19 16:00:29 2008
*filter

# According to man, there are three "chains"
#   INPUT = Packets from outside with a destination on this machine
#   FORWARD = Packets being routed by this machine
#             (happens when another machine in the network has this machine
#             configured as its gateway)
#   OUTPUT = Packets being sent from this machine
#

# These are the default rules. They will only apply if a packet makes it
# through our rule maze without matching any rule we set up.
#
:INPUT ACCEPT [158:13292]
:FORWARD DROP [4:224]
:OUTPUT ACCEPT [1123:117012]

# -------------------------------------------------------------------------- #
# INPUT (packets destined for this machine)

# Allow all packets originating from the local network to reach this
# machine. This in effect means we trust anyone in the intranet.
#
-A INPUT -s 192.168.124.0/24 -j ACCEPT

# Of course, we will also accept packets we sent to ourselves.
#
-A INPUT -s 127.0.0.1 -j ACCEPT

# This lets any connections, once established, keep running without
# forcing the packets through all the rules we set up.
#
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT

# Here would be the place to local open ports in your firewall. To allow
# a web server running on this macine to be contacted from the internet
# using your ppp0 adapter, use this example:
#
#-A INPUT -i ppp0 -p tcp -m state --state NEW --dport 80 -j ACCEPT
#-A INPUT -i ppp0 -p tcp -m state --state NEW --dport 443 -j ACCEPT

# All other packets are rejected
-A INPUT -j REJECT --reject-with icmp-port-unreachable

# -------------------------------------------------------------------------- #
# FORWARDING (packets being routed through this machine)

# Allow any packets from the local network to be routed to
# the internet connection on ppp0
#
-A FORWARD -s 192.168.124.0/24 -o ppp0 -j ACCEPT

# Allow any packets coming in from the internet connection on ppp0 to
# be routed to the local network
-A FORWARD -i ppp0 -d 192.168.124.0/24 -j ACCEPT

#-A FORWARD -i eth0 -
#-A FORWARD -s 192.168.124.0/24 -m state --state NEW -j ACCEPT

COMMIT
# Completed on Sat Jul 19 16:00:29 2008

# -------------------------------------------------------------------------- #
# NAT

# Generated by iptables-save v1.3.8 on Sat Jul 19 16:00:29 2008
*nat
:PREROUTING ACCEPT [38:2923]
:POSTROUTING ACCEPT [31:2379]
:OUTPUT ACCEPT [40:3005]

-A POSTROUTING -o ppp0 -j MASQUERADE

COMMIT
# Completed on Sat Jul 19 16:00:29 2008

# Generated by iptables-save v1.3.8 on Sat Jul 19 16:00:29 2008
*mangle
:PREROUTING ACCEPT [3568:275800]
:INPUT ACCEPT [3564:275576]
:FORWARD ACCEPT [4:224]
:OUTPUT ACCEPT [3551:635930]
:POSTROUTING ACCEPT [3551:635930]
COMMIT
# Completed on Sat Jul 19 16:00:29 2008

I’m currently recompiling the entire system to make sure the stage3 packages match my compiler settings. That will probably take a few hours, after which I will proceed to set up Samba, MySQL, Apache, KDE and, ultimately, 3D acceleration and Unreal Tournament 2004

Last week, I bought GT Legends, a racing simulation, from amazon.de. I really enjoy racing games and, while I do like arcade racers very much, I also play simulations. So for €9.95, what could I possibly do wrong?

Lots.

Installing the game killed my system. Obviously, the game employs the heinous StarForce copy protection system and the StarForce driver on the game DVD is incompatible with Windows Vista x64.

Luckily, I’m already prepared for these almost regular incidents and within 20 minutes, my drive image was restored. After some googling, I discovered a StarForce update that you can install after the game (but before rebooting) to get it working.

Next, I tried to register for an online account. The game simply displays the message “Registration Failed: Invalid CD-Key”. How great.

Being at least able to play offline, I tried to create an image of the game’s DVD in order to play with the image mounted in Daemon Tools (Advanced Pro’s IDE drive). Guess what? StarForce just hangs until I unmount the DVD, at which point it crashes.

Fantastic. So for buying the game, I can not play online and have to play disc jockey, fetching the CD each time I want to go for a drive.

Had I just downloaded a pirated version, I wouldn’t have been able to play online, too, just like it is now. But I wouldn’t have to keep the CD in my drive, my Vista x64 wouldn’t have been screwed up and I would not have the revolting StarForce drivers on my system.

What exactly was the point of all this key code and DVD verification stuff?

Maybe game producers could just stop pressing DVDs altogether and upload their game, cracked, to a warez site. Honest people then just transfer the money to their bank account and can play like everyone else - without fucking up their systems, trying to decipher badly printed CD keys and waiting for shipping.

Coming home, I turned the server back on and began implementing an idea I had during the day: Write a small utility that would reboot the server as soon as memory load went above 66%.

This is unthinkable for a server OS, but in my case, I expect an uptime increase of at least 300% - 400%.

I then decided to listen to some music. About 1 minute into the song, silence.

Several seconds later, another second of the song player, then silence again.

I tried to browse to the folder (using my remote desktop connection), the explorer window hung. Couldn’t kill explorer.exe. Had to push the reset button again.

After rebooting, PPPoE dialin failed. Then failed again. Then was automatically attempted and failed. Then succeeded.

Now I have full speed and no transmission errors. I’m curious what might cause such a hickup.

What does one have to do to install a server, go away and find it running the next time he returns?

As you can probably tell, I was a little bit pissed off when I wrote that last post.

Now, less than 7 hours later, I wanted to check on the status of my Windows Server 2008 machine. Already the activity LEDs on my Corsair RAMs were giving the tell-tale “zero usage with flashing spikes” indication.

Login worked. Launching the task manager worked, too. 1,92 GB memory load (out of 2 GB I have installed). Of course, no process was consuming that much memory (and login plus launching the Task Manager was extraordinarily quick)

The task bar only showed the hour glass cursor. Task Manager also had yet to show up in the task bar.

The first thing I did was use the task manager to kill Halite.exe, my BitTorrent client. Kill Process — “Are you sure” — yes. Process still runs.

After about 3 minutes of sitting and killing, the case was clear: My server was sitting like a dead fish in the water. It had paged out absolutely everything despite no memory actually being used.

There is no hope of of waiting this out. It wouldn’t make any progress if I gave it hours. No choice but to hit the reset button once more.

After rebooting, I waited (that’s what Windows Server 2008 is all about — waiting — mostly for something to fail) until the “Routing and Remote Access Service” launched. Clicked connect on my PPPoE uplink. Connecting 1…2…3…4…5…6…7…8…9…10…

Now I know from experience that if it hasn’t connected by 5, it’ll happily count away to 30 or so and then tell me some useless error code. I clicked cancel. My next click on Connect resulted in the (for me, well-known) “invalid username or password” error message.

I also know this from experience, this error message (which is WRONG by the way) will pop up whenever I connect now, even if I turn off my PPPoE modem. No choice but to reboot.

The sorry piece of **** is turned off for now. You cannot run Windows Server 2008 unattended. Period. Luckily my PPPoE modem has an integrated NAT router (that’s based on Linux and crashes about once a week). Gotta go to work now…

Great.

Just when I wanted to go to sleep, this sucker killed my RAID array again. Browsing into a network share got my Vista workstation’s explorer window to hang (of course — what else should windows do but rely 100% on some network peer to provide data in a timely and reliable manner). I couldn’t close the damn fucking thing and had to kill explorer.exe. My compliments for not learning to write a usable file browser in over 20 years, Microsoft.

On the server it looked no better. Tried to browse to the shared folder, but of course, the window just hung. I couldn’t restart the server either (the hung window stalked me into the reboot screen), so I had to hard-reset it. Windows Server 2008 isn’t even worth the $0.50 media it’s pressed on.

After rebooting, the RAID array was gone. I couldn’t reactivate it.

Shutdown, reboot, one RAID disk functioning, two missing, two foreign disks. WTF?

Tried to reactivate missing disks - device not ready.

Tried to import foreign disks - warning, you will loose data if you proceed, bla bla bla. Shocked, but remembering that I had that before and clicking yes actually doesn’t destroy my data. RAID array resynching. All shared folders unshared.

As I’m writing this, my Vista desktop is completely unresponsive, because I tried to browse into one of the server’s administrative shares. This sucks major ***.

Nowhere in the Event Logs can I find anything that gives me a clue as to what is happening on this freaking server. I’m going to buy a 1 TB drive, copy my RAID array onto that, then install Gentoo Linux on the server, create a new (soft) RAID-5 with the old drives, copy the data back and be done with this. Suck my ***** Windows Server 2008.