Cygon’s Blog

I’ve held a PayPal account for several years now with no problems. My only usage of the account were occasional donations to some artists whose work I enjoyed, though I planned on using it to accept payments in the future for my indie business.

Well, last year I received an email from Paypal telling me that my account access had been limited due to suspicious activity and that I would have to verify my informations. Fully expecting this to be another one of those annoying email scams, I had already started a mail to spoof@paypal.com so they can take the scammer’s site down.

However, when I searched the supposed scam email’s source for the URL of the scammer’s site, I found none. I logged into my Paypal account and indeed, while I could browse my previous transactions and view my account history (all of which was as expected), the account page displayed a note that my account access has been limited and that I was to verify my account informations and, amongst other things, provide Paypal with a birth certificate. I wasn’t even allowed to close my account.

I wrote to Paypal, informing them that I’m not willing to provide a birth certificate since I see that as a breach of privacy and that whatever repercussions that would lead to, they should take action. My email was ignored and never answered. For one year, I kept getting nagging mails from Paypal telling me to provide the neccessary information, then it stopped.

Behold, when I try to log into my account now, this is what I see:

My tip if you’re using PayPal: don’t try to leave them with too much of your money at once and keep an eye on possible alternatives.

Update: I sent an email to the address from the error message above. Guess what the answer was?

Dear <name removed>,

We apologize but we are unable to respond to inquiries sent to this
e-mail address. Your e-mail was routed to an unmonitored mailbox and as
such will not be reviewed.  

To resolve account limitations, please complete the following steps:

1. Log in to your PayPal account.
2. Click Resolution Center at the top of the page.
3. Go to the Action column.
4. Click the "Resolve" button and complete the requested steps on each
lifting requirement outlined.

Very funny. I can not log into my account, but to tell PayPal about it, I have to log into my account first. Furthermore, the email is asking me to again follow the steps I am unwilling to comply with.

It appears to me that PayPal is a fully automated system with most decisions defaulting to the disadvantage of the customer. The system will freeze accounts following mysterious rules and an unhelpful, time-wasting and not thought-through customer support system blocks off any attempts to rectify such situations.

Update 2: Out of distress, I created a second PayPal account, navigated to the resolution center and wrote PayPal about my problems again. Make sure you don’t have anything in your mouth that could spill out if you’re starting to lough - here’s the reply from PayPal:

Dear <name removed>,

Thank you for contacting PayPal Customer Support. I will be happy to assist
you with your locked account.

To help you, please contact from your registered email address
cygon@nuclex.org of your locked account.

PayPal works hard to protect our customers, and uses many security measures
to help ensure your protection. For this reason, we can only send answers
for account related information when the request comes from an email
address that is associated with your PayPal Account. We feel that this is
the best way to ensure that your personal information is not compromised.

If you are unable to find an answer to your question in the Help Center,
you can reach our Customer Service Department by following these steps:

   1.  Go to the PayPal website and log in to your account.
   2.  Click the "Contact Us" link.
   3.  Click the "Contact Customer Service" link.
   4.  Under "Choose a Topic" section, select the topic for your inquiry.
   5.  Under "Choose the Subtopic" section, choose the subtopic that best
fits your inquiry.
   6.  Enter your question in the "Summarize your question in one sentence"
box.
   7.  Click "Continue.".
   8.   On the "Contact Us" page you will find a few suggested search
results based on your question. If you do not receive suggested search
results or if the suggestions do not answer your question, please complete
the form provided.
   9.   Select the language you are writing your message in from the drop
down menu
   10.  Click "Continue."
If you have an issue of immediate concern and cannot find your answer in
the Help Center, see "Help by Phone" for assistance.

I understand your frustration regarding this matter and regret any
inconvenience it might have caused you, Mr. <name removed>.

For crying out loud. Could it be that PayPal is run by a group of mean jokers?

Let me summarize: Once your account is locked, you’re instructed to contact PayPal by email. The email adress provided auto-replies, telling you to log into your account (which you cannot) and contact PayPal from there. And just in case you create a second account, a friendly member of the support staff will tell you to please log into your other, locked account (which you still cannot) and contact PayPal from there.

After getting KDE and Samba to work, I started looking out for a good BitTorrent client. Those MythBusters episodes aren’t gonna download themselves and I was already showing the first signs of Top Gear Withdrawal Syndrome (TGWS)

What I was looking for was a fast downloading client that used little resources and that I could run in the background as a service or daemon. I had tried MLDonkey on Windows Server 2008, but it was quite a pain to get set up right and torrent download speeds weren’t all that great (plus some trackers have actually banned MLDonkey). Azureus has a textmode GUI that I could theoretically combine with a WebUI plugin, but that would still be a bit too heavyweight.

That’s when I discovered rTorrent. It runs in the console (and thus, you can run it with DTACH) and can be controlled via a simple XMLRPC interface. And then there’s wTorrent, a nice-looking, nifty Web 2.0 AJAX GUI written in PHP that you can run in Apache or lighthttpd. I’m now running a daemonized rTorrent with the wTorrent GUI and it’s working so well it’s almost too good to be true

BitTorrent Performance

Now I have tried µTorrent, Azureus, Halite, BitComet, MLDonkey und some more, double checked that I had opened the required ports (using nmap from a server on the internet), used random ports >50,000 to avoid throttling, tweaked my settings and what not, but download speeds were, at best, average.

I left rTorrent running overnight with a 7 GB download. One that had somehow caused my Windows Server 2008 system to commit suicide by paging in no time, or that would complete but still have missing chunks. After one night, the torrent was at 60%, the next day it was finished. And that’s no exception.

RAID / Samba Performance

I reported a stable 20 MB/s upload in my last post. Forget that, it’s a stable 40 MB/s now that I’ve got no compile or torrent rehash running in the background. And download speeds are at a stable 60 MB/s — I believe that’s pretty close to what the hard drive I’m downloading to can do.

This is just incredible. It’s still the same hardware, but the new system could run circles around my old setup.

Today, I continued setting up my Gentoo server. With internet access available to my windows machine through NAT and remote administration working through SSH, I could easily look up resources on the internet and copy & paste between the linux console and my web browser

RAID5 was a simple matter of activating the required kernel options (which I already did beforehand). Somehow, mdadm created a RAID5 array with one spare. I haven’t investigated this much further, but it seems this allows the array’s initial synch to work faster. Whatever, I didn’t want it, so after finding out how to take the RAID array down again, I used mdadm with –spare-devices=0 and –force to have all disks UP from the beginning.

Using –chunk to set a block size of 128 kb and mke2fs with the -E stride=n,stripe-width=n options allowed me to tailor the RAID array’s stripe size to the file system. Not that I expect any noticeable gain, but it’s easily done and can’t hurt.

Next was samba. Gentoo makes this surprisingly easy. emerge samba, edit /etc/samba/smb.conf and you’re done. It took some effort to figure out how to create a password protected share, so this is what I did:

I’ve got a linux user named ‘cygon‘ on this system. This was created with the good ol’ useradd script. I added this user to /etc/samba/smbusers as an alias for “administrator” and “cygon“, so the file now looks like this:

# Unix_name = SMB_name1 SMB_name2 ...
# $Header: /var/cvsroot/gentoo-x86/net-fs/samba/files/config/smbusers,v 1.1 2007/09/07 21:07:40 dev-zero Exp $
#root = administrator admin
nobody = guest pcguest smbguest
cygon = administrator cygon

Then I set up a password for this user with smbpasswd -a cygon.

Finally, I created a folder that I wanted to share, assigned it to the user “nobody” (this is what samba uses for all files creates by guests) with chown nobody:nobody /var/storage/raid -R. This I then added in my /etc/samba/smb.conf:

[Protected]
  comment = Administrative share for the entire RAID array
  path = /var/storage/raid
  public = yes
  guest ok = no
  writable = yes
  printable = no
  force user = nobody
  force group = nobody
  create mask = 744
  directory mask = 755
  valid users = cygon

‘Public‘ says the folder can be seen by other network users. ‘force user‘ and ‘force group‘ make samba assign all files and folders created from networked users to the ‘nobody‘ linux user account and ‘nobody‘ linux group. Likewise, ‘create mask‘ and ‘directory mask‘ are the attributes assigned to folders created by network users. Finally, ‘valid users‘ tells samba to only let the listed users access the share.

That’s all it took to get this working.

The funny thing, again, is that performance and reliability increased by an order or magnitude compared to Windows Server 2008. I configured Samba to always act as my domain master browser, so when I browse my network neighborhood in windows, one click and all PCs on the network appear — immediately. Instead of 20 seconds of searching and then maybe half of the local PCs showing up.

File copies to and from the array are easily 4 times faster than with windows. And, most notably, they run much smoother. Windows Server 2008 accepted an average 20 MB/s for several seconds, then blocked hard for a while (probably flushing its insane cache), then accepted data again. Download speed was kept up longer, but also wouldn’t go over around 25 MB/s (30 MB/s tops) during the whole transfer (at least until the cache grows to the size of the physical RAM and the OS starts paging out unimportant things, like its own kernel, the DHCP server, its TCP/IP stack, DNS database, the RDP server and any running foreground application the user is currently working with)

Now Samba, which is just an implementation of Microsoft’s SMB protocol, pieced together by logging network packets, combined with a (Soft-)RAID 5 partition manages a stable 20 MB/s upload and a stable 60 MB/s download. And the kernel just takes the load - no suicide by paging, swapping out of vital system components or anything noticeable happening at all! The copy beats the original - by far.

Now my enthusiasm has faded a bit. I worked under the presumption that anything linux would be tested by millions of users and nearly every obscure bug that might occur like if you are running your PSU near its limit and decide to attach an USB hairdryer to the USB hub in your keyboard would have been found eventually.

The install CD passes some unterminated string coming from GRUB to the linux kernel as a parameter. This string then creeps into your environment. When you chroot into your system as it is being installed (and maybe forget to env-update && source /etc/environment - might have happened to me because I chrooted quite often until the thing could boot itself), this unterminated string in the environment gets saved.

Linux version 2.6.24-gentoo-r7 (root@poseidon) (gcc version 4.1.2 (Gentoo 4.1.2 p1.0.2)) #1 SMP Sat Jun 21 06:59:43 UTC 2008
Command line: root=/dev/ram0 init=/linuxrc dokeymap looptype=squashfs
loop=/image.squashfs cdroot vga=791Y^ÛóØÐö^A^È´þ¸7Õ^Fð>å)Ë6B¿u
^Cª^×ÎV^Gé^K¿ú^Éþ¢Xm.^ÍdrRW½Y^×g^Ú^C¤^ÛQ’ñ÷^5÷Vb²Iuf2õ^Ê°ý^ÁÀ¶^U§^Ï7ãÞ÷;9^U^Òg^
Y^SÛKÈÅd^Ù_^Óê|äKºpN» ðK´¸¼ “NÇ@ܹ¤À×ÉTÊHõ^Rùc;¨øÁ²^Ñ=PcûmMùÒû^É^YH^Óò^É!

I think gentoo’s emerge stores the environment in which a package was installed so that, when the package is uninstalled, this can be done under the exact same environment. For example, if you installed PHP with the apache USE flag, it would have installed the apache extension for php. If you now remove the apache USE flag and uninstall PHP, it wouldn’t know that it has to uninstall the apache extension — unless you let it run in a sandbox with the exact environment at the time of installation.

/var/tmp/binpkgs/app-text/ghostscript-gpl-8.62/temp/environment

[…]
vga=$’791Y\233\363\330\320\366\001\001\001\001\001\001\001\001\001\001\001\001
\001\001\001\001\001\001\001\001\001\001\001\001\001\001\001\001\001\001\001
\001\001\001\001\001\001\001\001\001\001\001\001\001\001\001\001\001\001\001
\001\001\001\001\001\001\001\001\001\001\001\001\001\001\001\001\001\001…
[…]
vga=”791Y^ÛóØÐö^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A
^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A
^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A
^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A^A…

To make a long story short, this saved environment was close to 500 KB, maybe only limited by the maximum environment size being reached. Add just one more thing to the environment (something an uninstall sandbox would likely do) and restoring the environment causes it to overflow.

Whenever I tried to rebuild a package or update it, emerge would fail with an error message (and continue doing its stuff — ignoring that a required package just failed to install — wow!).

/usr/lib/portage/bin/ebuild.sh: line 1496: /bin/touch: Argument list too long

This sucks pretty hard. Not knowing how to solve this, I tried increasing my kernel’s environment size and then (after that didn’t work), tried to rob portage of its saved environments. For that, I used find|grep to make a list of all environment.bz2 files in /var/db/pkg. Then I tried to tar the files in the list with the –remove-files option and when couldn’t get tar to do what I wanted, I tried passing the list to rm.

Well, ultimately I managed to clean out my entire /var/db directory - with the exception of my list of files to delete.

I started a reinstall from scratch, this time explicitely unsetting the vga variable in my install CD’s environment before even so much as touching emerge. Hope it works out this time around.

Today, I finally got rid of my Windows Server 2008 trial server. No point in “trialing” this thing any more — Windows kept killing itself by paging out vital system services despite plenty of memory being available, created unkillable phantom processes and the firewall system in Windows Server 2008 is a class of itself in terms of unusability (<– I think I just created that word :))

My Gentoo Linux 2008.0 x64 install went extremely smooth. The install CD recognized my networking settings, PPPoE dial-in took just a few minutes to get working after launching the SSH daemon, I could conveniently install the system using PuTTY on my Vista box, allowing me to cross-reference the installation guide and playing some games during the longer tasks.

I still don’t have any idea how people get their linux kernels trimmed down — I simply don’t know what options I really need and which just sound like I might be needing them. Thus, I went ahead and selected any device drivers that sounded like my hardware, then added the most likely options for raid, vpn, ppp and routing.

After GRUB was installed (which was a lot easier this time around since I’ve only got one boot partition - the server hosting my blog uses two boot partitions on different hard drives as a fail-safe mechanism), the kernel booted, networking was working and basically everything just did what it should.

Amusing fact: my Corsair memory modules have LED indicators on them that display the current memory bandwidth similar to a volume indicator in a stereo. With Windows Server 2008 idling away, the LEDs were wildly flicking between 50% and 75% load. Now with linux idling, only one lonely LED (out of 20) is lit up.

It took me some time to get NAT (IP masquerading) working and I’m still not sure I got my iptables configuration right. The examples I could find on the net all had some confusing and from my limited knowledge erroneous rules in them, so I decided to try it myself. This is what I’ve come up with:

# Generated by iptables-save v1.3.8 on Sat Jul 19 16:00:29 2008
*filter

# According to man, there are three "chains"
#   INPUT = Packets from outside with a destination on this machine
#   FORWARD = Packets being routed by this machine
#             (happens when another machine in the network has this machine
#             configured as its gateway)
#   OUTPUT = Packets being sent from this machine
#

# These are the default rules. They will only apply if a packet makes it
# through our rule maze without matching any rule we set up.
#
:INPUT ACCEPT [158:13292]
:FORWARD DROP [4:224]
:OUTPUT ACCEPT [1123:117012]

# -------------------------------------------------------------------------- #
# INPUT (packets destined for this machine)

# Allow all packets originating from the local network to reach this
# machine. This in effect means we trust anyone in the intranet.
#
-A INPUT -s 192.168.124.0/24 -j ACCEPT

# Of course, we will also accept packets we sent to ourselves.
#
-A INPUT -s 127.0.0.1 -j ACCEPT

# This lets any connections, once established, keep running without
# forcing the packets through all the rules we set up.
#
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT

# Here would be the place to local open ports in your firewall. To allow
# a web server running on this macine to be contacted from the internet
# using your ppp0 adapter, use this example:
#
#-A INPUT -i ppp0 -p tcp -m state --state NEW --dport 80 -j ACCEPT
#-A INPUT -i ppp0 -p tcp -m state --state NEW --dport 443 -j ACCEPT

# All other packets are rejected
-A INPUT -j REJECT --reject-with icmp-port-unreachable

# -------------------------------------------------------------------------- #
# FORWARDING (packets being routed through this machine)

# Allow any packets from the local network to be routed to
# the internet connection on ppp0
#
-A FORWARD -s 192.168.124.0/24 -o ppp0 -j ACCEPT

# Allow any packets coming in from the internet connection on ppp0 to
# be routed to the local network
-A FORWARD -i ppp0 -d 192.168.124.0/24 -j ACCEPT

#-A FORWARD -i eth0 -
#-A FORWARD -s 192.168.124.0/24 -m state --state NEW -j ACCEPT

COMMIT
# Completed on Sat Jul 19 16:00:29 2008

# -------------------------------------------------------------------------- #
# NAT

# Generated by iptables-save v1.3.8 on Sat Jul 19 16:00:29 2008
*nat
:PREROUTING ACCEPT [38:2923]
:POSTROUTING ACCEPT [31:2379]
:OUTPUT ACCEPT [40:3005]

-A POSTROUTING -o ppp0 -j MASQUERADE

COMMIT
# Completed on Sat Jul 19 16:00:29 2008

# Generated by iptables-save v1.3.8 on Sat Jul 19 16:00:29 2008
*mangle
:PREROUTING ACCEPT [3568:275800]
:INPUT ACCEPT [3564:275576]
:FORWARD ACCEPT [4:224]
:OUTPUT ACCEPT [3551:635930]
:POSTROUTING ACCEPT [3551:635930]
COMMIT
# Completed on Sat Jul 19 16:00:29 2008

I’m currently recompiling the entire system to make sure the stage3 packages match my compiler settings. That will probably take a few hours, after which I will proceed to set up Samba, MySQL, Apache, KDE and, ultimately, 3D acceleration and Unreal Tournament 2004

I’ve discovered a cool use of google maps, a website where everyone can enter his own running routes: runmap.net

I think this is a very nice idea, since it allows me to exactly determine the length of my running routes. Plus I can try out other routes in my vicinity and perhaps even make contact with other runners (well, once my fitness is on a competitive level, that is, I don’t want to embarrass myself :)).

Today I ran this route. From start to finish it took me about 1 hour 18 minutes, with my heart rate averaging 168 bpm (max 190 bpm). The route is about 8.4 km long and, as you can see, includes some cross-country running.

  Not too good since I know that I was trying to beat the 1 hour mark some time ago. It was snowing throughout my whole run (yes, snow in April) and I woke up with a clogged nose, so I’m confident that I can hit the 1 hour mark again later in the year!

I’ve been using BitTorrent for several years now. Where else, for example, to obtain the latest Top Gear episode?

The first client I tried was Azureus. Azureus worked fine for some time, but when Azureus Vuze came, which tries to transform Azureus from a functional torrent client into a platform for searching, viewing and publishing files on the BitTorrent network, I decided to go looking for an alternative.

I then used uTorrent for some time. It was more responsive than Azureus and got the job done, but after I upgraded to Windows Server 2008 (Beta3), it regularly started hanging. Worse even, it took a “break” (>5 minutes of unresponsiveness) whenever I tried to add a new torrent to the list. So I checked out some other BitTorrent clients, but eventually came back to…

Azureus. As I found out, you can disable the fancy new GUI in Azureus. Azureus worked, but after leaving it running on my home server for a day, the whole RAM was taken.Not by Azureus, but something about Azureus allocation behavior caused the Windows Memory Manager to fragment its memory or whatever, so that after a day, 99% of memory were filled with …nothing and everything that could be paged out had been moved to the page file. So with my latest home server reinstall, I went hunting for another BitTorrent client.

That’s when I discovered Halite. Halite is written in C++ and uses the Boost (www.boost.org) library, so it’s completely built on technology I know and respect. For those who don’t know, Boost is a collection of pretty advanced C++ libraries that attract a certain mindset of programmers — using Boost is already a bold indication that no whacky hobby coder is at work here). There even is an x64 build of Halite. So for the first time ever, I’m running a 64-bit BitTorrent client. And it is working extremely well. It’s fast, responsive and uses very little memory.

Guess I’ll have to ease my frustration by letting out another rant about Windows Server 2008.

Don’t use this to guide your decision for or against Windows Server 2008. As I said in my previous post, resourcefully titled “Windows Server 2008“, I’m still running Beta 3, so things might have changed in the RTM build. I won’t know, however, because I don’t have the money to buy it, well, unless I can score a lucky hit on ebay

My home server is running a (soft)RAID-5 with 4 drives. The drives are configured so they get turned off after 60 minutes of inactivity. This is a pretty simple scheme since when I’m at my PC, I usually access them all the time (streaming music, movies recorded by my Siemens Gigaset M 750 S, which also isn’t without its problems) and when I’m not at home, the drives can go to sleep for 20+ hours straight.

So the rule should be simple: Drives used -> turn them on for 60 minutes.

Unfortunately, Windows Server 2008 appears to have its own, unique, understanding of this rule.

If I just browse my server’s file shares, the drives spin up, the directory is read, then the drives spin down again. Well, sometimes they spin down while I’m still browsing, causing my Vista PC’s explorer windows to hang - dear Vista team, I can almost feel the quality code that is in Explorer.exe :o)

Sometimes they will also spin up for no apparent reason. Maybe the Windows File Indexing service decided to go on patrol, but who knows.

At other times, I want mount an .iso image from a file share in Daemon Tools. By browsing for the .iso image, I cause the hard drives to spin up. But when I then don’t access the virtual drive within seconds, the drives spin down again. Unlike explorer.exe, which shits itself, Daemon Tools unmounts the .iso again when it cannot be accessed anymore. So, from the user point-of-view, you mount an .iso, attempt to open the virtual drive and discover the .iso is not mounted anymore.

When I’m just pressing play in my WinAmp and the drives aren’t spinning yet, there’s another funny thing to observe. First, WinAmp freezes until the first drive has spun up. Then you get to listen to some seconds of music and WinAmp freezes again. Now the next drive spins up. Then another few seconds of music and WinAmp freezes yet again. When finally all drives are running, you are about half a minute into the song.

Maybe this last thing was done to not push the PSU by spinning up all drives at the same time. Who knows. But what’s so hard about my rule?

When the drive is accessed, spin it up for 60 minutes.

Is that asking too much?

I’ve always been rather neutral about the whole Windows versus Linux thing. For me, Windows was easier to manage, had better development tools, my games ran on it and I didn’t have to put up with all the half-baked solutions following a thousand different ideologies on how a program should interact with the user.

However, there’s one thing that makes me seriously think about switching my home server, currently running Windows Server 2008 (Beta3), to Linux. You can observe it on any LAN and, if you’re running a home server, you’ve most likely run into this issue more than once, too:

As soon as someone in the LAN downloads a big file from your windows machine, let’s say an .avi of a recorded TV show, your system is dead. Just opening the task manager to look what’s wrong takes literally minutes. Heck, it already takes that long to get the context menu to appear. The system becomes an upload slave and cannot be used anymore until you friend has finished his download… and by that time, the only sane thing to do is to reboot — if you have the patience to wait for the start menu to open.

This has hit Windows XP, it has hit Windows Server 2003 and it is hitting Windows Server 2008 right now as I’m writing this.

What can be observed in the task manager is that windows uses all available memory and then some for its disk cache. There is no sane limit on RAM, there is no stopping, windows will page out all running applications, the desktop, explorer, the start menu, your calculator, your clock, absolutely everything so it can fill the entire RAM with the never-again used contents of the file your friend is downloading.

This is hitting Microsoft’s server OS just as good as its workstation and home Windows packages. The holy file cache is allowed to use every bit of RAM in your system, all else becomes secondary to that. Sharing a large file is basically a way of asking for your system to be taken down, or more specifically, to take down itself.

For me that means: (File Server) == !(Microsoft OS), or in words, if you want a file server, you cannot use a Microsoft OS.

I wonder if there is a way to limit windows’ cache, or to disable caching of files on a specific partition or something else that prevents Windows from dying when it serves a file. Anything.

…or is Microsoft’s latest and greatest Server OS really such a miserable joke?

Spam in my inbox has slowly increased to an inconvenient amount. I’m using SpamAssassin and some spam emails even had a negative spam score, meaning they get a bonus to get through even when other spam criterions were met.

Most of these emails had the BAYES_00 rule, so I did some research. The bayesian filter in SpamAssassin (that’s the one that dynamically learns what kind emails you consider spam) has three outputs in SpamAssassin: BAYES_00 (meaning the bayesian filter thinks the email is good), BAYES_50 (meaning the bayesian filter thinks there’s a 50% probability of the email being spam) and BAYES_99 (of course meaning the email is spam with a 99% probability).

It looked like my bayesian filter had somehow learned to see spam emails as good emails. This surprised me, because I’ve set up two special folders in my IMAP inbox named ‘ham’ and ’spam’ into which I move any emails wrongly classified by SpamAssassin. An hourly cron task then picks up any emails moved into those folders and train my bayesian filter. So I assumed the darn thing would be well trained for the kind of spam I’m receiving.

What actually happened was that the cron task ran under the user account cron was using while SpamAssassin scanned my emails under another user account I had set up specifically for simscan, the utility I use to scan my emails at SMTP upload time (normal email servers accept all emails and then scan for spam; mine rejects spam right when it is being uploaded - this allows my email account to appear as being permanently unavailable or disabled for any spammer).

So, to make a long story short, all my efforts to train SpamAssassin’s bayesian filter for nearly two years have been for naught because I had trained cron’s bayesian database instead of the one used by simscan to scan incoming emails. Unattended, the bayesian filter slowly auto-trained itself to regard spam emails as good emails.

I cleared my bayesian filter’s database and I’m now hoping that this time, I will be able to properly train it. Here’s the shell script I’m now using:

#!/bin/sh

# ------------------------------------------------------------------
# Train SpamAssassin
#
for homedirectory in /home/*
do
  if [ -e $homedirectory/.maildir ]; then
    chmod +t $homedirectory/.maildir

    if [ -e $homedirectory/.maildir/.SpamAssassin.Spam ]; then
      if ls $homedirectory/.maildir/.SpamAssassin.Spam/cur/* >/dev/null 2>&1; then
        sa-learn \
          --username=simscan \
          --dbpath /var/spool/simscan/.spamassassin \
          --spam $homedirectory/.maildir/.SpamAssassin.Spam/cur/*
        mv $homedirectory/.maildir/.SpamAssassin.Spam/cur/* \
           $homedirectory/.maildir/.Trash/cur/
      fi
    fi

    if [ -e $homedirectory/.maildir/.SpamAssassin.Ham ]; then
      if ls $homedirectory/.maildir/.SpamAssassin.Ham/cur/* >/dev/null 2>&1; then
        sa-learn \
          --username=simscan \
          --dbpath /var/spool/simscan/.spamassassin \
          --ham $homedirectory/.maildir/.SpamAssassin.Ham/cur/*
        mv $homedirectory/.maildir/.SpamAssassin.Ham/cur/* \
           $homedirectory/.maildir/.Trash/cur/
      fi
    fi

    chmod -t $homedirectory/.maildir
  fi
done