Cygon’s Blog

My web server is providing access to several subversion repositories by means of mod_dav_svn, an Apache module allowing subversion clients to access repositories byconnecting to an URL managed by the Apache server.

That server is also running a continuous integration system which does automated builds of my sources as soon as I commit something new to a repository. For this to work, it checks all of the repositories once per minute. This has bloated my access_log to several megabytes of useless chatter each day.

After some searching, I discovered this method to make Apache exclude requests from localhost in access_log:

# Set 'dontlog' variable if IP address of requester is our own
SetEnvIf Remote_Addr "85.25.133.66" dontlog

# Only log this access if 'dontlog' is NOT set
CustomLog
  "|/usr/sbin/cronolog /path/to/logs/%Y-%m-%d_access.log"
  "%h %l %u %t \"%r\" %>s %b"
  env=!dontlog

You will have to replace the logging target with your own settings, of course

I already tried getting rid of Qmail <a href=”#”>about one year ago</a>, for various reasons. Today, while scanning my logs, I discovered that for the past 2 days, qmail was happily distributing emails to unheard-of recipients, most likely SPAM.

Holy fricking cr*p! I hate HATE H-A-T-E ! ! spam and now I find that this stupid piece of sh*t is sending out unsolicited emails.

Probably I’m to blame because of some configuration issue. I’m just so happy having chosen an email client that, after a year of running, still isn’t configured 100% secure and pops up new issues. The qmail handbook tells me that part of the motivation for writing qmail was to create an email server that was safe by default instead of requiring an expert admin (like sendmail supposedly did). Is the SMTP protocol really so crude and complicated that any SMTP server has to be a nightmare to set up?

Is there any other SMTP server where I can get smtp time spam rejection to work?
Would it be better if I just dumped my personal email domain and switched to gmail?

Maybe I’ll give courier another try. Looked very good (althought still needlessly complicated if you ask me), but I capitulated when I tried to set up SpamAssassin at SMTP time through some Perl nonsense.

Some time ago, when I built my website (www.nuclex.org), I reviewed several content management systems (CMS) and finally decided to use drupal. Drupal has been nice, but there are still a lot of things (no good forum module or integration, breadcrumbs not working too well). Now I’ve got several other CMSs installed for testing and am thinking about whether to switch to one of them.

Xaraya

• A CMS should be intuitive to use. The entire navigation of a Xaraya site (prime examples are the articles by category view and the category refinement system) always hits the most undesirable, hard-to-understand way to present things with surprising accuracy. Just visit xaraya.org and try to get a clear grasp of the site’s structure for a start.
• Things like a module that provides downloads to the user being called the ‘uploads’ module further instill my awe at how efficient the developers are at breaking with conventions and doing things in the most unexpected way.
• Really bad SEF URL system. For once, there is no way to map an arbitrary URL to a page (which would make switching CMSs easier; see Drupal’s url_alias module; optimally each alias URL should have an option to report http 302 page moved to the user). The SEF URL path is hardcoded, you would have to modify your Xaraya distribution to get URLs like “news/2007/2/25/page-title”. SEF URLs are the responsibility of each module on its own, so the xarbb (forum) module will have your forums appear as “forum/category/c23″ and you can’t do a thing about it. The XLink module’s functionality is very limited. You can not make a short url actually become the url of a page, it’s just an alias for it and whereever the page is mentioned, you’ll probably see the old-style URL again (so you can be sure people linking to your page will probably use cryptic URLs)

Drupal

• The template system is too inflexible. All drupal themes have to provide a fixed set of regions: header, left_sidebar, right_sidebar, content and footer. You can change this by creating your own template engine for drupal, but then you’re responsible for maintaining it in and upgrading it to future versions.
• Drupal is much too lightweight. Being lightweight is not a bad thing in itself, of course, but drupal just doesn’t provide a robust framework. For example, there’s still no established way to handle files in drupal. One module allows you to attach files to articles that will all be stored in one directory when they’re uploaded, another will create per-user directories for the user’s avatar pictures, yet another will use a third scheme for storing images that are embedded in an article.
• There is no usable forum module in drupal. The forum provided by drupal looks unattractive and offers only the most basic functionality. You will have a hard time growing a community when people are appalled by your discussion area. Integration of external forums is also cumbersome most of the time.
• The url_alias module of drupal allows you to let any arbitrary URL point to any location in your page. This is outstanding. You can migrate from any other CMS with SEF URLs and just keep your old URLs working for search engines and other people linking to you. Even better, the first URL alias you assign to a page actually becomes the main URL for that page throughout the site.

Joomla

• The are no hierarchical categories in Joomla. Your site has sections and your sections have categories. That’s it. Two fixed levels. Well, the truth is that actually, most of the time, you don’t need more than that.
• Has a very professional appearance. New versions aren’t release nearly as often as for other CMSs, so you have much less trouble with outdated modules, not working templates and such. Plus, this kind of stability has attracted many commercial developers, leading to a wealth of high quality modules for this CMS.
• Lots of themes. Also, as an effect of aforementioned stability, there are lots of sites providing well designed, professional looking themes.

E107

• Consisted of a long number of .php files in its root directory. No classes, smells like unorganized copy & paste giantism. I wonder how many security vulnerabilities are sitting in there. Get away already…

Typo3

• This one makes me want to puke. I’ve heard professionals are using this, but I cannot understand why. What trust should I put in a CMS that is impossible to get installed 100% working (you need an antique version of some discontinued PHP image module to get Typo’s image features working as they were intended to). There’s no clear concept visible in the way Typo organizes its pages. Its writer seems unable to speak proper english. Buttons with important functionality are placed in tiny what must be 8×8 pixel icons under gigantic text boxes. It’s got its own page scripting language, TypoScript, to artificially increase the learning curve.

Exponent

• This is a pretty exotic CMS. There’s no admin area, you just browse around your page and edit anything you want and create sub pages whereever you want. Don’t just think “wiki”, this is something more, read on:
• Instead of modules providing a rigid design, you design by yourself where specific functionality of your website appears. For example, in the typical CMS, the ‘forum’ module will have a page that displays the forum and that’s it. In Exponent, the ‘forum’ module is a widget that you can put anywhere on any page you want.
• The missing admin area is a bad thing for me. Larger sites need a place where you can do administrative tasks in an efficient manner and get an overview of things instead of browsing around your page looking for things to fix.

Umbraco

• Ajax in its purest form. It’s really fun creating pages in this environment. You’re presented with a nice, interactive view where you can set up your pages, write content (in a state-of-the-art wysiwyg editor of course) and manage your site.
• In it’s current state, it’s only really suited for a mostly static site you want to edit in your browser. It doesn’t offer the required functionality to set up a community site, web shop or other such things.

I think I’ll stay with drupal for now!

It is done!

For the past months, I’ve secretly been working on an all-new and improved website to supersede www.lunaticsystems.com, the game programming site I built on Xaraya back in 2001.

Whilst Xaraya provided everything I ever needed, I never had the nerve to delve into Xaraya theme development to create an inviting site that didn’t cause visitors to run away screaming.

This time, things were different. The new site uses Drupal, which has a much more fluid user experience to begin with. Some modules needed minor customizations and I spent some design work changing the forum to look more like other forum systems my visitors may be accustomed to. But in overall, the decision to use Drupal saved me a lot of effort. As a result, I had much more time available to fill the site with content.

The new site now focuses on game programming in C# using the Microsoft XNA framework (successor to Managed DirectX). The new programming environment is much more productive than the C++ toolchain and I feel that the C# community in overall is also more enthusiastic.

I’d love to welcome you on my site. If you’re a fellow game programmer, you will surely find something of interest there!

If your client happens to be Windows XP, you’re bound for some trouble. Of course, for something as important as WebDAV that you’re likely going to use for accessing your entire htdocs folder, a password sent as clear text will not cut it. So the authentication method of choice is digest authentication. Windows XP clients do support htdigest authentication, however, since Windows 2000, someone at Microsoft broke the code, causing Windows to think our WebDAV share actually uses another kind of authentication and to transmit the windows domain part with it.

There have been various recommendations, ranging from adding a # to the end of the WebDAV URL when configuring your windows client up to a special apache authentication module which will cut the windows domain part from the transmitted credentials before handing them over to the WebDAV module. This guide will explain how you can configure your Apache2 server so it works with all clients - unmodified.

The most important thing first: You need SSL. Using an https:// URL makes Windows XP use WebDAV and goes around the authentication problem in its entirety.

In your <VirtualHost /> section for the SSL IP (see [[Configuring multiple domains in Apache2]] why you should have this) we will define two aliases for the WebDAV share:

Alias /webdav/ /var/www/
Alias /webdav /var/www

This avoids another common problem of WebDAV clients, including, but not limited to, Windows XP.

Of course we now need to tell apache that this directory is supposed to be accessed via WebDAV and that the server should require authentication before letting a user fiddle around in our sensitive web roots:

<Location /webdav>
    DAV on

    Options Indexes FollowSymLinks

    <LimitExcept OPTIONS>
        AuthType Digest
        AuthName "webdav"
        AuthDigestFile /var/www/ssl-xy/conf/.htdigest

        Require valid-user
    </LimitExcept>

</Location>

That’s all, folks. No additional modules, no hazzle for the clients, no non-conforming server. Windows XP clients will now be able to access this WebDAV server as will any other WebDAV client.

This article originally appeared in my wiki, but since I’ve got different plans with that site now, I’ve just blogged it because I think it might be useful to some people

Obviously, in the spam business it is economical to to hire an office of typing slaves that surf around websites and try to send you spam where you don’t expect it. Your website contact form for example.

At least this seems to be the case with eBrandz, by their own description a “search engine optimization” company, that was contracted by another indian company whose name I’ll not disclose because it is, as I’m willing to believe, not aware of eBrandz’ questionable business practices.

• The first time, some danny.ebrandz@hotmail.com used my website’s contact form to tell me that he had placed a link to my website on his own site and that I should kindly link back to his site.
  

  Dear Webmaster, My name is [name] and I just wanted to let you know that we have already placed a link to your site on the following webpage: [url of hidden page on advertising client’s site]. Your site details are As Follows: [target site description from google open directory]. Kindly link back to our site with the following details: [description of advertising client’s site].

  The page containing the link to my website was there, but was in no way connected with the website structure of the client. Just a hidden area to make poor webmasters believe there was an actual outgoing link. I friendly replied to the sender’s hotmail account that I wasn’t interested. I got no reply.

• Some days later a bloke whose name I don’t know anymore because I deleted the mail repeated the exact same message, only the name was different. I visited the client’s website, looked up the marketing department’s email and repeated my friendly notice. I got no reply.

• Today, some zavier.ebrandz@hotmail.com pulled the same. Again. Oh, wait, now the message template had been changed:
  

  My name is [name] and I have just gone through your site, and visited many pages. It would be better if we link to each other as reciprocal link place an important role in a search engine ranking algorithm.

  I have already placed a link to your site on the following webpage: [url of hidden page on advertising client’s site]. Your link details are here: [target site description from google open directory]. I humbly request you to link back to us with the following details: [description of advertising client’s site].

I have to admit this is one step up from the usual buy 100 million email adresses and send undecipherable advertising mails to everyone practice. It’s almost like… hm… crime vs. organized crime. Do we have organized spammers now?

On Friday evening, I decided to upgrade this server’s kernel, which had been running for more than 6 months, to the current stable release, 2.6.17-gentoo-r7. Somewhere in this process I decided to build my boot partition from scratch: rm -r /boot/*, reinstall grub, put the kernel image and I’m on my way again.

Or so I thought. For the last 4 days I’ve been messing around with this server’s configuration and only now have I managed to boot the darn thing up again.

I’m sure I would have solved this problem in less than 30 minutes if I were able to see the console output during boot time. The reason I had to work blindly was that this server is not at my location and the provider doesn’t have KVM-over-IP, console forwarding or serial line monitoring.

Why can’t the linux kernel just write stdout into some file somewhere? Even if it would mean to temporarily reuse my swap partition as an ext2 partition to store the log or something. I can only see what happened when the whole ding has mounted the root file system and started the logging daemon, which is pretty much near the end of the boot process.

My latest server update surprised me by suggesting to replace my GCC 3.4.6 with the all-new GCC 4.1.1. Looks like the new GCC is assumed to be stable enough for production usage now.

I tried using -march opteron but some configure scripts began whining that “GCC is not able to create executables” (a quick hello world program worked for me), so I’ll stay with -march athlon-xp for now. It took several hours to recompile my whole system but thanks to gentoo’s portage, I didn’t need to monitor the server during this time at all. Everything went smooth and everything that runs on this server is now compiled by GCC 4.1.1.

With one notable exception: qmail. Instead of including the required header, qmail decided to locally declare the strerror() function in TLS.c. Except that header still got included and since the function signature seems to have changed, this is now an ambiguous reference. So, Ctrl+Ztted out after the ebuild had extracted qmail, commented out the strerror() declaration in TLS.c and resumed with fg.

When I checked by mails a bit later, my inbox had been nailed with countless spam emails. Oh great, simscan wasn’t getting called anymore. The last time this happened I spent a week uninstalling and reinstalling qmail and then attempting to switch to the courier mail server. There’s no way I want to go through that again, so I very carefully examined my system.

Simscan was installed and configured in /etc/tcprules.d/tcp.qmail-smtp. I rebuilt the .cdb file and restarted svscan but simscan still didn’t run. To make a long story short, I still don’t know what went wrong, but I decided to try netqmail. It compiled on GCC 4.1.1 without any problems and after uninstalling and reinstalling simscan, it immediately sprang to work, too. Phew.

Next, authentication failed whenever I wanted to send an email. Turned out that /usr/bin/checkpassword-pam (which I’m using for smtp-auth) had been rebuilt and its rights were wrong again. A quick chmod u+s /usr/bin/checkpassword-pam got everything running again.

In my ongoing effort to build an appealing site in order to gather a .NET game development community, publish my articles and ultimately sell my games, I recently evaluated several CMS solutions. My top choices were Drupal and Xaraya and I’m still not fully sure which one to use.

Drupal has shown to have a surprisingly small codebase which makes it very easy to extend the system without first examining the its architecture from the ground up. On the downside, Drupal’s sources are a huge collection of of copy & paste redundancy, questionable programming practices and most of the official modules are pretty much tailored towards what the author’s requirements were.

As an example, the ‘article’ module, which is responsible for presenting the user with a nice, browsable overview of the articles on a site, by default shows types of nodes you ever created. That means your static page structure will be mangled together with your actual articles, weblinks, projects, issues and so on.

Drupal does not ship with a module to handle links to other sites in the way I’d like to. There’s the linksdb module which uses an ugly URL scheme and is fully detached from the node and taxonomy system (meaning you’ll have to rebuild the entire category tree for your links), the janode module which provides a new weblink node that is called “janode node” (wtf?) and the flexinode or CCK modules for creating your own node types.

I tried modding the article, story and janode modules to suit my needs. The package attached to this post contains the following four modules:

• weblink - The janode module modified so the node is named ‘weblink’

• article - The story module modified so the node is named ‘article’

• weblinks - The article module, renamed to weblinks and modified so it only shows nodes of type ‘weblink’

• articles - The article module, renamed to articles and modified so it only shows nodes of type ‘article’

So, effectively, you can now view all your articles as http://mysite/articles and all your weblinks as http://mysite/weblinks and everything is named the way you would expect it to be.

Download the package here: drupal-nuclex-nodes-4.7.tar.bz2

When I ran the weekly update on my server today, rebooting the system to ensure all running processes are using the latest version of all files, courier-imap failed to launch. No error message, nothing in the logs.

Thankfully, only four packages had been updated and knowing what to google for, I quickly discovered this bug report on gentoo’s bugzilla: Bug 98745 - courier-{imap,authlib} init scripts fail to work with baselayout-1.12.*

Basically, it seems as if the current baselayout package changes something in the way daemons are run, breaking the courier-imap init scripts which are badly written as it seems. The init scripts for the same daemons in the full courier package have been reported to be still working. So either switch to the full courier suite or replace your courier init scripts with the ones that have been attached to the bug above.

What actually shocks me is the bug’s age. It has been reported more than a year ago and today, it made its way into the gentoo stable branch. That’ll be a whole lot of fun for all those people installing courier-imap for the first time. How many days would you spend trying to get your configuration fixed until you finally consider the package itself is broken?

Update: The courier-imap package has been updated and includes working init scripts, so everything is back to normal now