Intrepid Blog

Sizing Up Post-Quantum Signatures

Sat, 13 Nov 2021 13:22:17 +0100

The future is Post-Quantum. Unfortunately, post-quantum cryptography is larger, especially the signatures. How much room is there in TLS? I had a look.

demandimport: load python modules on-demand

Sat, 05 Dec 2015 20:09:54 +0100

A typical python app starts with lots of imports of modules which are not always used. Usually this isn’t a problem. However, if you are writing a commandline-util in Python which should be snappy, every fraction of a second counts. Mercurial, the SCM written in Python, loads its modules on-demand to be snappy. I took their code and packaged it into a separate module demandimport such that everyone can use it for their own command-line utils.

Its use is simple:

import demandimport; demandimport.enable()

You can find demandimport on github.

Use MacPort’s ssh-agent

Mon, 07 Sep 2015 14:33:34 +0100

The version of ssh that ships with Mac OS X is quite old. If you want a more recent version to use, for instance, ECDSA or ED25519 keypairs you can install a more recent version of ssh using MacPorts with sudo port install openssh +gsskex.

Unfortunately, OS X won’t remember your fancy ECDSA or ED25519 key passwords in the KeyChain. This is because the old ssh-agent is running in the background. This is how to run MacPort’s ssh-agent:

Don’t forget to install openssh with the +gsskex variant. Otherwise ssh-agent won’t even try to talk to the KeyChain.
Run launchctl unload -w /System/Library/LaunchAgents/org.openbsd.ssh-agent.plist to disable Mac’s old ssh-agent. Note: run this under your user account — not as root.
Copy /System/Library/LaunchAgents/org.openbsd.ssh-agent.plist to ~/Library/LaunchAgents/org.macports.ssh-agent.plist and open the copy in an editor. In the copy, replace /usr/bin/ssh-agent with /opt/local/bin/ssh-agent and org.openbsd.ssh-agent with org.macports.ssh-agent.
Run launchctl load -S Aqua -w ~/Library/LaunchAgents/org.macports.ssh-agent.plist to enable the ssh-agent from MacPorts. Again, run this as under your user account — not as root.
Logout and login again.

That should do it. These steps are an updated version of this guide.

Setup Ubuntu IPv4/IPv6 router for VLAN-tagged PPPoE internet connection

Thu, 28 May 2015 11:46:45 +0100

My ISP, XS4ALL, delivers IPv4/IPv6 over fiber via VLAN-tagged PPPoE. Normally, you would use the (excellent) WiFi-router they lend you, but if you want a little more control you can install your own computer as router. In this post I will explain how to do this with Ubuntu 15.04. Some steps are specific to my ISP, but most of it will also apply to any other provider using PPPoE.

Rename interfaces
The router should have at least two interfaces: one to connect to the fiber-modem and the other for the LAN. It is convenient to rename the interfaces. Create a file /etc/udev/rules.d/10-persistent-network-names.rules containing

SUBSYSTEM=="net", ACTION=="add", KERNEL=="em1", ATTRS{address}=="12:34:45:67:89:0a", NAME="modem"
SUBSYSTEM=="net", ACTION=="add", KERNEL=="eth0", ATTRS{address}=="12:34:45:67:89:0a", NAME="modem"
SUBSYSTEM=="net", ACTION=="add", ATTRS{address}=="0a:89:67:45:23:12", NAME="lan"

Replace the MAC addresses by the corresponding MAC addresses of your interfaces (you can find them using ifconfig | grep HWaddr) and em1 by the current name of the interface connected to the modem. Note that we have two rules for the modem interface: this is to prevent udev form renaming the VLAN interface. If you know a nicer way, please let me know. Reboot and test whether this worked.

Configure interfaces
Add the following to /etc/network/interfaces

auto modem
iface modem inet manual
 mtu 1508

auto modem.6
iface modem.6 inet manual
 vlan-raw-device modem
 mtu 1508

auto lan
iface lan inet static
 address 10.0.0.1
 netmask 255.255.255.0

auto wan
iface wan inet ppp
 pre-up /bin/ip link set modem.6 up
 provider xs4all

iface wan inet6 auto

The block for modem is only used to set its MTU to 1508. We need this if we want to set the MTU of the PPPoE connection to the normal 1500 (and compensate for the 8 bytes PPPoE uses). If you run into trouble, you can leave all MTU declarations out.

The block modem.6 configures an interface for packets on modem tagged with VLAN ID 6. XS4ALL uses VLAN tagging such that it can deliver different services (internet on 6, TV on 4, …) via the same modem.

The block lan sets up the interface for a LAN on 10.0.0.x.

Finally, the wan block tells the system to set-up a PPP-interface using the configuration named xs4all. We will set that up in a moment.

Set-up PPPoE
Install the package pppoe and create /etc/ppp/peers/xs4all with the contents


ifname wan
noipdefault
+ipv6 ipv6cp-use-ipaddr
defaultroute
connect /bin/true
noauth
persist
mtu 1500
mru 1500
noaccomp
default-asyncmap
plugin rp-pppoe.so modem.6
user "someuser@xs4all.nl"

Also create /etc/ppp/chap-secrets with


"someuser@xs4all.nl" * "somepassword"

My ISP requires the authentication step, but (intentionally) does not check the username and password — you can fill in anything you like.

Reboot. You should have IPv4 on the router. (The rest: DNS, IPv6, IPv4 on the LAN shouldn’t work.)

LAN
We’re ready to set up the LAN. We will use dnsmasq as it will act as (caching) DNS server, DHCP server and RA (IPv6’s alternative to DHCP) server. Install dnsmasq and write the following to /etc/dnsmasq.conf


domain-needed
bogus-priv
filterwin2k
resolv-file=/etc/resolv.conf-dnsmasq
local=/lan/
expand-hosts
domain=lan
interface=lan
dhcp-range=lan,10.0.0.50,10.0.0.150,255.255.255.0,12h
dhcp-option=tag:lan,option:router,10.0.0.1
dhcp-authoritative
enable-ra
dhcp-range=::1,constructor:lan,ra-stateless,ra-names,12h

In /etc/resolv.conf, put:

nameserver 10.0.0.1

You might want to remove the package resolvconf as it might interfere. Now, in /etc/resolv.conf-dnsmasq, put the actual DNS server you would like to use. For Google’s public DNS, write:

nameserver 8.8.8.8

We’re almost done with the IPv4 LAN. To /etc/sysctl.conf append

net.ipv4.ip_forward=1
net.ipv6.conf.all.forwarding=1

This will allow the kernel to forward packages between all interfaces. If you have more interfaces, you might want to restrict the forwarding to lan and wan, but to configure this requires running a script at the creation of the interfaces.

And finally, install iptables-persistent and write to /etc/iptables/rules.v4, the following for the IPv4 NAT:

*nat
:POSTROUTING ACCEPT [0:0]
-A POSTROUTING -o wan -j MASQUERADE
COMMIT

Note that this will allow any connections from outside to the router on any port. Also, without configuration ip6tables, no connection is blocked form the outside. If you like a firewall (that is: whitelist which connections are allowed), you want to add some rules here.

Reboot. You should have IPv4 & DNS on the router and LAN.

IPv6
My ISP assigns a /48 via DHCPv6. Install wide-dhcpv6-client. Change /etc/wide-dhcpv6/dhcp6c.conf to

profile default
{
 script "/etc/wide-dhcpv6/dhcp6c-script";
};

interface wan {
 send ia-pd 0;
};

id-assoc pd 0 {
 prefix-interface lan {
 sla-len 16;
 sla-id 0;
 ifid 1;
 }; 
};

This will request a subnet and assign it to the lan interface. The IP of the router will be thesubnet::1 due to the ifid setting.

That’s it: reboot and enjoy your IPv4 & IPv6 connection.

Simple Elliptic Curve Cryptography in Python compatible with seccure

Tue, 16 Apr 2013 15:12:46 +0100

Instead of RSA, you can use an Elliptic Curve algorithm for public/private key cryptography. The main advantage is that keys are a lot smaller. With RSA you need keyservers to distribute public keys. With Elliptic Curves, you can just write: my public key is *jMVCU^[QC&q*v_8C1ZAFBAgD.

There are two drawbacks: first, Elliptic Curve cryptography is even harder to understand than plain RSA and secondly, there are only a few implementation of Elliptic Curve cryptography. In fact: I did not find any maintained Elliptic Curve package for Python.

Thus I wrote a Python package compatible with the excellent commandline utility seccure written by Poettering. Here are some examples of how to use the original commandline seccure and how to do the same thing in Python.

For a private key, you just pick a (long!) password. You can derive the public key with seccure as follows:

$ seccure-key Assuming curve p160. Enter private key: my private key The public key is: 8W;>i^H0qi|J&$coR5MFpR*Vn

In Python

>>> import seccure >>> str(seccure.passphrase_to_pubkey(b'my private key')) '8W;>i^H0qi|J&$coR5MFpR*Vn'

Now, to encrypt a message for the public key:

$ seccure-encrypt -o private.msg '8W;>i^H0qi|J&$coR5MFpR*Vn' Assuming MAC length of 80 bits. Go ahead and type your message ... This is a very secret message! ^D

In Python:

>>> ciphertext = seccure.encrypt(b'This is a very secret message\n', b'8W;>i^H0qi|J&$coR5MFpR*Vn') >>> ciphertext '\x00\x146\x17\xe9\xc1\x1a\x7fkX\xec\xa0n,h…

To decrypt the encrypted message:

$ seccure-decrypt -i private.msg Assuming MAC length of 80 bits. Assuming curve p160. Enter private key: my private key This is a very secret message! Integrity check successful, message unforged!

In Python:

>>> seccure.decrypt(ciphertext, b'my private key') 'This is a very secret message\n'

To create a signature

$ seccure-sign Assuming curve p160. Enter private key: my private key Go ahead and type your message ... This message will be signed ^D Signature: $HPI?t(I*1vAYsl$|%21WXND=6Br*[>k(OR9B!GOwHqL0s+3Uq

In Python:

>>> seccure.sign(b'This message will be signed\n', b'my private key') '$HPI?t(I*1vAYsl$|%21WXND=6Br*[>k(OR9B!GOwHqL0s+3Uq'

And to verify a signature:

$ seccure-verify '8W;>i^H0qi|J&$coR5MFpR*Vn' '$HPI?t(I*1vAYsl$|%21WXND=6Br*[>k(OR9B!GOwHqL0s+3Uq' Go ahead and type your message ... This message will be signed ^D Signature successfully verified!

In Python:

>>> seccure.verify(b'This message will be signed\n', b'$HPI?t(I*1vAYsl$|%21WXND=6Br*[>k(OR9B!GOwHqL0s+3Uq', b'8W;>i^H0qi|J&$coR5MFpR*Vn') True

You can find the Python library on Github.

Update: added support for Python 2.6, 2.7, 3.2 and 3.3.

msgpack for pypy

Tue, 29 Jan 2013 23:24:38 +0100

msgpack is a fast and small binary format for json. The Python msgpack module is even a drop-in replacement for the json module.

PyPy is an implementation of Python in Python with a tracing JIT. If your Python script runs for more than 1 second, it will probably be quite faster on PyPy.

PyPy is almost a drop-in replacement for Python. Only extensions modules written in C can be a problem — for instance the msgpack module.

Thus I wrote a pure Python fallback for the msgpack module, which has been merged upstream. As a rough benchmark, I measured the pack and unpack time of a 30MB msgpack-object from the wild.

Note that:

The execution times for the fallback with normal Python are all off the chart (15.4s, 15.1s, 10.1s, 10.0s).
PyPy is faster on the second run: it had time to trace and optimize at runtime.
For packing, PyPy (nightly) is almost as fast as the original C extension.
For unpacking, PyPy (nightly) is 340% slower. Yet it more than 10 times faster than normal Python.

This code should be in the next official release of msgpack-python. If you want to use it already, check out the git repository.

The first version did not run this fast on PyPy. With PyPy’s jitviewer the compiled code and assumptions of PyPy can be examined. These are the tweaks I used in descending order of impact.

Instead of StringIO, the fallback uses PyPy’s own StringBuilder. StringIO allows writing, reading, seeking and what more. StringBuilder only allows appending and thus it is easier for PyPy to optimize. This increased performance of packing by an order of magnitude.
Using constant format strings for struct.unpack allows PyPy to optimize it. Thus struct.unpack("I", f.read(4)); f.read(n) instead of struct.unpack("I%ds"%n, f.read(4+n))
Using stream.write(a); stream.write(b) instead of stream.write(a+b) increased performance.
Adding an explicit fastpath
PyPy usually specializes a function well for its most common path, however in some cases it needs help. In this case a function returns a concatenation of several strings. However: in the most common case it only does one: ret=''; ret+=something; return ret. PyPy does not recognize that ret is not needed at all in this case, so I added an if-statement before initializing it for the case where there is only one concatenation.

Clearly the unpacking code could be faster, if anyone with expertise on PyPy’s JIT would look at it, that would be great.

PyPy 2.0-beta1 for Debian Squeeze amd64

Sat, 26 Jan 2013 11:57:47 +0100

I compiled PyPy 2.0-beta1 for Debian Squeeze amd64.

FTL .dat packer and unpacker

Fri, 14 Sep 2012 17:04:25 +0100

I wrote a simple Python script to pack and unpack the data files of the indie game FTL, which is by the way quite enjoyable. You can find it here on github.

Fix excessive disk usage of Sparrow for Mac

Sun, 12 Aug 2012 15:43:25 +0100

It may happen that Sparrow for Mac uses an excessive amount of disk space. For only the headers of my 9GB of e-mail, Sparrow used 39GB. I found a work-around.

The problem
Sparrow uses a Tokyo Cabinet, called data.tch, as storage. Mine was 39GB. What could eat up all that space? I only have 9GB of e-mail in total. With a simple script I looked into the cabinet and found it actually contained less than 1GB of data. The cabinet was wasting a lot of space by not being packed tightly.

The work-around
To repack the Tokyo Cabinet, run:

tchmgr optimize "~/Library/Containers/com.sparrowmailapp.sparrow/Data/Library/Application Support/Sparrow/(your-email@gmail.com).sparrowdb/data.db/data.tch"

Replace (your-email@gmail.com) with your e-mail address. You can install the tchmgr program using sudo port install tokyocabinet, if you are a MacPorts user.

This reduced the size of the cabinet to 1.1GB!

Cause?
It is odd that the cabinet became so loosely packed. It may happen if Sparrow adds and removes a lot of entries, but there seems no reason for Sparrow to do this. There might be an underlying bug. I reported this issue to the developers, but they have yet to respond.

Bachelor thesis: on effective undecidability and Post’s problem

Wed, 11 Jul 2012 11:49:48 +0100

This is my bachelor thesis on effective undecidability and Post’s problem.

Henk Westerbaan

Sat, 09 Jun 2012 15:45:03 +0100

Could not open audio device for playback. with GStreamer on Rapsberry Pi

Sun, 13 May 2012 15:17:23 +0100

You can fix this error by explicitly setting the default audio device. For instance, put this:

pcm.!default { type hw card 0 }

ctl.!default {
type hw
card 0
}

in $HOME/.asoundrc

Split ape/flac with cue into oggs on Linux

Sat, 08 Oct 2011 15:56:02 +0100

$ shnsplit -o 'cust ext=ogg oggenc - -o %f' \ -f CDImage.cue -t "%n.%p - %a - %t" \ CDImage.ape $ cuetag CDImage.cue *.ogg

If you’re on Ubuntu, you’ll need to install cuetools and shntool. To split ape, compile and install this port of mac.

Radboud Universiteit Nieuws

Mon, 06 Jun 2011 15:50:24 +0100

De Radboud Universiteit censuurt onder het mom van een duidelijke scheiding tussen intern en extern nieuws sinds kort het niet-meer-onafhankelijke blad de Vox.

Gelukkig kunt u het nieuws alsnog lezen op RU Nieuws. Op deze site wordt al het interne nieuws gelekt.

Bertha: no-nonsense blob storage

Sat, 04 Jun 2011 11:53:43 +0100

For a project I need to store blobs of data (~10MB) and access them over TCP. I don’t need any features like removing, updating, authentication, statistics or replication: it’s simply not required or already handled by some other part of the project.

I only want to be able to store a blob and receive a key for it; retrieve a blob by its key and list all keys of the blobs stored.

I couldn’t find anything that fit it. Thus I created bertha. Lets delve right into it, shall we:

To run the server:

./berthad-vfs 0.0.0.0 1234 tmp data

There’s a python client

>>> from bertha import BerthaClient >>> c = BerthaClient('serf', 1234) >>> list(c.list()) [] >>> key = c.put_str('Hello world') >>> key '64ec88ca00b268e5ba1a35678a1b5316d212f4f366b2477232534a8aeca37f3c' >>> c.get(key).read() 'Hello world' >>> list(c.list()) ['64ec88ca00b268e5ba1a35678a1b5316d212f4f366b2477232534a8aeca37f3c'] >>> ctx = c.put() >>> ctx.f.write("Do some") >>> ctx.f.write("Streaming") >>> ctx.finish() '975001fb9bdc0f72a78ca6326c55af86348d4c84da7ba47b7ed785a03f6803b0' >>> c.get('975001fb9bdc0f72a78ca6326c55af86348d4c84da7ba47b7ed785a03f6803b0').read() 'Do someStreaming'

Which also install a bertha commandline tool:

$ bertha list 975001fb9bdc0f72a78ca6326c55af86348d4c84da7ba47b7ed785a03f6803b0 64ec88ca00b268e5ba1a35678a1b5316d212f4f366b2477232534a8aeca37f3c $ bertha get 975001fb9bdc0f72a78ca6326c55af86348d4c84da7ba47b7ed785a03f6803b0 Do someStreaming $ echo Hi | bertha put c01a4cfa25cb895cdd0bb25181ba9c1622e93895a6de6f533a7299f70d6b0cfb $ bertha get c01a4cfa25cb895cdd0bb25181ba9c1622e93895a6de6f533a7299f70d6b0cfb tmp $ cat tmp Hi

The berthad code is pretty small: at the moment under a thousand lines of C with lots of comments. GETs are pretty fast: berthad uses Linux’ splice syscall, which usually makes the network card read directly from the buffer the harddisk wrote to.

Lion display settings refresh-rate bug

Sat, 30 Apr 2011 18:36:35 +0100

When I try to set the displaymode of my external monitor to 1024×768@60 the preferences application doesn’t listen properly and sets it to 1024×768@120, which my monitor does not support. This seems to be a bug in the developer preview.

To remedy this bug, I wrote a simple commandline tool to set displaymodes in Mac OS X. It’s on github.

802.1X configuration profile on Lion (Mac OS X 10.7)

Wed, 16 Mar 2011 14:24:59 +0100

On the developer preview of Lion, the “+” button for 802.1X profiles was removed.

You can use the iPhoneConfigurationUtility application (Google for it) to create a mobile configuration profile with the 802.1X settings, that also imports perfectly fine on Lion.

MacPorts on Lion (Mac OS X 10.7)

Thu, 10 Mar 2011 22:40:17 +0100

Simply install from Subversion.

Thoughts on Flash

Thu, 29 Apr 2010 17:40:45 +0100

I have some remarks on Steve Jobs’s “Thoughts on Flash”.

“Adobe claims that we are a closed system, and that Flash is open, but in fact the opposite is true.”. I agree with Steve: Flash is pretty closed. However, the iP(hone/od/ad) isn’t open either.
1. H.264, which Jobs touts as a great modern replacement of flash, is patented. You have to pay whatever the MPEG LA fancies you to pay for use of the standard. This is not really different from the control of Adobe over Flash.
2. iPhone OS is closed. You need to buy yourself into the iPhone Development Program. Again Apple can shut the program down whenever they like, which is not really different from the control of Adobe over Flash development.
Safari is an intermediate layer and creates sub-standard apps. Jobs claims that third-party intermediate layers result in sub-standard apps. Jobs argues that an intermediate layer will keep developers from platform enhancements, won’t result in targeted great apps and won’t put apps directly on the shoulder of the platform. If he is thinking about Safari, he is right: web-apps for the iPhone just don’t feel great, can’t use all the platform enhancements and don’t result in great targeted apps.

Looking for an apartment

Sun, 28 Feb 2010 01:54:30 +0100

Dear readers,

I’m looking for an apartment in the neighborhood of Nijmegen. I’m not picky, but can’t afford more than €300,- in total per month.

Contact me per e-mail (bas@this domain).

Bas

Update I’m settled!

Dutch student protest

Sun, 31 Jan 2010 14:08:35 +0100

This week, various student organisations protest against major cuts in the funding of education and research.

A reader of dutch, can visit their site.

Unique followers on Twitter

Wed, 16 Dec 2009 19:27:12 +0100

As pointed out by Jorg Kennis, Twitter‘s new lists feature make it hard to determine the amount of unique followers. I’ve written a simple script, using a slightly modified TweePy, to determine the amount of unique followers.

Example usage:

bas@w-nz ~/twitter-unique-followers $ python twitter-unique-followers.py JorgK -ubwesterb Password: rate_limit_status: remaining_hits: 112 Jorg Kennis followed directly by 182 in lists RPtje/vriendenbekenden subscribed to by 0 JorgK/TechNL subscribed to by 0 sentfanwyaerda/nijmegen1 subscribed to by 9 JorgK/Friends subscribed to by 0 sjorsjes/Community subscribed to by 0 robinspeijer/iPhone subscribed to by 1 nielsschooneman/iPhone subscribed to by 0 JeanPaulH/iPhoneclub subscribed to by 5 number of unique followers: 198

You can download it here. I could write a simple webpage with the same functionality, if anyone would mind.

Normul

Sun, 01 Nov 2009 12:18:03 +0100

Normul normalizes URLs. It expands shortened URLs:

>>> from normul import normul >>> normul('http://bit.ly/1I4VQ') {'type': 'other', 'normalized': 'http://www.shinguz.ch/MySQL/mysql_mv.html', 'original': 'http://bit.ly/1I4VQ'}

And shows useful links for hosted-images:

>>> normul('http://yfrog.com/6c5krj') {'image': {'full': 'http://img228.imageshack.us/img228/1079/5kr.jpg', 'thumbnail': 'http://img228.imageshack.us/img228/1079/5kr.th.jpg'}, 'type': 'image', 'original': 'http://yfrog.com/6c5krj', 'normalized': 'http://yfrog.com/6c5krj'}

You can find the simple but convenient sourcecode here.

Cantor never bores (1)

Sat, 24 Oct 2009 21:11:23 +0100

Given a set of countable sets $K$ , such that $K$ is totally ordered by inclusion, videlicet for every $A,B\in K$ either $A\subseteq B$ or $A\supseteq B$ . Intuitively, for at every step in this chain one element at least must be added, one expects the set $K$ to be countable as well.

Suppose $K$ is countable. Then the union, $\bigcup K$ is a countable union of countable sets, hence countable. (Suppose $k: \mathbb N \to K$ is an enumeration of $K$ and $f_i: \mathbb N \to k(i)$ enumerations of the elements of the chain. Then $f_0(0), f_1(0), f_0(1), f_2(0), f_1(1), f_0(2), \ldots$ enumerates $\bigcup K$ .)

Thus $\bigcup K$ is an upper bound of $K$ . In the poset of countable subsets of some set $U$ , of which $\bigcup K$ is a subset, every non-empty chain has an upper bound. Hence, using Zorn’s lemma there is a maximal element, say $M$ .

Suppose $U$ is uncountable, then there exists a $\star \in U \backslash M$ . $M \cup \{\star\}$ is most definitely also countable and $M \subset M \cup \{\star\}$ which contradicts $M$ ’s maximality. We are forced to conclude that there exists an uncountable chain of countable sets.

Cantor’s set theory keeps surprising.

Update: an example of such a chain is the set of the countable ordinals.

Another update: a “more concrete” example are the downsets in $\mathbb Q$ without the empty set and $\mathbb Q$ itself. These downsets correspond to real numbers, see Dedekind Cuts.

Bond against Loanwords

Thu, 10 Sep 2009 00:37:14 +0100

Although I don’t bear any animosity against most dutch loanwords (except those Anglo-Saxon), the dictionary of the dutch Bond tegen Leenwoorden is a true joy to read.

21

Mon, 31 Aug 2009 11:18:48 +0100

Since yesterday I, for the first time, enjoy the anything-but-special age of 21 years.

Windows 7 Sins

Thu, 27 Aug 2009 14:06:16 +0100

Windows 7 Sins. I strongly agree with points 1, 3, 4, 5 and 6. The arguments for points 2 and 7 are a bit weak, but I do agree with the conclusions.

For the non US audience, it would have been nice to note that if Microsoft (or the US) would recall all foreign licenses for Windows, our entire government is crippled.

Twitter and Facebook

Fri, 07 Aug 2009 00:46:19 +0100

were both victim of a DDoS today. Silently, I always hoped that a really long-lasting one will convince them to put effort in a distributed scheme.

Maybe I should be waiting for another Wave.

PijsMarietje

Wed, 01 Jul 2009 16:58:44 +0100

At the faculty for sciences there are canteens for students. In each of these, there’s sound equipment connected to linux boxes. On each of those linux boxes, we run a music-request-server called Marietje. I just finished writing a front-end in Javascript. It wasn’t a pain. As instead, the use of jQuery was a bliss.

The frontend for one of those boxes and the source code (see the ajax folder).

Damned DOM (1)

Mon, 22 Jun 2009 15:34:51 +0100

When I wanted to react to any changes to a input textbox immediately, my first instrinct was to use onChange. onChange, however, is called when the input loses focus. onKeyPress then? Isn’t called on backspaces. onKeyDown, maybe? It does get called, but the effect of the keystroke isn’t yet applied, for the return value determines whether it that is done in the first place. (Same story for onKeyPress by the way.) onKeyUp does work a bit, except if someone is holding down a single key, for a while.

The solution: hook onKeyUp and use setTimeout with a timeout of 0. Yugh. I hate DOM.

Big Fat Disclaimer: I actually tested this only on one browser.

Unicode to ASCII (1)

Fri, 19 Jun 2009 14:19:10 +0100

When I want to generate usernames from real names, which can contain non-ascii characters, you can’t simply ignore the unicode characters. For instance, danielle@blaat.org is the right e-mail address for Daniëlle, danille@blaat.org isn’t.

There’s trick. Unicode has got a single code for ë itself, but it has also got a code which (simplified) adds ¨ on top of the previous character. The unicode standard defines a normal form in which (at least) all such characters, which can be, are represented using such modifiers. If you then simply ignore the non-ascii representable codes, you’ll get the desired result.

In python: unicodedata.normalize('NFKD', txt).encode('ASCII', 'ignore').

However, this isn’t the right solution. For instance, in german, one prefers ue as a replacement of ü over u.

Django annoyances: no reverse select_related

Sat, 30 May 2009 14:48:45 +0100

Consider

for page in Page.objects.all(): print page.title for comment in page.comments.all(): print comment.

There will be a single query to fetch all pages, but there will be for every page another query to fetch its comments. Luckily, Django has got a nice trick up its sleave: select_related. Would I use instead of Page.objects.all(), Page.objects.select_related('comments').all() then Django will use a single joined query to prefetch comments for each page.

However, Django’s select_related only supports forward one-to-many references. No many-to-many; certainly no reverce many-to-many; no reverse one-to-many and no, not even reverse one-to-one (yet). A developer claims it’s impossible (which is bullshit), another asks for patches, which means he doesn’t care doing it himself.

It’s quite easy to manually code around the missing reverse select_related, but it takes too many ugly lines compared to the single word it could’ve been.

Javascript’s stupid Date constructor

Wed, 29 Apr 2009 17:27:42 +0100

new Date(2009, 1, 1) represents the first of February 2009. Not the second of February nor the first of January. Why this stupidity?

GStreamer: accurate duration

Sun, 19 Apr 2009 14:26:01 +0100

When decoding, for instance, a variable-bitrate MP3, gstreamer reported durations are, to say the least, estimates. I’ve tried to get a better result in a few ways. First off, some files yield a duration tag, but even if you’re lucky and it is there, there are no guaranties about precision. After that I tried seeking to the end (GST_SEEK_END) of the stream and querying the position, which gstreamer didn’t like. Finally, routing the audio into a fakesink, waiting for the end of stream and then querying for the position gives the right result. It’s not the prettiest method, but it works.

This is a Python script that prints the duration of a media to stdout.

Spacing up- and downarrow properly for up- and downsets

Tue, 24 Mar 2009 23:34:44 +0100

$f(\uparrow x)$ is ugly, but $f(\left\uparrow x\right.)$ is nice! The solution: prefix \uparrow with \mathopen.

Fosdem (3)

Fri, 06 Feb 2009 13:57:29 +0100

In a few hours I’ll travel the short distance to Bruxelles to visit Fosdem. Once again I’m pretty excited :). Lets hope this time the pink elephants of the Delirium Cafe don’t crush me. If you’re also going, drop me a comment.

Timestamp 1234567890

Thu, 29 Jan 2009 23:37:42 +0100

It’s soon. The 14th of februari, 00:31:30 (Europe/Amsterdam). Will the world end? Will ancient libc code giggle and break?

Aperitif for order

Thu, 01 Jan 2009 02:16:09 +0100

Assign 1 to True and 0 to False. Now the minimum corresponds to “and” and maximum to “or”. If you give it a bit more though, less or equal to corresponds to implication. This is a lot more general than this specific case. Add .5 for a third value (eg. NULL) and it still yields natural results.

We can recognize the behaviour of minima and maxima in a lot of other things. Take for instance set inclusion as order with intersection as minimum and union as maximum. Actually, the link between general order and set inclusion is frequently made to then propose that “intersection of two set of cases” and “logical and” do look a lot alike.

This is just the tip of the huge iceberg. Order appears everywhere! Everywhere in Math. Everywhere in CS. And its just recognizing simple order I’ve demonstrated. Other useful concepts in order theory that I didn’t even touch are Galois Connections and formal concept analysis.

Oh, another example of an order are integers with bitwise or and bitwise and. It is left as an exercise to the reader when one integer is greater than another.

Interested? Buy an Introduction to Lattices and Order.

2009

Thu, 01 Jan 2009 00:41:53 +0100

Happy Newyear!

On-demand singleton for Python

Sat, 27 Dec 2008 20:28:33 +0100

Some singletons eat slightly more resources, than you want to give them for free. For instance, if you have a home-brew threadpool singleton, you don’t want it to create its threads if you are not going to use it. The solution: a simple function that creates a stub which proxies attribute access to an ad-hoc created instance.

Usage: create_ondemand_singleton('mylibrary.Threadpool', MyThreadPoolClass).

The Cuckoo Hashtable

Sun, 21 Dec 2008 23:22:41 +0100

A Hashtable algorithm is a specific algorithm to implement key-value pair datastructure with efficient by-key look-ups using hashing of the keys. A hashtable contains a list of buckets. In a simple implementation, the i-th bucket, contains the key-value pairs in a list of which the key has a hash that is i modulo the amount of buckets. The hash-table would increase the amount of buckets if any bucket contains more than a fixed amount of pairs. This results in a constant-time look-up, but an insertion might invoke a expensive rebuild.

There are a few variations on Hash Tables. I’d like to share a really smart one: the Cuckoo Hashtable.

The Cuckoo hashtable expects two different hash-functions for the keys. Instead of storing a list of pairs in each bucket, the Cuckoo Hash table stores a single pair in each bucket. When inserting a pair, it is inserted in one of the two possible locations. If it happens to be occupied, the old pair is replaced. Then the replaced pair is inserted at its other possible location, potentially kicking out another pair. This step is repeated, until there is no displaced pair or a loop is detected. When a loop is detected, the hash-functions can be changed, if the buckets are for the most part empty -or- when the table is almost full, the amount of buckets can be increased. It can be shown that an insertion has a amortized constant time. (The buckets can be resized in-place, and each entry is repositioned as if it was displaced by another.)

If I’d own a botnet… (1)

Wed, 26 Nov 2008 22:36:47 +0100

…and didn’t want to loose it if my control servers got shut down, I’d let every orphaned zombie randomly connect to hosts in a given IP range, and challenge them to give a preimage of a hardcoded hash. [ detail: add a salt to prevent replay attacks ]. With a sufficiently safe casu quo large range, it also might be helpfull to allow zombies to forward still orphan zombies.

Music Animation Machine

Fri, 10 Oct 2008 20:34:57 +0100

The sound and performance really aren’t that good, but the visual connection is so powerful:

Bach’s little fuge on a Music Animation Machine.

Vim Essentials

Fri, 19 Sep 2008 08:40:54 +0100

Vim Essentials. Usefull.

20

Sun, 31 Aug 2008 18:43:58 +0100

Yesterday we turned 20. I’ll miss that one.

Wacken (2)

Sat, 26 Jul 2008 14:04:57 +0100

Tomorrow morning, way too early, I’ll be leaving for the picturesque town of Wacken in the north of Germany to attend the Wacken Open Air festival. I’ll probably be back the third of august and recovered the fourth.

iPhone WiFi weirdness

Tue, 22 Jul 2008 18:32:44 +0100

I’m having trouble with the wireless on my 3G iPhone. Either an action works perfectly (for instance downloading e-mail or visiting a website), or it will just time-out. I just jailbroke it and installed sshd. ssh-ing to my it via 3G miraculously works perfectly. However, when connecting to it via WiFi it’ll simply refuse data. That is, until you let the iPhone try to get data itself by for instance visiting a website. Then suddenly the ssh packets come through again. In the background (with screen) I’m running a very frequent ping to my local gateway which makes ssh work perfectly again via WiFi. A very unsatisfying hack.

It seems like an aggressive power safe on the WiFi.

iPhone (0)

Fri, 18 Jul 2008 17:23:42 +0100

On the day of release, I bought an iPhone. We all know the great features; thus let’s talk about the annoyances.

It crashes and gets sluggish a lot. Especially Safari crashes a lot and the contact list at one moment took 10 seconds to get responsive. Over time random crashes in all applications appeared. Rebooting it, actually, seems to have solved it (for now).

Wifi is unreliable. Sitting right next to the router it’s still a game of chance whether it will load a website properly or just time-out. This often leaves me disabling Wifi, and using HSDPA instead (which is our 3G), which has a terrible coverage. Outside reception is great, but even inside a building near a window, it only shows one or two “pips”. This again, often leaves me disabling 3G in favor of the (slower) but the more reliable GRPS (?).

I use iTunes on Windows in VMWare to sync my iPhone, which works great for the music. For the other stuff, like contacts and photos it just does not work. Syncing with google contacts or the windows address book just doesn’t work on windows. There isn’t even any support for calendar syncing for windows. I still have to look intro a free exchange compatible server. (Anyone?)

T-Mobile in the Netherlands seems to have trouble with the administration of all new subscriptions. Billing information and visual voicemail still don’t work.

(Nevertheless, it’s a great toy)

PyX

Tue, 08 Jul 2008 01:57:56 +0100

The software I use to render those great graphs might be of interest to some. Instead of using a clumsy GUI or an inflexible graph language, I use a Python library called PyX which is able to generate beautiful graphs. At its core it’s a generator for TeX, and rendering graphs is just one thing it does well!

Take a look at the examples and convince yourself!

git‘s versus svn‘s storage efficiency

Tue, 08 Jul 2008 01:43:47 +0100

At Codeyard we maintain a git and a subversion repository (which are synced with each other) for each of the >115 projects. The following graph shows the repositories plotted logarithmically according to the size of their whole server side subversion repository horizontally and their git repository size vertically:

To make more sense of the logarithmic nature of the graph, I’ve added three lines. The first (solid black) indicates the points of which both sizes are equal. The second course dashed line indicates the points of which the subversion repository is twice as large as the git repository. And lastly, the third finely dashed line indicates the points of which the subversion repository is five times as large as the git repository.

All projects for which git is less storage efficient, are smaller than 100Kb. The projects for which git is most storage efficient (up to even 6 times for a certain C# project), are all of medium size (10–100MB) and code-heavy. For the other projects, which are blob heavy (eg. images), git and subversion are close (git beats svn by ~20%).

One notable disadvantage of huge (someone committed a livecd image) git repositories, is an apparent $\geq2N$ memory usage of git repack even if I tell it not to with --window-memory.

find: invalid predicate `-delete'

Sun, 29 Jun 2008 19:30:31 +0100

Check for a stray version of find in /bin.

“waiting for x server to begin accepting connections”

Sun, 29 Jun 2008 18:19:58 +0100

Try DisallowTCP = false under [Security] in /etc/X11/Sessions/Gnome/custom.conf or similar for other window managers. Obviously this isn’t a very desirable solution, emerge --emptytree gnome might do the trick too.

(and obviously this might be just one of the many underlying causes for the very generic symptom of X not accepting connections)

cyv: syncing git and svn

Mon, 26 May 2008 23:16:06 +0100

For codeyard I’m developing cyv, which is a (still quite specific) util (written in Python!) to keep svn and git repo’s in sync. On the serverside, at least. First, let me explain what exactly is synced.

When someone commits to a svn repo, the git repo is synced with git-svn. You can just clone the git repo and git pull instead of having to use git-svn yourself.

When pushing commits to the git repo on a branch that came from the svn repo, the commits are git-svn dcommit-ed. If it fails (svn doesn’t do merges that well), it’ll revert the updates and will receive the successful part of the dcommit on the next post-commit triggered fetch from the svn repo. The user will have to git-pull and fix the commits locally: the git manner.

If pushes don’t involve the svn backed branches, it won’t have any unusual side-effects. This allows for pushing and pulling of topic-branches separate from svn and pushing them, when mature enough, into subversion without ever having to hassle (as much) with git-svn.

An obvious huge advantage is that a git-clone of the git repo is a hell of a lot faster than a git svn clone. A second big advantage is that someone can choose to either use git or svn himself while not mutually excluding the other. This is of special concern to codeyard, where projects should be accessible to everyone: beginners and advanced. If instead we offered fully separated git repositories, the projects that prefer git would become inaccessible for most. And if we wouldn’t offer git repos, people would set them up themselves elsewhere, for they really don’t want to bother themselves with git-svn.

cyv contains some neat features. One I want to highlight is the cyv-layout file, you can place in the root of the svn repo. It tells cyv how the repository is laid out. Eg:

trunk:trunk branches/*: branches/* tags/*: releases/* some-git-branch: some/path/in/the/svn/repo

Another one is a wrapper around git-shell to have per repository permissions for different users depending on their ssh pub key.

To reiterate, cyv is still quite specific to the needs of codeyard. (If you’re a codeyard participant: be patient, it’ll be up mid june). However, if you’re interested, I’ll be glad to hear from you.

Don’t forget /etc/init.d/hibernate-cleanup

Fri, 23 May 2008 00:47:37 +0100

Or it’ll bite you when you switch kernels often and forgot you hibernated on one.

Benchmarking CouchDB (2)

Tue, 15 Apr 2008 13:59:42 +0100

This is a plot of the amount of documents created in a bulk update at the same time against the average amount of documents created per second it yields.

On the App Engine lock-in

Tue, 15 Apr 2008 13:31:13 +0100

Some people are afraid that Google App Engine won’t be a really big success because it’s a lock-in. Give it a month and no doubt there will be a project which allows you to run AppEngine applications on your own servers, which (in my opinion) is even more interesting than AppEngine itself.

Benchmarking CouchDB (1)

Wed, 02 Apr 2008 13:48:31 +0100

I’ve written a small benchmark for couchdb to test it’s document creation performance. A script creates $N$ documents in total using bulk update to create $B$ at the same time with $T$ concurrent threads. The following graph show the time it takes to create an amount of documents against that amount of document for different values of $B$ with $T=1$ .

And for $T=2$ (two concurrent threads. Tested on a dual core machine)

The values of B are 1, 2, 4, 5, 8, 11, 16, 22, 32, 45, 64, 90, 128, 181, 256, 362, 512, 724 and 1024

As you can see, a higher value of $B$ causes the graph to shift to the right which means more $N$ for the same time. Bulk update really does make a difference. Or non-bulk-update really sucks. Also adding threads does help a bit, but not as much as expected.

There are some more interesting graphs to plot ( $B$ against $\overline {N \over \Delta T}$ ). More graphs tomorrow.

(For those interested, the raw data from which these graphs were plotted.)

CouchDB document creation performance

Sun, 30 Mar 2008 00:24:10 +0100

CouchDB is a non-relational database which uses MapReduce inspired views to query data. There are lots of cool things to tell about its design, but I rather want to talk about its performance.

Today I’ve been busy hacking together a little script to import all e-mails of a long e-mail thread into a couchdb database to write views to extract all kinds of statistics. I already imported these e-mails into a MySQL database a few months ago, but was quite disappointed by the (performance) limitations of SQL. The e-mail thread contains over 20,000 messages which weren’t a real problem for MySQL. When importing, however, couchdb was adding them at a rate of only a few dozen per second with a lot of (seek)noise of my HDD.

So I decided to do a simple benchmark. First of, a simple script (ser.py) that adds empty documents sequentially. It’s averaging 16 per second. It occurred to me that couchdb waits for a fsync before sending a response and that asynchronously the performance would be way better. A simple modification to the script later (par.py) it still averaged 16 creations per second.

I guess, for I haven’t yet figured out how to let straces tell me, that it’s the fsync after each object creation which causes the mess. couchdb itself doesn’t write or seek a lot, but my journaling filesystem (XFS) does on a fsync.

Can anyone test it on a different filesystem?

Update Around 17/sec with reiserfs.

Update I had some trouble with the bulk update feature. I switched from svn to the 0.7.2 release. I got about 600/sec, which dropped to a steady-ish 350/sec when using sequential bulkupdates of 100 docs. Two bulk updates in parallel yield about 950/sec initially, dropping to 550/sec after a while. Three parallel updates yield similar performance.

Interrupting a select without a timeout

Tue, 11 Mar 2008 23:24:30 +0100

select is a POSIX syscall which allows you to wait on several different filedescriptors (including sockets) for the event that they won’t block on write; won’t block or read or are in error. This syscall is very convenient when you’re writing a server.

When I want to shutdown an instance of the server, I have to interrupt the select. I have yet to find a satisfying way of doing this. At the moment I create a pair of linked sockets with socketpair. I include one of them to the sockets on which to block until there is data to read in the select call. To interrupt, I simply write some data to the other socket which will cause data to be available on the socket which in turn will interrupt the select.

There must be a more elegant solution.

wlan0_rename

Sat, 08 Mar 2008 14:07:49 +0100

If udev has problems assigning wlan0 to wlan0 (as given by the kernel) and instead lets you end up with an ugly wlan0_rename or alike, you might want to check whether you’ve got stray udev config files in /etc/udev/rules.d/ like 70-persistent-net.rules.

X61s (2)

Fri, 07 Mar 2008 15:37:07 +0100

And here are some pictures of my X61s.

X61s (1)

Thu, 06 Mar 2008 23:35:47 +0100

I’ve been the lucky owner of a Thinkpad X61s for a bit more than a week now. It’s a light and small 12.1 inch notebook. It’s structurally very solid and has got almost the same full size keyboard as my 14.1inch T60. (The enter, tab, shift and alike keys are shortened in width).

The installation of Gentoo went quite smooth. For those interested the xorg.conf, make.conf and kernel .config I use.

Internal mic/speaker and external jacks; video; DRI and AIGLX; USB; PCMCIA; wireless; fingerprint reader and ethernet all seem to work just fine. I haven’t tested the firewire, ssd slot, bluetooth and n-capabilities of the wireless yet.

In comparison with my T60 the volume and the backlight buttons aren’t hardware controlled. Gnome recognizes the volume buttons, but not the backlight ones. I’m still working on those.

ssmtp segfault in sendmail on Gentoo

Tue, 26 Feb 2008 21:39:52 +0100

Disabling the md5sum useflag might fix it.

Thanks to Bram for the tip.

Fosdem (2)

Mon, 25 Feb 2008 00:25:35 +0100

I have just returned from Fosdem 2008. The trip by car to Bruxelles was by Dutch standards quite long, 3 hours. Volker, who did the hotel bookings for the FSFE, was kind enough to also book a room for me and my brother. For the 3-star hotel, which normally would cost you 200,– per night, we only had to pay about 70,– p/n. I did expect group discount, but that it would be so enormous was a surpise. I’m not complaining though :).

After the check-in we went for a quick diner and straight of to the Delirium Café for the “Beer Event”. The café, even though it’s the largest one I’ve ever visited, was barely big enough for the masses of geeks. One could buy a ‘red dot’ from one of the organizers for 20,– euros. With it you can order any beer (one of the 25 on the tap) at the bar without paying until you spend the 20,– euros. This wasn’t checked though. I feel it hard to imagine, and my head felt even harder the next morning, that one would consume more. On that next morning the real magnitude of the conference became apparent. The first talk, at a large (really, large) college room at the Free University of Bruxelles, was attended by at least a thousand if it weren’t two-thousand people.

The talk about Perl 6 was the most interesting for me. I didn’t really like Perl <5 primarily because of having too many ways to do the same in exactly the same way but with a different syntax. I knew that Perl 6 was a total and backwards incompatible redesign of the language, build on top of a generic and good virtual machine called Parrot. Parrot, which I hadn’t given a proper look yet, turned out to be a lot greater than expected. You write support for a new language in Parrot by writing in a subset of Perl 6, which with it’s new Regular Expressions and specializations (tokens: regex without backtracking, etc), was looking very suited for it.
Except for all the new syntactic very very sweet sugar (on which I won’t (yet) elaborate) they added in Perl 6, the greatest one (which is actually more of a Parrot thing) is being able to extend Perl during runtime: writing new parser rules. One application is being able to define ‘!’ as a faculty operator. I’m itching to play with it.

Another very interesting talk was the one about Gallium3D, which is an effort to rework the 3D API on Linux and Xorg for 3D accelerator drivers. It primarily abstracts most of the operations of the video card to being a very specialized language processor and not entirely surprising makes heavy use of the llvm. The abstraction was effective enough to being able to take a driver originally written for Linux, rework it for Gallium3D and actually being able to be run on a Windows version of Gallium. That it runs on Windows really can’t bother that much people. That it also now runs on Cell’s SMP’s, can. The speaker joked that one finally can play 3D games on your PS3 [ on Linux ].

Most of the talks I attended were pretty interesting. Most of the speakers, though, weren’t that great speakers. The thing I missed most with FOSDEM was that there was no easy way to get in touch with people of a certain project. There were booths of a lot of projects, but there the people behind the booth were just standing there to show you stuff in the hallway that was too narrow. There were separate rooms for projects where there were talks given about the project, but there was no real time in between. A few hours nothing but people of the project meeting in a project room would have been great.

Miscellaneously, the OLPC is a great toy; the EEE is even uglier than it looks on the web; the MacBook Air is even thinner than you think and the FSFE people with whom we stayed at the hotel were great. I could tell a lot more, but quite frankly I’m very tired and sleep calls.

Fosdem (1)

Fri, 22 Feb 2008 01:44:21 +0100

Tomorrow I’ll travel the short distance to Bruxelles to visit Fosdem. I’m pretty excited :).

Codeyard 2008

Tue, 12 Feb 2008 20:54:43 +0100

Codeyard is a project of the RU to stimulate high school students for Free and Opensource Software in the Netherlands. Primarily they offer hosting and guidance for setting up projects. They also organize (great!) monthly meetings at the university where participants meet.

To motivate people to join, they organize a yearly award sponsored by Capgemini for the best project. With a friend, Noud, and my brother Bram we participated in their first year (2006) and won the award. This year, I have the honor to join as a member of the jury.

If you’re interested, you can visit one of the “community days”. The next one is this Saterday. There will also be a presentation by the people who cracked the “OV-kaart”, which should be quite interesting.

Evolving the Object Paradigm

Thu, 31 Jan 2008 13:39:02 +0100

Kaja is writing a series of articles on the shortcomings and solutions to the current object paradigm. Very interesting.

Bye bye Reiser4

Wed, 23 Jan 2008 15:44:10 +0100

A few days ago my root partition (formatted Reiser4) corrupted on my notebook. [ the usual IO hangups and nasty output in dmesg ]. Probably due to the usual wear and tear a notebook has to suffer or a faulty suspend cycle causing bogus IO. Something I suffered a few times before and didn’t think it would be a great deal. This time, though, fsck.reiser4 said it was all ok. That meant I was pretty screwed, for I knew it didn’t work correctly.

I lended a USB hdd, booted up to a fallback installation on a separate ext2 partition and tried to copy over everything to the USB hdd. It was quite tricky to copy over as much as I could and remembering the point where it started to crash when reading it. Luckily, I salvaged my whole /home. /var, /bin, /usr/share and a lot of other trees weren’t that lucky.

Formatted to XFS, copied everything I got back to the HDD and copied a Gentoo stage 3 tarball over it. A stage 3 tarball contains a minimal installation to which can be chrooted and then booted and from which the rest of the system can be build: the usual method to install Gentoo. I didn’t lost my world nor /etc/make.conf file. A small script later I got portage re-emerging every package I had installed on the system. Still 200 to go at the moment, but at least I’m now in a partially functioning gnome desktop, which is a lot more usable than TWM (ugly default WM of Xorg).

XFS performs quite well. It’s latency under load is a lot smaller than Reiser4’s. (It’s a pity I haven’t yet come to try the new patches in mm to help Reiser4 a bit with that problem. And also becasuse Reiser4 seems so close to inclusion, reading Andrew’s merge plans). In contrast, XFS sucks at handling a lot of small files compared to Reiser4. This is all just a feeling though. I haven’t tested anything. The most important characteristic of a FS, though, is only apparent after long use: the influence of fragmentation. Having looked around a bit, btrfs seems interesting.

On a sidenote on latency: my mom runs Ubuntu with EXT3 and even though EXT3 sucks in practically every single performance benchmark it has seem to got a superb responsiveness. Ah, 150 packages to go.

Linux 2.6.24-bw-r34

Sun, 20 Jan 2008 01:24:50 +0100

Again an update for my bw-tree. There isn’t a tree that includes reiser4 and TuxOnIce without a lot of other bloat, so I created one myself.

Download the big diff or the seperate patches broken out.

Note I pulled in new patches for reiser4 from the -mm tree against -rc8 which should fix the Reiser4 flush problem a bit.

Update Patched against 2.6.24 stable. New TuxOnIce patched added and the genpatches. Please note that I haven’t tested Reiser4 thoroughly enough on this version.

2008

Tue, 01 Jan 2008 00:00:46 +0100

Happy newyear!

Tickle

Fri, 28 Dec 2007 18:34:15 +0100

Tickle is a small Python serializer like Pickle. It however aims at generating smaller output:

>>> len(tickle('hello')) 7 >>> s = StringIO.StringIO() >>> pickle.dump('hello', s) >>> len(s.getvalue()) 13

Though the difference is and remains quite small, this alone is useful for serialization of small things in the case of for instance RPC. However, usually you already know what kind of data to expect and you don’t really bother about the type information. This can be done by specifying a template:

>>> obj = [] >>> for i in xrange(100): obj.append((i, str(i))) >>> len(tickle(obj)) 629 >>> len(tickle(obj, template=(tuple, \ ((tuple,((int,), (str,))),)*100))) 390

(Instead the *100 an iterator could be constructed, but that would clutter the example even more than it already is.) In comparison:

>>> s = StringIO.StringIO(); pickle.dump(obj, s) >>> len(s.getvalue()) 1680

One big disadvantage of Tickle is speed. Pickle has got a nice C implementation, which is quite fast. Psyco helps a bit but not really enough for really big things. Even more so pickle is a bit smarter: it builds a LUT for instances to avoid duplicate data. However, in the situations where Tickle will be used (by me at least) that isn’t too big of an issue.

You can download tickle.py via gitweb.

Merry Christmas

Mon, 24 Dec 2007 22:59:00 +0100

For happy a christmas!

linux 2.6.24-bw-r12

Thu, 20 Dec 2007 23:07:05 +0100

There isn’t a (stable-ish) tree that includes reiser4 and TuxOnIce (suspend2), so I made one myself, based on 2.6.24~~-rc5~~-rc67.

You can grab it as one big patch, bw-r12-for-2.6.24~~-rc5~~-rc67.diff.bz2, or broken out: bw-r12-for-2.6.24~~-rc5~~-rc67.tar.bz2.

Java crashing on xcb_xlib_unlock.

Mon, 17 Dec 2007 23:34:31 +0100

When Java applications crash on linux with xcb_xlib_unlock: Assertion `c->xlib.lock' failed you should upgrade to libxcb-1.1 and add LIBXCB_ALLOW_SLOPPY_LOCK=1 to the environment.

rebuild-scm-ebuilds

Sat, 08 Dec 2007 19:51:37 +0100

Two very simple ruby scripts (I love ruby!) and a little bash script that searches all ebuilds with 9999 in the name, orders them by dependencies and then emerge -a them. Very useful when you got a lot of 9999 ebuilds.

CaCert.org

Sat, 08 Dec 2007 13:20:46 +0100

CaCert is a Certification Authority that works with a web of trust: people meet and assure (similar to keysigning) eachother. If you’ve been assured by enough people you’ll be able to let your ssl server key be certified by cacert. It’s a lot more secure than other CA’s who just give anyone a certificate who pays enough.

Still a hierarchical system with a CA is flawed. When the CA is compromised, the whole system fails. PGP’s web of trust hasn’t got this weakness.

(Got a nice shiny cacert certified ssl certificate on my webserver now)

Virtual packages in python

Sat, 10 Nov 2007 22:11:07 +0100

When writing an application in python, it can be very convenient to be able to import a module from the top of your module tree. Eg. import myapp.config instead of (python 2.5 only) relative imports: import ....config. To do this one would have to make myapp a package. The normal way to do this is to put your application directory (which now has to be named myapp) somewhere in python’s package search path. This isn’t all to convenient.

The solution: manually set up your package module:

import os import sys import imp def setup_virtual_package(name, path=os.curdir): """ Sets up a package at the given path with a given name """ modulePath = os.path.abspath(path) f, fn, suffix = imp.find_module('__init__', [modulePath]) imp.load_module(name, f, fn, suffix) sys.modules[name].__path__ = [modulePath]

Now import myapp.something works like a charm.

Xkcd

Sat, 03 Nov 2007 13:56:33 +0100

Maybe I’m a bit late, but xkcd is great.

Rome

Mon, 22 Oct 2007 07:39:14 +0100

I’m visiting the city of Rome next week.

Ulrich Drepper on Memory

Sat, 13 Oct 2007 19:54:59 +0100

Kernel hacker Ulrich Drepper has written a lengthy paper on the design and especially performance characteristics of memory of modern consumer hardware and it’s written for programmers. A great read so far (it’s published in parts by LWN):

What every programmer should know about memory by Ulrich Drepper.

Stupid PHP (1) (Strings are faster than Arrays)

Sun, 23 Sep 2007 18:05:14 +0100

When I slowly build a big string out of little bits, the worst thing to do in most languages is to just use string concatenation:

for(something) { str .= little_bit; }

Why? Everytime a little bit is added to str, there must be a new string allocated big enough to contain str and the new little bit. Then the actual str must be copied in. In most languages there are constructs to efficiently build strings like this instead of concatenating. StringBuffer in C#. StringIO in Python.

But no, PHP has to be stupid. There is no nice construct and you’ll end up using concatenation. So, I thought to be smart and make use of PHP array’s and implode. Arrays are here for having elements added and removed all the time so they are properly buffered and should be great at having lots of small elements added. And when I want to pack it all into one big string, I can use PHP’s builtin implode function.

I wanted to try it out and created two scripts: a.php concats a little (10byte) string one million times and b.php appends it to an array and then implodes it. And because I’m also interested in the performance of implode I got a script c.php that’s identical to b.php but doesn’t implode afterwards. These are the results:

a.php (concat)	0.320s
b.php (array append and implode)	0.814s
c.php (array append)	0.732s

Indeed, string concatenation with all its allocation and copying is actually faster than plain simple array appending. PHP is stupid.

Stupid IE (2)

Sun, 02 Sep 2007 14:02:46 +0100

To create a multiple select-box with javascript you need a very ugly hack in IE.

if (navigator.appName.match(/Internet Explorer/)) {
	fsel = document.createElement(\'<SELECT MULTIPLE>\');
} else {
	fsel = document.createElement(\'select\');
	fsel.multiple = true;
}

Stupid IE (1)

Sun, 02 Sep 2007 13:51:19 +0100

if(!Array.prototype.indexOf) {
	Array.prototype.indexOf = function(el) {
		var i = 0;
		for(; i < this.length; i++)
			if(this[i] == el)
				return i;
		return -1;
	};
}

19

Thu, 30 Aug 2007 21:38:22 +0100

Nuff said.

The (or at least a better) Solution to Spam

Tue, 14 Aug 2007 17:45:34 +0100

There is no easy way to distinguish between a human and a spambot. It’s an arms race which we’ll always be behind. I’m talking here about spam in more general—not only on e-mail but also on for instance Wikipedia or on blogposts. Even if we would have a perfect-solution to test whether there is a human behind something, we still have to combat cheap labour in India: real people spamming “by hand”.

I think the solution is a Web of Trust similar to that of PGP. An identity (not necessarily a person) publishes who she trusts (not to be spammy/phishy) or not trusts. Ideally everyone would be in one big web. Only someone who my blog via-via trusts may post.

Obviously one still may gather trust of people over time and enter the web of trust and then start spamming with that identity. However, then that identity will be marked untrusted by people and also the people who initially marked the identity as trusted will be less trusted. Also, there are way more sophisticated measures of establishment in the web/trust to conceive than just being trusted via-via by one identity.

There is no way to prevent spam perfectly, but the amount of work that has to go in to making an identity trusted and established in the web is several orders of magnitude greater than any other protection we have. The big problem: we don’t have such an ubiquitous web of trust yet. (Yes, it’ll be in SINP if I’ll get around to working on it)

Section 202c of the German computer crime laws

Mon, 13 Aug 2007 17:46:33 +0100

This section has come into effect over the weekend. It makes it illegal to create, possess, obtain, provide access to, yield, distribute or otherwise allow access to lots of widespread tools that can be used to breach security. Take for instance nmap.

This law does not only impede our freedom (of speech), research, decrease security and allow for misuse, but more importantly it won’t even stop the real criminals.

Stefan of the Month of PHP Bugs Project writes:

The law does not affect our freedom of speech to report and inform about security vulnerabilities and how to exploit them.

We are just not allowed to create/distribute/use software that could be used as “hacking tools”.

Where would they draw the line between reporting/informing about a vulnerability and how to exploit it and the actual source code to do it. Would pseudocode be illegal? Would literate code be illegal? Also there would be no way for security researchers to try out their work.

What will happen in the worst case if similar laws are accepted in other countries and enforced, is that vendors will rather cover up all vulnerabilities using these laws instead of securing it. That there are lots of ready-to-use exploits is good. It’s a very good incentive for security.

That there will always be a leak in a piece of software that someone will be able to find on his own will not be changed by this law. Also there will be no way to stop the real criminals from creating and distributing tools underground. Now everyone still knows what kind of tools are around and will know what to expect.

bas@fsfe.org

Mon, 06 Aug 2007 01:00:13 +0100

I just got back from Wacken Open Air 2007 (it was great!) and noticed that my bas@fsfe.org account has been activated :).

Wacken

Fri, 27 Jul 2007 02:57:56 +0100

I’ll be up early tomorrow, Saturday, to take the train with lots of friends to the little town of Wacken in Germany. (Actually, there is no train station in Wacken, so we need to hire a cab someway for the last dozen kilometers)

Sunday morning the camping of the Wacken Open Air festival will open. The festival itself will start on Thursday. I hope to be home again the first Sunday of August.

linux 2.6.22-bw-r1

Thu, 12 Jul 2007 16:53:17 +0100

There isn’t a tree that contains reiser4, suspend2 and the gentoo patches — so I created one. This revision adds the -stable patches and the new gentoo patches.

One big patch: bw-r1-for-2.6.22.diff.bz2
Patches broken out: bw-r1-for-2.6.22.tar.bz2

To apply the one big patch, use: bzcat bw-r1-for-2.6.22.diff.bz2 | patch -p1 inside a vanilla 2.6.22.

“Nothing to hide”

Wed, 11 Jul 2007 02:19:23 +0100

In this short essay, written for a symposium in the San Diego Law Review, Professor Daniel Solove examines the “nothing to hide” argument. When asked about government surveillance and data mining, many people respond by declaring: “I’ve got nothing to hide.” According to the “nothing to hide” argument, there is no threat to privacy unless the government uncovers unlawful activity, in which case a person has no legitimate justification to claim that it remain private. The “nothing to hide” argument and its variants are quite prevalent, and thus are worth addressing. In this essay, Solove critiques the “nothing to hide” argument and exposes its faulty underpinnings.

“I’ve Got Nothing to Hide” and Other Misunderstandings of Privacy

Not only is its subject very relevant, the Essay is very well written and a pleasure to read.

Online Cycle Detection in Directed Graphs

Mon, 02 Jul 2007 22:01:45 +0100

A short while ago I came across a quite interesting problem. Design a datastructure (and algorithms) to maintain a Directed Acyclic Graph.

There has to be only one operation: adding a link between two given nodes. This operation must be able to detect and deny any new link that would cause a cycle. For simplicity, nodes are identified by sequential ids starting with 0.

An example:

addLink 0, 1 -> True addLink 1, 2 -> True addLink 2, 0 -> False # Fails because it would create a cycle

addLink 0, 1 -> True addLink 1, 2 -> True addLink 0, 2 -> True # This isn't a real cycle, so it's perfectly fine

It’s rather trivial to create a $\mathcal{O}\left(n+\ell\right)$ (where $n$ is the number of nodes and $\ell$ the number of links). I conjecture there exists a $\mathcal{O}\left(\log n\right)$ algorithm.

Xgl with RandR 1.3 and Ati

Mon, 25 Jun 2007 10:18:26 +0100

You have to add Option "DesktopSetup" "single" to your ati’s Device section in xorg.conf otherwise Xgl will think you’ve got two screens if you’ve got two adapters even if you just use one (which is the case with my Thinkpad) and will miserably fail on the second unattached screen.

Graduated

Fri, 15 Jun 2007 00:20:48 +0100

Today I received a long anticipated phone call from my mentor who took away the doubt and let me know that I graduated for my VWO exams at the Stedelijk Gymnasium Nijmegen, which is a matriculation exam.

With mixed feeling I look back at seven long years. It certainly shouldn’t have took 7 years, nor 6, not even 3. But espacially in my last few years I came to appreciate the tons of very different people with whom I spend the days. A diversity I won’t find in that extend within the students of Maths and Physics which I intend to study next year at the Radboud University in Nijmegen.

For one thing I certainly wouldn’t have want to have missed these last few years. Some say they’re the best of your life. I do really look forward to university, though.

For those interested, I’ve planned an “Examenfeest” with some others in a nice club in Nijmegen. Send me an e-mail if you happen to be in the neighborhood and care to join.

Syslets and other untrusted kernelspace programming

Mon, 11 Jun 2007 01:39:04 +0100

In the wake of creating true async. IO in the linux kernel Ingo worked on so called syslets. A syslet is nothing more than one async system call to chain together a few arbitrary system calls. An example syslet would be:

1. Read from file descriptor 123
2. If 1 returned with error, break
3. Write which was read to file descriptor 321
4. If 3 succeeded jump to 1 otherwise break

This is represented by a few struct’s linked together of which the first is passed on to the kernel with the syslet system call. The systemcall returns practically directly and eventually the application can be notified (or can wait) on the syslet to complete. This could safe an enormous amount of system calls. This means way less context switches, which is very good for performance.

Syslets dawn, in a very primitive manner, kernelspace scripting. But hey, wasn’t kernelside scripting the thing that all linux dev’s dreaded? Wasn’t it Linus who joked that he would be in a mental institute whilst releasing Linux 3 with a VB message pump? Yes, they’re afraid of putting untrusted programs/scripts in kernelspace and they’ll barely acknowledge that syslets is the first step.

The problem with the current full-featured scripting languages is that they are, well, full-features gone wrong: they’re bloated and not really secure. In kernelspace you can’t allow any memory except for the scripts’ own to be accessed, not to mention the restrictions on resources and virtual memory won’t help you there. Most scripting languages weren’t developed with these restrictions in mind. Most languages have got evil functions in dark corners of the standard library that will allow you to do really evil stuff with memory.

As far as I know only .net (thus mono) have got a quite rigorous trust framework build in. .Net is bloated and proprietary and Mono is still (and probably will never be) feature complete though still being very bloated.

A very simple safe language is what we need, of which a compiler daemon is running as a system service, with which untrusted userspace programs can have scripts running in the kernel. I’m tempted to use one of the brainf*ck JIT’s, they’re small enough to thoroughly review :).

A kernelspace interpreter would do to, though, as a PoC.

Linux 2.6.21-bw-r1

Sun, 10 Jun 2007 23:51:48 +0100

There isn’t a tree that contains reiser4, suspend2 and the gentoo patches — so I created one. This revision adds the -stable patches, new gentoo patches and some powertop patches.

One big patch: bw-r1-for-2.6.21.diff.bz2
Patches broken out: bw-r1-for-2.6.21.tar.bz2

To apply the one big patch, use: bzcat bw-r1-for-2.6.21.diff.bz2 | patch -p1 inside a vanilla 2.6.21.

Upgrading wordpress with git

Thu, 07 Jun 2007 22:03:56 +0100

I didn’t like upgrading wodpress much. Everytime I did it, I needed to re-apply all my little tweaks to the new wordpress. It took too much time.

I tried to diff -uNr on the current version I was running and the newer version and then applying the resulting diff to the current version, but it seems wordpress has been backporting changes so I got conflicts, quite a lot of them.

Because I was quite tired of porting my changes, I’ve tried git, the Source Code Managment tool used by the linux kernel, to do it for me:

I did this all in the parent directory of the root of blog.w-nz.com. This folder contains:

htdocs current installation (2.1.2)
2.1.2 the unmodified wordpress
2.2.0 the new wordpress I want to upgrade to

First, I created an empty git repository:

mkdir git; cd git; git init-db; cd ..

Then I copied over the unmodified version of wordpress I was running, and commited them:

cp 2.1.2/* git -R cd git git add * git commit -a -s cd ..

Then I copied over my current installation:

cp htdocs/* git -R git status # lets see what changed

There are lots of files like uploads I want git to ignore, so I edit .gitignore to make git ignore them. There weren’t any files I added though, otherwise I’d had to run git add to let git know.

And let commit my changes:

git commit -a -s

Now, lets go back to the original commit — the clean 2.1.2 wordpress — and start a branch from there:

git checkout HEAD^ # HEAD^ means parent commit of HEAD: the previous commit git checkout -b tmp # create a new branch tmp from here

Now I’m in a branch without my own changes, which was forked from the master branch. Lets apply the new wordpress on this branch:

cd .. cp 2.2.0/* git -R cd git git status # see what changed

git-status showed me that there are a few new files in wordpress 2.2.0, I git-add-ed all of these new files. And then committed it all:

git commit -a -s

Now I’ve got two branches:

master which contains wordpress 2.1.2 with my own changes on top as a commit
tmp which is forked from the wordpress 2.1.2 from the master branch without my own changes but with the 2.2.0 changes on top

What I want to do is to reapply the 2.2.0 changes on top of my current changes’ commit instead of on top of the 2.1.2 commit. To do this, git has a very powerfull util called git-rebase:

git rebase master

This will search down the tree until the point where the current branch (tmp) forked from the target branch (master). Then it will re-apply all commits in between on the latest commit of the target branch.

Just like if I’d use diff/patch I get a merge conflict. git rebase lets me know this and git status shows me which one are these. The one little difference with the diff/patch approach is, that there are way less merge conflicts (git is smarter) and that the merge conflict are way easier to identify and they’re inline in the original files. Not to mention that when I would have fucked up I’d always have a way back.

After I fixed the merge conflict, I git update-index each conflicted file (to tell git it’s resolved) and git rebase --continue-ed.

Now I’ve got my updated wordpress in the git folder. Then I backuped the current, copied over from git and visited wp-admin/upgrade.php and I’m done :).

By the way: “I didn’t say Subversion doesn’t work. Subversion users are just ugly and stupid.” — Linus on this Google tech talk.

Sidenote, I switched from Hashcash to Akismet. Hashcash didn’t work anymore and Akismet theoretically should be the best solution because it isn’t based on security by obscurity.

Elastic tabstops plugins for gedit

Mon, 28 May 2007 19:24:59 +0100

Spaces vs Tabs? Nah: elastic tabstops, now for gedit.

The Future of the Internet

Sun, 06 May 2007 18:07:41 +0100

One of the prominent people behind the current internet discusses the history (telephony, wire oriented), the current (IP, endpoint oriented) and the future (?, data oriented) at google tech talks.

A short synopsis: the internet is having trouble at the moment for it has been designed in a time the problem was different. In these days most of the data is duplicate data, which is a tremendous waste. Also connecting to the internet (getting an address) (and resulting from that keeping everything in sync) is hard. Van suggests and predicts a data oriented internet. A bit like a secure P2P bittorrent network, but instead of on top of IP on the IP level.

It’s a very interesting talk, worth watching.

linux 2.6.21-bw

Sat, 28 Apr 2007 20:11:53 +0100

There aren’t official reiser4 patches for .20 or .21. There are quite a few branches that contain support for reiser4, but these are highly unstable. (for instance the -mm tree). Also because I wanted to give stacked git a shot, I started my own kernel tree:

One big bzip2-ed diff: bw-for-2.6.21.diff.bz2.
bzip2-ed tar with the separate patches: bw-for-2.6.21.tar.bz2.

This release contains reiser4, suspend2, the gentoo patches and a few patches to get everything working together nicely.

Virtual package for your python application

Sat, 21 Apr 2007 19:45:06 +0100

When you’ve got a big python application, you’ll usually split it up in modules. One big annoyance I’ve had is that a module inside a directory cannot (easily) import a module higher up in the tree. Eg: drawers/gtk.py cannot import state/bla.py.

This is usually solved by making the application a package. This allows for import myapp.drawers.gtk from everywhere inside your application. To make it a package though, you need to add the parent directory in the sys.path list. But unfortunately this also includes all other subdirectories of the parent directory as packages.

However, when the package module (eg: myapp) was already loaded, then the path from which myapp was loaded is used to find the submodules (eg: myapp.drawers.gtk) and sys.path isn’t looked at, at all. So, here is the trick:

import sys
import os.path

p = os.path.dirname(__file__)
sys.path.append(os.path.abspath(p+"/.."))
__import__(os.path.basename(p))
sys.path.pop()

Note that this script doesn’t work when directly executed, because the __file__ attribute is only available when loaded as a module.

Save this script as loader.py in the root of your application. import loader from the main script in your app, and you’ll be able to import modules by myapp.a.module, where myapp is the root directory name of your application.

This is How We Catch You Downloading

Sun, 15 Apr 2007 14:38:52 +0100

torrentfreak.com has acquired a document how a british company is tracking down illegal use of P2P: This is How We Catch You Downloading.

Basically they use a modified P2P client which searches for infringing content, download it and if that works and is indeed is the content then they do a whois on your IP and send a infringement notice to your ISP. The best thing is that they claim that this provides enough proof that you really are infringing.

They probably never heard about botnets.

Ruby and parenthesis

Wed, 11 Apr 2007 18:06:30 +0100

Do you prefer

p = Pathaname.new('.').realpath

p = (Pathname.new '.').realpath?

C&C 3 on Linux

Sun, 01 Apr 2007 15:20:10 +0100

I’ve got Command and Conquer 3: Tiberium Wars running on my Gentoo Linux installation with wine 0.9.34 by following the instructions here. I had to first install it on windows though, and copy the folder for the installer didn’t work, even with Crossover Office.

Except for (very glitchly) video and sometimes a crash everything seems to run. (Didn’t try multiplayer yet though). I had to put all quality settings to lowest, which makes me wonder whether that is my radeon X1400 being not so good as I expected or wine just being slow in emulating Direct3D.

The Filesystem Failed. Part I: introduction

Sat, 03 Mar 2007 19:12:12 +0100

The Filesystem (I’ll consider the Linux VFS as an example) has failed:

Database storage is implemented on top of the Filesystem, because the Filesystem is incapable of serving the needs of relational storage.
Metadata is stored inside files in many different formats which can only be guessed by clumsy ‘magic’ in the headers. This forces many media player and desktop search application to duplicate tag information in their own databases. Each of them has only limited support for each of the many different formats.
More and more device and service abstractions are moving from the Filesystem to seperate namespaces, because the Filesystem’s API is inadequate. Take for instance oss which used /dev/dsp, whereas alsa uses its own. Many new abstractions don’t even go near the filesystem anymore, for instance kevents, futexes, networking, dbus and hal.
Small files are stored in (compressed) packs and archives because the Filesystem can’t handle them. This happens with for instance your mailbox.

The problem comes down to fragmentation of data and metadata in too many namespaces because the Filesystem doesn’t seem to be an adequate one.

In a series of posts I’ll look at the possibilities to create one unified filesystem.

Watermarking media

Thu, 01 Mar 2007 17:01:01 +0100

It seems the new trend of the music industry against piracy is watermarking movies/audio/etc.

Content is water-marked by adding very small (unnoticeable) changes that could store something like a rsa based certificate to identify a given audiotrack.

Originally I thought they’d use it to track down the source of an illegal download. It sounds illogical to me because it’s hard to keep watermarks when format is changed (mp3, ogg and others really do mess up slight unnoticeable differences because otherwise they wouldn’t compress as good). And when someone has got two versions of the same audiotrack one can compare them and find out how something is watermarked.

Maybe the scheme of the industry isn’t that stupid, but the other way around (and a lot more evil). Maybe just sue everyone who hasn’t got a watermark on their movies or mp3.

Luckily a Fair Use bill was passed which they say (haven’t checked) allows fair-use conversion of format of media.

Reiser4 for linux 2.6.20

Sun, 25 Feb 2007 23:27:37 +0100

reiser4-for-2.6.20-0.patch.gz

Big fat warning, when hibernating using suspend2 it seems to corrupt some memory here and there. So don’t hibernate.

Internal Microphone

Sun, 11 Feb 2007 11:53:07 +0100

By accident, I discovered that my thinkpad has got an internal microphone. Now I can get rid of that stupid headset I have to use for skype.

(Yeah, I know macBook’s have got a camera too)

4 inches

Thu, 08 Feb 2007 21:38:30 +0100

Today several schools (including miine) stopped early because of ‘heavy’ snowfall. 10cm of snow, that are about 4 inches.

It’s ridiculous.

Codeyard Community Day

Sat, 03 Feb 2007 19:22:46 +0100

It was fun :).

Photo’s

ati-drivers-8.33.6 for Gentoo

Fri, 02 Feb 2007 21:29:29 +0100

This is a slightly adjusted 25.3 ebuild that will give you the 8.33.6 ati-drivers for Gentoo. Yes, it’s dirty. They aren’t in the main tree yet because they are considered broken, although it works just fine for me.

Download: ati-drivers-overlay-8.33.6.tar.bz2

Extract them to an overlay.

Update, the 8.33.6 drivers are in the mainline tree now, so you should use those instead of mine.

Intel’s free literature

Thu, 01 Feb 2007 20:14:06 +0100

I noticed that you are able to order hard-copy’s of Intel’s books on the IA-32, 64 and Itanium architecture. So I did.

Today, a few days later, I received them. They’re very informative thorough and well structured. And, in contrary to other professional literature, you are able to read it without crunching your brains on one jargon-filled sentence. Basically free, dense and easily read books and great references.

So, a big thank you to Intel for this nice free service.

By the way, Itanium seems to really rock. (256 registers, compiler branch prediction, 8-superscalar, etc)

Also, I’ve got no spare room anymore on my desk. 🙂

Python Imaging Library

Tue, 30 Jan 2007 00:54:39 +0100

PIL is great. In less than a thousand lines one can compute the polygons from regions, assess the best possible subimage layout and create all the subimages.

Hashcash 3.2

Wed, 03 Jan 2007 01:34:32 +0100

I upgraded to hashcash 3.2. I hope that will stop the new wave of spam I had on this blog. When you’re upgrading yourself, note that now wp-hashcash resides in its own subdirectory in the plugins folder, I didn’t notice that and couldn’t find what was wrong until I noticed the ‘/wp-hashcash/’ bit in the source.

By the way, happy new year!

Rotating Beryl cube using HDAPS on a Thinkpad

Wed, 27 Dec 2006 01:35:28 +0100

I hacked together this little ruby script to read the G-measuring device in the Thinkpad’s harddisk (HDAPS) and rotate the desktop cube of Beryl when sudden movement occurs. A bit like what has been done for the MacBook already.

http://w-nz.com/~darkshines/projects/rtollina.rb

GPL, obviously. It’s not really perfect yet. I’ll try to improve it tomorrow and add support for compiz and maybe even make a little widget out of it.

It’s based on code by Fer, which is for Compiz instead of Beryl.

XGL take 2

Wed, 27 Dec 2006 00:01:22 +0100

I tried to get AIGLX to work on my Thinkpad yesterday. AIGLX is an API similar to XGL, but is a better implementation. Unfortunately AIGLX requires implementation by the video-card driver (which is good because it allows more performance), but the proprietary drivers of ati still doesn’t support it. (nvidia’s do, note to myself: get nvidia next time).

So I had to revert to XGL. A lot has changed since the last time I installed XGL. Other gentoo overlays, other windows managers, other hacks.

I used the gentoo-xeffects overlay to get Xgl.

Installing Xgl was a lot more straight forward and less of a problem than it used to be. An emerge and writing a simple startxgl script was enough.

The compiz-quinstorm patchset seems to have evolved to a proper branch of the compositing window manager, now called Beryl. It also includes a nice settings manager now.

Even hibernation and suspend finally seem to work. 🙂

A nice screenshot:

One thing left to do: integrate Xgl into xdm.

Avoiding multiple lock dead-locks by memory addresses

Mon, 25 Dec 2006 20:18:43 +0100

Sometimes you need to lock several resources. If you don’t take great care you are likely to get yourself into dead-locks. A simple example with just two lockable resources A and B:

function foo {
 lock A;
 lock B;
 // Do something with A and B
 unlock A;
 unlock B;
}

function bar {
 lock B;
 lock A;
 // Do something different with A and B
 unlock B;
 unlock A;
}

When foo and bar are called at about the same time then there is the change that foo locks A and bar locks B which will make bar wait on foo‘s lock on A and vice versa.

Solution: fixed order on memory address
The simplest way to get rid of the deadlock is to always try to acquire a lock on A before on B. A generic solution would be to always lock the resource with the lowest memory address first.

This only works when memory addresses are fixed or that there is an otherwise fixed order-able property.

(auto)mounting removable media as user

Wed, 13 Dec 2006 18:46:26 +0100

I’ve always been bothered by the fact that you need to be root to mount anything (like an usb stick). It can be solved a bit by setting up udev rules and putting a specific device in /etc/fstab, but that only works for that single usb stick. Pretty annoying.

Googling only gives you stupid and silly solution (like allowing users to mount /dev/sd[a-z] — security risk).

Luckily I’ve recently been pointed to ivman, which is an automounter. It automatically mounts removable media for you in /media.

I looked at the internals of ivman, and noticed that it uses pmount, which is a wrapper around mount which allows users to mount removable media on a /media folder. Great!

Btw, you need to be in the plugdev group to use pmount.

Update It seems that gnome-mount also works fine when you’re in the plugdev group. Gnome-mount does about the same as pmount with the advantage that gnome-mount has got the nice gui integration everywhere in gnome.

Reiser4 for 2.6.19

Sat, 09 Dec 2006 16:23:23 +0100

I found this patch: reiser4-for-2.6.19.patch.gz.

Disappearing backgrounds in IE.

Fri, 08 Dec 2006 18:33:14 +0100

In Internet Explorer in some cases backgrounds disappear seemingly randomly. The fix? Add the position: relative; style entry to the concerned div tags.

Long mySQL keys

Sun, 03 Dec 2006 23:58:03 +0100

Instead of limiting your long key (for instance a path) to approximately 800 characters (on mySQL with generic UTF8), you can hash your key and store the hash as index.

The drawback is that you need to use a quite good hash function to avoid duplicates if you want to use it for a unique or primary key. These good hash functions tend to require some computing time.

md5(microtime())

Sun, 03 Dec 2006 23:52:50 +0100

Don’t use md5(microtime()). You might think it’s more secure than md5(rand()), but it isn’t.

With a decent amount of tries and a method of syncing (like a clock on your website) one can predict the result of microtime() to the millisecond. This only leaves about a 1000 different possible return values for microtime() to be guessed. That isn’t safe.

Just stick with md5(rand()), and if you’re lucky and rand() is backed by /dev/random you won’t even need the md5(). In both cases it will be quite a lot more secure than using microtime().

A loud present

Mon, 27 Nov 2006 21:10:37 +0100

Today, a friend of mine, Gerben, gave me a birthday present (a tad late). Not the usual beverages (turned 18 ;)) (usual, but still very well appreciated :)), but a home-made portable speaker with built-in amplifier. Here seen making more sound than the internal speaker of my Thinkpad:

It’s a nifty little device: it’s powered by either an internal battery or an external power source.

It contains a voltage converter and a rectifier which allows a variable AC power source to be connected. This has been added to be able to connect the speaker to the electrical generator of a bicycle. Why? The previous version of this speaker was taken on the bicycle trip with a few friends to the Hertog Jan Brewery in Arcen (~70km) which was only powered by a battery (which tends to be exhausted).

Even better, when set into external source mode it recharges the battery. And because of the voltage converter, the sound doesn’t get louder when you are cycling faster, but the quality of the sound gets better!

I’ve had quite some fun dissecting the little device. Thanks again Gerben!

P’tit-doux

Mon, 20 Nov 2006 23:14:02 +0100

For my dutch readers, a great analogy between microsoft and a fictional P’tit-doux.

http://www.vromans.org/johan/articles/ptitdoux.html

Simple Branch Prediction Analysis

Sun, 19 Nov 2006 16:08:27 +0100

This paper outlines simple branch prediction analysis attack against the RSA decryption algorithm.

At the core of RSA decryption is a loop over all bits of the secret key number d. When the bit 1 there is other code executed than when the bit is 0. The CPU branches on a different bit.

A spy process can be run on the CPU which measures the branch cache of the CPU by flooding the cache with branches and measuring the time it takes. When the sequentially running secret process doing RSA decryption makes a different branch (1 instead of 0) it can be noticed in a change of execution time on the spy process’s branches.

In this way quite a lot of secret bits can be derived.

There are some clear buts:

You must be able to insert a spy process on the computer itself and it should know exactly when the RSA process runs.
To attain clear readings, there shouldn’t be other processes claiming too much CPU time.
The spy and CPU process should run on the same physical processor and preferably at the same time (dual core)

An easy fix would be to allocate a whole processor for the RSA decryption time, so no process can spy. Another option would be to add noise in the Branch Prediction Buffer, but that would result in a performance loss.

E-Mail server issues

Thu, 09 Nov 2006 23:12:42 +0100

I had some problems with a recent mysql update — it broke some packages including qmail-send, which caused quite a lot of mail to be dropped the last few days. Sorry for that.

Update: mail was fortunately not dropped :), it’s still in the qmail queue, it just hasn’t been delivered yet. So there’ll only be a delay.

RTFM, where?

Fri, 27 Oct 2006 08:24:03 +0100

Recently a buddy on msn asked me a linux question, he just started linux so he had some problems getting stuff done.

He downloaded an installer, he said, a .run, but he doesn’t know how to execute it. He tried googling for it and asking on forums, but didn’t get an answer, so he asked me.

I solved his problem, but I still wondered, where you can find that you need to put ‘./’ in front of a file in bash to execute it and where can you find that you probably need to chmod +x the file too if you downloaded it from somewhere, if you are a total newcomer to linux.

The bash tutorial would’ve probably solved it, but do you know that that thing in which you are typing actually is a separate program? Probably not.

I basically learned all this trivial stuff while following the gentoo installation manual, but I guess that’s a bit too much to ask from each new linux user. There should be a good linux introduction that explains this trivial stuff somewhere to which I can redirect new users. Anyone knows one?

Reiser4 on 2.6.18

Fri, 27 Oct 2006 00:41:31 +0100

Try this patch:
http://vipernicus.evolution-mission.org/…

Fallacies (1)

Tue, 17 Oct 2006 13:38:11 +0100

I just read this on slashdot:

statistically significant relationship between autism rates and television watching

Offcourse this relation was explained as televisions causing children to become autistic. This is a very annoying fallacy. Why? Because just maybe children with autism like to watch television more than other children.

If there is a relation, and if it’s not coincidental, there are still two ways to explain it as a cause and effect relation — you cant just blindly assume that one of the two is the cause.

Same goes with relations like:

“Children that play violent games are more likely to become criminals” or “Children that are to become criminals are more likely to play games”

Stupid shout culture.

Good and bad CAPTCHA`s

Sun, 15 Oct 2006 12:02:25 +0100

CAPTCHA’s are images which content needs to be written into a textbox by a user to make sure it’s a human instead of some computer script. This is an example of a good CAPTCHA of yahoo:

This is an example of a really bad CAPTCHA:

What makes a CAPTCHA good, as in hard to solve by a computer? Lets look how a computer would solve a CAPTCHA, there basically are 3 parts:

Remove rubish background.
Remove rubish lines and partition the image into sections, with in each section a letter.
Recognize the letter with a neural network.

Part 1 is very easy in most cases — just filter everything out that isn’t black and isn’t a glyph-ic curve. It gets a bit more difficult if the font and background colors are random, but usually it’s simple to distinguish between a glyph (small, curve-ish, solid color) and a background (solid, usually gradients). Software is way better in this step than humans.

Part 2 is the most difficult part for software. Distorting fonts isn’t that much of a problem, as long as the software can recognize seperate curve-blobs. The real problem comes in when there are red-hering-curves or when several glyphs are connected with curves like in the yahoo CAPTCHA. When the captcha uses undistorted fixed aligned fonts, it isn’t a problem even if you add glyph connecting curves like in the dotmac CAPTCHA, because you only need to add a little bit of code to recognize an authentic glyph curve (small, thin) and then you can predict the position of the other curves. Humans are better in this step than computers.

Part 3 is a bit tedious for software, but usually easier for specifically trained neural networks than for humans.

How to make a good CAPTCHA:

Do not add stupid background or differently coloured polygons, they won’t work at all — they will only confuse the human.
Do not use a fixed font, size or alignment. Rotate the font a bit, transform it a bit and, most importantly, place them unpredictably.
Add glyph like curves the intersect preferably only two glyphs to make them less recognizable. Take care though that you don’t make them too font like, because that’ll prevent the human from recognizing. These extra intersecting curves make CAPTCHA’s strong, because it prevents proper partitioning.
Don’t use strange fonts that might seem hard to see, but are easy recognizable. For instance, dotted fonts are very easy to locate when everything else are solid curves.

Update: nice blogpost on breaking captcha’s: http://www.brains-n-brawn.com/default.aspx?vDir=aicaptcha

T60 (2)

Sat, 07 Oct 2006 23:57:57 +0100

After 2 days installing gentoo on my great T60 I’ve accomplished quite a lot so far:

WiFi works. 4 seperate components (userland, firmware, kernel module and yet another kernel module) increase complexity a bit. Although the fifth component caused trouble — I forgot to tweak conf.d/net
Radeon X1400 and video card worked almost directly after using aticonf — nice! When I upgraded to xorg-server-1.1, though, I got a few crashes when closing X in a not-so-nice manner.
Using the synaptics drivers my touchpad now support drag’n’drop, right and middle mouse click and scrolling emulation. Still got to find a way to bind the thinkpad shortcut to disable/enable the point or pad.
Finally got the DVD rewriter to work. It seems I overlooked one little stupid generic E-IDE option in the kernel config.
Bluetooth and infrared should work — haven’t got the gadgets to try them out
Never even saw a PCCard, so didn’t even bother to look into ti
Software suspend (hibernation) doens’t work yet, but I definitely be working on that.

T60

Thu, 05 Oct 2006 20:47:39 +0100

I just received the Thinkpad T60 notebook I ordered last week from nofost.de. And I’m thrilled!

I’ve been busy all day installing gentoo on the beauty (and removing the ugly windows sticker).

Some nice pictures:

New google reader

Fri, 29 Sep 2006 18:24:10 +0100

Google Reader has been updated. It resembles bloglines a bit more concerning the tree view. It has still got the usual items by popularity/date view as the “all items” view. It’s looks nicer and seems to work pretty well.

One thing still bothers me: the load times. Sometimes it’s just slow (same with gmail). It’s annoying.

18

Wed, 30 Aug 2006 09:37:20 +0100

My brother and I turned 18 today!

Feel free to visit us today and have some coffee and a slice of pie.

Send a mail to x@y, or call 024 6421829 (in holland). Where x=bas.westerbaan and y=gmail.com

Arcen

Fri, 18 Aug 2006 01:14:36 +0100

Last tuesday I cycled with some friends (about 11) to Arcen to visit the Hertog Jan brewery. The ~75km of cycling went pretty well — way better than expected. We stayed on a 4 star camping for the night and cycled back the next day.

Needless to say, this was a great experience and we all had a lot of fun.

Some photos were made, I’ll post them as soon as I get them.

WP 2.0.4

Fri, 18 Aug 2006 01:03:47 +0100

Upgraded to WordPress 2.0.4.

As usual, slam the F5 button (or similar) once (or twice on stupid browsers) to invalidate the old stylesheets and images in the cache. If your browser is really stupid (*cough* IE *cough*), you might need to purge the whole cache manually, but that isn’t my concern ;-).

bcmap 0.1

Sun, 13 Aug 2006 18:41:33 +0100

bcmap is a simply program to be linked against klibc to map Broadcom RAIDCORE disks to more sensible and stable device nodes.

Currently the bcraid drivers assign the next available sd node to a detected disk, which is very inconvenient when you have a usb stick inserted on boot which pushes your root disk from sda to sdb.

bcmap ensures that your first array will always be /dev/bca, which is a nice thing.

>>> bcmap-0.1.tar.bz2 (source code)

Licensed under the (holy) GNU GPL v.2

bcraid drivers part two

Sat, 05 Aug 2006 14:31:12 +0100

Here’s a small patch that tweak the broadcom raid drivers for the bc4000 series to run on the latest kernels:

http://w-nz.com/~darkshines/projects/bcraid-2.1.0-bw1.patch

Note that it is everything but complete. Some other pointers getting the bcraid to work:

Use sparse memory or conitguous memory instead of incontiguous memory in the memory layout configuration in the kernel. (CONFIG_SPARSEMEM=y)
Enable the magic sysrq key (CONFIG_MAGIC_SYSRQ=y)

At the moment I’m trying to get the bcraid module working in an initrd to allow the root partition on the raid drivers.

Oops

Thu, 03 Aug 2006 07:49:42 +0100

The mysql wasn’t enabled whilest updating dev-lang/php. This caused a big part of my website to fail to operate any longer after the update. I enabled and re-emerged and everything works fine again. Sorry ’bout the downtime.

The rm -r / typo

Thu, 27 Jul 2006 18:01:21 +0100

Today I accidently made a (yes, very stupid) typo in a root console:

rm -r /

I noticed the typo almost directly, but rm managed to wipe out my /bin and started removing parts of /boot. This situation wasn’t very helpful for the stability of my system, as you might understand.

For the windows user: it’s a bit like deleting half of all executables in the windows folder.

One key difference: when running linux, you can fix it easily. I booted a livecd, mounted my system, copied the /bin from a stage3 tarball to my root partition and rebooted.

And it’s working again! There were some complaints about a libproc version mismatch with the binaries, but that’ll be easily solved by a emerge -e system.

You just got to love linux. (and other nixes for that matter)

SINP: Push versus Pull

Thu, 27 Jul 2006 15:10:43 +0100

SINP is pull based — I give my SINP address to someone, and he will pull the information he wants from my SINP server.

Our competitor SXIP is push based. When I use my SXIP identity I push all information I want to provide to the service — there doesn’t even have to be a SXIP server (‘homesite’).

Push has got certain advantages over pull:

Pull is complexer: you need more traffic and more complicated traffic. Push is simpler.
You most likely need a seperate server for pull (you need one with SINP at least), this makes you rely on your SINP server. You don’t need a real one for push.

But pull too got advantages:

You don’t need to actively give your information. When I’m offline someone can still pull information from my SINP identity.
Pull doesn’t require the actual information to go via your computer. If someone requests my creditcard number and I allow it, it won’t be redirected through the computer I’m using, which is safer.

Tilda

Sat, 22 Jul 2006 18:59:12 +0100

Tilda is a drop-down terminal for linux. Press the assigned hotkey and the terminal will dropdown and gains focus, press it once again and it’ll dropout. And even better, the terminal isn’t closed, just hidden.

This is great for development. I write some code, press Ctrl+S to save, Alt+Q for the terminal and make. If there is a bug I can Alt+Q and return to my code, and if I didn’t look closely enough I can press Alt+Q again to see the output again in the terminal.

SINP Certificates and redirects

Wed, 19 Jul 2006 13:24:59 +0100

Tuesday the 11th, we (Noud, Bram and I) had a meeting with some guys of the Security of Systems Group at the Radboud University. We discussed the security of the current SINP protocol. There hasn’t been a hard verdict on whether SINP is secure, because the SINP specification leaves a lot of details to implementations and SINP doesn’t make hard claims on its security yet (which can be either proved or disproved).

The meeting has yielded two new additions to SINP: document certificates¹ and redirects.

First of, SINP document certificates. At the moment you can’t trust the information in a SINP document. I can forge my SINP document and claim that I’m Belgium, which I am not. To allow some trust which some people and services care about, we’ll allow certificates in your identity document. Basically you let someone sign a part of your SINP document and include that certificate.

Your goverment could sign your name in your SINP document for instance and you’d add that certificate into your document, which could be required by some services. These certificates are a bit tricky though to design, because they do need to be secure and they need to be a bit more dynamic than your usual certificate because of the way SINP documents are queried.

A second problem we encountered during the meeting was how to be able to trust your SINP server. I (and other tech savvy people) can set up their own SINP server, which we can fully trust because we set it up ourselves. Not so tech savvy people can’t — they need to rely on existing SINP servers. The problem is whether we can trust those servers with our secrets.

Cees (if I recall correctly) coined the idea that some of your secrets are already on the internet. If you’ve got a VISA creditcard number, then VISA obviously has that creditcard number, and you trust them with it. What if VISA would store the part of your SINP identity document with your creditcardnumber on its own SINP Server?

Basically I go to a big SINP provider (which I don’t trust), I create a SINP identity and put in my SINP document that you can find my creditcard number under the SINP identity bas@visa.com. This act of redirecting clients to other SINP identities is called a SINP Redirect. SINP Redirects could also proof very usefull when you change your SINP server. The only thing you’d have to do is to set up a SINP redirect in your old identity document to your new identitiy document.

Both SINP Certificates and SINP Redirects will require a lot of though to implement cleanly and simple, which is tricky.

Any thoughts would be welcome.

1: Actually, this certificates aren’t new, Bram came up with the idea quite a while ago.

Typed Yields: Non fatal exceptions

Sun, 16 Jul 2006 17:50:23 +0100

Wouldn’t it be nice to have:

begin { &nsbrLoadTheDatabase("foo.bar"); } rescue (Exception e) { print "Fatal exception happened: ", e } on (Warning w) { print "Database Warning: ", w } on (Message m) { print "Database Message: ", m }

The rescue (Exception e) should be familiar with everyone — something failed, maybe the database file was corrupted very bad, and raised an exception and the rescue block will be executed.

But what if the database has a small error, or something is only a little bit out of place. You wouldn’t want to just ignore it, but warn about it. Usually one would implement a ‘Logger’ class to which a function can log certain events, but that is ugly and inconvenient.

Enter non fatal exceptions. Basically there would be two ways to raise an exception, fatal like we all know it, and non fatal. When the on block for a non fatal exception has been executed, control will be returned to the function in which the raise was called.

This is done in about the same ways as a lot of languages implement yield. But this time the handling code depends on the type of the yielded object.

As far as Kaja and I concern this will be a feature of Paradox.

Thanks go to Bram for the idea.

sinp.rb

Mon, 03 Jul 2006 00:30:49 +0100

irb> require 'sinp' irb> c = SINP::Client.new nil, nil, [:http] irb> c.getPublicDocument('Kristy@w-nz.com').write <requested version='2'> <sinp-id> <name><nick>kristy</nick></name> <address type='email'>kbuiter@hotmail.com</address> <uri>hotmail.com</uri> </sinp-id> </requested>

As you can see, I’ve almost finished the implementation of a Ruby SINP client — I only got to finish SINP Negotiation.

SINP

Sat, 01 Jul 2006 12:51:59 +0100

SINP is a protocol based on HTTP(S) and XML that provides you with an identity on the web. You register a so called SINP Identity on a SINP Server of your choice. To address a certain identity, we use an email like notation: bas@w-nz.com is the SINP Identity of the user bas on the SINP Server w-nz.com.

The first big feature of SINP is authentication. If someone claims to be bram@w-nz.com, I can check that by asking w-nz.com to check it. I’ll redirect that guy to w-nz.com to let him be checked by his proclaimed SINP Server. If he really is bram@w-nz.com, he’ll have a nice session cookie for w-nz.com and w-nz.com will check that. After that w-nz.com will redirect him back and I’ll ask w-nz.com whether he succeeded.

One major application of this authentication is that someone who posts a blog comment as noud@w-nz.com, really is/are the same guy(s) that posted before as noud@w-nz.com, for they are allowed by w-nz.com.

The second big feature of SINP is that each identity comes bundled with a XML document, which can store information about the owner like his name, email address, date of birth, etc. The SINP Server stores this document. The identity owner, the guy who owns the identity, can pick an access policy for each little bit of information in this document. You might want to share your real address only to those who you’ve explicitly allowed. Everyone can see the parts of your document you’ve allowed everyone access to. This is the one for bas@w-nz.com.

To get specific parts instead of the whole thing, or get to stuff you’ve limited access to, one needs to use SINP Negotiation. To get some specific information from noud@w-nz.com, I ask w-nz.com for this specific information, in the form of a few xpaths. Along with the xpaths I can send my own SINP address, bas@w-nz.com. The server will respond on each request according to the access policy which the SINP Server has set. There are several possibilities:

Ok, the requested information will be included in the response.
Nope, you’re denied access to that.
Not found, that stuff isn’t in this document.
You’ll have to ask Noud. Basically you’ll have to redirect Noud to the server, where he will be authenticated and after that he can decide whether to allow you access to it, and you can try again lateron.
If you’re bas@w-nz.com, you can see it. You’ll have to authenticate, though. This is done via sinp authentication as described before.

Another big feature of SINP is versioning, which allows caching. The version of a specific bit of information is send back on each response in negotiation. In a negotation request, I can specify the current version I already have. In case that specific part of the document hasn’t updated, the SINP server will let me know, instead of sending the whole thing.

One advantage of caching and negotation is that information can be kept synchronized with your document when it updates. A blog, on which you’ve posted a comment, might periodically check whether the information it retreived from your SINP document has changed. This can be done cheaply with negotiation and versioning.

SINP is easy to implement, it is quite simple. It also is portable, it uses widely supported technologies like XML and HTTP(S) as its base.

SINP is under development, but you can already (and really should) take a look to:

The latest specification
An almost complete SINP Server implementation in PHP. Notice that it’s a subversion repository.
A proof of concept SINP client written in PHP. (subversion over http!)
The w-nz.com SINP server, you can register your own SINP identity, now!
A hacked wordpress blog to allow you to post comments with your SINP address, as a proof of concept.
The kind people who hosted the project and who even awarded us a price for the project :-).

SINP is based on things I’ve seen floating around on the web, for instance Zef`s SPTP.

At the moment of writing we’re developing a PHP client, a Python client and continueing development of the PHP server. You’re welcome to participate!

We hope you like it, comments or any other forms of participation would be very welcome.

Bas, Bram and Noud.

SINP & Codeyard

Sat, 01 Jul 2006 00:58:38 +0100

SINP, a protocol that provides you with an identity that allows authentication and negotation of information linked with the identity as an identity document (basically the whole functionality put into a way too short sentence), which was deviced by Noud, Bram and me, has won the Capgemini Opensource Award! Lots of rejoice! You will hear more about this :-).

On IE7

Mon, 26 Jun 2006 19:48:33 +0100

I’ve had to test some website I designed on IE (VMWare offcourse) and it went really bad. IE 6 sucks, that’s for sure. It was a blow in the face for me, because everything went so terribly smoothly, escpecially the CSS.

I was wondering whether it would be as bad in IE 7 as in 6, so I installed it (Genuine Advantage doesn’t work by the way).

I was amazed by the amount of improvement they made.

The website was a mess in IE6: backgrounds wouldn’t show up; no attribute selectors on CSS; blocks were positioned in absolute ridiculous locations. Whereas in IE7 it showed up pretty nicely compared to 6. It wasn’t perfect though, there were quite a few glitches in positioning (go IE box model!) and font sizes, but that’s managable.

I wonder whether IE7 took so long and why it still isn’t as bug-less and feature-complete as almost every other big browser. If microsoft wanted to make a perfect browser, they could do it easily! They’ve got more than enough very talented people and even more than enough money to hire/bribe any expert they’d want.

Maybe they’re afraid of big AJAX applications. I bet gmail took a pretty big market share of outlook. At least, that is what other people are thinking. They may be right, but google wasn’t stopped by the bugs in IE, google doesn’t use nice and semantically correct HTML (and it is the correct stuff that usually doesn’t work on IE), they just use the stuff that works anyway.

More likely they don’t care. They force website devs already to create sites that show up properly in IE, just because of its market share. Only website creators feel the pain of the buggy IE engine, not the users of the browser. Besides that it could be a pain for them to create a whole new IE, which needs to be backwards compatible with the old IE. A lot of people would be angry if their site doesn’t show up properly anymore in the new IE — and these people can’t be persuaded with the nice semantics ideology.

A little bit more pressure by alternative browsers could force microsoft to release something more profound than IE7. Although IE7 is an improvement, they could have done better.

DDOS on Hash Tables (Self Balancing Hash Tables)

Sun, 25 Jun 2006 17:08:01 +0100

Hash Tables are widely used in server software. A malicious user can easily forge keys in the communication with the server that will result in hashes from the keys so that they will end up in the same bucket. This will force the hashtable to grow each time this bucket is filled.

In a worst case implementation, this will result in an enormous amount of empty buckets and only one full bucket. Even if the implementation is smart and will therefore only will grow the targetted bucket, the hash table will fail its use. The hash table is meant to make lookups for a key fast by distributing the pairs in buckets, this effect is gone when all pairs are located in one bucket, which will cause linear time look-ups.

One malicious user won’t be able to crash a server, but a DDOS attack with several users would be tremendously more efficient when targeting this weakness in hash tables.

A simple solution would be to randomly pick a salt for the hash function on each new instance of a hash table. Basically we prefix each key with a random salt before we hash it. The malicious user might know the hash function, but he has to guess the salt, which neglects the effect.

Additionally logic could be added to detect an unbalanced hash table and switch the salt on when the table is unbalanced.

This could also be usefull to balance hash tables when there is no malicious user, but the keys are possibly unfortunate.

Gentoo and BC4000 series RAID cards

Thu, 08 Jun 2006 20:54:52 +0100

Raidcore hasn’t released drivers that support gentoo linux, only precompiled drivers for the other big distro’s.

I’ve managed to get the drivers for the bc4000 series to work on Gentoo Linux 2006.0 on linux-2.6.16-gentoo-r9 for the bc4852 card.

You need the driver SDK, which contains the source of the precompiled drivers they distribute.

I had to tweak some kernel configurations to get it to work — first of I had to turn the ‘sysrq magic key’, although it seems easy to remove all sysrq usage in the drivers itself iif you aren’t comfortable with sysrq.

I had to switch from ‘discontiguous memory’ to ‘sparse memory’ layout — the driver relies on the availability of ‘mem_map’.

In the Makefile of the bcraid there’s an issue with the PLATFORM define. It should be defines with ‘uname -m’ instead — otherwise it won’t link in the binary blob.

The bccfg init script they provide seems terribly useless, adding ‘bccfg’, and ‘bcraid’ for that matter, to /etc/modules.autoload.d/kernel-2.6 does just as fine, but cleaner.

Initially it seems that everything works fine, although I’m a bit worried about a lot of errors related to the bccfg device nodes in dmesg.

Downtime 1st June

Sat, 27 May 2006 19:57:57 +0100

The vserver will be upgraded the first of June. There will be some downtime while copying the FS to a larger partition.

Update: The server has been upgraded. A little bit later though than expected.

GMail inbox size plotted

Sat, 27 May 2006 15:38:57 +0100

Haochi has plotted the inbox size of gmail over time, interesting.

e

Thu, 25 May 2006 22:39:27 +0100

\displaystyle e=\lim_{n\to\infty} \left(1+\frac{1}{n}\right)^n

Yay, it seems the wordpress $\TeX$ plugin works.

sudo is evil

Thu, 25 May 2006 20:55:35 +0100

sudo, the *nix tool with which you can easily execute commands as a super user is harmful.

The idea behind sudo is good. You are able to execute super user commands — offcourse when the super user has allowed you — without having to log into the super user.

The problem though is that the default configuration of sudo only asks for the password of the current user, when he first uses it. That means that if the security is breached of the user, which could happen, it can simply use sudo to gain access to root, for it is very likely that users who have enabled sudo, use it a lot.

My recommendation: don’t use sudo, if you haven’t explicitly configured it to require your password every single time you use it — use su instead.

update: If you want to force sudo to ask your password every time you use it, add this in your /etc/sudoers:

Defaults timestamp_timeout=0

They could’ve used a more sensible name for that in my opinion. I had to read the manual carefully before I found out it was this entry.

Keep your email secure (and what just doesn’t work)

Sun, 21 May 2006 00:15:26 +0100

The best way to keep your e-mail address secure from evil spam bots is some kind of javascript and obfuscation, which unfortunately isn’t always available. There are enough alternatives though.

Usually people tend to replace the ‘@’ with some short replacement like ‘{at}’ or ‘bij’. This just doesn’t help.

Any programmer with a bit of knowledge of regex can create a program that scans for domain names and interprets every small bit of text in front of it as an @ sign.

Some smarter people also replace the dot. This works, unless your email-host uses a easily recognizable TLD (.com) or domainname (gmail.com).

Also putting ‘SPAM’ in your email-adress some.personREMOVETHISFORSPAM@foo.bar is easily filtered.

Best thing is to use something out of the box.

For instance, my email address is X@Y, where:
X = bas.westerbaan
Y = gmail.com
Also I’ve got an email-address on w-nz.com, namely bas.westerbaan.

Or even maybe w-nz.com@bas.

Anonymous comments disabled re-enabled

Fri, 19 May 2006 22:09:49 +0100

I’ve disabled anonymous comments for my blog, because my blog is spammed with ~100 spam-comments a day — it seems they’ve worked around Hashcash 3.0. I’ll look into this a bit more when I’ve got time. Sorry for the inconvenience.

Update Seems it were nasty trackbacks instead of comments. So I just disabled trackbacks. You can comment again.

First SINP draft

Tue, 16 May 2006 08:21:49 +0100

SINP is a set of protocols to transfer a profile/identity; to authenticate owners of identities and negotiate for restricted information in protocols. It’s designed to be simple, being based on HTTPS and XML.

You can find the first draft here.

Subversion repository: https://cvs.codeyard.net/svn/webid

Acknoledgements: it’s loosely based on other stuff that has been floating around the web, like Zef’s SPTP.

Comments would be appreciated.

Update: Photo’s are in of the presentation we’ve given about SINP last wednesday:

http://www.codeyard.net/fotos/capaward-1.php
Our presentation has got several penguin mascottes.

GMail`s strange perception of EMail size

Tue, 02 May 2006 00:44:26 +0100

I’ve just downloaded my whole GMail account via POP3. Each message is stored as-is, including headers and attachments in base64, in a seperate file.

du -ms on the folder with all emails tells me:

2117

While GMail tells me:

You are currently using 2079 MB

Why this difference, I asked myself. Was there a difference in the way e-mail was stored? Actually I stored my e-mail very inefficiently. E-Mail is 7bits encoded — every character is 7 bits — where my FS (like virtually every other) stores each character in a 8bits. Lets calculate the actual size of email on my disk:

>>> 2117 / 8 * 7
1848

It gets even more absurd considering that the most part of that space is used by attachments in emails, which are encoded in Base64, which uses 6bits per character. At least 50% of these 2GB are attachments, thus:

>>> 2117 / 8 * 6.5
1716

Why am I using 2079MB according to GMail?

GMail wouldn’t require 2079MB for my emails, they probably compress all attachments and old mails, they won’t even come near that 2079MB. It would therefore seem logical that they would use the real size of all emails — which should at least match the 1848MB or 2117MB, but it doesn’t.

Anyone got a good guess?

Disclaimer: This isn’t in any way meant to be anti-gmail — I love gmail, everyone loves gmail! I’m just curious.

Last.fm

Thu, 13 Apr 2006 20:08:40 +0100

last.fm is a (omfg! web2.0) service that tracks the music you play. You just install a plugin for your favourite music player and all songs you play will be send to the site. They will create nice charts, make recommendations and lots of other fun stuff.

Oh look, it even has got syndication! (my recently played tracks)

You can see nice charts about my music here.

SQL on rails

Sat, 01 Apr 2006 19:50:00 +0100

You’ve got to love the web2.0 impact on aprilfools:

A search engine in a few lines of code with SQL on rails.

The .tgz you can download is (deliberately) corrupted. You can still recover some amusing stuff from it. Give it a try.

IThreads

Wed, 22 Mar 2006 11:22:54 +0100

While working on paradox, Kaja and I felt that we needed a cross-platform threading library, which handles all the subtle differences in capabilities and semantics of each platform`s API.

After a rewrite of Kaja`s original code, I present to you ithreads. You can find the source in this subversion repository.

svn://w-nz.com/px

At this moment ithreads gives you implementations of:

Mutex

Event (aka. conditional variable)
Semaphore
R/w lock
Barrier
Gate
Thread

By using (one or more of) the following APIs:

pthreads, posix thread API (unix, linux …)
pth, portable user-space threads
futex, linux`s lightweight fast hybrid locking

ithreads will use generic implementations in case one of the primitives (eg. Semaphore for pthreads) isn’t available on your system.

I’m espacially happy with the futex based implementations. They outperform the pthreads implementation, and are a lot smaller (4 vs. ~16 bytes for a mutex in pthreads) – although they aren’t as robust (but still correct).
Support for BeOS, OS/2, Windows and more will be added, but testing them will be tricky.

The library will still need a lot of stress testing and probably still contains some bugs – you’re warned. When everything works nicely I’ll add all kinds of extra flavours of the primitives, eg. read-favouring r/w-lock.

I hope it will prove useful.

Thread synchronization mechanisms (Locks)

Thu, 02 Mar 2006 00:02:55 +0100

To share it with you and making sure I won’t forget, I present you with (some of) the different thread synchronization mechanisms known to mankind. Most of the mechanisms are used to lock operations or data, some aren’t. You know, those nifty little things that ought to help you with thread hell.

Semaphore
The semaphore is the cournerstone of locking – often other locks are implemented using one or more semaphores. A semaphore is a simple integer. The initial value of this integer is the amount of threads that can attain (aka. lock, down, enter, pass …) the semaphore before it will block incoming threads – a blocked thread will have to wait until a thread that has attained the semaphore releases (aka. unlocks, ups, leaves …) the semaphore.

The implementation of a semaphore is an atomic integer on most systems. When a thread wants to attain the semaphore it will check whether the value is greater than 0. If it is it will decrement the value and the semaphore has been attained. If the value is 0 it means that there are already as much threads that have attained the lock as the semaphore allows – and the thread will wait a while and try again. If a thread releases the semaphore it will increment the value.

You can compare a semaphore with a dining table in a mess hall at which only a certain amount of people can dine. People will either have to wait for someone to finish and stand up, or they’ll have to look for some other place.

Mutex
The mutual exclusion lock is the lock you’ll be most familiar with and is often just referred to as a lock, because of its popularity. A mutex is either locked or unlocked. Only one thread can hold the lock – all others will have to wait for that other thread to release the mutex. The mutex is often used to assure mutual exclusive access to a certain operation or piece of data.

A mutex is simply just implemented as a semphore with a starting value of 1. The first thread that would attain the mutex would decrement the value to 0 and block all other incoming threads – until it releases the lock.

You can compare a mutex with the door lock on your restroom, it will help you make sure that you got some privacy.

Event
An event will block all incoming threads, until it is signalled to release a thread or all of them at the same time. Events are quite useful when dealing with situations when you want to queue threads before they are allowed to pass. Events aren’t locks that you lock and after that have to unlock. A thread just waits for an event, and doesn’t have to ‘unlock’ it afterwards.

An event is usually also implemented using a semaphore. This time the default value of the semaphore would be 0. If a thread wants to pass the event it’ll try to decrement the semaphore`s value, unless it is 0 in which case it’ll wait, just as you would normally lock a semaphore. If the event is signalled to release a thread it will increment the semaphore`s value – one of the waiting threads will notice that the value isn’t 0 anymore, decrement it to 0 again and pass. If the event is signalled to release all waiting threads it’ll keep on incrementing the value until all threads have passed. Usually there is a mutex alongside the semaphore to make sure that a thread can’t slip in, while all threads are being released, and be released to soon.

You can compare an event with a (chaotic) queue in front of a fun park attraction. People have to wait until the operator of the attraction tells one of them to continue, or tells all of them to continue at once. (the attraction could be nearly empty with enough room for all of them).

Gate
A gate is a flavour of an event. It blocks incoming threads just like an event, and lets all of them through if signalled. The difference it that it won’t close again – once signalled (‘opened’) it’ll allow all threads through until it is explicitly closed. It also can’t let threads through one by one.

A gate is usually implemented using a semaphore, with the initial value of 0 so it will block incoming threads. Each incoming thread tries to attain the semaphore as is done too with the event – but when a semaphore lock is attained it’ll directly release the semaphore so it’ll be available for another thread waiting to pass too.

You can compare a gate with a … gate. Or a draw bridge if you like. If it’s closed none can pass. If it’s opened everyone can pass. Very straightforward.

Barrier
A barrier blocks all incoming thread until there are a certain number of threads. In that case the barrier will release all those threads, but will remain to block incoming threads (until they have reached the specified count together too, etc). A barrier is useful to make sure that a set of threads all start something at the same time and maybe even more important aren’t doing what they did before.

A barrier is usually implemented as a custom event, that will have a waiter-count-field (and a mutex to synchronize access to it), which will auto release all threads when the waiter count has reached the specified number.

You can compare a barrier with watter drops from a tap. The drop won’t fall until there is a specific amount of water that causes it to fall.

So far for the thread synchronization mechanisms today. Tomorrow I’ll maybe talk about the hassles of implementing them yourself.

Some credit goes to Kaja, who told me about all these locks in the first place :-).

Upgraded to WordPress 2.0.1

Wed, 01 Mar 2006 18:41:16 +0100

Just upgraded to WordPress 2.0.1, which was way too easy:

# in my blog.w-nz.com htdocs folder wget http://wordpress.org/latest.tar.gz tar -xvzpf latest.tar.gz cp wordpress/* . -R

Visit the upgrade script, which consisted out of one simple click and I was done.

Great stuff.

PS. I reuploaded my logo and google analytics code too, but they don’t count, really (even though they consisted out of more work that the rest of the upgrade).

Gentoo linux 2006.0 installer

Tue, 28 Feb 2006 10:36:40 +0100

Today Gentoo released a live-cd with a gentoo-installer on it. They also posted some nice screenshots.

Gentoo Linux is a linux redistribution that features its home-made package manager Portage. This package manager makes Gentoo special. There are a lot of other package managers, but Portage is one of the only package managers that compiles each package on your computer, instead of installing precompiled binary’s.

Compiling everything yourself has got the advantage that you can tweak your package to your specific needs. This means you can have a nice customized system. But the key advantage is performance. Most binaries that package managers install are compiled for generic x86. That means that it can even run on your old 486. The issue is that a lot has changed since the 486. A few new instruction sets (MMX, SSE, etc), specifically designed to increase performance for common tasks, are present in almost every processor which aren’t used by generic x86 compiled binaries. And even more performance advantages¹.

For this reason some people already did this (see Linux From Scratch). The problem is that it takes a lot of time to configure, build, install and trouble shoot at least 200 packages. If you take into account that your average server and desktop have at least 15 updated packages per week you are looking at a huge amount of dedication and time required to keep your own Linux From Scratch up to date.

This is where Gentoo and Portage comes in. If you want apache, you simply type emerge apache. If you want to update everything, you simply type emerge -u world. Portage checks which packages are required, it downloads them, it patches them, compiles them and installs them for you.

But wait? Maybe I don’t want ssl build into apache. How do I do that? Fairly simple actually. Portage has got a system called use-flags. There is a useflag called ssl. You can put ~ssl in the USE entry in /etc/make.conf to tell to portage that you don’t want ssl on any package. Or you could put net-www/apache ~ssl in /etc/portage/package.use, which tells that you want ssl to be disabled for apache.

Off course Portage isn’t perfect, and you will have some trouble once in a while, but it’s better than the headaches caused by LFS and the reward is similar2.

— A happy Gentoo user.

¹When compiling for generic x86 a lot of memory is aligned which causes a lot of overhead for processors that don’t really care a lot about alignment. Not only are there these nice new instruction sets, but each processor has got specific differences in the implementation of the instruction. On some processors instruction X may be faster than instruction Y. Also some processor information in tight loops can be useful. On one processor the loop may fit in L1-cache, on others it wouldn’t.

²You can offcourse do more with LFS. But usually it isn’t that important, and if it were it would be better to add it into Portage yourself than to switch to LFS for that reason.

Xgl

Sun, 19 Feb 2006 20:49:24 +0100

After a long night wrestling with alfa source code, I’ve managed to install Xgl.

Xgl is a Xorg-X11 layer that uses openGL to achieve some nice stunning effects.

One of these is to be able to switch desktops by pressing ‘Ctrl+Alt’ and dragging your desktop-cube.

There are a lot of other things that I can’t show with screenshots. Take a look at the Xgl release post. These things include that all forms behave flexible. If I drag a form it’s like it’s made of rubber instead of concrete. Also every form pops up gently animated. There’s also a mac osX expose-clone, which is really helpfull.

If you want to install it yourself then the gentoo wiki article and hanno’s blog post should be very helpful.

Virtualization isn’t hardware efficient

Thu, 09 Feb 2006 11:13:53 +0100

Virtualization is the thing right now. Escpacially because VMWare released VMServer for free.

I’ve read quite a lot of posts about the pro’s about virtualization, and I`m annoyed by some.

This for some claim that virtualization is hardware efficient. Quite frankly, it isn’t. It is quite inefficient.

The thing though is that you don’t use your server optimally usually, and that with virtualization you can more easily get most out of your server, although you could get more from your server if you wouldn’t use virtualization. But that is a lot trickier, virtualization is quite flexible.

Virtualization doesn’t make hardware more efficient, it makes you more efficient at allocating resources of your server.

This server, w-nz.com, runs on a virtual server. I can’t affort or would fully use a ful dedicated server, but I can affort and use a virtual server. Efficient. But running all sites on the real server on just one server would be more efficient, but harder to do, espacially related to security.

Virtual locality of reference

Tue, 07 Feb 2006 13:38:42 +0100

Quite a lot of programmers consider the advantages of locality of reference; if they put all the data they work with in about the same space the processor will be able to retreive them more easily, for the processor caches regions of memory in the Lx caches.

Quite a lot of memory managers even move objects for you in the same place which seem to depend on each other. This doesn’t only decrease fragmentation, for they would be freed in the same timespan, but also increase performance due to locality of reference.

Although the advantage of locality of reference isn’t as big as most people presume in some situations due to virtual memory.

The address space a process sees is a virtual address space, it is mapped in place by the OS. This means that 0x1234 can contain an image for one process and a normal integer for another process. This enables two processes to be loaded, virtually, in the same place.

One implication this has on locality of reference is that a seamingly contiguous piece of memory can in physically be distributed over quite a big span of real physical memory. One part of your list could be at 0x1300, the other at 0x30240.

If your program wants to enumerate through that list it`ll have to query the RAM for the second part, which it wouldn’t have to if it would be contiguous in the physical memory, for then it would most likely be in complete in the L2 cache.

The physical memory is mapped page by page into the virtual memory space. Usually pages are 2KB. Each page is always contiguous in physical memory for it can’t be split up by mapping. Therefore locality of reference always works within a page.

It doesn’t mean that if you are working with more than a page of memory it will be all fragmented. The OS usually does a good job at keeping pages contiguous in physical memory when it is contiguous in virtual memory.

Although when a computer is under some load and it’ll have to use each bit of memory and swap in and out memory it will likely be a lot more fragmented.

I don’t want people to be ignorant about the advantage of locality of reference when dealing with more than a page of memory. I want people that trade of literally everything for their holy locality of reference to see that it’s just a nice extra, but that it just doesn’t always work due to fragmentation of the physical memory.

Side notes
Virtual memory is a bit more complicated than I explained, read about it on wikipedia.
Processors will try to keep virtual memory address space in mind to avoid this issue, although it’s limited.

Mulholland Drive

Tue, 31 Jan 2006 00:05:58 +0100

Mulholland Drive is a great film. If you haven’t watched it yet, just watch it.

Those lucky enough to be astounded and puzzled by the film have all came up with their own interpretation of the movie.

The most common interpretation is that either the first or the last part is a dream of the opposite part. I’d have to disagree with that for both parts are just too bizarre to be real. However, both parts are pseudo-opposite.

Just take the ‘119’ on the firetruck for instance that hasn’t been mirrored, but still reversed.

It keeps you thinking in any case.

PHP’s hidden treasures

Mon, 30 Jan 2006 19:37:39 +0100

I’ve complained a lot when I worked with PHP about PHP’s terribly inefficient design; I’ve complained about it, just because it was PHP. The things I missed most in PHP, it seems, were actually there all along!

Behold shared memory and yet more IPC.

One reason PHP really sucked was that you need to load small data from and to databases or files if you want to share it between page views. The whole concept of handling a request per page view is ridiculous too, IMO.

With those two libraries however, PHP pages could be way more efficient.

I wonder why the big PHP software haven’t used it. Lets hope it isn’t portability.

Beacon with eggs and …

Sat, 28 Jan 2006 21:14:47 +0100

Spam, again on this blog.

It seems those nasty spammers are now using actual people (or an automated browser) to post spam comments, subverting the protection wp-hashcash delivers.

Luckily it are only about two per day, which is very managable, but still annoying.

Like DRM and most other copyright protections, SPAM protection is inherently insecure, for the original openess required to allow a not-spammer to use them is, well, too open.

One interesting thing is that we can fight back now that they are using full javascript VM’s. Matching the spam-ip and letting it execute a rather ‘memory inefficient’ little program would certainly make me smile.

Freelancer hash database

Sat, 28 Jan 2006 21:04:17 +0100

The game Freelancer uses hashes a lot in its data files, which gives modders a hard time.

I’ve put a database online, which modders can use to look-up hashes.

mail.w-nz.com

Sun, 15 Jan 2006 22:51:28 +0100

I’ve successfully installed vpopmail, qmail and courier today, after some hours of work on this server.
From now on you can send e-mail again to @w-nz.com adresses, which won’t end up in /dev/null. (like bas.westerbaan@~)

To the users of the server: if you want an email-address, mail me.

While testing my smtp server whether it would receive incoming email properly I was harshly confirmed by it being spam instead of my test mail entering my mailbox first:

It’s pretty funny to see spam junk in an ancient text-mode mailing program.

If anyone experiences problems with any application not mailing you any verification as it should, please email me. It could very possibly be that the qmail-send isn’t configured properly to allow unauthorized local mails.

Update: More good news, the amount of RAM on my virtual server has increased to 120MB! Which means that my server won’t be swapping instead of working all the time anymore.

Crash

Thu, 12 Jan 2006 21:41:03 +0100

For unkown reason lighttpd decided to pump out a log file of a few GB’s, which totally filled the last single piece of free memory on my server, resulting in the malfunction of several server daemons crashing the server.

Anyone has experience with this problem?

Welcome to lighttpd

Wed, 04 Jan 2006 14:38:59 +0100

After a long afternoon I’ve got lighttpd to work with my current apache based layout.

This means I can choose whether to run apache or lighttpd.

Lighttpd is a webserver, like apache. The key advantage of lighttpd over apache is that lighttpd is very light on your server. It uses a lot less memory, which is very nice espacially when considering that my server only has got ~90mB of memory.

The drawback of lighttpd is that it is light and doesn’t support as much as apache does.

It doesn’t do .htaccess files. Everything needs to be configured in the lighttpd.conf, which doesn’t support everything, or at least not in the same way as apache does.

However, lighttpd is pretty easy to configure when you get the hang of it.

One particulair pain in the ass is getting old mod_rewrite using .htaccess to work, for instance the one used by wordpress of this blog.

I’ve added this in my lighttpd.conf:

$HTTP["host"] =~ "blog.w-nz.com" { url.rewrite = ( "^/(page|archives|comments|search|feed)/" => "/index.php?error=404" ) }

One interesting thing to note is that the configuration file is nothing more than a script being executed for each request.

Welcome to wordpress 2.0

Sun, 01 Jan 2006 17:16:11 +0100

I’ve just upgraded my blog to the new wordpress 2.0, which seems to work properly.

Espacially the back-end has got a face lift, which looks and feels really nice.

Update: redid the logo and changed the theme to show the author

And offcourse: Happy newyear!

Rainbow Tables: Coverage

Mon, 26 Dec 2005 02:45:52 +0100

A rainbow table is generated by creating (m) chains using randomly picked starting keys. The reduction functions result (or ought to result at least) in evenly distributed new keys. Their is only a probability that a certain key/cipher pair is in the table.

I’ll gradually explain how the coverage can be calculated.

Take m chains, each 1 long to cover a keyspace of N keys. The chance that you can find the key is:

\displaystyle P = \frac{m}{N}

When we have 1/2N chains, we’ll subsequently have a P of 1/2, a 50% coverage. Obvious.

When each chain would be 2 in length it gets trickier.

$P = \frac{m}{N}$ still is the chance that we can find the key in the first column (first item of each chain of the table) of the table. The second column isn’t as straight forward. We’ve blindly assumed that each key in the first column is unique, which is true for the first column, but certainly isn’t for the second column. A few chains can merge. The second column is ‘random’.

The chance that a certain key is in a specific chain of the second column is $\frac{1}{N}$ . The chance that a certain key is in the second column is $1 - \left(1 - \frac{1}{N} \right)^m$ (One minus the chance that there isn’t a certain key in a certain chain multiplied by itself the-amount-of-chains times).

The amount of keys covered by this column is the chance that a certain key is in the column times the amount of keys: $N \left(1 - \left(1 - \frac{1}{N} \right)^m \right)$ .

The chance a certain key is in any of the two columns would be:

\left( \frac{m}{N} \right) \cdot N \left(1 - \left(1 - \frac{1}{N} \right)^m \right) \over N

The third column’s coverage can be calculated by taking the unique keys in the second column as m was taken for the amount of unique keys for the first column. With each chain t in length this formula applies:

\displaystyle m_1=m \\ m_{n+1}=N \left( 1 - \left( 1 - \frac {1}{N} \right) \right) \\ P=1-\prod^m_{i=1}1-\frac{m_i}{N}

Rainbow Tables: Introduction

Mon, 26 Dec 2005 01:52:20 +0100

A rainbow table is a data structure with which it is possible to retreive the key used to generate a ciphertext with a fixed plaintext. One common application is to reverse (password) hashes; which can be seen as cipher functions who take the message as the key and the ciphertext as the generated hash.

One important thing to note is that all keys and ciphertext pairs stored in the table have been precomputed during generation of the table. Generating a table that stores 1 miljon md5 password hashes, requires all those passwords to be hashed during precompution.

I’ve been doing some research about rainbow tables and I think it will be usefull to share some of my experiences.

But before I can dive into the matter I first got to explain how rainbow tables work. (or you can read the ‘official’ paper about rainbow tables by Philippe Oechslin).

Precomputing a significant amount of key/ciphertext (password/password hash) pairs is fairly easy. The issue is that a significant amount is fairly large, 2³⁷ for alphanumerical windows password hashes to give a real-life example. Because it isn’t practical to store 2³⁷ pairs of key/ciphertext’s (~2TB) on your computer; rainbow tables store chains of key/ciphertext’s pairs, more precisely they store only the starting and ending key of a chain.

Each chain starts with a key (randomly generated; in our example a password). The next item in the chain is the ciphertext which is calculated by the cipher (in our example the hash for the password). The next item is a key (password) again, which was generated from the ciphertext (hash) by a reduction function. This reduction function can be almost everything, it can be a XOR hash, it can be CRC32, as long as it changes the (larger) cipher text into a new key.

k₁ — cipher –> C₁ — reduction function 1–> k₂ — cipher –> C₂ — reduction function 2 –>…

Each chain has got a fixed length (t). In the table only the first and the last key are stored (k₁ and k_t).

To look up a certain ciphertext, the reduction function is applied on the cipher text and all the end-keys of all chains are checked whether they match. If one of them matches you can regenerate the chain from the begin key stored allong the end key of the chain, and you’ll find the key that when applied the reduction and the cipher on will result in your ciphertext (which is the last ciphertext in the chain). When no end key matches, the cipher and reduction functions are applied to generate a new key, and all the end key’s are checked again. If one of the end key’s matches you know that your ciphertext is in the chain in the second-last position. Repeat this t times for each position in the chain.

It sounds straight forward, but there are some issues that appear:

Loops, there are some cases where the reduction function generates a key which when being applied the cipher function will become the first plain text. (K = R(S(K)), where R is the reduction function, and S is the cipher). This will cause a the rest of the chain to be useless.
Merging chains/False alarms, because the ciphertext is larger than the key there will likely be a lot of chains with the same end point, this reduces the total coverage of the table and will cause a lot of additional time spent on the ‘false alarms’ caused by the fact that you have to check each matching endpoint, even if the starting key is the wrong branch.

To avoid loops; avoid merging chains a bit more and to increase lookup speed, rainbow tables use rainbow chains: each column in the table uses a different reduction function.

Because we use a different reduction function for each ciphertext in the chain there can’t be loops, or at least not loops that cause redundant data, because a certain ciphertext will become a different key in a different column of the chain.

The amount of merging chains is also cut by t times, because a certain key in a chain will only lead to the same chain when it is in the same place of the chain.

This is a very superficial explanation, read more (when you’re interested) in the original paper.

Merry Christmas

Sun, 25 Dec 2005 11:07:03 +0100

Merry Christmas!

Christmas carol

Sat, 24 Dec 2005 15:07:26 +0100

What would you do when you need to perform something ‘nice’ on a christmas celebration of your high school?

This is what my brother and Noud did.

Hopefully I`ll be able to grab some photo’s soon and upload them of the stunned faces of the suprised people there.

The problem with rainbow tables

Wed, 07 Dec 2005 23:13:47 +0100

Rainbow tables (Making a Faster Cryptanalytic Time-Memory Trade-Off) work fairly well, although – because they use different regression functions in the chains – they contain a lot of redundant data.

The example table in the paper (lanmanager hash; 8 char. alpha; 4666×38223872) contains a lot of redundant data, which can be calculated:

>>> CalcRedundancy(4666, 38223872, CalcN(26, 1, 7)) 21.50866258033486

The 300mb table would be stored in a 20mb table if the regression function was to be perfect.

The problem is that to make a regression function perfect, it should be ‘in paralel’ with the cipher function, which isn’t an option for most of these are engineered to prevent this.

Adding variable length chains back into the rainbow chains could allow exploiting existing chains more efficiently.

This probably sounds messy and to be honoust this idea is still a bit messy.

I`ll write something decent along the way.

Switching to Lighttpd

Tue, 22 Nov 2005 14:48:08 +0100

Recently a lot of people seem to be switching from Apache to Lighttpd, which is a webserver that is said to be a lot faster, but even better it is said to have a constant low memory footprint.

I`m currently compiling lighttpd on my vserver (on which this blog runs), and I`ll switch to lighttpd – which should be as easy as setting some configurations for lighttpd to fit in with the current /var/www model I`m using and simply switching off Apache and switching on lighttpd.

I hope there won’t be a lot downtime.

Update Lighttpd and fastcgi don’t seem to really go together on my server configuration, so no lighttpd for a while :(.

Westerbaan Huurman notation

Tue, 15 Nov 2005 00:11:07 +0100

[C_n-H]^1..8 [sigma(n,n+1)]^1..3,5..7; sigma(4,1);sigma(8,5); [sigma(n,n+4)]^1..4

and

C₁-5,H-C₂-6,H-C₃-7,H-C₄-8,H,1; C₅-H-C₆-H-C₇-C₈-H;

and

[C_n-n+4,H]^1..4; [C_n-H]^5..8; sigma(4,1); sigma(8,5);

They all represent the same carbon cube. (C₈H₈)

Purpose; because the normal naming of chemistry can’t define all structures they come up with trivial names, which I find very annoying. This notation is intended to be able to describe every structure uniformly. And no, don’t even try to pronounce it.

Google Analytics

Mon, 14 Nov 2005 23:15:03 +0100

Google’s new service to information domination, Google Analytics, it a traffic tracker for your website, originally used with their adwords service, seems pretty nice.

Well.. if it would be responsive, that is.

Worse than being slashdotted, a lot of high-traffic bloggers link to it.

Unintentional Distributed Denail of Service.

Code Golf 3: Googler

Wed, 02 Nov 2005 18:43:58 +0100

Code Golf. As with the real sport the goal is to reduce the number of strokes that it takes to complete a particular objective, although with Code Golf â€œstrokesâ€ refers to keystrokes rather than swings of a golf club.

Write code to return the url’s of the first page of search results for the query ‘perl’ on google.com

There’ll be two categories, one for using perl’s Socket module only, and for those using external libraries.

Good luck!

Code Golf 2: Line Sort

Tue, 01 Nov 2005 21:44:04 +0100

Create an algorithm that reads lines from the stdin until it hits an EOF and prints those lines sorted to the stdout

Don’t use the language’s native sorting capabilities, write your own sorting algorithm. (print for sort<> would be too easy.)
Any language is allowed, except for those that myseriously implement a 1 byte command that does exactly this.

Example implementation using inverted bubble sort:

@a = <>;
for ($i = 0; $i <= $#a; $i++) {
 for ($j = $i; $j <= $#a; $j++) {
 if (@a[$i] ge @a[$j]) {
 $t = @a[$j];
 @a[$j] = @a[$i];
 @a[$i] = $t;
 }
 }
}
foreach(@a) {
 print;
}

This implementation is very large and inefficient, and just an example.

Good luck!

Code Golf 1: Fibonacci

Tue, 01 Nov 2005 19:23:55 +0100

Code Golf. As with the real sport the goal is to reduce the number of strokes that it takes to complete a particular objective, although with Code Golf “strokes” refers to keystrokes rather than swings of a golf club.

You may compete using any language.¹

Create an algorithm that prints the 30 first fibonacci number to the screen, each followed by a newline.

An algorithm which could do this would be (148 bytes):

my $a = 1;
my $b = 0;
my $c;

my $limit = 30;
my $i = 0;

while ($i < $limit) {
 $i += 1;
 $c = $a + $b;
 $a = $b;
 $b = $c;
 print $c . "\n";
}

But that can be written way shorter.

My first:

$a=1;for((0..29)){$c=$a+$b;$a=$b;$b=$c;print"$c\n"}

My second:
$a=1;$c=$a+$b,$a=$b,$b=$c,print"$c\n"for 0..29

Noud’s reponse:
$i=1;print$i+=$a,"\n",$a+=$i,"\n"while$i<317811

My revenge:
$a=1;$c=$a+$b,$a=$b,$b=$c,print"$c\n"for 0..29

Noud was 30 seconds later with:
$a=1;print$a+=$b,"\n",$b+=$a,"\n"for 1..15

My latest one: partially by Bram too
print$a+=$b,"\n",$b+=$a,"\n"for$a++..14

Update:Twan:
print$a+=$b,$/,$b+=$a,$/for$a++..14

1. a language that implements fib returning mysteriously the first 30 fibonacci numbers isn’t allowed.

If only I could cash it

Tue, 25 Oct 2005 08:02:13 +0100

My blog is worth $2,258.16.
How much is your blog worth?

Memory Alignment Havoc

Sun, 23 Oct 2005 22:50:04 +0100

Memory Alignment is one of the most important concepts when working low-level with memory. The bliss of the x86 processor, which is very kind about alignment, has left me unaware of it for years. That was until Kaja mentioned it, and therefore I decided to rebuild my Gc.

Like err.. what you talking ’bout?

Data should be aligned in the memory. Why? Because the processor doesn’t see the memory as most programmers do. For the processor the memory isn’t an array of bytes, but rather an array of blocks. In the case of the x86 processor this is usually 4 bytes.

When I got a simple piece of code like ‘(*i)++‘ the processor will find the 4 byte block in which this 4 byte integer i is located, increments it and stores it.

Well.. it does this when i fits exactly in one block. When i is partially in one block, and the rest is in a second block (it is unaligned) the processor will have to fetch both blocks, reconstruct the integer, increment it, split it, and store it back.

This is a lot of extra work. The processor will spend a few times more work on a simple increment. Although the actual difference is only about 5% because requesting 2 blocks or 1 block doesn’t differ that much.

Because this means a lot of extra work for the processor, and even more transistors, some processor manufacturers decided to barely or not support unaligned data.

Summarized:

Some processors will just cope with unalligned data, although it takes a bit more time. (x68)
Other’s will throw an hardware exception and let the OS fix it up, which takes a hell of a lot more time.
A bunch will get unstable and the process crashes, or the whole processor will.
Ans some will silently ignore the unaligned data using the first aligned data that comes close instead which leads to silent corruptions and hard to trace bugs.

Bliss of C

Maybe this will shock you, at least it shocked me. Luckily you don’t need to worry a lot when you aren’t writing specific low-level applications like memory managers or compilers. Your C compiler will take care of alignment for you.

Gimmi more

Kaja wrote an article about memory alignment.
IBM has got an excelent article on it too.

Enter HashCash 3.0

Sun, 23 Oct 2005 22:26:07 +0100

A new anti-wordpress-spam release of Elliott Back: HashCash 3.0.

~~Basicly it uses md5, javascript and a bit of AJAX, which is very hard to force except when a spam bot executes the actual javascript.~~Basicaly it just keeps an â€œundecipherableâ€ secret which in theory youâ€™d need javascript to decode. The problem with that is that it takes some time to execute the javascript and hopefully will scare the spambot away.

The method isn’t intrusive (although it does require javascript to be turned on), and pretty effective.

Good work Elliot!

Paradox Wiki

Wed, 19 Oct 2005 01:16:15 +0100

Kaja has set up a wiki for paradox to document his Compiler + JIT + Interpreter in the making.

I added quite some documentation about my Garbage Collector IMGC there too for it will be used by Paradox.

Back from Rome

Sat, 15 Oct 2005 10:40:09 +0100

I`m back indeed.

Rome is a great city.

Giolitti‘s ice-cream tastes great as does tazza d’oro‘s espresso.

Best building I visited was the pantheon. I like it more than the sixtian chapel, which is overrated in my humble opinion.

(Go Bernini! Boo Michelangolo)

Anyway, I could fill a few pages with my experiences, but I`m to lazy and too excited with my new toy: google reader!

Rome, here I come

Wed, 05 Oct 2005 17:46:59 +0100

I’ll be on a school excursion to Rome for 10 days, from tomorrow Thursday the 6th of October untill Saterday the 15th of October.

NB Contact my twin, Bram, if you would have contacted me.

bram.westerbaan@gmail.com

My wordpress.com blog finally has got an usage

Sun, 02 Oct 2005 01:32:25 +0100

The free wordpress.com blog I got will be used to dump my stupidest, most impulsive and chaotic thoughts or things I want to say.

You’re warned.

Binary Multiples

Sat, 01 Oct 2005 12:34:03 +0100

Data size prefixes always have caused confusion.

How much is one MB?

Originally it’s 1024 kB, each kB is 1024B. This makes a total of 1024³ bytes (1073741824).

Using 1024 instead of 1000 has its roots in the fact that computers usually work with blocks of 1024 bytes (2¹⁰).

Other people stick to the SI and deem one MB 1000³ bytes (1000000000).

To get rid of confusion binary multiples were introduced. One MiB = 1024³, where one MB would be 1000³.

Factor	Name	Symbol	Factor	Name	Symbol
1000¹	Kilo	k	1024¹	Kibi	Ki
1000²	Mega	M	1024²	Mebi	Mi
1000³	Giga	G	1024³	Gibi	Gi
1000⁴	Tera	T	1024⁴	Tebi	Ti
1000⁵	Peta	P	1024⁵	Pebi	Pi
1000⁶	Exa	E	1024⁶	Exbi	Ei
1000⁷	Zetta	Z	1024⁷	Zebi	Zi
1000⁸	Yotta	Y	1024⁸	Yobi	Yi

When buying something one should take notice of this, because the difference between one GB and one GiB is 74MB (or 70 MiB).

With ever increasing storage capability there would be a time where one VendekaByte would be half of one VendebiByte. (Vendika = 10³⁰)

Downtime

Thu, 29 Sep 2005 20:34:43 +0100

Sorry for the downtime, it seems that the vServer software hosting this site crashed.

The GC: Tracing, reversing pointer to object

Sun, 25 Sep 2005 20:06:45 +0100

The problem

Whilest tracing down garbage during the Garbage Collection, as described in earlier posts, it is required to find mark an object pointed to from another object.

This could lead to problems when it is allowed to point anywhere inside an object, or when the object information bit can’t be found by a pointer to the start of the object.

In both cases the Gc should somehow translate a pointer (eg. 0x1234) to the head of the object it points in (eg. 0x1200). The problem here is how to efficiently get 0x1200 from 0x1234.

Avoiding the problem

This problem can be circumvented by disallowing pointers pointing to inside the object and instead require all pointers to point to the start of the object. This will only work when the header of the object has a fixed size, otherwise it will still be impossible to work back to the start of the header, which contains the flags you want to alter when tracing garbage.

Pointers inside a managed object aren’t required and could easily be removed. They can be very usefull though in some situations. One of these situations is being able to inline a private referenced instance into the contained class, of which is known that it won’t be modified, which would result in pointers inside the container class. It will only increase performance in some situations, but it will be usefull to have, and is optional.

A bigger issue are pointers that point to the start of an object with a header in front of it which has got a dynamic size. Either the header shouldn’t be of dynamic size, and a fixed (big) size, or instead of to the start of the object, pointers should point to the header of an object. In the latter case each pointer derefence should require an ‘add’ call to increase the address to match the start of the object, with the additional logic required to attain the size of the header which can be significant depending on the implementation. These extra opcodes for each pointer derefence could not be easily be worked away even when using a JIT because of multi-threading, and certainly could not be avoided when using C. The fixed header size isn’t nice either. For it probably would end up being at least 7 bytes on a 64bit platform (flags + pointer).

Oh, and there is also the java way which uses a handle for each object. Which basicly means that an integer pointer isn’t int*, but int** in java. Where the pointer in between contains the meta-data about the object. This is even more less ideal for it requires an extra derefence for each derefence and it results in twice the amount of pointers to track.

As far as I concern this problem can’t be avoided, when dealing with a Gc that is required to interop with non-managed code. With a JIT it could be done though, but this requires a runtime performance/garbage collection performance trade-off. And I personally like a bit slower garbage collect instead of slower runtime.

Look-up structure

Before tracing the garbage collection usually walks through the heap to mark all objects white and put those in the root-set in the gray-list. During this first walk through the heap a temporary look-up structure could be build to allow efficient look-up during trace.

When only allowing pointers to the start of the object it would be rather easy to do. A modulus-based bucket like implementation, a hash table without the hash-part, would do. This would allow a nice even redistribution of entries throughout the table, for it uses a modulus.

When allowing pointers in the middle of an object it becomes more tricky, because you can’t precompute the pointers to put in the table. You need to have a structure which allows you to look to the left and find the object the pointer points into. Although most pointers still point to the start of the object, the case that they could point inside the object shouldn’t be concidered.

Instead of a modulus-based bucket, division-based buckets should be used. This creates a problem for the likely distribution of blocks between the buckets will vary heavily. One whole bucket could be empty when there is a big allocation in the memory it represents, where almost all small objects which represent most of the pointers could be concentrated in just two buckets.

Bucket-tree

To compensate for this a bucket-tree could be used. Basicly when a bucket contains a lot of elements it would split itself in a new set of buckets. This way a minimal amount of buckets could be wasted on really big allocations, and a lot of buckets could be allocated for small allocations. The look-up time would be a bit worse, but it would be neglectable to the huge lookup time a normal search through the heap would require.

Special treatment for pointers pointing inside objects

When there are only a one or two pointers in the whole heap that point inside another object they could be treated seperately instead, and the rest could keep using the modulus-based buckets.

One way to give those pointers a special treatment is marking them, and resolving to which object they point during the initial walk through the heap where finding objects pointed to from pointers is a lot easier. (pointers needed to be looked up could be sorted by address, and be checked during the walk, which is very efficient).

Another way would be to simulate pointers pointing inside another pointer by making the pointer consist out of an object pointer and a offset from there, which could be cached by the JIT, or expanded before the garbage collection.

Yet another way would be to walk the modulus-based buckets by hand when the pointer isn’t found.

The best method would depend on the implementation of the rest of the Gc.

About modulus-based buckets

I`m still uncertain about whether the distribution in a modulus-based bucket can be trusted at all times. Putting a bucket-tree in default, for worst case would be nice, just to be sure.

Update: some more possibilities:

Binary search

When we put pointers to all allocated blocks sorted somewhere in the memory, one could use binary search, or other search algorithms to search in sorted lists, to find the pointer. When adding shortcuts each ~64K this could be quite efficient. I’ve implemented this method for my Garbage Collector temporarily for it is amongst the easier to implement and I suspect the fastest. Actually, I hope it is the fastest.

I’ll do some tests on different algorithms on a dump of a ‘modal’ application’s managed heap, when I’ve finished the Gc sufficiently to make an application work with it.

The GC: Concurrency

Wed, 14 Sep 2005 21:50:11 +0100

The tracing Gc`s (see previous posts) biggest disadvantage is the unpredictability of the garbage collection. It can happen anytime and the time it takes varies. This while normal execution is halted, can cause serious problems to certain applications which are required to have a low latency.

Incremental Gc

One trick to improve the latency of the gc is to make the garbage collection incremental. Instead of doing everything in one big garbage collection, tiny chunks of execution are interchanged with garbage collection.

Generations

When using a generations gc, a garbage collector which groups objects by their age, garbage collection could be done on each seperate generation. Objects that have survived a few collections, and settle in older generations, tend to remain persistent. (those are the objects that remain global during the duration of the program). Therefore the older generations don’t need a lot of garbage collections. Whereas the youngest collections typically contain a lot of garbage and require frequent garbage collections.

Granularity

Very much small garbage collections could have worse performance than one big garbage collection. Although very big garbage collections which are very infrequent are really bad for latency. When it gets extreme there even could be 100MB allocated memory for 1MB used memory at a moment, because garbage isn’t collected frequently enough. Having a lot of garbage between live data also gets in the way of locality of reference (used memory close together gets all in one load of the L2-cache which is a lot faster than normal RAM).

Although a figure like 100MB sounds ridiculous, take for instance a program that reads through a text file and print every line:

while there-is-a-new-line
 string line = get-line-from-file
 print line
end

Every line read would go on the heap as a string. After the line string goes out of scope it becomes unreferenced from the stack (part of the root-set) and therefore garbage. When reading a big file the heap fills up pretty fast. When doing another similar operations but then from the lot faster memory than file it could even get worse.

When doing a lot of garbage collects small pieces of memory are copied each time over short distances, which is less efficient than bigger pieces, but having to copy pieces over a very great distance, which doesn’t fit in the L2-cache really slows down instead of speeding up.

Trace-only garbage collections

When a lot of memory is allocated in a short time the garbage collector could get suspicious and fire away a full collection, which could be a waste of time when all allocated memory seems to still be live. Moving memory, marking freespaces and updating pointers takes most time of the whole garbage collection. When only tracing what is garbage and what isn’t and after that deciding what to do could provide a gain in performance.

Not only does tracing down garbage take less time, it also is easier to do while the rest of the program is running.

Internal data garbage collections

Most gc’s keep internal data structures optimized for quick access and modification for thing like tracking types, freeblocks, blocks, unmanaged pointers and others. An example could be a list which contains the pointers to unmanaged pointers to be able to link into managed memory from unmanaged memory. This list would be accessed a lot when in unmanaged code, which requires all pointers on its unmanaged stack to be registered on that list.

A chained-block stack would perform best. The whole stack contains out of a few chained blocks, where the last block is pointed to from the stack structure, which on itself points to the previous block, and so on all the way to the first block. Adding a pointer-pointer to the stack would be as easy as increment the pointer count, and put it in the last block. And possibly creating a new block when the current block is full.

Removing a pointer-pointer would work best by searching from end to begin for the pointer and simply NULL-ing it. This for the most frequent use for the stack-list would be for registering pointers from the unmanaged stack and deregistering them afterwards. Although when another global unmanaged pointer is registered there could be a few block registered for the stack, where only the very last contains that one single pointer-pointer to that global pointer which won’t be deregistered soon.

The algorithm could be changed to compact on every operation, or to track freespaces in the stack-list, but it can be done a lot easier. Simply collection the garbage and compacting the stack-list would work just as well.

It would be tempting to perform these garbage collections for internal structures during the normal garbage collection, but this would hurt latency. Instead scheduling them somewhere in between would be best.

Concurrency

When a garbage collection is in progress all other threads are usually stopped to avoid the program itself (the mutator) to change memory and corrupting the managed heap. Although it is possible to make most parts of the garbage collection concurrent with normal application execution.

Tracing

When a pointer is changed during tracing down garbage in an object which already is black (all pointers in the object have been checked), the object pointer to will remain white (no black object seems to point to it) and will be scheduled for garbage collection.

This condition is rare, but it could happen, and it could be fatal. Other conditions like an allocation during trace could be easily solved by marking the object black when allocated.

Write barrier

An easy way to cope with these conditions is to detect any reading or writing to pointers that already are marked black, and acting upon this. This is called a write barrier.

The problem with a write barrier is that it is very hard to implement for languages like C, and cause a performance hit, for each pointer-derefence+write would require a few extra opcodes, and would be in most cases unacceptible. When you are dealing with a JIT-ed language (Just In Time compiled) however, you can emit these extra opcodes to detect writes into the black objects only when a Gc is in progress.

Compacting and Pointer change

After garbage has been traced down it is time for the gc to compact the memory by moving live objects in freed garbage, and changing pointers to these objects accordingly. When traced the gc already would have made a translation table for pointers (usually offset based for efficiency) and would walk its way through the heap compacting and changing pointers in one go.

When the application is running at that time it would encounter problems when reading dereferencing pointers.

Read barrier

To combat this a read barrier would be required. Which is the same as a write barrier, but then for normal pointer-dereferencing. This too requires opcodes to be added on each dereferencing, which would be easy for a JIT and unacceptible for C. The added opcodes would simply look to the place where the garbage collection is at the moment, and convert the pointer as if it was the Gc itself using the Gc’s translation table.

Concurrency outside collection

Making a Gc work concurrent outside a garbage collection is relatively easy for no really collection is happening at runtime and a few rw-locks would do the trick.

Conclusion

Concurrency in a Gc is a must for some applications, and generally a good idea anyway. A better latency can be gained from a fine-coarsed incremental garbage collections. Real concurrency can be gained by using read and write barriers to allow execution during the garbage collection, which is only feasible when working with a JIT.

The GC: Tracing Garbage

Mon, 12 Sep 2005 18:27:58 +0100

Most Garbage Collectors are tracing. Instead of using Reference Counting, they trace down unused objects (garbage) by tracing from the root set of objects and mark all objects found. Those who weren’t marked have to garbage.

The Root-set

The root-set are all objects that are definitely live objects. The root set typically consists out of the objects linked from the stack and registers of each thread. Also object vital to the Gc and globals are in the root-set.

Tri-Color algorithm

An easy way to represent tracing through objects is the tri-color method. There are three colors an object can have: white, gray and black. All white objects haven’t seem to be referenced yet from black objects which are sure to be non-garbage. The gray objects are those that are referenced by black objects but didn’t have all of their references checked. Sounds confusing? A more practical explanation:

Colour all objects white.
Colour the root-set objects gray.
(Repeat this as long as there are gray coloured objects) Pick a gray object. Colour all objects referenced to from that gray object gray too. (Except those who are black). And colour itself black.
All black objects are now reachable, and all white objects are unreachable. Free those that are white!

Pop-mark-push referenced

An implementation of the Tri-Color algorithm can be accomplished by putting a mark on each object which either can be 1 or 0. 1 representing a black object, and 0 representing a white object. A stack list can be used to keep track of the gray objects:

NB do not confuse this stack, which is a normal Last In First Out list with the call stack.

Mark all objects 0 (white).
Push all objects in the root-set on the gray-stack.
(while there are objects in the gray-stack) Pop an object from the gray-stack. Mark it 1 (black) and push all referenced objects from that object on the gray-stack that aren’t marked 1 (thus white).
All objects still marked 0 (white) can be freed rid of.

Generations

Studies have shown that in almost all applications most allocations have a lifetime less than a few miliseconds, although the rest of the allocations would have a far greater lifespan. When collecting every few miliseconds would cause unnesessary tracing through the long living objects which would still take up most of the memory, and when collecting every few seconds the uncollected short-lifed objects would have spammed the heap and would make the garbage collector locality-of-reference unfriendly (everything worked on wouldn’t fit in one l2-cache load and therefore would be a lot slower).

The older objects which are in comparison with the newer objects highly unlikely to be garbage could be searched less frequently. This could be done by seperating the heap into generations. Each generation is basicly a piece of the heap. A typical garbage collection would only target the youngest few generation. Everytime an object survives a garbage collection it’s moved into an older generation, and finally becomes part of the oldest generation that is only garbage collected very infrequently.

Unpredictable garbage collection duration

The duration of the garbage collections would be hard to predict and could vary a lot. Although generation based gc’s could increase the amount of garbage collections and decrease the total time in comparison with one big, applications that use a garbage collection can not guarentee (practically) uninterupted execution.

Advantages

No extra fields required to keep track of the reference count.
There hasn’t got to be registration of references to objects, cause the gc keeps traces those down.
Hasn’t got any problem with circular references.
Outperforms reference Gc. The registration for each reference in the reference counter Gc takes more time than tracing down that reference. And it doesn’t do unnesessary checking of older objects when generations are kept in mind.

Disadvantages

Way harder to implement.
All pointers inside an object should be known.
Worse latency than reference counter. Although it outperforms refcounter, a garbage collects usually requires the whole program to be halted, which is a really bad thing for some applications.

Conclusion

Tracing gc‘s have shown to easily outperform reference counter gc‘s. The prerequeste to know all pointers in an object can be solved easily for interpreted and jit-ted languages using the gc and can also be managed even in C with effort. The real problem remains the arbitrary and unpredictable length of garbage collections. There are ways to make tracing gc’s have a better latency, which I`ll discuss in coming posts.

The GC: As A Managed Memory Allocator

Sun, 11 Sep 2005 14:35:25 +0100

Most Garbage Collectors aren’t limited to just tracing the garbage (unused objects) and scheduling it for deletion, but also manage memory as a whole including allocating it.

Data segment

The kernel of an Operating System takes care of mapping physical memory (RAM/Swap memory/IO memory/Cache’s) to virtual memory available to a process. A process is allocated a few segment’s of memory. The one that is important to memory allocators is the Data Segment.

The Data Segment is the segment which contains memory which is used for random access by the program. The size of the Data Segment can be changed using the sbrk or brk C function, which asks the kernel for more memory to be mapped for the process, which then returns a pointer to the start of the newly allocated space. (providing a negative value will result in decreasing the data segment size).

Malloc

A non-Gc memory manager/allocator are malloc and its friends. Malloc uses sbrk to obtain memory for the Data Segment to allocate memory from as almost all other allocators. It prefixes and suffixes all blocks of free or allocated memory by size fields so it can walk through the whole heap. It puts in each freeblock a pointer to the next and previous freeblock, which makes freeblocks easy to insert and find.

Freeing a block is simply adding into the freeblock chain and setting a bit to flag it is a free block. Allocating a block is simply walking the freeblock chain until a freeblock is found that can hold the required size, and adds new size’s and flags to the new freeblock and updates the pointers one block previous and next into the freeblock chain to match the new offset of the freeblock.

The big problem with malloc is fragmentation. sbrk could be required to be called to enlarge the heap size to create a freeblock big enough for the allocation although there is enough freespace for that allocation, which is scattered and not contiguous.

Modern implementations of malloc feature tricks to prefer perfect fits and automaticly coalesce freeblocks to create bigger blocks, which all makes malloc a whole lot slower than it should be with still quite a lot of memory fragmentation.

Gc managed pointers and `malloc`

Most Gc’s implement their own memory allocator routines instead of using malloc, usually even overriding the old malloc. This is because Gc’s usually know more about the requirements of the application than malloc itself.

It gets even better when the Gc knows all pointers to the managed heap. In this case a Gc could decide to move an object, update the pointers that linked to it, and save a lot of space from fragmentation. This is easily implemented when dealing with a interpreted language which uses a Gc for it knows all types runtime. It even can be done in C, but that requires all pointers both inside as outside the managed heap that point into the managed heap to be registered.

The difficult part is to find all pointers pointing to a certain object. This usually requires the whole heap to be walked, once or twice. Waiting until a reasonable amount of objects would be subject to move, and then doing one big move of all those objects, and possibly other operations requiring heap walks while doing it anyway as one big “garbage collection” would be way more efficient than doing this all ‘live’. In the first pass each object subject to move could be tagged with the new location and on the second pass all pointers would be checked and adjusted accordingly.

Unmanaged code interop

The big problem of being able to move managed memory is keeping track of all pointers into the managed memory. If you would want to store a file handle in managed memory you would have a hard time when the internals of file access keep contain pointers to your file handle.

Fixed allocations, which remain stationary would solve the issue but this could, sadly enough, result in fragmentation of your heap. When all objects can be moved it is very easy to move, but when there are fixed objects somewhere in the heap, complicated ‘fitting’ algorithms must be used which decrease performance.

Fixing memory temporarily, or preventing a GC temporarily is a much cleaner solution to the problem. In some cases this just isn’t possible and big blocks with their own malloc style allocator could be used to handle unmanaged memory so that it won’t interfere too much with the normal managed memeory. On most *nix systems mmap could be used to map additional usable memory to the virtual memory space of the process. Which is slower than bsrk and create some extra overhead, but it will avoid fragmentation in managed heap due to fixed blocks.

When

It is only usefull for a Gc to implement its own malloc when it can move its objects, or when there is a clear advantage to implementing the malloc close to the Gc, which could be for instance to more easily walk through all objects in the heap without having to implement a walk algorithm for each malloc used accross different platforms.

The GC: Reference Counter

Sun, 11 Sep 2005 13:53:58 +0100

The simplest form of Garbage Collector is the Reference Counter Garbage Collector.

Each time an object is referenced (a pointer to it is made) the reference count on the object is incremented. If the pointer isn’t used anymore the reference count is decreased. If the reference count hits 0 the object will be freed.

Python uses a Reference Counter.

Advantages

A reference counter does its tracking at runtime. There is no big garbage collection, and the execution of the program won’t be interupted for a while when the garbage is collected.
It’s very easy to implement.

Disadvantages

Instead of the required malloc free pair required for each allocation without Gc, it now requires add_ref free_ref for each pointer/reference to an allocation. When someone forgets to release its reference the object persists.
Circular references will never be collected. An example of a circular reference is a list that contains as an item a reference to itself. When the list is released its count will remain 1 because inside the list it refers to itself and will therefore never be collected.
A RefCounter GC doesn’t keep track of pointers into the managed heap, and therefore can’t move any object because it`ll break pointers. This makes the fragmentation of the RefCounter GC as bad as malloc‘s fragmentation.

Circular references

Circular references can be fixed by adding a trace algorithm. This algorithm will trace through all objects from the root objects (pointers on the stack, etc.) and mark them. The objects that aren’t marked aren’t accessible and therefore are subject to Garbage Collection.

The problem is that a RefCounter Gc usually doesn’t keep track of what pointers are in objects, and that there could be pointers outside the managed heap which only use the add_ref and free_ref. Python struggled with this problem and it implemented the a trace algorithm that depended on the programmer’s of not-python-modules (of which pointers aren’t known) to implement the list of references to objects that also contain objects. By traversing links that way circular dependencies could be found without hurting unmanaged pointers, but this has a high performance hit on Python, and it is still adviced to avoid circular references.

Conclusion

RefCounter Gc’s are easy to implement, and a good idea for scripting languages which know all references so that circular references can be dealt with accoringly. But they still have a lot of disadvantages which other kinds of Gc’s solve a lot better, and more elegantly.

The GC: Introduction

Sun, 11 Sep 2005 11:38:58 +0100

I’ve been working on Garbage Collectors recently, and I`ll share some of my experiences with them with you on my blog.

First off, what is a Garbage Collector?

A Garbage Collector, a.k.a GC is an algorithm that keeps track of your objects in the memory and gets rid of objects that aren’t used (referenced to) anymore, a.k.a garbage collecting. This can only be accomplished when the GC knows all pointers to the objects managed (objects on the managed part of the heap). And they are therefore almost always used with an interpreted/jit-ted language, like Java, Python or C#.

Since the first GC’s (reference counters) they have gained a lot more responsibility and perform a lot more tasks, where collecting the garbage is just one of the things they do.

I`ll discuss the various kinds and aspects of Garbage Collectors with their advantages and disadvantages in coming blog posts.

You can read all articles about the gc in its category.

Spam, spam and more spam

Sun, 11 Sep 2005 11:23:34 +0100

I noticed I had an enourmous amount of spam in my moderation queue.

The plugin I used to protect myself from spam wp-hashcash, seemed to have been mastered by spammers.

A download of the newest version did the trick.

If anyone experiences problems with posting comments, please mail me.

Update I: Seems some spam prevailed even over this version. I’d better get to making my own custom changes to wp-hashcsah.

Update II: I changed the secret codes in the plugin. And I broke it for a while. Either one of those could have resulted in the fortunate (hopefully not temporarilly) stop of spam.

Update III: According to Elliot Back, the creator of hashcash, the spammers bruteforce the secret value. Changing it usually is efficient enough to keep them at bay for a while. He’s working on a newer version which features bigger, thus harder to bruteforce values. I just hope they won’t suck my bandwidth too much.

Update IV: Unfortunately there seems to be a lot of computing power or a hack behind the breaking of the hashcash security -_-, I keep getting spam :-/

Yesterdays Crash

Fri, 02 Sep 2005 18:16:44 +0100

I`m sorry for today’s and yesterday’s downtime.

@noud, please don’t play with home brewed assembly in root the next time.

B-Day

Tue, 30 Aug 2005 20:19:15 +0100

Today it’s my birthday :-). My 17th to be exact.

Quantumic FS

Sun, 28 Aug 2005 14:17:30 +0100

Recently most linux file systems are atomic. An operation is either performed fully or not at all.

QNTFS even goes further down. Quantumic operations. Something is done, or not done, or done and not done at the same time, depending on the reader, everytime.

Strange pagerank

Tue, 23 Aug 2005 23:33:41 +0100

Google uses pagerank to give a site a ranking of importance. Pagerank is strange.

One thing to note is that pagerank is logarithmic-ish. A pagerank of 2 is a lot better than just the double of pagerank 1.

By far the most visited site on my server, this blog, has got a page rank of 3.

Other pages on my server which are visited sometimes like w-nz.com, board.w-nz.com have got a pagerank of 2.

What catched my attention is that the page xr12.com, which basicly is a filler containing a link to the xr12 wiki has got a pagerank of 5. This is the same pagerank as a big site like newgrounds!

Maybe google values xr12.com a lot because it is about one topic and is the only site about that topic and that is xr12.com, where this blog has got tons of links about practicly everything from very various sources.

Pagerank itself isn’t the sorting factor for google but rather the context, although pagerank still is an indicator. Maybe google values a few links which are very specific above tons of links about very different topics.

Using template variables in extension tags with Mediawiki

Tue, 23 Aug 2005 16:32:13 +0100

Mediawiki, the wiki used by wikipedia, is a powerfull wiki.

It allows you to include self made templates which you can pass variables too. A very usefull template on wikipedia is the stub template. When you add {{stub}} at the start of an article it will insert text explaining the user that the article isn’t finished yet, but rather a stub. With {{Album_infobox|Name=Sehnsucht|Cover=Sehnsucht_cover.jpg|…}} an infobox is inserted used at wikipedia for all albums.

When the features of wikipedia don’t suffice you don’t neccessarily have to hack wikipedia to add features. You can add an extension which hooks into a part of the parser to create your own logic for your own custom tags.

I created a wiki to store the lyrics which I already got on a forum and found espacially templates usefull.

However, there was a slight annoyance when working with templates. The problem is that in several cases there is a link in a template which can be ambiguous (there are two songs called the same) but you don’t want an ugly suffix after the name (like “Deliverance (Album of Opeth)” instead of “Deliverance”). To solve this you have to pass both the name of the album and the name of the page of the album, but this is annoying to do everytime.

MediaWiki knows no logic so I thought to solve this by adding some basic logic in a custom tag which either uses the same text as the page name when no custom text is specified and otherwise uses the custom text. But that just wouldn’t work.

MediaWiki, when parsing a page, first escapes all nowiki elements, extensions, comments, links and so on by a unique ID. Then parsed the format and replaced variables with their values. After that it just replaced the unique id’s back with the proper content, which in this case was the output of my extension too, which was flawed because the variables weren’t replaced before the extension tag was escaped out and also because the wiki-link ([[pagename]]) wasn’t replaced because the extension was still escaped. So I ended up with having [[{{{album}}}]] on my page instead of a Sehnsucht link.

Trying to fetch template variables inside the extension just wouldn’t work because the extension was evaluated at the moment it was already included in the main page. Avoiding this by disabling the cache would require parsing the whole template each time it is viewed which is very extensive. (each link on a page in a wiki has to be looked up when parsed to see whether it exists)

So I googled a bit around and found that Mikesch Nepomuk submitted a patch on one of the mailinglists to fix this by escaping extensions seperately after the veriables were expanded. Apparently it hadn’t been approved or maybe because it wasn’t included in my release yet so I tried to apply it but it failed. The patch was a bit too old. So I did it by hand instead of by GNUpatch and with some tinkering I got it to work.

You can download the patch for the 1.5 beta4 here, if you are interested.

And I thought I would never touch PHP again.

Spread Firefox Clone

Fri, 19 Aug 2005 15:57:25 +0100

SpreadInternetExplorer.com

It’s hilarious

Dual Cores, Traffic and Gaming

Wed, 17 Aug 2005 21:49:57 +0100

Dual Cores are the new thing in the processor bussiness. They claim to be perfect for people who want to do different things at the same time but in reality they are just a cheap solution to deliver more.

The truth is that two 500 mhz processors just don’t deliver the calculation power of one 1 000mhz processor. Given offcourse that they are of the same architecture. This is due to the fact that there is a lot of additional logic required to let two processors work together without getting eachother in the way. When two processors both want to write another thing to the same spot of memory you would get impredictable results. To solve this you have all kind of ways to solve it, by for instance locking a region of memory for an amount of time, making the other processor to wait for the first one to finish. But this all creates a lot of overhead and complexity.

You could compare it with driving a car. Driving a car is relatively easy. You just have to steer around some static obstacles, no big deal. When you know the way you could even do it with your eyes closed. That is unless there are other people driving a car. When driving a car you are keeping an eye, not on the road, but on the other people on the road. This not only slows down your maximum speed or decreases efficiency – you can’t just drive full speed over a junction – but it also increases the complexity and the likelyhood of errors.

The same thing goes in the case of dual core processors (or even hyper-threaded and normal multi processor platforms). Although the comparison isn’t really valid because having two processors doesn’t mean that you have to do two things at the same time. The issue is though that what you normally would do, would only be done by one processor and you are therefore wasting the other’s capability.

A good example of this are games. Games tend to be single threaded, which gives best performance for no processor time is wasted to multithreading and it is the easiest thing to do for multithreaded is rather complicated. Complicated enough that there have been a few lengthy discussions in the mono mailing lists how to lock a simple object.

Because we are getting dual core and propably ‘more’core processors lateron for the companies are too lazy to make decent processors¹ games should become multithreaded to exploit the full capability of the machine it runs on.

Although it makes creating performance applications a lot more difficult it will surelly benifit distributed computing for the change from different processors to different machines is less than from 1 processor to more.

[1] Native threaded CPU’s like dualcores/multi processor/hyperthreaded processors are ideal for server applications where multiple short living requests have to be resolved. Switching an allocating on a software-threaded processor would create too much overhead for such a simple request.

Practically Reversing CRC

Sun, 31 Jul 2005 19:27:46 +0100

The algorithm described in my previous post on how to compute a patch for a message to get the wanted crc computed from that text has a usefull application when wanting to get the possible texts that could compute to your own provided crc.

A little util I wrote, reversecrc (http://w-nz.com/projects/reversecrc/), lists (the endless) list of possible original texts, has been usefull for the Freelancer modding community, for in that game Crc is used a lot and was an obstacle.

Although it is almost impossible to reverse a 20 character string because it would be insane to check all the texts that could result in the hash, using one of the provided texts would do too as it results in the same hash anyways.

Subversion repositry: https://cvs.codeyard.net/svn/icrc

Opeth’s Ghost Reveries

Thu, 21 Jul 2005 21:57:23 +0100

Swedenâ€™s OPETH and Jens Bogren have finished the recording and mixing of the groupâ€™s latest opus, â€œGhost Reveriesâ€. The album was mastered on Thursday (June 16) at the Cutting Room facilities in Stockholm. The total running time will be around 65 minutes.

The two released tracks, Ghost of Perdition and The Grand Conjuration sound promising.

The album will be for sale on my birthday :-), the 30th of August.

And they will perform it during their Europian tour, which I`ll visit on the 11th of September in the 013 in Tilburg.

I`m thrilled.

Reversing CRC

Fri, 15 Jul 2005 12:44:05 +0100

Cyclic Redundancy Code

CRC is a hash which is frequently used as a checksum for data in for instance archives. Who hasn’t had bad CRC errors once when opening corrupted zips. CRC is a very old algorithm and over time it changed a lot from the original idea.

The original idea behind CRC was representing the data that you wanted the hash from as a big number and dividing it by a certain number called the polynomial and taking the remainder of the division as the hash. For instance: 23 % 3 = 2 (% is the modulus operator, which is the remainder of a division)

The initial problem was that dividing is a rather intensive operation. They wanted to simplify CRC to make it easier to implement in hardware and make it faster. They did this by getting rid of the carry used in the substraction of the division:

Normal substraction (binary): 10101 - 01100 = 01001
Without carry: 10101 - 01100 = 11001

Substraction without a carry is basicly a eXclusive bitwise OR (XOR), which returns only 1 when one of the operands is 1 and the other 0.

The algorithm required was faster bit still worked bit by bit, which isn’t really what a computer likes. A computer works best with one to four bytes. To make CRC faster they cached 8 operations at the time by precomputing the results for a certain start value and put it in a table called a XOR Table.

The required code for the CRC calculation itself now became very simple:

hash = (hash >> 8 ) ^ table[data ^ (0xff & hash)]

They changed the CRC algorithm once more by making it reflected. This means that the input data is reversed bitwise: 011101011 <-> 110101110. This was done because most of the hardware chips at the time reversed data bitwise. For it was too much work to reflect each byte of incoming data they changed the algorithm that generates the Crc table to create a table which has the effect of reflected data.

This is by the way not totally correct; the result still was different for a reflected than a non-reflected algorithm for they wouldn’t cache the whole piece of data to reverse it but did it per byte at calculation.

At this moment CRC barely resembles the original idea of a modulus.

Reversing CRC

First off, credits for the original idea of CRC patching go to anarchriz.

CRC is a cryptographicly very weak algorithm. It can be easily reversed for it has got the property that with 4 bytes you append to the current computed hash you can get every required hash. You can change the whole message and add 4 patch bytes to patch the hash to the one you like.

The ability to patch a CRC also makes it possible to very efficiently generate all possible source data of a checksum. Although it still is a bruteforce method you got 4 bytes freely and patching is faster than calculating.

Patching is basicly going back the way CRC works. Crc basicly takes the hash, moves it 1 byte to the right (dropping one byte) and xor-ring it with the table entry. The nice thing about normal CRC is that the first byte of a table entry is unique for that entry.

For the first byte of the entry is unique for that entry and it is put in the hash xor-red with 0 for that is what is shifted in from the right you can work back the whole entry used.

For instance:

My is: 0x012345678, this means that it was xorred with the entry in the CRC table that starts with 0x12. When you xor the hash with that full entry the only thing that the next byte was xorred with was the start of a table entry too.

When reversing the current hash you know what will be xorred on the patch you’ll give. Xorring this with your wanted hash is enough.

The resulting algorithm is suprisingly simple:

– Put the current hash byte wise reversed at the start of a buffer. Put the wanted hash byte wise reversed at the end of the current hash in the same buffer.
– Look up the entry in the table that starts with byte 7 in the buffer. Xor this value of position 4, and Xor the entry number on position 3. Repeat this 4 times with the positions each time one decremented. (thus 7,6,5,4 and 4,3,2,1 and 3,2,1,0)

When you’ve done this the required patch bytes are the first 4 bytes in the buffer.

Some Crc variants tend to use duplicates in the crc-table which means there could be more than one original table entry used. You should just branch and try all of them.

I’ve made a simple python script to work with crc32 and to patch a hash.

You can download it Here. And there is a C implementation here.

Update Fixed a bug in the example.

Update Andrea Pannitti wrote an implementation in java.

Update I found a great article by Stigge, Plï¿½tz, Mï¿½ller and Redlich about reversing CRC. They pretty much nailed it by bringing it down to elementary algebra. They explain an algorithm which is able to patch data at any given point to adjust the CRC to any desired value.

GMail invites

Mon, 11 Jul 2005 17:17:42 +0100

It seems that the place to get a gmail invite has been closed down: isnoop.net/gmail.

I haven’t got a place anymore to send my gmail invites so I have them spare again.

If you want a gmail invite, just mail to bas.westerbaan [curly thing] gmail.com.

Linux Mount Security

Sun, 10 Jul 2005 11:07:53 +0100

With the linux Set UID Attribute you can let the owner of the file be the one the execute it when another user executes the file. This feature has traditionaly be used for system tools in linux which require root access to run but also must be able to be run my users.

It came to mind that a floppy with the ext2 filesystem could contain files of the root user with this Set UID Attribute set. Which theoraticly would allow anyone who is allowed to mount floppy’s or other media with an filesystem that supports this attribute to gain root access for a program.

On my system I got this entry in my /mnt/fstab, which allows an user to mount the floppy:

/dev/floppy/0 /mnt/floppy auto noauto,user 0 0

I made a simple C program which would show the contents of /etc/shadow, which contains the password hashes of the users, and chmodded it accordingly. (chmod = showshadow; chmod go+x showshadow; chmod u+rs showshadow)

I ran my program, and it seemed to work! The contents of the /etc/shadow file was streaming on my console.

Euforicly I went to another linux computer and tried the same trick.

darkshines@darkshines-one /mnt/floppy $ ./showshadow bash: ./showshadow: Permission denied

Dissapointed but releived it seemed that linux had already some precaution against a root Set UID-ed executable.

I copied the contents of the folder whilest preserving permissions to another folder outside the /mnt/floppy and it all seemed to work again, although I couldn’t do it with a normal user account for I can’t preserve the owner when copying a file as a normal user.

I wondered how linux would secure it and tried to run the program while it was unchmodded.

darkshines@darkshines-one /mnt/floppy $ ./showshadow.unchmodded bash: ./showshadow.unchmodded: Permission denied

The warning is from bash which can’t seem to execute the program. (note that it isn’t the program that can’t acces shadow) . After recompiling it on the floppy itself it seems that linux prevents any program to be executed in an user mounted folder.

I recon that that security precaution is a bit too strict. Although copying the file from the medium to a normal folder and then executing is still possible, I find it a bit strange that nothing of the user itself can be executed.

This could result in trouble when dealing with netmounts where one user can has a share on a server where he puts his files and can access only that mount for space on a terminal, when dealing with an user mount which would be required for security.

London Bombings

Thu, 07 Jul 2005 21:25:48 +0100

One thing that amazed me about the bombings is that the people didn’t panic, but remained calm.

Even though the people of London and the rest of England were hurt, they weren’t terrorized.

Although it might sound harsh, the terrorists have lost this fight: they gained nothing by these bombings.

Update: although it is s a bit inconvenient, this post is my 100th blog post. And currently the 100th comment has been made too.

Welcome to 213.133.112.101

Wed, 06 Jul 2005 00:21:11 +0100

When you read this, this page has been served from my virtual server.

This also means that luckily the whole web content transfer has been a success :-D.

Sadly things haven’t gone as smoothly with the qmail configuration for my mail accounts.

So please mail me to bas.westerbaan@gmail.com instead of my @w-nz.com acccount.

Even though the minor setback with the mail (which is giving me headaches), it has been rather fun to do. And I learned a lot. (like that putting a restriction on the amount of memory that php uses isn’t just to annoy the customer but just to safe the server from some terribly php scripts)

When everything is working fine I`ll post some more on setting up your own server.

I desperately need some sleep.

Update: seems that there are quite some bugs in the transfer after all: ftp which only recursed a certain amount of times; .htaccess files not being transfered; permissions changing; php sucking too much memory (again); not enough sleep 😛
But seems to be working right now

213.133.112.101

Mon, 04 Jul 2005 21:50:53 +0100

That’s the Ip of the vserver I have bought from greenT, and to which this site will be transfered within the next week.

There will certainly be some downtime.

update: I have installed most of the software, now all that remains is transfering.

update 2: Xr12 transfered

rel=”nofollow”

Sat, 02 Jul 2005 22:51:07 +0100

Quite some time ago google started honouring the rel="nofollow" attribute value pair in a html tags, which should prevent spammers from gaining a high page rank by spamming blogs with comments.

It is useless.

Spamming costs almost nothing, if there is a slightest amount of gain in it for the spammers they will keep doing it. Of all those hundred thousands of people visiting blogs and seeing comment spam a few will still follow the link, those few are enough for the spammers.

One could argue that the websites the comment spam point to now haven’t got a really high pagerank anymore. This is only partially true, for only a selective amount of people install the code to add the rel="nofollow" attribute in links that can be spammed. Even though the highest ranking blogs have already installed it, the tons of small blogs are still enough to raise the page rank enourmously.

In my opinion search engines should just ignore links which are considered spam.

Function Recursion Overhead

Fri, 01 Jul 2005 21:22:39 +0100

In courses I followed at university and in most books about programming, recursion is praised as a good way to solve some problems.

An example of recursion (not-existing-api used for the sake of simplicity):

int GetFolderSize(Folder folder) { int sum = 0; foreach(File file in folder.files) { sum += file.Size; } foreach(Folder subFolder in folder.folders) { sum += GetFolderSize(subFolder); } return sum; }

This example function calculates the combined size of all files in a folder and those inside any subfolder.

It does its job and it is very clear how it works, but it is inefficient, take another way to write the algorithm:

int GetFolderSize(Folder folder) { Stack<Folder> stack = new Stack<Folder>(); int sum = 0; stack.Push(folder); while(stack.Count > 0) { Folder folder = stack.Pop(); foreach(File file in folder.files) { sum += file.Size; } foreach(Folder subFolder in folder.folders) { stack.Push(subFolder); } } return sum; }

This version is harder to understand. Basicly it maintains a stack of folder of which the total size of the files in it are not yet calculated. While going through each of the folders in the stack it adds new sub-folders from folders and removes the ones processed.

The latter method is way more efficient.

For each function function call in the first (recursive) version of the algoritm a new sum-instance, a subFolder-instance and an enumerator instance over the folder.Folders is required which stacks up. When 5 deep in the first algorithm you already use more than double of the amount of memory ever required in the second algorithm.

Additionally a function call itself requires more memory of its own, which depending on the language used, can be quite significant. Debugging also gets really hard when you have got a stack trace of hundreds of functions.

Using your own stack instead of poluting the call stack (the thing where function calls are kept) sound great. There is only one little problem, it can get pretty complex.

Take for instance a program that would put the files of folders and their subfolders in a file in that same folder, for instance for a playlist generater. This requires the algorithm to detect when all child items of a certain item have been processed. The problem with this is that the parent item is already gone from the stack when the child item is processed. This requires some additional fields and a few function pointers to get it to work and it can get a mess.

The best way to get it to work is to mimic what a normal recursive method would have done, which involves a lot of extra helper classes, which all depend on how the original recursive algorithm has worked. In a language with a Garbage Collector (which automaticly free’s unused classes) it is managable, but in a language without it like C(++) trouble doubles when you also need to monitor the lifetime of the helper classes.

I noticed that it has been very tempting to use recursion and that there are almost no occasions where something like your own stack is used, espacially in the complex cases. A shame, it is challenging :-P.

IEnumerable to IEnumerable<> wrapper

Thu, 30 Jun 2005 23:09:51 +0100

A simple .net class which maps an IEnumerator to a generic IEnumerator, which is very usefull to convert from generic to non-generic collections.

using System;
using System.Collections;
using System.Collections.Generic;

namespace Intrepid.Collections.Generic
{
	public class EnumerableWrapper<T> : IEnumerable<T>
	{
		class EnumeratorWrapper : IEnumerator<T>
		{
			IEnumerator enumerator;
			
			public EnumeratorWrapper(IEnumerator enumerator)
			{
				this.enumerator = enumerator;
			}
			
			public void Dispose()
			{
				// maybe we should dispose the underlying IEnumerable?
			}
			
			public bool MoveNext()
			{
				return enumerator.MoveNext();
			}
			
			public void Reset()
			{
				enumerator.Reset();
			}
			
			object System.Collections.IEnumerator.Current
			{
				get { return enumerator.Current; }
			}
			
			public T Current
			{
				get { return (T)enumerator.Current; }
			}
		}
		
		IEnumerator enumerator;
		
		public EnumerableWrapper (IEnumerable enumerable)
		{
			enumerator = enumerable.GetEnumerator();	
		}
		
		public IEnumerator<T> GetEnumerator()
		{
			return new EnumeratorWrapper(enumerator);
		}
		
		IEnumerator System.Collections.IEnumerable.GetEnumerator()
		{
			return enumerator;
		}
	}
}

Bye Bye Spam

Sat, 25 Jun 2005 23:19:54 +0100

I just installed Hash Cash, which is an anti spam plugin for WordPress.

Hash Cash protects this blog from spam by requiring the client to execute javascript which calculates a checksum of the content from a seed which is very hard to extract.

Since I installed it I haven’t got any spam comments :-).

The downside is that it disallows anyone who hasn’t got a javascript enabled browser to post a comment.

Now I still need to get some good means to combat trackback spam. Just putting them under moderation isn’t good enough for they keep coming

Shiftings in Architectures

Sat, 25 Jun 2005 15:06:59 +0100

Cell, the hype

Sony is working on a new kind of processor which they call Cell. A single Cell processor is said to have a 1/75th of the power of the Blue Gene/L supercomputer in teraflops, and that for 400 dollars.

The new Cell processor is hyped, a lot. People are shouting that the performance of the PlayStation 3, which will feature a Cell processor, will blow away those of the competers: the XBox and the Revolution, both using claimed to be slow PowerPC processors.

Teraflops

So what are these Teraflops? Why aren’t they talking about Hertzes anymore?

A TeraFLOPS stands for 1,000,000,000,000 Floating point Operations Per Second. (TeraFLOPS on wikipedia).

A normal computer has a few GigaFLOPS (one TeraFLOPS is 1000 GigaFLOPS).

The Cell processor in the PS3 will have 1,8 TeraFLOPS. It is very easy to believe that the PS3 will inheritly be about 500 times faster than a normal computer. But that is false.

The performance of a computer isn’t all about how fast a computer can calculate with floating point numbers. The performance of a computer has got to do with a lot more than that. Simply doubling the memory performance (not specificly memory bandwidth, but the whole) would have a bigger impact than doubling the amount of TeraFLOPS, even on applications that heavily rely on floating point operations.

So why have a lot of TeraFLOPS then? It is quite simple, the new platforms haven’t got seperate GPU’s (Graphical Processing Units), their task is now intergrated into the CPU. And what was their task? Calculating, a lot, with floating points.
The amount of TeraFLOPS on the new platforms are the sum of the amount of TeraFLOPS of both the GPU and the CPU. A GPU needs to calculate a lot with numbers for 3d rendering.

Your normal computer has got a lot more than those few GigaFLOPS in your CPU in your GPU already. Although the leap to over the one TeraFLOPS is certainly impressive.

Marketing

Super Computers need to do a lot of FLOPS too, they calculate models which got a lot of floating points. So why put the Cell processor in the Ps3 instead of directly in some supercomputer? Simple, marketing. Almost everyone that is superficially following the computer news is telling me over and over again that the Cell processor is the most incredible thing there is. It is hyped.

Linux on Ps3

Also the Cell processor uses a whole new architecture which still has to be tested a lot to mature. Sony will let you install Linux on your Ps3? Why? Simple, because they want the Linux community to add decent support for the Cell architecture.

New Architectures & CLI

I guess it will be a matter of time before new architectures will come. A problem with a lot new architectures is that there isn’t a lot of support for them yet. The solution? CLI’s like Java and .Net could bring the solution.

Microsoft rumoured to have MSIL running (semi)natively on their XBox360 and making Longhorn more and more rely on .Net (the ontop applications, not the kernel offcourse) means a lot less architecture dependency.

I wonder what will happen, but one thing is sure.. things are shifting.

Python Url Encoding

Sat, 25 Jun 2005 00:00:20 +0100

I looked in the Python Library Reference for a function to encode special characters in strings to be able to be put in Urls (and Uri’s), but found none.

Maybe I missed it?

Anyways, here is an implementation I wrote based on this article:

HexCharacters = "0123456789abcdef"

def UrlEncode(s):
 r = ''
 for c in s:
 o = ord(c)
 if (o >= 48 and o <= 57) or \
 (o >= 97 and o <= 122) or \
 (o >= 65 and o <= 90) or \
 o == 36 or o == 45 or o == 95 or \
 o == 46 or o == 43 or o == 33 or \
 o == 42 or o == 39 or o == 40 or \
 o == 41 or o == 44:
 r += c
 else:
 r += '%' + CleanCharHex(c)
 return r

def CleanCharHex(c):
 o = ord(c)
 r = HexCharacters[o / 16]
 r += HexCharacters[o % 16]
 return r

note: I used the character numbers instead the characters to compare with so I could do greater than and lesser than for the alphanumeric numbers. Maybe testing with a bitmask would be more efficient.

I have to write almost everytime I work with python my own something to hex function which doesn’t add the ‘0x’ in front the built-in hex does.

Either I can’t search or Python hasn’t got enough batteries included. I guess the first, if not I`ll submit my batteries.

Mono gone generic

Fri, 17 Jun 2005 23:42:30 +0100

Mono 1.1.8, released yesterday, has implemented the .net2 System.Collections.Generic namespace 😀

This hasn’t come a moment too early for microsoft will be launching .net2 soon.

BSD

Fri, 17 Jun 2005 23:38:06 +0100

BSD is Unix.. said to be the professional cousin of Linux.

This piece of propoganda for BSD against linux gives as reason that linux is bad for it is told to be just hacked together. This because of the people who develop linux are just people from the community who put a little bit of time in it to get a feature (they probably want) added into Linux and don’t really concern about making it perfect, was said.

They took some of the sarcastic ‘todo’ comments in the kernel as example, blaming that if that stuff is in the kernel linux can’t be trusted at all.

But why does BSD hasn’t got the widespread hardware support Linux has? They blame the big company’s like IBM for instance. I just wonder whether their 60 dedicated BSD programmers could code all the hardware drivers that the thousands of contributers of Linux have coded, even if it were as bad as it is now according to them.

I bet that *BSD has got more than enough of those comments in their code too. If they haven’t they are just hiding the truth and hiding points of improvement, for these TODO’s and FIXME’s are fixed in the end. And even if they get rid of all the TODO’s and FIXME’s before they release any of it they waste a lot of time it could have been used already (less efficiently though.. but it usualy doesn’t make such a big difference).

Services, a different approach

Mon, 06 Jun 2005 20:38:17 +0100

A Server is a computer (or a group of them) that offers services.

Such services could be a webmarket, a website, a game server and so on.

These services tend all to be implemented quite differently, hardly interacting at all. All these services require a different approach, a different way to manage them. This creates both overhead and chaos.

I`m working on a small framework intented to host services which is very flexible.

Framework
A service isn’t a single process as it usualy is, although it could be put in its own seperate process. A service is rather just an instance of its servervice type in the framework somewhere, meaning you can have multiple instances of one service with different configurations scattered accross processes and even machines as you like.

You can edit the settings and manage the services using the same API (and likely the same configuration tool).

A service isn’t aware where it is hosted, from its perspective it is just in a framework with other services and service modules with which it can interact in always the same way.

You as administrator however are aware of the services and its allocation. You can allocate a service (even runtime) to different locations which haven’t neccessarily got to be processes, but also can be lightweight processes found in .net like appdomains or on different machine’s processes.

Having different processes for each service decreases performance but increases stability, and visa versa.

Interaction
Services shouldn’t be restricted to their own process but should also be able to interact with other services by using their exposed interfaces.

An example could be a webserver-service which you provide with the name of the logger-service you want to be used for that webserver. The webserver could just interact by casting an object in the public interface dictionary of the logger to an log interface. The framework takes care of the required underlying communication like the remoting in case they are hosted on different processes or even different machines.

A more tighter type of interaction would be modules. A module of a webserver could be a webmarker. The webmarket would just access the object of the webserver which handles hooking to certain URI, add the hooks and await a visit. Other services could do the same but a module is specificly bound to the service it attached to.

Futhermore modules would be an easier way for administrators to manage extra functionality which is more module like than filling in the instance paths to use in the settings of a certain service.

Portability
I`m still in doubt whether I should make it cross-language. It would be a good thing but it would certainly create havoc to make everything run smooth and alike on all languages. Not even to mention the overhead to load each languages’ runtime.

Power PC dumped

Mon, 06 Jun 2005 07:12:57 +0100

Apple to ditch IBM, switch to Intel chips

When Apple will switch to x86 and ia64 it will be possible to run mac osX on your own home bought computer, given that they don’t avoid it by changing some opcodes.

One other things is sure; they will loose some power. As long as they provide binaries and provide one set for all apple computers they will have to add support for both ppc and x86 in one binary distribution which will undoubtely be a lot bigger if not slower too.

It will be interesting to see how this issue will evolve.

Subversion

Sat, 04 Jun 2005 23:41:38 +0100

I’ve recently been using CVS a lot, more specificly subversion. It makes it a lot easier to share source code. Where it wasn’t possible to work on the same projects or even on another project that depends on the first because sharing source code is tricky, subversion was the solution.

mount..?
As far as I know there isn’t a program that allows you to mount a subversion repositry into the linux filesystem. This would make managing a subversion reposity a bit easier for at the moment you need to use svn add to add every single file; svn cp to copy, svn rm to remove, and so on which could all be intergrated when wrapping a repositry up.

It is possible I haven’t been searching in the right place. Searching for it just gave me an enourmous list of other people graving for it.

pruning
To my knowladge there is no really easy way to get rid of older versions. Offcourse it would be against the basic idea behind the system storing every single version, but sometimes it just isn’t practicly. In case you are dealing with extra not sourcecode files, like images or data files, which usualy are quite large in comparison with source code, you’ll get a really big and slow repositry.

Adding a feature that will store all files before version x if it isn’t the top version in a compressed archive of some type would be really nice. Although it would drasticly decrease the access time for the older versions it does decrease the space used and increases the access time for the top versions which are used a lot more than the older ones.

Opeth Ghost Reveries tour

Sat, 04 Jun 2005 23:24:10 +0100

I just bought a ticket for an Opeth performance in the Netherlands (013, Tilburg) on 11 september 2005. I’m looking forward to the performance even more, for Opeth really seems to be a great live band. I just bought their Lamentations DVD, which was quite amazing.

XBox 360

Fri, 20 May 2005 19:55:12 +0100

Microsoft released the specifications of the XBox 360..
it is a beast..

With 3 CPU cores running on 3.2Ghz each with 2 hardware threads and 1 mb l2-cache each it is a powerhouse.

A remarkable feature of the XBox 360 is that it has got processor level support for MSIL (the bytecode used by .net applications).

The best thing of the XBox 360 is that it’ll be quite cheap (~150$), this probably lets the XBox360 deliver the most processor capability per dollar.

Maybe I’ll buy a few and make my own little linux server cluster with it, when the guys behind xbox-linux.org have added support for the XBox360 :-).

Music Fill Out Thingy

Thu, 19 May 2005 14:47:54 +0100

Zef handed me the music baton, meaning I got to fill out a form ’bout music. So well… here it is:

Total volume of music files on my computer
3.2 GB.. that isn’t a lot.. but in contrary to some with huge collections I do listen to them all.

The last CD I bought was
I got myself some albums of Children of Bodom recently. I`m trying to get the Lamentations DVD of Opeth.

Song playing right now
Opeth – Still Life – The Moor

Five songs I listen to a lot/mean a lot to me
I don’t really listen to a certain song a lot.. although recently I listened to these a bit more than the others:

– Opeth – The Drapery Falls
– Children of Bodom – Needled 24/7
– Nirvana – You Know You’re Right
– Opeth – Dirge for November
– Muse – Citizen Erased

Lets be creative and add another question:

Artists you like
Opeth, Children of Bodom, Nightwish, Nirvana, Linkin Park and last but not least Muse.

The five people where this meme will go on
– Kaja Fumei
– Noud Aldenhoven
– Gumuz
– you!
– (yes, i was out of idea`s)

Safe web authentication

Mon, 16 May 2005 15:52:55 +0100

The major problem with security of web applications is that the client sends the login name and password in plain text if https isn’t available. A nasty person with access to the network could use ARP poisening alongside packet sniffing to acquire the login, which wouldn’t really be desirable.

I stumbled accross a very interesting piece javascript which implements the md5 hash algorithm: http://pajhome.org.uk/crypt/md5/.

Using a hash makes it impossible to reverse engineer a password and makes authentication safer. An issue with this is that you only require the hash, not the password to get in. To prevent this the password should be salted before hashed.

Basicly a secure authentication via http would look like this:

Client sends request for login to server.
Server sends the login form which includes a login id and salt to the client.
Server stores the login id and salt it sent to the client.
Client sends the hash of the filled out password and received hash alongside the login id from the server to the server.
Server checks whether the hash of the password in the database and the received hash combined with the login id are valid.
Server sends whether authentication was a success.

Maybe I’ll implement an example application :-). In any case I hope that this will be employed.

Update, most authentication system used by webbased software are still vulnerable and would almost neglect the use of this by being able to hijack a session by just getting the session key. The client however could also implement javascript to use a similar method with a salt to protect the session key. The problem still is that it is extra overhead on the client and that not every client has got javascript enabled.

Assembly ‘programmers’ suck

Sat, 14 May 2005 21:49:28 +0100

Some programmers claim writing assembly is the solution to every single programming issue, for it would be great for being down to the basics, or it creates small programs, or even be fast. For all I care they are just trying to brag for not a lot of people know how to program in assembly and it is generally seen as difficult to do.

Programming in assembly is almost everytime a bad idea.

So first, what exactly is assembly?
Assembly is a text language with which you can write opcodes and arguments of machine code processed by the processor by hand. This gives you a very high amount of control of what the processor does. Whereas higher languages generate the machine code for you as they deem it to fit you can choose which machine code would fit.

Assembly isn’t difficult to learn. Assembly is very straight forward. It just is hard to program for you don’t have functions, you don’t have high level structures, you don’t have all the aid of a high level languages.

The reason programmers claim that assembly is superior is for with assembly you can write faster and smaller code. This is only partially true.

When you know all opcodes and tricks you can pull on all different architectures you can beat a compiler.. The problem here is that there are tens of architectures with totally different opcodes and even more subarchitectures with each a slightly different implementation and different performance characteristics of each opcode or way to do something.

So to truely beat a compiler which compiles one piece of code to assembly, you have to create for each different architecture a seperate piece of assembly source.

You’d also have to learn all the opcodes of one cpu, that aren’t just a few hundred which would suffice to get it working, but thousands which are required to get it working as fast as possible.

Compilers know all these extra opcodes and tricks for each architecture and would therefore a higher level programmer would do a better job on creating an executable than an experienced assembly programmer in the same amount of time. If the assembly programmer would want to beat the higher level programmer he would require not just 2 times more time but at least 10 times.

Also assembly isn’t very portable. If you want to change one little thing you got to change it for all optimalizations for all different processors. I haven’t seen a big application written in assembly. And that has got a reason.

Most programmers that claim to be able to write quick assembly don’t have got an idea how everything works exactly and are just thrilled that they made a little program work in assembly.

Assembly though can be usefull. An example is an arbitrary length integer class like GMP which uses optimized assembly for certain operations. It aren’t a lot of operations done in assembly, but it certainly has got a lot of assembly. And it is worth it.

Sticking with a good compiler and a high low level language like C is always the best option for good applications.

Gentoo Linux

Fri, 06 May 2005 22:06:52 +0100

3 days ago I came accross gentoo linux.

Gentoo Linux is a linux redist that provides you with a livecd from which you are able to build a whole new linux installation practicly from scratch.

Yes, that means compiling everything yourself.

Normally compiling means reading through tons of documentation, trying to find dependencies, trying to get stuff working, making tweaks, and reading again through tons of documentation, trying to find dependencies, hoping version won’t collide, and again, and again.

Luckily gentoo provides a very neat tool called portage.

Portage allows you to simply install for instance gnome by just typing:

emerge gnome

Portage will look up which dependencies are required, download them, configure them, compile them, and clean up garbage. The only thing you got to do is hang back and relax.

Portage also automaticly implements your preferenced optimalization settings to create the best possible build for every single application you compile.

Gentoo seems to have resulted in a lot faster linux installation, although there is one little issue, it takes time.. A lot of time…

Compiling gnome (with all dependencies) took about 8 hours…

Maybe gentoo will be the first linux to kick windows from my desktop :-).

Extending using Wrapping

Sat, 30 Apr 2005 01:11:02 +0100

Ever had a problem, not being able to add more functionality to an existing class? Some languages seem to support adding functionality to an existing class but it does make it less secure and leaves a lot of dangers of maluse.

A solution could be to create a new class which inherits the original class although it can only be instanciated using a cast from an instance of the original class or instanciating providing the original instance. Also it won’t be able to override anything otherwise it couldn’t act as its base object.

A possible implementation of this idea could be:

wrapper Int_IsOdd : int {
 public bool IsOdd() {
 return this % 2;
 }
}

And it could be used in several ways:

int myInt = 100;
print ((Int_IsOdd)myInt).IsOdd();
print myInt.IsOdd();
print new Int_IsOdd(myInt).IsOdd();

The first way would be the easiest for both a reader which isn’t familiar with the extension and for the compiler to guess errors. The second would be neat to use but it would be confusing for people that cannot access the rest of the source or do not know what function is meant. The third way would be very compatible.

I wondered by myself whether normal inheritance could provide this functionality.. and it does. The only problem is that polyforism only works up and not down as the wrapper requires. (when you got a class a and b inherits of a, you can with inheritance treat an instance of b as a but you cant treat an instance of a as b which is required for the wrapper)

Also a wrapper would be more cached and not really a new instance as suggested by the way it seems to work and can already be slightly implemented in existing languages.

Maybe it’ll be put in paradox.

update – I talked with Kaja, and it will most likely be a feature of paradox

It’s all about the model

Sat, 23 Apr 2005 20:23:21 +0100

With all those strange new physics theories and models which seem to conflict eachother it seems all rather too awquard to be true.

Like Quantum Mechanics stating that something can be two things at the same time, which seems impossible, but certainly is when keeping in mind that Quantum Mechanics states that something is what it seems to be when measured where measurement always has to interact which means that very unstable states of an object don’t exist but still can be applied on an object being two states at the same time with a third set of properties… great…

What people don’t seem to realize is that most physics theories apparently conflict for they are formulated rather awquard, but only formulated like that for then they are reasonably understandable as a model. Also one given theory isn’t superior above another, usualy one theory just stresses a certain part of apparent reality more by the way it represents everything.

There just isn’t the theory that explains how the universe works, there are only models covering a part in a certain way.

I don’t care if someone likes to represents an atom as a little UFO, as long as the conclusions and predictions in that theory are usefull in being able to predict behaviour and characteristics.

This is also where philosophie makes a fatal mistake, a philosopher would tend to use metaphores, scientificly called models, literaly assuming that reality is just a metaphore. And there ishardly anything you can do to persue them otherwise anyways :-).

Apparently it seems, assuming the string theory is an accurate model, that we only experience an infinite small amount of the whole universe. It maybe could be seen that we are just a little (end) fractal in a big fractal (inside another).

But why are we as we are? It would be logical to assume that we are nothing more than a logical result of one chaos like formula for the whole universe seems to be pseude randomly as many chaos formula’s seem. We’re just one of the infinite possibilities (stressing possibilities).

That would seem rather depressing and our lives would seem rather useless.. and actually.. they are rather useless for the whole static everything. But actually that makes just one more reason to maintain our human species for otherwise we couldn’t enjoy our uselessness :-).

And even if we would have to formula to predict everything (which maybe even has been found already by some mathematician, but is only experienced differently by us as models differ from eachother), we would only be interested in derived models for our perception of it to aid our uselessness :-).

Fun with hex

Fri, 22 Apr 2005 22:50:02 +0100

Yes, I was bored.

(and no, I did not do that by hand)

Linux style

Wed, 20 Apr 2005 15:57:35 +0100

Make your own tux toy:

They only redistribute source branches ^_^

You got to config and build it manually..

Foolish people

Tue, 19 Apr 2005 21:44:48 +0100

You can buy World of Warcraft gold and items by paying real money…

It actually seems they are having quite good bussiness for the amount of adds linking to them has increased IMHO.

So.. there must be a lot of people buying basicly nothing for damned high prices…

How can people be so foolish?

Pope

Tue, 19 Apr 2005 21:37:32 +0100

(dutch joke)

Our brand new german pope: “Bedankt voor de stranden”

Note on logo

Fri, 15 Apr 2005 23:42:52 +0100

I found some time to change the default logo of the wordpress theme to something I liked a bit more. I guess I`ll have to do a better layout for the website once but it takes too damned much time.

Anyways, I was required to change both the CSS as obviously the Image, so press refresh a few times if it doesn’t show up properly.

(the logo itself is a part of a ‘o’, yeah.. the letter ‘o’… of the word opeth.. that opeth:
)

The Linux Paradox

Fri, 15 Apr 2005 20:58:00 +0100

Linux wants to get popular. Everyone using linux wants linux to grow bigger and bigger to ultimately have linux be the operating system dominating. err.. correction.. independant distributions and inheritances of linux to avoid the nasty monopoly stuff. Even your mom should start using it.

Sounds good. Even if linux is in some people`s opinion not the best OS it still has the best community backing it and is the most secure OS. It also has got the most support from the community. If something is required or has to be fixed it`ll happen in notime.

So why doesn’t everyone use it already?

The problem is that linux demands too much from the user.

A linux user should be able to at least have a grip of how a console works. And preferably has to know how to compile application and fix bugs when it won’t compile. And a user should know how to get dependencies it misses. An user should read through the fast different documentation of each application to even begin to hope to get it running easily. And an user should start over every time a new version is released.

Sure, they got RPM’s and yumm which makes life a lot easier. But the yum repositries don’t contain everything. If you want some new stuff, or an application with a certain module installed you have to go dig into configuration files, master the console, read through documentation, compile, and offcourse check your version dependencies, etc.

The average windows user like your mom doesn’t even know how drives work and certainly have never heard about the command line. It just wouldn’t work out. They got a lot of trouble even with just one or two options. When confronted with linux with hundreds of possible options and even stranger errors it just won’t work out well between them.

It certainly is possible to stick to the main applications provided by your redistribution and using the yum repositry. But linux will never run as satisfying as when you get it all compiled yourself and configured yourself. This really makes a big difference! I made linux start up a lot faster by compiling the kernel with only the modules I require with computer specific optimalizations. These little things give linux a really big edge over windows.

Linux is and remains only interested for adventerous ‘power’ users with quite some free time on their hands.

Don’t rely on parsing

Wed, 13 Apr 2005 22:09:47 +0100

Most applications store their settings (espacially in *nix) in text configuration files. These text files need to be parsed every time the application starts.

Parsing is the action of (usualy streaming) dividing a certain piece of data into understandable parts. This usualy comes down to looking at the text file character by character and deciding what should be done. Usualy this is done by maintaining a state which contains the data collected sofar. And with a bit more complicated files this even means having a stack of states and complicated actions when a certain state is left.

The problems:

Parsing is slow, very slow
Formats required to be parsed contain overhead, a lot of overhead

But it certainly has got advantages:

Humans can easily edit it, you don’t need to rely on configuration tools
(Usualy) makes a configuration format more extensible by nature (adding one new field in the average programmer’s binary format would break it)

Now, there are attempts to help improve speed. This by standardizing the format, which makes the amount of oddities to expect less, which ultimately makes the whole parsing slightly faster. This at cost of the easiness it can be edited.

A good example would be Xml. Xml is damned ugly. Xml is too strict. And Xml still takes a hell of a lot of time to parse.

Yaml looked like a decent alternative: easy to edit, looks nice. But then I encountered this:

%YAML 1.1 --- !!map { ? !!str "sequence" : !!seq [ !!str "one", !!str "two" ], ? !!str "mapping" : !!map { ? !!str "sky" : !!str "blue", ? !!str "sea" : !!str "green", } }

Ugly…

So what to use instead?

Use binary configuration files, which are easy to load and save for the application. And create a parser to parse the configuration file and save it to the binary format! In other words: serialize the usefull data from the parsed document and only parse again when it is required.

When you only parse stuff when it has changed by the user than it doesn’t really matter how long it takes to parse. Which can get rid of the really ugly stuff and let us just have a very loose kind of format without the ugly rules and regulations.

‘Objects’ in C – Part 3

Tue, 05 Apr 2005 20:36:15 +0100

In Part 3 of this mini serie I`ll discuss how interfaces / (multi)inheritance like behaviour can be achieved in a language like C as in C++.

In the posts before (part 1, part 2) I discussed the possibility to inherit behaviour of an ‘object’ in C by adding a type pointer which handles initialization and destruction of the object; achieving the possibility to extend an ‘object’ by adding fields at the end and updating the type pointer to a new type.

Any object contains a pointer at the start to its type. The type of an object is passed to the CreateInstance function when an instance needs to be created of the object. The type itself just contains a few fields:

Length, the required length of the object, this allows an inheritance to extend the amount of data.
Constructor Function Pointer, the Constructor Function Pointer is called after the object is allocated to construct its contents.
Destructor Function Pointer, the Destructor Function Pointer is called before the object is freed to destruct its contents.

These fields, along with using function pointers in the instance itself instead of functions, allow an inherited type to expand the existing type. The problem is that an object can’t inherit from more than one object, for both object’s expect to start just after the pointer to the type. To solve this an additional field needs to be added:

QueryInterface Function Pointer, the QueryInterface Function Pointer can be used to get an instance of a supported type from the instance called.
Top Object Pointer, the Top pointer points to the top object. (An object returned from QueryInterface its Top pointer would point to the object where QueryInterface was called)

nb. The latter one will be added after the type pointer in the instance instead of in the type

A good example would be a type called ICloneable which itself only contains one field: Clone, a functionpointer to a function to clone the object. When you would want to clone an object given you would use:

ICloneable* interface = toClone->top->type->QueryInterface(toClone->top->type, toClone->top, ICloneableType); Object* clone = interface->Clone(interface);

Lets take a look to the first line:
ICloneable* interface = toClone->top->type->QueryInterface(toClone->top->type, toClone->top, ICloneableType);
This line queries the interface that the object that will be cloned wants you to use to clone it. The top object is used for that is the only way to be certain that you are dealing with the top object and the latest code instead of some base. Using this code an object can be cloned by just providing its interface!

(the QueryInterface takes 3 parameters: first one is the type itself where the function pointer relies, second is the object in question, third is the requested type where ICloneableType is an instance of the ICloneable type).

Object* clone = interface->Clone(interface);
The second line just calls the functionpointer in the interface queried.

To ensure that every bit of behaviour can be overriden, the object queried using ->top->type->QueryInterface on the provided object must be used instead of the provided object itself for the provided object can be a whole different object merely supporting that interface secondarily.

The nice thing about this architecture is that a top object merely wraps its implemented objects and returns them (possibly addapted) when queried for it. This allows way more access to the inherited types than in any other object orientated model.

This model is quite easy to implement in C and (I hope) will be easy to intergrate with for instance objects in a Virtual Machine.

How to GC in C(++)

Sun, 03 Apr 2005 22:51:04 +0100

GC`s, Garbage Collectors, are systems to manage memory. Even the good-old C-runtime library has its GC. free, malloc and the others. Now, these aren’t the best memory managers there are. They cause fragmented memory, and scatter the memory of your application over the whole memory rendering the great processor l1 and l2 cache almost useless.

I won’t talk about how to use the C ‘GC’, but how to implement new-generation alike Garbage Collectors in C(++).

The best GC`s can be found in Intermediate Language Virtual Machines which keep track of every object`s type in the memory and can therefore freely move objects in the memory. The great advantage of this is that object that have survived about as much Garbage Collects tend to link a lot to eachother and use eachother. That group survivors is called a generation. When you put a generation close to eachother in the memory, it will usualy only take 1 RAM memory load for the CPU, caching the whole generation in the CPU cache, for a while which is a lot quicker considering that l2-cache is about a thousand times faster than normal RAM.

The problem in C(++) is that you can’t just move objects. You don’t know what part of the memory of an object is a pointer, and what is just a mere integer. Therefore it is impossible to make a generation based Garbage Collector for you just can’t move stuff.

Allocating a big chunks and putting the objects of the C(++) application however using a custom allocation function will generate some additional performance above the traditional malloc although it still isn’t perfect.

One way to get it to work is to let the GC know what a pointer is and what now. This can be done by letting the first 4 (or 8 bytes in case of a 64bit CPU) be a pointer to a function which returns an array of offsets which resemble the pointers in the type.

Now the GC knows what the pointers are in a structure :-D.

The GC can now move the object by updating the pointers from the other objects to the new location of the object!

The only problem with this is that there can’t be pointers to the object, or inside the object from a not referenced pointer exposing object or a object that doesn’t exposes it pointers at all.

To overcome this I guess it could be feasable to add a flag byte in each object that exposes it pointers which allows it to specify a function that can be called when moving the object or when an object referenced is moved.

I’ve tried out some ways to get this all wrapped up in a nice framework (which is very feasable using my ‘objects in c-framework’ or something similar (easier) in C++ using inheritance).

I`m afraid however that such a GC comes too late for by the time it is reliable the Intermediate Languages would have gained way more ground for these can implement way more optimalizations, including in the GC, during runtime for they tend to use an Ahead of Time and Just in Time compiler.

Feedback on this would be welcome :-).

Softwire

Sat, 02 Apr 2005 23:43:25 +0100

Softwire

Softwire is a library to generate assembly runtime for different kind of processors. It has a great API. Very usefull for JIT alikes.

Kaja is converting it for use in C for the Paradox JIT. Hopefully it’ll be a lot faster replacing virtual class function calls with macro’s.

Great stuff to play with 🙂

April`s first

Fri, 01 Apr 2005 00:00:03 +0100

[Insert stupid joke here]

Downtime

Mon, 28 Mar 2005 01:00:00 +0100

Due to server upgrades taking a bit more time than expected there was a little bit too much downtime.. sorry about that.

PHPavascript

Wed, 23 Mar 2005 22:46:00 +0100

Due to the recent hypes around Ajax and web development I’ve been thinking about a more effective method to write web applications running both on the server as on the client by writing code in just one environment and language instead of two: PHPavascript.

Having all kinds of fancy libraries to help you does help a lot; but you still got to manage transfering all data from the client and the server and back by hand; which espacially with javascript is quite tricky to do not only because it isn’t the easiest language to debug but also because every browser tends to do things a bit different with javascript.

It would be nice, I thought, to have a language for developing sererside and client side at the same time:

client TextBox mytextbox = TextBox.FromId("mytextbox");

int server function CalculateSum(first, second) {
 return first + second;
}

void client function ShowSum(first, second) {
 mytextbox.text = CalculateSum(first, second);
}

ShowSum(10,1000);

Basicly you mark a language entity to be either on the client or server side. The compiler would take care of the rest.

This would be really cool if it would be implemented but there would be quite a few issues:

All client side functions and variables can be forged by the user; a naive programmer could put too much trust in functions on the client side
Synchronizing variables on server and client side and executing functions on server and client side could with a suffisticated algorithm be managed pretty decently although it would still create a lot of overhead when the programmer doesn’t pay attention on the usage of his function/variable location. Having hundreds of variable transfers for one page would be a lot, although very possible when a programmer doesn’t take care or the compiler is too dumb.
Language features available on the server side and only available on the client side with dirty hacks could be a bottleneck if the programmer doesnt take enough care. How to implement a mySQL connection for instance (not considering the fact that it wouldn’t be very safe in the first place)
etc

Basicly it would be way easier to develop a web application but I don’t know what would be better: having a hard to handle situation as it is now where you are forced to think efficiently, or an easy to use environment as such a language could provide where everything works with the temptingness of not even thinking about what happens beneath the bonnet.

Although there still are these advantages:

Vastly reduced development time
Highly maintainable code; a compiler could even create seperate javascript files for each specific browser and compile to the desired server side language

Ajax, the hype

Tue, 22 Mar 2005 17:37:51 +0100

It’s new!

It’s cool!

And now it even got a name! Ajax.

(what’s ajax?)

The funny this is that it is quite old and it has been used for a long time already. It just wasn’t hyped before.

Also this phenomenon clearly illustrates the dissatisfaction with the current static Html document standard common to the world wide web. Don’t deny it: html sucks for user interfaces! Html is a format for documents notuser interfaces.

A child of Html could make a nice User Interface definition; main thing would getting rid of awquard javascript. Using a more dynamic Document Object Model, and espacially a consistant one, with some kind of Intermediate Language providing power but still security like Java, or .Net or maybe a new one, which should be hosted by the browser itself instead of a nasty plugin noone has, would be perfect.

Phalanger

Sat, 19 Mar 2005 11:44:46 +0100

Phalanger is a PHP-Compiler for the .net framework. It can be used to run existing PHP applications on asp.net webservers with far greater performance than using PHP itself:

In contrary with PHP itself the bottleneck isn’t the execution of the code itself but the underlying functions for executing the code: phalanger still uses the PHP library for the PHP functions which creates a lot of overhead due to interop and makes the PHP objects not as native to .net as it could. In contrary to PHP you would be best off not using the functions but trying to use the .net ones or your own written.

In my own little benchmark for basic differences between normal .net and PHP, PHP came out to be 5000 times slower. When phalanger finaly compiles into proper .net code by avoiding any PHP library and PHP interop it would be a -lot- faster. When people would start to like it and install mod_mono on their apache webservers to run it they would probably find that they’ll be better of with asp.net with c# or vb.net. They after all got a way cleaner syntax than PHP and are happier working with the .net framework. (I don’t want to know what hacks they used at the phalanger compiler to get include and require working when compiling everything to one dll).

In the mean time microsoft is rubbing its hands

Copy protecting

Thu, 17 Mar 2005 19:15:19 +0100

From software, audio to video are being illegaly copied and everytime the major brands try to implement some kind of protection. They always claim their protection to be perfect, and yet it is always broken, for it is quite simple:

As long as the intended user has the platform on which he`ll run it in his own possession he can always adapt it in someway to extract the data. Even the best video protection can’t beat making a bypass in your monitor to acquire the image on your screen.

Even protecting something like a DVD is almost impossible. The dvd player hardware and software must be able to read what’s on the cd, and a protection must be able to be read to. Also there must be dvd writers to write a protection. Now all major brands can say they’ll put a protection in their DVD burners to protect from writing to the DVD protection section, but then another brand creates their DVD burners that can write to it and everyone will buy those which the big brands won’t let happen. And even if they got the disk itself truly protected someone can emulate the DVD using software or even hardware.

Also there are things that allow itself to be copied, but the original copier can be tracked; this by putting in every video/song/software a unique signature which can be tracked back to the store which then can track it back to the person who copied it. Sounds great, would be impossible to forge when they use strong RSA like cryptography, just one problem, when inserting random trash instead of the signature someone can know the piece is illegal but cannot track someone, hopeless.

The only, and only way, to stop illegal copying is making buying legaly less of an effort than acquiring illegaly. I hope they will relize this sooner or later for honoustly I`m becoming sick of all those ‘magic’ protections.

Enter The Unkown: Algorithm`s A programmer Has Got To Know

Wed, 16 Mar 2005 21:51:16 +0100

I don’t like to link to other blogs or articles for I am just a lame copier, but I’d like to point my select few readers to Enter The Unkown, the weblog of Kaja Fumei, where he will point out some Algorithms a Programmer should know. First one in the list are the Hash Tables.

You don’t need to know the stuff behind these kind of features a language library exposes; but it helps a lot if you know how it works and what the weak and strong points are.

Easy Crossbrowser JavaScript

Wed, 16 Mar 2005 12:47:22 +0100

The major problem when dealing with javascript for me was that javascript acts differently on each different browser. And there are a lot of browsers supporting javascript.
Usualy to get it working it would include having for each sensitive operation a big if block. And in the a bit nicer javascripts that would become a lot. Also it becomes hard to maintain.

So what to do about it?

Actually.. C(++) gave me a possible solution. Use macro`s. It ain’t possible to use macro`s in javascript itself feasable, so what you do is you compile your javascript with macro`s to a different javascript file for each browser. Then using a simple server side script you can let the browser get what it wants.

I’m not a javascript guru, so I hardly know all the problems of each browser.

If a javascript guru does read it, please contact me – it would be great to have such a goodie.

Introducing Paradox

Sat, 12 Mar 2005 19:53:18 +0100

Kaja Fumei and I are currently developing a light-weight rich jit-ed language called Paradox.

The main feature of Paradox will be that it will be very light weight in memory and startup time.

It will be great to use for scripting in other applications which demand high performance like for instance games. Also for normal scripting purposes or normal applications it will be way more suitable than normal interpreted languages.

Basicly it will feature a JIT Compiler, Generation GC, rich (modulair) base library and a being very extensible.

It will probably not perform as good as the .net framework but will rather come in the range of mono`s and java`s performance, which is very high compared to the performance of normal scripts.

When Kaja, who is currently working on the JITc and gGC finished the base I’ll start working on the core library and help optimizing and I will post a preview of its capabilities.

(for those wondering how long it will take:

The Unknown – æ£®ã®ä¸ã«æœ¨ãŒãªã„ã€‚ says:
reeally long week

)

Beating Spam

Fri, 11 Mar 2005 22:46:46 +0100

Most mail clients now include spam filters, which are learning and improving themselves.

The problem though is that when spam keeps getting smarter your program has got too which still means reporting half of your spam mail as spam and checking your spam mail for your regular mail. Also when you start again after a reinstall and the spam filter has lost its experience you got to start over again.

Now, I guess it would be great to create a centralized independant organization specificly to regognize (and when it becomes successfull also extract the spammers for prosecution) spam.

The problem is how to organize such a centralized system, for when someone receives a spam email it has got to check with the centralized server whether the spam is spam. The amounts of spam are huge and doubling the enourmous bandwidth and cpu spam has costed already isn’t a very pleasant foresight.

It would be feasable however to create a spam regognition service to run on a clients computer which updates itself with the latest definitions once in a while. This would undoubtly be way more efficient.

The only problem that we are left with is how to get such a system to be intergrated with existing applications, if no one uses it it would be rather useless.

When you have some idea`s on this, please share them.

Trying Thunderbird

Thu, 10 Mar 2005 22:53:09 +0100

When I got my windows reinstalled I tought it was time to try something else than Office Outlook (espacially because a big mailbox makes Outlook slow), so I tried out Mozilla Thunderbird.

What I noticed about Thunderbird in comparison with Outlook:

Faster, slicker UI
Although Thunderbird doesn’t look as great as Outlook it certainly has got a nice interface which is customizable enough.
The Interface itself is also a lot faster than Outlook`s, which I find more important than grafical splender.
Also the collapsable headerview above emails and information bars above the email (eg. a bar showing that thunderbird thinks the email or spam or that thunderbird is blocking images) are a nice addition and are worked out better than in Outlook containing them in a limited way.
Also thunderbird seems to interface directly with firefox letting it show html emails a lot faster than Outlook does.

Configuration
In the beginning the configuration was rather confusing. Although this also applied to the configuration with Outlook when I first tried it out. I still think that the account configuration interface could have been a lot more intuitive.

Outgoing mail
Thunderbird requires and actually uses only one smtp server for all your mail accounts. Although you can specify more it doesn’t seem to be able to specify which Smtp server to use to sent the actual mail. This is a shame for espacially when sending important emails it is more trustworthy if the email can be traced back to the smtp server of the original domain. Also some smtp servers don’t allow seemingly spoofed `from` fields.

Extensions
Thunderbird seems to have a very similar extension system as Firefox which I can appreciate. There are some very usefull extensions like Enigmail which is a front end for GPG for Assymetric email encryption. It doesn’t only manage your encrypted emails, it also got a very good interface for all kinds of GPG operations like all kinds of key managment.

Everybody should give thunderbird a try, it’s a great toy :).

Plugins, the web and python

Wed, 09 Mar 2005 14:51:16 +0100

Plugins for web applications seems to be discussed a lot more recently by bloggers (Zef did for instance).

Plugins usualy come in 2 shapes: hacks and modules.

The key difference between a hack and a module is that a hack changes existing behaviour and a module adds behaviour or maybe extends some. In Zef`s example he is talking about possibly adding a calander to google`s gmail – that is definitely a module.

When changing the whole interface by for instance being able to plan appointments inside other module`s (like viewing email) that would require changing those modules and basicly make it a hack.

Modules are easy to make and maintain for they form seperate entitites which don’t often conflict. Most web applications already support module-based plugins like IPB. Some more functionality can be added to modules by letting modules hook onto certain events in the other application, but this has its limits.

Where hacks are also widely used, although these are very hard to maintain and usualy conflict which eachother.

So what is the best way to write a plugin based application?

I was thinking about that when I was working on a computer version of the popular card game Magic. There are already a lot of computer version which let players play the game but the players have to apply the rules themselves.. the computer doesn’t do that for them for there are for every card a lot of exceptions and new rules are added every month. To make such a game plugins would be required.

The way I solved this is by putting everything in the game in modules/plugins. The only thing the game does is saying ‘start’ to all plugins. One plugin responds to that and that may for instance be the UI plugin which shows the main UI form and then sais ‘postgui-start’, where another plugin may respond that extends the UI form for some new features in the game and then calls ‘postseries4gui-start’. A card itself is represented by a big hashtable containing all info of that card dynamicly, including functions. Letting one create-card attack another basicly is just a big dynamicly getting properties and calling functions which all are overloaded for new rules which seems to work pretty fine.

Guess what I used to develop this highly dynamic interaction which eachother?

Python!

Python is the perfect language for plugins.

Now.. why don’t we already got really nice web applications allowing this kind of highly dynamic plugins? Web applications are although using sessions, primarily stateless.. On every page view the whole script is loaded in the memory, and executed again and then unloaded. Nothing keeps persistant. Using plugins generates a lot of overhead on espacially loading stuff for every plugin must be able to hook onto and interact with everything. Doing this with the current model webservers just don’t work.

I already made some suggestions how to solve this at the server side, but changing this would take a lot of time.

So to get back to how to get this done -now- and not later is to keep working with a modulair way exposing the most used hooks and data access.

A suggestion for gmail:

Allow SOAP based access to gmail`s features. Exactly the same amount as access as the user has got.
Then allow adding new menu items and hooks on which a call is made to your specified url, which then can work with gmail on the client-access level using SOAP calls.

Best would be for google to just expose the api of gmail and give a limited functionality dev download so people can make modules in the source and send them to google back. If they like it they’ll use it.. I guess google will come up with something like that once. Or at least I hope it.

Dyslexia

Wed, 09 Mar 2005 14:19:48 +0100

They just found out me (and my brother) are affected by Dyslexia. A bit late. The strange thing is that I didn’t really notice it that much, it seems that not having motivation for all those foreign languages isn’t the only reason for those really bad marks.

selfish driver control panels

Tue, 08 Mar 2005 18:30:43 +0100

A week ago my windows was agonizing slow. Starting up would take ages, and getting those ‘windows is out of virtual memory’ messages was common, so I decided to reinstall windows.
When reinstalled windows was using 120 mb without anything special installed.. so I started to install my usual application: apache, mysql, mssql.net express, php.. etc..
At the point I got those installed windows was using 250 mb instead of 120mb! msSQL server uses 50mb`s (although that number isn’t displayed in the task manager properly (i guess due to modules). MySQL and Apache both 25mb`s…
So I put the apache/sql servers services off by default and made a little bat script to start those when I want and put that one on my desktop. I also made a backup of windows so I didn’t need that horribly slow reinstall..
I installed some basic drivers via the provided driver cd’s loading a lot of junk like tray icons on my computer. They did not appear to use a lot of memory in the taskmanager but when quiting them it saved 100 mb :-/. 100mb used by selfish driver control panels thinking they are the single-most-used application and the only one sucking that amount of memory…
So instead of using those horrible flashy flash autorun installers I let windows find the correct driver .inf files and install it.
This really saves a lot of performance.. (I’ve cut down my startup time by 4 times).

So, better let windows find the drivers you require on your driver cd’s instead of using those selfish flashy traybar spamming driver control panel installers.

Xr12 concept

Mon, 07 Mar 2005 00:43:46 +0100

I’ve been wanting to develop a massive online browser based game for a long time, for I have been addicted to those for quite some time and have been missing a lot of features and user input.

And here it is, I got myself already a few other people to help, and I just finished the server architecture for when it gets big:

Xr12 concept

I just hope I`ll be able to affort the system due to user activity, from experience I’ve seen that these online games tend to grow really fast.

Python Html Document Abstraction

Sun, 06 Mar 2005 19:15:58 +0100

Python is great!

>>> d = document()
>>> d.html.body.h1.value = "My Site!"
>>> d.html.body.p.value = "Welcome to this python generated site"
>>> str(d)
'<?xml version="1.0" encoding="UTF-8"?>< !DOCTYPE html PUBLIC "-//W3C//DTD XHTML
 1.1//EN" "http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en">
<body><p>Welcome to this python generated site</p>
<h1>My site!</h1></body><head><title>
</title></head></html>'

(Ignore the added slashes and the additional line breaks caused by wordpress)

By overloading the __get/set/delattr__ functions a html document can be represented like a real python object model.

I’ve just experimented a little bit with python code to ultimately go to write a framework to write nice dynamic python webbased applications in.

Although it appears that the names of the objects (html, body, p, etc) are the tag names, they aren’t. They are the identifiers of the tags.. in case the tag isn’t set by yourself but just created for it didn’t existed it uses as tag its alleged id.

The default created object when no object exists already with that id is a tag. This abstract document won’t be limited to tags. I’ve just made a styleTag class which allows:

d.html.head.style.body["font-family"] = 'verdana'

which is basicly the same as

d.html.head.style.["body"].font-family = 'verdana'

In contrary to the normal tag class where an item is an attribute, this is different in the style tag for CSS got a lot of characters which python doesn’t like (like #).

Being able to manipulate a style sheet that easily allows every custom tag (maybe a datetimepickercontrol) to set its own style information by just using simple python code.

For the styletag isn’t bound to putting its emitted css in the emitted-html string itself in case it is emit-ed in a specific context like a webserver, it can even create a seperate css for this purpose.

Python allows much more dynamic features in a dynamic framework like this than any other language, I`m quite enthousiastic about it and am playing with new idea’s like a little child :-).

All kinds of idea’s would be welcome..

Just wondering whether such a thing has already been written for Python.. anyone knows?

Strange spam

Tue, 01 Mar 2005 22:54:49 +0100

The last few days this blog has been under heavy attack of comment spam. Although the excellent wordpress filters have put all of it in the moderation queue it still is quite some work so sift out any comments that actually are of a real person.

The odd thing I noticed out of curiousity is that the links don’t even seem to work on more than half of all the spam comments. They are basicly flooding you with for them hopefully tempting comments and if someone finaly has been tempted enough to click one it doesn’t work!

Using exception or assertation

Mon, 28 Feb 2005 18:54:03 +0100

People tend to mix up the intended use of exceptions and assertations.

Assertations

Most assertation implementations will show up a form and most of them allow you to ignore the assertation. I once was using a program which even asked me whether I wanted to ignore that assertation in the future.

Using asserations as normal errors is wrong. Assertations aren’t meant to be errors, neither exceptions.

Using assertations (as you should) is to make sure that the conditions that should be right are right, if they aren’t right you will know and you know you made an error and not the program or the user.

A seemingly ridiculous example is the following assertation:

ASSERT(1 + 1 == 2)

If a programmer uses the assumption that 1 + 1 is 2 and isn’t sure about this he would use an assertation to be notified when he was wrong or missed something.

Exceptions

Most people treat exceptions as failures of a piece of program. And some programmers don’t tend to clean up stuff before they throw an exception for they just think that noone wants to do anything anymore, for an exception, they think, is fatal.

Exception aren’t fatal. Exceptions aren’t errors. Exceptions occur, for they are just as the word says exceptions. You just got to handle any exception that might occur. An exception becomes fatal when this isn’t handled for it is either totaly unexpected or just unhandable, where in both cases letting the exception stay unhandled is a better alternative than using an assertation.

Schnappi

Mon, 28 Feb 2005 16:05:35 +0100

For all those people that ‘like’ all those great jamba commercials:

Schnappi 1
http://7dhls.7d.funpic.de/schnappishooter.swf Schnappi 2, direct link doesn’t work
Sweety

Enjoy!

Command line parser update

Sun, 27 Feb 2005 11:45:24 +0100

I made some updates on my command line parser for .net .

It now supports aliases and you can set whether the parameter should accept a value or not so you can use -foo bar where foo doesn’t accept a value and therefore bar is regognized as a position dependand parameter.

An example parameter definition:

[Parameter(Position = 0, IsMandatory = true,
AcceptsNoValue = false, Name = "folder",
Description = "The folder", 
Aliases = new string[]{"target"})]
public string FolderParameter
{
 get { return _FolderParameter; }
 set { _FolderParameter = value; }
}

(Ignore the slashes placed before the quotes, wordpress seems to add them automaticly) I also have added the Intrepid.Automation.CommandLine.OutputHelp function, which outputs help for the parameters of an object to a stream. Like this:

SUMMARY
Creates playlists in a folder and its subfolders

PARAMETERS
-target

[String excludefilter = NULL] (efilter)
Only files not matching this regex are included

[String includefilter = NULL] (ifilter)
Only files matching this regex are included

String folder (target)
The folder

[Boolean expand]
Whether to expand referenced m3u’s

REMARKS
m3u playlists are automaticly excluded; use -excludefilter to add additional excludes.

You can still download the Intrepid.Clorelib assembly here and you still may only use it for non-commercial open-source usage and may not change/reverse engineer/etc it in any way.

Hope it will prove usefull :-).

Command line parser for .net

Sat, 26 Feb 2005 19:37:53 +0100

I just wrote a command line parser for .net which parses the command line similar to the way Monad’s cmdlets do it, with the key difference that this still is for the normal .exe executables.

It’s very simple, first get yourself a program class, which contains some properties you want the user able to set via the commandline and apply the Parameter attribute:

class CopyProgram
{
 private string _Source;
 [Parameter(Position = 0, IsMandatory = true)]
 public string Source
 {
 get { return _Source;}
 set { _Source = value; }
 }
 private string _Target;
 
 [Parameter(Position = 1, IsMandatory = true)]
 public string Target
 {
 get { return _Target;}
 set { _Target= value; }
 }
 
 private bool _WalkRecursivly;
 [Parameter(Name = "recursivly")]
 public bool WalkRecursivly
 {
 get { return _WalkRecursivly;}
 set { _WalkRecursivly= value; }
 } 

 static void Main(string[] args)
 {
 new Program().Run(args);
 }

 void Run(string[] args)
 {
 CommandLine.ParseCommandLine(args, this);
 
 // Do stuff here
 }
}

The CommandLine.ParseCommandLine function takes care of parsing the command line string array and filling the required properties of the class provided.

You can call this program in a few different ways:

copy.exe c:\source.txt c:\target.txt
copy.exe -s c:\source.txt -target c:\target.txt
copy.exe -targ c:\target.txt c:\source.txt -r

Although it only required around 250 lines of code it certainly is an enourmous time safer and makes stuff a lot easier for the user.

At the moment I’m programming a few small utilities and will change the code a bit and fix some bugs if any.

For those interesting in testing it themselves (I encourage it), just download the Intrepid.Corelib .net 2 assembly*. The classes required are in the Intrepid.Automation namespace. Please send any bug reports to: bas.westerbaan@gmail.com.

(* You may use everything in the Intrepid.Corelib assembly freely for non-commercial open-source usage. Do not modify or reverse engineer the assembly in any way.)

Update:
Got rid of some bugs and added the RequiresValue property to ParameterAttribute. Will replace lateron with Type, this allows you to use -e bleh where -e takes no value and bleh is interpreted as a nameless parameter. Will add alliases too.

Running linux in windows

Wed, 23 Feb 2005 21:31:59 +0100

Opinion Column by PC Magazine: How to Kill Linux

We better not tell him about Cygwin 😉

Serialization

Tue, 22 Feb 2005 20:13:29 +0100

Yey, I am your average developer and I made yet another program with some kind of data I want to be able to dump on the hard drive and be able to grab it again. Let’s use serialization!

Serialization was meant to be a tool for developers to pick up their data from the memory sqeesh it a bit and let it be dropped onto the hard drive, sparing hours of work making a custom data serializer to an existing format or even worse: a home made format. However there are a lot of reasons why not to serialize in .net.

Ok, so Iâ€™ll spend endless hours to make my own algorithms to save my data producing thousands of almost similar lines of code, but just not nearly similar enough to prevent copy and paste bugs. After admiring my very own labor there will be twice as much time debugging the code.

… mm… isnâ€™t there another way to serialize, which doesnâ€™t have too much adverse sides?
ok, what are the demands:

It shouldn’t effect the way you design classes.
Thus:
It should not be based on public fields or properties.
This is probably the foremost cause of type design restrictions due to serialization support. While dropping public fields as basis of the serialization allows protection of certain fields, it also prevent those fields to be accessed by some kind of automated serialization. This however is only a minor setback: youâ€™ll need to create Serialize and Deserialize methods to control serialization. A part of the current .net serialization also relies on this principle, however there should be some modification:
The actual implementation of the (de)serialization algorithm should only optionaly be in the type declaration.
The original .net serialization restricts the user to serializing souly his/her own types. If string was to be unserializable you wouldnâ€™t be able to tell some kind of serialization handler: “hey, here is a type, serialize it” because this handler would find that you are a moron, trying to serialize a string which is not serializable because there is no Serialize nor a Deserialize method. You would be forced to create your own string serialization algorith in every single type you want to serialize and which contains a string. While this is doable image that the string has been changed in a complex data structure containing inter references… You still would be spending a lot of time creating a serialization algorithm for someone elses type. The most obvious solution to this problem is allowing some kind of TypeSerializer which could be ‘registered’ to a serialization provider. The only drawback to this solution is that you canâ€™t access any private fields, therefore you arenâ€™t able to truly serialize every type and there are scenarioâ€™s imaginable where serialization is impossible. There is no easy solution to this. Luckily this should be a rare event.
You should be able to handle reference types as reference types.
The most fundamental flaw in the current .net serialization in my opinion is being unable to serialize a from multiple locations referenced type only once in other words preserving the ‘reference equals’. Programmers have been known to use ‘ID’s’ to preserve some kind of referencing ability. This seems like a nice simple solution, only draining processor time every time an Id needs to be solved.
There should be a Serialization and Deserialization Host/Provider.
Such a provider has 3 main function, justifying its existence:
- Storing type serializers
  All different type serializers could also be stored in a static list, but this would enforce the ‘use’ of every single one of them (during lookup). You donâ€™t want to know how to deserialize meat when you are vegetarian!
- Storing serialized signatures together with their (deserialized) objects
  The only way you are able to preserve references while being able to serialize on demand is to keep track of the already serialized reference type object or the already deserialize reference objects. This could also be managed in a static list however the ‘same’ rule applies: A ice cream shop doesnâ€™t need to know what kinds of meats already have been serialized.
- Providing Serialize and Deserialize methods to the Serialize and Deserialize algorithms by being an argument.
  The provider should automatically redirect the serialization request for a field of a serialize method to the serialize method of the type of the field, while passing itself as serialization provider along to the next ones in the chain. While this improves user comfort it also gives the serializer the ability to store the created ‘serialized object’ or in the case of a deserializer the deserialized object. There is a catch to this system, when there is the possibility of inter referencing this could cause an endless (until the stack runs out) chain. When the pork meat isnâ€™t finished serializing itâ€™s fields, the olives donâ€™t know that it is busy and will invoke a second serialization of pork meat causing a second serialization of the olives and so on. This is easily solved by a not so elegant use of a Register method inside the Serialize or Deserialize method. This method adds the not yet totally (De)serialized object to the referenced object and serialized objects pairs allowing any objects down the graph to use the correct reference.
It should be able to coup with any changes.
This is a hard one and this is a problem bugging all areas of development. There is no easy solution to this. The only way to handle this is to make some sort of conversion for older files or to have different types of serialized object of the same (sometimes changed) type. It would be something like that you have the serialized type of salade mix containing the amount of salad and another one which also contains the amount of tomatoes.
It should get rid of the string key value pair used by the microsoft serialization.
Strings are slow and interpreting them is even slower. Itâ€™s like describing the forms of the figures in your bank account with metaphors. There is one good thing about key value pairs and that it that they are unordered. This barely manages to try to hide the fact that itâ€™s too slow. An alternative could look like this:
- A Guid regogniced by the deserialization provider as a specific version of the serialized version of a specific type. This would trash any problems with extensibility of the serialized type because you would simple copy and paste the algorithm and kick iit a bit (to fit your demands) and supply it with a new Guid while preserving the old algorithm and possibly adding a friendly obsolete exception. Any type which supplies serialization and deserialization to itself or to another type should include a list of accepted guids.
- A list of referenced serialized objects which could contain the fields of the type. How this is used is to the creator of the serialization and deserialization algorithm.
- A byte array containing any ‘personal’ data of a type whos data isnâ€™t distributed among fields (like natives).
It should be secure.
The memory is pretty save due to access restrictions and the fact that only the application controlling the memory really knows what a byte means, however the hard disk or even worse the internet isnâ€™t really any match for the protection the memory offers. There are a few ways to protect data on your hard disk. One of those is access policy. However this is somewhat unpractical and canâ€™t be applied to internet traffic. Maybe the best solution is using encryption, this could be applied to the hole file, inefficient, but effective or sensitive data inside of a serialized file could be stored into the ‘raw’ data and be encrypted by the type serializer.

I think that there could be an implementation of serialization able to meet these demands. If it does meet these demands there would be little objection left to use and it would be favorable above even the most optimized hand crafted data ‘dumpers’. I will do some more research and there could be a sequential more concrete article with some closer-to-code talk.

P.S. As you may have noticed the posts now contain an author specifies and you may have also noticed this post wasnâ€™t written by the usual author. Iâ€™ve joined Bas Westerbaan writing posts for Intrepid Blog.

Regular Expressions in .Net

Mon, 21 Feb 2005 19:05:44 +0100

Regular Expressions in .Net (System.Text.RegularExpression) are fast!

When creating a Regex with the compile flag the regex will actually be compiled to very optimized native assembly which makes the .net regular expression library faster than any c# code you could write for one specific case.

This was a pleasant suprise 🙂 for now I know I can use regular expression for all those small things like parsing input.

Why Php sucks (and I still use it)

Mon, 21 Feb 2005 17:57:09 +0100

Php is slow, not just slow, but really slow. A simple benchmark runs in 1 milisecond in C. It takes 2 miliseconds for .Net. Python takes 600 miliseconds for instead of a native assembly language or jit-ted language it is an interpreted language. But Php, even though it also is an interpreted language and hasn’t got the enourmous object overhead Python has got it still takes 12000 miliseconds, that are 12 seconds.
Stupid work arounds, for Php itself is rather slow you got to rely as much as you can on function calls instead of doing anything yourself. It is for instance faster to load a list by deserializing a string than just reading line by line through a file although the deserialization would be way faster if both methods would be implemented nativly. Another little issue here is that most very quick functions in Php are only available in the newer versions (eg. file_set_contents), this requires you to add an if statement with a home made implementation of the function which usualy is slower by a factor of 10. You can choose to use an alternative way which doesn’t exactly implements the functions of the function you require to use but still does the job for the circumstance (better) (eg. not rewriting file_put_contents when it isn’t available when you want to streamingly write data to a file but rather call fwrite a few times which gets rid of having the whole file in the memory in a string at a time.
State less Although the Http protocol is a stateless protocol that doesn’t have to mean that a server side scripting framework should be stateless too. Although Php attempts to become statefull by using a session implementation by serializing a session array on the harddisk for every session this isn’t very efficient. Even one global array that persists between requests would result in such a performance boost. Not only for it doesn’t require file reads, writes, serialization and deserialization (and optionaly queries when you don’t like the php session system), but also for it would allow you to store that little bit of important cache between sessions that otherwise would have needed to be read, deserialized, serialized and written again, for every mere page view!
Allowing such a persistant array however poses a security risk in the way Php works at the moment. They should add contexts to allow one instance of apache on which mod_php runs to execute files in different context, each with its own settings (and persistant data).
There is no satisfying solution in Php For every single common issue in Php there is no simple solution, that works perfectly or even reasonbly well.
Take for instance templates. There are basicly a few ways to handle templates in Php. Usualy it comes down to either caching .php-scripts which are than executed as smarty does, or using a class for every major template section where every function fetches a template bit. In both these methods executing Php code is required to just only replace a certain tag with an replacement. Php has been designed to do a lot more than that and contains a lot of overhead during interpretation. Using str_replace‘s is a lot faster than a php block inline or even using instring php variables ("Example: {$example}"). The second way using classes and functions is even worse for the whole class and all functions first need to be loaded in the memory and basicly are a lot slower.
The proper way to use templates is streaminly inputting and replacing tags with their values and outputting it. This isn’t possible for php is slow and loading the whole template in a string is even faster.

The only reason why I still use Php is for it is just the number one supported server side scripting language.

Why is world wide web not fair?

Got GMail invites left?

Sat, 19 Feb 2005 14:05:43 +0100

Don’t go to every forum and post a topic that people can get them by mailing you.

Just donate them to http://isnoop.net/gmail/!

It is the second link in google when searching for gmail invites. (first is google itself)

If you want your invites to be used send your invites to them, as I did.

Microsoft Anti Spyware

Thu, 17 Feb 2005 18:11:31 +0100

Microsoft Anti Spyware beta

It looks to work well, it even detects some adware that adaware didn’t remove.

It is said by many people that using more than one anti-adware program is the best way for they all don’t catch everything. Joel suggests why Microsoft Anti Spyware wouldn’t catch everything: conflicts of interest.

A lot of money is made by redistributing spyware/adaware/malware on people’s computers. They could easily bribe some anti-adware software developers to ignore their adware.

I just got to find an anti-adware program that is proffesional enough to clean my computer, and isn’t bribed to leave some.

The problem is that there are a lot of anti-adware programs, most of them are adware themselves.

Blog updated

Thu, 17 Feb 2005 17:21:22 +0100

The blog has been updated to 1.5.

It adds a lot of nice new features and I would like to advice other users to upgrade now.

The new skin is the new default skin that comes with WP1.5.

When I find time I’ll adapt it a bit to make it a bit less default.

Edit Just noticed that the wordpress ACP looks horribly scrambled in Firefox, looks all right in internet explorer though :(.

Skins and performance in PHP

Sun, 13 Feb 2005 23:58:10 +0100

There are several ways to use skins in PHP, I’ve put some through a performance test.

Basicly you can use either evaluated PHP or a string that will undergo str_replace’s.

When evaluating PHP in a file it seems to be faster than replacing tags in a string. This for PHP streams through the file during execution instead of handling one big string. The difference is minimal though (15% in my tests).

Although when the PHP code is placed in a string instead of in a file which has to be done in case a string is cached in a database or is generated from compiling from another format it is significantly slower than using str_replace’s on a normal string (600%!), this is because the original sourcecode, the intermediate code and the return from the code all take a lot of memory.

Either use cached PHP files or a string with tags instead of PHP code in a database, never the otherway around (what happens very often).

Caching in PHP

Fri, 11 Feb 2005 17:06:03 +0100

It is usefull to cache certain things between Php script executions.
Some boards written in Php cache the forum architecture so a difficult query hasn’t got to be run every time a guest views the board.

There are a few ways to cache data:

Php script. Data will be stored as a normal php file which will be included during execution
Serialized object in file. Data will be serialized and dumped to a file which will be read every page view
Database storage. Data will be serialized and stored in a database and queried every page view.

There are a lot of myths about using a database would be way slower than a normal php file.

I’ve run a few tests caching a ~16kB php array, the results:

Serialized object stored in file: 0.0015ms
Object in PHP script: 0.0121ms
Serialized object stored in mySQL database: 0.0015ms

It seems to be quicker to use a serialized array in a file as configuration file than a config.php php script!

Databases although just as quick as normal files are favored by me for they are much more scalable.

Gamma Wave Effect on a real electric guitar

Mon, 07 Feb 2005 15:15:05 +0100

Finaly I got grip of a recording of an electric guitar without distortion and applied the gamma wave distortion on it:

Sine gamma effect results and positive linear gamma results

Negative linear gamma effect results

Gamma Wave Effect

Mon, 07 Feb 2005 12:40:09 +0100

As promised a few pictures of the electric guitar effects we’re working on,

first up, the gamma wave effect.

The gamma wave effect pulls the amplitudo’s either to the 0 line or the -1/1 line, just like the gamma on your monitor does:

The effect basicly makes the wave a lot louder and gets rid of the faint sounds. When on fully applied as in the images above it also creates some distortions due to either making the wave unharmonic-ish or getting rid of the nuances.

Another image taken on a higher oscilation (smaller zoom):

This clearly shows the distortion

(The inversion of peak and valley doesn’t matter for for sound only the transition from a valley to a peak matters)

Electric Guitar Distortion

Mon, 07 Feb 2005 00:02:08 +0100

As I said before I am working with some others on a software based distortion. I finaly got the base running pretty smoothly but the main obstacle is how to create that distortion electric guitars use.

We came up with about 3 different methods:

(When I got my osciloscope control working I’ll take before and after pictures, for now I ain’t sure how the formula’s effect real-world sound yet)

Gamma corrections
This works a bit like the gamma of your monitor. The input for graph for gamma correction is called the epsilon which comes in 2 forms, linear and sine. The first one creates a more distortion like sound than the latter but also makes the wave not harmonic anymore which possible means that most of the distortion like sound is caused by the speakers not handling unharmonic waves very well. The latter one only sounds near the amplitudo distorted.

Sharpening valleys/tops
This method requires some buffering of the current top (or valley) of the wave and sharpenes it by a specific amount. This works a bit like the gamma correction method although it works on every volume making it usefull too for low-volume sounds. The major problem is that it requires tracking a top or a valley, which with a high bitrate requires a really big buffer to analyze, and it has a delay of one top/valley. The big problem is that this isn’t ideal for live playing for which it was designed, a delay of a few extra hundreds of a second would be noticable and it is fairly possible that this algorithm requires too much resources.

Adjusting speed resistance
At a certain point in a wave you can derive the speed and the angle. By registering the original speed and resistance of the wave a derived one can be created which could leed to sharper or softer edges of a top/valley just as the previous method but without being required to analyze the whole top/valley. We’re hoping this method will work best but this is just a concept hoping to work out as it should.

If anyone actually knows how analog distortions work exactly we would be more than happy to learn about it, just comment.

Working on a software based distortion for the electric guitar

Sat, 05 Feb 2005 21:27:58 +0100

At the moment I am busy creating a software based distortion for an electric guitar.

The most challenging part is getting effects to work, to explain this in more detail you need to know how a computer handles audio.

Sound itself is nothing more than a vibration in the air. It can be represented by the amount of force the air is pushing or pulling.

A tone of a specific frequency would look like:

A computer stores sound by sampling the amplitude of the air at regular interfals.

It would be easy to write a program to increase the frequency of the sound above, but a normal sound doesn’t look that regular:

Increasing the bass or treble of that wave would require some advanced algorithms, which take time to execute which creates a larger delay. One thing that a distortion shouldn’t do is lag.

More on this when some stuff is working.

PHP Security Consortium

Fri, 04 Feb 2005 21:19:55 +0100

The PHPSC is a site managing a lot of resources on PHP security.

For all those starting or sometimes using PHP this is a must read.

Also I’d advice for people who want to know whether there site is safe enough is to try to play the other site by trying out hacking yourself: hackthissite.org. It is easier than you might have thought.

Image echoing

Wed, 02 Feb 2005 00:01:37 +0100

An almost realtime webcam with a monitor is always fun, espacialy when the bad quality requires the software to compensate resulting in some nice effects after several passes:

C# rapid game development

Sun, 30 Jan 2005 01:32:16 +0100

I’ve been working on a Direct 3d game in C++ for quite some while now. I recently tried using Direct 3d in C# and it took me only 15 minutes to get a basic framework running instead of the 5 hours in C++. Now I know that C# isn’t favoured for game developing for it is slower; it is rapid development and a lot more maintainable.

I did some test on how much the difference would be between having a C# or C++ game engine.

The results were quite supprising:

C# is slower, but when letting as much as possible be done by the graphics card and direct X instead of C# code the difference between C# and C++ is neglected also due to superior threading control and runtime memory managment of C#
C# develops a lot faster than C++, far faster than I expected initialy
Threading and timing the renders and other operations is a lot faster in C# than it is in C++. Also multithreading doesn’t generate as much problems in C# as it does in C++. And maybe even important; debugging threads in C# is a lot easier than in C++
The second critical area (after the rendering) is the gameplay engine. Most games need to use scripts for it is unfeasable to hardcode everything. C# doesn’t has to use scripts for using seperate .net dll’s works quite well and is hundreds times faster too

I wonder what will be the first major game that will be written in .net. It certainly should atract some attention by game developers.

Server transfer: downtime

Sun, 30 Jan 2005 00:39:20 +0100

I will soon own a quite cheap virtual server (by greenT, great company) which will give me way more flexibility than the current shared server account on which w-nz.com is located.

I’ll transfer my current site to the new virtual server from the current shared server account (w-nz.com; xr12.com; intrepidsoft.net).

In the worst case this could result in domain problems, transfer problems and lack of time to get it done and therefore a lot of downtime.

So when w-nz.com is down please be patient and if you want to email me use bas.westerbaan@gmail.com instead of my @w-nz.com email addresses.

Negative .Net myths busted

Thu, 27 Jan 2005 16:04:31 +0100

There are a lot of negative myths about .net which people tend to use to favor the traditional languages like C++ above .net. I’ve busted the ones I read frequently:

The GC is really slow
malloc is way slower! The Garbage Collect of .net actually is faster than any Unmanaged code for it nows whether a value is a reference (pointer) and therefore can move objects in the memory. The GC puts objects of about the same age (generation) close to eachother in the memory. Objects tend to refer and use objects in the same generation. The processor itself doesn’t directly load a value from the memory but loads a whole block of a few KiloBytes in the Cache. When the processor directly caches all the objects which one object uses it just runs a lot faster for working from the cache is a lot faster than recaching different parts of the memory over and over again which happens with unmanaged languages which just put objects where there is free space.
Interpreting that stupid Intermediate Language is damned slow
.Net doesn’t interpret its IL, it compiles and optimizes IL runtime
Compiling runtime is very slow anyways
(That compiling a C++ is slow doesn’t mean that .Net is slow) It saves a lot of time for compiling at runtime allows great optimalisations like getting rid of unreachable code and inlining depending on the current runtime variables. Also operations can be compiled with processor specific optimalisations from one IL source. Most of the resource intensive compiling is done at the startup of the application, it is done while the program is running too but that really makes it a lot faster instead of slower
If I write assembly myself it will be way superior to anything .Net can generate
.Net can’t make all the optimalisations possible for it would take longer to analyse code than the optimalisation would gain. But usualy it creates still very optimised code. The big problem with writing very optimised assembly yourself is that the most optimised code is very processor specific and would be very hard to port, and even worse to maintain. Wanting to add one little extra feature could let you rewrite the whole code again. (Yes I indeed have made programs with assembly). Languages which avoid this a bit like C++ still require you to make a different build for every specific processor when fully optimising. Also it is nearly impossible to debug fully optimised unmanaged code but in .Net it still provides you with at least the functionname in which it has happened with the offset (try to accomplish that with C++ in release mode)
The runtime is soooo damned big, it sucks
20 Mb’s isn’t a lot. It only has got to be downloaded once, and the .net framework is in Windows Update so everyone who updates his computer would have it installed by now. Usualy there is room enough on your software installation CD to include .net, it is more than worth those 20 mb. Also languages like C++ require certain runtimes which arent that cooperative. Does ‘DLLHell’ ring a bell?
The .net library naming SUCKS
Yeah.. its naming is different than what MFC uses. At least the naming is very consistant which is way more important than ‘nice naming’, although when seeing some C++ API names used I still wonder why someone could prefer that above the clear .Net naming
The .net library itself sucks
Really? Like what? What can’t it do?
You can’t use API calls like CreateFile
Now I can’t…
[DllImport("kernel32.dll", SetLastError = true)]public static extern IntPtr CreateFile(string lpFileName, uint dwDesiredAccess, uint dwShareMode, IntPtr lpSecurityAttributes, uint dwCreationDisposition, uint dwFlagsAndAttributes, IntPtr hTemplateFile);
… now I can!
.Net sucks cause it is Microsoft
Yeah, so what. .Net is a ECMA standard so you are pretty free to use it, and if there is a catch then that one hasn’t been exploited yet for on linux people are happily using mono to run .net stuff

Avalon

Tue, 25 Jan 2005 22:42:46 +0100

http://www.microsoft.com/downloads/details.aspx?familyid=C8F904E1-B4CA-402B-ACCF-AAA2BD60DA74&displaylang=en

When having Avalon installed it adds a few new project templates to my Visual C# 2005 express, with which avalon applications can be made. Avalon is a Window system which uses xml files to define a form. At the moment I couldn’t find a designer neither reference to it in the help files and assume it isn’t implemented yet, which is rather a pain for creating a form by editing Xml by hand is just a pain: Xml is hard to write and I am just too dependant on the user friendlyness of the form designer.

When compiling Avalon generates classes and serialized data files in your application to replace the Xml files.

So what does it basicly do? Lets you design your forms easily (although there isn’t a proper designer yet) maintaining a good performance by replacing the slow Xml files by generated classes and serialized resources compile-time.
Having a few test applications decompiled and having looked at the sdk it seems that Avalon can do practicly the same as the current Windows.Forms dll.
So now I wonder, in what way would Avalon be better than using the great current form designer and the Windows.Forms dll?

Intrepid Countdown

Tue, 25 Jan 2005 22:31:26 +0100

Very simple program I wrote in about 1 hour with c# (I love its productivity) to keep track of certain events:

http://w-nz.com/update/IntrepidCountDown-1.0.0.1.exe

It basicly runs in the background and notifies you of an event or a few minutes before one which you can add yourself.

Electrical Resistance

Tue, 25 Jan 2005 10:58:49 +0100

R = t / C!

Electrical Resistance equals Time per Electrical Capacity.

Variable sized floating points

Sat, 22 Jan 2005 11:13:50 +0100

A lot of selfrespecting programming languages have got Integer classes which theoraticly can have an unlimited size by using a dynamic array underneath it.
Python got it, and lots of others too.
But still, they haven’t got a variable sized floating point, which I find odd for it shouldn’t be a big problem to create:

Wrap one existing variable sized Integer
Add an integer which points to the place where the point will be
Add operator overrides to get it working
Add an integer which specifies how much numbers after the dot there will be

The later one is quite important to have got. We wouldn’t want 1 /3 causing an infinite loop.

You even could do it a second way:

Wrap two existing variable sized Integers.
Add overriden operators to get it working

These 2 integers would represent a fraction: a / b.

By storing the fraction in stead of the result you are actually very precise, although some numbers cannot be represented in this way, but can be approached.

Subtext

Sun, 16 Jan 2005 15:13:51 +0100

Subtext

I like subtext, I am just wondering how it would be possible to create a big practical project in it:

How would it do async operations? GUI’s; networking… If there is no difference between runtime and execution time this could be very hard to visualize
How would this ever perform properly? Compiling would just do but when debugging a big application it would get rather slow. When it would get dynamic, meaning that the code can drag-n-drop itself, it would be virtually impossible to get a compiled version running quick.
What about internal arguments? Lets make a static field in a function, and if the language wisely doesn’t support it this behaviour could also be replicated by having a file in which a number is incremented. So now we got a field in that function which number gets incremented on every call, an internal argument not known by the engine. Every function generates the same results with the same arguments but now it doesn’t and for there is no difference between runtime and developtime it would behave unpredictable.

Subtext is rather a way to create a function which is ‘instant’, it is a defenition rather than an operation. Computers can’t just do stuff instantly like read a whole file or execute operations.

I like the idea, but I just see no real practical use.

C&C Generals Scud Hack

Thu, 13 Jan 2005 13:57:14 +0100

I’ve found a rather interesting ‘hack’ for C&C generals on a forum: the Scud hack.

Build a Scud Launcher
Select a unit which can shoot and press Ctrl+1
Select the Scud Launcher and press Ctrl+2
Press 1, than press shift+2
Now you’ve got the Scud launcher and the unit selected. Mouseclick on an area while holding Ctrl to let the unit and the Scud force fire there.

EA won’t fix the hack but when you use the Scud hack whilst playing a stat-online-game you’re probably get banned.

Chain emails suck & Asia

Mon, 10 Jan 2005 20:01:58 +0100

I have received 5 chain emails in my mailbox today claiming that when I forward it to a douzen other people putting my name in it would help the victims of the tsunami in asia.
How? How can miljons of emails help those who most of them haven’t got computers (anymore, or never had) to receive email!

Usualy people forward one email to 10 others at least, the count on most emails I received was 400. So lets assume that every forwards the email 10 times, and this continues for 400 times:

10 ** 400 == 1e400

That are actually more emails than people in the world, usualy people get the same mail back from someone else later in the chain.

Lets assume that 500 miljon people receive a certain chain mail Ã 100 KB bandwidth for the sender and receiver combined.

That makes 500 milj times 100 KB is 50 TB..

1 Gig usualy costs a provider lets say 5 cents: 5 cents times 50 TB is $2500.

If we would just don’t forward chain mails but all send a simple postcard to asia we would let them show we care and we would save $2500 for the mail providers who will lower prises, which will result in more money for users which eventually results in more money for the world economy including asia!

Never forward a chain mail

I would like to use this moment to say I am shocked by the tsunami in Asia and I do care for the miljons over there, I hope this single post will convince at least one person to stop forwarding mails, that would save an average of about $100 over some time, my donation for them.

Torrent sharing p2p network

Sat, 08 Jan 2005 10:55:00 +0100

In my previous post I discussed Exeem. Exeem is (or actually will be for it hasn’t been launched, just announced) a p2p network for sharing, rating and commenting torrents.

What is a torrent? A torrent is a small file which is used for thebittorent p2p file redistribution system to identify a certain file, or files you can download. You first need the torrent for a file/folder before you can download it.

The major problem with this is that it is impossible to use a bittorent client itself to search for the downloads you want, therefore a lot of sites have been created over time which contain huge searchable collections of torrents. One of these sites was suprnova.org, which has recently been terminated due to legal issues.

As I elaborated in my previous post Exeem probably will suck. So someone will need to do stuff right by making an alternative.

What issues would have need to be solved to create such a p2p torrent sharing network?

No centralized client list, most p2p networks were terminated because they had a centralised tracker to which a client connected to receive the file list and all the users available for a certain file. Instead of a centralised server every single client should tell other clients who else is in the network and what files are there. When giving every client a buildin list of IP’s it can update these by querying these for better IP’s. By rating an IP by uptime and connection bandwidth a big changing group of frequently online users could provide the other IP’s and port search queries for the rest.
Searching, how to handle a search query? At this moment our client is connected to a few big clients who are frequently online in their neighbourhood, lets call them super nodes for now. When we send them a search query they would look in their cache whether they got the result and if not they look in their own torrents to see whether one of those matched the query and if it doesn’t they’ll just forward it to another a-bit-smaller supernode. The problem with this method is that one query could travel a huge amount of nodes and when you are connected with a good bandwidth you are doing nothing more than passing through queries to other nodes. To solve this the query feedback (when they found a result) should contain the source along with the estamated amount of different search queries the node which had the result can provide. By doing this a shortcut can be formed by one client if it finds a node which either has a lot of files searchable or which has an enourmous cache and offcourse along with that is online often and has a neat bandwidth
Rating Alongside every torrent you download or expose for upload there would be a meta data file containing a description, rating and comments on the torrent itself. The problem with this system is that descriptions and ratings can change and it is very hard to keep every instance of a torrent on the whole network synchronized. It is possible to send a message through the network to the original node from which you received the file with the new comment message, or you could search for the torrent again by unique id and message the nodes found to have the torrent too. All these methods still include a lot of passing through messages.
Privacy
Client side ‘hacking’, When everyone would use the default client which automaticly selects super nodes and lets people pass through queries everything will work fine. The big problem is that it is very possible that people would start using illegal client applications which would just leech from the network. To incorpirate methods to get rid of leechers would work when most people are still using the default client, but when people massivly start using illegal clients the network won’t block itself anymore but would certainly get rid off itself for everyone is leeching. This is the major problem that could happen to this p2p network which heavily relies on the fact that everyone should help others whether they like it or not by proxying, caching and passing through various queries to maintain privacy and decentralization.

I’d be rather interested in how exeem will address these issues. I guess they would just outrule client site hacking by incorperating various encrypting tricks in their protocol.

Exeem the hype

Thu, 06 Jan 2005 19:02:04 +0100

Slashdot on exeem

Since suprnova.org has been offline due to it being illegal the main source for torrents has dissapeared and has lead to a hype around the replacement made by the original maintainers of suprnova, exeem.

First, what actually was suprnova.org. It was a site which maintained torrents for legal and illegal files. And it did even more; all torrents were thouroughly checked, commented and rated by an enourmous team of editors, making sure that the torrents on Suprnova.org were the best you could possibly find.

Because Suprnova moderated, commented, rated and checked every single torrent they offered they were without any doubt illegal. If they would have only offered user uploaded torrents with a nice disclaimer that the torrents are the property of their respective owners they would have probably gotten away with it, but they also wouldn’t have got as big as they were.

Exeem basicly works offers the same as Supernova.org, except that it is a p2p application, not a centralised website.

It basicly stores torrents and comments on these on a Peer to Peer network similar to Kazaa, which basicly makes every single user of it just as legal instead of just the main servers as it was the case with Supernova’s servers. It is very hard for authorities to punish every single user of a p2p network. There would have to be a trail for every single user which would never be profitable. Governments have tried to confict the big p2p users for this actually is profitable. The main problem is that there aren’t a lot of really big p2p users, just an incredible amount of small users who combined are even worse than a few big ones.

Exeem sounds great, exeem is an enourmous hype. But I think Exeem will suck:

Exeem is p2p, this will most likely cause the rating, comments and moderation on torrents go down a lot and make it less attractive for the user. When having a very secure system so that only a few people can add new torrents to the network you have to have some kind of centralised authority which will be very vulnerable to legal persueds.
Exeem will be addware, this will cause a lot of people to drop off. Noone wants to have addware on his computer.

Although I could be mistaking, most hypes like this one tend to turn out really dissapointing.

In my opinion the only way to get a neat new system like exeem which works cool is to get a p2p torrent redistribution network for legal purposes. Bittorent grew big for it was used to redistribute linux redists. Although it will probably will get used for illegal purposes it just would be a very handy system for legal purposes too.

Freelancer Binary INI format

Wed, 29 Dec 2004 18:29:23 +0100

A paper I wrote about the binary INI format used in the game Freelancer by Microsoft:

>>> Freelancer Binary INI format

The dynamic of python

Wed, 29 Dec 2004 18:21:30 +0100

Python is dynamic (duh)

Dynamic typing
You don’t need to specify the type of an object anywhere.

def PrintSomething(something):
 print something

Everything is an object
Everything, yes, everything in Python is an object. Even an integer, a method, a class, etc.
Every object has got a type (__class__), which actually can be changed at runtime, which usually results in conflicts about the allocated size for the PyObject.

No typing needed
Python doesn’t know interfaces, for if a certain object wants to expose certain behaviour it just implements the required methods.
When you want a object to be comparible in python you don’t inherit something like IComparible like in .Net, but you just create the __cmp__ method.

Fields and methods can be changed (and are the same)
A method and a field in python are 2 the same things. They both rely in the object’s dictionary (__dict__) or of the object’s type’s dictionary (__class__.__dict__).
This allows you to use an object which implements __call__ instead of a method.
For __dict__ is writeable you can change / create attributes (members) at runtime:

>>> sys.__dict__['foo'] = "bar"
>>> sys.foo
'bar'

A class member function is a normal method, wrapped

>>> class exampleclass:
	value = ""
	def examplefunction(self):
		print self.value
>>> def examplereplacement(self):
	print "replacement!"
	print self.value
>>> instance = exampleclass()
>>> instance.examplefunction()
>>> instance.value = "foobar"
>>> instance.examplefunction()
foobar
>>> instance.examplefunction = examplereplacement
>>> instance.examplefunction()
Traceback (most recent call last):
 File "<pyshell#15>", line 1, in -toplevel-
 instance.examplefunction()
TypeError: examplereplacement() takes exactly 1 argument (0 given)
>>> instance.__class__.examplefunction = examplereplacement
>>> instance.examplefunction()
Traceback (most recent call last):
 File "<pyshell#17>", line 1, in -toplevel-
 instance.examplefunction()
TypeError: examplereplacement() takes exactly 1 argument (0 given)
>>> instance.__class__.__dict__['examplefunction'] = examplereplacement
>>> instance.examplefunction()
Traceback (most recent call last):
 File "<pyshell#19>", line 1, in -toplevel-
 instance.examplefunction()
TypeError: examplereplacement() takes exactly 1 argument (0 given)
>>> instance = exampleclass()
>>> instance.__class__.__dict__['examplefunction'] = examplereplacement
>>> instance.examplefunction()
replacement!

The bound methods seem to work a bit more tricky than they appear to work. Usually editing member functions via the __class__‘s __dict__ will result in a proper replacement.

These were just a few examples of what Python has to offer. These features aren’t limited to Python. Iron Python, an implementation of Python in .Net is capable of letting usual .Net objects to be manipulated in similar ways with Python via Iron Python. Although this doesn’t actually change the .Net objects, changing the wrapper resulting in the required similar behaviour is good enough.

The author of Iron Python has been recruited by Microsoft and is now working on making the CLR more dynamic.

I’d love a static dynamic language.

Base64 encoding/decoding algorithm

Mon, 27 Dec 2004 17:58:14 +0100

I’ve made some python functions to encode/decode base64. I’ve been trying to develop my own algorithm for base64 for the email protection script which can be found here.

Python again has proved itself again to be a great language for quickly developing stuff.

def tobase64(s, padd = False):
 b64s = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/"
 b64p = "="
 ret = ""
 left = 0
 for i in range(0, len(s)):
 if left == 0:
 ret += b64s[ord(s[i]) >> 2]
 left = 2
 else:
 if left == 6:
 ret += b64s[ord(s[i - 1]) & 63]
 ret += b64s[ord(s[i]) >> 2]
 left = 2
 else:
 index1 = ord(s[i - 1]) & (2 ** left - 1)
 index2 = ord(s[i]) >> (left + 2)
 index = (index1 << (6 - left)) | index2
 ret += b64s[index]
 left += 2
 if left != 0:
 ret += b64s[(ord(s[len(s) - 1]) & (2 ** left - 1)) << (6 - left)]
 if(padd):
 for i in range(0, (4 - len(ret) % 4) % 4):
 ret += b64p
 return ret
def frombase64(s):
 b64s = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/"
 b64p = "="
 ret = ""
 s2 = s.replace(b64p, "")
 left = 0
 for i in range(0, len(s2)):
 if left == 0:
 left = 6
 else:
 value1 = b64s.index(s2[i - 1]) & (2 ** left - 1)
 value2 = b64s.index(s2[i]) >> (left - 2)
 value = (value1 << (8 - left)) | value2
 ret += chr(value)
 left -= 2
 return ret

The algorithm doesn’t automaticly add the required =‘s while encoding, nor does it require while deencoding.

Inactivity

Sun, 26 Dec 2004 00:09:03 +0100

For some undisclosed reasons I won’t be able to access the internet as often as I would like to in the next few weeks. Therefore my activity on the internet will drop, so will the amount of posts on this blog propably. I will try to find some time to post at least 3 articles per week.

If you want to reach me, try e-mail instead of MSN:

Update on the anti-email-harvester mailto links

Wed, 22 Dec 2004 22:50:13 +0100

In the previous post I described a simple though effective method to get rid of the constantly cleverer spam email harvester bots.

I’ve made a little update on the algorithm, it now uses only 1 number for each character and uses a cascading incremental xor transform.

Python code for the algorithm itself:

def alphaicx(s):
 ret = ""
 cascvalue = 0
 for i in range(0, len(s)):
 ret = ret + chr(ord(s[i]) ^ cascvalue)
 cascvalue = (ord(ret[i]) + 1) % 255 
 return ret
def betaicx(s):
 ret = ""
 cascvalue = 0
 for i in range(0, len(s)):
 ret = ret + chr(ord(s[i]) ^ cascvalue)
 cascvalue = ((ord(ret[i]) ^ cascvalue) + 1) % 255
 return ret

I designed the algorithm in Python. Python is great for that kind of stuff.

As you can see there are 2 functions, when you encode something with alphaicx you can decode it with betaicx, and visa versa. betaicx creates tougher code though. This encryption is pretty lousy, but hard enough to stop spam bots.

I’ve ported betaicx to PHP, and alphaicx to Javascript. The running example (very usefull though) has been updated.

The PHP/Javascript code for the function:

function JSBotProtect($text){
	$cxred = "0";
	$cascval = 0;
	for($i = 0; $i < strlen($text); $i++){
		$value = (ord($text[$i]) ^ $cascval);
		$cxred .= "," . $value;
		$cascval = (($value ^ $cascval) + 1) % 255;
	}
	return <<<EOF
<script type="text/javascript">var cxred=String.fromCharCode({$cxred});
var uncxred=""; var cascval=0;for(i=1;i<cxred .length; i++)
{uncxred+=String.fromCharCode(cxred.charCodeAt(i)^cascval);
cascval=((uncxred.charCodeAt(i-1))+1)%255;}document.write(uncxred);</script>
EOF;
}

I’ll more compact uncxred storage. Probable just normal hex, or when I can get it working BASE64.

Protecting your email address against spam bots

Tue, 21 Dec 2004 23:43:26 +0100

Spam bots get smarter these days in harvesting email addresses. They usualy use a regex which searches for ‘.. dot .. ltd’, which isn’t that resource intensive. When that is done a more advanced regex is put in there to get the email adress somehow removing stuff like ‘spam’.

Using normal javascript encoding doesn’t work anymore, for it isn’t that hard for a spider to regognize encoded strings and decode them, whether this is in javascript code or normal html escapes.

Therefore we need to get more inventive:

function JSBotProtect($text){
	$xorred = "0";
	$layer = "0";
	for($i = 0; $i < strlen($text); $i++){
		$layerbit = mt_rand(0, 255);
		$xorred .= "," . (string)(ord($text[$i]) ^ $layerbit);
		$layer .= "," . (string)$layerbit;
	}
	return <<<EOF
	<script type="text/javascript">
		var xorred = String.fromCharCode({$xorred});
		var layer = String.fromCharCode({$layer});
		var unxorred = "";
		for(i = 1; i < xorred.length; i++){
			unxorred += String.fromCharCode(
				xorred.charCodeAt(i)^layer.charCodeAt(i));
		}
		document.write(unxorred);
	</script>
EOF;
}

This PHP function returns a javascript block of code which stores the sensitive string like an email address in 2 parts, which when xorred with eachother result in the original email address.

An implementation to get a mailto: link

Developing plugin-based applications

Tue, 21 Dec 2004 21:40:40 +0100

When you want to write an application, whether this is a website, or a database manager which is based on plugin provided functionality you should remember two things when programming:

Can I change this behaviour easily with another plugin? You want to be able to edit a menu item for instance.
Do I want to be able to change this with another plugin? You don’t want a plugin to edit a menu item if it is meant to only be a simple format interpreter.

Basicly there are 2 big groups of plugin based applications, the static and dynamic. Where the static plugins are the easiest to program and most commonly used. An example would be a plugin which would load and save an image from and to a ‘.jpg’. These plugins usualy contain one function (gateway) which exposes a few plugins which usualy expose a static interface.

On the other side there are the over-dynamic plugins, which rather should be called ‘modules’. These applications would use a high level slow ‘message’ system where basicly everything is able to be intercepted, hooked up to, responded to…

Deciding what and how you want base behaviour to be able to be extended, edited or removed by plugins is the hardest part of developing a plugin based application. If everything should be able to be changed you get stuck with complicated code, and a lot of overhead due to extra ‘hookable’ calls. If almost nothing is overridable you problably get stuck with worthless plugins.

While developing a plugin extendable application it is very helpfull to put the base code in several plugins themselves, which prevents a lot of mistakes.

I’ve found python and c# to be very capable languages to write plugins, although C# is a bit more trouble than Python, Python allows you to do too much and just call functions without knowing for sure with what object you are handling.

I’ve been working on a modular server (see the modular server article I’ve posted), programmed in C#.

The base of the server is nothing more than a framework for loading / unloading libraries & plugins, managing configuration files, and hosting ‘server’ plugin instances. Every server plugin itself can load plugins (like for instance a custom authentication handler for a HTTP server). I’ve noticed that having a very dynamic configuration storage is very usefull for every instance of a plugin requires its own configurations and usualy its own plugins to be loaded. I’m currently using multiple tree based files which are merged at runtime, which I’ll replace with a concept format called XTL about which I’ll tell more lateron.

RGB to Hex (and why the python interactive mode is so damned handy)

Mon, 20 Dec 2004 00:18:52 +0100

Update 2010-02-01 Thanks to Sameer for pointing out the short way of doing this: return “#%02X%02X%02X” % (r,g,b)

def tohex(r,g,b):
	hexchars = "0123456789ABCDEF"
	return "#" + hexchars[r / 16] + hexchars[r % 16] + hexchars[g / 16] + hexchars[g % 16] + hexchars[b / 16] + hexchars[b % 16]

Python is very convenient when you need to program a simple algorithm. I programmed this RGB to Hex converter in less than 1 minute, including using the function for a few RGB values I needed to convert.

Usualy I just grab a pen and a paper and do the calculations myself for getting either the calculator of windows itself to show up and actually calculate stuff properly (looking to screen, writing down, forgetting to press C, starting again…), or writing a program in a language like C# would just take too much time.

I’ve been using Idle for quite some time as a very good replacement for both my calculator and my pen and paper.

Markup? Nah, wysiwyg!

Sat, 18 Dec 2004 19:06:10 +0100

When you post something on a forum, on a guestbook, on anything on the web which supports some kind of formatting of your code it works with some sort of BB-Code.
BB-Code is just like HTML formatting, with ‘[]’s instead of ‘<>‘s, and a lot less features. It is hard to type and doesn’t look neat.

But who cares? Everyone uses BB-Code, almost everyone knows BB-Code, and most people don’t find it hard to type for people are just too used to it!

There are alternatives to BB-Code, like Textile and Markdown, which use a more convenient syntax.
Personally I don’t like using them for I ain’t used to the syntax.

However.. why would we want to write formating anyway? Why not just use wysiwyg, and I do not mean a java applet but rather a standard for browsers; a new tag: “<input type="formatted" name="example" />“, which would act for the server as a normal input field returning the formatted text in Html.

The problem is that it would be very hard to get every browser to support such a new tag. Most browsers I guess would be very willing to comply. But browsers like for instance Internet Explorer wouldn’t. They don’t even comply with the simplest of CSS at the moment which gives website developers a headache.

Intrepid C – ‘Objects’ in C

Thu, 16 Dec 2004 23:35:19 +0100

I’m working on a library which can be used to achieve most OOP functionality of OO languages in C, by providing an API (not by extending the syntax).

I’ve written 2 articles (artc. 1, artc. 2) so far about getting OO like behaviour in C, and will write a few more about the more advanced stuff which I’m still experimenting with.

You can download the source of the API I have developed so far here, which by the way is far from finished. Stuff will change, and I am aware of some issues.

I’ll explain how it works in the next parts in the ‘Objects in C’ serie, but to give you a kickstart in the code I’ll explain it briefly:

Every object has got a pointer to its type and its toplevel object, which usualy is itself. The type instance pointed to from the instance provides the size, constructor, destructor and the interfacegetter.
The GetInterface function in the type (the interfacegetter) returns a pointer to an instance of an object of the given type if supported by the object.
Using interfaces can serve different purposes:

Exposing certain objectwide functionality

((ICloneable*)object->top->type->GetInterface(object->top->type, object->top, GetICloneableType()))->Clone(object->top);

This code will return a Clone of the object, even if only the object is passed by a not top level pointer. Using the top pointer ensures proper polyforism, and enables the ability for the top most class to hide, or show some internal functionality of wrapped objects.

Exposing interface specific functionality

((String*)object->type->GetInterface(object->type, object, GetIToStringAble()))->ToString(object);

It now depends on which interface of a certain object is passed which behaviour results.

And more.. and more.. which I will explain in the coming articles.

Download the source so far

Some more on my blog

Thu, 16 Dec 2004 23:02:28 +0100

I’m keeping this blog mainly to channel some thoughts about computer science (but probably about physics and other subjects which I find interesting too). I hope people will find the stuff I’ll write about interesting, and hopefully usefull. Most stuff I’ll post will as far as I know be quite new, or a new approach to an existing issue.

It seems that so far 3 people have linked to an article on this blog (all of them to the Objects in C article) already. (Thanks! without readers a webblog is quite useless!)

‘Objects’ in C – part 2, ‘type’ing stuff

Wed, 15 Dec 2004 20:32:49 +0100

In my previous article I’ve talked about the basics of using ‘object like’ programming in C. Actually they rather were more like examples. To continue further I’ll discuss the stuff behind it more detailed, but first I’ll explain the basics of structs, if you know them already, just skip that part of this part.

Structs in C (and in the memory)
An example structure in C:

struct example{int a; int b;};

A structure is nothing more than a way of giving some parts in a fixed size amount of memory a specific name, and a recommended type.

When one would create a new instance of the example structure it would require 8 bytes of space in the memory, 4 for a, the first integer and again 4 bytes for b, the second integer:

00 a 01 a 02 a 03 a 04 a 05 b 06 b 07 b 08 b

An instance of the example struct could begin at the memory address0x0A001000. In that case the value of a would be at the memory adress 0x0A001000 too, for a is the first field in the structure. The value of b however would be located at 0x0A001004 for b is the second field and is located 4 bytes after the start of the structure. I could use that knowladge to access b in several different ways:

example* e = malloc(sizeof(example)); e->a = 1; e->b = 10; printf("%d", e->b); // The usual way printf("%d", *((int*)((int)e + 4)); // Converts e to an integer, adds 4 and then interprets the result as a pointer to an integer printf("%d", *((int*)((int)(&(e->a)) + 4)); // Basicly the same way, but now via 'a' which is at the offset of the struct anyway

Structure inheritance

typedef struct {
	int a; 
	int b; 
	int result;
} SumHandle;
void GetResult(SumHandle* handle){
	handle->result = handle->a + handle->b;
}

The example SumHandle struct is a very simple (and useless) structure to store 2 operands, and the result which is filled by calling the GetResult function providing a pointer to the handle.

This works fine, but we could want to add more functionality, for instance why not want to know the result of multiplying those 2 operands? We could make a new struct defenition and a new GetResult function which still uses the old GetResult:

typedef struct {
	int a; 
	int b; 
	int sumResult;
	int multResult;
} SumHandle2;
void GetResult2(SumHandle2* handle){
	handle->multResult = handle->a * handle->b;
	GetResult((SumHandle*)handle);
}

As you can see changing the name of the originaly named result field hasn’t got any effect for only the offset from the offset from the struct offset is used at runtime, therefore I could add the extra field and still use the original GetResult function. This can be done for the original GetResult function only uses the first 12 bytes of the pointed to piece of memory. Adding stuff after that does not effect the original function. Note that the oposite is not possible for changing memory outside your own allocated memory could do really nasty stuff causing the all-too-known GPF‘s.

Virtual functions
In the previous example only an extension has been made to the original behaviour. To overwrite the previous behaviour you need to use virtual functions to overwrite the previous call, the original struct would look like:

typedef void (*GetResultHandler)(void* result);
typedef struct {
	int a; 
	int b; 
	int result;
	GetResultHandler GetResult;
} SumHandle;
void GetResultImpl(void* handle){
	((SumHandle*)handle)->result = ((SumHandle*)handle)->a + ((SumHandle*)handle)->b;
}

The GetResult field is a function pointer which would be set in an initialization function of SumHandle to the GetResultImpl method.

Overwriting the behaviour would be as simple as writing a new initialization function used which would use another GetResult implementation. This for calling resultHandleInstance->GetResult(resultHandleInstance) is nothing more than calling the function at the address specified in the GetResult field.

A type
Every inheritance at the moment still needs a custom constructor and destructor function. Therefore some functionality could not be ashieved like making a copy of a handle for that would need both the constructor function and the actual size of the struct, which both aren’t available when only a pointer is passed. Possibly a custom clone function would need to be provided.
If every object would expose a pointer to a type struct containing information about the struct this problem would be eliminated:

#include <malloc.h>
#include <stdio.h>
// Default defenitions
typedef struct _Type	Type;
typedef struct _Object	Object;
typedef void (*ConstructorHandler)	(Type* type, Object* instance);
typedef void (*DestructorHandler)	(Type* type, Object* instance);
struct _Type {
	int					size;
	ConstructorHandler	Construct;
	DestructorHandler	Destruct;
};
struct _Object {
	Type*				type;
};
Object* Construct(Type* type) {
	Object* object	= malloc(type->size);
	object->type	= type;
	type->Construct(type, object);
	return object;
}
void Destruct(Object* object) {
	object->type->Destruct(object->type, object);
	free(object);
}
Type* CreateType(int size, ConstructorHandler constructor, DestructorHandler destructor) {
	Type* type		= malloc(sizeof(Type));
	type->size		= size;
	type->Construct	= constructor;
	type->Destruct	= destructor;
	return type;
}
// An example implementation
typedef struct _Example Example;
typedef void (*GenerateResultHandler)(Example* example);
struct _Example {
	Type*					type;
	int						a;
	int						b;
	int						result;
	GenerateResultHandler	GenerateResult;
};
void GenerateResultImpl(Example* example) {
	example->result = example->a + example->b;
}
void ConstructExample (Type* type, Object* instance) {
	((Example*)instance)->a					= 0;
	((Example*)instance)->b					= 0;
	((Example*)instance)->result			= 0;
	((Example*)instance)->GenerateResult	= GenerateResultImpl;
}
void DestructExample (Type* type, Object* instance) {
}
Type* CreateExampleType() {
	return CreateType(sizeof(Example), ConstructExample, DestructExample); 
}
void main() {
	Type* exampleType = CreateExampleType();
	Example* example = (Example*)Construct(exampleType);
	example->a = 5;
	example->b = 5;
	example->GenerateResult(example);
	printf("%d", example->result);
	Destruct((void*)example);
	getchar();
}

To add an inheritance:

#include <malloc.h>
#include <stdio.h>
// Default defenitions
typedef struct _Type	Type;
typedef struct _Object	Object;
typedef void (*ConstructorHandler)	(Type* type, Object* instance);
typedef void (*DestructorHandler)	(Type* type, Object* instance);
struct _Type {
	int					size;
	ConstructorHandler	Construct;
	DestructorHandler	Destruct;
};
struct _Object {
	Type*				type;
};
Object* Construct(Type* type) {
	Object* object	= malloc(type->size);
	object->type	= type;
	type->Construct(type, object);
	return object;
}
void Destruct(Object* object) {
	object->type->Destruct(object->type, object);
	free(object);
}
Type* CreateType(int size, ConstructorHandler constructor, DestructorHandler destructor) {
	Type* type		= malloc(sizeof(Type));
	type->size		= size;
	type->Construct	= constructor;
	type->Destruct	= destructor;
	return type;
}
// An example implementation
typedef struct _Example Example;
typedef void (*GenerateResultHandler)(Example* example);
struct _Example {
	Type*					type;
	int						a;
	int						b;
	int						result;
	GenerateResultHandler	GenerateResult;
};
void GenerateResultImpl(Example* example) {
	example->result = example->a + example->b;
}
void ConstructExample (Type* type, Object* instance) {
	((Example*)instance)->a					= 0;
	((Example*)instance)->b					= 0;
	((Example*)instance)->result			= 0;
	((Example*)instance)->GenerateResult	= GenerateResultImpl;
}
void DestructExample (Type* type, Object* instance) {
}
Type* CreateExampleType() {
	return CreateType(sizeof(Example), ConstructExample, DestructExample); 
}
// An inheritance
typedef struct _Example2 Example2;
typedef void (*GenerateResult2Handler)(Example2* example);
struct _Example2 {
	Type*					type;
	int						a;
	int						b;
	int						result;
	GenerateResult2Handler	GenerateResult;
	int						result2;
};
void GenerateResult2Impl(Example2* example) {
	example->result = example->a * example->b;
	example->result2 = example->a + example->b;
}
void ConstructExample2 (Type* type, Object* instance) {
	((Example2*)instance)->a				= 0;
	((Example2*)instance)->b				= 0;
	((Example2*)instance)->result			= 0;
	((Example2*)instance)->result2			= 0;
	((Example2*)instance)->GenerateResult	= GenerateResult2Impl;
}
void DestructExample2 (Type* type, Object* instance) {
}
Type* CreateExampleType2() {
	return CreateType(sizeof(Example2), ConstructExample2, DestructExample2); 
}
void main() {
	Type* exampleType = CreateExampleType();
	Type* example2Type = CreateExampleType2();
	Example* example = (Example*)Construct(exampleType);
	Example2* example2 = (Example2*)Construct(example2Type);
	example->a = 5;	example->b = 5;
	example2->a = 5;	example2->b = 5;
	example->GenerateResult(example);
	example2->GenerateResult(example2);
	printf("example: result: %dn", example->result);
	printf("example2: result: %d result2: %dn", example2->result, example2->result2);
	Destruct((void*)example);
	Destruct((void*)example2);
	getchar();
}

Limitations
Although using types, and casts creates a lot more freedom, it still has its limitations:

No multi inheritance possible
No ‘new’ functions (functions that are only used when a pointer is specificly cast to a struct)
No static support throughout types
No type behaviour inheritance (a type still has its own basetype and isn’t an object itself)

I’ll explain some stuff about wrapping old implementations, or even multiple implementations in the next part, which will overcome these limitations.

‘Objects’ in C – part 1, the basics

Tue, 14 Dec 2004 23:20:30 +0100

(Next part)

C is on memory allocation quite faster than C++ for C++ contains a lot of overhead due to its object orientated nature.
Although not as convenient in C it is possible to get ‘Object’ like stuff in C as in C++.

I’ll explain the basics in this first part, and continue with more complicated (and neater stuff) in the later parts. I hope you’ll find them usefull.

The base
For this example we’ll use a ‘class’ called ‘example’, first the base:

typedef struct{
	int dummy;
} Example;
Example* ConstructExample(){
	return malloc(sizeof(Example));
}
void DestructExample(Example* example){
	free(example);
}
void main(){
	Example* e;
	e = ConstructExample();
	DestructExample(e);
}

Simple inheritance
Lets add some values, and some simple functions and inheritance. Using the inherited class as the base class is just a simple matter of casting via void*.

typedef struct {
	int a;
} Example;
typedef struct {
	int a;
	int b;
} Example2;
Example2* ConstructExample2() {
	Example2* example = malloc(sizeof(Example2));
	example->b = 10;
	example->a = 11;
	return example;
}
Example* ConstructExample() {
	Example* example = malloc(sizeof(Example));
	example->a = 1;
	return example;
}
void DestructExample(Example* example) {
	free(example);
}
void DestructExample2(Example2* example) {
	free(example);
}
void PrintExample(Example* example) {
	printf("%d ", example->a);
}
void PrintExample2(Example2* example) {
	printf("%d ", example->b);
}
void main() {
	Example* e;
	Example2* e2;
	e = ConstructExample();
	e2 = ConstructExample2();
	PrintExample((void*)e);
	PrintExample((void*)e2);
	PrintExample2((void*)e2);
	DestructExample(e);
	DestructExample2(e2);
}

When inheriting you have to copy the original defenition and only append at the bottom, changing nothing of the previous stuff for otherwise you’ll get nasty errors. When you want to override stuff you got to use neat tricks, more on that lateron.

Virtual functions
Using function pointers, virtual functions can be used:

typedef void (*PrintExample)(void* example);
typedef struct {
	int a;
	PrintExample Print;
} Example;
typedef struct {
	int a;
	PrintExample Print;
	int b;
} Example2;
void PrintExampleImpl(void* example) {
	printf("(printexample) %d", ((Example*)example)->a);
}
void PrintExample2Impl(void* example) {
	printf("(printexample2) %d ", ((Example2*)example)->b);
	PrintExampleImpl(example);
}
Example2* ConstructExample2() {
	Example2* example = malloc(sizeof(Example2));
	example->b = 10;
	example->a = 11;
	example->Print = PrintExample2Impl;
	return example;
}
Example* ConstructExample() {
	Example* example = malloc(sizeof(Example));
	example->a = 1;
	example->Print = PrintExampleImpl;
	return example;
}
void DestructExample(Example* example) {
	free(example);
}
void DestructExample2(Example2* example) {
	free(example);
}
void main() {
	Example* e;
	Example2* e2;
	e = ConstructExample();
	e2 = ConstructExample2();
	e->Print(e);
	e2->Print(e2);
	DestructExample(e);
	DestructExample2(e2);
	getchar();
}

A virtual function still has to be supplied with the function in which it has been called. 2 simple macro can be made to make a this call a bit easier, (for some :p):

#define THISCALLPAR(x,y,z) x->y(x,z) #define THISCALL(x,y) x->y(x)

Wrapped inheritance
To override some functionality and retain other functionality while adding your own functionality is virtually impossible by using one simple object. Multi-inheritance would be virtually impossible.
2 parts ahead I will talk about these limitations and how to overcome them, the next part discusses inheritance in more detail, and adding ‘Types’ in the mix.

Intrepid.IO.FragmentedStream

Tue, 14 Dec 2004 19:30:13 +0100

While working on a design of a .pak files, which are nothing more then archives, I noticed most existing .pak files are just very simple. This for they put all files after eachother which makes it really hard to even just add a file, or delete one.

Therefor I started thinking about a way to make an efficient archive file for read and write access.

There are basicly 2 ways to accomplish this:

Clustering
Fragmenting

Fragmenting.
The archive consists out of several files, and an index which is counted as a file. The start of every file is pointed to from the index by its offset. When a file is deleted, its record in the index is null-ed, which is lateron filled with a new entry and in the meantime ignored (for the only file that can start at position 0 is the index itself).

A file itself is fragmented, every fragment starts with 4 bytes marking the length of the fragment and after that again 4 bytes marking the offset of the next fragment this time. In case there is no next fragment present the next-cluster-offset is 0. The first fragment of every file contains after the 8 bytes marking the length and offset of the next fragment some information about the file like the name.

For the index itself is a file, it also is fragmented and can grow, and shrink.

The 4 byte offset value could be default set to 8 bytes in case files are going to be bigger than 2 Gig, and the oposite in case you are dealing with small files. Although this can decrease the size of files which are frequently edited it isn’t possible to increase or decrease to 8 or 2 bytes in runtime.

Clustered
The clustered way is practicly the same as the fragmented way except for that every fragment is padded to a multiple of the cluster size. Therefore the ‘next fragment’ value can now be a cluster number instead of an offset. This will both increase the speed of writing due to having always a buffer, also defragmenting will be faster. The files itself will also be able to retain a 2 byte next-cluster pointer longer. The only downside is that files will just be larger.

FragmentedStream
The fragmented stream wraps these methods pretty efficiently at the moment.
I am thinking to implement an inteligent algorithm which checks which streams are often used which are then placed at the end of the file with a buffer. This also could be applied for files which are marked to be expanding.

Rich Client Side Framework

Tue, 14 Dec 2004 17:13:00 +0100

On several blogs the idea of having a rich java script passed, for example on ZefHemel.com: Rich Web UI: Search As You Type

Guess due to google, which has made a neat Webmail interface for gmail and Google suggest with find as you type.

The demands on java script keeps growing. People want to make better webUI’s and features with Javascript although javascript is defenitely not designed for this stuff.

Using flash, and java is an overkill, but using javascript is espacially an overkill for javascript isnt handled consistantly on different browsers, and isn’t as quick and maintainable as it could be.

I guess it would be time to extend HTML itself with a more advanced script; java like preferably although then directly supported by the browser, and less aimed at custom drawing but using an API provided by the browser.

I’m currently experimenting with Microsoft .net assemblies which are downloaded in slimmed form as webpage which are executed with very limited access. Which works neat although it is still an overkill (a .dll is about 20 Kb, even if you got only one line of code..)

Just a thought.

OO Stated Stackbased Parsing

Sun, 12 Dec 2004 13:25:29 +0100

Every parsable format (like INI, XML) consists out of certain area’s. You can be parsing the section name at one moment, or be parsing a comment when parsing an INI. These certain area’s where you can parse result in a parse state. In every state you expect something else, and you gather other kinds of information.

When you are parsing in a certain state you can find that the state has changed (the parser found a new xml node in a xml file), then the old state is pushed on the state stack.

In certain circumstances you need to have the ability to fall back to a previous state, this can happen when you are parsing a apparently new section name and suddenly there is a comment character instead of a sectionname end character. In this case you need to be able to fall back on the previous section you were parsing. Although when you successfully have parsed the sectionname you want the old sectionstate removed from the stack (and the data of it emited).

An example of a state based parser is this INI parser I wrote in C#:


 public override void Load(Stream stream)
 {
 StreamReader sr = new StreamReader(stream);
 Stack<parsestate> stack = new Stack<parsestate>();
 stack.Push(ParseState.NoSection);
 StringBuilder sb = null;
 IniSection section = null;
 IniEntry entry = null;
 while (true)
 {
 char c = (char)sr.Read();
 ParseState s = stack.Peek();
 if (s == ParseState.NoSection)
 {
 if (c == 32 || c == 9 || c == 10 || c == 13)
 {
 // To nothing.
 }
 else if (c == '[')
 {
 stack.Push(ParseState.SectionName);
 sb = new StringBuilder();
 }
 else
 {
 stack.Push(ParseState.TillNewLine);
 }
 }
 else if (s == ParseState.TillNewLine)
 {
 if (c == 13 || c == 10)
 {
 stack.Pop();
 }
 }
 else if (s == ParseState.SectionName)
 {
 if (c == 13 || c == 10)
 {
 stack.Pop();
 }
 else if (c == ']')
 {
 stack.Clear();
 stack.Push(ParseState.EntryName);
 stack.Push(ParseState.TillNewLine);
 _Sections.Add(section = new IniSection(sb.ToString()));
 sb = new StringBuilder();
 }
 else if (c == ';')
 {
 stack.Pop();
 stack.Push(ParseState.TillNewLine);
 }
 else
 {
 sb.Append( c );
 }
 }
 else if (s == ParseState.EntryName)
 {
 if (c == 13 || c == 10)
 {
 string result = sb.ToString().Trim();
 if (result != "")
 {
 section.Entries.Add(new IniEntry(result));
 }
 sb = new StringBuilder();
 }
 else if (c == ';')
 {
 string result = sb.ToString().Trim();
 if (result != "")
 {
 section.Entries.Add(new IniEntry(result));
 }
 stack.Push(ParseState.TillNewLine);
 sb = new StringBuilder();
 }
 else if (c == '=')
 {
 section.Entries.Add(entry = new IniEntry(sb.ToString().Trim()));
 stack.Push(ParseState.EntryValue);
 sb = new StringBuilder();
 }
 else if (c == '[')
 {
 if (sb.ToString().Trim() == "")
 {
 stack.Push(ParseState.SectionName);
 sb = new StringBuilder();
 }
 else
 {
 sb.Append( c );
 }
 }
 else
 {
 sb.Append( c );
 }
 }
 else if (s == ParseState.EntryValue)
 {
 if (c == ';')
 {
 entry.Values.Add(new TextIniValue(sb.ToString().Trim()));
 sb = new StringBuilder();
 stack.Pop();
 stack.Push(ParseState.EntryName);
 stack.Push(ParseState.TillNewLine);
 }
 else if (c == 10 || c == 13)
 {
 entry.Values.Add(new TextIniValue(sb.ToString().Trim()));

 sb = new StringBuilder();
 stack.Pop();
 stack.Push(ParseState.EntryName);
 }
 else if (c == ',')
 {
 entry.Values.Add(new TextIniValue(sb.ToString().Trim()));
 sb = new StringBuilder();
 }
 else
 {
 sb.Append( c );
 }
 }
 }
 }

I am now investigating whether creating a more Object Orientated implementation would be feasible, and even more importantly whether with this technique it would be possible to write a state description file which the parser reads, and just produces the data which is meant to be captured as described in the state description file.

Parsing $_SERVER[‘PATH_INFO’]

Sun, 12 Dec 2004 12:20:34 +0100

The PHP global variable $_SERVER['PATH_INFO'] contains the path suffixed to a PHP script, if I would call the URL:

http://domain.ext/path/to/script.php/foo/bar.htm?a=b&c=d

Then $_SERVER['PATH_INFO'] would contain:

/foo/bar.htm

Traditionaly the $_GET variables are used for certain parameters like a page to display:

http://domain.ext/page.php?page=about.htm

This method is easy to program, but not only looks strange, but also is very search engine unfriendly. Most searchengines ignore the QueryString (the part of the URL after the ?). And therefor would index the first page.php?page=x they would find and ignore the rest.
Some searchengines like Google do not ignore the query string, but would give a page without using a querystring for different content a way higher ranking.

Parsing the $_SERVER['PATH_INFO'] is relatively easy, this code would do most of the stuff just fine:

if (!isset($_SERVER['PATH_INFO'])){
	$pathbits= array('');
}else{
	$pathbits = explode("/", $_SERVER['PATH_INFO']);
}

The $pathbits array would always contain / as first element if a path info was provided, otherwise it will be an empty array.

Here is a quite simple example which parses the path info to decide which file to include:

<?php
if (!isset($_SERVER['PATH_INFO'])){
	$pathbits= array('');
}else{
	$pathbits = explode("/", $_SERVER['PATH_INFO']);
}
if (!isset($pathbits[1]) || $pathbits[1] == ""){
	$page = "default"
}else{
	$page = basename($pathbits[1]);
}
$file = "./pages/{$page}.php";
if (!is_file($file)){
	echo "File not found";
}else{
	require $file;
}
?>

Modular Server

Sat, 11 Dec 2004 19:01:24 +0100

(Understanding URIs)

“A common mistake, responsible for many HTTP implementations problems, is to think this is equivalent to a filename within a computer system. This is wrong. URIs have, conceptually, nothing to do with a file system. One should remember that at all times when dealing with the World Wide Web.”

So why do most HTTP servers still heavily rely on the filesystem dictating URL’s?

This not only tends to create Uncool URI’s, but also makes it seem logical for filebased dynamic content (more on that in my previous post).

A http server, actually every server should be nothing more then a wrapper for modules which handle requests of clients.

The server would only limit itself to a very selected amount of functions

Handling connections
Exposing an API for the protocol which the modules can us
Hosting modules

On startup the server loads modules and binds them to certain URL. Modules remain persistant in the memory and are just signalled a request is made passing the module an API to handle the request.

I’ll be busy exploring the posibilities to implement this.. there will be more about this.

Why server side scripts aren’t that scalable at all

Sat, 11 Dec 2004 14:58:27 +0100

There are a lot of different types of server scripts, a few examples:

Practicly all of these were to be used via CGI, but most of them created tighter intergrations for popular webservers like for the Apache Http Server. Usualy the tight implementations for apache are in the form of modules like mod_python to use the general purpose python scripting language as a server side scripts.

When you, as client, request a page powered by a server side script via CGI (like for instance this blog at the time of writing), the webserver starts the responsible interpreter providing it with some arguments like the POST, GET variables, which then are processed by the PHP interpreter executing the php script, which provides a stream to return to the client.

This works pretty well with little, not too demanding, scripts.
However, problems arise when a lot of people use the scripts or when the script itself is quite demanding.

An example could be this blog, this blog uses a mySQL database to store its posts and comments. Every time the index page is requested it makes a new connection to the mySQL database server and send a query for the categories, links, latest posts, latest comments, etc. Creating a mySQL connection takes time, sending queries takes time, processing queries takes time for the mySQL server, retreiving the results takes time, processing the results take time, this all just to produce the same content for the index page over and over again, for there are at least 100 times more visits than updates to this blog.

Some blogs build there content. When you are viewing posts on these webblogs these posts are not generated for your request but are cached (usualy as normal .html files on the webserver). The control panel to post new blog items is however written in a server side script, and utilizes some sort of database. When you are finished creating new posts the php script will rebuild the cached pages from the database which will significantly reduce the server stress.

The downside of these types of blogs is that they contain only a very limited amount of dynamic features, for that would require scripts. Another downside of these blogs is that it is very hard to get such a blog hosted by multiple servers at the same time which usualy happens when a site is very popular and one server can not handle the demand. This is possible for the database powered blog for it stores its data on one centralized datbase server. However, a database powered blog will most likely require more than one server quite soon for it is a far greater strain on the server then a caching weblog. However, it is possible to be done by setting up the blog in such a manner that it stores its cached files on a centralized server too (by normal filesharing). However, I would be suprised when a server using cached pages will ever reach the limit of its server capacities.

The problem gets bigger when you are dealing with more dynamic server side software like forums. A forum requires some queries. Like a query which retreives the posts, it has no use to cache them for visitors of forums tend to post (yeah.. I know, it’s strange), which would require caches to be updated (which would be an even bigger strain then just using the database anyway).

A forum however still has got a lot of stuff that is quite static and would run a lot faster if it could be cached. This would be stuff like the templates, the categories architecture, the help pages, the statistics like user count, and the sessions, which are very dynamic but are queried by every page view. These things usualy are requested in every time you download a page.

Some forums use a cache which consists out of a table in the database which contains all cached stuff serialized so it can be immediately used. But this still means about 10 Kb transfered from the databaseevery time someone views your page!

This problem even grows bigger when you are developing even more demanding server side projects like a browser based online game.

I started developing an online game as just a hobby project in PHP, but I soon switched to making my own webserver in C# which caches all the stuff in the memory of the webserver which makes the rewritten stuff 3 times faster.

I figured it would be great if http servers and server side would be less aimed to just handling one request but create more support for inter-request caching. There is limited support for caching in JSP and ASP.net but this is used quite rarely for JSP and ASP.net still focus on just requesting. A server side script should not be loaded on request but rather be loaded already in the form of being able to cache objects like a provider of mySQL connections, which just recycles a connection; a function class which contains all the common used functions already loaded; and off course stuff like templates, sessions, and other cachable things.

The problem about caching in the memory is that memory can’t be shared between multiple servers, so it isn’t realy scalable. When you would multiple servers and store sessions in the memory it is quite possible that members would just get a “session not found” error when clicking a page which is served by the other server. A possible solution to this problem could be to redirect people when they access “domain.ext”, to a mirror (“s12.domain.ext”). This would avoid session loss. When adding a ‘cacheversion’ value in a table on the shared database server which is changed everytime something is changed for which a cache is created. Requesting just this very small number would be enough to check whether the cache should be rebuild for another server has changed something.

Just a thought…

My blog

Sat, 11 Dec 2004 13:28:22 +0100

This is my blog, don’t expect a lot of posts, for I just aren’t good in writing text. (had to correct this post twice :-))
I’ll just post some thoughts, I hope others will find them interesting.