Category: General

Tickle

Tickle is a small Python serializer like Pickle. It however aims at generating smaller output:

>>> len(tickle('hello'))
7
>>> s = StringIO.StringIO()
>>> pickle.dump('hello', s)
>>> len(s.getvalue())
13

Though the difference is and remains quite small, this alone is useful for serialization of small things in the case of for instance RPC. However, usually you already know what kind of data to expect and you don’t really bother about the type information. This can be done by specifying a template:

>>> obj = []
>>> for i in xrange(100):
       obj.append((i, str(i)))
>>> len(tickle(obj))
629
>>> len(tickle(obj, template=(tuple, \
   ((tuple,((int,), (str,))),)*100)))
390

(Instead the *100 an iterator could be constructed, but that would clutter the example even more than it already is.) In comparison:

>>> s = StringIO.StringIO(); pickle.dump(obj, s)
>>> len(s.getvalue())
1680

One big disadvantage of Tickle is speed. Pickle has got a nice C implementation, which is quite fast. Psyco helps a bit but not really enough for really big things. Even more so pickle is a bit smarter: it builds a LUT for instances to avoid duplicate data. However, in the situations where Tickle will be used (by me at least) that isn’t too big of an issue.

You can download tickle.py via gitweb.

Virtual packages in python

When writing an application in python, it can be very convenient to be able to import a module from the top of your module tree. Eg. import myapp.config instead of (python 2.5 only) relative imports: import ....config. To do this one would have to make myapp a package. The normal way to do this is to put your application directory (which now has to be named myapp) somewhere in python’s package search path. This isn’t all to convenient.

The solution: manually set up your package module:

import os
import sys
import imp
def setup_virtual_package(name, path=os.curdir):
    """ Sets up a package at the given path with a given
     name """
    modulePath = os.path.abspath(path)
    f, fn, suffix = imp.find_module('__init__',
         [modulePath])
    imp.load_module(name, f, fn, suffix)
    sys.modules[name].__path__ = [modulePath]

Now import myapp.something works like a charm.

Section 202c of the German computer crime laws

This section has come into effect over the weekend. It makes it illegal to create, possess, obtain, provide access to, yield, distribute or otherwise allow access to lots of widespread tools that can be used to breach security. Take for instance nmap.

This law does not only impede our freedom (of speech), research, decrease security and allow for misuse, but more importantly it won’t even stop the real criminals.

Stefan of the Month of PHP Bugs Project writes:

The law does not affect our freedom of speech to report and inform about security vulnerabilities and how to exploit them.

We are just not allowed to create/distribute/use software that could be used as “hacking tools”.

Where would they draw the line between reporting/informing about a vulnerability and how to exploit it and the actual source code to do it. Would pseudocode be illegal? Would literate code be illegal? Also there would be no way for security researchers to try out their work.

What will happen in the worst case if similar laws are accepted in other countries and enforced, is that vendors will rather cover up all vulnerabilities using these laws instead of securing it. That there are lots of ready-to-use exploits is good. It’s a very good incentive for security.

That there will always be a leak in a piece of software that someone will be able to find on his own will not be changed by this law. Also there will be no way to stop the real criminals from creating and distributing tools underground. Now everyone still knows what kind of tools are around and will know what to expect.

Wacken

I’ll be up early tomorrow, Saturday, to take the train with lots of friends to the little town of Wacken in Germany. (Actually, there is no train station in Wacken, so we need to hire a cab someway for the last dozen kilometers)

Sunday morning the camping of the Wacken Open Air festival will open. The festival itself will start on Thursday. I hope to be home again the first Sunday of August.

“Nothing to hide”

In this short essay, written for a symposium in the San Diego Law Review, Professor Daniel Solove examines the “nothing to hide” argument. When asked about government surveillance and data mining, many people respond by declaring: “I’ve got nothing to hide.” According to the “nothing to hide” argument, there is no threat to privacy unless the government uncovers unlawful activity, in which case a person has no legitimate justification to claim that it remain private. The “nothing to hide” argument and its variants are quite prevalent, and thus are worth addressing. In this essay, Solove critiques the “nothing to hide” argument and exposes its faulty underpinnings.

“I’ve Got Nothing to Hide” and Other Misunderstandings of Privacy

Not only is its subject very relevant, the Essay is very well written and a pleasure to read.

Graduated

Today I received a long anticipated phone call from my mentor who took away the doubt and let me know that I graduated for my VWO exams at the Stedelijk Gymnasium Nijmegen, which is a matriculation exam.

With mixed feeling I look back at seven long years. It certainly shouldn’t have took 7 years, nor 6, not even 3. But espacially in my last few years I came to appreciate the tons of very different people with whom I spend the days. A diversity I won’t find in that extend within the students of Maths and Physics which I intend to study next year at the Radboud University in Nijmegen.

For one thing I certainly wouldn’t have want to have missed these last few years. Some say they’re the best of your life. I do really look forward to university, though.

For those interested, I’ve planned an “Examenfeest” with some others in a nice club in Nijmegen. Send me an e-mail if you happen to be in the neighborhood and care to join.

Upgrading wordpress with git

I didn’t like upgrading wodpress much. Everytime I did it, I needed to re-apply all my little tweaks to the new wordpress. It took too much time.

I tried to diff -uNr on the current version I was running and the newer version and then applying the resulting diff to the current version, but it seems wordpress has been backporting changes so I got conflicts, quite a lot of them.

Because I was quite tired of porting my changes, I’ve tried git, the Source Code Managment tool used by the linux kernel, to do it for me:

I did this all in the parent directory of the root of blog.w-nz.com. This folder contains:

  • htdocs current installation (2.1.2)
  • 2.1.2 the unmodified wordpress
  • 2.2.0 the new wordpress I want to upgrade to

First, I created an empty git repository:

mkdir git; cd git; git init-db; cd ..

Then I copied over the unmodified version of wordpress I was running, and commited them:

cp 2.1.2/* git -R
cd git
git add *
git commit -a -s
cd ..

Then I copied over my current installation:

cp htdocs/* git -R
git status # lets see what changed

There are lots of files like uploads I want git to ignore, so I edit .gitignore to make git ignore them. There weren’t any files I added though, otherwise I’d had to run git add to let git know.

And let commit my changes:

git commit -a -s

Now, lets go back to the original commit — the clean 2.1.2 wordpress — and start a branch from there:

git checkout HEAD^ # HEAD^ means parent commit of HEAD: the previous commit
git checkout -b tmp # create a new branch tmp from here

Now I’m in a branch without my own changes, which was forked from the master branch. Lets apply the new wordpress on this branch:

cd ..
cp 2.2.0/* git -R
cd git
git status # see what changed

git-status showed me that there are a few new files in wordpress 2.2.0, I git-add-ed all of these new files. And then committed it all:

git commit -a -s

Now I’ve got two branches:

  • master which contains wordpress 2.1.2 with my own changes on top as a commit
  • tmp which is forked from the wordpress 2.1.2 from the master branch without my own changes but with the 2.2.0 changes on top

What I want to do is to reapply the 2.2.0 changes on top of my current changes’ commit instead of on top of the 2.1.2 commit. To do this, git has a very powerfull util called git-rebase:

git rebase master

This will search down the tree until the point where the current branch (tmp) forked from the target branch (master). Then it will re-apply all commits in between on the latest commit of the target branch.

Just like if I’d use diff/patch I get a merge conflict. git rebase lets me know this and git status shows me which one are these. The one little difference with the diff/patch approach is, that there are way less merge conflicts (git is smarter) and that the merge conflict are way easier to identify and they’re inline in the original files. Not to mention that when I would have fucked up I’d always have a way back.

After I fixed the merge conflict, I git update-index each conflicted file (to tell git it’s resolved) and git rebase --continue-ed.

Now I’ve got my updated wordpress in the git folder. Then I backuped the current, copied over from git and visited wp-admin/upgrade.php and I’m done :).

By the way: “I didn’t say Subversion doesn’t work. Subversion users are just ugly and stupid.” — Linus on this Google tech talk.

Sidenote, I switched from Hashcash to Akismet. Hashcash didn’t work anymore and Akismet theoretically should be the best solution because it isn’t based on security by obscurity.

This is How We Catch You Downloading

torrentfreak.com has acquired a document how a british company is tracking down illegal use of P2P: This is How We Catch You Downloading.

Basically they use a modified P2P client which searches for infringing content, download it and if that works and is indeed is the content then they do a whois on your IP and send a infringement notice to your ISP. The best thing is that they claim that this provides enough proof that you really are infringing.

They probably never heard about botnets.

C&C 3 on Linux

I’ve got Command and Conquer 3: Tiberium Wars running on my Gentoo Linux installation with wine 0.9.34 by following the instructions here. I had to first install it on windows though, and copy the folder for the installer didn’t work, even with Crossover Office.

Except for (very glitchly) video and sometimes a crash everything seems to run. (Didn’t try multiplayer yet though). I had to put all quality settings to lowest, which makes me wonder whether that is my radeon X1400 being not so good as I expected or wine just being slow in emulating Direct3D.

Watermarking media

It seems the new trend of the music industry against piracy is watermarking movies/audio/etc.

Content is water-marked by adding very small (unnoticeable) changes that could store something like a rsa based certificate to identify a given audiotrack.

Originally I thought they’d use it to track down the source of an illegal download. It sounds illogical to me because it’s hard to keep watermarks when format is changed (mp3, ogg and others really do mess up slight unnoticeable differences because otherwise they wouldn’t compress as good). And when someone has got two versions of the same audiotrack one can compare them and find out how something is watermarked.

Maybe the scheme of the industry isn’t that stupid, but the other way around (and a lot more evil). Maybe just sue everyone who hasn’t got a watermark on their movies or mp3.

Luckily a Fair Use bill was passed which they say (haven’t checked) allows fair-use conversion of format of media.