February 2006 – Intrepid Blog

Gentoo linux 2006.0 installer

Today Gentoo released a live-cd with a gentoo-installer on it. They also posted some nice screenshots.

Gentoo Linux is a linux redistribution that features its home-made package manager Portage. This package manager makes Gentoo special. There are a lot of other package managers, but Portage is one of the only package managers that compiles each package on your computer, instead of installing precompiled binary’s.

Compiling everything yourself has got the advantage that you can tweak your package to your specific needs. This means you can have a nice customized system. But the key advantage is performance. Most binaries that package managers install are compiled for generic x86. That means that it can even run on your old 486. The issue is that a lot has changed since the 486. A few new instruction sets (MMX, SSE, etc), specifically designed to increase performance for common tasks, are present in almost every processor which aren’t used by generic x86 compiled binaries. And even more performance advantages¹.

For this reason some people already did this (see Linux From Scratch). The problem is that it takes a lot of time to configure, build, install and trouble shoot at least 200 packages. If you take into account that your average server and desktop have at least 15 updated packages per week you are looking at a huge amount of dedication and time required to keep your own Linux From Scratch up to date.

This is where Gentoo and Portage comes in. If you want apache, you simply type emerge apache. If you want to update everything, you simply type emerge -u world. Portage checks which packages are required, it downloads them, it patches them, compiles them and installs them for you.

But wait? Maybe I don’t want ssl build into apache. How do I do that? Fairly simple actually. Portage has got a system called use-flags. There is a useflag called ssl. You can put ~ssl in the USE entry in /etc/make.conf to tell to portage that you don’t want ssl on any package. Or you could put net-www/apache ~ssl in /etc/portage/package.use, which tells that you want ssl to be disabled for apache.

Off course Portage isn’t perfect, and you will have some trouble once in a while, but it’s better than the headaches caused by LFS and the reward is similar2.

— A happy Gentoo user.

¹When compiling for generic x86 a lot of memory is aligned which causes a lot of overhead for processors that don’t really care a lot about alignment. Not only are there these nice new instruction sets, but each processor has got specific differences in the implementation of the instruction. On some processors instruction X may be faster than instruction Y. Also some processor information in tight loops can be useful. On one processor the loop may fit in L1-cache, on others it wouldn’t.

²You can offcourse do more with LFS. But usually it isn’t that important, and if it were it would be better to add it into Portage yourself than to switch to LFS for that reason.

Xgl

After a long night wrestling with alfa source code, I’ve managed to install Xgl.

Xgl is a Xorg-X11 layer that uses openGL to achieve some nice stunning effects.

One of these is to be able to switch desktops by pressing ‘Ctrl+Alt’ and dragging your desktop-cube.

There are a lot of other things that I can’t show with screenshots. Take a look at the Xgl release post. These things include that all forms behave flexible. If I drag a form it’s like it’s made of rubber instead of concrete. Also every form pops up gently animated. There’s also a mac osX expose-clone, which is really helpfull.

If you want to install it yourself then the gentoo wiki article and hanno’s blog post should be very helpful.

Virtualization isn’t hardware efficient

Virtualization is the thing right now. Escpacially because VMWare released VMServer for free.

I’ve read quite a lot of posts about the pro’s about virtualization, and I`m annoyed by some.

This for some claim that virtualization is hardware efficient. Quite frankly, it isn’t. It is quite inefficient.

The thing though is that you don’t use your server optimally usually, and that with virtualization you can more easily get most out of your server, although you could get more from your server if you wouldn’t use virtualization. But that is a lot trickier, virtualization is quite flexible.

Virtualization doesn’t make hardware more efficient, it makes you more efficient at allocating resources of your server.

This server, w-nz.com, runs on a virtual server. I can’t affort or would fully use a ful dedicated server, but I can affort and use a virtual server. Efficient. But running all sites on the real server on just one server would be more efficient, but harder to do, espacially related to security.

Virtual locality of reference

Quite a lot of programmers consider the advantages of locality of reference; if they put all the data they work with in about the same space the processor will be able to retreive them more easily, for the processor caches regions of memory in the Lx caches.

Quite a lot of memory managers even move objects for you in the same place which seem to depend on each other. This doesn’t only decrease fragmentation, for they would be freed in the same timespan, but also increase performance due to locality of reference.

Although the advantage of locality of reference isn’t as big as most people presume in some situations due to virtual memory.

The address space a process sees is a virtual address space, it is mapped in place by the OS. This means that 0x1234 can contain an image for one process and a normal integer for another process. This enables two processes to be loaded, virtually, in the same place.

One implication this has on locality of reference is that a seamingly contiguous piece of memory can in physically be distributed over quite a big span of real physical memory. One part of your list could be at 0x1300, the other at 0x30240.

If your program wants to enumerate through that list it`ll have to query the RAM for the second part, which it wouldn’t have to if it would be contiguous in the physical memory, for then it would most likely be in complete in the L2 cache.

The physical memory is mapped page by page into the virtual memory space. Usually pages are 2KB. Each page is always contiguous in physical memory for it can’t be split up by mapping. Therefore locality of reference always works within a page.

It doesn’t mean that if you are working with more than a page of memory it will be all fragmented. The OS usually does a good job at keeping pages contiguous in physical memory when it is contiguous in virtual memory.

Although when a computer is under some load and it’ll have to use each bit of memory and swap in and out memory it will likely be a lot more fragmented.

I don’t want people to be ignorant about the advantage of locality of reference when dealing with more than a page of memory. I want people that trade of literally everything for their holy locality of reference to see that it’s just a nice extra, but that it just doesn’t always work due to fragmentation of the physical memory.

Side notes
Virtual memory is a bit more complicated than I explained, read about it on wikipedia.
Processors will try to keep virtual memory address space in mind to avoid this issue, although it’s limited.