Hacking and Other Thoughts

Wed, 27 Jun 2007


Linux guest support for Sun LDOMS...

I just recently finished writing preliminary support for Linux to run as a guest under Sun LDOMs. This was the impetus behind my recent itch for virtualization issues.

First, I'd like to thanks Ashley Saulsbury, Narayan Venkat, Greg Onufer, and other engineers at Sun for helping me out. I'd also like to thank Fabio Massimo Di Netto for helping me put together early ramdisk userland and installer images. His tireless work and useful feedback allowed this to be brought up so quickly.

Things are in a bit of a rough state, but you can play around with installing a basic Linux guest with Solaris running the control node. There is a lot of missing functionality, and several major problems to resolve.

Anyways if you want to play, start with something like Solaris 10 U3 on your t1000 or t2000 box. Make sure all current patches are applied. Update your firmare to LDOMs capable firmware and grab the LDOMS-1.0 userspace et al. from: Sun's LDOM site .

Once you have the control node configured and rebooted into (follow Sun's instructions, the documentation is rather clear on how to set things up), you can create a guest on which to test Linux. For my testing I gave it one virtual network interface, and exported an 8GB file (created with 'mkfile 8g file') as the virtual disk.

Startup the Linux guest and net-boot the test Ubuntu gutsy installer image at this location.

The install should go pretty smoothly. It will say that it cannot find any kernel modules, just say "Yes" and let it keep going. Since the ubuntu tree doesn't have a LDOM capable kernel yet you'll need a kernel to boot into your new Linux guest, I've provided a netboot image here just for that purpose. You should be able to boot into your newly installed guest using something like:

ok boot net root=/dev/vdiska2
or similar.

There are two major issues which are being resolved, which you should be aware of once you have a guest installed.

First, if you unbind the guest from the control node or reboot the control node, the vds Solaris driver is going to write it's version of the disk label back to the disk. You will lose your Linux disk label when this happens. One way to work around this is to save a copy of your disk label after your install, for example:

	dd if=/path/to/exported_disk_file of=linux_label.img bs=512 count=1
If vds scrambles your disk label, start the guest and leave it at the OBP prompt, then restore your disk label with something like this:
	dd conv=notrunc if=linux_label.img of=/path/to/exported_disk_file \
		bs=512 count=1
Be VERY VERY careful when doing this stuff, in particular if you forget the conv=notrunc option when restoring the label or get the bs or count args wrong, you will destroy your Linux guest install.

Trust me, I've done it 5 or 6 times already :-) Anyways, the Solaris folks are hard at work fixing this vds bug so that it will leave the disk label alone and we won't have to deal with this problem any more.

Secondly, there is a bug I'm still tracking down that hangs the machine during lots of network activity. Full installs and normal usage work fine.

All of the current LDOM kernel code can be found in a GIT tree up at:

	kernel.org:/pub/scm/linux/kernel/git/davem/ldom-2.6.git
The only major piece of missing functionality on the guest is lack of a domain services driver, which allows one to reboot, shutdown, and hotplug cpus from the control node. That's the next item I plan on implementing.

After getting all of the guest functionality in place I hope to work on the infrastructure necessary to allow Linux to act as a control or service node, but that's quite a ways off.

There are all sorts of technical details I'd like to discuss about Sun LDOMs and the Linux support, but this is a long enough blog entry for now. Stay tuned :-)