I have a sweet deal on a Virtual Private Server. I opted for the “Dev” model, which comes with little support and expects that you are very familiar with your chosen OS. I chose Ubuntu, which comes in 9.04. The host provides this through Virtuozzo, which is a specialized kernel that provides separation of each OS. If you can configure Linux, this is a great VPS solution. However every time I issued an apt-get, I ran into problems. The first problem was messages from any “apt-get install” which told me that there were permission denied problems:

Can't exec "[blah]": Permission denied at /usr/share/perl/[version]/IPC/Open3.pm

What’s happening is the VPS host mounts /tmp with noexec,nosuid. There is nothing wrong with that, I prefer it that way, but apt-get was not configured to deal with that. The solution is to add two lines to the end of /etc/apt/apt.conf.d/70debconf:

DPkg::Pre-Invoke{"mount -o remount,exec /tmp";};
DPkg::Post-Invoke {"mount -o remount /tmp";};

The second problem I ran into was the fact that every time I issued apt-get upgrade and then rebooted, I could no longer connect to my host. I would have to reinstall from the template using the control panel provided by my host. Ugh. Trying to troubleshoot this becomes arduous at best, because I have no alternate way to connect once the server becomes unresponsive. I can see that all the processes are running from the web-based control panel, including sshd, but I just can’t connect. My solution? I wanted to see if there was something about the upgrade that would break, so I copied my entire OS to a folder within the root folder:

mkdir /root/os
cp -a bin boot etc home lib media mnt opt sbin selinux srv tmp usr var /root/os
mkdir /root/os/root
cp -rp /root/.* /root/os/root
mkdir /root/os/dev
mkdir /root/os/proc
mkdir /root/os/sys

I chose to put things in the root folder just because I don’t like to sully the root directory. I also took care to copy in my dot files so that my bash prompt would still be pretty. Also you should not copy the system folder: dev, proc, sys. Those need to mounted. I issued the following commands to move into the copied OS. I later turned that into a script while experimenting with other OS’s:

#!/bin/sh
#
# switchto OSFOLDER

if [ -e "/root/$1" ]; then
        mount -t proc none /root/$1/proc
        mount -o bind /dev /root/$1/dev
        mount -o bind /sys /root/$1/sys
        mount -t tmpfs none /root/$1/tmp

        chroot /root/$1 /bin/bash -l

        umount /root/$1/tmp
        umount /root/$1/sys
        umount /root/$1/dev
        umount /root/$1/proc
else
        echo Negative.  "$1" is not an option.
fi

This takes care of setting up the system folders and changing the root. I type /root/switchto os and am seated comfortably in a copy of the original OS. I issue apt-get update followed by apt-get upgrade. After occupying myself for a little while I find that all my packages have been upgraded. Everything works. Just to be sure, given the previous problems I ran into, I reboot. Everything is fine. I shell in and issue another /root/switchto os, and here I am, same OS, upgraded packages. Wonderful. I run a distribution upgrade:

apt-get install update-manager-core
do-release-upgrade

After running the distribution upgrade I do the reboot test again, and I still find that I my host OS functions normally, and from a shell I can switch to the new distribution. Wonderful. I ran another distribution upgrade. Ubuntu 10.04 is the current stable distribution upgrade. You can force it upgrade to 10.10, but that is not officially supported through the upgrade path.

So last problem, I’m not actually running 10.04, I am switching to it in a single shell. I was considering a call the my switchos script from within the init.d process, but my work with my router tells me that’s a bad idea. After a bit of experimenting I decided to replace my host operating system’s init since it’s what the kernel does after it is ready to rock. I was hoping I could replace /sbin/init with a script, but no such luck. I ended up writing a C++ program to do the job.

// switchinit.c
//
// Written on: 2010.12.08
// Chris Hanson (c) 2010
//
// Copy all you want, change all you want, leave/append this header, give credit.
//
// Compile as "init"
// Rename old init as "init.original"
// This replaces that init
// Create a file named /root/.initos
// Place one string on one line indicating the path of the new root.
//
// There's lots to fix.  Like checking for trailing spaces in ".initos"
// Better usage of memory, maybe proper placement of the switched OS

#include <time.h>
#include <fcntl.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <sys/mount.h>
#include <sys/stat.h>
#include <sys/types.h>

char* ReadOsLocation( char* name );
int IsValidLocation( char* location );
void DumpEnv( char* name, char** envp, char** argv, int argc, char* os );

int main( int argc, char* argv[], char* envp[] )
{
        // Locals
        char szProc[1024];
        char  szSys[1024];
        char  szDev[1024];
        char szRoot[1024];
        int stream;

        // My host likes to mount tmps for me, undo that
        umount( "/tmp" );
        umount( "/var/tmp" );

        // Run injection
        if( IsValidLocation( "/sbin/injection"  ) )
        {
                // Launch primary injection (can be script)
                system( "/sbin/injection" );
        }

        // Read location
        char* location = ReadOsLocation( "/root/.initos" );

        // Ensure trailing slash
        if( location[strlen( location ) - 1] != '/' )
                location = strcat( location, "/" );

        // Do we have a valid, new location?
        if( strcmp( location, "/" ) != 0 && IsValidLocation( location ) != 0 )
        {
                // Load up string
                strcpy( szProc, location );
                strcpy( szSys, location );
                strcpy( szDev, location );
                strcpy( szRoot, location );
                strcat( szProc, "proc" );
                strcat( szSys, "sys" );
                strcat( szDev, "dev" );
                strcat( szRoot, "host" );

                // Mount system folders into destination
                mount( "none", szProc, "proc",  0,       "" );
                mount( "/sys",  szSys, "sysfs", MS_BIND, "" );
                mount( "/dev",  szDev, "",      MS_BIND, "" );
                mount(    "/", szRoot, "",      MS_BIND, "" );

                // Change root
                setsid();
                chdir( location );
                chroot( "." );
        }

        // Otherwise change target file to avoid looping
        else argv[0] = "init.original";

        // Mount our tmp
        mount( "none", "/tmp", "tmpfs", MS_NOEXEC | MS_NOSUID, "" );

        // Log
        DumpEnv( "/var/log/initos.log", envp, argv, argc, location );

        // Redirect streams for debugging
        if( ( stream = open( "/var/log/initos.log", O_RDWR | O_APPEND ) ) != -1 )
        {
                dup2( stream, STDOUT_FILENO );
                dup2( stream, STDERR_FILENO );
                close( stream );
        }

        // Run injection
        if( IsValidLocation( "/root/injection"  ) )
        {
                // Launch secondary injection (can be script)
                system( "/root/injection" );
        }

        // Move to sbin
        chdir( "/sbin" );

        // Reload using new init
        execve( argv[0], argv, envp );
}

// Reads a string from a file.  Calls malloc no matter what.
char* ReadOsLocation( char* name )
{
        // Locals
        FILE* file;
        char* buffer;
        unsigned long length;
        int c;

        // Open file
        file = fopen( name, "rb" );

        // File doesn't exist?
        if( !file )
        {
                // We're going to just use this OS
                return "/";
        }

        // Get file length
        fseek( file, 0, SEEK_END );
        length = ftell( file );
        fseek( file, 0, SEEK_SET );

        // Allocate memory
        buffer = (char *)malloc( length + 1 );

        // Memory problem?
        if( !buffer )
        {
                // Clean up
                fclose( file );

                // Run away!
                return "/";
        }

        // Read the file contents
        fread( buffer, length, 1, file );

        // Clean up
        fclose(file);

        // Trim string
        for( c = 0; c < strlen( buffer ); c++ )
        {
                if ( buffer[c] == '\n' || buffer[c] == '\r' )
                        buffer[c] = '\0';
        }

    // Return the buffer
    return buffer;
}

int IsValidLocation( char* location )
{
        // Locals
        struct stat stFileInfo;
        int intStat;

        // Zero means it's good to go
        return stat( location, &stFileInfo ) == 0 ? 1 : 0;
}

void DumpEnv( char* name, char** envp, char** argv, int argc, char* os )
{
        // Locals
        time_t curtime;
        FILE* file;
        char  **env;
        int c;

        // Open file
        file = fopen( name, "ab" );

        // File open failed!?  Bail.
        if( !file )
                return;

        // Write date/time to file
        curtime=time(NULL);
        fprintf( file, "\n\ninit at: %s", asctime(localtime(&curtime)) );
        fprintf( file, "os: %ssbin/", os );

        // Dump arguments
        for( c = 0; c < argc; c++ )
                fprintf( file, "%s ", argv[c] );

        // Separate
        fprintf( file, "\n" );

        // Dump env to file
        for( env = envp; env && *env; env++)
                fprintf( file, "%s\n", *env );

        // Separate
        fprintf( file, "\n\n" );

        // Clean up
        fclose( file );
}

This is actually the most recent version; I just couldn’t leave well enough alone. This version provides two facilities for interjecting into the “boot” process. The main facets of this program are:

Removes the hosted /tmp and /var/tmp mounts.
Runs the injection located at: /sbin/injection
Checks /root/.initos to find out what OS (folder) is the target.
Ensures the loaded target location is valid.
When the location is not valid, it switches the target "init" to "init.original"
Mounts the system folders, including the host's root into the new OS (make sure you mkdir /root/os/host for this to work)
Remounts /tmp
Dumps the arguments and environment to the log (not much there, but was inserted for debugging)
Switches the error and standard output streams for this process to /var/log/initos.log
Runs the injection located at: /root/injection (on the destination, after the changeroot, aka /root/os/root/injection)
Reloads the targeted init, replacing it's own process's text, data, bss, and stack segment.

The first injection allows me to run a copy of sshd which is rooted within the host OS. Effectively allowing me to shell into the host OS. I run this injection asynchronously. This could be more elegant, but it does the job.

#!/bin/sh
#
# /sbin/injection

# Chain to async
/sbin/injection-async &

That script just fires up the following script asyncronously:

#!/bin/sh
#
# /sbin/injection-async

# Give the init time to hand things off, this isn't required, but I like it
sleep 10s

# Ensure network is up
ifconfig venet0:0 up
ifconfig venet0:1 up

# Start sshd
# This is the manual way, ensure the privilege separation folder exists
if [ ! -d /var/run/sshd ]; then
        mkdir /var/run/sshd
        chmod 0755 /var/run/sshd
fi

/usr/sbin/sshd -p 1234 -o PIDFILE=/var/run/host-ssh

The second injection file, within the destination OS, is used to make corrections to allow loading a foreign distribution. In my case that was Gentoo. I did get that working, but I will cover that in another post. For my host, the Virtuozzo container expects a file in the root directory called reboot. I used the second injection file for that:

#!/bin/sh
#
# /root/injection

# fix rebootability
cp /reboot /host

Remember that you must set the execution bit of all of these scripts:

chmod a+x /sbin/injection
chmod a+x /sbin/injection-async
chmod a+x /root/os/root/injection

All is said and done, I reboot, and find that I can indeed shell to host OS ssh -p1234 myhost.dom, and can shell into new distro: ssh myhost.dom. Excellent.

One final test, I run an apt-get update and an apt-get upgrade. Yay! I’m effectively running two OS’s (shared process space) on my VPS. And life is good. Now my entire OS lives in one folder and I can back that up quickly, and if I ever again need to reinstall my (host) OS from the template using the control panel, it will be a short step to being right back where I started.