Running OpenWRT on an External Drive

Ever since the GPL violation which made open source home routers a reality, I’ve been playing with both OpenWRT and DD-WRT. All and all I prefer OpenWRT, although they both have their strengths. What I don’t like is that most devices have very limited flash space, and to get all the neat Linux features available, I’d need far more than the space allotted. One of my wireless router’s has a USB port, and is supported by OpenWRT. Excellent.

Today’s challenge was getting my router to run off of the USB stick so that I can install all the goodies I want without worrying about running out of space. I don’t think that booting from an external device is an easy option, so I went down the path of finding a way to chroot my way over. Initially I just tried to modify /etc/init.d and /etc/rc.d, but my changes just wouldn’t work out. Here’s what I figured out:

The router and OS supports a flash-based file system called JFFS.
Changes to the loaded file structure actually exist inside /jffs.
Upon booting, the bootstrap mounts the ROM filesystem, without the JFFS overlay.
The JFFS overlay does not start until sometime during init.d scripts.

Because JFFS is loaded during rc.d, trying to insert a file wasn’t being picked up. There is a segment of code in /etc/init.d/rcS which read:

run_scripts() {
        for i in /etc/rc.d/$1*; do
                [ -x $i ] && $i $2 2>&1
        done | $LOGGER
}

This will generate a list of in memory of all the rc.d files to load. When the JFFS overlay loads and modifies the directory structure, the for loop that started the process will not account for it. My changes aren’t making a difference.

Okay, now I understand the problem, how do we fix it? An elite coder named Jeremy Collake has created a project that helps us out. You can find this project at http://code.google.com/p/firmware-mod-kit/, and directions here: http://www.bitsum.com/firmware_mod_kit.htm. There are two basic commands to accomplish our task (which is to change the ROM file system): extract_firmware.sh and the counterpart build_firmware.sh. I’m going to retrace my steps in hopes that this will help someone else. I should note that a lot of this information has been pieced together from random pages, including several poages from the OpenWRT sites.

Start by extracting the payload you have downloaded from here. Then make your way into the root file system.

./extract_firmware.sh OpenWRT.BIN /tmp/OpenWRT-work
cd /tmp/OpenWRT-work/rootfs
ls -al

I spent a fair amount of time playing with my options here, before I decided to take the flexible alternative. I had originally started working on my init scripts, building the image, flashing the router, sometimes having some success and sometimes having none. That’s the long and hard way. Instead, I finally just inserted a stub file into /etc/rc.d. This made the rcS script account for the file, but I could change it’s functionality because it will get processed after JFFS is up. So, with your favorite text editor, create a file called bootext inside the /etc/init.d folder of the payload’s filesystem (/tmp/OpenWRT-work/rootfs/etc/init.d in my example):

#!/bin/sh

# Create a temporary file to prove that this ran
touch /tmp/bootext.good

Now make it executable, and create two links to that file from /etc/rc.d:

chmod a+x /tmp/OpenWRT-work/rootfs/etc/init.d/bootext
cd /tmp/OpenWRT-work/rootfs/etc/rc.d
ln -s ../init.d/bootext S11bootext
ln -s ../init.d/bootext K91bootext

The first file will load just after S10boot during startup. The second file will be triggered during shutdown.

Even though I find it easier to have the script loaded from JFFS, you could include the needed modules to perform the mount. There are a few ways to go about this, the easiest is using the menu driven compilation of OpenWRT via make menuconfig. Then you can navigate the menu to find the needed kernel modules. Specifically I used usbcore, usb-ohci, ehci-hcd, scsi_mod, sd_mod, usb-storage, and ext2. You may want vfat, although I formatted my USB stick with ext2. I’m going to precede with the assumption that you will load the kernel modules into JFFS, and not into the original image. If you want to make a ready-to-go image before flashing your router, read the rest of this post and apply the changes to the rootfs prior to repacking and flashing.

We need to repack the image, so we turn back to firmware-mod-kit. Repacking is simple:

./build_firmware.sh /tmp/OpenWRT-build /tmp/OpenWRT-work
cd /tmp/OpenWRT-build

The next part is a bit tricky. I’m hoping you’ve done this before, otherwise you may want to read over the documentation provided at http://openwrt.org. Here’s what I do:

1. Unplug the router
2. Issue this command: service network-interface restart INTERFACE=eth0

   (Your interface may not be eth0, but my version of Linux likes to down that interface when the link goes down.)

3. Prepare the TFTP session. Don’t press enter on the last line yet!

   someone@somewhere:/tmp/OpenWRT-build# tftp 192.168.1.1
   tftp> bin
   tftp> PUT custom_image-generic.bin

4. Plug in the router
5. Wait 1 second, and press enter on the TFTP PUT command

The Link-Light LED should begin to flash fairly quickly… 7-10 seconds later your firmware should be loaded and unpacking itself. Leave the router alone for a couple minutes. I wait three minutes to be safe. Power-cycle the router again. Wait for it to come back and you should be able to either telnet (if no root password has been set) or shell into the router. With a little luck you’ll be presented with:

login as: root
root@router's password:
  _______                     ________        __
 |       |.-----.-----.-----.|  |  |  |.----.|  |_
 |   -   ||  _  |  -__|     ||  |  |  ||   _||   _|
 |_______||   __|_____|__|__||________||__|  |____|
          |__| W I R E L E S S   F R E E D O M
 KAMIKAZE (8.09.2, r18801) -------------------------
  * 10 oz Vodka       Shake well with ice and strain
  * 10 oz Triple sec  mixture into 10 shot glasses.
  * 10 oz lime juice  Salute!
 ---------------------------------------------------
root@router:~$

Good news, it’s not a brick. Now we check to see if your modification worked out:

root@router:~$ ls /tmp/bootext.good
/tmp/bootext.good

The second line tells us it worked. You can use the web interface or shell to configure the router to what’s best for your environment. Don’t worry, we maintain one configuration when hooking up the external root.

Now that the router is configured, we can add the modules we need to be able to mount the external device. We use opkg, which is a lot like apt-get. In my case, available packages came from here. On the router’s shell, issue these commands as root:

opkg update
opkg install kmod-usb-core
opkg install kmod-usb-ohci
opkg install kmod-usb-uhci
opkg install kmod-scsi-core
opkg install kmod-usb-storage
opkg install kmod-usb2
# Optionally vfat, for most USB stick filesystems
opkg install kmod-fs-vfat
# Optionally ext2, if you're going my way
opkg install kmod-fs-ext2
# Optionally ext3, If you want journalism on your ext2 filesystem
opkg install kmod-fs-ext3

The modules will be installed to /lib/modules/[kernel version]/. Now we can make sure the usb stick gets loaded. First we have to get our modules loaded. Then we insert the USB stick.

insmod usbcore
insmod usb-ohci
insmod ehci-hcd
insmod scsi_mod
insmod sd_mod
insmod usb-storage
insmod ext2
insmod ext3 (if you got it)
insmod vfat (if you got it)
[insert USB stick]
cd /dev/scsi
ls -al
[look around, hopefully you'll find: /dev/scsi/host0/bus0/target0/lun0/part1, although the location be different on another system]

Sweet, let’s mount that and copy our file system to it.

mount -t ext2 /dev/scsi/host0/bus0/target0/lun0/part1 /mnt
cp -a bin home lib opt root sbin tmp usr www /mnt
mkdir /mnt/backup
mkdir /mnt/dev
mkdir /mnt/etc
mkdir /mnt/jffs
mkdir /mnt/mnt
mkdir /mnt/old
mkdir /mnt/proc
mkdir /mnt/rom
mkdir /mnt/sys
mkdir /mnt/tmp
ln -s /tmp /mnt/var

This is done this way because many of the folders are provided by the operating system. We are going to also not copy /etc, because we do not want to a pre-switch and post-switch version of our configuration to maintain. This will instead be mounted over into the new root of the external device. We also want our device to boot and be accessible if the external device is removed. Now to setup the configuration.

Create the configuration file /etc/config/bootfromexternalmedia. If you do not have an editor installed, issue a opkg install nano. You don’t want to install too much, we’re still storing everything in the router’s flash.

config bootfromexternalmedia
        option enabled  '1'
        option device   '/dev/scsi/host0/bus0/target0/lun0/part1'
        option name     'usb'
        option target   '/mnt'
        option putold   '/old'
        option modules  'usbcore usb-ohci ehci-hcd scsi_mod sd_mod usb-storage ext2'
        option gpiomask '0x9c'
        option waitdev  '10'
        option filesys  'ext2'
        option mountopt 'noatime'

This controls the init script. If you are using other file systems, you need to modify the modules line. If your external drive takes a long time to initialize, you will want to increase the waitdev parameter.

Then modify the insert point we created early /etc/init.d/bootext:

#!/bin/sh /etc/rc.common

START=11
STOP=91

bootext_cleanup() { # [cleanup_level]
 [ "$1" -ge 3 ] && grep -q '^[^ ]* /rom ' /proc/mounts && mount -o move /rom $putold/rom
 [ "$1" -ge 2 ] && { . $putold/bin/firstboot ; pivot $putold $target ; }
 [ "$1" -ge 1 ] && umount -l $target/etc
 return 0
}

bootext_fail() { # <error_message> [cleanup_level]
 echo "$1" >&2
 [ ! "$2" ] || bootext_cleanup $2
 exit 1
}

bootext_quit() { # <message>
 echo "$1" >&2
 exit 0
}

bootext_start() {
 ! grep -q "^$device / " /proc/mounts || bootext_quit "$name already on /"

 if ! grep -q "^$device $target " /proc/mounts
 then
  ! grep -q "^$device " /proc/mounts || bootext_fail "$name already mounted"

  for module in $modules
  do
   if ! grep -q "^$module " /proc/modules
   then
    [ $module != mmc ] || [ ! "$gpiomask" ] || echo "$gpiomask" > /proc/diag/gpiomask || bootext_fail "could not set gpiomask"
    insmod $module || bootext_fail "could not insert $module module"
   fi
  done

  while [ ! -b $device ]
  do
   [ "$waitdev" -gt 0 ] || bootext_fail "$device does not exist"
   waitdev=$(( $waitdev - 1 ))
   sleep 1
  done

  [ -d $target ] || mkdir $target || bootext_fail "could not create mountpoint $target"
  echo mount ${filesys:+-t $filesys} ${mountopt:+-o $mountopt} $device $target || bootext_fail "could not mount $name on $target"
  mount ${filesys:+-t $filesys} ${mountopt:+-o $mountopt} $device $target || bootext_fail "could not mount $name on $target"
 fi

 [ -d $target$putold ] || mkdir $target$putold || bootext_fail "could not create mountpoint $putold"
 [ -d $target/etc ] || mkdir $target/etc || bootext_fail "could not create mountpoint /etc"
 mount -o bind /etc $target/etc || bootext_fail "could not bind mount /etc"

 . /bin/firstboot
 pivot $target $putold || bootext_fail "could not pivot to $target" 1

 ! grep -q "^[^ ]* $putold/rom " /proc/mounts || { [ -d /rom ] || mkdir /rom && mount -o move $putold/rom /rom ; }

 cp -a /etc /backup

 return 0
}

bootext_stop() {
 grep -q "^$device / " /proc/mounts || bootext_quit "$name not on /"

 bootext_cleanup 999
}

bootext_config() { # <section> <action>
 local section=$1
 local action=$2
 local enabled device name target putold modules gpiomask waitdev filesys mountopt

 config_get_bool enabled $section enabled 1
 [ "$enabled" -gt 0 ] || return 0
 config_get device $section device
 [ "$device" ] || bootext_fail "external boot device not configured"
 config_get name $section name
 config_get target $section target
 config_get putold $section putold
 config_get modules $section modules
 config_get gpiomask $section gpiomask
 config_get waitdev $section waitdev
 config_get filesys $section filesys
 config_get mountopt $section mountopt

 [ "$name" ] || name="$device"
 [ "$putold" ] || putold="${target:-/old}"
 [ "$target" ] || target="/${filesys:-new}"

 bootext_$action
}

start() {
 config_load bootfromexternalmedia
 config_foreach bootext_config bootfromexternalmedia start
}

stop() {
 config_load bootfromexternalmedia
 config_foreach bootext_config bootfromexternalmedia stop
}

That’s it. This script originally came from the OpenWRT site, I have made some modifications. Most notably, (and perhaps all that I remember at the moment) is a cp /etc /backup which keeps a backup copy of our configuration on the external device for disaster recovery purposes. Now we’re ready to test it:

root@router:/# /etc/init.d/bootext start
root@router:/# mount
rootfs on / type rootfs (rw)
/dev/root on /rom type squashfs (ro)
none on /dev type devfs (rw)
proc on /proc type proc (rw)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev)
devpts on /dev/pts type devpts (rw)
/dev/mtdblock/4 on /jffs type jffs2 (rw)
mini_fo:/jffs on /old type mini_fo (rw)
/dev/scsi/host0/bus0/target0/lun0/part1 on / type ext2 (rw,noatime)
none on /proc/bus/usb type usbfs (rw)
mini_fo:/jffs on /etc type mini_fo (rw)

From this I can see that the device is mounted as root. One last thing. When I fdisk’d my usb stick I put an addition partition for swap space. Configuring that is much easier. Simply edit your /etc/config/fstab file and put in (or edit) the lines:

config swap
        option device   /dev/scsi/host0/bus0/target0/lun0/part5
        option enabled  1

Mine was part5 (first partition of extended partition 2). Your setup may differ. You can test this by issueing the following:

root@router:/etc/config# /etc/init.d/fstab restart
root@router:/etc/config# swapon -s
Filename                        Type            Size    Used    Priority
/dev/scsi/host0/bus0/target0/lun0/part5 partition       147840  0       -3

I see that my swap space has been loaded. Now, the final test. Reboot your router, login, and ensure that the mount’s appear correct, and that the swap space is enabled. Congrats! You have moved your root file system to the external device. Now feel free to install all kinds of goodies. I recommend iftop. This will bring up a screen which shows you your traffic throughput by device. In some cases you must specify the interface to see something useful; I listen to the bridge:

opkg update
opkg install iftop
iftop -i br-lan