Instructions for running NetBSD on EC2

To: port-xen%netbsd.org@localhost
Subject: Instructions for running NetBSD on EC2
From: Alistair Crooks <agc%pkgsrc.org@localhost>
Date: Wed, 26 Jan 2011 02:46:08 +0100
Hi,

I've been asked to distribute the notes and instructions I received
about running NetBSD/EC2 to the port-xen list.  These are from some
time ago (June 2010, I believe), but should be useful to people.  I
can't remember whether anonymity was requested in this (it's not
obvious from my notes of the phone call), I don't think so, but just
in case, I'll remove identifying traces.  If I've got it wrong, then
I'll re-post with the correct attributions.  Please advise.

Please feel free to contact me if anything is unclear.

Thanks,
Alistair

> Here's my NetBSD-AMI-building notes.  It's more or less in a "copy and paste
> into console" format.  As written it creates an i386 AMI; creating an amd64
> AMI needs a global s/i386/amd64/ plus some adjustment to the ec2-register
> command.  It also creates the AMI in Amazon's US-East region; creating an AMI
> in a different region is just a matter of adjusting the ec2-* command lines.
> 
> The basic approach here is
> 1. Build NetBSD locally with some minor changes (the XEN3PAE_DOMU 
> configuration
> is adjusted to find the root disk; root's password is starred out; root logins
> via ssh are enabled; dhcp+ssh are turned on; and an extra rc.d script fetches
> root's public ssh key the first time the instance boots).
> 2. Create a Linux EC2 instance, upload the NetBSD bits to it, and write them
> out to EC2 disks.
> 3. Snapshot those disks and register a new AMI using them.
> 
> Let me know if you have any problems.

#!/bin/sh

# PROVIDE: ec2_firstboot
# REQUIRE: NETWORKING
# BEFORE: LOGIN

. /etc/rc.subr

name="ec2_firstboot"
start_cmd="ec2_firstboot_run"
stop_cmd=":"

SSHKEYURL="http://169.254.169.254/1.0/meta-data/public-keys/0/openssh-key";
SSHKEYFILE="/root/.ssh/authorized_keys"

ec2_firstboot_run()
{
        if ! [ -f /etc/ec2_firstboot_done ]; then
                # Grab the provided SSH public key and authorize it to log in
                # as root.  (Note that /etc/ssh/sshd_config has been edited
                # to allow root to login directly.)
                mkdir -p `dirname ${SSHKEYFILE}`
                chmod 700 `dirname ${SSHKEYFILE}`
                ftp -o ${SSHKEYFILE}.ec2 -a ${SSHKEYURL}
                if [ -f ${SSHKEYFILE}.ec2 ]; then
                        touch ${SSHKEYFILE}
                        sort -u ${SSHKEYFILE} ${SSHKEYFILE}.ec2         \
                            > ${SSHKEYFILE}.tmp
                        mv ${SSHKEYFILE}.tmp ${SSHKEYFILE}
                        rm ${SSHKEYFILE}.ec2
                fi

                # Mark that we've done the first boot.
                echo "DO NOT REMOVE THIS FILE" > /etc/ec2_firstboot_done
        fi
}

load_rc_config $name
run_rc_command "$1"

### NetBSD AMI building instructions
### ================================

### Build EC2ized NetBSD bits
### -------------------------

# Fetch src.tgz, syssrc.tgz, sharesrc.tgz, and gnusrc.tgz into the cwd.
# TODO

# Extract source tree
tar -xf gnusrc.tgz
tar -xf sharesrc.tgz
tar -xf syssrc.tgz
tar -xf src.tgz

# Patch source tree for EC2
cat root-starpass.patch | (cd usr && patch -p0)
cp usr/src/sys/arch/i386/conf/XEN3PAE_DOMU usr/src/sys/arch/i386/conf/EC2
echo 'config netbsd root on xbd1 type ffs dumps on none' >> 
usr/src/sys/arch/i386/conf/EC2
echo 'options DDB_COMMANDONENTER=bt' >> usr/src/sys/arch/i386/conf/EC2
sed -EI '' -e 's/^#*config.*//' usr/src/sys/arch/i386/conf/XEN2_DOMU

# Build NetBSD
rm -rf DEST OBJ TOOLS
mkdir DEST OBJ TOOLS
export DIRS="-D ../../DEST -O ../../OBJ -T ../../TOOLS"
cd usr/src
sh build.sh $DIRS -m i386 -N 0 -U tools kernel=EC2 distribution
cd ../..

# Create tarball of grub boot filesystem contents
mkdir GRUB GRUB/boot GRUB/boot/grub
cp OBJ/sys/arch/i386/compile/EC2/netbsd GRUB/boot/kern.netbsd
cat > GRUB/boot/grub/menu.lst <<- EOF
        default=0
        timeout=0
        hiddenmenu

        title NetBSD AMI
            root (hd0)
            kernel /boot/kern.netbsd root=xbd1
EOF
tar -cf GRUB.tar -C GRUB .
rm -rf GRUB

# Add fstab file
cat >DEST/etc/fstab <<EOF
/dev/xbd1 / ffs rw 1 1
/dev/xbd0 /grub ext2 rw 2 2
kernfs /kern kernfs rw
ptyfs /dev/pts ptyfs rw
procfs /proc procfs rw
EOF

# Add EC2 first-boot configuration script
cp ec2_firstboot DEST/etc/rc.d

# Tell NetBSD to go ahead and boot properly; to use dhclient to set up its
# network; to enable sshd; and to use a hostname of NetBSD-EC2.
sed -I '' -e 's/rc_configured=NO/rc_configured=YES/' DEST/etc/rc.conf
cat >>DEST/etc/rc.conf <<EOF
dhclient="YES"
dhclient_flags="xennet0"
hostname="NetBSD-EC2"
sshd="YES"
EOF

# Tell sshd to allow root logins (standard for EC2 instances, and necessary
# since there are no other users yet).
sed -I '' -e 's/.*PermitRootLogin.*/PermitRootLogin yes/' 
DEST/etc/ssh/sshd_config

# Remove some lines from METALOG which have (now incorrect) size values.
cat DEST/METALOG |
        grep -vE '^./etc/rc.conf ' |
        grep -vE '^./etc/ssh/sshd_config ' > DEST/METALOG.tmp
mv DEST/METALOG.tmp DEST/METALOG

# Add some necessary bits to METALOG, including replacing the lines we
# removed above.
cat >> DEST/METALOG <<EOF
./etc/fstab type=file uname=root gname=wheel mode=644
./kern type=dir uname=root gname=wheel mode=0755
./proc type=dir uname=root gname=wheel mode=0755
./grub type=dir uname=root gname=wheel mode=0755
./etc/rc.d/ec2_firstboot type=file uname=root gname=wheel mode=0555
./etc/rc.conf type=file uname=root gname=wheel mode=0644
./etc/ssh/sshd_config type=file uname=root gname=wheel mode=0444
EOF

# Create the world disk image
./TOOLS/bin/nbmakefs -B le -F DEST/METALOG -s 10g -N DEST/etc/ -o density=32k 
-x NetBSD-i386.disk DEST
gzip -f NetBSD-i386.disk

### Upload NetBSD bits to EC2
### -------------------------

# Download and extract the latest EC2 API tools.
# Create an AWS x.509 private key & certificate (pk-XXX.pem and cert-XXX.pem).
# Create an SSH keypair via EC2.
# Set shell variables:
#     EC2_HOME=/path/to/ec2-api-tools
#     JAVA_HOME=/path/to/jdk
#     EC2_PRIVATE_KEY=/path/to/pk-XXX.pem
#     EC2_CERT=/path/to/cert-XXX.pem
#     EC2_SSH_KEY=/path/to/ssh_private_key
#     EC2_SSH_KEYNAME=ec2_ssh_keyname

# We will use a linux AMI as a staging point
AMI_LINUX=ami-3ac33653

# Create an Amazon Linux instance
${EC2_HOME}/bin/ec2-run-instances ${AMI_LINUX} -z us-east-1b -t t1.micro \
    -k ${EC2_SSH_KEYNAME} | tee tmp.run-instances
export I_LINUX=`cat tmp.run-instances | grep ^INSTANCE | cut -f 2`
while ${EC2_HOME}/bin/ec2-describe-instances ${I_LINUX} | grep -q pending; do
        sleep 1;
done

# Get IP address of Linux instance
${EC2_HOME}/bin/ec2-describe-instances ${I_LINUX} | tee tmp.describe-instance
export IP_LINUX=`cat tmp.describe-instance | grep ^INSTANCE | cut -f 17`

# Create and attach a 1G EBS volume for the grub boot filesystem
${EC2_HOME}/bin/ec2-create-volume -s 1 -z us-east-1b | tee tmp.create-volume
export VOL_KERN=`cat tmp.create-volume | grep ^VOLUME | cut -f 2`
${EC2_HOME}/bin/ec2-attach-volume ${VOL_KERN} -i ${I_LINUX} -d "/dev/sdf"

# Copy grub boot filesystem contents across
scp -i ${EC2_SSH_KEY} GRUB.tar ec2-user@${IP_LINUX}:

# Create ext2 filesystem for grub boot
ssh -t -i ${EC2_SSH_KEY} ec2-user@${IP_LINUX} sudo mke2fs /dev/xvdf

# Mount filesystem
ssh -t -i ${EC2_SSH_KEY} ec2-user@${IP_LINUX} sudo mount /dev/xvdf /mnt

# Populate filesystem
ssh -t -i ${EC2_SSH_KEY} ec2-user@${IP_LINUX} sudo tar -xvf GRUB.tar -C /mnt

# Unmount filesystem
ssh -t -i ${EC2_SSH_KEY} ec2-user@${IP_LINUX} sudo umount /mnt

# Detach the volume and create a snapshot
${EC2_HOME}/bin/ec2-detach-volume ${VOL_KERN}
${EC2_HOME}/bin/ec2-create-snapshot ${VOL_KERN} | tee tmp.create-snapshot
export SNAP_KERN=`cat tmp.create-snapshot | grep ^SNAPSHOT | cut -f 2`
while ${EC2_HOME}/bin/ec2-describe-snapshots ${SNAP_KERN} | grep -q pending; do
        sleep 1;
done
${EC2_HOME}/bin/ec2-delete-volume ${VOL_KERN}

# Create and attach a 10G EBS volume for the world filesystem
${EC2_HOME}/bin/ec2-create-volume -s 10 -z us-east-1b | tee tmp.create-volume
export VOL_WORLD=`cat tmp.create-volume | grep ^VOLUME | cut -f 2`
${EC2_HOME}/bin/ec2-attach-volume ${VOL_WORLD} -i ${I_LINUX} -d "/dev/sdg"

# Copy world filesystem across
scp -i ${EC2_SSH_KEY} NetBSD-i386.disk.gz ec2-user@${IP_LINUX}:

# Write filesystem to EBS volume
ssh -t -i ${EC2_SSH_KEY} ec2-user@${IP_LINUX} \
    'gunzip < NetBSD-i386.disk.gz | sudo dd of=/dev/xvdg bs=128k'

# Detach the volume and create a snapshot
${EC2_HOME}/bin/ec2-detach-volume ${VOL_WORLD}
${EC2_HOME}/bin/ec2-create-snapshot ${VOL_WORLD} | tee tmp.create-snapshot
export SNAP_WORLD=`cat tmp.create-snapshot | grep ^SNAPSHOT | cut -f 2`
while ${EC2_HOME}/bin/ec2-describe-snapshots ${SNAP_WORLD} | grep -q pending; do
        sleep 1;
done
${EC2_HOME}/bin/ec2-delete-volume ${VOL_WORLD}

# Kill Linux instance
${EC2_HOME}/bin/ec2-terminate-instances ${I_LINUX}

# Register an AMI
${EC2_HOME}/bin/ec2-register -a i386 --kernel aki-407d9529 \
    -b "/dev/sda1=${SNAP_KERN}" -b "/dev/sda2=${SNAP_WORLD}" \
    -n "NetBSD `date | tr : -`" | tee tmp.register
export AMI_NETBSD=`cat tmp.register | grep ^IMAGE | cut -f 2`

### Test the AMI we just created
### ----------------------------

# Launch a NetBSD instance
${EC2_HOME}/bin/ec2-run-instances ${AMI_NETBSD} -z us-east-1b -t t1.micro \
    -k ${EC2_SSH_KEYNAME} | tee tmp.run-instances
export I_NETBSD=`cat tmp.run-instances | grep ^INSTANCE | cut -f 2`
while ${EC2_HOME}/bin/ec2-describe-instances ${I_NETBSD} | grep -q pending; do
        sleep 1;
done

# Read console output (EC2 only polls the instances for output occasionally,
# so there will be some delay before the output is available)
sleep 240 && ${EC2_HOME}/bin/ec2-get-console-output ${I_NETBSD}

# Kill NetBSD instance
${EC2_HOME}/bin/ec2-terminate-instances ${I_NETBSD}


----- End forwarded message -----
Follow-Ups:
- Re: Instructions for running NetBSD on EC2
  - From: Jean-Yves Migeon
- Re: Instructions for running NetBSD on EC2
  - From: Andreas Gustafsson
Prev by Date: Re: No Intel Speedstep in 5.1_STABLE ?
Next by Date: Re: Instructions for running NetBSD on EC2
Previous by Thread: No Intel Speedstep in 5.1_STABLE ?
Next by Thread: Re: Instructions for running NetBSD on EC2
Indexes:
Home | Main Index | Thread Index | Old Index