Port-xen archive
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index][Old Index]
Amazon EC2 build script (was: Re: EC2 images)
Dear all,
I wrote a set of shell scripts to help build and upload NetBSD images
(see attached). I am presently using them to update the 5.1 images.
I am well aware that they are ugly :) Feel free to send me patches/ideas
to improve them, and/or commit them to localsrc.
The set_ec2_env.sh.example script is a skeleton to set up environment
for EC2 use. Edit it to fit your EC2 account.
What you can do:
- build_ec2_img.sh: compile and generate a NetBSD image, suitable for
upload to EC2. Feel free to adapt it to your needs, I wrote a function
for each step (building src, patching installed files, makefs, etc).
Please keep the various ec2 scripts, and only modify them when you know
what you are doing. EC2 AMIs require some automation to fetch ssh keys,
for example.
Usage:
# ./build_ec2_img.sh <path-to-NetBSD-src> <tmp>
- create_ec2_ami.sh:
- list NetBSD AMIs by region (when called without arguments).
- Upload the image and kernel built previously, and create the new AMI.
Usage:
$ ./create_ec2_ami.sh # Outputs NetBSD's AMI information
$ ./create_ec2_ami.sh <tmp> # Upload and create a NetBSD AMI
=== Shortcomings and possible improvements ===
- build_ec2_img.sh must be executed as root. Using mtree specs with
makefs can avoid that, but as I am using ./build.sh install=$EC2DIR with
specific sets, I did not find a way around this.
- scripts are ugly, and make extensive use of awk.
- i386 is not supported, but should be trivial to implement.
- there may be some unhandled corner cases with AMI creation, especially
when communicating via EC2 API. Amazon did not design their tools in a
shell friendly way. Errors can be reported either by command's return
value, or out-of-band (in the output).
- the upload/creation of the AMI can take some time. It checks the SSH
host key from the console output, and since it's buffered on Amazon's
side upload will wait for them before starting AMI creation.
--
Jean-Yves Migeon
jeanyves.migeon%free.fr@localhost
#!/bin/sh
# This script is used to generate a correct NetBSD AMI image, suitable
# for uploading.
# The script must be run as root (eventually, we will not require
# this when fileutils are ready)
# XXX does not support i386 and PAE. Fix build + summary paths.
# User used to build the NetBSD's src tree
BUILD_USER="nobody"
BUILD_FLAGS="-j14 -U"
usage() {
echo "Usage: ${0##*/} src [tmp]"
echo " src: path to NetBSD's src directory"
echo " tmp: temporary directory where EC2 image and kernel are built."
exit 1
}
build_amd64() {
cd $SRCDIR
su "$BUILD_USER" \
./build.sh -O ../obj -T ../tools \
-D ../dest -R ../release \
$BUILD_FLAGS release
}
install_amd64() {
cd $SRCDIR
./build.sh -O ../obj -T ../tools \
-D ../dest -R ../release \
-V INSTALLSETS="base comp etc man tests" \
$BUILD_FLAGS install=${EC2DIR}
cp "../obj/sys/arch/amd64/compile/XEN3_DOMU/netbsd" $KERNEL
}
patch_install() {
cd $EC2DIR
TMPFILE=$(mktemp /tmp/netbsd-ec2.XXXXXX)
trap "rm -f $TMPFILE; exit 1" INT EXIT QUIT
sed 's/rc_configured=NO/rc_configured=YES/g' \
etc/rc.conf > $TMPFILE
cp "$TMPFILE" etc/rc.conf
# required to allow connection through EC2 SSH keys
cat >> etc/rc.conf << 'EOF'
ec2_init=YES # for setup of SSH key pair(s)
sshd=YES # for remote shell access to instance
dhcpcd_flags="-t 0" # Wait for the DHCP server forever
EOF
sed 's/^#PermitRootLogin.*/PermitRootLogin without-password/g' \
etc/ssh/sshd_config > $TMPFILE
cp "$TMPFILE" etc/ssh/sshd_config
# DO NOT MODIFY THE FOLLOWING FILE WITHOUT PRIOR CONSENT.
# Its output is parsed by custom scripts to handle SSH host
# key fingerprints.
cat > etc/rc.d/ec2_init << 'EOF'
#!/bin/sh
#
# PROVIDE: ec2_init
# REQUIRE: NETWORKING sshd
# BEFORE: LOGIN
$_rc_subr_loaded . /etc/rc.subr
name="ec2_init"
rcvar=${name}
start_cmd="ec2_init"
stop_cmd=":"
METADATA_URL="http://169.254.169.254/latest/meta-data/";
SSH_KEY_URL="public-keys/0/openssh-key"
HOSTNAME_URL="hostname"
SSH_KEY_FILE="/root/.ssh/authorized_keys"
ec2_init()
{
(
umask 022
# fetch the key pair from Amazon Web Services
EC2_SSH_KEY=$(ftp -o - "${METADATA_URL}${SSH_KEY_URL}")
if [ -n "$EC2_SSH_KEY" ]; then
# A key pair is associated with this instance, add it
# to root 'authorized_keys' file
mkdir -p $(dirname "$SSH_KEY_FILE")
touch "$SSH_KEY_FILE"
cd $(dirname "$SSH_KEY_FILE")
grep -q "$EC2_SSH_KEY" "$SSH_KEY_FILE"
if [ $? -ne 0 ]; then
echo "ec2: Setting EC2 SSH key pair: ${EC2_SSH_KEY##* }"
echo "$EC2_SSH_KEY" >> "$SSH_KEY_FILE"
fi
fi
# set hostname
HOSTNAME=$(ftp -o - "${METADATA_URL}${HOSTNAME_URL}")
echo "ec2: Setting EC2 hostname: ${HOSTNAME}"
echo "$HOSTNAME" > /etc/myname
hostname "$HOSTNAME"
# Output the SSH host keys
echo "ec2: ###########################################################"
echo "ec2: -----BEGIN SSH HOST KEY FINGERPRINTS-----"
for FILE in /etc/ssh/ssh_host_*_key.pub; do
echo -n "ec2: "
ssh-keygen -l -f "$FILE"
done
echo "ec2: -----END SSH HOST KEY FINGERPRINTS-----"
echo "ec2: ###########################################################"
)
}
load_rc_config $name
run_rc_command "$1"
EOF
chmod 555 etc/rc.d/ec2_init
# Some required files and directories
# Add proc and kern directories
mkdir grub kern proc
# EC2 network configuration, via DHCP
echo "dhcp" > etc/ifconfig.xennet0
# Basic fstab entries
cat > etc/fstab << 'EOF'
/dev/xbd1a / ffs rw 1 1
/dev/xbd0a /grub ext2fs rw 2 2
kernfs /kern kernfs rw
ptyfs /dev/pts ptyfs rw
procfs /proc procfs rw
EOF
cat > etc/motd << 'EOF'
Welcome to NetBSD - Amazon EC2 image!
This system is currently running a snapshot of a stable branch of the NetBSD
operating system, adapted for running on the Amazon EC2 infrastructure.
The environment is very similar to one provided within a typical Xen domU
installation. It contains a small, autonomous environment (including a
compiler toolchain) that you can run to build-up your own system.
The file-system is lightly populated so you have plenty of space to play with.
Should you need a src or pkgsrc tree, please use the "bootstrap" script found
under /usr to download them:
/usr/bootstrap.sh [src|pkgsrc]
You are encouraged to test this image as thoroughly as possible. Should you
encounter any problem, please report it back to the development team using the
send-pr(1) utility (requires a working MTA). If yours is not properly set up,
use the web interface at: http://www.NetBSD.org/support/send-pr.html
Thank you for helping us test and improve NetBSD's quality!
EOF
cat > usr/bootstrap.sh << 'EOF'
#! /bin/sh
URLROOT="http://ftp.netbsd.org/pub/";
usage() {
echo "Usage: ${0##*/} [src|pkgsrc]"
exit
}
if [ $# -ne 1 ]; then
usage
fi
cd /tmp
case $1 in
src)
echo "Downloading -current src..."
ftp -a "$URLROOT/NetBSD/NetBSD-current/tar_files/src.tar.gz"
progress -f src.tar.gz tar -xzpf - -C /usr/
rm -f src.tar.gz
;;
pkgsrc)
echo "Downloading latest pkgsrc stable release..."
ftp -a "$URLROOT/pkgsrc/stable/pkgsrc.tar.gz"
progress -f pkgsrc.tar.gz tar -xzpf - -C /usr/
rm -f pkgsrc.tar.gz
;;
*)
usage
;;
esac
EOF
chmod 755 usr/bootstrap.sh
}
generate_img() {
makefs -t ffs -B le -s 5g -M 5g -N $EC2DIR/etc/ \
-o version=2 -f600000 $IMGFILE $EC2DIR
gzip -9n $IMGFILE
}
summary() {
echo "Creation of NetBSD Amazon EC2 image successful."
echo "Kernel DOMU: $KERNEL"
echo "Root image (compressed): $IMGFILE.gz"
}
if [ $(id -u) -ne 0 ]; then
echo "This script must be run as root."
exit 2
fi
case $# in
1)
SRCDIR="$1"
BUILDDIR="/tmp"
;;
2)
SRCDIR="$1"
BUILDDIR="$2"
;;
*)
usage
;;
esac
set -e
ulimit -p 1024
ulimit -n 1024
EC2DIR="$BUILDDIR/ec2"
KERNEL="$BUILDDIR/netbsd"
IMGFILE="$BUILDDIR/NetBSD-AMI.img"
build_amd64
install_amd64
patch_install
generate_img
summary
#!/bin/sh
export EC2_PRIVATE_KEY=$HOME/.ec2/pk-XXXXX.pem
export EC2_CERT=$HOME/.ec2/cert-XXXX.pem
export EC2_SSH_KEY=$HOME/.ec2/id_rsa.ec2
export EC2_SSH_PUBKEY=$EC2_SSH_KEY.pub
export EC2_ACCOUNT_NUM=XXXX-XXXX-XXXX
export EC2_ACCESS_KEY=XXXXXXXXXXXXXXXXXXXX
export EC2_SECRET_KEY=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
export EC2_SSH_KEYNAME=xxxxxxxxx
#!/bin/sh
# This script is used to:
# - query for current state of NetBSD AMIs
# - update an image disk and create an AMI remotely
# In update mode, it performs the following steps:
# - create volumes for Grub (1GiB, ext2), and NetBSD's root (5GiB, ffs)
# - upload the kernel and populate the Grub boot volume
# - upload the image and create NetBSD's root volume
# - snapshots the grub and / disks
# - output identifiers and some info, then exit.
export LC_ALL=C
ARCH="x86_64"
VERSION="5.1"
DESCRIPTION="built: $(date)"
# path to file that sets EC2 credentials (SSH keys, user's x509 certificate)
EC2CREDS="$HOME/set_ec2_env.sh"
# load the Amazon EC2 credentials
. "$EC2CREDS"
usage() {
echo "Usage: ${0##*/} [region] [build-dir]"
echo "Without parameter, returns per-region information about " \
"NetBSD's AMIs"
echo " region: the EC2 region we want to create an AMI for"
echo " build-dir: path to the directory where EC2 files were built"
exit 1
}
if [ $# -eq 0 ]; then
# Assume we just want information about NetBSD AMIs
# The name should contain NetBSD, at least...
echo "REGION ID NAME ARCH ACCESS"
for REGION in $(ec2-describe-regions | awk '{print $2}'); do
ec2-describe-images --region $REGION -F name=*NetBSD* | (
while read TYPE ID NAME OWNER AVAIL PUBLIC ARCH VOID; do
# Skip all except IMAGEs
if [ "$TYPE" != "IMAGE" ]; then continue; fi
echo "$REGION $ID $NAME $ARCH $OWNER
$PUBLIC"
done
)
done
exit
fi
if [ $# -eq 2 ]; then
REGION=$1
BUILDDIR=$2
else
usage
fi
KERNEL="$BUILDDIR/netbsd"
IMGFILE="$BUILDDIR/NetBSD-AMI.img.gz"
# We are in creation mode. Very simple preliminary checks
if [ ! -r $KERNEL ]; then
echo "Kernel file '$KERNEL' does not exist!"
exit 1
fi
if [ ! -r $IMGFILE ]; then
echo "Image files '$IMGFILE' does not exist!"
exit 1
fi
# Start a micro instance for the region we want.
# Any Amazon Linux instance should do.
AMIID=$(ec2-describe-images -o amazon --region $REGION \
-F name=amzn-ami-* -F architecture=$ARCH |\
awk 'NR == 1 {print $2}')
if [ "$AMIID" = "" ]; then
# No Amazon Linux AMI for this region. Bail out.
echo "There is no Amazon Linux AMI available for '$REGION'."
echo "Consult http://wiki.netbsd.org/amazon_ec2/build_your_own_ami/";
echo "to see how you can build one from scratch."
exit 1
fi
# Run an instance
INSTANCE=$(ec2-run-instances $AMIID -k $EC2_SSH_KEYNAME \
-t t1.micro --region $REGION)
INSTANCEID=$(echo $INSTANCE | awk '{print $6}')
ZONEID=$(echo $INSTANCE | awk '{print $13}')
echo "Creating: $ZONEID $AMIID $INSTANCEID"
# Create the 1GiB, ext2fs volume. Will contain kernel and grub.conf files
VOL1ID=$(ec2-create-volume -s 1 --region $REGION -z $ZONEID | awk '{print $2}')
echo "Creating: volume $VOL1ID (1GiB), Grub configuration and kernel"
# Create the 5GiB, ffs volume. Will contain the root file-system.
VOL2ID=$(ec2-create-volume -s 5 --region $REGION -z $ZONEID | awk '{print $2}')
echo "Creating: volume $VOL2ID (5GiB), the root file-system"
# Wait for the instance.
COUNT=0
while true; do
ADDRESS=$(ec2-describe-instances $INSTANCEID --region $REGION |\
awk '/running/ {print $4}')
if [ "$ADDRESS" != "" ]; then
echo "$INSTANCEID running, address: $ADDRESS"
break
fi
COUNT=$(($COUNT + 1))
if [ $COUNT -gt 5 ]; then
echo "Instance was not reported as 'started' after $COUNT " \
"tries. Leaving."
exit 1
fi
echo "Waiting for instance $INSTANCEID..."
sleep 30
done
# Now grab the correct SSH host key. This can take a while, the console
# output is buffered on EC2's side.
# By convention, the EC2 initialization script should prefix all its output
# lines by "ec2: "
TMPFILE1=$(mktemp /tmp/ec2-ssh-host-keys.XXXXXX)
TMPFILE2=$(mktemp /tmp/ec2-ssh-host-keys2.XXXXXX)
LOGFILE="/tmp/system.log.$INSTANCEID"
trap "rm -f $TMPFILE1 $TMPFILE2; exit 1" INT EXIT QUIT
COUNT=0
while true; do
ec2-get-console-output $INSTANCEID --region $REGION |\
sed -n "s/^ec2: //gp" > $TMPFILE1
if grep -q "^-----END SSH HOST KEY FINGERPRINTS-----" \
$TMPFILE1; then
echo "EC2 instance reported:"
cat $TMPFILE1
break
fi
COUNT=$(($COUNT + 1))
if [ $COUNT -gt 5 ]; then
ec2-get-console-output $INSTANCEID --region $REGION > $LOGFILE
echo "Failed contacting instance ($COUNT tries). The system "
echo "log is available under:"
echo " $LOGFILE"
echo "Connect to AWS Console to terminate and clean instance."
exit 1
fi
echo "Waiting for console output of EC2 init script..."
sleep 90
done
# Compare the fingerprints between EC2 console output and direct connection.
FP1=$(awk '$NF ~ /(RSA)/ {print $2}' $TMPFILE1)
ssh-keyscan -T60 -t rsa $ADDRESS 2>/dev/null > $TMPFILE2
FP2=$(ssh-keygen -l -f $TMPFILE2 | awk '{print $2}')
if [ "$FP1" != "$FP2" ]; then
echo "SSH server fingerprint mismatch:"
echo "EC2 Console reported: '$FP1'"
echo "Host keyscan reported: '$FP2'"
echo "You are most likely trying to connect to an instance you do"
echo "not own, or are subject to MITM. Continue? [yn]"
read YESNO
if [ "$YESNO" != "y" ]; then
echo "Leaving..."
exit 1
fi
fi
ssh-keygen -R $ADDRESS
cat $TMPFILE2 >> ~/.ssh/known_hosts
# Attach 1GiB volume
echo "Attaching $VOL1ID to $INSTANCEID"
ec2-attach-volume $VOL1ID --region $REGION -i $INSTANCEID -d "/dev/sdf" \
> /dev/null
# Attach 5GiB volume
echo "Attaching $VOL2ID to $INSTANCEID"
ec2-attach-volume $VOL2ID --region $REGION -i $INSTANCEID -d "/dev/sdg" \
> /dev/null
# Wait for the 5GiB and 1GiB volume to be attached
COUNT=0
while true; do
PRESENT=$(ssh -i $EC2_SSH_KEY ec2-user@$ADDRESS \
"/bin/dmesg | /bin/grep -c 'xvd[fg]: unknown'")
if [ "$PRESENT" -eq 2 ]; then
break
fi
COUNT=$(($COUNT + 1))
if [ $COUNT -gt 5 ]; then
echo "Volume did not attach after $COUNT tries."
echo "Connect to AWS Console to terminate and clean volumes."
exit 1
fi
echo "Waiting for volumes $VOL1ID,$VOL2ID to be attached..."
sleep 15
done
# Quick hack, we do not care about sudo tty safety checks. Avoids a lot
# of piping magic through sudo/ssh/fifo
ssh -t -i $EC2_SSH_KEY ec2-user@$ADDRESS \
"sudo sed -i '/.*requiretty.*/d' /etc/sudoers"
# Format the 1GiB boot partition, mount, copy grub.conf + NetBSD kernel
echo "Creating the Grub boot partition"
ssh -i $EC2_SSH_KEY ec2-user@$ADDRESS \
"sudo mkfs.ext3 /dev/xvdf; sudo mount /dev/xvdf /mnt;" \
"sudo mkdir -p /mnt/boot/grub/"
ssh -i $EC2_SSH_KEY ec2-user@$ADDRESS \
"sudo dd of=/mnt/boot/grub/menu.lst" << EOF
default=0
timeout=0
hiddenmenu
title NetBSD AMI
root (hd0)
kernel /boot/netbsd root=xbd1
EOF
echo "Copying $KERNEL to $INSTANCEID"
progress -f "$KERNEL" ssh -i $EC2_SSH_KEY ec2-user@$ADDRESS \
"sudo dd of=/mnt/boot/netbsd"
# Overwrite the 5GiB volume with the prepared image
echo "Copying $IMGFILE to $INSTANCEID"
progress -f "$IMGFILE" ssh -i $EC2_SSH_KEY ec2-user@$ADDRESS \
"gunzip | sudo dd of=/dev/xvdg"
# Everything done. sync and umount.
ssh -i $EC2_SSH_KEY ec2-user@$ADDRESS "sudo sync; sudo umount /mnt"
# Snapshot the volumes.
SNAP1ID=$(ec2-create-snapshot $VOL1ID --region $REGION -d "Boot volume" |\
awk '{print $2}')
SNAP2ID=$(ec2-create-snapshot $VOL2ID --region $REGION -d "Root volume" |\
awk '{print $2}')
# Wait for the snapshots to be complete
COUNT=0
while true; do
COMPLETE=$(ec2-describe-snapshots --region $REGION $SNAP1ID $SNAP2ID |\
grep -c 'completed')
if [ "$COMPLETE" -eq 2 ]; then
echo "Created snapshots:"
echo "$SNAP1ID (1GiB -- boot volume)"
echo "$SNAP2ID (5GiB -- root file-system)"
break
fi
COUNT=$(($COUNT + 1))
if [ $COUNT -gt 5 ]; then
echo "One of $SNAP1ID,$SNAP2ID was not reported as " \
"completed after $COUNT tries. Leaving."
exit 1
fi
echo "Waiting for snapshots $SNAP1ID,$SNAP2ID to be completed..."
sleep 60
done
# Destroy the instance
ec2-terminate-instances $INSTANCEID --region $REGION
# We need the proper AKI to build the AMI. Our partitioning being:
# - 1GiB volume, contains grub.conf/menu.lst + boot kernel
# - 5GiB volume, the root partition
# we need an AKI where the volume is itself the boot partition, ie. "hd0" in
# Amazon terminology
AKIID=$(ec2-describe-images -o amazon --region $REGION \
-F image-type=kernel \
-F manifest-location=*pv-grub-hd0-* -F architecture=$ARCH |\
awk 'NR == 1 {print $2}')
# Get the old NetBSD AMI ID
ONBAMIID=$(ec2-describe-images -o self --region $REGION \
-F name=*NetBSD* -F architecture=$ARCH |\
awk 'NR == 1 {print $2}')
# Register the new AMI
NEWAMIID=$(ec2-register -a $ARCH --kernel $AKIID --region $REGION \
-b "/dev/sda1=$SNAP1ID" -b "/dev/sda2=$SNAP2ID" -n "NetBSD-$VERSION" \
-d "$DESCRIPTION" | awk '{print $2}')
# Summary:
echo
echo "A new AMI has been created: $NEWAMIID"
echo "You are advised to check that it is bootable by running the"
echo "following command:"
echo
echo " ec2-run-instances $NEWAMIID -t t1.micro --region $REGION"
echo
echo "then make the AMI public if it is bootable:"
echo " ec2-modify-image-attribute $NEWAMIID --region $REGION -l -a all"
echo
echo "If the new image boots correctly, you can proceed to the deletion"
echo "of the build volumes:"
echo
for VOL in $VOL1ID $VOL2ID; do
echo " ec2-delete-volume $VOL --region $REGION"
done
if [ "$ONBAMIID" != "" ]; then
# Obtain the snapshot IDs used to create the old AMI
SNAPIDS=$(ec2-describe-images $ONBAMIID --region $REGION |\
awk '/BLOCKDEVICEMAPPING/ {print $3}')
echo
echo "Do not forget to delete the old NetBSD AMI and its snapshots:"
echo
echo " ec2-deregister $ONBAMIID --region $REGION"
for SNAPID in $SNAPIDS; do
echo " ec2-delete-snapshot $SNAPID --region $REGION"
done
fi
Home |
Main Index |
Thread Index |
Old Index