Amazon EC2 Experience

To: port-xen%NetBSD.org@localhost
Subject: Amazon EC2 Experience
From: Thomas Laus <lausts%acm.org@localhost>
Date: Wed, 11 Mar 2020 10:41:50 -0400
List:

I was finally able to upload and install a NetBSD 9.0 image on an Amazon
HVM EC2 instance.  I was interested in checking out the performance of
the NetBSD 'npf' packet filter and compare it to the performance of the
OpenBSD 'pf' that I use on other running instances.  The pre-made AMI's
are a little old and they are all for the smallest PV older generation
t1 instances that Amazon offers.  I ran across one article on the NetBSD
wiki that lists the building procedure and script for a larger
production server.  I wanted to start on a smaller scale.

I ended up installing NetBSD Release 9.0 on another virtual machine with
MBR disk format.  I uploaded it as a snapshot to Amazon S3.  I converted
the snapshot to a volume and then registered it as an AMI on a t2
instance.  It booted but the vioif0 interface was not functional on EC2.
 The EC2 instance also could not find the disk drives.  The EC2 log did
start and I was able to read the error messages.  I destroyed the VM
images and this time formatted the disk using GPT.  This time the
instance booted cleanly, but no NIC was operational.  Once more, I
destroyed the VM image and this time added the Enhanced Network Adapter
(ENA) for the NIC.  The instance booted and I was able to login.  This
is the DMESG:


Mar 10 18:45:18 ip-172-31-13-54 /netbsd: [   1.0000000] Copyright (c)
1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005,
Mar 10 18:45:18 ip-172-31-13-54 /netbsd: [   1.0000000]     2006, 2007,
2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017,
Mar 10 18:45:18 ip-172-31-13-54 /netbsd: [   1.0000000]     2018, 2019,
2020 The NetBSD Foundation, Inc.  All rights reserved.
Mar 10 18:45:18 ip-172-31-13-54 /netbsd: [   1.0000000] Copyright (c)
1982, 1986, 1989, 1991, 1993
Mar 10 18:45:18 ip-172-31-13-54 /netbsd: [   1.0000000]     The Regents
of the University of California.  All rights reserved.
Mar 10 18:45:18 ip-172-31-13-54 /netbsd:
Mar 10 18:45:18 ip-172-31-13-54 /netbsd: [   1.0000000] NetBSD 9.0
(GENERIC) #0: Fri Feb 14 00:06:28 UTC 2020
Mar 10 18:45:18 ip-172-31-13-54 /netbsd: [   1.0000000]
mkrepro%mkrepro.NetBSD.org@localhost:/usr/src/sys/arch/amd64/compile/GENERIC
Mar 10 18:45:18 ip-172-31-13-54 /netbsd: [   1.0000000] total memory =
489 MB
Mar 10 18:45:18 ip-172-31-13-54 /netbsd: [   1.0000000] avail memory =
451 MB
Mar 10 18:45:18 ip-172-31-13-54 /netbsd: [   1.0000000] cpu_rng: RDSEED
Mar 10 18:45:18 ip-172-31-13-54 /netbsd: [   1.0000000] rnd: seeded with
256 bits
Mar 10 18:45:18 ip-172-31-13-54 /netbsd: [   1.0000000] timecounter:
Timecounters tick every 10.000 msec
Mar 10 18:45:18 ip-172-31-13-54 /netbsd: [   1.0000000] Kernelized
RAIDframe activated
Mar 10 18:45:18 ip-172-31-13-54 /netbsd: [   1.0000000] running cgd
selftest aes-xts-256 aes-xts-512 done
Mar 10 18:45:18 ip-172-31-13-54 /netbsd: [   1.0000000] timecounter:
Timecounter "i8254" frequency 1193182 Hz quality 100
Mar 10 18:45:18 ip-172-31-13-54 /netbsd: [   1.0000030] Amazon EC2 t3.nano
Mar 10 18:45:18 ip-172-31-13-54 /netbsd: [   1.0000030] mainbus0 (root)
Mar 10 18:45:18 ip-172-31-13-54 /netbsd: [   1.0000030] ACPI: RSDP
0x00000000000F8FA0 000014 (v00 AMAZON)
Mar 10 18:45:18 ip-172-31-13-54 /netbsd: [   1.0000030] ACPI: RSDT
0x000000001E9FE360 00003C (v01 AMAZON AMZNRSDT 00000001 AMZN 00000001)
Mar 10 18:45:18 ip-172-31-13-54 /netbsd: [   1.0000030] ACPI: FACP
0x000000001E9FFF80 000074 (v01 AMAZON AMZNFACP 00000001 AMZN 00000001)
Mar 10 18:45:18 ip-172-31-13-54 /netbsd: [   1.0000030] ACPI: DSDT
0x000000001E9FE3A0 0010E9 (v01 AMAZON AMZNDSDT 00000001 AMZN 00000001)
Mar 10 18:45:18 ip-172-31-13-54 /netbsd: [   1.0000030] ACPI: FACS
0x000000001E9FFF40 000040
Mar 10 18:45:18 ip-172-31-13-54 /netbsd: [   1.0000030] ACPI: SSDT
0x000000001E9FF6C0 00087A (v01 AMAZON AMZNSSDT 00000001 AMZN 00000001)
Mar 10 18:45:18 ip-172-31-13-54 /netbsd: [   1.0000030] ACPI: APIC
0x000000001E9FF5D0 000076 (v01 AMAZON AMZNAPIC 00000001 AMZN 00000001)
Mar 10 18:45:18 ip-172-31-13-54 /netbsd: [   1.0000030] ACPI: SRAT
0x000000001E9FF530 0000A0 (v01 AMAZON AMZNSRAT 00000001 AMZN 00000001)
Mar 10 18:45:18 ip-172-31-13-54 /netbsd: [   1.0000030] ACPI: SLIT
0x000000001E9FF4C0 00006C (v01 AMAZON AMZNSLIT 00000001 AMZN 00000001)
Mar 10 18:45:18 ip-172-31-13-54 /netbsd: [   1.0000030] ACPI: WAET
0x000000001E9FF490 000028 (v01 AMAZON AMZNWAET 00000001 AMZN 00000001)
Mar 10 18:45:18 ip-172-31-13-54 /netbsd: [   1.0000030] ACPI: 2 ACPI AML
tables successfully acquired and loaded
Mar 10 18:45:18 ip-172-31-13-54 /netbsd: [   1.0000030] ioapic0 at
mainbus0 apid 0: pa 0xfec00000, version 0x11, 24 pins
Mar 10 18:45:18 ip-172-31-13-54 /netbsd: [   1.0000030] cpu0 at mainbus0
apid 0
Mar 10 18:45:18 ip-172-31-13-54 /netbsd: [   1.0000030] cpu0: Intel(R)
Xeon(R) Platinum 8175M CPU @ 2.50GHz, id 0x50654
Mar 10 18:45:18 ip-172-31-13-54 /netbsd: [   1.0000030] cpu0: package 0,
core 0, smt 0
Mar 10 18:45:18 ip-172-31-13-54 /netbsd: [   1.0000030] cpu1 at mainbus0
apid 1
Mar 10 18:45:18 ip-172-31-13-54 /netbsd: [   1.0000030] cpu1: Intel(R)
Xeon(R) Platinum 8175M CPU @ 2.50GHz, id 0x50654
Mar 10 18:45:18 ip-172-31-13-54 /netbsd: [   1.0000030] cpu1: package 0,
core 0, smt 1
Mar 10 18:45:18 ip-172-31-13-54 /netbsd: [   1.0000030] acpi0 at
mainbus0: Intel ACPICA 20190405
Mar 10 18:45:18 ip-172-31-13-54 /netbsd: [   1.0000030] acpi0: X/RSDT:
OemId <AMAZON,AMZNRSDT,00000001>, AslId <AMZN,00000001>
Mar 10 18:45:18 ip-172-31-13-54 /netbsd: [   1.0000030] LNKD: ACPI:
Found matching pin for 0.4.INTA at func 0: 11
Mar 10 18:45:18 ip-172-31-13-54 /netbsd: [   1.0000030] acpi0: SCI
interrupting at int 9
Mar 10 18:45:18 ip-172-31-13-54 /netbsd: [   1.0000030] acpi0: fixed
power button present
Mar 10 18:45:18 ip-172-31-13-54 /netbsd: [   1.0000030] acpi0: fixed
sleep button present
Mar 10 18:45:18 ip-172-31-13-54 /netbsd: [   1.0000030] timecounter:
Timecounter "ACPI-Safe" frequency 3579545 Hz quality 900
Mar 10 18:45:18 ip-172-31-13-54 /netbsd: [   1.0150016] pckbc1 at acpi0
(KBD, PNP0303) (kbd port): io 0x60,0x64 irq 1
Mar 10 18:45:18 ip-172-31-13-54 /netbsd: [   1.0150016] pckbc2 at acpi0
(MOU, PNP0F13) (aux port): irq 12
Mar 10 18:45:18 ip-172-31-13-54 /netbsd: [   1.0150016] LPT (PNP0400) at
acpi0 not configured
Mar 10 18:45:18 ip-172-31-13-54 /netbsd: [   1.0150016] COM1 (PNP0501)
at acpi0 not configured
Mar 10 18:45:18 ip-172-31-13-54 /netbsd: [   1.0150016] ACPI: Enabled 16
GPEs in block 00 to 0F
Mar 10 18:45:18 ip-172-31-13-54 /netbsd: [   1.0150016] pckbd0 at pckbc1
(kbd slot)
Mar 10 18:45:18 ip-172-31-13-54 /netbsd: [   1.0150016] pckbc1: using
irq 1 for kbd slot
Mar 10 18:45:18 ip-172-31-13-54 /netbsd: [   1.0150016] wskbd0 at pckbd0
mux 1
Mar 10 18:45:18 ip-172-31-13-54 /netbsd: [   1.0150016] pms0 at pckbc1
(aux slot)
Mar 10 18:45:18 ip-172-31-13-54 /netbsd: [   1.0150016] pckbc1: using
irq 12 for aux slot
Mar 10 18:45:18 ip-172-31-13-54 /netbsd: [   1.0150016] wsmouse0 at pms0
mux 0
Mar 10 18:45:18 ip-172-31-13-54 /netbsd: [   1.0150016] pci0 at mainbus0
bus 0: configuration mode 1
Mar 10 18:45:18 ip-172-31-13-54 /netbsd: [   1.0150016] pci0: i/o space,
memory space enabled, rd/line, rd/mult, wr/inv ok
Mar 10 18:45:18 ip-172-31-13-54 /netbsd: [   1.0150016] pchb0 at pci0
dev 0 function 0: vendor 8086 product 1237 (rev. 0x00)
Mar 10 18:45:18 ip-172-31-13-54 /netbsd: [   1.0150016] pcib0 at pci0
dev 1 function 0: vendor 8086 product 7000 (rev. 0x00)
Mar 10 18:45:18 ip-172-31-13-54 /netbsd: [   1.0150016] piixpm0 at pci0
dev 1 function 3: vendor 8086 product 7113 (rev. 0x08)
Mar 10 18:45:18 ip-172-31-13-54 /netbsd: [   1.0150016] timecounter:
Timecounter "piixpm0" frequency 3579545 Hz quality 1000
Mar 10 18:45:18 ip-172-31-13-54 /netbsd: [   1.7632547] piixpm0: 24-bit
timer
Mar 10 18:45:18 ip-172-31-13-54 /netbsd: [   1.7632547] piixpm0: polling
Mar 10 18:45:18 ip-172-31-13-54 /netbsd: [   1.7632547] iic0 at piixpm0
port 0: I2C bus
Mar 10 18:45:18 ip-172-31-13-54 /netbsd: [   1.7632547] vga0 at pci0 dev
3 function 0: vendor 1d0f product 1111 (rev. 0x00)
Mar 10 18:45:18 ip-172-31-13-54 /netbsd: [   1.7632547] wsdisplay0 at
vga0 kbdmux 1
Mar 10 18:45:18 ip-172-31-13-54 /netbsd: [   1.7632547] wsmux1:
connecting to wsdisplay0
Mar 10 18:45:18 ip-172-31-13-54 /netbsd: [   1.7632547] wskbd0:
connecting to wsdisplay0
Mar 10 18:45:18 ip-172-31-13-54 /netbsd: [   1.7632547] drm at vga0 not
configured
Mar 10 18:45:18 ip-172-31-13-54 /netbsd: [   1.7632547] nvme0 at pci0
dev 4 function 0: vendor 1d0f product 8061 (rev. 0x00)
Mar 10 18:45:18 ip-172-31-13-54 /netbsd: [   1.7632547] nvme0: NVMe 1.0
Mar 10 18:45:18 ip-172-31-13-54 /netbsd: [   1.7632547] nvme0: for admin
queue interrupting at msix0 vec 0
Mar 10 18:45:18 ip-172-31-13-54 /netbsd: [   1.7632547] nvme0: Amazon
Elastic Block Store, firmware 1.0, serial vol0e74e5d51938c154d
Mar 10 18:45:18 ip-172-31-13-54 /netbsd: [   1.7632547] nvme0: for io
queue 1 interrupting at msix0 vec 1 affinity to cpu0
Mar 10 18:45:18 ip-172-31-13-54 /netbsd: [   1.7632547] nvme0: for io
queue 2 interrupting at msix0 vec 2 affinity to cpu1
Mar 10 18:45:18 ip-172-31-13-54 /netbsd: [   1.7632547] ld0 at nvme0 nsid 1
Mar 10 18:45:18 ip-172-31-13-54 /netbsd: [   1.7632547] ld0: 5120 MB,
2600 cyl, 64 head, 63 sec, 512 bytes/sect x 10485760 sectors
Mar 10 18:45:18 ip-172-31-13-54 /netbsd: [   1.7632547] ena0 at pci0 dev
5 function 0: vendor 1d0f product ec20 (rev. 0x00)
Mar 10 18:45:18 ip-172-31-13-54 /netbsd: [   1.7632547] pci0: Elastic
Network Adapter (ENA)ena v0.8.1
Mar 10 18:45:18 ip-172-31-13-54 /netbsd: [   1.7632547] ena0: initalize
2 io queues
Mar 10 18:45:18 ip-172-31-13-54 /netbsd: [   1.7632547] ena0: for MGMNT
interrupting at msix1 vec 0
Mar 10 18:45:18 ip-172-31-13-54 /netbsd: [   1.7632547] isa0 at pcib0
Mar 10 18:45:18 ip-172-31-13-54 /netbsd: [   1.7632547] com0 at isa0
port 0x3f8-0x3ff irq 4: ns16550a, working fifo
Mar 10 18:45:18 ip-172-31-13-54 /netbsd: [   1.7632547] com0: console
Mar 10 18:45:18 ip-172-31-13-54 /netbsd: [   1.7632547] attimer0 at isa0
port 0x40-0x43
Mar 10 18:45:18 ip-172-31-13-54 /netbsd: [   1.7632547] pcppi0 at isa0
port 0x61
Mar 10 18:45:18 ip-172-31-13-54 /netbsd: [   1.7632547] spkr0 at pcppi0:
PC Speaker
Mar 10 18:45:18 ip-172-31-13-54 /netbsd: [   1.7632547] wsbell at spkr0
not configured
Mar 10 18:45:18 ip-172-31-13-54 /netbsd: [   1.7632547] midi0 at pcppi0:
PC speaker
Mar 10 18:45:18 ip-172-31-13-54 /netbsd: [   1.7632547] sysbeep0 at pcppi0
Mar 10 18:45:18 ip-172-31-13-54 /netbsd: [   1.7632547] attimer0:
attached to pcppi0
Mar 10 18:45:18 ip-172-31-13-54 /netbsd: [   1.7632547] acpicpu0 at
cpu0: ACPI CPU
Mar 10 18:45:18 ip-172-31-13-54 /netbsd: [   1.7632547] acpicpu0: C1:
HLT, lat   0 us, pow     0 mW
Mar 10 18:45:18 ip-172-31-13-54 /netbsd: [   1.7632547] acpicpu1 at
cpu1: ACPI CPU
Mar 10 18:45:18 ip-172-31-13-54 /netbsd: [   1.7632547] ena0: link is UP
Mar 10 18:45:18 ip-172-31-13-54 /netbsd: [   1.7632547] timecounter:
Timecounter "clockinterrupt" frequency 100 Hz quality 0
Mar 10 18:45:18 ip-172-31-13-54 /netbsd: [   2.5925646] timecounter:
Timecounter "TSC" frequency 2500094220 Hz quality 3000
Mar 10 18:45:18 ip-172-31-13-54 /netbsd: [   2.5926462] ld0: GPT GUID:
46591d1b-d8ec-4b39-b48a-4b137d3d2013
Mar 10 18:45:18 ip-172-31-13-54 /netbsd: [   2.5926462] dk0 at ld0:
"47cd9546-021b-4996-92f7-189692dbe517", 9961405 blocks at 64, type: ffs
Mar 10 18:45:18 ip-172-31-13-54 /netbsd: [   2.5926462] dk1 at ld0:
"e73aa67d-d54e-47a7-89bc-d132764f9393", 524255 blocks at 9961472, type: swap
Mar 10 18:45:18 ip-172-31-13-54 /netbsd: [   2.5926462] IPsec:
Initialized Security Association Processing.
Mar 10 18:45:18 ip-172-31-13-54 /netbsd: [   2.6726080] boot device: ld0
Mar 10 18:45:18 ip-172-31-13-54 /netbsd: [   2.6726080] root on dk0
dumps on dk1
Mar 10 18:45:18 ip-172-31-13-54 /netbsd: [   2.6726080] root file system
type: ffs
Mar 10 18:45:18 ip-172-31-13-54 /netbsd: [   2.6726080]
kern.module.path=/stand/amd64/9.0/modules
Mar 10 18:45:18 ip-172-31-13-54 /netbsd: [   7.2755494] ena0: device is
going UP
Mar 10 18:45:18 ip-172-31-13-54 /netbsd: [   7.2755494] ena0: for IO
queue 0 interrupting at msix1 vec 1 affinity to 1
Mar 10 18:45:18 ip-172-31-13-54 /netbsd: [   7.2855526] ena0: for IO
queue 1 interrupting at msix1 vec 2 affinity to 0
Mar 10 18:45:18 ip-172-31-13-54 /netbsd: [  13.8897818] wsdisplay0:
screen 1 added (80x25, vt100 emulation)
Mar 10 18:45:18 ip-172-31-13-54 /netbsd: [  13.8897818] wsdisplay0:
screen 2 added (80x25, vt100 emulation)
Mar 10 18:45:18 ip-172-31-13-54 /netbsd: [  13.8897818] wsdisplay0:
screen 3 added (80x25, vt100 emulation)
Mar 10 18:45:18 ip-172-31-13-54 /netbsd: [  13.8897818] wsdisplay0:
screen 4 added (80x25, vt100 emulation)
Mar 10 18:45:23 ip-172-31-13-54 dhcpcd[111]: ena0: no IPv6 Routers available
Mar 10 18:45:29 ip-172-31-13-54 ntpdate[281]: step time server
209.51.161.238 offset 1.056384 sec
Mar 10 18:45:29 ip-172-31-13-54 savecore: no core dump
Mar 10 18:45:30 ip-172-31-13-54 ntpd[410]: ntpd 4.2.8p11-o Sat Sep 29
17:04:56 EDT 2018 (import): Starting
Mar 10 18:45:30 ip-172-31-13-54 ntpd[410]: Command line: /usr/sbin/ntpd
-p /var/run/ntpd.pid
Mar 10 18:45:30 ip-172-31-13-54 ntpd[315]: proto: precision = 0.360 usec
(-21)
Mar 10 18:45:30 ip-172-31-13-54 ntpd[315]: Listen and drop on 0
v6wildcard [::]:123
Mar 10 18:45:30 ip-172-31-13-54 ntpd[315]: Listen and drop on 1
v4wildcard 0.0.0.0:123
Mar 10 18:45:30 ip-172-31-13-54 ntpd[315]: Listen normally on 2 ena0
[fe80::10cf:724:801d:f3f9%1]:123
Mar 10 18:45:30 ip-172-31-13-54 ntpd[315]: Listen normally on 3 ena0
172.31.13.54:123
Mar 10 18:45:30 ip-172-31-13-54 ntpd[315]: Listen normally on 4 lo0
127.0.0.1:123
Mar 10 18:45:30 ip-172-31-13-54 ntpd[315]: Listen normally on 5 lo0
[::1]:123
Mar 10 18:45:30 ip-172-31-13-54 ntpd[315]: Listen normally on 6 lo0
[fe80::1%2]:123
Mar 10 18:45:30 ip-172-31-13-54 ntpd[315]: Listening on routing socket
on fd #27 for interface updates
Mar 10 18:45:30 ip-172-31-13-54 ntpd[315]: kernel reports TIME_ERROR:
0x41: Clock Unsynchronized
Mar 10 18:45:30 ip-172-31-13-54 ntpd[315]: kernel reports TIME_ERROR:
0x41: Clock Unsynchronized
Mar 10 18:54:21 ip-172-31-13-54 ntpd[315]: kernel reports TIME_ERROR:
0x41: Clock Unsynchronized

Installation and conversion of a running VM is a lot easier than the
previous NetBSD instructions that involve the Linux shim and only having
a PV instance.  Amazon doesn't have many t1 instance offerings anymore.
 Using the ENA adapter works on:

instance types: A1, C5, C5d, C5n, F1, G3, G4, H1, I3, I3en, Inf1,
m4.16xlarge, M5, M5a, M5ad, M5d,
M5dn, M5n, P2, P3, R4, R5, R5a, R5ad, R5d, R5dn, R5n, T3, T3a,
u-6tb1.metal, u-9tb1.metal,
u-12tb1.metal, u-18tb1.metal, u-24tb1.metal, X1, X1e, and z1d.

That rules out the t2-small offered for the free tier.  I used a t3-nano
which has a very low cost for this test.  I plan on uploading and
converting a generic AMI on us-east-1 in a few days if anyone wants to
try NetBSD on an Amazon EC2 instance.

Tom





-- 
Public Keys:
PGP KeyID = 0x5F22FDC1
GnuPG KeyID = 0x620836CF
Prev by Date: xennet vs Windows checksum offload - help wanted
Next by Date: XEN command parameters for pciback(4)?
Previous by Thread: xennet vs Windows checksum offload - help wanted
Next by Thread: XEN command parameters for pciback(4)?
Indexes:
Home | Main Index | Thread Index | Old Index