On Wed, May 14, 2003 at 05:50:32PM +0200, Milos Urbanek wrote:

Actually the problem is not easily reproducible.

I've run 3 times below mentioned cvs update of a 45 MB MP3 file; the machine
went to swap quite easily (about 74 MB of swap was allocated). But I did not
locked up..

At least I think that it is unusual to go to swap so hard when only
writing some file (although large).

Before I start the write, top shows:

Memory: 47M Act, 22M Inact, 716K Wired, 7196K Exec, 56M File, 35M Free
(note: total memory = 127 MB avail memory = 114 MB)

When I run the large file write, the machine goes somewhere at 768K of Free RAM
(seen in top).
It looks like the page daemon does not free enough pages. Actually
from time to time (in period of about 1 minute) there is a quick jump of free
RAM from those 768K to about 41MB of free RAM (looks like many pages from
UBC are written to disk). But most of the time there is only about 200 KB of
memory free; the pageout wokes up, frees some memory (to about 4MB of Free RAM)
then goes to sleep and during the time the swap is filling with new
pages.. 

Is it an expected behaviour? Should not pageout daemon first free those pages
that belong to UBC and only then continue with pages that belong e.g.
to executable programs, anons etc. that are modified and has to go to swap?
(I have rather standard vm settings:
vm.anonmax = 80
vm.execmax = 30
vm.filemax = 50
vm.anonmin = 10
vm.execmin = 5
vm.filemin = 10
)

Milos


> I get aswell from time to time "hard lockup" with NetBSD 1.6P.
> Last time it happened (two days ago) when updating my local cvs tree that
> contained two 45 MB MP3 files. I'm running X all the time so no
> chance to get to debugger.
> 
> Hard lockup:
> 	- unable to ping machine from another one
> 	- unable to do anything in X (including mouse movement etc)
> 	- unable to get to debugger / console
> 	- no logs available
> 	- shortly before lockup there is a period if high disk activity
> 	(writing a file); during that some pages are swapped out
> 	(like X server etc.) machine goes less interactively but it runs
> 	- when the lockup occurs all activity is stopped (disk LEDs
> 	are silent etc.)
> 	- I'm still able to switch numlock; so some interrupts are probably
> 	working
> 
> Machine with 128 MB of memory. Dmesg follows;
> Note that I use local kernel config that is slightly different from GENERIC
> options         DIAGNOSTIC      # expensive kernel consistency checks
> options         DEBUG           # expensive debugging checks/support
> options         DDB             # in-kernel debugger
> options         DDB_HISTORY_SIZE=512    # enable history editing in DDB  
> options         NEW_BUFQ_STRATEGY
> 
> 	- drivers that are not needed are removed from kernel
> 
> I was able to crash the machine the same way when running NetBSD 1.5ZC
> (that time it was with GENERIC kernel). I was never able to get to debugger
> to get valuable information.
> 
> Milos
> 
> Dmesg>
> 
> NetBSD 1.6P (OAKLAND) #1: Mon Mar 31 13:10:26 MEST 2003
>         root@oakland:/usr/src/sys/arch/i386/compile/OAKLAND
> total memory = 127 MB
> avail memory = 114 MB
> using 1658 buffers containing 6632 KB of memory
> BIOS32 rev. 0 found at 0xfb310
> mainbus0 (root)
> cpu0 at mainbus0: (uniprocessor)
> cpu0: AMD Duron (686-class), 800.12 MHz, id 0x631
> cpu0: features c1c7f9ff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,SEP,MTRR>
> cpu0: features c1c7f9ff<PGE,MCA,CMOV,PAT,PSE36,PN,MMXX,MMX>
> cpu0: features c1c7f9ff<FXSR,3DNOW2,3DNOW>
> cpu0: I-cache 64 KB 64b/line 2-way, D-cache 64 KB 64b/line 2-way
> cpu0: L2 cache 64 KB 64b/line 16-way
> cpu0: ITLB 16 4 KB entries fully associative, 8 4 MB entries fully associative
> cpu0: DTLB 24 4 KB entries fully associative, 8 4 MB entries 4-way
> cpu0: 8 page colors
> pci0 at mainbus0 bus 0: configuration mode 1
> pci0: i/o space, memory space enabled, rd/line, rd/mult, wr/inv ok
> pchb0 at pci0 dev 0 function 0
> pchb0: VIA Technologies VT8363 KT133 System Controller (rev. 0x03)
> agp0 at pchb0: aperture at 0xd0000000, size 0x10000000
> ppb0 at pci0 dev 1 function 0: VIA Technologies VT8363 KT133 PCI to AGP Bridge (rev. 0x00)
> pci1 at ppb0 bus 1
> pci1: i/o space, memory space enabled
> vga1 at pci1 dev 0 function 0: ATI Technologies Rage XL (AGP) (rev. 0x65)
> wsdisplay0 at vga1 kbdmux 1: console (80x25, vt100 emulation)
> wsmux1: connecting to wsdisplay0
> pcib0 at pci0 dev 7 function 0
> pcib0: VIA Technologies VT82C686A (Apollo KX133) PCI-ISA Bridge (rev. 0x40)
> pciide0 at pci0 dev 7 function 1: VIA Technologies VT82C686A (Apollo KX133) ATA1
> 00 controller
> pciide0: bus-master DMA support present
> pciide0: primary channel configured to compatibility mode
> wd0 at pciide0 channel 0 drive 0: <IBM-DJNA-371350>
> wd0: drive supports 16-sector PIO transfers, LBA addressing
> wd0: 12949 MB, 16383 cyl, 16 head, 63 sec, 512 bytes/sect x 26520480 sectors
> wd0: 32-bit data port
> wd0: drive supports PIO mode 4, DMA mode 2, Ultra-DMA mode 4 (Ultra/66)
> wd1 at pciide0 channel 0 drive 1: <WDC WD300AB-00BVA0>
> wd1: drive supports 16-sector PIO transfers, LBA addressing
> wd1: 28629 MB, 16383 cyl, 16 head, 63 sec, 512 bytes/sect x 58633344 sectors
> wd1: 32-bit data port
> wd1: drive supports PIO mode 4, DMA mode 2, Ultra-DMA mode 5 (Ultra/100)
> pciide0: primary channel interrupting at irq 14
> wd0(pciide0:0:0): using PIO mode 4, Ultra-DMA mode 4 (Ultra/66) (using DMA data 
> transfers)
> wd1(pciide0:0:1): using PIO mode 4, Ultra-DMA mode 5 (Ultra/100) (using DMA data
>  transfers)
> pciide0: secondary channel configured to compatibility mode
> pciide0: disabling secondary channel (no drives)
> uhci0 at pci0 dev 7 function 2: VIA Technologies VT83C572 USB Controller (rev. 0
> x16)
> uhci0: interrupting at irq 9
> usb0 at uhci0: USB revision 1.0
> uhub0 at usb0
> uhub0: VIA Technologies UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
> uhub0: 2 ports with 2 removable, self powered
> uhci1 at pci0 dev 7 function 3: VIA Technologies VT83C572 USB Controller (rev. 0
> x16)
> uhci1: interrupting at irq 9
> usb1 at uhci1: USB revision 1.0
> uhub1 at usb1
> uhub1: VIA Technologies UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
> uhub1: 2 ports with 2 removable, self powered
> viapm0 at pci0 dev 7 function 4
> viaenv0 at viapm0
> auvia0 at pci0 dev 7 function 5: VIA VT82C686A AC'97 Audio (rev 0x50)
> auvia0: ICEnsemble ICE1232/VT1611A codec; headphone, 18 bit DAC, 18 bit ADC, KS 
> Waves 3D
> auvia0: variable rate audio
> audio0 at auvia0: full duplex, mmap, independent
> ex0 at pci0 dev 18 function 0: 3Com 3c905C-TX 10/100 Ethernet with mngmt (rev. 0
> x74)
> ex0: interrupting at irq 11
> ex0: MAC address 00:01:02:db:4f:a5
> bmtphy0 at ex0 phy 24: Broadcom 3c905C internal PHY, rev. 6
> bmtphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
> isa0 at pcib0
> com0 at isa0 port 0x3f8-0x3ff irq 4: ns16550a, working fifo
> com1 at isa0 port 0x2f8-0x2ff irq 3: ns16550a, working fifo
> pckbc0 at isa0 port 0x60-0x64
> pckbd0 at pckbc0 (kbd slot)
> pckbc0: using irq 1 for kbd slot
> wskbd0 at pckbd0: console keyboard, using wsdisplay0
> pms0 at pckbc0 (aux slot)
> pckbc0: using irq 12 for aux slot
> wsmouse0 at pms0 mux 0
> lpt0 at isa0 port 0x378-0x37b irq 7
> pcppi0 at isa0 port 0x61
> midi0 at pcppi0: PC speaker
> sysbeep0 at pcppi0
> isapnp0 at isa0 port 0x279: ISA Plug 'n Play device support
> npx0 at isa0 port 0xf0-0xff: using exception 16
> fdc0 at isa0 port 0x3f0-0x3f7 irq 6 drq 2
> isapnp0: no ISA Plug 'n Play devices found
> apm0 at mainbus0: Power Management spec V1.2 (slowidle)
> IPsec: Initialized Security Association Processing.
> uhub0: port error, restarting port 1
> uhub0: port error, giving up port 1
> uhub0: port error, restarting port 2
> uhub0: port error, giving up port 2
> uhub1: port error, restarting port 1
> uhub1: port error, giving up port 1
> uhub1: port error, restarting port 2
> uhub1: port error, giving up port 2
> boot device: wd0
> root on wd0a dumps on wd0b
> mountroot: trying smbfs...
> mountroot: trying coda...
> mountroot: trying msdos...
> mountroot: trying cd9660...
> isofs: session offset (part 0) 0
> mountroot: trying ntfs...
> mountroot: trying nfs...
> mountroot: trying ext2fs...
> mountroot: trying ffs...
> root file system type: ffs
> init: copying out path `/sbin/init' 11
> 
> It happens when running e.g. 
> > I am running 1.6.1 on a /i386 machine with 128Mb memory. I get regular
> > lock-ups during times of high-memory, high-swap activity, typically
> > during a nightly rsync run against a large filesystem on a remote
> > machine.  I can still break into the debugger, but all higher level
> > function are frozen. (I don't even get character echo on the serial
> > console). The tracebacks for the last two lock-ups are:
> > 
> > pmap_extract(c0542400,c0cf1000,cbc98cac,c0241d56) at pmap_extract+0xc
> > uvm_km_pgremove_intrsafe(c0cdf000,c0d28000,c0cdf000,c0307bf0,c04fc8e0,cf8000,300
> > 00,0,1727,0,cbc98d0c,c04fc908,c04fc908,0,cbc98d30,c0301b5f,c04fc8e0,c0cdf000,c0d
> > 28000,cbc98d2c,c0cf8000,cf8000,30000,e000ffe6,1,3,cbc98d70,cbc98d2c,0,0,cbc98d70
> > ,c0300c87,c04fc8e0,c0cdf000,c0d28000,1727,c052c5e0,49000,c052bce0,c024217a,49000
> > ,c0c16000,cbc98db0,c023bdf1,49000,c0cdf000,cbc98dc0,c023bdf1,c04fc8e0,0,49000,1,
> > cbcac3d0,49000,1,ffffffff,49000,0,cbc98dd0,ffffffff,49000,0) at uvm_km_pgremove_
> > intrsafe+0x2a
> > uvm_unmap_remove(c04fc8e0,c0cdf000,c0d28000,cbc98d2c) at uvm_unmap_remove+0x100
> > uvm_unmap(c04fc8e0,c0cdf000,c0d28000,1727,c052c5e0) at uvm_unmap+0x8f
> > uvm_km_kmemalloc(c04fc8e0,0,49000,1,cbcac3d0) at uvm_km_kmemalloc+0x7f
> > malloc(49000,52,1,0,cbb952f8) at malloc+0x249
> > amap_copy(cbb952f8,cbcb0780,1,1,f9eb000,f9eb001,cbc98f48,286) at amap_copy+0x16e
> > 
> > uvmfault_amapcopy(cbc98f34,6,0,1,0) at uvmfault_amapcopy+0x128
> > uvm_fault(cbb952f8,f9eb000,0,2,4811f094) at uvm_fault+0x1b6
> > trap() at trap+0x4d4
> > --- trap (number 6) ---
> > 0x481112e4:
> > 
> > 
> > and
> > 
> > uvm_pagealloc_strat(0,c7e000,0,0,1,0,0,1727) at uvm_pagealloc_strat+0x141
> > uvm_km_kmemalloc(c04fc8e0,0,49000,1,cba5fd48) at uvm_km_kmemalloc+0xbd
> > malloc(49000,52,1,0,cb9c0478) at malloc+0x249
> > amap_copy(cb9c0478,cba465d0,1,1,1a2bb000,1a2bb001,cbbf2f48,202) at amap_copy+0x1
> > 6e
> > uvmfault_amapcopy(cbbf2f34,6,0,1,0) at uvmfault_amapcopy+0x128
> > uvm_fault(cb9c0478,1a2bb000,0,2,4811f094) at uvm_fault+0x1b6
> > trap() at trap+0x4d4
> > --- trap (number 6) ---
> > 0x481112e4:
> > 
> > 
> > I cought the last one on systat vm and top:
> > 
> >     2 users    Load  2.68  1.96  1.05                  Wed May 14 05:57:51
> > 
> >           memory totals (in KB)             PAGING   SWAPPING      Interrupts
> >          real   virtual    free             in  out   in  out      1425 total
> > Active  67988    374524    1020     ops   1320    3                 100 irq0
> > All    123180    429716  469092     pages        20                     irq4
> >                                                                         irq6
> > Proc:r  d  s  w    Csw   Trp   Sys  Int  Sof   Flt        forks         irq10
> >         1  9      1344  2651    40 1426 1432  1330        fkppw       2 irq11
> >                                                           fksvm    1323 irq12
> >    6.4% Sy  20.8% Us   0.0% Ni   2.0% In  70.9% Id        pwait         irq15
> > |    |    |    |    |    |    |    |    |    |    |  1320 relck
> > ===>>>>>>>>>>>%                                      1320 rlkok
> >                                                           noram
> > Namei         Sys-cache     Proc-cache                    ndcpy
> >     Calls     hits    %     hits     %                    fltcp
> >                                                           zfod
> >                                                           cow
> > Discs  cd0  wd0  wd1  fd0  md0                         64 fmin
> > seeks                                                  85 ftarg
> > xfers       659  664                                 8596 itarg
> > Kbyte      2671 2704                                  436 wired
> > %busy      45.7 28.7                                 1308 pdfre
> > 
> > 
> > load averages:  2.71,  1.97,  1.06                                     05:57:55
> > 62 processes:  1 runnable, 60 sleeping, 1 on processor
> > CPU states:  3.0% user,  0.0% nice,  6.5% system,  2.5% interrupt, 88.1% idle
> > Memory: 67M Act, 34M Inact, 1744K Wired, 2408K Exec, 1868K File, 300K Free
> > Swap: 756M Total, 299M Used, 457M Free
> > 
> >   PID USERNAME PRI NICE   SIZE   RES STATE      TIME   WCPU    CPU COMMAND
> >  2012 root      -5    0   290M   96M RUN        2:20 25.68% 25.68% rsync
> >     6 root     -18    0     0K   14M pgdaemon   0:12  0.44%  0.44% [pagedaemon]
> >     8 root      18    0     0K   14M syncer     0:24  0.00%  0.00% [ioflush]
> >  1009 root      28    0   244K  804K CPU        0:16  0.00%  0.00% top
> >  1008 root       3    0   472K  676K ttyin      0:07  0.00%  0.00% systat
> >   315 andreas    2    0   524K 1308K select     0:06  0.00%  0.00% sshd
> >   243 root       2    0  4908K 1200K select     0:06  0.00%  0.00% squid
> >   238 root       2    0   380K    4K select     0:05  0.00%  0.00% <sshd>
> >   293 andreas    2    0   524K 1308K select     0:03  0.00%  0.00% sshd
> >   219 root      18  -12   716K 1552K pause      0:02  0.00%  0.00% ntpd
> >   121 root      10    0   672K  468K nanoslee   0:01  0.00%  0.00% ipmon
> >  2015 root       2    0   476K    4K select     0:01  0.00%  0.00% <rcmd>
> >     9 root     -18    0     0K   14M aiodoned   0:01  0.00%  0.00% [aiodoned]
> >   377 root      18    0   540K    4K pause      0:00  0.00%  0.00% <ksh>
> >   335 root      18    0   540K    4K pause      0:00  0.00%  0.00% <ksh>
> >   316 andreas   18    0   504K    4K pause      0:00  0.00%  0.00% <ksh>
> >   294 andreas   18    0   504K    4K pause      0:00  0.00%  0.00% <ksh>
> > 
> > 
> > ----
> > 
> > The problem did occur with a -current kernel from early February.
> > I am about to build a -current kernel for the machine to see if the
> > problem persists.
> > 
> > I suspect I can "fix" this problem by installing more memory and I'll
> > probably do that after the real problem is fixed.
> > 
> > Any suggestions as to how I should proceed?
> > 
> > -- 
> >     - aew
> > 
> > 
> > NetBSD 1.6.1 (PLANIX) #2: Sun May 11 12:00:04 EDT 2003
> >     root@wonder.wrede.pvt:/usr/src/sys/arch/i386/compile/PLANIX
> > cpu0: Intel Pentium III (Coppermine) (686-class), 701.68 MHz
> > cpu0: I-cache 16 KB 32b/line 4-way, D-cache 16 KB 32b/line 2-way
> > cpu0: L2 cache 256 KB 32b/line 8-way
> > cpu0: features 387f9ff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,SEP,MTRR>
> > cpu0: features 387f9ff<PGE,MCA,CMOV,FGPAT,PSE36,PN,MMX>
> > cpu0: features 387f9ff<FXSR,SSE>
> > cpu0: serial number 0000-0683-0003-017B-4660-0B30
> > total memory = 127 MB
> > avail memory = 113 MB
> > using 1658 buffers containing 6632 KB of memory
> > BIOS32 rev. 0 found at 0xf04e0
> > mainbus0 (root)
> > pci0 at mainbus0 bus 0: configuration mode 1
> > pci0: i/o space, memory space enabled, rd/line, rd/mult, wr/inv ok
> > pchb0 at pci0 dev 0 function 0
> > pchb0: VIA Technologies VT82C691 (Apollo Pro) Host-PCI (rev. 0x42)
> > agp0 at pchb0: aperture at 0xe4000000, size 0x10000000
> > ppb0 at pci0 dev 1 function 0: VIA Technologies VT82C598 (Apollo MVP3) CPU-AGP Bridge (rev. 0x00)
> > pci1 at ppb0 bus 1
> > pci1: i/o space, memory space enabled
> > vga1 at pci1 dev 0 function 0: 3Dfx Interactive Voodoo3 (rev. 0x01)
> > wsdisplay0 at vga1 kbdmux 1
> > wsmux1: connecting to wsdisplay0
> > pcib0 at pci0 dev 7 function 0
> > pcib0: VIA Technologies VT82C596A (Apollo Pro) PCI-ISA Bridge (rev. 0x12)
> > pciide0 at pci0 dev 7 function 1: VIA Technologies VT82C596A (Apollo Pro) ATA66 controller
> > pciide0: bus-master DMA support present
> > pciide0: primary channel configured to compatibility mode
> > pciide0: primary channel ignored (disabled)
> > pciide0: secondary channel configured to compatibility mode
> > atapibus0 at pciide0 channel 1: 2 targets
> > cd0 at atapibus0 drive 0: <MATSHITA CR-571, , 1.0d> type 5 cdrom removable
> > cd0: 32-bit data port
> > cd0: drive supports PIO mode 3
> > pciide0: secondary channel interrupting at irq 15
> > cd0(pciide0:1:0): using PIO mode 3
> > uhci0 at pci0 dev 7 function 2: VIA Technologies VT83C572 USB Controller (rev. 0x08)
> > uhci0: interrupting at irq 14
> > usb0 at uhci0: USB revision 1.0
> > uhub0 at usb0
> > uhub0: VIA Technologie UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
> > uhub0: 2 ports with 2 removable, self powered
> > pchb1 at pci0 dev 7 function 3
> > pchb1: VIA Technologies product 0x3050 (rev. 0x20)
> > eap0 at pci0 dev 9 function 0: Ensoniq AudioPCI 97 ES1373B (rev. 0x06)
> > eap0: interrupting at irq 14
> > eap0: Crystal CS4297 codec; headphone, 18 bit DAC, 18 bit ADC, no 3D stereo
> > audio0 at eap0: full duplex, mmap, independent
> > midi0 at eap0: AudioPCI MIDI UART
> > pciide1 at pci0 dev 10 function 0: Promise Ultra100TX2/ATA Bus Master IDE Accelerator (rev. 0x01)
> > pciide1: bus-master DMA support present
> > pciide1: primary channel configured to native-PCI mode
> > pciide1: using irq 12 for native-PCI interrupt
> > wd0 at pciide1 channel 0 drive 0: <WDC WD200BB-00AUA1>
> > wd0: drive supports 16-sector PIO transfers, LBA addressing
> > wd0: 19092 MB, 16383 cyl, 16 head, 63 sec, 512 bytes/sect x 39102336 sectors
> > wd0: 32-bit data port
> > wd0: drive supports PIO mode 4, DMA mode 2, Ultra-DMA mode 5 (Ultra/100)
> > wd0(pciide1:0:0): using PIO mode 4, Ultra-DMA mode 5 (Ultra/100) (using DMA data transfers)
> > pciide1: secondary channel configured to native-PCI mode
> > wd1 at pciide1 channel 1 drive 0: <IC35L120AVVA07-0>
> > wd1: drive supports 16-sector PIO transfers, LBA addressing
> > wd1: 115 GB, 16383 cyl, 16 head, 63 sec, 512 bytes/sect x 241254720 sectors
> > wd1: 32-bit data port
> > wd1: drive supports PIO mode 4, DMA mode 2, Ultra-DMA mode 5 (Ultra/100)
> > wd1(pciide1:1:0): using PIO mode 4, Ultra-DMA mode 5 (Ultra/100) (using DMA data transfers)
> > vr0 at pci0 dev 11 function 0: VIA VT3043 (Rhine) 10/100 Ethernet
> > vr0: interrupting at irq 10
> > vr0: Ethernet address: 00:50:ba:aa:23:6f
> > ukphy0 at vr0 phy 8: Generic IEEE 802.3u media interface
> > ukphy0: Am79C873 10/100 media interface (OUI 0x000676, model 0x0000), rev. 0
> > ukphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
> > ex0 at pci0 dev 12 function 0: 3Com 3c905C-TX 10/100 Ethernet with mngmt (rev. 0x74)
> > ex0: interrupting at irq 11
> > ex0: MAC address 00:50:da:c6:d3:ef
> > bmtphy0 at ex0 phy 24: Broadcom 3c905C internal PHY, rev. 6
> > bmtphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
> > isa0 at pcib0
> > com0 at isa0 port 0x3f8-0x3ff irq 4: ns16550a, working fifo
> > com0: console
> > com1 at isa0 port 0x2f8-0x2ff irq 3: ns16550a, working fifo
> > pckbc0 at isa0 port 0x60-0x64
> > az0 at isa0 port 0x350: Aztech/PackardBell
> > radio0 at az0
> > pcppi0 at isa0 port 0x61
> > midi1 at pcppi0: PC speaker
> > spkr0 at pcppi0
> > sysbeep0 at pcppi0
> > isapnp0 at isa0 port 0x279: ISA Plug 'n Play device support
> > npx0 at isa0 port 0xf0-0xff: using exception 16
> > fdc0 at isa0 port 0x3f0-0x3f7 irq 6 drq 2
> > fd0 at fdc0 drive 0: 1.44MB, 80 cyl, 2 head, 18 sec
> > isapnp0: no ISA Plug 'n Play devices found
> > apm0 at mainbus0: Power Management spec V1.2
> > APM power mgmt engage (device 1): power management disabled (0x10f)
> > biomask f3e7 netmask ffe7 ttymask ffe7
> > Kernelized RAIDframe activated
> > IPsec: Initialized Security Association Processing.
> > boot device: wd0
> > root on wd0a dumps on wd0b
> > root file system type: ffs
> > IP Filter: v3.4.29 initialized.  Default = pass all, Logging = enabled
> > wsdisplay0: screen 1 added (80x25, vt100 emulation)
> > wsdisplay0: screen 2 added (80x25, vt100 emulation)
> > wsdisplay0: screen 3 added (80x25, vt100 emulation)
> > wsdisplay0: screen 4 added (80x25, vt100 emulation)
> 
> -- 
> 

--