Re: SCP file transfer speed

[Stephan <stephanwib%googlemail.com@localhost>]

A local copy is quite fast on by box.

1) With pkgsrc openssh:

/tmp> scp NetBSD-trunk.zip localhost:/var/tmp
NetBSD-trunk.zip
                                          100%  450MB  50.0MB/s
48.5MB/s   00:09

2) With NetBSD builtin ssh:

NetBSD-trunk.zip
                                          100%  450MB  32.2MB/s
34.9MB/s   00:14

2015-03-20 12:01 GMT+01:00 Dan LaBell <dan4l-nospam%verizon.net@localhost>:
>
> On Mar 19, 2015, at 4:57 AM, Stephan wrote:
>
>> Hi
>>
>> When I copy large files through scp to a NetBSD box, I get transfer
>> speeds of only 7 MB/s on a 100 MBit connection. This should be around
>> 11 MB/s. I´ve seen this on different x86/amd64 hardware with NetBSD 5,
>> 6 and 7-BETA. The NICs are largely wm, fxp and bnx.
>>
>> Can others reproduce this?
>
>
> I've noticed it can be slower.   Do you ever shell into the box, and push
> a file back to you -- and notice it's faster? Or, maybe shell into to the
> box
> cd to to where you want to put, and use sftp and get , and pull it down, and
> find it's faster?   I believe I noticed the a difference between my netbsd
> box,
> and my mac's,  basically, by mistake, since I was just trying to avoid
> typos.
>
> And, kind of forgot about it, until your post.,  Lately, I'm using my netbsd
> box
> to recover the root file system of a toaster, it's  just over a gigabyte,
> linux ext3,
> and transferring the image seemed slower, than I expected.  After that,
> I plan to make a ccd , and send  141 gib image over,  a 100 MBIT connection,
> so, I decided to some tests of my own.
> I made a 50 megabyte file in /tmp and tried scp:
>
> [root@qix:/tmp]
> #scp root@qix:/tmp/test.scp /va r/root/test.scp
> ...
> test.scp                             92%   46MB   1.6MB/s   1.5MB/s   00:02
> ETA
> test.scp                             95%   48MB   1.6MB/s   1.5MB/s   00:01
> ETA
> test.scp                             99%   50MB   1.6MB/s   1.6MB/s   00:00
> ETA
> test.scp                            100%   50MB   1.6MB/s 452.0KB/s   00:31
>
> And, I basically see 31/32 seconds on every test.
> I try
> #ssh root@qix 'dd if=/tmp/test.scp bs=32k ' | dd of=/var/root/test.scp
> bs=32k progress=40
>
> and try:
> #ssh root@qix 'dd if=/tmp/test.scp bs=64k ' | dd of=/var/root/test.scp
> bs=16k progress=160
>
> ..........................................800+0 records in
> 800+0 records out
> 52428800 bytes transferred in 32.253 secs (1625548 bytes/sec)
>
> 1831+4925 records in
> 1831+4925 records out
> 52428800 bytes transferred in 33.172 secs (1580513 bytes/sec)
> [root@qix:/tmp]
> 800+0 means no issues, but I interpret  1831+4925 for  the receiving dd or
> outer dd
> to mean 1831 times 16 kb were ready for dd's  READ(2) , so dd could  fill
> it's buffer with 1 call,
> but for 4925 records, it took more than 1 call to get 16k.
>
> With 64k, and 64k and progress=60 or (progress=50, or 80? ) it came out
> 0+800, BUT
> I could see it start strong with maybe 8 dots, then start to 'strobe'  or ,
> startup, and slow,
> down , 4 dots then stall/pause, etc.   Basically, sputtering.
>
> I thought what if its having trouble because the test file is a stream of
> zeroes . I create
> a 52mb file like this (twice) :
> cat /data2/4targ/105mb.dmg.rz >> something
> and try again, and twiddle with bs a few times.
> #ssh root@qix 'dd if=/tmp/something bs=64k ' | dd of=/var/root/something
> bs=4k
> 835+1 records in
> 835+1 records out
> 54784466 bytes transferred in 32.372 secs (1692341 bytes/sec)
>
> 13375+1 records in
> 13375+1 records out
> 54784466 bytes transferred in 33.297 secs (1645327 bytes/sec)
> Blocksize doesn't seem to help,
> I kind of stupidly try this:
> time ssh root@qix 'gzip -1c something' | gzcat > something
> CTRL-T a few times...
> load: 1.68  cmd: ssh 1571 [select] 8.03u 1.72s 13% 4880k
> load: 1.78  cmd: ssh 1571 [runnable] 8.53u 1.90s 13% 4880k
> load: 1.88  cmd: ssh 1571 [runnable] 8.80u 1.96s 13% 4880k
>   1m34.00s real     0m15.52s user     0m7.62s system
> 3 times as long, but it's already compressed data so... kinda of stupid.
>
> Periodically, in my tests, I do a cmp -l  against the 2 files, it's always
> good..
>
> Finally, I try netcat.
> In another vc, I do: nc -l -p 1234 >| something
> then:
> #time nc -w 3 qix 1234 < something
>     0m10.49s real     0m0.02s user     0m1.50s system
>
> 10 - 11 seconds,  so I stop peering at sysctl. net.inet
>
>