Re: Network very very slow... was iSCSI and jumbo frames

[BERTRAND Joël <joel.bertrand%systella.fr@localhost>]

Michael van Elst a écrit :
> On Mon, Feb 08, 2021 at 11:40:43AM +0100, BERTRAND Joël wrote:
>> Michael van Elst a écrit :
>>> For the "dd"-test of iSCSI, use the raw devices (not block devices,
>>> not files in a filesystem) and a block size of 1MB. Only if that
>>> works ok, it makes sense to look at filesystem I/O, i.e. your /opt
>>> filesystem.
>>
>> 	If I do that in write mode, I suppose I will kill filesystem on
>> /dev/rdk5, isn't it ?
> 
> Definitely.
> 
> You need a test volume for a write test.

	I haven't spare disks enough. Thus, I have created a disk image on a
Linux laptop (with an old AMD E-450), installed istgt and exported this
disk.

legendre# dd if=/dev/rsd1a of=/dev/null iflag=direct bs=1m count=10000
10000+0 records in
10000+0 records out
10485760000 bytes transferred in 156.383 secs (67051789 bytes/sec)

	Throughput is more or less 500 Mbps on network, but there are some
limiting factors :
- CPU (but istgt only takes 60% of one thread) ;
- hard disk ;
- Ethernet adapter (Qualcomm Atheros)...

	But 67 MBps is greater than 10 MBps I obtain with NAS.

	Other test, when I try to write on iSCSI target :
legendre# dd if=/dev/zero of=/dev/rsd1d iflag=direct oflag=direct bs=1m
count=10000
10000+0 records in
10000+0 records out
10485760000 bytes transferred in 282.990 secs (37053464 bytes/sec)

	Throughput is only about 37 MBps, but CPU cannot offer more power...

	I have captured tcpdump output on both adapters (I post only one second) :

wm0 (to euclide) :
14:06:09.014969 IP legendre.systella.fr.65508 >
euclide.systella.fr.iscsi-target: Flags [.], seq 107229888:107238836,
ack 78529, win 6528, options [nop,nop,TS val 642769 ecr 43049463],
length 8948
14:06:09.014981 IP legendre.systella.fr.65508 >
euclide.systella.fr.iscsi-target: Flags [.], seq 107238836:107247784,
ack 78529, win 6528, options [nop,nop,TS val 642769 ecr 43049463],
length 8948
14:06:09.014993 IP legendre.systella.fr.65508 >
euclide.systella.fr.iscsi-target: Flags [.], seq 107247784:107256732,
ack 78529, win 6528, options [nop,nop,TS val 642769 ecr 43049463],
length 8948
14:06:09.015005 IP legendre.systella.fr.65508 >
euclide.systella.fr.iscsi-target: Flags [.], seq 107256732:107265680,
ack 78529, win 6528, options [nop,nop,TS val 642769 ecr 43049463],
length 8948
14:06:09.015057 IP legendre.systella.fr.65508 >
euclide.systella.fr.iscsi-target: Flags [.], seq 107265680:107274628,
ack 78529, win 6528, options [nop,nop,TS val 642769 ecr 43049463],
length 8948
14:06:09.015083 IP legendre.systella.fr.65508 >
euclide.systella.fr.iscsi-target: Flags [.], seq 107274628:107283576,
ack 78529, win 6528, options [nop,nop,TS val 642769 ecr 43049463],
length 8948
14:06:09.015096 IP legendre.systella.fr.65508 >
euclide.systella.fr.iscsi-target: Flags [.], seq 107283576:107292524,
ack 78529, win 6528, options [nop,nop,TS val 642769 ecr 43049463],
length 8948
14:06:09.015104 IP legendre.systella.fr.65508 >
euclide.systella.fr.iscsi-target: Flags [P.], seq 107292524:107295472,
ack 78529, win 6528, options [nop,nop,TS val 642769 ecr 43049463],
length 2948
14:06:09.015122 IP legendre.systella.fr.65508 >
euclide.systella.fr.iscsi-target: Flags [.], seq 107295472:107304420,
ack 78529, win 6528, options [nop,nop,TS val 642769 ecr 43049463],
length 8948
14:06:09.015136 IP legendre.systella.fr.65508 >
euclide.systella.fr.iscsi-target: Flags [.], seq 107304420:107313368,
ack 78529, win 6528, options [nop,nop,TS val 642769 ecr 43049463],
length 8948
legendre.systella.fr.65508: Flags [.], ack 107396848, win 24576, options
[nop,nop,TS val 43049472 ecr 642769], length 0
14:06:09.016495 IP legendre.systella.fr.65508 >
euclide.systella.fr.iscsi-target: Flags [.], seq 107568880:107577828,
ack 78625, win 6528, options [nop,nop,TS val 642769 ecr 43049472],
length 8948
...
14:06:09.023177 IP legendre.systella.fr.65508 >
euclide.systella.fr.iscsi-target: Flags [.], ack 79153, win 6522,
options [nop,nop,TS val 642769 ecr 43049472], length 0
14:06:09.023375 IP euclide.systella.fr.iscsi-target >
legendre.systella.fr.65508: Flags [P.], seq 79153:79201, ack 108213648,
win 24576, options [nop,nop,TS val 43049472 ecr 642769], length 48
14:06:09.023670 IP euclide.systella.fr.iscsi-target >
legendre.systella.fr.65508: Flags [P.], seq 79201:79249, ack 108213648,
win 24576, options [nop,nop,TS val 43049472 ecr 642769], length 48
14:06:09.023677 IP legendre.systella.fr.65508 >
euclide.systella.fr.iscsi-target: Flags [.], ack 79249, win 6522,
options [nop,nop,TS val 642769 ecr 43049472], length 0
14:06:09.115011 IP legendre.systella.fr.65508 >
euclide.systella.fr.iscsi-target: Flags [.], seq 108213648:108222596,
ack 79249, win 6528, options [nop,nop,TS val 642769 ecr 43049472],
length 8948

	Often, network stops during 90 or 100 ms (see the last lines). On agr0
(to riemann), I don't see these 'stops'.

	On euclide, there is no istgt, but a iscsid (iscsid version 2.0-871)
and I don't understand how this daemon is started as
/etc/config/iscsi/sbin/iscsid.conf doesn't exists !
[/] # /sbin/iscsid -v
iscsid version 2.0-871
[/] # ps auwx | grep iscsi
 6110 admin           SW  [iscsi_np]
 6134 admin      3144 S   /sbin/iscsi_logd
 6712 admin           SW< [iscsi_eh]
 6753 admin      1092 S   /sbin/iscsid
--config=/etc/config/iscsi/sbin/iscsid.conf
--initiatorname=/etc/iscsi/initiatorname.iscsi
 6754 admin      1860 S < /sbin/iscsid
--config=/etc/config/iscsi/sbin/iscsid.conf
--initiatorname=/etc/iscsi/initiatorname.iscsi
19978 admin       416 S   grep iscsi
25674 admin           SW  [iscsi_ttx]
25675 admin           SW  [iscsi_trx]
30073 admin           SW  [iscsi_np]
30078 admin           SW  [iscsi_np]
[/] # cd /etc/config/iscsi
-sh: cd: /etc/config/iscsi: No such file or directory
[/] # ls /etc/config/iscsi/sbin/iscsid.conf
ls: /etc/config/iscsi/sbin/iscsid.conf: No such file or directory
[/] #

	OK, iscsid is started without config file (I don't understand why it
doesn't stop immediatly), but it runs fine with Linux initiator...

	JKB