port-arm/49145: readonly mbuf is not writebacked by _bus_dmamap_sync_segment() nor sosend_loan()

To: port-arm-maintainer%netbsd.org@localhost,gnats-admin%netbsd.org@localhost,netbsd-bugs%netbsd.org@localhost
Subject: port-arm/49145: readonly mbuf is not writebacked by _bus_dmamap_sync_segment() nor sosend_loan()
From: ryo%nerv.org@localhost
Date: Sat, 23 Aug 2014 19:30:00 +0000 (UTC)
>Number:         49145
>Category:       port-arm
>Synopsis:       readonly mbuf is not writebacked by _bus_dmamap_sync_segment() 
>nor sosend_loan()
>Confidential:   no
>Severity:       critical
>Priority:       high
>Responsible:    port-arm-maintainer
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Sat Aug 23 19:30:00 +0000 2014
>Originator:     Ryo Shimizu
>Release:        NetBSD 7.99.1
>Organization:
>Environment:
System: NetBSD nitro 7.99.1 NetBSD 7.99.1 (NITROGEN6X) #938: Sat Aug 23 
04:24:30 JST 2014  
ryo@phenomena:/src/cvs/NetBSD/sys/arch/evbarm/compile/NITROGEN6X evbarm
Architecture: earm
Machine: evbarm
>Description:
sys/arch/arm/arm32/bus_dma.c:_bus_sync_segment() doesn't writeback
when argument readonly_p is true. therefore readonly mbuf (M_EXT_ROMAP)
is not writeback.

Readonly mbufs are made by sys/kern/uipc_socket.c:sosend_loan().
but even at this point, cachelines are not writebacked.

I found this problem when writing ethernet driver for iMX6.
Ethernet driver cannot writeback mbufs that contains M_EXT_ROMAP.

I think because of relatively large *1Mbyte* L2 cache, this problem
has occurred.

>How-To-Repeat:
try ping -s10000 172.16.0.12.
payload must all zero, but strange data has appeared at 0x02f4 in this packet.

03:24:02.003801 00:19:b8:00:f2:f8 > 00:0e:0c:83:5c:f4, ethertype IPv4 (0x0800), 
length 1514: (tos 0x0, ttl 255, id 5, offset 5920, flags [+], proto ICMP (1), 
length 1500)
    172.16.0.23 > 172.16.0.12: ip-proto-1
        0x0000:  4500 05dc 0005 22e4 ff01 3af4 ac10 0017  E....."...:.....
        0x0010:  ac10 000c 0000 0000 0000 0000 0000 0000  ................
        0x0020:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0030:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0040:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0050:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0060:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0070:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0080:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0090:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x00a0:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x00b0:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x00c0:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x00d0:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x00e0:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x00f0:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0100:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0110:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0120:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0130:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0140:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0150:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0160:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0170:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0180:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0190:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x01a0:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x01b0:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x01c0:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x01d0:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x01e0:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x01f0:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0200:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0210:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0220:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0230:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0240:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0250:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0260:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0270:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0280:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0290:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x02a0:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x02b0:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x02c0:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x02d0:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x02e0:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x02f0:  0000 0000 5f63 6f6d 7061 743a 0964 6e73  ...._compat:.dns      
# must be zero
        0x0300:  2c20 6e69 730a 2320 7368 656c 6c73 3a09  ,.nis.#.shells:.      
#
        0x0310:  0964 6e73 0000 0000 0000 0000 0000 0000  .dns............      
#
        0x0320:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0330:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0340:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0350:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0360:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0370:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0380:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0390:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x03a0:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x03b0:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x03c0:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x03d0:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x03e0:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x03f0:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0400:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0410:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0420:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0430:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0440:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0450:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0460:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0470:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0480:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0490:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x04a0:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x04b0:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x04c0:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x04d0:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x04e0:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x04f0:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0500:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0510:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0520:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0530:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0540:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0550:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0560:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0570:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0580:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0590:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x05a0:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x05b0:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x05c0:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x05d0:  0000 0000 0000 0000 0000 0000            ............


#
# debug log by imx6 ethernet driver
# (https://github.com/ryo/netbsd-src/blob/ryo-imx6/sys/arch/arm/imx/if_enet.c)
#
Aug 20 09:51:13 nitro /netbsd: TX START
Aug 20 09:51:13 nitro /netbsd: mbuf paddr=4fef3538 size=14
Aug 20 09:51:13 nitro /netbsd: mbuf paddr=4fef3648 size=20
Aug 20 09:51:13 nitro /netbsd: mbuf paddr=4e427f60 size=160 (M_READONLY)
Aug 20 09:51:13 nitro /netbsd: mbuf paddr=4e3e6000 size=1320 (M_READONLY)
Aug 20 09:51:13 nitro /netbsd: DONE

#
# debug log by sys/arch/arm/arm32/bus_dma.c:_bus_sync_segment() and
# sys/arch/arm/cortex/pl310.c (arml2cc_sdcache_*_range() printf patched)
#
Aug 20 09:51:13 nitro /netbsd: sync_segment: va=0xbfef3538 pa=0x4fef3538 
len=0xe ops=0x5 ro=0
Aug 20 09:51:13 nitro /netbsd: arml2cc_sdcache_wbinv_range(pa=0x4fef3538, 
len=14)
Aug 20 09:51:13 nitro /netbsd: sync_segment: va=0xbfef3648 pa=0x4fef3648 
len=0x14 ops=0x5 ro=0
Aug 20 09:51:13 nitro /netbsd: arml2cc_sdcache_wbinv_range(pa=0x4fef3648, 
len=20)
Aug 20 09:51:13 nitro /netbsd: sync_segment: va=0xca259f60 pa=0x4e427f60 
len=0xa0 ops=0x5 ro=1
Aug 20 09:51:13 nitro /netbsd: arml2cc_sdcache_inv_range(pa=0x4e427f60, len=160)
Aug 20 09:51:13 nitro /netbsd: sync_segment: va=0xca25a000 pa=0x4e3e6000 
len=0x528 ops=0x5 ro=1
Aug 20 09:51:13 nitro /netbsd: arml2cc_sdcache_inv_range(pa=0x4e3e6000, 
len=1312)
Aug 20 09:51:13 nitro /netbsd: arml2cc_sdcache_wbinv_range(pa=0x4e3e6520, 
len=32)


>Fix:
for the nonce, below quick-fix.
but I don't know accurately.

cvs -q diff -aup bus_dma.c
Index: bus_dma.c
===================================================================
RCS file: /cvsroot/src/sys/arch/arm/arm32/bus_dma.c,v
retrieving revision 1.86
diff -a -u -p -r1.86 bus_dma.c
--- bus_dma.c   10 Apr 2014 02:44:05 -0000      1.86
+++ bus_dma.c   23 Aug 2014 17:58:18 -0000
@@ -775,13 +775,10 @@ _bus_dmamap_sync_segment(vaddr_t va, pad
 
        switch (ops) {
        case BUS_DMASYNC_PREREAD|BUS_DMASYNC_PREWRITE:
-               if (!readonly_p) {
-                       STAT_INCR(sync_prereadwrite);
-                       cpu_dcache_wbinv_range(va, len);
-                       cpu_sdcache_wbinv_range(va, pa, len);
-                       break;
-               }
-               /* FALLTHROUGH */
+               STAT_INCR(sync_prereadwrite);
+               cpu_dcache_wbinv_range(va, len);
+               cpu_sdcache_wbinv_range(va, pa, len);
+               break;
 
        case BUS_DMASYNC_PREREAD: {
                const size_t line_size = arm_dcache_align;
Follow-Ups:
- Re: port-arm/49145: readonly mbuf is not writebacked by _bus_dmamap_sync_segment() nor sosend_loan()
  - From: Matt Thomas
Prev by Date: bin/49144: udf_times test takes 951 times longer on i386 than amd64
Next by Date: Re: xsrc/48317: openchrome driver crashes with undefined PLT symbol
Previous by Thread: bin/49144: udf_times test takes 951 times longer on i386 than amd64
Next by Thread: Re: port-arm/49145: readonly mbuf is not writebacked by _bus_dmamap_sync_segment() nor sosend_loan()
Indexes:
Home | Main Index | Thread Index | Old Index