Re: kern/53903: ixgbe cannot access some PCI registers

[Masanobu SAITOH <msaitoh%execsw.org@localhost>]

On 2019/01/23 23:05, maekawa%daemon-systems.org@localhost wrote:
> > Number:         53903
> > Category:       kern
> > Synopsis:       ixgbe cannot access some PCI registers
> > Confidential:   no
> > Severity:       serious
> > Priority:       medium
> > Responsible:    kern-bug-people
> > State:          open
> > Class:          sw-bug
> > Submitter-Id:   net
> > Arrival-Date:   Wed Jan 23 14:05:01 +0000 2019
> > Originator:     MAEKAWA Masahide
> > Release:        8.99.31
> > Organization:
> M-Systems, Inc.
> > Environment:
> NetBSD hpegen10.daemon-systems.org 8.99.31 NetBSD 8.99.31 (GENERIC) #0: Wed Jan 23 21:59:19 JST 2019  maekawa%kerberos.daemon-systems.org@localhost:/home/release/objs/sys/arch/amd64/compile/GENERIC amd64
> > Description:
> X540-T1 doesn't work on MicroServer Gen10 when "UEFI only", enabling "Above 4G Decoding" in setup menu.
> When disable "Above 4G Decoding", ixg(4) is attached normaly.
>
> Full dmesg is : https://www.daemon-systems.org/~maekawa/ixgbe/dmesg_64bit_regdump.log
>
> The ixg(4)-related part of boot message:
> ixg0 at pci3 dev 0 function 0: Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 4.0.1-k
> ixg0: clearing prefetchable bit
> ixgbe_init_shared_code
> ixgbe_set_mac_type
>
> ixgbe_set_mac_type found mac: 4, returns: 0
>
> ixgbe_init_ops_X540
> ixgbe_init_phy_ops_generic
> ixgbe_init_ops_generic
> ixgbe_get_pcie_msix_count_generic
> ixgbe_init_ops_X540 end
> ixg0: device X540
> ixgbe_set_lan_id_multi_port_pcie
> ixgbe_get_device_caps_generic
> ixgbe_read_eerd_X540
> ixgbe_acquire_swfw_sync_X540
> ixgbe_get_swfw_sync_semaphore
> ixg0: Software semaphore SMBI between device drivers not granted.
> Failed to get NVM access and register semaphore, returning IXGBE_ERR_SWFW_SYNC
>
> 00000: ffffffff
> 00004: ffffffff
> 00008: ffffffff
> 0000c: ffffffff
> 00010: ffffffff
> 00014: ffffffff
> 00018: ffffffff
> 0001c: ffffffff
> 00020: ffffffff
> 00024: ffffffff
> 00028: ffffffff
> 0002c: ffffffff
> 00030: ffffffff
> 00034: ffffffff
> 00038: ffffffff
> 0003c: ffffffff
> 10140: ffffffff
> 10160: ffffffff
> 10140: ffffffff
> 10160: ffffffff

The above diff was generated from the following diff:
Index: ixgbe.c
===================================================================
RCS file: /cvsroot/src/sys/dev/pci/ixgbe/ixgbe.c,v
retrieving revision 1.173
diff -u -p -r1.173 ixgbe.c
--- ixgbe.c	23 Jan 2019 09:47:52 -0000	1.173
+++ ixgbe.c	23 Jan 2019 11:44:40 -0000
@@ -876,6 +871,23 @@ ixgbe_attach(device_t parent, device_t d
 	} else
 		adapter->num_segs = IXGBE_82598_SCATTER;
 
+#if 1
+	{
+		uint32_t i;
+
+		for (i = 0; i < 64; i += 4)
+			printf("%05x: %08x\n", i, IXGBE_READ_REG(hw, i));
+		printf("%05x: %08x\n", IXGBE_SWSM_BY_MAC(hw),
+		    IXGBE_READ_REG(hw, IXGBE_SWSM_BY_MAC(hw)));
+		printf("%05x: %08x\n", IXGBE_SWFW_SYNC_BY_MAC(hw),
+		    IXGBE_READ_REG(hw, IXGBE_SWFW_SYNC_BY_MAC(hw)));
+		printf("%05x: %08x\n", IXGBE_SWSM_BY_MAC(hw),
+		    IXGBE_READ_REG(hw, IXGBE_SWSM_BY_MAC(hw)));
+		printf("%05x: %08x\n", IXGBE_SWFW_SYNC_BY_MAC(hw),
+		    IXGBE_READ_REG(hw, IXGBE_SWFW_SYNC_BY_MAC(hw)));
+	}
+#endif
+
 	/* Ensure SW/FW semaphore is free */
 	ixgbe_init_swfw_semaphore(hw);
 

> ixgbe_get_swfw_sync_semaphore
> ixg0: Software semaphore SMBI between device drivers not granted.
> ixgbe_release_swfw_sync_semaphore
> ixgbe_acquire_swfw_sync_X540
> ixgbe_get_swfw_sync_semaphore
> ixg0: Software semaphore SMBI between device drivers not granted.
> Failed to get NVM access and register semaphore, returning IXGBE_ERR_SWFW_SYNC
>
> ixgbe_release_swfw_sync_X540
> ixgbe_get_swfw_sync_semaphore
> ixg0: Software semaphore SMBI between device drivers not granted.
> ixgbe_release_swfw_sync_semaphore
> ixgbe_reset_hw_X540
> ixgbe_stop_adapter_generic
> ixgbe_disable_pcie_master
> GIO Master Disable bit didn't clear - requesting resets
>
> ixgbe_acquire_swfw_sync_X540
> ixgbe_get_swfw_sync_semaphore
> ixg0: Software semaphore SMBI between device drivers not granted.
> Failed to get NVM access and register semaphore, returning IXGBE_ERR_SWFW_SYNC
>
> ixg0: semaphore failed with -16ixg0: autoconfiguration error: Hardware initialization failed
> > How-To-Repeat:
> 0.) Install X540-T1 to MicroServer Gen10
> 1.) Restore default settings in setup menu.
> 2.) Enable "Above 4G Decoding" in setup menu.
> 3.) Boot
> > Fix:

The following directory has some logs:	

	https://www.daemon-systems.org/~maekawa/ixgbe/

pcidump_{32,64}bit.log are dump of all pci devices
(64 is from "Above 4G Decoding" enabled).

 Only X540 is mapped to over 4G. I suspect the reason is that
the BAR0 is marked prefetchable and it's used as a annotation
that the device requires large amounts of memory space. See also:

	http://mail-index.netbsd.org/tech-kern/2017/03/22/msg021678.html

I think this problem is not ixgbe specific.
Has our implementation of amd64 PCI bus access 32bit limitation?
(extent_create() or extent_alloc())

 I'm not familiar with x86 PCI memory map, so any test patch are
welcomed :)

--
-----------------------------------------------
                SAITOH Masanobu (msaitoh%execsw.org@localhost
                                 msaitoh%netbsd.org@localhost)