Errors initializing a PLX 8717 NTB on SCALE 25.04.2.1

TrueNAS General
,
1

Hey Community & TrueNAS Kernel Devs,

I’ve recently acquired a Supermicro SuperStorageBridgeBay system (6038R-DE2CR16L) and I’m curious to play around a little with the PCIe non-transparent bridge (NTB). The system has a PLX8717 switch, but I’m getting an initialization error on boot from the ntb_hw_plx driver. Reviewing the code, I gather this chip should be supported - the comments + code show what looks to be a straight port of the FreeBSD driver, and it does support the PLX8717, and being new to PCIe NTB hardware, I’m having a hard time figuring out what to try next.

The error messages:

[ 24.240037 ] ntb_hw_plx 0000:08:00.0: 256 A-LUT entries
[ 24.240048 ] ntb_hw_plx 0000:08:00.0: NTB-to-NTB (back-to-back) mode
[ 24.240051 ] ntb_hw_plx 0000:08:00.0: Can't map B2B BAR.
[ 24.240087 ] ntb_hw_plx 0000:08:00.0: probe with driver ntb_hw_plx failed with error -12

relevant lspci -vvv:

06:00.0 PCI bridge: PLX Technology, Inc. PEX 8717 16-lane, 8-Port PCI Express Gen 3 (8.0 GT/s) Switch with DMA (rev ca) (prog-if 00 [Normal decode])
        Subsystem: PLX Technology, Inc. PEX 8717 16-lane, 8-Port PCI Express Gen 3 (8.0 GT/s) Switch with DMA
        Physical Slot: 14
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 32 bytes
        Interrupt: pin A routed to IRQ 30
        NUMA node: 0
        IOMMU group: 77
        Region 0: Memory at c7400000 (32-bit, non-prefetchable) [size=256K]
        Bus: primary=06, secondary=07, subordinate=08, sec-latency=0
        I/O behind bridge: 0000f000-00000fff [disabled] [32-bit]
        Memory behind bridge: c7300000-c73fffff [size=1M] [32-bit]
        Prefetchable memory behind bridge: 383fc0000000-383fdfffffff [size=512M] [32-bit]
        Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
        BridgeCtl: Parity+ SERR+ NoISA- VGA- VGA16+ MAbort- >Reset- FastB2B-
                PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
        Capabilities: [40] Power Management version 3
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [48] MSI: Enable- Count=1/8 Maskable+ 64bit+
                Address: 0000000000000000  Data: 0000
                Masking: 00000000  Pending: 00000000
        Capabilities: [68] Express (v2) Upstream Port, MSI 00
                DevCap: MaxPayload 2048 bytes, PhantFunc 0
                        ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ SlotPowerLimit 25W
                DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
                        RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+
                        MaxPayload 256 bytes, MaxReadReq 128 bytes
                DevSta: CorrErr+ NonFatalErr- FatalErr- UnsupReq+ AuxPwr- TransPend-
                LnkCap: Port #0, Speed 8GT/s, Width x8, ASPM L1, Exit Latency L1 <4us
                        ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
                LnkCtl: ASPM L1 Enabled; Disabled- CommClk-
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 8GT/s, Width x8
                        TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Not Supported, TimeoutDis- NROPrPrP- LTR+
                         10BitTagComp- 10BitTagReq- OBFF Via message, ExtFmt- EETLPPrefix-
                         EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
                         FRS-
                         AtomicOpsCap: Routing+ 32bit- 64bit- 128bitCAS-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- 10BitTagReq- OBFF Disabled,
                         AtomicOpsCtl: EgressBlck-
                LnkCap2: Supported Link Speeds: 2.5-8GT/s, Crosslink+ Retimer- 2Retimers- DRS-
                LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
                LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+ EqualizationPhase1+
                         EqualizationPhase2+ EqualizationPhase3+ LinkEqualizationRequest-
                         Retimer- 2Retimers- CrosslinkRes: unsupported
        Capabilities: [a4] Subsystem: PLX Technology, Inc. PEX 8717 16-lane, 8-Port PCI Express Gen 3 (8.0 GT/s) Switch with DMA
        Capabilities: [100 v1] Device Serial Number ca-87-00-10-b5-df-0e-00
        Capabilities: [fb4 v1] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
                AERCap: First Error Pointer: 1f, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
                        MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
                HeaderLog: 00000000 00000000 00000000 00000000
        Capabilities: [138 v1] Power Budgeting <?>
        Capabilities: [10c v1] Secondary PCI Express
                LnkCtl3: LnkEquIntrruptEn- PerformEqu-
                LaneErrStat: 0
        Capabilities: [148 v1] Virtual Channel
                Caps:   LPEVC=0 RefClk=100ns PATEntryBits=8
                Arb:    Fixed- WRR32- WRR64- WRR128-
                Ctrl:   ArbSelect=Fixed
                Status: InProgress-
                VC0:    Caps:   PATOffset=03 MaxTimeSlots=1 RejSnoopTrans-
                        Arb:    Fixed- WRR32- WRR64+ WRR128- TWRR128- WRR256-
                        Ctrl:   Enable+ ID=0 ArbSelect=WRR64 TC/VC=01
                        Status: NegoPending- InProgress-
                        Port Arbitration Table <?>
        Capabilities: [e00 v1] Multicast
                McastCap: MaxGroups 64, ECRCRegen+
                McastCtl: NumGroups 1, Enable-
                McastBAR: IndexPos 0, BaseAddr 0000000000000000
                McastReceiveVec:      0000000000000000
                McastBlockAllVec:     0000000000000000
                McastBlockUntransVec: 0000000000000000
                McastOverlayBAR: OverlaySize 0 (disabled), BaseAddr 0000000000000000
        Capabilities: [b00 v1] Latency Tolerance Reporting
                Max snoop latency: 0ns
                Max no snoop latency: 0ns
        Capabilities: [b70 v1] Vendor Specific Information: ID=0001 Rev=0 Len=010 <?>
        Kernel driver in use: pcieport

06:00.1 System peripheral: PLX Technology, Inc. PEX PCI Express Switch DMA interface (rev ca)
        Subsystem: PLX Technology, Inc. PEX PCI Express Switch DMA interface
        Physical Slot: 14
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 32 bytes
        Interrupt: pin B routed to IRQ 10
        NUMA node: 0
        IOMMU group: 77
        Region 0: Memory at c7446000 (32-bit, non-prefetchable) [size=8K]
        Capabilities: [40] Power Management version 3
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [48] MSI: Enable- Count=1/8 Maskable+ 64bit+
                Address: 0000000000000000  Data: 0000
                Masking: 00000000  Pending: 00000000
        Capabilities: [68] Express (v2) Endpoint, MSI 00
                DevCap: MaxPayload 2048 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
                        ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0W
                DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
                        RlxdOrd- ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
                        MaxPayload 256 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr+ NonFatalErr- FatalErr- UnsupReq+ AuxPwr- TransPend-
                LnkCap: Port #0, Speed 8GT/s, Width x8, ASPM L1, Exit Latency L1 <4us
                        ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
                LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk-
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 8GT/s, Width x8
                        TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ NROPrPrP- LTR+
                         10BitTagComp- 10BitTagReq- OBFF Via message, ExtFmt- EETLPPrefix-
                         EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
                         FRS- TPHComp- ExtTPHComp-
                         AtomicOpsCap: 32bit- 64bit- 128bitCAS-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- 10BitTagReq- OBFF Disabled,
                         AtomicOpsCtl: ReqEn-
                LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-
                         EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
                         Retimer- 2Retimers- CrosslinkRes: unsupported
        Capabilities: [100 v1] Device Serial Number ca-87-00-10-b5-df-0e-00
        Capabilities: [fb4 v1] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
                AERCap: First Error Pointer: 1f, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
                        MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
                HeaderLog: 00000000 00000000 00000000 00000000
        Capabilities: [1f0 v1] Vendor Specific Information: ID=0010 Rev=0 Len=0c4 <?>
        Capabilities: [b70 v1] Vendor Specific Information: ID=0001 Rev=0 Len=010 <?>

06:00.2 System peripheral: PLX Technology, Inc. PEX PCI Express Switch DMA interface (rev ca)
        Subsystem: PLX Technology, Inc. PEX PCI Express Switch DMA interface
        Physical Slot: 14
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 32 bytes
        Interrupt: pin B routed to IRQ 10
        NUMA node: 0
        IOMMU group: 77
        Region 0: Memory at c7444000 (32-bit, non-prefetchable) [size=8K]
        Capabilities: [40] Power Management version 3
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [48] MSI: Enable- Count=1/8 Maskable+ 64bit+
                Address: 0000000000000000  Data: 0000
                Masking: 00000000  Pending: 00000000
        Capabilities: [68] Express (v2) Endpoint, MSI 00
                DevCap: MaxPayload 2048 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
                        ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0W
                DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
                        RlxdOrd- ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
                        MaxPayload 256 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr+ NonFatalErr- FatalErr- UnsupReq+ AuxPwr- TransPend-
                LnkCap: Port #0, Speed 8GT/s, Width x8, ASPM L1, Exit Latency L1 <4us
                        ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
                LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk-
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 8GT/s, Width x8
                        TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ NROPrPrP- LTR+
                         10BitTagComp- 10BitTagReq- OBFF Via message, ExtFmt- EETLPPrefix-
                         EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
                         FRS- TPHComp- ExtTPHComp-
                         AtomicOpsCap: 32bit- 64bit- 128bitCAS-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- 10BitTagReq- OBFF Disabled,
                         AtomicOpsCtl: ReqEn-
                LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-
                         EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
                         Retimer- 2Retimers- CrosslinkRes: unsupported
        Capabilities: [100 v1] Device Serial Number ca-87-00-10-b5-df-0e-00
        Capabilities: [fb4 v1] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
                AERCap: First Error Pointer: 1f, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
                        MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
                HeaderLog: 00000000 00000000 00000000 00000000
        Capabilities: [1f0 v1] Vendor Specific Information: ID=0010 Rev=0 Len=0c4 <?>
        Capabilities: [b70 v1] Vendor Specific Information: ID=0001 Rev=0 Len=010 <?>

06:00.3 System peripheral: PLX Technology, Inc. PEX PCI Express Switch DMA interface (rev ca)
        Subsystem: PLX Technology, Inc. PEX PCI Express Switch DMA interface
        Physical Slot: 14
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 32 bytes
        Interrupt: pin B routed to IRQ 10
        NUMA node: 0
        IOMMU group: 77
        Region 0: Memory at c7442000 (32-bit, non-prefetchable) [size=8K]
        Capabilities: [40] Power Management version 3
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [48] MSI: Enable- Count=1/8 Maskable+ 64bit+
                Address: 0000000000000000  Data: 0000
                Masking: 00000000  Pending: 00000000
        Capabilities: [68] Express (v2) Endpoint, MSI 00
                DevCap: MaxPayload 2048 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
                        ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0W
                DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
                        RlxdOrd- ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
                        MaxPayload 256 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr+ NonFatalErr- FatalErr- UnsupReq+ AuxPwr- TransPend-
                LnkCap: Port #0, Speed 8GT/s, Width x8, ASPM L1, Exit Latency L1 <4us
                        ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
                LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk-
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 8GT/s, Width x8
                        TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ NROPrPrP- LTR+
                         10BitTagComp- 10BitTagReq- OBFF Via message, ExtFmt- EETLPPrefix-
                         EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
                         FRS- TPHComp- ExtTPHComp-
                         AtomicOpsCap: 32bit- 64bit- 128bitCAS-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- 10BitTagReq- OBFF Disabled,
                         AtomicOpsCtl: ReqEn-
                LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-
                         EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
                         Retimer- 2Retimers- CrosslinkRes: unsupported
        Capabilities: [100 v1] Device Serial Number ca-87-00-10-b5-df-0e-00
        Capabilities: [fb4 v1] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
                AERCap: First Error Pointer: 1f, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
                        MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
                HeaderLog: 00000000 00000000 00000000 00000000
        Capabilities: [1f0 v1] Vendor Specific Information: ID=0010 Rev=0 Len=0c4 <?>
        Capabilities: [b70 v1] Vendor Specific Information: ID=0001 Rev=0 Len=010 <?>

06:00.4 System peripheral: PLX Technology, Inc. PEX PCI Express Switch DMA interface (rev ca)
        Subsystem: PLX Technology, Inc. PEX PCI Express Switch DMA interface
        Physical Slot: 14
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 32 bytes
        Interrupt: pin B routed to IRQ 10
        NUMA node: 0
        IOMMU group: 77
        Region 0: Memory at c7440000 (32-bit, non-prefetchable) [size=8K]
        Capabilities: [40] Power Management version 3
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [48] MSI: Enable- Count=1/8 Maskable+ 64bit+
                Address: 0000000000000000  Data: 0000
                Masking: 00000000  Pending: 00000000
        Capabilities: [68] Express (v2) Endpoint, MSI 00
                DevCap: MaxPayload 2048 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
                        ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0W
                DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
                        RlxdOrd- ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
                        MaxPayload 256 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr+ NonFatalErr- FatalErr- UnsupReq+ AuxPwr- TransPend-
                LnkCap: Port #0, Speed 8GT/s, Width x8, ASPM L1, Exit Latency L1 <4us
                        ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
                LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk-
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 8GT/s, Width x8
                        TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ NROPrPrP- LTR+
                         10BitTagComp- 10BitTagReq- OBFF Via message, ExtFmt- EETLPPrefix-
                         EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
                         FRS- TPHComp- ExtTPHComp-
                         AtomicOpsCap: 32bit- 64bit- 128bitCAS-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- 10BitTagReq- OBFF Disabled,
                         AtomicOpsCtl: ReqEn-
                LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-
                         EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
                         Retimer- 2Retimers- CrosslinkRes: unsupported
        Capabilities: [100 v1] Device Serial Number ca-87-00-10-b5-df-0e-00
        Capabilities: [fb4 v1] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
                AERCap: First Error Pointer: 1f, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
                        MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
                HeaderLog: 00000000 00000000 00000000 00000000
        Capabilities: [1f0 v1] Vendor Specific Information: ID=0010 Rev=0 Len=0c4 <?>
        Capabilities: [b70 v1] Vendor Specific Information: ID=0001 Rev=0 Len=010 <?>

07:01.0 PCI bridge: PLX Technology, Inc. PEX 8717 16-lane, 8-Port PCI Express Gen 3 (8.0 GT/s) Switch with DMA (rev ca) (prog-if 00 [Normal decode])
        Subsystem: PLX Technology, Inc. PEX 8717 16-lane, 8-Port PCI Express Gen 3 (8.0 GT/s) Switch with DMA
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 32 bytes
        Interrupt: pin A routed to IRQ 36
        NUMA node: 0
        IOMMU group: 77
        Bus: primary=07, secondary=08, subordinate=08, sec-latency=0
        I/O behind bridge: 0000f000-00000fff [disabled] [32-bit]
        Memory behind bridge: c7300000-c73fffff [size=1M] [32-bit]
        Prefetchable memory behind bridge: 383fc0000000-383fdfffffff [size=512M] [32-bit]
        Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
        BridgeCtl: Parity+ SERR+ NoISA- VGA- VGA16+ MAbort- >Reset- FastB2B-
                PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
        Capabilities: [40] Power Management version 3
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [48] MSI: Enable+ Count=1/8 Maskable+ 64bit+
                Address: 00000000fee00398  Data: 0000
                Masking: 000000ff  Pending: 00000000
        Capabilities: [68] Express (v2) Downstream Port (Slot-), MSI 00
                DevCap: MaxPayload 2048 bytes, PhantFunc 0
                        ExtTag- RBE+
                DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
                        RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+
                        MaxPayload 256 bytes, MaxReadReq 128 bytes
                DevSta: CorrErr+ NonFatalErr- FatalErr- UnsupReq+ AuxPwr- TransPend-
                LnkCap: Port #1, Speed 8GT/s, Width x8, ASPM L1, Exit Latency L1 <4us
                        ClockPM- Surprise- LLActRep- BwNot+ ASPMOptComp+
                LnkCtl: ASPM Disabled; Disabled- CommClk-
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 2.5GT/s, Width x0
                        TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Not Supported, TimeoutDis- NROPrPrP- LTR+
                         10BitTagComp- 10BitTagReq- OBFF Via message, ExtFmt- EETLPPrefix-
                         EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
                         FRS- ARIFwd+
                         AtomicOpsCap: Routing+
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- 10BitTagReq- OBFF Disabled, ARIFwd-
                         AtomicOpsCtl: EgressBlck-
                LnkCap2: Supported Link Speeds: 2.5-8GT/s, Crosslink+ Retimer- 2Retimers- DRS-
                LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
                LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete- EqualizationPhase1-
                         EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
                         Retimer- 2Retimers- CrosslinkRes: unsupported
        Capabilities: [a4] Subsystem: PLX Technology, Inc. PEX 8717 16-lane, 8-Port PCI Express Gen 3 (8.0 GT/s) Switch with DMA
        Capabilities: [100 v1] Device Serial Number ca-87-00-10-b5-df-0e-00
        Capabilities: [fb4 v1] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
                AERCap: First Error Pointer: 1f, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
                        MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
                HeaderLog: 00000000 00000000 00000000 00000000
        Capabilities: [138 v1] Power Budgeting <?>
        Capabilities: [10c v1] Secondary PCI Express
                LnkCtl3: LnkEquIntrruptEn- PerformEqu-
                LaneErrStat: 0
        Capabilities: [148 v1] Virtual Channel
                Caps:   LPEVC=0 RefClk=100ns PATEntryBits=8
                Arb:    Fixed- WRR32- WRR64- WRR128-
                Ctrl:   ArbSelect=Fixed
                Status: InProgress-
                VC0:    Caps:   PATOffset=03 MaxTimeSlots=1 RejSnoopTrans-
                        Arb:    Fixed- WRR32- WRR64+ WRR128- TWRR128- WRR256-
                        Ctrl:   Enable+ ID=0 ArbSelect=WRR64 TC/VC=01
                        Status: NegoPending+ InProgress-
                        Port Arbitration Table <?>
        Capabilities: [e00 v1] Multicast
                McastCap: MaxGroups 64, ECRCRegen+
                McastCtl: NumGroups 1, Enable-
                McastBAR: IndexPos 0, BaseAddr 0000000000000000
                McastReceiveVec:      0000000000000000
                McastBlockAllVec:     0000000000000000
                McastBlockUntransVec: 0000000000000000
                McastOverlayBAR: OverlaySize 0 (disabled), BaseAddr 0000000000000000
        Capabilities: [f24 v1] Access Control Services
                ACSCap: SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl+ DirectTrans+
                ACSCtl: SrcValid+ TransBlk- ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans-
        Capabilities: [b70 v1] Vendor Specific Information: ID=0001 Rev=0 Len=010 <?>
        Kernel driver in use: pcieport
08:00.0 Bridge: PLX Technology, Inc. PEX PCI Express Switch NT0 Port Virtual Interface (rev ca)
        Subsystem: PLX Technology, Inc. PEX PCI Express Switch NT0 Port Virtual Interface
        Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Interrupt: pin A routed to IRQ 35
        NUMA node: 0
        IOMMU group: 77
        Region 0: Memory at c7300000 (32-bit, non-prefetchable) [size=256K]
        Region 2: Memory at 383fc0000000 (64-bit, prefetchable) [size=512M]
        Capabilities: [40] Power Management version 3
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [48] MSI: Enable- Count=1/8 Maskable+ 64bit+
                Address: 0000000000000000  Data: 0000
                Masking: 00000000  Pending: 00000000
        Capabilities: [68] Express (v2) Endpoint, MSI 00
                DevCap: MaxPayload 2048 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
                        ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 0W
                DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
                        RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+
                        MaxPayload 256 bytes, MaxReadReq 128 bytes
                DevSta: CorrErr+ NonFatalErr- FatalErr- UnsupReq+ AuxPwr- TransPend-
                LnkCap: Port #1, Speed 8GT/s, Width x8, ASPM L0s L1, Exit Latency L0s <2us, L1 <4us
                        ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
                LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk-
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 2.5GT/s (downgraded), Width x0 (downgraded)
                        TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Not Supported, TimeoutDis- NROPrPrP- LTR-
                         10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
                         EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
                         FRS- TPHComp- ExtTPHComp-
                         AtomicOpsCap: 32bit+ 64bit+ 128bitCAS+
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- 10BitTagReq- OBFF Disabled,
                         AtomicOpsCtl: ReqEn-
                LnkCap2: Supported Link Speeds: 2.5-8GT/s, Crosslink- Retimer- 2Retimers- DRS-
                LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
                LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-
                         EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
                         Retimer- 2Retimers- CrosslinkRes: unsupported
        Capabilities: [c8] Vendor Specific Information: Len=00 <?>
        Capabilities: [100 v1] Device Serial Number ca-87-00-10-b5-df-0e-00
        Capabilities: [fb4 v1] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
                AERCap: First Error Pointer: 1f, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
                        MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
                HeaderLog: 00000000 00000000 00000000 00000000
        Capabilities: [148 v1] Virtual Channel
                Caps:   LPEVC=0 RefClk=100ns PATEntryBits=1
                Arb:    Fixed- WRR32- WRR64- WRR128-
                Ctrl:   ArbSelect=Fixed
                Status: InProgress-
                VC0:    Caps:   PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
                        Arb:    Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
                        Ctrl:   Enable+ ID=0 ArbSelect=Fixed TC/VC=01
                        Status: NegoPending+ InProgress-
        Capabilities: [c34 v1] Vendor Specific Information: ID=0003 Rev=0 Len=078 <?>
        Capabilities: [b70 v1] Vendor Specific Information: ID=0001 Rev=0 Len=010 <?>
        Kernel modules: ntb_hw_plx
2

Same symptom as this thread, on different hardware — and I’ve been disassembling NetApp’s ONTAP driver, which I think pinpoints what ntb_hw_plx is missing. Sharing in case it helps.

Hardware: two repurposed NetApp FAS2750 controllers (Xeon D-1557, Broadwell-DE), shared backplane, now on FreeBSD 15.0-RELEASE after removing ONTAP. Switch is a PLX PEX 8725 (10b5:87b0, subvendor 1275 = NetApp). NetApp platform codename for this board family is “katana” (their driver has katana_plx87xx_* functions).

Symptom: ntb_hw_plx attaches cleanly on both nodes and detects back-to-back mode:

ntb_hw0: NTB-to-NTB (back-to-back) mode
ntb_hw0: 1 memory windows, 12 scratchpads, 16 doorbells
ntb_transport0: <NTB Transport> mw 0 spad 0-11 db 0-15 on ntb_hw0

but ntb_transport never gets link-up and ntb0 is never created. The NT0 Virtual endpoint shows the inter-switch link dead:

cap 10[68] = PCI-Express 2 endpoint
             link x0(x8) speed 0.0(8.0)

So the back-to-back link between the two switches never trains (Width x0), same as the 8717 case here.

Findings from disassembling ONTAP’s platform.ko (its PLX 87xx / katana driver):

  1. CSR window uses per-port blocks at port_number << 12. plx_setup_probe_mode addresses ports as (port<<12)+reg (e.g. +0x578); plxsw87xx_attach writes ports 0/1/8 (offsets 0x24/0x8024/0x1024…).

  2. Station/VS register map (read via BAR0 CSR, bus_space_read_4):

    • 0x300 per-port config (3-bit field per port)
    • 0x354 bit 0x20 = VS (virtual-switch) mode enable
    • 0x358 VS-enable bitmap
    • 0x360 + vs*4 that VS’s upstream port (low 5 bits)
    • 0x380 + vs*4 that VS’s port vector

  3. The retrain routine (in the plx probe_mode diagnostic sysctl) is just the standard PCIe op: read the port’s Link Control reg (PCIe cap + 0x10), set bit 5 (0x20 = Retrain Link), write back — done in config space on the enumerated port.

  4. plxsw87xx_attach programs the NTB windows (port<<12 + 0x24/0x28/0x2c on ports 0/1/8, plus station regs 0x3e018/01c, 0x3e0d4/0d8, 0x3ec3c/40) but never retrains or clears link-disable. So ONTAP’s normal boot path assumes the inter-switch link is already trained by the switch at power-on; retrain only exists as a manual diagnostic.

Questions:

  • Given ONTAP relies on power-on training, what actually brings the link-side port up on these boards — EEPROM strap, or a step ntb_hw_plx should do?
  • On a katana/8725 b2b layout, which port number is the inter-switch link side (the ports attach touches are 0/1/8)?
  • Has ntb_hw_plx ever been brought up on hardware where firmware did NOT pre-train the b2b link?

Both controllers are on my bench and I have the ONTAP platform.ko decompiled — happy to dump any function or register on request.

root@netapp-01:~ # pciconf -lc ntb_hw0
ntb_hw0@pci0:6:0:0: class=0x068000 rev=0xca hdr=0x00 vendor=0x10b5 device=0x87b0 subvendor=0x1275 subdevice=0x4a1c
cap 01[40] = powerspec 3 supports D0 D3 current D0
cap 05[48] = MSI supports 8 messages, 64 bit, vector masks
cap 10[68] = PCI-Express 2 endpoint max data 256(1024) RO NS
max read 128
link x0(x8) speed 0.0(8.0) ASPM disabled(L0s/L1)
cap 09[c8] = vendor (length 0)
ecap 0003[100] = Serial 1 ca870010b5df0e00
ecap 0001[fb4] = AER 1 0 fatal 0 non-fatal 0 corrected
ecap 0002[148] = VC 1 max VC1 lowpri VC0-VC1
ecap 000b[c34] = Vendor [1] ID 0003 Rev 0 Length 120
ecap 000b[b70] = Vendor [1] ID 0001 Rev 0 Length 16
root@netapp-01:~ #

root@netapp-01:~ # pciconf -lc
hostb0@pci0:0:0:0: class=0x060000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x6f00 subvendor=0x8086 subdevice=0x6f00
cap 10[90] = PCI-Express 2 root port max data 128(128)
max read 128
link x0(x4) speed 0.0(5.0) ASPM disabled(L1)
cap 01[e0] = powerspec 3 supports D0 D3 current D0
ecap 000b[100] = Vendor [1] ID 0002 Rev 0 Length 12
ecap 000b[144] = Vendor [1] ID 0004 Rev 1 Length 60
ecap 000b[1d0] = Vendor [1] ID 0003 Rev 1 Length 10
ecap 000b[280] = Vendor [1] ID 0005 Rev 3 Length 24
ecap 000b[300] = Vendor [1] ID 0008 Rev 0 Length 56
pcib1@pci0:0:1:0: class=0x060400 rev=0x04 hdr=0x01 vendor=0x8086 device=0x6f02 subvendor=0x8086 subdevice=0x6f02
cap 0d[40] = PCI Bridge subvendor=0x8086 subdevice=0x6f02
cap 05[60] = MSI supports 2 messages, vector masks
cap 10[90] = PCI-Express 2 root port max data 256(256) ARI disabled
max read 128
link x8(x8) speed 8.0(8.0)
cap 01[e0] = powerspec 3 supports D0 D3 current D0
ecap 000b[100] = Vendor [1] ID 0002 Rev 0 Length 12
ecap 000d[110] = ACS 1 Source Validation disabled, Translation Blocking disabled
P2P Req Redirect disabled, P2P Cmpl Redirect disabled
P2P Upstream Forwarding disabled, P2P Egress Control unavailable
P2P Direct Translated unavailable, Enhanced Capability unavailable
ecap 0001[148] = AER 1 0 fatal 0 non-fatal 0 corrected
ecap 000b[1d0] = Vendor [1] ID 0003 Rev 1 Length 10
ecap 0019[250] = PCIe Sec 1 lane errors 0
ecap 000b[280] = Vendor [1] ID 0005 Rev 3 Length 24
ecap 000b[300] = Vendor [1] ID 0008 Rev 0 Length 56
pcib7@pci0:0:2:0: class=0x060400 rev=0x04 hdr=0x01 vendor=0x8086 device=0x6f04 subvendor=0x8086 subdevice=0x6f04
cap 0d[40] = PCI Bridge subvendor=0x8086 subdevice=0x6f04
cap 05[60] = MSI supports 2 messages, vector masks
cap 10[90] = PCI-Express 2 root port max data 256(256) ARI disabled
max read 128
link x1(x1) speed 2.5(2.5)
cap 01[e0] = powerspec 3 supports D0 D3 current D0
ecap 000b[100] = Vendor [1] ID 0002 Rev 0 Length 12
ecap 000d[110] = ACS 1 Source Validation disabled, Translation Blocking disabled
P2P Req Redirect disabled, P2P Cmpl Redirect disabled
P2P Upstream Forwarding disabled, P2P Egress Control unavailable
P2P Direct Translated unavailable, Enhanced Capability unavailable
ecap 0001[148] = AER 1 0 fatal 0 non-fatal 0 corrected
ecap 000b[300] = Vendor [1] ID 0008 Rev 0 Length 56
pcib8@pci0:0:2:2: class=0x060400 rev=0x04 hdr=0x01 vendor=0x8086 device=0x6f06 subvendor=0x8086 subdevice=0x6f06
cap 0d[40] = PCI Bridge subvendor=0x8086 subdevice=0x6f06
cap 05[60] = MSI supports 2 messages, vector masks
cap 10[90] = PCI-Express 2 root port max data 256(256) ARI enabled
max read 128
link x1(x1) speed 2.5(2.5)
cap 01[e0] = powerspec 3 supports D0 D3 current D0
ecap 000b[100] = Vendor [1] ID 0002 Rev 0 Length 12
ecap 000d[110] = ACS 1 Source Validation disabled, Translation Blocking disabled
P2P Req Redirect disabled, P2P Cmpl Redirect disabled
P2P Upstream Forwarding disabled, P2P Egress Control unavailable
P2P Direct Translated unavailable, Enhanced Capability unavailable
ecap 0001[148] = AER 1 0 fatal 0 non-fatal 0 corrected
ecap 000b[300] = Vendor [1] ID 0008 Rev 0 Length 56
pcib9@pci0:0:3:0: class=0x060400 rev=0x04 hdr=0x01 vendor=0x8086 device=0x6f08 subvendor=0x8086 subdevice=0x6f08
cap 0d[40] = PCI Bridge subvendor=0x8086 subdevice=0x6f08
cap 05[60] = MSI supports 2 messages, vector masks
cap 10[90] = PCI-Express 2 root port max data 256(256) ARI disabled
max read 128
link x8(x8) speed 8.0(8.0)
cap 01[e0] = powerspec 3 supports D0 D3 current D0
ecap 000b[100] = Vendor [1] ID 0002 Rev 0 Length 12
ecap 000d[110] = ACS 1 Source Validation disabled, Translation Blocking disabled
P2P Req Redirect disabled, P2P Cmpl Redirect disabled
P2P Upstream Forwarding disabled, P2P Egress Control unavailable
P2P Direct Translated unavailable, Enhanced Capability unavailable
ecap 0001[148] = AER 1 0 fatal 0 non-fatal 0 corrected
ecap 000b[1d0] = Vendor [1] ID 0003 Rev 1 Length 10
ecap 0019[250] = PCIe Sec 1 lane errors 0
ecap 000b[280] = Vendor [1] ID 0005 Rev 3 Length 24
ecap 000b[300] = Vendor [1] ID 0008 Rev 0 Length 56
pcib10@pci0:0:3:2: class=0x060400 rev=0x04 hdr=0x01 vendor=0x8086 device=0x6f0a subvendor=0x8086 subdevice=0x6f0a
cap 0d[40] = PCI Bridge subvendor=0x8086 subdevice=0x6f0a
cap 05[60] = MSI supports 2 messages, vector masks
cap 10[90] = PCI-Express 2 root port max data 256(256) ARI enabled
max read 128
link x4(x4) speed 8.0(8.0)
cap 01[e0] = powerspec 3 supports D0 D3 current D0
ecap 000b[100] = Vendor [1] ID 0002 Rev 0 Length 12
ecap 000d[110] = ACS 1 Source Validation disabled, Translation Blocking disabled
P2P Req Redirect disabled, P2P Cmpl Redirect disabled
P2P Upstream Forwarding disabled, P2P Egress Control unavailable
P2P Direct Translated unavailable, Enhanced Capability unavailable
ecap 0001[148] = AER 1 0 fatal 0 non-fatal 0 corrected
ecap 000b[1d0] = Vendor [1] ID 0003 Rev 1 Length 10
ecap 0019[250] = PCIe Sec 1 lane errors 0
ecap 000b[280] = Vendor [1] ID 0005 Rev 3 Length 24
ecap 000b[300] = Vendor [1] ID 0008 Rev 0 Length 56
pcib11@pci0:0:3:3: class=0x060400 rev=0x04 hdr=0x01 vendor=0x8086 device=0x6f0b subvendor=0x8086 subdevice=0x6f0b
cap 0d[40] = PCI Bridge subvendor=0x8086 subdevice=0x6f0b
cap 05[60] = MSI supports 2 messages, vector masks
cap 10[90] = PCI-Express 2 root port max data 256(256) ARI enabled
max read 128
link x4(x4) speed 8.0(8.0)
cap 01[e0] = powerspec 3 supports D0 D3 current D0
ecap 000b[100] = Vendor [1] ID 0002 Rev 0 Length 12
ecap 000d[110] = ACS 1 Source Validation disabled, Translation Blocking disabled
P2P Req Redirect disabled, P2P Cmpl Redirect disabled
P2P Upstream Forwarding disabled, P2P Egress Control unavailable
P2P Direct Translated unavailable, Enhanced Capability unavailable
ecap 0001[148] = AER 1 0 fatal 0 non-fatal 0 corrected
ecap 000b[1d0] = Vendor [1] ID 0003 Rev 1 Length 10
ecap 0019[250] = PCIe Sec 1 lane errors 0
ecap 000b[280] = Vendor [1] ID 0005 Rev 3 Length 24
ecap 000b[300] = Vendor [1] ID 0008 Rev 0 Length 56
none0@pci0:0:5:0: class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x6f28 subvendor=0x8086 subdevice=0x6f28
cap 10[40] = PCI-Express 2 root endpoint max data 128(128)
max read 128
none1@pci0:0:5:1: class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x6f29 subvendor=0x8086 subdevice=0x6f29
cap 10[40] = PCI-Express 1 root endpoint max data 128(128)
max read 128
cap 05[80] = MSI supports 1 message, 64 bit
ecap 000b[100] = Vendor [1] ID 0006 Rev 1 Length 16
ecap 000b[110] = Vendor [1] ID 0006 Rev 1 Length 16
ecap 000b[120] = Vendor [1] ID 0006 Rev 1 Length 16
ecap 000b[130] = Vendor [1] ID 0006 Rev 1 Length 16
none2@pci0:0:5:2: class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x6f2a subvendor=0x8086 subdevice=0x6f2a
cap 10[40] = PCI-Express 2 root endpoint max data 128(128)
max read 128
ioapic0@pci0:0:5:4: class=0x080020 rev=0x04 hdr=0x00 vendor=0x8086 device=0x6f2c subvendor=0x8086 subdevice=0x6f2c
cap 10[44] = PCI-Express 1 root endpoint max data 128(128)
max read 128
cap 01[e0] = powerspec 3 supports D0 D3 current D0
xhci0@pci0:0:20:0: class=0x0c0330 rev=0x05 hdr=0x00 vendor=0x8086 device=0x8c31 subvendor=0x8086 subdevice=0x8c31
cap 01[70] = powerspec 2 supports D0 D3 current D0
cap 05[80] = MSI supports 8 messages, 64 bit enabled with 1 message
none3@pci0:0:22:0: class=0x078000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x8c3a subvendor=0x8086 subdevice=0x8c3a
cap 01[50] = powerspec 3 supports D0 D3 current D0
cap 05[8c] = MSI supports 1 message, 64 bit
pcib12@pci0:0:28:0: class=0x060400 rev=0xd5 hdr=0x01 vendor=0x8086 device=0x8c10 subvendor=0x8086 subdevice=0x8c10
cap 10[40] = PCI-Express 2 root port max data 128(128)
max read 128
link x0(x4) speed 0.0(5.0) ASPM disabled(L0s/L1)
slot 0 power limit 0 mW HotPlug(empty) surprise
cap 05[80] = MSI supports 1 message enabled with 1 message
cap 0d[90] = PCI Bridge subvendor=0x8086 subdevice=0x8c10
cap 01[a0] = powerspec 3 supports D0 D3 current D0
pcib13@pci0:0:28:4: class=0x060400 rev=0xd5 hdr=0x01 vendor=0x8086 device=0x8c18 subvendor=0x8086 subdevice=0x8c18
cap 10[40] = PCI-Express 2 root port max data 128(128)
max read 128
link x1(x1) speed 2.5(5.0) ASPM disabled(L0s/L1)
slot 4 power limit 100 mW
cap 05[80] = MSI supports 1 message
cap 0d[90] = PCI Bridge subvendor=0x8086 subdevice=0x8c18
cap 01[a0] = powerspec 3 supports D0 D3 current D0
ehci0@pci0:0:29:0: class=0x0c0320 rev=0x05 hdr=0x00 vendor=0x8086 device=0x8c26 subvendor=0x8086 subdevice=0x8c26
cap 01[50] = powerspec 2 supports D0 D3 current D0
cap 0a[58] = EHCI Debug Port at offset 0xa0 in map 0x14
cap 13[98] = PCI Advanced Features: FLR TP
isab0@pci0:0:31:0: class=0x060100 rev=0x05 hdr=0x00 vendor=0x8086 device=0x8c54 subvendor=0x8086 subdevice=0x8c54
cap 09[e0] = vendor (length 12) Intel cap 1 version 0
features: AMT, 4 PCI-e x1 slots
ahci0@pci0:0:31:2: class=0x010601 rev=0x05 hdr=0x00 vendor=0x8086 device=0x8c02 subvendor=0x8086 subdevice=0x8c02
cap 05[80] = MSI supports 1 message enabled with 1 message
cap 01[70] = powerspec 3 supports D0 D3 current D0
cap 12[a8] = SATA Index-Data Pair
ichsmb0@pci0:0:31:3: class=0x0c0500 rev=0x05 hdr=0x00 vendor=0x8086 device=0x8c22 subvendor=0x8086 subdevice=0x8c22
pcib2@pci0:1:0:0: class=0x060400 rev=0xca hdr=0x01 vendor=0x10b5 device=0x8725 subvendor=0x10b5 subdevice=0x8725
cap 01[40] = powerspec 3 supports D0 D3 current D0
cap 05[48] = MSI supports 8 messages, 64 bit, vector masks
cap 10[68] = PCI-Express 2 upstream port max data 256(1024) RO NS
max read 128
link x8(x8) speed 8.0(8.0) ASPM disabled(L1)
cap 0d[a4] = PCI Bridge subvendor=0x10b5 subdevice=0x8725
ecap 0003[100] = Serial 1 ca870010b5df0e00
ecap 0001[fb4] = AER 1 0 fatal 0 non-fatal 1 corrected
ecap 0004[138] = Power Budgeting 1
ecap 0019[10c] = PCIe Sec 1 lane errors 0
ecap 0002[148] = VC 1 max VC1 lowpri VC0-VC1
ecap 0012[e00] = Multicast 1
ecap 0018[b00] = LTR 1
ecap 000b[b70] = Vendor [1] ID 0001 Rev 0 Length 16
none4@pci0:1:0:1: class=0x088000 rev=0xca hdr=0x00 vendor=0x10b5 device=0x87d0 subvendor=0x10b5 subdevice=0x87d0
cap 01[40] = powerspec 3 supports D0 D3 current D0
cap 05[48] = MSI supports 8 messages, 64 bit, vector masks
cap 10[68] = PCI-Express 2 endpoint max data 256(1024) FLR RO NS
max read 512
link x8(x8) speed 8.0(8.0) ASPM disabled(L1)
ecap 0003[100] = Serial 1 ca870010b5df0e00
ecap 0001[fb4] = AER 1 0 fatal 0 non-fatal 1 corrected
ecap 000b[1f0] = Vendor [1] ID 0010 Rev 0 Length 196
ecap 000b[b70] = Vendor [1] ID 0001 Rev 0 Length 16
none5@pci0:1:0:3: class=0x088000 rev=0xca hdr=0x00 vendor=0x10b5 device=0x87d0 subvendor=0x10b5 subdevice=0x87d0
cap 01[40] = powerspec 3 supports D0 D3 current D0
cap 05[48] = MSI supports 8 messages, 64 bit, vector masks
cap 10[68] = PCI-Express 2 endpoint max data 256(1024) FLR RO NS
max read 512
link x8(x8) speed 8.0(8.0) ASPM disabled(L1)
ecap 0003[100] = Serial 1 ca870010b5df0e00
ecap 0001[fb4] = AER 1 0 fatal 0 non-fatal 1 corrected
ecap 000b[1f0] = Vendor [1] ID 0010 Rev 0 Length 196
ecap 000b[b70] = Vendor [1] ID 0001 Rev 0 Length 16
pcib3@pci0:2:1:0: class=0x060400 rev=0xca hdr=0x01 vendor=0x10b5 device=0x8725 subvendor=0x10b5 subdevice=0x8725
cap 01[40] = powerspec 3 supports D0 D3 current D0
cap 05[48] = MSI supports 8 messages, 64 bit, vector masks
cap 10[68] = PCI-Express 2 downstream port max data 256(1024) RO NS ARI disabled
max read 128
link x4(x4) speed 8.0(8.0) ASPM disabled(L0s/L1)
cap 0d[a4] = PCI Bridge subvendor=0x10b5 subdevice=0x8725
ecap 0003[100] = Serial 1 ca870010b5df0e00
ecap 0001[fb4] = AER 1 0 fatal 0 non-fatal 1 corrected
ecap 0004[138] = Power Budgeting 1
ecap 0019[10c] = PCIe Sec 1 lane errors 0
ecap 0002[148] = VC 1 max VC1 lowpri VC0-VC1
ecap 0012[e00] = Multicast 1
ecap 000d[f24] = ACS 1 Source Validation disabled, Translation Blocking disabled
P2P Req Redirect disabled, P2P Cmpl Redirect disabled
P2P Upstream Forwarding disabled, P2P Egress Control disabled
P2P Direct Translated disabled, Enhanced Capability unavailable
ecap 000b[b70] = Vendor [1] ID 0001 Rev 0 Length 16
pcib4@pci0:2:2:0: class=0x060400 rev=0xca hdr=0x01 vendor=0x10b5 device=0x8725 subvendor=0x10b5 subdevice=0x8725
cap 01[40] = powerspec 3 supports D0 D3 current D0
cap 05[48] = MSI supports 8 messages, 64 bit, vector masks enabled with 1 message
cap 10[68] = PCI-Express 2 downstream port max data 256(1024) RO NS ARI disabled
max read 128
link x0(x4) speed 0.0(8.0) ASPM disabled(L1)
slot 34 power limit 25000 mW HotPlug(empty) Attn Button PC(off) MRL(open)
cap 0d[a4] = PCI Bridge subvendor=0x10b5 subdevice=0x8725
ecap 0003[100] = Serial 1 ca870010b5df0e00
ecap 0001[fb4] = AER 1 0 fatal 0 non-fatal 1 corrected
ecap 0004[138] = Power Budgeting 1
ecap 0019[10c] = PCIe Sec 1 lane errors 0
ecap 0002[148] = VC 1 max VC1 lowpri VC0-VC1
ecap 0012[e00] = Multicast 1
ecap 000d[f24] = ACS 1 Source Validation disabled, Translation Blocking disabled
P2P Req Redirect disabled, P2P Cmpl Redirect disabled
P2P Upstream Forwarding disabled, P2P Egress Control disabled
P2P Direct Translated disabled, Enhanced Capability unavailable
ecap 000b[b70] = Vendor [1] ID 0001 Rev 0 Length 16
pcib5@pci0:2:8:0: class=0x060400 rev=0xca hdr=0x01 vendor=0x10b5 device=0x8725 subvendor=0x10b5 subdevice=0x8725
cap 01[40] = powerspec 3 supports D0 D3 current D0
cap 05[48] = MSI supports 8 messages, 64 bit, vector masks
cap 10[68] = PCI-Express 2 downstream port max data 256(1024) RO NS ARI disabled
max read 128
link x0(x8) speed 0.0(8.0) ASPM disabled(L1)
slot 40 power limit 25000 mW
cap 0d[a4] = PCI Bridge subvendor=0x10b5 subdevice=0x8725
ecap 0003[100] = Serial 1 ca870010b5df0e00
ecap 0001[fb4] = AER 1 0 fatal 0 non-fatal 1 corrected
ecap 0004[138] = Power Budgeting 1
ecap 0019[10c] = PCIe Sec 1 lane errors 0
ecap 0002[148] = VC 1 max VC1 lowpri VC0-VC1
ecap 0012[e00] = Multicast 1
ecap 000d[f24] = ACS 1 Source Validation disabled, Translation Blocking disabled
P2P Req Redirect disabled, P2P Cmpl Redirect disabled
P2P Upstream Forwarding disabled, P2P Egress Control disabled
P2P Direct Translated disabled, Enhanced Capability unavailable
ecap 000b[b70] = Vendor [1] ID 0001 Rev 0 Length 16
pcib6@pci0:2:9:0: class=0x060400 rev=0xca hdr=0x01 vendor=0x10b5 device=0x8725 subvendor=0x10b5 subdevice=0x8725
cap 01[40] = powerspec 3 supports D0 D3 current D0
cap 05[48] = MSI supports 8 messages, 64 bit, vector masks
cap 10[68] = PCI-Express 2 downstream port max data 256(1024) RO NS ARI disabled
max read 128
link x0(x8) speed 0.0(8.0) ASPM disabled(L0s/L1)
cap 0d[a4] = PCI Bridge subvendor=0x10b5 subdevice=0x8725
ecap 0003[100] = Serial 1 ca870010b5df0e00
ecap 0001[fb4] = AER 1 0 fatal 0 non-fatal 1 corrected
ecap 0004[138] = Power Budgeting 1
ecap 0019[10c] = PCIe Sec 1 lane errors 0
ecap 0002[148] = VC 1 max VC1 lowpri VC0-VC1
ecap 0012[e00] = Multicast 1
ecap 000d[f24] = ACS 1 Source Validation disabled, Translation Blocking disabled
P2P Req Redirect disabled, P2P Cmpl Redirect disabled
P2P Upstream Forwarding disabled, P2P Egress Control disabled
P2P Direct Translated disabled, Enhanced Capability unavailable
ecap 000b[b70] = Vendor [1] ID 0001 Rev 0 Length 16
nvme0@pci0:3:0:0: class=0x010802 rev=0x00 hdr=0x00 vendor=0x144d device=0xa808 subvendor=0x144d subdevice=0xa801
cap 01[40] = powerspec 3 supports D0 D3 current D0
cap 05[50] = MSI supports 32 messages, 64 bit
cap 10[70] = PCI-Express 2 endpoint max data 256(256) FLR RO NS
max read 512
link x4(x4) speed 8.0(8.0)
cap 11[b0] = MSI-X supports 33 messages, enabled
Table in map 0x10[0x3000], PBA in map 0x10[0x2000]
ecap 0001[100] = AER 2 0 fatal 0 non-fatal 0 corrected
ecap 0003[148] = Serial 1 0000000000000000
ecap 0004[158] = Power Budgeting 1
ecap 0019[168] = PCIe Sec 1 lane errors 0
ecap 0018[188] = LTR 1
ecap 001e[190] = L1 PM Substates 1
ntb_hw0@pci0:6:0:0: class=0x068000 rev=0xca hdr=0x00 vendor=0x10b5 device=0x87b0 subvendor=0x1275 subdevice=0x4a1c
cap 01[40] = powerspec 3 supports D0 D3 current D0
cap 05[48] = MSI supports 8 messages, 64 bit, vector masks
cap 10[68] = PCI-Express 2 endpoint max data 256(1024) RO NS
max read 128
link x0(x8) speed 0.0(8.0) ASPM disabled(L0s/L1)
cap 09[c8] = vendor (length 0)
ecap 0003[100] = Serial 1 ca870010b5df0e00
ecap 0001[fb4] = AER 1 0 fatal 0 non-fatal 0 corrected
ecap 0002[148] = VC 1 max VC1 lowpri VC0-VC1
ecap 000b[c34] = Vendor [1] ID 0003 Rev 0 Length 120
ecap 000b[b70] = Vendor [1] ID 0001 Rev 0 Length 16
ioat0@pci0:7:0:0: class=0x088000 rev=0x00 hdr=0x00 vendor=0x8086 device=0x6f50 subvendor=0x8086 subdevice=0x6f50
cap 10[40] = PCI-Express 2 endpoint max data 256(256) FLR
max read 256
cap 01[80] = powerspec 3 supports D0 D3 current D0
cap 05[90] = MSI supports 1 message, vector masks
cap 11[ac] = MSI-X supports 1 message, enabled
Table in map 0x10[0x1000], PBA in map 0x10[0x1800]
ecap 0001[100] = AER 1 0 fatal 0 non-fatal 1 corrected
ecap 0012[150] = Multicast 1
ioat1@pci0:7:0:1: class=0x088000 rev=0x00 hdr=0x00 vendor=0x8086 device=0x6f51 subvendor=0x8086 subdevice=0x6f51
cap 10[40] = PCI-Express 2 endpoint max data 256(256) FLR
max read 256
cap 01[80] = powerspec 3 supports D0 D3 current D0
cap 05[90] = MSI supports 1 message, vector masks
cap 11[ac] = MSI-X supports 1 message, enabled
Table in map 0x10[0x1000], PBA in map 0x10[0x1800]
ecap 0001[100] = AER 1 0 fatal 0 non-fatal 1 corrected
ecap 0012[150] = Multicast 1
ioat2@pci0:7:0:2: class=0x088000 rev=0x00 hdr=0x00 vendor=0x8086 device=0x6f52 subvendor=0x8086 subdevice=0x6f52
cap 10[40] = PCI-Express 2 endpoint max data 256(256) FLR
max read 256
cap 01[80] = powerspec 3 supports D0 D3 current D0
cap 05[90] = MSI supports 1 message, vector masks
cap 11[ac] = MSI-X supports 1 message, enabled
Table in map 0x10[0x1000], PBA in map 0x10[0x1800]
ecap 0001[100] = AER 1 0 fatal 0 non-fatal 1 corrected
ecap 0012[150] = Multicast 1
ioat3@pci0:7:0:3: class=0x088000 rev=0x00 hdr=0x00 vendor=0x8086 device=0x6f53 subvendor=0x8086 subdevice=0x6f53
cap 10[40] = PCI-Express 2 endpoint max data 256(256) FLR
max read 256
cap 01[80] = powerspec 3 supports D0 D3 current D0
cap 05[90] = MSI supports 1 message, vector masks
cap 11[ac] = MSI-X supports 1 message, enabled
Table in map 0x10[0x1000], PBA in map 0x10[0x1800]
ecap 0001[100] = AER 1 0 fatal 0 non-fatal 1 corrected
ecap 0012[150] = Multicast 1
ix0@pci0:8:0:0: class=0x020000 rev=0x00 hdr=0x00 vendor=0x8086 device=0x15ac subvendor=0x8086 subdevice=0x0000
cap 01[40] = powerspec 3 supports D0 D3 current D0
cap 05[50] = MSI supports 1 message, 64 bit, vector masks
cap 11[70] = MSI-X supports 64 messages, enabled
Table in map 0x20[0x0], PBA in map 0x20[0x2000]
cap 10[a0] = PCI-Express 2 endpoint max data 256(512) FLR RO
max read 512
link x1(x1) speed 2.5(2.5) ASPM disabled(L0s/L1)
ecap 0001[100] = AER 2 0 fatal 0 non-fatal 1 corrected
ecap 0003[140] = Serial 1 0000c9ffff000000
ecap 000e[150] = ARI 1
ecap 0010[160] = SR-IOV 1 IOV disabled, Memory Space disabled, ARI disabled
0 VFs configured out of 64 supported
First VF RID Offset 0x0180, VF RID Stride 0x0002
VF Device ID 0x15a8
Page Sizes: 4096 (enabled), 8192, 65536, 262144, 1048576, 4194304
ecap 000d[1b0] = ACS 1 Source Validation unavailable, Translation Blocking unavailable
P2P Req Redirect unavailable, P2P Cmpl Redirect unavailable
P2P Upstream Forwarding unavailable, P2P Egress Control unavailable
P2P Direct Translated unavailable, Enhanced Capability unavailable
ecap 0018[1c0] = LTR 1
ix1@pci0:8:0:1: class=0x020000 rev=0x00 hdr=0x00 vendor=0x8086 device=0x15ac subvendor=0x8086 subdevice=0x0000
cap 01[40] = powerspec 3 supports D0 D3 current D0
cap 05[50] = MSI supports 1 message, 64 bit, vector masks
cap 11[70] = MSI-X supports 64 messages, enabled
Table in map 0x20[0x0], PBA in map 0x20[0x2000]
cap 10[a0] = PCI-Express 2 endpoint max data 256(512) FLR RO
max read 512
link x1(x1) speed 2.5(2.5) ASPM disabled(L0s/L1)
ecap 0001[100] = AER 2 0 fatal 0 non-fatal 1 corrected
ecap 0003[140] = Serial 1 0000c9ffff000000
ecap 000e[150] = ARI 1
ecap 0010[160] = SR-IOV 1 IOV disabled, Memory Space disabled, ARI disabled
0 VFs configured out of 64 supported
First VF RID Offset 0x0180, VF RID Stride 0x0002
VF Device ID 0x15a8
Page Sizes: 4096 (enabled), 8192, 65536, 262144, 1048576, 4194304
ecap 000d[1b0] = ACS 1 Source Validation unavailable, Translation Blocking unavailable
P2P Req Redirect unavailable, P2P Cmpl Redirect unavailable
P2P Upstream Forwarding unavailable, P2P Egress Control unavailable
P2P Direct Translated unavailable, Enhanced Capability unavailable
pmspcv0@pci0:9:0:0: class=0x010700 rev=0x06 hdr=0x00 vendor=0x11f8 device=0x8070 subvendor=0x0000 subdevice=0x0000
cap 01[80] = powerspec 3 supports D0 D1 D3 current D0
cap 03[88] = VPD
cap 05[90] = MSI supports 32 messages, 64 bit, vector masks
cap 11[b0] = MSI-X supports 64 messages, enabled
Table in map 0x10[0x400], PBA in map 0x10[0x800]
cap 10[c0] = PCI-Express 2 endpoint max data 256(512) RO NS
max read 512
link x8(x8) speed 8.0(8.0)
ecap 0001[100] = AER 2 0 fatal 0 non-fatal 0 corrected
ecap 0019[300] = PCIe Sec 1 lane errors 0
ql0@pci0:10:0:0: class=0x020000 rev=0x02 hdr=0x00 vendor=0x1077 device=0x8830 subvendor=0x1275 subdevice=0xfb05
cap 01[44] = powerspec 3 supports D0 D3 current D0
cap 10[4c] = PCI-Express 2 endpoint max data 256(1024) FLR RO NS
max read 512
link x4(x8) speed 8.0(8.0)
cap 03[88] = VPD
cap 11[90] = MSI-X supports 64 messages, enabled
Table in map 0x18[0x0], PBA in map 0x18[0x1000]
ecap 0001[100] = AER 1 0 fatal 0 non-fatal 1 corrected
ecap 0003[138] = Serial 1 d039eafffe257eb5
ecap 000e[154] = ARI 1
ecap 0019[1b0] = PCIe Sec 1 lane errors 0
ql1@pci0:10:0:1: class=0x020000 rev=0x02 hdr=0x00 vendor=0x1077 device=0x8830 subvendor=0x1275 subdevice=0xfb05
cap 01[44] = powerspec 3 supports D0 D3 current D0
cap 10[4c] = PCI-Express 2 endpoint max data 256(1024) FLR RO NS
max read 512
link x4(x8) speed 8.0(8.0)
cap 03[88] = VPD
cap 11[90] = MSI-X supports 64 messages, enabled
Table in map 0x18[0x0], PBA in map 0x18[0x1000]
ecap 0001[100] = AER 1 0 fatal 0 non-fatal 1 corrected
ecap 0003[138] = Serial 1 d039eafffe257eb5
ecap 000e[154] = ARI 1
none6@pci0:10:0:2: class=0x0c0400 rev=0x02 hdr=0x00 vendor=0x1077 device=0x8831 subvendor=0x1275 subdevice=0xfb05
cap 01[44] = powerspec 3 supports D0 D3 current D0
cap 10[4c] = PCI-Express 2 endpoint max data 256(1024) FLR RO NS
max read 512
link x4(x8) speed 8.0(8.0)
cap 03[88] = VPD
cap 11[90] = MSI-X supports 16 messages
Table in map 0x18[0x0], PBA in map 0x18[0x1000]
ecap 0001[100] = AER 1 0 fatal 0 non-fatal 1 corrected
ecap 0003[138] = Serial 1 d039eafffe257eb5
ecap 000e[154] = ARI 1
none7@pci0:10:0:3: class=0x0c0400 rev=0x02 hdr=0x00 vendor=0x1077 device=0x8831 subvendor=0x1275 subdevice=0xfb05
cap 01[44] = powerspec 3 supports D0 D3 current D0
cap 10[4c] = PCI-Express 2 endpoint max data 256(1024) FLR RO NS
max read 512
link x4(x8) speed 8.0(8.0)
cap 03[88] = VPD
cap 11[90] = MSI-X supports 16 messages
Table in map 0x18[0x0], PBA in map 0x18[0x1000]
ecap 0001[100] = AER 1 0 fatal 0 non-fatal 1 corrected
ecap 0003[138] = Serial 1 d039eafffe257eb5
ecap 000e[154] = ARI 1
none8@pci0:10:0:4: class=0x028000 rev=0x02 hdr=0x00 vendor=0x1077 device=0x8832 subvendor=0x1275 subdevice=0xfb05
cap 01[44] = powerspec 3 supports D0 D3 current D0
cap 10[4c] = PCI-Express 2 endpoint max data 256(1024) FLR RO NS
max read 512
link x4(x8) speed 8.0(8.0)
cap 03[88] = VPD
cap 11[90] = MSI-X supports 64 messages
Table in map 0x18[0x0], PBA in map 0x18[0x1000]
ecap 0001[100] = AER 1 0 fatal 0 non-fatal 1 corrected
ecap 0003[138] = Serial 1 d039eafffe257eb5
ecap 000e[154] = ARI 1
none9@pci0:10:0:5: class=0x028000 rev=0x02 hdr=0x00 vendor=0x1077 device=0x8832 subvendor=0x1275 subdevice=0xfb05
cap 01[44] = powerspec 3 supports D0 D3 current D0
cap 10[4c] = PCI-Express 2 endpoint max data 256(1024) FLR RO NS
max read 512
link x4(x8) speed 8.0(8.0)
cap 03[88] = VPD
cap 11[90] = MSI-X supports 64 messages
Table in map 0x18[0x0], PBA in map 0x18[0x1000]
ecap 0001[100] = AER 1 0 fatal 0 non-fatal 1 corrected
ecap 0003[138] = Serial 1 d039eafffe257eb5
ecap 000e[154] = ARI 1
ql2@pci0:11:0:0: class=0x020000 rev=0x02 hdr=0x00 vendor=0x1077 device=0x8830 subvendor=0x1275 subdevice=0xfb05
cap 01[44] = powerspec 3 supports D0 D3 current D0
cap 10[4c] = PCI-Express 2 endpoint max data 256(1024) FLR RO NS
max read 512
link x4(x8) speed 8.0(8.0)
cap 03[88] = VPD
cap 11[90] = MSI-X supports 64 messages, enabled
Table in map 0x18[0x0], PBA in map 0x18[0x1000]
ecap 0001[100] = AER 1 0 fatal 0 non-fatal 1 corrected
ecap 0003[138] = Serial 1 d039eafffe257eb3
ecap 000e[154] = ARI 1
ecap 0019[1b0] = PCIe Sec 1 lane errors 0
ql3@pci0:11:0:1: class=0x020000 rev=0x02 hdr=0x00 vendor=0x1077 device=0x8830 subvendor=0x1275 subdevice=0xfb05
cap 01[44] = powerspec 3 supports D0 D3 current D0
cap 10[4c] = PCI-Express 2 endpoint max data 256(1024) FLR RO NS
max read 512
link x4(x8) speed 8.0(8.0)
cap 03[88] = VPD
cap 11[90] = MSI-X supports 64 messages, enabled
Table in map 0x18[0x0], PBA in map 0x18[0x1000]
ecap 0001[100] = AER 1 0 fatal 0 non-fatal 1 corrected
ecap 0003[138] = Serial 1 d039eafffe257eb3
ecap 000e[154] = ARI 1
none10@pci0:11:0:2: class=0x0c0400 rev=0x02 hdr=0x00 vendor=0x1077 device=0x8831 subvendor=0x1275 subdevice=0xfb05
cap 01[44] = powerspec 3 supports D0 D3 current D0
cap 10[4c] = PCI-Express 2 endpoint max data 256(1024) FLR RO NS
max read 512
link x4(x8) speed 8.0(8.0)
cap 03[88] = VPD
cap 11[90] = MSI-X supports 16 messages
Table in map 0x18[0x0], PBA in map 0x18[0x1000]
ecap 0001[100] = AER 1 0 fatal 0 non-fatal 1 corrected
ecap 0003[138] = Serial 1 d039eafffe257eb3
ecap 000e[154] = ARI 1
none11@pci0:11:0:3: class=0x0c0400 rev=0x02 hdr=0x00 vendor=0x1077 device=0x8831 subvendor=0x1275 subdevice=0xfb05
cap 01[44] = powerspec 3 supports D0 D3 current D0
cap 10[4c] = PCI-Express 2 endpoint max data 256(1024) FLR RO NS
max read 512
link x4(x8) speed 8.0(8.0)
cap 03[88] = VPD
cap 11[90] = MSI-X supports 16 messages
Table in map 0x18[0x0], PBA in map 0x18[0x1000]
ecap 0001[100] = AER 1 0 fatal 0 non-fatal 1 corrected
ecap 0003[138] = Serial 1 d039eafffe257eb3
ecap 000e[154] = ARI 1
none12@pci0:11:0:4: class=0x028000 rev=0x02 hdr=0x00 vendor=0x1077 device=0x8832 subvendor=0x1275 subdevice=0xfb05
cap 01[44] = powerspec 3 supports D0 D3 current D0
cap 10[4c] = PCI-Express 2 endpoint max data 256(1024) FLR RO NS
max read 512
link x4(x8) speed 8.0(8.0)
cap 03[88] = VPD
cap 11[90] = MSI-X supports 64 messages
Table in map 0x18[0x0], PBA in map 0x18[0x1000]
ecap 0001[100] = AER 1 0 fatal 0 non-fatal 1 corrected
ecap 0003[138] = Serial 1 d039eafffe257eb3
ecap 000e[154] = ARI 1
none13@pci0:11:0:5: class=0x028000 rev=0x02 hdr=0x00 vendor=0x1077 device=0x8832 subvendor=0x1275 subdevice=0xfb05
cap 01[44] = powerspec 3 supports D0 D3 current D0
cap 10[4c] = PCI-Express 2 endpoint max data 256(1024) FLR RO NS
max read 512
link x4(x8) speed 8.0(8.0)
cap 03[88] = VPD
cap 11[90] = MSI-X supports 64 messages
Table in map 0x18[0x0], PBA in map 0x18[0x1000]
ecap 0001[100] = AER 1 0 fatal 0 non-fatal 1 corrected
ecap 0003[138] = Serial 1 d039eafffe257eb3
ecap 000e[154] = ARI 1
igb0@pci0:13:0:0: class=0x020000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x1533 subvendor=0xffff subdevice=0x0000
cap 01[40] = powerspec 3 supports D0 D3 current D0
cap 05[50] = MSI supports 1 message, 64 bit, vector masks
cap 11[70] = MSI-X supports 5 messages, enabled
Table in map 0x1c[0x0], PBA in map 0x1c[0x2000]
cap 10[a0] = PCI-Express 2 endpoint max data 128(512) FLR RO NS
max read 512
link x1(x1) speed 2.5(2.5) ASPM disabled(L0s/L1)
ecap 0001[100] = AER 2 0 fatal 0 non-fatal 0 corrected
ecap 0003[140] = Serial 1 d039eaffff257ebf
ecap 0017[1a0] = TPH Requester 1
none14@pci0:255:11:0: class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x6f81 subvendor=0x8086 subdevice=0x6f81
none15@pci0:255:11:1: class=0x110100 rev=0x04 hdr=0x00 vendor=0x8086 device=0x6f36 subvendor=0x8086 subdevice=0x6f36
none16@pci0:255:11:2: class=0x110100 rev=0x04 hdr=0x00 vendor=0x8086 device=0x6f37 subvendor=0x8086 subdevice=0x6f37
none17@pci0:255:11:3: class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x6f76 subvendor=0x0000 subdevice=0x0000
none18@pci0:255:12:0: class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x6fe0 subvendor=0x8086 subdevice=0x6fe0
none19@pci0:255:12:1: class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x6fe1 subvendor=0x8086 subdevice=0x6fe1
none20@pci0:255:12:2: class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x6fe2 subvendor=0x8086 subdevice=0x6fe2
none21@pci0:255:12:3: class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x6fe3 subvendor=0x8086 subdevice=0x6fe3
none22@pci0:255:12:4: class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x6fe4 subvendor=0x8086 subdevice=0x6fe4
none23@pci0:255:12:5: class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x6fe5 subvendor=0x8086 subdevice=0x6fe5
none24@pci0:255:12:6: class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x6fe6 subvendor=0x8086 subdevice=0x6fe6
none25@pci0:255:12:7: class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x6fe7 subvendor=0x8086 subdevice=0x6fe7
none26@pci0:255:13:0: class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x6fe8 subvendor=0x8086 subdevice=0x6fe8
none27@pci0:255:13:1: class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x6fe9 subvendor=0x8086 subdevice=0x6fe9
none28@pci0:255:13:2: class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x6fea subvendor=0x8086 subdevice=0x6fea
none29@pci0:255:13:3: class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x6feb subvendor=0x8086 subdevice=0x6feb
none30@pci0:255:15:0: class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x6ff8 subvendor=0x8086 subdevice=0x6ff8
none31@pci0:255:15:4: class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x6ffc subvendor=0x8086 subdevice=0x6fe0
none32@pci0:255:15:5: class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x6ffd subvendor=0x8086 subdevice=0x6fe0
none33@pci0:255:15:6: class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x6ffe subvendor=0x8086 subdevice=0x6fe0
none34@pci0:255:16:0: class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x6f1d subvendor=0x8086 subdevice=0x6f1d
none35@pci0:255:16:1: class=0x110100 rev=0x04 hdr=0x00 vendor=0x8086 device=0x6f34 subvendor=0x8086 subdevice=0x6f34
none36@pci0:255:16:5: class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x6f1e subvendor=0x8086 subdevice=0x6f1e
none37@pci0:255:16:6: class=0x110100 rev=0x04 hdr=0x00 vendor=0x8086 device=0x6f7d subvendor=0x8086 subdevice=0x6f7d
none38@pci0:255:16:7: class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x6f1f subvendor=0x8086 subdevice=0x6f1f
none39@pci0:255:18:0: class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x6fa0 subvendor=0x8086 subdevice=0x6fa0
none40@pci0:255:18:1: class=0x110100 rev=0x04 hdr=0x00 vendor=0x8086 device=0x6f30 subvendor=0x8086 subdevice=0x6f30
none41@pci0:255:19:0: class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x6fa8 subvendor=0x8086 subdevice=0x6fa8
cap 10[40] = PCI-Express 1 root endpoint max data 128(128)
max read 128
none42@pci0:255:19:1: class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x6f71 subvendor=0x8086 subdevice=0x6f71
cap 10[40] = PCI-Express 1 root endpoint max data 128(128)
max read 128
none43@pci0:255:19:2: class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x6faa subvendor=0x8086 subdevice=0x6faa
cap 10[40] = PCI-Express 1 root endpoint max data 128(128)
max read 128
none44@pci0:255:19:3: class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x6fab subvendor=0x8086 subdevice=0x6fab
cap 10[40] = PCI-Express 1 root endpoint max data 128(128)
max read 128
none45@pci0:255:19:4: class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x6fac subvendor=0x8086 subdevice=0x6fac
cap 10[40] = PCI-Express 1 root endpoint max data 128(128)
max read 128
none46@pci0:255:19:5: class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x6fad subvendor=0x8086 subdevice=0x6fad
cap 10[40] = PCI-Express 1 root endpoint max data 128(128)
max read 128
none47@pci0:255:19:6: class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x6fae subvendor=0x0000 subdevice=0x0000
none48@pci0:255:19:7: class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x6faf subvendor=0x0000 subdevice=0x0000
none49@pci0:255:20:0: class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x6fb0 subvendor=0x8086 subdevice=0x6fb0
cap 10[40] = PCI-Express 1 root endpoint max data 128(128)
max read 128
none50@pci0:255:20:1: class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x6fb1 subvendor=0x8086 subdevice=0x6fb1
cap 10[40] = PCI-Express 1 root endpoint max data 128(128)
max read 128
none51@pci0:255:20:2: class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x6fb2 subvendor=0x8086 subdevice=0x6fb2
cap 10[40] = PCI-Express 1 root endpoint max data 128(128)
max read 128
none52@pci0:255:20:3: class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x6fb3 subvendor=0x8086 subdevice=0x6fb3
cap 10[40] = PCI-Express 1 root endpoint max data 128(128)
max read 128
none53@pci0:255:20:4: class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x6fbc subvendor=0x0000 subdevice=0x0000
none54@pci0:255:20:5: class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x6fbd subvendor=0x0000 subdevice=0x0000
none55@pci0:255:20:6: class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x6fbe subvendor=0x0000 subdevice=0x0000
none56@pci0:255:20:7: class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x6fbf subvendor=0x0000 subdevice=0x0000
none57@pci0:255:21:0: class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x6fb4 subvendor=0x8086 subdevice=0x6fb4
cap 10[40] = PCI-Express 1 root endpoint max data 128(128)
max read 128
none58@pci0:255:21:1: class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x6fb5 subvendor=0x8086 subdevice=0x6fb5
cap 10[40] = PCI-Express 1 root endpoint max data 128(128)
max read 128
none59@pci0:255:21:2: class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x6fb6 subvendor=0x8086 subdevice=0x6fb6
cap 10[40] = PCI-Express 1 root endpoint max data 128(128)
max read 128
none60@pci0:255:21:3: class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x6fb7 subvendor=0x8086 subdevice=0x6fb7
cap 10[40] = PCI-Express 1 root endpoint max data 128(128)
max read 128
none61@pci0:255:30:0: class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x6f98 subvendor=0x8086 subdevice=0x6f98
none62@pci0:255:30:1: class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x6f99 subvendor=0x8086 subdevice=0x6f99
none63@pci0:255:30:2: class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x6f9a subvendor=0x8086 subdevice=0x6f9a
none64@pci0:255:30:3: class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x6fc0 subvendor=0x8086 subdevice=0x6fc0
none65@pci0:255:30:4: class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x6f9c subvendor=0x8086 subdevice=0x6f9c
none66@pci0:255:31:0: class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x6f88 subvendor=0x0000 subdevice=0x0000
none67@pci0:255:31:2: class=0x088000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x6f8a subvendor=0x0000 subdevice=0x0000
root@netapp-01:~ #