PCIe问题举例: Enable SR-IOV导致资源分配失败

发布时间 2023-06-17 14:13:04作者: zhuqingzhu
资源分配问题分析过程:
1.在shell下和OS都能复制到现象;
2.测试CRB BIOS,发现CRB BIOS在shell下没有复制到现象,在OS下复制到现象;
3.比较Aoqin BIOS和CRB BIOS,发现在setup里面把PCIe SR-IOV disable,shell下就能正常分配到Region0/2的资源;---- 看来跟PCIe SR-IOV功能有关
4.Region0/2的资源是MMIO32类型的,Ampere平台给每个RC预留的MMIO32的资源一共是128MB(Ampere的代码里面有定义);同时Tencent反馈摇光卡默认支持SR-IOV功能,仅该功能就需要申请128MB的MMIO32资源,从而导致我们的MMIO32资源不够。
5.但是把PCIe SR-IOV disable后进OS还是分配不到Region0/2的资源,是因为linux系统默认也是支持SR-IOV功能的(如果要在OS下也能正确分配资源,需要在OS里面把PCIe SR-IOV disable)。
6.最终的解决方案是摇光卡固件里面把PCIe SR-IOV功能disable。
 

 

问题:

接上PCIe设备之后,不能开机,log看到是pcie资源fail。

结论:

把SR-IOV [disable]后,可以开机,能认到设备。

 

 

         

 

 

问题:

接上PCIe卡之后,hang机,post log显示pci 资源分配失败,OUT_OF_RESOURCE

现象:

1.在BIOS里面把SR-IOV disable之后,可以开机,进shell看设备资源分配正常;

2.但是进OS之后发现BAR0/2没有分配到资源,BAR4分配到资源

3.修改pcie卡的FW,disable 卡的SR-IOV,卡工作正常

分析:

卡本身需要的资源:BAR0需要MMIO32, BAR2需要MMIO32, BAR4需要MMIO64 ,如果开启SRIOV,VF也需要MMIO32的资源。

BIOS里面定义了(某个RC)的MMIO32资源只有128MB,而仅仅VF 需要的资源就需要 16MB*8个VF=128MB,从而导致bar0、2分配不到资源。

[   10.234281] pci 000a:03:00.0: [1ea0:2a18] type 00 class 0x008004
[   10.234311] pci 000a:03:00.0: reg 0x10: [mem 0x50800000-0x50800fff]
[   10.234325] pci 000a:03:00.0: reg 0x18: [mem 0x50000000-0x507fffff]
[   10.234344] pci 000a:03:00.0: reg 0x20: [mem 0x680000000000-0x6801ffffffff 64bit pref]
[   10.234358] pci 000a:03:00.0: enabling Extended Tags
[   10.239404] pci 000a:03:00.0: supports D1 D2
[   10.239405] pci 000a:03:00.0: PME# supported from D0 D1 D2 D3hot D3cold
[   10.239438] pci 000a:03:00.0: reg 0x164: [mem 0x00000000-0x0000ffff]
[   10.239441] pci 000a:03:00.0: VF(n) BAR0 space: [mem 0x00000000-0x0007ffff] (contains BAR0 for 8 VFs)
[   10.248658] pci 000a:03:00.0: reg 0x16c: [mem 0x00000000-0x00ffffff]
[   10.248659] pci 000a:03:00.0: VF(n) BAR2 space: [mem 0x00000000-0x07ffffff] (contains BAR2 for 8 VFs)
[   10.257890] pci 000a:03:00.0: reg 0x174: [mem 0x00000000-0x1ffffffff 64bit pref]
[   10.257892] pci 000a:03:00.0: VF(n) BAR4 space: [mem 0x00000000-0xfffffffff 64bit pref] (contains BAR4 for 8 VFs)
[   10.268210] pci 000a:03:00.0: 16.000 Gb/s available PCIe bandwidth, limited by 5 GT/s x4 link at 000a:00:05.0 (capable of 63.012 Gb/s with 16 GT/s x4 link)
[   10.282183] pci_bus 000a:00: on NUMA node 1
[   10.282190] pci 000a:00:05.0: BAR 15: assigned [mem 0x680000000000-0x6811ffffffff 64bit pref]
[   10.290700] pci 000a:00:05.0: BAR 14: no space for [mem size 0x08c00000]
[   10.297397] pci 000a:00:05.0: BAR 14: failed to assign [mem size 0x08c00000]
[   10.304431] pci 000a:00:01.0: BAR 14: assigned [mem 0x50000000-0x501fffff]
[   10.311293] pci 000a:00:01.0: BAR 15: assigned [mem 0x681200000000-0x6812001fffff 64bit pref]
[   10.319802] pci 000a:00:03.0: BAR 14: assigned [mem 0x50200000-0x503fffff]
[   10.326672] pci 000a:00:03.0: BAR 15: assigned [mem 0x681200200000-0x6812003fffff 64bit pref]
[   10.335182] pci 000a:00:01.0: BAR 13: no space for [io  size 0x1000]
[   10.341525] pci 000a:00:01.0: BAR 13: failed to assign [io  size 0x1000]
[   10.348212] pci 000a:00:03.0: BAR 13: no space for [io  size 0x1000]
[   10.354561] pci 000a:00:03.0: BAR 13: failed to assign [io  size 0x1000]
[   10.361247] pci 000a:00:05.0: BAR 13: no space for [io  size 0x1000]
[   10.367596] pci 000a:00:05.0: BAR 13: failed to assign [io  size 0x1000]
[   10.374283] pci 000a:00:01.0: PCI bridge to [bus 01]
[   10.379241] pci 000a:00:01.0:   bridge window [mem 0x50000000-0x501fffff]
[   10.386016] pci 000a:00:01.0:   bridge window [mem 0x681200000000-0x6812001fffff 64bit pref]
[   10.394450] pci 000a:00:03.0: PCI bridge to [bus 02]
[   10.399403] pci 000a:00:03.0:   bridge window [mem 0x50200000-0x503fffff]
[   10.406187] pci 000a:00:03.0:   bridge window [mem 0x681200200000-0x6812003fffff 64bit pref]
[   10.414613] pci 000a:03:00.0: BAR 4: assigned [mem 0x680000000000-0x6801ffffffff 64bit pref]
[   10.423052] pci 000a:03:00.0: BAR 11: assigned [mem 0x680200000000-0x6811ffffffff 64bit pref]
[   10.431566] pci 000a:03:00.0: BAR 9: no space for [mem size 0x08000000]
[   10.438176] pci 000a:03:00.0: BAR 9: failed to assign [mem size 0x08000000]
[   10.445124] pci 000a:03:00.0: BAR 2: no space for [mem size 0x00800000]
[   10.451725] pci 000a:03:00.0: BAR 2: failed to assign [mem size 0x00800000]
[   10.458673] pci 000a:03:00.0: BAR 7: no space for [mem size 0x00080000]
[   10.465283] pci 000a:03:00.0: BAR 7: failed to assign [mem size 0x00080000]
[   10.472230] pci 000a:03:00.0: BAR 0: no space for [mem size 0x00001000]
[   10.478839] pci 000a:03:00.0: BAR 0: failed to assign [mem size 0x00001000]
[   10.485787] pci 000a:00:05.0: PCI bridge to [bus 03]
[   10.490747] pci 000a:00:05.0:   bridge window [mem 0x680000000000-0x6811ffffffff 64bit pref]

BIOS可以正常分配资源(因为BIOS里面把SRIOV disable了)

进OS之后,OS会重新给卡分配资源,这时Linux检测到卡支持SRIOV,所以会分配VF资源,最终导致资源不够。

如果是windows的话可能就不会再去分配了,直接使用BIOS分配的) 

        Region 0: Memory at <ignored> (32-bit, non-prefetchable) [disabled]
        Region 2: Memory at <ignored> (32-bit, non-prefetchable) [disabled]
        Region 4: Memory at 680000000000 (64-bit, prefetchable) [disabled] [size=8G]

 

 

 

ee