PCIE Tutorial: Enumeration

Make it to the Right and Larger Audience

Press Release

PCIE Tutorial: Enumeration

We are going to write a series of PCIE blogs. Here is the 1st one about PCIE enumeration. When system first boots up and PCIE enumeration is not done yet, our example PCIE system looks like below.

pcie_enum1

Some observation. We have root complex (RC) which connects to host CPU via host/PCI bridge. Note root complex also resides on host side. Inside root complex, we have two devices, dev 0 and dev 1. Dev 0 is a multi-function device and consists of function 0 and function 1. Dev 1 is single function device. RC dev 0 and dev 1 are not end point devices and they are of bridge type, also called PCIE to PCIE (P2P) virtual bridge.

There is a PCIE switch in the system which is connected to RC dev0 func0 via a PCIE link. This PCIE switch has four P2P bridges and three downstream bridges are connected to a PCIE end point respectively.

Note
1. As highlighted in diagram, all devices attached to downstream side of a PCIE link must be device 0.
2. PCIE link is a point to point connection and P2P bridge, either in RC or in switch, is needed to connected multiple PCIE devices. However, the connection among P2P bridges, either inside RC or inside switch, is multi-drop and it is NOT a PCIE link.

What PCIE enumeration does is to assign PCIE bus number to each PCIE link and P2P bridge connection, and properly fill up secondary bus number and subordinate bus number inside each P2P bridge so that software running in CPU can uniquely identify each PCIE device and P2P bridges combined can properly route the transaction to the correct target PCIE device.

 

Now let’s get started. Software first assigns bus 0 to RC connection. Then it reads bus 0 dev0 configuration space and figures this device is a bridge since configuration header is type 1. Then it assigns bus 1 to this device’s downstream PCIE link and updates secondary (sec) bus number to be 1 and subordinate (sub) bus number to be 255. Sec bus number specifies the minimum bus number in the tree under this bridge and sub bus number is the max bus number. Software uses 255 for now since it hasn’t gone through the tree yet so it doesn’t know how many devices down there.

pcie_enum2
Software reads bus 1 dev0 and figures it is a P2P bridge and then assigns bus 2 to its down stream link (note it is not a PCIE link but a connection among P2P bridges) and updates bus 1 dev0 sec bus number to 2 and sub bus number to 255.

pcie_enum3

 

Software reads bus 2 dev 0 and figures it is a P2P bridge. It does the same thing as above, assign bus 3 and updates sec bus num to 3 and sub bus num to 255.

pcie_enum4

 

 
Profile Photo
We are a couple of RTL design engineers with combined more than 40 years of experience in ASIC and FPGA digital design. We successfully led several chips through the whole process from feature define, RTL design, verification, to backend/DFT support, TO, and bring-up. We are familiar with Xilinx FPGA design too and have experience in using multiple FPGAs to verify complicated ASIC RTL design or for the final product. We are interested in working as independent contractor for your projects. Please send us private message if there is a match. Thank you!
2 Comments
  1. chandra_vs9 1 year ago
    0
    -0

    when two devices are there on same bus, say bus 1, how device number is decided ? when enumeration SW tries to read on B/F 1,0,0 which of these two devices responds ?

    0
  2. maheswarudu 1 year ago
    0
    -0

    Nice explanation.

    0

Contact Us

Thanks for helping us better serve the community. You can make a suggestion, report a bug, a misconduct, or any other issue. We'll get back to you using your private message ASAP.

Sending

©2017  ValPont.com

Forgot your details?