We are going to write a series of PCIE blogs. Here is the 1st one about PCIE enumeration. When system first boots up and PCIE enumeration is not done yet, our example PCIE system looks like below.
Some observation. We have root complex (RC) which connects to host CPU via host/PCI bridge. Note root complex also resides on host side. Inside root complex, we have two devices, dev 0 and dev 1. Dev 0 is a multi-function device and consists of function 0 and function 1. Dev 1 is single function device. RC dev 0 and dev 1 are not end point devices and they are of bridge type, also called PCIE to PCIE (P2P) virtual bridge.
There is a PCIE switch in the system which is connected to RC dev0 func0 via a PCIE link. This PCIE switch has four P2P bridges and three downstream bridges are connected to a PCIE end point respectively.
1. As highlighted in diagram, all devices attached to downstream side of a PCIE link must be device 0.
2. PCIE link is a point to point connection and P2P bridge, either in RC or in switch, is needed to connected multiple PCIE devices. However, the connection among P2P bridges, either inside RC or inside switch, is multi-drop and it is NOT a PCIE link.
What PCIE enumeration does is to assign PCIE bus number to each PCIE link and P2P bridge connection, and properly fill up secondary bus number and subordinate bus number inside each P2P bridge so that software running in CPU can uniquely identify each PCIE device and P2P bridges combined can properly route the transaction to the correct target PCIE device.
Now let’s get started. Software first assigns bus 0 to RC connection. Then it reads bus 0 dev0 configuration space and figures this device is a bridge since configuration header is type 1. Then it assigns bus 1 to this device’s downstream PCIE link and updates secondary (sec) bus number to be 1 and subordinate (sub) bus number to be 255. Sec bus number specifies the minimum bus number in the tree under this bridge and sub bus number is the max bus number. Software uses 255 for now since it hasn’t gone through the tree yet so it doesn’t know how many devices down there.
Software reads bus 1 dev0 and figures it is a P2P bridge and then assigns bus 2 to its down stream link (note it is not a PCIE link but a connection among P2P bridges) and updates bus 1 dev0 sec bus number to 2 and sub bus number to 255.
Software reads bus 2 dev 0 and figures it is a P2P bridge. It does the same thing as above, assign bus 3 and updates sec bus num to 3 and sub bus num to 255.