1 ============ 2 Introduction 3 ============ 4 5 The RapidIO standard is a packet-based fabric interconnect standard designed for 6 use in embedded systems. Development of the RapidIO standard is directed by the 7 RapidIO Trade Association (RTA). The current version of the RapidIO specification 8 is publicly available for download from the RTA web-site [1]. 9 10 This document describes the basics of the Linux RapidIO subsystem and provides 11 information on its major components. 12 13 1 Overview 14 ========== 15 16 Because the RapidIO subsystem follows the Linux device model it is integrated 17 into the kernel similarly to other buses by defining RapidIO-specific device and 18 bus types and registering them within the device model. 19 20 The Linux RapidIO subsystem is architecture independent and therefore defines 21 architecture-specific interfaces that provide support for common RapidIO 22 subsystem operations. 23 24 2. Core Components 25 ================== 26 27 A typical RapidIO network is a combination of endpoints and switches. 28 Each of these components is represented in the subsystem by an associated data 29 structure. The core logical components of the RapidIO subsystem are defined 30 in include/linux/rio.h file. 31 32 2.1 Master Port 33 --------------- 34 35 A master port (or mport) is a RapidIO interface controller that is local to the 36 processor executing the Linux code. A master port generates and receives RapidIO 37 packets (transactions). In the RapidIO subsystem each master port is represented 38 by a rio_mport data structure. This structure contains master port specific 39 resources such as mailboxes and doorbells. The rio_mport also includes a unique 40 host device ID that is valid when a master port is configured as an enumerating 41 host. 42 43 RapidIO master ports are serviced by subsystem specific mport device drivers 44 that provide functionality defined for this subsystem. To provide a hardware 45 independent interface for RapidIO subsystem operations, rio_mport structure 46 includes rio_ops data structure which contains pointers to hardware specific 47 implementations of RapidIO functions. 48 49 2.2 Device 50 ---------- 51 52 A RapidIO device is any endpoint (other than mport) or switch in the network. 53 All devices are presented in the RapidIO subsystem by corresponding rio_dev data 54 structure. Devices form one global device list and per-network device lists 55 (depending on number of available mports and networks). 56 57 2.3 Switch 58 ---------- 59 60 A RapidIO switch is a special class of device that routes packets between its 61 ports towards their final destination. The packet destination port within a 62 switch is defined by an internal routing table. A switch is presented in the 63 RapidIO subsystem by rio_dev data structure expanded by additional rio_switch 64 data structure, which contains switch specific information such as copy of the 65 routing table and pointers to switch specific functions. 66 67 The RapidIO subsystem defines the format and initialization method for subsystem 68 specific switch drivers that are designed to provide hardware-specific 69 implementation of common switch management routines. 70 71 2.4 Network 72 ----------- 73 74 A RapidIO network is a combination of interconnected endpoint and switch devices. 75 Each RapidIO network known to the system is represented by corresponding rio_net 76 data structure. This structure includes lists of all devices and local master 77 ports that form the same network. It also contains a pointer to the default 78 master port that is used to communicate with devices within the network. 79 80 2.5 Device Drivers 81 ------------------ 82 83 RapidIO device-specific drivers follow Linux Kernel Driver Model and are 84 intended to support specific RapidIO devices attached to the RapidIO network. 85 86 2.6 Subsystem Interfaces 87 ------------------------ 88 89 RapidIO interconnect specification defines features that may be used to provide 90 one or more common service layers for all participating RapidIO devices. These 91 common services may act separately from device-specific drivers or be used by 92 device-specific drivers. Example of such service provider is the RIONET driver 93 which implements Ethernet-over-RapidIO interface. Because only one driver can be 94 registered for a device, all common RapidIO services have to be registered as 95 subsystem interfaces. This allows to have multiple common services attached to 96 the same device without blocking attachment of a device-specific driver. 97 98 3. Subsystem Initialization 99 =========================== 100 101 In order to initialize the RapidIO subsystem, a platform must initialize and 102 register at least one master port within the RapidIO network. To register mport 103 within the subsystem controller driver's initialization code calls function 104 rio_register_mport() for each available master port. 105 106 After all active master ports are registered with a RapidIO subsystem, 107 an enumeration and/or discovery routine may be called automatically or 108 by user-space command. 109 110 RapidIO subsystem can be configured to be built as a statically linked or 111 modular component of the kernel (see details below). 112 113 4. Enumeration and Discovery 114 ============================ 115 116 4.1 Overview 117 ------------ 118 119 RapidIO subsystem configuration options allow users to build enumeration and 120 discovery methods as statically linked components or loadable modules. 121 An enumeration/discovery method implementation and available input parameters 122 define how any given method can be attached to available RapidIO mports: 123 simply to all available mports OR individually to the specified mport device. 124 125 Depending on selected enumeration/discovery build configuration, there are 126 several methods to initiate an enumeration and/or discovery process: 127 128 (a) Statically linked enumeration and discovery process can be started 129 automatically during kernel initialization time using corresponding module 130 parameters. This was the original method used since introduction of RapidIO 131 subsystem. Now this method relies on enumerator module parameter which is 132 'rio-scan.scan' for existing basic enumeration/discovery method. 133 When automatic start of enumeration/discovery is used a user has to ensure 134 that all discovering endpoints are started before the enumerating endpoint 135 and are waiting for enumeration to be completed. 136 Configuration option CONFIG_RAPIDIO_DISC_TIMEOUT defines time that discovering 137 endpoint waits for enumeration to be completed. If the specified timeout 138 expires the discovery process is terminated without obtaining RapidIO network 139 information. NOTE: a timed out discovery process may be restarted later using 140 a user-space command as it is described below (if the given endpoint was 141 enumerated successfully). 142 143 (b) Statically linked enumeration and discovery process can be started by 144 a command from user space. This initiation method provides more flexibility 145 for a system startup compared to the option (a) above. After all participating 146 endpoints have been successfully booted, an enumeration process shall be 147 started first by issuing a user-space command, after an enumeration is 148 completed a discovery process can be started on all remaining endpoints. 149 150 (c) Modular enumeration and discovery process can be started by a command from 151 user space. After an enumeration/discovery module is loaded, a network scan 152 process can be started by issuing a user-space command. 153 Similar to the option (b) above, an enumerator has to be started first. 154 155 (d) Modular enumeration and discovery process can be started by a module 156 initialization routine. In this case an enumerating module shall be loaded 157 first. 158 159 When a network scan process is started it calls an enumeration or discovery 160 routine depending on the configured role of a master port: host or agent. 161 162 Enumeration is performed by a master port if it is configured as a host port by 163 assigning a host destination ID greater than or equal to zero. The host 164 destination ID can be assigned to a master port using various methods depending 165 on RapidIO subsystem build configuration: 166 167 (a) For a statically linked RapidIO subsystem core use command line parameter 168 "rapidio.hdid=" with a list of destination ID assignments in order of mport 169 device registration. For example, in a system with two RapidIO controllers 170 the command line parameter "rapidio.hdid=-1,7" will result in assignment of 171 the host destination ID=7 to the second RapidIO controller, while the first 172 one will be assigned destination ID=-1. 173 174 (b) If the RapidIO subsystem core is built as a loadable module, in addition 175 to the method shown above, the host destination ID(s) can be specified using 176 traditional methods of passing module parameter "hdid=" during its loading: 177 178 - from command line: "modprobe rapidio hdid=-1,7", or 179 - from modprobe configuration file using configuration command "options", 180 like in this example: "options rapidio hdid=-1,7". An example of modprobe 181 configuration file is provided in the section below. 182 183 NOTES: 184 (i) if "hdid=" parameter is omitted all available mport will be assigned 185 destination ID = -1; 186 187 (ii) the "hdid=" parameter in systems with multiple mports can have 188 destination ID assignments omitted from the end of list (default = -1). 189 190 If the host device ID for a specific master port is set to -1, the discovery 191 process will be performed for it. 192 193 The enumeration and discovery routines use RapidIO maintenance transactions 194 to access the configuration space of devices. 195 196 NOTE: If RapidIO switch-specific device drivers are built as loadable modules 197 they must be loaded before enumeration/discovery process starts. 198 This requirement is cased by the fact that enumeration/discovery methods invoke 199 vendor-specific callbacks on early stages. 200 201 4.2 Automatic Start of Enumeration and Discovery 202 ------------------------------------------------ 203 204 Automatic enumeration/discovery start method is applicable only to built-in 205 enumeration/discovery RapidIO configuration selection. To enable automatic 206 enumeration/discovery start by existing basic enumerator method set use boot 207 command line parameter "rio-scan.scan=1". 208 209 This configuration requires synchronized start of all RapidIO endpoints that 210 form a network which will be enumerated/discovered. Discovering endpoints have 211 to be started before an enumeration starts to ensure that all RapidIO 212 controllers have been initialized and are ready to be discovered. Configuration 213 parameter CONFIG_RAPIDIO_DISC_TIMEOUT defines time (in seconds) which 214 a discovering endpoint will wait for enumeration to be completed. 215 216 When automatic enumeration/discovery start is selected, basic method's 217 initialization routine calls rio_init_mports() to perform enumeration or 218 discovery for all known mport devices. 219 220 Depending on RapidIO network size and configuration this automatic 221 enumeration/discovery start method may be difficult to use due to the 222 requirement for synchronized start of all endpoints. 223 224 4.3 User-space Start of Enumeration and Discovery 225 ------------------------------------------------- 226 227 User-space start of enumeration and discovery can be used with built-in and 228 modular build configurations. For user-space controlled start RapidIO subsystem 229 creates the sysfs write-only attribute file '/sys/bus/rapidio/scan'. To initiate 230 an enumeration or discovery process on specific mport device, a user needs to 231 write mport_ID (not RapidIO destination ID) into that file. The mport_ID is a 232 sequential number (0 ... RIO_MAX_MPORTS) assigned during mport device 233 registration. For example for machine with single RapidIO controller, mport_ID 234 for that controller always will be 0. 235 236 To initiate RapidIO enumeration/discovery on all available mports a user may 237 write '-1' (or RIO_MPORT_ANY) into the scan attribute file. 238 239 4.4 Basic Enumeration Method 240 ---------------------------- 241 242 This is an original enumeration/discovery method which is available since 243 first release of RapidIO subsystem code. The enumeration process is 244 implemented according to the enumeration algorithm outlined in the RapidIO 245 Interconnect Specification: Annex I [1]. 246 247 This method can be configured as statically linked or loadable module. 248 The method's single parameter "scan" allows to trigger the enumeration/discovery 249 process from module initialization routine. 250 251 This enumeration/discovery method can be started only once and does not support 252 unloading if it is built as a module. 253 254 The enumeration process traverses the network using a recursive depth-first 255 algorithm. When a new device is found, the enumerator takes ownership of that 256 device by writing into the Host Device ID Lock CSR. It does this to ensure that 257 the enumerator has exclusive right to enumerate the device. If device ownership 258 is successfully acquired, the enumerator allocates a new rio_dev structure and 259 initializes it according to device capabilities. 260 261 If the device is an endpoint, a unique device ID is assigned to it and its value 262 is written into the device's Base Device ID CSR. 263 264 If the device is a switch, the enumerator allocates an additional rio_switch 265 structure to store switch specific information. Then the switch's vendor ID and 266 device ID are queried against a table of known RapidIO switches. Each switch 267 table entry contains a pointer to a switch-specific initialization routine that 268 initializes pointers to the rest of switch specific operations, and performs 269 hardware initialization if necessary. A RapidIO switch does not have a unique 270 device ID; it relies on hopcount and routing for device ID of an attached 271 endpoint if access to its configuration registers is required. If a switch (or 272 chain of switches) does not have any endpoint (except enumerator) attached to 273 it, a fake device ID will be assigned to configure a route to that switch. 274 In the case of a chain of switches without endpoint, one fake device ID is used 275 to configure a route through the entire chain and switches are differentiated by 276 their hopcount value. 277 278 For both endpoints and switches the enumerator writes a unique component tag 279 into device's Component Tag CSR. That unique value is used by the error 280 management notification mechanism to identify a device that is reporting an 281 error management event. 282 283 Enumeration beyond a switch is completed by iterating over each active egress 284 port of that switch. For each active link, a route to a default device ID 285 (0xFF for 8-bit systems and 0xFFFF for 16-bit systems) is temporarily written 286 into the routing table. The algorithm recurs by calling itself with hopcount + 1 287 and the default device ID in order to access the device on the active port. 288 289 After the host has completed enumeration of the entire network it releases 290 devices by clearing device ID locks (calls rio_clear_locks()). For each endpoint 291 in the system, it sets the Discovered bit in the Port General Control CSR 292 to indicate that enumeration is completed and agents are allowed to execute 293 passive discovery of the network. 294 295 The discovery process is performed by agents and is similar to the enumeration 296 process that is described above. However, the discovery process is performed 297 without changes to the existing routing because agents only gather information 298 about RapidIO network structure and are building an internal map of discovered 299 devices. This way each Linux-based component of the RapidIO subsystem has 300 a complete view of the network. The discovery process can be performed 301 simultaneously by several agents. After initializing its RapidIO master port 302 each agent waits for enumeration completion by the host for the configured wait 303 time period. If this wait time period expires before enumeration is completed, 304 an agent skips RapidIO discovery and continues with remaining kernel 305 initialization. 306 307 4.5 Adding New Enumeration/Discovery Method 308 ------------------------------------------- 309 310 RapidIO subsystem code organization allows addition of new enumeration/discovery 311 methods as new configuration options without significant impact to the core 312 RapidIO code. 313 314 A new enumeration/discovery method has to be attached to one or more mport 315 devices before an enumeration/discovery process can be started. Normally, 316 method's module initialization routine calls rio_register_scan() to attach 317 an enumerator to a specified mport device (or devices). The basic enumerator 318 implementation demonstrates this process. 319 320 4.6 Using Loadable RapidIO Switch Drivers 321 ----------------------------------------- 322 323 In the case when RapidIO switch drivers are built as loadable modules a user 324 must ensure that they are loaded before the enumeration/discovery starts. 325 This process can be automated by specifying pre- or post- dependencies in the 326 RapidIO-specific modprobe configuration file as shown in the example below. 327 328 File /etc/modprobe.d/rapidio.conf:: 329 330 # Configure RapidIO subsystem modules 331 332 # Set enumerator host destination ID (overrides kernel command line option) 333 options rapidio hdid=-1,2 334 335 # Load RapidIO switch drivers immediately after rapidio core module was loaded 336 softdep rapidio post: idt_gen2 idtcps tsi57x 337 338 # OR : 339 340 # Load RapidIO switch drivers just before rio-scan enumerator module is loaded 341 softdep rio-scan pre: idt_gen2 idtcps tsi57x 342 343 -------------------------- 344 345 NOTE: 346 In the example above, one of "softdep" commands must be removed or 347 commented out to keep required module loading sequence. 348 349 5. References 350 ============= 351 352 [1] RapidIO Trade Association. RapidIO Interconnect Specifications. 353 http://www.rapidio.org. 354 355 [2] Rapidio TA. Technology Comparisons. 356 http://www.rapidio.org/education/technology_comparisons/ 357 358 [3] RapidIO support for Linux. 359 https://lwn.net/Articles/139118/ 360 361 [4] Matt Porter. RapidIO for Linux. Ottawa Linux Symposium, 2005 362 https://www.kernel.org/doc/ols/2005/ols2005v2-pages-43-56.pdf
Linux® is a registered trademark of Linus Torvalds in the United States and other countries.
TOMOYO® is a registered trademark of NTT DATA CORPORATION.