TEXT   35

Open Coherent Accelerator Processor Interface

Guest on 13th May 2022 02:01:48 AM

  1. ========================================================
  2. OpenCAPI (Open Coherent Accelerator Processor Interface)
  3. ========================================================
  4.  
  5. OpenCAPI is an interface between processors and accelerators. It aims
  6. at being low-latency and high-bandwidth. The specification is
  7. developed by the `OpenCAPI Consortium <http://opencapi.org/>`_.
  8.  
  9. It allows an accelerator (which could be a FPGA, ASICs, ...) to access
  10. the host memory coherently, using virtual addresses. An OpenCAPI
  11. device can also host its own memory, that can be accessed from the
  12. host.
  13.  
  14. OpenCAPI is known in linux as 'ocxl', as the open, processor-agnostic
  15. evolution of 'cxl' (the driver for the IBM CAPI interface for
  16. powerpc), which was named that way to avoid confusion with the ISDN
  17. CAPI subsystem.
  18.  
  19.  
  20. High-level view
  21. ===============
  22.  
  23. OpenCAPI defines a Data Link Layer (DL) and Transaction Layer (TL), to
  24. be implemented on top of a physical link. Any processor or device
  25. implementing the DL and TL can start sharing memory.
  26.  
  27. ::
  28.  
  29.   +-----------+                         +-------------+
  30.   |           |                         |             |
  31.   |           |                         | Accelerated |
  32.   | Processor |                         |  Function   |
  33.   |           |  +--------+             |    Unit     |  +--------+
  34.   |           |--| Memory |             |    (AFU)    |--| Memory |
  35.   |           |  +--------+             |             |  +--------+
  36.   +-----------+                         +-------------+
  37.        |                                       |
  38.   +-----------+                         +-------------+
  39.   |    TL     |                         |    TLX      |
  40.   +-----------+                         +-------------+
  41.        |                                       |
  42.   +-----------+                         +-------------+
  43.   |    DL     |                         |    DLX      |
  44.   +-----------+                         +-------------+
  45.        |                                       |
  46.        |                   PHY                 |
  47.        +---------------------------------------+
  48.  
  49.  
  50.  
  51. Device discovery
  52. ================
  53.  
  54. OpenCAPI relies on a PCI-like configuration space, implemented on the
  55. device. So the host can discover AFUs by querying the config space.
  56.  
  57. OpenCAPI devices in Linux are treated like PCI devices (with a few
  58. caveats). The firmware is expected to abstract the hardware as if it
  59. was a PCI link. A lot of the existing PCI infrastructure is reused:
  60. devices are scanned and BARs are assigned during the standard PCI
  61. enumeration. Commands like 'lspci' can therefore be used to see what
  62. devices are available.
  63.  
  64. The configuration space defines the AFU(s) that can be found on the
  65. physical adapter, such as its name, how many memory contexts it can
  66. work with, the size of its MMIO areas, ...
  67.  
  68.  
  69.  
  70. MMIO
  71. ====
  72.  
  73. OpenCAPI defines two MMIO areas for each AFU:
  74.  
  75. * the global MMIO area, with registers pertinent to the whole AFU.
  76. * a per-process MMIO area, which has a fixed size for each context.
  77.  
  78.  
  79.  
  80. AFU interrupts
  81. ==============
  82.  
  83. OpenCAPI includes the possibility for an AFU to send an interrupt to a
  84. host process. It is done through a 'intrp_req' defined in the
  85. Transaction Layer, specifying a 64-bit object handle which defines the
  86. interrupt.
  87.  
  88. The driver allows a process to allocate an interrupt and obtain its
  89. 64-bit object handle, that can be passed to the AFU.
  90.  
  91.  
  92.  
  93. char devices
  94. ============
  95.  
  96. The driver creates one char device per AFU found on the physical
  97. device. A physical device may have multiple functions and each
  98. function can have multiple AFUs. At the time of this writing though,
  99. it has only been tested with devices exporting only one AFU.
  100.  
  101. Char devices can be found in /dev/ocxl/ and are named as:
  102. /dev/ocxl/<AFU name>.<location>.<index>
  103.  
  104. where <AFU name> is a max 20-character long name, as found in the
  105. config space of the AFU.
  106. <location> is added by the driver and can help distinguish devices
  107. when a system has more than one instance of the same OpenCAPI device.
  108. <index> is also to help distinguish AFUs in the unlikely case where a
  109. device carries multiple copies of the same AFU.
  110.  
  111.  
  112.  
  113. Sysfs class
  114. ===========
  115.  
  116. An ocxl class is added for the devices representing the AFUs. See
  117. /sys/class/ocxl. The layout is described in
  118. Documentation/ABI/testing/sysfs-class-ocxl
  119.  
  120.  
  121.  
  122. User API
  123. ========
  124.  
  125. open
  126. ----
  127.  
  128. Based on the AFU definition found in the config space, an AFU may
  129. support working with more than one memory context, in which case the
  130. associated char device may be opened multiple times by different
  131. processes.
  132.  
  133.  
  134. ioctl
  135. -----
  136.  
  137. OCXL_IOCTL_ATTACH:
  138.  
  139.   Attach the memory context of the calling process to the AFU so that
  140.   the AFU can access its memory.
  141.  
  142. OCXL_IOCTL_IRQ_ALLOC:
  143.  
  144.   Allocate an AFU interrupt and return an identifier.
  145.  
  146. OCXL_IOCTL_IRQ_FREE:
  147.  
  148.   Free a previously allocated AFU interrupt.
  149.  
  150. OCXL_IOCTL_IRQ_SET_FD:
  151.  
  152.   Associate an event fd to an AFU interrupt so that the user process
  153.   can be notified when the AFU sends an interrupt.
  154.  
  155. OCXL_IOCTL_GET_METADATA:
  156.  
  157.   Obtains configuration information from the card, such at the size of
  158.   MMIO areas, the AFU version, and the PASID for the current context.
  159.  
  160. OCXL_IOCTL_ENABLE_P9_WAIT:
  161.  
  162.   Allows the AFU to wake a userspace thread executing 'wait'. Returns
  163.   information to userspace to allow it to configure the AFU. Note that
  164.   this is only available on POWER9.
  165.  
  166. OCXL_IOCTL_GET_FEATURES:
  167.  
  168.   Reports on which CPU features that affect OpenCAPI are usable from
  169.   userspace.
  170.  
  171.  
  172. mmap
  173. ----
  174.  
  175. A process can mmap the per-process MMIO area for interactions with the
  176. AFU.

Raw Paste


Login or Register to edit or fork this paste. It's free.