1 .. SPDX-License-Identifier: GPL-2.0 OR GFDL-1. 1 .. SPDX-License-Identifier: GPL-2.0 OR GFDL-1.1-no-invariants-or-later
2 .. c:namespace:: MC 2 .. c:namespace:: MC
3 3
4 .. _media-request-api: 4 .. _media-request-api:
5 5
6 Request API 6 Request API
7 =========== 7 ===========
8 8
9 The Request API has been designed to allow V4L 9 The Request API has been designed to allow V4L2 to deal with requirements of
10 modern devices (stateless codecs, complex came 10 modern devices (stateless codecs, complex camera pipelines, ...) and APIs
11 (Android Codec v2). One such requirement is th 11 (Android Codec v2). One such requirement is the ability for devices belonging to
12 the same pipeline to reconfigure and collabora 12 the same pipeline to reconfigure and collaborate closely on a per-frame basis.
13 Another is support of stateless codecs, which 13 Another is support of stateless codecs, which require controls to be applied
14 to specific frames (aka 'per-frame controls') 14 to specific frames (aka 'per-frame controls') in order to be used efficiently.
15 15
16 While the initial use-case was V4L2, it can be 16 While the initial use-case was V4L2, it can be extended to other subsystems
17 as well, as long as they use the media control 17 as well, as long as they use the media controller.
18 18
19 Supporting these features without the Request 19 Supporting these features without the Request API is not always possible and if
20 it is, it is terribly inefficient: user-space 20 it is, it is terribly inefficient: user-space would have to flush all activity
21 on the media pipeline, reconfigure it for the 21 on the media pipeline, reconfigure it for the next frame, queue the buffers to
22 be processed with that configuration, and wait 22 be processed with that configuration, and wait until they are all available for
23 dequeuing before considering the next frame. T 23 dequeuing before considering the next frame. This defeats the purpose of having
24 buffer queues since in practice only one buffe 24 buffer queues since in practice only one buffer would be queued at a time.
25 25
26 The Request API allows a specific configuratio 26 The Request API allows a specific configuration of the pipeline (media
27 controller topology + configuration for each m 27 controller topology + configuration for each media entity) to be associated with
28 specific buffers. This allows user-space to sc 28 specific buffers. This allows user-space to schedule several tasks ("requests")
29 with different configurations in advance, know 29 with different configurations in advance, knowing that the configuration will be
30 applied when needed to get the expected result 30 applied when needed to get the expected result. Configuration values at the time
31 of request completion are also available for r 31 of request completion are also available for reading.
32 32
33 General Usage 33 General Usage
34 ------------- 34 -------------
35 35
36 The Request API extends the Media Controller A 36 The Request API extends the Media Controller API and cooperates with
37 subsystem-specific APIs to support request usa 37 subsystem-specific APIs to support request usage. At the Media Controller
38 level, requests are allocated from the support 38 level, requests are allocated from the supporting Media Controller device
39 node. Their life cycle is then managed through 39 node. Their life cycle is then managed through the request file descriptors in
40 an opaque way. Configuration data, buffer hand 40 an opaque way. Configuration data, buffer handles and processing results
41 stored in requests are accessed through subsys 41 stored in requests are accessed through subsystem-specific APIs extended for
42 request support, such as V4L2 APIs that take a 42 request support, such as V4L2 APIs that take an explicit ``request_fd``
43 parameter. 43 parameter.
44 44
45 Request Allocation 45 Request Allocation
46 ------------------ 46 ------------------
47 47
48 User-space allocates requests using :ref:`MEDI 48 User-space allocates requests using :ref:`MEDIA_IOC_REQUEST_ALLOC`
49 for the media device node. This returns a file 49 for the media device node. This returns a file descriptor representing the
50 request. Typically, several such requests will 50 request. Typically, several such requests will be allocated.
51 51
52 Request Preparation 52 Request Preparation
53 ------------------- 53 -------------------
54 54
55 Standard V4L2 ioctls can then receive a reques 55 Standard V4L2 ioctls can then receive a request file descriptor to express the
56 fact that the ioctl is part of said request, a 56 fact that the ioctl is part of said request, and is not to be applied
57 immediately. See :ref:`MEDIA_IOC_REQUEST_ALLOC 57 immediately. See :ref:`MEDIA_IOC_REQUEST_ALLOC` for a list of ioctls that
58 support this. Configurations set with a ``requ 58 support this. Configurations set with a ``request_fd`` parameter are stored
59 instead of being immediately applied, and buff 59 instead of being immediately applied, and buffers queued to a request do not
60 enter the regular buffer queue until the reque 60 enter the regular buffer queue until the request itself is queued.
61 61
62 Request Submission 62 Request Submission
63 ------------------ 63 ------------------
64 64
65 Once the configuration and buffers of the requ 65 Once the configuration and buffers of the request are specified, it can be
66 queued by calling :ref:`MEDIA_REQUEST_IOC_QUEU 66 queued by calling :ref:`MEDIA_REQUEST_IOC_QUEUE` on the request file descriptor.
67 A request must contain at least one buffer, ot 67 A request must contain at least one buffer, otherwise ``ENOENT`` is returned.
68 A queued request cannot be modified anymore. 68 A queued request cannot be modified anymore.
69 69
70 .. caution:: 70 .. caution::
71 For :ref:`memory-to-memory devices <mem2mem 71 For :ref:`memory-to-memory devices <mem2mem>` you can use requests only for
72 output buffers, not for capture buffers. At 72 output buffers, not for capture buffers. Attempting to add a capture buffer
73 to a request will result in an ``EBADR`` er 73 to a request will result in an ``EBADR`` error.
74 74
75 If the request contains configurations for mul 75 If the request contains configurations for multiple entities, individual drivers
76 may synchronize so the requested pipeline's to 76 may synchronize so the requested pipeline's topology is applied before the
77 buffers are processed. Media controller driver 77 buffers are processed. Media controller drivers do a best effort implementation
78 since perfect atomicity may not be possible du 78 since perfect atomicity may not be possible due to hardware limitations.
79 79
80 .. caution:: 80 .. caution::
81 81
82 It is not allowed to mix queuing requests w 82 It is not allowed to mix queuing requests with directly queuing buffers:
83 whichever method is used first locks this i 83 whichever method is used first locks this in place until
84 :ref:`VIDIOC_STREAMOFF <VIDIOC_STREAMON>` i 84 :ref:`VIDIOC_STREAMOFF <VIDIOC_STREAMON>` is called or the device is
85 :ref:`closed <func-close>`. Attempts to dir 85 :ref:`closed <func-close>`. Attempts to directly queue a buffer when earlier
86 a buffer was queued via a request or vice v 86 a buffer was queued via a request or vice versa will result in an ``EBUSY``
87 error. 87 error.
88 88
89 Controls can still be set without a request an 89 Controls can still be set without a request and are applied immediately,
90 regardless of whether a request is in use or n 90 regardless of whether a request is in use or not.
91 91
92 .. caution:: 92 .. caution::
93 93
94 Setting the same control through a request 94 Setting the same control through a request and also directly can lead to
95 undefined behavior! 95 undefined behavior!
96 96
97 User-space can :c:func:`poll()` a request file 97 User-space can :c:func:`poll()` a request file descriptor in
98 order to wait until the request completes. A r 98 order to wait until the request completes. A request is considered complete
99 once all its associated buffers are available 99 once all its associated buffers are available for dequeuing and all the
100 associated controls have been updated with the 100 associated controls have been updated with the values at the time of completion.
101 Note that user-space does not need to wait for 101 Note that user-space does not need to wait for the request to complete to
102 dequeue its buffers: buffers that are availabl 102 dequeue its buffers: buffers that are available halfway through a request can
103 be dequeued independently of the request's sta 103 be dequeued independently of the request's state.
104 104
105 A completed request contains the state of the 105 A completed request contains the state of the device after the request was
106 executed. User-space can query that state by c 106 executed. User-space can query that state by calling
107 :ref:`ioctl VIDIOC_G_EXT_CTRLS <VIDIOC_G_EXT_C 107 :ref:`ioctl VIDIOC_G_EXT_CTRLS <VIDIOC_G_EXT_CTRLS>` with the request file
108 descriptor. Calling :ref:`ioctl VIDIOC_G_EXT_C 108 descriptor. Calling :ref:`ioctl VIDIOC_G_EXT_CTRLS <VIDIOC_G_EXT_CTRLS>` for a
109 request that has been queued but not yet compl 109 request that has been queued but not yet completed will return ``EBUSY``
110 since the control values might be changed at a 110 since the control values might be changed at any time by the driver while the
111 request is in flight. 111 request is in flight.
112 112
113 .. _media-request-life-time: 113 .. _media-request-life-time:
114 114
115 Recycling and Destruction 115 Recycling and Destruction
116 ------------------------- 116 -------------------------
117 117
118 Finally, a completed request can either be dis 118 Finally, a completed request can either be discarded or be reused. Calling
119 :c:func:`close()` on a request file descriptor 119 :c:func:`close()` on a request file descriptor will make
120 that file descriptor unusable and the request 120 that file descriptor unusable and the request will be freed once it is no
121 longer in use by the kernel. That is, if the r 121 longer in use by the kernel. That is, if the request is queued and then the
122 file descriptor is closed, then it won't be fr 122 file descriptor is closed, then it won't be freed until the driver completed
123 the request. 123 the request.
124 124
125 The :ref:`MEDIA_REQUEST_IOC_REINIT` will clear 125 The :ref:`MEDIA_REQUEST_IOC_REINIT` will clear a request's state and make it
126 available again. No state is retained by this 126 available again. No state is retained by this operation: the request is as
127 if it had just been allocated. 127 if it had just been allocated.
128 128
129 Example for a Codec Device 129 Example for a Codec Device
130 -------------------------- 130 --------------------------
131 131
132 For use-cases such as :ref:`codecs <mem2mem>`, 132 For use-cases such as :ref:`codecs <mem2mem>`, the request API can be used
133 to associate specific controls to 133 to associate specific controls to
134 be applied by the driver for the OUTPUT buffer 134 be applied by the driver for the OUTPUT buffer, allowing user-space
135 to queue many such buffers in advance. It can 135 to queue many such buffers in advance. It can also take advantage of requests'
136 ability to capture the state of controls when 136 ability to capture the state of controls when the request completes to read back
137 information that may be subject to change. 137 information that may be subject to change.
138 138
139 Put into code, after obtaining a request, user 139 Put into code, after obtaining a request, user-space can assign controls and one
140 OUTPUT buffer to it: 140 OUTPUT buffer to it:
141 141
142 .. code-block:: c 142 .. code-block:: c
143 143
144 struct v4l2_buffer buf; 144 struct v4l2_buffer buf;
145 struct v4l2_ext_controls ctrls; 145 struct v4l2_ext_controls ctrls;
146 int req_fd; 146 int req_fd;
147 ... 147 ...
148 if (ioctl(media_fd, MEDIA_IOC_REQUEST_ 148 if (ioctl(media_fd, MEDIA_IOC_REQUEST_ALLOC, &req_fd))
149 return errno; 149 return errno;
150 ... 150 ...
151 ctrls.which = V4L2_CTRL_WHICH_REQUEST_ 151 ctrls.which = V4L2_CTRL_WHICH_REQUEST_VAL;
152 ctrls.request_fd = req_fd; 152 ctrls.request_fd = req_fd;
153 if (ioctl(codec_fd, VIDIOC_S_EXT_CTRLS 153 if (ioctl(codec_fd, VIDIOC_S_EXT_CTRLS, &ctrls))
154 return errno; 154 return errno;
155 ... 155 ...
156 buf.type = V4L2_BUF_TYPE_VIDEO_OUTPUT; 156 buf.type = V4L2_BUF_TYPE_VIDEO_OUTPUT;
157 buf.flags |= V4L2_BUF_FLAG_REQUEST_FD; 157 buf.flags |= V4L2_BUF_FLAG_REQUEST_FD;
158 buf.request_fd = req_fd; 158 buf.request_fd = req_fd;
159 if (ioctl(codec_fd, VIDIOC_QBUF, &buf) 159 if (ioctl(codec_fd, VIDIOC_QBUF, &buf))
160 return errno; 160 return errno;
161 161
162 Note that it is not allowed to use the Request 162 Note that it is not allowed to use the Request API for CAPTURE buffers
163 since there are no per-frame settings to repor 163 since there are no per-frame settings to report there.
164 164
165 Once the request is fully prepared, it can be 165 Once the request is fully prepared, it can be queued to the driver:
166 166
167 .. code-block:: c 167 .. code-block:: c
168 168
169 if (ioctl(req_fd, MEDIA_REQUEST_IOC_QU 169 if (ioctl(req_fd, MEDIA_REQUEST_IOC_QUEUE))
170 return errno; 170 return errno;
171 171
172 User-space can then either wait for the reques 172 User-space can then either wait for the request to complete by calling poll() on
173 its file descriptor, or start dequeuing CAPTUR 173 its file descriptor, or start dequeuing CAPTURE buffers. Most likely, it will
174 want to get CAPTURE buffers as soon as possibl 174 want to get CAPTURE buffers as soon as possible and this can be done using a
175 regular :ref:`VIDIOC_DQBUF <VIDIOC_QBUF>`: 175 regular :ref:`VIDIOC_DQBUF <VIDIOC_QBUF>`:
176 176
177 .. code-block:: c 177 .. code-block:: c
178 178
179 struct v4l2_buffer buf; 179 struct v4l2_buffer buf;
180 180
181 memset(&buf, 0, sizeof(buf)); 181 memset(&buf, 0, sizeof(buf));
182 buf.type = V4L2_BUF_TYPE_VIDEO_CAPTURE 182 buf.type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
183 if (ioctl(codec_fd, VIDIOC_DQBUF, &buf 183 if (ioctl(codec_fd, VIDIOC_DQBUF, &buf))
184 return errno; 184 return errno;
185 185
186 Note that this example assumes for simplicity 186 Note that this example assumes for simplicity that for every OUTPUT buffer
187 there will be one CAPTURE buffer, but this doe 187 there will be one CAPTURE buffer, but this does not have to be the case.
188 188
189 We can then, after ensuring that the request i 189 We can then, after ensuring that the request is completed via polling the
190 request file descriptor, query control values 190 request file descriptor, query control values at the time of its completion via
191 a call to :ref:`VIDIOC_G_EXT_CTRLS <VIDIOC_G_E 191 a call to :ref:`VIDIOC_G_EXT_CTRLS <VIDIOC_G_EXT_CTRLS>`.
192 This is particularly useful for volatile contr 192 This is particularly useful for volatile controls for which we want to
193 query values as soon as the capture buffer is 193 query values as soon as the capture buffer is produced.
194 194
195 .. code-block:: c 195 .. code-block:: c
196 196
197 struct pollfd pfd = { .events = POLLPR 197 struct pollfd pfd = { .events = POLLPRI, .fd = req_fd };
198 poll(&pfd, 1, -1); 198 poll(&pfd, 1, -1);
199 ... 199 ...
200 ctrls.which = V4L2_CTRL_WHICH_REQUEST_ 200 ctrls.which = V4L2_CTRL_WHICH_REQUEST_VAL;
201 ctrls.request_fd = req_fd; 201 ctrls.request_fd = req_fd;
202 if (ioctl(codec_fd, VIDIOC_G_EXT_CTRLS 202 if (ioctl(codec_fd, VIDIOC_G_EXT_CTRLS, &ctrls))
203 return errno; 203 return errno;
204 204
205 Once we don't need the request anymore, we can 205 Once we don't need the request anymore, we can either recycle it for reuse with
206 :ref:`MEDIA_REQUEST_IOC_REINIT`... 206 :ref:`MEDIA_REQUEST_IOC_REINIT`...
207 207
208 .. code-block:: c 208 .. code-block:: c
209 209
210 if (ioctl(req_fd, MEDIA_REQUEST_IOC_RE 210 if (ioctl(req_fd, MEDIA_REQUEST_IOC_REINIT))
211 return errno; 211 return errno;
212 212
213 ... or close its file descriptor to completely 213 ... or close its file descriptor to completely dispose of it.
214 214
215 .. code-block:: c 215 .. code-block:: c
216 216
217 close(req_fd); 217 close(req_fd);
218 218
219 Example for a Simple Capture Device 219 Example for a Simple Capture Device
220 ----------------------------------- 220 -----------------------------------
221 221
222 With a simple capture device, requests can be 222 With a simple capture device, requests can be used to specify controls to apply
223 for a given CAPTURE buffer. 223 for a given CAPTURE buffer.
224 224
225 .. code-block:: c 225 .. code-block:: c
226 226
227 struct v4l2_buffer buf; 227 struct v4l2_buffer buf;
228 struct v4l2_ext_controls ctrls; 228 struct v4l2_ext_controls ctrls;
229 int req_fd; 229 int req_fd;
230 ... 230 ...
231 if (ioctl(media_fd, MEDIA_IOC_REQUEST_ 231 if (ioctl(media_fd, MEDIA_IOC_REQUEST_ALLOC, &req_fd))
232 return errno; 232 return errno;
233 ... 233 ...
234 ctrls.which = V4L2_CTRL_WHICH_REQUEST_ 234 ctrls.which = V4L2_CTRL_WHICH_REQUEST_VAL;
235 ctrls.request_fd = req_fd; 235 ctrls.request_fd = req_fd;
236 if (ioctl(camera_fd, VIDIOC_S_EXT_CTRL 236 if (ioctl(camera_fd, VIDIOC_S_EXT_CTRLS, &ctrls))
237 return errno; 237 return errno;
238 ... 238 ...
239 buf.type = V4L2_BUF_TYPE_VIDEO_CAPTURE 239 buf.type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
240 buf.flags |= V4L2_BUF_FLAG_REQUEST_FD; 240 buf.flags |= V4L2_BUF_FLAG_REQUEST_FD;
241 buf.request_fd = req_fd; 241 buf.request_fd = req_fd;
242 if (ioctl(camera_fd, VIDIOC_QBUF, &buf 242 if (ioctl(camera_fd, VIDIOC_QBUF, &buf))
243 return errno; 243 return errno;
244 244
245 Once the request is fully prepared, it can be 245 Once the request is fully prepared, it can be queued to the driver:
246 246
247 .. code-block:: c 247 .. code-block:: c
248 248
249 if (ioctl(req_fd, MEDIA_REQUEST_IOC_QU 249 if (ioctl(req_fd, MEDIA_REQUEST_IOC_QUEUE))
250 return errno; 250 return errno;
251 251
252 User-space can then dequeue buffers, wait for 252 User-space can then dequeue buffers, wait for the request completion, query
253 controls and recycle the request as in the M2M 253 controls and recycle the request as in the M2M example above.
Linux® is a registered trademark of Linus Torvalds in the United States and other countries.
TOMOYO® is a registered trademark of NTT DATA CORPORATION.