1 .. SPDX-License-Identifier: GPL-2.0 OR GFDL-1. !! 1 .. This file is dual-licensed: you can use it either under the terms
2 .. c:namespace:: MC !! 2 .. of the GPL 2.0 or the GFDL 1.1+ license, at your option. Note that this
>> 3 .. dual licensing only applies to this file, and not this project as a
>> 4 .. whole.
>> 5 ..
>> 6 .. a) This file is free software; you can redistribute it and/or
>> 7 .. modify it under the terms of the GNU General Public License as
>> 8 .. published by the Free Software Foundation version 2 of
>> 9 .. the License.
>> 10 ..
>> 11 .. This file is distributed in the hope that it will be useful,
>> 12 .. but WITHOUT ANY WARRANTY; without even the implied warranty of
>> 13 .. MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> 14 .. GNU General Public License for more details.
>> 15 ..
>> 16 .. Or, alternatively,
>> 17 ..
>> 18 .. b) Permission is granted to copy, distribute and/or modify this
>> 19 .. document under the terms of the GNU Free Documentation License,
>> 20 .. Version 1.1 or any later version published by the Free Software
>> 21 .. Foundation, with no Invariant Sections, no Front-Cover Texts
>> 22 .. and no Back-Cover Texts. A copy of the license is included at
>> 23 .. Documentation/userspace-api/media/fdl-appendix.rst.
>> 24 ..
>> 25 .. TODO: replace it to GPL-2.0 OR GFDL-1.1-or-later WITH no-invariant-sections
3 26
4 .. _media-request-api: 27 .. _media-request-api:
5 28
6 Request API 29 Request API
7 =========== 30 ===========
8 31
9 The Request API has been designed to allow V4L 32 The Request API has been designed to allow V4L2 to deal with requirements of
10 modern devices (stateless codecs, complex came 33 modern devices (stateless codecs, complex camera pipelines, ...) and APIs
11 (Android Codec v2). One such requirement is th 34 (Android Codec v2). One such requirement is the ability for devices belonging to
12 the same pipeline to reconfigure and collabora 35 the same pipeline to reconfigure and collaborate closely on a per-frame basis.
13 Another is support of stateless codecs, which 36 Another is support of stateless codecs, which require controls to be applied
14 to specific frames (aka 'per-frame controls') 37 to specific frames (aka 'per-frame controls') in order to be used efficiently.
15 38
16 While the initial use-case was V4L2, it can be 39 While the initial use-case was V4L2, it can be extended to other subsystems
17 as well, as long as they use the media control 40 as well, as long as they use the media controller.
18 41
19 Supporting these features without the Request 42 Supporting these features without the Request API is not always possible and if
20 it is, it is terribly inefficient: user-space 43 it is, it is terribly inefficient: user-space would have to flush all activity
21 on the media pipeline, reconfigure it for the 44 on the media pipeline, reconfigure it for the next frame, queue the buffers to
22 be processed with that configuration, and wait 45 be processed with that configuration, and wait until they are all available for
23 dequeuing before considering the next frame. T 46 dequeuing before considering the next frame. This defeats the purpose of having
24 buffer queues since in practice only one buffe 47 buffer queues since in practice only one buffer would be queued at a time.
25 48
26 The Request API allows a specific configuratio 49 The Request API allows a specific configuration of the pipeline (media
27 controller topology + configuration for each m 50 controller topology + configuration for each media entity) to be associated with
28 specific buffers. This allows user-space to sc 51 specific buffers. This allows user-space to schedule several tasks ("requests")
29 with different configurations in advance, know 52 with different configurations in advance, knowing that the configuration will be
30 applied when needed to get the expected result 53 applied when needed to get the expected result. Configuration values at the time
31 of request completion are also available for r 54 of request completion are also available for reading.
32 55
33 General Usage 56 General Usage
34 ------------- 57 -------------
35 58
36 The Request API extends the Media Controller A 59 The Request API extends the Media Controller API and cooperates with
37 subsystem-specific APIs to support request usa 60 subsystem-specific APIs to support request usage. At the Media Controller
38 level, requests are allocated from the support 61 level, requests are allocated from the supporting Media Controller device
39 node. Their life cycle is then managed through 62 node. Their life cycle is then managed through the request file descriptors in
40 an opaque way. Configuration data, buffer hand 63 an opaque way. Configuration data, buffer handles and processing results
41 stored in requests are accessed through subsys 64 stored in requests are accessed through subsystem-specific APIs extended for
42 request support, such as V4L2 APIs that take a 65 request support, such as V4L2 APIs that take an explicit ``request_fd``
43 parameter. 66 parameter.
44 67
45 Request Allocation 68 Request Allocation
46 ------------------ 69 ------------------
47 70
48 User-space allocates requests using :ref:`MEDI 71 User-space allocates requests using :ref:`MEDIA_IOC_REQUEST_ALLOC`
49 for the media device node. This returns a file 72 for the media device node. This returns a file descriptor representing the
50 request. Typically, several such requests will 73 request. Typically, several such requests will be allocated.
51 74
52 Request Preparation 75 Request Preparation
53 ------------------- 76 -------------------
54 77
55 Standard V4L2 ioctls can then receive a reques 78 Standard V4L2 ioctls can then receive a request file descriptor to express the
56 fact that the ioctl is part of said request, a 79 fact that the ioctl is part of said request, and is not to be applied
57 immediately. See :ref:`MEDIA_IOC_REQUEST_ALLOC 80 immediately. See :ref:`MEDIA_IOC_REQUEST_ALLOC` for a list of ioctls that
58 support this. Configurations set with a ``requ 81 support this. Configurations set with a ``request_fd`` parameter are stored
59 instead of being immediately applied, and buff 82 instead of being immediately applied, and buffers queued to a request do not
60 enter the regular buffer queue until the reque 83 enter the regular buffer queue until the request itself is queued.
61 84
62 Request Submission 85 Request Submission
63 ------------------ 86 ------------------
64 87
65 Once the configuration and buffers of the requ 88 Once the configuration and buffers of the request are specified, it can be
66 queued by calling :ref:`MEDIA_REQUEST_IOC_QUEU 89 queued by calling :ref:`MEDIA_REQUEST_IOC_QUEUE` on the request file descriptor.
67 A request must contain at least one buffer, ot 90 A request must contain at least one buffer, otherwise ``ENOENT`` is returned.
68 A queued request cannot be modified anymore. 91 A queued request cannot be modified anymore.
69 92
70 .. caution:: 93 .. caution::
71 For :ref:`memory-to-memory devices <mem2mem 94 For :ref:`memory-to-memory devices <mem2mem>` you can use requests only for
72 output buffers, not for capture buffers. At 95 output buffers, not for capture buffers. Attempting to add a capture buffer
73 to a request will result in an ``EBADR`` er 96 to a request will result in an ``EBADR`` error.
74 97
75 If the request contains configurations for mul 98 If the request contains configurations for multiple entities, individual drivers
76 may synchronize so the requested pipeline's to 99 may synchronize so the requested pipeline's topology is applied before the
77 buffers are processed. Media controller driver 100 buffers are processed. Media controller drivers do a best effort implementation
78 since perfect atomicity may not be possible du 101 since perfect atomicity may not be possible due to hardware limitations.
79 102
80 .. caution:: 103 .. caution::
81 104
82 It is not allowed to mix queuing requests w 105 It is not allowed to mix queuing requests with directly queuing buffers:
83 whichever method is used first locks this i 106 whichever method is used first locks this in place until
84 :ref:`VIDIOC_STREAMOFF <VIDIOC_STREAMON>` i 107 :ref:`VIDIOC_STREAMOFF <VIDIOC_STREAMON>` is called or the device is
85 :ref:`closed <func-close>`. Attempts to dir 108 :ref:`closed <func-close>`. Attempts to directly queue a buffer when earlier
86 a buffer was queued via a request or vice v 109 a buffer was queued via a request or vice versa will result in an ``EBUSY``
87 error. 110 error.
88 111
89 Controls can still be set without a request an 112 Controls can still be set without a request and are applied immediately,
90 regardless of whether a request is in use or n 113 regardless of whether a request is in use or not.
91 114
92 .. caution:: 115 .. caution::
93 116
94 Setting the same control through a request 117 Setting the same control through a request and also directly can lead to
95 undefined behavior! 118 undefined behavior!
96 119
97 User-space can :c:func:`poll()` a request file !! 120 User-space can :ref:`poll() <request-func-poll>` a request file descriptor in
98 order to wait until the request completes. A r 121 order to wait until the request completes. A request is considered complete
99 once all its associated buffers are available 122 once all its associated buffers are available for dequeuing and all the
100 associated controls have been updated with the 123 associated controls have been updated with the values at the time of completion.
101 Note that user-space does not need to wait for 124 Note that user-space does not need to wait for the request to complete to
102 dequeue its buffers: buffers that are availabl 125 dequeue its buffers: buffers that are available halfway through a request can
103 be dequeued independently of the request's sta 126 be dequeued independently of the request's state.
104 127
105 A completed request contains the state of the 128 A completed request contains the state of the device after the request was
106 executed. User-space can query that state by c 129 executed. User-space can query that state by calling
107 :ref:`ioctl VIDIOC_G_EXT_CTRLS <VIDIOC_G_EXT_C 130 :ref:`ioctl VIDIOC_G_EXT_CTRLS <VIDIOC_G_EXT_CTRLS>` with the request file
108 descriptor. Calling :ref:`ioctl VIDIOC_G_EXT_C 131 descriptor. Calling :ref:`ioctl VIDIOC_G_EXT_CTRLS <VIDIOC_G_EXT_CTRLS>` for a
109 request that has been queued but not yet compl 132 request that has been queued but not yet completed will return ``EBUSY``
110 since the control values might be changed at a 133 since the control values might be changed at any time by the driver while the
111 request is in flight. 134 request is in flight.
112 135
113 .. _media-request-life-time: 136 .. _media-request-life-time:
114 137
115 Recycling and Destruction 138 Recycling and Destruction
116 ------------------------- 139 -------------------------
117 140
118 Finally, a completed request can either be dis 141 Finally, a completed request can either be discarded or be reused. Calling
119 :c:func:`close()` on a request file descriptor !! 142 :ref:`close() <request-func-close>` on a request file descriptor will make
120 that file descriptor unusable and the request 143 that file descriptor unusable and the request will be freed once it is no
121 longer in use by the kernel. That is, if the r 144 longer in use by the kernel. That is, if the request is queued and then the
122 file descriptor is closed, then it won't be fr 145 file descriptor is closed, then it won't be freed until the driver completed
123 the request. 146 the request.
124 147
125 The :ref:`MEDIA_REQUEST_IOC_REINIT` will clear 148 The :ref:`MEDIA_REQUEST_IOC_REINIT` will clear a request's state and make it
126 available again. No state is retained by this 149 available again. No state is retained by this operation: the request is as
127 if it had just been allocated. 150 if it had just been allocated.
128 151
129 Example for a Codec Device 152 Example for a Codec Device
130 -------------------------- 153 --------------------------
131 154
132 For use-cases such as :ref:`codecs <mem2mem>`, 155 For use-cases such as :ref:`codecs <mem2mem>`, the request API can be used
133 to associate specific controls to 156 to associate specific controls to
134 be applied by the driver for the OUTPUT buffer 157 be applied by the driver for the OUTPUT buffer, allowing user-space
135 to queue many such buffers in advance. It can 158 to queue many such buffers in advance. It can also take advantage of requests'
136 ability to capture the state of controls when 159 ability to capture the state of controls when the request completes to read back
137 information that may be subject to change. 160 information that may be subject to change.
138 161
139 Put into code, after obtaining a request, user 162 Put into code, after obtaining a request, user-space can assign controls and one
140 OUTPUT buffer to it: 163 OUTPUT buffer to it:
141 164
142 .. code-block:: c 165 .. code-block:: c
143 166
144 struct v4l2_buffer buf; 167 struct v4l2_buffer buf;
145 struct v4l2_ext_controls ctrls; 168 struct v4l2_ext_controls ctrls;
146 int req_fd; 169 int req_fd;
147 ... 170 ...
148 if (ioctl(media_fd, MEDIA_IOC_REQUEST_ 171 if (ioctl(media_fd, MEDIA_IOC_REQUEST_ALLOC, &req_fd))
149 return errno; 172 return errno;
150 ... 173 ...
151 ctrls.which = V4L2_CTRL_WHICH_REQUEST_ 174 ctrls.which = V4L2_CTRL_WHICH_REQUEST_VAL;
152 ctrls.request_fd = req_fd; 175 ctrls.request_fd = req_fd;
153 if (ioctl(codec_fd, VIDIOC_S_EXT_CTRLS 176 if (ioctl(codec_fd, VIDIOC_S_EXT_CTRLS, &ctrls))
154 return errno; 177 return errno;
155 ... 178 ...
156 buf.type = V4L2_BUF_TYPE_VIDEO_OUTPUT; 179 buf.type = V4L2_BUF_TYPE_VIDEO_OUTPUT;
157 buf.flags |= V4L2_BUF_FLAG_REQUEST_FD; 180 buf.flags |= V4L2_BUF_FLAG_REQUEST_FD;
158 buf.request_fd = req_fd; 181 buf.request_fd = req_fd;
159 if (ioctl(codec_fd, VIDIOC_QBUF, &buf) 182 if (ioctl(codec_fd, VIDIOC_QBUF, &buf))
160 return errno; 183 return errno;
161 184
162 Note that it is not allowed to use the Request 185 Note that it is not allowed to use the Request API for CAPTURE buffers
163 since there are no per-frame settings to repor 186 since there are no per-frame settings to report there.
164 187
165 Once the request is fully prepared, it can be 188 Once the request is fully prepared, it can be queued to the driver:
166 189
167 .. code-block:: c 190 .. code-block:: c
168 191
169 if (ioctl(req_fd, MEDIA_REQUEST_IOC_QU 192 if (ioctl(req_fd, MEDIA_REQUEST_IOC_QUEUE))
170 return errno; 193 return errno;
171 194
172 User-space can then either wait for the reques 195 User-space can then either wait for the request to complete by calling poll() on
173 its file descriptor, or start dequeuing CAPTUR 196 its file descriptor, or start dequeuing CAPTURE buffers. Most likely, it will
174 want to get CAPTURE buffers as soon as possibl 197 want to get CAPTURE buffers as soon as possible and this can be done using a
175 regular :ref:`VIDIOC_DQBUF <VIDIOC_QBUF>`: 198 regular :ref:`VIDIOC_DQBUF <VIDIOC_QBUF>`:
176 199
177 .. code-block:: c 200 .. code-block:: c
178 201
179 struct v4l2_buffer buf; 202 struct v4l2_buffer buf;
180 203
181 memset(&buf, 0, sizeof(buf)); 204 memset(&buf, 0, sizeof(buf));
182 buf.type = V4L2_BUF_TYPE_VIDEO_CAPTURE 205 buf.type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
183 if (ioctl(codec_fd, VIDIOC_DQBUF, &buf 206 if (ioctl(codec_fd, VIDIOC_DQBUF, &buf))
184 return errno; 207 return errno;
185 208
186 Note that this example assumes for simplicity 209 Note that this example assumes for simplicity that for every OUTPUT buffer
187 there will be one CAPTURE buffer, but this doe 210 there will be one CAPTURE buffer, but this does not have to be the case.
188 211
189 We can then, after ensuring that the request i 212 We can then, after ensuring that the request is completed via polling the
190 request file descriptor, query control values 213 request file descriptor, query control values at the time of its completion via
191 a call to :ref:`VIDIOC_G_EXT_CTRLS <VIDIOC_G_E 214 a call to :ref:`VIDIOC_G_EXT_CTRLS <VIDIOC_G_EXT_CTRLS>`.
192 This is particularly useful for volatile contr 215 This is particularly useful for volatile controls for which we want to
193 query values as soon as the capture buffer is 216 query values as soon as the capture buffer is produced.
194 217
195 .. code-block:: c 218 .. code-block:: c
196 219
197 struct pollfd pfd = { .events = POLLPR 220 struct pollfd pfd = { .events = POLLPRI, .fd = req_fd };
198 poll(&pfd, 1, -1); 221 poll(&pfd, 1, -1);
199 ... 222 ...
200 ctrls.which = V4L2_CTRL_WHICH_REQUEST_ 223 ctrls.which = V4L2_CTRL_WHICH_REQUEST_VAL;
201 ctrls.request_fd = req_fd; 224 ctrls.request_fd = req_fd;
202 if (ioctl(codec_fd, VIDIOC_G_EXT_CTRLS 225 if (ioctl(codec_fd, VIDIOC_G_EXT_CTRLS, &ctrls))
203 return errno; 226 return errno;
204 227
205 Once we don't need the request anymore, we can 228 Once we don't need the request anymore, we can either recycle it for reuse with
206 :ref:`MEDIA_REQUEST_IOC_REINIT`... 229 :ref:`MEDIA_REQUEST_IOC_REINIT`...
207 230
208 .. code-block:: c 231 .. code-block:: c
209 232
210 if (ioctl(req_fd, MEDIA_REQUEST_IOC_RE 233 if (ioctl(req_fd, MEDIA_REQUEST_IOC_REINIT))
211 return errno; 234 return errno;
212 235
213 ... or close its file descriptor to completely 236 ... or close its file descriptor to completely dispose of it.
214 237
215 .. code-block:: c 238 .. code-block:: c
216 239
217 close(req_fd); 240 close(req_fd);
218 241
219 Example for a Simple Capture Device 242 Example for a Simple Capture Device
220 ----------------------------------- 243 -----------------------------------
221 244
222 With a simple capture device, requests can be 245 With a simple capture device, requests can be used to specify controls to apply
223 for a given CAPTURE buffer. 246 for a given CAPTURE buffer.
224 247
225 .. code-block:: c 248 .. code-block:: c
226 249
227 struct v4l2_buffer buf; 250 struct v4l2_buffer buf;
228 struct v4l2_ext_controls ctrls; 251 struct v4l2_ext_controls ctrls;
229 int req_fd; 252 int req_fd;
230 ... 253 ...
231 if (ioctl(media_fd, MEDIA_IOC_REQUEST_ 254 if (ioctl(media_fd, MEDIA_IOC_REQUEST_ALLOC, &req_fd))
232 return errno; 255 return errno;
233 ... 256 ...
234 ctrls.which = V4L2_CTRL_WHICH_REQUEST_ 257 ctrls.which = V4L2_CTRL_WHICH_REQUEST_VAL;
235 ctrls.request_fd = req_fd; 258 ctrls.request_fd = req_fd;
236 if (ioctl(camera_fd, VIDIOC_S_EXT_CTRL 259 if (ioctl(camera_fd, VIDIOC_S_EXT_CTRLS, &ctrls))
237 return errno; 260 return errno;
238 ... 261 ...
239 buf.type = V4L2_BUF_TYPE_VIDEO_CAPTURE 262 buf.type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
240 buf.flags |= V4L2_BUF_FLAG_REQUEST_FD; 263 buf.flags |= V4L2_BUF_FLAG_REQUEST_FD;
241 buf.request_fd = req_fd; 264 buf.request_fd = req_fd;
242 if (ioctl(camera_fd, VIDIOC_QBUF, &buf 265 if (ioctl(camera_fd, VIDIOC_QBUF, &buf))
243 return errno; 266 return errno;
244 267
245 Once the request is fully prepared, it can be 268 Once the request is fully prepared, it can be queued to the driver:
246 269
247 .. code-block:: c 270 .. code-block:: c
248 271
249 if (ioctl(req_fd, MEDIA_REQUEST_IOC_QU 272 if (ioctl(req_fd, MEDIA_REQUEST_IOC_QUEUE))
250 return errno; 273 return errno;
251 274
252 User-space can then dequeue buffers, wait for 275 User-space can then dequeue buffers, wait for the request completion, query
253 controls and recycle the request as in the M2M 276 controls and recycle the request as in the M2M example above.
Linux® is a registered trademark of Linus Torvalds in the United States and other countries.
TOMOYO® is a registered trademark of NTT DATA CORPORATION.