1 .. SPDX-License-Identifier: GFDL-1.1-no-invari
2
3 .. _subdev:
4
5 ********************
6 Sub-device Interface
7 ********************
8
9 The complex nature of V4L2 devices, where hard
10 several integrated circuits that need to inter
11 controlled way, leads to complex V4L2 drivers.
12 reflect the hardware model in software, and mo
13 components as software blocks called sub-devic
14
15 V4L2 sub-devices are usually kernel-only objec
16 implements the media device API, they will aut
17 media entities. Applications will be able to e
18 and discover the hardware topology using the m
19 links enumeration API.
20
21 In addition to make sub-devices discoverable,
22 make them directly configurable by application
23 sub-device driver and the V4L2 device driver s
24 will feature a character device node on which
25
26 - query, read and write sub-devices controls
27
28 - subscribe and unsubscribe to events and ret
29
30 - negotiate image formats on individual pads
31
32 - inspect and modify internal data routing be
33
34 Sub-device character device nodes, conventiona
35 ``/dev/v4l-subdev*``, use major number 81.
36
37 Drivers may opt to limit the sub-device charac
38 operations that do not modify the device state
39 are referred to as ``read-only`` in the rest o
40 related restrictions are documented in individ
41
42
43 Controls
44 ========
45
46 Most V4L2 controls are implemented by sub-devi
47 usually merge all controls and expose them thr
48 Applications can control all sub-devices throu
49
50 Complex devices sometimes implement the same c
51 of hardware. This situation is common in embed
52 sensors and image processing hardware implemen
53 such as contrast adjustment, white balance or
54 As the V4L2 controls API doesn't support sever
55 single device, all but one of the identical co
56
57 Applications can access those hidden controls
58 node with the V4L2 control API described in :r
59 behave identically as when issued on V4L2 devi
60 exception that they deal only with controls im
61 sub-device.
62
63 Depending on the driver, those controls might
64 one (or several) V4L2 device nodes.
65
66
67 Events
68 ======
69
70 V4L2 sub-devices can notify applications of ev
71 :ref:`event`. The API behaves identically as w
72 nodes, with the exception that it only deals w
73 the sub-device. Depending on the driver, those
74 reported on one (or several) V4L2 device nodes
75
76
77 .. _pad-level-formats:
78
79 Pad-level Formats
80 =================
81
82 .. warning::
83
84 Pad-level formats are only applicable to v
85 need to expose low-level format configurat
86 V4L2 applications do *not* need to use the
87 section.
88
89 .. note::
90
91 For the purpose of this section, the term
92 combination of media bus data format, fram
93
94 Image formats are typically negotiated on vide
95 devices using the format and
96 :ref:`selection <VIDIOC_SUBDEV_G_SELECTION>` i
97 responsible for configuring every block in the
98 to the requested format at the pipeline input
99
100 For complex devices, such as often found in em
101 image sizes at the output of a pipeline can be
102 hardware configurations. One such example is s
103 :ref:`pipeline-scaling`, where image scaling c
104 the video sensor and the host image processing
105
106
107 .. _pipeline-scaling:
108
109 .. kernel-figure:: pipeline.dot
110 :alt: pipeline.dot
111 :align: center
112
113 Image Format Negotiation on Pipelines
114
115 High quality and high speed pipeline confi
116
117
118
119 The sensor scaler is usually of less quality t
120 scaling on the sensor is required to achieve h
121 Depending on the use case (quality vs. speed),
122 configured differently. Applications need to c
123 every point in the pipeline explicitly.
124
125 Drivers that implement the :ref:`media API <me
126 can expose pad-level image format configuratio
127 they do, applications can use the
128 :ref:`VIDIOC_SUBDEV_G_FMT <VIDIOC_SUBDEV_G_FMT
129 :ref:`VIDIOC_SUBDEV_S_FMT <VIDIOC_SUBDEV_G_FMT
130 negotiate formats on a per-pad basis.
131
132 Applications are responsible for configuring c
133 whole pipeline and making sure that connected
134 formats. The pipeline is checked for formats m
135 :ref:`VIDIOC_STREAMON <VIDIOC_STREAMON>` time,
136 code is then returned if the configuration is
137
138 Pad-level image format configuration support c
139 the :ref:`VIDIOC_SUBDEV_G_FMT` ioctl on pad
140 0. If the driver returns an ``EINVAL`` error c
141 configuration is not supported by the sub-devi
142
143
144 Format Negotiation
145 ------------------
146
147 Acceptable formats on pads can (and usually do
148 external parameters, such as formats on other
149 even controls. Finding a combination of format
150 pipeline, acceptable to both application and d
151 formats enumeration only. A format negotiation
152
153 Central to the format negotiation mechanism ar
154 operations. When called with the ``which`` arg
155 :ref:`V4L2_SUBDEV_FORMAT_TRY <VIDIOC_SUBDEV_G_
156 :ref:`VIDIOC_SUBDEV_G_FMT <VIDIOC_SUBDEV_G_FMT
157 :ref:`VIDIOC_SUBDEV_S_FMT <VIDIOC_SUBDEV_G_FMT
158 a set of formats parameters that are not conne
159 configuration. Modifying those 'try' formats l
160 untouched (this applies to both the software s
161 and the hardware state stored in the device it
162
163 While not kept as part of the device state, tr
164 the sub-device file handles. A
165 :ref:`VIDIOC_SUBDEV_G_FMT <VIDIOC_SUBDEV_G_FMT
166 the last try format set *on the same sub-devic
167 applications querying the same sub-device at t
168 interact with each other.
169
170 To find out whether a particular format is sup
171 applications use the
172 :ref:`VIDIOC_SUBDEV_S_FMT <VIDIOC_SUBDEV_G_FMT
173 verify and, if needed, change the requested ``
174 requirements and return the possibly modified
175 then choose to try a different format or accep
176 continue.
177
178 Formats returned by the driver during a negoti
179 guaranteed to be supported by the device. In p
180 guarantee that a returned format will not be f
181 to an :ref:`VIDIOC_SUBDEV_S_FMT <VIDIOC_SUBDEV
182 (as long as external parameters, such as forma
183 configuration are not changed).
184
185 Drivers automatically propagate formats inside
186 or active format is set on a pad, correspondin
187 the same sub-device can be modified by the dri
188 modify formats as required by the device. Howe
189 with the following rules when possible:
190
191 - Formats should be propagated from sink pads
192 a format on a source pad should not modify
193 pad.
194
195 - Sub-devices that scale frames using variabl
196 reset the scale factors to default values w
197 modified. If the 1:1 scaling ratio is suppo
198 source pads formats should be reset to the
199
200 Formats are not propagated across links, as th
201 propagating them from one sub-device file hand
202 Applications must then take care to configure
203 explicitly with compatible formats. Identical
204 a link are guaranteed to be compatible. Driver
205 different formats matching device requirements
206
207 :ref:`sample-pipeline-config` shows a sample c
208 for the pipeline described in :ref:`pipeline-s
209 list entity names and pad numbers).
210
211
212 .. raw:: latex
213
214 \begingroup
215 \scriptsize
216 \setlength{\tabcolsep}{2pt}
217
218 .. tabularcolumns:: |p{2.0cm}|p{2.1cm}|p{2.1cm
219
220 .. _sample-pipeline-config:
221
222 .. flat-table:: Sample Pipeline Configuration
223 :header-rows: 1
224 :stub-columns: 0
225 :widths: 5 5 5 5 5 5 5
226
227 * -
228 - Sensor/0
229
230 format
231 - Frontend/0
232
233 format
234 - Frontend/1
235
236 format
237 - Scaler/0
238
239 format
240 - Scaler/0
241
242 compose selection rectangle
243 - Scaler/1
244
245 format
246 * - Initial state
247 - 2048x1536
248
249 SGRBG8_1X8
250 - (default)
251 - (default)
252 - (default)
253 - (default)
254 - (default)
255 * - Configure frontend sink format
256 - 2048x1536
257
258 SGRBG8_1X8
259 - *2048x1536*
260
261 *SGRBG8_1X8*
262 - *2046x1534*
263
264 *SGRBG8_1X8*
265 - (default)
266 - (default)
267 - (default)
268 * - Configure scaler sink format
269 - 2048x1536
270
271 SGRBG8_1X8
272 - 2048x1536
273
274 SGRBG8_1X8
275 - 2046x1534
276
277 SGRBG8_1X8
278 - *2046x1534*
279
280 *SGRBG8_1X8*
281 - *0,0/2046x1534*
282 - *2046x1534*
283
284 *SGRBG8_1X8*
285 * - Configure scaler sink compose selectio
286 - 2048x1536
287
288 SGRBG8_1X8
289 - 2048x1536
290
291 SGRBG8_1X8
292 - 2046x1534
293
294 SGRBG8_1X8
295 - 2046x1534
296
297 SGRBG8_1X8
298 - *0,0/1280x960*
299 - *1280x960*
300
301 *SGRBG8_1X8*
302
303 .. raw:: latex
304
305 \endgroup
306
307 1. Initial state. The sensor source pad format
308 size and V4L2_MBUS_FMT_SGRBG8_1X8 media bus
309 host frontend and scaler sink and source pa
310 values, as well as the compose rectangle on
311
312 2. The application configures the frontend sin
313 2048x1536 and its media bus code to V4L2_MB
314 driver propagates the format to the fronten
315
316 3. The application configures the scaler sink
317 2046x1534 and the media bus code to V4L2_MB
318 match the frontend source size and media bu
319 on the sink pad is set to V4L2_MBUS_FMT_SGR
320 propagates the size to the compose selectio
321 scaler's sink pad, and the format to the sc
322
323 4. The application configures the size of the
324 rectangle of the scaler's sink pad 1280x960
325 the size to the scaler's source pad format.
326
327 When satisfied with the try results, applicati
328 formats by setting the ``which`` argument to
329 ``V4L2_SUBDEV_FORMAT_ACTIVE``. Active formats
330 formats by drivers. To avoid modifying the har
331 negotiation, applications should negotiate try
332 modify the active settings using the try forma
333 last negotiation iteration. This guarantees th
334 be applied as-is by the driver without being m
335
336
337 .. _v4l2-subdev-selections:
338
339 Selections: cropping, scaling and composition
340 ---------------------------------------------
341
342 Many sub-devices support cropping frames on th
343 (or possible even on both). Cropping is used t
344 interest in an image, typically on an image se
345 It can also be used as part of digital zoom im
346 the area of the image that will be scaled up.
347
348 Crop settings are defined by a crop rectangle
349 struct :c:type:`v4l2_rect` by the coordinates
350 left corner and the rectangle size. Both the c
351 expressed in pixels.
352
353 As for pad formats, drivers store try and acti
354 selection targets :ref:`v4l2-selections-common
355
356 On sink pads, cropping is applied relative to
357 The pad format represents the image size as re
358 from the previous block in the pipeline, and t
359 represents the sub-image that will be transmit
360 sub-device for processing.
361
362 The scaling operation changes the size of the
363 dimensions. The scaling ratio isn't specified
364 from the original and scaled image sizes. Both
365 struct :c:type:`v4l2_rect`.
366
367 Scaling support is optional. When supported by
368 rectangle on the subdev's sink pad is scaled t
369 using the
370 :ref:`VIDIOC_SUBDEV_S_SELECTION <VIDIOC_SUBDEV
371 using ``V4L2_SEL_TGT_COMPOSE`` selection targe
372 subdev supports scaling but not composing, the
373 not used and must always be set to zero.
374
375 On source pads, cropping is similar to sink pa
376 that the source size from which the cropping i
377 COMPOSE rectangle on the sink pad. In both sin
378 crop rectangle must be entirely contained insi
379 for the crop operation.
380
381 The drivers should always use the closest poss
382 requests on all selection targets, unless spec
383 ``V4L2_SEL_FLAG_GE`` and ``V4L2_SEL_FLAG_LE``
384 the image size either up or down. :ref:`v4l2-s
385
386
387 Types of selection targets
388 --------------------------
389
390
391 Actual targets
392 ^^^^^^^^^^^^^^
393
394 Actual targets (without a postfix) reflect the
395 configuration at any point of time. There is a
396 corresponding to every actual target.
397
398
399 BOUNDS targets
400 ^^^^^^^^^^^^^^
401
402 BOUNDS targets is the smallest rectangle that
403 rectangles. It may not be possible to set the
404 as the BOUNDS rectangle, however. This may be
405 pixel array is not rectangular but cross-shape
406 size may also be smaller than the BOUNDS recta
407
408
409 .. _format-propagation:
410
411 Order of configuration and format propagation
412 ---------------------------------------------
413
414 Inside subdevs, the order of image processing
415 the sink pad towards the source pad. This is a
416 in which the configuration must be performed b
417 made will be propagated to any subsequent stag
418 not desired, the user must set ``V4L2_SEL_FLAG
419 flag causes no propagation of the changes are
420 circumstances. This may also cause the accesse
421 by the driver, depending on the properties of
422
423 The coordinates to a step always refer to the
424 previous step. The exception to this rule is t
425 rectangle, which refers to the sink compose bo
426 is supported by the hardware.
427
428 1. Sink pad format. The user configures the si
429 defines the parameters of the image the ent
430 pad for further processing.
431
432 2. Sink pad actual crop selection. The sink pa
433 performed to the sink pad format.
434
435 3. Sink pad actual compose selection. The size
436 rectangle defines the scaling ratio compare
437 pad crop rectangle. The location of the com
438 the location of the actual sink compose rec
439 bounds rectangle.
440
441 4. Source pad actual crop selection. Crop on t
442 performed to the image in the sink compose
443
444 5. Source pad format. The source pad format de
445 format of the subdev, as well as the other
446 exception of the image width and height. Wi
447 by the size of the source pad actual crop s
448
449 Accessing any of the above rectangles not supp
450 return ``EINVAL``. Any rectangle referring to
451 rectangle coordinates will instead refer to th
452 rectangle. For example, if sink crop is not su
453 selection will refer to the sink pad format di
454
455
456 .. _subdev-image-processing-crop:
457
458 .. kernel-figure:: subdev-image-processing-cro
459 :alt: subdev-image-processing-crop.svg
460 :align: center
461
462 **Figure 4.5. Image processing in subdevs:
463
464 In the above example, the subdev supports crop
465 configure it, the user sets the media bus form
466 pad. Now the actual crop rectangle can be set
467 location and size of this rectangle reflect th
468 rectangle to be cropped from the sink format.
469 rectangle will also be the size of the format
470 pad.
471
472
473 .. _subdev-image-processing-scaling-multi-sour
474
475 .. kernel-figure:: subdev-image-processing-sca
476 :alt: subdev-image-processing-scaling-mu
477 :align: center
478
479 **Figure 4.6. Image processing in subdevs:
480
481 In this example, the subdev is capable of firs
482 and finally cropping for two source pads indiv
483 scaled image. The location of the scaled image
484 ignored in sink compose target. Both of the lo
485 rectangles refer to the sink scaling rectangle
486 an area at location specified by the source cr
487
488
489 .. _subdev-image-processing-full:
490
491 .. kernel-figure:: subdev-image-processing-ful
492 :alt: subdev-image-processing-full.svg
493 :align: center
494
495 **Figure 4.7. Image processing in subdevs:
496
497 The subdev driver supports two sink pads and t
498 from both of the sink pads are individually cr
499 further composed on the composition bounds rec
500 independent streams are cropped and sent out o
501 source pads.
502
503
504 .. toctree::
505 :maxdepth: 1
506
507 subdev-formats
508
509 .. _subdev-routing:
510
511 Streams, multiplexed media pads and internal r
512 ----------------------------------------------
513
514 Simple V4L2 sub-devices do not support multipl
515 and only a single stream can pass through a me
516 Thus each pad contains a format and selection
517 single stream. A subdev can do stream processi
518 two or compose two streams into one, but the i
519 subdev are still a single stream per pad.
520
521 Some hardware, e.g. MIPI CSI-2, support multip
522 data streams are transmitted on the same bus,
523 link connecting a transmitter source pad with
524 example, a camera sensor can produce two disti
525 metadata stream, which are transmitted on the
526 by a media link which connects the single sens
527 sink pad. The stream-aware receiver will de-mu
528 the its sink pad and allows to route them indi
529 pads.
530
531 Subdevice drivers that support multiplexed str
532 non-multiplexed subdev drivers. However, if th
533 does not support streams, then only stream 0 o
534 There may be additional limitations specific t
535
536 Understanding streams
537 ^^^^^^^^^^^^^^^^^^^^^
538
539 A stream is a stream of content (e.g. pixel da
540 the media pipeline from a source (e.g. a senso
541 receiver and demultiplexer in a SoC). Each med
542 streams from one end of the link to the other,
543 tables which describe how the incoming streams
544 source pads.
545
546 A stream ID is a media pad-local identifier fo
547 the same stream must be equal on both ends of
548 a particular stream ID must exist on both side
549 link, but another stream ID can be used for th
550 of the sub-device.
551
552 A stream at a specific point in the media pipe
553 sub-device and a (pad, stream) pair. For sub-d
554 multiplexed streams the 'stream' field is alwa
555
556 Interaction between routes, streams, formats a
557 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
558
559 The addition of streams to the V4L2 sub-device
560 formats and selections from pads to (pad, stre
561 usual pad, also the stream ID needs to be prov
562 selections. The order of configuring formats a
563 the same as without streams (see :ref:`format-
564
565 Instead of the sub-device wide merging of stre
566 towards all source pads, data flows for each r
567 other. Any number of routes from streams on si
568 source pads is allowed, to the extent supporte
569 stream on a source pad, however, only a single
570
571 Any configurations of a stream within a pad, s
572 are independent of similar configurations on o
573 subject to change in the future.
574
575 Device types and routing setup
576 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
577
578 Different kinds of sub-devices have differing
579 depending on the hardware. In all cases, howev
580 ``V4L2_SUBDEV_STREAM_FL_ACTIVE`` flag set are
581
582 Devices generating the streams may allow enabl
583 routes or have a fixed routing configuration.
584 declaring the routes (or declaring them withou
585 ``V4L2_SUBDEV_STREAM_FL_ACTIVE`` flag set) in
586 disable the routes. ``VIDIOC_SUBDEV_S_ROUTING`
587 back to the user in the routes array, with the
588 flag unset.
589
590 Devices transporting the streams almost always
591 respect to routing. Typically any route betwee
592 pads is possible, and multiple routes (usually
593 be active simultaneously. For such devices, no
594 and user-created routes are fully replaced whe
595 called on the sub-device. Such newly created r
596 configuration for format and selection rectang
597
598 Configuring streams
599 ^^^^^^^^^^^^^^^^^^^
600
601 The configuration of the streams is done indiv
602 the validity of the streams between sub-device
603 is started.
604
605 There are three steps in configuring the strea
606
607 1. Set up links. Connect the pads between sub-
608 :ref:`Media Controller API <media_controlle
609
610 2. Streams. Streams are declared and their rou
611 routing table for the sub-device using :ref
612 <VIDIOC_SUBDEV_G_ROUTING>` ioctl. Note that
613 reset formats and selections in the sub-dev
614
615 3. Configure formats and selections. Formats a
616 configured separately as documented for pla
617 :ref:`format-propagation`. The stream ID is
618 associated with either sink or source pads
619 :ref:`VIDIOC_SUBDEV_S_ROUTING <VIDIOC_SUBDE
620
621 Multiplexed streams setup example
622 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
623
624 A simple example of a multiplexed stream setup
625
626 - Two identical sensors (Sensor A and Sensor B
627 pad (pad 0) which carries a pixel data strea
628
629 - Multiplexer bridge (Bridge). The bridge has
630 sensors (pads 0, 1), and one source pad (pad
631
632 - Receiver in the SoC (Receiver). The receiver
633 connected to the bridge, and two source pads
634 engine. The receiver demultiplexes the incom
635
636 - DMA Engines in the SoC (DMA Engine), one for
637 connected to a single source pad in the rece
638
639 The sensors, the bridge and the receiver are m
640 exposed to userspace via /dev/v4l-subdevX devi
641 modeled as V4L2 devices, exposed to userspace
642
643 To configure this pipeline, the userspace must
644
645 1. Set up media links between entities: connec
646 bridge to the receiver, and the receiver to
647 not differ from normal non-multiplexed medi
648
649 2. Configure routing
650
651 .. flat-table:: Bridge routing table
652 :header-rows: 1
653
654 * - Sink Pad/Stream
655 - Source Pad/Stream
656 - Routing Flags
657 - Comments
658 * - 0/0
659 - 2/0
660 - V4L2_SUBDEV_ROUTE_FL_ACTIVE
661 - Pixel data stream from Sensor A
662 * - 1/0
663 - 2/1
664 - V4L2_SUBDEV_ROUTE_FL_ACTIVE
665 - Pixel data stream from Sensor B
666
667 .. flat-table:: Receiver routing table
668 :header-rows: 1
669
670 * - Sink Pad/Stream
671 - Source Pad/Stream
672 - Routing Flags
673 - Comments
674 * - 0/0
675 - 1/0
676 - V4L2_SUBDEV_ROUTE_FL_ACTIVE
677 - Pixel data stream from Sensor A
678 * - 0/1
679 - 2/0
680 - V4L2_SUBDEV_ROUTE_FL_ACTIVE
681 - Pixel data stream from Sensor B
682
683 3. Configure formats and selections
684
685 After configuring routing, the next step is
686 selections for the streams. This is similar
687 streams, with just one exception: the ``str
688 to the value of the stream ID.
689
690 A common way to accomplish this is to start
691 the configurations along the stream towards
692 :ref:`VIDIOC_SUBDEV_S_FMT <VIDIOC_SUBDEV_G_
693 stream endpoint in each sub-device.
Linux® is a registered trademark of Linus Torvalds in the United States and other countries.
TOMOYO® is a registered trademark of NTT DATA CORPORATION.