1 .. SPDX-License-Identifier: GPL-2.0
2
3 Media Controller devices
4 ------------------------
5
6 Media Controller
7 ~~~~~~~~~~~~~~~~
8
9 The media controller userspace API is documented in
10 :ref:`the Media Controller uAPI book <media_controller>`. This document focus
11 on the kernel-side implementation of the media framework.
12
13 Abstract media device model
14 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
15
16 Discovering a device internal topology, and configuring it at runtime, is one
17 of the goals of the media framework. To achieve this, hardware devices are
18 modelled as an oriented graph of building blocks called entities connected
19 through pads.
20
21 An entity is a basic media hardware building block. It can correspond to
22 a large variety of logical blocks such as physical hardware devices
23 (CMOS sensor for instance), logical hardware devices (a building block
24 in a System-on-Chip image processing pipeline), DMA channels or physical
25 connectors.
26
27 A pad is a connection endpoint through which an entity can interact with
28 other entities. Data (not restricted to video) produced by an entity
29 flows from the entity's output to one or more entity inputs. Pads should
30 not be confused with physical pins at chip boundaries.
31
32 A link is a point-to-point oriented connection between two pads, either
33 on the same entity or on different entities. Data flows from a source
34 pad to a sink pad.
35
36 Media device
37 ^^^^^^^^^^^^
38
39 A media device is represented by a struct media_device
40 instance, defined in ``include/media/media-device.h``.
41 Allocation of the structure is handled by the media device driver, usually by
42 embedding the :c:type:`media_device` instance in a larger driver-specific
43 structure.
44
45 Drivers initialise media device instances by calling
46 :c:func:`media_device_init()`. After initialising a media device instance, it is
47 registered by calling :c:func:`__media_device_register()` via the macro
48 ``media_device_register()`` and unregistered by calling
49 :c:func:`media_device_unregister()`. An initialised media device must be
50 eventually cleaned up by calling :c:func:`media_device_cleanup()`.
51
52 Note that it is not allowed to unregister a media device instance that was not
53 previously registered, or clean up a media device instance that was not
54 previously initialised.
55
56 Entities
57 ^^^^^^^^
58
59 Entities are represented by a struct media_entity
60 instance, defined in ``include/media/media-entity.h``. The structure is usually
61 embedded into a higher-level structure, such as
62 :c:type:`v4l2_subdev` or :c:type:`video_device`
63 instances, although drivers can allocate entities directly.
64
65 Drivers initialize entity pads by calling
66 :c:func:`media_entity_pads_init()`.
67
68 Drivers register entities with a media device by calling
69 :c:func:`media_device_register_entity()`
70 and unregistered by calling
71 :c:func:`media_device_unregister_entity()`.
72
73 Interfaces
74 ^^^^^^^^^^
75
76 Interfaces are represented by a
77 struct media_interface instance, defined in
78 ``include/media/media-entity.h``. Currently, only one type of interface is
79 defined: a device node. Such interfaces are represented by a
80 struct media_intf_devnode.
81
82 Drivers initialize and create device node interfaces by calling
83 :c:func:`media_devnode_create()`
84 and remove them by calling:
85 :c:func:`media_devnode_remove()`.
86
87 Pads
88 ^^^^
89 Pads are represented by a struct media_pad instance,
90 defined in ``include/media/media-entity.h``. Each entity stores its pads in
91 a pads array managed by the entity driver. Drivers usually embed the array in
92 a driver-specific structure.
93
94 Pads are identified by their entity and their 0-based index in the pads
95 array.
96
97 Both information are stored in the struct media_pad,
98 making the struct media_pad pointer the canonical way
99 to store and pass link references.
100
101 Pads have flags that describe the pad capabilities and state.
102
103 ``MEDIA_PAD_FL_SINK`` indicates that the pad supports sinking data.
104 ``MEDIA_PAD_FL_SOURCE`` indicates that the pad supports sourcing data.
105
106 .. note::
107
108 One and only one of ``MEDIA_PAD_FL_SINK`` or ``MEDIA_PAD_FL_SOURCE`` must
109 be set for each pad.
110
111 Links
112 ^^^^^
113
114 Links are represented by a struct media_link instance,
115 defined in ``include/media/media-entity.h``. There are two types of links:
116
117 **1. pad to pad links**:
118
119 Associate two entities via their PADs. Each entity has a list that points
120 to all links originating at or targeting any of its pads.
121 A given link is thus stored twice, once in the source entity and once in
122 the target entity.
123
124 Drivers create pad to pad links by calling:
125 :c:func:`media_create_pad_link()` and remove with
126 :c:func:`media_entity_remove_links()`.
127
128 **2. interface to entity links**:
129
130 Associate one interface to a Link.
131
132 Drivers create interface to entity links by calling:
133 :c:func:`media_create_intf_link()` and remove with
134 :c:func:`media_remove_intf_links()`.
135
136 .. note::
137
138 Links can only be created after having both ends already created.
139
140 Links have flags that describe the link capabilities and state. The
141 valid values are described at :c:func:`media_create_pad_link()` and
142 :c:func:`media_create_intf_link()`.
143
144 Graph traversal
145 ^^^^^^^^^^^^^^^
146
147 The media framework provides APIs to traverse media graphs, locating connected
148 entities and links.
149
150 To iterate over all entities belonging to a media device, drivers can use
151 the media_device_for_each_entity macro, defined in
152 ``include/media/media-device.h``.
153
154 .. code-block:: c
155
156 struct media_entity *entity;
157
158 media_device_for_each_entity(entity, mdev) {
159 // entity will point to each entity in turn
160 ...
161 }
162
163 Helper functions can be used to find a link between two given pads, or a pad
164 connected to another pad through an enabled link
165 (:c:func:`media_entity_find_link()`, :c:func:`media_pad_remote_pad_first()`,
166 :c:func:`media_entity_remote_source_pad_unique()` and
167 :c:func:`media_pad_remote_pad_unique()`).
168
169 Use count and power handling
170 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
171
172 Due to the wide differences between drivers regarding power management
173 needs, the media controller does not implement power management. However,
174 the struct media_entity includes a ``use_count``
175 field that media drivers
176 can use to track the number of users of every entity for power management
177 needs.
178
179 The :c:type:`media_entity<media_entity>`.\ ``use_count`` field is owned by
180 media drivers and must not be
181 touched by entity drivers. Access to the field must be protected by the
182 :c:type:`media_device`.\ ``graph_mutex`` lock.
183
184 Links setup
185 ^^^^^^^^^^^
186
187 Link properties can be modified at runtime by calling
188 :c:func:`media_entity_setup_link()`.
189
190 Pipelines and media streams
191 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
192
193 A media stream is a stream of pixels or metadata originating from one or more
194 source devices (such as a sensors) and flowing through media entity pads
195 towards the final sinks. The stream can be modified on the route by the
196 devices (e.g. scaling or pixel format conversions), or it can be split into
197 multiple branches, or multiple branches can be merged.
198
199 A media pipeline is a set of media streams which are interdependent. This
200 interdependency can be caused by the hardware (e.g. configuration of a second
201 stream cannot be changed if the first stream has been enabled) or by the driver
202 due to the software design. Most commonly a media pipeline consists of a single
203 stream which does not branch.
204
205 When starting streaming, drivers must notify all entities in the pipeline to
206 prevent link states from being modified during streaming by calling
207 :c:func:`media_pipeline_start()`.
208
209 The function will mark all the pads which are part of the pipeline as streaming.
210
211 The struct media_pipeline instance pointed to by the pipe argument will be
212 stored in every pad in the pipeline. Drivers should embed the struct
213 media_pipeline in higher-level pipeline structures and can then access the
214 pipeline through the struct media_pad pipe field.
215
216 Calls to :c:func:`media_pipeline_start()` can be nested.
217 The pipeline pointer must be identical for all nested calls to the function.
218
219 :c:func:`media_pipeline_start()` may return an error. In that case,
220 it will clean up any of the changes it did by itself.
221
222 When stopping the stream, drivers must notify the entities with
223 :c:func:`media_pipeline_stop()`.
224
225 If multiple calls to :c:func:`media_pipeline_start()` have been
226 made the same number of :c:func:`media_pipeline_stop()` calls
227 are required to stop streaming.
228 The :c:type:`media_entity`.\ ``pipe`` field is reset to ``NULL`` on the last
229 nested stop call.
230
231 Link configuration will fail with ``-EBUSY`` by default if either end of the
232 link is a streaming entity. Links that can be modified while streaming must
233 be marked with the ``MEDIA_LNK_FL_DYNAMIC`` flag.
234
235 If other operations need to be disallowed on streaming entities (such as
236 changing entities configuration parameters) drivers can explicitly check the
237 media_entity stream_count field to find out if an entity is streaming. This
238 operation must be done with the media_device graph_mutex held.
239
240 Link validation
241 ^^^^^^^^^^^^^^^
242
243 Link validation is performed by :c:func:`media_pipeline_start()`
244 for any entity which has sink pads in the pipeline. The
245 :c:type:`media_entity`.\ ``link_validate()`` callback is used for that
246 purpose. In ``link_validate()`` callback, entity driver should check
247 that the properties of the source pad of the connected entity and its own
248 sink pad match. It is up to the type of the entity (and in the end, the
249 properties of the hardware) what matching actually means.
250
251 Subsystems should facilitate link validation by providing subsystem specific
252 helper functions to provide easy access for commonly needed information, and
253 in the end provide a way to use driver-specific callbacks.
254
255 Pipeline traversal
256 ^^^^^^^^^^^^^^^^^^
257
258 Once a pipeline has been constructed with :c:func:`media_pipeline_start()`,
259 drivers can iterate over entities or pads in the pipeline with the
260 :c:macro:´media_pipeline_for_each_entity` and
261 :c:macro:´media_pipeline_for_each_pad` macros. Iterating over pads is
262 straightforward:
263
264 .. code-block:: c
265
266 media_pipeline_pad_iter iter;
267 struct media_pad *pad;
268
269 media_pipeline_for_each_pad(pipe, &iter, pad) {
270 /* 'pad' will point to each pad in turn */
271 ...
272 }
273
274 To iterate over entities, the iterator needs to be initialized and cleaned up
275 as an additional steps:
276
277 .. code-block:: c
278
279 media_pipeline_entity_iter iter;
280 struct media_entity *entity;
281 int ret;
282
283 ret = media_pipeline_entity_iter_init(pipe, &iter);
284 if (ret)
285 ...;
286
287 media_pipeline_for_each_entity(pipe, &iter, entity) {
288 /* 'entity' will point to each entity in turn */
289 ...
290 }
291
292 media_pipeline_entity_iter_cleanup(&iter);
293
294 Media Controller Device Allocator API
295 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
296
297 When the media device belongs to more than one driver, the shared media
298 device is allocated with the shared struct device as the key for look ups.
299
300 The shared media device should stay in registered state until the last
301 driver unregisters it. In addition, the media device should be released when
302 all the references are released. Each driver gets a reference to the media
303 device during probe, when it allocates the media device. If media device is
304 already allocated, the allocate API bumps up the refcount and returns the
305 existing media device. The driver puts the reference back in its disconnect
306 routine when it calls :c:func:`media_device_delete()`.
307
308 The media device is unregistered and cleaned up from the kref put handler to
309 ensure that the media device stays in registered state until the last driver
310 unregisters the media device.
311
312 **Driver Usage**
313
314 Drivers should use the appropriate media-core routines to manage the shared
315 media device life-time handling the two states:
316 1. allocate -> register -> delete
317 2. get reference to already registered device -> delete
318
319 call :c:func:`media_device_delete()` routine to make sure the shared media
320 device delete is handled correctly.
321
322 **driver probe:**
323 Call :c:func:`media_device_usb_allocate()` to allocate or get a reference
324 Call :c:func:`media_device_register()`, if media devnode isn't registered
325
326 **driver disconnect:**
327 Call :c:func:`media_device_delete()` to free the media_device. Freeing is
328 handled by the kref put handler.
329
330 API Definitions
331 ^^^^^^^^^^^^^^^
332
333 .. kernel-doc:: include/media/media-device.h
334
335 .. kernel-doc:: include/media/media-devnode.h
336
337 .. kernel-doc:: include/media/media-entity.h
338
339 .. kernel-doc:: include/media/media-request.h
340
341 .. kernel-doc:: include/media/media-dev-allocator.h
Linux® is a registered trademark of Linus Torvalds in the United States and other countries.
TOMOYO® is a registered trademark of NTT DATA CORPORATION.