1 .. SPDX-License-Identifier: GPL-2.0
2
3 .. include:: <isonum.txt>
4
5 ==============================================
6 Intel Image Processing Unit 3 (IPU3) Imaging U
7 ==============================================
8
9 Copyright |copy| 2018 Intel Corporation
10
11 Introduction
12 ============
13
14 This file documents the Intel IPU3 (3rd genera
15 Imaging Unit drivers located under drivers/med
16 as under drivers/staging/media/ipu3 (ImgU).
17
18 The Intel IPU3 found in certain Kaby Lake (as
19 platforms (U/Y processor lines) is made up of
20 (ImgU) and the CIO2 device (MIPI CSI2 receiver
21
22 The CIO2 device receives the raw Bayer data fr
23 frames in a format that is specific to the IPU
24 ImgU). The CIO2 driver is available as drivers
25 and is enabled through the CONFIG_VIDEO_IPU3_C
26
27 The Imaging Unit (ImgU) is responsible for pro
28 by the IPU3 CIO2 device. The ImgU driver sourc
29 drivers/staging/media/ipu3 directory. The driv
30 CONFIG_VIDEO_IPU3_IMGU config option.
31
32 The two driver modules are named ipu3_csi2 and
33
34 The drivers has been tested on Kaby Lake platf
35
36 Both of the drivers implement V4L2, Media Cont
37 interfaces. The IPU3 CIO2 driver supports came
38 MIPI CSI-2 interfaces through V4L2 sub-device
39
40 CIO2
41 ====
42
43 The CIO2 is represented as a single V4L2 subde
44 interface to the user space. There is a video
45 with a single media controller interface for t
46
47 The CIO2 contains four independent capture cha
48 receiver and DMA engine. Each channel is model
49 to userspace as a V4L2 sub-device node and has
50
51 .. tabularcolumns:: |p{0.8cm}|p{4.0cm}|p{4.0cm
52
53 .. flat-table::
54 :header-rows: 1
55
56 * - Pad
57 - Direction
58 - Purpose
59
60 * - 0
61 - sink
62 - MIPI CSI-2 input, connected to the sen
63
64 * - 1
65 - source
66 - Raw video capture, connected to the V4
67
68 The V4L2 video interfaces model the DMA engine
69 as V4L2 video device nodes.
70
71 Capturing frames in raw Bayer format
72 ------------------------------------
73
74 CIO2 MIPI CSI2 receiver is used to capture fra
75 from the raw sensors connected to the CSI2 por
76 as input to the ImgU driver.
77
78 Image processing using IPU3 ImgU requires tool
79 yavta [#f2]_ due to the following unique requi
80 to IPU3.
81
82 -- The IPU3 CSI2 receiver outputs the captured
83 raw Bayer format that is specific to IPU3.
84
85 -- Multiple video nodes have to be operated si
86
87 Let us take the example of ov5670 sensor conne
88 2592x1944 image capture.
89
90 Using the media controller APIs, the ov5670 se
91 frames in packed raw Bayer format to IPU3 CSI2
92
93 .. code-block:: none
94
95 # This example assumes /dev/media0 as the
96 export MDEV=/dev/media0
97
98 # and that ov5670 sensor is connected to i
99 export SDEV=$(media-ctl -d $MDEV -e "ov567
100
101 # Establish the link for the media devices
102 media-ctl -d $MDEV -l "ov5670:0 -> ipu3-cs
103
104 # Set the format for the media devices
105 media-ctl -d $MDEV -V "ov5670:0 [fmt:SGRBG
106 media-ctl -d $MDEV -V "ipu3-csi2 0:0 [fmt:
107 media-ctl -d $MDEV -V "ipu3-csi2 0:1 [fmt:
108
109 Once the media pipeline is configured, desired
110 (such as exposure and gain settings) can be se
111
112 e.g
113
114 .. code-block:: none
115
116 yavta -w 0x009e0903 444 $SDEV
117 yavta -w 0x009e0913 1024 $SDEV
118 yavta -w 0x009e0911 2046 $SDEV
119
120 Once the desired sensor settings are set, fram
121
122 e.g
123
124 .. code-block:: none
125
126 yavta --data-prefix -u -c10 -n5 -I -s2592x
127 -f IPU3_SGRBG10 $(media-ctl -d $MDEV
128
129 With the above command, 10 frames are captured
130 sGRBG10 format and output as IPU3_SGRBG10 form
131
132 The captured frames are available as /tmp/fram
133
134 ImgU
135 ====
136
137 The ImgU is represented as two V4L2 subdevs, e
138 subdev interface to the user space.
139
140 Each V4L2 subdev represents a pipe, which can
141 This helps to support advanced camera features
142 and Snapshot During Video(SDV).
143
144 The ImgU contains two independent pipes, each
145 exposed to userspace as a V4L2 sub-device node
146
147 Each pipe has two sink pads and three source p
148
149 .. tabularcolumns:: |p{0.8cm}|p{4.0cm}|p{4.0cm
150
151 .. flat-table::
152 :header-rows: 1
153
154 * - Pad
155 - Direction
156 - Purpose
157
158 * - 0
159 - sink
160 - Input raw video stream
161
162 * - 1
163 - sink
164 - Processing parameters
165
166 * - 2
167 - source
168 - Output processed video stream
169
170 * - 3
171 - source
172 - Output viewfinder video stream
173
174 * - 4
175 - source
176 - 3A statistics
177
178 Each pad is connected to a corresponding V4L2
179 userspace as a V4L2 video device node.
180
181 Device operation
182 ----------------
183
184 With ImgU, once the input video node ("ipu3-im
185 <entity>:<pad-number> format) is queued with b
186 format), ImgU starts processing the buffer and
187 format and statistics output on respective out
188 to have buffers ready for all of parameter, ou
189 input video node is queued with buffer.
190
191 At a minimum, all of input, main output, 3A st
192 video nodes should be enabled for IPU3 to star
193
194 Each ImgU V4L2 subdev has the following set of
195
196 input, output and viewfinder video nodes
197 ----------------------------------------
198
199 The frames (in packed raw Bayer format specifi
200 input video node is processed by the IPU3 Imag
201 nodes, with each targeting a different purpose
202 output).
203
204 Details onand the Bayer format specific to the
205 :ref:`v4l2-pix-fmt-ipu3-sbggr10`.
206
207 The driver supports V4L2 Video Capture Interfa
208
209 Only the multi-planar API is supported. More d
210 :ref:`planar-apis`.
211
212 Parameters video node
213 ---------------------
214
215 The parameters video node receives the ImgU al
216 to configure how the ImgU algorithms process t
217
218 Details on processing parameters specific to t
219 :ref:`v4l2-meta-fmt-params`.
220
221 3A statistics video node
222 ------------------------
223
224 3A statistics video node is used by the ImgU d
225 focus, auto exposure and auto white balance) s
226 being processed by the ImgU to user space appl
227 can use this statistics data to compute the de
228 the ImgU.
229
230 Configuring the Intel IPU3
231 ==========================
232
233 The IPU3 ImgU pipelines can be configured usin
234 :ref:`media_controller`.
235
236 Running mode and firmware binary selection
237 ------------------------------------------
238
239 ImgU works based on firmware, currently the Im
240 in time-sharing with single input frame data.
241 - "VIDEO" or "STILL", "VIDEO" mode is commonly
242 and "STILL" is used for still frame capture. H
243 "VIDEO" to capture still frames if you want to
244 load and power. For "STILL" mode, ImgU will tr
245 output larger bayer frame for further YUV proc
246 high quality images. Besides, "STILL" mode nee
247 hence "STILL" mode will need more power and me
248 TNR will be enabled in "VIDEO" mode and bypass
249 running at "VIDEO" mode by default, the user c
250 V4L2_CID_INTEL_IPU3_MODE (currently defined in
251 drivers/staging/media/ipu3/include/uapi/intel-
252 running mode. For user, there is no difference
253 "VIDEO" and "STILL" mode, mandatory input and
254 enabled and buffers need be queued, the statis
255 are optional.
256
257 The firmware binary will be selected according
258 "using binary if_to_osys_striped " or "using b
259 could be observed if you enable the ImgU dynam
260 if_to_osys_striped is selected for "VIDEO" and
261 "if_to_osys_primary_striped" is selected for "
262
263
264 Processing the image in raw Bayer format
265 ----------------------------------------
266
267 Configuring ImgU V4L2 subdev for image process
268 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
269
270 The ImgU V4L2 subdevs have to be configured wi
271 all the video nodes setup correctly.
272
273 Let us take "ipu3-imgu 0" subdev as an example
274
275 .. code-block:: none
276
277 media-ctl -d $MDEV -r
278 media-ctl -d $MDEV -l "ipu3-imgu 0 input":
279 media-ctl -d $MDEV -l "ipu3-imgu 0":2 -> "
280 media-ctl -d $MDEV -l "ipu3-imgu 0":3 -> "
281 media-ctl -d $MDEV -l "ipu3-imgu 0":4 -> "
282
283 Also the pipe mode of the corresponding V4L2 s
284 (e.g 0 for video mode or 1 for still mode) thr
285 below.
286
287 .. code-block:: none
288
289 yavta -w "0x009819A1 1" /dev/v4l-subdev7
290
291 Certain hardware blocks in ImgU pipeline can c
292 cropping or scaling, these hardware blocks inc
293 Scaler (BDS) and Geometric Distortion Correcti
294 There is also a block which can change the fra
295 only applicable to the secondary output.
296
297 RAW Bayer frames go through these ImgU pipelin
298 processed image output to the DDR memory.
299
300 .. kernel-figure:: ipu3_rcb.svg
301 :alt: ipu3 resolution blocks image
302
303 IPU3 resolution change hardware blocks
304
305 **Input Feeder**
306
307 Input Feeder gets the Bayer frame data from th
308 of lines and columns from the frame and then s
309 pixel buffer which are ready to readout by fol
310
311 **Bayer Down Scaler**
312
313 Bayer Down Scaler is capable of performing ima
314 downscale factor can be configured from 1X to
315 configuration steps of 0.03125 (1/32).
316
317 **Geometric Distortion Correction**
318
319 Geometric Distortion Correction is used to per
320 and image filtering. It needs some extra filte
321 work, so the input resolution of GDC should be
322 resolution.
323
324 **YUV Scaler**
325
326 YUV Scaler which similar with BDS, but it is m
327 YUV domain, it can support up to 1/12X down sc
328 to the main output.
329
330 The ImgU V4L2 subdev has to be configured with
331 the above hardware blocks, for a given input r
332 For a given supported resolution for an input
333 Down Scaler and GDC blocks should be configure
334 as each hardware block has its own alignment r
335
336 You must configure the output resolution of th
337 the hardware requirement along with keeping th
338 intermediate resolutions can be generated by s
339
340 https://github.com/intel/intel-ipu3-pipecfg
341
342 This tool can be used to generate intermediate
343 be obtained by looking at the following IPU3 I
344
345 https://chromium.googlesource.com/chromiumos/o
346
347 Under baseboard-poppy/media-libs/cros-camera-h
348 directory, graph_settings_ov5670.xml can be us
349
350 The following steps prepare the ImgU pipeline
351
352 1. The ImgU V4L2 subdev data format should be
353 VIDIOC_SUBDEV_S_FMT on pad 0, using the GDC wi
354
355 2. The ImgU V4L2 subdev cropping should be set
356 VIDIOC_SUBDEV_S_SELECTION on pad 0, with V4L2_
357 using the input feeder height and width.
358
359 3. The ImgU V4L2 subdev composing should be se
360 VIDIOC_SUBDEV_S_SELECTION on pad 0, with V4L2_
361 using the BDS height and width.
362
363 For the ov5670 example, for an input frame wit
364 (which is input to the ImgU subdev pad 0), the
365 for input feeder, BDS and GDC are 2592x1944, 2
366 respectively.
367
368 Once this is done, the received raw Bayer fram
369 V4L2 subdev as below, using the open source ap
370
371 For an image captured with 2592x1944 [#f4]_ re
372 resolution as 2560x1920 and viewfinder resolut
373 v4l2n command can be used. This helps process
374 the desired results for the main output image
375 format.
376
377 .. code-block:: none
378
379 v4l2n --pipe=4 --load=/tmp/frame-#.bin --o
380 --fmt=type:VIDEO_OUTPUT_MPLANE,width
381 --reqbufs=type:VIDEO_OUTPUT_MPLANE,c
382 --output=/tmp/frames.out --open=/dev
383 --fmt=type:VIDEO_CAPTURE_MPLANE,widt
384 --reqbufs=type:VIDEO_CAPTURE_MPLANE,
385 --output=/tmp/frames.vf --open=/dev/
386 --fmt=type:VIDEO_CAPTURE_MPLANE,widt
387 --reqbufs=type:VIDEO_CAPTURE_MPLANE,
388 --output=/tmp/frames.3A --fmt=type:M
389 --reqbufs=count:1,type:META_CAPTURE
390
391 You can also use yavta [#f2]_ command to do sa
392
393 .. code-block:: none
394
395 yavta --data-prefix -Bcapture-mplane -c10
396 --file=frame-#.out-f NV12 /dev/video
397 yavta --data-prefix -Bcapture-mplane -c10
398 --file=frame-#.vf -f NV12 /dev/video
399 yavta --data-prefix -Bmeta-capture -c10 -n
400 --file=frame-#.3a /dev/video7 & \
401 yavta --data-prefix -Boutput-mplane -c10 -
402 --file=/tmp/frame-in.cio2 -f IPU3_SG
403
404 where /dev/video4, /dev/video5, /dev/video6 an
405 input, output, viewfinder and 3A statistics vi
406
407 Converting the raw Bayer image into YUV domain
408 ----------------------------------------------
409
410 The processed images after the above step, can
411 as below.
412
413 Main output frames
414 ~~~~~~~~~~~~~~~~~~
415
416 .. code-block:: none
417
418 raw2pnm -x2560 -y1920 -fNV12 /tmp/frames.o
419
420 where 2560x1920 is output resolution, NV12 is
421 by input frame and output PNM file.
422
423 Viewfinder output frames
424 ~~~~~~~~~~~~~~~~~~~~~~~~
425
426 .. code-block:: none
427
428 raw2pnm -x2560 -y1920 -fNV12 /tmp/frames.v
429
430 where 2560x1920 is output resolution, NV12 is
431 by input frame and output PNM file.
432
433 Example user space code for IPU3
434 ================================
435
436 User space code that configures and uses IPU3
437
438 https://chromium.googlesource.com/chromiumos/p
439
440 The source can be located under hal/intel dire
441
442 Overview of IPU3 pipeline
443 =========================
444
445 IPU3 pipeline has a number of image processing
446 set of parameters as input. The major stages o
447
448 .. kernel-render:: DOT
449 :alt: IPU3 ImgU Pipeline
450 :caption: IPU3 ImgU Pipeline Diagram
451
452 digraph "IPU3 ImgU" {
453 node [shape=box]
454 splines="ortho"
455 rankdir="LR"
456
457 a [label="Raw pixels"]
458 b [label="Bayer Downscaling"]
459 c [label="Optical Black Correction"]
460 d [label="Linearization"]
461 e [label="Lens Shading Correction"]
462 f [label="White Balance / Exposure / Fo
463 g [label="Bayer Noise Reduction"]
464 h [label="ANR"]
465 i [label="Demosaicing"]
466 j [label="Color Correction Matrix"]
467 k [label="Gamma correction"]
468 l [label="Color Space Conversion"]
469 m [label="Chroma Down Scaling"]
470 n [label="Chromatic Noise Reduction"]
471 o [label="Total Color Correction"]
472 p [label="XNR3"]
473 q [label="TNR"]
474 r [label="DDR", style=filled, fillcolor
475 s [label="YUV Downscaling"]
476 t [label="DDR", style=filled, fillcolor
477
478 { rank=same; a -> b -> c -> d -> e -> f
479 { rank=same; j -> k -> l -> m -> n -> o
480
481 a -> j [style=invis, weight=10]
482 i -> j
483 q -> r
484 }
485
486 The table below presents a description of the
487
488 ======================== =====================
489 Name Description
490 ======================== =====================
491 Optical Black Correction Optical Black Correct
492 value from the respec
493 image quality.
494 Defined in struct ipu
495 Linearization This algo block uses
496 address non-linearity
497 table is defined in
498 struct ipu3_uapi_isp_
499 SHD Lens shading correcti
500 non-uniformity of the
501 lens shading. This is
502 for each pixel. The g
503 configured in struct
504 BNR Bayer noise reduction
505 applying a bilateral
506 See struct ipu3_uapi_
507 ANR Advanced Noise Reduct
508 that performs noise r
509 convolution matrix et
510 struct ipu3_uapi_anr_
511 DM Demosaicing converts
512 into RGB (Red, Green,
513 outputs of estimation
514 processing by Firmwar
515 struct ipu3_uapi_dm_c
516 Color Correction Color Correction algo
517 space to the standard
518 by applying 3x3 matri
519 struct ipu3_uapi_ccm_
520 Gamma correction Gamma correction stru
521 basic non-linear tone
522 applied per pixel for
523 CSC Color space conversio
524 RGB primary presentat
525 UV: Luminance) presen
526 a 3x3 matrix defined
527 struct ipu3_uapi_csc_
528 CDS Chroma down sampling
529 After the CSC is perf
530 is applied for a UV p
531 of 2 in each directio
532 configurable filter s
533 CHNR Chroma noise reductio
534 This block processes
535 performs noise reduct
536 frequency noise.
537 See struct struct ipu
538 TCC Total color correctio
539 struct ipu3_uapi_yuvp
540 XNR3 eXtreme Noise Reducti
541 noise reduction algor
542 quality. This removes
543 captured image. Two r
544 struct ipu3_uapi_isp_
545 and struct ipu3_uapi_
546 memory.
547 TNR Temporal Noise Reduct
548 frames in time to rem
549 values. struct ipu3_u
550 struct ipu3_uapi_isp_
551 vector and data memor
552 ======================== =====================
553
554 Other often encountered acronyms not listed in
555
556 ACC
557 Accelerator cluster
558 AWB_FR
559 Auto white balance filter resp
560 BDS
561 Bayer downscaler parameters
562 CCM
563 Color correction matrix coeffi
564 IEFd
565 Image enhancement filter direc
566 Obgrid
567 Optical black level compensati
568 OSYS
569 Output system configuration
570 ROI
571 Region of interest
572 YDS
573 Y down sampling
574 YTM
575 Y-tone mapping
576
577 A few stages of the pipeline will be executed
578 processor, while many others will use a set of
579 called accelerator cluster (ACC) to crunch pix
580
581 ACC parameters of individual algorithms, as de
582 struct ipu3_uapi_acc_param, can be chosen to b
583 space through struct struct ipu3_uapi_flags em
584 struct ipu3_uapi_params structure. For paramet
585 not enabled by the user space, the correspondi
586 driver, in which case the existing configurati
587 preserved.
588
589 References
590 ==========
591
592 .. [#f5] drivers/staging/media/ipu3/include/ua
593
594 .. [#f1] https://github.com/intel/nvt
595
596 .. [#f2] http://git.ideasonboard.org/yavta.git
597
598 .. [#f3] http://git.ideasonboard.org/?p=media-
599
600 .. [#f4] ImgU limitation requires an additiona
Linux® is a registered trademark of Linus Torvalds in the United States and other countries.
TOMOYO® is a registered trademark of NTT DATA CORPORATION.