1 .. SPDX-License-Identifier: GPL-2.0 1 .. SPDX-License-Identifier: GPL-2.0
2 2
3 .. include:: <isonum.txt> 3 .. include:: <isonum.txt>
4 4
5 ============================================== 5 ===============================================================
6 Intel Image Processing Unit 3 (IPU3) Imaging U 6 Intel Image Processing Unit 3 (IPU3) Imaging Unit (ImgU) driver
7 ============================================== 7 ===============================================================
8 8
9 Copyright |copy| 2018 Intel Corporation 9 Copyright |copy| 2018 Intel Corporation
10 10
11 Introduction 11 Introduction
12 ============ 12 ============
13 13
14 This file documents the Intel IPU3 (3rd genera 14 This file documents the Intel IPU3 (3rd generation Image Processing Unit)
15 Imaging Unit drivers located under drivers/med 15 Imaging Unit drivers located under drivers/media/pci/intel/ipu3 (CIO2) as well
16 as under drivers/staging/media/ipu3 (ImgU). 16 as under drivers/staging/media/ipu3 (ImgU).
17 17
18 The Intel IPU3 found in certain Kaby Lake (as 18 The Intel IPU3 found in certain Kaby Lake (as well as certain Sky Lake)
19 platforms (U/Y processor lines) is made up of 19 platforms (U/Y processor lines) is made up of two parts namely the Imaging Unit
20 (ImgU) and the CIO2 device (MIPI CSI2 receiver 20 (ImgU) and the CIO2 device (MIPI CSI2 receiver).
21 21
22 The CIO2 device receives the raw Bayer data fr 22 The CIO2 device receives the raw Bayer data from the sensors and outputs the
23 frames in a format that is specific to the IPU 23 frames in a format that is specific to the IPU3 (for consumption by the IPU3
24 ImgU). The CIO2 driver is available as drivers 24 ImgU). The CIO2 driver is available as drivers/media/pci/intel/ipu3/ipu3-cio2*
25 and is enabled through the CONFIG_VIDEO_IPU3_C 25 and is enabled through the CONFIG_VIDEO_IPU3_CIO2 config option.
26 26
27 The Imaging Unit (ImgU) is responsible for pro 27 The Imaging Unit (ImgU) is responsible for processing images captured
28 by the IPU3 CIO2 device. The ImgU driver sourc 28 by the IPU3 CIO2 device. The ImgU driver sources can be found under
29 drivers/staging/media/ipu3 directory. The driv 29 drivers/staging/media/ipu3 directory. The driver is enabled through the
30 CONFIG_VIDEO_IPU3_IMGU config option. 30 CONFIG_VIDEO_IPU3_IMGU config option.
31 31
32 The two driver modules are named ipu3_csi2 and 32 The two driver modules are named ipu3_csi2 and ipu3_imgu, respectively.
33 33
34 The drivers has been tested on Kaby Lake platf 34 The drivers has been tested on Kaby Lake platforms (U/Y processor lines).
35 35
36 Both of the drivers implement V4L2, Media Cont 36 Both of the drivers implement V4L2, Media Controller and V4L2 sub-device
37 interfaces. The IPU3 CIO2 driver supports came 37 interfaces. The IPU3 CIO2 driver supports camera sensors connected to the CIO2
38 MIPI CSI-2 interfaces through V4L2 sub-device 38 MIPI CSI-2 interfaces through V4L2 sub-device sensor drivers.
39 39
40 CIO2 40 CIO2
41 ==== 41 ====
42 42
43 The CIO2 is represented as a single V4L2 subde 43 The CIO2 is represented as a single V4L2 subdev, which provides a V4L2 subdev
44 interface to the user space. There is a video 44 interface to the user space. There is a video node for each CSI-2 receiver,
45 with a single media controller interface for t 45 with a single media controller interface for the entire device.
46 46
47 The CIO2 contains four independent capture cha 47 The CIO2 contains four independent capture channel, each with its own MIPI CSI-2
48 receiver and DMA engine. Each channel is model 48 receiver and DMA engine. Each channel is modelled as a V4L2 sub-device exposed
49 to userspace as a V4L2 sub-device node and has 49 to userspace as a V4L2 sub-device node and has two pads:
50 50
51 .. tabularcolumns:: |p{0.8cm}|p{4.0cm}|p{4.0cm 51 .. tabularcolumns:: |p{0.8cm}|p{4.0cm}|p{4.0cm}|
52 52
53 .. flat-table:: 53 .. flat-table::
54 :header-rows: 1 <<
55 54
56 * - Pad !! 55 * - pad
57 - Direction !! 56 - direction
58 - Purpose !! 57 - purpose
59 58
60 * - 0 59 * - 0
61 - sink 60 - sink
62 - MIPI CSI-2 input, connected to the sen 61 - MIPI CSI-2 input, connected to the sensor subdev
63 62
64 * - 1 63 * - 1
65 - source 64 - source
66 - Raw video capture, connected to the V4 65 - Raw video capture, connected to the V4L2 video interface
67 66
68 The V4L2 video interfaces model the DMA engine 67 The V4L2 video interfaces model the DMA engines. They are exposed to userspace
69 as V4L2 video device nodes. 68 as V4L2 video device nodes.
70 69
71 Capturing frames in raw Bayer format 70 Capturing frames in raw Bayer format
72 ------------------------------------ 71 ------------------------------------
73 72
74 CIO2 MIPI CSI2 receiver is used to capture fra 73 CIO2 MIPI CSI2 receiver is used to capture frames (in packed raw Bayer format)
75 from the raw sensors connected to the CSI2 por 74 from the raw sensors connected to the CSI2 ports. The captured frames are used
76 as input to the ImgU driver. 75 as input to the ImgU driver.
77 76
78 Image processing using IPU3 ImgU requires tool 77 Image processing using IPU3 ImgU requires tools such as raw2pnm [#f1]_, and
79 yavta [#f2]_ due to the following unique requi 78 yavta [#f2]_ due to the following unique requirements and / or features specific
80 to IPU3. 79 to IPU3.
81 80
82 -- The IPU3 CSI2 receiver outputs the captured 81 -- The IPU3 CSI2 receiver outputs the captured frames from the sensor in packed
83 raw Bayer format that is specific to IPU3. 82 raw Bayer format that is specific to IPU3.
84 83
85 -- Multiple video nodes have to be operated si 84 -- Multiple video nodes have to be operated simultaneously.
86 85
87 Let us take the example of ov5670 sensor conne 86 Let us take the example of ov5670 sensor connected to CSI2 port 0, for a
88 2592x1944 image capture. 87 2592x1944 image capture.
89 88
90 Using the media controller APIs, the ov5670 se !! 89 Using the media contorller APIs, the ov5670 sensor is configured to send
91 frames in packed raw Bayer format to IPU3 CSI2 90 frames in packed raw Bayer format to IPU3 CSI2 receiver.
92 91
93 .. code-block:: none !! 92 # This example assumes /dev/media0 as the CIO2 media device
>> 93
>> 94 export MDEV=/dev/media0
>> 95
>> 96 # and that ov5670 sensor is connected to i2c bus 10 with address 0x36
94 97
95 # This example assumes /dev/media0 as the !! 98 export SDEV=$(media-ctl -d $MDEV -e "ov5670 10-0036")
96 export MDEV=/dev/media0 <<
97 99
98 # and that ov5670 sensor is connected to i !! 100 # Establish the link for the media devices using media-ctl [#f3]_
99 export SDEV=$(media-ctl -d $MDEV -e "ov567 !! 101 media-ctl -d $MDEV -l "ov5670:0 -> ipu3-csi2 0:0[1]"
100 102
101 # Establish the link for the media devices !! 103 # Set the format for the media devices
102 media-ctl -d $MDEV -l "ov5670:0 -> ipu3-cs !! 104 media-ctl -d $MDEV -V "ov5670:0 [fmt:SGRBG10/2592x1944]"
103 105
104 # Set the format for the media devices !! 106 media-ctl -d $MDEV -V "ipu3-csi2 0:0 [fmt:SGRBG10/2592x1944]"
105 media-ctl -d $MDEV -V "ov5670:0 [fmt:SGRBG !! 107
106 media-ctl -d $MDEV -V "ipu3-csi2 0:0 [fmt: !! 108 media-ctl -d $MDEV -V "ipu3-csi2 0:1 [fmt:SGRBG10/2592x1944]"
107 media-ctl -d $MDEV -V "ipu3-csi2 0:1 [fmt: <<
108 109
109 Once the media pipeline is configured, desired 110 Once the media pipeline is configured, desired sensor specific settings
110 (such as exposure and gain settings) can be se 111 (such as exposure and gain settings) can be set, using the yavta tool.
111 112
112 e.g 113 e.g
113 114
114 .. code-block:: none !! 115 yavta -w 0x009e0903 444 $SDEV
115 116
116 yavta -w 0x009e0903 444 $SDEV !! 117 yavta -w 0x009e0913 1024 $SDEV
117 yavta -w 0x009e0913 1024 $SDEV !! 118
118 yavta -w 0x009e0911 2046 $SDEV !! 119 yavta -w 0x009e0911 2046 $SDEV
119 120
120 Once the desired sensor settings are set, fram 121 Once the desired sensor settings are set, frame captures can be done as below.
121 122
122 e.g 123 e.g
123 124
124 .. code-block:: none !! 125 yavta --data-prefix -u -c10 -n5 -I -s2592x1944 --file=/tmp/frame-#.bin \
125 !! 126 -f IPU3_SGRBG10 $(media-ctl -d $MDEV -e "ipu3-cio2 0")
126 yavta --data-prefix -u -c10 -n5 -I -s2592x <<
127 -f IPU3_SGRBG10 $(media-ctl -d $MDEV <<
128 127
129 With the above command, 10 frames are captured 128 With the above command, 10 frames are captured at 2592x1944 resolution, with
130 sGRBG10 format and output as IPU3_SGRBG10 form 129 sGRBG10 format and output as IPU3_SGRBG10 format.
131 130
132 The captured frames are available as /tmp/fram 131 The captured frames are available as /tmp/frame-#.bin files.
133 132
134 ImgU 133 ImgU
135 ==== 134 ====
136 135
137 The ImgU is represented as two V4L2 subdevs, e 136 The ImgU is represented as two V4L2 subdevs, each of which provides a V4L2
138 subdev interface to the user space. 137 subdev interface to the user space.
139 138
140 Each V4L2 subdev represents a pipe, which can 139 Each V4L2 subdev represents a pipe, which can support a maximum of 2 streams.
141 This helps to support advanced camera features 140 This helps to support advanced camera features like Continuous View Finder (CVF)
142 and Snapshot During Video(SDV). 141 and Snapshot During Video(SDV).
143 142
144 The ImgU contains two independent pipes, each 143 The ImgU contains two independent pipes, each modelled as a V4L2 sub-device
145 exposed to userspace as a V4L2 sub-device node 144 exposed to userspace as a V4L2 sub-device node.
146 145
147 Each pipe has two sink pads and three source p 146 Each pipe has two sink pads and three source pads for the following purpose:
148 147
149 .. tabularcolumns:: |p{0.8cm}|p{4.0cm}|p{4.0cm 148 .. tabularcolumns:: |p{0.8cm}|p{4.0cm}|p{4.0cm}|
150 149
151 .. flat-table:: 150 .. flat-table::
152 :header-rows: 1 <<
153 151
154 * - Pad !! 152 * - pad
155 - Direction !! 153 - direction
156 - Purpose !! 154 - purpose
157 155
158 * - 0 156 * - 0
159 - sink 157 - sink
160 - Input raw video stream 158 - Input raw video stream
161 159
162 * - 1 160 * - 1
163 - sink 161 - sink
164 - Processing parameters 162 - Processing parameters
165 163
166 * - 2 164 * - 2
167 - source 165 - source
168 - Output processed video stream 166 - Output processed video stream
169 167
170 * - 3 168 * - 3
171 - source 169 - source
172 - Output viewfinder video stream 170 - Output viewfinder video stream
173 171
174 * - 4 172 * - 4
175 - source 173 - source
176 - 3A statistics 174 - 3A statistics
177 175
178 Each pad is connected to a corresponding V4L2 176 Each pad is connected to a corresponding V4L2 video interface, exposed to
179 userspace as a V4L2 video device node. 177 userspace as a V4L2 video device node.
180 178
181 Device operation 179 Device operation
182 ---------------- 180 ----------------
183 181
184 With ImgU, once the input video node ("ipu3-im 182 With ImgU, once the input video node ("ipu3-imgu 0/1":0, in
185 <entity>:<pad-number> format) is queued with b 183 <entity>:<pad-number> format) is queued with buffer (in packed raw Bayer
186 format), ImgU starts processing the buffer and 184 format), ImgU starts processing the buffer and produces the video output in YUV
187 format and statistics output on respective out 185 format and statistics output on respective output nodes. The driver is expected
188 to have buffers ready for all of parameter, ou 186 to have buffers ready for all of parameter, output and statistics nodes, when
189 input video node is queued with buffer. 187 input video node is queued with buffer.
190 188
191 At a minimum, all of input, main output, 3A st 189 At a minimum, all of input, main output, 3A statistics and viewfinder
192 video nodes should be enabled for IPU3 to star 190 video nodes should be enabled for IPU3 to start image processing.
193 191
194 Each ImgU V4L2 subdev has the following set of 192 Each ImgU V4L2 subdev has the following set of video nodes.
195 193
196 input, output and viewfinder video nodes 194 input, output and viewfinder video nodes
197 ---------------------------------------- 195 ----------------------------------------
198 196
199 The frames (in packed raw Bayer format specifi 197 The frames (in packed raw Bayer format specific to the IPU3) received by the
200 input video node is processed by the IPU3 Imag 198 input video node is processed by the IPU3 Imaging Unit and are output to 2 video
201 nodes, with each targeting a different purpose 199 nodes, with each targeting a different purpose (main output and viewfinder
202 output). 200 output).
203 201
204 Details onand the Bayer format specific to the 202 Details onand the Bayer format specific to the IPU3 can be found in
205 :ref:`v4l2-pix-fmt-ipu3-sbggr10`. 203 :ref:`v4l2-pix-fmt-ipu3-sbggr10`.
206 204
207 The driver supports V4L2 Video Capture Interfa 205 The driver supports V4L2 Video Capture Interface as defined at :ref:`devices`.
208 206
209 Only the multi-planar API is supported. More d 207 Only the multi-planar API is supported. More details can be found at
210 :ref:`planar-apis`. 208 :ref:`planar-apis`.
211 209
212 Parameters video node 210 Parameters video node
213 --------------------- 211 ---------------------
214 212
215 The parameters video node receives the ImgU al 213 The parameters video node receives the ImgU algorithm parameters that are used
216 to configure how the ImgU algorithms process t 214 to configure how the ImgU algorithms process the image.
217 215
218 Details on processing parameters specific to t 216 Details on processing parameters specific to the IPU3 can be found in
219 :ref:`v4l2-meta-fmt-params`. 217 :ref:`v4l2-meta-fmt-params`.
220 218
221 3A statistics video node 219 3A statistics video node
222 ------------------------ 220 ------------------------
223 221
224 3A statistics video node is used by the ImgU d 222 3A statistics video node is used by the ImgU driver to output the 3A (auto
225 focus, auto exposure and auto white balance) s 223 focus, auto exposure and auto white balance) statistics for the frames that are
226 being processed by the ImgU to user space appl 224 being processed by the ImgU to user space applications. User space applications
227 can use this statistics data to compute the de 225 can use this statistics data to compute the desired algorithm parameters for
228 the ImgU. 226 the ImgU.
229 227
230 Configuring the Intel IPU3 228 Configuring the Intel IPU3
231 ========================== 229 ==========================
232 230
233 The IPU3 ImgU pipelines can be configured usin 231 The IPU3 ImgU pipelines can be configured using the Media Controller, defined at
234 :ref:`media_controller`. 232 :ref:`media_controller`.
235 233
236 Running mode and firmware binary selection 234 Running mode and firmware binary selection
237 ------------------------------------------ 235 ------------------------------------------
238 236
239 ImgU works based on firmware, currently the Im !! 237 ImgU works based on firmware, currently the ImgU firmware support run 2 pipes in
240 in time-sharing with single input frame data. !! 238 time-sharing with single input frame data. Each pipe can run at certain mode -
241 - "VIDEO" or "STILL", "VIDEO" mode is commonly !! 239 "VIDEO" or "STILL", "VIDEO" mode is commonly used for video frames capture, and
242 and "STILL" is used for still frame capture. H !! 240 "STILL" is used for still frame capture. However, you can also select "VIDEO" to
243 "VIDEO" to capture still frames if you want to !! 241 capture still frames if you want to capture images with less system load and
244 load and power. For "STILL" mode, ImgU will tr !! 242 power. For "STILL" mode, ImgU will try to use smaller BDS factor and output
245 output larger bayer frame for further YUV proc !! 243 larger bayer frame for further YUV processing than "VIDEO" mode to get high
246 high quality images. Besides, "STILL" mode nee !! 244 quality images. Besides, "STILL" mode need XNR3 to do noise reduction, hence
247 hence "STILL" mode will need more power and me !! 245 "STILL" mode will need more power and memory bandwidth than "VIDEO" mode. TNR
248 TNR will be enabled in "VIDEO" mode and bypass !! 246 will be enabled in "VIDEO" mode and bypassed by "STILL" mode. ImgU is running at
249 running at "VIDEO" mode by default, the user c !! 247 “VIDEO” mode by default, the user can use v4l2 control V4L2_CID_INTEL_IPU3_MODE
250 V4L2_CID_INTEL_IPU3_MODE (currently defined in !! 248 (currently defined in drivers/staging/media/ipu3/include/intel-ipu3.h) to query
251 drivers/staging/media/ipu3/include/uapi/intel- !! 249 and set the running mode. For user, there is no difference for buffer queueing
252 running mode. For user, there is no difference !! 250 between the "VIDEO" and "STILL" mode, mandatory input and main output node
253 "VIDEO" and "STILL" mode, mandatory input and !! 251 should be enabled and buffers need be queued, the statistics and the view-finder
254 enabled and buffers need be queued, the statis !! 252 queues are optional.
255 are optional. <<
256 253
257 The firmware binary will be selected according 254 The firmware binary will be selected according to current running mode, such log
258 "using binary if_to_osys_striped " or "using b 255 "using binary if_to_osys_striped " or "using binary if_to_osys_primary_striped"
259 could be observed if you enable the ImgU dynam 256 could be observed if you enable the ImgU dynamic debug, the binary
260 if_to_osys_striped is selected for "VIDEO" and 257 if_to_osys_striped is selected for "VIDEO" and the binary
261 "if_to_osys_primary_striped" is selected for " 258 "if_to_osys_primary_striped" is selected for "STILL".
262 259
263 260
264 Processing the image in raw Bayer format 261 Processing the image in raw Bayer format
265 ---------------------------------------- 262 ----------------------------------------
266 263
267 Configuring ImgU V4L2 subdev for image process 264 Configuring ImgU V4L2 subdev for image processing
268 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 265 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
269 266
270 The ImgU V4L2 subdevs have to be configured wi 267 The ImgU V4L2 subdevs have to be configured with media controller APIs to have
271 all the video nodes setup correctly. 268 all the video nodes setup correctly.
272 269
273 Let us take "ipu3-imgu 0" subdev as an example 270 Let us take "ipu3-imgu 0" subdev as an example.
274 271
275 .. code-block:: none !! 272 media-ctl -d $MDEV -r
>> 273
>> 274 media-ctl -d $MDEV -l "ipu3-imgu 0 input":0 -> "ipu3-imgu 0":0[1]
276 275
277 media-ctl -d $MDEV -r !! 276 media-ctl -d $MDEV -l "ipu3-imgu 0":2 -> "ipu3-imgu 0 output":0[1]
278 media-ctl -d $MDEV -l "ipu3-imgu 0 input": !! 277
279 media-ctl -d $MDEV -l "ipu3-imgu 0":2 -> " !! 278 media-ctl -d $MDEV -l "ipu3-imgu 0":3 -> "ipu3-imgu 0 viewfinder":0[1]
280 media-ctl -d $MDEV -l "ipu3-imgu 0":3 -> " !! 279
281 media-ctl -d $MDEV -l "ipu3-imgu 0":4 -> " !! 280 media-ctl -d $MDEV -l "ipu3-imgu 0":4 -> "ipu3-imgu 0 3a stat":0[1]
282 281
283 Also the pipe mode of the corresponding V4L2 s 282 Also the pipe mode of the corresponding V4L2 subdev should be set as desired
284 (e.g 0 for video mode or 1 for still mode) thr 283 (e.g 0 for video mode or 1 for still mode) through the control id 0x009819a1 as
285 below. 284 below.
286 285
287 .. code-block:: none !! 286 yavta -w "0x009819A1 1" /dev/v4l-subdev7
288 <<
289 yavta -w "0x009819A1 1" /dev/v4l-subdev7 <<
290 287
291 Certain hardware blocks in ImgU pipeline can c 288 Certain hardware blocks in ImgU pipeline can change the frame resolution by
292 cropping or scaling, these hardware blocks inc 289 cropping or scaling, these hardware blocks include Input Feeder(IF), Bayer Down
293 Scaler (BDS) and Geometric Distortion Correcti 290 Scaler (BDS) and Geometric Distortion Correction (GDC).
294 There is also a block which can change the fra 291 There is also a block which can change the frame resolution - YUV Scaler, it is
295 only applicable to the secondary output. 292 only applicable to the secondary output.
296 293
297 RAW Bayer frames go through these ImgU pipelin 294 RAW Bayer frames go through these ImgU pipeline hardware blocks and the final
298 processed image output to the DDR memory. 295 processed image output to the DDR memory.
299 296
300 .. kernel-figure:: ipu3_rcb.svg 297 .. kernel-figure:: ipu3_rcb.svg
301 :alt: ipu3 resolution blocks image 298 :alt: ipu3 resolution blocks image
302 299
303 IPU3 resolution change hardware blocks 300 IPU3 resolution change hardware blocks
304 301
305 **Input Feeder** 302 **Input Feeder**
306 303
307 Input Feeder gets the Bayer frame data from th 304 Input Feeder gets the Bayer frame data from the sensor, it can enable cropping
308 of lines and columns from the frame and then s 305 of lines and columns from the frame and then store pixels into device's internal
309 pixel buffer which are ready to readout by fol 306 pixel buffer which are ready to readout by following blocks.
310 307
311 **Bayer Down Scaler** 308 **Bayer Down Scaler**
312 309
313 Bayer Down Scaler is capable of performing ima 310 Bayer Down Scaler is capable of performing image scaling in Bayer domain, the
314 downscale factor can be configured from 1X to 311 downscale factor can be configured from 1X to 1/4X in each axis with
315 configuration steps of 0.03125 (1/32). 312 configuration steps of 0.03125 (1/32).
316 313
317 **Geometric Distortion Correction** 314 **Geometric Distortion Correction**
318 315
319 Geometric Distortion Correction is used to per !! 316 Geometric Distortion Correction is used to performe correction of distortions
320 and image filtering. It needs some extra filte !! 317 and image filtering. It needs some extra filter and envelop padding pixels to
321 work, so the input resolution of GDC should be 318 work, so the input resolution of GDC should be larger than the output
322 resolution. 319 resolution.
323 320
324 **YUV Scaler** 321 **YUV Scaler**
325 322
326 YUV Scaler which similar with BDS, but it is m 323 YUV Scaler which similar with BDS, but it is mainly do image down scaling in
327 YUV domain, it can support up to 1/12X down sc 324 YUV domain, it can support up to 1/12X down scaling, but it can not be applied
328 to the main output. 325 to the main output.
329 326
330 The ImgU V4L2 subdev has to be configured with 327 The ImgU V4L2 subdev has to be configured with the supported resolutions in all
331 the above hardware blocks, for a given input r 328 the above hardware blocks, for a given input resolution.
332 For a given supported resolution for an input 329 For a given supported resolution for an input frame, the Input Feeder, Bayer
333 Down Scaler and GDC blocks should be configure 330 Down Scaler and GDC blocks should be configured with the supported resolutions
334 as each hardware block has its own alignment r 331 as each hardware block has its own alignment requirement.
335 332
336 You must configure the output resolution of th 333 You must configure the output resolution of the hardware blocks smartly to meet
337 the hardware requirement along with keeping th 334 the hardware requirement along with keeping the maximum field of view. The
338 intermediate resolutions can be generated by s 335 intermediate resolutions can be generated by specific tool -
339 336
340 https://github.com/intel/intel-ipu3-pipecfg 337 https://github.com/intel/intel-ipu3-pipecfg
341 338
342 This tool can be used to generate intermediate 339 This tool can be used to generate intermediate resolutions. More information can
343 be obtained by looking at the following IPU3 I 340 be obtained by looking at the following IPU3 ImgU configuration table.
344 341
345 https://chromium.googlesource.com/chromiumos/o 342 https://chromium.googlesource.com/chromiumos/overlays/board-overlays/+/master
346 343
347 Under baseboard-poppy/media-libs/cros-camera-h 344 Under baseboard-poppy/media-libs/cros-camera-hal-configs-poppy/files/gcss
348 directory, graph_settings_ov5670.xml can be us 345 directory, graph_settings_ov5670.xml can be used as an example.
349 346
350 The following steps prepare the ImgU pipeline 347 The following steps prepare the ImgU pipeline for the image processing.
351 348
352 1. The ImgU V4L2 subdev data format should be 349 1. The ImgU V4L2 subdev data format should be set by using the
353 VIDIOC_SUBDEV_S_FMT on pad 0, using the GDC wi 350 VIDIOC_SUBDEV_S_FMT on pad 0, using the GDC width and height obtained above.
354 351
355 2. The ImgU V4L2 subdev cropping should be set 352 2. The ImgU V4L2 subdev cropping should be set by using the
356 VIDIOC_SUBDEV_S_SELECTION on pad 0, with V4L2_ 353 VIDIOC_SUBDEV_S_SELECTION on pad 0, with V4L2_SEL_TGT_CROP as the target,
357 using the input feeder height and width. 354 using the input feeder height and width.
358 355
359 3. The ImgU V4L2 subdev composing should be se 356 3. The ImgU V4L2 subdev composing should be set by using the
360 VIDIOC_SUBDEV_S_SELECTION on pad 0, with V4L2_ 357 VIDIOC_SUBDEV_S_SELECTION on pad 0, with V4L2_SEL_TGT_COMPOSE as the target,
361 using the BDS height and width. 358 using the BDS height and width.
362 359
363 For the ov5670 example, for an input frame wit 360 For the ov5670 example, for an input frame with a resolution of 2592x1944
364 (which is input to the ImgU subdev pad 0), the 361 (which is input to the ImgU subdev pad 0), the corresponding resolutions
365 for input feeder, BDS and GDC are 2592x1944, 2 362 for input feeder, BDS and GDC are 2592x1944, 2592x1944 and 2560x1920
366 respectively. 363 respectively.
367 364
368 Once this is done, the received raw Bayer fram 365 Once this is done, the received raw Bayer frames can be input to the ImgU
369 V4L2 subdev as below, using the open source ap 366 V4L2 subdev as below, using the open source application v4l2n [#f1]_.
370 367
371 For an image captured with 2592x1944 [#f4]_ re 368 For an image captured with 2592x1944 [#f4]_ resolution, with desired output
372 resolution as 2560x1920 and viewfinder resolut 369 resolution as 2560x1920 and viewfinder resolution as 2560x1920, the following
373 v4l2n command can be used. This helps process 370 v4l2n command can be used. This helps process the raw Bayer frames and produces
374 the desired results for the main output image 371 the desired results for the main output image and the viewfinder output, in NV12
375 format. 372 format.
376 373
377 .. code-block:: none !! 374 v4l2n --pipe=4 --load=/tmp/frame-#.bin --open=/dev/video4
378 !! 375 --fmt=type:VIDEO_OUTPUT_MPLANE,width=2592,height=1944,pixelformat=0X47337069
379 v4l2n --pipe=4 --load=/tmp/frame-#.bin --o !! 376 --reqbufs=type:VIDEO_OUTPUT_MPLANE,count:1 --pipe=1 --output=/tmp/frames.out
380 --fmt=type:VIDEO_OUTPUT_MPLANE,width !! 377 --open=/dev/video5
381 --reqbufs=type:VIDEO_OUTPUT_MPLANE,c !! 378 --fmt=type:VIDEO_CAPTURE_MPLANE,width=2560,height=1920,pixelformat=NV12
382 --output=/tmp/frames.out --open=/dev !! 379 --reqbufs=type:VIDEO_CAPTURE_MPLANE,count:1 --pipe=2 --output=/tmp/frames.vf
383 --fmt=type:VIDEO_CAPTURE_MPLANE,widt !! 380 --open=/dev/video6
384 --reqbufs=type:VIDEO_CAPTURE_MPLANE, !! 381 --fmt=type:VIDEO_CAPTURE_MPLANE,width=2560,height=1920,pixelformat=NV12
385 --output=/tmp/frames.vf --open=/dev/ !! 382 --reqbufs=type:VIDEO_CAPTURE_MPLANE,count:1 --pipe=3 --open=/dev/video7
386 --fmt=type:VIDEO_CAPTURE_MPLANE,widt !! 383 --output=/tmp/frames.3A --fmt=type:META_CAPTURE,?
387 --reqbufs=type:VIDEO_CAPTURE_MPLANE, !! 384 --reqbufs=count:1,type:META_CAPTURE --pipe=1,2,3,4 --stream=5
388 --output=/tmp/frames.3A --fmt=type:M <<
389 --reqbufs=count:1,type:META_CAPTURE <<
390 385
391 You can also use yavta [#f2]_ command to do sa 386 You can also use yavta [#f2]_ command to do same thing as above:
392 387
393 .. code-block:: none 388 .. code-block:: none
394 389
395 yavta --data-prefix -Bcapture-mplane -c10 !! 390 yavta --data-prefix -Bcapture-mplane -c10 -n5 -I -s2592x1944 \
396 --file=frame-#.out-f NV12 /dev/video !! 391 --file=frame-#.out-f NV12 /dev/video5 & \
397 yavta --data-prefix -Bcapture-mplane -c10 !! 392 yavta --data-prefix -Bcapture-mplane -c10 -n5 -I -s2592x1944 \
398 --file=frame-#.vf -f NV12 /dev/video !! 393 --file=frame-#.vf -f NV12 /dev/video6 & \
399 yavta --data-prefix -Bmeta-capture -c10 -n !! 394 yavta --data-prefix -Bmeta-capture -c10 -n5 -I \
400 --file=frame-#.3a /dev/video7 & \ !! 395 --file=frame-#.3a /dev/video7 & \
401 yavta --data-prefix -Boutput-mplane -c10 - !! 396 yavta --data-prefix -Boutput-mplane -c10 -n5 -I -s2592x1944 \
402 --file=/tmp/frame-in.cio2 -f IPU3_SG !! 397 --file=/tmp/frame-in.cio2 -f IPU3_SGRBG10 /dev/video4
403 398
404 where /dev/video4, /dev/video5, /dev/video6 an 399 where /dev/video4, /dev/video5, /dev/video6 and /dev/video7 devices point to
405 input, output, viewfinder and 3A statistics vi 400 input, output, viewfinder and 3A statistics video nodes respectively.
406 401
407 Converting the raw Bayer image into YUV domain 402 Converting the raw Bayer image into YUV domain
408 ---------------------------------------------- 403 ----------------------------------------------
409 404
410 The processed images after the above step, can 405 The processed images after the above step, can be converted to YUV domain
411 as below. 406 as below.
412 407
413 Main output frames 408 Main output frames
414 ~~~~~~~~~~~~~~~~~~ 409 ~~~~~~~~~~~~~~~~~~
415 410
416 .. code-block:: none !! 411 raw2pnm -x2560 -y1920 -fNV12 /tmp/frames.out /tmp/frames.out.ppm
417 <<
418 raw2pnm -x2560 -y1920 -fNV12 /tmp/frames.o <<
419 412
420 where 2560x1920 is output resolution, NV12 is 413 where 2560x1920 is output resolution, NV12 is the video format, followed
421 by input frame and output PNM file. 414 by input frame and output PNM file.
422 415
423 Viewfinder output frames 416 Viewfinder output frames
424 ~~~~~~~~~~~~~~~~~~~~~~~~ 417 ~~~~~~~~~~~~~~~~~~~~~~~~
425 418
426 .. code-block:: none !! 419 raw2pnm -x2560 -y1920 -fNV12 /tmp/frames.vf /tmp/frames.vf.ppm
427 <<
428 raw2pnm -x2560 -y1920 -fNV12 /tmp/frames.v <<
429 420
430 where 2560x1920 is output resolution, NV12 is 421 where 2560x1920 is output resolution, NV12 is the video format, followed
431 by input frame and output PNM file. 422 by input frame and output PNM file.
432 423
433 Example user space code for IPU3 424 Example user space code for IPU3
434 ================================ 425 ================================
435 426
436 User space code that configures and uses IPU3 427 User space code that configures and uses IPU3 is available here.
437 428
438 https://chromium.googlesource.com/chromiumos/p 429 https://chromium.googlesource.com/chromiumos/platform/arc-camera/+/master/
439 430
440 The source can be located under hal/intel dire 431 The source can be located under hal/intel directory.
441 432
442 Overview of IPU3 pipeline 433 Overview of IPU3 pipeline
443 ========================= 434 =========================
444 435
445 IPU3 pipeline has a number of image processing 436 IPU3 pipeline has a number of image processing stages, each of which takes a
446 set of parameters as input. The major stages o 437 set of parameters as input. The major stages of pipelines are shown here:
447 438
448 .. kernel-render:: DOT 439 .. kernel-render:: DOT
449 :alt: IPU3 ImgU Pipeline 440 :alt: IPU3 ImgU Pipeline
450 :caption: IPU3 ImgU Pipeline Diagram 441 :caption: IPU3 ImgU Pipeline Diagram
451 442
452 digraph "IPU3 ImgU" { 443 digraph "IPU3 ImgU" {
453 node [shape=box] 444 node [shape=box]
454 splines="ortho" 445 splines="ortho"
455 rankdir="LR" 446 rankdir="LR"
456 447
457 a [label="Raw pixels"] 448 a [label="Raw pixels"]
458 b [label="Bayer Downscaling"] 449 b [label="Bayer Downscaling"]
459 c [label="Optical Black Correction"] 450 c [label="Optical Black Correction"]
460 d [label="Linearization"] 451 d [label="Linearization"]
461 e [label="Lens Shading Correction"] 452 e [label="Lens Shading Correction"]
462 f [label="White Balance / Exposure / Fo 453 f [label="White Balance / Exposure / Focus Apply"]
463 g [label="Bayer Noise Reduction"] 454 g [label="Bayer Noise Reduction"]
464 h [label="ANR"] 455 h [label="ANR"]
465 i [label="Demosaicing"] 456 i [label="Demosaicing"]
466 j [label="Color Correction Matrix"] 457 j [label="Color Correction Matrix"]
467 k [label="Gamma correction"] 458 k [label="Gamma correction"]
468 l [label="Color Space Conversion"] 459 l [label="Color Space Conversion"]
469 m [label="Chroma Down Scaling"] 460 m [label="Chroma Down Scaling"]
470 n [label="Chromatic Noise Reduction"] 461 n [label="Chromatic Noise Reduction"]
471 o [label="Total Color Correction"] 462 o [label="Total Color Correction"]
472 p [label="XNR3"] 463 p [label="XNR3"]
473 q [label="TNR"] 464 q [label="TNR"]
474 r [label="DDR", style=filled, fillcolor 465 r [label="DDR", style=filled, fillcolor=yellow, shape=cylinder]
475 s [label="YUV Downscaling"] 466 s [label="YUV Downscaling"]
476 t [label="DDR", style=filled, fillcolor 467 t [label="DDR", style=filled, fillcolor=yellow, shape=cylinder]
477 468
478 { rank=same; a -> b -> c -> d -> e -> f 469 { rank=same; a -> b -> c -> d -> e -> f -> g -> h -> i }
479 { rank=same; j -> k -> l -> m -> n -> o 470 { rank=same; j -> k -> l -> m -> n -> o -> p -> q -> s -> t}
480 471
481 a -> j [style=invis, weight=10] 472 a -> j [style=invis, weight=10]
482 i -> j 473 i -> j
483 q -> r 474 q -> r
484 } 475 }
485 476
486 The table below presents a description of the 477 The table below presents a description of the above algorithms.
487 478
488 ======================== ===================== 479 ======================== =======================================================
489 Name Description 480 Name Description
490 ======================== ===================== 481 ======================== =======================================================
491 Optical Black Correction Optical Black Correct 482 Optical Black Correction Optical Black Correction block subtracts a pre-defined
492 value from the respec 483 value from the respective pixel values to obtain better
493 image quality. 484 image quality.
494 Defined in struct ipu !! 485 Defined in :c:type:`ipu3_uapi_obgrid_param`.
495 Linearization This algo block uses 486 Linearization This algo block uses linearization parameters to
496 address non-linearity 487 address non-linearity sensor effects. The Lookup table
497 table is defined in 488 table is defined in
498 struct ipu3_uapi_isp_ !! 489 :c:type:`ipu3_uapi_isp_lin_vmem_params`.
499 SHD Lens shading correcti 490 SHD Lens shading correction is used to correct spatial
500 non-uniformity of the 491 non-uniformity of the pixel response due to optical
501 lens shading. This is 492 lens shading. This is done by applying a different gain
502 for each pixel. The g 493 for each pixel. The gain, black level etc are
503 configured in struct !! 494 configured in :c:type:`ipu3_uapi_shd_config_static`.
504 BNR Bayer noise reduction 495 BNR Bayer noise reduction block removes image noise by
505 applying a bilateral 496 applying a bilateral filter.
506 See struct ipu3_uapi_ !! 497 See :c:type:`ipu3_uapi_bnr_static_config` for details.
507 ANR Advanced Noise Reduct 498 ANR Advanced Noise Reduction is a block based algorithm
508 that performs noise r 499 that performs noise reduction in the Bayer domain. The
509 convolution matrix et 500 convolution matrix etc can be found in
510 struct ipu3_uapi_anr_ !! 501 :c:type:`ipu3_uapi_anr_config`.
511 DM Demosaicing converts 502 DM Demosaicing converts raw sensor data in Bayer format
512 into RGB (Red, Green, 503 into RGB (Red, Green, Blue) presentation. Then add
513 outputs of estimation 504 outputs of estimation of Y channel for following stream
514 processing by Firmwar 505 processing by Firmware. The struct is defined as
515 struct ipu3_uapi_dm_c !! 506 :c:type:`ipu3_uapi_dm_config`.
516 Color Correction Color Correction algo 507 Color Correction Color Correction algo transforms sensor specific color
517 space to the standard 508 space to the standard "sRGB" color space. This is done
518 by applying 3x3 matri 509 by applying 3x3 matrix defined in
519 struct ipu3_uapi_ccm_ !! 510 :c:type:`ipu3_uapi_ccm_mat_config`.
520 Gamma correction Gamma correction stru !! 511 Gamma correction Gamma correction :c:type:`ipu3_uapi_gamma_config` is a
521 basic non-linear tone 512 basic non-linear tone mapping correction that is
522 applied per pixel for 513 applied per pixel for each pixel component.
523 CSC Color space conversio 514 CSC Color space conversion transforms each pixel from the
524 RGB primary presentat 515 RGB primary presentation to YUV (Y: brightness,
525 UV: Luminance) presen 516 UV: Luminance) presentation. This is done by applying
526 a 3x3 matrix defined 517 a 3x3 matrix defined in
527 struct ipu3_uapi_csc_ !! 518 :c:type:`ipu3_uapi_csc_mat_config`
528 CDS Chroma down sampling 519 CDS Chroma down sampling
529 After the CSC is perf 520 After the CSC is performed, the Chroma Down Sampling
530 is applied for a UV p 521 is applied for a UV plane down sampling by a factor
531 of 2 in each directio 522 of 2 in each direction for YUV 4:2:0 using a 4x2
532 configurable filter s !! 523 configurable filter :c:type:`ipu3_uapi_cds_params`.
533 CHNR Chroma noise reductio 524 CHNR Chroma noise reduction
534 This block processes 525 This block processes only the chrominance pixels and
535 performs noise reduct 526 performs noise reduction by cleaning the high
536 frequency noise. 527 frequency noise.
537 See struct struct ipu !! 528 See struct :c:type:`ipu3_uapi_yuvp1_chnr_config`.
538 TCC Total color correctio 529 TCC Total color correction as defined in struct
539 struct ipu3_uapi_yuvp !! 530 :c:type:`ipu3_uapi_yuvp2_tcc_static_config`.
540 XNR3 eXtreme Noise Reducti 531 XNR3 eXtreme Noise Reduction V3 is the third revision of
541 noise reduction algor 532 noise reduction algorithm used to improve image
542 quality. This removes 533 quality. This removes the low frequency noise in the
543 captured image. Two r 534 captured image. Two related structs are being defined,
544 struct ipu3_uapi_isp_ !! 535 :c:type:`ipu3_uapi_isp_xnr3_params` for ISP data memory
545 and struct ipu3_uapi_ !! 536 and :c:type:`ipu3_uapi_isp_xnr3_vmem_params` for vector
546 memory. 537 memory.
547 TNR Temporal Noise Reduct 538 TNR Temporal Noise Reduction block compares successive
548 frames in time to rem 539 frames in time to remove anomalies / noise in pixel
549 values. struct ipu3_u !! 540 values. :c:type:`ipu3_uapi_isp_tnr3_vmem_params` and
550 struct ipu3_uapi_isp_ !! 541 :c:type:`ipu3_uapi_isp_tnr3_params` are defined for ISP
551 vector and data memor 542 vector and data memory respectively.
552 ======================== ===================== 543 ======================== =======================================================
553 544
554 Other often encountered acronyms not listed in 545 Other often encountered acronyms not listed in above table:
555 546
556 ACC 547 ACC
557 Accelerator cluster 548 Accelerator cluster
558 AWB_FR 549 AWB_FR
559 Auto white balance filter resp 550 Auto white balance filter response statistics
560 BDS 551 BDS
561 Bayer downscaler parameters 552 Bayer downscaler parameters
562 CCM 553 CCM
563 Color correction matrix coeffi 554 Color correction matrix coefficients
564 IEFd 555 IEFd
565 Image enhancement filter direc 556 Image enhancement filter directed
566 Obgrid 557 Obgrid
567 Optical black level compensati 558 Optical black level compensation
568 OSYS 559 OSYS
569 Output system configuration 560 Output system configuration
570 ROI 561 ROI
571 Region of interest 562 Region of interest
572 YDS 563 YDS
573 Y down sampling 564 Y down sampling
574 YTM 565 YTM
575 Y-tone mapping 566 Y-tone mapping
576 567
577 A few stages of the pipeline will be executed 568 A few stages of the pipeline will be executed by firmware running on the ISP
578 processor, while many others will use a set of 569 processor, while many others will use a set of fixed hardware blocks also
579 called accelerator cluster (ACC) to crunch pix 570 called accelerator cluster (ACC) to crunch pixel data and produce statistics.
580 571
581 ACC parameters of individual algorithms, as de 572 ACC parameters of individual algorithms, as defined by
582 struct ipu3_uapi_acc_param, can be chosen to b !! 573 :c:type:`ipu3_uapi_acc_param`, can be chosen to be applied by the user
583 space through struct struct ipu3_uapi_flags em !! 574 space through struct :c:type:`ipu3_uapi_flags` embedded in
584 struct ipu3_uapi_params structure. For paramet !! 575 :c:type:`ipu3_uapi_params` structure. For parameters that are configured as
585 not enabled by the user space, the correspondi 576 not enabled by the user space, the corresponding structs are ignored by the
586 driver, in which case the existing configurati 577 driver, in which case the existing configuration of the algorithm will be
587 preserved. 578 preserved.
588 579
589 References 580 References
590 ========== 581 ==========
591 582
592 .. [#f5] drivers/staging/media/ipu3/include/ua !! 583 .. [#f5] drivers/staging/media/ipu3/include/intel-ipu3.h
593 584
594 .. [#f1] https://github.com/intel/nvt 585 .. [#f1] https://github.com/intel/nvt
595 586
596 .. [#f2] http://git.ideasonboard.org/yavta.git 587 .. [#f2] http://git.ideasonboard.org/yavta.git
597 588
598 .. [#f3] http://git.ideasonboard.org/?p=media- 589 .. [#f3] http://git.ideasonboard.org/?p=media-ctl.git;a=summary
599 590
600 .. [#f4] ImgU limitation requires an additiona 591 .. [#f4] ImgU limitation requires an additional 16x16 for all input resolutions
Linux® is a registered trademark of Linus Torvalds in the United States and other countries.
TOMOYO® is a registered trademark of NTT DATA CORPORATION.