1 .. SPDX-License-Identifier: GPL-2.0 1 .. SPDX-License-Identifier: GPL-2.0
2 2
3 ============================================== 3 =========================================================
4 Raspberry Pi PiSP Back End Memory-to-Memory IS 4 Raspberry Pi PiSP Back End Memory-to-Memory ISP (pisp-be)
5 ============================================== 5 =========================================================
6 6
7 The PiSP Back End 7 The PiSP Back End
8 ================= 8 =================
9 9
10 The PiSP Back End is a memory-to-memory Image 10 The PiSP Back End is a memory-to-memory Image Signal Processor (ISP) which reads
11 image data from DRAM memory and performs image 11 image data from DRAM memory and performs image processing as specified by the
12 application through the parameters in a config 12 application through the parameters in a configuration buffer, before writing
13 pixel data back to memory through two distinct 13 pixel data back to memory through two distinct output channels.
14 14
15 The ISP registers and programming model are do 15 The ISP registers and programming model are documented in the `Raspberry Pi
16 Image Signal Processor (PiSP) Specification do 16 Image Signal Processor (PiSP) Specification document`_
17 17
18 The PiSP Back End ISP processes images in tile 18 The PiSP Back End ISP processes images in tiles. The handling of image
19 tessellation and the computation of low-level 19 tessellation and the computation of low-level configuration parameters is
20 realized by a free software library called `li 20 realized by a free software library called `libpisp
21 <https://github.com/raspberrypi/libpisp>`_. 21 <https://github.com/raspberrypi/libpisp>`_.
22 22
23 The full image processing pipeline, which invo 23 The full image processing pipeline, which involves capturing RAW Bayer data from
24 an image sensor through a MIPI CSI-2 compatibl 24 an image sensor through a MIPI CSI-2 compatible capture interface, storing them
25 in DRAM memory and processing them in the PiSP 25 in DRAM memory and processing them in the PiSP Back End to obtain images usable
26 by an application is implemented in `libcamera 26 by an application is implemented in `libcamera <https://libcamera.org>`_ as
27 part of the Raspberry Pi platform support. 27 part of the Raspberry Pi platform support.
28 28
29 The pisp-be driver 29 The pisp-be driver
30 ================== 30 ==================
31 31
32 The Raspberry Pi PiSP Back End (pisp-be) drive 32 The Raspberry Pi PiSP Back End (pisp-be) driver is located under
33 drivers/media/platform/raspberrypi/pisp-be. It 33 drivers/media/platform/raspberrypi/pisp-be. It uses the `V4L2 API` to register
34 a number of video capture and output devices, 34 a number of video capture and output devices, the `V4L2 subdev API` to register
35 a subdevice for the ISP that connects the vide 35 a subdevice for the ISP that connects the video devices in a single media graph
36 realized using the `Media Controller (MC) API` 36 realized using the `Media Controller (MC) API`.
37 37
38 The media topology registered by the `pisp-be` 38 The media topology registered by the `pisp-be` driver is represented below:
39 39
40 .. _pips-be-topology: 40 .. _pips-be-topology:
41 41
42 .. kernel-figure:: raspberrypi-pisp-be.dot 42 .. kernel-figure:: raspberrypi-pisp-be.dot
43 :alt: Diagram of the default media pipel 43 :alt: Diagram of the default media pipeline topology
44 :align: center 44 :align: center
45 45
46 46
47 The media graph registers the following video 47 The media graph registers the following video device nodes:
48 48
49 - pispbe-input: output device for images to be 49 - pispbe-input: output device for images to be submitted to the ISP for
50 processing. 50 processing.
51 - pispbe-tdn_input: output device for temporal 51 - pispbe-tdn_input: output device for temporal denoise.
52 - pispbe-stitch_input: output device for image 52 - pispbe-stitch_input: output device for image stitching (HDR).
53 - pispbe-output0: first capture device for pro 53 - pispbe-output0: first capture device for processed images.
54 - pispbe-output1: second capture device for pr 54 - pispbe-output1: second capture device for processed images.
55 - pispbe-tdn_output: capture device for tempor 55 - pispbe-tdn_output: capture device for temporal denoise.
56 - pispbe-stitch_output: capture device for ima 56 - pispbe-stitch_output: capture device for image stitching (HDR).
57 - pispbe-config: output device for ISP configu 57 - pispbe-config: output device for ISP configuration parameters.
58 58
59 pispbe-input 59 pispbe-input
60 ------------ 60 ------------
61 61
62 Images to be processed by the ISP are queued t 62 Images to be processed by the ISP are queued to the `pispbe-input` output device
63 node. For a list of image formats supported as 63 node. For a list of image formats supported as input to the ISP refer to the
64 `Raspberry Pi Image Signal Processor (PiSP) Sp 64 `Raspberry Pi Image Signal Processor (PiSP) Specification document`_.
65 65
66 pispbe-tdn_input, pispbe-tdn_output 66 pispbe-tdn_input, pispbe-tdn_output
67 ----------------------------------- 67 -----------------------------------
68 68
69 The `pispbe-tdn_input` output video device rec 69 The `pispbe-tdn_input` output video device receives images to be processed by
70 the temporal denoise block which are captured 70 the temporal denoise block which are captured from the `pispbe-tdn_output`
71 capture video device. Userspace is responsible 71 capture video device. Userspace is responsible for maintaining queues on both
72 devices, and ensuring that buffers completed o 72 devices, and ensuring that buffers completed on the output are queued to the
73 input. 73 input.
74 74
75 pispbe-stitch_input, pispbe-stitch_output 75 pispbe-stitch_input, pispbe-stitch_output
76 ----------------------------------------- 76 -----------------------------------------
77 77
78 To realize HDR (high dynamic range) image proc 78 To realize HDR (high dynamic range) image processing the image stitching and
79 tonemapping blocks are used. The `pispbe-stitc 79 tonemapping blocks are used. The `pispbe-stitch_output` writes images to memory
80 and the `pispbe-stitch_input` receives the pre 80 and the `pispbe-stitch_input` receives the previously written frame to process
81 it along with the current input image. Userspa 81 it along with the current input image. Userspace is responsible for maintaining
82 queues on both devices, and ensuring that buff 82 queues on both devices, and ensuring that buffers completed on the output are
83 queued to the input. 83 queued to the input.
84 84
85 pispbe-output0, pispbe-output1 85 pispbe-output0, pispbe-output1
86 ------------------------------ 86 ------------------------------
87 87
88 The two capture devices write to memory the pi 88 The two capture devices write to memory the pixel data as processed by the ISP.
89 89
90 pispbe-config 90 pispbe-config
91 ------------- 91 -------------
92 92
93 The `pispbe-config` output video devices recei 93 The `pispbe-config` output video devices receives a buffer of configuration
94 parameters that define the desired image proce 94 parameters that define the desired image processing to be performed by the ISP.
95 95
96 The format of the ISP configuration parameter 96 The format of the ISP configuration parameter is defined by
97 :c:type:`pisp_be_tiles_config` C structure and 97 :c:type:`pisp_be_tiles_config` C structure and the meaning of each parameter is
98 described in the `Raspberry Pi Image Signal Pr 98 described in the `Raspberry Pi Image Signal Processor (PiSP) Specification
99 document`_. 99 document`_.
100 100
101 ISP configuration 101 ISP configuration
102 ================= 102 =================
103 103
104 The ISP configuration is described solely by t 104 The ISP configuration is described solely by the content of the parameters
105 buffer. The only parameter that userspace need 105 buffer. The only parameter that userspace needs to configure using the V4L2 API
106 is the image format on the output and capture 106 is the image format on the output and capture video devices for validation of
107 the content of the parameters buffer. 107 the content of the parameters buffer.
108 108
109 .. _Raspberry Pi Image Signal Processor (PiSP) 109 .. _Raspberry Pi Image Signal Processor (PiSP) Specification document: https://datasheets.raspberrypi.com/camera/raspberry-pi-image-signal-processor-specification.pdf
Linux® is a registered trademark of Linus Torvalds in the United States and other countries.
TOMOYO® is a registered trademark of NTT DATA CORPORATION.