1 .. SPDX-License-Identifier: GPL-2.0 1 .. SPDX-License-Identifier: GPL-2.0
2 2
3 .. include:: <isonum.txt> 3 .. include:: <isonum.txt>
4 4
5 Qualcomm Camera Subsystem driver 5 Qualcomm Camera Subsystem driver
6 ================================ 6 ================================
7 7
8 Introduction 8 Introduction
9 ------------ 9 ------------
10 10
11 This file documents the Qualcomm Camera Subsys 11 This file documents the Qualcomm Camera Subsystem driver located under
12 drivers/media/platform/qcom/camss. 12 drivers/media/platform/qcom/camss.
13 13
14 The current version of the driver supports the 14 The current version of the driver supports the Camera Subsystem found on
15 Qualcomm MSM8916/APQ8016 and MSM8996/APQ8096 p 15 Qualcomm MSM8916/APQ8016 and MSM8996/APQ8096 processors.
16 16
17 The driver implements V4L2, Media controller a 17 The driver implements V4L2, Media controller and V4L2 subdev interfaces.
18 Camera sensor using V4L2 subdev interface in t 18 Camera sensor using V4L2 subdev interface in the kernel is supported.
19 19
20 The driver is implemented using as a reference 20 The driver is implemented using as a reference the Qualcomm Camera Subsystem
21 driver for Android as found in Code Linaro [#f 21 driver for Android as found in Code Linaro [#f1]_ [#f2]_.
22 22
23 23
24 Qualcomm Camera Subsystem hardware 24 Qualcomm Camera Subsystem hardware
25 ---------------------------------- 25 ----------------------------------
26 26
27 The Camera Subsystem hardware found on 8x16 / 27 The Camera Subsystem hardware found on 8x16 / 8x96 processors and supported by
28 the driver consists of: 28 the driver consists of:
29 29
30 - 2 / 3 CSIPHY modules. They handle the Physic 30 - 2 / 3 CSIPHY modules. They handle the Physical layer of the CSI2 receivers.
31 A separate camera sensor can be connected to 31 A separate camera sensor can be connected to each of the CSIPHY module;
32 - 2 / 4 CSID (CSI Decoder) modules. They handl 32 - 2 / 4 CSID (CSI Decoder) modules. They handle the Protocol and Application
33 layer of the CSI2 receivers. A CSID can deco 33 layer of the CSI2 receivers. A CSID can decode data stream from any of the
34 CSIPHY. Each CSID also contains a TG (Test G 34 CSIPHY. Each CSID also contains a TG (Test Generator) block which can generate
35 artificial input data for test purposes; 35 artificial input data for test purposes;
36 - ISPIF (ISP Interface) module. Handles the ro 36 - ISPIF (ISP Interface) module. Handles the routing of the data streams from
37 the CSIDs to the inputs of the VFE; 37 the CSIDs to the inputs of the VFE;
38 - 1 / 2 VFE (Video Front End) module(s). Conta 38 - 1 / 2 VFE (Video Front End) module(s). Contain a pipeline of image processing
39 hardware blocks. The VFE has different input 39 hardware blocks. The VFE has different input interfaces. The PIX (Pixel) input
40 interface feeds the input data to the image 40 interface feeds the input data to the image processing pipeline. The image
41 processing pipeline contains also a scale an 41 processing pipeline contains also a scale and crop module at the end. Three
42 RDI (Raw Dump Interface) input interfaces by 42 RDI (Raw Dump Interface) input interfaces bypass the image processing
43 pipeline. The VFE also contains the AXI bus 43 pipeline. The VFE also contains the AXI bus interface which writes the output
44 data to memory. 44 data to memory.
45 45
46 46
47 Supported functionality 47 Supported functionality
48 ----------------------- 48 -----------------------
49 49
50 The current version of the driver supports: 50 The current version of the driver supports:
51 51
52 - Input from camera sensor via CSIPHY; 52 - Input from camera sensor via CSIPHY;
53 - Generation of test input data by the TG in C 53 - Generation of test input data by the TG in CSID;
54 - RDI interface of VFE 54 - RDI interface of VFE
55 55
56 - Raw dump of the input data to memory. 56 - Raw dump of the input data to memory.
57 57
58 Supported formats: 58 Supported formats:
59 59
60 - YUYV/UYVY/YVYU/VYUY (packed YUV 4:2:2 - 60 - YUYV/UYVY/YVYU/VYUY (packed YUV 4:2:2 - V4L2_PIX_FMT_YUYV /
61 V4L2_PIX_FMT_UYVY / V4L2_PIX_FMT_YVYU / 61 V4L2_PIX_FMT_UYVY / V4L2_PIX_FMT_YVYU / V4L2_PIX_FMT_VYUY);
62 - MIPI RAW8 (8bit Bayer RAW - V4L2_PIX_FMT 62 - MIPI RAW8 (8bit Bayer RAW - V4L2_PIX_FMT_SRGGB8 /
63 V4L2_PIX_FMT_SGRBG8 / V4L2_PIX_FMT_SGBRG 63 V4L2_PIX_FMT_SGRBG8 / V4L2_PIX_FMT_SGBRG8 / V4L2_PIX_FMT_SBGGR8);
64 - MIPI RAW10 (10bit packed Bayer RAW - V4L 64 - MIPI RAW10 (10bit packed Bayer RAW - V4L2_PIX_FMT_SBGGR10P /
65 V4L2_PIX_FMT_SGBRG10P / V4L2_PIX_FMT_SGR 65 V4L2_PIX_FMT_SGBRG10P / V4L2_PIX_FMT_SGRBG10P / V4L2_PIX_FMT_SRGGB10P /
66 V4L2_PIX_FMT_Y10P); 66 V4L2_PIX_FMT_Y10P);
67 - MIPI RAW12 (12bit packed Bayer RAW - V4L 67 - MIPI RAW12 (12bit packed Bayer RAW - V4L2_PIX_FMT_SRGGB12P /
68 V4L2_PIX_FMT_SGBRG12P / V4L2_PIX_FMT_SGR 68 V4L2_PIX_FMT_SGBRG12P / V4L2_PIX_FMT_SGRBG12P / V4L2_PIX_FMT_SRGGB12P).
69 - (8x96 only) MIPI RAW14 (14bit packed Bay 69 - (8x96 only) MIPI RAW14 (14bit packed Bayer RAW - V4L2_PIX_FMT_SRGGB14P /
70 V4L2_PIX_FMT_SGBRG14P / V4L2_PIX_FMT_SGR 70 V4L2_PIX_FMT_SGBRG14P / V4L2_PIX_FMT_SGRBG14P / V4L2_PIX_FMT_SRGGB14P).
71 71
72 - (8x96 only) Format conversion of the input 72 - (8x96 only) Format conversion of the input data.
73 73
74 Supported input formats: 74 Supported input formats:
75 75
76 - MIPI RAW10 (10bit packed Bayer RAW - V4L 76 - MIPI RAW10 (10bit packed Bayer RAW - V4L2_PIX_FMT_SBGGR10P / V4L2_PIX_FMT_Y10P).
77 77
78 Supported output formats: 78 Supported output formats:
79 79
80 - Plain16 RAW10 (10bit unpacked Bayer RAW 80 - Plain16 RAW10 (10bit unpacked Bayer RAW - V4L2_PIX_FMT_SBGGR10 / V4L2_PIX_FMT_Y10).
81 81
82 - PIX interface of VFE 82 - PIX interface of VFE
83 83
84 - Format conversion of the input data. 84 - Format conversion of the input data.
85 85
86 Supported input formats: 86 Supported input formats:
87 87
88 - YUYV/UYVY/YVYU/VYUY (packed YUV 4:2:2 - 88 - YUYV/UYVY/YVYU/VYUY (packed YUV 4:2:2 - V4L2_PIX_FMT_YUYV /
89 V4L2_PIX_FMT_UYVY / V4L2_PIX_FMT_YVYU / 89 V4L2_PIX_FMT_UYVY / V4L2_PIX_FMT_YVYU / V4L2_PIX_FMT_VYUY).
90 90
91 Supported output formats: 91 Supported output formats:
92 92
93 - NV12/NV21 (two plane YUV 4:2:0 - V4L2_PI 93 - NV12/NV21 (two plane YUV 4:2:0 - V4L2_PIX_FMT_NV12 / V4L2_PIX_FMT_NV21);
94 - NV16/NV61 (two plane YUV 4:2:2 - V4L2_PI 94 - NV16/NV61 (two plane YUV 4:2:2 - V4L2_PIX_FMT_NV16 / V4L2_PIX_FMT_NV61).
95 - (8x96 only) YUYV/UYVY/YVYU/VYUY (packed 95 - (8x96 only) YUYV/UYVY/YVYU/VYUY (packed YUV 4:2:2 - V4L2_PIX_FMT_YUYV /
96 V4L2_PIX_FMT_UYVY / V4L2_PIX_FMT_YVYU / 96 V4L2_PIX_FMT_UYVY / V4L2_PIX_FMT_YVYU / V4L2_PIX_FMT_VYUY).
97 97
98 - Scaling support. Configuration of the VFE 98 - Scaling support. Configuration of the VFE Encoder Scale module
99 for downscalling with ratio up to 16x. 99 for downscalling with ratio up to 16x.
100 100
101 - Cropping support. Configuration of the VFE 101 - Cropping support. Configuration of the VFE Encoder Crop module.
102 102
103 - Concurrent and independent usage of two (8x9 103 - Concurrent and independent usage of two (8x96: three) data inputs -
104 could be camera sensors and/or TG. 104 could be camera sensors and/or TG.
105 105
106 106
107 Driver Architecture and Design 107 Driver Architecture and Design
108 ------------------------------ 108 ------------------------------
109 109
110 The driver implements the V4L2 subdev interfac 110 The driver implements the V4L2 subdev interface. With the goal to model the
111 hardware links between the modules and to expo 111 hardware links between the modules and to expose a clean, logical and usable
112 interface, the driver is split into V4L2 sub-d 112 interface, the driver is split into V4L2 sub-devices as follows (8x16 / 8x96):
113 113
114 - 2 / 3 CSIPHY sub-devices - each CSIPHY is re 114 - 2 / 3 CSIPHY sub-devices - each CSIPHY is represented by a single sub-device;
115 - 2 / 4 CSID sub-devices - each CSID is repres 115 - 2 / 4 CSID sub-devices - each CSID is represented by a single sub-device;
116 - 2 / 4 ISPIF sub-devices - ISPIF is represent 116 - 2 / 4 ISPIF sub-devices - ISPIF is represented by a number of sub-devices
117 equal to the number of CSID sub-devices; 117 equal to the number of CSID sub-devices;
118 - 4 / 8 VFE sub-devices - VFE is represented b 118 - 4 / 8 VFE sub-devices - VFE is represented by a number of sub-devices equal to
119 the number of the input interfaces (3 RDI an 119 the number of the input interfaces (3 RDI and 1 PIX for each VFE).
120 120
121 The considerations to split the driver in this 121 The considerations to split the driver in this particular way are as follows:
122 122
123 - representing CSIPHY and CSID modules by a se 123 - representing CSIPHY and CSID modules by a separate sub-device for each module
124 allows to model the hardware links between t 124 allows to model the hardware links between these modules;
125 - representing VFE by a separate sub-devices f 125 - representing VFE by a separate sub-devices for each input interface allows
126 to use the input interfaces concurrently and 126 to use the input interfaces concurrently and independently as this is
127 supported by the hardware; 127 supported by the hardware;
128 - representing ISPIF by a number of sub-device 128 - representing ISPIF by a number of sub-devices equal to the number of CSID
129 sub-devices allows to create linear media co 129 sub-devices allows to create linear media controller pipelines when using two
130 cameras simultaneously. This avoids branches 130 cameras simultaneously. This avoids branches in the pipelines which otherwise
131 will require a) userspace and b) media frame 131 will require a) userspace and b) media framework (e.g. power on/off
132 operations) to make assumptions about the d 132 operations) to make assumptions about the data flow from a sink pad to a
133 source pad on a single media entity. 133 source pad on a single media entity.
134 134
135 Each VFE sub-device is linked to a separate vi 135 Each VFE sub-device is linked to a separate video device node.
136 136
137 The media controller pipeline graph is as foll 137 The media controller pipeline graph is as follows (with connected two / three
138 OV5645 camera sensors): 138 OV5645 camera sensors):
139 139
140 .. _qcom_camss_graph: 140 .. _qcom_camss_graph:
141 141
142 .. kernel-figure:: qcom_camss_graph.dot 142 .. kernel-figure:: qcom_camss_graph.dot
143 :alt: qcom_camss_graph.dot 143 :alt: qcom_camss_graph.dot
144 :align: center 144 :align: center
145 145
146 Media pipeline graph 8x16 146 Media pipeline graph 8x16
147 147
148 .. kernel-figure:: qcom_camss_8x96_graph.dot 148 .. kernel-figure:: qcom_camss_8x96_graph.dot
149 :alt: qcom_camss_8x96_graph.dot 149 :alt: qcom_camss_8x96_graph.dot
150 :align: center 150 :align: center
151 151
152 Media pipeline graph 8x96 152 Media pipeline graph 8x96
153 153
154 154
155 Implementation 155 Implementation
156 -------------- 156 --------------
157 157
158 Runtime configuration of the hardware (updatin 158 Runtime configuration of the hardware (updating settings while streaming) is
159 not required to implement the currently suppor 159 not required to implement the currently supported functionality. The complete
160 configuration on each hardware module is appli 160 configuration on each hardware module is applied on STREAMON ioctl based on
161 the current active media links, formats and co 161 the current active media links, formats and controls set.
162 162
163 The output size of the scaler module in the VF 163 The output size of the scaler module in the VFE is configured with the actual
164 compose selection rectangle on the sink pad of 164 compose selection rectangle on the sink pad of the 'msm_vfe0_pix' entity.
165 165
166 The crop output area of the crop module in the 166 The crop output area of the crop module in the VFE is configured with the actual
167 crop selection rectangle on the source pad of 167 crop selection rectangle on the source pad of the 'msm_vfe0_pix' entity.
168 168
169 169
170 Documentation 170 Documentation
171 ------------- 171 -------------
172 172
173 APQ8016 Specification: 173 APQ8016 Specification:
174 https://developer.qualcomm.com/download/sd410/ 174 https://developer.qualcomm.com/download/sd410/snapdragon-410-processor-device-specification.pdf
175 Referenced 2016-11-24. 175 Referenced 2016-11-24.
176 176
177 APQ8096 Specification: 177 APQ8096 Specification:
178 https://developer.qualcomm.com/download/sd820e 178 https://developer.qualcomm.com/download/sd820e/qualcomm-snapdragon-820e-processor-apq8096sge-device-specification.pdf
179 Referenced 2018-06-22. 179 Referenced 2018-06-22.
180 180
181 References 181 References
182 ---------- 182 ----------
183 183
184 .. [#f1] https://git.codelinaro.org/clo/la/ker 184 .. [#f1] https://git.codelinaro.org/clo/la/kernel/msm-3.10/
185 .. [#f2] https://git.codelinaro.org/clo/la/ker 185 .. [#f2] https://git.codelinaro.org/clo/la/kernel/msm-3.18/
Linux® is a registered trademark of Linus Torvalds in the United States and other countries.
TOMOYO® is a registered trademark of NTT DATA CORPORATION.