1 .. SPDX-License-Identifier: GPL-2.0+ 1 .. SPDX-License-Identifier: GPL-2.0+
2 2
3 ========================================= 3 =========================================
4 Automated testing of the DRM subsystem 4 Automated testing of the DRM subsystem
5 ========================================= 5 =========================================
6 6
7 Introduction 7 Introduction
8 ============ 8 ============
9 9
10 Making sure that changes to the core or driver 10 Making sure that changes to the core or drivers don't introduce regressions can
11 be very time-consuming when lots of different 11 be very time-consuming when lots of different hardware configurations need to
12 be tested. Moreover, it isn't practical for ea 12 be tested. Moreover, it isn't practical for each person interested in this
13 testing to have to acquire and maintain what c 13 testing to have to acquire and maintain what can be a considerable amount of
14 hardware. 14 hardware.
15 15
16 Also, it is desirable for developers to check 16 Also, it is desirable for developers to check for regressions in their code by
17 themselves, instead of relying on the maintain 17 themselves, instead of relying on the maintainers to find them and then
18 reporting back. 18 reporting back.
19 19
20 There are facilities in gitlab.freedesktop.org 20 There are facilities in gitlab.freedesktop.org to automatically test Mesa that
21 can be used as well for testing the DRM subsys 21 can be used as well for testing the DRM subsystem. This document explains how
22 people interested in testing it can use this s 22 people interested in testing it can use this shared infrastructure to save
23 quite some time and effort. 23 quite some time and effort.
24 24
25 25
26 Relevant files 26 Relevant files
27 ============== 27 ==============
28 28
29 drivers/gpu/drm/ci/gitlab-ci.yml 29 drivers/gpu/drm/ci/gitlab-ci.yml
30 -------------------------------- 30 --------------------------------
31 31
32 This is the root configuration file for GitLab 32 This is the root configuration file for GitLab CI. Among other less interesting
33 bits, it specifies the specific version of the 33 bits, it specifies the specific version of the scripts to be used. There are
34 some variables that can be modified to change 34 some variables that can be modified to change the behavior of the pipeline:
35 35
36 DRM_CI_PROJECT_PATH 36 DRM_CI_PROJECT_PATH
37 Repository that contains the Mesa software 37 Repository that contains the Mesa software infrastructure for CI
38 38
39 DRM_CI_COMMIT_SHA 39 DRM_CI_COMMIT_SHA
40 A particular revision to use from that rep 40 A particular revision to use from that repository
41 41
42 UPSTREAM_REPO 42 UPSTREAM_REPO
43 URL to git repository containing the targe 43 URL to git repository containing the target branch
44 44
45 TARGET_BRANCH 45 TARGET_BRANCH
46 Branch to which this branch is to be merge 46 Branch to which this branch is to be merged into
47 47
48 IGT_VERSION 48 IGT_VERSION
49 Revision of igt-gpu-tools being used, from 49 Revision of igt-gpu-tools being used, from
50 https://gitlab.freedesktop.org/drm/igt-gpu 50 https://gitlab.freedesktop.org/drm/igt-gpu-tools
51 51
52 drivers/gpu/drm/ci/testlist.txt 52 drivers/gpu/drm/ci/testlist.txt
53 ------------------------------- 53 -------------------------------
54 54
55 IGT tests to be run on all drivers (unless men 55 IGT tests to be run on all drivers (unless mentioned in a driver's \*-skips.txt
56 file, see below). 56 file, see below).
57 57
58 drivers/gpu/drm/ci/${DRIVER_NAME}-${HW_REVISIO 58 drivers/gpu/drm/ci/${DRIVER_NAME}-${HW_REVISION}-fails.txt
59 ---------------------------------------------- 59 ----------------------------------------------------------
60 60
61 Lists the known failures for a given driver on 61 Lists the known failures for a given driver on a specific hardware revision.
62 62
63 drivers/gpu/drm/ci/${DRIVER_NAME}-${HW_REVISIO 63 drivers/gpu/drm/ci/${DRIVER_NAME}-${HW_REVISION}-flakes.txt
64 ---------------------------------------------- 64 -----------------------------------------------------------
65 65
66 Lists the tests that for a given driver on a s 66 Lists the tests that for a given driver on a specific hardware revision are
67 known to behave unreliably. These tests won't 67 known to behave unreliably. These tests won't cause a job to fail regardless of
68 the result. They will still be run. 68 the result. They will still be run.
69 69
70 Each new flake entry must be associated with a 70 Each new flake entry must be associated with a link to the email reporting the
71 bug to the author of the affected driver, the 71 bug to the author of the affected driver, the board name or Device Tree name of
72 the board, the first kernel version affected, 72 the board, the first kernel version affected, the IGT version used for tests,
73 and an approximation of the failure rate. 73 and an approximation of the failure rate.
74 74
75 They should be provided under the following fo 75 They should be provided under the following format::
76 76
77 # Bug Report: $LORE_OR_PATCHWORK_URL 77 # Bug Report: $LORE_OR_PATCHWORK_URL
78 # Board Name: broken-board.dtb 78 # Board Name: broken-board.dtb
79 # Linux Version: 6.6-rc1 79 # Linux Version: 6.6-rc1
80 # IGT Version: 1.28-gd2af13d9f 80 # IGT Version: 1.28-gd2af13d9f
81 # Failure Rate: 100 81 # Failure Rate: 100
82 flaky-test 82 flaky-test
83 83
84 drivers/gpu/drm/ci/${DRIVER_NAME}-${HW_REVISIO 84 drivers/gpu/drm/ci/${DRIVER_NAME}-${HW_REVISION}-skips.txt
85 ---------------------------------------------- 85 -----------------------------------------------------------
86 86
87 Lists the tests that won't be run for a given 87 Lists the tests that won't be run for a given driver on a specific hardware
88 revision. These are usually tests that interfe 88 revision. These are usually tests that interfere with the running of the test
89 list due to hanging the machine, causing OOM, 89 list due to hanging the machine, causing OOM, taking too long, etc.
90 90
91 91
92 How to enable automated testing on your tree 92 How to enable automated testing on your tree
93 ============================================ 93 ============================================
94 94
95 1. Create a Linux tree in https://gitlab.freed 95 1. Create a Linux tree in https://gitlab.freedesktop.org/ if you don't have one
96 yet 96 yet
97 97
98 2. In your kernel repo's configuration (eg. 98 2. In your kernel repo's configuration (eg.
99 https://gitlab.freedesktop.org/janedoe/linux/- 99 https://gitlab.freedesktop.org/janedoe/linux/-/settings/ci_cd), change the
100 CI/CD configuration file from .gitlab-ci.yml t 100 CI/CD configuration file from .gitlab-ci.yml to
101 drivers/gpu/drm/ci/gitlab-ci.yml. 101 drivers/gpu/drm/ci/gitlab-ci.yml.
102 102
103 3. Request to be added to the drm/ci-ok group 103 3. Request to be added to the drm/ci-ok group so that your user has the
104 necessary privileges to run the CI on https:// 104 necessary privileges to run the CI on https://gitlab.freedesktop.org/drm/ci-ok
105 105
106 4. Next time you push to this repository, you 106 4. Next time you push to this repository, you will see a CI pipeline being
107 created (eg. https://gitlab.freedesktop.org/ja 107 created (eg. https://gitlab.freedesktop.org/janedoe/linux/-/pipelines)
108 108
109 5. The various jobs will be run and when the p 109 5. The various jobs will be run and when the pipeline is finished, all jobs
110 should be green unless a regression has been f 110 should be green unless a regression has been found.
111 111
112 112
113 How to update test expectations 113 How to update test expectations
114 =============================== 114 ===============================
115 115
116 If your changes to the code fix any tests, you 116 If your changes to the code fix any tests, you will have to remove one or more
117 lines from one or more of the files in 117 lines from one or more of the files in
118 drivers/gpu/drm/ci/${DRIVER_NAME}_*_fails.txt, 118 drivers/gpu/drm/ci/${DRIVER_NAME}_*_fails.txt, for each of the test platforms
119 affected by the change. 119 affected by the change.
120 120
121 121
122 How to expand coverage 122 How to expand coverage
123 ====================== 123 ======================
124 124
125 If your code changes make it possible to run m 125 If your code changes make it possible to run more tests (by solving reliability
126 issues, for example), you can remove tests fro 126 issues, for example), you can remove tests from the flakes and/or skips lists,
127 and then the expected results if there are any 127 and then the expected results if there are any known failures.
128 128
129 If there is a need for updating the version of 129 If there is a need for updating the version of IGT being used (maybe you have
130 added more tests to it), update the IGT_VERSIO 130 added more tests to it), update the IGT_VERSION variable at the top of the
131 gitlab-ci.yml file. 131 gitlab-ci.yml file.
132 132
133 133
134 How to test your changes to the scripts 134 How to test your changes to the scripts
135 ======================================= 135 =======================================
136 136
137 For testing changes to the scripts in the drm- 137 For testing changes to the scripts in the drm-ci repo, change the
138 DRM_CI_PROJECT_PATH and DRM_CI_COMMIT_SHA vari 138 DRM_CI_PROJECT_PATH and DRM_CI_COMMIT_SHA variables in
139 drivers/gpu/drm/ci/gitlab-ci.yml to match your 139 drivers/gpu/drm/ci/gitlab-ci.yml to match your fork of the project (eg.
140 janedoe/drm-ci). This fork needs to be in http 140 janedoe/drm-ci). This fork needs to be in https://gitlab.freedesktop.org/.
141 141
142 142
143 How to incorporate external fixes in your test 143 How to incorporate external fixes in your testing
144 ============================================== 144 =================================================
145 145
146 Often, regressions in other trees will prevent 146 Often, regressions in other trees will prevent testing changes local to the
147 tree under test. These fixes will be automatic 147 tree under test. These fixes will be automatically merged in during the build
148 jobs from a branch in the target tree that is 148 jobs from a branch in the target tree that is named as
149 ${TARGET_BRANCH}-external-fixes. 149 ${TARGET_BRANCH}-external-fixes.
150 150
151 If the pipeline is not in a merge request and 151 If the pipeline is not in a merge request and a branch with the same name
152 exists in the local tree, commits from that br 152 exists in the local tree, commits from that branch will be merged in as well.
153 153
154 154
155 How to deal with automated testing labs that m 155 How to deal with automated testing labs that may be down
156 ============================================== 156 ========================================================
157 157
158 If a hardware farm is down and thus causing pi 158 If a hardware farm is down and thus causing pipelines to fail that would
159 otherwise pass, one can disable all jobs that 159 otherwise pass, one can disable all jobs that would be submitted to that farm
160 by editing the file at 160 by editing the file at
161 https://gitlab.freedesktop.org/gfx-ci/lab-stat 161 https://gitlab.freedesktop.org/gfx-ci/lab-status/-/blob/main/lab-status.yml.
Linux® is a registered trademark of Linus Torvalds in the United States and other countries.
TOMOYO® is a registered trademark of NTT DATA CORPORATION.