1 .. _development_process:
2
3 How the development process works
4 =================================
5
6 Linux kernel development in the early 1990's w
7 with relatively small numbers of users and dev
8 user base in the millions and with some 2,000
9 course of one year, the kernel has since had t
10 processes to keep development happening smooth
11 how the process works is required in order to
12
13 The big picture
14 ---------------
15
16 The kernel developers use a loosely time-based
17 major kernel release happening every two or th
18 release history looks like this:
19
20 ====== =================
21 5.0 March 3, 2019
22 5.1 May 5, 2019
23 5.2 July 7, 2019
24 5.3 September 15, 2019
25 5.4 November 24, 2019
26 5.5 January 6, 2020
27 ====== =================
28
29 Every 5.x release is a major kernel release wi
30 API changes, and more. A typical release can
31 changesets with changes to several hundred tho
32 the leading edge of Linux kernel development;
33 rolling development model which is continually
34
35 A relatively straightforward discipline is fol
36 merging of patches for each release. At the b
37 cycle, the "merge window" is said to be open.
38 deemed to be sufficiently stable (and which is
39 community) is merged into the mainline kernel.
40 new development cycle (and all of the major ch
41 this time, at a rate approaching 1,000 changes
42 per day.
43
44 (As an aside, it is worth noting that the chan
45 merge window do not come out of thin air; they
46 and staged ahead of time. How that process wo
47 detail later on).
48
49 The merge window lasts for approximately two w
50 time, Linus Torvalds will declare that the win
51 first of the "rc" kernels. For the kernel whi
52 for example, the release which happens at the
53 be called 5.6-rc1. The -rc1 release is the si
54 merge new features has passed, and that the ti
55 kernel has begun.
56
57 Over the next six to ten weeks, only patches w
58 submitted to the mainline. On occasion a more
59 allowed, but such occasions are rare; develope
60 features outside of the merge window tend to g
61 As a general rule, if you miss the merge windo
62 best thing to do is to wait for the next devel
63 exception is made for drivers for previously-u
64 touch no in-tree code, they cannot cause regre
65 add at any time).
66
67 As fixes make their way into the mainline, the
68 time. Linus releases new -rc kernels about on
69 will get up to somewhere between -rc6 and -rc9
70 considered to be sufficiently stable and the f
71 At that point the whole process starts over ag
72
73 As an example, here is how the 5.4 development
74 2019):
75
76 ============== ======================
77 September 15 5.3 stable release
78 September 30 5.4-rc1, merge window
79 October 6 5.4-rc2
80 October 13 5.4-rc3
81 October 20 5.4-rc4
82 October 27 5.4-rc5
83 November 3 5.4-rc6
84 November 10 5.4-rc7
85 November 17 5.4-rc8
86 November 24 5.4 stable release
87 ============== ======================
88
89 How do the developers decide when to close the
90 the stable release? The most significant metr
91 regressions from previous releases. No bugs a
92 break systems which worked in the past are con
93 serious. For this reason, patches which cause
94 unfavorably and are quite likely to be reverte
95 period.
96
97 The developers' goal is to fix all known regre
98 release is made. In the real world, this kind
99 achieve; there are just too many variables in
100 There comes a point where delaying the final r
101 worse; the pile of changes waiting for the nex
102 larger, creating even more regressions the nex
103 kernels go out with a handful of known regress
104 of them are serious.
105
106 Once a stable release is made, its ongoing mai
107 "stable team," currently Greg Kroah-Hartman. T
108 occasional updates to the stable release using
109 To be considered for an update release, a patc
110 bug, and (2) already be merged into the mainli
111 kernel. Kernels will typically receive stable
112 than one development cycle past their initial
113 5.2 kernel's history looked like this (all dat
114
115 ============== ======================
116 July 7 5.2 stable release
117 July 14 5.2.1
118 July 21 5.2.2
119 July 26 5.2.3
120 July 28 5.2.4
121 July 31 5.2.5
122 ... ...
123 October 11 5.2.21
124 ============== ======================
125
126 5.2.21 was the final stable update of the 5.2
127
128 Some kernels are designated "long term" kernel
129 for a longer period. Please refer to the foll
130 long term kernel versions and their maintainer
131
132 https://www.kernel.org/category/releas
133
134 The selection of a kernel for long-term suppor
135 maintainer having the need and the time to mai
136 are no known plans for long-term support for a
137 release.
138
139
140 The lifecycle of a patch
141 ------------------------
142
143 Patches do not go directly from the developer'
144 kernel. There is, instead, a somewhat involve
145 process designed to ensure that each patch is
146 each patch implements a change which is desira
147 This process can happen quickly for minor fixe
148 and controversial changes, go on for years. M
149 comes from a lack of understanding of this pro
150 circumvent it.
151
152 In the hopes of reducing that frustration, thi
153 a patch gets into the kernel. What follows be
154 describes the process in a somewhat idealized
155 treatment will come in later sections.
156
157 The stages that a patch goes through are, gene
158
159 - Design. This is where the real requirement
160 those requirements will be met - are laid o
161 done without involving the community, but i
162 in the open if at all possible; it can save
163 things later.
164
165 - Early review. Patches are posted to the re
166 developers on that list reply with any comm
167 process should turn up any major problems w
168 well.
169
170 - Wider review. When the patch is getting cl
171 inclusion, it should be accepted by a relev
172 though this acceptance is not a guarantee t
173 all the way to the mainline. The patch wil
174 subsystem tree and into the -next trees (de
175 process works, this step leads to more exte
176 the discovery of any problems resulting fro
177 patch with work being done by others.
178
179 - Please note that most maintainers also have
180 your patch may not be their highest priorit
181 getting feedback about changes that are nee
182 make those changes or justify why they shou
183 patch has no review complaints but is not b
184 appropriate subsystem or driver maintainer,
185 in updating the patch to the current kernel
186 and keep sending it for review and merging.
187
188 - Merging into the mainline. Eventually, a s
189 merged into the mainline repository managed
190 comments and/or problems may surface at thi
191 the developer be responsive to these and fi
192
193 - Stable release. The number of users potent
194 is now large, so, once again, new problems
195
196 - Long-term maintenance. While it is certain
197 to forget about code after merging it, that
198 leave a poor impression in the development
199 eliminates some of the maintenance burden,
200 problems caused by API changes. But the or
201 continue to take responsibility for the cod
202 in the longer term.
203
204 One of the largest mistakes made by kernel dev
205 is to try to cut the process down to a single
206 step. This approach invariably leads to frust
207 involved.
208
209 How patches get into the Kernel
210 -------------------------------
211
212 There is exactly one person who can merge patc
213 repository: Linus Torvalds. But, for example,
214 which went into the 2.6.38 kernel, only 112 (a
215 chosen by Linus himself. The kernel project ha
216 where no single developer could possibly inspe
217 unassisted. The way the kernel developers have
218 through the use of a lieutenant system built a
219
220 The kernel code base is logically broken down
221 networking, specific architecture support, mem
222 devices, etc. Most subsystems have a designat
223 who has overall responsibility for the code wi
224 subsystem maintainers are the gatekeepers (in
225 of the kernel they manage; they are the ones w
226 patch for inclusion into the mainline kernel.
227
228 Subsystem maintainers each manage their own ve
229 tree, usually (but certainly not always) using
230 tool. Tools like git (and related tools like
231 maintainers to track a list of patches, includ
232 and other metadata. At any given time, the ma
233 patches in his or her repository are not found
234
235 When the merge window opens, top-level maintai
236 the patches they have selected for merging fro
237 Linus agrees, the stream of patches will flow
238 becoming part of the mainline kernel. The amo
239 pays to specific patches received in a pull op
240 that, sometimes, he looks quite closely. But,
241 trusts the subsystem maintainers to not send b
242
243 Subsystem maintainers, in turn, can pull patch
244 For example, the networking tree is built from
245 first in trees dedicated to network device dri
246 etc. This chain of repositories can be arbitr
247 exceeds two or three links. Since each mainta
248 those managing lower-level trees, this process
249 trust."
250
251 Clearly, in a system like this, getting patche
252 finding the right maintainer. Sending patches
253 normally the right way to go.
254
255
256 Next trees
257 ----------
258
259 The chain of subsystem trees guides the flow o
260 but it also raises an interesting question: wh
261 at all of the patches which are being prepared
262 Developers will be interested in what other ch
263 whether there are any conflicts to worry about
264 core kernel function prototype, for example, w
265 patches which use the older form of that funct
266 want access to the changes in their integrated
267 changes land in the mainline kernel. One coul
268 the interesting subsystem trees, but that woul
269 job.
270
271 The answer comes in the form of -next trees, w
272 collected for testing and review. The older o
273 Andrew Morton, is called "-mm" (for memory man
274 started). The -mm tree integrates patches fro
275 trees; it also has some patches aimed at helpi
276
277 Beyond that, -mm contains a significant collec
278 been selected by Andrew directly. These patch
279 mailing list, or they may apply to a part of t
280 no designated subsystem tree. As a result, -m
281 subsystem tree of last resort; if there is no
282 patch into the mainline, it is likely to end u
283 patches which accumulate in -mm will eventuall
284 an appropriate subsystem tree or be sent direc
285 development cycle, approximately 5-10% of the
286 mainline get there via -mm.
287
288 The current -mm patch is available in the "mmo
289 directory at:
290
291 https://www.ozlabs.org/~akpm/mmotm/
292
293 Use of the MMOTM tree is likely to be a frustr
294 there is a definite chance that it will not ev
295
296 The primary tree for next-cycle patch merging
297 Stephen Rothwell. The linux-next tree is, by
298 the mainline is expected to look like after th
299 Linux-next trees are announced on the linux-ke
300 lists when they are assembled; they can be dow
301
302 https://www.kernel.org/pub/linux/kerne
303
304 Linux-next has become an integral part of the
305 all patches merged during a given merge window
306 their way into linux-next some time before the
307
308
309 Staging trees
310 -------------
311
312 The kernel source tree contains the drivers/st
313 many sub-directories for drivers or filesystem
314 being added to the kernel tree live. They rem
315 they still need more work; once complete, they
316 kernel proper. This is a way to keep track of
317 up to Linux kernel coding or quality standards
318 them and track development.
319
320 Greg Kroah-Hartman currently maintains the sta
321 still need work are sent to him, with each dri
322 subdirectory in drivers/staging/. Along with
323 TODO file should be present in the directory a
324 the pending work that the driver needs for acc
325 proper, as well as a list of people that shoul
326 the driver. Current rules require that driver
327 must, at a minimum, compile properly.
328
329 Staging can be a relatively easy way to get ne
330 where, with luck, they will come to the attent
331 improve quickly. Entry into staging is not th
332 code in staging which is not seeing regular pr
333 removed. Distributors also tend to be relativ
334 staging drivers. So staging is, at best, a st
335 a proper mainline driver.
336
337
338 Tools
339 -----
340
341 As can be seen from the above text, the kernel
342 heavily on the ability to herd collections of
343 directions. The whole thing would not work an
344 does without suitably powerful tools. Tutoria
345 are well beyond the scope of this document, bu
346 pointers.
347
348 By far the dominant source code management sys
349 community is git. Git is one of a number of d
350 systems being developed in the free software c
351 for kernel development, in that it performs qu
352 large repositories and large numbers of patche
353 for being difficult to learn and use, though i
354 time. Some sort of familiarity with git is al
355 developers; even if they do not use it for the
356 to keep up with what other developers (and the
357
358 Git is now packaged by almost all Linux distri
359 page at:
360
361 https://git-scm.com/
362
363 That page has pointers to documentation and tu
364
365 Among the kernel developers who do not use git
366 almost certainly Mercurial:
367
368 https://www.selenic.com/mercurial/
369
370 Mercurial shares many features with git, but i
371 many find easier to use.
372
373 The other tool worth knowing about is Quilt:
374
375 https://savannah.nongnu.org/projects/q
376
377 Quilt is a patch management system, rather tha
378 system. It does not track history over time;
379 toward tracking a specific set of changes agai
380 Some major subsystem maintainers use quilt to
381 upstream. For the management of certain kinds
382 quilt is the best tool for the job.
383
384
385 Mailing lists
386 -------------
387
388 A great deal of Linux kernel development work
389 lists. It is hard to be a fully-functioning m
390 without joining at least one list somewhere.
391 represent a potential hazard to developers, wh
392 load of electronic mail, running afoul of the
393 lists, or both.
394
395 Most kernel mailing lists are hosted at kernel
396 be found at:
397
398 https://subspace.kernel.org
399
400 There are lists hosted elsewhere; please check
401 the list relevant for any particular subsystem
402
403 The core mailing list for kernel development i
404 This list is an intimidating place to be; volu
405 day, the amount of noise is high, the conversa
406 technical, and participants are not always con
407 degree of politeness. But there is no other p
408 development community comes together as a whol
409 list will miss important information.
410
411 There are a few hints which can help with linu
412
413 - Have the list delivered to a separate folder
414 mailbox. One must be able to ignore the str
415 time.
416
417 - Do not try to follow every conversation - no
418 important to filter on both the topic of int
419 long-running conversations can drift away fr
420 without changing the email subject line) and
421 participating.
422
423 - Do not feed the trolls. If somebody is tryi
424 response, ignore them.
425
426 - When responding to linux-kernel email (or th
427 the Cc: header for all involved. In the abs
428 as an explicit request), you should never re
429 sure that the person you are responding to i
430 convention also makes it unnecessary to expl
431 replies to your postings.
432
433 - Search the list archives (and the net as a w
434 questions. Some developers can get impatien
435 have not done their homework.
436
437 - Use interleaved ("inline") replies, which ma
438 read. (i.e. avoid top-posting -- the practic
439 the quoted text you are responding to.) For
440 :ref:`Documentation/process/submitting-patch
441
442 - Ask on the correct mailing list. Linux-kern
443 point, but it is not the best place to find
444 subsystems.
445
446 The last point - finding the correct mailing l
447 beginning developers to go wrong. Somebody wh
448 question on linux-kernel will almost certainly
449 to ask on the netdev list instead, as that is
450 networking developers. Other lists exist for
451 filesystem, etc. subsystems. The best place t
452 in the MAINTAINERS file packaged with the kern
453
454
455 Getting started with Kernel development
456 ---------------------------------------
457
458 Questions about how to get started with the ke
459 common - from both individuals and companies.
460 which make the beginning of the relationship h
461
462 Companies often look to hire well-known develo
463 group started. This can, in fact, be an effec
464 tends to be expensive and does not do much to
465 kernel developers. It is possible to bring in
466 on Linux kernel development, given the investm
467 this time can endow an employer with a group o
468 the kernel and the company both, and who can h
469 Over the medium term, this is often the more p
470
471 Individual developers are often, understandabl
472 start. Beginning with a large project can be
473 to test the waters with something smaller firs
474 some developers jump into the creation of patc
475 minor coding style issues. Unfortunately, suc
476 noise which is distracting for the development
477 increasingly, they are looked down upon. New
478 introduce themselves to the community will not
479 they wish for by these means.
480
481 Andrew Morton gives this advice for aspiring k
482
483 ::
484
485 The #1 project for all kernel beginner
486 that the kernel runs perfectly at all
487 you can lay your hands on". Usually t
488 with others on getting things fixed up
489 persistence!) but that's fine - it's a
490
491 (https://lwn.net/Articles/283982/).
492
493 In the absence of obvious problems to fix, dev
494 at the current lists of regressions and open b
495 never any shortage of issues in need of fixing
496 developers will gain experience with the proce
497 building respect with the rest of the developm
Linux® is a registered trademark of Linus Torvalds in the United States and other countries.
TOMOYO® is a registered trademark of NTT DATA CORPORATION.