1 tdc - Linux Traffic Control (tc) unit testing 1 tdc - Linux Traffic Control (tc) unit testing suite
2 2
3 Author: Lucas Bates - lucasb@mojatatu.com 3 Author: Lucas Bates - lucasb@mojatatu.com
4 4
5 tdc is a Python script to load tc unit tests f 5 tdc is a Python script to load tc unit tests from a separate JSON file and
6 execute them inside a network namespace dedica 6 execute them inside a network namespace dedicated to the task.
7 7
8 8
9 REQUIREMENTS 9 REQUIREMENTS
10 ------------ 10 ------------
11 11
12 * Minimum Python version of 3.8. !! 12 * Minimum Python version of 3.4. Earlier 3.X versions may work but are not
>> 13 guaranteed.
13 14
14 * The kernel must have network namespace supp 15 * The kernel must have network namespace support if using nsPlugin
15 16
16 * The kernel must have veth support available 17 * The kernel must have veth support available, as a veth pair is created
17 prior to running the tests when using nsPlu 18 prior to running the tests when using nsPlugin.
18 19
19 * The kernel must have the appropriate infras 20 * The kernel must have the appropriate infrastructure enabled to run all tdc
20 unit tests. See the config file in this dir 21 unit tests. See the config file in this directory for minimum required
21 features. As new tests will be added, confi 22 features. As new tests will be added, config options list will be updated.
22 23
23 * All tc-related features being tested must b 24 * All tc-related features being tested must be built in or available as
24 modules. To check what is required in curr 25 modules. To check what is required in current setup run:
25 ./tdc.py -c 26 ./tdc.py -c
26 27
27 Note: 28 Note:
28 In the current release, tdc run will abort 29 In the current release, tdc run will abort due to a failure in setup or
29 teardown commands - which includes not bein 30 teardown commands - which includes not being able to run a test simply
30 because the kernel did not support a specif 31 because the kernel did not support a specific feature. (This will be
31 handled in a future version - the current w 32 handled in a future version - the current workaround is to run the tests
32 on specific test categories that your kerne 33 on specific test categories that your kernel supports)
33 34
34 35
35 BEFORE YOU RUN 36 BEFORE YOU RUN
36 -------------- 37 --------------
37 38
38 The path to the tc executable that will be mos 39 The path to the tc executable that will be most commonly tested can be defined
39 in the tdc_config.py file. Find the 'TC' entry 40 in the tdc_config.py file. Find the 'TC' entry in the NAMES dictionary and
40 define the path. 41 define the path.
41 42
42 If you need to test a different tc executable 43 If you need to test a different tc executable on the fly, you can do so by
43 using the -p option when running tdc: 44 using the -p option when running tdc:
44 ./tdc.py -p /path/to/tc 45 ./tdc.py -p /path/to/tc
45 46
46 47
47 RUNNING TDC 48 RUNNING TDC
48 ----------- 49 -----------
49 50
50 To use tdc, root privileges are required. Thi 51 To use tdc, root privileges are required. This is because the
51 commands being tested must be run as root. Th 52 commands being tested must be run as root. The code that enforces
52 execution by root uid has been moved into a pl 53 execution by root uid has been moved into a plugin (see PLUGIN
53 ARCHITECTURE, below). 54 ARCHITECTURE, below).
54 55
55 Tests that use a network device should have ns 56 Tests that use a network device should have nsPlugin.py listed as a
56 requirement for that test. nsPlugin executes a 57 requirement for that test. nsPlugin executes all commands within a
57 network namespace and creates a veth pair whic 58 network namespace and creates a veth pair which may be used in those test
58 cases. To disable execution within the namespa 59 cases. To disable execution within the namespace, pass the -N option
59 to tdc when starting a test run; the veth pair 60 to tdc when starting a test run; the veth pair will still be created
60 by the plugin. 61 by the plugin.
61 62
62 Running tdc without any arguments will run all 63 Running tdc without any arguments will run all tests. Refer to the section
63 on command line arguments for more information 64 on command line arguments for more information, or run:
64 ./tdc.py -h 65 ./tdc.py -h
65 66
66 tdc will list the test names as they are being 67 tdc will list the test names as they are being run, and print a summary in
67 TAP (Test Anything Protocol) format when they 68 TAP (Test Anything Protocol) format when they are done. If tests fail,
68 output captured from the failing test will be 69 output captured from the failing test will be printed immediately following
69 the failed test in the TAP output. 70 the failed test in the TAP output.
70 71
71 72
72 OVERVIEW OF TDC EXECUTION 73 OVERVIEW OF TDC EXECUTION
73 ------------------------- 74 -------------------------
74 75
75 One run of tests is considered a "test suite" 76 One run of tests is considered a "test suite" (this will be refined in the
76 future). A test suite has one or more test ca 77 future). A test suite has one or more test cases in it.
77 78
78 A test case has four stages: 79 A test case has four stages:
79 80
80 - setup 81 - setup
81 - execute 82 - execute
82 - verify 83 - verify
83 - teardown 84 - teardown
84 85
85 The setup and teardown stages can run zero or 86 The setup and teardown stages can run zero or more commands. The setup
86 stage does some setup if the test needs it. T 87 stage does some setup if the test needs it. The teardown stage undoes
87 the setup and returns the system to a "neutral 88 the setup and returns the system to a "neutral" state so any other test
88 can be run next. These two stages require any 89 can be run next. These two stages require any commands run to return
89 success, but do not otherwise verify the resul 90 success, but do not otherwise verify the results.
90 91
91 The execute and verify stages each run one com 92 The execute and verify stages each run one command. The execute stage
92 tests the return code against one or more acce 93 tests the return code against one or more acceptable values. The
93 verify stage checks the return code for succes 94 verify stage checks the return code for success, and also compares
94 the stdout with a regular expression. 95 the stdout with a regular expression.
95 96
96 Each of the commands in any stage will run in 97 Each of the commands in any stage will run in a shell instance.
97 98
98 Each test is an atomic unit. A test that for w <<
99 definitions is a bug. <<
100 <<
101 A test that runs inside a namespace (requires <<
102 with other tests. <<
103 <<
104 Tests that use netdevsim or don't run inside a <<
105 to each other. <<
106 <<
107 99
108 USER-DEFINED CONSTANTS 100 USER-DEFINED CONSTANTS
109 ---------------------- 101 ----------------------
110 102
111 The tdc_config.py file contains multiple value 103 The tdc_config.py file contains multiple values that can be altered to suit
112 your needs. Any value in the NAMES dictionary 104 your needs. Any value in the NAMES dictionary can be altered without affecting
113 the tests to be run. These values are used in 105 the tests to be run. These values are used in the tc commands that will be
114 executed as part of the test. More will be add 106 executed as part of the test. More will be added as test cases require.
115 107
116 Example: 108 Example:
117 $TC qdisc add dev $DEV1 ingress 109 $TC qdisc add dev $DEV1 ingress
118 110
119 The NAMES values are used to substitute into t 111 The NAMES values are used to substitute into the commands in the test cases.
120 112
121 113
122 COMMAND LINE ARGUMENTS 114 COMMAND LINE ARGUMENTS
123 ---------------------- 115 ----------------------
124 116
125 Run tdc.py -h to see the full list of availabl 117 Run tdc.py -h to see the full list of available arguments.
126 118
>> 119 usage: tdc.py [-h] [-p PATH] [-D DIR [DIR ...]] [-f FILE [FILE ...]]
>> 120 [-c [CATG [CATG ...]]] [-e ID [ID ...]] [-l] [-s] [-i] [-v] [-N]
>> 121 [-d DEVICE] [-P] [-n] [-V]
>> 122
>> 123 Linux TC unit tests
>> 124
>> 125 optional arguments:
>> 126 -h, --help show this help message and exit
>> 127 -p PATH, --path PATH The full path to the tc executable to use
>> 128 -v, --verbose Show the commands that are being run
>> 129 -N, --notap Suppress tap results for command under test
>> 130 -d DEVICE, --device DEVICE
>> 131 Execute test cases that use a physical device, where
>> 132 DEVICE is its name. (If not defined, tests that require
>> 133 a physical device will be skipped)
>> 134 -P, --pause Pause execution just before post-suite stage
>> 135
>> 136 selection:
>> 137 select which test cases: files plus directories; filtered by categories
>> 138 plus testids
>> 139
>> 140 -D DIR [DIR ...], --directory DIR [DIR ...]
>> 141 Collect tests from the specified directory(ies)
>> 142 (default [tc-tests])
>> 143 -f FILE [FILE ...], --file FILE [FILE ...]
>> 144 Run tests from the specified file(s)
>> 145 -c [CATG [CATG ...]], --category [CATG [CATG ...]]
>> 146 Run tests only from the specified category/ies, or if
>> 147 no category/ies is/are specified, list known
>> 148 categories.
>> 149 -e ID [ID ...], --execute ID [ID ...]
>> 150 Execute the specified test cases with specified IDs
>> 151
>> 152 action:
>> 153 select action to perform on selected test cases
>> 154
>> 155 -l, --list List all test cases, or those only within the
>> 156 specified category
>> 157 -s, --show Display the selected test cases
>> 158 -i, --id Generate ID numbers for new test cases
>> 159
>> 160 netns:
>> 161 options for nsPlugin (run commands in net namespace)
>> 162
>> 163 -N, --no-namespace
>> 164 Do not run commands in a network namespace.
>> 165
>> 166 valgrind:
>> 167 options for valgrindPlugin (run command under test under Valgrind)
>> 168
>> 169 -V, --valgrind Run commands under valgrind
>> 170
>> 171
127 PLUGIN ARCHITECTURE 172 PLUGIN ARCHITECTURE
128 ------------------- 173 -------------------
129 174
130 There is now a plugin architecture, and some o 175 There is now a plugin architecture, and some of the functionality that
131 was in the tdc.py script has been moved into t 176 was in the tdc.py script has been moved into the plugins.
132 177
133 The plugins are in the directory plugin-lib. 178 The plugins are in the directory plugin-lib. The are executed from
134 directory plugins. Put symbolic links from pl 179 directory plugins. Put symbolic links from plugins to plugin-lib,
135 and name them according to the order you want 180 and name them according to the order you want them to run. This is not
136 necessary if a test case being run requires a 181 necessary if a test case being run requires a specific plugin to work.
137 182
138 Example: 183 Example:
139 184
140 bjb@bee:~/work/tc-testing$ ls -l plugins 185 bjb@bee:~/work/tc-testing$ ls -l plugins
141 total 4 186 total 4
142 lrwxrwxrwx 1 bjb bjb 27 Oct 4 16:12 10-r 187 lrwxrwxrwx 1 bjb bjb 27 Oct 4 16:12 10-rootPlugin.py -> ../plugin-lib/rootPlugin.py
143 lrwxrwxrwx 1 bjb bjb 25 Oct 12 17:55 20-n 188 lrwxrwxrwx 1 bjb bjb 25 Oct 12 17:55 20-nsPlugin.py -> ../plugin-lib/nsPlugin.py
144 -rwxr-xr-x 1 bjb bjb 0 Sep 29 15:56 __in 189 -rwxr-xr-x 1 bjb bjb 0 Sep 29 15:56 __init__.py
145 190
146 The plugins are a subclass of TdcPlugin, defin 191 The plugins are a subclass of TdcPlugin, defined in TdcPlugin.py and
147 must be called "SubPlugin" so tdc can find the 192 must be called "SubPlugin" so tdc can find them. They are
148 distinguished from each other in the python pr 193 distinguished from each other in the python program by their module
149 name. 194 name.
150 195
151 This base class supplies "hooks" to run extra 196 This base class supplies "hooks" to run extra functions. These hooks are as follows:
152 197
153 pre- and post-suite 198 pre- and post-suite
154 pre- and post-case 199 pre- and post-case
155 pre- and post-execute stage 200 pre- and post-execute stage
156 adjust-command (runs in all stages and receive 201 adjust-command (runs in all stages and receives the stage name)
157 202
158 The pre-suite hook receives the number of test 203 The pre-suite hook receives the number of tests and an array of test ids.
159 This allows you to dump out the list of skippe 204 This allows you to dump out the list of skipped tests in the event of a
160 failure during setup or teardown stage. 205 failure during setup or teardown stage.
161 206
162 The pre-case hook receives the ordinal number 207 The pre-case hook receives the ordinal number and test id of the current test.
163 208
164 The adjust-command hook receives the stage id 209 The adjust-command hook receives the stage id (see list below) and the
165 full command to be executed. This allows for 210 full command to be executed. This allows for last-minute adjustment
166 of the command. 211 of the command.
167 212
168 The stages are identified by the following str 213 The stages are identified by the following strings:
169 214
170 - pre (pre-suite) 215 - pre (pre-suite)
171 - setup 216 - setup
172 - command 217 - command
173 - verify 218 - verify
174 - teardown 219 - teardown
175 - post (post-suite) 220 - post (post-suite)
176 221
177 222
178 To write a plugin, you need to inherit from Td 223 To write a plugin, you need to inherit from TdcPlugin in
179 TdcPlugin.py. To use the plugin, you have to 224 TdcPlugin.py. To use the plugin, you have to put the
180 implementation file in plugin-lib, and add a s 225 implementation file in plugin-lib, and add a symbolic link to it from
181 plugins. It will be detected at run time and 226 plugins. It will be detected at run time and invoked at the
182 appropriate times. There are a few examples i 227 appropriate times. There are a few examples in the plugin-lib
183 directory: 228 directory:
184 229
185 - rootPlugin.py: 230 - rootPlugin.py:
186 implements the enforcement of running as 231 implements the enforcement of running as root
187 - nsPlugin.py: 232 - nsPlugin.py:
188 sets up a network namespace and runs all 233 sets up a network namespace and runs all commands in that namespace,
189 while also setting up dummy devices to b 234 while also setting up dummy devices to be used in testing.
190 - valgrindPlugin.py 235 - valgrindPlugin.py
191 runs each command in the execute stage u 236 runs each command in the execute stage under valgrind,
192 and checks for leaks. 237 and checks for leaks.
193 This plugin will output an extra test fo 238 This plugin will output an extra test for each test in the test file,
194 one is the existing output as to whether 239 one is the existing output as to whether the test passed or failed,
195 and the other is a test whether the comm 240 and the other is a test whether the command leaked memory or not.
196 (This one is a preliminary version, it m 241 (This one is a preliminary version, it may not work quite right yet,
197 but the overall template is there and it 242 but the overall template is there and it should only need tweaks.)
>> 243 - buildebpfPlugin.py:
>> 244 builds all programs in $EBPFDIR.
198 245
199 246
200 ACKNOWLEDGEMENTS 247 ACKNOWLEDGEMENTS
201 ---------------- 248 ----------------
202 249
203 Thanks to: 250 Thanks to:
204 251
205 Jamal Hadi Salim, for providing valuable test 252 Jamal Hadi Salim, for providing valuable test cases
206 Keara Leibovitz, who wrote the CLI test driver 253 Keara Leibovitz, who wrote the CLI test driver that I used as a base for the
207 first version of the tc testing suite. This 254 first version of the tc testing suite. This work was presented at
208 Netdev 1.2 Tokyo in October 2016. 255 Netdev 1.2 Tokyo in October 2016.
209 Samir Hussain, for providing help while I dove 256 Samir Hussain, for providing help while I dove into Python for the first time
210 and being a second eye for this code. 257 and being a second eye for this code.
Linux® is a registered trademark of Linus Torvalds in the United States and other countries.
TOMOYO® is a registered trademark of NTT DATA CORPORATION.