1tdc - Linux Traffic Control (tc) unit testing suite 2 3Author: Lucas Bates - lucasb@mojatatu.com 4 5tdc is a Python script to load tc unit tests from a separate JSON file and 6execute them inside a network namespace dedicated to the task. 7 8 9REQUIREMENTS 10------------ 11 12* Minimum Python version of 3.4. Earlier 3.X versions may work but are not 13 guaranteed. 14 15* The kernel must have network namespace support 16 17* The kernel must have veth support available, as a veth pair is created 18 prior to running the tests. 19 20* All tc-related features being tested must be built in or available as 21 modules. To check what is required in current setup run: 22 ./tdc.py -c 23 24 Note: 25 In the current release, tdc run will abort due to a failure in setup or 26 teardown commands - which includes not being able to run a test simply 27 because the kernel did not support a specific feature. (This will be 28 handled in a future version - the current workaround is to run the tests 29 on specific test categories that your kernel supports) 30 31 32BEFORE YOU RUN 33-------------- 34 35The path to the tc executable that will be most commonly tested can be defined 36in the tdc_config.py file. Find the 'TC' entry in the NAMES dictionary and 37define the path. 38 39If you need to test a different tc executable on the fly, you can do so by 40using the -p option when running tdc: 41 ./tdc.py -p /path/to/tc 42 43 44RUNNING TDC 45----------- 46 47To use tdc, root privileges are required. This is because the 48commands being tested must be run as root. The code that enforces 49execution by root uid has been moved into a plugin (see PLUGIN 50ARCHITECTURE, below). 51 52If nsPlugin is linked, all tests are executed inside a network 53namespace to prevent conflicts within the host. 54 55Running tdc without any arguments will run all tests. Refer to the section 56on command line arguments for more information, or run: 57 ./tdc.py -h 58 59tdc will list the test names as they are being run, and print a summary in 60TAP (Test Anything Protocol) format when they are done. If tests fail, 61output captured from the failing test will be printed immediately following 62the failed test in the TAP output. 63 64 65OVERVIEW OF TDC EXECUTION 66------------------------- 67 68One run of tests is considered a "test suite" (this will be refined in the 69future). A test suite has one or more test cases in it. 70 71A test case has four stages: 72 73 - setup 74 - execute 75 - verify 76 - teardown 77 78The setup and teardown stages can run zero or more commands. The setup 79stage does some setup if the test needs it. The teardown stage undoes 80the setup and returns the system to a "neutral" state so any other test 81can be run next. These two stages require any commands run to return 82success, but do not otherwise verify the results. 83 84The execute and verify stages each run one command. The execute stage 85tests the return code against one or more acceptable values. The 86verify stage checks the return code for success, and also compares 87the stdout with a regular expression. 88 89Each of the commands in any stage will run in a shell instance. 90 91 92USER-DEFINED CONSTANTS 93---------------------- 94 95The tdc_config.py file contains multiple values that can be altered to suit 96your needs. Any value in the NAMES dictionary can be altered without affecting 97the tests to be run. These values are used in the tc commands that will be 98executed as part of the test. More will be added as test cases require. 99 100Example: 101 $TC qdisc add dev $DEV1 ingress 102 103The NAMES values are used to substitute into the commands in the test cases. 104 105 106COMMAND LINE ARGUMENTS 107---------------------- 108 109Run tdc.py -h to see the full list of available arguments. 110 111usage: tdc.py [-h] [-p PATH] [-D DIR [DIR ...]] [-f FILE [FILE ...]] 112 [-c [CATG [CATG ...]]] [-e ID [ID ...]] [-l] [-s] [-i] [-v] 113 [-d DEVICE] [-n NS] [-V] 114 115Linux TC unit tests 116 117optional arguments: 118 -h, --help show this help message and exit 119 -p PATH, --path PATH The full path to the tc executable to use 120 -v, --verbose Show the commands that are being run 121 -d DEVICE, --device DEVICE 122 Execute the test case in flower category 123 124selection: 125 select which test cases: files plus directories; filtered by categories 126 plus testids 127 128 -D DIR [DIR ...], --directory DIR [DIR ...] 129 Collect tests from the specified directory(ies) 130 (default [tc-tests]) 131 -f FILE [FILE ...], --file FILE [FILE ...] 132 Run tests from the specified file(s) 133 -c [CATG [CATG ...]], --category [CATG [CATG ...]] 134 Run tests only from the specified category/ies, or if 135 no category/ies is/are specified, list known 136 categories. 137 -e ID [ID ...], --execute ID [ID ...] 138 Execute the specified test cases with specified IDs 139 140action: 141 select action to perform on selected test cases 142 143 -l, --list List all test cases, or those only within the 144 specified category 145 -s, --show Display the selected test cases 146 -i, --id Generate ID numbers for new test cases 147 148netns: 149 options for nsPlugin(run commands in net namespace) 150 151 -n NS, --namespace NS 152 Run commands in namespace NS 153 154valgrind: 155 options for valgrindPlugin (run command under test under Valgrind) 156 157 -V, --valgrind Run commands under valgrind 158 159 160PLUGIN ARCHITECTURE 161------------------- 162 163There is now a plugin architecture, and some of the functionality that 164was in the tdc.py script has been moved into the plugins. 165 166The plugins are in the directory plugin-lib. The are executed from 167directory plugins. Put symbolic links from plugins to plugin-lib, 168and name them according to the order you want them to run. 169 170Example: 171 172bjb@bee:~/work/tc-testing$ ls -l plugins 173total 4 174lrwxrwxrwx 1 bjb bjb 27 Oct 4 16:12 10-rootPlugin.py -> ../plugin-lib/rootPlugin.py 175lrwxrwxrwx 1 bjb bjb 25 Oct 12 17:55 20-nsPlugin.py -> ../plugin-lib/nsPlugin.py 176-rwxr-xr-x 1 bjb bjb 0 Sep 29 15:56 __init__.py 177 178The plugins are a subclass of TdcPlugin, defined in TdcPlugin.py and 179must be called "SubPlugin" so tdc can find them. They are 180distinguished from each other in the python program by their module 181name. 182 183This base class supplies "hooks" to run extra functions. These hooks are as follows: 184 185pre- and post-suite 186pre- and post-case 187pre- and post-execute stage 188adjust-command (runs in all stages and receives the stage name) 189 190The pre-suite hook receives the number of tests and an array of test ids. 191This allows you to dump out the list of skipped tests in the event of a 192failure during setup or teardown stage. 193 194The pre-case hook receives the ordinal number and test id of the current test. 195 196The adjust-command hook receives the stage id (see list below) and the 197full command to be executed. This allows for last-minute adjustment 198of the command. 199 200The stages are identified by the following strings: 201 202 - pre (pre-suite) 203 - setup 204 - command 205 - verify 206 - teardown 207 - post (post-suite) 208 209 210To write a plugin, you need to inherit from TdcPlugin in 211TdcPlugin.py. To use the plugin, you have to put the 212implementation file in plugin-lib, and add a symbolic link to it from 213plugins. It will be detected at run time and invoked at the 214appropriate times. There are a few examples in the plugin-lib 215directory: 216 217 - rootPlugin.py: 218 implements the enforcement of running as root 219 - nsPlugin.py: 220 sets up a network namespace and runs all commands in that namespace 221 - valgrindPlugin.py 222 runs each command in the execute stage under valgrind, 223 and checks for leaks. 224 This plugin will output an extra test for each test in the test file, 225 one is the existing output as to whether the test passed or failed, 226 and the other is a test whether the command leaked memory or not. 227 (This one is a preliminary version, it may not work quite right yet, 228 but the overall template is there and it should only need tweaks.) 229 230 231ACKNOWLEDGEMENTS 232---------------- 233 234Thanks to: 235 236Jamal Hadi Salim, for providing valuable test cases 237Keara Leibovitz, who wrote the CLI test driver that I used as a base for the 238 first version of the tc testing suite. This work was presented at 239 Netdev 1.2 Tokyo in October 2016. 240Samir Hussain, for providing help while I dove into Python for the first time 241 and being a second eye for this code. 242