1Table of Contents 2================= 3 4 A. How to use lldb for kernel debugging 5 B. Design of lldb kernel debugging platform. 6 C. Kernel debugging commands. 7 i. Using commands. 8 ii. Writing new commands. 9 D. Kernel type summaries. 10 i. Using summaries 11 ii. Writing new summary functions 12 E. FAQ and General Coding Guidelines 13 i. Frequently Asked Questions 14 ii. Formatted Output printing guidelines [MUST READ] 15 iii. Coding conventions. [MUST READ] 16 iv. Submitting changes in lldbmacros [MUST READ] 17 v. Common utility functions and paradigms 18 F. Development and Debugging on lldb kernel debugging platform. 19 i. Reading a exception backtrace 20 ii. Loading custom or local lldbmacros and operating_system plugin 21 iii. Adding debug related 'printf's 22 23A. How to use lldb for kernel debugging 24======================================== 25 26lldb can be used for kernel debugging the same way as gdb. The simplest way is to start lldb with kernel symbol file. The lldb environment by default does not allow loading automatic python modules. Please add the following setting in 27 28 File: ~/.lldbinit 29 settings set target.load-script-from-symbol-file true 30 31Now lldb will be ready to connect over kdp-remote '\<hostname:port>' or 'gdb-remote \<hostname:port>'. 32If you need to debug a core file, you can add '--core /path/to/corefile' to your lldb command (e.g. `xcrun --sdk macosx.internal lldb --core /path/to/corefile`) 33 34Following are detailed steps on how to debug a panic'ed / NMI'ed machine (For the curious souls). 35 36lldb debugging in detail:- 37 38 * start lldb with the right symbols file. If you do not know the version apriori, then enable dsymForUUID to load symbols dynamically. 39 bash$ dsymForUUID --enable 40 bash$ lldb /path/to/mach_kernel.symbols 41 Current executable set to '/Sources/Symbols/xnu/xnu-2253~2/mach_kernel' (x86_64). 42 (lldb) 43 44 * connect to remote device or load a core file 45 #for kdp 46 (lldb) process connect --plugin kdp-remote udp://17.123.45.67:41139 47 #for gdb (eg with astris) 48 (lldb) process connect --plugin gdb-remote gdb://17.123.45.67:8000 49 #for loading a core file 50 (lldb) file --core /path/to/core/file /path/to/kernel_symbol_file 51 52 * Once connected you can debug with basic lldb commands like print, bt, expr etc. The xnu debug macros will also be loaded automatically from the dSYM files. 53 In case if you are working with older kernel files you can load kernel specific commands by doing - 54 (lldb) command script import /path/to/xnu/tools/lldbmacros/xnu.py 55 (lldb) showbootargs 56 debug=0x14e ncpus=2 57 58 * You can do `kgmhelp` to get a list of commands available through xnu.py 59 60SPECIAL: The `xnu.py` script brings in kernel type summary functions. To enable these please do - 61 62 (lldb) showlldbtypesummaries 63 64These could be very handy in printing important information from structures easily. 65For ex. 66 67 (lldb) print (thread_t)0x80d6a620 68 (thread_t) $45 = 0x80d6a620 69 thread thread_id processor pri io_policy state wait_queue wait_event wmesg thread_name 70 0x80d6a620 0x317 0x902078c8 61 W 0x910cadd4 0x0 SystemSoundServer 71 72 73 74B. Design of lldb kernel debugging platform. 75============================================= 76 77The lldb debugger provides python scripting bridge for customizing commands and summaries in lldb. Following is the stack of platforms and how commands and summaries interact with it. 78 79 |------- xnu scripts ----------| 80 | |- lldb Command/Scripting-| | <-- provides scriptability for kernel data structures through summary/command invocation. 81 | | |--lldb core--| | | <-- interacts with remote kernel or corefile. 82 | |-------------------------| | 83 |------------------------------| 84 85The xnu script in xnu/tools/lldbmacros provides the following: 86 87 * Custom functions to do plumbing of lldb command invocation to python function call. (see doc strings for @lldb_command) 88 The command interface provides some common features (which can be invoked after passing '--' on cmd line) like - 89 90 i. send the output of command to file on disk 91 ii. search for a string in the output and selectively print the line containing it. 92 iii. -v options to increase verbosity levels in commands. 93 For example: (lldb)showalltasks -- -s kernel_task --o /tmp/kernel_task.output -v 94 will show task summary output with lines matching string 'kernel_task' into a file /tmp/kernel_task.output and with a verbosity level of (default +1) 95 96 * Customization for plugging in summary functions for lldb type summaries. (see doc strings for @lldb_summary) 97 It will automatically register given types with the functions within the kernel category. 98 99 * Ability to register test cases for macros (see doc strings for @xnudebug_test). 100 101The file layout is as follows 102 103 xnu/ 104 |-tools/ 105 |-lldbmacros/ 106 |-core/ # Core logic about kernel, lldb value abstraction, configs etc. **DO NOT TOUCH THIS DIR** 107 |-plugins/ # Holds plugins for kernel commands. 108 |-xnu.py # xnu debug framework along with kgmhelp, xnudebug commands. 109 |-xnudefines.py 110 |-utils.py 111 |-process.py # files containing commands/summaries code for each subsystem 112 |-memory.py 113 |-... 114 |-tests/ 115 |-lldb_tests/ # unit tests for macros, using lldb scripted process to simulate debugging a core file 116 |-standalone_tests/ # standalone tests for functionality that's seperate from lldb/macros (but used by them) 117 118 119The lldbmacros directory has a Makefile that follows the build process for xnu. This packages lldbmacros scripts into the dSYM of each kernel build. This helps in rev-locking the lldb commands with changes in kernel sources. 120 121 122C. Kernel debugging commands. 123============================== 124i. Using commands. 125------------------ 126Using xnu debug commands is very similar to kgmacros in gdb. You can use 'kgmhelp' to get a listing of available commands. 127If you need detailed help for a command please type 'help <command name>' and the documentation for the command will be displayed. 128For ex. 129 130 (lldb) help pmap_walk 131 Perform a page-table walk in <pmap> for <virtual_address>. 132 You can pass -- -v for verbose output. To increase the verbosity add more -v args after the '--'. 133 Syntax: pmap_walk <pmap> <virtual_address> 134 135The basic format for every command provided under kgmhelp is like follows 136 137 (lldb) command_name [cmd_args..] [-CMDOPTIONS] [-xnuoptions] 138 where: 139 command_name : name of command as registed using the @lldb_command decorator and described in 'kgmhelp' 140 cmd_args : shell like arguments that are passed as is to the registered python function. 141 If there is error in these arguments than the implementor may display according error message. 142 xnuoptions : common options for stream based operations on the output of command_name. 143 Allowed options are 144 -h : show help string of a command 145 -s <regexp> : print only the lines matching <regexp> 146 -o <file> : direct the output of command to <file>. Will not display anything on terminal 147 -v : increase the verbosity of the command. Each '-v' encountered will increase verbosity by 1. 148 -p <plugin> : pass the output of command to <plugin> for processing and followup with command requests by it. 149 CMDOPTIONS : These are command level options (always a CAPITAL letter option) that are defined by the macro developer. Please do 150 help <cmdname> to know how each option operates on that particular command. For an example of how to use CMDOPTIONS, take a look at vm_object_walk_pages in memory.py 151 152ii. Writing new commands. 153-------------------------- 154The python modules are designed in such a way that the command from lldb invokes a python function with the arguments passed at lldb prompt. 155 156It is recommended that you do a decoupled development for command interface and core utility function so that any function/code can be called as a simple util function and get the same output. i.e. 157 158 (lldb)showtask 0xabcdef000 is same as python >>> GetTaskSummary(0xabcdef000) or equivalent 159 160Following is a step by step guideline on how to add a new command ( e.g showtaskvme ). [extra tip: Always good idea to wrap your macro code within # Macro: , # EndMacro.] 161 162 1. register a command to a function. Use the lldb_command decorator to map a 'command_name' to a function. Optionally you can provide getopt compatible option string for customizing your command invocation. Note: Only CAPITAL letter options are allowed. lowercase options are reserved for the framework level features. 163 164 2. Immediately after the register define the function to handle the command invocation. The signature is always like Abc(cmd_args=None, cmd_options={}) 165 166 3. Add documentation for Abc(). This is very important for lldb to show help for each command. [ Follow the guidelines above with documentation ] 167 168 4. Use cmd_args array to get args passed on command. For example a command like `showtaskvme 0xabcdef00` will put have cmd_args=['0xabcdef00'] 169 - note that we use core.value class as an interface to underlying C structures. Refer [Section B] for more details. 170 - use kern.globals.\<variable_name> & kern.GetValueFromAddress for building values from addresses. 171 - remember that the ideal type of object to be passed around is core.value 172 - Anything you 'print' will be relayed to lldb terminal output. 173 174 5. If the user has passed any custom options they would be in cmd_options dict. the format is `{'-<optionflag>':'<value>'}`. The \<value> will be '' (empty string) for non-option flags. 175 176 6. If your function finds issue with the passed argument then you can `raise ArgumentError('error_message')` to notify the user. The framework will automatically catch this and show appropriate help using the function doc string. 177 178 7. Please use "##" for commenting your code. This is important because single "#" based strings may be mistakenly considered in `unifdef` program. 179 180 Time for some code example? Try reading the code for function ShowTaskVmeHelper in memory.py. 181 182SPECIAL Note: Very often you will find yourself making changes to a file for some command/summary and would like to test it out in lldb. 183 184To easily reload your changes in lldb please follow the below example. 185 186 * you fire up lldb and start using zprint. And soon you need to add functionality to zprint. 187 188 * you happily change a function code in memory.py file to zprint macro. 189 190 * now to reload that particular changes without killing your debug session do 191 (lldb) xnudebug reload memory 192 memory is reloaded from ./memory.py 193 (lldb) 194 195 * Alternatively, you can use lldb`s command for script loading as 196 (lldb) command script import /path/to/memory.py 197 You can re-run the same command every time you update the code in file. 198 199 It is very important that you do reload using xnudebug command as it does the plumbing of commands and types for your change in the module. Otherwise you could easily get confused 200 why your changes are not reflected in the command. 201 202 203D. Kernel type summaries. 204========================== 205i. Using summaries 206------------------ 207The lldb debugger provides ways for user to customize how a particular type of object be described when printed. These are very useful in displaying complex and large structures 208where only certain fields are important based on some flag or value in some field or variable. The way it works is every time lldb wants to print an object it checks 209for registered summaries. We can define python functions and hook it up with lldb as callbacks for type summaries. For example. 210 211 (lldb) print first_zone 212 (zone_t) $49 = 0xd007c000 213 ZONE TOT_SZ ALLOC_ELTS FREE_ELTS FREE_SZ ELT_SZ ALLOC(ELTS PGS SLK) FLAGS NAME 214 0x00000000d007c000 29808 182 25 3600 144 4096 28 1 64 X$ zones 215 (lldb) 216Just printing the value of first_zone as (zone_t) 0xd007c000 wouldnt have been much help. But with the registered summary for zone_t we can see all the interesting info easily. 217 218You do not need to do anything special to use summaries. Once they are registered with lldb they show info automatically when printing objects. However if you wish to 219see all the registered type summaries run the command `type summary list -w kernel` on lldb prompt. 220Also if you wish to quickly disable the summaries for a particular command use the `showraw` command. 221 222ii. Writing new summary functions 223--------------------------------- 224lldb provides really flexible interface for building summaries for complex objects and data. If you find that a struct or list can be 225diagnosed better if displayed differently, then feel free to add a type summary for that type. Following is an easy guide on how to do that. 226 227 1. Register a function as a callback for displaying information for a type. Use the `@lldb_type_summary()` decorator with an array of types you wish to register for callback 228 229 2. Provide a header for the summary using `@header()` decorator. This is a strong requirement for summaries. This gets displayed before the output 230 of `GetTypeSummary()` is displayed. [In case you do not wish to have header then still define it as "" (empty string) ] 231 232 3. Define the function with signature of `GetSomeTypeSummary(valobj)`. It is highly recommended that the naming be consistent to `Get.*?Summary(valobj)` 233 The valobj argument holds the core.value object for display. 234 235 4. Use the utility functions and memory read operations to pull out the required information. 236 [ use `kern.globals` & `kern.GetValueFromAddress` for building args to core functions. ] 237 [ remember that the ideal type of object to be passed around is core.value ] 238 239 5. return a string that would be printed by the caller. When lldb makes a call back it expects a str to be returned. So do not print 240 directly out to console. [ debug info or logs output is okay to be printed anywhere :) ] 241 242Time for some code example? Try reading the code for GetTaskSummary() in process.py. 243 244 245 246E. FAQs and Generel Coding Guidelines 247====================================== 248 249i. Frequently Asked Questions 250----------------------------- 251 252 Q. How do I avoid printing the summary and see the actual data in a structure? 253 254 A. There is a command called `showraw`. This will disable all kernel specific type summaries and execute any command you provide. For ex. 255 256 (lldb) print (thread_t) 0x80d6a620 257 (thread_t) $45 = 0x80d6a620 258 thread thread_id processor pri io_policy state wait_queue wait_event wmesg thread_name 259 0x80d6a620 0x317 0x902078c8 61 W 0x910cadd4 0x0 SystemSoundServer 260 (lldb) showraw print (thread_t) 0x80d6a620 261 (thread_t) $48 = 0x80d6a620 262 263 Q. I typed `showallvnodes` and nothing happens for a long time? OR How do I get output of long running command instantly on the terminal? 264 265 A. The lldb command interface tries to build result object from output of a python function. So in case of functions with very long output or runtime it may 266 seem that the lldb process is hung. But it is not. You can use "-i" option to get immediate output on terminal. 267 268 ex. (lldb) showallvnodes -- -i 269 Immediate Output 270 .... 271 272 Q. I made a change in a python file for a command or summary, but the output is not reflected in the lldb command? 273 274 A. The python framework does not allow for removing a loaded module and then reloading it. So sometimes if a command has a cached value from 275 old code that it will still call the old function and hence will not display new changes in file on disk. If you find yourself in such a situation 276 please see [Section C. -> SPECIAL Note]. If the change is to basic class or caching mechanism than it is advised to quit lldb and re-load all modules again. 277 278 Q. I am new to python. I get an error message that I do not understand. what should I do? 279 280 A. The syntax for python is different from conventional programming languages. If you get any message with SyntaxError or TypeError or ValueError then please review your code and look for common errors like 281 282 - wrong level of indentation? 283 - missed a ':' at the end of an if, elif, for, while statement? 284 - referencing a key in dictionary that doesn't exist? You might see KeyError in such cases. 285 - mistakenly used python reserved keyword as variable? (check http://docs.python.org/release/3.0.1/reference/lexical_analysis.html#id8) 286 - Trying to modify a string value? You can only create new strings but never modify existing ones. 287 - Trying to add a non string value to a string? This typically happens in print "time is " + gettime(). here gettime() returns int and not str. 288 - using a local variable with same name as global variable? 289 - assigning a value to global variable without declaring first? Its highly recommended to always declare global variable with 'global' keyword 290 If you still have difficulty you can look at the python documentation at http://docs.python.org 291 292 293 Q. I wish to pass value of variable/expression to xnu lldb macro that accepts only pointers. How can I achieve that? 294 295 A. Many lldb macros have syntax that accepts pointers (eg showtaskstacks etc). In order to have your expression be evaluated before passing to command use `back ticks`. For example: 296 297 (lldb) showtaskstacks `(task_t)tasks.next` 298 This way the expressing withing ` ` is evaluated by lldb and the value is passed to the command. 299 Note that if your argument pointer is bad or the memory is corrupted lldb macros will fail with a long backtrace that may not make sense. gdb used to fail silently but lldb does not. 300 Please see Section F(i) for more information on reading backtraces. 301 302 Q. I connected to a coredump file with lldb --core corefile and I got RuntimeError: Unable to find lldb thread for tid=XYZ. What should I do? 303 304 A. This is most likely the case that lldb ignored the operating system plugin in the dSYM and hence threads are not populated. Please put the line 'settings set target.load-script-from-symbol-file true' in your ~/.lldbinit file. If you do not have access you can alternatively do 305 306 bash# lldb 307 (lldb) settings set target.load-script-from-symbol-file true 308 (lldb) file --core corefile 309 310 311ii. Formatted output printing - zen and peace for life 312------------------------------------------------------ 313 314To avoid the horrors of printing a tabular data on console and then 2 weeks later again messing with it for a new field, it is recommended to follow these guidelines. 315 316 * any python string can be invoked to "".format() and hence makes it very easy to play with formats 317 318 * As a convention, I suggest that for printing pointer values in hex use "{0: <#020x}".format(some_int_value). This will print nice 0x prefixed strings with length padded to 20. 319 320 * If you need help with format options take a look at http://docs.python.org/library/string.html#format-string-syntax 321 322 * [ I'd first create a format string for data and then for the header just change the x's and d's to s and pass the header strings to format command. see GetTaskSummary()] 323 324 * If you need to print a string from a core.value object then use str() to get string representation of value. 325 326 327iii. Coding conventions 328----------------------- 329It is very very HIGHLY RECOMMENDED to follow these guidelines for writing any python code. 330 331 * Python is very sensitive to tabs and spaces for alignment. So please make sure you **INDENT YOUR CODE WITH SPACES** at all times. 332 333 * The standard tab width is 4 spaces. Each increasing indent adds 4 spaces beginning of the line. 334 335 * The format for documentation is - 336 """ A one line summary describing what this function / class does 337 Detailed explanation if necessary along with params and return values. 338 """ 339 340 * All Classes and functions should have a doc string describing what the function does 341 A consistent format is expected. For ex. 342 def SumOfNumbers(a, b, c, d): 343 """ Calculate sum of numbers. 344 params: 345 a - int, value to be added. can be 0 346 b - int/float, value to be added. 347 returns: 348 int/float - Sum of two values 349 raises: 350 TypeError - If any type is not identified in the params 351 """ 352 353 * A Class or Function should always start with CAPITAL letter and be CamelCase. If a function is for internal use only than it starts with '_'. 354 355 * Function params should always be lower_case and be word separated with '_' 356 357 * A local variable inside a function should be lower_case and separated with '_' 358 359 * A variable for internal use in object should start with '_'. 360 361 * if a class variable is supposed to hold non native type of object, it is good idea to comment what type it holds 362 363 * A class function with name matching `Get(.*?)Summary()` is always supposed to return a string which can be printed on stdout or any file. 364 365 * Functions beginning with "Get" (eg. GetVnodePath()) mean they return a value and will not print any output to stdout. 366 367 * Functions beginning with "Show" (eg. ShowZTrace()) mean they will print data on screen and may not return any value. 368 369 370iv. Submitting changes in lldbmacros 371------------------------------------ 372 373To contribute new commands or fixes to existing one, it is recommended that you follow the procedure below. 374 375 * Save the changes requried for new command or fix into lldbmacros directory. 376 377 * Make sure that the coding conventions are strictly followed. 378 379 * Run syntax checker on each of the modified files. It will find basic formatting errors in the changed files for you. 380 381 * If you are adding new file then please update the Makefile and xnu.py imports to ensure they get compiled during kernel build. 382 383 * Do a clean build of kernel from xnu top level directory. 384 385 * Verify that your changes are present in the dSYM directory of new build. 386 387 * Re-run all your test and verification steps with the lldbmacros from the newly packaged dSYM/Contents/Resources/Python/lldbmacros. 388 389v. Common utility functions and paradigms 390----------------------------------------- 391Please search and look around the code for common util functions and paradigm 392 393 * Take a peek at utils.py for common utility like sizeof_fmt() to humanize size strings in KB, MB etc. The convention is to have functions that do self contained actions and does not require intricate knowledge of kernel structures in utils.py 394 395 * If you need to get pagesize of the traget system, do not hard code any value. kern.globals.page_size is your friend. Similarly use config['verbosity'] for finding about configs. 396 397 * If you are developing a command for structure that is different based on development/release kernels please use "hasattr()" functionality to conditionalize referencing #ifdef'ed fields in structure. See example in def GetTaskSummary(task) in process.py 398 399 * `ArgumentStringToInt()` is recommended for argument parsing, as it supports binary/octal/decimal/hexadecimal literal 400 representations, as well as lldb expressions, which allows for convenient for usage e.g. `showmapvme foo_map_ptr` 401 402 403F. Development and Debugging on lldb kernel debugging platform. 404=============================================================== 405 406i. Reading a exception backtrace 407-------------------------------- 408In case of an error the lldbmacros may print out an exception backtrace and halt immediately. The important thing is to 409isolate possible causes of failure, and eventually filing a bug with kernel team. Following are some common ways where 410you may see an exception instead of your expected result. 411 412 * The lldbmacros cannot divine the type of memory by inspection. If a wrong pointer is passed from commandline then, 413 the command code will try to read and show some results. It may still be junk or plain erronous. Please make sure 414 your command arguments are correct. For example: a common mistake is to pass task address to showactstack. In such 415 a case lldb command may fail and show you a confusing backtrace. 416 417 * Kernel debugging is particularly tricky. Many parts of memory may not be readable. There could be failure in network, 418 debugging protocol or just plain bad memory. In such a case please try to see if you can examine memory for the object 419 you are trying to access. 420 421 * In case of memory corruption, the lldbmacros may have followed wrong pointer dereferencing. This might lead to failure 422 and a exception to be thrown. 423 424There are few more options that you can use when a macro is raising exceptions: 425 426 * Add --debug to your macro invocation to provide more detailed/verbose exception output. 427 * Add --radar to generate tar.gz archive when filling a new radar for kernel team. 428 * Add --pdb to attach pdb to exception stack for debugging. 429 430ii. Loading custom or local lldbmacros and operating_system plugin 431------------------------------------------------------------------ 432 433The lldbmacros are packaged right into the dSYM for the kernel executable. This makes debugging very easy since they can get loaded automatically when symbols are loaded. 434However, this setup makes it difficult for a lldbmacro developer to load custom/local macros. Following is the suggested solution for customizing your debugging setup: 435 436 * set up environment variable DEBUG_XNU_LLDBMACROS=1 on your shell. This will disable the automatic setup of lldbmacros and the operating_system.py from the symbols. 437 - bash$ export DEBUG_XNU_LLDBMACROS=1 438 439 * start lldb from the shell 440 - bash$ lldb 441 442 * [optional] If you are making changes in the operating_system plugin then you need to set the plugin path for lldb to find your custom operating_system plugin file. 443 - (lldb)settings set target.process.python-os-plugin-path /path/to/xnu/tools/lldbmacros/core/operating_system.py 444 If you do not wish to change anything in operating_system plugin then just leave the setting empty. The symbol loading module will set one up for you. 445 446 * Load the xnu debug macros from your custom location. 447 - (lldb)command script import /path/to/xnu/tools/lldbmacros/xnu.py 448 449 450iii. Adding debug related 'printf's 451----------------------------------- 452 453The xnu debug framework provides a utility function (debuglog) in utils.py. Please use this for any of your debugging needs. It will not print any output unless the user turns on debug logging on the command. Please check the documentaiton of debuglog for usage and options. 454 455 * To enable/disable logging 456 - (lldb) xnudebug debug 457 Enabled debug logging. 458 459 460