1Table of Contents 2================= 3 4 A. How to use lldb for kernel debugging 5 B. Design of lldb kernel debugging platform. 6 C. Kernel debugging commands. 7 i. Using commands. 8 ii. Writing new commands. 9 D. Kernel type summaries. 10 i. Using summaries 11 ii. Writing new summary functions 12 E. FAQ and General Coding Guidelines 13 i. Frequently Asked Questions 14 ii. Formatted Output printing guidelines [MUST READ] 15 iii. Coding conventions. [MUST READ] 16 iv. Submitting changes in lldbmacros [MUST READ] 17 v. Common utility functions and paradigms 18 F. Development and Debugging on lldb kernel debugging platform. 19 i. Reading a exception backtrace 20 ii. Loading custom or local lldbmacros and operating_system plugin 21 iii. Adding debug related 'printf's 22 23A. How to use lldb for kernel debugging 24======================================== 25 26lldb can be used for kernel debugging the same way as gdb. The simplest way is to start lldb with kernel symbol file. The lldb environment by default does not allow loading automatic python modules. Please add the following setting in 27 28 File: ~/.lldbinit 29 settings set target.load-script-from-symbol-file true 30 31Now lldb will be ready to connect over kdp-remote '\<hostname:port>' or 'gdb-remote \<hostname:port>'. In case using a core file please do 'file --core /path/to/corefile' 32 33Following are detailed steps on how to debug a panic'ed / NMI'ed machine (For the curious souls). 34 35lldb debugging in detail:- 36 37 * start lldb with the right symbols file. If you do not know the version apriori, then enable dsymForUUID to load symbols dynamically. 38 bash$ dsymForUUID --enable 39 bash$ lldb /path/to/mach_kernel.symbols 40 Current executable set to '/Sources/Symbols/xnu/xnu-2253~2/mach_kernel' (x86_64). 41 (lldb) 42 43 * connect to remote device or load a core file 44 #for kdp 45 (lldb) process connect --plugin kdp-remote udp://17.123.45.67:41139 46 #for gdb (eg with astris) 47 (lldb) process connect --plugin gdb-remote gdb://17.123.45.67:8000 48 #for loading a core file 49 (lldb) file --core /path/to/core/file /path/to/kernel_symbol_file 50 51 * Once connected you can debug with basic lldb commands like print, bt, expr etc. The xnu debug macros will also be loaded automatically from the dSYM files. 52 In case if you are working with older kernel files you can load kernel specific commands by doing - 53 (lldb) command script import /path/to/xnu/tools/lldbmacros/xnu.py 54 (lldb) showbootargs 55 debug=0x14e ncpus=2 56 57 * You can do `kgmhelp` to get a list of commands available through xnu.py 58 59SPECIAL: The `xnu.py` script brings in kernel type summary functions. To enable these please do - 60 61 (lldb) showlldbtypesummaries 62 63These could be very handy in printing important information from structures easily. 64For ex. 65 66 (lldb) print (thread_t)0x80d6a620 67 (thread_t) $45 = 0x80d6a620 68 thread thread_id processor pri io_policy state wait_queue wait_event wmesg thread_name 69 0x80d6a620 0x317 0x902078c8 61 W 0x910cadd4 0x0 SystemSoundServer 70 71 72 73B. Design of lldb kernel debugging platform. 74============================================= 75 76The lldb debugger provides python scripting bridge for customizing commands and summaries in lldb. Following is the stack of platforms and how commands and summaries interact with it. 77 78 |------- xnu scripts ----------| 79 | |- lldb Command/Scripting-| | <-- provides scriptability for kernel data structures through summary/command invocation. 80 | | |--lldb core--| | | <-- interacts with remote kernel or corefile. 81 | |-------------------------| | 82 |------------------------------| 83 84The xnu script in xnu/tools/lldbmacros provides the following: 85 86 * Custom functions to do plumbing of lldb command invocation to python function call. (see doc strings for @lldb_command) 87 The command interface provides some common features (which can be invoked after passing '--' on cmd line) like - 88 89 i. send the output of command to file on disk 90 ii. search for a string in the output and selectively print the line containing it. 91 iii. -v options to increase verbosity levels in commands. 92 For example: (lldb)showalltasks -- -s kernel_task --o /tmp/kernel_task.output -v 93 will show task summary output with lines matching string 'kernel_task' into a file /tmp/kernel_task.output and with a verbosity level of (default +1) 94 95 * Customization for plugging in summary functions for lldb type summaries. (see doc strings for @lldb_summary) 96 It will automatically register given types with the functions within the kernel category. 97 98 * Ability to register test cases for macros (see doc strings for @xnudebug_test). 99 100The file layout is like following 101 102 xnu/ 103 |-tools/ 104 |-lldbmacros/ 105 |-core/ # Core logic about kernel, lldb value abstraction, configs etc. **DO NOT TOUCH THIS DIR** 106 |-plugins/ # Holds plugins for kernel commands. 107 |-xnu.py # xnu debug framework along with kgmhelp, xnudebug commands. 108 |-xnudefines.py 109 |-utils.py 110 |-process.py # files containing commands/summaries code for each subsystem 111 |-... 112 113 114The lldbmacros directory has a Makefile that follows the build process for xnu. This packages lldbmacros scripts into the dSYM of each kernel build. This helps in rev-locking the lldb commands with changes in kernel sources. 115 116 117C. Kernel debugging commands. 118============================== 119i. Using commands. 120------------------ 121Using xnu debug commands is very similar to kgmacros in gdb. You can use 'kgmhelp' to get a listing of available commands. 122If you need detailed help for a command please type 'help <command name>' and the documentation for the command will be displayed. 123For ex. 124 125 (lldb) help pmap_walk 126 Perform a page-table walk in <pmap> for <virtual_address>. 127 You can pass -- -v for verbose output. To increase the verbosity add more -v args after the '--'. 128 Syntax: pmap_walk <pmap> <virtual_address> 129 130The basic format for every command provided under kgmhelp is like follows 131 132 (lldb) command_name [cmd_args..] [-CMDOPTIONS] [-xnuoptions] 133 where: 134 command_name : name of command as registed using the @lldb_command decorator and described in 'kgmhelp' 135 cmd_args : shell like arguments that are passed as is to the registered python function. 136 If there is error in these arguments than the implementor may display according error message. 137 xnuoptions : common options for stream based operations on the output of command_name. 138 Allowed options are 139 -h : show help string of a command 140 -s <regexp> : print only the lines matching <regexp> 141 -o <file> : direct the output of command to <file>. Will not display anything on terminal 142 -v : increase the verbosity of the command. Each '-v' encountered will increase verbosity by 1. 143 -p <plugin> : pass the output of command to <plugin> for processing and followup with command requests by it. 144 CMDOPTIONS : These are command level options (always a CAPITAL letter option) that are defined by the macro developer. Please do 145 help <cmdname> to know how each option operates on that particular command. For an example of how to use CMDOPTIONS, take a look at vm_object_walk_pages in memory.py 146 147ii. Writing new commands. 148-------------------------- 149The python modules are designed in such a way that the command from lldb invokes a python function with the arguments passed at lldb prompt. 150 151It is recommended that you do a decoupled development for command interface and core utility function so that any function/code can be called as a simple util function and get the same output. i.e. 152 153 (lldb)showtask 0xabcdef000 is same as python >>> GetTaskSummary(0xabcdef000) or equivalent 154 155Following is a step by step guideline on how to add a new command ( e.g showtaskvme ). [extra tip: Always good idea to wrap your macro code within # Macro: , # EndMacro.] 156 157 1. register a command to a function. Use the lldb_command decorator to map a 'command_name' to a function. Optionally you can provide getopt compatible option string for customizing your command invocation. Note: Only CAPITAL letter options are allowed. lowercase options are reserved for the framework level features. 158 159 2. Immediately after the register define the function to handle the command invocation. The signature is always like Abc(cmd_args=None, cmd_options={}) 160 161 3. Add documentation for Abc(). This is very important for lldb to show help for each command. [ Follow the guidelines above with documentation ] 162 163 4. Use cmd_args array to get args passed on command. For example a command like `showtaskvme 0xabcdef00` will put have cmd_args=['0xabcdef00'] 164 - note that we use core.value class as an interface to underlying C structures. Refer [Section B] for more details. 165 - use kern.globals.\<variable_name> & kern.GetValueFromAddress for building values from addresses. 166 - remember that the ideal type of object to be passed around is core.value 167 - Anything you 'print' will be relayed to lldb terminal output. 168 169 5. If the user has passed any custom options they would be in cmd_options dict. the format is `{'-<optionflag>':'<value>'}`. The \<value> will be '' (empty string) for non-option flags. 170 171 6. If your function finds issue with the passed argument then you can `raise ArgumentError('error_message')` to notify the user. The framework will automatically catch this and show appropriate help using the function doc string. 172 173 7. Please use "##" for commenting your code. This is important because single "#" based strings may be mistakenly considered in `unifdef` program. 174 175 Time for some code example? Try reading the code for function ShowTaskVmeHelper in memory.py. 176 177SPECIAL Note: Very often you will find yourself making changes to a file for some command/summary and would like to test it out in lldb. 178 179To easily reload your changes in lldb please follow the below example. 180 181 * you fire up lldb and start using zprint. And soon you need to add functionality to zprint. 182 183 * you happily change a function code in memory.py file to zprint macro. 184 185 * now to reload that particular changes without killing your debug session do 186 (lldb) xnudebug reload memory 187 memory is reloaded from ./memory.py 188 (lldb) 189 190 * Alternatively, you can use lldb`s command for script loading as 191 (lldb) command script import /path/to/memory.py 192 You can re-run the same command every time you update the code in file. 193 194 It is very important that you do reload using xnudebug command as it does the plumbing of commands and types for your change in the module. Otherwise you could easily get confused 195 why your changes are not reflected in the command. 196 197 198D. Kernel type summaries. 199========================== 200i. Using summaries 201------------------ 202The lldb debugger provides ways for user to customize how a particular type of object be described when printed. These are very useful in displaying complex and large structures 203where only certain fields are important based on some flag or value in some field or variable. The way it works is every time lldb wants to print an object it checks 204for registered summaries. We can define python functions and hook it up with lldb as callbacks for type summaries. For example. 205 206 (lldb) print first_zone 207 (zone_t) $49 = 0xd007c000 208 ZONE TOT_SZ ALLOC_ELTS FREE_ELTS FREE_SZ ELT_SZ ALLOC(ELTS PGS SLK) FLAGS NAME 209 0x00000000d007c000 29808 182 25 3600 144 4096 28 1 64 X$ zones 210 (lldb) 211Just printing the value of first_zone as (zone_t) 0xd007c000 wouldnt have been much help. But with the registered summary for zone_t we can see all the interesting info easily. 212 213You do not need to do anything special to use summaries. Once they are registered with lldb they show info automatically when printing objects. However if you wish to 214see all the registered type summaries run the command `type summary list -w kernel` on lldb prompt. 215Also if you wish to quickly disable the summaries for a particular command use the `showraw` command. 216 217ii. Writing new summary functions 218--------------------------------- 219lldb provides really flexible interface for building summaries for complex objects and data. If you find that a struct or list can be 220diagnosed better if displayed differently, then feel free to add a type summary for that type. Following is an easy guide on how to do that. 221 222 1. Register a function as a callback for displaying information for a type. Use the `@lldb_type_summary()` decorator with an array of types you wish to register for callback 223 224 2. Provide a header for the summary using `@header()` decorator. This is a strong requirement for summaries. This gets displayed before the output 225 of `GetTypeSummary()` is displayed. [In case you do not wish to have header then still define it as "" (empty string) ] 226 227 3. Define the function with signature of `GetSomeTypeSummary(valobj)`. It is highly recommended that the naming be consistent to `Get.*?Summary(valobj)` 228 The valobj argument holds the core.value object for display. 229 230 4. Use the utility functions and memory read operations to pull out the required information. 231 [ use `kern.globals` & `kern.GetValueFromAddress` for building args to core functions. ] 232 [ remember that the ideal type of object to be passed around is core.value ] 233 234 5. return a string that would be printed by the caller. When lldb makes a call back it expects a str to be returned. So do not print 235 directly out to console. [ debug info or logs output is okay to be printed anywhere :) ] 236 237Time for some code example? Try reading the code for GetTaskSummary() in process.py. 238 239 240 241E. FAQs and Generel Coding Guidelines 242====================================== 243 244i. Frequently Asked Questions 245----------------------------- 246 247 Q. How do I avoid printing the summary and see the actual data in a structure? 248 249 A. There is a command called `showraw`. This will disable all kernel specific type summaries and execute any command you provide. For ex. 250 251 (lldb) print (thread_t) 0x80d6a620 252 (thread_t) $45 = 0x80d6a620 253 thread thread_id processor pri io_policy state wait_queue wait_event wmesg thread_name 254 0x80d6a620 0x317 0x902078c8 61 W 0x910cadd4 0x0 SystemSoundServer 255 (lldb) showraw print (thread_t) 0x80d6a620 256 (thread_t) $48 = 0x80d6a620 257 258 Q. I typed `showallvnodes` and nothing happens for a long time? OR How do I get output of long running command instantly on the terminal? 259 260 A. The lldb command interface tries to build result object from output of a python function. So in case of functions with very long output or runtime it may 261 seem that the lldb process is hung. But it is not. You can use "-i" option to get immediate output on terminal. 262 263 ex. (lldb) showallvnodes -- -i 264 Immediate Output 265 .... 266 267 Q. I made a change in a python file for a command or summary, but the output is not reflected in the lldb command? 268 269 A. The python framework does not allow for removing a loaded module and then reloading it. So sometimes if a command has a cached value from 270 old code that it will still call the old function and hence will not display new changes in file on disk. If you find yourself in such a situation 271 please see [Section C. -> SPECIAL Note]. If the change is to basic class or caching mechanism than it is advised to quit lldb and re-load all modules again. 272 273 Q. I am new to python. I get an error message that I do not understand. what should I do? 274 275 A. The syntax for python is different from conventional programming languages. If you get any message with SyntaxError or TypeError or ValueError then please review your code and look for common errors like 276 277 - wrong level of indentation? 278 - missed a ':' at the end of an if, elif, for, while statement? 279 - referencing a key in dictionary that doesn't exist? You might see KeyError in such cases. 280 - mistakenly used python reserved keyword as variable? (check http://docs.python.org/release/3.0.1/reference/lexical_analysis.html#id8) 281 - Trying to modify a string value? You can only create new strings but never modify existing ones. 282 - Trying to add a non string value to a string? This typically happens in print "time is " + gettime(). here gettime() returns int and not str. 283 - using a local variable with same name as global variable? 284 - assigning a value to global variable without declaring first? Its highly recommended to always declare global variable with 'global' keyword 285 If you still have difficulty you can look at the python documentation at http://docs.python.org 286 287 288 Q. I wish to pass value of variable/expression to xnu lldb macro that accepts only pointers. How can I achieve that? 289 290 A. Many lldb macros have syntax that accepts pointers (eg showtaskstacks etc). In order to have your expression be evaluated before passing to command use `back ticks`. For example: 291 292 (lldb) showtaskstacks `(task_t)tasks.next` 293 This way the expressing withing ` ` is evaluated by lldb and the value is passed to the command. 294 Note that if your argument pointer is bad or the memory is corrupted lldb macros will fail with a long backtrace that may not make sense. gdb used to fail silently but lldb does not. 295 Please see Section F(i) for more information on reading backtraces. 296 297 Q. I connected to a coredump file with lldb --core corefile and I got RuntimeError: Unable to find lldb thread for tid=XYZ. What should I do? 298 299 A. This is most likely the case that lldb ignored the operating system plugin in the dSYM and hence threads are not populated. Please put the line 'settings set target.load-script-from-symbol-file true' in your ~/.lldbinit file. If you do not have access you can alternatively do 300 301 bash# lldb 302 (lldb) settings set target.load-script-from-symbol-file true 303 (lldb) file --core corefile 304 305 306ii. Formatted output printing - zen and peace for life 307------------------------------------------------------ 308 309To avoid the horrors of printing a tabular data on console and then 2 weeks later again messing with it for a new field, it is recommended to follow these guidelines. 310 311 * any python string can be invoked to "".format() and hence makes it very easy to play with formats 312 313 * As a convention, I suggest that for printing pointer values in hex use "{0: <#020x}".format(some_int_value). This will print nice 0x prefixed strings with length padded to 20. 314 315 * If you need help with format options take a look at http://docs.python.org/library/string.html#format-string-syntax 316 317 * [ I'd first create a format string for data and then for the header just change the x's and d's to s and pass the header strings to format command. see GetTaskSummary()] 318 319 * If you need to print a string from a core.value object then use str() to get string representation of value. 320 321 322iii. Coding conventions 323----------------------- 324It is very very HIGHLY RECOMMENDED to follow these guidelines for writing any python code. 325 326 * Python is very sensitive to tabs and spaces for alignment. So please make sure you **INDENT YOUR CODE WITH SPACES** at all times. 327 328 * The standard tab width is 4 spaces. Each increasing indent adds 4 spaces beginning of the line. 329 330 * The format for documentation is - 331 """ A one line summary describing what this function / class does 332 Detailed explanation if necessary along with params and return values. 333 """ 334 335 * All Classes and functions should have a doc string describing what the function does 336 A consistent format is expected. For ex. 337 def SumOfNumbers(a, b, c, d): 338 """ Calculate sum of numbers. 339 params: 340 a - int, value to be added. can be 0 341 b - int/float, value to be added. 342 returns: 343 int/float - Sum of two values 344 raises: 345 TypeError - If any type is not identified in the params 346 """ 347 348 * A Class or Function should always start with CAPITAL letter and be CamelCase. If a function is for internal use only than it starts with '_'. 349 350 * Function params should always be lower_case and be word separated with '_' 351 352 * A local variable inside a function should be lower_case and separated with '_' 353 354 * A variable for internal use in object should start with '_'. 355 356 * if a class variable is supposed to hold non native type of object, it is good idea to comment what type it holds 357 358 * A class function with name matching `Get(.*?)Summary()` is always supposed to return a string which can be printed on stdout or any file. 359 360 * Functions beginning with "Get" (eg. GetVnodePath()) mean they return a value and will not print any output to stdout. 361 362 * Functions beginning with "Show" (eg. ShowZTrace()) mean they will print data on screen and may not return any value. 363 364 365iv. Submitting changes in lldbmacros 366------------------------------------ 367 368To contribute new commands or fixes to existing one, it is recommended that you follow the procedure below. 369 370 * Save the changes requried for new command or fix into lldbmacros directory. 371 372 * Make sure that the coding conventions are strictly followed. 373 374 * Run syntax checker on each of the modified files. It will find basic formatting errors in the changed files for you. 375 376 * If you are adding new file then please update the Makefile and xnu.py imports to ensure they get compiled during kernel build. 377 378 * Do a clean build of kernel from xnu top level directory. 379 380 * Verify that your changes are present in the dSYM directory of new build. 381 382 * Re-run all your test and verification steps with the lldbmacros from the newly packaged dSYM/Contents/Resources/Python/lldbmacros. 383 384v. Common utility functions and paradigms 385----------------------------------------- 386Please search and look around the code for common util functions and paradigm 387 388 * Take a peek at utils.py for common utility like sizeof_fmt() to humanize size strings in KB, MB etc. The convention is to have functions that do self contained actions and does not require intricate knowledge of kernel structures in utils.py 389 390 * If you need to get pagesize of the traget system, do not hard code any value. kern.globals.page_size is your friend. Similarly use config['verbosity'] for finding about configs. 391 392 * If you are developing a command for structure that is different based on development/release kernels please use "hasattr()" functionality to conditionalize referencing #ifdef'ed fields in structure. See example in def GetTaskSummary(task) in process.py 393 394 395F. Development and Debugging on lldb kernel debugging platform. 396=============================================================== 397 398i. Reading a exception backtrace 399-------------------------------- 400In case of an error the lldbmacros may print out an exception backtrace and halt immediately. The backtrace is very verbose and may be confusing. The important thing is to isolate possible causes of failure, and eventually filing a bug with kernel team. Following are some common ways where you may see an exception instead of your expected result. 401 402 * The lldbmacros cannot divine the type of memory by inspection. If a wrong pointer is passed from commandline then, the command code will try to read and show some results. It may still be junk or plain erronous. Please make sure your command arguments are correct. 403 For example: a common mistake is to pass task address to showactstack. In such a case lldb command may fail and show you a confusing backtrace. 404 405 * Kernel debugging is particularly tricky. Many parts of memory may not be readable. There could be failure in network, debugging protocol or just plain bad memory. In such a case please try to see if you can examine memory for the object you are trying to access. 406 407 * In case of memory corruption, the lldbmacros may have followed wrong pointer dereferencing. This might lead to failure and a exception to be thrown. 408 409 410ii. Loading custom or local lldbmacros and operating_system plugin 411------------------------------------------------------------------ 412 413The lldbmacros are packaged right into the dSYM for the kernel executable. This makes debugging very easy since they can get loaded automatically when symbols are loaded. 414However, this setup makes it difficult for a lldbmacro developer to load custom/local macros. Following is the suggested solution for customizing your debugging setup: 415 416 * set up environment variable DEBUG_XNU_LLDBMACROS=1 on your shell. This will disable the automatic setup of lldbmacros and the operating_system.py from the symbols. 417 - bash$ export DEBUG_XNU_LLDBMACROS=1 418 419 * start lldb from the shell 420 - bash$ lldb 421 422 * [optional] If you are making changes in the operating_system plugin then you need to set the plugin path for lldb to find your custom operating_system plugin file. 423 - (lldb)settings set target.process.python-os-plugin-path /path/to/xnu/tools/lldbmacros/core/operating_system.py 424 If you do not wish to change anything in operating_system plugin then just leave the setting empty. The symbol loading module will set one up for you. 425 426 * Load the xnu debug macros from your custom location. 427 - (lldb)command script import /path/to/xnu/tools/lldbmacros/xnu.py 428 429 430iii. Adding debug related 'printf's 431----------------------------------- 432 433The xnu debug framework provides a utility function (debuglog) in utils.py. Please use this for any of your debugging needs. It will not print any output unless the user turns on debug logging on the command. Please check the documentaiton of debuglog for usage and options. 434 435 * To enable/disable logging 436 - (lldb) xnudebug debug 437 Enabled debug logging. 438 439 440