1*2c2f96dcSApple OSS Distributions""" 2*2c2f96dcSApple OSS Distributions Triage Macros for zone related panics 3*2c2f96dcSApple OSS Distributions 4*2c2f96dcSApple OSS Distributions Supported panic strings from xnu/osfmk/kern/zalloc.c: 5*2c2f96dcSApple OSS Distributions "a freed zone element has been modified in zone %s: expected %p but found %p, bits changed %p, at offset %d of %d in element %p, cookies %p %p" and 6*2c2f96dcSApple OSS Distributions "zalloc: zone map exhausted while allocating from zone %s, likely due to memory leak in zone %s (%lu total bytes, %d elements allocated)" 7*2c2f96dcSApple OSS Distributions These macros are dependant on the above panic strings. If the strings are modified in any way, this script must be updated to reflect the change. 8*2c2f96dcSApple OSS Distributions 9*2c2f96dcSApple OSS Distributions To support more zone panic strings: 10*2c2f96dcSApple OSS Distributions 1. Add the panic string regex to the globals and include in the named capture group 'zone' (the zone to be 11*2c2f96dcSApple OSS Distributions logged) as well as any other info necessary to parse out of the panic string. 12*2c2f96dcSApple OSS Distributions 2. Add a check for the panic string regex in ZoneTriage(), which then calls into the function you create. 13*2c2f96dcSApple OSS Distributions 3. Add a check for the panic string regex in CheckZoneBootArgs() which sets the variable panic_string_regex to your 14*2c2f96dcSApple OSS Distributions panic string regex if found. 15*2c2f96dcSApple OSS Distributions 4. Create a function that can be called either through the zonetriage macro ZoneTriage() or using its own macro. 16*2c2f96dcSApple OSS Distributions This function should handle all lldb commands you want to run for this type of zone panic. 17*2c2f96dcSApple OSS Distributions""" 18*2c2f96dcSApple OSS Distributionsfrom xnu import * 19*2c2f96dcSApple OSS Distributionsimport sys, shlex 20*2c2f96dcSApple OSS Distributionsfrom utils import * 21*2c2f96dcSApple OSS Distributionsimport xnudefines 22*2c2f96dcSApple OSS Distributionsimport re 23*2c2f96dcSApple OSS Distributionsimport os.path 24*2c2f96dcSApple OSS Distributions 25*2c2f96dcSApple OSS Distributions## Globals 26*2c2f96dcSApple OSS Distributionspanic_string = None 27*2c2f96dcSApple OSS Distributions## If the following panic strings are modified in xnu/osfmk/kern/zalloc.c, they must be updated here to reflect the change. 28*2c2f96dcSApple OSS Distributionszone_element_modified = ".*a freed zone element has been modified in zone (?P<zone>.+): expected (0x)?([0-9A-Fa-f]*)? but found (0x)?([0-9A-Fa-f]*)?, bits changed (0x)?([0-9A-Fa-f]*)?, at offset ([0-9]*)? of ([0-9]*)? in element (?P<element>0x[0-9A-Fa-f]*), cookies (0x)?([0-9A-Fa-f]*)? (0x)?([0-9A-Fa-f]*)?.*" 29*2c2f96dcSApple OSS Distributionszone_map_exhausted = ".*zalloc: zone map exhausted while allocating from zone .+, likely due to memory leak in zone (?P<zone>.+) \(([0-9]*)? total bytes, ([0-9]*)? elements allocated\).*" 30*2c2f96dcSApple OSS Distributions 31*2c2f96dcSApple OSS Distributions# Macro: zonetriage, zonetriage_freedelement, zonetriage_memoryleak 32*2c2f96dcSApple OSS Distributions@lldb_command('zonetriage') 33*2c2f96dcSApple OSS Distributionsdef ZoneTriage(cmd_args=None): 34*2c2f96dcSApple OSS Distributions """ Calls function specific to type of zone panic based on the panic string 35*2c2f96dcSApple OSS Distributions """ 36*2c2f96dcSApple OSS Distributions global panic_string 37*2c2f96dcSApple OSS Distributions if panic_string is None: 38*2c2f96dcSApple OSS Distributions try: 39*2c2f96dcSApple OSS Distributions panic_string = lldb_run_command("paniclog").split('\n', 1)[0] 40*2c2f96dcSApple OSS Distributions except: 41*2c2f96dcSApple OSS Distributions return 42*2c2f96dcSApple OSS Distributions if re.match(zone_element_modified, panic_string) is not None: 43*2c2f96dcSApple OSS Distributions ZoneTriageFreedElement() 44*2c2f96dcSApple OSS Distributions elif re.match(zone_map_exhausted, panic_string) is not None: 45*2c2f96dcSApple OSS Distributions ZoneTriageMemoryLeak() 46*2c2f96dcSApple OSS Distributions else: 47*2c2f96dcSApple OSS Distributions print("zonetriage does not currently support this panic string.") 48*2c2f96dcSApple OSS Distributions 49*2c2f96dcSApple OSS Distributions@lldb_command('zonetriage_freedelement') 50*2c2f96dcSApple OSS Distributionsdef ZoneTriageFreedElement(cmd_args=None): 51*2c2f96dcSApple OSS Distributions """ Runs zstack_findelem on the element and zone being logged based on the panic string regex 52*2c2f96dcSApple OSS Distributions """ 53*2c2f96dcSApple OSS Distributions global panic_string 54*2c2f96dcSApple OSS Distributions if panic_string is None: 55*2c2f96dcSApple OSS Distributions try: 56*2c2f96dcSApple OSS Distributions panic_string = lldb_run_command("paniclog").split('\n', 1)[0] 57*2c2f96dcSApple OSS Distributions except: 58*2c2f96dcSApple OSS Distributions return 59*2c2f96dcSApple OSS Distributions CheckZoneBootArgs() 60*2c2f96dcSApple OSS Distributions ## Run showzonesbeinglogged. 61*2c2f96dcSApple OSS Distributions print("(lldb) zstack_showzonesbeinglogged\n%s\n" % lldb_run_command("zstack_showzonesbeinglogged")) 62*2c2f96dcSApple OSS Distributions ## Capture zone and element from panic string. 63*2c2f96dcSApple OSS Distributions values = re.search(zone_element_modified, panic_string) 64*2c2f96dcSApple OSS Distributions if values is None or 'zone' not in values.group() or 'element' not in values.group(): 65*2c2f96dcSApple OSS Distributions return 66*2c2f96dcSApple OSS Distributions element = values.group('element') 67*2c2f96dcSApple OSS Distributions zone = values.group('zone') 68*2c2f96dcSApple OSS Distributions btlog = FindZoneBTLog(zone) 69*2c2f96dcSApple OSS Distributions if btlog is not None: 70*2c2f96dcSApple OSS Distributions print("(lldb) zstack_findelem " + btlog + " " + element) 71*2c2f96dcSApple OSS Distributions findelem_output = lldb_run_command("zstack_findelem " + btlog + " " + element) 72*2c2f96dcSApple OSS Distributions findelem_output = re.sub('Scanning is ongoing. [0-9]* items scanned since last check.\n', '', findelem_output) 73*2c2f96dcSApple OSS Distributions print(findelem_output) 74*2c2f96dcSApple OSS Distributions 75*2c2f96dcSApple OSS Distributions@lldb_command('zonetriage_memoryleak') 76*2c2f96dcSApple OSS Distributionsdef ZoneTriageMemoryLeak(cmd_args=None): 77*2c2f96dcSApple OSS Distributions """ Runs zstack_findtop and zstack_findleak on all zones being logged 78*2c2f96dcSApple OSS Distributions """ 79*2c2f96dcSApple OSS Distributions global kern 80*2c2f96dcSApple OSS Distributions CheckZoneBootArgs() 81*2c2f96dcSApple OSS Distributions ## Run showzonesbeinglogged. 82*2c2f96dcSApple OSS Distributions print("(lldb) zstack_showzonesbeinglogged\n%s\n" % lldb_run_command("zstack_showzonesbeinglogged")) 83*2c2f96dcSApple OSS Distributions for zval, _ in kern.zones: 84*2c2f96dcSApple OSS Distributions btlog = getattr(zval, 'z_btlog', None) 85*2c2f96dcSApple OSS Distributions if btlog: 86*2c2f96dcSApple OSS Distributions print('%s:' % zval.z_name) 87*2c2f96dcSApple OSS Distributions print("(lldb) zstack_findtop -N 5 0x%lx" % btlog) 88*2c2f96dcSApple OSS Distributions print(lldb_run_command("zstack_findtop -N 5 0x%lx" % btlog)) 89*2c2f96dcSApple OSS Distributions print("(lldb) zstack_findleak 0x%lx" % btlog) 90*2c2f96dcSApple OSS Distributions print(lldb_run_command("zstack_findleak 0x%lx" % btlog)) 91*2c2f96dcSApple OSS Distributions 92*2c2f96dcSApple OSS Distributionsdef CheckZoneBootArgs(cmd_args=None): 93*2c2f96dcSApple OSS Distributions """ Check boot args to see if zone is being logged, if not, suggest new boot args 94*2c2f96dcSApple OSS Distributions """ 95*2c2f96dcSApple OSS Distributions global panic_string 96*2c2f96dcSApple OSS Distributions if panic_string is None: 97*2c2f96dcSApple OSS Distributions try: 98*2c2f96dcSApple OSS Distributions panic_string = lldb_run_command("paniclog").split('\n', 1)[0] 99*2c2f96dcSApple OSS Distributions except: 100*2c2f96dcSApple OSS Distributions return 101*2c2f96dcSApple OSS Distributions panic_string_regex = "" 102*2c2f96dcSApple OSS Distributions if re.match(zone_element_modified, panic_string) is not None: 103*2c2f96dcSApple OSS Distributions panic_string_regex = zone_element_modified 104*2c2f96dcSApple OSS Distributions if re.match(zone_map_exhausted, panic_string) is not None: 105*2c2f96dcSApple OSS Distributions panic_string_regex = zone_map_exhausted 106*2c2f96dcSApple OSS Distributions values = re.search(panic_string_regex, panic_string) 107*2c2f96dcSApple OSS Distributions if values is None or 'zone' not in values.group(): 108*2c2f96dcSApple OSS Distributions return 109*2c2f96dcSApple OSS Distributions zone = values.group('zone') 110*2c2f96dcSApple OSS Distributions bootargs = lldb_run_command("showbootargs") 111*2c2f96dcSApple OSS Distributions correct_boot_args = re.search('zlog([1-9]|10)?=' + re.sub(' ', '.', zone), bootargs) 112*2c2f96dcSApple OSS Distributions if correct_boot_args is None: 113*2c2f96dcSApple OSS Distributions print("Current boot-args:\n" + bootargs) 114*2c2f96dcSApple OSS Distributions print("You may need to include: -zc -zp zlog([1-9]|10)?=" + re.sub(' ', '.', zone)) 115*2c2f96dcSApple OSS Distributions 116*2c2f96dcSApple OSS Distributionsdef FindZoneBTLog(zone): 117*2c2f96dcSApple OSS Distributions """ Returns the btlog address in the format 0x%lx for the zone name passed as a parameter 118*2c2f96dcSApple OSS Distributions """ 119*2c2f96dcSApple OSS Distributions global kern 120*2c2f96dcSApple OSS Distributions for zval, _ in kern.zones: 121*2c2f96dcSApple OSS Distributions btlog = getattr(zval, 'z_btlog', None) 122*2c2f96dcSApple OSS Distributions if btlog: 123*2c2f96dcSApple OSS Distributions if zone == "%s" % zval.z_name: 124*2c2f96dcSApple OSS Distributions return "0x%lx" % btlog 125*2c2f96dcSApple OSS Distributions return None 126*2c2f96dcSApple OSS Distributions# EndMacro: zonetriage, zonetriage_freedelement, zonetriage_memoryleak 127