1*d4514f0bSApple OSS DistributionsTask References 2*d4514f0bSApple OSS Distributions=============== 3*d4514f0bSApple OSS Distributions 4*d4514f0bSApple OSS DistributionsFinding the source of task reference count leaks. 5*d4514f0bSApple OSS Distributions 6*d4514f0bSApple OSS DistributionsBackground 7*d4514f0bSApple OSS Distributions---------- 8*d4514f0bSApple OSS Distributions 9*d4514f0bSApple OSS DistributionsTasks in XNU are reference counted. When a task is created it starts with two 10*d4514f0bSApple OSS Distributionsreferences - one for the caller and one for the task itself. Over the lifetime 11*d4514f0bSApple OSS Distributionsof the task this reference count is modified, for example when a thread is 12*d4514f0bSApple OSS Distributionscreated it increments the reference count and when it exits that count drops. 13*d4514f0bSApple OSS DistributionsWhen a reference count reaches zero, the task is freed. 14*d4514f0bSApple OSS Distributions 15*d4514f0bSApple OSS DistributionsTo grab a reference: 16*d4514f0bSApple OSS Distributions```c 17*d4514f0bSApple OSS Distributionstask_reference() 18*d4514f0bSApple OSS Distributions``` 19*d4514f0bSApple OSS Distributions 20*d4514f0bSApple OSS DistributionsTo release a reference: 21*d4514f0bSApple OSS Distributions```c 22*d4514f0bSApple OSS Distributionstask_deallocate() 23*d4514f0bSApple OSS Distributions``` 24*d4514f0bSApple OSS Distributions 25*d4514f0bSApple OSS DistributionsOne of the big problems seen with task references is that difficult to debug 26*d4514f0bSApple OSS Distributions_leaks_ commonly occur. This happens when a reference is taken but never 27*d4514f0bSApple OSS Distributionsreleased. The task is kept around indefinitely and eventually the system runs 28*d4514f0bSApple OSS Distributionsout of a finite resource (for example ASIDs). At this point there is very little 29*d4514f0bSApple OSS Distributionsinformation to determine what code was responsible for the leak. 30*d4514f0bSApple OSS Distributions 31*d4514f0bSApple OSS Distributions 32*d4514f0bSApple OSS DistributionsTask Reference Groups 33*d4514f0bSApple OSS Distributions-------------------- 34*d4514f0bSApple OSS Distributions 35*d4514f0bSApple OSS DistributionsReference groups are a feature which keep track of statistics (and when 36*d4514f0bSApple OSS Distributionsconfigured backtrace information) for a set of references. Reference groups are 37*d4514f0bSApple OSS Distributionshierarchical. To help with debugging the following task reference group 38*d4514f0bSApple OSS Distributionshierarchy is used: 39*d4514f0bSApple OSS Distributions 40*d4514f0bSApple OSS Distributions``` 41*d4514f0bSApple OSS Distributionstask 42*d4514f0bSApple OSS Distributions -> task_internal 43*d4514f0bSApple OSS Distributions -> task_local_internal 44*d4514f0bSApple OSS Distributions -> task_kernel 45*d4514f0bSApple OSS Distributions -> task_local_internal 46*d4514f0bSApple OSS Distributions -> task_mig 47*d4514f0bSApple OSS Distributions -> task_local_internal 48*d4514f0bSApple OSS Distributions -> task_external 49*d4514f0bSApple OSS Distributions -> task_local_external 50*d4514f0bSApple OSS Distributions -> task_com.apple.security.sandbox 51*d4514f0bSApple OSS Distributions -> task_com.apple.security.sandbox 52*d4514f0bSApple OSS Distributions -> task_com.apple.driver.AppleHV 53*d4514f0bSApple OSS Distributions -> task_com.apple.driver.AppleHV 54*d4514f0bSApple OSS Distributions ... 55*d4514f0bSApple OSS Distributions``` 56*d4514f0bSApple OSS Distributions 57*d4514f0bSApple OSS DistributionsThe `task` group contains a count of all task references in the system. The 58*d4514f0bSApple OSS Distributionsfirst-level groups are static and sub-divide task references based on the 59*d4514f0bSApple OSS Distributionssub-system they come from. `task_external` is used for kext references and each 60*d4514f0bSApple OSS Distributionskext will be dynamically assigned a reference group as needed (if there's 61*d4514f0bSApple OSS Distributionsone available). At the bottom level, there's a per-task (local) ref group under 62*d4514f0bSApple OSS Distributionseach global group. 63*d4514f0bSApple OSS DistributionsThe exact hierarchy of task references (specifically what per-task reference 64*d4514f0bSApple OSS Distributionsgroups are created) changes depending on the 'task_refgrp' boot arg. 65*d4514f0bSApple OSS Distributions 66*d4514f0bSApple OSS DistributionsTask reference groups can be explored in `lldb` as follows: 67*d4514f0bSApple OSS Distributions 68*d4514f0bSApple OSS Distributions``` 69*d4514f0bSApple OSS Distributions(lldb) showglobaltaskrefgrps 70*d4514f0bSApple OSS Distributionsos_refgrp name count retain release log 71*d4514f0bSApple OSS Distributions0xffffff801ace9250 task_kernel 68 367663 367595 0x0 72*d4514f0bSApple OSS Distributions0xffffff801ace9288 task_internal 974 4953 3979 0x0 73*d4514f0bSApple OSS Distributions0xffffff801ace92c0 task_mig 0 3670 3670 0x0 74*d4514f0bSApple OSS Distributions0xffffff801ace9218 task_external 35 108 73 0x0 75*d4514f0bSApple OSS Distributions0xffffff9369dc7b20 task_com.apple.iokit.IOAcceleratorFamily2 29 77 48 0x0 76*d4514f0bSApple OSS Distributions0xffffff936a3f0a20 task_com.apple.iokit.CoreAnalyticsFamily 1 1 0 0x0 77*d4514f0bSApple OSS Distributions0xffffff936a22cb20 task_com.apple.iokit.EndpointSecurity 0 1 1 0x0 78*d4514f0bSApple OSS Distributions0xffffff936a283f60 task_com.apple.iokit.IOSurface 5 5 0 0x0 79*d4514f0bSApple OSS Distributions0xffffff936a3f08a0 task_com.apple.security.sandbox 0 24 24 0x0 80*d4514f0bSApple OSS Distributions 81*d4514f0bSApple OSS Distributions``` 82*d4514f0bSApple OSS Distributions 83*d4514f0bSApple OSS DistributionsDisplay a task's reference groups: 84*d4514f0bSApple OSS Distributions 85*d4514f0bSApple OSS Distributions``` 86*d4514f0bSApple OSS Distributions(lldb) showtaskrefgrps kernel_task 87*d4514f0bSApple OSS Distributionsos_refgrp name count retain release log 88*d4514f0bSApple OSS Distributions0xffffff936a4b9200 task_local_kernel 1 6 5 0x0 89*d4514f0bSApple OSS Distributions0xffffff936a4b9238 task_local_internal 132 619 487 0x0 90*d4514f0bSApple OSS Distributions``` 91*d4514f0bSApple OSS Distributions 92*d4514f0bSApple OSS DistributionsThe reference group hierarchy for a specific group can be displayed as follows: 93*d4514f0bSApple OSS Distributions 94*d4514f0bSApple OSS Distributions``` 95*d4514f0bSApple OSS Distributions(lldb) showosrefgrphierarchy 0xffffff936a3f08a0 96*d4514f0bSApple OSS Distributions0xffffff801ace9988 all 1121 377740 376619 0x0 97*d4514f0bSApple OSS Distributions0xffffff801ace91e0 task 1077 376394 375317 0x0 98*d4514f0bSApple OSS Distributions0xffffff801ace9218 task_external 35 108 73 0x0 99*d4514f0bSApple OSS Distributions0xffffff936a3f08a0 task_com.apple.security.sandbox 0 24 24 0x0 100*d4514f0bSApple OSS Distributions``` 101*d4514f0bSApple OSS Distributions 102*d4514f0bSApple OSS DistributionsReference groups are normally disabled, but task reference group statistics 103*d4514f0bSApple OSS Distributions*are* enabled by default (for `RELEASE` builds, reference groups are not available 104*d4514f0bSApple OSS Distributionsat all). Backtrace logging for all groups is disabled, including task reference 105*d4514f0bSApple OSS Distributionsgroups. To enable backtrace logging and reference group statistics, the `rlog` 106*d4514f0bSApple OSS Distributionsboot-arg must be used. Backtrace logging for task reference groups is only 107*d4514f0bSApple OSS Distributionsenabled when `rlog` has been set to a suitable value. 108*d4514f0bSApple OSS Distributions 109*d4514f0bSApple OSS DistributionsFor example 110*d4514f0bSApple OSS Distributions 111*d4514f0bSApple OSS DistributionsTo enable statistics for all reference groups and backtrace logging for the 112*d4514f0bSApple OSS Distributions*task_external* reference group in particular: 113*d4514f0bSApple OSS Distributions 114*d4514f0bSApple OSS Distributions``` 115*d4514f0bSApple OSS Distributionsnvram boot-args="rlog=task_external ..." 116*d4514f0bSApple OSS Distributions``` 117*d4514f0bSApple OSS Distributions 118*d4514f0bSApple OSS Distributions``` 119*d4514f0bSApple OSS Distributions(lldb) showglobaltaskrefgrps 120*d4514f0bSApple OSS Distributionsos_refgrp name count retain release log 121*d4514f0bSApple OSS Distributions0xffffff801e0e9250 task_kernel 1259 132739 131480 0x0 122*d4514f0bSApple OSS Distributions0xffffff801e0e9218 task_external 35 100 65 0xffffffa05b3fc000 123*d4514f0bSApple OSS Distributions0xffffff936d117be0 task_com.apple.iokit.IOAcceleratorFamily2 29 77 48 0x0 124*d4514f0bSApple OSS Distributions0xffffff936db9fa20 task_com.apple.iokit.CoreAnalyticsFamily 1 1 0 0x0 125*d4514f0bSApple OSS Distributions0xffffff936d9dbb20 task_com.apple.iokit.EndpointSecurity 0 1 1 0x0 126*d4514f0bSApple OSS Distributions0xffffff936da324e0 task_com.apple.iokit.IOSurface 5 5 0 0x0 127*d4514f0bSApple OSS Distributions0xffffff936db9f8a0 task_com.apple.security.sandbox 0 16 16 0x0 128*d4514f0bSApple OSS Distributions 129*d4514f0bSApple OSS Distributions 130*d4514f0bSApple OSS Distributions(lldb) showbtlogrecords 0xffffffa05b3fc000 131*d4514f0bSApple OSS Distributions-------- OP 1 Stack Index 0 with active refs 1 of 165 -------- 132*d4514f0bSApple OSS Distributions0xffffff801da7c1cb <kernel.development`ref_log_op at refcnt.c:107> 133*d4514f0bSApple OSS Distributions0xffffff801d27c35d <kernel.development`task_reference_grp at task_ref.c:274> 134*d4514f0bSApple OSS Distributions0xffffff801ecc014e <EndpointSecurity`VMMap::taskSelf()> 135*d4514f0bSApple OSS Distributions0xffffff801eccc845 <EndpointSecurity`EndpointSecurityClient::create(ScopedPointer<MachSendWrapper> const&, proc*, ScopedPointer<EndpointSecurityExternalClient> const&, es_client_config_t const&)> 136*d4514f0bSApple OSS Distributions... 137*d4514f0bSApple OSS Distributions``` 138