1# Memorystatus Subsystem 2 31. [Overview](#overview) 41. [Code layout](#code-layout) 51. [Design](#design) 61. [Threads](#threads) 71. [Snapshots](#snapshots) 81. [Dumping Caches](#dumping-caches) 9 10## Overview 11<a name="overview"></a> 12 13The xnu memorystatus subsystem is responsible for recovering the system when we're running dangerously low 14certain resources. Currently it monitors the following resources: 15 16- memory 17- vnodes 18- compressor space 19- swap space 20- zone map VA 21 22Depending on the resource, there are a variety of actions that memorystatus might take. 23One of the most common actions is to kill 1 or more processes in an attempt to recover the system. 24In addition to monitoring system level resources, the memorystatus code is also responsible 25for killing processes that go over their per-process memory limits. 26 27The memorystatus contains code to perform four actions in response to resource shortages: 28- Kill Processes 29- Freeze Processes 30- Send warning notifications 31- Swap memory from apps 32 33Each of these actions are covered in their own document in this folder. 34 35## Code Layout 36<a name="code-layout"></a> 37 38The memorystatus code lives on the BSD side of xnu. It's comprised of the following C files: 39 40- `bsd/kern/kern_memorystatus_policy.c` 41 Contains the policy decisions around when to perform which action. 42- `bsd/kern/kern_memorystatus_freeze.c` 43 Implementation of the freezer. See `doc/memorystatus/freezer.md` for details. 44- `bsd/kern/kern_memorystatus.c` 45 Contains mechanical code to implement the kill and swap actions. Should not contain any policy 46 (that should be in `bsd/kern/kern_memorystatus_policy.c`), but that's a recent refactor so 47 is a bit of a WIP. 48- `bsd/kern/kern_memorystatus_notify.c` 49 Contains both the policy and mechanical bits to send out memory pressure notifications. See `doc/memorystatus/notify.md` 50 51And the following headers: 52- `bsd/kern/kern_memorystatus_internal.h` 53- `bsd/sys/kern_memorystatus_notify.h` 54- `bsd/sys/kern_memorystatus_freeze.h` 55- `bsd/sys/kern_memorystatus.h` 56 57## Design 58<a name="design"></a> 59 60The memorystatus subsystem is designed around a central health check. 61All of the fields in this health check are defined in the `memorystatus_system_health_t` struct. See `bsd/kern/kern_memorystatus_internal.h` for the struct definition. 62 63Most of the monitoring and actions taken by the memorystatus subsystem happen in the `memorystatus_thread` (`bsd/kern/kern_memorystatus.c`). However, there are some synchronous actions that happen on other threads. See `doc/memorystatus/kill.md` for more documentation on specific kill types. 64 65Whenever it's woken up the memorystatus thread does the following: 661. Fill in the system health state by calling `memorystatus_health_check`) 671. Log this state to the os log (or serial if we're early in boot) 681. Check if the system is healthy via `memorystatus_is_system_healthy` 691. If the system is unhealthy, pick a recovery action and perform it. See `memorystatus_pick_action` (in `bsd/kern/kern_memorystatus_policy.c`) for the conditions that trigger specific actions. Note that we sometimes do pre-emptive actions on a healthy system if we're somewhat low on a specific resource. For example, we'll kill procs over their soft limit if we're under 15% available pages even if the system is otherwise healthy. 701. Go back to step 1 until the system is healthy and the thread can block. 71 72Notice that the memorystatus thread does not explicitly check why it was woken up. 73To keep the synchronization simple, anytime a resource shortage is detected the memorystatus 74thread is woken up *blindly* and it will do a full system health check. 75 76### Jetsam Bands 77 78The memorystatus subsystem has 210 priority levels. Every process in the system (except launchd) has a jetsam priority level. Higher numbers are more important. 79 80Each priority level is tracked as a TAILQ linked list . There is one global array, `memstat_bucket`, containing all of these TAILQ lists. 81A process's priority is tracked in the proc structure (See `bsd/sys/proc_internal.h`). `p_memstat_effective_priority` stores the proc's current jetsam priority, and `p_memstat_list` stores the TAILQ linkage. All lists are protected by the `proc_list_mlock` (Yes this is bad for scalability. Ideally we'd use finer grain locking or at least not share the global lock with the scheduler. See [rdar://36390487](rdar://36390487)) . 82 83Many kill types kill in ascending jetsam priority level. See `doc/memorystatus/kill.md` for more details. 84The jetsam band is either asserted by [RunningBoard](https://stashweb.sd.apple.com/projects/COREOS/repos/runningboard/browse) (apps and runningboard managed daemons) or determined by the jetsam priority set in the [JetsamProperties](https://stashweb.sd.apple.com/projects/COREOS/repos/jetsamproperties/browse) database. 85 86For reference, here are some of the band numbers: 87| Band Number | Name | Description | 88| ----------- | ---- | ----------- | 89| 0 | `JETSAM_PRIORITY_IDLE` | Idle processes | 90| 30 | `JETSAM_PRIORITY_BACKGROUND` | Docked apps on iOS. Some active daemons on other platforms. | 91| 40 | `JETSAM_PRIORITY_MAIL` | Docked apps on watchOS. Some active daemons on other platforms. | 92| 75 | `JETSAM_PRIORITY_FREEZER` | Suspended & frozen processes | 93| 100 | `JETSAM_PRIORITY_FOREGROUND` | Foreground app processes | 94| 140 | - | mediaserverd | 95| 160 | `JETSAM_PRIORITY_HOME` | SpringBoard | 96| 180 | `JETSAM_PRIORITY_IMPORTANT` | RunningBoard, watchdogd, thermalmonitord, etc.. | 97| 190 | `JETSAM_PRIORITY_CRITICAL` | CommCenter | 98 99See the full jetsam band reference on [confluence](https://confluence.sd.apple.com/display/allOSSystemsInternals/Jetsam#Jetsam-JetsamPriorities). 100 101### Daemon lifecycle 102 103The memorystatus subsystem is heavily intertwined with daemon lifecycle. A full discussion of daemon lifecycle is outside the scope of this document. If you're curious, here are some good resources: 104- [Daemon Overview](https://confluence.sd.apple.com/display/allOSSystemsInternals/Daemons#) 105- [RunningBoard's Process Management Documentation](https://confluence.sd.apple.com/display/allOSSystemsInternals/Process+Management+Paradigms) 106- [PressuredExit (A.K.A. activity tracking)](https://confluence.sd.apple.com/display/allOSSystemsInternals/Pressured+Exit) 107 108From the perspective of memorystatus there are essentially two kinds of processes: managed and unmanaged. Managed processes have their lifecycle managed by RunningBoard and have the `P_MEMSTAT_MANAGED` bit set on the `p_memstat_state` field. RunningBoard moves these processes between different jetsam bands based on their open assertions. 109 110Unmanaged processes go into their active jetsam band when they take out transactions. 111 112Daemons have different memory limits when they're inactive (in band 0) vs. active (above band 0). The inactive memory limit, active memory limit, and active jetsam band are determined via [JetsamProperties](https://stashweb.sd.apple.com/projects/COREOS/repos/jetsamproperties/browse). [Launchd](https://stashweb.sd.apple.com/projects/COREOS/repos/libxpc/browse) reads the JetsamProperties database and passes these values down to the kernel via posix_spawn(2) attributes. memorystatus stashes these values on the proc structure (`p_memstat_memlimit_active`, `p_memstat_memlimit_inactive`, `p_memstat_requestedpriority`), and applies them as daemons move between states. 113 114### Memory Monitoring 115 116Memorystatus makes most memory decisions based on the `memorystatus_available_pages` metric. This metric reflects the number of pages that memorystatus thinks could quickly be made free. This metric is defined in the `VM_CHECK_MEMORYSTATUS` macro in `osfmk/vm/vm_page.h`. 117 118Currently on non-macOS systems, it's defined as `pageable_external + free + secluded_over_target + purgeable`. Breaking that down: 119- pageable_external: file backed page count 120- free: free page count 121- secluded_over_target: `(vm_page_secluded_count - vm_page_secluded_target)`. This target comes from the device tree `kern.secluded_mem_mb`. Secluded memory is a special pool of memory that's intended for the camera so that it can startup faster on memory constrained systems. 122- purgeable: The number of purgeable volatile pages in the system. Purgeable memory is an API for clients to specify that the VM can treat the contents of a range of pages as volatile and quickly free the backing pages under pressure. See `osfmk/mach/vm_purgable.h` for the API. Note that the API was accidentally exported with incorrect spelling ("purgable" instead of "purgeable") 123 124Since we purge purgeable memory and trim the secluded pool quickly under memory pressure, this can generally be approximated to `free + file_backed` for a system under pressure. 125 126 127The `VM_CHECK_MEMORYSTATUS` macro is called whenever a page is allocated, wired, freed, etc... Basically `memorystatus_available_pages` is supposed to always be accurate down to a page level. On our larger memory systems (8 and 16GB iPads in particular) this might be overkill. 128And it calls into `memorystatus_pages_update` to actually update `memorystatus_available_pages` and issue the blind wakeup of the memorystatus thread if necessary. `memorystatus_pages_update` is also responsible for waking the freezer and memory pressure notification threads. 129 130<a name="threads"></a> 131 132This section lists the threads that comprise the memorystatus subsystem. More details on each thread are below. 133 134| Thread name | Main function | wake event | 135| ----------- | ------------- | ---------- | 136| VM\_memorystatus\_1 | `memorystatus_thread` | `jt_wakeup_cond` in `jetsam_thread_state_t` | 137| VM\_freezer | `memorystatus_freeze_thread` | `memorystatus_freeze_wakeup` | 138| VM\_pressure | `vm_pressure_thread` | `vm_pressure_thread` | 139 140### VM\_memorystatus_1 141 142This is the jetsam thread. It's responsible for running the system health check and performing most jetsam kills (see `doc/memorystatus/kill.md` for a kill breakdown). 143 144It's woken up via a call to `memorystatus_thread_wake` whenever any subsystem determines we're running low on a monitored resource. The wakeup is blind and the thread will immediately do a health check to determine what's wrong with the system. 145 146NB: There are technically three memorystatus threads: `VM_memorystatus_1`, `VM_memorystatus_2`, and `VM_memorystatus_3`. But we currently only use `VM_memorystatus_1`. At one point we tried to parallelize jetsam to speed it up, but this effort was unsuccessful. The other threads are just dead code at this point. 147 148### VM\_freezer 149 150This is the freezer thread. It's responsible for freezing processes under memory pressure and demoting processes when the freezer is full. See `doc/memorystatus/freeze.md` for more details on the freezer. 151 152It's woken up by issuing a `thread_wakeup` call to the `memorystatus_freeze_wakeup` global. This is done in `memorystatus_pages_update` if `memorystatus_freeze_thread_should_run` returns true. It's also done whenever `memorystatus_on_inactivity` runs. 153 154Upon wakeup the freezer thread will call `memorystatus_pick_freeze_count_for_wakeup` and attempt 155to freeze up to that many processes before blocking. `memorystatus_pick_freeze_count_for_wakeup` returns 1 on most platforms. But if app swap is enabled (M1 and later iPad Pros) it will return the total number of procs in all eligible bands. 156 157### VM\_pressure 158 159This is the memorystatus notification thread. It's woken up by the pageout thread via `vm_pressure_response`. `vm_pressure_response` is also called in `memorystatus_pages_update`. 160 161When awoken it calls `consider_vm_pressure_events` which winds its way to `memorystatus_update_vm_pressure`. This routine checks if the pressure level has changed and issues memory pressure notifications. It also schedules the thread call for sustained pressure kills. 162 163On macOS this thread also does idle exit kills. 164 165## Snapshots 166<a name="snapshots"></a> 167The memorystatus subsystem provides a snapshot mechanism so that 168ReportCrash can generate JetsamEvent.ips files. These files contain 169a snapshot of the system at the time that memorystatus performed 170some kills. The snapshot data structure is `memorystatus_jetsam_snapshot_t` defined in `bsd/sys/kern_memorystatus.h`. Generally speaking the snapshot contains system level memory statistics along with entries for each process in the system. Since we do not want to wake up ReportCrash while the system is low on memory, we maintain one global snapshot (`memorystatus_jetsam_snapshot` in `bsd/kern/kern_memorystatus.c`) while we're performing kills and only wake up ReportCrash once the system is healthy again. See `memorystatus_post_snapshot` in `bsd/kern/kern_memorystatus.c` which is called right before the jetsam thread blocks. 171 172**NB**: Posting the snapshot just means sending a notification to userspace that the snapshot is ready. Userspace (currently OSAnalytics) must make the `memorystatus_control` syscall with the `MEMORYSTATUS_CMD_GET_JETSAM_SNAPSHOT` subcommand to retrieve the snapshot. See `memorystatus_cmd_get_jetsam_snapshot` in `bsd/kern/kern_memorystatus.c` for details. Since we only have one global snapshot its cleared on read and thus can only have 1 consumer in userspace. 173 174### Freezer Snapshot 175The freezer snapshot, `memorystatus_jetsam_snapshot_freezer`, is a second global jetsam snapshot object. It reuses the snapshot struct definition but only contains apps that have been jetsammed. 176dasd reads this snapshot and uses it as an input for its freezer recommendation algorithm. However, we're not currently using the dasd recommendation algorithm for the freezer so this snapshot really only serves a diagnostic purpose today. 177This snapshot is also reset when dasd reads it. Note that it has to be separate from the OSAnalytics snapshot so that these daemons can read the snapshots independently. 178 179## Dumping Caches 180<a name="dumping-caches"></a> 181 182In general system caches should be cleared before we do higher band jetsams. Userspace entities should do this via purgeable memory if possible, or memory pressure notifications if not. In the kernel, memorystatus calls `memorystatus_approaching_fg_band` when we're about to do a fg band kill. This in turn calls `memorystatus_dump_caches` to clear the PPLs cache and purge all task corpses. This also sends out a notification to other entities to clear their caches (see `memorystatus_issue_fg_band_notify`). To avoid unnecessary corpse forking and purging, memorystatus blocks all additional corpse creation after it purges them until the system returns to a healthy state. 183