1# XNU Build Consolidation 2 3## Introduction and motivation 4 5XNU is supported on approximately 20 different targets. Whilst in some cases the differences between two 6given targets are small (e.g. when they both support the same ISA), XNU has traditionally required to have 7separate builds in cases where the topology of the targets differ (for example, when they feature different 8core/cluster counts or cache sizes). Similarly, SoC-specific fix-ups are usually conditionally compiled 9based on the target. 10 11Given the time it takes to compile all three different variants (release, debug and development) for each 12supported SoC, usually several times a day for various teams across Apple, the goal of this project was to 13reduce the number of existing builds, as well as to set up a simple framework that makes it easier to share 14builds across different SoCs moving forward. 15 16Although this effort could be extended to KEXTs, and hence lead to shared KernelCaches across devices, the 17scope of this document only includes XNU. In cases where KEXTs also differ across targets, or perhaps the 18required KEXTs are completely different in the first place, the kernel still needs to be linked 19appropriately with different sets of KEXTs and hence KernelCaches cannot be shared. 20 21 22## Changes required in XNU 23 24The kernel itself is relatively SoC-agnostic, although strongly architecture-dependent; this is because most 25of the SoC-specific aspects of the KernelCache are abstracted by the KEXTs. Things that pertain to the 26kernel include: 27 28* Number of cores/clusters in the system, their physical IDs and type. 29* Addresses of PIO registers that are to be accessed from the XNU side. 30* L1/L2 cache geometry parameters (e.g. size, number of set/ways). 31* Just like other components, the kernel has its share of responsibility when it comes to setting up HID 32registers and applying fix-ups at various points during boot or elsewhere at runtime. 33* Certain kernel-visible architectural features are optional, which means that two same-generation SoCs may 34still differ in their feature set. 35 36All of these problems can be solved through a mix of relying more heavily on device tree information and 37performing runtime checks. The latter is possible because both the ARM architecture and the Apple's 38extensions provide r/o registers that can be checked at runtime to discover supported features as well as 39various CPU-specific parameters. 40 41### Obtaining cache geometry parameters at runtime 42 43Although not often, the kernel may still require deriving, one way or another, parameters like cache sizes 44and number of set/ways. XNU needs most of this information to perform set/way clean/invalidate operations. 45Prior to this work, these values were hardcoded for each supported target in `proc_reg.h`, and used in 46`caches_asm.s`. The ARM architecture provides the `CCSIDR_EL1` register, which can be used in conjunction 47with `CSSELR_EL1` to select the target cache and obtain geometry information. 48 49 50### Performing CPU/Revision-specific checks at runtime 51 52CPU and revision checks may be required at various places, although the focus here has been the application 53of tunables at boot time. 54 55Tunables are often applied: 56 57* On a specific core type of a specific SoC. 58* On a subset of all of the CPU revisions. 59* On all P-cores or all E-cores. 60 61This has led in the past to a number of nested, conditionally-compiled blocks of code that are not easy to 62understand or manage as new tunables are added or SoCs/revisions are deprecated. 63 64The changes applied as part of this work focus mainly on: 65 661. Decoupling the tunable-application code from `start.s`. 672. Splitting the tunable-application code across different files, one per supported architecture (e.g. 68`tunables_h7.h`, or `tunables_h11.h`). 693. Providing "templates" for the most commonly-used combinations of tunables. 704. Providing a family of assembly macros that can be used to conditionally execute code on a specific core 71type, CPU ID, revision(s), or a combination of these. 72 73All of the macros live in the 64-bit version of `proc_reg.h`, and are SoC-agnostic; they simply check the 74`MIDR_EL1` register against a CPU revision that is passed as a parameter to the macro, where applicable. 75Similarly, where a block of code is to be executed on a core type, rather than a specific core ID, a couple 76of the provided macros can check this against `MPIDR_EL1`. 77 78 79### Checking for feature compatibility at runtime 80 81Some architectural features are optional, which means that, when disabled at compile-time, this may cause 82two same-generation SoCs to diverge. 83 84 85Rather than disabling features, and assuming this does not pose security risks or performance regressions, 86the preferred approach is to compile them in, but perform runtime checks to enable/disable them, possibly in 87early boot. The way these checks are performed varies from feature to feature (for example, VHE is an ARM 88feature, and the ARM ARM specifies how it can be discovered). For Apple-specific features, these are all 89advertised through the `AIDR_EL1` register. One of the changes is the addition of a function, 90ml_feature_supported(), that may be used to check for the presence of a feature at runtime. 91 92 93### Deriving core/cluster counts from device tree 94 95One of the aspects that until now has been hardcoded in XNU is the system topology: number of cores/clusters 96and their physical IDs. This effort piggybacks on other recent XNU changes which aimed to consolidate 97topology-related information into XNU, by parsing it from the device tree and exporting it to KEXTs through 98well-defined APIs. 99 100Changes applied as part of the XNU consolidation project include: 101 102* Extending the `ml_*` API to extract cluster information from the topology parser. New APIs include the following: 103 * `ml_get_max_cluster_number()` 104 * `ml_get_cluster_count()` 105 * `ml_get_first_cpu_id()` 106* Removing hardcoded core counts (`CPU_COUNT`) and cluster counts (`ARM_CLUSTER_COUNT`) from XNU, and 107replacing them with `ml_*` calls. 108* Similarly, deriving CPU physical IDs from the topology parser. 109 110 111### Allocating memory that is core size/cluster size/cache size aligned 112 113In some cases, certain statically-allocated arrays/structures need to be cache line-aligned, or have one 114element per core or cluster. Whilst this information is not known precisely at compile time anymore, the 115following macros have been added to provide a reasonably close upper bound: 116 117* `MAX_CPUS` 118* `MAX_CPU_CLUSTERS` 119* `MAX_L2_CLINE` 120 121These macros are defined in `board_config.h`, and should be set to the same value for a group of targets 122sharing a single build. Note that these no longer reflect actual counts and sizes, and the real values need 123to be queried at runtime through the `ml_` API. 124 125The L1 cache line size is still hardcoded, and defined as `MMU_CLINE`. Since this value is always the same 126and very often checked at various places across XNU and elsewhere, it made sense to keep it as a compile 127time macro rather than relying on runtime checks. 128 129### Restrictions on conditional compilation 130 131Currently, a family of per-SoC macros are defined at build time to enable XNU to conditionally compile code 132for different targets. These are named `ARM[64]_BOARD_CONFIG_[TARGET_NAME]`, and have historically been used 133in different places across the kernel; for example, when applying tunables, various fixes, or enabling 134disabling features. In order not to create divergences in the future across same-generation SoCs, but also 135to keep the codebase consistent, the recommendation is to avoid the use of these macros whenever possible. 136 137Instead, XNU itself defines yet another family of macros that are defined for all targets of a particular 138generation. These are named after the P-CORE introduced by each (for example, `APPLEMONSOON`, or 139`APPLEVORTEX`), and are preferred over the SoC-specific ones. Where a generation macro is not enough to 140provide correctness (which happens, for example, when the code block at hand should not be executed on a 141given SoC of the same family), appropriate runtime checks can be performed inside the conditionally-compiled 142code block. `machine_read_midr()` and `get_arm_cpu_version()` may be used for this purpose. 143