Cloud Mac CI Memory Pressure and Concurrency Control

The same CI pipeline may finish in 18 minutes on one run, stall during testing on another, and then succeed after a retry. Do not immediately attribute this behavior to a machine that “occasionally slows down.” On a cloud Mac, compilers, linkers, simulators, and test processes compete for unified memory at the same time. Build results alone cannot tell you whether the cause is a code issue, excessive concurrency, or a system that has entered a sustained cycle of memory compression and swapping.

Define a reproducible observation window

For a valid diagnosis, keep the commit, dependency cache state, build command, and test set fixed. Choose a job that reproduces the problem reliably, then record its start time, end time, exit code, and failure stage. During the first sampling run, do not clear caches and change concurrency at the same time, or you will not know which variable affected the result.

First, establish a baseline for physical memory and swap usage:

sysctl -n hw.memsize
sysctl vm.swapusage
memory_pressure -Q
vm_stat

hw.memsize reports physical memory in bytes. vm.swapusage shows current swap usage, while the page size, compressed-page counts, and page-in/page-out counters from vm_stat help reveal trends. Focus on the change between the start and end of a job rather than treating a single point-in-time value as a threshold.

Low free memory does not necessarily mean the system is out of memory. macOS actively uses available memory to cache file pages. What matters is sustained memory pressure, continuously increasing swap usage, and a simultaneous increase in build duration or unexpected process termination.

Sample continuously during the build

Run diagnostic commands outside the build command so that evidence remains available after a child process exits. The following script records memory pressure, swap usage, and the 15 processes with the highest RSS every 10 seconds:

#!/bin/zsh
set -eu

out="${1:-memory-samples.log}"

while true; do
  printf '
=== %s ===
' "$(date -u '+%Y-%m-%dT%H:%M:%SZ')" >> "$out"
  memory_pressure -Q >> "$out" 2>&1
  sysctl vm.swapusage >> "$out" 2>&1
  ps -axo pid,ppid,rss,etime,command | sort -nrk3 | head -n 16 >> "$out"
  sleep 10
done

Run zsh sample-memory.zsh in a separate terminal before starting the pipeline, then stop sampling when the job finishes. RSS is typically reported in KB. It does not represent a process’s entire virtual address space, but it is sufficient to identify compiler, simulator, or test processes whose memory use keeps growing.

Preserve stage markers as well

Print UTC timestamps before and after dependency resolution, compilation, linking, simulator startup, and test execution. This lets you map an inflection point in swap usage to a specific stage instead of ending up with a system snapshot that lacks pipeline context.

Classify the source of memory pressure

The following table provides an initial classification of common patterns:

Immediately after an abnormal termination, search the relevant system logs:

log show --last 30m --style compact \
  | grep -Ei 'memory pressure|memorystatus|killed process' \
  > memory-events.log

The absence of matching log entries does not prove that memory behavior was normal, so you must still correlate sampling trends with the pipeline exit code. Likewise, do not diagnose a leak from a single process peak. Compilation and linking naturally produce short-lived spikes; sustained growth that does not subside after the stage completes is more suspicious.

Turn concurrency into an explicit budget

Set concurrency limits from measured peak usage. Reserve about 20% of physical memory for the system, remote sessions, and logging processes, then fit compilers and simulators into the remaining capacity. If compilation and testing never overlap, you can budget for them separately. If the pipeline overlaps those stages, their memory requirements must be combined.

For Xcode builds, start with a conservative experiment using -jobs:

set -o pipefail

xcodebuild \
  -workspace App.xcworkspace \
  -scheme App \
  -configuration Release \
  -jobs 4 \
  build

For simulator tests, limit the number of parallel destinations:

xcodebuild \
  -workspace App.xcworkspace \
  -scheme AppTests \
  -parallel-testing-enabled YES \
  -maximum-concurrent-test-simulator-destinations 2 \
  test

Do not copy the example values 4 and 2 without validation. Start by halving the current concurrency, run the same job at least three consecutive times, and then increase concurrency one step at a time. The goal is not to keep every CPU core saturated every second, but to achieve stable throughput without continuously growing swap usage.

Finish tuning with an acceptance checklist

Change only one parameter per round and record the following results:

1. Whether the same commit completes successfully three consecutive times.
2. Peak swap usage and the change between the start and end of the job.
3. The pipeline stage in which memory pressure rises.
4. Whether any processes still terminate unexpectedly.
5. Total duration, median duration, and the variability across the three runs.
6. Whether lower concurrency improves the number of useful jobs completed per unit of time.

If lower concurrency makes an individual build slightly slower but eliminates retries and random process exits, overall throughput is usually higher. Once the settings are validated, store the concurrency values in the pipeline configuration and keep the sampling script as an on-demand troubleshooting tool rather than running it permanently at high frequency. Measure again after changing the machine configuration or expanding the test matrix. After confirming the currently available configuration in the console, use the same benchmark job to establish a new budget.

Frequently asked questions