Diagnosing File Descriptor Exhaustion on Cloud Mac CI

After the same Xcode pipeline runs successfully dozens of times, it may suddenly fail with Too many open files or EMFILE, or a test process may exit unpredictably. A retry might then succeed. These failures rarely indicate disk corruption. More often, the build agent, test processes, and file watchers have collectively exhausted the file descriptors available to a process. Resolving the issue requires more than running ulimit once in a terminal: you need to identify what is growing, determine which limits the agent inherited, and verify that the current concurrency stays within the available budget.

Identify Which Layer Is Failing

File descriptors represent more than regular files. They also cover sockets, pipes, directory handles, and some interprocess communication objects. The count can rise quickly when dependency resolution, parallel compilation, simulator testing, and log collection run at the same time.

Start by recording the limits from the same execution environment in which the job failed:

printf 'soft limit: '
ulimit -Sn
printf 'hard limit: '
ulimit -Hn
launchctl limit maxfiles
sysctl kern.maxfiles kern.maxfilesperproc

ulimit reports the limits inherited by the current shell and its child processes. launchctl limit shows the limits applied in the launch context, while sysctl reports system-level boundaries. These values are not interchangeable. If an interactive terminal reports 65536 but the CI log still reports 256, the agent did not inherit the terminal’s settings.

Use the following table to narrow down common failure patterns:

Do not treat a successful retry as a fix. A change in concurrent timing may temporarily keep a leak or peak below the limit, but the conditions that caused the failure still exist.

Capture Process-Level Evidence to Find the Growth Source

First obtain the build agent PID, then count open items by process. The following commands do not alter the running state:

runner_pid="$(pgrep -n -f 'ci-runner|build-runner')"
test -n "$runner_pid" || exit 1

lsof -nP -p "$runner_pid" > "/tmp/runner-lsof-${runner_pid}.txt"
lsof -nP -p "$runner_pid" | awk 'NR > 1 {count[$5]++} END {
  for (type in count) print type, count[type]
}' | sort

The agent will usually spawn xcodebuild, test hosts, and script subprocesses, so inspecting only the parent process can miss the source of the problem. You can record the descriptor count for each PID in the process tree every ten seconds:

root_pid="$runner_pid"
for sample in 1 2 3 4 5 6; do
  ps -axo pid=,ppid=,command= |
    awk -v root="$root_pid" '$1 == root || $2 == root {print $1}' |
    while read -r pid; do
      count="$(lsof -nP -p "$pid" 2>/dev/null | tail -n +2 | wc -l | tr -d ' ')"
      printf '%s pid=%s open=%s
' "$(date '+%H:%M:%S')" "$pid" "$count"
    done
  sleep 10
done

The important signal is not a single high reading, but whether the count drops after the job finishes. If the same subprocess keeps accumulating descriptors after each test run, preserve the complete lsof output and narrow the investigation by types such as REG, IPv4, IPv6, and PIPE. Do not terminate the process before collecting evidence, or the most valuable diagnostic state will be lost.

Give the Agent Explicit Launch Limits

Adding ulimit -n 65536 only to a login shell configuration file usually has no effect on an agent started by launchd. A more reliable approach is to have the agent’s entry script validate the limit before starting the job:

#!/bin/zsh
set -euo pipefail

required=65536
hard="$(ulimit -Hn)"

if [[ "$hard" != "unlimited" ]] && (( hard < required )); then
  print -u2 "File descriptor hard limit is below ${required}"
  exit 78
fi

ulimit -Sn "$required"
exec /Users/runner/ci/bin/runner

If the agent is managed by a user-level LaunchAgent, declare the limits explicitly in its plist:

<key>SoftResourceLimits</key>
<dict>
  <key>NumberOfFiles</key>
  <integer>65536</integer>
</dict>
<key>HardResourceLimits</key>
<dict>
  <key>NumberOfFiles</key>
  <integer>65536</integer>
</dict>

After making the change, reload the agent in the corresponding user session instead of assuming that the current terminal settings will propagate:

uid="$(id -u)"
plist="$HOME/Library/LaunchAgents/com.minibin.ci-runner.plist"

launchctl bootout "gui/${uid}" "$plist" 2>/dev/null || true
launchctl bootstrap "gui/${uid}" "$plist"
launchctl kickstart -k "gui/${uid}/com.minibin.ci-runner"

Then print ulimit -Sn and ulimit -Hn again from an actual CI job. If the agent does not run in a graphical user session, adjust the loading procedure for the launchd domain it actually belongs to rather than copying gui/<uid> unchanged.

Derive Safe Concurrency from Peak Usage

Raising the limit is not a reason to increase parallelism without bound. First run a single job three to five times and record both steady-state and peak usage. For example, if one job peaks at 8200, the background agent and collection processes have a baseline of 1800, and you want to run 4 jobs simultaneously, the budget is:

8200 × 4 + 1800 = 34600
34600 × 1.3 = 44980

This indicates that 65536 leaves headroom, but memory, disk I/O, and simulator count must still be monitored. If descriptor peaks are under control but jobs remain unstable, reduce test sharding or compilation concurrency instead of continuing to adjust only the file limit.

You can enforce a hard threshold before each job starts:

limit="$(ulimit -Sn)"
open_now="$(lsof -nP -p $$ | tail -n +2 | wc -l | tr -d ' ')"
reserve=$((limit - open_now))

if (( reserve < 10000 )); then
  printf 'Insufficient descriptor reserve: %s
' "$reserve" >&2
  exit 75
fi

The threshold should be based on observed peak usage, not permanently hard-coded as a value for every project. Large test matrices and lightweight builds have very different requirements.

Turn the Fix into a Verifiable Baseline

After making the changes, run the same commit and test set repeatedly at a fixed concurrency. For every run, record the starting count, peak count, ending count, and PID of any failed process. Acceptance criteria should cover at least the following:

• The soft limit inside the CI job matches the expected value.
• The hard limit is not lower than the soft limit and remains effective after the agent restarts.
• After a single job finishes, the number of open items returns to an explainable baseline.
• No subprocess grows monotonically across repeated runs.
• The calculated headroom remains available at the target concurrency.
• Failure logs preserve the process tree, lsof samples, and job stage together.
• When concurrency is reduced, peak usage changes in line with the number of jobs.

If only one test suite continues to grow, run it in isolation and progressively narrow the set of test cases. If every job fails near the same threshold, inspect the launch limits and concurrency budget first. The goal is not merely to make the error disappear temporarily, but to establish a reviewable calculation connecting limits, peak usage, and concurrency.

Frequently asked questions