Skip to main content

Developing Ceph Natively on macOS

Problem​

I develop Ceph on a Mac with Apple silicon. Can I build Ceph and run a local vstart.sh test cluster natively, without a Linux VM or container?

Solution​

Yes. Clyso has developed a port of Ceph to macOS on Apple silicon, submitted upstream as a series of pull requests. With the full series applied, the client libraries, the mon, the OSD, the mgr, the MDS, the RGW and the offline tools build with the Apple clang from the Xcode Command Line Tools. vstart.sh --memstore brings up all five daemons, and RADOS, RBD, S3 and CephFS (through ceph-fuse) all work against it.

The main limitation is that BlueStore does not build on macOS, so the OSD runs on memstore and its data does not survive a restart. macOS is also not covered by upstream CI, so it is a best-effort development platform, not a place to run production clusters.

Status of the Upstream Work​

At the time of writing (September 2026), Clyso's port is not yet merged into Ceph main. It was split into seven PRs, each based on main:

PRBranchContent
#71883wip-macos-1-portabilityPortability fixes that a stricter toolchain turns up
#71884wip-macos-2-inttypesFixed-width types where the encoding depends on them
#71885wip-macos-3-builddepsBuild dependencies that are not present on every platform
#71886wip-macos-4-cephfs-proxyBuild libcephfs_proxy on Linux only
#71887wip-macos-5-cmake-applecmake: recognise AppleClang and arm64
#71888wip-macos-6-darwin-compatThe Darwin side of the compatibility layer
#71889wip-macos-7-macosmacFUSE, ARM CPU feature probing, do_cmake_macos.sh and doc/dev/macos.rst

All seven are needed for a working build. The original combined PR, #71232 (branch wip-macos-build), is kept as a draft for reference but predates the review changes in the split series.

Getting the Code​

Until the series is merged, fetch the seven branches from github.com/dvanders/ceph and merge them together:

$ git clone https://github.com/ceph/ceph.git && cd ceph
$ git remote add dvanders https://github.com/dvanders/ceph.git
$ git fetch dvanders
$ git checkout -b macos dvanders/wip-macos-1-portability
$ for n in 2-inttypes 3-builddeps 4-cephfs-proxy 5-cmake-apple 6-darwin-compat 7-macos; do
    git merge --no-edit dvanders/wip-macos-$n
  done

Merge them as above rather than onto the tip of main: main has moved on since the branches were cut, and at the time of writing #71885 needs a rebase and conflicts with it. Once the PRs are merged, a plain checkout of main is enough.

Installing Dependencies​

Install the build dependencies with Homebrew:

$ brew install cmake ninja boost ccache openssl@3 snappy lz4 zstd nss \
    icu4c sqlite libevent utf8proc c-ares

The build needs Cython and PyYAML even if the Python bindings are not wanted. Homebrew's Python is externally managed, so install them into a virtualenv in the source tree:

$ uv venv --python 3.12 .venv
$ uv pip install --python .venv/bin/python cython pyyaml setuptools \
    prettytable python-dateutil

Initialise only the submodules that are needed, and do not pass --depth 1. Several submodules pin a commit that is not the tip of their default branch, and a shallow clone silently leaves them on master (git submodule status then shows a leading +):

$ git submodule update --init src/fmt src/BLAKE3 src/xxHash src/rocksdb \
    src/zstd src/googletest src/utf8proc \
    src/erasure-code/jerasure/jerasure src/erasure-code/jerasure/gf-complete

Building​

do_cmake_macos.sh configures a base build of the mon, the OSD, the client libraries and the tools. It points cmake at the keg-only Homebrew packages, uses the virtualenv's Python, and turns off what cannot build on macOS:

$ ./do_cmake_macos.sh
$ ninja -C build vstart-base librbd rbd

The mgr, MDS, RGW and ceph-fuse need extra packages and options. Build them in a separate directory:

$ brew install oath-toolkit
$ brew install --cask macfuse
$ git submodule update --init src/libkmip
$ git submodule update --init --recursive src/s3select
$ BUILD_DIR=build-all ./do_cmake_macos.sh \
    -DWITH_SYSTEM_BOOST=OFF \
    -DWITH_MGR=ON -DWITH_CEPHFS=ON -DWITH_RADOSGW=ON -DWITH_FUSE=ON \
    -DWITH_MGR_ROOK_CLIENT=OFF -DWITH_MGR_DASHBOARD_FRONTEND=OFF \
    -DWITH_RADOSGW_AMQP_ENDPOINT=OFF -DWITH_RADOSGW_KAFKA_ENDPOINT=OFF \
    -DWITH_RADOSGW_LUA_PACKAGES=OFF -DWITH_RADOSGW_D4N=OFF \
    -DWITH_RADOSGW_POSIX=OFF -DWITH_RADOSGW_DBSTORE=OFF \
    -DWITH_RADOSGW_SELECT_PARQUET=OFF
$ ninja -C build-all vstart-base ceph-mgr ceph-mds radosgw radosgw-admin \
    ceph-fuse cython_cephfs cython_rbd

-DWITH_SYSTEM_BOOST=OFF makes Ceph build its own Boost, so that Boost.Python matches the virtualenv's interpreter. The cython_cephfs and cython_rbd targets must be named explicitly: ninja all does not build them, and without cython_cephfs the volumes mgr module cannot load, which makes vstart.sh retry ceph fs volume create forever.

Running a vstart Cluster​

$ cd build-all
$ export OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES
$ export PYTHONPATH=$PWD/../.venv/lib/python3.12/site-packages
$ MON=1 OSD=1 MDS=1 MGR=1 RGW=1 ../src/vstart.sh -n --memstore -X
$ export CEPH_CONF=$PWD/ceph.conf
$ ./bin/ceph -s
  services:
    mon: 1 daemons, quorum a
    mgr: x(active)
    mds: 1/1 daemons up
    osd: 1 osds: 1 up, 1 in
    rgw: 1 daemon active (1 hosts, 1 zones)

For RADOS and RBD work alone, MON=1 OSD=1 MDS=0 MGR=0 RGW=0 against the base build directory is enough and needs neither environment variable.

To mount CephFS, approve the macFUSE kernel extension in System Settings and restart once, then:

$ mkdir -p /tmp/cephfs
$ ./bin/ceph-fuse -c ceph.conf --client_fs a /tmp/cephfs
$ umount /tmp/cephfs

Discussion​

Things That Look Like Bugs But Are Not​

  • radosgw aborts at startup with only libobjc and CoreFoundation frames in the backtrace. radosgw forks without exec when it daemonises, and touching the Objective-C runtime after fork() is fatal on macOS. Set OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES.
  • Writes hang after a few seconds of benchmarking. memstore_device_bytes defaults to 1 GiB, and memstore is fast enough to fill it in seconds. The OSD then marks itself full and blocks writes. Bound the test (rados bench 5 write --max-objects 16) or raise memstore_device_bytes.
  • ceph pg stat and other commands hang without a mgr. Commands implemented by mgr modules hang instead of failing when no mgr is running, and ceph -s shows an empty data section. Use ceph health detail or the daemons' admin sockets instead.
  • mgr modules fail to load. The embedded interpreter only sees the virtualenv through PYTHONPATH. Install the missing pip packages (e.g. requests cherrypy scipy urllib3 cryptography) into .venv. The dashboard additionally needs -DWITH_MGR_DASHBOARD_FRONTEND=ON and npm.
  • The cluster is in HEALTH_ERR because of devicehealth. This module reports its failures as errors, and macOS cannot answer its SMART queries.

Hardware Acceleration​

The ARMv8 accelerated paths are used automatically on Apple silicon: hardware crc32c (about 7400 MB/s against 1260 MB/s for the portable implementation), NEON gf-complete for jerasure and shec, the ISA-L erasure code plugin, and ISA-L igzip for the zlib compressor. This makes a Mac a reasonable platform for developing and testing erasure code and checksum changes on ARM.

Limitations​

  • BlueStore. Darwin's POSIX aio does not implement kqueue completion delivery (SIGEV_KEVENT) and allows only 16 outstanding requests per process, so BlueStore needs a new IO backend rather than the FreeBSD one. Develop and test BlueStore changes on Linux.
  • Tests. With -DWITH_TESTS=ON, all but seven of the test targets build. The upstream document lists which. Building is not the same as passing: the test suite has not been run on macOS as a whole.
  • No /proc, no per-process POSIX timers, no NUMA or CPU affinity. The code paths that depend on them return errors, and the raise admin socket command is unavailable.

For the complete and current guide, see doc/dev/macos.rst in the source tree once the series is merged, or on the wip-macos-7-macos branch until then.

References​