Skip to main content

Developing Ceph Natively on macOS

Problem​

I develop Ceph on a Mac with Apple silicon. Can I build Ceph and run a local vstart.sh test cluster natively, without a Linux VM or container?

Solution​

Yes. Clyso has developed a port of Ceph to macOS on Apple silicon, submitted upstream as a series of pull requests. With the full series applied, the client libraries, the mon, the OSD, the mgr, the MDS, the RGW and the offline tools build with the Apple clang from the Xcode Command Line Tools. vstart.sh --memstore brings up all five daemons, and RADOS, RBD, S3 and CephFS (through ceph-fuse) all work against it.

The main limitation is that BlueStore does not build on macOS, so the OSD runs on memstore and its data does not survive a restart. macOS is also not covered by upstream CI, so it is a best-effort development platform, not a place to run production clusters.

Status of the Upstream Work​

Clyso's port was split into seven upstream PRs. As of October 2026, three of them are merged into Ceph main, together with three related PRs that the port depends on:

PRContentStatus
#71883Portability fixes that a stricter toolchain turns upopen
#71884Fixed-width types where the encoding depends on themmerged
#71885Build dependencies that are not present on every platformmerged
#71886Build libcephfs_proxy on Linux onlyopen
#71887cmake: recognise AppleClang and arm64merged
#71888The Darwin side of the compatibility layeropen
#71889macFUSE, ARM CPU feature probing, do_cmake_macos.sh and doc/dev/macos.rstopen
#72446Put Linux's AF_INET6 on the wire on FreeBSD and macOSopen
#72484Unversioned object class libraries, which #71888 relies onopen

The merged dependencies are #71445 (entity_addr_t encoding), #72479 (container_ios) and #72482 (le32toh()).

Getting the Code​

Until the remaining PRs are merged, use the draft PR #71232: its branch wip-macos-build is main plus all the open PRs above, and is refreshed when they change:

$ git clone https://github.com/ceph/ceph.git && cd ceph
$ git remote add dvanders https://github.com/dvanders/ceph.git
$ git fetch dvanders wip-macos-build
$ git checkout -b macos dvanders/wip-macos-build

It was last refreshed on 30 October 2026 and tested on macOS 27.0.1: every target below builds, and vstart.sh --memstore brings up all five daemons, creates a CephFS file system, round-trips RADOS objects, creates RBD images and answers S3 requests. Once all the PRs are merged, a plain checkout of main is enough.

Installing Dependencies​

Install the build dependencies with Homebrew:

$ brew install cmake ninja boost ccache openssl@3 snappy lz4 zstd nss \
    icu4c sqlite libevent utf8proc c-ares

The build needs Cython and PyYAML even if the Python bindings are not wanted. Homebrew's Python is externally managed, so install them into a virtualenv in the source tree:

$ uv venv --python 3.12 .venv
$ uv pip install --python .venv/bin/python cython pyyaml setuptools \
    prettytable python-dateutil

Initialise only the submodules that are needed, and do not pass --depth 1. Several submodules pin a commit that is not the tip of their default branch, and a shallow clone silently leaves them on master (git submodule status then shows a leading +):

$ git submodule update --init src/fmt src/BLAKE3 src/xxHash src/rocksdb \
    src/zstd src/googletest src/utf8proc \
    src/erasure-code/jerasure/jerasure src/erasure-code/jerasure/gf-complete

Building​

do_cmake_macos.sh configures a base build of the mon, the OSD, the client libraries and the tools. It points cmake at the keg-only Homebrew packages, uses the virtualenv's Python, and turns off what cannot build on macOS:

$ ./do_cmake_macos.sh
$ ninja -C build vstart-base librbd rbd

The mgr, MDS, RGW and ceph-fuse need extra packages and options. Build them in a separate directory:

$ brew install oath-toolkit
$ brew install --cask macfuse
$ git submodule update --init src/libkmip
$ git submodule update --init --recursive src/s3select
$ BUILD_DIR=build-all ./do_cmake_macos.sh \
    -DWITH_SYSTEM_BOOST=OFF \
    -DWITH_MGR=ON -DWITH_CEPHFS=ON -DWITH_RADOSGW=ON -DWITH_FUSE=ON \
    -DWITH_MGR_ROOK_CLIENT=OFF -DWITH_MGR_DASHBOARD_FRONTEND=OFF \
    -DWITH_RADOSGW_AMQP_ENDPOINT=OFF -DWITH_RADOSGW_KAFKA_ENDPOINT=OFF \
    -DWITH_RADOSGW_LUA_PACKAGES=OFF -DWITH_RADOSGW_D4N=OFF \
    -DWITH_RADOSGW_POSIX=OFF -DWITH_RADOSGW_DBSTORE=OFF \
    -DWITH_RADOSGW_SELECT_PARQUET=OFF
$ ninja -C build-all vstart-base ceph-mgr ceph-mds radosgw radosgw-admin \
    ceph-fuse cython_cephfs cython_rbd

-DWITH_SYSTEM_BOOST=OFF makes Ceph build its own Boost, so that Boost.Python matches the virtualenv's interpreter. The cython_cephfs and cython_rbd targets must be named explicitly: ninja all does not build them, and without cython_cephfs the volumes mgr module cannot load, which makes vstart.sh retry ceph fs volume create forever.

Running a vstart Cluster​

$ cd build-all
$ export OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES
$ export PYTHONPATH=$PWD/../.venv/lib/python3.12/site-packages
$ MON=1 OSD=1 MDS=1 MGR=1 RGW=1 ../src/vstart.sh -n --memstore -X
$ export CEPH_CONF=$PWD/ceph.conf
$ ./bin/ceph -s
  services:
    mon: 1 daemons, quorum a
    mgr: x(active)
    mds: 1/1 daemons up
    osd: 1 osds: 1 up, 1 in
    rgw: 1 daemon active (1 hosts, 1 zones)

For RADOS and RBD work alone, MON=1 OSD=1 MDS=0 MGR=0 RGW=0 against the base build directory is enough and needs neither environment variable.

To mount CephFS, approve the macFUSE kernel extension in System Settings and restart once, then:

$ mkdir -p /tmp/cephfs
$ ./bin/ceph-fuse -c ceph.conf --client_fs a /tmp/cephfs
$ umount /tmp/cephfs

Discussion​

Things That Look Like Bugs But Are Not​

  • radosgw aborts at startup with only libobjc and CoreFoundation frames in the backtrace. radosgw forks without exec when it daemonises, and touching the Objective-C runtime after fork() is fatal on macOS. Set OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES.
  • Writes hang after a few seconds of benchmarking. memstore_device_bytes defaults to 1 GiB, and memstore is fast enough to fill it in seconds. The OSD then marks itself full and blocks writes. Bound the test (rados bench 5 write --max-objects 16) or raise memstore_device_bytes.
  • ceph pg stat and other commands hang without a mgr. Commands implemented by mgr modules hang instead of failing when no mgr is running, and ceph -s shows an empty data section. Use ceph health detail or the daemons' admin sockets instead.
  • mgr modules fail to load. The embedded interpreter only sees the virtualenv through PYTHONPATH. Install the missing pip packages (e.g. requests cherrypy scipy urllib3 cryptography) into .venv. The dashboard additionally needs -DWITH_MGR_DASHBOARD_FRONTEND=ON and npm.
  • The cluster is in HEALTH_ERR because of devicehealth. This module reports its failures as errors, and macOS cannot answer its SMART queries.
  • cython_cephfs or cython_rbd fails with 'stdlib.h' file not found. Homebrew's python@3.12 records the SDK it was built with, and older builds point at a macOS 15 SDK that a macOS upgrade removes. Run brew upgrade python@3.12.
  • The mon does not start, or vstart.sh stalls at the mon. The mon refuses to run when the disk holding its data has less than 5% free, which is easy to reach on a laptop with a Ceph build tree (13 GB or more) and a ccache. Free some space, or for a throwaway cluster pass -o "mon_data_avail_crit = 0" to vstart.sh.

Hardware Acceleration​

The ARMv8 accelerated paths are used automatically on Apple silicon: hardware crc32c (about 7400 MB/s against 1260 MB/s for the portable implementation), NEON gf-complete for jerasure and shec, the ISA-L erasure code plugin, and ISA-L igzip for the zlib compressor. This makes a Mac a reasonable platform for developing and testing erasure code and checksum changes on ARM.

Limitations​

  • BlueStore. Darwin's POSIX aio does not implement kqueue completion delivery (SIGEV_KEVENT) and allows only 16 outstanding requests per process, so BlueStore needs a new IO backend rather than the FreeBSD one. Develop and test BlueStore changes on Linux.
  • Tests. With -DWITH_TESTS=ON, all but seven of the test targets build. The upstream document lists which. Building is not the same as passing: the test suite has not been run on macOS as a whole.
  • No /proc, no per-process POSIX timers, no NUMA or CPU affinity. The code paths that depend on them return errors, and the raise admin socket command is unavailable.

For the complete and current guide, see doc/dev/macos.rst in the source tree once the series is merged, or on the wip-macos-build branch until then.

References​