RGW SSE-KMS Secrets Caching: Operational Runbook

The RGW SSE-KMS cache introduces an in-memory layer that stores decrypted secrets within the Linux kernel keyring. This mechanism is designed to eliminate redundant round-trips to the KMS backend (such as Barbican) during repeated access to encrypted objects, effectively lowering latency and reducing throughput bottlenecks for the KMS itself.

However, configuring this feature requires significant attention to your production environment's specific constraints. You must carefully size the cache to match your active key-per-bucket workload and ensure that system-wide kernel keyring quotas are adjusted to accommodate these stored secrets. Failing to properly tune these parameters can lead to frequent cache evictions, latency spikes following service restarts, or operational failures in secret retrieval. Properly sizing your cache and configuring your keystore quotas is critical to maintaining stability and preventing performance degradation under heavy load.

Reference

• KMS secrets cache settings
• Encryption caching

Guidelines

Sizing the Cache for Long-Lived Encryption Keys

It is important to understand that the KMS keys used for SSE-KMS are long-lived fixed secrets, rather than ephemeral, per-request data. Consequently, your cache should be sized specifically to accommodate the total number of distinct, long-lived encryption keys actively used across all buckets. In a well-tuned environment, cache evictions should be minimal under normal operations.

Be Mindful about Cache Addiction

Relying on the cache to mask an undersized or unreliable KMS is an anti-pattern. While caching optimizes performance, the system must remain resilient to cache misses. Always architect your environment such that even if the cache were flushed or the service restarted, the underlying KMS has sufficient throughput and availability to handle the resulting "thundering herd" of requests without causing a cascading failure.

Select Positive TTLs as Long as Security Permits

The Time-To-Live (TTL) is the primary lever for balancing latency against security requirements. A longer TTL reduces the frequency of KMS round-trips but increases the window during which a key revocation might not immediately propagate to all RGW nodes. Evaluate your compliance and threat model to set the longest acceptable TTL.

Security Considerations for Cached Secrets

The cache stores decrypted secrets in the Linux kernel keyring. Treat these cached keys with the same security sensitivity as your KMS backend.

Be Mindful about Service Restarts (that cause cold caches)

Service restarts and rolling updates flush the in-memory cache, leading to an immediate "cold start" spike in KMS load. During these periods, expect a surge in KMS requests and corresponding latency for object operations. To mitigate this, ensure that your deployment strategies are staggered and that your KMS capacity planning accounts for the worst-case scenario: the sudden re-authentication of all active buckets following a global restart.

Observability Strategy

Effective monitoring of the RGW SSE-KMS cache relies on correlating cache performance with your underlying KMS backend and system-level resources.

Key Performance Indicators

• Cache Efficiency (Hit/Miss Ratio): Monitor the ratio between ceph_kms_cache_hit and ceph_kms_cache_miss. A low hit ratio indicates that the cache is likely undersized.
• Cache Capacity Saturation: Compare ceph_kms_cache_size against ceph_kms_cache_capacity. If the cache size consistently hits the maximum capacity, you are likely experiencing excessive evictions.
• KMS Backend Load: Track ceph_rgw_kms_fetch_lat_count (total fetch ops) and compare it against rgw_ops_status_total. A spike in fetch operations relative to total RGW operations signals a cache-bypass or cache exhaustion event.
• Latency Impact: Calculate average fetch latency using ceph_rgw_kms_fetch_lat_sum / ceph_rgw_kms_fetch_lat_count. Use this to baseline the "cost" of a cache miss.

Error Observability

• Secret Store Errors (ceph_rgw_kms_error_secret_store): These indicate that the cache itself has encountered a failure, effectively disabling it and forcing all requests to the KMS backend. This bypass significantly increases load on the KMS, and immediate investigation is required to restore caching functionality.
• Transient Errors (ceph_rgw_kms_error_transient): (e.g., KMS system overload, network timeouts). These errors are also cached with a defined TTL to prevent overwhelming the backend during ongoing outages. Monitor their frequency, but they typically resolve automatically once the backend recovers.
• Permanent Errors (ceph_rgw_kms_error_permanent): (e.g., key unavailable, access denied, invalid configuration). These require immediate operator intervention. Unlike transient errors, these will persist and potentially block client requests until the configuration or key state is corrected.

Configuration and Tuning

The following parameters control cache behavior. Adjust these values based on your unique key-per-bucket density and system throughput capacity.

Parameter	Recommended Tuning for Performance	Note
rgw_crypt_s3_kms_cache_enabled	true	Required to enable the in-memory cache.
rgw_crypt_s3_kms_cache_max_size	Matches active key working set size	Each key consumes 32 bytes + overhead.
rgw_crypt_s3_kms_cache_positive_ttl	> 3600 (seconds)	As high as your security policy allows.
rgw_crypt_s3_kms_cache_negative_ttl	60 (seconds)	Caches non-existent key results to reduce KMS backend lookups.
rgw_crypt_s3_kms_cache_transient_error_ttl	10 (seconds)	Controls time before retrying after KMS outages.
/proc/…/maxkeys (System)	> rgw_crypt_s3_kms_cache_max_size	Must accommodate cached secrets or the cache will fail.
/proc/…/maxbytes (System)	> rgw_crypt_s3_kms_cache_max_size * 256	Limits total payload bytes for all keys. Heuristic accounts for payload + structural overhead. Reference: man 7 keyrings.

Verification

To verify that the RGW SSE-KMS cache is functioning correctly, confirm the configuration and monitor the cache activity through Prometheus.

Configuration Check

Confirm that rgw_crypt_s3_kms_cache_enabled is set to true in your RGW configuration.

Active Usage & Capacity Check

Monitor the following Prometheus metrics to confirm the cache is active, populated, and serving requests:

• Active Cache Utilization: Observe the ceph_kms_cache_size metric. A value greater than 0 indicates the cache is populated and actively managing keys.
• Operational Throughput: Monitor the ceph_kms_cache_hit counter. A monotonically increasing value confirms the cache is successfully intercepting requests and reducing backend KMS load.
• Capacity Saturation: Compare ceph_kms_cache_size against ceph_kms_cache_capacity to ensure the cache has sufficient room for your working set. If size equals capacity, you must increase rgw_crypt_s3_kms_cache_max_size.

Common Failure Modes

• "Thundering Herd" on Restart: Service restarts flush the in-memory cache. If all RGW nodes restart simultaneously, they will immediately overwhelm the KMS backend. Mitigation: Use staggered rolling updates.
• Kernel Keyring Exhaustion: The cache stores secrets in the Linux kernel keyring. If the system-wide quota is exceeded, the cache will fail to store new keys. Mitigation: Ensure /proc limits are tuned appropriately.
• Stale Key Revocation: A key revoked in the external KMS backend may remain available for decryption in RGW until the TTL expires. Mitigation: Lower your positive TTL if your threat model requires immediate key revocation enforcement. Restart RGW, but be mindful of the cold start problem documented above.
• Secret Store Error Propagation: If the cache encounters a permanent failure in the secret store, it may disable the cache entirely. Mitigation: Alerting on ceph_rgw_kms_error_secret_store is critical.