Skip to main content

Operations

Resource requirements​

Proxy​

  • stateless - horizontally scalable
  • low-memory
  • low-CPU
  • high-network

Worker​

  • stateless - horizontally scalable
  • high-memory
  • low-CPU
  • high-network

Worker instance network and memory consumption can be limited by the following worker config parameters:

concurrency: 10 # max number of simultaneously processed tasks (copied abjects)

and memory rate-limit:

rclone:
  memoryLimit:
    enabled: true
    limit: 300M # Max RAM memory for file copy. Possible values: 50M <= mem <= (total app RAM - 50M). 
                # More memory -> faster s3 replication. # For example: if worker container resources.memory.limit = 500M, then mem = 450M
  localFileLimit:
    enabled: false
    limit: 3 # max number of objects copied in parallel by one instance
  globalFileLimit:
    enabled: true
    limit: 3 # max number of objects copied in parallel across all instances - useful if worker replicas share the same network resource

Strict worker limits leads to delay in replications and increasing number of objects in the work queue. Big amount of unprocessed tasks increases Redis RAM consumption. Try to balance worker limits according to your workload and desired replication delay.

Work Queue (Redis)​

  • scale: Redis-Cluster, Redis-Sentinel.
  • persistence: Redis-AOF, Redis-Snapshot (RDB).
  • Fault-tolerance - not critical. In case of data loss, bucket replication can be restarted.
  • memory: 1M migrated obj ~ 105MB, 1M tasks in the queue ~ 700MB
  • low cpu: 100-1000 rps

Monitoring​

All components expose prometheus metrics on :9090/metrics endpoint. The metrics can be used in Chorus Grafana dashboard to monitor workload, task rate, error rate, s3 objects and bytes uploaded/downloaded.

TODO: attach Grafana dashboard json.

TODO: describe metrics

Tracing​

All components supports open-tracing protocol. Currently only tested with Jaeger. To enable tracing edit config:

trace:
  enabled: false
  endpoint: # url to Jaeger or other open trace provider

Logging​

All components supports structured logging. To enable json logs edit config:

log:
  json: false # false for dev console logger, true - json log for prod to export to Grafana&Loki.
  level: info