Skip to main content

Operations

Resource requirements​

Proxy​

  • stateless - horizontally scalable
  • low-memory
  • low-CPU
  • high-network

Worker​

  • stateless - horizontally scalable
  • high-memory
  • low-CPU
  • high-network

Set the following worker configuration parameters to limit the worker-instance network and memory consumption:

concurrency: 10 # max number of simultaneously processed tasks (copied abjects)

Set the following parameters to set the memory rate-limit:

rclone:
  memoryLimit:
    enabled: true
    limit: 300M # Max RAM memory for file copy. Possible values: 50M <= mem <= (total app RAM - 50M). 
                # More memory -> faster s3 replication. # For example: if worker container resources.memory.limit = 500M, then mem = 450M
  localFileLimit:
    enabled: false
    limit: 3 # max number of objects copied in parallel by one instance
  globalFileLimit:
    enabled: true
    limit: 3 # max number of objects copied in parallel across all instances - useful if worker replicas share the same network resource

Strict worker limits can cause delay in replications and increasing number of objects in the work queue. A large amount of unprocessed tasks increases Redis RAM consumption. Balance worker limits according to your workload and desired replication delay.

Work Queue (Redis)​

  • scale: Redis-Cluster, Redis-Sentinel.
  • persistence: Redis-AOF, Redis-Snapshot (RDB).
  • Fault-tolerance - not critical. In case of data loss, bucket replication can be restarted.
  • memory: 1M migrated obj ~ 105MB, 1M tasks in the queue ~ 700MB
  • low cpu: 100-1000 rps

Monitoring​

All components expose prometheus metrics on the :9090/metrics endpoint. The metrics can be used in the Chorus Grafana dashboard to monitor workload, task rate, error rate, S3 objects, and bytes uploaded and downloaded.

TODO: attach Grafana dashboard json.

TODO: describe metrics

Tracing​

All components support open-tracing protocol. Currently this has been tested only with Jaeger.

To enable tracing, modify the configuration file so that it includes this::

trace:
  enabled: false
  endpoint: # url to Jaeger or other open trace provider

Logging​

All components support structured logging. To enable JSON logs, modify the configuration file so that it includes the following:

log:
  json: false # false for dev console logger, true - json log for prod to export to Grafana&Loki.
  level: info