Open-telemetry Jaeger Tracing in Rook

Ceph Version Support

Jaeger distributed tracing support was officially introduced in Ceph Pacific (version 16.x) and continues to be supported in your current Ceph Squid (v19.2.3) deployment. I used Squid(v19.2.0+) for this article. The tracing functionality is available for both RGW and OSD components in Rook-managed Kubernetes deployments.

Rook Kubernetes Compatibility

Opentelemetry(Jaeger tracing) is supported in Ceph deployments using method of sidecar injection for now for Rook.

Here’s a step by step guide on how to configure Opentelemetry(Jaegertracing) using sidecar injection for Rook-Ceph.

In an existing Rook Ceph Kubernetes cluster we will be deploying cert-manager, for the validation of Jaeger-operator.

$ kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.11.0/cert-manager.yaml

Deploy Jaeger operator:

$ kubectl create namespace observability
$  kubectl apply -f https://github.com/jaegertracing/jaeger-operator/releases/download/v1.42.0/jaeger-operator.yaml -n observability

• For Production cluster please refer to: https://www.jaegertracing.io/docs/1.21/operator/

• We deploy simplest (agent, collector, query, ingestor, Jaeger UI) in a single pod, using in-memory storage by default

$ cat << EOF | kubectl apply -f -
apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
  name: simplest
  namespace: observability
EOF

Expose Query API:

$ cat << EOF | kubectl apply -f -
apiVersion: v1
kind: Service
metadata:
  name: simplest-query-external
  namespace: observability
  labels:
    app: jaeger
    app.kubernetes.io/component: service-query
    app.kubernetes.io/instance: simplest
    app.kubernetes.io/managed-by: jaeger-operator
    app.kubernetes.io/name: simplest-query
    app.kubernetes.io/part-of: jaeger
spec:
  ports:
  - name: http-query
    port: 16686
    protocol: TCP
    targetPort: 16686
  selector:
    app.kubernetes.io/component: all-in-one
    app.kubernetes.io/instance: simplest
    app.kubernetes.io/managed-by: jaeger-operator
    app.kubernetes.io/name: simplest
    app.kubernetes.io/part-of: jaeger
    app: jaeger
  sessionAffinity: None
  type: NodePort
EOF

add jaeger configuration to cluster-test ConfigMap under [global] settings

jaeger_tracing_enable = true
jaeger_agent_port = 6831

optionally you can add debug_trace to see any debug logs based on tracing for the cluster.

  cephConfig:
    global:
      jaeger_tracing_enable: true
      jaeger_agent_port: "6831"
      debug_trace: "20"
      

Jaeger requires a sidecar agent for the instrumented code to collect traces from, for that we need to inject Jaeger as a sidecar on OSDs and RGW

add annotations to the cluster. so that jaeger will inject an agent side-car to OSD pods:

spec:
  annotations:
    osd:
      sidecar.jaegertracing.io/inject: "true"

$ kubectl create -f cluster-test.yaml

add annotations to the cluster. so that jaeger will inject an agent side-car to RGW object store object-test.yaml:

gateway:
  annotations:
    sidecar.jaegertracing.io/inject: "true"

$ kubectl create -f object-test.yaml

• fetch the URL that allow access to the RGW service from the host running the minikube VM:

$ export AWS_URL=$(minikube service --url rook-ceph-rgw-my-store-external -n rook-ceph)

• user credentials and bucket name:

$ export AWS_ACCESS_KEY_ID=$(kubectl -n default get secret ceph-delete-bucket -o jsonpath='{.data.AWS_ACCESS_KEY_ID}' | base64 --decode)
$ export AWS_SECRET_ACCESS_KEY=$(kubectl -n default get secret ceph-delete-bucket -o jsonpath='{.data.AWS_SECRET_ACCESS_KEY}' | base64 --decode)
$ export BUCKET_NAME=$(kubectl get objectbucketclaim ceph-delete-bucket -o jsonpath='{.spec.bucketName}')

• now use them to upload an object:

$ echo "hello world" > hello.txt
$ aws --endpoint-url "$AWS_URL" s3 cp hello.txt s3://"$BUCKET_NAME"

• fetch the URL that allow access to the jaeger query service from the host running the minikube VM:

$ export JAEGER_URL=$(minikube service --url simplest-query-external -n observability)

• query traces:

$ curl "$JAEGER_URL/api/traces?service=rgw&limit=20&lookback=1h" | jq