Auto-Instrumentation with OpenTelemetry and Jaeger: Solving with Simple Annotation

Microservices have multiple benefits for development organizations. They make applications easily scalable, highly resilient, and easier to maintain, and keeping track of hundreds of microservices can be challenging.

As many organizations move towards microservices architecture to boost developer productivity, they are also facing new operational challenges. "We can't fix something which we can't observe," so it is important to have a strong monitoring and observability solution in place to track the performance of microservices applications. That’s where observability comes into the picture. Observability platforms help businesses understand system behavior and predict outages or problems before they occur. Observability tools are used by businesses to take preventive action and stop system issues. In this post, we will see how to use OpenTelemetry and Jaeger to trace the data of applications.

OpenTelemetry 

OpenTelemetry (OTel) is a set of open-source libraries and tools that enable the collection, processing, and export of telemetry data from cloud-native software applications. It provides a vendor-agnostic approach to instrumenting applications and collecting telemetry data.

Source: https://opentelemetry.io/docs/

Instrumenting

Instrumenting is the process of adding monitoring and telemetry code to software applications. This code collects data about the application's performance, behavior, and usage. This data can then be used to gain insights into how the application is performing, identify issues and errors, and optimize its performance.

OpenTelemetry provides both manual and automatic instrumentation capabilities. Manual instrumentation allows developers to add instrumentation code to their applications themselves. Automatic instrumentation allows developers to let OpenTelemetry automatically add instrumentation code to their applications.

OpenTelemetry can be used to monitor and trace applications in a distributed environment. Distributed tracing allows developers to see how requests flow through a distributed system, which can be helpful for identifying performance bottlenecks and errors.

1. Manual Instrumentation Manual instrumentation involves developers explicitly creating and recording spans using the Opentelemetry SDK API. It allows developers to trace specific parts of their application code and provides greater control over the instrumentation process.

2. Automatic Instrumentation Automatic instrumentation utilizes pre-built integrations to instrument an application automatically, eliminating the need for developers to write any code. Opentelemetry provides automatic instrumentation for popular frameworks and libraries like Spring, Django, and Node.js.

Here, we will find out how you can use automatic instrumentation in your application.

Prerequisites:

1. You must have the Kubernetes cluster up and running

2. You must have Kubectl access to your Kubernetes cluster

3. Make sure to install cert-manager

Let's assume that you have all the prerequisites and now you are ready for the installation.

Jaeger

Data trace workflow with OpenTelemetry and Jaeger

Jaeger is an open-source distributed tracing system that provides end-to-end tracing, allowing developers to track the flow of requests through a complex system of microservices. To start visualizing data, we need to set up Jaeger. 

Jaeger installation contains two steps:

1. Jaeger Operator Installation

2. Deploying the AllInOne image

 

Jaeger Operator Installation

The Jaeger Operator is an open-source Kubernetes operator that simplifies the deployment and management of Jaeger. Firstly, we will create an observability namespace and deploy the Jaeger Operator in that namespace. By default, the operator will monitor all namespaces. 

$ kubectl create namespace observability  
  
$ kubectl create -f https://github.com/jaegertracing/jaeger-operator/releases/download/v1.44.0/jaeger-operator.yaml -n observability  
  

Once it is deployed, you can view it by running the following command.

$ kubectl get deployment jaeger-operator -n observability  
  
NAME                      READY   UP-TO-DATE   AVAILABLE   AGE  
jaeger-operator            1/1        1           1        27m  

$ kubectl get svc -n observability  
  
NAME                         TYPE     CLUSTER-IP   EXTERNAL-IP PORT(S)
jaeger-operator-metrics    ClusterIP    10.64.15.107   <none>  8443/TCP
jaeger-operator-webhook-service ClusterIP 10.64.10.106 <none>  443/TCP

Deploying the AllInOne image

The Jaeger All-In-One (AIO) image is a pre-built Docker image that includes all the necessary components of Jaeger, including the Jaeger collector, Jaeger query, Jaeger agent, Jaeger UI, as well as the necessary dependencies, all packaged in a single image and this is the simplest possible way to create a Jaeger instance is by creating a YAML file.

Create a file named “jeager-instance.yaml” and add the following YAML content to the file.

apiVersion: jaegertracing.io/v1 
kind: Jaeger 
metadata: 
   name: jaeger 
   namespace: observability  
spec: 
  ingress: 
    enabled: true 

Deploy Jeager all-in-one image by running the following:

$ Kubectl apply -f jeager-instance.yaml

Verify the deployment by running the command:

$ kubectl get jaegers -n observability 
NAME        STATUS      VERSION     STRATEGY   STORAGE   AGE 
jaeger     Running      1.44.0      allinone    memory   43m

When the Jeager all-in-one image is deployed, you can access the Jaeger UI at a specific ingress URL, allowing you to explore the traces generated by your instrumentation.

OpenTelemetry Installation

For the installation, we need to follow below steps:

1. OpenTelemetry Operator

2. Injecting Auto-instrumentation

3. Add annotations to the deployment manifest

OpenTelemetry Operator 

The OpenTelemetry Operator is designed to deliver auto-instrumentation to export traces and metrics in new and existing applications without any code changes (Source). Now, we would introduce OpenTelemetry Operator which makes it very easy to set up OpenTelemetry collector and instrument workloads deployed on Kubernetes.

To install the OpenTelemetry Operator in an existing Kubernetes Cluster, make sure you have the cert-manager installed and running.

$ kubectl apply -f https://github.com/open-telemetry/opentelemetry-operator/releases/latest/download/opentelemetry-operator.yaml

The above command creates a new namespace called opentelemetry-operator-system and deploys the OpenTelemetry Operator within that namespace.

You can verify the deployment of the OpenTelemetry operator by running the following command:

$ kubectl get all -n opentelemetry-operator-system

NAME                                                          READY   STATUS   RESTARTS   AGE 
pod/opentelemetry-operator-controller-manager-b9f568969-lpzc5   2/2     Running     0       3h8m

NAME                                                              TYPE      CLUSTER-IP    EXTERNAL-IP   PORT(S) AGE 
service/opentelemetry-operator-controller-manager-metrics-service  ClusterIP     10.104.3.204    <none>         8443/TCP    3h8m 
service/opentelemetry-operator-webhook-service                     ClusterIP   10.104.15.157     <none>        443/TCP    3h8m

NAME                                                       READY   UP-TO-DATE   AVAILABLE   AGE 
deployment.apps/opentelemetry-operator-controller-manager          1/1       1             1       3h8m 

NAME                                                                 DESIRED   CURRENT   READY   AGE 
replicaset.apps/opentelemetry-operator-controller-manager-b9f568969      1         1        1     3h8m 

Once the opentelemetry-operator deployment is ready, define a custom Opentelemetry configuration by creating a new file “otelcol.yaml” and adding the desired configuration. Here is the following configuration file that enables the collection and export of traces to a backend. As you know that we are using Jaeger as the backend.

apiVersion: opentelemetry.io/v1alpha1 
kind: OpenTelemetryCollector 
metadata: 
  name: otelcol 
spec: 
  config: | 
    receivers: 
      otlp: 
        protocols: 
          grpc: 
          http: 
    processors: 
 
    exporters: 
      logging: 
      jaeger: 
        endpoint: "jaeger-collector.observability.svc.cluster.local:14250" 
        tls: 
          insecure: true 
 
    service: 
      pipelines: 
        traces: 
          receivers: [otlp] 
          processors: [] 
          exporters: [logging, jaeger] 

Run the following command to confirm the deployment of the collector:

$ kubectl get deploy otelcol-collector 
 
NAME                         READY     UP-TO-DATE   AVAILABLE    AGE 
otelcol-collector             1/1        1              1        178m 

Injecting Auto-Instrumentation

The operator can inject and configure OpenTelemetry auto-instrumentation libraries for various programming languages and frameworks. Configure an Instrumentation resource with the SDK and instrumentation configurations in order to use auto-instrumentation.

The following YAML will create a basic Auto-instrumentation resource:

apiVersion: opentelemetry.io/v1alpha1 
kind: Instrumentation 
metadata: 
  name: demo-instrumentation 
spec: 
  exporter: 
    endpoint: http://otelcol-collector:4317 
  propagators: 
    - tracecontext 
    - baggage 
  sampler: 
    type: parentbased_traceidratio 
    argument: "1"

Verify the instrumentation by running the below-mentioned command:

$ kubectl get instrumentation

NAME                            AGE        ENDPOINT          SAMPLER                SAMPLER ARG 
otel-operator-instrumentation   32s  http://otel-collector:4317   parentbased_traceidratio     1 

Add Annotations To The Deployment Manifest

The final step is to add an annotation to a pod to enable injection. The annotation can be added to a namespace, so that all pods within that namespace will get instrumentation. Alternatively, the annotation can be added to individual PodSpec objects, available as part of a Deployment.

As we are going to demo a Nodejs application, the annotation which we will have to use here is:

instrumentation.opentelemetry.io/inject-nodejs: "true"

Sample App 

Here, we are utilizing an open-source two-tier Node.js application. We have added annotations to the Node.js application to enable injection, as shown in the Kubernetes manifest below.

apiVersion: v1 
kind: Service 
metadata: 
  name: knote 
spec: 
  selector: 
    app: knote 
  ports: 
    - port: 80 
      targetPort: 3000 
  type: LoadBalancer 
--- 
apiVersion: apps/v1 
kind: Deployment 
metadata: 
  name: knote 
spec: 
  replicas: 1 
  selector: 
    matchLabels: 
      app: knote 
  template: 
    metadata: 
      labels: 
        app: knote 
      annotations: 
        instrumentation.opentelemetry.io/inject-nodejs: "true" 
        instrumentation.opentelemetry.io/container-names: "knote" 
    spec: 
      containers: 
        - name: knote 
          image: learnk8s/knote-js:1.0.0 
          ports: 
            - containerPort: 3000 
          env: 
            - name: MONGO_URL 
              value: mongodb://mongo:27017/dev 
          imagePullPolicy: Always 

You can verify the deployment through the following command:

$ kubectl get pod  
NAME                            READY      STATUS    RESTARTS   AGE 
knote-58fdfcc89d-b9tnn           1/1        Running      0       18m 
mongo-586867d494-wgf8d           1/1        Running      0       18m 

To generate traces, you can navigate through all pages of the website and access the Jaeger UI. By selecting the specified service name and operation, you will be able to view the traces generated by those requests.

Below, you will find all the traces captured within a specific time window and their associated spans.

Distributed tracing is invaluable in modern software systems and application architectures. It aids in understanding and troubleshooting complex, distributed systems by tracking the flow of requests across various services and components.

In this blog post, we have explored how to instrument our application without making changes to the code using the OpenTelemetry Auto-Instrumentation method. This method allows you to monitor transactions, conduct root cause analysis, optimize performance and latency, and visualize service dependencies.

---

Are you looking to improve your organization's DevOps capabilities?

At Vikasietum, we have a team of experienced DevOps engineers who can help you improve your organization's DevOps capabilities. Contact us today to learn more about our DevSecOps services!