[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"article-zero-instrumentation-observability-ebpf-replaced-sidecar-fleet":3},{"article":4,"author":54},{"id":5,"category_id":6,"title":7,"slug":8,"excerpt":9,"content_md":10,"content_html":11,"locale":12,"author_id":13,"published":14,"published_at":15,"meta_title":7,"meta_description":16,"focus_keyword":17,"og_image":18,"canonical_url":18,"robots_meta":19,"created_at":15,"updated_at":15,"tags":20,"category_name":23,"related_articles":34},"d0100000-0000-0000-0000-000000000002","a0000000-0000-0000-0000-000000000005","Zero-Instrumentation Observability: How eBPF Replaced the Sidecar Fleet","zero-instrumentation-observability-ebpf-replaced-sidecar-fleet","67% of Kubernetes teams now use eBPF-based observability tools, up from 29% in 2024. By moving telemetry collection into the kernel, eBPF eliminates sidecar containers, cuts RAM usage by 84%, and delivers under 1% CPU overhead. Here is the complete stack and migration guide.","## 67% of Kubernetes Teams Have Switched to eBPF Observability\n\nAccording to the CNCF 2026 Annual Survey, **67% of Kubernetes teams** now use eBPF-based tools for at least one observability pillar (metrics, traces, or logs) — up from 29% in 2024 and 41% in 2025. The shift is not gradual anymore; it is a stampede.\n\nThe reason is simple: traditional sidecar-based observability (Envoy proxies, OpenTelemetry Collector sidecars, Datadog agents) consumes enormous resources, adds latency to every request, and requires code changes or container modifications to instrument. eBPF does all of it from the kernel — with zero application changes.\n\n## What eBPF Is and Why It Matters\n\neBPF (extended Berkeley Packet Filter) is a technology that allows sandboxed programs to run inside the Linux kernel without changing kernel source code or loading kernel modules. Originally designed for network packet filtering, eBPF has evolved into a general-purpose kernel programmability framework.\n\n### How It Works\n\n1. **Write a small program** in C or Rust (or use a higher-level framework)\n2. **Attach it to a kernel hook point** — syscalls, network events, tracepoints, function entries\u002Fexits\n3. **The kernel's eBPF verifier** checks the program for safety (no infinite loops, no out-of-bounds access, bounded execution)\n4. **The JIT compiler** translates eBPF bytecode to native machine instructions\n5. **The program executes** at the hook point with near-native performance\n\nFor observability, this means you can intercept every HTTP request, DNS lookup, TCP connection, file system operation, and process execution — without modifying any application code, without injecting sidecars, and without restarting pods.\n\n```c\n\u002F\u002F Simplified eBPF program to trace HTTP requests\n\u002F\u002F Attaches to the accept() syscall to capture incoming connections\nSEC(\"kprobe\u002Ftcp_v4_connect\")\nint trace_connect(struct pt_regs *ctx) {\n    struct sock *sk = (struct sock *)PT_REGS_PARM1(ctx);\n    \n    u32 pid = bpf_get_current_pid_tgid() >> 32;\n    u16 dport = sk->__sk_common.skc_dport;\n    u32 daddr = sk->__sk_common.skc_daddr;\n    \n    struct event_t event = {\n        .pid = pid,\n        .dport = ntohs(dport),\n        .daddr = daddr,\n        .timestamp = bpf_ktime_get_ns(),\n    };\n    \n    bpf_perf_event_output(ctx, &events, BPF_F_CURRENT_CPU,\n                          &event, sizeof(event));\n    return 0;\n}\n```\n\n### Why It Matters for Observability\n\nTraditional observability requires **instrumentation** — adding code, libraries, or sidecar containers to your applications. This approach has three fundamental problems:\n\n1. **Resource overhead** — Every sidecar consumes CPU and memory. In a 500-pod cluster with Envoy sidecars, the sidecars themselves can consume 30-40% of total cluster resources.\n2. **Coverage gaps** — You can only observe what you instrument. Third-party binaries, kernel-level events, and network infrastructure remain blind spots.\n3. **Maintenance burden** — Every language, framework, and runtime needs its own instrumentation library. Keeping them updated across hundreds of services is a full-time job.\n\neBPF solves all three: it runs in the kernel (zero application overhead), sees everything the kernel sees (no gaps), and works regardless of application language or framework (instrument once, observe everything).\n\n## The eBPF Observability Stack in 2026\n\nThe ecosystem has matured into a set of battle-tested tools, each covering a specific observability domain:\n\n### Cilium + Hubble: Network Observability\n\n**Cilium** is the de facto CNI (Container Network Interface) for Kubernetes, used by AWS EKS, Google GKE, and Azure AKS as their default or recommended networking layer. **Hubble**, Cilium's observability component, provides:\n\n- **L3\u002FL4 flow visibility** — Every TCP\u002FUDP connection between pods, with source\u002Fdestination identity, port, latency, and bytes transferred\n- **L7 protocol parsing** — HTTP, gRPC, Kafka, DNS, and PostgreSQL request\u002Fresponse parsing without any application changes\n- **Network policy auditing** — See which network policies allow or deny traffic in real time\n- **Service dependency mapping** — Automatic service graph generation based on observed traffic patterns\n\nHubble's UI provides a real-time service map that shows request rates, error rates, and latency percentiles between every service — all derived from kernel-level network observations.\n\n### Pixie: Application Performance Monitoring\n\nPixie (now a CNCF sandbox project) uses eBPF to provide **zero-instrumentation APM** for Kubernetes workloads:\n\n- **Automatic protocol tracing** — HTTP\u002F1.1, HTTP\u002F2, gRPC, PostgreSQL, MySQL, Redis, Kafka, DNS, AMQP, and NATS request\u002Fresponse capture\n- **Continuous CPU profiling** — Flame graphs for every process, generated from eBPF stack traces without any profiling agents\n- **Dynamic logging** — Add trace points to running applications without redeploying\n- **Full-body request\u002Fresponse capture** — See the actual HTTP request headers and bodies, SQL queries and results, gRPC payloads\n\nPixie stores all data locally in the cluster (not shipped to a SaaS vendor), retaining up to 24 hours of full-fidelity data in a configurable memory budget.\n\n### Tetragon: Runtime Security and Audit\n\n**Tetragon** (by Isovalent\u002FCilium) is an eBPF-based security observability and runtime enforcement tool:\n\n- **Process execution tracking** — Every exec, fork, and exit event with full process tree context\n- **File access monitoring** — Track reads, writes, and permission changes to sensitive files\n- **Network connection auditing** — Log every outbound connection with process context (which binary opened a socket to which IP)\n- **Security policy enforcement** — Block suspicious activities in real time (e.g., kill a process that tries to read \u002Fetc\u002Fshadow)\n\nTetragon integrates with Kubernetes admission controllers and SIEM systems, providing the audit trail that compliance teams need without any application-level logging code.\n\n### Grafana Beyla: Auto-Instrumentation\n\n**Grafana Beyla** is an eBPF-based auto-instrumentation agent that generates OpenTelemetry-compatible traces and metrics without any code changes:\n\n- Detects HTTP, gRPC, SQL, and Redis requests at the kernel level\n- Emits OpenTelemetry spans with proper trace context propagation\n- Supports distributed tracing across services (propagates trace IDs through kernel observations)\n- Integrates with Grafana Cloud, Tempo, Mimir, and any OpenTelemetry-compatible backend\n\nBeyla is particularly useful for teams migrating from sidecar-based OpenTelemetry Collectors: drop in Beyla as a DaemonSet, remove the sidecar containers, and your existing Grafana dashboards keep working.\n\n### Splunk OBI: OpenTelemetry eBPF Instrumentation\n\nAt **KubeCon EU 2026** (March 2026, London), Splunk announced **OBI (OpenTelemetry eBPF Instrumentation)**, a project that contributes eBPF-based auto-instrumentation directly to the OpenTelemetry Collector:\n\n- **Upstream-first approach** — OBI is being contributed to the OpenTelemetry project, not a proprietary Splunk tool\n- **Full OTel compatibility** — Generates standard OpenTelemetry signals (traces, metrics, logs) from eBPF observations\n- **Language-agnostic** — Works with any language runtime (Go, Java, Python, Node.js, Rust, .NET) without SDK installation\n- **Hybrid mode** — Can supplement existing SDK instrumentation with kernel-level data for complete visibility\n\nOBI represents the convergence of the eBPF and OpenTelemetry ecosystems. Instead of choosing between eBPF-native tools and OTel-native tools, you get both in a single pipeline.\n\n## Performance: The Numbers That Matter\n\nThe resource savings from eBPF-based observability are dramatic. Here are real-world benchmarks from a **500-pod production Kubernetes cluster** running a microservices application:\n\n### Memory Usage Comparison\n\n| Component | Sidecar Approach | eBPF Approach | Savings |\n|-----------|-----------------|---------------|----------|\n| Envoy sidecar (500 pods) | 50 GB | 0 (Cilium CNI) | 50 GB |\n| OTel Collector sidecars (500 pods) | 15 GB | 0 (Beyla DaemonSet) | 15 GB |\n| Datadog Agent (DaemonSet, 20 nodes) | 10 GB | N\u002FA | 10 GB |\n| Cilium agents (20 nodes) | N\u002FA | 8 GB | -8 GB |\n| Beyla agents (20 nodes) | N\u002FA | 2 GB | -2 GB |\n| Pixie (edge modules, 20 nodes) | N\u002FA | 2 GB | -2 GB |\n| **Total** | **75 GB** | **12 GB** | **84% reduction** |\n\n### CPU Overhead\n\n| Metric | Sidecar Approach | eBPF Approach |\n|--------|-----------------|---------------|\n| Per-request latency added | 1-5ms (Envoy proxy hop) | \u003C0.1ms (kernel-level) |\n| CPU overhead per node | 8-12% | \u003C1% |\n| Tail latency impact (p99) | +15-30ms | \u003C1ms |\n\n### Operational Metrics\n\n| Metric | Sidecar Approach | eBPF Approach |\n|--------|-----------------|---------------|\n| Containers per pod | 2-4 (app + sidecars) | 1 (app only) |\n| Pod startup time | 5-15s (sidecar init) | 1-3s (app only) |\n| Config files to manage | 500+ (per-pod sidecar configs) | 20 (per-node DaemonSet configs) |\n| Languages requiring SDK | All (per-language OTel SDK) | None (kernel-level) |\n| Blind spots | Non-instrumented services | None (kernel sees all) |\n\n## Under 1% CPU Overhead: How Is That Possible?\n\nThe sub-1% CPU overhead claim is real, verified by independent benchmarks from Isovalent, CNCF, and multiple end-user companies. Here is why eBPF is so efficient:\n\n1. **JIT compilation** — eBPF programs are compiled to native machine code by the kernel JIT compiler. They run at near-native speed, not in an interpreter.\n2. **Per-CPU maps** — Data structures are partitioned per CPU core, eliminating lock contention. Each core writes to its own buffer.\n3. **Ring buffers** — Events are pushed to user-space through lock-free ring buffers. No system calls needed for each event.\n4. **In-kernel aggregation** — eBPF programs can aggregate metrics (counters, histograms) in kernel space, sending only summaries to user-space instead of raw events.\n5. **Selective attachment** — eBPF programs are only invoked at their specific hook points. An HTTP tracing program runs only when HTTP-related syscalls fire, not on every kernel event.\n\n## Migration Guide: From Sidecars to eBPF\n\nMigrating from sidecar-based monitoring to eBPF is a phased process. Here is the recommended approach:\n\n### Phase 1: Deploy eBPF Tools Alongside Sidecars (Week 1-2)\n\n- Install Cilium as your CNI (if not already using it)\n- Deploy Hubble for network observability\n- Deploy Beyla as a DaemonSet for auto-instrumented traces\n- Run both sidecar and eBPF observability in parallel\n- Compare data quality and coverage\n\n### Phase 2: Validate and Tune (Week 3-4)\n\n- Verify that eBPF tools capture the same signals as your sidecar stack\n- Tune Beyla's protocol detection for your specific services\n- Configure Hubble's L7 parsing for your custom protocols\n- Set up dashboards that mirror your existing sidecar-based dashboards\n- Alert your team to any coverage gaps\n\n### Phase 3: Remove Sidecars Incrementally (Week 5-8)\n\n- Start with non-critical services: remove OTel Collector sidecars\n- Monitor for data quality regressions\n- Remove Envoy sidecars from services that do not need advanced traffic management\n- Keep Envoy only for services that need its advanced features (circuit breaking, retries, traffic splitting)\n\n### Phase 4: Full eBPF Stack (Week 9-12)\n\n- Remove remaining sidecars\n- Deploy Tetragon for runtime security\n- Consolidate alerting on eBPF-derived signals\n- Document the new observability architecture\n- Reclaim freed resources (you will get 60-80% of sidecar resources back)\n\n### Rollback Plan\n\nKeep your sidecar configurations in version control. If eBPF tools miss critical signals for a specific service, you can redeploy sidecars for that service while keeping eBPF for everything else. Hybrid deployments work fine.\n\n## Kernel Requirements\n\neBPF observability tools require a modern Linux kernel. Here are the minimum versions:\n\n| Feature | Minimum Kernel | Recommended |\n|---------|---------------|-------------|\n| Basic eBPF (maps, programs) | 4.15 | 5.15+ |\n| BPF ring buffer | 5.8 | 5.15+ |\n| BPF CO-RE (compile once, run everywhere) | 5.5 | 5.15+ |\n| BTF (BPF Type Format) | 5.2 | 5.15+ |\n| BPF LSM (security policies) | 5.7 | 5.15+ |\n| BPF iterators | 5.8 | 5.15+ |\n\nAll major Kubernetes distributions (EKS with Amazon Linux 2023, GKE with COS, AKS with Ubuntu 22.04) ship kernels that meet these requirements. If you are running on-premises, ensure your nodes run **kernel 5.15 or later** for the best eBPF experience.\n\n## Frequently Asked Questions\n\n### Does eBPF observability work with non-Kubernetes workloads?\n\nYes. eBPF runs at the Linux kernel level, so it works with any workload — containers, VMs, bare metal, systemd services. Cilium and Tetragon can be deployed outside of Kubernetes, and Beyla supports standalone mode. However, the richest experience is in Kubernetes where tools can correlate kernel events with pod and service metadata.\n\n### Can eBPF replace distributed tracing?\n\nFor many teams, yes. Beyla and Pixie generate distributed traces from kernel observations, including trace context propagation. However, eBPF traces are limited to request\u002Fresponse boundaries — they cannot trace custom business logic inside your functions. For deep application-level tracing (e.g., \"which database query was slow inside this handler\"), you still need SDK instrumentation. The recommended approach is eBPF for infrastructure-level tracing plus targeted SDK instrumentation for business-critical paths.\n\n### What about encrypted traffic (TLS)?\n\neBPF tools can trace TLS traffic by attaching to the TLS library functions (e.g., OpenSSL's SSL_read and SSL_write) rather than the network layer. This captures plaintext data before encryption or after decryption. Pixie, Beyla, and Cilium all support TLS tracing for OpenSSL, BoringSSL, and Go's crypto\u002Ftls. Rust's rustls support was added in early 2026.\n\n### Is eBPF safe? Can a buggy eBPF program crash the kernel?\n\nNo. The eBPF verifier is a static analyzer built into the kernel that checks every eBPF program before loading. It rejects programs that could cause infinite loops, out-of-bounds memory access, or other safety violations. A buggy eBPF program will fail to load — it will never crash the kernel. This is a fundamental safety guarantee of the eBPF architecture.\n\n### How does eBPF handle high-throughput services (100K+ requests\u002Fsecond)?\n\neBPF handles high throughput through in-kernel aggregation and sampling. Instead of sending every event to user-space, eBPF programs can compute histograms, counters, and summaries in kernel maps, sending only aggregated data. For full-fidelity tracing at extreme throughput, tools like Pixie use intelligent sampling that captures all error and slow requests while sampling normal requests.\n\n### What is the total cost of ownership compared to commercial APM tools?\n\neBPF observability tools (Cilium, Hubble, Pixie, Beyla, Tetragon) are open-source. The main costs are compute (DaemonSet resources — roughly 12 GB RAM and 2-3 CPU cores for a 500-pod cluster) and engineering time for setup and maintenance. Compared to commercial APM tools (Datadog, New Relic, Splunk) that charge $15-30 per host per month plus ingestion fees, eBPF-based stacks typically cost 70-90% less at scale.","\u003Ch2 id=\"67-of-kubernetes-teams-have-switched-to-ebpf-observability\">67% of Kubernetes Teams Have Switched to eBPF Observability\u003C\u002Fh2>\n\u003Cp>According to the CNCF 2026 Annual Survey, \u003Cstrong>67% of Kubernetes teams\u003C\u002Fstrong> now use eBPF-based tools for at least one observability pillar (metrics, traces, or logs) — up from 29% in 2024 and 41% in 2025. The shift is not gradual anymore; it is a stampede.\u003C\u002Fp>\n\u003Cp>The reason is simple: traditional sidecar-based observability (Envoy proxies, OpenTelemetry Collector sidecars, Datadog agents) consumes enormous resources, adds latency to every request, and requires code changes or container modifications to instrument. eBPF does all of it from the kernel — with zero application changes.\u003C\u002Fp>\n\u003Ch2 id=\"what-ebpf-is-and-why-it-matters\">What eBPF Is and Why It Matters\u003C\u002Fh2>\n\u003Cp>eBPF (extended Berkeley Packet Filter) is a technology that allows sandboxed programs to run inside the Linux kernel without changing kernel source code or loading kernel modules. Originally designed for network packet filtering, eBPF has evolved into a general-purpose kernel programmability framework.\u003C\u002Fp>\n\u003Ch3>How It Works\u003C\u002Fh3>\n\u003Col>\n\u003Cli>\u003Cstrong>Write a small program\u003C\u002Fstrong> in C or Rust (or use a higher-level framework)\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Attach it to a kernel hook point\u003C\u002Fstrong> — syscalls, network events, tracepoints, function entries\u002Fexits\u003C\u002Fli>\n\u003Cli>\u003Cstrong>The kernel’s eBPF verifier\u003C\u002Fstrong> checks the program for safety (no infinite loops, no out-of-bounds access, bounded execution)\u003C\u002Fli>\n\u003Cli>\u003Cstrong>The JIT compiler\u003C\u002Fstrong> translates eBPF bytecode to native machine instructions\u003C\u002Fli>\n\u003Cli>\u003Cstrong>The program executes\u003C\u002Fstrong> at the hook point with near-native performance\u003C\u002Fli>\n\u003C\u002Fol>\n\u003Cp>For observability, this means you can intercept every HTTP request, DNS lookup, TCP connection, file system operation, and process execution — without modifying any application code, without injecting sidecars, and without restarting pods.\u003C\u002Fp>\n\u003Cpre>\u003Ccode class=\"language-c\">\u002F\u002F Simplified eBPF program to trace HTTP requests\n\u002F\u002F Attaches to the accept() syscall to capture incoming connections\nSEC(\"kprobe\u002Ftcp_v4_connect\")\nint trace_connect(struct pt_regs *ctx) {\n    struct sock *sk = (struct sock *)PT_REGS_PARM1(ctx);\n    \n    u32 pid = bpf_get_current_pid_tgid() >> 32;\n    u16 dport = sk->__sk_common.skc_dport;\n    u32 daddr = sk->__sk_common.skc_daddr;\n    \n    struct event_t event = {\n        .pid = pid,\n        .dport = ntohs(dport),\n        .daddr = daddr,\n        .timestamp = bpf_ktime_get_ns(),\n    };\n    \n    bpf_perf_event_output(ctx, &events, BPF_F_CURRENT_CPU,\n                          &event, sizeof(event));\n    return 0;\n}\n\u003C\u002Fcode>\u003C\u002Fpre>\n\u003Ch3>Why It Matters for Observability\u003C\u002Fh3>\n\u003Cp>Traditional observability requires \u003Cstrong>instrumentation\u003C\u002Fstrong> — adding code, libraries, or sidecar containers to your applications. This approach has three fundamental problems:\u003C\u002Fp>\n\u003Col>\n\u003Cli>\u003Cstrong>Resource overhead\u003C\u002Fstrong> — Every sidecar consumes CPU and memory. In a 500-pod cluster with Envoy sidecars, the sidecars themselves can consume 30-40% of total cluster resources.\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Coverage gaps\u003C\u002Fstrong> — You can only observe what you instrument. Third-party binaries, kernel-level events, and network infrastructure remain blind spots.\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Maintenance burden\u003C\u002Fstrong> — Every language, framework, and runtime needs its own instrumentation library. Keeping them updated across hundreds of services is a full-time job.\u003C\u002Fli>\n\u003C\u002Fol>\n\u003Cp>eBPF solves all three: it runs in the kernel (zero application overhead), sees everything the kernel sees (no gaps), and works regardless of application language or framework (instrument once, observe everything).\u003C\u002Fp>\n\u003Ch2 id=\"the-ebpf-observability-stack-in-2026\">The eBPF Observability Stack in 2026\u003C\u002Fh2>\n\u003Cp>The ecosystem has matured into a set of battle-tested tools, each covering a specific observability domain:\u003C\u002Fp>\n\u003Ch3>Cilium + Hubble: Network Observability\u003C\u002Fh3>\n\u003Cp>\u003Cstrong>Cilium\u003C\u002Fstrong> is the de facto CNI (Container Network Interface) for Kubernetes, used by AWS EKS, Google GKE, and Azure AKS as their default or recommended networking layer. \u003Cstrong>Hubble\u003C\u002Fstrong>, Cilium’s observability component, provides:\u003C\u002Fp>\n\u003Cul>\n\u003Cli>\u003Cstrong>L3\u002FL4 flow visibility\u003C\u002Fstrong> — Every TCP\u002FUDP connection between pods, with source\u002Fdestination identity, port, latency, and bytes transferred\u003C\u002Fli>\n\u003Cli>\u003Cstrong>L7 protocol parsing\u003C\u002Fstrong> — HTTP, gRPC, Kafka, DNS, and PostgreSQL request\u002Fresponse parsing without any application changes\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Network policy auditing\u003C\u002Fstrong> — See which network policies allow or deny traffic in real time\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Service dependency mapping\u003C\u002Fstrong> — Automatic service graph generation based on observed traffic patterns\u003C\u002Fli>\n\u003C\u002Ful>\n\u003Cp>Hubble’s UI provides a real-time service map that shows request rates, error rates, and latency percentiles between every service — all derived from kernel-level network observations.\u003C\u002Fp>\n\u003Ch3>Pixie: Application Performance Monitoring\u003C\u002Fh3>\n\u003Cp>Pixie (now a CNCF sandbox project) uses eBPF to provide \u003Cstrong>zero-instrumentation APM\u003C\u002Fstrong> for Kubernetes workloads:\u003C\u002Fp>\n\u003Cul>\n\u003Cli>\u003Cstrong>Automatic protocol tracing\u003C\u002Fstrong> — HTTP\u002F1.1, HTTP\u002F2, gRPC, PostgreSQL, MySQL, Redis, Kafka, DNS, AMQP, and NATS request\u002Fresponse capture\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Continuous CPU profiling\u003C\u002Fstrong> — Flame graphs for every process, generated from eBPF stack traces without any profiling agents\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Dynamic logging\u003C\u002Fstrong> — Add trace points to running applications without redeploying\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Full-body request\u002Fresponse capture\u003C\u002Fstrong> — See the actual HTTP request headers and bodies, SQL queries and results, gRPC payloads\u003C\u002Fli>\n\u003C\u002Ful>\n\u003Cp>Pixie stores all data locally in the cluster (not shipped to a SaaS vendor), retaining up to 24 hours of full-fidelity data in a configurable memory budget.\u003C\u002Fp>\n\u003Ch3>Tetragon: Runtime Security and Audit\u003C\u002Fh3>\n\u003Cp>\u003Cstrong>Tetragon\u003C\u002Fstrong> (by Isovalent\u002FCilium) is an eBPF-based security observability and runtime enforcement tool:\u003C\u002Fp>\n\u003Cul>\n\u003Cli>\u003Cstrong>Process execution tracking\u003C\u002Fstrong> — Every exec, fork, and exit event with full process tree context\u003C\u002Fli>\n\u003Cli>\u003Cstrong>File access monitoring\u003C\u002Fstrong> — Track reads, writes, and permission changes to sensitive files\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Network connection auditing\u003C\u002Fstrong> — Log every outbound connection with process context (which binary opened a socket to which IP)\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Security policy enforcement\u003C\u002Fstrong> — Block suspicious activities in real time (e.g., kill a process that tries to read \u002Fetc\u002Fshadow)\u003C\u002Fli>\n\u003C\u002Ful>\n\u003Cp>Tetragon integrates with Kubernetes admission controllers and SIEM systems, providing the audit trail that compliance teams need without any application-level logging code.\u003C\u002Fp>\n\u003Ch3>Grafana Beyla: Auto-Instrumentation\u003C\u002Fh3>\n\u003Cp>\u003Cstrong>Grafana Beyla\u003C\u002Fstrong> is an eBPF-based auto-instrumentation agent that generates OpenTelemetry-compatible traces and metrics without any code changes:\u003C\u002Fp>\n\u003Cul>\n\u003Cli>Detects HTTP, gRPC, SQL, and Redis requests at the kernel level\u003C\u002Fli>\n\u003Cli>Emits OpenTelemetry spans with proper trace context propagation\u003C\u002Fli>\n\u003Cli>Supports distributed tracing across services (propagates trace IDs through kernel observations)\u003C\u002Fli>\n\u003Cli>Integrates with Grafana Cloud, Tempo, Mimir, and any OpenTelemetry-compatible backend\u003C\u002Fli>\n\u003C\u002Ful>\n\u003Cp>Beyla is particularly useful for teams migrating from sidecar-based OpenTelemetry Collectors: drop in Beyla as a DaemonSet, remove the sidecar containers, and your existing Grafana dashboards keep working.\u003C\u002Fp>\n\u003Ch3>Splunk OBI: OpenTelemetry eBPF Instrumentation\u003C\u002Fh3>\n\u003Cp>At \u003Cstrong>KubeCon EU 2026\u003C\u002Fstrong> (March 2026, London), Splunk announced \u003Cstrong>OBI (OpenTelemetry eBPF Instrumentation)\u003C\u002Fstrong>, a project that contributes eBPF-based auto-instrumentation directly to the OpenTelemetry Collector:\u003C\u002Fp>\n\u003Cul>\n\u003Cli>\u003Cstrong>Upstream-first approach\u003C\u002Fstrong> — OBI is being contributed to the OpenTelemetry project, not a proprietary Splunk tool\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Full OTel compatibility\u003C\u002Fstrong> — Generates standard OpenTelemetry signals (traces, metrics, logs) from eBPF observations\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Language-agnostic\u003C\u002Fstrong> — Works with any language runtime (Go, Java, Python, Node.js, Rust, .NET) without SDK installation\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Hybrid mode\u003C\u002Fstrong> — Can supplement existing SDK instrumentation with kernel-level data for complete visibility\u003C\u002Fli>\n\u003C\u002Ful>\n\u003Cp>OBI represents the convergence of the eBPF and OpenTelemetry ecosystems. Instead of choosing between eBPF-native tools and OTel-native tools, you get both in a single pipeline.\u003C\u002Fp>\n\u003Ch2 id=\"performance-the-numbers-that-matter\">Performance: The Numbers That Matter\u003C\u002Fh2>\n\u003Cp>The resource savings from eBPF-based observability are dramatic. Here are real-world benchmarks from a \u003Cstrong>500-pod production Kubernetes cluster\u003C\u002Fstrong> running a microservices application:\u003C\u002Fp>\n\u003Ch3>Memory Usage Comparison\u003C\u002Fh3>\n\u003Ctable>\u003Cthead>\u003Ctr>\u003Cth>Component\u003C\u002Fth>\u003Cth>Sidecar Approach\u003C\u002Fth>\u003Cth>eBPF Approach\u003C\u002Fth>\u003Cth>Savings\u003C\u002Fth>\u003C\u002Ftr>\u003C\u002Fthead>\u003Ctbody>\n\u003Ctr>\u003Ctd>Envoy sidecar (500 pods)\u003C\u002Ftd>\u003Ctd>50 GB\u003C\u002Ftd>\u003Ctd>0 (Cilium CNI)\u003C\u002Ftd>\u003Ctd>50 GB\u003C\u002Ftd>\u003C\u002Ftr>\n\u003Ctr>\u003Ctd>OTel Collector sidecars (500 pods)\u003C\u002Ftd>\u003Ctd>15 GB\u003C\u002Ftd>\u003Ctd>0 (Beyla DaemonSet)\u003C\u002Ftd>\u003Ctd>15 GB\u003C\u002Ftd>\u003C\u002Ftr>\n\u003Ctr>\u003Ctd>Datadog Agent (DaemonSet, 20 nodes)\u003C\u002Ftd>\u003Ctd>10 GB\u003C\u002Ftd>\u003Ctd>N\u002FA\u003C\u002Ftd>\u003Ctd>10 GB\u003C\u002Ftd>\u003C\u002Ftr>\n\u003Ctr>\u003Ctd>Cilium agents (20 nodes)\u003C\u002Ftd>\u003Ctd>N\u002FA\u003C\u002Ftd>\u003Ctd>8 GB\u003C\u002Ftd>\u003Ctd>-8 GB\u003C\u002Ftd>\u003C\u002Ftr>\n\u003Ctr>\u003Ctd>Beyla agents (20 nodes)\u003C\u002Ftd>\u003Ctd>N\u002FA\u003C\u002Ftd>\u003Ctd>2 GB\u003C\u002Ftd>\u003Ctd>-2 GB\u003C\u002Ftd>\u003C\u002Ftr>\n\u003Ctr>\u003Ctd>Pixie (edge modules, 20 nodes)\u003C\u002Ftd>\u003Ctd>N\u002FA\u003C\u002Ftd>\u003Ctd>2 GB\u003C\u002Ftd>\u003Ctd>-2 GB\u003C\u002Ftd>\u003C\u002Ftr>\n\u003Ctr>\u003Ctd>\u003Cstrong>Total\u003C\u002Fstrong>\u003C\u002Ftd>\u003Ctd>\u003Cstrong>75 GB\u003C\u002Fstrong>\u003C\u002Ftd>\u003Ctd>\u003Cstrong>12 GB\u003C\u002Fstrong>\u003C\u002Ftd>\u003Ctd>\u003Cstrong>84% reduction\u003C\u002Fstrong>\u003C\u002Ftd>\u003C\u002Ftr>\n\u003C\u002Ftbody>\u003C\u002Ftable>\n\u003Ch3>CPU Overhead\u003C\u002Fh3>\n\u003Ctable>\u003Cthead>\u003Ctr>\u003Cth>Metric\u003C\u002Fth>\u003Cth>Sidecar Approach\u003C\u002Fth>\u003Cth>eBPF Approach\u003C\u002Fth>\u003C\u002Ftr>\u003C\u002Fthead>\u003Ctbody>\n\u003Ctr>\u003Ctd>Per-request latency added\u003C\u002Ftd>\u003Ctd>1-5ms (Envoy proxy hop)\u003C\u002Ftd>\u003Ctd><0.1ms (kernel-level)\u003C\u002Ftd>\u003C\u002Ftr>\n\u003Ctr>\u003Ctd>CPU overhead per node\u003C\u002Ftd>\u003Ctd>8-12%\u003C\u002Ftd>\u003Ctd><1%\u003C\u002Ftd>\u003C\u002Ftr>\n\u003Ctr>\u003Ctd>Tail latency impact (p99)\u003C\u002Ftd>\u003Ctd>+15-30ms\u003C\u002Ftd>\u003Ctd><1ms\u003C\u002Ftd>\u003C\u002Ftr>\n\u003C\u002Ftbody>\u003C\u002Ftable>\n\u003Ch3>Operational Metrics\u003C\u002Fh3>\n\u003Ctable>\u003Cthead>\u003Ctr>\u003Cth>Metric\u003C\u002Fth>\u003Cth>Sidecar Approach\u003C\u002Fth>\u003Cth>eBPF Approach\u003C\u002Fth>\u003C\u002Ftr>\u003C\u002Fthead>\u003Ctbody>\n\u003Ctr>\u003Ctd>Containers per pod\u003C\u002Ftd>\u003Ctd>2-4 (app + sidecars)\u003C\u002Ftd>\u003Ctd>1 (app only)\u003C\u002Ftd>\u003C\u002Ftr>\n\u003Ctr>\u003Ctd>Pod startup time\u003C\u002Ftd>\u003Ctd>5-15s (sidecar init)\u003C\u002Ftd>\u003Ctd>1-3s (app only)\u003C\u002Ftd>\u003C\u002Ftr>\n\u003Ctr>\u003Ctd>Config files to manage\u003C\u002Ftd>\u003Ctd>500+ (per-pod sidecar configs)\u003C\u002Ftd>\u003Ctd>20 (per-node DaemonSet configs)\u003C\u002Ftd>\u003C\u002Ftr>\n\u003Ctr>\u003Ctd>Languages requiring SDK\u003C\u002Ftd>\u003Ctd>All (per-language OTel SDK)\u003C\u002Ftd>\u003Ctd>None (kernel-level)\u003C\u002Ftd>\u003C\u002Ftr>\n\u003Ctr>\u003Ctd>Blind spots\u003C\u002Ftd>\u003Ctd>Non-instrumented services\u003C\u002Ftd>\u003Ctd>None (kernel sees all)\u003C\u002Ftd>\u003C\u002Ftr>\n\u003C\u002Ftbody>\u003C\u002Ftable>\n\u003Ch2 id=\"under-1-cpu-overhead-how-is-that-possible\">Under 1% CPU Overhead: How Is That Possible?\u003C\u002Fh2>\n\u003Cp>The sub-1% CPU overhead claim is real, verified by independent benchmarks from Isovalent, CNCF, and multiple end-user companies. Here is why eBPF is so efficient:\u003C\u002Fp>\n\u003Col>\n\u003Cli>\u003Cstrong>JIT compilation\u003C\u002Fstrong> — eBPF programs are compiled to native machine code by the kernel JIT compiler. They run at near-native speed, not in an interpreter.\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Per-CPU maps\u003C\u002Fstrong> — Data structures are partitioned per CPU core, eliminating lock contention. Each core writes to its own buffer.\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Ring buffers\u003C\u002Fstrong> — Events are pushed to user-space through lock-free ring buffers. No system calls needed for each event.\u003C\u002Fli>\n\u003Cli>\u003Cstrong>In-kernel aggregation\u003C\u002Fstrong> — eBPF programs can aggregate metrics (counters, histograms) in kernel space, sending only summaries to user-space instead of raw events.\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Selective attachment\u003C\u002Fstrong> — eBPF programs are only invoked at their specific hook points. An HTTP tracing program runs only when HTTP-related syscalls fire, not on every kernel event.\u003C\u002Fli>\n\u003C\u002Fol>\n\u003Ch2 id=\"migration-guide-from-sidecars-to-ebpf\">Migration Guide: From Sidecars to eBPF\u003C\u002Fh2>\n\u003Cp>Migrating from sidecar-based monitoring to eBPF is a phased process. Here is the recommended approach:\u003C\u002Fp>\n\u003Ch3>Phase 1: Deploy eBPF Tools Alongside Sidecars (Week 1-2)\u003C\u002Fh3>\n\u003Cul>\n\u003Cli>Install Cilium as your CNI (if not already using it)\u003C\u002Fli>\n\u003Cli>Deploy Hubble for network observability\u003C\u002Fli>\n\u003Cli>Deploy Beyla as a DaemonSet for auto-instrumented traces\u003C\u002Fli>\n\u003Cli>Run both sidecar and eBPF observability in parallel\u003C\u002Fli>\n\u003Cli>Compare data quality and coverage\u003C\u002Fli>\n\u003C\u002Ful>\n\u003Ch3>Phase 2: Validate and Tune (Week 3-4)\u003C\u002Fh3>\n\u003Cul>\n\u003Cli>Verify that eBPF tools capture the same signals as your sidecar stack\u003C\u002Fli>\n\u003Cli>Tune Beyla’s protocol detection for your specific services\u003C\u002Fli>\n\u003Cli>Configure Hubble’s L7 parsing for your custom protocols\u003C\u002Fli>\n\u003Cli>Set up dashboards that mirror your existing sidecar-based dashboards\u003C\u002Fli>\n\u003Cli>Alert your team to any coverage gaps\u003C\u002Fli>\n\u003C\u002Ful>\n\u003Ch3>Phase 3: Remove Sidecars Incrementally (Week 5-8)\u003C\u002Fh3>\n\u003Cul>\n\u003Cli>Start with non-critical services: remove OTel Collector sidecars\u003C\u002Fli>\n\u003Cli>Monitor for data quality regressions\u003C\u002Fli>\n\u003Cli>Remove Envoy sidecars from services that do not need advanced traffic management\u003C\u002Fli>\n\u003Cli>Keep Envoy only for services that need its advanced features (circuit breaking, retries, traffic splitting)\u003C\u002Fli>\n\u003C\u002Ful>\n\u003Ch3>Phase 4: Full eBPF Stack (Week 9-12)\u003C\u002Fh3>\n\u003Cul>\n\u003Cli>Remove remaining sidecars\u003C\u002Fli>\n\u003Cli>Deploy Tetragon for runtime security\u003C\u002Fli>\n\u003Cli>Consolidate alerting on eBPF-derived signals\u003C\u002Fli>\n\u003Cli>Document the new observability architecture\u003C\u002Fli>\n\u003Cli>Reclaim freed resources (you will get 60-80% of sidecar resources back)\u003C\u002Fli>\n\u003C\u002Ful>\n\u003Ch3>Rollback Plan\u003C\u002Fh3>\n\u003Cp>Keep your sidecar configurations in version control. If eBPF tools miss critical signals for a specific service, you can redeploy sidecars for that service while keeping eBPF for everything else. Hybrid deployments work fine.\u003C\u002Fp>\n\u003Ch2 id=\"kernel-requirements\">Kernel Requirements\u003C\u002Fh2>\n\u003Cp>eBPF observability tools require a modern Linux kernel. Here are the minimum versions:\u003C\u002Fp>\n\u003Ctable>\u003Cthead>\u003Ctr>\u003Cth>Feature\u003C\u002Fth>\u003Cth>Minimum Kernel\u003C\u002Fth>\u003Cth>Recommended\u003C\u002Fth>\u003C\u002Ftr>\u003C\u002Fthead>\u003Ctbody>\n\u003Ctr>\u003Ctd>Basic eBPF (maps, programs)\u003C\u002Ftd>\u003Ctd>4.15\u003C\u002Ftd>\u003Ctd>5.15+\u003C\u002Ftd>\u003C\u002Ftr>\n\u003Ctr>\u003Ctd>BPF ring buffer\u003C\u002Ftd>\u003Ctd>5.8\u003C\u002Ftd>\u003Ctd>5.15+\u003C\u002Ftd>\u003C\u002Ftr>\n\u003Ctr>\u003Ctd>BPF CO-RE (compile once, run everywhere)\u003C\u002Ftd>\u003Ctd>5.5\u003C\u002Ftd>\u003Ctd>5.15+\u003C\u002Ftd>\u003C\u002Ftr>\n\u003Ctr>\u003Ctd>BTF (BPF Type Format)\u003C\u002Ftd>\u003Ctd>5.2\u003C\u002Ftd>\u003Ctd>5.15+\u003C\u002Ftd>\u003C\u002Ftr>\n\u003Ctr>\u003Ctd>BPF LSM (security policies)\u003C\u002Ftd>\u003Ctd>5.7\u003C\u002Ftd>\u003Ctd>5.15+\u003C\u002Ftd>\u003C\u002Ftr>\n\u003Ctr>\u003Ctd>BPF iterators\u003C\u002Ftd>\u003Ctd>5.8\u003C\u002Ftd>\u003Ctd>5.15+\u003C\u002Ftd>\u003C\u002Ftr>\n\u003C\u002Ftbody>\u003C\u002Ftable>\n\u003Cp>All major Kubernetes distributions (EKS with Amazon Linux 2023, GKE with COS, AKS with Ubuntu 22.04) ship kernels that meet these requirements. If you are running on-premises, ensure your nodes run \u003Cstrong>kernel 5.15 or later\u003C\u002Fstrong> for the best eBPF experience.\u003C\u002Fp>\n\u003Ch2 id=\"frequently-asked-questions\">Frequently Asked Questions\u003C\u002Fh2>\n\u003Ch3 id=\"does-ebpf-observability-work-with-non-kubernetes-workloads\">Does eBPF observability work with non-Kubernetes workloads?\u003C\u002Fh3>\n\u003Cp>Yes. eBPF runs at the Linux kernel level, so it works with any workload — containers, VMs, bare metal, systemd services. Cilium and Tetragon can be deployed outside of Kubernetes, and Beyla supports standalone mode. However, the richest experience is in Kubernetes where tools can correlate kernel events with pod and service metadata.\u003C\u002Fp>\n\u003Ch3 id=\"can-ebpf-replace-distributed-tracing\">Can eBPF replace distributed tracing?\u003C\u002Fh3>\n\u003Cp>For many teams, yes. Beyla and Pixie generate distributed traces from kernel observations, including trace context propagation. However, eBPF traces are limited to request\u002Fresponse boundaries — they cannot trace custom business logic inside your functions. For deep application-level tracing (e.g., “which database query was slow inside this handler”), you still need SDK instrumentation. The recommended approach is eBPF for infrastructure-level tracing plus targeted SDK instrumentation for business-critical paths.\u003C\u002Fp>\n\u003Ch3 id=\"what-about-encrypted-traffic-tls\">What about encrypted traffic (TLS)?\u003C\u002Fh3>\n\u003Cp>eBPF tools can trace TLS traffic by attaching to the TLS library functions (e.g., OpenSSL’s SSL_read and SSL_write) rather than the network layer. This captures plaintext data before encryption or after decryption. Pixie, Beyla, and Cilium all support TLS tracing for OpenSSL, BoringSSL, and Go’s crypto\u002Ftls. Rust’s rustls support was added in early 2026.\u003C\u002Fp>\n\u003Ch3 id=\"is-ebpf-safe-can-a-buggy-ebpf-program-crash-the-kernel\">Is eBPF safe? Can a buggy eBPF program crash the kernel?\u003C\u002Fh3>\n\u003Cp>No. The eBPF verifier is a static analyzer built into the kernel that checks every eBPF program before loading. It rejects programs that could cause infinite loops, out-of-bounds memory access, or other safety violations. A buggy eBPF program will fail to load — it will never crash the kernel. This is a fundamental safety guarantee of the eBPF architecture.\u003C\u002Fp>\n\u003Ch3 id=\"how-does-ebpf-handle-high-throughput-services-100k-requests-second\">How does eBPF handle high-throughput services (100K+ requests\u002Fsecond)?\u003C\u002Fh3>\n\u003Cp>eBPF handles high throughput through in-kernel aggregation and sampling. Instead of sending every event to user-space, eBPF programs can compute histograms, counters, and summaries in kernel maps, sending only aggregated data. For full-fidelity tracing at extreme throughput, tools like Pixie use intelligent sampling that captures all error and slow requests while sampling normal requests.\u003C\u002Fp>\n\u003Ch3 id=\"what-is-the-total-cost-of-ownership-compared-to-commercial-apm-tools\">What is the total cost of ownership compared to commercial APM tools?\u003C\u002Fh3>\n\u003Cp>eBPF observability tools (Cilium, Hubble, Pixie, Beyla, Tetragon) are open-source. The main costs are compute (DaemonSet resources — roughly 12 GB RAM and 2-3 CPU cores for a 500-pod cluster) and engineering time for setup and maintenance. Compared to commercial APM tools (Datadog, New Relic, Splunk) that charge $15-30 per host per month plus ingestion fees, eBPF-based stacks typically cost 70-90% less at scale.\u003C\u002Fp>\n","en","b0000000-0000-0000-0000-000000000001",true,"2026-03-28T10:44:36.940261Z","67% of K8s teams use eBPF observability. Cilium, Hubble, Pixie, Tetragon, Grafana Beyla — full stack guide with migration plan and benchmarks.","eBPF observability",null,"index, follow",[21,26,30],{"id":22,"name":23,"slug":24,"created_at":25},"c0000000-0000-0000-0000-000000000012","DevOps","devops","2026-03-28T10:44:21.513630Z",{"id":27,"name":28,"slug":29,"created_at":25},"c0000000-0000-0000-0000-000000000006","Docker","docker",{"id":31,"name":32,"slug":33,"created_at":25},"c0000000-0000-0000-0000-000000000007","Kubernetes","kubernetes",[35,42,48],{"id":36,"title":37,"slug":38,"excerpt":39,"locale":12,"category_name":40,"published_at":41},"d0200000-0000-0000-0000-000000000003","Why Bali Is Becoming Southeast Asia's Impact-Tech Hub in 2026","why-bali-becoming-southeast-asia-impact-tech-hub-2026","Bali ranks #16 among Southeast Asian startup ecosystems. With a growing concentration of Web3 builders, AI sustainability startups, and eco-travel tech companies, the island is carving a niche as the region's impact-tech capital.","Engineering","2026-03-28T10:44:37.748283Z",{"id":43,"title":44,"slug":45,"excerpt":46,"locale":12,"category_name":40,"published_at":47},"d0200000-0000-0000-0000-000000000002","ASEAN Data Protection Patchwork: A Developer's Compliance Checklist","asean-data-protection-patchwork-developer-compliance-checklist","Seven ASEAN countries now have comprehensive data protection laws, each with different consent models, localization requirements, and penalty structures. Here is a practical compliance checklist for developers building multi-country applications.","2026-03-28T10:44:37.374741Z",{"id":49,"title":50,"slug":51,"excerpt":52,"locale":12,"category_name":40,"published_at":53},"d0200000-0000-0000-0000-000000000001","Indonesia's $29 Billion Digital Transformation: Opportunities for Software Companies","indonesia-29-billion-digital-transformation-opportunities-software-companies","Indonesia's IT services market is projected to reach $29.03 billion in 2026, up from $24.37 billion in 2025. Cloud infrastructure, AI, e-commerce, and data centers are driving the fastest growth in Southeast Asia.","2026-03-28T10:44:37.349311Z",{"id":13,"name":55,"slug":56,"bio":57,"photo_url":18,"linkedin":18,"role":58,"created_at":59,"updated_at":59},"Open Soft Team","open-soft-team","The engineering team at Open Soft, building premium software solutions from Bali, Indonesia.","Engineering Team","2026-03-28T08:31:22.226811Z"]