Kubernetes events disappear after an hour

kubectl get events is the first place most people look when a pod misbehaves. Kubernetes events tell you exactly what’s happening in your cluster (pod scheduling failures, image pull errors, OOM kills), but by default they disappear from etcd after 60 minutes.

This guide walks through why that’s a problem and how to export events to the tools your team already uses.

What are Kubernetes events?

Every time something happens in your cluster, Kubernetes creates an Event object. A pod gets scheduled, a container crashes, a volume fails to mount. Events are first-class API resources, same as Pods or Deployments.

Here’s what one looks like:

apiVersion: v1
kind: Event
metadata:
  name: my-app-pod.17a6f8e2c
  namespace: production
reason: BackOff
message: "Back-off restarting failed container"
type: Warning
involvedObject:
  kind: Pod
  name: my-app-pod
  namespace: production
count: 15
firstTimestamp: "2026-03-07T10:00:00Z"
lastTimestamp: "2026-03-07T10:15:00Z"

The useful fields:

• Reason: the machine-readable cause (BackOff, FailedScheduling, Unhealthy, Pulling, Killing)
• Type: either Normal or Warning
• Message: a human-readable description of what happened
• InvolvedObject: the resource this event is about
• Count: how many times this event has occurred

Why the defaults are a problem

Three things make stock Kubernetes events nearly useless for debugging:

Events expire after 1 hour

The API server retains events for 60 minutes, so the record of a pod crash at 2 AM is gone by 9 AM.

No search or aggregation

kubectl get events gives you a flat, unsorted list with no cross-namespace search, severity filtering or time correlation.

No alerting

Kubernetes won’t tell you when something goes wrong. A FailedScheduling event could fire hundreds of times before anyone notices, unless someone happens to be watching.

Export events to external systems

The fix is an event exporter: a lightweight component that watches the Kubernetes event stream via the API server and forwards events to external destinations (sinks) like Elasticsearch, Slack, Loki, Kafka, or webhooks.

The pipeline:

The exporter uses a SharedInformer to watch events without polling, applies configurable routing rules, and delivers them to one or more sinks.

Setting up the exporter

The Ownkube Kubernetes Events Exporter is a fork we maintain. The upstream project by Opsgenie (later Resmo) has been dormant since 2023 and has some bugs that cause silent event loss, covered in the last section.

Install with Helm

helm install event-exporter \
  oci://ghcr.io/ownkube/charts/kubernetes-events-exporter \
  --namespace monitoring \
  --create-namespace

Basic configuration

The exporter takes a YAML config (deployed as a ConfigMap). A minimal config that sends everything to stdout:

logLevel: info
route:
  routes:
    - match:
        - receiver: "stdout"
receivers:
  - name: "stdout"
    stdout: {}

Routing

You can filter and direct events to different sinks based on any event property: namespace, event type, reason, involved object kind, etc.

Route warning events to Slack

route:
  routes:
    - match:
        - receiver: "slack-warnings"
      match:
        kind: ".*"
        type: "Warning"
    - match:
        - receiver: "elasticsearch"
receivers:
  - name: "slack-warnings"
    slack:
      token: "${SLACK_TOKEN}"
      channel: "#k8s-alerts"
      message: |
        *{{ .Reason }}* in {{ .InvolvedObject.Namespace }}/{{ .InvolvedObject.Name }}
        ```{{ .Message }}```

  - name: "elasticsearch"
    elasticsearch:
      hosts:
        - "https://elasticsearch:9200"
      index: "k8s-events"

Drop noisy events

Some events are just noise. Drop them before they hit any sink:

route:
  drop:
    - type: "Normal"
      reason: "LeaderElection"
    - namespace: "kube-system"
      reason: "ScalingReplicaSet"
  routes:
    - match:
        - receiver: "elasticsearch"

Route by namespace

This sends production warnings to PagerDuty, staging events to Slack, and everything to Elasticsearch.

route:
  routes:
    - match:
        - receiver: "pagerduty-webhook"
      match:
        namespace: "production"
        type: "Warning"
    - match:
        - receiver: "slack-staging"
      match:
        namespace: "staging"
    - match:
        - receiver: "elasticsearch"

Sink examples

The exporter supports 30+ sinks. Here are the ones most people use.

Elasticsearch

Store all events for long-term analysis and Kibana dashboards:

receivers:
  - name: "elasticsearch"
    elasticsearch:
      hosts:
        - "https://elasticsearch.monitoring:9200"
      index: "k8s-events"
      username: "${ES_USER}"
      password: "${ES_PASSWORD}"
      useEventID: true

Loki

Feed events into Grafana Loki for log-style querying:

receivers:
  - name: "loki"
    loki:
      url: "http://loki.monitoring:3100/loki/api/v1/push"
      streamLabels:
        app: "kubernetes-events"
      basicAuth:
        username: "${LOKI_USER}"
        password: "${LOKI_PASSWORD}"

Slack

receivers:
  - name: "slack"
    slack:
      token: "${SLACK_TOKEN}"
      channel: "#k8s-events"
      message: |
        *{{ .Type }}* event on *{{ .InvolvedObject.Kind }}* `{{ .InvolvedObject.Name }}`
        Namespace: `{{ .InvolvedObject.Namespace }}`
        Reason: `{{ .Reason }}`
        Message: {{ .Message }}

Kafka

receivers:
  - name: "kafka"
    kafka:
      brokers:
        - "kafka-broker:9092"
      topic: "kubernetes-events"
      tls:
        enable: true

Webhook

Send events to any HTTP endpoint:

receivers:
  - name: "webhook"
    webhook:
      endpoint: "https://your-api.example.com/k8s-events"
      headers:
        Authorization: "Bearer ${WEBHOOK_TOKEN}"
        Content-Type: "application/json"

Prometheus

Turn events into Prometheus metrics for alerting with Alertmanager:

receivers:
  - name: "prometheus"
    prometheus:
      labels:
        - "reason"
        - "type"
        - "involvedObject.kind"
        - "involvedObject.namespace"

Events worth watching

These are the events that usually point to real problems.

Pod lifecycle issues

Scheduling problems

Node issues

Practical tips

Drop noise early

Leader election events, successful image pulls, and routine scaling events generate a lot of volume. Drop them at the top of your routing tree to keep both storage costs and Slack noise down.

Don’t hardcode secrets

The exporter supports ${ENV_VAR} syntax. Use it with Kubernetes Secrets:

env:
  - name: SLACK_TOKEN
    valueFrom:
      secretKeyRef:
        name: event-exporter-secrets
        key: slack-token

Separate sinks for separate jobs

• Elasticsearch or Loki for historical analysis
• Slack for real-time awareness of warnings
• Webhook or PagerDuty for on-call alerting
• Prometheus for dashboards and SLO tracking

Run with leader election

If you’re running multiple replicas, enable leader election so only one instance processes events:

leaderElection:
  enabled: true
  leaderElectionID: "event-exporter-leader"

Monitor the exporter itself

It exposes Prometheus metrics at /metrics. Set up alerts for high error rates on sink delivery, event processing lag, and exporter pod restarts.

Why we maintain this fork

The original kubernetes-event-exporter by Opsgenie has been dormant since 2023. We ran into bugs in production and started fixing them. The fork now includes:

• A fix for silent event loss. The upstream only handled add and delete operations, missing updates entirely.
• Accurate event age calculation using max(EventTime, FirstTimestamp, LastTimestamp) instead of just LastTimestamp, which was causing stale events to get re-exported.
• Loki improvements: basic auth, stream label templating, TLS transport fixes.
• A Prometheus sink for converting events directly into metrics.
• Elasticsearch v8 compatibility (the v8 API broke the upstream).
• SNS FIFO support with proper message group IDs.

The full code and Helm chart are on GitHub.

Getting started

1. Install the exporter via Helm into your monitoring namespace
2. Start with stdout + one sink (Slack or Elasticsearch) to validate routing works
3. Add drop rules for the noisy stuff
4. Expand sinks as you need them

With that in place, events stay searchable, filterable, and alertable long after the 60-minute window.

If you’d rather skip the plumbing

If you’re running a single-instance cluster or EKS in your own AWS account, Ownkube gives every app logs, metrics and alerts out of the box, including an alert when a container keeps restarting, without installing a chart or writing a routing YAML. It runs on your cluster in your AWS account, so that data stays inside your perimeter.

Related reading

• Kubernetes cost optimization for startups
• EKS vs a single-instance cluster on AWS: cost and when to choose
• Ingress-NGINX EOL: migrating to Gateway API

See Ownkube pricing for the per-minute wallet model, from $5 a month. Deploy my first app and event monitoring is one of the things you stop thinking about.