---
title: "How Mastodon Runs OpenTelemetry Collectors in Production"
source: OpenTelemetry Documentation
source_url: https://github.com/open-telemetry/opentelemetry.io
licence: CC-BY-4.0
domain: engineering
subdomain: opentelemetry-docs
date_added: 2026-04-25
---

At the beginning of 2025, the OpenTelemetry Developer Experience SIG
[published the results of its first community survey](/blog/2025/devex-survey/).
One of the strongest themes was clear: teams want more real-world examples of
how the OpenTelemetry SDKs and the OpenTelemetry Collector are actually used in
production.

To help close that gap, the SIG began collecting stories directly from end
users—across industries, architectures, and company sizes. This post kicks off a
new series focused specifically on organizations' real world stories, starting
with a small but uniquely challenging case.

This first story features Mastodon, a non-profit organization operating at
global scale with a remarkably small team.

## Mastodon at a glance

[Mastodon](https://joinmastodon.org) is a free, open source, decentralized
social media platform operated by a non-profit organization.

Decentralization is not a marketing term here; it is a core architectural
principle. Anyone can
[run their own Mastodon server](https://docs.joinmastodon.org/user/run-your-own/),
and those independently operated servers interoperate using open protocols as
part of what is called the _Fediverse_—a federated network of independent social
platforms that communicate with one another using standardized protocols such as
ActivityPub. Much like email, users can communicate across instances regardless
of who operates them.

This philosophy shapes not just Mastodon’s feature decisions, but also its
approach to observability.

### Organizational structure

The entire Mastodon organization consists of around 20 people, and observability
infrastructure (including the OpenTelemetry Collector) is managed by a single
engineer.

Despite the small team size, Mastodon operates two large, production Mastodon
instances:

- [mastodon.social](https://mastodon.social)

  Runs on Kubernetes with autoscaling between 9 and 15 nodes (16 cores, 64 GB
  RAM each). Web frontend scale between 5 and 20 pods, while various Sidekiq
  worker pools scale between 10 and 40 pods. On average, mastodon.social has
  70–80 pods running at any given time. This platform handles up to **300,000
  active users** per day and approximately 10 million requests per minute.

- [mastodon.online](https://mastodon.online)

  Runs on Kubernetes with autoscaling between 3 and 6 nodes (8 cores, 32 GB RAM
  each). Web frontend scale between 3 and 10 pods, and Sidekiq pools scale
  between 5 and 15 pods, resulting in an average of 20–30 pods in total. This
  instance operates at a smaller but still substantial scale.

With such limited operational bandwidth, simplicity and reliability are
non-negotiable.

### OpenTelemetry adoption: Freedom of choice by design

Because Mastodon is open source and designed to be run by others, the team
wanted a telemetry solution that preserved operator freedom.

OpenTelemetry became the default because it allows each Mastodon server operator
to decide how—or whether—telemetry is collected.

Using simple
[environment variable configuration](https://docs.joinmastodon.org/admin/config/#otel),
operators can choose to:

- Send telemetry directly to an observability backend (using only the Ruby SDK
  configuration)
- Route telemetry through an OpenTelemetry Collector
- Disable telemetry entirely

The core Mastodon organization does not track how external instances handle
observability. What matters is that the emitted telemetry adheres strictly to
**[OpenTelemetry semantic conventions](/docs/specs/semconv/)**, making it usable
everywhere.

This approach avoids vendor-specific data models and ensures compatibility with
the broader OpenTelemetry ecosystem—without Mastodon having to maintain its own
conventions.

## Collector architecture: One per namespace, no more

Mastodon’s Collector architecture is intentionally minimal.

A single OpenTelemetry Collector per Kubernetes namespace handles all telemetry
signals: traces, metrics, and logs. There are no separate gateway and agent
tiers, no complex routing layers, and no custom deployment tooling.

![Mastodon Nodes architecture diagram](mastodon-nodes.png)

Given the scale and traffic, this has proven more than sufficient.

During the interview, [Tim Campbell](https://github.com/timetinytim), Software
Engineer at Mastodon, shared that in the ~2 years they’ve been running the
Collector, they’ve _never had a single issue_ with it.

> “To my surprise, to my very pleasant surprise, I haven't run into a single
> issue. Because we're using a Kubernetes operator for it, if it ever does have
> any issue, it just restarts automatically. At least as far as the actual
> traces and logs go in Datadog, I haven't seen any gaps. Memory and
> process-wise, it's stayed perfectly happy within the limits that we've set.”

## Deployment and lifecycle management

To keep operational overhead as low as possible, Mastodon relies on:

- The [OpenTelemetry Operator](/docs/platforms/kubernetes/operator/) for
  Kubernetes
- Argo CD for Git-based deployments and promotion

Each Collector is defined as an `OpenTelemetryCollector` custom resource. From
there, Kubernetes handles reconciliation, restarts, and lifecycle management
automatically.

> “Basically we just need to create a yaml file for each
> `OpenTelemetryCollector` object we need to create, and Argo will automatically
> deploy/update what we need.”

This model provides:

- Declarative configuration
- Automatic recovery on failure
- Clear auditability through Git history

Notably, Mastodon does not enforce strict CPU or memory limits on Collector
pods. In practice, resource consumption has remained negligible compared to the
rest of the platform.

## Traffic management through sampling

Rather than relying on resource limits, Mastodon controls observability overhead
primarily through tail-based sampling.

- On mastodon.social, successful traces are sampled at roughly 0.1%, resulting
  in only a few dozen traces per minute despite extremely high traffic.
- On mastodon.online, sampling is slightly more permissive but follows the same
  principles.
- All error traces are always collected, ensuring full visibility into failures.

This approach keeps data volume predictable while preserving high-value
diagnostic data.

## Configuration: Opinionated, but minimal

Mastodon uses the OpenTelemetry Collector Contrib distribution, primarily for
convenience—it includes everything they need without requiring custom builds.

The configuration focuses on:

- OTLP ingestion for all signals
- Kubernetes metadata enrichment
- Resource detection
- Tail-based sampling
- Transformation for backend compatibility

A full production configuration is included below for reference (you can also
view it on [otelbin][otelbin-mastodon]):

```yaml
apiVersion: opentelemetry.io/v1beta1
kind: OpenTelemetryCollector
metadata:
  name: mastodon-social
  namespace: mastodon-social
spec:
  nodeSelector:
    joinmastodon.org/property: mastodon.social
  env:
    - name: DD_API_KEY
      valueFrom:
        secretKeyRef:
          name: datadog-secret
          key: api-key
    - name: DD_SITE
      valueFrom:
        secretKeyRef:
          name: datadog-secret
          key: site
  config:
    receivers:
      otlp:
        protocols:
          grpc:
            endpoint: 0.0.0.0:4317
          http:
            endpoint: 0.0.0.0:4318
            cors:
              allowed_origins:
                - 'http://*'
                - 'https://*'

    processors:
      batch: {}
      resource:
        attributes:
          - key: deployment.environment.name
            value: 'production'
            action: upsert
          - key: property
            value: 'mastodon.social'
            action: upsert
          - key: git.commit.sha
            from_attribute: vcs.repository.ref.revision
            action: insert
          - key: git.repository_url
            from_attribute: vcs.repository.url.full
            action: insert
      k8sattributes:
        auth_type: 'serviceAccount'
        passthrough: false
        extract:
          metadata:
            - k8s.namespace.name
            - k8s.pod.name
            - k8s.pod.start_time
            - k8s.pod.uid
            - k8s.deployment.name
            - k8s.node.name
          labels:
            - tag_name: app.label.component
              key: app.kubernetes.io/component
              from: pod
        pod_association:
          - sources:
              - from: resource_attribute
                name: k8s.pod.ip
          - sources:
              - from: resource_attribute
                name: k8s.pod.uid
          - sources:
              - from: connection
      resourcedetection:
        detectors: [system]
        system:
          resource_attributes:
            os.description:
              enabled: true
            host.arch:
              enabled: true
            host.cpu.vendor.id:
              enabled: true
            host.cpu.family:
              enabled: true
            host.cpu.model.id:
              enabled: true
            host.cpu.model.name:
              enabled: true
            host.cpu.stepping:
              enabled: true
            host.cpu.cache.l2.size:
              enabled: true
      transform:
        error_mode: ignore

        # Proper code function naming
        trace_statements:
          - context: span
            conditions:
              - attributes["code.namespace"] != nil
            statements:
              - set(attributes["resource.name"],
                Concat([attributes["code.namespace"],
                attributes["code.function"]], "#"))

          # Proper kubernetes hostname
          - context: resource
            conditions:
              - attributes["k8s.node.name"] != nil
            statements:
              - set (attributes["k8s.node.name"],
                Concat([attributes["k8s.node.name"], "k8s-1"], "-"))
        metric_statements:
          - context: resource
            conditions:
              - attributes["k8s.node.name"] != nil
            statements:
              - set (attributes["k8s.node.name"],
                Concat([attributes["k8s.node.name"], "k8s-1"], "-"))
        log_statements:
          - context: resource
            conditions:
              - attributes["k8s.node.name"] != nil
            statements:
              - set (attributes["k8s.node.name"],
                Concat([attributes["k8s.node.name"], "k8s-1"], "-"))
      attributes/sidekiq:
        include:
          match_type: strict
          attributes:
            - key: messaging.sidekiq.job_class
        actions:
          - key: resource.name
            from_attribute: messaging.sidekiq.job_class
            action: upsert
      tail_sampling:
        policies:
          [
            {
              name: errors-policy,
              type: status_code,
              status_code: { status_codes: [ERROR] },
            },
            {
              name: randomized-policy,
              type: probabilistic,
              probabilistic: { sampling_percentage: 0.1 },
            },
          ]

    connectors:
      datadog/connector:
        traces:
          compute_stats_by_span_kind: true

    exporters:
      datadog:
        api:
          site: ${DD_SITE}
          key: ${DD_API_KEY}
        traces:
          compute_stats_by_span_kind: true
          trace_buffer: 500

    service:
      pipelines:
        traces/all:
          receivers: [otlp]
          processors:
            [
              resource,
              k8sattributes,
              resourcedetection,
              transform,
              attributes/sidekiq,
              batch,
            ]
          exporters: [datadog/connector]
        traces/sample:
          receivers: [datadog/connector]
          processors: [tail_sampling, batch]
          exporters: [datadog]
        metrics:
          receivers: [datadog/connector, otlp]
          processors:
            [resource, k8sattributes, resourcedetection, transform, batch]
          exporters: [datadog]
        logs:
          receivers: [otlp]
          processors:
            [
              resource,
              k8sattributes,
              resourcedetection,
              transform,
              attributes/sidekiq,
              batch,
            ]
          exporters: [datadog]
```

### Staying up to date

Mastodon typically upgrades the OpenTelemetry Collector within a day or two of
each release.

> “Everything is documented, and all breaking changes are properly detailed,”
> Tim noted, praising the clarity of the release notes.

While frequent releases sometimes introduce breaking changes, the team views
this as a sign of healthy and active development—as long as you stay current.

### Lessons and pain points

The most challenging part of the journey was simply getting started.
Understanding how the Collector’s components fit together took time, especially
for a team without dedicated observability specialists. More recently, the
biggest complexity has come from advanced use of the transform processor,
particularly when adapting span attributes for backend-specific naming
requirements.

```yaml
transform:
  error_mode: ignore

  # Proper code function naming
  trace_statements:
    - context: span
      conditions:
        - attributes["code.namespace"] != nil
      statements:
        - set(attributes["resource.name"], Concat([attributes["code.namespace"],
          attributes["code.function"]], "#"))
```

In the transform processor rule above, they have configured a condition to set
`resource.name` (a Datadog specific attribute) to the value of
`code.namespace#code.function`. With that set, whenever the span arrived at the
backend, it was able to map to the name they had defined. Despite that learning
curve, the overall experience has exceeded expectations.

> “You can basically do anything you want. It went beyond my expectations.
> Everything works pretty well.”

That reliability and flexibility are the reasons why Mastodon continues to use
the OpenTelemetry Collector in production.

## Advice for small teams

Based on Mastodon’s experience, a few lessons stand out:

- **Keep the architecture simple**: one Collector can go a long way
- **Rely on Kubernetes operators** for lifecycle management
- **Use sampling** to control cost
- **Stick to semantic conventions** to avoid long-term lock-in
- **Upgrade frequently** to reduce the pain of breaking changes

## What’s next

Mastodon’s story shows that even a very small team can successfully operate
OpenTelemetry Collectors in production—at global scale—without significant
operational burden.

This is just the first story in the series.

In upcoming posts, we’ll explore how medium and large organizations deploy and
operate the OpenTelemetry Collector, manage instrumentation across services, and
how their challenges—and solutions—change with scale.

If you’re running OpenTelemetry in production and want to share your experience,
join the CNCF [#otel-devex](https://cloud-native.slack.com/archives/C01S42U83B2)
Slack channel. We’d love to hear your story—and learn how we can keep improving
the OpenTelemetry developer experience together.

[otelbin-mastodon]:
  https://www.otelbin.io/?#config=receivers%3A*N__otlp%3A*N____protocols%3A*N______grpc%3A*N________endpoint%3A_0.0.0.0%3A4317*N______http%3A*N________endpoint%3A_0.0.0.0%3A4318*N________cors%3A*N__________allowed*_origins%3A*N____________-_*%22http%3A%2F%2F***%22*N____________-_*%22https%3A%2F%2F***%22*N*Nprocessors%3A*N__batch%3A_%7B%7D*N__resource%3A*N____attributes%3A*N______-_key%3A_deployment.environment.name*N________value%3A_*%22production*%22*N________action%3A_upsert*N______-_key%3A_property*N________value%3A_*%22mastodon.social*%22*N________action%3A_upsert*N______-_key%3A_git.commit.sha*N________from*_attribute%3A_vcs.repository.ref.revision*N________action%3A_insert*N______-_key%3A_git.repository*_url*N________from*_attribute%3A_vcs.repository.url.full*N________action%3A_insert*N__k8sattributes%3A*N____auth*_type%3A_*%22serviceAccount*%22*N____passthrough%3A_false*N____extract%3A*N______metadata%3A*N________-_k8s.namespace.name*N________-_k8s.pod.name*N________-_k8s.pod.start*_time*N________-_k8s.pod.uid*N________-_k8s.deployment.name*N________-_k8s.node.name*N______labels%3A*N________-_tag*_name%3A_app.label.component*N__________key%3A_app.kubernetes.io%2Fcomponent*N__________from%3A_pod*N____pod*_association%3A*N______-_sources%3A*N__________-_from%3A_resource*_attribute*N____________name%3A_k8s.pod.ip*N______-_sources%3A*N__________-_from%3A_resource*_attribute*N____________name%3A_k8s.pod.uid*N______-_sources%3A*N__________-_from%3A_connection*N__resourcedetection%3A*N____detectors%3A_%5Bsystem%5D*N____system%3A*N______resource*_attributes%3A*N________os.description%3A*N__________enabled%3A_true*N________host.arch%3A*N__________enabled%3A_true*N________host.cpu.vendor.id%3A*N__________enabled%3A_true*N________host.cpu.family%3A*N__________enabled%3A_true*N________host.cpu.model.id%3A*N__________enabled%3A_true*N________host.cpu.model.name%3A*N__________enabled%3A_true*N________host.cpu.stepping%3A*N__________enabled%3A_true*N________host.cpu.cache.l2.size%3A*N__________enabled%3A_true*N__transform%3A*N____error*_mode%3A_ignore*N*N____*H_Proper_code_function_naming*N____trace*_statements%3A*N______-_context%3A_span*N________conditions%3A*N__________-_attributes%5B%22code.namespace%22%5D_%21*E_nil*N________statements%3A*N__________-_set*Cattributes%5B%22resource.name%22%5D%2C*N____________Concat*C%5Battributes%5B%22code.namespace%22%5D%2C*N____________attributes%5B%22code.function%22%5D%5D%2C_%22*H%22*D*D*N*N______*H_Proper_kubernetes_hostname*N______-_context%3A_resource*N________conditions%3A*N__________-_attributes%5B%22k8s.node.name%22%5D_%21*E_nil*N________statements%3A*N__________-_set_*Cattributes%5B%22k8s.node.name%22%5D%2C*N____________Concat*C%5Battributes%5B%22k8s.node.name%22%5D%2C_%22k8s-1%22%5D%2C_%22-%22*D*D*N____metric*_statements%3A*N______-_context%3A_resource*N________conditions%3A*N__________-_attributes%5B%22k8s.node.name%22%5D_%21*E_nil*N________statements%3A*N__________-_set_*Cattributes%5B%22k8s.node.name%22%5D%2C*N____________Concat*C%5Battributes%5B%22k8s.node.name%22%5D%2C_%22k8s-1%22%5D%2C_%22-%22*D*D*N____log*_statements%3A*N______-_context%3A_resource*N________conditions%3A*N__________-_attributes%5B%22k8s.node.name%22%5D_%21*E_nil*N________statements%3A*N__________-_set_*Cattributes%5B%22k8s.node.name%22%5D%2C*N____________Concat*C%5Battributes%5B%22k8s.node.name%22%5D%2C_%22k8s-1%22%5D%2C_%22-%22*D*D*N__attributes%2Fsidekiq%3A*N____include%3A*N______match*_type%3A_strict*N______attributes%3A*N________-_key%3A_messaging.sidekiq.job*_class*N____actions%3A*N______-_key%3A_resource.name*N________from*_attribute%3A_messaging.sidekiq.job*_class*N________action%3A_upsert*N__tail*_sampling%3A*N____policies%3A*N______%5B*N________%7B*N__________name%3A_errors-policy%2C*N__________type%3A_status*_code%2C*N__________status*_code%3A_%7B_status*_codes%3A_%5BERROR%5D_%7D%2C*N________%7D%2C*N________%7B*N__________name%3A_randomized-policy%2C*N__________type%3A_probabilistic%2C*N__________probabilistic%3A_%7B_sampling*_percentage%3A_0.1_%7D%2C*N________%7D%2C*N______%5D*N*Nconnectors%3A*N__datadog%2Fconnector%3A*N____traces%3A*N______compute*_stats*_by*_span*_kind%3A_true*N*Nexporters%3A*N__datadog%3A*N____api%3A*N______site%3A_*S%7BDD*_SITE%7D*N______key%3A_*S%7BDD*_API*_KEY%7D*N____traces%3A*N______compute*_stats*_by*_span*_kind%3A_true*N______trace*_buffer%3A_500*N*Nservice%3A*N__pipelines%3A*N____traces%2Fall%3A*N______receivers%3A_%5Botlp%5D*N______processors%3A*N________%5B*N__________resource%2C*N__________k8sattributes%2C*N__________resourcedetection%2C*N__________transform%2C*N__________attributes%2Fsidekiq%2C*N__________batch%2C*N________%5D*N______exporters%3A_%5Bdatadog%2Fconnector%5D*N____traces%2Fsample%3A*N______receivers%3A_%5Bdatadog%2Fconnector%5D*N______processors%3A_%5Btail*_sampling%2C_batch%5D*N______exporters%3A_%5Bdatadog%5D*N____metrics%3A*N______receivers%3A_%5Bdatadog%2Fconnector%2C_otlp%5D*N______processors%3A*N________%5Bresource%2C_k8sattributes%2C_resourcedetection%2C_transform%2C_batch%5D*N______exporters%3A_%5Bdatadog%5D*N____logs%3A*N______receivers%3A_%5Botlp%5D*N______processors%3A*N________%5B*N__________resource%2C*N__________k8sattributes%2C*N__________resourcedetection%2C*N__________transform%2C*N__________attributes%2Fsidekiq%2C*N__________batch%2C*N________%5D*N______exporters%3A_%5Bdatadog%5D%7E
