Configure OpenTelemetry

📌 If you have any questions shoot us an email or join us on Discord! 💜

Overview

---

Multiplayer leverages OpenTelemetry for two features:

• System Auto-Documentation: When enabled, it automatically discovers all components, APIs, dependencies, platforms, and environments within your software system and lists them within the System Dashboard.
• Platform Debugger: Record and share deep session replays that include relevant data from frontend screens to distributed traces metrics and logs from your backend platform.

ℹ️ These two features share the same set up window and the following OpenTelemetry configuration steps.

1. Set up the OpenTelemetry Integration

---

• Open your project
• In the left-side menu, click "System" to open the System Dashboard
• Click "Enable Auto-Docs + Debugger"
• Name your integration
• Hit "Create"
• An OpenTelemetry (OTel) token will be created for your OTel integration

If you would like Auto-Documentation to automatically create newly detected components that are not currently present in your Multiplayer project, select:

• The name of the platform from the drop down menu where the new components will be auto-created
• Toggle on "Auto-add"

ℹ️ If a platform is not selected, the new components are added to the project (i.e. Components tab) but not linked to a specific Platform. If the platform is selected the components are created in the project and also added to the platform. ℹ️

Once you click “Confirm”, we will proceed with the Multiplayer OTel integration and start automatically documenting your system and adding backend data (e.g. traces and logs) to your Platform Debugger sessions.

Please note that if you close the set up window before finishing the set up (or before copying the generated tokens), you will no longer be able to see the Otel tokens. You need to delete the integration and start again to generate new OTel tokens.

2. Configuring OTel

---

JavaScript (web)

1. Install session debugger library with following command:

  npm i -S @multiplayer-app/session-debugger

2. Initialize session debugger library:

  // NOTE: this import should be done before all other imports in your main file.
  import SessionDebugger from "@multiplayer-app/session-debugger";

  SessionDebugger.init({
    version: "<web-app-version>",
    application: "<web-app-name>",
    environment: "<web-app-environment>",
    apiKey: "<multiplayer-key>",
    canvasEnabled: true,
    showWidget: true,
    ignoreUrls: [
      /https:\/\/domain\.to\.ignore\/.*/, // can be regex or string
      /https:\/\/another\.domain\.to\.ignore\/.*/,
    ],
    // NOTE: if frontend domain doesn't match to backend one, set backend domain to `propagateTraceHeaderCorsUrls` parameter
    propagateTraceHeaderCorsUrls: [
      new RegExp("https://your.backend.api.domain", "i"), // can be regex or string
      new RegExp("https://another.backend.api.domain", "i")
    ],
    schemifyDocSpanPayload: true,
    maskDebSpanPayload: true,
    docTraceRatio: 0.15 // 15% of traces will be sent for auto-documentation
  });

Node.Js

1. Install required packages run command:

  npm i -S @multiplayer-app/otlp-core \
    @opentelemetry/api \
    @opentelemetry/auto-instrumentations-node \
    @opentelemetry/core \
    @opentelemetry/exporter-trace-otlp-http \
    @opentelemetry/resources \
    @opentelemetry/sdk-node \
    @opentelemetry/sdk-trace-base \
    @opentelemetry/semantic-convention \
    @opentelemetry/sdk-logs \
    @opentelemetry/api-logs

2. Create a file opentelemetry.js. Put following content into it:

  import { hostname } from "os"
  import { node, NodeSDK } from "@opentelemetry/sdk-node"
  import {
    BatchSpanProcessor,
    ParentBasedSampler,
  } from "@opentelemetry/sdk-trace-base"
  import {
    getNodeAutoInstrumentations,
    getResourceDetectors,
  } from "@opentelemetry/auto-instrumentations-node"
  import { detectResourcesSync, Resource } from "@opentelemetry/resources"
  import {
    ATTR_SERVICE_NAME,
    ATTR_SERVICE_VERSION,
    SEMRESATTRS_HOST_NAME,
    SEMRESATTRS_DEPLOYMENT_ENVIRONMENT,
    SEMRESATTRS_PROCESS_RUNTIME_VERSION,
    SEMRESATTRS_PROCESS_PID,
  } from "@opentelemetry/semantic-conventions"
  import api from "@opentelemetry/api"
  import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-http"
  import { W3CTraceContextPropagator } from "@opentelemetry/core"
  import {
    MultiplayerHttpTraceExporterNode,
    MultiplayerTraceIdRatioBasedSampler,
    MultiplayerIdGenerator,
    MultiplayerFilterTraceExporter,
    MultiplayerHttpInstrumentationHooks,
    MultiplayerHttpLogExporterNode,
  } from "@multiplayer-app/otlp-core"
  import { LoggerProvider, BatchLogRecordProcessor } from "@opentelemetry/sdk-logs"
  import apiLogs from "@opentelemetry/api-logs"
  // import { OTLPLogExporter } from '@opentelemetry/exporter-logs-otlp-http'

  // NOTE: Change those variables before testing
  const SERVICE_NAME = "<example-service-name>"
  const SERVICE_VERSION = "<service-version>"
  const PLATFORM_ENV = "<environment-name>"
  const MULTIPLAYER_OTLP_KEY = "<multiplayer-key>"

  // NOTE: Update instrumentation configuration as needed
  // For more see: https://www.npmjs.com/package/@opentelemetry/auto-instrumentations-node
  const instrumentations = [
    getNodeAutoInstrumentations({
      "@opentelemetry/instrumentation-http": {
        requestHook: MultiplayerHttpInstrumentationHooks.requestHook({
          headersToMask: ["X-Api-Key"],
          maxPayloadSize: 5000,
          schemifyDocSpanPayload: true
          maskDebSpanPayload: true
        }),
        responseHook: MultiplayerHttpInstrumentationHooks.responseHook({
          headersToMask: ["X-Api-Key"],
          maxPayloadSize: 5000,
          schemifyDocSpanPayload: true
          maskDebSpanPayload: true
        }),
      },
    }),
  ]

  const getResource = () => {
    const resourcesWithDetectors = detectResourcesSync({
      detectors: getResourceDetectors(),
    })
    const resourceWithAttributes = new Resource({
      [ATTR_SERVICE_NAME]: SERVICE_NAME,
      [ATTR_SERVICE_VERSION]: SERVICE_VERSION,
      [SEMRESATTRS_HOST_NAME]: hostname(),
      [SEMRESATTRS_DEPLOYMENT_ENVIRONMENT]: PLATFORM_ENV,
      [SEMRESATTRS_PROCESS_RUNTIME_VERSION]: process.version,
      [SEMRESATTRS_PROCESS_PID]: process.pid,
    })
    const resource = resourceWithAttributes.merge(resourcesWithDetectors)

    return resource
  }

  const opentelemetry = () => {
    // NOTE: Either use MultiplayerHttpTraceExporterNode or OTLPTraceExporter
    const multiplayerTraceExporter = new MultiplayerHttpTraceExporterNode({
      apiKey: MULTIPLAYER_OTLP_KEY,
    })
    // const multiplayerTraceExporter = new OTLPTraceExporter({
    //   url: 'https://api.multiplayer.app/v1/traces',
    //   headers: {
    //     Authorization: MULTIPLAYER_OTLP_KEY
    //   },
    // })

    // NOTE: if you want to send traces to one more OpenTelemetry provider
    // uncomment lines below use these variables: `apmProviderTraceExporter`, `apmProviderTraceExporter`
    // const apmProviderTraceExporter = new OTLPTraceExporter({
    //   url: 'http://some.collector.url/v1/traces',
    // })
    // NOTE: Use MultiplayerFilterTraceExporter exporter wrapper to remove Multiplayer attributes (starts with `multiplayer.*` prefix)
    // before sending traces to your APM provider
    // const apmFilteredTraceExporter = new MultiplayerFilterTraceExporter(apmProviderTraceExporter) 
    const resource = getResource()

    const provider = new node.NodeTracerProvider({
      resource,
      spanProcessors: [
        // new BatchSpanProcessor(apmFilteredTraceExporter),
        new BatchSpanProcessor(multiplayerTraceExporter),
      ],
      sampler: new ParentBasedSampler({
        // NOTE: this config will send 80% of all traces + 100% of debug traces + 30% of 80% document traces
        root: new MultiplayerTraceIdRatioBasedSampler(0.8),
      }),
      // NOTE: this will set 30% of the traces sampled above for auto documentation
      idGenerator: new MultiplayerIdGenerator({ autoDocTracesRatio: 0.3 }),
    })

    // Initialise logger provider and exporting logs to collector
    const loggerProvider = new LoggerProvider({
      resource,
    })

    // NOTE: Either use `MultiplayerHttpLogExporterNode` or `OTLPLogExporter`
    const multiplayerLogExporter = new MultiplayerHttpLogExporterNode({
      apiKey: MULTIPLAYER_OTLP_KEY,
    })
    // const multiplayerLogExporter = new OTLPLogExporter({
    //   url: 'https://api.multiplayer.app/v1/logs',
    //   headers: {
    //     Authorization: MULTIPLAYER_OTLP_KEY
    //   },
    // })

    const logRecordProcessor = new BatchLogRecordProcessor(multiplayerLogExporter)
    loggerProvider.addLogRecordProcessor(logRecordProcessor)
    apiLogs.logs.setGlobalLoggerProvider(loggerProvider)

    provider.register()
    api.trace.setGlobalTracerProvider(provider)
    api.propagation.setGlobalPropagator(new W3CTraceContextPropagator())

    const sdk = new NodeSDK({
      instrumentations,
    })

    sdk.start()
  }

  opentelemetry()

3. Import opentelemetry.js into your main app file (for example index.js).

  // Note: import `opentelemetry.js` at the top your main app file.
  import "./opentelemetry.js"

  import http from "http"
  const httpServer = http.createServer(app)
  // ...
  httpServer.listen(3000)

If you're using common.js syntax, here is example:

const { hostname } = require("os")
const { node, NodeSDK } = require("@opentelemetry/sdk-node")
const {
 BatchSpanProcessor,
 ParentBasedSampler,
} = require("@opentelemetry/sdk-trace-base")
const {
 getNodeAutoInstrumentations,
 getResourceDetectors
} = require("@opentelemetry/auto-instrumentations-node")
const { detectResourcesSync, Resource } = require("@opentelemetry/resources")
const {
 ATTR_SERVICE_NAME,
 ATTR_SERVICE_VERSION,
 ATTR_HOST_NAME,
 SEMRESATTRS_DEPLOYMENT_ENVIRONMENT,
 SEMRESATTRS_PROCESS_RUNTIME_VERSION,
 SEMRESATTRS_PROCESS_PID,
} = require("@opentelemetry/semantic-conventions")
const api = require("@opentelemetry/api")
const { OTLPTraceExporter } = require("@opentelemetry/exporter-trace-otlp-http")
const { W3CTraceContextPropagator } = require("@opentelemetry/core")
const {
 MultiplayerHttpTraceExporterNode,
 MultiplayerTraceIdRatioBasedSampler,
 MultiplayerIdGenerator,
 MultiplayerFilterTraceExporter,
 MultiplayerHttpInstrumentationHooks,
 MultiplayerHttpLogExporterNode
} = require("@multiplayer-app/otlp-core")
const { LoggerProvider, BatchLogRecordProcessor } = require("@opentelemetry/sdk-logs")
const apiLogs = require("@opentelemetry/api-logs")
// const { OTLPLogExporter } = require('@opentelemetry/exporter-logs-otlp-http')

// NOTE: Change those variables before testing
const SERVICE_NAME = "<example-service-name>"
const SERVICE_VERSION = "<service-version>"
const PLATFORM_ENV = "<environment-name>"
const MULTIPLAYER_OTLP_KEY = "<multiplayer-key>"

// NOTE: Update instrumentation configuration as needed
// For more see: https://www.npmjs.com/package/@opentelemetry/auto-instrumentations-node
const instrumentations = [
 getNodeAutoInstrumentations({
   "@opentelemetry/instrumentation-http": {
     // Add http request payload and headers to span
     requestHook: MultiplayerHttpInstrumentationHooks.requestHook({
       headersToMask: ["X-Api-Key"],
       maxPayloadSize: 5000,
       schemifyDocSpanPayload: true
       maskDebSpanPayload: true
     }),
     // Add http response payload and headers to span
     responseHook: MultiplayerHttpInstrumentationHooks.responseHook({
       headersToMask: [],
       maxPayloadSize: 5000,
       schemifyDocSpanPayload: true
       maskDebSpanPayload: true
     })
   }
 }),
]

const getResource = () => {
 const resourcesWithDetectors = detectResourcesSync({
   detectors: getResourceDetectors()
 })
 const resourceWithAttributes = new Resource({
   [ATTR_SERVICE_NAME]: SERVICE_NAME,
   [ATTR_SERVICE_VERSION]: SERVICE_VERSION,
   [ATTR_HOST_NAME]: hostname(),
   [SEMRESATTRS_DEPLOYMENT_ENVIRONMENT]: PLATFORM_ENV,
   [SEMRESATTRS_PROCESS_RUNTIME_VERSION]: process.version,
   [SEMRESATTRS_PROCESS_PID]: process.pid
 })
 const resource = resourceWithAttributes.merge(resourcesWithDetectors)

 return resource
}

const opentelemetry = () => {
 // NOTE: Either use MultiplayerHttpTraceExporterNode or OTLPTraceExporter
 const multiplayerTraceExporter = new MultiplayerHttpTraceExporterNode({
   apiKey: MULTIPLAYER_OTLP_KEY,
 })
 // const multiplayerTraceExporter = new OTLPTraceExporter({
 //   url: 'https://api.multiplayer.app/v1/traces',
 //   headers: {
 //     Authorization: MULTIPLAYER_OTLP_KEY
 //   },
 // })

 // NOTE: if you want to send traces to one more OpenTelemetry provider
 // uncomment lines below use these variables: `apmProviderTraceExporter`, `apmProviderTraceExporter`
 // const apmProviderTraceExporter = new OTLPTraceExporter({
 //   url: 'http://some.collector.url/v1/traces',
 // })
 // NOTE: Use MultiplayerFilterTraceExporter exporter wrapper to remove Multiplayer attributes (starts with `multiplayer.*` prefix)
 // before sending traces to your APM provider
 // const apmFilteredTraceExporter = new MultiplayerFilterTraceExporter(apmProviderTraceExporter) 
 const resource = getResource()

 const provider = new node.NodeTracerProvider({
   resource,
   spanProcessors: [
     // new BatchSpanProcessor(apmFilteredTraceExporter),
     new BatchSpanProcessor(multiplayerTraceExporter),
   ],
   sampler: new ParentBasedSampler({
     // NOTE: this config will send 80% of all traces + 100% of debug traces + 30% of 80% document traces
     root: new MultiplayerTraceIdRatioBasedSampler(0.8),
   }),
   // NOTE: this will set 30% of the traces sampled above for auto documentation
   idGenerator: new MultiplayerIdGenerator({ autoDocTracesRatio: 0.3 }),
 })

 // Initialise logger provider and exporting logs to collector
 const loggerProvider = new LoggerProvider({
   resource,
 })

 // NOTE: Either use `MultiplayerHttpLogExporterNode` or `OTLPLogExporter`
 const multiplayerLogExporter = new MultiplayerHttpLogExporterNode({
   apiKey: MULTIPLAYER_OTLP_KEY,
 })
 // const multiplayerLogExporter = new OTLPLogExporter({
 //   url: 'https://api.multiplayer.app/v1/logs',
 //   headers: {
 //     Authorization: MULTIPLAYER_OTLP_KEY
 //   },
 // })

 const logRecordProcessor = new BatchLogRecordProcessor(multiplayerLogExporter)
 loggerProvider.addLogRecordProcessor(logRecordProcessor)
 apiLogs.logs.setGlobalLoggerProvider(loggerProvider)

 provider.register()
 api.trace.setGlobalTracerProvider(provider)
 api.propagation.setGlobalPropagator(new W3CTraceContextPropagator())

 const sdk = new NodeSDK({
   instrumentations
 })

 sdk.start()
}

opentelemetry()

module.exports.opentelemetry = opentelemetry

Go

  // Coming soon

Python

  # Coming soon

Ruby

  # Coming soon

.NET

  // Coming soon

Java

  // Coming soon

PHP

  // Coming soon

Rust

  // Coming soon

Swift

  // Coming soon

Next Steps

---

You did it! What’s next?

• Record a deep session replay of your bugs
• Visualize your system architecture