Part 4 — Instrument Now, Collect Later: OpenTelemetry-Ready .NET with Only the BCL

ActivitySource, Meter, and ILogger ship in the .NET runtime, not in OpenTelemetry. Instrument your app with them today at almost zero cost, read it back with built-in tooling, and bolt on a backend whenever you are ready.

Part 4 of 4 in a series on observing a .NET system with OpenTelemetry and SigNoz (series index). Parts 1–3 wired the collection side. This post is about the instrumentation side, and why it is a separate decision.

Parts 1 through 3 spent a lot of words on collection: a Collector, OTLP, exporters, SigNoz. That can leave the wrong impression — that instrumenting your code and choosing a backend are one project you take on all at once.

They are two decisions, and you can make them years apart. The API you instrument with — ActivitySource for spans, Meter for metrics, ILogger for logs — is part of the .NET runtime. OpenTelemetry is one consumer of that API, attached later. This post is about writing the instrumentation well now, so that "turn on OpenTelemetry" is a one-line change whenever it comes.

You have already seen these primitives at work: the custom spans and metrics in Parts 1 and 2 are nothing but ActivitySource and Meter. This post is the proper treatment of them — how to use each one well, and why they cost almost nothing until something listens. It reads fine on its own, so it works as a primer before the rest of the series or as the foundation underneath it.

What you'll learn

Why instrumenting with the BCL costs almost nothing until something listens
How to write custom spans (ActivitySource), with tags, events, and status
How nesting spans gives you a whole-stack duration breakdown for free
The four metric instruments and when each one fits
Structured logging that stays queryable, with scopes
How to read all of it back with dotnet-counters and a tiny ActivityListener — no backend
The one-line change that later ships it to SigNoz

Why this is almost free

The reason you can instrument early is that the primitives do nothing until something subscribes to them.

An ActivitySource creates an Activity (a span) only when an ActivityListener is registered and samples it. With no listener, StartActivity returns null and allocates nothing. Your activity?.SetTag(...) calls become no-ops on a null reference.
A Meter instrument records into nothing until a MeterListener (or a tool like dotnet-counters) subscribes. counter.Add(1) with no subscriber is a couple of cheap checks, not an allocation or an I/O.
ILogger is the exception: it always runs, but it writes only to the providers you configured. In a fresh console app that is the console, or nothing.

So the cost of instrumenting code you might never collect is close to zero. That changes the economics: you instrument as you write the feature, while you still remember what matters, instead of bolting it on during an incident.

The rest of this post writes a small order-processing loop with exactly these primitives. The full program is in the appendix — a console app with no NuGet package and, importantly, no OpenTelemetry reference.

Custom spans with ActivitySource

A span is one timed operation. You create them from a single ActivitySource, made once for the whole process:

private static readonly ActivitySource Source = new("Sample.Primer", "1.0.0");

To time a piece of work, start an activity and let using end it:

using var order = Source.StartActivity("process-order");
order?.SetTag("order.id", id);                 // a queryable dimension on the span

Three things go on a span, all null-safe so they cost nothing when nobody is listening:

Tags — order?.SetTag("order.id", id) — key/value context you can later filter and group by.
Status — order?.SetStatus(ActivityStatusCode.Error, "order failed at persist") — marks the span failed without throwing, the same trick the API in Part 2 used for an out-of-stock purchase.
Events — order?.AddEvent(new ActivityEvent("retry-scheduled")) — a timestamped marker for something that happened inside the span.

The order?. is not defensive habit, it is the design. When there is no listener, StartActivity returns null and every line above is skipped.

Tags vs. events vs. logs. A tag is a property of the whole span (order.id). An event is a moment within it (retry-scheduled). A log is a standalone record that happens to be correlated. Reach for a tag first — it is what you filter and group on.

Whole-stack durations are just nested spans

Start one span inside another and .NET links them through Activity.Current. The child knows its parent, so the set of spans for one operation forms a tree — a per-layer stopwatch you did not have to wire up:

using var order = Source.StartActivity("process-order");
Step("validate");     // each Step starts a child span under "process-order"
Step("charge");
Step("persist");

Printed back, that is a waterfall: the parent's duration on top, each child's beneath it. It is the same shape as the distributed trace in Part 2, except there the "children" also crossed into other services. Local methods or remote calls, the mechanism is identical, and you read where the time went by subtracting a child from its parent.

This is the answer to "measure the whole stack of method durations": nest spans around the methods you care about. One honest limit, the same one Part 2 noted: the tree starts when your server code starts. The time a page spends rendering in the browser needs client-side instrumentation (RUM), which is outside the .NET process and outside this series.

Timing without a stopwatch object

When you do want a raw duration — to record a metric, say — reach for Stopwatch's static timestamp API rather than allocating a Stopwatch:

var startedAt = Stopwatch.GetTimestamp();
// ... work ...
var elapsedMs = Stopwatch.GetElapsedTime(startedAt).TotalMilliseconds;

GetTimestamp returns a long and allocates nothing, which matters on a hot path. Stopwatch.StartNew() is fine when you are not counting allocations; the API in Part 2 uses it per request. A span already records its own duration, so only measure by hand when you need the number for something else, like the histogram below.

Metrics: four instruments, four jobs

Metrics come from a Meter, created once like the ActivitySource:

private static readonly Meter Meter = new("Sample.Primer", "1.0.0");

There are four instrument kinds, and picking the right one is most of the skill:

Counter<T> — a value that only goes up: requests, orders, errors. You chart its rate. meter.CreateCounter<long>("primer.orders.processed").
UpDownCounter<T> — a value that rises and falls: items in a queue, active circuits. primer.queue.depth. Part 1 used one for live Blazor circuits.
Histogram<T> — a distribution you want percentiles of: durations, payload sizes. primer.work.duration, charted as p50/p95/p99.
ObservableGauge<T> — a value you sample on demand via a callback, when there is no natural moment to record it: current temperature, current worker count.

Add dimensions with a TagList, and keep them low-cardinality. outcome (a handful of values) is a good tag; order.id (unbounded) is not — every distinct value is a new time series, and high-cardinality tags are how you melt a metrics backend.

var tags = new TagList { { "outcome", failed ? "failed" : "ok" } };
workDuration.Record(elapsedMs, tags);
ordersProcessed.Add(1, tags);

Name instruments in a stable, dotted namespace (primer.orders.processed) and set a unit. Those names are your public contract once dashboards and alerts depend on them, so it is worth getting them right while there is no listener and nothing to break.

Structured logging that stays queryable

ILogger is already in your app. The one rule that makes its output useful later is to log with message templates, not interpolated strings:

log.LogInformation("processed in {ElapsedMs:F0}ms", elapsedMs);   // good
log.LogInformation($"processed in {elapsedMs:F0}ms");             // loses the field

The first keeps ElapsedMs as a named value a backend can filter and aggregate on. The second bakes it into text, and now you are writing substring searches.

A scope attaches the same context to every line written inside it:

using var scope = log.BeginScope("order {OrderId}", id);
// every log line in here carries OrderId, structured, not just in the text

And because Parts 1–3 turn on the OpenTelemetry logging provider, a log written while a span is active is automatically stamped with that span's trace_id — so a log line links back to the exact operation that produced it. That correlation is configured once in AddObservability (IncludeScopes, ParseStateValues), not per call site.

See it with no backend at all

Here is the part that proves the point. The order loop above, running as a plain console app with no OpenTelemetry anywhere, is already observable — because the listener primitives OpenTelemetry uses are in the box.

Metrics, with dotnet-counters. Install the tool (dotnet tool install -g dotnet-counters), then point it at the running process:

dotnet-counters monitor -n InstrumentationPrimer --counters Sample.Primer

The dotnet-counters command-line tool reading our custom meter live. Every instrument is here -- the order counter split by outcome, the up/down queue depth, the duration percentiles, and the worker gauge -- with no exporter and no backend running.

Spans, with a tiny ActivityListener. A dozen lines subscribe to your ActivitySource and hand you every span as it ends. OpenTelemetry registers a richer version of exactly this:

using var listener = new ActivityListener
{
    ShouldListenTo = s => s.Name == "Sample.Primer",
    Sample = (ref ActivityCreationOptions<ActivityContext> _) => ActivitySamplingResult.AllData,
    ActivityStopped = a => Console.WriteLine($"{a.DisplayName} {a.Duration.TotalMilliseconds:F1} ms"),
};
ActivitySource.AddActivityListener(listener);

With that registered, StartActivity starts returning real activities, and the sample prints each order as a waterfall:

The sample's own console output. A 12-line ActivityListener prints each order's spans as a waterfall -- process-order with its validate, charge, and persist children -- and the failed third order shows its error status. No OpenTelemetry involved.

There is a MeterListener too, the metrics equivalent, if you want to consume measurements in-process. dotnet-trace and dotnet-monitor are the next rung up — they capture the same ActivitySource and Meter data over the diagnostics socket without a line of OpenTelemetry. The point of all of them: your instrumentation is real and readable long before you choose a backend.

Turning collection on later

When you do want it in SigNoz, you do not touch any of the code above. You register the source and meter names with the shared bootstrap from Part 1:

builder.AddObservability("orders-service", options =>
{
    options.ActivitySources.Add("Sample.Primer");
    options.Meters.Add("Sample.Primer");
});

That attaches OpenTelemetry's listener and an OTLP exporter. The spans, metrics, and logs you already wrote start flowing. Nothing in the instrumentation changed.

This separation is visible in the project files of the demo system. The Blazor frontend and the background worker define spans and metrics, yet their .csproj files reference zero OpenTelemetry packages — every span and counter is System.Diagnostics. Only the shared Shared.Telemetry project, which owns AddObservability, references OpenTelemetry, and only the composition root pulls it in. Instrumentation lives in your domain code; the exporter is a detail at the edge.

Best-practices checklist

One static readonly ActivitySource and one Meter per library or service. Create once; they are thread-safe.
Keep instrumentation in the domain code. Keep exporters at the composition root, behind something like AddObservability.
Always activity?. — assume there is no listener.
Stable, dotted instrument and span names with units. They are a contract once dashboards depend on them.
Tags are low-cardinality by default. No unbounded values (IDs, emails, raw input) on metrics.
Log with message templates, never interpolation. Add a scope for per-operation context.
For short-lived processes, flush on exit so the last batch is not lost (Part 3's tracer_provider.shutdown() note applies to .NET's exporter too).

Wrapping up

Instrumentation and collection are different jobs. ActivitySource, Meter, and ILogger let you do the first one now, in plain .NET, at almost no cost and with no commitment to a vendor — and you can already read it back with dotnet-counters and a few lines of ActivityListener. When you are ready for the second job, the one-line AddObservability from this series sends everything you already wrote to SigNoz.

That closes the series: Part 1 (Blazor Server and the shared foundation), Part 2 (a C# API down to the SQL), Part 3 (background jobs and non-.NET work), and this one on the instrumentation underneath them all. The full series is on the index page.

The complete code

A self-contained console app that demonstrates every primitive above. It has no NuGet dependency and no OpenTelemetry reference: ActivitySource and Meter are in the runtime, and the console logger comes from the shared framework.

Run dotnet run and it loops; dotnet run -- --once processes three orders and exits. In another terminal, dotnet-counters monitor -n InstrumentationPrimer --counters Sample.Primer shows the metrics.

samples/InstrumentationPrimer/InstrumentationPrimer.csproj

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>net10.0</TargetFramework>
    <Nullable>enable</Nullable>
    <ImplicitUsings>enable</ImplicitUsings>
    <AssemblyName>InstrumentationPrimer</AssemblyName>
    <RootNamespace>InstrumentationPrimer</RootNamespace>
  </PropertyGroup>

  <!--
    The whole point of this sample: there is no OpenTelemetry package here.
    ActivitySource and Meter are part of the .NET runtime. ILogger's console
    provider lives in the shared framework, so a FrameworkReference pulls it in
    with no NuGet dependency at all.
  -->
  <ItemGroup>
    <FrameworkReference Include="Microsoft.AspNetCore.App" />
  </ItemGroup>

</Project>

samples/InstrumentationPrimer/Program.cs

using System.Diagnostics;
using System.Diagnostics.Metrics;
using Microsoft.Extensions.Logging;

// InstrumentationPrimer -- prepare a .NET app for OpenTelemetry using only the BCL.
//
// ActivitySource (spans) and Meter (metrics) are part of the .NET runtime, not OpenTelemetry.
// They are no-ops until something *listens*: with no ActivityListener, StartActivity returns
// null and allocates nothing; with no MeterListener (or `dotnet-counters`), instruments
// aggregate nothing. So you can instrument now and decide on a backend later.

// One ActivitySource and one Meter for the whole process. Both are thread-safe; create once.
var source = new ActivitySource("Sample.Primer", "1.0.0");
var meter = new Meter("Sample.Primer", "1.0.0");

// Four instrument kinds, one of each, so you can see when each one fits.
var ordersProcessed = meter.CreateCounter<long>(
    "primer.orders.processed", unit: "{order}", description: "Orders processed, tagged by outcome.");
var queueDepth = meter.CreateUpDownCounter<long>(
    "primer.queue.depth", unit: "{order}", description: "Orders currently being processed.");
var workDuration = meter.CreateHistogram<double>(
    "primer.work.duration", unit: "ms", description: "Wall-clock duration of one order.");

var active = 0L;
meter.CreateObservableGauge(
    "primer.workers.active", () => Interlocked.Read(ref active),
    unit: "{worker}", description: "Workers currently inside ProcessOrder, sampled on collect.");

// A ~12-line ActivityListener is all it takes to SEE spans without any backend.
// OpenTelemetry attaches a richer version of exactly this. We buffer each order's spans
// and print them as a top-down waterfall when the root span stops.
var spans = new List<Activity>();
using var listener = new ActivityListener
{
    ShouldListenTo = s => s.Name == "Sample.Primer",
    Sample = (ref ActivityCreationOptions<ActivityContext> _) => ActivitySamplingResult.AllData,
    ActivityStopped = a =>
    {
        spans.Add(a);
        if (a.Parent is null) { RenderWaterfall(spans); spans.Clear(); }
    },
};
ActivitySource.AddActivityListener(listener);

// Structured logging straight to the console. Message templates, not string interpolation,
// so each field stays a queryable value rather than baked-into text.
using var loggerFactory = LoggerFactory.Create(b => b
    .SetMinimumLevel(LogLevel.Information)
    .AddSimpleConsole(o => { o.SingleLine = true; o.IncludeScopes = true; o.TimestampFormat = "HH:mm:ss "; }));
var log = loggerFactory.CreateLogger("primer");

var once = args.Contains("--once");
var sleep = new Random(7);

Console.WriteLine("InstrumentationPrimer: spans print below; metrics are live in `dotnet-counters`. Ctrl+C to stop.\n");

for (var id = 1; once ? id <= 3 : true; id++)
{
    ProcessOrder(id);
    if (once) continue;
    Thread.Sleep(1500);
}

void ProcessOrder(int id)
{
    Interlocked.Increment(ref active);
    queueDepth.Add(1);
    var startedAt = Stopwatch.GetTimestamp();   // allocation-free; no Stopwatch object

    // A logging scope stamps the order id onto every log line written inside it. The "{OrderId}"
    // placeholder stays a structured value, so a backend can filter on it, not just match text.
    using var scope = log.BeginScope("order {OrderId}", id);

    // The parent span. Nesting child spans under it builds a method-duration waterfall for free.
    using var order = source.StartActivity("process-order");
    order?.SetTag("order.id", id);

    Step("validate", 10, 25);
    Step("charge", 20, 55);

    var failed = id % 3 == 0;   // deterministic: every third order fails at persist
    if (failed)
    {
        using var persist = source.StartActivity("persist");
        Work(15, 30);
        persist?.AddEvent(new ActivityEvent("retry-scheduled"));     // a point-in-time marker on the span
        persist?.SetStatus(ActivityStatusCode.Error, "write conflict");
        order?.SetStatus(ActivityStatusCode.Error, "order failed at persist");
        log.LogWarning("failed at persist");
    }
    else
    {
        Step("persist", 15, 30);
    }

    var elapsedMs = Stopwatch.GetElapsedTime(startedAt).TotalMilliseconds;
    var tags = new TagList { { "outcome", failed ? "failed" : "ok" } };
    workDuration.Record(elapsedMs, tags);
    ordersProcessed.Add(1, tags);

    queueDepth.Add(-1);
    Interlocked.Decrement(ref active);
    if (!failed) log.LogInformation("processed in {ElapsedMs:F0}ms", elapsedMs);

    void Step(string name, int lo, int hi)
    {
        using var span = source.StartActivity(name);
        Work(lo, hi);
    }

    void Work(int lo, int hi) => Thread.Sleep(sleep.Next(lo, hi));
}

// Print one order's spans as an indented, top-down waterfall.
void RenderWaterfall(List<Activity> recorded)
{
    foreach (var a in recorded.OrderBy(s => s.StartTimeUtc))
    {
        var depth = 0;
        for (var p = a.Parent; p is not null; p = p.Parent) depth++;

        var indent = new string(' ', depth * 3);
        var branch = depth == 0 ? "" : "|- ";
        var error = a.Status == ActivityStatusCode.Error ? $"   [ERROR: {a.StatusDescription}]" : "";
        var id = a.GetTagItem("order.id");
        var label = id is null ? "" : $"   (order {id})";

        Console.WriteLine($"{indent}{branch}{a.DisplayName,-13}{a.Duration.TotalMilliseconds,7:F1} ms{label}{error}");
    }

    Console.WriteLine();
}

That is the series. Part 1 — Blazor Server observability carries the shared foundation; Part 2 — full-stack C# API observability follows a request to the database; Part 3 — on-prem background jobs brings in everything non-.NET. The index page ties them together.