Sendry — Event-Driven API Observability Platform

Overview

Sendry is a self-hosted API observability platform that captures, processes, and visualizes high-throughput API metrics — without adding any latency to the applications it monitors.

Most monitoring tools either cost too much, require a dedicated DevOps team to operate, or slow down the very systems they're supposed to observe. Sendry was built to solve all three of those problems at once.

The core idea is simple: telemetry should never block the request path. By separating ingestion from processing, Sendry can ingest events in under 2ms and handle everything else — aggregation, storage, dashboards — asynchronously in the background.

The Problem

Existing observability solutions fall into three frustrating categories:

Expensive SaaS tools that charge per event at scale, with monthly bills that often exceed infrastructure costs themselves.

Complex open-source stacks like Prometheus or ELK that demand dedicated DevOps resources and significant operational overhead.

Synchronous instrumentation that writes to databases during the request lifecycle, adding measurable latency to production applications.

What most engineering teams actually need is simple: which endpoints are slow, where errors are spiking, and which services are degrading — without paying premium rates or running a separate ops team to get those answers.

How It Works

Sendry sits between an application and its metrics. A lightweight SDK middleware captures request data and sends it to the ingestion API after the HTTP response is already delivered to the user — so there is zero impact on application performance.

From there, the pipeline takes over:

1. Ingest API validates the API key and publishes the event to RabbitMQ
2. Returns HTTP 202 immediately — no waiting
3. Background consumer pulls from the queue independently
4. Event is deduplicated, validated, and written to both databases
5. Dashboard queries serve from pre-aggregated PostgreSQL tables

The result is a system that handles 10,000+ events per second on commodity hardware, with ingestion latency under 2ms.

Architecture

What I Built

Lightweight SDK Middleware

A minimal Express.js middleware that captures request method, endpoint, status code, latency, IP, and payload sizes — then sends telemetry asynchronously after the response is delivered. Zero performance impact by design.

Event-Driven Ingestion Pipeline

The ingestion API never touches a database. It validates the API key, publishes to RabbitMQ, and immediately returns 202 Accepted. All processing happens downstream, independently of the request.

Circuit Breaker

Both the ingestion layer and consumer implement a three-state circuit breaker. If RabbitMQ goes down, the API returns 503 immediately instead of hanging. Infrastructure failures never cascade into application failures.

Idempotent Message Processing

RabbitMQ guarantees at-least-once delivery, which means duplicates can happen. Every event gets a unique hash, and an in-memory deduplication cache prevents duplicate writes to the database. Analytics stay accurate even under retry conditions.

Dual Database Architecture

MongoDB stores raw event payloads with a 30-day TTL — schemaless, write-optimized, auto-expiring. PostgreSQL stores hourly pre-aggregated metrics — read-optimized, fast for dashboard queries. The separation means dashboard queries complete in 50–100ms regardless of total event volume.

Real-Time Dashboard

A React dashboard built with TanStack Query and ApexCharts that shows request volume, error rates, p50/p95/p99 latency, top endpoints by traffic, and service health — all served from pre-aggregated tables.

Multi-Tenant Support

Each organization gets isolated API keys, separate metric aggregations, and namespaced data. The architecture is billing-ready with per-customer event counting built in.

Technical Challenges

Making failure invisible to users

The hardest constraint was that a monitoring system should never affect the system it monitors. If Sendry's queue goes down, the monitored application should keep running normally.

The circuit breaker solved this: when the queue is unhealthy, the ingestion layer fails fast with a 503 instead of blocking the caller. From the application's perspective, sending telemetry is a fire-and-forget operation.

Handling duplicates without database overhead

Message brokers retry on failure, which can deliver the same event more than once. Writing a deduplication check to the database on every event would undo the performance gains of async processing.

The solution was an in-memory Set capped at 100,000 entries using an LRU eviction strategy. Deduplication happens in memory with no database round-trip, keeping throughput high.

Keeping dashboard queries fast at scale

Scanning raw events for every dashboard request gets expensive fast. The consumer aggregates events into hourly PostgreSQL buckets during processing — not at query time. Dashboard queries never touch raw event tables, so they stay fast regardless of how many total events exist.

Cross-domain cookies in development

When the frontend and API run on different ports locally, browsers block session cookies due to SameSite policies. This was solved with environment-aware cookie configuration: SameSite=Lax in development, SameSite=None; Secure in production — switching automatically based on NODE_ENV.

Key Results

• Sub-2ms telemetry ingestion, zero impact on monitored application latency
• 10,000+ events per second on commodity hardware
• Dashboard queries complete in 50–100ms at any event volume
• Graceful degradation — monitoring failures never reach production users
• Self-hosted, with no per-event SaaS pricing

What I Learned

Building Sendry forced me to think carefully about system boundaries and failure modes in a way that most web projects don't require.

The most important lesson: decoupling is not just about scalability. It's about resilience. When each layer fails independently and predictably, the whole system becomes much easier to debug, operate, and extend.

The dual-database architecture showed me that picking the right tool for each access pattern — rather than forcing one database to do everything — has a real, measurable impact on performance. And circuit breakers taught me that explicit failure handling is always better than letting failures propagate silently through a system.