Building a Real-Time Dashboard at Scale: A Raptorzx Case Study

Real-time dashboards are the nervous system of modern machine learning operations. They surface model performance, data drift, and system health as they happen, enabling teams to react within seconds rather than hours. Yet building such a dashboard at scale is notoriously difficult: streaming data pipelines must handle high throughput, storage systems must balance speed with cost, and the user interface must remain responsive under load. In this guide, we walk through a composite case study inspired by the Raptorzx community, illustrating how one team approached these challenges. We will cover the architectural decisions, tooling trade-offs, and operational practices that made their dashboard both fast and reliable.

Why Real-Time Dashboards Fail at Scale

Many teams start with a simple polling approach: a backend service queries a database every few seconds and serves the results to a frontend chart. This works for small datasets but breaks down as data volume grows. The database becomes a bottleneck, queries take longer, and the dashboard feels sluggish. Worse, polling can mask underlying issues like data staleness or inconsistent aggregation windows.

The Latency Trap

In the composite case study, the team initially used a PostgreSQL database with a 5-second polling interval. As their event stream grew to over 10,000 events per second, the database CPU spiked to 90%, and queries took over 30 seconds to return. The dashboard became useless for real-time decisions. They realized that polling is fundamentally limited: it creates a tight coupling between query load and data freshness, and it wastes resources on repeated full scans.

Cost Explosion

Another common failure is cost. Storing every raw event in a hot storage tier and querying it repeatedly can drive cloud bills into the thousands of dollars per month. The team found that their initial approach—keeping all events in a time-series database with no downsampling—led to storage costs that grew linearly with data volume. They needed a strategy that separated hot, warm, and cold data tiers, and that pre-aggregated metrics where possible.

Inconsistent Semantics

A third pitfall is inconsistent aggregation. When different parts of the dashboard compute the same metric (e.g., average latency over the last minute) using different time windows or filtering logic, users lose trust. The team discovered that their frontend JavaScript was recalculating aggregates from raw data, while the backend API used a different window function. Aligning these semantics required a centralized aggregation layer.

These three problems—latency, cost, and inconsistency—are the main reasons real-time dashboards fail at scale. The solution lies in event-driven architectures that decouple ingestion from querying, and in careful trade-offs between freshness, accuracy, and expense.

Core Frameworks: Event-Driven Pipelines and Materialized Views

To overcome the limitations of polling, the team adopted an event-driven architecture. Instead of the frontend asking for data, the backend pushes updates whenever new events arrive. This is typically implemented using a message broker (like Apache Kafka or Amazon Kinesis) and a stream processor (like Apache Flink or Kafka Streams) that computes aggregates in real time.

Event-Driven Ingestion

The pipeline starts with a producer that publishes each event to a topic. The stream processor consumes these events, applies transformations (e.g., filtering, enrichment, windowed aggregation), and writes the results to a materialized view—a precomputed table that stores the latest values for each metric. The frontend then subscribes to updates from this view, either via WebSocket or by polling a lightweight cache (like Redis). This decouples the heavy computation from the query path.

Materialized Views vs. Ad-Hoc Queries

Materialized views are the key to low latency. Instead of running a complex SQL query over raw data each time a user loads the dashboard, the system precomputes the needed aggregates (e.g., count, average, p99 latency) and stores them in a fast key-value store. The team used Apache Druid for its ability to handle real-time ingestion and sub-second queries on pre-aggregated data. They also maintained a separate Redis cache for the most recent minute of data to avoid hitting Druid for every refresh.

Comparison of Three Approaches

The team chose the streaming + materialized views approach for its low latency and ability to scale horizontally. They accepted eventual consistency within a few seconds, which was acceptable for their use case of monitoring model drift and system health.

Execution: A Repeatable Workflow for Building the Dashboard

Building a real-time dashboard at scale is not a one-time effort; it requires an iterative process of design, implementation, testing, and refinement. The team followed a structured workflow that can be adapted by other teams.

Step 1: Define Metrics and SLAs

Start by listing the key metrics the dashboard must display (e.g., prediction latency, error rate, data volume). For each metric, define the required freshness (e.g., within 5 seconds) and accuracy (e.g., ±1% error). This drives architectural decisions: metrics that need sub-second freshness may require streaming, while others can tolerate batch updates.

Step 2: Choose the Stream Processor

The team evaluated Apache Flink, Kafka Streams, and Apache Spark Structured Streaming. Flink offered the best latency and state management for windowed aggregates; Kafka Streams was simpler but lacked advanced windowing; Spark was better for micro-batch but added seconds of latency. They chose Flink for its exactly-once semantics and support for event-time processing.

Step 3: Design the Materialized View Schema

For each metric, define the dimensions (e.g., model version, region) and the aggregation function (e.g., average, count, percentile). The team used a star schema in Druid, with a fact table for each metric and dimension tables for metadata. They pre-joined dimensions to avoid joins at query time.

Step 4: Implement the Data Pipeline

Write the Flink job to consume events from Kafka, apply windowed aggregations (e.g., 1-minute tumbling windows), and sink the results to Druid. Use Kafka Connect to ingest events from the source systems. Monitor lag and throughput using Kafka consumer lag metrics.

Step 5: Build the Frontend

The frontend should subscribe to updates from the materialized view. The team used a WebSocket server that polls Druid every second and pushes updates to the browser. They also implemented a fallback to HTTP polling for clients that do not support WebSocket. Use a charting library like D3.js or Chart.js with efficient update mechanisms (e.g., only redraw changed series).

Step 6: Test at Scale

Simulate production traffic using a load testing tool (e.g., Apache JMeter or Locust) that generates events at the expected peak rate. Monitor pipeline latency, dashboard responsiveness, and resource usage. The team found that their initial Flink job had a bottleneck in the sink operator; they resolved it by increasing parallelism and using batch writes to Druid.

Step 7: Iterate on User Feedback

After deployment, gather feedback from dashboard users. Common requests include adding new metrics, changing time windows, and improving drill-down capabilities. The team set up a process for adding new metrics by defining the aggregation logic in a configuration file, which automatically generates the Flink job and Druid schema.

Tools, Stack, and Economics

Choosing the right tools is critical for both performance and cost. The team's stack included Apache Kafka for message brokering, Apache Flink for stream processing, Apache Druid for materialized views, Redis for hot cache, and a React frontend with WebSocket support. Here we discuss the economic trade-offs.

Cost Breakdown

Running a streaming pipeline incurs costs for compute (Flink cluster), storage (Kafka, Druid, Redis), and data transfer. The team estimated that their monthly cloud bill for the dashboard component was around $2,500, with 40% going to Flink, 30% to Druid, 20% to Kafka, and 10% to Redis and networking. They optimized by using spot instances for Flink workers and by tuning retention policies in Kafka (7 days) and Druid (30 days hot, then move to cold storage).

When to Avoid Streaming

Not every dashboard needs real-time updates. If your metrics change slowly (e.g., daily model accuracy) or your data volume is low (e.g., <100 events per second), a simpler polling approach with a relational database may be more cost-effective. The team noted that their streaming pipeline added operational complexity that would not be justified for a small-scale project.

Maintenance Realities

Streaming pipelines require ongoing monitoring of consumer lag, checkpointing failures, and schema evolution. The team set up alerts for lag exceeding 10 seconds and for checkpoint duration exceeding 1 minute. They also implemented a schema registry to handle changes in event structure without breaking the pipeline. Regular maintenance tasks include upgrading Flink versions, tuning parallelism, and rotating Kafka topics.

Growth Mechanics: Scaling the Dashboard

As the team's ML platform grew, the dashboard needed to handle more metrics, more users, and higher query loads. Scaling a real-time dashboard involves both vertical and horizontal strategies.

Horizontal Scaling of Stream Processor

Flink allows scaling by increasing the parallelism of operators. The team started with 4 task slots and scaled to 16 as event volume grew. They also partitioned Kafka topics by metric type to allow independent processing of different streams. This avoided a single slow metric from blocking others.

Caching and Query Optimization

To reduce load on Druid, the team introduced a Redis cache for the most recent 60 seconds of data. The frontend first checks Redis; if the data is missing, it falls back to Druid. This cut Druid query load by 70%. They also implemented query result caching in Druid for identical queries within a short time window.

User Segmentation

Different user groups need different views. The team created role-based dashboards: operators see system health and latency; data scientists see model performance and drift; managers see high-level business metrics. Each dashboard queries a subset of metrics, reducing the overall query load. They also used lazy loading: only load charts that are visible in the viewport.

Data Retention and Downsampling

As data accumulates, query performance degrades. The team implemented a tiered storage strategy: hot data (last 7 days) in Druid with full granularity; warm data (7–30 days) with 1-minute downsampling; cold data (30+ days) in Parquet files on S3, queried only on demand. This kept the hot tier small and fast.

Risks, Pitfalls, and Mitigations

Even with a well-designed architecture, real-time dashboards can fail in subtle ways. The team encountered several pitfalls and developed mitigations that are broadly applicable.

Pitfall 1: Late-Arriving Data

Events may arrive out of order or after the aggregation window has closed. This can cause incorrect metrics if not handled properly. The team used Flink's allowed lateness feature with a 10-second grace period, and configured Druid to handle late data via segment replacement. They also added a warning label on the dashboard for metrics that may be incomplete.

Pitfall 2: Schema Evolution

When the event schema changes (e.g., a new field is added), the Flink job and Druid schema must be updated without downtime. The team used Avro with a schema registry and designed Druid tables with nullable columns for optional fields. They also implemented a versioning strategy: new events are written to a new Druid segment, and the frontend can handle both old and new schemas.

Pitfall 3: Dashboard Overload

If too many users open the dashboard simultaneously, the backend can become overwhelmed. The team implemented rate limiting at the WebSocket layer and used a CDN to serve static assets. They also added a