Logo CoinCryptoRank
Login Register
Infrastructure
Last updated: August 2025

Price Feed Aggregation Architecture: TTL, Outliers & Signal Reliability

Robust price feed aggregation is the backbone of arbitrage scanners, risk dashboards and execution engines. A single stale tick, exchange outage, or manipulated wick can cascade into false spread signals, mispriced hedges and forced liquidations. This engineering guide details a production‑grade architecture: multi‑venue intake (REST + WebSocket + on‑chain oracle), normalization, median/MAD outlier filtering, adaptive TTL staleness policies, reconciliation layers, weighted aggregation (liquidity, latency, reliability), circuit breakers, redundancy and monitoring KPIs. Implementing these patterns converts noisy fragmented crypto market data into deterministic, low‑latency, auditable reference prices powering safer arbitrage strategies.

Multi-Source Intake & Venue Classification

Primary Real-Time Channels

Consolidate WebSocket trade & best bid/ask streams from top liquid CEXs (Binance, OKX, Coinbase, Kraken) plus DEX oracle or AMM subgraph references for synthetic cross validation.

Secondary & Recovery Sources

Fallback REST midpoints, consolidated vendors (Kaiko, CryptoCompare), and on‑chain oracle snapshots (Chainlink, Pyth) activated when primary freshness SLA breached.

Metadata Enrichment

Attach venue latency_ms, reliability_score, liquidity_tier, and regime_tag (normal / stressed) for adaptive weighting downstream.

Data Normalization & Canonical Tick Model

Canonical Schema

Unify into {ts, symbol, venue, bid, ask, mid, last, volume_1s, latency_ms}. Pre-compute mid = (bid+ask)/2; ensure symbol mapping & base/quote decimals normalized.

Timestamp Harmonization

Convert to monotonic microsecond UTC; reject ticks where |client_ts - server_ts| > 250ms unless in stressed regime; record skew for latency dashboards.

Unit & Precision Control

Standardize to quote currency (e.g. USDT) and float64 internal representation; preserve original precision for audit. Apply tick size rounding only for display, not internal calculations.

Weighted Aggregation & Robust Statistics

1

Median + MAD Core

Compute median mid price; flag outliers where |mid - median| / MAD > 6. Robust to single venue manipulation and thin book wicks.

2

Liquidity / Reliability Weights

After pruning outliers, compute weighted mid: Σ(mid_i * w_i) / Σ w_i with w_i = log(liquidity_depth_usd+1)*reliability_score/(1+latency_ms).

3

Derived Benchmarks (TWAP/VWAP)

Maintain rolling TWAP and VWAP windows (1s, 5s, 60s). Use 5s TWAP as arb validation threshold: ignore instantaneous spreads < 1.2x TWAP divergence multiplier to reduce noise chasing.

Adaptive TTL, Staleness Detection & Circuit Breakers

  1. Base TTL Policy: per symbol 1000ms normal regime; shrink to 500ms during high volatility (abs(returns_1s) > threshold).
  2. Venue Expiry: Drop a venue from weighting if last tick age > 2 * TTL; mark status = STALE.
  3. Aggregation Degradation: If active venues < 3, downgrade confidence; if < 2 trigger circuit breaker and freeze reference price (TWAP fallback).
  4. Clock Drift: Reject ticks with negative monotonic delta or drift > 300ms vs NTP sync baseline.
  5. Confidence Score: Publish alongside price: composite of sample size, variance, staleness factor.

Outlier Rejection & Reconciliation Workflow

Multi-Stage Filters

Stage 1: sanity (bid ≤ ask, positive spreads). Stage 2: statistical (MAD / z-score). Stage 3: volatility gating (reject spikes > 5 * rolling σ unless corroborated by ≥ 50% of venues).

Reconciliation Logic

If venue excluded > X minutes, spawn health probe: REST depth snapshot + small notional test order (sandbox) to confirm connectivity before reinclusion.

Audit & Replay

Persist raw + filtered tick streams (parquet / columnar) with reason codes for exclusion enabling deterministic backtesting & incident forensics.

Redundancy, Failover & Deployment Topology

Run dual ingestion clusters (Region A / Region B) with active-active Kafka topics. Apply deterministic partition key symbol. Heartbeat consumer arbitration: if Region A gap > 3s, Region B elevates priority. Maintain cold standby vendor aggregator for black swan exchange outages. Introduce chaos drills monthly (synthetic venue blackout + latency injection) to validate failover correctness and confidence scoring adjustments.

Observability KPIs & Data Quality Metrics

Latency & Freshness

Track P50/P95 ingestion latency, tick age distribution, staleness ratio. Alert if P95 > 2 * baseline or staleness ratio > 5% for Tier-1 venues.

Data Integrity

Monitor invalid spread count, outlier rejection rate, reconciliation re-entry success rate, and variance between median & weighted mid (should remain < 4 bps in normal regimes).

Risk Impact

Correlate price feed deviations with downstream arbitrage trigger false positives; maintain rolling precision / recall metrics on historical replay to validate adjustments.

Price Feed Deployment Checklist

  1. 1. Schema: Canonical tick structure validated & contract tested.
  2. 2. Outliers: MAD thresholds calibrated across low / high volatility regimes.
  3. 3. TTL: Adaptive staleness policies simulated with 24h historical replay.
  4. 4. Failover: Region chaos drill executed; latency delta within SLA.
  5. 5. Metrics: Prometheus dashboards + alert routes reviewed.
  6. 6. Storage: Parquet archival + reason codes reproducibility verified.
  7. 7. Confidence: Downstream consumers reading confidence_score & gating actions.

Essential Tools & APIs

  • Kafka / NATS (tick stream backbone)
  • Redis (low latency cache + TTL enforcement)
  • Prometheus + Grafana (latency, staleness, rejection metrics)
  • Faust / asyncio (Python stream processing)
  • Chainlink / Pyth (oracle cross checks)
  • Kaiko / CryptoCompare (vendor fallback)
  • S3 + Parquet (immutable tick archives)
  • Great Expectations (data quality validation suite)

Upgrade Your Arbitrage Stack

Integrate this aggregation model with our Perpetual Arbitrage Module, compare cross‑venue mispricing via Market Overview and convert spreads instantly with the BTC/USDT Converter.

Conclusion

A resilient crypto price feed is not a single API call—it is an adaptive, multi-layer system balancing latency, integrity and robustness. By combining robust statistics (median/MAD), adaptive TTL, staged filtering, redundancy, confidence scoring and transparent monitoring you materially reduce false arbitrage triggers and liquidation risk. Treat every design decision as a trade‑off in the Latency vs Robustness triangle, continuously replay historical stress windows, and iterate weighting & rejection logic with quantitative precision / recall metrics. This transforms raw exchange noise into institutional‑grade reference pricing.

Tags

Price Feeds Data Aggregation TTL Staleness Outlier Detection Arbitrage Signals Data Quality Risk Management

Categories

Share this article

Facebook Twitter Telegram

Sources & References

Skip to main content