All case studies

MongoDB Reporting Across 26.6M Records

Replacing full-data downloads and application-side chart calculations with pre-aggregated collections and chart-specific queries.

Independent client work · Time-series reporting · 2022

MongoDBTime-series aggregation$merge

I reworked time-series reporting with aggregation pipelines, materialized collections and chart-specific queries, replacing full-data downloads and Python chart calculations.

THE OUTCOME26.6M records · ~1s chart queries

The problem

Two weeks of data contained approximately 26.6 million records and occupied 12.59 GB. The application downloaded complete datasets to the server and calculated chart values in Python. Reporting became expensive before a chart could even be drawn.

Queries involving large datasets were slow and could time out. The reporting path needed to serve charts without transferring the full raw dataset.

What I did

I analyzed the reporting pipeline and identified unnecessary data transfer and application-side processing.

The charts needed aggregated traffic metrics rather than complete raw datasets.

I moved the reporting work into time-series aggregation and materialized collections, then wrote chart-specific queries against the precomputed results.

Technical details

Aggregation pipelines grouped time into hourly buckets and summarized traffic metrics. MongoDB $merge wrote the results into materialized collections. The chart query could then read prepared data rather than download the complete source dataset.

Workflow overview

Before

  1. Raw MongoDB records
  2. Full dataset download
  3. Python chart calculations
  4. Chart response

After

  1. Hourly aggregation
  2. $merge materialized collections
  3. Chart-specific query
  4. ~1s reported chart query

Materialized collections require refresh or update logic and additional storage. The time-bucket size determines how much detail remains available. Late-arriving records and backfills should be considered when applying this design elsewhere.

The result

Chart-specific queries completed in approximately one second after moving aggregation into MongoDB and reading materialized collections.

More case studies.

All case studies

Facing a similar engineering challenge?

I help companies solve difficult backend, data and AI integration problems.

Book a technical call
Hiring a senior backend, applied AI or forward-deployed engineer?Explore my experienceDownload résumé