The problem
Two weeks of data contained approximately 26.6 million records and occupied 12.59 GB. The application downloaded complete datasets to the server and calculated chart values in Python. Reporting became expensive before a chart could even be drawn.
Queries involving large datasets were slow and could time out. The reporting path needed to serve charts without transferring the full raw dataset.
What I did
I analyzed the reporting pipeline and identified unnecessary data transfer and application-side processing.
The charts needed aggregated traffic metrics rather than complete raw datasets.
I moved the reporting work into time-series aggregation and materialized collections, then wrote chart-specific queries against the precomputed results.
Technical details
Aggregation pipelines grouped time into hourly buckets and summarized traffic metrics. MongoDB $merge wrote the results into materialized collections. The chart query could then read prepared data rather than download the complete source dataset.
Before
- Raw MongoDB records
- Full dataset download
- Python chart calculations
- Chart response
After
- Hourly aggregation
- $merge materialized collections
- Chart-specific query
- ~1s reported chart query
Materialized collections require refresh or update logic and additional storage. The time-bucket size determines how much detail remains available. Late-arriving records and backfills should be considered when applying this design elsewhere.
The result
Chart-specific queries completed in approximately one second after moving aggregation into MongoDB and reading materialized collections.