Problem
Two weeks of data contained approximately 26.6 million records and occupied 12.59 GB. The application downloaded complete datasets to the server and calculated chart values in Python. Reporting became expensive before a chart could even be drawn.
Context and constraints
A query took about 53 seconds locally and timed out remotely. Larger-data queries could take 10–20 minutes. Those observations describe different conditions, not one consistent baseline; the case does not turn them into a single speedup ratio.
Existing architecture
MongoDB supplied large raw datasets to the application server. The server then calculated chart values outside the database, coupling transfer volume and application computation to the full dataset.
Investigation
I examined the reporting requirements and aggregation structure. The supplied pipeline notes group timestamps into hourly buckets and aggregate source and destination traffic metrics—the information the charts actually needed.
Root cause
The chart path moved and processed much more data than the visualization required. Repeating full-data extraction and calculation made each request pay the cost of preparing reporting data.
Solution
I moved the reporting work into time-series aggregation and materialized collections, then wrote chart-specific queries against the precomputed results.
Architecture and implementation
Aggregation pipelines grouped time into hourly buckets and summarized traffic metrics. MongoDB $merge wrote the results into materialized collections. The chart query could then read prepared data rather than download the complete source dataset.
Before
- Raw MongoDB records
- Full dataset download
- Python chart calculations
- Chart response
After
- Hourly aggregation
- $merge materialized collections
- Chart-specific query
- ~1s reported chart query
Result
I reported chart queries running in approximately one second after the change. The client review confirms that the proposed solutions worked. This is a project-reported query result, not a claim about end-to-end page latency or all possible datasets.
Tradeoffs
Materialized collections require refresh or update logic and additional storage. The time-bucket size determines how much detail remains available. Late-arriving records and backfills should be considered when applying this design elsewhere.
What I would carry forward
Choose the chart’s required granularity before optimizing the raw-data query. In a future implementation, retain tests for bucket boundaries, late data and reconciliation with raw totals.
CLIENT EVIDENCE
“Ali picked up our problem very fast and gave us solutions which in the end worked fabolously. Great commitment towards customer”