Concord reviewed the existing pipeline and broke it down into parts to identify the troublesome areas. We learned the pipeline slowed down because it was processing historical data every time the pipeline ran.
To reduce pipeline strain, our team developed and deployed a static historical table that the pipeline could reference that only processed current year data.
As a result:
- The client saw a 35% improvement in the speed of processing, reducing the time from 110 minutes to 71 minutes.
- The data source became reliable again and downstream reporting experienced minimal disruption in it’s daily refresh.
- Analysts were trained on optimization tactics in pipeline development to ensure pipelines are developed to engineering best practices going forward.