Context

In 2016 I joined an insurance company that was the recent merger of two separate companies.

The reporting architecture had grown organically without much formal design, it was undocumented and poorly understood.

The CEO was getting complaints from the brokers saying that we “couldn’t do the basics”.

The data & analytics team approach to every requirement at the time was to always bolt on more and more layers at the end.

Current State

Above, this was the architecture when I joined. The problem statements could be defined as below:

1 – Simple reports were dependent on a combination of unreliable nightly loads taking around 22 hours.
2 – The financial data marts were having to ingest data unrelated to finance to support group reporting.
3 – Data lineage was next to impossible given the 500K lines of SQL.
4 – Business operations that should take 5-minutes, could potentially take 5 days if nightly loads failed.
5 – Testing was extremely hard and reporting data was not trusted.
6 – New reports often required very expensive vendor engagement.

New State

Above is the design I implemented. The impact was:

1 – Simple reports were always available.
2 – The financial data marts were able to be drastically slimmed down.
3 – Data lineage for a large number of critical reports was very easy to track.
4 – Business operations became reliable.
5 – The new reports were completely trusted.
6 – New reports took weeks to build instead of months.

I did all the data architecture, data modelling, database development, testing and a few core reports in around 9 months.

After I left…

Initially the solution was transformational, however, after I left it became a downstream consumer of the very problematic architecture that it was designed to alleviate – and so it inherited the unreliability, latency and complexity of the original solution.