Some discussions around data architecture and modern data platforms are somewhat over simplistic. For example, if you asked someone to explain the difference between ETL and ELT you might get the following:

“ETL is old, ELT is new”
“ETL is bad, ELT is good”
“ETL is databases, ELT is data lakes”

Let’s actually understand the terms first with the help of the AWS blog, which has a lot of great content.

Firstly, AWS points out that the E (Extract) is the same in both cases – happy days, we can move on.

Now, using both the above diagrams and below table we can compare the T (Transform) and L (Load).

Once I had taken this information in, I started to think back to all the SQL Server based Data Warehouses I have worked on over the last 15 years or so.

I started to realise those architectures followed the second image; SSIS would copy (Extract & Load) the source system data “as is” to a staging database on the Data Warehouse server, and we would use Stored Procedures to transform that data into dimensions and facts.

It is entirely possible that some people used SSIS (or other ETL tools) and transformed the data mid-flight and loaded it directly into their star schema, but these were rare as Stored Procedures were often simpler, faster and more elegant.

So although the ELT in the world of Fivetran, semi-structured data and Spark may look quite different to what we were doing ‘back in the day’, if you look at the “dictionary definitions” of the terms, you will see that often, it’s not a fundamental departure from where we were.

A while back I stumbled across a great blog post from a guy called Greg Ferro with his musings on ITIL. It’s a very good read in its own right and can be found here.

Unusually however, the pages of discussion and comments underneath the article, arguing both for and against ITIL, contained just as much insight as the main article itself. A lot of people felt so passionate about the subject that they took the time to relay their own personal experiences in a really considered and articulate way.

This slideshow below is just a collection of some of the best comments against ITIL – I’m not trying to be balanced here. They are largely posted ‘as is’, but some comments have been combined on the same slide just to use the space wisely.

Btw, I’m not personally commenting on ITIL’s merits – but as ITIL is seemingly ubiquitous in organisations everywhere, it is rarely challenged and doesn’t seem to have a feedback loop – which is one of the reason’s I found the post and comments so interesting.

So here you go…

Nobody Gets Hurt from Ben Brown

Introduction

The original Joel Test is a work of genius – it allowed you to assess the quality of your development teams in under 5 minutes.  By answering the yes/no questions, it steered you into thinking through each of the points and understand weak points in your process.

It was written in August 2000 (that’s before Windows XP was even released) and is still very relevant today, which suggests that a lot of IT teams are still making the same mistakes they were making 18 years ago.

I’ve heard one developer sum up the Joel Test perfectly when he said, “the beauty of the Joel Test is its simplicity vs its effectiveness”.

Just as the original did, these tests deliberately cover the basics. I have intentionally not included tests such as “are you agile?” or “do you do DevOps?” as a lot of people misunderstand these concepts and unfortunately they sometimes get thrown around as management buzz-words.

Just for the record, I have worked mainly as a data warehouse & BI developer for financial services firms in London, within IT teams ranging from 2 people to 100s; if I had spent my career working as a web developer for Facebook or Amazon then I’m sure my tests would be quite different.

Remember, Yes = 1 point.  No = 0 points.

11 or 12 points	Keep doing what you’re doing
8, 9 or 10 points	Keep going but analyse where you can gain efficiencies
7 or under	Stop what you’re doing, call a team meeting and make a plan

 

01 – Does your team have a goal… and do all your team members know what it is?

Do your team members also know what their individual goals are and those of the company?

This is universal and applies to any team/individual and is not really about having goals – which all companies will have – it’s about whether or not they are being communicated down the layers and making sure everyone is striving for the same success.

Benefits of having clearly understood goals

• Much more cohesion between teams when people pull in the same direction.
• Goal-relevant activities take up a lot more of the overall effort than goal-irrelevant activities.
• Managers cannot motivate teams by assigning out a series of seemingly unconnected tasks.

Continue reading →

Back in 2009 Netflix released a slide deck called ‘Freedom & Responsibility’ that explained some of their strategy and culture.

Facebook COO Sheryl Sandberg said that it “may well be the most important document ever to come out of the Valley”.

I first heard about it a few years back when I read Dave Coplin’s brilliant (and succinct) book Business Reimagined, which you can download here for free.

‘Freedom & Responsibility’ was something that evolved at Netflix over many years.  Here are what I consider to be the 10 most notable ideas from the document.

Continue reading →

Broken Window theory goes something like this:

1.   Some broken windows are left unrepaired in a neighbourhood…
2.   People see this state of disrepair and feel like no one cares about their surroundings…
3.   Because nobody cares, people feel like they can cause further damage without repercussion…
4.   Further damage is done, perpetuating the cycle.

We see this exact same pattern in all areas of life including software development.

Continue reading →

Data Dictionary w/ Search Functionality (2016)

At CNA-Hardy, I put together a data dictionary for an Underwriting/Actuarial MI system I looked after.

I created the data dictionary in Excel and put a search facility built in.  With around 250 calculations and attributes it made understanding and troubleshooting a lot easier.

The .xlsx version of the tool can be seen here.

The .xlsm version which also filters the rows can be found here.

 

Technical debt is a metaphor that equates software development to monetary debt.  In my opinion it is one of the most crucial concepts to be aware of when planning projects or road-maps.

Imagine that you have a project that has two potential options; one is quick and easy but will require modification in the future, the other has a better design but will take more time to implement.

In development, releasing code with a ‘quick-and-dirty’ approach is like incurring debt – it comes with the obligation of interest, which, for technical debt, comes in the form of extra work in the future.

Just like monetary debt, technical debt is interest-bearing and compounds.  You always have the option to pay down the debt (long-term thinking) or to take out additional credit (short-term) but your project can become insolvent where the only option is to write-off the debt (re-write from scratch).

To summarise, it is a debt that you incur every time you avoid doing the right thing like removing duplication or redundancy.  It will add an overhead to everything you do from thereon in, whether that is troubleshooting, maintenance, upgrading or making changes.

[Some parts taken from MartinFowler.com and Techopedia]