Software Development & Broken Window Theory

broken_window

Broken Window theory goes something like this:

  1.   Some broken windows are left unrepaired in a neighbourhood…
  2.   People see this state of disrepair and feel like no one cares about their surroundings…
  3.   Because nobody cares, people feel like they can cause further damage without repercussion…
  4.   Further damage is done, perpetuating the cycle.

We see this exact same pattern in all areas of life including software development.

Continue reading

SSRS Standards

When I joined my current employer, they had just started an informal project with 6 BI developers to deliver a host of new reports against their new data warehouse using SSRS 2016 Standard.

These are the standards I put together to try and bring some consistency to both the reports and the environments.

Like with any standards, my goal was to address the main areas whilst not being too onerous by concerning ourselves with minute details.  It was also to provide a steer on items which end up being arbitrary decisions such as the names of data-sources.

Hopefully it can act as a starting point for your own organisations.  I’ll look to cover the rationale behind the access model in another post.

The document, which also contains details on how to setup the project configurations can be found here.

SSRS Standards

Related Links:

How to enable Remote Errors in SSRS

How to set retention period of the execution log

Data Dictionary

Data Dictionary w/ Search Functionality (2016)

At CNA-Hardy, I put together a data dictionary for an Underwriting/Actuarial MI system I looked after.

I created the data dictionary in Excel and put a search facility built in.  With around 250 calculations and attributes it made understanding and troubleshooting a lot easier.

data-dictionary-img

The .xlsx version of the tool can be seen here.

The .xlsm version which also filters the rows can be found here.

 

Technical Debt

Technical debt is a metaphor that equates software development to monetary debt.  In my opinion it is one of the most crucial concepts to be aware of when planning projects or road-maps.

Imagine that you have a project that has two potential options; one is quick and easy but will require modification in the future, the other has a better design but will take more time to implement.

In development, releasing code with a ‘quick-and-dirty’ approach is like incurring debt – it comes with the obligation of interest, which, for technical debt, comes in the form of extra work in the future.

Just like monetary debt, technical debt is interest-bearing and compounds.  You always have the option to pay down the debt (long-term thinking) or to take out additional credit (short-term) but your project can become insolvent where the only option is to write-off the debt (re-write from scratch).

To summarise, it is a debt that you incur every time you avoid doing the right thing like removing duplication or redundancy.  It will add an overhead to everything you do from thereon in, whether that is troubleshooting, maintenance, upgrading or making changes.

[Some parts taken from MartinFowler.com and Techopedia]

Default SQL Server Settings

Introduction

You can change server-wide default settings by using facets and properties at server level or by modifying the model system database (which acts as a template for new databases created).

I don’t change much but here are the settings I do change to make life easier for myself.

Note: I fully understand that everything needs to be considered on a case by case basis – I’m just presenting these as possible ideas.


#1 – Backup locations – (In SSMS object explorer, right-click the server >> Facets >> Backup Directory)

When configuring my backup procedures, I like to setup two folders on each database server such as the below;

E:\SQL_Backups_Nightly\ | Used with nightly backup maintenance job.
E:\SQL_Backups_Adhoc\   | Used for manual backups/restores.

I then setup the latter as my default backup directory in the server facets pane.

This means that when I do manual backups/restores the dialog box will take me straight to this folder.  I find that when restoring a database under pressure, trying to navigate to the default paths (which usually look like below) just adds unnecessary confusion.

\\Servername\e$\Program Files\Microsoft SQL Server\MSSQL10_50.MSSQLSERVER\MSSQL\Backup

#2 – File Locations – (In SSMS object explorer, right-click the server >> Properties)

This is similar to the backup locations above.  In Database Settings under Database Default Location I use the following values.

E:\SQL_Server_Data\
F:\SQL_Server_Logs\

Again, they are just easier to find and if you ever have to write MOVE statements and such like, this will make it a lot easier.

For the record, it’s a good idea to have your log files on a separate physical disk to your data files, please see this article for a full explanation.

Continue reading

Cubes (& PowerPivot) vs Traditional Excel Pivot tables

This is a question I’ve been asked a few times before; “apart from handling larger data-sets, what do you get with cubes that you don’t get with pivot tables?”

This isn’t an exhaustive list but it covers what I think are the most important functionality differences.

#1 – Cubes are organised into dimensions / attributes.

With traditional Excel pivot tables, you are building on top of a flat data-set and as a result you will get your dimensions / attributes / measures in one big list.

You can take a flat data-set straight into PowerPivot as well (and there are definitely use cases for doing this) but if you build a dimensional model (as you are forced to do in SSAS MD) you will logically organise your reporting attributes by the correct dimension and make a much more intuitive model for the user.

dim-vs-flat

#2 – Hierarchies.

Traditional pivot tables will show you the implicit relationships in your data but you will have to create a column for each attribute you wish to show.  You can then tidy this up by adding manual grouping but the whole process is very clunky.

With PowerPivot & Cubes, you explicitly define hierarchies and when exploring the data you can drag this single field in and have instant drill-up and drill-down, plus on large dimensions performance is going to be significantly better.  Displaying natural relationships this way is key in making the data easy to understand at a glance.

ssas_pvt_hier_v2

#3 – MDX / DAX.

In cubes and PowerPivot you can write your own MDX & DAX functions to build more complex calculations.  There really isn’t any direct comparison for MDX/DAX for traditional pivot tables although when creating a measure in a traditional pivot table there are some options available (as shown below) but nothing with the advanced analytics capability of MDX/ DAX.

pivot_measure_opts

Users also sometimes build calculations outside of the traditional excel pivot but this then means the pivot table has to remain static otherwise the calculations are overwritten.

#4 – Client Tools.

Traditional Pivot Tables are themselves the end-product.  PowerPivot provides an upgrade path to SSAS tabular and cubes have a whole host of cutting edge reporting tools that can use them as a source.  Personally I don’t like excel pivot tables for browsing cubes and have tried out some different reporting tools that can connect to SSAS.  My favourite experience was delivered by Pyramid Analytic’s BI Office solution, an example of which can be shown below.

Power View inside Excel is not bad but in my opinion it’s not the easiest thing to use and I can’t see it gaining wide adoption.

bioxl

#5 – Other Considerations

There are plenty of other important considerations such as

security,
pre-aggregation,
partitioning,
formatting of dimensions/measures,
deployment possibilities

…and of course scalability but hopefully the above should demonstrate to people new to BI that SSAS offers key functionality benefits compared with pivot tables.