Can you just...?

I spent some time working at a property portal, where users could look at online listings of homes for sale or rent, then go on to book a viewing appointment with an estate agent. On one occasion we were asked to build a Power BI report showing:

• the number of appointments booked by portal users
• the percentage of appointments where the user had viewed the online listing more than once.

Sounds easy enough, right?

Adding an appointment count was easy enough – the information was stored in our own booking system and was already included in our semantic model (a shared Power BI dataset). Adding whether or not the user had viewed the listing more than once was a whole different ballgame:

• the data about page visits was stored elsewhere, in Google Analytics (GA)
• at that point we weren't reliably passing user IDs to GA, so we couldn't reconcile page visits to appointments
• (we'd never be able to attach a user ID to every visit, because users might not log in, or might opt out of cookies)
• even if we solved all these problems, we'd still have a whole bunch of engineering to do.

Communicating the difference to non-technical people can be a challenge, because – as the cartoon points out – two things can sound objectively similar but be of very different complexity.

Trying to respond to the “can you just…” question prompted me to draw this picture. It's one slide, with a high level view of a data analytics pipeline (click it to enlarge):

The questions here illustrate the sorts of things data professionals have to think about – and how much work we need to do – to get data from source to report. A non-technical colleague doesn't need to understand every detail, but at the very least I can point at a place in the diagram and say “the data you're asking for is here!”.

In case you're wondering, appointments were already in the semantic model (on the right) but page visits were on the far left (insofar as they existed at all). Pointing out their locations in the diagram made it simpler to demonstrate why one question was easy to answer and the other not even possible (yet).

If this picture's helpful to you, feel free to use it 😊.