About me
I'm a data engineer and Microsoft Data Platform MVP. I specialise in building and automating analytics platforms using Microsoft technologies from the SQL Server stack to the Azure cloud. I’m a fervent advocate of DataOps, with a technical focus on bringing automation to both analytics development and operations. An active member of the data community and keen knowledge-sharer, I'm a volunteer, organiser, speaker, blogger, open-source contributor and author – my book Azure Data Factory by Example was published by Apress in June 2021. I have a PhD in Computer Science and participate enthusiastically in choral singing, craft ale and cocktails, most successfully in that order.
Contact me at:
Pro DevEx for Power BI
A multi-part series on creating a pro dev experience for Power BI, aimed at bringing DevOps or DataOps approaches to Power BI report development. In the series I'll be assembling an approach to creating developer workflows that provide meaningful version control and automate deployment to multiple environments. I'll be writing a new post every couple of weeks and will link them here when they're done:
- Streamlining Power BI report creation
- Better version control for Power BI reports
- Automated testing for Power BI datasets
Learning Azure Synapse Analytics
The pace of change in data analytics has never been faster, and with each innovation the job of engineers and architects becomes ever more complicated. Azure Synapse Analytics reduces this complexity by combining data integration, data warehousing, and big data analytics into a single developer experience.
In our forthcoming book Learning Azure Synapse Analytics, Paul Andrew (b|t) and I provide a practical overview of this cloud native highly scalable analytics platform, exploring what, why and how for each service within it. While Azure Synapse Analytics offers a unified flexible toolset for data professionals, navigating the technical and architectural options can be bewildering. You'll learn about each service individually, and how to combine them into a powerful end to end platform for delivering next-generation data analytics.
This book will help you to:
- Explore the fundamentals of Azure Synapse Analytics
- Obtain the tools required to deliver cloud native data analytics
- Understand how to interact with technical capabilities offered
- Learn how to build and operate solutions and resources of your own
- Develop dynamic integration and orchestration pipelines
- Learn the role of Azure Synapse Analytics within a given data architecture
- Unlock the potential of the unified Azure Synapse Analytics cloud resource
Prior to publication (scheduled for July 2023), new chapters will regularly be released in draft on the O'Reilly Learning Platform.
Azure Data Factory by Example
Hit the ground running in Azure Data Factory v2 (ADF) with my book Azure Data Factory by Example. The tutorial-first approach gets you working from the first chapter, explaining key ideas naturally as you encounter them. From creating your first data factory to building complex, metadata-driven nested pipelines, the book guides you through essential concepts in Microsoft’s cloud-based ETL/ELT platform. It introduces components indispensable for the movement and transformation of data in the cloud. Then it demonstrates the tools necessary to orchestrate, monitor, and manage those components.
The hands-on introduction to ADF found in this book is equally well-suited to data engineers embracing their first ETL/ELT toolset as it is to seasoned veterans of Microsoft’s SQL Server Integration Services (SSIS). The example-driven approach leads you through ADF pipeline construction from the ground up, introducing important ideas and making learning natural and engaging. SSIS users will find concepts with familiar parallels, while ADF-first readers will quickly master those concepts through the book’s steady building up of knowledge in successive chapters. Summaries of key concepts at the end of each chapter provide a ready reference that you can return to again and again.
Azure Data Factory testing
A six-part series on test automation for Azure Data Factory pipelines. Automated testing is a key component of CI/CD development pipelines: it allows you to run more tests, more often, more quickly, with guaranteed repeatability.
In the series I build up an approach to writing tests for ADF and triggering them automatically.
If you're wondering if all this testing is worth the trouble, I have some thoughts on the subject here 😃.
Sprockit
Sprockit is a powerful but lightweight ETL process controller for managing SSIS package and T-SQL stored procedure execution. It's free, open-source and written purely in T-SQL. You can download it here.