Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
speaking [2019/03/31 14:50]
admin
speaking [2019/04/07 16:14] (current)
admin ↷ Links adapted because of a move operation
Line 2: Line 2:
  
 ===== Design patterns for metadata-first ETL process control ===== ===== Design patterns for metadata-first ETL process control =====
-{{:sprockit:etl_metadata.pptx|Download slides}}+{{etl_metadata.pptx|Download slides}}
  
 As data engineers we're great at putting together and managing complex process flows, but what happens if we stop trying to control the flow and start thinking about the metadata it needs instead? In this session we'll look at a variety of ETL metadata, how we can use it to drive process execution, and see benefits quickly emerge. I'll talk about design principles for metadata-first process control and show how using this approach reduces complexity, enhances resilience and allows a suite of ETL processes adaptively to reorganise itself. As data engineers we're great at putting together and managing complex process flows, but what happens if we stop trying to control the flow and start thinking about the metadata it needs instead? In this session we'll look at a variety of ETL metadata, how we can use it to drive process execution, and see benefits quickly emerge. I'll talk about design principles for metadata-first process control and show how using this approach reduces complexity, enhances resilience and allows a suite of ETL processes adaptively to reorganise itself.
Line 24: Line 24:
  
 ===== ETL process management with TSQL ===== ===== ETL process management with TSQL =====
-{{:sprockit:etl_process_management_datarelay.pptx|Download slides}}+{{etl_process_management_datarelay.pptx|Download slides}}
  
 There are many techniques for orchestrating ETL processes, but the difference between good ones and great ones is how they perform when things go wrong. Desirable behaviours – like fault tolerance, quick fault finding and easy resume after error – often aren't available and sometimes seem hard to achieve. In my session I'll present an approach to doing this using only TSQL and the SQL Server Agent, and which also enables parallel processing, adapts to evolving workloads and provides a wide variety of monitoring and diagnostic information. There are many techniques for orchestrating ETL processes, but the difference between good ones and great ones is how they perform when things go wrong. Desirable behaviours – like fault tolerance, quick fault finding and easy resume after error – often aren't available and sometimes seem hard to achieve. In my session I'll present an approach to doing this using only TSQL and the SQL Server Agent, and which also enables parallel processing, adapts to evolving workloads and provides a wide variety of monitoring and diagnostic information.