Ingest Streaming Data into Delta Lake with Auto Loader and Databricks

In part one of our series, we’ll go deep on Auto Loader, an optimized cloud file source for Delta Lake that loads data incrementally and efficiently as new data arrives. Auto Loader is provided as a managed offering as part of your Azure Databricks account, and helps you load raw data at low latency with minimal DevOps effort. In this session you will:

• Get an overview of Auto Loader, Delta Lake, Azure Databricks and Azure Event Hubs
• Ingest streaming data using Auto Loader and Azure Databricks
• Automatically subscribe to file events from the input directory and process new files as they arrive
• Infer schema from streaming data to create a clean and reliable data lake
• Work with basic Auto Loader functionality for both streaming and batch data loads