In the last chapter, you learned how to use Azure Data Lake Analytics (ADLA) to build reusable objects. You also used C# to enhance, and sometimes replace, the functions of SQL. In this chapter, you’ll build on that by adding features to improve your U-SQL scripts. You’ll use the Data Lake store to serve assembly files for use in ADLA jobs. You’ll run Azure PowerShell and U-SQL scripts to modify the ADLA and Data Lake environments. You’ll add new types of data extraction classes to ADLA, and add C# functions for modifying data. You’ll also connect to external providers to add even more data with minimal effort. This extensibility is facilitated by the compiled nature of ADLA jobs.
The ADLA cluster translates each U-SQL script submitted into a .NET compiled application as a new ADLA job. This creates a new set of code to be executed on the cluster nodes assigned to the job. Because the script is compiled, each job includes a step that allows external code libraries to be included. The compiler includes SQL and .NET assemblies in every job, which lets jobs use many C# and SQL functions. Adding custom assemblies to a job works this way: