concept datum in category azure

This is an excerpt from Manning's book Azure Storage, Streaming, and Batch Analytics: A guide for data engineers.
Increased connectivity has led to increased sophistication and user interaction in software systems. New deployments of connected “smart” electronics also rely on increased connectivity. In response, businesses now collect and store data from all aspects of their products. This has led to an enormous increase in compute and storage infrastructure. Writing for Gartner, Mark Beyer defines “Big Data.”[2]
In this chapter, you’ll learn how to move data in Azure services into SQLDB. This includes the outputs of ADLA batch processing, data in Blob Storage, and data in other SQLDBs. Providing data via an RDBMS gives more users access, with a wider range of tools, than data files only.
This chapter explores two methods of importing data into SQLDB: external tables and ADF. External tables read data from data stores other than the one storing the SQLDB’s native tables. These data stores can be external files or separate databases. ADF connects to both file services and databases, and copies data between services. Along the way you’ll enhance your use of both SQLDB and ADF. You’ll learn how to share data between otherwise isolated SQLDB instances, which works the same as accessing data from file stores. You’ll also attach Git repositories to your ADF services, providing change management for your configuration files.
One problem with normalized tables comes when lookup tables change over time. In the simplest form of lookup table, with a single-column relational key and single data column, updating the lookup value loses the historical value. In some scenarios this is a feature rather than a drawback. For example, you can discover and correct textual errors. In other scenarios, data sets drawn from a particular period should reflect the extant values of that period. For example, when a user changes addresses, referencing an address stored in a user lookup table should show the previous address for orders before the move occurred. This history can be kept either in a separate version of the main table, or by using a from/to date range. We’re not going to delve deeply into data warehouse designs, but we’ll use remote data sources to look up data. Dealing with slowly changing values in analytics systems is a topic for another book.
In this scenario, your users want a report that shows a single value for a column that changed over the period of time the report covers. Or, the data in the lookup table in your analytics database has become out of sync with the data in the development team’s database. In either case, you need to let your users query the latest data. You can do this by loading the updated data into your database or by querying the data remotely.
SQLDB is inherently insular. Unlike its cousins, the Managed Instance and full SQL Server install, SQLDB can’t connect to multiple databases in a single query. Server-level logins aren’t supported for connecting between SQLDBs, so these cross-database queries don’t connect. There’s no method for linking servers together either, as with a full SQL Server install. SQLDB does support connections from outside applications that can insert data into and read data from the database. The SQLDB is an island with no bridges.
You can build bridges from SQLDB to other databases using external data sources. An external data source uses a saved credential, a connection definition, and a table definition to let queries read from outside databases. This saved credential is called a database scoped credential. The external data source defines the location of the external database and the credential to use. An external table provides a schema definition to use when reading data from the external connection. Together, these three objects allow read access to data in separate databases.
An external data source can use either another SQLDB or a Blob file. Different schema, read, and authentication steps are used for each source. Figure 10.6 shows the process of reading data from external data sources using a SQL query.
Figure 12.2. Connecting SQLDB to an external data source
![]()
SQL queries using external tables retrieve rows from the external data source, and perform processing like JOINs and column calculations afterward. The retrieved rows are stored in an intermediate temp table during processing. To use data from Blob files in SQL queries, you must first import the data into a database table. Then you can use the table in your query.
Using external data sources with Blob files works differently than with remote SQLDBs. External tables can read directly from remote databases, and can be used in queries directly. Blob file data sources must import the data first, before using it in queries. The import target can be temp tables, table variables, or local tables.
To import data from Blob files, use the
BULK INSERTSQL command. TheFROMparameter provides the file to import, andWITHspecifies other options. Pass the external data source usingDATA_SOURCE. UseFORMAT = 'CSV'to specify a comma-separated file with carriage return and line feed row endings. The target table already has a header defined for the columns; you can skip the header row in the Blob file by passingFIRSTROW = 2. Listing 12.8 shows an example script for importing PlayerDetails.txt.

This is an excerpt from Manning's book Learn Azure in a Month of Lunches, Second Edition.
Infrastructure as a Service (IaaS) --You purchase the base compute resources from a vendor that manages the core infrastructure. You create and manage the VMs, data, and applications. The cloud provider is responsible for the physical infrastructure, host management, and resiliency. You may still have an infrastructure team to help support and deploy VMs, but the team is free from the time and cost of managing the physical equipment. This approach is good when you start to move applications out of your own on-premises environment. The management and operations are often similar to an on-premises environment, so IaaS provides a natural progression for the business, IT, and application owners to become comfortable with the cloud. Platform as a Service (PaaS) --You purchase the underlying platform stack from a vendor that manages the OS and patches, and bring your applications and data. You don’t worry about VMs or the virtual network, and your operations team can focus more of their time on application reliability and performance. This approach often starts making the IT organization and the business comfortable with running applications in the cloud. Your focus is on the applications and your customers, with fewer worries about the infrastructure needed to run those apps.
Data. You can’t get by without it. Almost every application that you build and run creates, processes, or retrieves data. Traditionally, this data has been stored in a structured database such as MySQL, Microsoft SQL, or PostgreSQL. These large, structured databases are established and well known, have ample documentation and tutorials, and can be accessed from most major programming languages.