concept datum in category cloud

appears as: data, data, The data
Cloud Native Spring in Action

This is an excerpt from Manning's book Cloud Native Spring in Action.

Therefore, containers enable agility, portability across different environments, and deployment repeatability. Being so lightweight and less resource-demanding, they are perfect for running in the cloud where applications are disposable and should be scaled dynamically and quickly. In comparison, building and destroying virtual machines is much more expensive and time-consuming.

Figure 2.17. Architecture of Polar Bookshop. The cloud native system comprises different services with different responsibilities: business logic, shared concerns, data storage, and UI. For clarity, observability services are not shown.
CH02 F17 Vitale
Listing 3.10. Unit tests to verify book validation constraints
package com.polarbookshop.catalogservice.domain;

import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.Test;

import javax.validation.ConstraintViolation;
import javax.validation.Validation;
import javax.validation.Validator;
import javax.validation.ValidatorFactory;
import java.time.Year;
import java.util.Set;

import static org.assertj.core.api.Assertions.assertThat;

class BookValidationTests {
    private static Validator validator;

    @BeforeAll #1
    static void setUp() {
        ValidatorFactory factory = Validation.buildDefaultValidatorFactory();
        validator = factory.getValidator();
    }

    @Test #2
    void whenAllFieldsCorrectThenValidationSucceeds() {
        Book book = new Book("1234567890", "Title", "Author", Year.of(2000)); #3
        Set<ConstraintViolation<Book>> violations = validator.validate(book);
        assertThat(violations).isEmpty(); #4
    }

    @Test
    void whenIsbnDefinedButIncorrectThenValidationFails() {
        Book book = new Book("a234567890", "Title", "Author", Year.of(2000)); #5
        Set<ConstraintViolation<Book>> violations = validator.validate(book);
        assertThat(violations).hasSize(1);
        assertThat(violations.iterator().next().getMessage()) #6
                .isEqualTo("The ISBN format must follow the standards ISBN-10 or ISBN-13.");
    }
}
Cloud Operations

This is an excerpt from Manning's book Cloud Operations.

  • Least privilege and separation of duties—Only the entity that needs a specific resource or data should be able to access it. A common output of this principle is to design your cloud operations in a way that there are no perpetual credentials granted.
  • Principle

    Cloud Operations

    Encryption of data

    Enable encryption – as simple as that. Many times, that can be done as a simple API call or flag to enable from the GUI. Many times, that action is not considered as priority because the data we handle are not critical or confidential.

    Identity and access management

    Enable federation (one-off)

    Review the expiring date of the certificates of the identity provider.

    Check the latest access list – i.e. the latest users who has access the systems

    Check the latest activity list

    Monitor alarms – i.e. use of MFA for crticial cloud operations.

    Check the presence of old access keys and old password.

    Rotate access keys and od passwords.

    Disable access to suspicious users or keys based on their activities.

    Separation between production and non-production

    Verify routing tables

    Governance framework

    Check platform logs

    Check pipelines status

    Run/schedule pipelines for daily operations

    Set baseline for images, OS versions and software updates.

    Run updates

    Check integrity of global policies

    Secure management: console and integration points

    Check the approval lists

    Approve/Deny access

    Check auto-approved access requests.

    Audit information provision

    Check anomalies

    Look for trends, peaks and drops

    Inspect critical alerts

    Set a flowing process of information and alerts through infrastructure and application owners.

    Expose to the Information Security, Risk and Compliance office key indicator of incidents, data breaches and a list of critical actions from the users.

    Cloud Native Machine Learning: Deploying PyTorch models on AWS

    This is an excerpt from Manning's book Cloud Native Machine Learning: Deploying PyTorch models on AWS.

    If you take a closer look at the illustration on figure 1.1, you should be able to describe some of the capabilities you need from a machine learning platform. Obviously, the platform needs to ingest and store data, process data (which includes applying machine learning and other computations to data), and serve the insights discovered by machine learning to the users of the platform. The less obvious observation is that the platform should be able to handle multiple, concurrent machine learning projects and enable multiple users to run the projects in isolation from each other. Otherwise, replacing just the machine learning code translates into re-work of 95% of the system.

    The Office of the Chief Technology Officer for Washington, DC maintains a website hosting data from taxi rides that took place in the DC area boundaries [4]. In this book, you will use this historical dataset of the taxi rides from 2015 through 2019 to build machine learning models to estimate how much it costs to travel by taxi around DC. The key advantage of the machine learning approach is that it will not depend on an expensive external service for route planning and distance calculations. The model will learn from the taxi rides data to estimate the fares based on trips taken by taxis across different locations in DC.

    The CSV data format used by the DC taxi data set is an example of a row-oriented format. In case of CSV files, every line in a file stores a single row of data from the structured dataset. Row-oriented data formats (illustrated on the left side of figure 2.3) are commonly used by traditional relational databases to store sequences of data records. The row-oriented format works well for transactional workloads that are typical for relational databases. Transactional workloads operate on individual rows of data and often just on a single row at a time. For example, consider a transactional database that stores a record of a taxi trip. If a passenger decides to change the destination of the trip halfway there, a transactional database can easily handle identifying the row of data about the trip, updating the latitude and the longitude coordinates for the destination, and then saving the changes back to the database.

    Figure 2.3. Row oriented storage (left) used by CSV files and traditional relational databases, is designed for transational processing, enabling changes to a row of data at a time. Column oriented storage (right) used by Apache Parquet and many modern data warehouses, works best with analytical queries over immutable datasets.
    CH02 F03 Osipov
    Serverless Architectures on AWS, Second Edition

    This is an excerpt from Manning's book Serverless Architectures on AWS, Second Edition.

    Let’s take the example of a typical data-driven web application, not unlike the systems powering most of today’s web-enabled software. These consist of a backend that accepts HTTP requests from the front end and then processes the requests. The backend servers perform various forms of computation and the client-side front ends provide an interface for users to operate via their browser, mobile, or desktop device. Data might travel through numerous application layers before being saved to a database. The backend, finally, generates a response—it could be in the form of JSON or fully rendered markup—which is sent back to the client (figure 1.1). These kinds of applications are conventionally architected as “tiers” – a presentation tier that controls how the information is captured and provided to the user, an application tier that controls the business logic of the application, and a data tier with the database and corresponding access controls.

    Figure 1.1 This is a basic request-response (client-server) message exchange pattern that most developers are familiar with. There’s only one web server and one database in this figure. Most systems are much more complex.

    Software architectures has evolved from the days of code running on a mainframe to multitier architecture where the presentation, data, and application/logic tiers are traditionally separated. Within each tier there may be multiple logical layers that deal with particular aspects of functionality or domain. There are also cross-cutting components, such as logging or exception-handling systems, that can span numerous layers. The preference for layering is understandable. Layering allows developers to decouple concerns and have more maintainable applications. Figure 1.2 shows an example of a tiered architecture with multiple layers including the API, the business logic, the user authentication component, and the database.

    Figure 5.8 The GraphQL endpoint serves as the central point for clients that need data.
  • First you split data into many independent subsets. In case of “War and Peace,” it could be individual pages or paragraphs.
  • Designing Cloud Data Platforms

    This is an excerpt from Manning's book Designing Cloud Data Platforms.

    One of the most common data platform use cases is driven by the need for a 360-degree view of an organization’s customers. Customers engage with or talk about organizations in many ways using many different systems, from social media, to e-commerce, to online chats, to call center conversations and more. The data from these engagements is both structured and unstructured, originates from many different sources, and is of varying degrees of quality, volume and velocity. Integrating all these touchpoints into a single view of the customer opens the doors to a plethora of improved business outcomes – an improved customer experience as they interact with different parts of the business, better personalization in marketing, dynamic pricing, reduced churn, improved cross-selling and much more.

    A second common use case for a data lake is IoT, where data from machines and sensors can be combined to create operational insights and efficiencies ranging from proactively predicting equipment failures on a factory floor or in the field, to monitoring skier (a person who moves over snow on skis) performance and location via RFID tags. The data from sensors tends to be very high volume and with a high degree of uncertainty, which makes it well-suited for a data lake. Traditional data warehouses don’t just struggle with this data: the sheer amount of data produced in an IoT use case makes a traditional data warehouse-based project extremely expensive and shifts the balance away from a good return on that investment for all but a small number of cases.

    Relational databases (referred to as RDBMS) are one of the most common data sources to be integrated into a data platform. Today RDBMS power most of the applications responsible for business operations and contain valuable information about core business transactions such as payments, orders, bookings, etc. Data in RDBMS is organized into tables, each containing one or more columns. Columns have clear data types (such as strings, integers, dates etc.) associated with them and RDBMS takes care of making sure that data with the wrong type doesn’t get inserted into the database. Data in relational databases is typically highly normalized, meaning data entities are split into multiple tables that can be joined using a common key at query time. As such it’s not unusual to have databases with hundreds or even thousands of different tables in them. Finally, because of the operational nature of these databases, data in these databases changes all the time: new orders arrive, get changed or canceled or delivered.

    If we look at this list from another angle, we will see a few things. First of all, we need some kind of connector that allows our application to read data from RDBMS. Luckily for us, RDBMS have been around for decades and there are database connectors (or drivers) available for just about every RDBMS vendor that can be used with most popular programming languages.

    Next, if you look carefully at the diagram you will see that in the RDBMS ingestion process, data will need to travel through at least three different levels as shown below:

    Figure 4. 3. Different levels of an ingestion process

    On Level 1, data is stored using native RDBMS types. Then when a SQL query is executed by an ingestion application, these native RDBMS data types will be converted into a data type that the specific programming language in which the ingestion application is implemented, supports (Level 2). Finally when the ingestion application saves data into the landing area of the cloud data platform, data types must be converted once again into the data types that your chosen file format supports.(Level 3). Since data types mapping is not always 1 to 1, and given that during the ingestion process from RDBMS database data types change at least twice, data types mapping is important and we will talk more about this later in this chapter.

    Finally, if all your ingestion application can do is execute SELECT * FROM some_table then you are limited to doing only full table ingestion. In many scenarios this is not enough and a more sophisticated way of identifying and ingesting new or changed data is required.

    Listing 1. Python 3 example of reading a JSON file from a landing area on GCS and saving it into staging area in Avro format. You need to make sure your GCP Dataproc cluster has permissions to read and write data to the appropriate GCS buckets.
    import datetime
    from pyspark.sql import SparkSession
     
    spark = SparkSession.builder ... # we omit Spark session creation for brevity
     
    namespace = “ETL”    #A
    pipeline_name = “click_stream_ingest”
    source_name = “clicks”
    batch_id = “01DH3XE2MHJBG6ZF4QKK6RF2Q9”
    current_date = datetime.datetime.now()
    in_path = f“gs://landing/{namespace}/{pipeline_name}/{source_name}/{batch_id}/*”    #B
    out_path = f”gs://staging/{namespace}/{pipeline_name}/{source_name}/year={current_date.year}/month={current_date.month}/day={current_date.day}/{batch_id}”
     
    clicks_df = spark.read.json(in_path)    #C
    clicks_df = spark.write.format(“avro”).save(out_path)    #D
    Cloud Native Patterns: Designing change-tolerant software

    This is an excerpt from Manning's book Cloud Native Patterns: Designing change-tolerant software.

    Figure 1.8. Key entities in the model for cloud-native software: apps, data, and interactions

    Notice that although at the start I talked about services, in the end they aren’t one of the three entities in this mental model. In large part, this is because pretty much everything is a service, both apps and data. But more so, I suggest that the interactions between services are even more interesting than a service alone. Services pervade through the entire cloud-native software model.

    With this model established, let’s come back to the modern software requirements covered in section 1.1 and consider their implications on the apps, data, and interactions of your cloud-native software.

    Take, for example, redundancy. Although the value of having multiple copies of data has been long understood, the patterns for achieving it in the past were often bolted on, and sometimes came through operational practices. For example, in active/passive deployment topologies, the active node is serving all read and write traffic, while the passive node is being updated with writes behind the scenes. If something happens to the active node, the whole system could fail over to what had been the passive node. Modern cloud-native data services (recall that in section 5.4.1, I talked about the special types of services that are the stateful ones) have redundancy built deeply into their multinode designs and employ patterns such as Paxos and leader/follower to drive the consistency and availability characteristics they deliver.

    Figure 12.8. The event-driven approach allows the Connections’ Posts service to operate with complete autonomy, using the data in the local storage. Missed events, however, can corrupt that local storage, “successfully” returning incorrect results.

    To Connections’ Posts, everything seems right in the world. It will return a result based on the data in its local storage, exactly as it should. It has no idea that the data is now not accurate. And it gets worse. It’s not usually the case that an event is of interest to only a single party; instead, many other entities may care if a user in the system changes their username, for example. If that change event doesn’t reach many intended participants, inconsistencies can propagate widely through your system, as shown in figure 12.9.

    Because of this level of indirection, the Posts service needs to map user IDs to usernames. The Posts service doesn’t manage users (the Connections service does), but in order to remain decoupled from the Connections service, it must keep in its own database an up-to-date copy of some of the data managed by the Connections service; namely, the correlation of ID to username. You can see that table through the following SQL query:

    mysql> select * from cookbookposts.user;
    +----+-----------+
    | id | username  |
    +----+-----------+
    |  1 | cdavisafc |
    |  2 | madmax    |
    |  3 | gmaxdavis |
    +----+-----------+

    Okay, so in figure 12.12, both the Connections and the Posts services are generating events, and the Posts and the Connections’ Posts services are consuming events. In that diagram, you can also see the data stores associated with each of your services. The Connections and Posts services each own data that’s stored in a source-of-truth database, and the Posts and Connections’ Posts services store copies of data owned by another service in local storage databases. These local databases are kept up to date through eventing.

  • When you give a microservice a database to store the data that it needs to fulfill its job, it realizes a significant gain in autonomy. As a result, your system as a whole will be more resilient.
  • Google Cloud Platform in Action

    This is an excerpt from Manning's book Google Cloud Platform in Action.

    Storage, although not the most glamorous part of computing, is incredibly necessary. Imagine if you weren’t able to save your data when you were done working on it? Cloud’s take on storage follows the same pattern you’ve seen so far with computing, abstracting away the management of your physical resources. This might seem unimpressive, but the truth is that storing data is a complicated thing to do. For example, do you want your data to be edge-cached to speed up downloads for users on the internet? Are you optimizing for throughput or latency? Is it OK if the “time to first byte” is a few seconds? How available do you need the data to be? How many concurrent readers do you need to support?

    Listing 5.10. Exporting data to Cloud Storage
    $ gcloud beta datastore export gs://my-data-export/export-1
    Waiting for [projects/your-project-id-here/operations/
         ASA1MTIwNzE4OTIJGnRsdWFmZWQHEmxhcnRuZWNzdS1zYm9qLW5pbWRhFAosEg] to
         finish...done.
    metadata:
      '@type':
         type.googleapis.com/google.datastore.admin.v1beta1.ExportEntitiesMetadata
      common:
        operationType: EXPORT_ENTITIES
        startTime: '2018-01-16T14:26:02.626380Z'
        state: PROCESSING
      outputUrlPrefix: gs://my-data-export/export-1
    name: projects/your-project-id-here/operations/
         ASA1MTIwNzE4OTIJGnRsdWFmZWQHEmxhcnRuZWNzdS1zYm9qLW5pbWRhFAosEg
    Figure 6.22. Reading data after it’s been changed doesn’t cause transaction failures.

    In addition to storing data correctly, a primary design goal of these systems is to reduce complexity of the underlying disks and data centers and instead provide a simple API for uploading and retrieving files, a bit like key-value storage for large values with automatic replication and caching around the world.

    8.9.1. Amount of data stored

    Data storage is the simplest and most obvious component of your Cloud Storage bill and should remind you of other storage providers like Drop Box. Every month, Cloud Storage charges you based on the amount of data you keep in your bucket measured in gigabytes per month, prorated on how long the object was stored. If you store an object for 15 out of 30 days, your bill for a single 2 GB object will be 2 (GB) * 0.026 (USD) * 15/30 (months), which is 31 cents. And if you store it for only 1 hour (1/24th of one day) out of a 31-day month, your data storage cost will be 2 (GB) * 0.026 (USD) * (1/24) days / 31 (days in the month), which is effectively zero ($0.000069892). The data storage component gets even cheaper if you change to different storage classes such as Nearline or Coldline.

    The Cloud at Your Service: The when, how, and why of enterprise cloud computing

    This is an excerpt from Manning's book The Cloud at Your Service: The when, how, and why of enterprise cloud computing.

    Over the past four decades, there have been tremendous changes in hardware capabilities, specifically in computing power and storage. The ability to quickly process prodigious amounts of data on inexpensive and mass-produced commodity servers means that a few inexpensive racks of servers can handle problems that were tackled on NSA-sized budgets as recently as the early 1990s.

    Figure 5.6. Rebalancing data. Initially, the GetDatabaseFor() function pushed requests for partition A to the center server. But that server’s database shard has gotten too large and needs to be rebalanced. All or a portion of database A is moved to the server on the left. After that data is successfully moved, the function GetDatabaseFor() is modified so that future requests for shard A are directed to the leftmost server.
    Data are Kept Small

    The larger a set of data a server handles, the harder it is to cache intelligently, because you have such a wide diversity of data being accessed. You need huge amounts of RAM that may not even be enough to cache the data when you need it. By isolating data into smaller shards, the data you’re accessing is more likely to stay in cache. Smaller sets of data are also easier to back up, restore, and manage.

    Azure in Action

    This is an excerpt from Manning's book Azure in Action.

    Applications can’t subsist on code alone; they usually need to store and retrieve data to provide any real value. In the next section, we’ll discuss how Azure provides you with shared storage, and then we’ll take a quick tour of the BLOB storage service, messaging, and the Table storage service. Each of these is covered in detail in their related sections later in this book.

    As you can see, the data listed is fairly static, and it’s applied across all shirts. All shirts will have a size and a material (admittedly not wool). This data can be considered static, because it’s unlikely that it would ever be changed once it’s defined. (Hawaiian shirts are unlikely to suddenly start being made in platinum.)

    When developing an ASP.NET web page, you should consider making asynchronous calls to improve performance, but most developers will typically write synchronous calls to retrieve the data. Figure 14.3 shows the synchronous call sequence for the product details web page.

    Figure 14.3. Synchronous call sequence; each result must be returned before you can make the next call

    As you can see, the synchronous nature of the page means you have to wait until you receive data back from the data provider (SQL Azure or the Table service) before you can process the next section of the page. Due to the blocking nature of synchronous calls and the latency involved in cross-server communication, the rendering of this page will be much slower than it needs to be.

    One of the issues with splitting data across multiple servers is keeping referential integrity.

    In the shopping cart example, the data that the shopping cart is referencing is either static (size types, materials, or personage types) or infrequently changing (and administratively controlled), so referential integrity isn’t such a big deal. You’re ultimately controlling the data, so you can break foreign-key relationships and store the various tables on separate servers.

    If you do need to maintain referential integrity, you could keep a copy of the static data on each instance of the database. Although there is data duplication, there’s such a small amount of data that you can easily fix the problem by synchronizing the databases. In this case, you’d generally keep one master database allowing one-way synchronization.