Appendix. AWS primer
This appendix provides a brief primer on Amazon Web Services to get you up to speed with the AWS environment and services.
Please note that, because Amazon Kinesis Data Streams is not currently available in AWS Free Tier, the procedures in this book necessarily involve creating live resources in your Amazon Web Services account, which can incur some charges.[1] Don’t worry—we will tell you as soon as you can safely delete a given resource. In addition, you can also set alerts on your spending in order to be notified whenever the charges go above a certain threshold.[2]
1 For detailed pricing information on the AWS Kinesis Data Streams service, see https://aws.amazon.com/kinesis/streams/pricing/.
2 You can read how to set spending alerts on your AWS usage here: https://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/billing-getting-started.html#d0e1069.
Amazon Kinesis is a fully hosted service, available only to users of the Amazon Web Services platform. Don’t worry if you haven’t worked with AWS before. This appendix introduces the key building blocks of AWS and will help you get set up on the platform.
To get the most out of this book, you need an AWS account. If you don’t have one, you can sign up by clicking the Get Started for Free or Create a Free Account button on the AWS homepage:
https://aws.amazon.com/
If you are planning on working through these examples in your company’s own AWS account, we highly recommend instead that you ask your company to create a new AWS account, called Developers’ Sandbox or something similar, and connect it to your main company AWS account by using Consolidated Billing.[3] This way, you can experiment in the sandbox safe in the knowledge that you cannot impact (for example, accidentally delete) existing resources in any way.
3 You can read more about how to set up consolidated billing here: https://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/consolidated-billing.html.
After you are signed up, log in to AWS and you should see a dashboard something like the one in figure A.1. The AWS offering is something of a zoo; by the time you read this, the dashboard may even display some new services. We’ve highlighted the services we will be using in this section; see table A.1 for a brief rundown on each of the highlighted services.
Table A.1. AWS Services we’ll be using in this book
| Service |
Short form |
Description |
|---|---|---|
| Identity & Access Management | iam | For securely controlling access to AWS services and resources for your users |
| Kinesis | kinesis | A fully managed unified log service |
Figure A.1. In this book, we will be working with several AWS services, including Kinesis and Identity & Access Management. These are shown highlighted in the AWS dashboard. We will also use other services such as Redshift, S3, and Elastic MapReduce, but these will be introduced directly in the corresponding chapters.
;%22/%3E%3C/svg%3E)
Before we dive into Kinesis, we will first configure some sensible security settings by using Amazon’s Identity & Access Management service. Let’s get started.
As a first step, we are going to use Identity & Access Management (IAM) to create a user with the permissions on AWS resources that we will need for this book. From the AWS dashboard, follow these steps:
- Click the Identity & Access Management icon.
- Click Users in the left-hand navigation pane.
- Click the Add User button.
The next screen is a four-step wizard for creating a user. Go ahead and add in a new user, called ulp for unified log processing, and make sure to select the Programmatic Access option, as shown in figure A.2. Then click the Next: Permissions button.
Figure A.2. We create a new IAM user called ulp in the AWS user interface, making sure to select the Programmatic Access option.
;%22/%3E%3C/svg%3E)
On the next screen, shown in figure A.3, you need to set the permissions for the ulp user. You need to attach what AWS calls a managed policy so that our new ulp user can create new streams in Kinesis, as well as write to and read from those streams. To do this:
- Click the Attach Existing Policies Directly button.
- In the search box, type AmazonKinesisFullAccess.
- Select the policy called AmazonKinesisFullAccess.
- Click the Next: Review button.
These permissions on Kinesis are more generous than they need to be, but they will help you get started with Kinesis with a minimum of fuss. You should reduce the scope of these permissions later as you get more comfortable with AWS.
On the next screen, you can review the details before effectively creating the user, as shown in figure A.4.
;%22/%3E%3C/svg%3E)
If everything seems correct, click the Create User button. This brings us to the next screen, shown in figure A.5. This screen shows a set of user security credentials—consisting of an Access Key ID and a Secret Access key. You can think of these as the username and password for access to the various AWS APIs.
Figure A.5. The user security credentials for our new IAM user consist of an Access Key ID and a Secret Access Key.
;%22/%3E%3C/svg%3E)
Make sure to download these or otherwise note them, as you will need these shortly to set up the AWS CLI. Note that the Secret Access Key is hidden, and make sure to reveal it fully by clicking Show. Also note the link with which you’ll be able to access the AWS Management Console by using the ulp user later.
Now click Close, click Users in the left navigation pane again, and then click your new ulp user. The screen should look like figure A.6.
Next, you need to give our ulp user a password so that you can access the AWS dashboard as this user, rather than as our all-powerful (and thus dangerous) root administrator. Click the Security Credentials tab, and then, on the line that reads Console Password, click the Manage link. A pop-up appears, as shown in figure A.7. Enable Console Access, leave the Autogenerated Password option selected, and click Apply.
;%22/%3E%3C/svg%3E)
A new pop-up appears, showing the generated password. Click the Show option, and make sure to note the password shown under User Security Credentials, as illustrated in figure A.8. Then close the pop-up.
As a final step, let’s log out of AWS by using the Sign Out option in the top-right drop-down, and then log back in as our new ulp user by using the link shown in figure A.4 (also present in the Security Credentials tab), not forgetting the password you wrote down earlier. Now click from the dashboard through to the Kinesis service, and you should see a screen like figure A.9. The prominent Create Kinesis Stream button tells you that you should have all the appropriate permissions to work with Kinesis.
Figure A.9. When you do not yet have any Kinesis streams set up, clicking the Kinesis icon on the AWS dashboard takes you to this Amazon Kinesis Streams screen, with a prominent Create Kinesis Stream button.
;%22/%3E%3C/svg%3E)
With our user set up with appropriate security credentials and Kinesis permissions, the next step is to configure the AWS CLI.
Although the AWS web interface is easy to use, where possible in this book we will work with AWS resources using the official AWS command-line interface (CLI) application. The AWS CLI isn’t much to look at, but its command syntax is quite intuitive, and it tends to get new AWS features before the web interface. The team behind the AWS CLI is also relatively good at maintaining backward compatibility, so hopefully these instructions will continue to work for the foreseeable future!
First, you need to get hold of the CLI application. If you are running this book’s development environment in Vagrant, you are in luck—the AWS CLI is preinstalled. Check that it’s available and working by navigating to your local copy of the Unified-Log-Processing repository and typing this:
host$ vagrant up && vagrant ssh guest$ aws usage: aws [options] <command><subcommand> [parameters] aws: error: too few arguments
If you are not using the prepackaged Vagrant environment, this page in the user guide has all the information you need to install the AWS CLI onto your system:
https://docs.aws.amazon.com/cli/latest/userguide/installing.html
Done? Next, you need to configure an AWS CLI profile for all of your work. By default, the AWS CLI will use an implicit global profile, but it’s safer to work with an explicit, named profile. So type in the following:
$ aws configure --profile=ulp
When prompted, fill in the requested details, supplying the Access Key ID and Secret Access Key you saved earlier:
AWS Access Key ID [None]: AKIAIWSMFSNA2ZH6W4UQ AWS Secret Access Key [None]: uOGIOXssDw/ZtzXxxXxXXxpQvgB3Dus0zFnywWr9 Default region name [eu-west-1]: us-east-1 Default output format [None]:
For this appendix, we are going to assume that you chose us-east-1 as your default AWS region, but it’s fine to choose another region if you prefer; just remember to update the code accordingly. You are now ready to start experimenting with Amazon Kinesis by using the AWS CLI.