Chapter 8: Integrate third-party APIs
The section introduces considerations for integrating external HTTP calls, including failure handling, concurrent operations, and HTTP client choices.
After this chapter you will be able to
- External Request Failure Considerations. Explain why external requests in route handlers require attention to speed, logging, and HTTP responses.
- Concurrent Async Operations. Explain what join! allows a runtime to do and how a blocking operation affects the others.
- HTTP Client Abstraction Choices. Distinguish Hyper and Reqwest by their abstraction levels, and explain how crate features affect what functionality is available.
- Asynchronous HTTP client choice. Explain why the application chooses an asynchronous HTTP client and what compatibility to check.
- Multiple runtime costs. Explain the resource and debugging consequences of using an HTTP client that depends on a different runtime.
- API Selection for Use Case. Explain why the book chooses the Bad Words API and how third-party APIs can enhance a question-and-answer service.
- Endpoint Documentation and Verification. Explain how to use API documentation and direct requests to investigate an endpoint’s requirements and responses.
- Isolated crate exploration. Explain how and why to try a new crate in a minimal project before integrating it into the application.
- Reqwest dependency and example call. Describe how the example prepares Reqwest and adds an HTTP call to the question handler.
- Profanity check placement. Explain the alternative points for checking profanity and the choice made for the first test.
- Reqwest and Warp error mismatch. Explain why the example fails to compile and the role of error conversion in handling the external request.
- Reqwest error adaptation. Explain why the route wraps reqwest::Error in its own Error enum and how map_err and ? propagate it.
- Request and response errors. Distinguish errors from sending a request from unsuccessful HTTP responses, and explain the role of error_for_status.
- Typed response struct advantages. Explain why the external API's responses and errors should be represented with structs such as BadWordsResponse and BadWordsError.
- API response discovery. Describe how to gather enough information to define response structs using API documentation and a curl request.
- Response guided type design. Explain how to use an API response to sketch a Rust type and why the type need not include every JSON field.
- Generic JSON response inspection. Describe how Serde JSON's generic Value helps inspect responses before defining a custom type.
- External API error classification. Explain how the route distinguishes API HTTP errors from Reqwest client errors and handles 4xx and 5xx responses.
- Internal and external errors. Explain why the application retains detailed API errors internally but returns a predefined error to its client.
- API content sanitization. Explain how the application will use the API to handle profane question and answer content.
- Reusable profanity check module. Explain how the external API code is extracted into a reusable profanity module and made accessible to the route handler.
- Question profanity checking. Describe how the updated handler checks a question's title and content, and explain the tradeoff against sending them together.
- Question Update Data Types. Distinguish the types used by update_question and add_question, explaining how the question ID affects the choice.
- Answer content profanity check. Explain what the add_answer handler checks and what the profanity API returns when it finds blocklisted words.
- Timeout Retry Strategy. Explain how retries can address a delayed external API response before the service returns an error.
- Concurrent and Parallel Calls. Distinguish the Tokio options described for running multiple HTTP calls concurrently or in parallel.
- Exponential backoff behavior. Explain how exponential backoff retries a failed HTTP call and why its added complexity may not always be worthwhile.
- Reqwest middleware integration. Describe how the example adds retry support to Reqwest using middleware and the profanity abstraction.
- Middleware error handling. Describe how the middleware changes the error returned by the client's POST call and how the example adapts its error handling.
- Concurrency and parallelism. Distinguish concurrency from parallelism when executing HTTP calls.
- Tokio execution choices. Compare tokio::spawn and tokio::join! for the HTTP calls and explain how to choose between them.
What it builds on
- Introducing the Tracing crate (chapter 6) · The Tracing crate provides context for observability.
- Error handling and tracing database interactions (chapter 7) · Database interaction error handling and tracing provide a prior example of managing an external dependency.
- Rust’s handling of async/await (chapter 2) · Rust's async and await concepts provide context for async operations.
- Choosing a runtime (chapter 2) · The application has a runtime whose compatibility with an async client must be considered.
- Taking an excursion into moving, borrowing, and ownership (chapter 2) · Rust ownership and borrowing can complicate new code.
- Handling results (chapter 2) · The explanation assumes familiarity with Rust results and error propagation.
- Handle errors gracefully (chapter 3) · The service has previously addressed error handling for Warp responses.
- Implementing logging in your web service (chapter 6) · Logging in the web service
Section by section
Introduction
The section introduces considerations for integrating external HTTP calls, including failure handling, concurrent operations, and HTTP client choices.
- External Request Failure Considerations: Explain why external requests in route handlers require attention to speed, logging, and HTTP responses.
Why it matters: External dependencies can fail independently of your service, so their effects need to be visible and understandable. - Concurrent Async Operations: Explain what join! allows a runtime to do and how a blocking operation affects the others.
Why it matters: Understanding this distinction helps you avoid assuming that joining operations protects them from a blocking task. - HTTP Client Abstraction Choices: Distinguish Hyper and Reqwest by their abstraction levels, and explain how crate features affect what functionality is available.
Why it matters: Choosing an appropriate abstraction and enabling needed features makes HTTP integration more convenient.
Key terms: route handler, Tracing, HTTP responses, join!, async operations, concurrently, parallel, thread, Hyper, Reqwest
8.1 Preparing the codebase
This section explains the choice of an asynchronous HTTP client and the costs of using one that depends on a different runtime.
- Asynchronous HTTP client choice: Explain why the application chooses an asynchronous HTTP client and what compatibility to check.
Why it matters: The client choice affects both request performance and compatibility with the application’s runtime. - Multiple runtime costs: Explain the resource and debugging consequences of using an HTTP client that depends on a different runtime.
Why it matters: Choosing a client for its API alone can add runtime overhead and make the application harder to diagnose.
Key terms: Reqwest, HTTP client, asynchronous, runtime, Tokio runtime, operating system thread, runtime complexity, bottlenecks
8.1.1 Picking an API
The section selects a profanity-checking API for the question-and-answer service and explains how to investigate its requirements, responses, and integration choices.
- API Selection for Use Case: Explain why the book chooses the Bad Words API and how third-party APIs can enhance a question-and-answer service.
Why it matters: The API should support the specific functionality your service needs and be practical to integrate. - Profanity Handling Design Choices: Describe the design questions that arise when deciding how the service should handle profane text.
Why it matters: A profanity check does not by itself decide how your application stores or displays content. - Endpoint Documentation and Verification: Explain how to use API documentation and direct requests to investigate an endpoint’s requirements and responses.
Why it matters: Checking real responses prevents you from relying on documentation that may be out of date. - Complex Endpoint Logic Separation: Describe when endpoint integration logic might be moved to a separate microservice and what that service would do.
Why it matters: Separating complex integration logic is an option when a route handler would otherwise take on too much responsibility.
Key terms: third-party APIs, Bad Words API, profanity check, profane words, questions and answers, website setting, API documentation, API key, header, JSON
8.1.2 Getting to know our HTTP crate
This section introduces a small-project workflow for exploring Reqwest and testing an API call before integrating it into the application.
- Isolated crate exploration: Explain how and why to try a new crate in a minimal project before integrating it into the application.
Why it matters: A small experiment helps you separate unfamiliar crate behavior from complications in your main application. - API call exploration: Describe how the section adapts and explores an example HTTP request for the chosen API.
Why it matters: A working example gives you a baseline for learning the HTTP crate and API together. - API key safety: State why the example's hard-coded API key should not be copied into production code.
Why it matters: Example code can make compromises that would expose secrets in a real application.
Key terms: Cargo project, Rust Playground, happy path, borrow checker, ownership model, HTTP POST request, API key, parameters, Reqwest, documentation
8.1.3 Adding an example HTTP call with Reqwest
This section adds an example Reqwest call to the question handler and identifies the error-type mismatch that prevents it from compiling.
- Reqwest dependency and example call: Describe how the example prepares Reqwest and adds an HTTP call to the question handler.
Why it matters: This establishes the first external HTTP call in the service. - Profanity check placement: Explain the alternative points for checking profanity and the choice made for the first test.
Why it matters: The placement determines whether the example changes content before storage or allows display-time filtering. - Reqwest and Warp error mismatch: Explain why the example fails to compile and the role of error conversion in handling the external request.
Why it matters: The external call cannot compile in the handler until its errors are handled in a way compatible with Warp.
Key terms: Reqwest, Cargo.toml, JSON feature set, add_question, Bad Words API, censored sentence, warp::Reject, Reqwest error, try!, handle-errors crate
8.1.4 Handling errors for external API requests
This section adapts Reqwest errors to Warp, detects unsuccessful API responses, and separates client-facing errors from diagnostic logs.
- Reqwest error adaptation: Explain why the route wraps reqwest::Error in its own Error enum and how map_err and ? propagate it.
Why it matters: This lets external request failures fit the route handler's existing error handling. - Request and response errors: Distinguish errors from sending a request from unsuccessful HTTP responses, and explain the role of error_for_status.
Why it matters: Checking only the request error can let an API failure look like success. - Client and diagnostic messages: Explain why the example returns a general error to the client while logging more detail internally.
Why it matters: Different messages protect internal details while still helping developers diagnose failures.
Key terms: reqwest::Error, Error enum, Reject, map_err, Result, send, Response, error_for_status, 4xx, 5xx
8.2 Deserializing JSON responses to structs
This section explains why typed API responses are useful and how to gather information for defining their structs.
- Typed response struct advantages: Explain why the external API's responses and errors should be represented with structs such as BadWordsResponse and BadWordsError.
Why it matters: Typed structs make external responses easier to understand, check, and work with. - API response discovery: Describe how to gather enough information to define response structs using API documentation and a curl request.
Why it matters: Inspecting both documentation and actual responses helps you model an external API despite imperfect documentation.
Key terms: typed responses, BadWordsError, BadWordsResponse, JSON, type-check, API documentation, curl, prototyping, structs
8.2.1 Gathering API response information
This section shows how to inspect API responses and use their contents to guide Rust type design and error handling.
- Response guided type design: Explain how to use an API response to sketch a Rust type and why the type need not include every JSON field.
Why it matters: Inspecting a response helps you design a useful type without modeling fields you do not need. - Generic JSON response inspection: Describe how Serde JSON's generic Value helps inspect responses before defining a custom type.
Why it matters: Generic parsing lets you explore responses before deciding what custom types to write. - Response error information: Describe what error information can be inspected in a response and how response status can be classified.
Why it matters: Response details and status categories give you information for more sophisticated error handling.
Key terms: Transform tool, deserialize, JSON, struct, Serde JSON, Value enum, generic JSON, error code, pretty-print, res.is_success
8.2.2 Creating types for our API responses
This section introduces structs for API responses and handles external API errors while keeping internal details from clients.
- API response structs: State how the section represents successful API responses in Rust.
Why it matters: Response structs give the application a Rust representation of the API's successful data. - External API error classification: Explain how the route distinguishes API HTTP errors from Reqwest client errors and handles 4xx and 5xx responses.
Why it matters: An HTTP error returned by the API needs handling even when the Reqwest client itself has not thrown an error. - Internal and external errors: Explain why the application retains detailed API errors internally but returns a predefined error to its client.
Why it matters: This separation supports production diagnosis without exposing sensitive implementation details to clients.
Key terms: structs, API responses, success path, response.status, is_client_error, is_server_error, APILayerError, 4xx, 5xx, internal and external errors
8.3 Sending questions and answers to the API
This section introduces the chosen approach for sanitizing question and answer content through a third-party API.
- API content sanitization: Explain how the application will use the API to handle profane question and answer content.
Why it matters: This approach defines how new content is sanitized before the route handlers use it.
Key terms: profane words, sanitized, API
8.3.1 Refactoring the add_question route handler
This section extracts profanity checking into a reusable module and updates question creation to check text before storing it.
- Reusable profanity check module: Explain how the external API code is extracted into a reusable profanity module and made accessible to the route handler.
Why it matters: The extraction lets other routes reuse the same external-call logic. - Question profanity checking: Describe how the updated handler checks a question's title and content, and explain the tradeoff against sending them together.
Why it matters: This explains how question text is checked before the application stores it. - Censorship verification: Describe how to test that a new question is saved with censored words.
Why it matters: This gives you a way to verify the result in storage, not just the HTTP response.
Key terms: profanity.rs, mod profanity, public function, error handling, title, content, profanity check, censored content, cargo run, curl
8.3.2 Making profanity checks for updating questions
This section distinguishes the data types used to update an existing question and add a new one.
- Question Update Data Types: Distinguish the types used by update_question and add_question, explaining how the question ID affects the choice.
Why it matters: The distinction explains why creating and updating a question use different types.
Key terms: update_question, add_question, Question, NewQuestion, id
8.3.3 Updating the add_answer route handler
The add_answer handler checks answer content for profanity before content is added to the database.
- Answer content profanity check: Explain what the add_answer handler checks and what the profanity API returns when it finds blocklisted words.
Why it matters: This check extends profanity filtering to answers before content reaches the database.
Key terms: add_answer, profanity check, blocklist, censored version
8.4 Handling timeouts and multiple requests at once
This section introduces retrying timed-out external API calls and distinguishes concurrent from parallel HTTP calls using Tokio.
- Timeout Retry Strategy: Explain how retries can address a delayed external API response before the service returns an error.
Why it matters: Retries give the service a chance to handle a timeout before reporting an error to the user. - Concurrent and Parallel Calls: Distinguish the Tokio options described for running multiple HTTP calls concurrently or in parallel.
Why it matters: Running multiple calls together can serve performance or usability needs.
Key terms: timeout, retries, 500 server error, tokio::join!, tokio::spawn, futures, concurrently, parallel, task
8.4.1 Implementing a retry for external HTTP calls
This section adds exponential-backoff retries to the profanity HTTP client using Reqwest middleware and adjusts error handling.
- Exponential backoff behavior: Explain how exponential backoff retries a failed HTTP call and why its added complexity may not always be worthwhile.
Why it matters: This strategy avoids retrying every failed call at a fixed interval, but adds complexity to the client. - Reqwest middleware integration: Describe how the example adds retry support to Reqwest using middleware and the profanity abstraction.
Why it matters: The wrapper makes retry middleware available without changing every caller of the profanity abstraction. - Middleware error handling: Describe how the middleware changes the error returned by the client's POST call and how the example adapts its error handling.
Why it matters: Wrapping the client changes the error type that the application must handle. - Offline retry test: Describe the section's suggested way to test the retry functionality.
Why it matters: Disconnecting the network provides a simple way to exercise the retry behavior.
Key terms: exponential backoff, retry, timeout, reqwest_retry, reqwest_middleware, Reqwest, profanity abstraction, reqwest_middleware::Error, reqwest::Error, handling-errors
8.4.2 Executing futures concurrently or in parallel
This section distinguishes concurrency from parallelism, compares Tokio approaches for running HTTP calls, and reflects on Rust’s support for the API integration.
- Concurrency and parallelism: Distinguish concurrency from parallelism when executing HTTP calls.
Why it matters: This distinction helps you reason about different ways to run multiple HTTP calls. - Tokio execution choices: Compare tokio::spawn and tokio::join! for the HTTP calls and explain how to choose between them.
Why it matters: The two approaches offer different execution behavior, and neither is automatically the best choice. - Rust integration support: Explain how Rust’s typing and ecosystem helped with the chapter’s HTTP-client integration.
Why it matters: These features helped make the integration safer and reduced the amount of retry code to write.
Key terms: concurrency, parallelism, tokio::spawn, tokio::join!, futures, context switching, return type, strictly typed, crate, retry
8.5 Summary
Where this is used later
- Read and parse environment variables into your web service (chapter 10) · Addresses reading environment variables into the web service, a way to keep configuration out of source code.
- Testing the profanity module with a newly created mock server (chapter 11) · The profanity check used when adding a question.
Concepts explained in this chapter
- reqwest::Errorreqwest::Error is Reqwest's error type for failures from HTTP client operations.
- reqwestReqwest is a Rust HTTP client library supporting both synchronous and asynchronous requests.
- request messageAn HTTP request is a message sent by a client to a server using the Hypertext Transfer Protocol.
- crateA crate is a Rust package unit that can be compiled and distributed as a library or executable.
- documentationSoftware documentation is written information describing software, its behavior, or how to use it.
- authenticationAuthentication is the process of verifying the identity of a user or other entity.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
