Chapter 3. Handling requests

Chapter 2 showed the steps for creating a simple internet forum web application. The chapter mapped out the various parts of a Go web application, and you saw the big picture of how a Go web application is structured. But there’s little depth in each of those parts. In the next few chapters, we’ll delve into the details of each of these parts and explore in depth how they can be put together.

In this and the next chapter, we’ll focus on the brains of the web application: the handlers that receive and process requests from the client. In this chapter, you’ll learn how to create a web server with Go, and then we’ll move on to handling requests from the client.

3.1. The Go net/http library

Although using a mature and sophisticated web application framework to write web applications is usually easy and fast, the same frameworks often impose their own conventions and patterns. Many assume that these conventions and patterns are best practices, but best practices have a way of growing into cargo cult programming when they aren’t understood properly. Programmers following these conventions without understanding why they’re used often follow them blindly and reuse them when it’s unnecessary or even harmful.

The reason data is persisted as cookies in the client and sessions in the server is because HTTP is a connection-less protocol, and each call to the server has no stored knowledge of the previous call. Without this understanding, using cookies and sessions seems a convoluted way of persisting information between connections. Using a framework to get around this complexity is smart because a framework normally hides the complexity and presents a uniform interface for persistence between connections. As a result, a new programmer would simply assume all it takes to persist data between connections is to use this interface. This uniform interface is based on the conventions of a specific framework, though, and such practices might or might not be consistent across all frameworks. What’s worse, the same interface name might be used in different frameworks, with different implementations and different names, adding to the confusion. This means that the web application that’s developed is now tied to the framework; moving it to another framework or even extending the application or adding new features requires deep knowledge of the framework (or customized versions of the framework).

This book isn’t about rejecting frameworks or conventions or patterns. A good web application framework is often the best way to build scalable and robust web applications quickly. But it’s important to understand the underlying concepts infrastructure that these frameworks are built on. In the case of the Go programming language, using the standard libraries typically means using the net/http and html/template libraries. With proper understanding, it becomes easier to see why certain conventions and patterns are what they are. This helps us to avoid pitfalls, gives clarity, and stops us from following patterns blindly.

In this and the next chapter, we’ll be focusing on net/http; chapter 5 covers html/template.

The net/http library is divided into two parts, with various structs and functions supporting either one or both (see figure 3.1):

We’ll start by using the net/http library as the server, and in this chapter we’ll talk about how Go handles requests from the client. In the next chapter, we’ll continue with the net/http library but focus on using it to process the request.

In this book, we’ll focus on using the net/http library’s server capabilities and not its client capabilities.

3.2. Serving Go

The net/http library provides capabilities for starting up an HTTP server that handles requests and sends responses to those requests (see figure 3.2). It also provides an interface for a multiplexer and a default multiplexer.

3.2.1. The Go web server

Unlike most standard libraries in other programming languages, Go provides a set of libraries to create a web server. Creating a server is trivial and can be done with a call to ListenAndServe, with the network address as the first parameter and the handler that takes care of the requests the second parameter, as shown in the following listing. If the network address is an empty string, the default is all network interfaces at port 80. If the handler parameter is nil, the default multiplexer, DefaultServeMux, is used.

This simple server doesn’t allow much configuration, but Go also provides a Server struct that’s essentially a server configuration.

The following listing does almost the same thing as the previous code but now allows more configurations. Configurations include setting the timeout for reading the request and writing the response and setting an error logger for the Server struct.

3.2.2. Serving through HTTPS

Most major websites use HTTPS to encrypt and protect the communications between the client and the server when confidential information like passwords and credit card information is shared. In some cases, this protection is mandated. If you accept credit card payments, you need to be compliant with the Payment Card Industry (PCI) Data Security Standard, and to be compliant you need to encrypt the communications between the client and the server. Some sites like Gmail and Facebook use HTTPS throughout their entire site. If you’re planning to run a site that requires the user to log in, you’ll need to use HTTPS.

HTTPS is nothing more than layering HTTP on top of SSL (actually, Transport Security Layer [TLS]). To serve our simple web application through HTTPS, we’ll use the ListenAndServeTLS function, shown in listing 3.4.

In the previous listing, the cert.pem file is the SSL certificate whereas key.pem is the private key for the server. In a production scenario you’ll need to get the SSL certificate from a CA like VeriSign, Thawte, or Comodo SSL. But if you need a certificate and private key only to try things out, you can generate your own certificates. There are many ways of generating them, including using Go standard libraries, mostly under the crypto library group.

Although you won’t use them (the certificate and private key created here) in a production server, it’s useful to understand how an SSL certificate and private key can be generated for development and testing purposes. This listing shows how we can do this.

Generating the SSL certificate and private key is relatively easy. An SSL certificate is essentially an X.509 certificate with the extended key usage set to server authentication, so we’ll be using the crypto/x509 library to create the certificate. The private key is required to create the certificate, so we simply take the private key we created for the certificate and save it into a file for the server private key file.

Let’s go through the code. First, we need to have a Certificate struct, which allows us to set the configuration for our certificate:

We need a certificate serial number, which is a unique number issued by the CA. For our purposes, it’s good enough to use a very large integer that’s randomly generated. Next, we create the distinguished name and set it up as the subject for the certificate, and we also set up the validity period to last for one year from the day the certificate is created. The KeyUsage and ExtKeyUsage fields are used to indicate that this X.509 certificate is used for server authentication. Finally, we set up the certificate to run from the IP 127.0.0.1 only.

Next, we need to generate a private key. We use the crypto/rsa library and call the GenerateKey function to create an RSA private key:

The RSA private key struct that’s created has a public key that we can access, useful when we use the x509.CreateCertificate function to create our SSL certificate:

The CreateCertificate function takes a number of parameters, including the Certificate struct and the public and private keys, to create a slice of DER-formatted bytes. The rest is relatively straightforward: we use the encoding/pem library to encode the certificate into the cert.pem file:

We also PEM encode and save the key we generated earlier into the key.pem file:

Note that if the certificate is signed by a CA, the certificate file should be the concatenation of the server’s certificate followed by the CA’s certificate.

3.3. Handlers and handler functions

Starting up a server is easy, but it doesn’t do anything. If you access the server, you’ll get only a 404 HTTP response code. The default multiplexer that will be used if the handler parameter is nil can’t find any handlers (because we haven’t written any) and will respond with the 404. To do any work, we need to have handlers.

3.3.1. Handling requests

So what exactly is a handler? We talked briefly about handlers and handler functions in chapters 1 and 2, so let’s elaborate here. In Go, a handler is an interface that has a method named ServeHTTP with two parameters: an HTTPResponseWriter interface and a pointer to a Request struct. In other words, anything that has a method called ServeHTTP with this method signature is a handler:

Let me digress and answer a question that might have occurred to you as you’re reading this chapter. If the second parameter for ListenAndServe is a handler, then why is the default value a multiplexer, DefaultServeMux?

That’s because ServeMux (which is what DefaultServeMux is an instance of) has a method named ServeHTTP with the same signature! In other words, a ServeMux is also an instance of the Handler struct. DefaultServeMux is an instance of ServeMux, so it is also an instance of the Handler struct. It’s a special type of handler, though, because the only thing it does is redirect your requests to different handlers depending on the URL that’s provided. If we use a handler instead of the default multiplexer, we’ll be able to respond, as shown in this listing.

Now let’s start the server (if you’re a bit hazy on how to do this, please flip to section 2.7). If you go to http://localhost:8080 in your browser you’ll see Hello World!

Here’s the tricky bit: if you go to http://localhost:8080/anything/at/all you’ll still get the same response! Why this is so should be quite obvious. We just created a handler and attached it to our server, so we’re no longer using any multiplexers. This means there’s no longer any URL matching to route the request to a particular handler, so all requests going into the server will go to this handler.

In our handler, the ServeHTTP method does all the processing. It doesn’t do anything except return Hello World!, so that’s what it does for all requests into the server.

This is the reason why we’d normally use a multiplexer. Most of the time we want the server to respond to more than one request, depending on the request URL. Naturally if you’re writing a very specialized server for a very specialized purpose, simply creating one handler will do the job marvelously.

3.3.2. More handlers

Most of the time, we don’t want to have a single handler to handle all the requests like in listing 3.6; instead we want to use different handlers instead for different URLs. To do this, we don’t specify the Handler field in the Server struct (which means it will use the DefaultServeMux as the handler); we use the http.Handle function to attach a handler to DefaultServeMux. Notice that some of the functions like Handle are functions for the http package and also methods for ServeMux. These functions are actually convenience functions; calling them simply calls DefaultServeMux’s corresponding functions. If you call http.Handle you’re actually calling DefaultServeMux’s Handle method.

In the following listing, we create two handlers and then attach the handler to the respective URL. If you now go to http://localhost:8080/hello you’ll get Hello! whereas if you go to http://localhost:8080/world, you’ll get World!.

3.3.3. Handler functions

We talked about handlers, but what are handler functions? Handler functions are functions that behave like handlers. Handler functions have the same signature as the ServeHTTP method; that is, they accept a ResponseWriter and a pointer to a Request. The following listing shows how this works with our server.

How does this work? Go has a function type named HandlerFunc, which will adapt a function f with the appropriate signature into a Handler with a method f. For example, take the hello function:

If we do this:

then helloHandler becomes a Handler. Confused? Let’s go back to our earlier server, which accepts handlers.

The line that registers the hello function to the URL /hello is

This shows us how the Handle function registers a pointer to a Handler to a URL. To simplify things, the HandleFunc function converts the hello function into a Handler and registers it to DefaultServeMux. In other words, handler functions are merely convenient ways of creating handlers. The following listing shows the code for the http.HandleFunc function.

Here’s the source code for the HandleFunc function:

Notice that handler, a function, is converted into an actual handler by HandlerFunc.

Because using handler functions is cleaner and it does the job just as well, why use handlers at all? It all boils down to design. If you have an existing interface or if you want a type that can also be used as a handler, simply add a ServeHTTP method to that interface and you’ll get a handler that you can assign to a URL. It can also allow you to build web applications that are more modular.

3.3.4. Chaining handlers and handler functions

Although Go isn’t considered a functional language, it has some features that are common to functional languages, including function types, anonymous functions, and closures. As you noticed earlier, we passed a function into another function and we referred to a named function by its identifier. This means we can pass a function f1 into another function f2 for f2 to do its processing, and then call f1 (see figure 3.3).

Let’s work through an example. Say every time we call a handler we want to log it down somewhere that it was called. We can always add this code into the handler, or we can refactor a utility function (as we did in chapter 2) that can be called by every function. Doing this can be intrusive, though; we usually want our handler to contain logic for processing the request only.

Logging, along with a number of similar functions like security and error handling, is what’s commonly known as a cross-cutting concern. These functions are common and we want to avoid adding them everywhere, which causes code duplication and dependencies. A common way of cleanly separating cross-cutting concerns away from your other logic is chaining. This listing shows how we can chain handlers.

We have our usual hello handler function. We also have a log function, which takes in a HandlerFunc and returns a HandlerFunc. Remember that hello is a HandlerFunc, so this sends the hello function into the log function; in other words it chains the log and the hello functions.

The log function returns an anonymous function that takes a ResponseWriter and a pointer to a Request, which means that the anonymous function is a HandlerFunc. Inside the anonymous function, we print out the name of the HandlerFunc (in this case it’s main.hello), and then call it. As a result, we’ll get hello! in the browser and a printed statement on the console that says this:

Naturally if we can chain together two handler functions, we can chain more. The same principle allows us to stack handlers to perform multiple actions, like Lego bricks. This is sometimes called pipeline processing (see figure 3.4).

Say we have another function named protect, which checks for the user’s authorization before executing the handler:

Then to use protect, we simply chain them together:

You might have noticed that while I mentioned earlier that we’re chaining handlers, the code in listing 3.10 actually shows chaining handler functions. The mechanisms for both chaining handlers and handler functions are very similar, as shown next.

Let’s see what’s different. We have our HelloHandler from earlier, which is the last handler in the chain, as before. The log function, instead of taking in a HandlerFunc and returning a HandlerFunc, takes in a Handler and returns a Handler:

Instead of returning an anonymous function, now we adapt that anonymous function using HandlerFunc, which, if you remember from earlier, returns a Handler. Also, instead of executing the handler function, we now take the handler and call its ServeHTTP function. Everything else remains mostly the same, except that instead of registering a handler function, we register the handler:

Chaining handlers or handler functions is a common idiom you’ll find in many web application frameworks.

3.3.5. ServeMux and DefaultServeMux

We discussed ServeMux and DefaultServeMux earlier in this chapter and in the previous chapter. ServeMux is an HTTP request multiplexer. It accepts an HTTP request and redirects it to the correct handler according to the URL in the request, illustrated in figure 3.5.

ServeMux is a struct with a map of entries that map a URL to a handler. It’s also a handler because it has a ServeHTTP method. ServeMux’s ServeHTTP method finds the URL most closely matching the requested one and calls the corresponding handler’s ServeHTTP (see figure 3.6).

So what is DefaultServeMux? ServeMux isn’t an interface, so DefaultServeMux isn’t an implementation of ServeMux. DefaultServeMux is an instance of ServeMux that’s publicly available to the application that imports the net/http library. It’s also the instance of ServeMux that’s used when no handler is provided to the Server struct.

Stepping back a bit, you’ll also come to a realization that ServeMux is also another take on chaining handlers, because ServeMux is a handler.

In these examples the requested URL /hello matches nicely with the registered URL in the multiplexer. What happens if we call the URL /random? Or if we call the URL /hello/there?

It all depends on how we register the URLs. If we register the root URL (/) as in figure 3.6, any URLs that don’t match will fall through the hierarchy and land on the root URL. If we now call /random and don’t have the handler for this URL, the root URL’s handler (in this case indexHandler) will be called.

How about /hello/there then? The Principle of Least Surprise would dictate that because we have the URL /hello registered we should default to that URL and helloHandler should be called. But in figure 3.6, indexHandler is called instead. Why is that so?

The reason is because we registered the helloHandler to the URL /hello instead of /hello/. For any registered URLs that don’t end with a slash (/), ServeMux will try to match the exact URL pattern. If the URL ends with a slash (/), ServeMux will see if the requested URL starts with any registered URL.

If we’d registered the URL /hello/ instead, then when /hello/there is requested, if ServeMux can’t find an exact match, it’ll start looking for URLs that start with /hello/. There’s a match, so helloHandler will be called.

3.3.6. Other multiplexers

Because what it takes to be a handler or even a multiplexer is to implement the ServeHTTP, it’s possible to create alternative multiplexers to net/http’s ServeMux. Sure enough, a number of third-party multiplexers are available, including the excellent Gorilla Toolkit (www.gorillatoolkit.org). The Gorilla Toolkit has two different multiplexers that work quite differently: mux and pat. In this section, we’ll go through a lightweight but effective third-party multiplexer called HttpRouter.

One of the main complaints about ServeMux is that it doesn’t support variables in its pattern matching against the URL. ServeMux handles /threads pretty well to retrieve and display all threads in the forum, but it’s difficult to handle /thread/123 for retrieving and displaying the thread with id 123. To process such URLs, your handler will need to parse the request path and extract the relevant sections. Also, because of the way ServeMux does pattern matching for the URLs, you can’t use something like /thread/123/post/456 if you want to retrieve the post with id 456 from the thread with id 123 (at least not with a lot of unnecessary parsing complexity).

The HttpRouter library overcomes some of these limitations. In this section, we’ll explore some of the more important features of this library, but you can always look up the rest of the documentation at https://github.com/julienschmidt/httprouter. This listing shows an implementation using HttpRouter.

Most of the code should look familiar to you now. We create the multiplexer by calling the New function.

Instead of using HandleFunc to register the handler functions, we use the method names:

In this case we’re registering a URL for the GET method to the hello function. If we send a GET request, the hello function will be called; if we send any other HTTP requests it won’t. Notice that the URL now has something called a named parameter. These named parameters can be replaced by any values and can be retrieved later by the handler.

The handler function has changed too; instead of taking two parameters, we now take a third, a Params type. Params contain the named parameters, which we can retrieve using the ByName method.

Finally, instead of using DefaultServeMux, we pass our multiplexer into the Server struct and let Go use that instead:

But wait. How exactly do we include the third-party library? If we do what we did for the other examples and run go build at the console, we’ll get something like this:

A simple package management system is one of the strengths of Go. We simply need to run

at the console and if we’re connected to the internet, it’ll download the code from the HttpRouter repository (in GitHub) and store it in the $GOPATH/src directory. Then when we run go build, it’ll import the code and compile our server.

3.4. Using HTTP/2

Before leaving this chapter, let me show you how you can use HTTP/2 in Go with what you have learned in this chapter.

In chapter 1, you learned about HTTP/2 and how Go 1.6 includes HTTP/2 by default when you start up a server with HTTPS. For older versions of Go, you can enable this manually through the golang.org/x/net/http2 package.

If you’re using a Go version prior to 1.6, the http2 package is not installed by default, so you need to get it using go get:

Modify the code from listing 3.6 by importing the http2 package and also adding a line to set up the server to use HTTP/2.

In the following listing you can see by calling the ConfigureServer method in the http2 package, and passing it the server configuration, you have set up the server to run in HTTP/2.

Now, run the server:

To check whether the server is running in HTTP/2, you can use cURL. You will be using cURL quite a bit in this book, because it’s widely available on most platforms, so it’s a good time to get familiar with it.

Starting from version 7.43.0, cURL supports HTTP/2. You can perform a request using the HTTP/2 protocol passing the --http2 flag. To use cURL with HTTP/2, you need to link it to nghttp2, a C library that provides support for HTTP/2. As of this writing, many default cURL implementations don’t yet support HTTP/2 (including the one shipped with OS X), so if you need to recompile cURL, link it with nghttp2 and replace the previous cURL version with the one you just built.

Once you have done that, you can use cURL to check your HTTP/2 web application:

Remember, you need to run it against HTTPS. Because you created your own certificate and private key, by default cURL will not proceed as it will try to verify the certificate. To force cURL to accept your certificate, you need to set the insecure flag.

You should get an output similar to this:

We’ve discussed how to handle requests, but we mostly glossed over how to process the incoming request and send responses to the client. Handlers and handler functions are the key to writing web applications in Go, but processing requests and sending responses is the real reason why web applications exist. In the next chapter, we’ll turn to the details on requests and responses and you’ll see how to extract information from requests and pass on information through responses.

3.5. Summary