Articles on Jan Pieter Bruins Slot - Software Engineer | Software Consultant

Making a simple HTTP webserver in C

Mon, 29 Nov 2021 08:00:00 +0000

Introduction

While reading, and working through Operating Systems: Three Easy pieces¹ I started with one of the assignments for making a webserver concurrent. The webserver was graciously provided by the authors, and the authors recommend figuring stuff out yourself, and do some more in-depth research. So, I thought it would be a cool idea to implement the webserver myself, as the basis for making it concurrent as the assignment² asks you to do, and in the process learn more about socket programming. Additionally, I thought it would be a good learning experience, to create a post of the process, and share it.

So, in this post we will go over creating a simple webserver in C, step-by-step. First, we’ll start with some background into webservers, and subsequently add more code to our program. I’ve tried to make it so that you’ll be able to figure out what you need to code before looking at the results. However, if you just want to look at a specific part or just at the end result, then check the Implementation sections throughout the article. You can also look at the resulting code in this repository.

So, let’s get to it!

Basics

First, it might be good idea to get a feel of what we’re trying to create, to investigate what it is that we actually want to make. So, let’s figure out what a webserver actually is, and does.

webserver

A webserver, and in this case on the software side is able to satisfy client requests over HTTP, and other several related protocols. Its primary function is to store, process, deliver files to that client. At minimum this is an HTTP Server, which is a piece of software that understand URLs (Universal Resource Locator) and HTTP (Hypertext Transfer Protocol).

A webserver can be either be, or a combination of, a static webserver which simply serves files “as-is”. Or, a dynamic webserver in which the webserver runs an executable file on the webserver, and the output is returned to the client. It is dynamic because the webserver updates the hosted files before sending them to the client it does this “on-the-fly”.³

HTTP

As mentioned above a webserver in our case should be able to understand the HTTP protocol. What does a protocol mean in this context? Well, it is a set of rules for communication between two computers. In this case it specifies how to transfer hyper text documents, meaning documents that are interconnected by hyperlinks. The protocol is textual and stateless. Textual because that all the commands are plain text, and you’ll be able to read and inspect it. Stateless, because that neither the client nor server remembers previous communications.⁴

It means on the client side the application (for instance a web browser) needs to speak the same ‘language’ as the webserver in order to communicate. The ‘language’ that is used is HTTP.

A message is constructed that is either a request or a response from either the client or the server. This message needs to be transported, and that is where TCP comes in.

TCP

HTTP presumes an underlying transport layer protocol to establish host-to-host data transfer channels, and manage the data exchange in a client-to-server or peer-to-peer networking model. The protocol that is commonly used for HTTP server is TCP (Transmission Control Protocol), but it can also be adapted to be used with for instance UDP (User Datagram Protocol). However, because of RFC 2616⁵ that states that the transport layer should be reliable, we will be using TCP instead of UDP.⁶ ⁷

TCP maintains communications between application processes between hosts (client and server), and they use port numbers to track sessions. HTTP, and TCP protocols are part of a suite of several multiple protocols on each layer of the request/response cycle. This suite is also known as: TCP/IP.

Internet Protocol Suite a.k.a. TCP/IP

HTTP is part of the Internet Protocol Suite and it is called an application layer protocol. The Internet Protocol Suite is a model that is commonly known as TCP/IP because of the foundational protocols that make up the Internet Protocol Suite. Namely, the Transmission Control Protocol (TCP, present on the transport layer), and Internet Protocol (IP, present on the internet layer). ⁸

This suite is a conceptual model, and it consists out of set of protocols used in internet, and similar computer networks. It specifies how data should be packetized, addressed, transmitted, routed and received. The model is made up of 4 abstraction layers: the application, transport, internet and link layer.

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ APPLICATION LAYER                ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ http, ftp, smtp, ssh, etc.       │
└──────────────────────────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ TRANSPORT LAYER                  ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ tcp, udp, etc.                   │
└──────────────────────────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ INTERNET LAYER                   ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ ipv4, ipv6, etc.                 │
└──────────────────────────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ LINK LAYER                       ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ arp, mac (ethernet, wifi, etc.)  │
└──────────────────────────────────┘

We stated above that: “HTTP presumes an underlying transport layer”. From the Internet Protocol Suite we can see that there are several available, and we already stated that TCP is commonly used. We can also see that the HTTP is in the application layer, next to other protocols that you might be familiar with such as FTP, Telnet, SSH, SMTP, etc. In essence HTTP is layered over TCP and uses it to transport its message data. In turn TCP is layered over IP, to make sure it ends up at the right location.

When data to the application layer is received, by which every program you are using (in the case of HTTP, a browser for instance), it talks to the transport layer through a port. Each port can be assigned to a different protocol in the application layer. In the case of HTTP this is port 80, so that TCP knows where the data is coming from.

Request / Response

As mentioned above: “HTTP is a set of rules for communication”. These rules are implemented in the request and response messages. You’re probably already familiar with its structure. The request message consist out of the following: a request line, the request header fields, an empty line, and an optional message body. In the following diagram you can see how a request and response message is built up when we access the webserver we are going to create using curl.

                                                                        
            ┌─┐                                        ┌─┐              
            └┬┘              ―――――――――――――▶            ╞ │              
            ▔▔▔              ◀―――――――――――――            └─┘              
    client request message                    server response message   
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┓              ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ GET / HTTP/1.1            ┃ request line ┃ HTTP/1.0 200 OK           ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━┩              ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ Host: localhost:8080      │    headers   │ Server: webserver-c       │
│ User-Agent: curl/7.52.1   │              │ Content-type: text/html   │
│ Accept: */*               │              │                           │
└───────────────────────────┘              ├───────────────────────────┤
                                  body     │ hello, world │
                                           └───────────────────────────┘

We can even inspect what curl is sending and receiving, and we can see that everything is just in plain text. Pretty cool!

$ curl -vs http://localhost:8080

* Rebuilt URL to: http://localhost:8080/
*   Trying ::1...
* TCP_NODELAY set
* connect to ::1 port 8080 failed: Connection refused
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 8080 (#0)
> GET / HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/7.52.1
> Accept: */*
>
* HTTP 1.0, assume close after body
< HTTP/1.0 200 OK
< Server: webserver-c
< Content-type: text/html
<
hello, world
* Curl_http_done: called premature == 0
* Closing connection 0

When a request is made by the application layer, the message passes through the layers on one side, and back up through layer on the other side (represented by the solid line). Logically, one layer talks to the corresponding layer at the other side (represented by the dashed line).

                                         
      ┌─┐                       ┌─┐      
      └┬┘                       ╞ │      
      ▔▔▔                       └─┘      
     client                    server    
┏━━━━━━━━━━━━━┓╷╷       ╭╮┏━━━━━━━━━━━━━┓
┃ APPLICATION ┃││ ◀---▶ ││┃ APPLICATION ┃
┗━━━━━━━━━━━━━┛││       ││┗━━━━━━━━━━━━━┛
┏━━━━━━━━━━━━━┓│▲       ▼│┏━━━━━━━━━━━━━┓
┃ TRANSPORT   ┃││ ◀---▶ ││┃ TRANSPORT   ┃
┗━━━━━━━━━━━━━┛││       ││┗━━━━━━━━━━━━━┛
┏━━━━━━━━━━━━━┓││       ││┏━━━━━━━━━━━━━┓
┃ INTERNET    ┃││ ◀---▶ ││┃ INTERNET    ┃
┗━━━━━━━━━━━━━┛▼│       │▲┗━━━━━━━━━━━━━┛
┏━━━━━━━━━━━━━┓││       ││┏━━━━━━━━━━━━━┓
┃ LINK        ┃││ ◀---▶ ││┃ LINK        ┃
┗━━━━━━━━━━━━━┛│╰───────╯│┗━━━━━━━━━━━━━┛
               ╰─────────╯

So, to recap: we’re creating an application (a webserver) that is able to receive/send plain text messages that adhere to rules of HTTP. Those messages are received through a transport layer. Our application will use TCP as this transport layer, and we need to implement that transport layer of HTTP. Both of these protocols are part of the Internet Protocol Suite (TCP/IP) and is provided by the operating system.⁹

Implementation

We can use the man pages to reference on how to start implementing this. Throughout this article we will be using man pages to get all the information we need to implement our webserver. The first man page we can look at is:

$ man 7 tcp

NAME
       tcp - TCP protocol

SYNOPSIS
       #include 
       #include 
       #include 

       tcp_socket = socket(AF_INET, SOCK_STREAM, 0);

DESCRIPTION

       ...

       A newly created TCP socket has no remote or local address and is
       not fully specified. To create an outgoing TCP connection use
       connect(2) to establish a connection to another TCP socket. To
       receive new incoming connections, first bind(2) the socket to a
       local address and port and then call listen(2) to put the socket
       into the listening state. After that a new socket for each
       incoming connection can be accepted using accept(2). A socket
       which has had accept(2) or connect(2) successfully called on it
       is fully specified and may transmit data. Data cannot be
       transmitted on listening or not yet connected sockets.  

       ...

NOTE The 7 stands for the section number the page is from, and you can check what section it is by typing man man. Typically, man pages referred to using the notation name(section), since the same name can be present in different sections. Throughout this document we will use this notation so that you’ll be able to inspect the man pages. If you’re trying to find a specific man page, you can use the apropos {name} command to find name usage through the man pages.

From this man page we can read that we need to implement a ‘socket’ on which we can ‘listen’ for incoming connections, then we need to ‘bind’ the socket to a local address, and port. Then put the socket in a ‘listen’ state. After that we’re able to ‘accept’ incoming connections, for each accepted connection a new socket will be created, and we will be able to read and write to this socket. The following diagram gives a bit of an overview of what we need to implement.

                                                                                    
┏━━━━━━━━━━━━┓   ┏━━━━━━━━━━━━┓   ┏━━━━━━━━━━━━┓   ┏━━━━━━━━━━━━┓   ┏━━━━━━━━━━━━━━┓
┃   SOCKET   ┃ ▶ ┃    BIND    ┃ ▶ ┃   LISTEN   ┃ ▶ ┃   ACCEPT   ┃ ▶ ┃  READ/WRITE  ┃
┗━━━━━━━━━━━━┛   ┗━━━━━━━━━━━━┛   ┗━━━━━━━━━━━━┛   ┗━━━━━━━━━━━━┛   ┗━━━━━━━━━━━━━━┛

The tcp(7) man page also states, as we’ve uncovered from above, that it is layered upon ip(7), so let’s also take a look at the man page for that.

$ man 7 ip

NAME
       ip - Linux IPv4 protocol implementation

SYNOPSIS
       #include 
       #include 
       #include  /* superset of previous */

       tcp_socket = socket(AF_INET, SOCK_STREAM, 0);
       udp_socket = socket(AF_INET, SOCK_DGRAM, 0);
       raw_socket = socket(AF_INET, SOCK_RAW, protocol);

Ok, cool! This gives us some more information on how to create sockets using other protocols, like udp, and raw. We’ll keep that in mind for further on in this article. But first let’s start with setting up our project.

Setting up

First, let’s start with setting up our environment in which we want to develop. We will start very simple, just to make sure everything is working:

// ./steps/step000.c
#include 

int main() {
    printf("hello, world\n");
    return 0;
}

Next we need to compile it, and we will be using gcc.

$ gcc -Wall webserver.c -o webserver

Let’s check if every worked.

$ ./webserver
hello, world

Ok, now that we’re set up. Lets get started with implementing our socket.

Implement the socket

From what we’ve read from the man page tcp(7), we need to implement a tcp socket. But, let’s also inspect what a socket is, and we can also use the man pages for this.

$ man socket

NAME
       socket - create an endpoint for communication

SYNOPSIS
       #include           /* See NOTES */
       #include 

       int socket(int domain, int type, int protocol);

DESCRIPTION
       socket() creates an endpoint for communication and returns a file
       descriptor that refers to that endpoint. The file descriptor returned by
       a successful call will be the lowest-numbered file descriptor not
       currently open for the process.
       ...

So, a socket creates an endpoint for communication. Furthermore, we can read that we need to include the header file, and that we can create a socket endpoint to communicate by using the function: socket(2). This function returns a file descriptor which is an integer. The arguments that it accepts are: domain, type, protocol. We will look at the individual arguments and investigate how they need to be set in the following sections.

domain

The argument domain is an integer that specifies a communication domain, and it selects the protocol family that which will be used for communication. These families are defined in these families are defined as constants in the header file, and we can reference them by their name and use them as the domain argument.

See the man page for an overview of what kind formats you’re able to choose. Since we’re creating a webserver that uses TCP we will be using AF_INET, which uses the IPv4 Internet protocols.

NOTE Now, I wanted to know how this header file looked like, and I was able to inspect it further by installing the POSIX man page. These are described as: “Manual pages about using a POSIX system for development”. And will give more information about the specification of a C standard library for POSIX systems. It’s a specification for a number of routines that should be available in a basic C standard library, and it depends on how this standard C library is implemented on a system. The most commonly used implementation on Linux is the GNU C Library: glibc. With these manpages we can thus reference the specification.¹⁰ You can also find these pages on the following site: The Open Group.

How I installed the posix man pages on a debian based distro of linux:
$ sudo apt install manpages-posix-dev
And was able to reference the man page for with the following command:
$ man sys_socket.h

type

The argument type specifies the ‘communication semantics’. So, which socket type do we need to use here? Well, we said we wanted to create a TCP webserver, so which of the options resembles that? Let’s refer back to man pages of tcp(7), and ip(7). There we can see that the valid socket type for a TCP socket is SOCK_STREAM. SOCK_STREAM is a full-duplex byte stream, and it is characterized as a type that ensures that data is not lost or duplicated.

protocol

protocol, according to the man page, is the particular protocol to be used with the socket. It is common that there exists only one protocol that will support a specific socket type. In our case where we are choosing SOCK_STREAM as the type, and as stated by the man 7 ip man page, protocol is the IP protocol in the IP header to be received or sent. And in this case of creating a TCP socket were the valid value is 0 for TCP sockets.

return value

The int socket(int domain, int type, int protocol) function returns an integer which is an file descriptor for the socket.¹¹ The file descriptor is an unique number that identifies an open file, in this case this is our socket and just as a regular file we will be able to read and write to it. When an error occurred it will return the value -1, it will also set an errno which we can use to properly handle errors.

errno

From the last section the return value of the socket function the errno will be set. So what is this errno? Let’s check if there is a man page about it.

$ man errno

NAME
       errno - number of last error

SYNOPSIS
       #include 

DESCRIPTION
       The  header file defines the integer variable errno, which is
       set by system calls and some library functions in the event of an error
       to indicate what went wrong.

errno is an integer variable that can be set to signify what exactly has gone wrong. And in order to inspect what kind of error was raised, we can use perror(3) to print the error, it will translate the error code that has been set in the variable errno to a human-readable form. Lets check the man page for perror(3).

$ man 3 perror

NAME
       perror - print a system error message

SYNOPSIS
       #include 

       void perror(const char *s);

DESCRIPTION
       The perror() function produces a message on standard error describing
       the last error encountered during a call to a system or library
       function.
       ...

We can use perror(3) and set the argument s with a string, and it then will be appended with an error message that corresponds with the current value of errno.

Implementation

Now that we know how we should implement the socket(2) function, let’s update our file with what we have discussed above. It should resemble something like this:

// ./steps/step001.c
#include 
#include 
#include 

int main() {
    // Create a socket
    int sockfd = socket(AF_INET, SOCK_STREAM, 0);
    if (sockfd == -1) {
        perror("webserver (socket)");
        return 1;
    }
    printf("socket created successfully\n");

    return 0;
}

Now when we check our diagram from the section Basics we’ve now created the socket, but we need to bind it to an address otherwise no communication can be sent or received to this socket as we have read from the man page tcp(7).

                                                                                  
┏━━━━━━━━━━━━┓   ┏━━━━━━━━━━━━┓   ┏━━━━━━━━━━━━┓   ┏━━━━━━━━━━━━┓   ┏━━━━━━━━━━━━┓
┃   SOCKET   ┃ ▶ ┃    BIND    ┃ ▶ ┃   LISTEN   ┃ ▶ ┃   ACCEPT   ┃ ▶ ┃ READ/WRITE ┃
┗━━━━━━━━━━━━┛   ┗━━━━━━━━━━━━┛   ┗━━━━━━━━━━━━┛   ┗━━━━━━━━━━━━┛   ┗━━━━━━━━━━━━┛
 ───────────▶

So our next step will be to bind the socket to a local address and port.

Bind the socket to an address

The socket is created and exists in a namespace (an address family, the AF in AF_INET stands for address family), and we need to bind the socket to a local address, in order for the socket to receive connections. We need to be using the bind(2) function for this, so let’s check out the bind(2) man pages on how we need to implement this.

$ man bind

NAME
       bind - bind a name to a socket

SYNOPSIS
       #include           /* See NOTES */
       #include 

       int bind(int sockfd, const struct sockaddr *addr,
                socklen_t addrlen);

DESCRIPTION                                                              
       When a socket is created with socket(2), it exists in a name space
       (address family) but has no address assigned to it. bind() assigns the
       address specified by addr to the socket referred to by the file
       descriptor sockfd. addrlen specifies the size, in bytes, of the address
       structure pointed to by addr. Traditionally, this operation is called
       “assigning a name to a socket”.
    
       It is normally necessary to assign a local address using bind() before
       a SOCK_STREAM socket may receive connections (see accept(2)).

       ...

We can see that bind(2) is included in the header file, . And, on success it will return zero. It accepts as arguments: sockfd, *addr, and addrln. Let’s go over the arguments, and make sense of what we need to do in order to implement it.

sockfd

This is the file descriptor that we’ve created with socket(2) in the last section. And we need to use that here as the first argument.

addr

This defines the address structure to which we want to bind the socket to, and it depends on the address family we’re using. So let’s check what addr needs to look like. We can inspect the rules used in the name binding, by referencing the man page of the communication domain we’re using: AF_INET.

$ man 7 ip

From the section ‘Address format’ we can see an example. The address structure will look like the following:

struct sockaddr_in {
   sa_family_t    sin_family; /* address family: AF_INET */
   in_port_t      sin_port;   /* port in network byte order */
   struct in_addr sin_addr;   /* internet address */
};

/* Internet address. */
struct in_addr {
   uint32_t       s_addr;     /* address in network byte order */
};

sin_family is always set to AF_INET, sin_port contains the port in network byte order. Network byte order represents how bytes are arranged when sending data over a network, an order must be chosen to make sure that on both ends, the machines interpret the numbers the same way independent of the cpu architecture.

For example an integer value of 1 represented as 4 bytes would be represented on ‘big endian’ machines as 0 0 0 1, on a ‘little endian’ machines this would be 1 0 0 0. The value of 0 0 0 1 of the ‘big endian’ machine would then be interpreted by the ‘little endian’ machine as the value 16777216, and vice versa.¹² ¹³

And, as such, like the man page states, we need to call htons(3) on the number that is assigned to the port. Like so: htons(8080). It will convert the host byte order to network byte order. See the man page at for htons(3) for more information.

sin_addr contains the host interface address in network byte order, and it is a member of the struct named in_addr. The man page states that it should be one of the INADDR_* values. These are defined as symbolic constants in the header file , or can set it by using one of the inet_aton(3), inet_addr(3), or inet_makeaddr(3) library functions, to specify a specific address. We can also inspect the POSIX man page to see how the header file should be implemented on systems: man netinet_in.h.

We will make use of the symbolic constant INADDR_ANY, and it means ‘any address’, which translates to the 0.0.0.0. INADDRY_ANY is already in network byte order, so we don’t really have to convert it. The man page advices us to convert, so lets just implement it. We do this by calling htonl(3) on the address. But why are we using 0.0.0.0, here? This is just your machine’s IP address. Your machine will have one IP address for each network interface. When your machine has for example Wi-Fi, and an ethernet connection, then that machine will have two addresses, one for each interface. When we don’t care what interface is going to be used we use the special address for this, 0.0.0.0 which is defined in the symbolic constant INADDR_ANY translates to this address.

addrlen

The addrlen argument specifies the size of the address structure addr in bytes. To get this we can use the sizeof() operator (it looks like a function, but it is an operator like &&, ||, etc.). The argument is of type socklen_t which is an integer type, and we can get some background on specific type by inspecting the man page of accept(2).

In the original BSD sockets implementation (and on other older systems)
the third argument of accept() was declared as an int *. A POSIX.1g
draft standard wanted to change it into a size_t *C; later POSIX
standards and glibc 2.x have socklen_t *.

return value

On success the return value for bind(2) will be zero, when an error occurred it will return -1, and errno will also be set.

Implementation

From what we have seen we’re able to implement the bind(2) function. First, we will create the address structure, and then we can bind it to the socket. The updated code will look something like this:

// ./steps/step002.c
#include 
#include 
#include 
#include 

#define PORT 8080

int main() {
    // Create a socket
    int sockfd = socket(AF_INET, SOCK_STREAM, 0);
    if (sockfd == -1) {
        perror("webserver (socket)");
        return 1;
    }
    printf("socket created successfully\n");

    // Create the address to bind the socket to
    struct sockaddr_in host_addr;
    int host_addrlen = sizeof(host_addr);

    host_addr.sin_family = AF_INET;
    host_addr.sin_port = htons(PORT);
    host_addr.sin_addr.s_addr = htonl(INADDR_ANY);

    // Bind the socket to the address
    if (bind(sockfd, (struct sockaddr *)&host_addr, host_addrlen) != 0) {
        perror("webserver (bind)");
        return 1;
    }
    printf("socket successfully bound to address\n");

    return 0;
}

Note that we are typecasting addr to the struct pointer struct sockaddr * in the argument of the bind(2) function. Since addr is of the type struct sockaddr_in we need to cast it to struct sockaddr *. From the man page bind(2) we can read: “The only purpose of this structure (sockaddr) is to cast the structure pointer passed in addr in order to avoid compiler warnings.” In essence what we are doing here is: whatever addr is pointing to, act like a sockaddr.¹⁴

                                                                                  
┏━━━━━━━━━━━━┓   ┏━━━━━━━━━━━━┓   ┏━━━━━━━━━━━━┓   ┏━━━━━━━━━━━━┓   ┏━━━━━━━━━━━━┓
┃   SOCKET   ┃ ▶ ┃    BIND    ┃ ▶ ┃   LISTEN   ┃ ▶ ┃   ACCEPT   ┃ ▶ ┃ READ/WRITE ┃
┗━━━━━━━━━━━━┛   ┗━━━━━━━━━━━━┛   ┗━━━━━━━━━━━━┛   ┗━━━━━━━━━━━━┛   ┗━━━━━━━━━━━━┛
 ───────────▶     ───────────▶

Referring back to our diagram, we’ve now also bound the socket to a specific address. Now we can are ready to listen for incoming connections. So lets implement that.

Listen

We’ve created a socket and bounded it to a local address, now we need to make sure that the socket is listening for incoming connection. We do that by using listen(2) function. This will make the socket available for incoming connections. Let’s see what the man pages can show us on how to use listen(2).

$ man 2 listen

NAME
       listen - listen for connections on a socket

SYNOPSIS
       #include           /* See NOTES */
       #include 

       int listen(int sockfd, int backlog);

DESCRIPTION
       listen() marks the socket referred to by sockfd as a passive socket,
       that is, as a socket that will be used to accept incoming connection
       requests using accept(2).
       ...

As we can see, the listen(2) function will put the socket into ‘passive’ mode. Stream sockets are often ‘active’ or ‘passive’.

When a socket is created with the socket(2) function, it is set to active. This socket can then be used in the connect(2) function to establish a connection to a ‘passive’ socket.
A passive socket can allow incoming connections by passing it to the listen(2) function.

In most applications that use stream sockets, the servers we will perform the so called ‘passive socket open’, and on the client an ‘active socket open’. Since we’re creating a http webserver, and using the listen(2) function in order to listen for incoming connections, the socket that we’ve created will be a passive socket, and will be used to accept connect connections from other (active) sockets. ¹⁵

sockfd

Again, this is the file descriptor of the socket, and thus we will use the sockfd that we’ve created in section 3.

backlog

This integer will define how many pending connections will be queued up for sockfd socket, before it will be refused. For now, we will set this to 128. Further connection requests block until a pending connection is accepted. So, it defines the number of connections that are accepted, but not yet handled by the application, until accept(2) gets it off the queue.

From listen(2) the ‘NOTES’ section:

...

The behavior of the backlog argument on TCP sockets changed with Linux 2.2. Now
it specifies the queue length for completely established sockets waiting to be
accepted, instead of the number of incomplete connection requests. The maximum
length of the queue for incomplete sockets can be set using
/proc/sys/net/ipv4/tcp_max_syn_backlog. When syncookies are enabled there is no
logical maximum length and this setting is ignored. See tcp(7) for more
information.

If the backlog argument is greater than the value in
/proc/sys/net/core/somaxconn, then it is silently truncated to that value;
the default value in this file is 128. In kernels before 2.4.25, this limit was
a hard coded value, SOMAXCONN, with the value 128.

...

The symbolic constant SOMAXCONN in is defined by our system (128 in the case of Linux), and we can use it to set the backlog argument (man sys_socket.h).

return value

On success, zero will be returned, on failure -1 will be returned, and as before errno will also set, so we can check and handle it accordingly.

Implementation

With the above information we are able to implement the listen(2) function, so let’s update our code:

// ./steps/step003.c
#include 
#include 
#include 
#include 

#define PORT 8080

int main() {
    // Create a socket
    int sockfd = socket(AF_INET, SOCK_STREAM, 0);
    if (sockfd == -1) {
        perror("webserver (socket)");
        return 1;
    }
    printf("socket created successfully\n");

    // Create the address to bind the socket to
    struct sockaddr_in host_addr;
    int host_addrlen = sizeof(host_addr);

    host_addr.sin_family = AF_INET;
    host_addr.sin_port = htons(PORT);
    host_addr.sin_addr.s_addr = htonl(INADDR_ANY);

    // Bind the socket to the address
    if (bind(sockfd, (struct sockaddr *)&host_addr, host_addrlen) != 0) {
        perror("webserver (bind)");
        return 1;
    }
    printf("socket successfully bound to address\n");

    // Listen for incoming connections
    if (listen(sockfd, SOMAXCONN) != 0) {
        perror("webserver (listen)");
        return 1;
    }
    printf("server listening for connections\n");

    return 0;
}

Referring back to our diagram, we’ve created a socket, bound it to a local address, and we’ve put the socket into ‘passive’ mode. Now we can listen for incoming connections.

                                                                                  
┏━━━━━━━━━━━━┓   ┏━━━━━━━━━━━━┓   ┏━━━━━━━━━━━━┓   ┏━━━━━━━━━━━━┓   ┏━━━━━━━━━━━━┓
┃   SOCKET   ┃ ▶ ┃    BIND    ┃ ▶ ┃   LISTEN   ┃ ▶ ┃   ACCEPT   ┃ ▶ ┃ READ/WRITE ┃
┗━━━━━━━━━━━━┛   ┗━━━━━━━━━━━━┛   ┗━━━━━━━━━━━━┛   ┗━━━━━━━━━━━━┛   ┗━━━━━━━━━━━━┛
 ───────────▶     ───────────▶     ───────────▶

So, on to accept those connections.

Accept

Now we’re ready to make sure the socket will accept connections. We need to use the accept(2) function, and let’s check the man pages again on how we need to implement this.

$ man 2 accept

NAME
       accept, accept4 - accept a connection on a socket

SYNOPSIS
       #include           /* See NOTES */
       #include 

       int accept(int sockfd, struct sockaddr *addr, socklen_t *addrlen);

DESCRIPTION
       The accept() system call is used with connection-based socket types
       (SOCK_STREAM, SOCK_SEQ‐ PACKET). It extracts the first connection
       request on the queue of pending connections for the listening socket,
       sockfd, creates a new connected socket, and returns a new file
       descriptor referring to that socket. The newly created socket is not in
       the listening state. The original socket sockfd is unaffected by this
       call.
       ...

So the accept(2) function will get the first connection from the queue of the listening socket sockfd. Then it will create a new connected socket, and the file descriptor that points to that socket will be returned. The newly created socket is however not in a listening state, and thus the original socket is not affected by this call and can be used to accept other connections. When there are no pending connections when the accept(2) function is called, the call blocks until a new connection arrives.

Again, let’s look at the arguments that we need to provide to accept connections.

sockfd

Like before, we will use the original socket that was created in Implement the socket and here sockfd is the file descriptor of the socket.

addr

The argument addr is a pointer that refers to a sockaddr struct, this needs to be the address of the original socket that we’ve created, and we need the pointer to that struct here.

addrlen

The addrlen is a value result argument, it points to the size of the buffer pointed to by the argument addr. Because accept() can accept multiple protocol families we need to provide the size of the address that we are using. A pointer is used because: “the caller must initialize it to contain the size (in bytes) of the structure pointed to by addr; on return it will contain the actual size of the peer address.” The kernel then knows how much space is available to return the socket address. Upon return from the accept(2) function, the value of addrlen is set to indicate the number of bytes of data actually stored by the kernel in the socket address structure. ¹⁶

When binding our socket (Bind the socket to an address), we’ve already created our addrlen variable with the size of the sockaddr struct, so we can just pass it to the accept(2) function. However, the original variable was an int, so we need to typecast it to socklen_t * to make it work.

return value

It will return a non-negative integer that is a file descriptor for the accepted socket. On error, it will return -1, and errno will be set.

Implementation

Because we want to continue accepting new connections we will put the accept(2) function in a continuous loop. Important to note is that we also need to close the file descriptor we’ve created by using accept(2). We can close the socket by calling the close(2) function.

$ man 2 close

NAME
       close - close a file descriptor

SYNOPSIS
       #include 

       int close(int fd);

DESCRIPTION
       close() closes a file descriptor, so that it no longer refers to any
       file and may be reused. Any record locks (see fcntl(2)) held on the file
       it was associated with, and owned by the process, are removed
       (regardless of the file descriptor that was used to obtain the lock).
       ...

When we’re done with the socket we can just use it as the argument fd in the function close(2) this will close the file descriptor, so that it no longer refers to any file and may be reused. When we update our code and implement the accept(2) function it should resemble the following:

// ./steps/step004.c
#include 
#include 
#include 
#include 
#include 

#define PORT 8080

int main() {
    // Create a socket
    int sockfd = socket(AF_INET, SOCK_STREAM, 0);
    if (sockfd == -1) {
        perror("webserver (socket)");
        return 1;
    }
    printf("socket created successfully\n");

    // Create the address to bind the socket to
    struct sockaddr_in host_addr;
    int host_addrlen = sizeof(host_addr);

    host_addr.sin_family = AF_INET;
    host_addr.sin_port = htons(PORT);
    host_addr.sin_addr.s_addr = htonl(INADDR_ANY);

    // Bind the socket to the address
    if (bind(sockfd, (struct sockaddr *)&host_addr, host_addrlen) != 0) {
        perror("webserver (bind)");
        return 1;
    }
    printf("socket successfully bound to address\n");

    // Listen for incoming connections
    if (listen(sockfd, SOMAXCONN) != 0) {
        perror("webserver (listen)");
        return 1;
    }
    printf("server listening for connections\n");

    for (;;) {
        // Accept incoming connections
        int newsockfd = accept(sockfd, (struct sockaddr *)&host_addr,
                               (socklen_t *)&host_addrlen);
        if (newsockfd < 0) {
            perror("webserver (accept)");
            continue;
        }
        printf("connection accepted\n");

        close(newsockfd);
    }

    return 0;
}

And when we check our diagram again, we can that we’ve implemented the accept(2) function.

                                                                                  
┏━━━━━━━━━━━━┓   ┏━━━━━━━━━━━━┓   ┏━━━━━━━━━━━━┓   ┏━━━━━━━━━━━━┓   ┏━━━━━━━━━━━━┓
┃   SOCKET   ┃ ▶ ┃    BIND    ┃ ▶ ┃   LISTEN   ┃ ▶ ┃   ACCEPT   ┃ ▶ ┃ READ/WRITE ┃
┗━━━━━━━━━━━━┛   ┗━━━━━━━━━━━━┛   ┗━━━━━━━━━━━━┛   ┗━━━━━━━━━━━━┛   ┗━━━━━━━━━━━━┛
 ───────────▶     ───────────▶     ───────────▶     ───────────▶

Now, and we’re now ready to starting reading and writing to the socket.

Read

If we recall back from man page of socket(2) we read the following:

...

A connection to another socket is created with a connect(2) call. Once
connected, data may be transferred using read(2) and write(2) calls or some
variant of  the send(2) and recv(2) calls.  When a session has been completed
a close(2) may be performed.

...

We can read and write by using the read(2) and write(2) functions, or some variant of send(2) and recv(2) calls. From the man page of send(2) we can read that send(2) provides extra flags that we might use. In this case we won’t be using those, and as such we can stick with read(2) and write(2). ¹⁷

Because we’ve setup a connection between the client and the server we can read the request of the client. Since, we got a file descriptor with the accept(2) function, we will be able to use the read(2) function to read the data that has been sent by the client. Let’s check the man pages on how we’re able to use the read(2) function.

$ man 2 read

NAME
       read - read from a file descriptor

SYNOPSIS
       #include 

       ssize_t read(int fd, void *buf, size_t count);

DESCRIPTION:
       read() attempts to read up to count bytes from file descriptor fd into
       the buffer starting at buf.
       ...

The function read(2) will read up to count bytes from the file descriptor fd into the buffer *buf. On success it will return the number of bytes that were read, and the file position is advanced by this number. On error -1 is returned and errno will be set as well.

The file position keeps track of where in the file the next character is to be read or written. This is ‘offset’ being recorded by the kernel. ¹⁸ On all POSIX.1 systems, the file position is an integer representing the number of bytes from the beginning of the file. The file position is normally set to the beginning of the file when it is opened, and each time a character is read or written, the file position is incremented sequentially. ¹⁹

fd

This argument fd needs to be the file descriptor, this is the new socket that was returned by the accept(2) function.

buf

The buf argument needs to be a pointer to the address of the memory buffer that you want the contents of the file descriptor fd to be read into as a temporary storage. This buffer must be at least count bytes long. In our case we will be creating a buffer that will be an array of the type char. And because the array name is converted to pointer, we can use the variable name of the buffer as the argument.

count

We need to provide how many bytes we want to ready from the file descriptor fd into the buffer. This depends on how large of a buffer you’re creating. In this example we’ll create an array of 2048 characters.

The type size_t is a unsigned integer type. It is commonly used by the standard library to represent sizes and counts. Its specific size is platform dependent.

return value

The return value is the number of bytes that were read into the buffer, or 0 if the end of the file has been reached. On error, -1 is returned, and errno is set appropriately. The ssize_t is a ‘signed’ integer type, again it is commonly used by the standard library to represent sizes and counts, and it holds the byte count of what was read into the buffer.

Implementation

First, we need to implement the buffer, we will do this as soon as the program begins. For now, we will create an array of the type char with a size of 2048. (each char is 1 byte). Don’t forget to include the header file uninstd.h.

You’ll also be able to check the number of bytes that are read, and continue reading when the limit of the buffer is reached. However for now we’ll keep it simple.

Additionally, because the requests is likely sent using the HTTP protocol. We can expect a certain format of the request, as we’ve uncovered that in the beginning. Here you’ll be able to inspect the contents of the request and return data based on the request, like return specific requested html file, handling GET, POST requests, as well as error handling for when the request wasn’t using the HTTP protocol for instance, etc.

// ./steps/step005.c
#include 
#include 
#include 
#include 
#include 

#define PORT 8080
#define BUFFER_SIZE 1024

int main() {
    char buffer[BUFFER_SIZE];

    // Create a socket
    int sockfd = socket(AF_INET, SOCK_STREAM, 0);
    if (sockfd == -1) {
        perror("webserver (socket)");
        return 1;
    }
    printf("socket created successfully\n");

    // Create the address to bind the socket to
    struct sockaddr_in host_addr;
    int host_addrlen = sizeof(host_addr);

    host_addr.sin_family = AF_INET;
    host_addr.sin_port = htons(PORT);
    host_addr.sin_addr.s_addr = htonl(INADDR_ANY);

    // Bind the socket to the address
    if (bind(sockfd, (struct sockaddr *)&host_addr, host_addrlen) != 0) {
        perror("webserver (bind)");
        return 1;
    }
    printf("socket successfully bound to address\n");

    // Listen for incoming connections
    if (listen(sockfd, SOMAXCONN) != 0) {
        perror("webserver (listen)");
        return 1;
    }
    printf("server listening for connections\n");

    for (;;) {
        // Accept incoming connections
        int newsockfd = accept(sockfd, (struct sockaddr *)&host_addr,
                               (socklen_t *)&host_addrlen);
        if (newsockfd < 0) {
            perror("webserver (accept)");
            continue;
        }
        printf("connection accepted\n");

        // Read from the socket
        int valread = read(newsockfd, buffer, BUFFER_SIZE);
        if (valread < 0) {
            perror("webserver (read)");
            continue;
        }

        close(newsockfd);
    }

    return 0;
}

Write

Now that we are able to read the message that the client has sent to us, we also want to relay something back to the client again. Because we’re implementing a webserver, we’re going to return a simple webpage. Again, because we are implementing a HTTP webserver we need to adhere to the HTTP protocol. That means that we need to structure our response to these rules.

We will be using the same socket the accept(2) function that we’ve just read from. Because this socket is a file descriptor we will, just as with read(2), be able to write to this socket using the write(2) function.

$ man 2 write

NAME
       write - write to a file descriptor

SYNOPSIS
       #include 

       ssize_t write(int fd, const void *buf, size_t count);

       write() writes up to count bytes from the buffer starting at buf to the
       file referred to by the file descriptor fd.
       ...

The function write(2) will write bytes up to count from the buffer pointed to by buf to the file referenced by the file descriptor fd. On success it will return number of bytes written. On error -1 is returned and errno will be set appropriately.

fd

As mentioned above the argument fd is the file descriptor that references the socket we’ve created by calling the accept(2) function. This is the also the same file descriptor from which we read the request with the function read(2).

buf

The buf arguments need to be a pointer to what we want to write to as a response. For now we will use a pre-defined string that we will add to the server. We can eventually extend this webserver to serve actual html files. But for now we will add the following:

char resp[] = "HTTP/1.0 200 OK\r\n"
"Server: webserver-c\r\n"
"Content-type: text/html\r\n\r\n"
"hello, world\r\n";

Note, that the string is formatted following the HTTP protocol. (See: Basics) We start with the request line, followed by the headers, and it ends with the body. The escape code \r\n is used to separate the different sections of the request. The escape code \r stands for carriage return and will set the cursor at the beginning of the line, and \n for new line and will move the cursor to a new line.

count

The argument count is the number of bytes we need write to the file fd from buffer buf. Because we want to write the complete contents we need to know how many bytes there are in the buffer. We do that by using strlen().

$ man 3 strlen

NAME
       strlen - calculate the length of a string

SYNOPSIS
       #include 

       size_t strlen(const char *s);

DESCRIPTION
       The  strlen() function calculates the length of the string pointed to by
       s, excluding the terminating null byte ('\0').

RETURN VALUE
       The strlen() function returns the number of characters in the string
       pointed to by s.

So, we can provide the s argument and we will get the number of characters we provided in the string point to by s.

return value

The return value of the write(2) function will be the number of bytes written to the file and its type is ssize_t. From before we’ve noted that size_t was used to represent sizes and counts, this version is the signed version of size_t, which means that it can hold values less than zero. So, in this case a value of less than zero, -1 is means an error occurred and errno will be set appropriately. When the return value is zero, it indicates that nothing was written. It will not mean that an error occurred when this number is smaller than the number of bytes that were requested.

Implementation

We will implement this first by creating the buffer, and we will follow the instructions we’ve mentioned above. Because, we are done with writing to the newly created socket we need to also close it. We’ve already called close(2) in the last section when we used the read(2) function. Now before we close it though, we want to write to it. So put the write(2) function above the close(2) function.

#include 
#include 
#include 
#include 
#include 
#include 

#define PORT 8080
#define BUFFER_SIZE 1024

int main() {
    char buffer[BUFFER_SIZE];
    char resp[] = "HTTP/1.0 200 OK\r\n"
                  "Server: webserver-c\r\n"
                  "Content-type: text/html\r\n\r\n"
                  "hello, world\r\n";

    // Create a socket
    int sockfd = socket(AF_INET, SOCK_STREAM, 0);
    if (sockfd == -1) {
        perror("webserver (socket)");
        return 1;
    }
    printf("socket created successfully\n");

    // Create the address to bind the socket to
    struct sockaddr_in host_addr;
    int host_addrlen = sizeof(host_addr);

    host_addr.sin_family = AF_INET;
    host_addr.sin_port = htons(PORT);
    host_addr.sin_addr.s_addr = htonl(INADDR_ANY);

    // Bind the socket to the address
    if (bind(sockfd, (struct sockaddr *)&host_addr, host_addrlen) != 0) {
        perror("webserver (bind)");
        return 1;
    }
    printf("socket successfully bound to address\n");

    // Listen for incoming connections
    if (listen(sockfd, SOMAXCONN) != 0) {
        perror("webserver (listen)");
        return 1;
    }
    printf("server listening for connections\n");

    for (;;) {
        // Accept incoming connections
        int newsockfd = accept(sockfd, (struct sockaddr *)&host_addr,
                               (socklen_t *)&host_addrlen);
        if (newsockfd < 0) {
            perror("webserver (accept)");
            continue;
        }
        printf("connection accepted\n");

        // Read from the socket
        int valread = read(newsockfd, buffer, BUFFER_SIZE);
        if (valread < 0) {
            perror("webserver (read)");
            continue;
        }

        // Write to the socket
        int valwrite = write(newsockfd, resp, strlen(resp));
        if (valwrite < 0) {
            perror("webserver (write)");
            continue;
        }

        close(newsockfd);
    }

    return 0;
}

And that concludes the implementation of all the steps from the section Basics.

                                                                                  
┏━━━━━━━━━━━━┓   ┏━━━━━━━━━━━━┓   ┏━━━━━━━━━━━━┓   ┏━━━━━━━━━━━━┓   ┏━━━━━━━━━━━━┓
┃   SOCKET   ┃ ▶ ┃    BIND    ┃ ▶ ┃   LISTEN   ┃ ▶ ┃   ACCEPT   ┃ ▶ ┃ READ/WRITE ┃
┗━━━━━━━━━━━━┛   ┗━━━━━━━━━━━━┛   ┗━━━━━━━━━━━━┛   ┗━━━━━━━━━━━━┛   ┗━━━━━━━━━━━━┛
 ───────────▶     ───────────▶     ───────────▶     ───────────▶     ───────────▶

All what is left to do is to compile and run the program!

Let’s run it!

We’ll go to the command line and run the following command:

$ gcc -Wall webserver.c -o webserver
$ ./webserver

Now, you should be able to open your browser and check: http://localhost:8080, and you should be greeted by the ‘hello, world’ message. Let’s also try to implement some logging to the terminal, so we can see who is making the request and what the request was.

Client address

In order to get the client address information we can use the function getsockname(2). Let’s see what the man page says:

NAME
       getsockname - get socket name

SYNOPSIS
       #include 

       int getsockname(int sockfd, struct sockaddr *addr, socklen_t *addrlen);

DESCRIPTION
       getsockname() returns the current address to which the socket sockfd is
       bound, in the buffer pointed to by addr. The addrlen argument should be
       initialized to indicate the amount of space (in bytes) pointed to by
       addr. On return it contains the actual size of the socket address.

       The returned address is truncated if the buffer provided is too small;
       in this case, addrlen will return a value greater than was supplied to
       the call.

RETURN VALUE
       On success, zero is returned. On error, -1 is returned, and errno is
       set appropriately.

sockfd

So from the man page we read that we can get the current address to which the socket sockfd is bound. We got a new connected socket with the client from accept that is called newsockfd so we can use that in the getsockname(2) function.

addr

This argument should be a pointer to a struct sockaddr structure, and should looking familiar since we’ve used it before when we used bind(2). So we will use the same structure.

addrlen

Like the addr argument we’ve used in the bind(2) function, this argument will also be a pointer to a socklen_t struct. Again we will be doing the same as we did with bind(2).

Implementation

When we’ve implemented the getsockname(2) we will be able to relay the client’s ip address and port. These are available from the sin_addr and sin_port fields of the struct sockaddr_in structure.

We need to convert them to a string representation, that we can use to print. For that we will use the inet_ntoa(3)function (Internet host address, given in network byte order, to a string in IPv4 dotted decimal notation), and the ntohs(3) (network byte order to short integer byte order)

// ./steps/step007.c
#include 
#include 
#include 
#include 
#include 
#include 

#define PORT 8080
#define BUFFER_SIZE 1024

int main() {
    char buffer[BUFFER_SIZE];
    char resp[] = "HTTP/1.0 200 OK\r\n"
                  "Server: webserver-c\r\n"
                  "Content-type: text/html\r\n\r\n"
                  "hello, world\r\n";

    // Create a socket
    int sockfd = socket(AF_INET, SOCK_STREAM, 0);
    if (sockfd == -1) {
        perror("webserver (socket)");
        return 1;
    }
    printf("socket created successfully\n");

    // Create the address to bind the socket to
    struct sockaddr_in host_addr;
    int host_addrlen = sizeof(host_addr);

    host_addr.sin_family = AF_INET;
    host_addr.sin_port = htons(PORT);
    host_addr.sin_addr.s_addr = htonl(INADDR_ANY);

    // Create client address
    struct sockaddr_in client_addr;
    int client_addrlen = sizeof(client_addr);

    // Bind the socket to the address
    if (bind(sockfd, (struct sockaddr *)&host_addr, host_addrlen) != 0) {
        perror("webserver (bind)");
        return 1;
    }
    printf("socket successfully bound to address\n");

    // Listen for incoming connections
    if (listen(sockfd, SOMAXCONN) != 0) {
        perror("webserver (listen)");
        return 1;
    }
    printf("server listening for connections\n");

    for (;;) {
        // Accept incoming connections
        int newsockfd = accept(sockfd, (struct sockaddr *)&host_addr,
                               (socklen_t *)&host_addrlen);
        if (newsockfd < 0) {
            perror("webserver (accept)");
            continue;
        }
        printf("connection accepted\n");

        // Get client address
        int sockn = getsockname(newsockfd, (struct sockaddr *)&client_addr,
                                (socklen_t *)&client_addrlen);
        if (sockn < 0) {
            perror("webserver (getsockname)");
            continue;
        }

        // Read from the socket
        int valread = read(newsockfd, buffer, BUFFER_SIZE);
        if (valread < 0) {
            perror("webserver (read)");
            continue;
        }
        printf("[%s:%u]\n", inet_ntoa(client_addr.sin_addr),
               ntohs(client_addr.sin_port));

        // Write to the socket
        int valwrite = write(newsockfd, resp, strlen(resp));
        if (valwrite < 0) {
            perror("webserver (write)");
            continue;
        }

        close(newsockfd);
    }

    return 0;
}

Get request headers

We’ve read the request from the client from the socket with read(2), now we can print the contents of the request message from the buffer, and we should see how the request of the client looks like. Since the client is sending the request adhering the HTTP protocol, we can use sscanf(3) to parse the request. Since the request line is the first line of the request, and is structured as follows: , we can use sscanf(3) to parse the request line, and get the method, path and version.

Let’s check the man page for sscanf(3):

NAME
       scanf, fscanf, sscanf, vscanf, vsscanf, vfscanf - input format conversion

SYNOPSIS
       #include 

       int sscanf(const char *str, const char *format, ...);

DESCRIPTION

        ...
        The scanf() function reads input from the standard input stream
        stdin, fscanf() reads input from the stream pointer stream, and
        sscanf() reads its input from the character string pointed to by
        str.
        ...

str

This argument is a pointer to a character string that contains the input data. We will thus use the buffer that we’ve read from the socket.

format

The format argument is a string that specifies the format of the input data. And we can use string formatting to parse the request line.

return value

On success, the number of input items successfully converted and assigned is returned. On failure, the return value is EOF, and no characters are assigned to the arguments.

Implementation

// webserver.c
#include 
#include 
#include 
#include 
#include 
#include 

#define PORT 8080
#define BUFFER_SIZE 1024

int main() {
    char buffer[BUFFER_SIZE];
    char resp[] = "HTTP/1.0 200 OK\r\n"
                  "Server: webserver-c\r\n"
                  "Content-type: text/html\r\n\r\n"
                  "hello, world\r\n";

    // Create a socket
    int sockfd = socket(AF_INET, SOCK_STREAM, 0);
    if (sockfd == -1) {
        perror("webserver (socket)");
        return 1;
    }
    printf("socket created successfully\n");

    // Create the address to bind the socket to
    struct sockaddr_in host_addr;
    int host_addrlen = sizeof(host_addr);

    host_addr.sin_family = AF_INET;
    host_addr.sin_port = htons(PORT);
    host_addr.sin_addr.s_addr = htonl(INADDR_ANY);

    // Create client address
    struct sockaddr_in client_addr;
    int client_addrlen = sizeof(client_addr);

    // Bind the socket to the address
    if (bind(sockfd, (struct sockaddr *)&host_addr, host_addrlen) != 0) {
        perror("webserver (bind)");
        return 1;
    }
    printf("socket successfully bound to address\n");

    // Listen for incoming connections
    if (listen(sockfd, SOMAXCONN) != 0) {
        perror("webserver (listen)");
        return 1;
    }
    printf("server listening for connections\n");

    for (;;) {
        // Accept incoming connections
        int newsockfd = accept(sockfd, (struct sockaddr *)&host_addr,
                               (socklen_t *)&host_addrlen);
        if (newsockfd < 0) {
            perror("webserver (accept)");
            continue;
        }
        printf("connection accepted\n");

        // Get client address
        int sockn = getsockname(newsockfd, (struct sockaddr *)&client_addr,
                                (socklen_t *)&client_addrlen);
        if (sockn < 0) {
            perror("webserver (getsockname)");
            continue;
        }

        // Read from the socket
        int valread = read(newsockfd, buffer, BUFFER_SIZE);
        if (valread < 0) {
            perror("webserver (read)");
            continue;
        }

        // Read the request
        char method[BUFFER_SIZE], uri[BUFFER_SIZE], version[BUFFER_SIZE];
        sscanf(buffer, "%s %s %s", method, uri, version);
        printf("[%s:%u] %s %s %s\n", inet_ntoa(client_addr.sin_addr),
               ntohs(client_addr.sin_port), method, version, uri);

        // Write to the socket
        int valwrite = write(newsockfd, resp, strlen(resp));
        if (valwrite < 0) {
            perror("webserver (write)");
            continue;
        }

        close(newsockfd);
    }

    return 0;
}

Conclusion / Next Steps

And that concludes our implementation of a simple HTTP server. We’ve covered the basics of implementing a simple HTTP web server, doing so we’ve also learned a bit about TCP/IP, HTTP, socket programming, and system calls. The resulting code gives you a basis to start implementing a more complex HTTP webserver that can handle multiple concurrent connections, and that can serve html files from the filesystem based on the path of the request.

References

Arpaci-Dusseau, Remzi H., and Andrea C. Arpaci-Dusseau. 2018. Operating Systems: Three Easy Pieces. Arpaci-Dusseau Books. http://pages.cs.wisc.edu/~remzi/OSTEP/.

Github. 2020. “Remzi-Arpacidusseau - Ostep Concurrency Webserver - Github.” 2020. https://github.com/remzi-arpacidusseau/ostep-projects/tree/master/concurrency-webserver.

Kerrisk, N. 2010. The Linux Programming Interface: a Linux and UNIX system programming handbook. No Starch Press.

Mozilla. 2020. “What Is a Webserver - Mozilla.” 2020. https://developer.mozilla.org/en-US/docs/Learn/Common_questions/What_is_a_web_server.

“The GNU C Library Reference Manual.” n.d. https://www.gnu.org/software/libc/manual/.

Wikipedia. 2020. “Webserver - Wikipedia, The Free Encyclopedia.” 2020. https://en.wikipedia.org/wiki/Web_server.

Arpaci-Dusseau and Arpaci-Dusseau (2018)↩︎
https://github.com/remzi-arpacidusseau/ostep-projects/tree/master/concurrency-webserver ↩︎
Github (2020); Mozilla (2020); Wikipedia (2020)↩︎
Mozilla (2020)↩︎
https://tools.ietf.org/html/rfc2616 ↩︎
https://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol ↩︎
https://en.wikipedia.org/wiki/Application_layer ↩︎
https://en.wikipedia.org/wiki/Internet_protocol_suite ↩︎
https://github.com/remzi-arpacidusseau/ostep-projects/tree/master/concurrency-webserver#http-background ↩︎
https://en.wikipedia.org/wiki/C_POSIX_library ↩︎
https://en.wikipedia.org/wiki/File_descriptor ↩︎
https://tools.ietf.org/html/draft-newman-network-byte-order-01 ↩︎
The terms derive from Jonathan Swift’s 1726 satirical novel Gulliver’s Travels, in which the terms refer to opposing political factions who open their boiled eggs at opposite ends. [Kerrisk (2010); ch. 59.2]↩︎
http://www.cplusplus.com/forum/general/14828/↩︎
Kerrisk (2010); ch. 56.5↩︎
Kerrisk (2010); ch. 56.5.2↩︎
Kerrisk (2010); ch. 4, ch. 56.1↩︎
Kerrisk (2010); ch. 4.7↩︎
“The GNU C Library Reference Manual” (n.d.); ch. 11.1.2↩︎

Securing Go HTTP webservers

Sun, 24 May 2020 08:00:00 +0000

Introduction

For a project I am working on, I wanted to be able to expose the http server implementation in Go to the internet. After listening to the Go Time podcast and specifically the episode 101¹, I wanted to go over the common measures you can employ to make your Go HTTP webserver more secure. But before we begin:

DISCLAIMER: Web security is sensitive and vast subject, and I want to make sure that you do own research on how to set up a secure HTTP webserver. So don’t rely on this post solely, and always do some rigorous testing. To make it a bit easier to find resources I’ve added references and footnotes to the post. So I implore you to read sources to get a better understanding of the subject. Additionally, some measures taken here will not always be supported by all clients, such as browsers. This is because in this post I tried to use the latest versions and recommendations, and these are not always supported.

Since we’ve got that out of the way, let’s begin.

Simple webserver

So, let’s start with creating a simple HTTP webserver, you’ve probably come across this one several times.²

// server.go
package main

import (
    "log"
    "net/http"
)

func main() {
    http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
        w.Write([]byte("hello, world\n"))
    })

    log.Fatal(http.ListenAndServe(":http", nil))
}

Let’s run it and check if it works:

$ go run server.go
$ curl http://localhost
hello world

Cool, we have a starting point. Let’s add our first security measures.

Timeouts / Limits

To make our HTTP webserver more secure we have the option to set some timeouts and limits for it. This will make sure that connections aren’t stuck, not making progress, or are stalling. To help us with setting some defaults We will be checking out the article written by Filippo Valsorda: “So you want to expose Go on the Internet” (Valsorda 2016) to guide us.

When we implement those defaults, the end result will resemble something like this:

// server.go
package main

import (
    "log"
    "net/http"
    "time"
)

func main() {
    srv := &http.Server{
        Addr: ":http",
        Handler: http.HandlerFunc(
            func(w http.ResponseWriter, r *http.Request) {
                r.Body = http.MaxBytesReader(w, r.Body, 1<<20)
                w.Write([]byte("hello, world\n"))
            },
        ),
        ReadTimeout:       5 * time.Second,
        WriteTimeout:      10 * time.Second,
        IdleTimeout:       120 * time.Second,
        ReadHeaderTimeout: 5 * time.Second,
        MaxHeaderBytes:    1 << 20,
    }

    log.Fatal(srv.ListenAndServe())
}

So, let’s go over it and investigate what fields we need to set.

ReadTimeout:       5 * time.Second,
WriteTimeout:      10 * time.Second,
IdleTimeout:       120 * time.Second,
ReadHeaderTimeout: 5 * time.Second,

The timeout ReadTimeout covers, from the server perspective, when the connection is established to when the request from the client is read. The timeout WriteTimeout covers the duration of the server writing its response. The timeout IdleTimeout covers the duration to wait for a new connection when ‘keep-alive’ is enabled. And the ReadHeaderTimeout covers the time that the request header (from the client) is completely read.³

Accept                                                          New request
                                         ┌──────────ServeHTTP─────────────┐
┏━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━┓
┃ Wait ┃ TLS Handshake ┃ Request Headers ┃ Request Body ┃ Response ┃ Idle ┃
┗━━━━━━┻━━━━━━━━━━━━━━━┻━━━━━━━━━━━━━━━━━┻━━━━━━━━━━━━━━┻━━━━━━━━━━┻━━━━━━┛
│                      └ReadHeaderTimeout┘              │          └──┬───┘
│                                        │              │       IdleTimeout
└──────────────ReadTimeout──────────────────────────────┘          │       
│                                        │                         │       
└┄┄┄┄┄┄┄┄┄┄┄┄┄┄WriteTimeout┄┄┄┄┄┄┄┄┄┄┄┄┄┄┴───────WriteTimeout──────┘

Above an overview of the timeouts in the request/response cycle, note that when TLS is used, WriteTimeout will include the TLS handshake, as well as the header read as well.

We continue with setting the field MaxHeaderBytes:

MaxHeaderBytes: 1 << 20,  // Resolves to 1048576, and equals 1MB

This field makes sure that there is a maximum of bytes that the server will read when parsing the headers. Note that it does not limit the size of incoming request body. You’ll need to implement http.MaxBytesReader() in your handlers.⁴ And we can implement it as follows:

Handler: http.HandlerFunc(
    func(w http.ResponseWriter, r *http.Request) {
        r.Body = http.MaxBytesReader(w, r.Body, 1<<20)
        io.WriteString(w, "hello, world\n")
    },
),

Remember, that this is just an example, and you should think about the specific timeouts, and limits that serve your use-case.

TLS

To make our HTTP webserver more secure, and expose it to the internet we do well to implement a secure communication between the client and the server. And we do that by leveraging TLS, which stands for: Transport Layer Security. We use it to secure communication between the webserver and the client over the internet. The protocol is implemented at the application layer, directly on top of TCP, which enables protocols ‘above’⁵ it to continue working while leveraging the added communication security. (Grigorik 2013, 48)

First, let’s explore what TLS exactly does, so we get a sense of why we choose certain methods when coding an implementation in Go.

Overview

There are 3 main components to TLS:

Encryption: hides the data being transferred from third parties.
Authentication: ensures that the parties exchanging information are who they claim to be.
Integrity: verifies that the data has not been forged or tampered with.

We hide the data by encrypting it, and we do that by using symmetric cryptography.⁶ This means that a generated key is used for both to encrypt and decrypt the data. The keys used for this are created for each connection, and are based on a shared secret that was negotiated at the start of the session. (Wikipedia 2020f, 2020e) The chosen ‘ciphersuite’ determines details such as, which shared encryption keys, or session keys, will be used for the session. (Cloudflare 2020)

We make sure that the identity of both the server and client is correct by using public-key cryptography. In contrast to symmetric cryptography, this employs asymmetric cryptography where a pair of keys are used: the public and private key. Everyone with the public key is able to encrypt data, but only the holder of the private key is able to decrypt it. In our case that will be the server. (Wikipedia 2020f, 2020d)

This public-key cryptography is used only during session setup of the TLS tunnel. The server provides its public key to the client, and then the client generates a symmetric key, which it encrypts with the server’s public key, and returns the encrypted symmetric key, with which the data between the client and server is encrypted, to the server. Finally, the server can decrypt the symmetric key sent from the client with its own private key. This symmetric key is then used for all further encrypted communication since now both the server and client have the key to encrypt and decrypt data. (Grigorik 2013, 52)

When the data is encrypted and authenticated, it will be signed with a message authentication code (MAC) to ensure its integrity. (Cloudflare 2020) The MAC algorithm is a one-way cryptographic hash function, in a way a sort of checksum. The keys in order to generate the MAC are negotiated between the client and server. When a TLS record is sent, a MAC value is generated and appended to that message. The receiver is then able to compute and verify the MAC value to ensure that the message hasn’t been changed or altered. (Grigorik 2013, 49)

OK, so how is this all implemented? A TLS connection between the server and client will be established by the so-called “TLS Handshake”. This “handshake” creates the agreement on the version of the TLS protocol, the ciphersuite, and certificate verification. In the following diagram we’ll go over the steps of how such a “handshake” is realised.⁷

                                                     
        ┌─┐                             ┌─┐          
        └┬┘                             ╞ │          
        ▔▔▔                             └─┘          
       client                          server        
┏━━━━━━━━━━━━━━━━━━━┓                                
┃ SYN               ┃ ――――――――▶ ┏━━━━━━━━━━━━━━━━━━━┓
┗━━━━━━━━━━━━━━━━━━━┛           ┃ SYN ACK           ┃
┏━━━━━━━━━━━━━━━━━━━┓ ◀―――――――― ┗━━━━━━━━━━━━━━━━━━━┛
┃ ACK               ┃ ――――――――▶ ┏━━━━━━━━━━━━━━━━━━━┓
┣━━━━━━━━━━━━━━━━━━━┫           ┃ ServerHello       ┃
┃ ClientHello       ┃           ┣━━━━━━━━━━━━━━━━━━━┫
┗━━━━━━━━━━━━━━━━━━━┛           ┃ Certificate       ┃
┏━━━━━━━━━━━━━━━━━━━┓ ◀―――――――― ┣━━━━━━━━━━━━━━━━━━━┫
┃ ClientKeyExchange ┃           ┃ ServerHelloDone   ┃
┣━━━━━━━━━━━━━━━━━━━┫           ┗━━━━━━━━━━━━━━━━━━━┛
┃ ChangeCipherSpec  ┃ ――――――――▶ ┏━━━━━━━━━━━━━━━━━━━┓
┣━━━━━━━━━━━━━━━━━━━┫           ┃ ChangeCipherSpec  ┃
┃ Finished          ┃           ┣━━━━━━━━━━━━━━━━━━━┫
┗━━━━━━━━━━━━━━━━━━━┛           ┃ Finished          ┃
┏━━━━━━━━━━━━━━━━━━━┓ ◀―――――――― ┗━━━━━━━━━━━━━━━━━━━┛
┃ Application Data  ┃           ┏━━━━━━━━━━━━━━━━━━━┓
┗━━━━━━━━━━━━━━━━━━━┛ ――――――――▶ ┃ Application Data  ┃
                                ┗━━━━━━━━━━━━━━━━━━━┛

SYN, SYN ACK, ACK - Start with a TCP connection, and a three-way handshake. After this TCP connection, the TLS Handshake begins.

ClientHello - Client sends a ClientHello message and it will include a number of specifications in plain text, such as version of TLS protocol, list of supported ciphersuites (ciphers and hash functions).

ServerHello, Certificate, ServerHelloDone - Server selects the version, and the ciphersuite. It then presents its certificate. This contains: the domain name, the trusted certificate authority (CA), and the public encryption key, and sends its response back to the client.

ClientKeyExchange, ChangeCipherSpec, Finished - Client confirms the validity of the certificate. Then the client generates a new symmetric key, and encrypts that key with the server’s public key.

ChangeCipherSpec, Finished - The server decrypts the symmetric key sent by the client, checks the integrity of the message by verifying the MAC, and returns an encrypted “Finished” message back to the client.

Application Data - The client decrypts the message with the symmetric key it generated earlier, verifies the MAC, and if everything checks out application data can be sent.

So, this gives us a bit of a basic understanding of how TLS works. In the next section we’ll implement it into our HTTP webserver.

Implementation

We’ll be using the crypto/tls package for implementing TLS for our HTTP webserver.⁸ We also take note of the Mozilla SSL Configuration Generator, which allows us to implement the “Modern” recommended configuration made by Mozilla. (Mozilla 2020a) When implemented, our code will look something like this:

// server.go
package main

import (
    "crypto/tls"
    "flag"
    "log"
    "net/http"
    "time"
)

func main() {
    flgCert := flag.String("cert", "cert.pem", "path to cert")
    flgKey := flag.String("key", "key.pem", "path to key")
    flag.Parse()

    cert, err := tls.LoadX509KeyPair(*flgCert, *flgKey)
    if err != nil {
        log.Fatal(err)
    }

    tlsConfig := &tls.Config{
        Certificates:             []tls.Certificate{cert},
        CipherSuites:             nil,
        PreferServerCipherSuites: true,
        MinVersion:               tls.VersionTLS13,
        CurvePreferences: []tls.CurveID{
            tls.CurveP256,
            tls.X25519,
        },
    }

    go func() {
        srv := &http.Server{
            Addr: ":http",
            Handler: http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
                http.Redirect(
                    w, r,
                    "https://"+r.Host+r.URL.RequestURI(),
                    http.StatusMovedPermanently,
                )
            }),
        }
        log.Fatal(srv.ListenAndServe())
    }()

    srv := &http.Server{
        Addr: ":https",
        Handler: http.HandlerFunc(
            func(w http.ResponseWriter, r *http.Request) {
                r.Body = http.MaxBytesReader(w, r.Body, 1<<20)
                w.Write([]byte("hello, world\n"))
            },
        ),
        TLSConfig: tlsConfig,
        TLSNextProto: make(
            map[string]func(*http.Server, *tls.Conn, http.Handler), 0,
        ),
        ReadTimeout:    5 * time.Second,
        WriteTimeout:   10 * time.Second,
        IdleTimeout:    120 * time.Second,
        MaxHeaderBytes: 1 << 20,
    }

    log.Fatal(srv.ListenAndServeTLS("", ""))
}

Again, let’s go over the code and what investigate what we’ve changed.

flgCert := flag.String("cert", "cert.pem", "path to cert")
flgKey := flag.String("key", "key.pem", "path to key")
flag.Parse()

Here we’re parsing some command line arguments that we will use later on. Eventually, we’ll create self-signed certificates, whose paths we pass into our program here.

cert, err := tls.LoadX509KeyPair(*flgCert, *flgKey)
if err != nil {
    log.Fatal(err)
}

The LoadX509KeyPair() function reads and parses a public/private key pair from a pair of files. Both files must be ‘PEM’ (Privacy-Enhanced Mail) encoded.⁹

tlsConfig := tls.Config{
    // ...
}

Here we create the tls.Config struct that we use to configure our TLS functionality for our HTTP webserver.¹⁰

Certificates: []tls.Certificate{cert},

To the Certificates field we add the certificates we’ve loaded with the parsed public/private key pair that we provided.

CipherSuites: nil,

The field CipherSuites is a list of ciphersuites that the server will support. This is up to TLS version 1.2, when using version 1.3 this isn’t configurable. Because we want to use the most up to date version we keep it empty, which results in a default list of ciphersuites to be used with a preference order based on hardware performance.

PreferServerCipherSuites: true,

This boolean controls the server’s preferred ciphersuite to use, as provided by the CipherSuites, when false it will select the client’s preferred ciphersuite. Setting this will ensure that safer and faster ciphersuites are used. (Valsorda 2016)

MinVersion: tls.VersionTLS13,

Here we can specify the minimal TLS version the server is going to use. We’ll select version recommended by the “Modern” configuration from Mozilla. (Mozilla 2020a)

CurvePreferences: []tls.CurveID{
    tls.CurveP256,
    tls.X25519,
},

The array CurvePreferences contain the elliptic curves that will be used in an ECDHE handshake. We again use the Mozilla recommendation, however without tls.CurveP384, because a client using tls.CurveP384 would cause up to a second of CPU to be consumed on the server. (Valsorda 2016)

go func() {
    srv := &http.Server{
        Addr: ":http",
        Handler: http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
            http.Redirect(
                w, r,
                "https://"+r.Host+r.URL.RequestURI(),
                http.StatusMovedPermanently,
            )
        }),
    }
    log.Fatal(srv.ListenAndServe())
}()

Here, we create, and run an additional http.Server in a goroutine that will redirect all ‘http’ traffic to our ‘https’ port.

Addr: ":https",

For our ‘main’ http.Server we update the port number and set it to :https to resemble that we’re communicating on port 443.

TLSConfig: tlsConfig,

Here we load our TLS configuration we’ve defined before.

TLSNextProto: make(
    map[string]func(*http.Server, *tls.Conn, http.Handler), 0,
),

Setting TLSNextProto to an empty map will disable HTTP/2 for this server. If you want to enable HTTP/2 set it to nil or remove the field. Since Go 1.6 it is enabled by default. (Valsorda 2016)

log.Fatal(srv.ListenAndServeTLS("", ""))

Because we’re now using TLS, we start the server with ListenAndServeTLS(). It will then listen on the serv.Addr for incoming TLS connections. Because we already set the paths of the key and cert in the tlsConfig, we can set certFile and keyFile arguments to empty strings.

We’ve added TLS functionality to our HTTP webserver, but we’re still missing one part of the TLS components: the certificate.

Certificates

We’re almost ready to test out the program, but we need have the cert and key file. As was mentioned before with the 3 main components TLS consists of we need to ensure the authentication between the server and the client. Meaning that the parties exchanging information, are who they claim to be. This is employed by using public-key cryptography, with the creation of a public and private key. Where everyone with the public key is able to encrypt data, but only the holder of the private key is able to decrypt it.

So what methods are there to trust someone?

Manually specified certificates, in which you manually import any certificate you trust.
A Certificate Authority (CA), where a third party generates the certificate that both parties trust.
The operating system also contains a list with well-known certificate authorities that it trusts.

The most common solution is to make use of a ‘Certificate Authority’. However, first we’re going to create a self-signed certificate, because using a CA to generate a certificate will come with some costs, because you’ll be using a service to verify, audit, and revoke when a certificate is misused or compromised. And it’ll give us a bit more information on how such a thing is generated. (Grigorik 2013, 59)

By checking out the “Modern” configuration again from Mozilla, we can see that it states that the certificate type should be ‘ECDSA (P-256)’¹¹, which stand for elliptical curve digital signature algorithm, and that the maximum certificate lifespan should be 90 days. (Mozilla 2020a)

We’ll be using the openssl req command for this from the commandline, which is a certificate request and certificate generating utility.

# -X509
#   Output a self-signed certificate instead of a certificate request.
#
# -nodes
#   No DES, private key not protected by a passphrase, thus it won't be
#   encrypted.
#
# -newkey ec
#   Create a new certificate request and a new EC private key, usable for
#   both ECDSA or ECDH algorithms
#
# -pkeyopt ec_paramgen_curve:prime256v1
#   Set the public key algorithm ec_paramgen_curve to prime256v1
#
# -keyout key.pem
#   Filename to write the private key to.
#
# -out cert.pem
#   Specifies output filename to write to.
#
# -days 90
#   When using -X509 this specifies the number of days to certify for.

$ openssl req -x509 -nodes -newkey ec -pkeyopt ec_paramgen_curve:prime256v1 \
    -keyout key.pem -out cert.pem -days 90

This will output both the key and cert files in the PEM format, exactly what we needed for tls.Load509KeyPair() function. The ‘X509’ is the format for public-key certificates, an X.509 certificate contains a public key and an identity (a hostname, or an organization, or an individual) (Wikipedia 2020g)

We can then inspect the content of the certificate by again using openssl. By using the openssl x509 command we can look at the contents of a ‘X509’ certificate.

# -text
#   Prints out the certificate in text form.
#
# -in
#   Specifies the input filename to a certificate from. 
#
# -noout
#   Prevents output of the encoded version of the actual certificate.

$ openssl x509 -text -in cert.pem -noout

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            6b:41:90:1d:ad:c2:bc:0b:9b:61:f5:14:f8:06:92:70:49:d7:99:81
        Signature Algorithm: ecdsa-with-SHA256
        Issuer: C = NL, ST = Some-State, L = Amsterdam, O = Internet Widgets Pty Ltd
        Validity
            Not Before: Apr 15 18:24:08 2020 GMT
            Not After : Jul 14 18:24:08 2020 GMT
        Subject: C = NL, ST = Some-State, L = Amsterdam, O = Internet Widgets Pty Ltd
        Subject Public Key Info:
            Public Key Algorithm: id-ecPublicKey
                Public-Key: (256 bit)
                pub:
                    04:a6:fa:3c:86:99:a9:30:04:65:7f:00:7a:18:e2:
                    bc:4f:b2:69:45:75:b5:a0:06:37:2c:08:d9:2e:d5:
                    bc:a5:e5:14:fb:0a:20:28:b6:c8:e7:30:78:4d:be:
                    7f:5f:12:2c:6e:0a:76:4d:e9:2b:38:c8:9c:cf:d2:
                    db:2b:22:9d:1f
                ASN1 OID: prime256v1
                NIST CURVE: P-256
        X509v3 extensions:
            X509v3 Subject Key Identifier:
                75:BB:CA:93:7A:CF:FB:CF:E5:0A:FB:1C:32:9E:C8:A9:61:D0:76:3F
            X509v3 Authority Key Identifier:
                keyid:75:BB:CA:93:7A:CF:FB:CF:E5:0A:FB:1C:32:9E:C8:A9:61:D0:76:3F

            X509v3 Basic Constraints: critical
                CA:TRUE
    Signature Algorithm: ecdsa-with-SHA256
         30:45:02:21:00:b4:5b:81:65:bd:6e:48:a8:f4:fb:d5:47:1a:
         a0:e2:44:ec:a0:90:7f:b0:9c:de:ba:64:33:d6:98:c3:97:60:
         ba:02:20:26:41:74:a9:bc:c9:78:8b:0e:e3:12:c6:fd:ba:10:
         93:87:cb:8f:4b:15:99:e3:6e:89:d2:f6:b2:e0:48:bb:fc

Here we can see some important information such as the issuer, its validity, and the public key. With this clients knows how to recognize it when this server has signed a message with the corresponding private key.

Testing it out

Now, that we’ve got the certificate ready, let’s test out our program.

$ go run server.go -cert=cert.pem -key=key.pem

When our server is running, we’ll be able to test it with curl. Note that we’re going to add the --insecure flag to our command. This is because we’ve self-signed the certificate and curl rightfully doesn’t trust that. We’re also adding the --verbose flag to output more information about the request.¹²

$ curl --verbose --insecure https://localhost

*   Trying ::1:443...
* Connected to localhost (::1) port 443 (#0)
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: C=NL; ST=Some-State; O=Internet Widgets Pty Ltd
*  start date: Apr  3 09:20:12 2020 GMT
*  expire date: Jul  2 09:20:12 2020 GMT
*  issuer: C=NL; ST=Some-State; O=Internet Widgets Pty Ltd
*  SSL certificate verify result: self signed certificate (18), continuing anyway.
> GET / HTTP/1.1
> Host: localhost:443
> User-Agent: curl/7.52.1
> Accept: */*
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Strict-Transport-Security: max-age=63072000
< Date: Fri, 03 Apr 2020 09:20:15 GMT
< Content-Length: 14
< Content-Type: text/plain; charset=utf-8
<
hello, world
* Connection #0 to host localhost left intact

OK, that’s pretty cool. We’re able to see exactly what the request/response cycle is like, including the ‘TLS Handshake’ that we’ve explored in the ‘TLS’ section. You can however see the following line: SSL certificate verify result: self signed certificate (18), continuing anyway. So is there however a possibility to make curl trust our certificate without skipping and/or ignoring the certificate verification? We can with mkcert, and it allows us to create locally-trusted development certificates. It automatically creates and installs a local CA in the system root store, and generates locally-trusted certificates. So let’s create one for localhost.

$ mkcert -install
$ mkcert -ecdsa -key-file=key.pem -cert-file=cert.pem localhost

Now, run restart the server, and make another request with curl without the --insecure flag.

$ curl --verbose https://localhost

*   Trying ::1:443...
* Connected to localhost (127.0.0.1) port 443 (#0)
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: O=mkcert development certificate; OU=jp@aztlan
*  start date: Jun  1 00:00:00 2019 GMT
*  expire date: Apr 15 18:58:07 2030 GMT
*  subjectAltName: host "localhost" matched cert's "localhost"
*  issuer: O=mkcert development CA; OU=jp@aztlan; CN=mkcert jp@aztlan
*  SSL certificate verify ok.
> GET / HTTP/1.1
> Host: localhost:443
> User-Agent: curl/7.52.1
> Accept: */*
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Date: Wed, 15 Apr 2020 19:07:29 GMT
< Content-Length: 13
< Content-Type: text/plain; charset=utf-8
<
hello, world
* Connection #0 to host localhost left intact

However, we aren’t able to actually use this in production, since we still used a self-signed certificate. Considering getting a certificate from a CA has some costs associated with it, we’ll create our certificate with ‘Let’s Encrypt’.

Let’s Encrypt

‘Let’s Encrypt’ allows us to create free TLS certificates from nonprofit Certificate Authority. And we’ll be able to implement it in our program with the golang.org/x/crypto/acme/autocert package.¹³ This package automatically generates, and when necessary, renews your certificate. Let’s update our code and implement the autocert functionality.

// server.go
package main

import (
    "crypto/tls"
    "flag"
    "log"
    "net/http"
    "time"

    "golang.org/x/crypto/acme/autocert"
)

func main() {
    flgProd := flag.Bool("prod", false, "run server in production")
    flgHost := flag.String("host", "", "host name of the server")
    flgCert := flag.String("cert", "cert.pem", "path to cert")
    flgKey := flag.String("key", "key.pem", "path to key")
    flag.Parse()

    tlsConfig := &tls.Config{
        CipherSuites:             nil,
        PreferServerCipherSuites: true,
        MinVersion:               tls.VersionTLS13,
        CurvePreferences: []tls.CurveID{
            tls.CurveP256,
            tls.X25519,
        },
    }

    if *flgProd {
        if *flgHost == "" {
            log.Fatal("please specify a host")
        }

        m := autocert.Manager{
            Prompt:     autocert.AcceptTOS,
            HostPolicy: autocert.HostWhitelist(*flgHost),
            Cache:      autocert.DirCache("certs"),
        }

        tlsConfig.GetCertificate = m.GetCertificate

        go func() {
            h := m.HTTPHandler(nil)
            log.Fatal(http.ListenAndServe(":http", h))
        }()
    } else {
        cert, err := tls.LoadX509KeyPair(*flgCert, *flgKey)
        if err != nil {
            log.Fatal(err)
        }

        tlsConfig.Certificates = []tls.Certificate{cert}

        go func() {
            srv := &http.Server{
                Addr: ":http",
                Handler: http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
                    http.Redirect(
                        w, r, "https://"+r.Host+r.URL.RequestURI(),
                        http.StatusMovedPermanently,
                    )
                }),
            }
            log.Fatal(srv.ListenAndServe())
        }()
    }

    srv := &http.Server{
        Addr: ":https",
        Handler: http.HandlerFunc(
            func(w http.ResponseWriter, r *http.Request) {
                r.Body = http.MaxBytesReader(w, r.Body, 1<<20)
                w.Write([]byte("hello, world\n"))
            },
        ),
        TLSConfig: tlsConfig,
        TLSNextProto: make(
            map[string]func(*http.Server, *tls.Conn, http.Handler), 0,
        ),
        ReadTimeout:    5 * time.Second,
        WriteTimeout:   10 * time.Second,
        IdleTimeout:    120 * time.Second,
        MaxHeaderBytes: 1 << 20,
    }

    log.Fatal(srv.ListenAndServeTLS("", ""))
}

Again, let’s go over the code and explore what we’ve changed.

flgProd := flag.Bool("prod", false, "run server in production")
flgHost := flag.String("host", "", "host name of the server")

We’ve added two new flags, flgProd and flgHost. The flag flgProd allows us to signal to the program, by setting the command line flag, that the program is running in production, and should be using the ‘Let’s Encrypt’ certificate created by autocert. The flag flgHost is necessary for autocert to let it know for which host we’re generating a certificate.

if *flgProd {
    if *flgHost == "" {
        log.Fatal("please specify a host")
    }

    m := autocert.Manager{
        Prompt:     autocert.AcceptTOS,
        HostPolicy: autocert.HostWhitelist(*flgHost),
        Cache:      autocert.DirCache("certs"),
    }

    tlsConfig.GetCertificate = m.GetCertificate

    go func() {
        h := m.HTTPHandler(nil)
        log.Fatal(http.ListenAndServe(":http", h))
    }()
} else {
    cert, err := tls.LoadX509KeyPair(*flgCert, *flgKey)
    if err != nil {
        log.Fatal(err)
    }

    tlsConfig.Certificates = []tls.Certificate{cert}

    go func() {
        srv := &http.Server{
            Addr: ":http",
            Handler: http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
                http.Redirect(
                    w, r, "https://"+r.Host+r.URL.RequestURI(),
                    http.StatusMovedPermanently,
                )
            }),
        }
        log.Fatal(srv.ListenAndServe())
    }()
}

We’ve removed the Certificates field from tlsConfig. And with this conditional we choose either by generating a certificate with autocert, or using our self-signed certificate.

m := autocert.Manager{
    Prompt:     autocert.AcceptTOS,
    HostPolicy: autocert.HostWhitelist(*flgHost),
    Cache:      autocert.DirCache("certs"),
}

The field Prompt specifies a function to accept a CA’s Terms of Service, here we choose autocert.AcceptTOS which always accept the terms. The field HostPolicy controls which domains the Manager will attempt to retrieve new certificates for. It accepts a HostPolicy, which specifies which host names the Manager is allowed to respond to. We create that by calling HostWhiteList() with the host name for our server. The field Cache is needed to write our key and certificate to, so that we can keep reusing it with server restarts for the days that the certificate is valid. We also do this because there is a rate-limit for requesting certificates from Let’s Encrypt.¹⁴ We set the directory to "certs" in this example.

tlsConfig.GetCertificate = m.GetCertificate

We specify the GetCertificate field of tls.Config to use the GetCertificate function of the autocert.Manager, because this implements the tls.Config.GetCertificate hook.

go func() {
    h := m.HTTPHandler(nil)
    log.Fatal(http.ListenAndServe(":http", h))
}()

Additionally, we will use the Manager.HTTPHandler from autocert that implements a handler fallback for requests received on port 80, and redirects it to the ‘https’ port.¹⁵

tlsConfig.Certificates = []tls.Certificate{cert}

Because we removed the Certificates field from tlsConfig, and we’re using the autocert.Manager.GetCertificate function when using Let’s Encrypt to get our certificate when we are using Let’s Encrypt. We need to specify our tlsConfig.Certificates field, when we aren’t using ‘Let’s Encrypt’ and using our self-signed certificate.

You can run it locally, but the certificate won’t be valid. Since, we’re running it locally and using localhost, ‘Let’s Encrypt’ isn’t able to create certificates for this because nobody uniquely owns localhost. So local development deployments we continue using our self-signed certificates. (See Let’s Encrypt 2020 for some alternate solutions)

OK, now if you have a production environment and domain name that points to your production environment, you’ll be able to run the server as follows:

$ go run server.go -prod=true -host=yourawesomedomain.org

Secure headers

In order to make our server more secure it is advised that we add some secure headers to our server.¹⁶ This is done in order to mitigate several vulnerabilities. (OWASP 2020b). To our Handler we will add these response headers:

Handler: http.HandlerFunc(
    func(w http.ResponseWriter, r *http.Request) {
        w.Header().Add("Strict-Transport-Security", "max-age=63072000; includeSubdomains")
        w.Header().Add("Content-Security-Policy", "default-src 'self'")
        w.Header().Add("X-XSS-Protection", "1; mode=block")
        w.Header().Add("X-Frame-Options", "DENY")
        w.Header().Add("Referrer-Policy", "strict-origin-when-cross-origin")
        w.Header().Add("X-Content-Type-Options", "nosniff")
        w.Header().Add("Content-Type", "text/plain")

        r.Body = http.MaxBytesReader(w, r.Body, 1<<20)
        w.Write([]byte("hello, world!\n"))
    },
),

Let’s go over the code again, and check what headers we’ve added, what they protect us from, and how we set it.

w.Header().Add("Strict-Transport-Security", "max-age=63072000; includeSubdomains")

Here we set the ‘HSTS’ header (HTTP Strict Transport Security). This will tell that clients should automatically interact with the server using HTTPS connections. By setting the response header we specify the period of time the client should only access the server in a secure fashion. It protects against ‘protocol downgrade attacks’, and ‘cookie hijacking’. (Wikipedia 2020c; OWASP 2020b) The value max-age specifies the time the client should remember that the site should be accessed by HTTPS. The value includeSubdomains implies that this rule applies to all subdomains as well.¹⁷

w.Header().Add("Content-Security-Policy", "default-src 'self'")

The ‘CSP’ header (Content Security Policy), allows you to define a policy of what resources the client is allowed to load/run. It specifies the domains the client should consider valid sources. Let’s say you have a web application that allows content to be loaded within the web application that originates from somewhere else. With CSP you’re able to white list origins for scripts, images, fonts, stylesheets, etc. It protects against certain ‘cross-site scripting attacks’ (XSS), which exploit the clients’ trust of the content received from the server. Malicious scripts can be thus be executed by the client because it trusts the source of the content, even when it not from where it seems to be from. The policy "default-src 'self'" defines that all content should come from the site’s own origin, this excludes subdomains.¹⁸

w.Header().Add("X-XSS-Protection", "1; mode=block")

The ‘X-XSS-Protection’ header stops pages from loading when they detect XSS (Cross-site scripting) attacks. This is where the attacker causes a page to load some malicious JavaScript, and does so by sending the malicious payload as part of the request. With this header specify to load or block the page from loading. The setting "1; mode=block;" will completely block the page from loading. This is a special feature header for the Chrome and Internet Explorer browsers, and seems to be unnecessary for API’s and should be covered by a restrictive Content Security Policy.¹⁹

w.Header().Add("X-Frame-Options", "DENY")

The ‘X-Frame-Options’ header let’s the browser know whether it is allowed to render a page in a , , or tag. This is used to avoid ‘click-jacking’ such that the content isn’t embedded into other sites. Here we set it to "DENY", which blocks the content from being embedded in other pages.²⁰

w.Header().Add("Referrer-Policy", "strict-origin-when-cross-origin")

The ‘Referrer-Policy’ header lets us define what information is being sent to external sites/resources in the Referer request header. When a client accesses an url from hyperlink, or a webpage loads an external resource, the browser adds the header Referer (yes that is intentionally misspelled), to let the destination know the origin of that request. Imagine that your webpage has a link to an external site, and that external site receives the Referer header with information that should only be used internally. With this header we can control what is being sent to the destination. In this example we will set it to strict-origin-when-cross-origin, which sends the full referer to the sources on the same origin, and url without path to external origin destinations, and send no header to less-secure destinations (HTTPS→HTTP).²¹

w.Header().Add("X-Content-Type-Options", "nosniff")

The ‘X-Content-Type-Options’ header indicates the MIME types set by the "Content-Type" header, which should not be changed and be followed. It prevents browsers from interpreting files as something else than what was specified the "Content-Type" header. It opts out of ‘Mime type sniffing’: guessing the correct Mime type by looking at the bytes of the resource done by browser. Without this header, browsers can incorrectly detect files as scripts and stylesheets, leading to XSS attacks. Setting the header to "nosniff" indicates that browsers should prevent incorrectly detecting non-scripts as scripts.²²

w.Header().Add("Content-Type", "text/plain; charset=UTF-8")

In order to make ‘X-Content-Type-Options’ to work correctly, we need to set the ‘Content-Type’ to the correct MIME type. For this example we’ll set it to "text/plain; charset=UTF-8". This tells the client that the type is text, the subtype is plain, and that the character encoding is utf-8.²³

In this section we’ve set up some response headers that will mitigate some common vulnerabilities.²⁴ For brevity, I’ve added the response header to the Handler for a single response. In an actual implementation you would likely add this to a sort of middleware for your handlers.²⁵

Again, we can test out our HTTP webserver:

$ go run server.go -cert=cert.pem -key=key.pem
$ curl --verbose --insecure https://localhost

*   Trying ::1:443...
* Connected to localhost (::1) port 443 (#0)
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: O=mkcert development certificate; OU=jp@aztlan
*  start date: Jun  1 00:00:00 2019 GMT
*  expire date: Apr 15 18:58:07 2030 GMT
*  subjectAltName: host "localhost" matched cert's "localhost"
*  issuer: O=mkcert development CA; OU=jp@aztlan; CN=mkcert jp@aztlan
*  SSL certificate verify ok.
> GET / HTTP/1.1
> Host: localhost
> User-Agent: curl/7.52.1
> Accept: */*
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Content-Security-Policy: default-src 'self'
< Content-Type: text/plain
< Referrer-Policy: strict-origin-when-cross-origin
< Strict-Transport-Security: max-age=63072000; includeSubdomains
< X-Content-Type-Options: nosniff
< X-Frame-Options: DENY
< X-Xss-Protection: 1; mode=block
< Date: Sun, 24 May 2020 14:21:49 GMT
< Content-Length: 13
<
hello, world
* Connection #0 to host localhost left intact

And, now you can see that we’ve implemented the secure headers.

CORS / CSRF

In this post we’ve added timeouts, a cryptographic protocol, and secure headers to our HTTP webserver. And depending on the size, utility, and what the server is going to be used for, you likely want to add additional security measures such as CORS, and CSRF. Let’s check out what they are, and investigate at what point we want to implement those measures.

CORS

CORS²⁶ (Cross-Origin Resource Sharing) specifies which cross-domain requests for a server are allowed from a webbrowser. Meaning: are request to, for instance, an api allowed from somewhere else than from the domain of the original api. One would likely disable CORS header, when cross-domain calls are not supported or expected. (OWASP 2020c)

CORS are disabled by default by your browser, meaning you can’t access resources from different origins. As a server you specify to the client which domains (origins), are allowed to make requests. It allows the client’s browser to determine if and when cross-domain requests should be allowed. Suppose a user visits example.com and the page attempts a cross-origin request to fetch the user’s data from service.example.com. The server (service.example.com) can then reply that requests from all domains are allowed. So requests from the origin example.com (and any other for that matter), are allowed to be made against the server ‘service.example.com’. (Wikipedia 2020a)

You can set CORS with the Access-Control headers (Mozilla 2020b):

Access-Control-Allow-Origin - specifies either a single origin, which tells browsers to allow that origin to access the resource; or else — for requests without credentials — the * wildcard, to tell browsers to allow any origin to access the resource.
Access-Control-Expose-Headers - lets a server whitelist headers that browsers are allowed to access.
Access-Control-Max-Age - indicates how long the results of a preflight request can be cached.
Access-Control-Allow-Credentials - indicates whether the response to the request can be exposed when the credentials flag is true.
Access-Control-Allow-Methods - specifies the method or methods allowed when accessing the resource.
Access-Control-Allow-Headers - is used in response to a preflight request to indicate which HTTP headers can be used when making the actual request.

CSRF

CSRF²⁷ (Cross Site Request Forgery) is to execute an undesired action on the server when the user is authenticated. So executing requests on your behalf from somewhere else. Unlike cross-site scripting (XSS), which exploits the trust a user has for a particular site, CSRF exploits the trust that a site has in a user’s browser. (Wikipedia 2020b)

Imagine being authenticated at myexamplebank.com, and an attacker is able to get your browser to send a request to myexamplebank.com. By either letting you, for instance, click a link or visit a page that has instruction to send a request to myexamplebank.com. Now, the request is sent by the attacker with the instructions to transfer money from your account to his, by including cookies that are associated with myexamplebank.com. And when the cookies contain authentication data, the attacker is able to execute the transfer.

In order to mitigate this we need to make sure that HTTP methods such as GET, HEAD, OPTIONS are not allowed to change any server side state. And that the HTTP methods POST, PUT, PATCH, DELETE use a token based mitigation protocol. This method is the most popular, and recommended method for protecting your web application from CSRF attacks. This method works by requiring that requests that are able to change server side state to include a token. The token is generated server side once per user session, or for each request. It should be unique per user session, secret, and unpredictable. This will prevent CSRF attacks because an attack will not be able to create a valid request without this token. (OWASP 2020a) We can implement these token in Go by using the package gorilla/mux.

Conclusion

In this posts we’ve explored how we can make our Go based HTTP webserver more secure by implementing timeouts, TLS, and secure headers. This gives us a good starting-point, and as your application grows, and depending on its application and utility, additional security measures should be implemented. To reiterate what was mentioned in the introduction, we do well to check, and continue checking for best-practices, and there are some very good checklists²⁸ available that can help you make your HTTP webservers and web applications more secure.

References

Cloudflare. 2020. “What Is Transport Layer Security.” 2020. https://www.cloudflare.com/learning/ssl/transport-layer-security-tls/.

Grigorik, Ilya. 2013. High-Performance Browser Networking. O’Reilly Media. https://hpbn.co/.

Let’s Encrypt. 2020. “Certificates for Localhost.” 2020. https://letsencrypt.org/docs/certificates-for-localhost/.

Mozilla. 2020a. “Security/Server Side TLS.” 2020. https://wiki.mozilla.org/Security/Server_Side_TLS.

———. 2020b. “Web Technology for Developers - CORS.” 2020. https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS.

OWASP. 2020a. “Cross-Site Request Forgery (CSRF) Prevention Cheat Sheet.” 2020. https://cheatsheetseries.owasp.org/cheatsheets/Cross-Site_Request_Forgery_Prevention_Cheat_Sheet.html.

———. 2020b. “OWASP Secure Headers Project.” 2020. https://www.owasp.org/index.php/OWASP_Secure_Headers_Project#tab=Headers.

———. 2020c. “REST Security Cheat Sheet.” 2020. https://github.com/OWASP/CheatSheetSeries/blob/master/cheatsheets/REST_Security_Cheat_Sheet.md.

Valsorda, Filippo. 2016. “So You Want to Expose Go on the Internet.” 2016. https://blog.gopheracademy.com/advent-2016/exposing-go-on-the-internet/.

Wikipedia. 2020a. “Cross-Origin Resource Sharing.” 2020. https://en.wikipedia.org/wiki/Cross-origin_resource_sharing.

———. 2020b. “Cross-Site Request Forgery.” 2020. https://en.wikipedia.org/wiki/Cross-site_request_forgery.

———. 2020c. “HTTP Strict Transport Security.” 2020. https://en.wikipedia.org/wiki/HTTP_Strict_Transport_Security.

———. 2020d. “Public-Key Cryptography.” 2020. https://en.wikipedia.org/wiki/Public-key_cryptography.

———. 2020e. “Symmetric-Key Algorithm.” 2020. https://en.wikipedia.org/wiki/Symmetric-key_algorithm.

———. 2020f. “Transport Layer Security.” 2020. https://en.wikipedia.org/wiki/Transport_Layer_Security.

———. 2020g. “X.509.” 2020. https://en.wikipedia.org/wiki/X.509.

Go Time - 101 ↩︎
Go Documentation: Writing Web Applications ↩︎
Additional information on timeouts: Ilija Eftimov - Make resilient Go net/http servers using timeouts, deadlines and context cancellation, Simon Frey - Standard net/http config will break your production environment ↩︎
Go Documentation: net/http ↩︎
More information on other application protocols: Wikipedia - Application Layer ↩︎
Some more information on session keys used in TLS: Cloudflare - What Is a Session Key?↩︎
In depth look at the TLS handshake: Cloudflare - What Happens in a TLS Handshake?↩︎
Go Documentation: TLS ↩︎
Wikipedia - Privacy-Enhanced Mail ↩︎
Go Documentation: TLS - Config ↩︎
More information on ECDSA: Cloudflare - ECDSA: The digital signature algorithm of a better internet ↩︎
You might need to update curl and openssl to be able to use TLS version 1.3. Here are some links that can help you install/upgrade them on a linux system. cURL 1, openssl 1, 2. Note for cURL it is important to ./configure --with-ssl.↩︎
Go Documentation: autocert ↩︎
Let’s Encrypt - Rate Limits ↩︎
Go Documentation: autocert - Manager.HTTPHandler ↩︎
Some information about secure headers: OWASP - Secure headers project, Appcanary - Everything you need to know about HTTP security headers ↩︎
More information on HSTS: Mozilla - Web Security - HSTS, Mozilla - Web technology for developers - HSTS OWASP - Secure Headers - HSTS, Appcanary - Everything you need to know about HTTP security headers - HSTS ↩︎
More information on CSP: Mozilla - Web Security - HSTS, Mozilla - Web technology for developers - CSP 1, Mozilla - Web technology for developers - CSP 2, OWASP - Secure Headers - CSP, Appcanary - Everything you need to know about HTTP security headers - CSP ↩︎
More information on XSS: Mozilla - Web Security - XSS, Mozilla - Web technology for developers - XSS, OWASP - Secure Headers - XSS, Appcanary - Everything you need to know about HTTP security headers - XSS ↩︎
More information on X-Frame: Mozilla - Web Security - XFO, Mozilla - Web technology for developers - XFO, OWASP - Secure Headers - XFO, Appcanary - Everything you need to know about HTTP security headers - XFO ↩︎
More information on Referrer Policy: Mozilla - Web Security- Referrer Policy, Mozilla - Web technology for developers- Referrer Policy, OWASP - Secure Headers - Referrer Policy, Appcanary - Everything you need to know about HTTP security headers - Referrer Policy ↩︎
More information on X-Content-Type-Options: Mozilla - Web Security - XTCO, Mozilla - Web technology for developers - XTCO, OWASP - Secure Headers - XTCO, Appcanary - Everything you need to know about HTTP security headers - XTCO ↩︎
More information on Content-Type: Mozilla - Web Technology - Content-Type ↩︎
Some information about secure headers: OWASP - Secure headers project, Appcanary - Everything you need to know about HTTP security headers ↩︎
Some examples of response headers middleware implementations: Github - unrolled/secure, Mat Reyer - How I Write HTTP Web Services after Eight Years,↩︎
More information on CORS: Mozilla - Web technology for developers - Cross-Origin Resource Sharing (CORS), Wikipedia - Cross-origin resource sharing ↩︎
More information on CSRF: OWASP - Cross-Site Request Forgery (CSRF) Prevention Cheat Sheet, StackExchange - Should I use CSRF protection on Rest API endpoints?↩︎
Some useful security checklists/guidelines: Mozilla - Web security, Shieldfy - API Security Checklist, OWASP - Web Checklist, OWASP - REST Security Cheatsheet, OWASP - Cheat Sheet Series Project ↩︎

Encrypt data with a password in Go

Tue, 25 Feb 2020 08:00:00 +0000

Introduction

When we’re encrypting data, typically we will create a random key that is able to decrypt that data. In some specific cases one wants to use a user specified key to decrypt that data like a password. However, the key that is used for cryptographic algorithms typically needs to be at least 32 bytes. But, it is likely that our password won’t make that criteria, so we need to have a solution for that. Recently, I needed such a method, and in this post I’ll lay out what I’ve done in order to solve it. But before we get into the nitty-gritty.

DISCLAIMER: I’m not an expert at encryption, I’ve mentioned the sources that I’ve used to come to the solutions provided in this post. I implore you read/watch those sources to better understand it. And, as such if there are any errors in the post/code please let me know or leave a comment so I can update it, so that there is no perpetuation of wrong methods/techniques.

OK, since we’ve got that out of the way, let’s begin!

Encrypt

Let’s first start with encrypting our data. We’ll start with creating the Encrypt function that will accept a key and a data argument. Based on that we will encrypt the data that can be decrypted using the key. First, we will generate that key by using 32 random bytes, later on we’ll replace that with our password. Below, shows the code that is able to encrypt our data, provided by a generated key.

import (
    "crypto/aes"
    "crypto/cipher"
    "crypto/rand"
)

func Encrypt(key, data []byte) ([]byte, error) {
    blockCipher, err := aes.NewCipher(key)
    if err != nil {
        return nil, err
    }

    gcm, err := cipher.NewGCM(blockCipher)
    if err != nil {
        return nil, err
    }

    nonce := make([]byte, gcm.NonceSize())
    if _, err = rand.Read(nonce); err != nil {
        return nil, err
    }

    ciphertext := gcm.Seal(nonce, nonce, data, nil)

    return ciphertext, nil
}

So, let’s go over the code, and inspect what we’re doing.

func Encrypt(key, data []byte) ([]byte, error)

First, we start by creating our Encrypt function, and it will accept a key and a data argument. We’ll be using a byte slice instead of an io.Reader as the data argument. While using io.Reader would allow us to use the Encrypt function with every other type that implements the io.Reader interface. (Ryer 2015) It is however because of the nature of io.Reader, being a stream of data, that when we want to decrypt the ciphertext, we need to see it in its entirety. A solution would be to break the stream into discrete chunks, however this would add significant complexity to the problem. ¹(Isom 2015)

blockCipher, err := aes.NewCipher(key)

We’re initializing the block cipher based on the key that we provided. Here we’re using the crypto/aes² package that implements the AES³ ⁴ (Advanced Encryption Standard) encryption algorithm. AES is a symmetric-key encryption algorithm, that will be secure enough for modern use cases. Additionally, AES uses hardware acceleration on most platforms, so it’ll be pretty fast to use. (Tankersley 2016)

gcm, err := cipher.NewGCM(blockCipher)

Here we’re wrapping the block cipher, with a specific mode. We do this because we shouldn’t use a cipher.Block interface directly. This is because the block cipher only encrypts 16 bytes of data, nothing more. So if you would call blockCiper.Encrypt() it would only encrypt the first 16 bytes. Thus we need something on top of that, and wrap the block cipher, and those are called modes. Again we have several modes to choose from, and here we’re going to use the Galois Counter Mode (GCM)⁵, with a standard nonce length.

Only GCM provides authenticated encryption, and it implements the cipher.AEAD interface (Authenticated Encryption with Associated Data)⁶. Authenticated encryption means that not only is your data going to be confidential, secret, and encrypted, it’s also now going to be tamper proof. If someone alters the ciphertext you will not then be able to validly decrypt it. When you’re using authenticated encryption and someone messes with your data it just fails to decrypt. (Tankersley 2016; Isom 2015)

nonce := make([]byte, gcm.NonceSize())
if _, err = rand.Read(nonce); err != nil {
    return nil, err
}

Before we can encrypt our bytes we need to generate a randomized nonce, and its length is specified by the GCM. The nonce stands for: number once used, and it’s a piece of data that should not be repeated and only used once in combination with any particular key. Meaning: don’t repeat the combination of a key and a nonce more than once. But, how do you keep track of that? If we use sufficiently large numbers for a nonce we should probably be fine for this use-case. (Isom 2015; Viega and Messier 2003, 134–35) We do that by using Go’s crypto/rand package to read randomized bytes into the nonce byte slice.⁷

encryptedData := gcm.Seal(nonce, nonce, data, nil)

The nonce that we’re going to use for encrypting our data, is also needed to decrypt it. So we need to be able to refer to it while decrypting, and one of the strategies is to add it to the encrypted data. In this example we will prepend the nonce to the encrypted data. We do that by passing in the nonce as the first argument dst of the Seal function, and as such the encrypted data will be appended to it.⁸ We can do this because the nonce doesn’t have to be secret, it just has to be unique. (Tankersley 2016)

Decrypt

Now, we’re able to encrypt our data, and let’s implement the Decrypt function.

import (
    "crypto/aes"
    "crypto/cipher"
)

func Decrypt(key, data []byte) ([]byte, error) {
    blockCipher, err := aes.NewCipher(key)
    if err != nil {
        return nil, err
    }

    gcm, err := cipher.NewGCM(blockCipher)
    if err != nil {
        return nil, err
    }

    nonce, ciphertext := data[:gcm.NonceSize()], data[gcm.NonceSize():]

    plaintext, err := gcm.Open(nil, nonce, ciphertext, nil)
    if err != nil {
        return nil, err
    }

    return plaintext, nil
}

Again let’s go over the code and check what it does. It is largely the same code as the Encrypt function, so let’s inspect the parts that differ.

nonce, ciphertext := data[:gcm.NonceSize()], data[gcm.NonceSize():]

Remember from the last section, that we prepended the nonce to the data using gcm.Seal to create the ciphertext? Now we need to split those parts so we can use them independently. And we’re creating those part by slicing the data based on the size of the nonce that gcm provides.

plaintext, err := gcm.Open(nil, nonce, ciphertext, nil)

Now, we’re using gcm.Open to decrypt the ciphertext into plaintext. ⁹

Key

We’ve been passing in a key to both the Encrypt and Decrypt functions, but we have yet to make it, so let’s do that.

import (
    "crypto/rand"
)

func GenerateKey() ([]byte, error) {
    key := make([]byte, 32)

    _, err := rand.Read(key)
    if err != nil {
        return nil, err
    }

    return key, nil
}

Here we’re generating a random key using Go’s crypto/rand package. For AES we need a key that has the length of 32 bytes, so we make a byte slice of size 32. Then we let rand.Read() fill the slice with random bytes.¹⁰

Now we have enough to encrypt and decrypt some data, so let’s put it all together and test it out:

// crypto.go
package main

import (
    "crypto/aes"
    "crypto/cipher"
    "crypto/rand"
    "encoding/hex"
    "fmt"
    "log"
)

func Encrypt(key, data []byte) ([]byte, error) {
    blockCipher, err := aes.NewCipher(key)
    if err != nil {
        return nil, err
    }

    gcm, err := cipher.NewGCM(blockCipher)
    if err != nil {
        return nil, err
    }

    nonce := make([]byte, gcm.NonceSize())
    if _, err = rand.Read(nonce); err != nil {
        return nil, err
    }

    ciphertext := gcm.Seal(nonce, nonce, data, nil)

    return ciphertext, nil
}

func Decrypt(key, data []byte) ([]byte, error) {
    blockCipher, err := aes.NewCipher(key)
    if err != nil {
        return nil, err
    }

    gcm, err := cipher.NewGCM(blockCipher)
    if err != nil {
        return nil, err
    }

    nonce, ciphertext := data[:gcm.NonceSize()], data[gcm.NonceSize():]

    plaintext, err := gcm.Open(nil, nonce, ciphertext, nil)
    if err != nil {
        return nil, err
    }

    return plaintext, nil
}

func GenerateKey() ([]byte, error) {
    key := make([]byte, 32)

    _, err := rand.Read(key)
    if err != nil {
        return nil, err
    }

    return key, nil
}

func main() {
    data := []byte("our super secret text")

    key, err := GenerateKey()
    if err != nil {
        log.Fatal(err)
    }

    ciphertext, err := Encrypt(key, data)
    if err != nil {
        log.Fatal(err)
    }

    fmt.Printf("ciphertext: %s\n", hex.EncodeToString(ciphertext))

    plaintext, err := Decrypt(key, ciphertext)
    if err != nil {
        log.Fatal(err)
    }

    fmt.Printf("plaintext: %s\n", plaintext)
}

And, we can run this example with:

$ go run crypto.go

Now, we have enough to encrypt and decrypt our data with a randomized key. This is cool and now we have a key that allows us to encrypt and decrypt our data. But that means that the key now becomes our password and weren’t able to choose it ourselves, additionally it has a length of 32 bytes.

But, as mentioned in the start of the post, we want to be able to encrypt and decrypt the data by providing our own key namely a password that we’ve chosen to use. We will be doing that in the following section.

Password

Now, the aes.NewCipher() needs a 16, 24, or a 32 byte key, and in this example we are using a 32 byte key. However, our password likely isn’t going to be 32 bytes. So we need to transform our password to a suitable key. And we do that by using a key derivation function (KDF)¹¹ to ‘stretch’ the password to make it a suitable cryptographic key. This key-stretching¹² characterizes itself by being slow. This is done in order to make it that, an attacker needs to spend a lot of resources to attempt to brute force an attack the on the password. We have several options for KDF’s: Argon2¹³, scrypt¹⁴, bcrypt¹⁵, and pbkdf2¹⁶. Choosing one depends on several factors, but mainly how safe it is. ¹⁷ ¹⁸ ¹⁹ ²⁰ ²¹ ²²

Typically in a KDF we have a password, a salt, and an iterations argument. The salt²³ is used to prevent an attacker from just storing password/key pairs, and prevents an attacker from precomputing a dictionary of derived keys, as a different salt yields a different output. Each password has to be checked with the salt used to derive the key. (Isom 2015; Wikipedia 2020) The salt is related to the nonce in that it also needs to be randomly generated. And as with the nonce, the salt doesn’t need to be secret, it needs to be unique. The iterations argument or the difficulty parameter, signifies how many times to repeat the process. This is because, even with salt, a dictionary attack is still possible, but with the iterations count, it will slow down the time it takes to compute a key from a password. (Viega and Messier 2003, 141–42)

In this example we’ll be using scrypt, so let’s see how we can implement that into our program.

import (
    "crypto/rand"

    "golang.org/x/crypto/scrypt"
)

func DeriveKey(password, salt []byte) ([]byte, []byte, error) {
    if salt == nil {
        salt = make([]byte, 32)
        if _, err := rand.Read(salt); err != nil {
            return nil, nil, err
        }
    }

    key, err := scrypt.Key(password, salt, 1048576, 8, 1, 32)
    if err != nil {
        return nil, nil, err
    }

    return key, salt, nil
}

Again let’s go over the code and see what it does.

func DeriveKey(password, salt []byte) ([]byte, []byte, error)

Here we accept the password as a slice of bytes as the argument, and we return the resulting key, and salt.

salt := make([]byte, 32)
if _, err := rand.Read(salt); err != nil {
    return err
}

Just like our Encrypt function, we’ll be creating the salt with 32 random bytes.

key, err := scrypt.Key(password, salt, 1048576, 8, 1, 32)

Here we’re using the scrypt package from golang.org/x/ library. ²⁴ From the documentation we can read that the Key function accepts the following arguments:

func Key(password, salt []byte, N, r, p, keyLen int) ([]byte, error)

The arguments password and salt speak for themselves. N is the number of iterations. In a presentation given by C. Percival it is recommended that for interactive logins $16384$ ( $2^{14}$ ) iterations, and for file encryption $1048576$ ( $2^{20}$ ) iterations are used. (Percival 2005a, 2005b; Isom 2015) The arguments r and p must satisfy $r * p < 2^{30}$ , if it doesn’t satisfy the limits, the function returns a nil byte slice and an error. (Golang Documentation 2020). The r argument defines the relative memory cost parameter it controls the blocksize in the underlying hash, the recommended value is 8. The p argument is the relative CPU cost parameter and the recommended value for this is 1. (Isom 2015; Percival 2005a) The keyLen argument defines the length of the bytes that are returned as key, as discussed this will be 32 bytes.

Result

Now that we’ve created our DeriveKey function we need to update our code to support it. So let’s do that, it should resemble the code below:

// scrypt.go
package main

import (
    "crypto/aes"
    "crypto/cipher"
    "crypto/rand"
    "crypto/sha256"
    "encoding/hex"
    "fmt"
    "log"

    "golang.org/x/crypto/scrypt"
)

func Encrypt(key, data []byte) ([]byte, error) {
    key, salt, err := DeriveKey(key, nil)
    if err != nil {
        return nil, err
    }

    blockCipher, err := aes.NewCipher(key)
    if err != nil {
        return nil, err
    }

    gcm, err := cipher.NewGCM(blockCipher)
    if err != nil {
        return nil, err
    }

    nonce := make([]byte, gcm.NonceSize())
    if _, err = rand.Read(nonce); err != nil {
        return nil, err
    }

    ciphertext := gcm.Seal(nonce, nonce, data, nil)

    ciphertext = append(ciphertext, salt...)

    return ciphertext, nil
}

func Decrypt(key, data []byte) ([]byte, error) {
    salt, data := data[len(data)-32:], data[:len(data)-32]

    key, _, err := DeriveKey(key, salt)
    if err != nil {
        return nil, err
    }

    blockCipher, err := aes.NewCipher(key)
    if err != nil {
        return nil, err
    }

    gcm, err := cipher.NewGCM(blockCipher)
    if err != nil {
        return nil, err
    }

    nonce, ciphertext := data[:gcm.NonceSize()], data[gcm.NonceSize():]

    plaintext, err := gcm.Open(nil, nonce, ciphertext, nil)
    if err != nil {
        return nil, err
    }

    return plaintext, nil
}

func DeriveKey(password, salt []byte) ([]byte, []byte, error) {
    if salt == nil {
        salt = make([]byte, 32)
        if _, err := rand.Read(salt); err != nil {
            return nil, nil, err
        }
    }

    key, err := scrypt.Key(password, salt, 1048576, 8, 1, 32)
    if err != nil {
        return nil, nil, err
    }

    return key, salt, nil
}

func main() {
    var (
        password = []byte("mysecretpassword")
        data     = []byte("our super secret text")
    )

    ciphertext, err := Encrypt(password, data)
    if err != nil {
        log.Fatal(err)
    }

    fmt.Printf("ciphertext: %s\n", hex.EncodeToString(ciphertext))

    plaintext, err := Decrypt(password, ciphertext)
    if err != nil {
        log.Fatal(err)
    }

    fmt.Printf("plaintext: %s\n", plaintext)
}

And, we’re able to run and test it:

# First we need to get the scrypt package
$ go get -u golang.org/x/crypto/scrypt

$ go run scrypt.go

We’ve updated some parts, so let’s go over it.

key, salt, err := DeriveKey(key, nil)

In the Encrypt function we create our key by passing in our password, which is contained in the key argument. We pass in nil as the salt argument, that is because we want to create the salt since it is the first time we encrypt our data.

ciphertext = append(ciphertext, salt...)

Additionally, in the Encrypt function, we append the salt to our ciphertext.

salt, data := data[len(data)-32:], data[:len(data)-32]

And, because we append the salt to the ciphertext, we need to split and slice it in the Decrypt function, because we’re going to use it in the DeriveKey function.

key, _, err := DeriveKey(key, salt)

As you can see here we pass in the salt to the DeriveKey function and we’ll be able to retrieve the key that we used in order to encrypt our data.

Conclusion

With that, we’ve created two ways in order to encrypt and decrypt our data in Go. First we’ve encrypted our data by using the AES encryption algorithm, for which we’ve created a randomized key to be used for decrypting our data. Subsequently, we’ve updated our code to support using a password as our key. We’ve done that by key-stretching our password using a key derivation function, and we’ve used scrypt to achieve that. Hopefully, you found this post useful, and again I advice you to read and watch the sources that I’ve listed, and check out other sources to get a good overview on how to correctly and securely encrypt your data, and if you have any suggestions let me know.

References

Golang Documentation. 2020. “Package Scrypt.” 2020. https://godoc.org/golang.org/x/crypto/scrypt.

Isom, Kyle. 2015. Practical Cryptography with Go. Leanpub. https://leanpub.com/gocrypto/read.

Percival, C. 2005a. “scrypt: A new key derivation function.” https://www.bsdcan.org/2009/schedule/attachments/86_scrypt_slides.pdf.

———. 2005b. “Strong Key Derivation Via Sequential Memory-Hard Functions.” https://www.tarsnap.com/scrypt/scrypt.pdf.

Ryer, M. 2015. “Io.Reader in Depth.” 2015. https://medium.com/@matryer/golang-advent-calendar-day-seventeen-io-reader-in-depth-6f744bb4320b.

Tankersley, G. 2016. “Go for Crypto Developers.” https://youtu.be/2r_KMzXB74w.

Viega, J., and M. Messier. 2003. Secure Programming Cookbook for C and C++. O’Reilly Media.

Wikipedia. 2020. “Key Derivation Function - Wikipedia, The Free Encyclopedia.” 2020. https://en.wikipedia.org/wiki/Key_derivation_function.

How to profile Go applications inside a docker container

Thu, 12 Dec 2019 12:00:00 +0000

Introduction

In this post I’ll give a quick overview of several methods you can use for profiling/debugging Go applications that are running in a docker container. To get a more in-depth overview of the several methods, I’ve added the source links you can reference at the end of the post. I really advise you to read the sources listed, and read up on how to use and interpret the created profiles in order to debug/optimize your application. This post aims to help you to create those profiles when using a Docker container to run your Go applications. The main idea is that we are going to leverage the remote profiling capabilities that is supplied by the net/http package.

Application

We will be using pprof to profile your Go applications, you’ll typically use pprof when you’re able to access the binary of your application locally. When you’re using a docker container you’d need to have the go tool installed in the container to be able to profile it. However there is another way for profiling Go applications running in a docker container. For this we need to create an HTTP webserver, which will enable the pprof HTTP endpoints. From these endpoints we’re able to download profiles of the running application.

So first we need to create an HTTP webserver, that we will run in a goroutine. We can then import the net/http/pprof package, which automatically enables the pprof HTTP endpoints of the webserver we’ve created, and it will install the handlers under the http://localhost:6060/debug/pprof URL.

So we can add the following to our code:

// main.go
package main

import (
	"fmt"
	"math/rand"
	"net/http"
	_ "net/http/pprof"
)

func main() {

    // Add the following
    go func() {
        http.ListenAndServe(":6060", nil)
    }()

    // Some sample code
    for {
        n := rand.Intn(10) + 1
        fmt.Printf("Fibonacci sequence for %d:\n", n)
        for i := 1; i <= n; i++ {
            fmt.Println(fibonacci(i))
        }
    }
}

func fibonacci(n int) int {
	if n < 2 {
		return n
	}
	return fibonacci(n-1) + fibonacci(n-2)
}

When you are using a different router, we can do the following to make the pprof HTTP endpoints work. Below is an example how you would do this if you’re using gorilla/mux

// main.go
package main

import (
    _ "net/http/pprof"
    _ "expvar"
    "github.com/gorilla/mux"
)

func main() {
    router := mux.NewRouter()

    router.HandleFunc(
        "/debug/pprof/", pprof.Index,
    )
    router.HandleFunc(
        "/debug/pprof/cmdline", pprof.Cmdline,
    )
    router.HandleFunc(
        "/debug/pprof/profile", pprof.Profile,
    )
    router.HandleFunc(
        "/debug/pprof/symbol", pprof.Symbol,
    )
    router.HandleFunc(
        "/debug/pprof/trace", pprof.Trace,
    )
    router.Handle(
        "/debug/pprof/goroutine", pprof.Handler("goroutine"),
    )
    router.Handle(
        "/debug/pprof/heap", pprof.Handler("heap"),
    )
    router.Handle(
        "/debug/pprof/threadcreate", pprof.Handler("threadcreate"),
    )
    router.Handle(
        "/debug/pprof/block", pprof.Handler("block"),
    )
    router.Handle(
        "/debug/vars", http.DefaultServeMux,
    )

    http.ListenAndServe(":6060", router)
}

Docker

For the docker container we’ll create the following Dockerfile:

# Dockerfile
FROM golang:1.13

WORKDIR /go/src/app
COPY . .

RUN go get -d -v ./...
RUN go install -v ./...

CMD ["app"]

Next, we can build and start our Docker container:

$ docker build -t pprof-app .
$ docker run -p 6060:6060 -it --rm --name my-running-app pprof-app

Once you’ve updated the files you’ll be able to access the endpoints by using your browser and to go to http://localhost:6060/debug/pprof/, or you can use go tool pprof. We’ll explore a few of the methods we can profile the application using go tool pprof

pprof

Now, that we’ve set everything up to use pprof, we’ll profile our application for cpu, and heap. Additionally we’ll also trace the application.

CPU

When we want to start to profile our application to determine where an application spends time, we first need to make a CPU profile. And we have several methods to do so. Below I’ll give the common ones. The gist of it is that we’re fetching the profile over HTTP, on the endpoints that we’ve created in our application.

Go to http://localhost:6060/debug/pprof/profile, and after 30 seconds you’ll automatically download the profile, or
Use go tool pprof. It will tell you where the profile is being stored, and will open up an interactive terminal that you can use to interpret the results. Like so: go tool pprof http://localhost:6060/debug/pprof/profile

After that we can interpret the results, and for this we again have several options, and tools we can use to interpret the results You can read more about additional ways to interpret the results at the sites I’ve listed in the “Sources” section at the end of the post.

Here I’ll show you a few of them. One of them is that we’re able to inspect the results in an automatically created web interface.

$ go tool pprof -http=":8080" [path-to-profile-dump]

This will open up a webbrowser, in which we can inspect, for instance, the flamegraph of the created profile.

The horizontal axis represents the total number of samples collected. So the larger the area, the more time has been spent executing the associated function. The vertical axis represents the depth of the call stack. So the higher the peak, the deeper the call stack. Colors don’t represent anything specific; they’re just there to make a visual contrast.¹

Another method is that we can use the command line based interactive interface by using the go tool:

$ go tool pprof [path-to-profile-dump]

We can also instantly go to the web interface, and step over the whole download profile step, by using the following command:

$ go tool pprof -http=":8080" http://localhost:6060/debug/pprof/profile

One thing to remember is: if you’re making cpu profiles that takes a certain amount of time you need to specify that your server is able to handle those long request times. You typically want to add the following to your http.Server:

srv := &http.Server{
    WriteTimeout: 30 * time.Second,
}

Heap

In order to get a heap profile, so that we can inspect the memory usage of the application, we again can issue the following commands:

Go to http://localhost:6060/debug/pprof/heap, and we will automatically download the heap profile, or
Using the go tool, it will tell you were the profile is being stored, and it opens up an interactive terminal that you can use to interpret the results:

# Heap profile
#
# optional flags:
#   –alloc_space tells you how many megabytes have been allocated.
#   –inuse_space tells you know how many are still in use.
$ go tool pprof http://localhost:6060/debug/pprof/heap

And, just like with the CPU profile, we’re able to interpret it with go tool pprof, and the web interface:

# Go tool
$ go tool pprof [path-to-profile-dump]

# Web interface
$ go tool pprof -http=":8080" [path-to-profile-dump]

# Create profile and start web interface
$ go tool pprof -http=":8080" http://localhost:6060/debug/pprof/heap

Trace

To trace the runtime activities/events of an application, we can do the following. First, we need to download the trace file, and we can use curl for this. Specify for how many seconds you want to create a trace and make sure to create some load in that time frame. Once downloaded you can use the go tool to start up a web interface in which you can inspect the results.

The trace tool will capture execution events like the creation, blocking, unblocking of goroutines, garbage collection events, processor starting and stopping, enter, exit, and block syscalls.²

Issue the following commands:

$ curl http://localhost:6060/debug/pprof/trace?seconds=20 > trace.out
$ go tool trace trace.out

This will open up your webbrowser, and you’ll be able to interpret the results of your trace with the provided web interface.

Conclusion

We’ve explored some methods that allows us to profile our Go applications when it is run in a Docker container. There’s a lot more information out there, that can help you optimize your applications using these methods. I’ve listed several of those below in the “Sources” section. Be sure to check them out they have some great additional information that can help you.

Sources

How to profile Python applications inside a docker container

Thu, 21 Nov 2019 12:00:00 +0000

In the following post I’ll explain how you can profile a running Python program in a Docker container using py-spy. py-spy is able to generate flame graphs, and it can give us profiling capabilities in order to debug our Python programs. Now, when your Python code is running in a Docker container it can be a bit more difficult to profile your code, and this post sets out to show you how this can be done with these tools.

1. Setting up

Before we can profile the Python application, we need to set up the tools, and example files, that we will be using in this project. You can inspect the finished result here.

1.1 Project

Let’s get started with the project outline, and create the following files and folders, the docker-compose.yml is optional because I will show you how to use plain docker as well.

$ tree -L 1 --dirsfirst
.
├── app/
├── pyspy/
└── docker-compose.yml

1.2 py-spy

In the pyspy/ folder we will be creating a Dockerfile that we’re going to use to create a Docker container that contains the py-spy program. The Dockerfile should contain the following:

# pyspy/Dockerfile
FROM python:3.6
RUN pip install py-spy
WORKDIR /profiles
ENTRYPOINT [ "py-spy" ]
CMD []

Let’s test if is is working:

$ cd pyspy/
$ docker build -t pyspy .
$ docker run -it pyspy

py-spy 0.3.0
Sampling profiler for Python programs

USAGE:
    py-spy 

OPTIONS:
    -h, --help       Prints help information
    -V, --version    Prints version information

SUBCOMMANDS:
    record    Records stack trace information to a flamegraph, speedscope
              or raw file
    top       Displays a top like view of functions consuming CPU
    dump      Dumps stack traces for a target program to stdout
    help      Prints this message or the help of the given subcommand(s)

Cool, that worked! Now, you’re also able to install py-spy locally on your host system. When you do, be sure to read the documentation at: https://github.com/benfred/py-spy, on how to do this.

1.3 Python

Next, we want to create a Python program that we will use to profile. Create a new file in the app/ folder name run.py and copy the contents into it.

# app/run.py
import random


def factorial(n):
    factorial = 1
    for i in range(1, n + 1):
        factorial *= i

    return factorial


if __name__ == "__main__":
    while True:
        n = random.choice(range(1, 5))
        f = factorial(n=n)

        print("Factorial of {n} is {f}".format(n=n, f=f))

Next, create a Dockerfile in the app/ folder:

# app/Dockerfile
FROM python:3.6
WORKDIR /usr/src/app
COPY run.py .
CMD [ "python", "./run.py" ]

And again, let’s see if it is working:

$ cd app/
$ docker build -t app .
$ docker run -t app

Press Control-C to exit the program.

1.4 Docker Compose

I promised that I’ll show you how to also use Docker Compose to orchestrate our containers. Create docker-compose.yml file and add the following to it:

# docker-compose.yml
version: "3"
services:
  pyspy:
    build:
      context: pyspy/
    pid: "host"
    privileged: "true"
    volumes:
      - .:/profiles

  app:
    build:
      context: app/
    cap_add:
      - sys_ptrace

Now, we can build the containers like so:

$ docker-compose build

You’ll probably wondering about what the pid, privileged and cap_add is used for, and I’ll get to that in the next section.

2. Profiling the program

We’ve set everything up and we’re ready to test everything. First, let’s run the python program. We need to add --cap-add sys_ptrace, because by default Docker images do not have the SYS_PTRACE capability. And this is because Docker is restricting the process_vm_readv system call that py-spy uses to directly read the memory of the Python program.

# docker
$ docker run --cap-add sys_ptrace -t app --name py-app

# docker-compose
$ docker-compose up app

Now, before we’re able to use the py-spy program, we need to get the PID of the Python program that is running in the container. We will be using the PID of the program, and py-spy is going to sample from it.

# docker
$ docker inspect --format '{{.State.Pid}}' py-app
26982

# docker-compose
$ docker inspect --format '{{.State.Pid}}' profile-python-docker_app_1
26982

Now, that we’ve got the PID we will be using it with py-spy to profile our Python application. Now that we’re using the PID of the running program in the Docker container. We need the py-spy container to be able to use this PID inside it’s container. So we to use the host’s PID namespace inside this container, and we do that by adding the flag --pid=host to the run command. Additionally, we need to add the --privileged flag, so that it allows the container the same access to the host as processes running outside containers on the host.

# docker: change $(pwd) where you want the profiles to be saved
$ docker run \
    --pid=host \
    --privileged \
    -v $(pwd):/profiles \
    -it pyspy record -o myprofile.svg --pid 26982

# docker-compose
$ docker-compose run pyspy record -o myprofile.svg --pid 26982

You can now open the myprofile.svg file with a browser and look at the flamegraph that was created, and you’ll be able to interpret the results.

A cool feature of py-spy is that we can also create our profiles in the speedscope and inspect it with the online tool. Add the --format speedscope flag to the command and you’ll be able to import the profile in the speedscope format.

Conclusion

Now, you’ve got a basic setup to profile your Python applications in a Docker container. So, be sure to read up on all the features and options of py-spy so that you can fully utilize it and, profile your Python applications in Docker containers.

A minimal REST API Django setup as a microservices

Tue, 12 Nov 2019 12:00:00 +0000

In this post I will set out on how to set up a django project that can be used as a REST API microservice. To see the end result, you can investigate the code here. An important disclaimer: the code presented here is to be used in a development environment, review security practices for the framework(s) when you want to use it in production or expose it to the internet.

1. Project structure

We will using the following stack:

Name	Version
Docker	18.03.1-ce
Python	3.8
Django	2.2.7
PostgreSQL	12.0
Gunicorn	20.0.0
DRF	3.10.3

Let’s first start with the project structure and create some files and folders. Create the following files and folders:

$ tree -L 1 --dirsfirst
.
├── config/
├── docker-compose.yml
├── Dockerfile
└── README.md

First update our Dockerfile:

$ cat Dockerfile

FROM python:3.8

COPY . /srv/minimal-django
RUN pip3 install -r /srv/minimal-django/config/requirements.txt

CMD [ "gunicorn", "--config", \
    "/srv/minimal-django/config/gunicorn.py", \
    "minimal_django.wsgi" \
]

We will use a Python base image, and of course you can choose your own base image as you like. In the Dockerfile we’ll define that we need to copy the code, and install the Python package requirements. Lastly, we use Gunicorn as the WSGI application server.

Now, for convenience sake we’re using docker-compose to orchestrate our docker containers. So we will add the following contents to our docker-compose.yml file:

$ cat docker-compose.yml

version: "3"

volumes:
  postgres:
    driver: local

services:
    postgres:
      image: postgres:12.0
      environment:
        - POSTGRES_NAME=postgres
        - POSTGRES_USER=postgres
        - POSTGRES_PASSWORD=mysecretpassword
        - PGDATA=pgdata
      volumes:
        - postgres:/var/lib/postgresql/data/pgdata

    minimal_django:
      build:
        context: .
      ports:
        - "4001:4001"
      volumes:
        - .:/srv/minimal_django
      depends_on:
        - postgres

We will be using PostgreSQL as our datastore in this example, and we will calling our Django application minimal_django.

Next, we need to add the configuration files in the folder config/, and we will add the following files:

$ ls config/
gunicorn.py  requirements.txt

But first, let’s add the python packages we will using in the Django application, and we will be adding those in the requirements.txt file.

$ cat config/requirements.txt

django==2.2.7
gunicorn==19.9.0
psycopg2-binary==2.8.4
djangorestframework==3.10.3

As you can see we will be using Django REST Framework for the REST API. Next up we will create the Django project.

2. Create the Django project

Now, that we have the basic structure set up. We need to create an actual Django project, and we will be using the Docker container for this. This way we don’t necessarily install Django on our host system.

First, build the minimal_django container.

$ docker-compose build minimal_django

Second, we will start an interactive shell in the minimal_django container.

$ docker-compose run minimal_django sh

Third, we will create the Django project by executing the following command in the attached shell:

# cd /srv/minimal-django
# django-admin startproject minimal_django

We’ve created the project ‘inside’ the container, and because we’ve attached a volume to the location of the files. So now we need to fix the file permissions on our host system, because they have been created as the root account. So, exit from the shell in the docker container, and change the owner of the folder that we’ve just created.

$ chown -R $USER:$USER minimal_django/

3. Update `settings.py`

Because this will be a minimal Django implementation we will updating the settings.py file, and remove the non-essential elements.

Update ALLOWED_HOSTS
```
ALLOWED_HOSTS = ["*"]
```

Update INSTALLED_APPS

INSTALLED_APPS = [
    'rest_framework',
    'api',
]

Remove MIDDLEWARE, and its contents
Remove TEMPLATES, and its contents

Update the DATABASES dict

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql_psycopg2',
        'NAME': "postgres",
        'USER': "postgres",
        'PASSWORD': "mysecretpassword",
        'HOST': "postgres",
        'PORT': "5432",
    }
}

Remove AUTH_PASSWORD_VALIDATORS, and its contents
Remove STATIC_URL, and its contents
Remove LANGUAGE_CODE, USE_I18N, USE_L10N, and its contents

Add the following Django REST Framework configuration

REST_FRAMEWORK = {
    "DEFAULT_RENDERER_CLASSES": [
        'rest_framework.renderers.JSONRenderer',
    ],
    "DEFAULT_AUTHENTICATION_CLASSES": [],
    "DEFAULT_PERMISSION_CLASSES": [],
    "UNAUTHENTICATED_USER": None,
    'PAGE_SIZE': 10,
    'DEFAULT_PAGINATION_CLASS': \
        'rest_framework.pagination.LimitOffsetPagination',
    'DEFAULT_FILTER_BACKENDS': [
        'rest_framework.filters.OrderingFilter'
    ],
}

Adding logging configuration, this is optional and you can edit/update it to your preferences.

LOGGING = {
    'version': 1,
    'disable_existing_loggers': True,
    'formatters': {
        'verbose': {
            'format':
            '%(pathname)s:%(lineno)d (%(funcName)s) '
            '%(levelname)s %(message)s',
        },
        'simple': {
            'format': '%(levelname)s %(message)s'
        },
    },
    'handlers': {
        'null': {
            'level': 'DEBUG', 'class': 'logging.NullHandler',
        },
        'stderr': {
            'level': 'DEBUG',
            'formatter': 'simple',
            'class': 'logging.StreamHandler',
        },
    },
    'loggers': {
        'django': {
            'handlers': ['stderr'],
            'propagate': True,
            'level': 'WARNING',
        },
    },
}

NOTE: Again, this is not a production ready configuration!

4. Configuring the Django `api` app

Now we are ready to create the Django app. We will call it api, and within it we will create a simple REST API. First, create a folder in which it will be contained.

$ mkdir minimal_django/api/

Inside this folder create the following files:

$ ls minimal_django/api/
__init__.py  models.py  serializers.py  urls.py  views.py

We will start with the file models.py, and we will create a very simple model.

# minimal_django/api/models.py
from django.db import models


class Pet(models.Model):
    name = models.CharField(max_length=64)

Next, update the views.py file where we will add the Django Rest Framework ViewSet.

# minimal_django/api/views.py
from rest_framework import viewsets

from api.models import Pet
from api.serializers import PetSerializer


class PetViewSet(viewsets.ModelViewSet):
    queryset = Pet.objects.all()
    serializer_class = PetSerializer

Create and update the serializers.py file that will be used in our application.

# minimal_django/api/serializers.py
from rest_framework import serializers

from api.models import Pet


class PetSerializer(serializers.ModelSerializer):
    class Meta:
        model = Pet
        fields = ('name', )

Update the urls.py file, and we will use the Django Rest Framework default routers.

# minimal_django/api/urls.py
from django.conf.urls import include, url
from rest_framework.routers import DefaultRouter

from api import views

router = DefaultRouter()
router.register(r'pets', views.PetViewSet)

urlpatterns = [url(r'^', include(router.urls))]

Also update the urls.py file, of the Django project itself and remove the paths to the admin environment.

# minimal_django/urls.py
from django.conf.urls import url, include

urlpatterns = [
    url(r'^', include('api.urls')),
]

5. Configuring Gunicorn

We will be using Gunicorn as the WSGI application server. Create and update the gunicorn.py file with the following content. Refer to the configuration documentation for more configuration options.

# config/gunicorn.py

# Documentation at: http://docs.gunicorn.org/en/latest/index.html
django_project_name = "minimal_django"

# Chdir to specified directory before apps loading
chdir = "/srv/minimal-django/minimal_django"

# The socket to bind to
bind = ":4001"

# The class of worker processes for handling requests
worker_class = "sync"

# Is a number of OS processes for handling requests
workers = 4

# Is a maximum count of active greenlets grouped in a pool that will be
# allowed in each process
worker_connections = 1000

# The maximum number of requests a worker will process before restarting
max_requests = 5000

# Workers silent for more than this many seconds are killed and restarted
timeout = 120

# Set environment variable
raw_env = \
    ["DJANGO_SETTINGS_MODULE={}.settings".format(django_project_name)]

# The access log file to write to, "-" means to stderr
accesslog = "-"

# The error log file to write to, "-" means to stderr
errorlog = "-"

6. Start it up

Before we can run the application, we need to make initial migrations and propagate the model definition into our database schema. We will be using our Docker container to create those migrations, and again as a result we need to change the folder permissions in the end.

# First start the postgresql container
$ docker-compose up -d postgres

# Create the migrations
$ docker-compose run minimal_django \
    /srv/minimal-django/minimal_django/manage.py \
    makemigrations api

# Actually migrate the schema
$ docker-compose run minimal_django \
    /srv/minimal-django/minimal_django/manage.py \
    migrate

# Reset the folder permissions of the created migrations folder
$ chown -R $USER:$USER minimal_django

With that done we’re able to run our application, since we already started our postgresql container we will able to execute the following command.

$ docker-compose up minimal_django

Now, the application should be running and you’ll be able to access the API with curl or any other method.

$ curl http://localhost:4001/pets/

How to profile Go applications inside a docker container

Fri, 11 Aug 2017 12:00:00 +0000

This article has been updated and you can view its updated version here

general sources

pprof

When you want to use pprof to profile your Go applications, that are running in a container, we need to make sure we turn on the internal pprof HTTP endpoints. We’ll do this by updating the source code of your application. What follows is an example of how you would enable those endpoints.

// main.go
import (
    _ "net/http/pprof"
    _ "expvar"
    "github.com/gorilla/mux"
)

func main() {

    // OPTION 1: Add this to your application
    go func() {
        http.ListenAndServe("0.0.0.0:6060", nil)
    }()

    // OPTION 2: When using gorilla mux you can do the following:
    router := mux.NewRouter()
    router.HandleFunc("/debug/pprof/", pprof.Index)
    router.HandleFunc("/debug/pprof/cmdline", pprof.Cmdline)
    router.HandleFunc("/debug/pprof/profile", pprof.Profile)
    router.HandleFunc("/debug/pprof/symbol", pprof.Symbol)
    router.HandleFunc("/debug/pprof/trace", pprof.Trace)
    router.Handle("/debug/pprof/goroutine", pprof.Handler("goroutine"))
    router.Handle("/debug/pprof/heap", pprof.Handler("heap"))
    router.Handle("/debug/pprof/threadcreate", pprof.Handler("threadcreate"))
    router.Handle("/debug/pprof/block", pprof.Handler("block"))
    router.Handle("/debug/vars", http.DefaultServeMux)
    http.ListenAndServe("0.0.0.0:6060", router)
}

Now, we need to make sure that we’re able to access those endpoints. When you’re using docker-compose update your docker-compose.yaml file as follows:

// docker-compose.yaml
version: "3"
services:
  app:
    build: ./build-dir
    ports:
      - "6060:6060"

Once you’ve updated the files you’ll be able to access the endpoints by using your browser and to go to http://localhost:6060/debug/pprof/, or you can use go tool pprof:

# Heap profile
#
# optional flags:
#   –alloc_space tells you how many megabytes have been allocated.
#   –inuse_space tells you know how many are still in use.
$ go tool pprof http://localhost:6060/debug/pprof/heap

# CPU profile
$ go tool pprof http://localhost:6060/debug/pprof/profile

# Using trace tool
$ curl http://localhost:6060/debug/pprof/trace?seconds=5 > trace.out
$ go tool trace trace.out

# Create a heap dump and analyze it locally
$ curl http://localhost:6060/debug/pprof/heap > heap.out
$ go tool pprof -runtime --inuse_space [your-binary-here] heap.out

srv := &http.Server{
    WriteTimeout: 30 * time.Second,
}

sources

Alec Holmes - always be closing

gcvis

To use gcvis to profile our applications we need to first install it:

go get -u github.com/davecheney/gcvis

Now update your docker-compose.yaml file, important is to set the GODEBUG environment variable:

// docker-compose.yaml
version: "3"
services:
  app:
    build: ./build-dir
  environment:
    - GODEBUG=gctrace=1

Once you’ve updated the docker-compose.yaml file, run your container with the command below, and gcvis will start a browser for you automatically.

$ docker-compose up app | gcvis

sources

go-torch

When using go-torch, first update your code to enable the pprof endpoints. Reference the pprof section on how to do that. Next, you need to get go-torch:

$ go get -u github.com/uber/go-torch
$ git clone git@github.com:brendangregg/FlameGraph.git
$ export PATH=$PATH:/path/to/FlameGraph

And you can run it as follows:

$ go-torch -u http://localhost:6060/

sources

Github - go-torch

How to restore a PostgreSQL database on Kubernetes

Sat, 06 Aug 2016 12:00:00 +0000

As I wanted to transfer the database from one kubernetes cluster to a new one, I had to find a way to do this. And with a bit of trial and error I succeeded. So for future reference, and the hope that this might help someone, I figured I would write it down.

First we will transfer the database to our file system:

# pod-name         name of the postgres pod
# postgres-user    database user that is able to access the database
# database-name    name of the database
kubectl exec [pod-name] -- \
    bash -c "pg_dump -U [postgres-user] [database-name]" > database.sql

This will download the specified database on your file system as database.sql. Now the following command will restore the database:

# pod-name         name of the postgres pod
# postgres-user    database user that is able to access the database
# database-name    name of the database
cat database.sql | kubectl exec -i [pod-name] -- \
    psql -U [postgres-user] -d [database-name]

Sources:

Create command line applications with a config file

Fri, 10 Jun 2016 12:00:00 +0000

In this tutorial we’ll be using engage to create a cli application by only using a json config file. Enage was created because at FuelUp we have several bash scripts placed in different folders in the project. I wanted a way to bundle them so that I didn’t have to remember the location of these scripts. Additionally when developing it isn’t uncommon to have to type several commands after another, and it would be nice to have them easy accessible. And I really wanted to have the command fuel up to start the platform from the commandline. :)

So, the solution was to create a cli application and I wanted to do this in Go. However, adding new commands to the application would require co-workers to know Go, update the code and recreate the binaries. I wanted something simpler, just a json file that was easy to edit and would reference the scripts and commands, that would make up the cli application.

This is where ’engage’ comes in, and in the following sections I’ll explain how to install, configure and use it. So that you’ll be able to create your own cli application.

In short, it’ll make this:

{
    "name": "riker",
    "usage": "our first engage app",
    "authors": [
        {
            "name": "William T. Riker",
            "email": "w.t.riker@starfleet.org"
        }
    ],
    "version": "1.0",
    "commands": [
        {
            "name": "say",
            "action": "echo",
            "usage": "use `say` to relay a message"
        }
    ]
}

Into this:

$ riker

NAME:
   riker - our first enage app

USAGE:
   riker [global options] command [command options] [arguments...]

VERSION:
   1.0

AUTHOR(S):
   William T. Riker 

COMMANDS:
    say use `say` to relay a message

GLOBAL OPTIONS:
   --help, -h           show help
   --version, -v        print the version

Installation

First make sure you have the Go Programming Language installed on your system. Then clone the ’engage’ project on your system.

$ git clone git@github.com:erroneousboat/engage.git

Now think of a name you want to give your new cli application. In this example we’ll use riker. Move to the project folder and execute the following command:

$ make APP_NAME=riker

This will place your new application in the bin/ folder. Now lets create our config file and some commands.

Configuration

Create a engage.json file next to your application and add the following lines to it:

{
    "name": "riker",
    "usage": "our first engage app",
    "authors": [
        {
            "name": "William T. Riker",
            "email": "w.t.riker@starfleet.org"
        }
    ],
    "version": "1.0",
    "commands": [
        {
            "name": "say",
            "action": "echo",
            "usage": "use `say` to relay a message"
        }
    ]
}

Now, let’s test it out. Direct yourself to the bin/ folder, which should now contain the riker binary and the engage.json file.

$ ./riker

NAME:
   riker - our first enage app

USAGE:
   riker [global options] command [command options] [arguments...]

VERSION:
   1.0

AUTHOR(S):
   William T. Riker 

COMMANDS:
    say use `say` to relay a message

GLOBAL OPTIONS:
   --help, -h           show help
   --version, -v        print the version

Now let’s try to use our say command.

$ ./riker say "What's a knock-out like you doing in a computer-generated gin joint like this?"
What's a knock-out like you doing in a computer-generated gin joint like this?

Awesome! Now we’ll create a command that will execute two scripts. We’ll create two simple scripts to show that we’re able to chain commands with the use of a semicolon ;.

Create the following files: script1.sh and script2.sh. And add the following lines:

# script1.sh
#!/bin/bash
echo to boldy go where

# script2.sh
#!/bin/bash
echo no man has gone before

Make them executable:

$ sudo chmod +x script1.sh script2.sh

Now update the engage.json:

{
    "name": "riker",
    "usage": "our first engage app",
    "authors": [
        {
            "name": "William T. Riker",
            "email": "w.t.riker@starfleet.org"
        }
    ],
    "version": "1.0",
    "commands": [
        {
            "name": "say",
            "action": "echo",
            "usage": "use `say` to relay a message"
        },
        {
            "name": "tagline",
            "action": "./script1.sh; ./script2.sh",
            "usage": "space the final frontier"
        }
    ]
}

This should give us the following:

$ ./riker tagline
to boldy go where
no man has gone before

We can also use commandline arguments with our cli app. Create a file script3.sh and the lines:

# script3.sh
ARG="$1"
echo Hello ${ARG}

Now update the engage.json:

{
    "name": "riker",
    "usage": "our first engage app",
    "authors": [
        {
            "name": "William T. Riker",
            "email": "w.t.riker@starfleet.org"
        }
    ],
    "version": "1.0",
    "commands": [
        {
            "name": "say",
            "action": "echo",
            "usage": "use `say` to relay a message"
        },
        {
            "name": "tagline",
            "action": "./script1.sh; ./script2.sh",
            "usage": "space the final frontier"
        },
        {
            "name": "greetings",
            "action": "./script3.sh",
            "usage": "use to greet someone"
        }
    ]
}

And, now you can do the following:

$ riker greetings World
Hello World

$ riker greetings Picard
Hello Picard

Usage

Now, that we have created our cli app it would be nice to have it run directly from the commandline, instead of referencing the location of the binary on the filesystem.

When you want to create a cli app for a project for instance, then you might want to place the cli app in that project together with its engage.json and create a symlink in one of the following locations.

Folder	Access
`$HOME/bin`	yourself only
`/usr/local/bin`	you and other local users
`/usr/local/sbin`	root only

And, to create the symlink:

$ ln -s ~/location/of/app $HOME/bin

Now you’ll be able to access your cli app directly from the commandline!

Conclusion

In this post I’ve shown you have you can create your own cli application with a config file. If you like to know more about ’engage’, have feedback/suggestions want to follow its development. Then visit the project page on github

How to enable true-color for neovim, tmux, and gnome-terminal

Sun, 29 May 2016 12:00:00 +0000

With the release of tmux version 2.2 we’re able to get true-color support inside of tmux. And because both neovim and gnome-terminal are also supporting this, we can have great looking interfaces and colorschemes inside of tmux.

However, there are some steps we need to take in order to make this work. So I created this blog post for future reference, and hopefully you might find it useful too.

gnome-terminal

So lets start with gnome-terminal. First, we need to make sure that gnome-terminal is version linked with libvte >= 0.36. In order to check this, run the following command, and if you have the correct version of gnome-terminal. The version should be greater than or equal to 0.36.

$ ldd /usr/lib/gnome-terminal/gnome-terminal-server | grep libvte

If this isn’t the case, we need to get gnome-terminal from the staging ppa and install the latest version.

$ sudo add-apt-repository ppa:gnome3-team/gnome3-staging
$ sudo apt-get update
$ sudo apt-get install gnome-terminal
# the following will remove the staging repository because we only want to
# install gnome-terminal
$ sudo add-apt-repository -r ppa:gnome3-team/gnome3-staging

You might need to restart your computer or you could restart the gnome-terminal-server. And, just to be sure, check if libvte is the correct version.

Now, let’s check if we can really get the true-color spectrum. Download the following script, and execute it.

# download script
$ wget https://raw.githubusercontent.com/robertknight/konsole/master/tests/color-spaces.pl

# make the script executable
$ sudo chmod +x color-spaces.pl

# run the script
$ ./color-spaces.pl

It should look something like the following:

Optional: support for italics

I wanted to be able to have support for italics in tmux, and this is how you do it. Create a file name xterm-256color-italic.terminfo with the following contents:

# A xterm-256color based TERMINFO that adds the escape sequences for italic.
xterm-256color-italic|xterm with 256 colors and italic,
        sitm=\E[3m, ritm=\E[23m,
                use=xterm-256color,

Then install it by doing the following:

$ tic xterm-256color-italic.terminfo

Now you need to make sure that the terminfo is used by both your shell and tmux. Make sure the following line is present in your .bashrc or .zshrc. Depending on which shell you’re using.

export TERM=xterm-256color-italic

In your .tmux.conf file add the following line:

set -g default-terminal "xterm-256color-italic"

Now you can check if it is set correctly by typing in the following command in both the gnome-terminal and tmux.

$ echo $TERM
xterm-256color-italic

Now we should be able to get italics, and to check this, type the following in both gnome-terminal and tmux. This should output the word italics.

echo `tput sitm`italics`tput ritm`

tmux

On to tmux. First we need to check if we have the correct version.

$ tmux -V
# it should be >= 2.2
tmux 2.2

If it isn’t the correct version, then remove tmux from you system and install it. Get the latest tar.gz file from the following page. Extract the contents and install it. Instructions are also available on the tmux github page.

# on ubuntu:
$ sudo apt-get install -y libevent-dev libncurses-dev build-essential

$ tar -xzvf tmux-2.2.tar.gz
$ cd tmux-2.2
$ ./configure && make
$ sudo make install

Now that we have the correct version we need to enable Tc terminal capability for the outer terminal (to which tmux is attached) by setting the terminal-overrides option in tmux.

First it will be reported as missing.

$ tmux info | grep Tc
Tc: missing

And by adding the following line to your .tmux.conf file (you might need to restart your terminal):

set-option -ga terminal-overrides ",xterm-256color:Tc"

# When you're using the italic setup use the following line
set-option -ga terminal-overrides ",xterm-256color-italic:Tc"

The flag will be set to true:

$ tmux
$ tmux info | grep Tc
197: Tc: (flag) true

Now check the true-color capabilities of tmux by executing the script we used with the gnome-terminal.

$ ./color-spaces.pl

neovim

To enable true color capabilities for neovim add the following line to your init.vim file (located at ~/.config/nvim/):

set termguicolors

Now you can choose a colorscheme that doesn’t force neovim to use 256 colors, or one that doesn’t contain the line if !has('gui_running'). When it doesn’t it will use the guifg and guibg colors from the colorscheme.

Sources: