Mastering Curl: Going from junior to senior level debugging skills

Mastering Curl: Going from junior to senior level debugging skills

Debugging is just the process of creating hypotheses about what is causing a bug and then testing those hypotheses. One of the biggest differences between junior and senior developers is that senior developers are much better debuggers. They have a better sense of where a bug may be coming from and they have better ideas about how to test their hypotheses.

If you are a full-stack engineer or a backend engineer, part of your job is to debug servers. In many cases, you cannot debug an API server unless you send a request to the server. There are many ways to do this: using fetch in a browser, using a third party application, or  using cURL. Many great engineers who I have met can use cURL with ease. Being able to use it is not required to be a good engineer, but if you can use it comfortably, it will definitely help you to level up your debugging skills. In this article, I will explain a practical guide on how to get started with cURL.

What’s wrong with fetch or Postman?

Some people might wonder why you need to use a command-line tool in 2020. I have used Postman before, and I agree that it is a great application. In particular, being able to save the history of your requests is a neat feature. However, there is a big limitation. You cannot easily use it inside a Kubernetes cluster, VM, or Docker container. You sometimes have to go inside another machine and send requests from the machine. You cannot easily use fetch or a third-party application in this situation. On the other hand, you can install cURL with one command and start using it right away. cURL also supports many different protocols such as FTP and SMTP along with HTTP/HTTPS protocols. If you need to learn just one approach, then it is better to learn the one that you can use in any situation.

Utilize a browser’s network tab

If you are new to cURL and want to read the documentation, you can just type man curl on your command line. However, if you are not a big fan of reading documentation and prefer to actually start using a tool right away, you can start using cURL quickly by utilizing a browser:

  1. Go to a page you want to debug
  2. Open up the browser’s developer tools
  3. Go to the network tab
  4. Hover over a network request you want to get cUrl command for
  5. Right-click on the network and select “Copy as cURL”
Browser's network tab

Once you copy the request as cURL, then paste that in your command line. You will see something similar to the following image.

Copied cURL command

If you hit the Enter, it will send a request. If you look at the command closely, you will find that each line is followed by -H, which represents a header property. When you are debugging, you don’t need most of the header properties except for a few of them that I will talk more about later. In this case, you can simplify your cURL command to curl 'https://www.google.com/' and it works perfectly. Here  is a general tip for beginners when you want to use a cURL command and don’t know it exactly: Copy a request as a cURL command from a browser and simplify the command by removing unnecessary header properties.

Basic usage of cURL

Now you know how to get started with cURL, so let’s look at the basics of cURL so that you don’t have to copy a request from a browser every time.

GET request

The first use case is sending a GET request, which is super simple.

GET request

POST, PUT, and PATCH request

Now let’s take a look at the case where you send a request with a body massage. There are multiple ways to do it, and how a server is implemented determines which method you can use.

  1. When a server accepts application/json

If a server accepts JSON format, you can send JSON data by setting the header property Content-Type to be application/json.

POST request with application/json header

2. When a server accepts application/x-www-form-urlencoded

Another popular way to send a request with a body is to use x-www-form-urlencoded format. Data you want to send will be transferred with the format of key1=value1&key2=value2.

POST request with application/x-www-form-urlencoded header

3. When a server accepts multipart/form-data

When you send a request using the above 2 methods, you can only send one data type at once. However, a server can accept multiple data types. This is typically the case when you can upload files to the server. When you use multipart/form-data, you specify the data using the -F flag. In order to upload a file, you need to add @ at the beginning of the file path.

POST request with multipart/form-data header

There are more content types you can specify, but knowing these 3 will cover most use cases. Sometimes the body message is way too long to type on the command line. In this situation, you can let a server read data from a file. I will talk more about this  later.

Authentication

Another common use case is to send authorization information. Many services have an authentication system that requires you to be authorized to make a certain request. When you make a request from a browser, your authorization information is usually stored as a cookie and attached to requests. However, if you make a request outside of a browser, you need to attach authorization information to a request. What method works depends on how an authentication system is implemented on a server.

Basic Authentication

Basic authentications are one of the most popular  authentication systems. They send user credentials encoded in base64. A request has an authorization header that contains encoded values of the username and password. When you use cURL, you don’t need to encode the user credentials on your own because curl will do it for you if you include the -u flag like the following.

POST request with a basic authentication

Bearer Authentication

Bearer authentications are mainly used for OAuth 2.0, which are industry-standard for applications that send requests on behalf of a user. An example of OAuth 2.0 is a GitHub app that sends requests to the GitHub API server when a user performs a specific action. Bearer tokens are usually generated by a server in response to  user authentication. Once you get a bearer token, you can send a cURL request like the following.

POST request with a bearer token

If you cannot authenticate correctly using the methods above, this page explains more about authentication using cURL. You might need to pass optional flags as described on the page.

Advanced usage

Now you know how to send requests using cURL. Once you get used to it, you can write up a command to send a request really quickly. However, in reality, you need to know a little bit more about cURL to debug like a pro. Here are some of the advanced usages that are really helpful when I debug a server.

Avoid SSL certificate error

When you send a cURL request using HTTPS, it performs SSL certificate verification. However, if your server does not have a certificate signed by the CA, then your cURL command will fail. This often happens when you send requests to a server with a self-signed SSL. If you want to know the details about SSL, CloudFlare has a good article about it. You can disable certificate verification by passing a -k flag.

cURL command without SSL certificate verification

Write the content of the body message in a file, and pass the file

Your body message can be too big to write all of it in the command line. You might want to save the content for  debugging purposes. In these situations, You can write a body message in a file and send the contents of the file as a body message. There are two slightly different ways to do it: The one on the top would strip all newlines in a file, and the one below does not. Choose an appropriate method depending on your use case.

Read a body message from a file

Stress-test a server


cURL can be also used to stress-test your server. I have encountered a bug where a program worked fine when there were few requests to a server, but it broke when there were loads of requests. cURL is really handy in this situation. You can use [] to create dynamic URLs, and mimic the stress test. The following program sends the request 1000 times.

Stress-test a server

There is a lot more you can do with cURL, but now you probably know enough to explore cURL on your own! If you want to learn more about cURL, just type man curl in your command line. Mastering cURL will help you to debug more quickly.

If you liked this post, we recommend you subscribe to the Antcode blog (see below) and follow us on Twitter at @o_nagen and @mkinoshita12.

If you want to get reviews on your code and learn from reviews on other people’s code, we recommend that you check out what we’re building at Antcode and join our code review Slack group.

Happy coding!



Show Comments