Peeking into Requests

Here, we briefly look at a few command-line tools that can be used to study whether a server responds to requests and to study the content of requests (and responses).

Ping

First, the ping command is used to study the response time of a service -- it is not really a tool for making HTTP requests, but a tool for checking whether an address responds to requests. It is used in conjunction with a server name (or an IP address).

In the following example, we have "pinged" the address www.aalto.fi. The ping time corresponds to the time that it took to send a small packet to the server and back from the server. In this case, the first 64 byte packet took 23.7 milliseconds, while the second one took 50.2 milliseconds.

The below example also shows how commands given in terminal are presented in the materials. The lines that have a terminal $ prefix demonstrate given commands, while the lines that do not have that prefix demonstrate terminal output.

ping www.aalto.fi
PING www.aalto.fi.cdn.cloudflare.net (104.17.221.22) 56(84) bytes of data.
64 bytes from 104.17.221.22 (104.17.221.22): icmp_seq=1 ttl=58 time=23.7 ms
64 bytes from 104.17.221.22 (104.17.221.22): icmp_seq=2 ttl=58 time=50.2 ms
(and so on)

Stopping ping (and, more generally processed that are run in terminal) is done by the key combination ctrl+c. In addition to checking the time that a server took to respond, ping can be used to check if a server is responding. It is, however, possible to also configure the server so that responses are not sent to ping requests.

cURL

Client URL (cURL) is a program that can be used to make HTTP requests to servers. By default, curl is used to make a HTTP GET request to a server. The example below shows two simple GET requests to the address https://aged-cherry-5206.fly.dev/sum?one=19&two=23.

curl "https://aged-cherry-5206.fly.dev/sum?one=19&two=23"
Sum: 42%
curl "https://aged-cherry-5206.fly.dev/sum?one=19&two=23"
Sum: 42%

The server responds with the message Sum: 42. There is no line break at the end of the message, which is -- in this terminal -- shown with a percentage sign %.

The program can be also used to describe the request in more detail. If curl is used with the --verbose flag (or -v), the output contains also request and response details, including the headers.

$ curl -v "https://aged-cherry-5206.fly.dev/sum?one=19&two=23"
*   Trying (address)...
* Connected to aged-cherry-5206.fly.dev (address) port 443 (#0)
* (connection and certificate information)
> GET /sum?one=19&two=23 HTTP/1.1
> Host: aged-cherry-5206.fly.dev
> User-Agent: curl/7.58.0
> Accept: */*
... more content
< HTTP/2 200
< content-type: text/plain;charset=UTF-8
.. more headers
<
* Connection #0 to host aged-cherry-5206.fly.dev left intact
Sum: 42%

The above output shows both the HTTP request that curl made and the HTTP response sent from the server.

Developer tools

Browsers come with developer tools that provide means for studying requests and responses. In Google Chrome, Developer Tools can be accessed by clicking the context menu, selecting "More tools", and selecting "Developer tools". Alternatively, one can use the key combination Ctrl+Shift+I.

In Chrome, Developer Tools look as follows. In the figure below, the Console tab has been opened.

Clicking the Network-tab opens up a Network monitoring facility, which is used to record and display the requests that the browser makes.

Now, when we reload the current page that is open in the browser -- here https://aged-cherry-5206.fly.dev/sum?one=19&two=23 -- we see that the request is shown in the network tab. In addition to the request to the actual address, denoted as sum?one=19&two=23, the browser also looks for an icon that could be used to represent the page in the browser (favicon.ico). In practice, unbeknownst to us, the browser actually made two requests when we loaded the page.

When we click on the request sum?one=19&two=23, we can see the details of the request and response. In the example below, we have first selected the request, and then opened the Headers tab to see the HTTP request and response headers.

Above, we can see that the request is a HTTP GET request, and that the browser has received a HTTP response (HTTP status code 200) from the server. Additional information about the request is also available, including the query parameters.

Chrome Developer Tools also provide the opportunity for copying a request as a curl command that can be then executed in the terminal. This is done by clicking the request, selecting "Copy", and then selecting "Copy as cURL". This will copy the curl command to the clipboard, where it can then be pasted from. The command will also include all the headers sent to the server during the request. As an example, the following snippet has been copied from Chrome Devtools.

curl 'https://aged-cherry-5206.fly.dev/sum?one=19&two=23' \
  -H 'Connection: keep-alive' \
  -H 'Cache-Control: max-age=0' \
  -H 'sec-ch-ua: "Google Chrome";v="87", " Not;A Brand";v="99", "Chromium";v="87"' \
  -H 'sec-ch-ua-mobile: ?0' \
  -H 'DNT: 1' \
  -H 'Upgrade-Insecure-Requests: 1' \
  -H 'User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.141 Safari/537.36' \
  -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9' \
  -H 'Sec-Fetch-Site: none' \
  -H 'Sec-Fetch-Mode: navigate' \
  -H 'Sec-Fetch-User: ?1' \
  -H 'Sec-Fetch-Dest: document' \
  -H 'Accept-Language: en-US,en;q=0.9' \
  --compressed