Http Requests Cheatsheet

When working around APIs, whether it is building or playing with, you often end up making API requests over and over that all look alike, and yet there are some stuff that you can’t remember. Like how would you provide form data with cURL? Or basic authentication with requests? This article serves as a cheatsheet on all those common stuff.

Selecting the HTTP method

`requests` (Python)

3 possible ways:

Using the requests.request method:

resp = requests.request('GET', 'https://httpbin.org/get')
resp = requests.request('POST', 'https://httpbin.org/post')
resp = requests.request('PUT', 'https://httpbin.org/put')
resp = requests.request('PATCH', 'https://httpbin.org/patch')
resp = requests.request('DELETE', 'https://httpbin.org/delete')
resp = requests.request('HEAD', 'https://httpbin.org/get')

Using the requests.<method> methods:

resp = requests.get('https://httpbin.org/get')
resp = requests.post('https://httpbin.org/post')
resp = requests.put('https://httpbin.org/put')
resp = requests.patch('https://httpbin.org/patch')
resp = requests.delete('https://httpbin.org/delete')
resp = requests.head('https://httpbin.org/get')

Using a prepared request:

req = requests.Request('GET', 'https://httpbin.org/get')
r = req.prepare()
s = requests.Session()
s.send(r)

From requests.PreparedRequest documentation.

cURL

curl -X GET https://httpbin.org/get
curl -X POST https://httpbin.org/post
curl -X PUT https://httpbin.org/put
curl -X PATCH https://httpbin.org/patch
curl -X DELETE https://httpbin.org/delete

If the -X/--request is not provided, cURL will default to GET.

[doc of -X/--request option]

Note on the HEAD requests: to make a proper HEAD request (ie. requesting only the headers), the -X HEAD option will not work, you will need to use -I/--head instead:

> curl -X HEAD https://httpbin.org/get
Warning: Setting custom HTTP method to HEAD with -X/--request may not work the
Warning: way you want. Consider using -I/--head instead.
curl: (18) transfer closed with 255 bytes remaining to read
> curl -I https://httpbin.org/get
HTTP/2 200
date: Sun, 19 Dec 2021 11:03:31 GMT
content-type: application/json
content-length: 255
server: gunicorn/19.9.0
access-control-allow-origin: *
access-control-allow-credentials: true

(Tested on cURL 7.64.0 and 7.77.0)

Headers

`requests` (Python)

requests.get(
  'https://httpbin.org/headers',
  headers={
      'X-My-Header': 'Value',
  }
)

[doc of headers parameter]

cURL

curl -H 'X-My-Header: Value' https://httpbin.org/headers

[doc of -H/--headers option]

Authentication

HTTP Basic

`requests` (Python)

2 possible ways:

Using the auth option

requests.get(
  'https://httpbin.org/basic-auth/user/passwd',
  auth=('user', 'passwd')
)

Using the headers option

requests.get(
  'https://httpbin.org/basic-auth/user/passwd',
  headers={
      'Authorization': 'Basic dXNlcjpwYXNzd2QK',
  }
)

from base64 import b64encode
requests.get(
  'https://httpbin.org/basic-auth/user/passwd',
  headers={
      'Authorization': f'Basic {b64encode('user:passwd')}',
  }
)

Note on using this along with a .netrc file: Authorization headers that are provided this way will be overriden by credentials set in the .netrc file [doc].

As a workaround you can either temporarily comment the authentication part in your .netrc, set the NETRC env var to an empty file, or temporarily set requests.utils.NETRC_FILES = [] [source code] .

cURL

2 possible ways:

Using the -u/--user option

curl -u user:passwd https://httpbin.org/basic-auth/user/passwd

Few things to note when using this option:

cURL will split the username and password on the first colon. This means you can’t have one in the username, but you can in the password. If needed, use the solution below with the -H/--header option.
If only the username is provided, cURL will prompt for the password before sending the request

Using the -H/--header option

curl -H 'Authorization: Basic dXNlcjpwYXNzd2QK' https://httpbin.org/basic-auth/user/passwd
curl -H "Authorization: Basic $(openssl base64 <<< user:passwd)" https://httpbin.org/basic-auth/user/passwd

HTTP Bearer

`requests` (Python)

requests.get(
  'https://httpbin.org/bearer',
  headers={
      'Authorization': 'Basic dGhpc0lzQUJlYXJlclRva2VuCg==',
  }
)

cURL

curl -H 'Authorization: Bearer dGhpc0lzQUJlYXJlclRva2VuCg==' https://httpbin.org/bearer

Query parameters

`requests` (Python)

requests.get(
  'https://httpbin.org/anything',
  params={
      'arg1': 'val1',
      'arg2': 'val2',
  }
)

[doc of params parameter]

cURL

No useful options here unfortunately

curl https://httpbin.org/anything?arg1=val1&arg2=val2

Body parameters

`requests` (Python)

There are multiple ways to provide a body, each with their side consequences:

Providing a dict in the data parameter: this will set the Content-Type header to application/x-www-form-urlencoded, unless told otherwise by the headers parameter.

requests.post(
  "https://httpbin.org/post",
  data={'key1': 'value1', 'key2': 'value2'},
)

Providing a raw string in the data parameter: this will not set any value to the Content-Type header.

requests.post(
  "https://httpbin.org/post",
  data='theBodyOfMyRequest',
)

Providing a dict in the json parameter: this will dumps the provided payload as JSON and set the Content-Type header to application/json, unless told otherwise by the headers parameter. The following two request are equivalents:

import json
requests.post(
  "https://httpbin.org/post",
  data=json.dumps({'key1': 'value1 and value2', 'key2': 'value2'}),
  headers={'Content-Type': 'application/json'},
)
requests.post(
  "https://httpbin.org/post",
  json={'key1': 'value1 and value2', 'key2': 'value2'},
)

[doc]

cURL

cURL offers multiple options to send a request with a body, each with their side consequences as well.

Using the -F/--form option

curl -F key1=value1 -F key2=value2 https://httpbin.org/post

Few things to note when using this option:

it forces the method to POST, unless told otherwise with -X/--request option
it sets the Content-Type header to multipart/form-data, unless told otherwise aswell with -H/--header option

Using this option, it is possible to send files or read input from a file by prefixing the filename with @ or <, respectively. This means that if you need to have either a @ or < in the key and/or the value, you need to use the --form-string option.

[doc of -F/--form option]

[doc of --form-string option]

Using -d/--data option

curl -d key1=value1 -d key2=value2 https://httpbin.org/post

Few things to note when using this option:

it forces the method to POST, unless told otherwise with -X/--request option
it sets the Content-Type header to application/x-www-form-urlencoded, unless told otherwise aswell with -H/--header option

Using this option, it is possible to read input from a file by prefixing the filename with @. This means that if you need to have a @ in the key and/or the value you need to use the --data-raw option.

It is also possible to URL-encode the parameter using the --data-urlencode option.

[doc of -d/--data option]

[doc of --data-raw option]

[doc of --data-urlencode option]

Sending files

`requests` (Python)

requests.post(
  'https://httpbin.org/post',
  files={'file': ('index.html', open('index.html', 'rb'), 'text/html', {'Expires': '0'})}
)

[doc]

cURL

curl -F "file=@index.html" https://httpbin.org/post
curl -F "file=@index.html;type=text/html" https://httpbin.org/post

[doc of -F/--form option]

Disabling SSL verification

`requests` (Python)

resp = requests.get(
    'https://httpbin.org/get',
    verify=False,
)

[doc of verify parameter]

cURL

curl -k https://httpbin.org/get

[doc of -k/--insecure option]

3XX redirects

`requests` (Python)

requests default behavior is to follow redirect, unless making a HEAD request (see below for more informations). Use the parameter allow_redirects to disable it:

resp = requests.get(
    'https://httpbin.org/redirect-to?url=bastien-antoine.fr&status_code=302',
    allow_redirects=False,
)

Note on redirects with HEAD requests: when making HEAD requests, requests will not follow 3XX redirects in most case, depending on how the request is made:

How the request is made	Follow 3XX on default
`requests.head(...)`	No
`requests.Session(...).head(...)`	No
`requests.request('HEAD', ...)`	Yes

cURL

cURL’s default behavior is to not follow 3XX redirects. To enable this use the -L/--location option:

curl -L https://httpbin.org/redirect-to?url=bastien-antoine.fr&status_code=302

[doc of -L/--location option]

Logging

`requests` (Python)

import requests
import logging
import http.client

http.client.HTTPConnection.debuglevel = 1

logging.basicConfig()
logging.getLogger().setLevel(logging.DEBUG)
requests_log = logging.getLogger("requests.packages.urllib3")
requests_log.setLevel(logging.DEBUG)
requests_log.propagate = True

[source]

cURL

curl -v https://httpbin.org/get

Note that this option only shows requests headers and full response. If you want to see the whole process, including request body, checkout the --trace and --trace-ascii options.

[doc of -v/--verbose option]