Python Requests - How to Send POST Requests | ScrapeOps (2024)

import requests

response = requests.post("https://httpbin.org/post", 
 data={"key": "value"},
 headers={"Content-Type": "application/json"},
)

print(response.json())

In this guide for The Python Web Scraping Playbook, we will look at how to make POST requests with the Python Requests library.

In this guide we will walk you through the most common ways of sending POST requests with Python Requests:

• POST JSON Data Using Python Requests
• POST Form Data Using Python Requests
• Configuring Data Types
• Using POST Requests With Sessions

Let's begin...

First, let's quickly go over some the very basics.

POST JSON Data Using Python Requests​

A common scenario for using POST requests is to send JSON data to an API endpoint, etc. Doing this with Python requests is very simple.

We simply just need to add the data to the request using the json parameter of the POST request:

import requests

url = 'https://example.com/api'
data = {'key': 'value'}

# Send POST request with JSON data using the json parameter
response = requests.post(url, json=data)

# Print the response
print(response.json())

The requests library will automatically encode the data as JSON and set the Content-Type header to application/json.

This approach can be simpler and more concise than manually encoding the data and setting the headers. Additionally, it may offer some performance benefits, as the Requests library can use a more efficient encoding method for JSON data.

POST Form Data Using Python Requests​

Another common use case for using POST requests is to send form data to an endpoint.

We simply just need to add the data to the request using the data parameter of the POST request:

import requests

url = 'https://example.com/api'
data = {'key': 'value'}

# Send POST request with FORM data using the data parameter
response = requests.post(url, data=data)

# Print the response
print(response.text)

The requests library will automatically encode the data as JSON and set the Content-Type header to application/x-www-form-urlencoded so you don't have to set any headers.

Configuring Data Types​

As we've seen above when you use the data or json parameter to send data with the POST request is defaults the Content-Type header to either application/json or application/x-www-form-urlencoded.

However, if you would like to override this or send data with another Content-Type then you can do so by just adding the Content-Type header to the POST request.

In the following example, we will send JSON data using the data parameter instead of the json parameter as we did previously.

import requests
import json

url = 'https://example.com/api'
data = {'key': 'value'}

# Convert data to JSON format
json_data = json.dumps(data)

# Set the Content-Type header to application/json
headers = {'Content-Type': 'application/json'}

# Send POST request with JSON data
response = requests.post(url, data=json_data, headers=headers)

# Print the response
print(response.json())

Using POST Requests With Sessions​

You can also use Python Request's Session functionality to send POST requests. Using sessions can be useful if you need to send multiple requests to the same server, as sessions can keep track of cookies and other stateful information across requests.

Here's an example of how to make a POST request with JSON data using a session in Python Requests:

import requests

url = 'https://example.com/api'
data = {'key': 'value'}

# Create a session object
session = requests.Session()

# Set the Content-Type header to application/json for all requests in the session
session.headers.update({'Content-Type': 'application/json'})

# Send a POST request with JSON data using the session object
response = session.post(url, json=data)

# Print the response
print(response.json())

In this example, we create a Session object using requests.Session(). We then update the headers attribute of the session object to set the Content-Type header to application/json. This will ensure that all requests sent through the session will have this header.

Finally, we use the session.post() method to send a POST request with JSON data, passing the URL and the data dictionary as arguments. The session will automatically encode the data as JSON and set the Content-Type header to application/json.

The session.post() method returns a Response object, just like the regular requests.post() method.

More Web Scraping Tutorials​

So that's how you can send POST requests using Python Requests.

If you would like to learn more about Web Scraping, then be sure to check out The Web Scraping Playbook.

Or check out one of our more in-depth guides:

• How to Scrape The Web Without Getting Blocked Guide
• The State of Web Scraping 2020
• The Ethics of Web Scraping