RedTeam Pentesting GmbH - Blog

2 December 2020

Introducing monsoon ‒ a Lean and Versatile HTTP Enumerator

We recently released our first open-source project, monsoon. monsoon is a so-called command-line HTTP enumerator: A tool that iterates over a list of values, for example a word list or a range of integers, and sends one HTTP request per item towards a given server. The target server, path and HTTP headers can be configured on the command line. There, one can replace parts of the HTTP request with the placeholder FUZZ. All its occurrences are replaced with the current item during each iteration. Optionally, monsoon can filter or even parse the HTTP response.

monsoon can be very helpful during penetration testing and in this blog post we would like to motivate its development and introduce some common scenarios that can be tackled using monsoon.

Motivation

There are quite a few similar software projects on the internet, namely DirBuster, gobuster, wfuzz or ffuf. We have used some of them during penetration tests and while these might be great tools for certain jobs, we felt that none of them reflect our penetration testing workflow very well.

Penetration tests at RedTeam Pentesting mostly consist of a manual, creative search for vulnerabilities, since we, as well as our customers, feel that this is the most efficient way to deliver high-quality results. However, this does not imply that we do not use automated tools for highly repetitive tasks, such as port scanning or brute-force attacks. In order to integrate corresponding software into our creative workflow, tools have to be very versatile. The most versatile tool sets are made up of multiple single-purpose, easy to use tools working on open/standardised data formats. Great examples can be found in the basic Unix tool set: cat, grep, cut, head, tail, … all of which act on simple text streams and strive to fulfill a single purpose.

Thus, our goal when developing monsoon was to stay close to the minimally necessary feature set for an HTTP enumerator and to add new features only if they are actually relevant for our penetration testing workflow. The first commits to the monsoon repository are from 2017; since then we have used and refined monsoon in many many penetration tests.

File and Directory Discovery

The most common usage scenario for monsoon is the discovery of files or folders which are served by a web server, but are not directly linked from any previously known page. This might for example include administration panels, different applications or remnants of development, such as debugging endpoints or even source code. If these resources are not protected by separate authentication, any attacker who knows the corresponding URL can interact with them.

This is where monsoon comes in: monsoon allows us to query many URLs in sequence, with the goal of guessing URLs to aforementioned hidden resources. In this process, our success rate depends on the speed and quality of guesses. A good starting point for guessing hidden file and directory names is the SecLists project by Daniel Miessler. SecLists is a large collection of word lists for many different purposes. The files that we are interested in during this attack phase are contained in the folder Discovery/Web-Content. Among others, the folder contains the RAFT word lists originally compiled by FishNet Security, such as raft-small-files.txt and raft-small-directories.txt.

The following screen cast shows an invocation of monsoon on a sample application with the goal of discovering hidden subdirectories below the path /app:

The HTTP responses are summarised in the result table. With the command-line option --hide-status set to 404, we have hidden 996 responses for which the web server has responded with 404 Not Found. We’re rather interested in the four results for which the web server has responded with status code 302, indicating a redirection. It’s a very common behavior of web servers to redirect requests from /foo to /foo/ if the directory foo exists. In this case there are several interesting directories, such as internal and phpMyAdmin which we subsequently would inspect manually.

Enumeration

Another common scenario is the exploitation of a so-called insecure direct object reference (IDOR) vulnerability. Such vulnerabilities exist whenever access to a resource only depends on a predictable reference, such as a numerical ID, a known algorithm generating a serial number or otherwise obtained UUIDs. Imagine a web application where data for your user profile is fetched from the endpoint /api.php?userid=11:

$ curl -s 'http://app.example.com/api.php?userid=11' | jq
{
  "Id": 11,
  "FirstName": "RedTeam",
  "LastName": "Pentesting",
  "UserName": "rtpt",
  "Email": "jobs@redteam-pentesting.de"
}

What happens if /api.php?userid=10 is requested? In this example, we assume that the analysed web application does not intend to offer the functionality to display other users personal information and should therefore deny the access to it. Despite the intention of the web application, user data is returned which indicates an IDOR vulnerability:

$ curl -s 'http://app.example.com/api.php?userid=10' | jq
{
  "Id": 10,
  "FirstName": "Ben",
  "LastName": "Grimm",
  "UserName": "b.grimm",
  "Email": "b.grimm@example.com"
}

We often use monsoon in this scenario in order to demonstrate the impact of such a vulnerability to our customers. With the -r/--range option, monsoon iterates over ascending integer numbers instead of a word list. This is demonstrated in the following screen cast:

That was fast! Note that the results appear out of order because monsoon by default sends five concurrent requests. Furthermore, we have parsed the JSON response with the command-line JSON parser jq. Its output (the fictional person’s name) is displayed in the result table.

Additionally, monsoon can perform basic content extraction without an external process. For this purpose, one can provide a regular expression to the --extract parameter. monsoon will try to match the regular expression to the HTTP response body. If one or more matches have been found, the content of the first capture group of each match will be displayed in the results table. This can be useful if, for example, you want to extract content from responses containing HTML. Imagine the following endpoint:

$ curl -s 'http://app.example.com/dashboard.php?userid=11'
<!DOCTYPE html>
<html>
    <head>
        <meta charset="utf-8">
        <title>Profile of RedTeam Pentesting</title>
    </head>
    <body>
        <h1>Profile of RedTeam Pentesting</h1>
        <table>
            <tr><th>Key</th><th>Value</th></tr>
            <tr><td>First Name</td><td>RedTeam</td></tr>
            <tr><td>Last Name</td><td>Pentesting</td></tr>
            <tr><td>Email</td><td>jobs@redteam-pentesting.de</td></tr>
        </table>
    </body>
</html>

Here, we can use the regular expression <title>Profile of ([^<]+)</title> to extract a name from the title. Watch to see monsoon quickly extract names from our sample application:

Brute-Force

HTTP requests are not always as straight-forward as the ones shown before. You might encounter requests which require certain HTTP headers or some complex body structure. In those cases it might be easier to start from a known HTTP request, one that has for example been captured with an HTTP attack proxy like ZAP. One example could be the following login request to a our sample application:

POST /login.php HTTP/1.1
Host: app.example.com
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Firefox/68.0
Content-Type: application/x-www-form-urlencoded
Accept: */*
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br
Referer: http://127.0.0.1:8080/
Connection: keep-alive
Pragma: no-cache
Cache-Control: no-cache

username=admin&password=FUZZ

We save this request as a file with the name login-request.txt. As the request body shows, we would like to authenticate with the username admin, but we don’t know the password (yet). Therefore, we substitute the password parameter with the placeholder FUZZ. Now, we can specify the file name login-request.txt as the value of the --template-file parameter. For the URL we only have to specify the correct server, as this cannot be deduced from the template file. Note that some fields such as the Content-Length header will be automatically adjusted by monsoon.

Again, decent word lists to start from can be found in the SecLists project. This time, we will use a word list from the Passwords directory. Most files in this directory contain common passwords and default passwords. For this demonstration, we will use the word list xato-net-10-million-passwords-10000.txt which has been originally compiled by Mark Burnett (xato.net). The invocation with the aforementioned template file is as follows:

Et voilá, we discovered that the password is wonderful! 😁

Conclusion

monsoon is a very versatile HTTP enumerator suited for penetration testing of web applications. Its limited feature set helps the project stay lean while allowing for creative use. In the end, it’s a tool, developed to support your creative workflow.