Response time logging and filtering #433

denandz · 2021-04-23T08:54:53Z

Description

This pull request implements response time logging and filtering. Timing is performed using Golang's HTTP Tracing, so the logged time is the duration between the request being fully written (so we avoid logging overhead from DNS and TLS connection setup) and the first byte of the response being received.

Matchers and filters have been implemented with -mt and -ft. This is particularly useful when fuzzing things and looking for time discrepancies. The idea is to let the user tweak these parameters and see responses that fall outside of the expected response time.

Here is an example of using this to find blind sqli in the OWASP DVWA (using the SQLI Blind fuzzing list from Seclists)

$ ./ffuf -timeout 30 -t 1 -w ./urlsqli -u "http://172.17.0.1.nip.io:8088/vulnerabilities/sqli_blind/?id=FUZZ&Submit=Submit" -H 'Cookie: PHPSESSID=cd02gr40qdcrifs0g52paofqv1; security=low' -mc all -ft "<1000"

        /'___\  /'___\           /'___\       
       /\ \__/ /\ \__/  __  __  /\ \__/       
       \ \ ,__\\ \ ,__\/\ \/\ \ \ \ ,__\      
        \ \ \_/ \ \ \_/\ \ \_\ \ \ \ \_/      
         \ \_\   \ \_\  \ \____/  \ \_\       
          \/_/    \/_/   \/___/    \/_/       

       v1.3.0-dev
________________________________________________

 :: Method           : GET
 :: URL              : http://172.17.0.1.nip.io:8088/vulnerabilities/sqli_blind/?id=FUZZ&Submit=Submit
 :: Wordlist         : FUZZ: ./urlsqli
 :: Header           : Cookie: PHPSESSID=cd02gr40qdcrifs0g52paofqv1; security=low
 :: Follow redirects : false
 :: Calibration      : false
 :: Timeout          : 30
 :: Threads          : 1
 :: Matcher          : Response status: all
 :: Filter           : Response time: <1000
________________________________________________

%27+or+sleep%282%29%23  [Status: 404, Size: 4567, Words: 179, Lines: 111, Duration: 10.004019898s]
%27+or+sleep%282%29%3D%27 [Status: 200, Size: 4561, Words: 178, Lines: 111, Duration: 10.011460493s]
%27+or+benchmark%2810000000%2CMD5%281%29%29%23 [Status: 404, Size: 4567, Words: 179, Lines: 111, Duration: 6.763547328s]
:: Progress: [42/42] :: Job [1/1] :: 754 req/sec :: Duration: [0:00:26] :: Errors: 0 ::

And another example of finding SSRF based on timing discrepancy (using the https://owasp-skf.gitbook.io/asvs-write-ups/kbid-262-server-side-request-forgery container as the target)

$ ./ffuf -w fuzzlist -t 10 -timeout 30 -u 'http://127.0.0.1:5000/check_existence' -request request -mc all -ft '<2000'

        /'___\  /'___\           /'___\       
       /\ \__/ /\ \__/  __  __  /\ \__/       
       \ \ ,__\\ \ ,__\/\ \/\ \ \ \ ,__\      
        \ \ \_/ \ \ \_/\ \ \_\ \ \ \ \_/      
         \ \_\   \ \_\  \ \____/  \ \_\       
          \/_/    \/_/   \/___/    \/_/       

       v1.3.0-dev
________________________________________________

 :: Method           : POST
 :: URL              : http://127.0.0.1:5000/check_existence
 :: Wordlist         : FUZZ: fuzzlist
 :: Header           : Host: 127.0.0.1:5000
 :: Header           : Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
 :: Header           : Dnt: 1
 :: Header           : Accept-Encoding: deflate, gzip
 :: Header           : Accept-Language: en-US,en;q=0.9
 :: Header           : Referer: http://127.0.0.1:5000/check_existence
 :: Header           : Sec-Fetch-Site: same-origin
 :: Header           : Origin: http://127.0.0.1:5000
 :: Header           : Proxy-Connection: Keep-Alive
 :: Header           : Upgrade-Insecure-Requests: 1
 :: Header           : User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.72 Safari/537.36
 :: Header           : Cache-Control: max-age=0
 :: Header           : Connection: keep-alive
 :: Header           : Content-Type: multipart/form-data; boundary=----WebKitFormBoundaryBteNwndvo1K1kAc3
 :: Header           : Sec-Ch-Ua: " Not A;Brand";v="99", "Chromium";v="90", "Google Chrome";v="90"
 :: Header           : Sec-Ch-Ua-Mobile: ?0
 :: Header           : Sec-Fetch-Dest: document
 :: Header           : Sec-Fetch-Mode: navigate
 :: Header           : Sec-Fetch-User: ?1
 :: Data             : ------WebKitFormBoundaryBteNwndvo1K1kAc3
Content-Disposition: form-data; name="url"

FUZZ
------WebKitFormBoundaryBteNwndvo1K1kAc3--

 :: Follow redirects : false
 :: Calibration      : false
 :: Timeout          : 30
 :: Threads          : 10
 :: Matcher          : Response status: all
 :: Filter           : Response time: <2000
________________________________________________

https://httpstat.us/200?sleep=5000 [Status: 200, Size: 4178, Words: 1185, Lines: 95, Duration: 2.323751622s]
http://www.google.com   [Status: 200, Size: 4162, Words: 1183, Lines: 95, Duration: 2.49734596s]
https://www.google.com/  [Status: 200, Size: 4162, Words: 1183, Lines: 95, Duration: 2.67432851s]
http://www.google.com:80/search?q=wingus [Status: 200, Size: 4162, Words: 1183, Lines: 95, Duration: 3.294715376s]
https://www.google.com/search?q=dingus [Status: 200, Size: 4162, Words: 1183, Lines: 95, Duration: 3.702227468s]

Fixes: #196

denandz · 2021-04-23T09:33:43Z

One bazillion decimal points on stdout output seemed kind of silly. Latest commit rounds stdout output to nearest millisecond. This is not granular enough for some local-host based services but I figure that's kind of an edge case that user's can fall back to file output on.

Here's what the stdout looks like now:

https://httpstat.us/200?sleep=5000 [Status: 200, Size: 4178, Words: 1185, Lines: 95, Duration: 2.84s]
http://www.google.com   [Status: 200, Size: 4162, Words: 1183, Lines: 95, Duration: 3.009s]
https://www.google.com/ [Status: 200, Size: 4162, Words: 1183, Lines: 95, Duration: 3.379s]
http://www.google.com:80/search?q=wingus [Status: 200, Size: 4162, Words: 1183, Lines: 95, Duration: 4.16s]
https://www.google.com/search?q=dingus [Status: 200, Size: 4162, Words: 1183, Lines: 95, Duration: 4.775s]

joohoi

Thanks for the PR! This is cool!

I see that you changed up the output (since the last comment) to use milliseconds throughout. I think this is perfect.

The code in general looks very good (thanks for writing a few tests too!), there's just a small addition I'd love to see: adding the template to the ffufrc.example file.

Something I also noticed: sometimes with for example -ft '>30' the responses show durations of 31ms, but this must be related to the rounding in the output, but I'm a bit confused. I think the Time.Milliseconds() and Duration.Round... have a bit of discrepancy there. I don't think this is critical to change, but if you have a simple way of doing it, it'd be great :)

denandz · 2021-05-14T13:48:20Z

Hmm, seems like there must be some discrepancy between Milliseconds() and Round() like you mention. I've moved the stdout output to use Milliseconds() to match the filter.

I'm not sure what you mean by adding the template to the ffufrc.example file. I've added in the Time fields under filter and matcher. Do you want an example value like '>100' included in ffufrc.example?

denandz · 2021-05-14T13:57:07Z

Ahh that's right, I used Round() so we get pretty output with %v. At this point, I'll take a rounding error on the time output in exchange for prettier stdout output. I'd rather have 20.21s than 20210ms. Have changed it back to Round()

joohoi · 2021-05-16T20:36:14Z

Thanks for the update. I'd prefer using a well defined, static output format for the duration, and in this case the best fit would be milliseconds.

Currently, durations under a second get printed in ms, and durations over a second in seconds:

slow                    [Status: 200, Size: 185, Words: 6, Lines: 8, Duration: 2.494s]
still_slo               [Status: 200, Size: 185, Words: 6, Lines: 8, Duration: 1.211s]
under_second            [Status: 200, Size: 185, Words: 6, Lines: 8, Duration: 713ms]

denandz · 2021-05-16T21:01:54Z

Fair enough, have switched it to always print milliseconds.

||/usr/bin/id|          [Status: 200, Size: 4176, Words: 1187, Lines: 95, Duration: 2ms]
}                       [Status: 200, Size: 4176, Words: 1187, Lines: 95, Duration: 2ms]
https://www.google.com  [Status: 200, Size: 4162, Words: 1183, Lines: 95, Duration: 1614ms]

joohoi

LGTM, merging! Thanks again for the high quality pull request.

* Added response time reporting and filtering * Update to use the http config context * Added changelog and contributor info * Round time output in stdout to nearest millisecond * Change stdout duration rounding to use Milliseconds() * Go back to Round() for timing output * Changed stdout to display millisecond durations Co-authored-by: Joona Hoikkala <joohoi@users.noreply.github.com>

denandz added 4 commits April 23, 2021 20:32

Added response time reporting and filtering

aabc70b

Update to use the http config context

01a413b

Added changelog and contributor info

274635c

Round time output in stdout to nearest millisecond

b14a99d

joohoi requested changes May 13, 2021

View reviewed changes

Change stdout duration rounding to use Milliseconds()

e7e4f31

Go back to Round() for timing output

60db88a

denandz requested a review from joohoi May 14, 2021 14:00

Changed stdout to display millisecond durations

d8e27a2

Merge branch 'master' into response-time-filtering

0f8f9c3

joohoi approved these changes May 16, 2021

View reviewed changes

joohoi merged commit 965f282 into ffuf:master May 16, 2021

This was referenced May 17, 2021

[WIP] Adds request timings #316

Closed

Record request durations #317

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Response time logging and filtering #433

Response time logging and filtering #433

Uh oh!

denandz commented Apr 23, 2021 •

edited

Loading

Uh oh!

denandz commented Apr 23, 2021

Uh oh!

joohoi left a comment

Uh oh!

denandz commented May 14, 2021 •

edited

Loading

Uh oh!

denandz commented May 14, 2021

Uh oh!

joohoi commented May 16, 2021

Uh oh!

denandz commented May 16, 2021 •

edited

Loading

Uh oh!

joohoi left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Response time logging and filtering #433

Response time logging and filtering #433

Uh oh!

Conversation

denandz commented Apr 23, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Uh oh!

denandz commented Apr 23, 2021

Uh oh!

joohoi left a comment

Choose a reason for hiding this comment

Uh oh!

denandz commented May 14, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

denandz commented May 14, 2021

Uh oh!

joohoi commented May 16, 2021

Uh oh!

denandz commented May 16, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

joohoi left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

denandz commented Apr 23, 2021 •

edited

Loading

denandz commented May 14, 2021 •

edited

Loading

denandz commented May 16, 2021 •

edited

Loading