[FEATURE] Add audit logging functionality #771

denandz · 2024-03-21T03:50:06Z

This PR adds an audit log which writes every request and response, along with the config, to a JSON lines formatted log file (https://jsonlines.org/). The audit log is not affected by filter logic, and can be used to log everything sent/received by FFUF regardless of what the user specifies as a matcher/filter.

The JSON lines format was chosen so requests/responses can be written as they're sent/received. This avoids memory pressure and ensures immediate audit logging that isn't lost if FFUF crashes.

The audit logging feature has the following use cases:

Audit logging when attacking sensitive infrastructure. Allows us to answer the question of "what requests exactly did you send when this service crashed", which prior to this pull request would have required piping all fuzz runs through a separate proxy for logging.
Hunting for one-time discrepancies, such as finding hidden functions/header-values in JIT compiled languages like dotnet by analysing subtle timing discrepancies associated with JIT compilation across multiple runs. A messed up matcher/filter requiring a re-run of a fuzzing session would miss this.
Performing statistical analysis and "machine-learning" based analysis on fuzz run output after-the-fact.

The audit logging output also enables future functionality where the audit log can be re-read by FFUF. Such as replaying the audit log to refine matcher/filters without sending all the fuzz-run requests again, and sending requests to replay proxies after the fact. This is going to require some more development and is still on my to-do list.

Fixes: #759

Example of the output below:

$ ./ffuf -w wlist -u http://127.0.0.1:8000/FUZZ -audit-log foo.json 

        /'___\  /'___\           /'___\       
       /\ \__/ /\ \__/  __  __  /\ \__/       
       \ \ ,__\\ \ ,__\/\ \/\ \ \ \ ,__\      
        \ \ \_/ \ \ \_/\ \ \_\ \ \ \ \_/      
         \ \_\   \ \_\  \ \____/  \ \_\       
          \/_/    \/_/   \/___/    \/_/       

       v2.1.0-dev
________________________________________________

 :: Method           : GET
 :: URL              : http://127.0.0.1:8000/FUZZ
 :: Wordlist         : FUZZ: /home/doi/go/src/github.com/ffuf/ffuf/wlist
 :: Follow redirects : false
 :: Calibration      : false
 :: Timeout          : 10
 :: Threads          : 40
 :: Matcher          : Response status: 200-299,301,302,307,401,403,405,500
________________________________________________

:: Progress: [4/4] :: Job [1/1] :: 0 req/sec :: Duration: [0:00:00] :: Errors: 0 ::

The filter didn't match on any responses; however, the foo.json audit log has complete requests/responses and config:

$ jq . < foo.json 
{
  "Type": "*ffuf.Config",
  "Data": {
    "auditlog": "foo.json",
    "autocalibration": false,
    "autocalibration_keyword": "FUZZ",
    "autocalibration_perhost": false,
    "autocalibration_strategies": [
      "basic"
    ],
    "autocalibration_strings": [],
    "colors": false,
    "cmdline": "./ffuf -w wlist -u http://127.0.0.1:8000/FUZZ -audit-log foo.json",
    "configfile": "",
    "postdata": "",
    "debuglog": "",
    "delay": {
      "value": "0.00"
    },
    "dirsearch_compatibility": false,
    "encoders": [],
    "extensions": [],
    "fmode": "or",
    "follow_redirects": false,
    "headers": {},
    "ignorebody": false,
    "ignore_wordlist_comments": false,
    "inputmode": "clusterbomb",
    "cmd_inputnum": 100,
    "inputproviders": [
      {
        "name": "wordlist",
        "keyword": "FUZZ",
        "value": "/home/doi/go/src/github.com/ffuf/ffuf/wlist",
        "encoders": "",
        "template": ""
      }
    ],
    "inputshell": "",
    "json": false,
    "matchers": null,
    "mmode": "or",
    "maxtime": 0,
    "maxtime_job": 0,
    "method": "GET",
    "noninteractive": false,
    "outputdirectory": "",
    "outputfile": "",
    "outputformat": "",
    "OutputSkipEmptyFile": false,
    "proxyurl": "",
    "quiet": false,
    "rate": 0,
    "raw": false,
    "recursion": false,
    "recursion_depth": 0,
    "recursion_strategy": "default",
    "replayproxyurl": "",
    "requestfile": "",
    "requestproto": "https",
    "scraperfile": "",
    "scrapers": "all",
    "sni": "",
    "stop_403": false,
    "stop_all": false,
    "stop_errors": false,
    "threads": 40,
    "timeout": 10,
    "url": "http://127.0.0.1:8000/FUZZ",
    "verbose": false,
    "wordlists": [
      "/home/doi/go/src/github.com/ffuf/ffuf/wlist"
    ],
    "http2": false,
    "client-cert": "",
    "client-key": ""
  }
}
{
  "Type": "*ffuf.Request",
  "Data": {
    "Method": "GET",
    "Host": "",
    "Url": "http://127.0.0.1:8000/asdva",
    "Headers": {},
    "Data": "",
    "Input": {
      "FFUFHASH": "MmJmMDgx",
      "FUZZ": "YXNkdmE="
    },
    "Position": 1,
    "Raw": "",
    "Timestamp": "2024-03-21T16:43:21.895628032+13:00"
  }
}
{
  "Type": "*ffuf.Request",
  "Data": {
    "Method": "GET",
   ...yoink...
  }
}
{
  "Type": "*ffuf.Request",
  "Data": {
    "Method": "GET",
...yoink...
  }
}
{
  "Type": "*ffuf.Request",
  "Data": {
    "Method": "GET",
    ...yoink...
  }
}
{
  "Type": "*ffuf.Response",
  "Data": {
    "StatusCode": 404,
    "Headers": {
      "Connection": [
        "close"
      ],
      "Content-Length": [
        "469"
      ],
      "Content-Type": [
        "text/html;charset=utf-8"
      ],
      "Date": [
        "Thu, 21 Mar 2024 03:43:21 GMT"
      ],
      "Server": [
        "SimpleHTTP/0.6 Python/3.9.2"
      ]
    },
    "Data": "PCFET...yoink...Cg==",
    "ContentLength": 469,
    "ContentWords": 96,
    "ContentLines": 15,
    "ContentType": "text/html;charset=utf-8",
    "Cancelled": false,
    "Request": {
      "Method": "GET",
      "Host": "127.0.0.1:8000",
      "Url": "http://127.0.0.1:8000/asdva",
      "Headers": {
        "User-Agent": "Fuzz Faster U Fool v2.1.0-dev"
      },
      "Data": "",
      "Input": {
        "FFUFHASH": "MmJmMDgx",
        "FUZZ": "YXNkdmE="
      },
      "Position": 1,
      "Raw": "",
      "Timestamp": "2024-03-21T16:43:21.897260281+13:00"
    },
    "Raw": "",
    "ResultFile": "",
    "ScraperData": {},
    "Duration": 3196397,
    "Timestamp": "2024-03-21T16:43:21.900456678+13:00"
  }
}
{
  "Type": "*ffuf.Response",
  "Data": {
    "StatusCode": 404,
 ...yoink...
  }
}
{
  "Type": "*ffuf.Response",
  "Data": {
    "StatusCode": 404,
 ...yoink...
  }
}
{
  "Type": "*ffuf.Response",
  "Data": {
    "StatusCode": 404,
...yoink...
  }
}

denandz · 2024-05-20T01:05:28Z

Yo @joohoi anything else you need to get this merged? There is some future functionality around statistical analysis of response data I'd like to work on but it's relying on this feature getting merged.

joohoi

Hi,
in general this looks good!

In order to reflect the reality better, there's couple of potential improvements to be made, lmk if you agree.

I think the spot where the request is written to audit log should be changed. ffuf adds for example User-Agent header later in the Execute function of the runner and that isn't caught by the current behavior.

Also, it would be good to save the raw requests (Request struct has a spot for the raw data, but it's currently only written if -od if defined), that behavior can be changed in following spots:

Request:
https://github.com/ffuf/ffuf/blob/master/pkg/runner/simple.go#L149-L151

Response:
https://github.com/ffuf/ffuf/blob/master/pkg/runner/simple.go#L171-L175

denandz · 2024-05-20T22:00:02Z

Oh nice! Good catch with the request data. I'll get those changes rolled in

denandz · 2024-05-28T10:55:35Z

Have tweaked the logic so we're logging the request object after any modifications are made by the runner. Have also tweaked the raw request/response logic to log that data.

How set are you on storing the raw request/response as a string? I can see the json getting messy when working with binary data in requests and responses. Might be an idea to move it to a byte[] and then it'll serialize out as base64.

denandz · 2024-06-24T01:42:24Z

Let me know if there's anything else I've missed that's stopping this from merging @joohoi :)

Auditlogging

jarodchase · 2024-09-20T17:03:04Z

Would love to see this! Would be very useful for providing full request/response examples in pentest reports.

Support any header case and multiple headers with the same name.

Merge payload-response delta feature

Add run summaries

This reverts commit ee8be89, reversing changes made to ed58fb6.

denandz · 2025-04-05T02:40:19Z

Apologies for the commit noise here. Wanted to merge some fixes into both this PR and my master branch.

The lastest set of fixes address an issue with err variable overriding and include an Error string object in the Request struct. This allows users to see what requests caused what errors (eg - timeouts) in the audit log.

joohoi · 2025-04-05T17:01:01Z

All good, apologies from my end; I have been insanely busy and as such, dropped the ball on everything not absolutely critical and urgent. But thanks to the notification from the aformentioned commit noise, I'm back on it :)

joohoi

LGTM, thanks for your patience and continued contributions. Highly appreciated <3

denandz · 2025-04-09T00:32:33Z

Thanks @joohoi ! I'm currently working on a handful of other features, if there's anything specific you'd like to see merged upstream from github.com/denandz/ffuf just let me know and I can work out the PRs :)

denandz added 5 commits March 19, 2024 21:51

Added audit log functionality

0f9a520

remove redundant comments

db5d6cb

Add tests for audit log writing

b9a0080

Fix comment, add error checking on config auditing

2149b24

Fix auditlog tests

8e8d8fc

denandz changed the title ~~Add audit logging functionality~~ [FEATURE] Add audit logging functionality Mar 21, 2024

Update CHANGELOG.md

8603c39

joohoi requested changes May 20, 2024

View reviewed changes

Re-order request logging and log raw request/responses when auditing

ecc4649

denandz mentioned this pull request Jun 24, 2024

Feature request : Filter for (response size - payload size) #779

Open

denandz added 2 commits September 20, 2024 19:02

Merge pull request #5 from denandz/auditlogging

161112a

Auditlogging

Remove header cannonicalization

da4a34a

denandz added 3 commits September 27, 2024 20:01

Add support for multiple headers with the same name

4eb65ba

Merge branch 'master' into header-case-fixes

f59a40c

Merge pull request #6 from denandz/header-case-fixes

8f59a95

Support any header case and multiple headers with the same name.

denandz requested a review from joohoi November 10, 2024 23:01

denandz added 9 commits February 18, 2025 12:28

Add payload/response delta output

8f38e94

Added payload/response delta into file outputs

c6d1b53

Merge pull request #7 from denandz/payloadresponsedelta

71d4214

Merge payload-response delta feature

Fix audit log tests

8732dca

Minimize CLI output - remove words and lines

7f5649f

Implement responsestats array for summary output

bd238e7

Fix help

51bede8

Add response timing chart

f2d14ca

Add checks to response stats generation

1fd1367

denandz added 12 commits March 30, 2025 22:33

Add statistics and timing line chart to the response summary

b60c887

Fix minimum content-length size reporting in the summary

948d6f2

Add error check to HostURLFromRequest

d9dd613

Fix audit log err overwriting

3eff8bf

Fix erroneous err variable overwrite

53b5327

Implement box plots for summary view

8b95bc4

Update README.md

0bad70f

Merge pull request #8 from denandz/run-summary

a69e711

Add run summaries

Add errors to Request objects for logging

3bfb71c

Fix audit logger error message

ed58fb6

Merge branch 'master' into auditlogging

ee8be89

Revert "Merge branch 'master' into auditlogging"

9b41413

This reverts commit ee8be89, reversing changes made to ed58fb6.

joohoi approved these changes Apr 5, 2025

View reviewed changes

joohoi merged commit ce3cf6b into ffuf:master Apr 5, 2025
3 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

[FEATURE] Add audit logging functionality #771

[FEATURE] Add audit logging functionality #771

Uh oh!

denandz commented Mar 21, 2024 •

edited

Loading

Uh oh!

denandz commented May 20, 2024

Uh oh!

joohoi left a comment

Uh oh!

denandz commented May 20, 2024

Uh oh!

denandz commented May 28, 2024

Uh oh!

denandz commented Jun 24, 2024

Uh oh!

jarodchase commented Sep 20, 2024

Uh oh!

denandz commented Apr 5, 2025 •

edited

Loading

Uh oh!

joohoi commented Apr 5, 2025

Uh oh!

joohoi left a comment

Uh oh!

Uh oh!

denandz commented Apr 9, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

[FEATURE] Add audit logging functionality #771

[FEATURE] Add audit logging functionality #771

Uh oh!

Conversation

denandz commented Mar 21, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

denandz commented May 20, 2024

Uh oh!

joohoi left a comment

Choose a reason for hiding this comment

Uh oh!

denandz commented May 20, 2024

Uh oh!

denandz commented May 28, 2024

Uh oh!

denandz commented Jun 24, 2024

Uh oh!

jarodchase commented Sep 20, 2024

Uh oh!

denandz commented Apr 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

joohoi commented Apr 5, 2025

Uh oh!

joohoi left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

denandz commented Apr 9, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

denandz commented Mar 21, 2024 •

edited

Loading

denandz commented Apr 5, 2025 •

edited

Loading