Add -json option #509

justinsteven · 2022-01-17T06:13:30Z

Prints newline-delimited JSON output to STDOUT

Resolves #205

Description

Adds the -json option and makes it mutually exclusive with -v. Operates similarly to -v in that it changes the behavior of the stdout outputter, causing result lines to be stringified json.Marshal(result) output.

By simply changing stdout to be JSON-formatted, new use cases include:

Piping JSON stdout into another tool, or into an analysis pipeline, without needing to go via the filesystem
Sending the JSON stdout to a file on disk, allowing machine-readable results to be produced incrementally (mitigates the need for ffuf should write output while a job is in progress #205)
- Optional use of tee to do this while monitoring results via the terminal

Aside from making -json be mutually exclusive with -v, no other thought was given to compatibility with other features.

Demo

% cat ~/domains.txt
bugcrowd.com
www.hackerone.com

% cat ~/wordlist.txt
index.html
index.php
robots.txt

The progress output causes typical output to be messy:

% ffuf -u https://FUZZ1/FUZZ2 -w ~/domains.txt:FUZZ1 -w ~/wordlist.txt:FUZZ2 -json -H "User-Agent: Mozilla"

        /'___\  /'___\           /'___\
       /\ \__/ /\ \__/  __  __  /\ \__/
       \ \ ,__\\ \ ,__\/\ \/\ \ \ \ ,__\
        \ \ \_/ \ \ \_/\ \ \_\ \ \ \ \_/
         \ \_\   \ \_\  \ \____/  \ \_\
          \/_/    \/_/   \/___/    \/_/

       v1.3.1-dev
________________________________________________

 :: Method           : GET
 :: URL              : https://FUZZ1/FUZZ2
 :: Wordlist         : FUZZ1: /home/justin/domains.txt
 :: Wordlist         : FUZZ2: /home/justin/wordlist.txt
 :: Header           : User-Agent: Mozilla
 :: Follow redirects : false
 :: Calibration      : false
 :: Timeout          : 10
 :: Threads          : 40
 :: Matcher          : Response status: 200,204,301,302,307,401,403,405
________________________________________________

:: Progress: [6/6] :: Job [1/1] :: 0 req/sec :: Duration: [0:00:00] :: Errors: 0 ::{"input":{"FUZZ1":"d3d3LmhhY2tlcm9uZS5jb20=","FUZZ2":"cm9ib3RzLnR4dA=="},"position":6,"status":200,"length":1586,"words":129,"lines":62,"content-type":"text/plain","redirectlocation":"","url":"https://www.hackerone.com/robots.txt","duration":38538999,"resultfile":"","host":"www.hackerone.com"}
:: Progress: [6/6] :: Job [1/1] :: 0 req/sec :: Duration: [0:00:00] :: Errors: 0 ::{"input":{"FUZZ1":"YnVnY3Jvd2QuY29t","FUZZ2":"cm9ib3RzLnR4dA=="},"position":5,"status":200,"length":53,"words":4,"lines":5,"content-type":"text/plain; charset=UTF-8","redirectlocation":"","url":"https://bugcrowd.com/robots.txt","duration":41794316,"resultfile":"","host":"bugcrowd.com"}
:: Progress: [6/6] :: Job [1/1] :: 0 req/sec :: Duration: [0:00:00] :: Errors: 0 ::{"input":{"FUZZ1":"d3d3LmhhY2tlcm9uZS5jb20=","FUZZ2":"aW5kZXgucGhw"},"position":4,"status":301,"length":31324,"words":562,"lines":12,"content-type":"text/html; charset=UTF-8","redirectlocation":"https://www.hackerone.com/","url":"https://www.hackerone.com/index.php","duration":48637368,"resultfile":"","host":"www.hackerone.com"}
:: Progress: [6/6] :: Job [1/1] :: 77 req/sec :: Duration: [0:00:01] :: Errors: 0 ::

Use of tee and jq while throwing away stderr to clean up the output:

% ffuf -u https://FUZZ1/FUZZ2 -w ~/domains.txt:FUZZ1 -w ~/wordlist.txt:FUZZ2 -json -H "User-Agent: Mozilla" 2>/dev/null | tee output.json | jq .
{
  "input": {
    "FUZZ1": "d3d3LmhhY2tlcm9uZS5jb20=",
    "FUZZ2": "cm9ib3RzLnR4dA=="
  },
  "position": 6,
  "status": 200,
  "length": 1586,
  "words": 129,
  "lines": 62,
  "content-type": "text/plain",
  "redirectlocation": "",
  "url": "https://www.hackerone.com/robots.txt",
  "duration": 45728708,
  "resultfile": "",
  "host": "www.hackerone.com"
}
{
  "input": {
    "FUZZ1": "YnVnY3Jvd2QuY29t",
    "FUZZ2": "cm9ib3RzLnR4dA=="
  },
  "position": 5,
  "status": 200,
  "length": 53,
  "words": 4,
  "lines": 5,
  "content-type": "text/plain; charset=UTF-8",
  "redirectlocation": "",
  "url": "https://bugcrowd.com/robots.txt",
  "duration": 52167547,
  "resultfile": "",
  "host": "bugcrowd.com"
}
[... SNIP ...]

% cat output.json
{"input":{"FUZZ1":"d3d3LmhhY2tlcm9uZS5jb20=","FUZZ2":"cm9ib3RzLnR4dA=="},"position":6,"status":200,"length":1586,"words":129,"lines":62,"content-type":"text/plain","redirectlocation":"","url":"https://www.hackerone.com/robots.txt","duration":45728708,"resultfile":"","host":"www.hackerone.com"}
{"input":{"FUZZ1":"YnVnY3Jvd2QuY29t","FUZZ2":"cm9ib3RzLnR4dA=="},"position":5,"status":200,"length":53,"words":4,"lines":5,"content-type":"text/plain; charset=UTF-8","redirectlocation":"","url":"https://bugcrowd.com/robots.txt","duration":52167547,"resultfile":"","host":"bugcrowd.com"}
{"input":{"FUZZ1":"d3d3LmhhY2tlcm9uZS5jb20=","FUZZ2":"aW5kZXgucGhw"},"position":4,"status":301,"length":31324,"words":562,"lines":12,"content-type":"text/html; charset=UTF-8","redirectlocation":"https://www.hackerone.com/","url":"https://www.hackerone.com/index.php","duration":140936220,"resultfile":"","host":"www.hackerone.com"}

Additonally

If this is the first time you are contributing to ffuf, add your name to CONTRIBUTORS.md.
The file should be alphabetically ordered.
Add a short description of the fix to CHANGELOG.md

Prints newline-delimited JSON output to STDOUT

joohoi

Thanks for the PR! I think jsonline format is really useful to have for different integration purposes.

I added a few change requests and / or proposals, but in general the PR look good

pkg/output/stdout.go

CONTRIBUTORS.md

For each JSON result being printed, prepend it with a TERMINAL_CLEAR_LINE via STDERR. This clears the progress line (which is also being emitted via STDERR) and leaves us with a clean stream of JSON lines in the terminal.

joohoi

Thanks for the fixes, it seems all good. One more small thing, I'd like the flag to be called -jsonlines instead of -json in order to not to confuse people too much. I think JSONlines is the somewhat official name for that data format, re: https://jsonlines.org/

joohoi · 2022-02-07T12:48:12Z

Hi @justinsteven , are you interested in making the last very small change? I can also do this for you if you wish though. (also there seems to be a merge conflict caused by other PR's getting merged while this has been open)

justinsteven · 2022-02-07T13:05:58Z

Yes, sorry, I've been tied up and I missed that you'd said that abstracting the TERMINAL_CLEAR_LINE can be left until down the track.

I'm happy to change -json to -jsonlines. But could I persuade you in saying -json is easier to type and remember, and would also match the syntax of httpx and nuclei? Their help says:

OUTPUT:
   [... SNIP ...]
   -json                             store output in JSONL(ines) format

I could clarify via the help that -json is in fact JSON Lines format.

If you think it's clearer and easier for users if it's -jsonlines I'll get that done :)

joohoi

Ok, I think you have a fair point here. My thinking was that -json could confuse people with the json output file format and jsonlines output. Let's do it your way.

* Add -json option Prints newline-delimited JSON output to STDOUT * sort * Clear terminal line via STDERR foreach JSON result For each JSON result being printed, prepend it with a TERMINAL_CLEAR_LINE via STDERR. This clears the progress line (which is also being emitted via STDERR) and leaves us with a clean stream of JSON lines in the terminal.

Add -json option

0c28d13

Prints newline-delimited JSON output to STDOUT

joohoi requested changes Jan 22, 2022

View reviewed changes

pkg/output/stdout.go Outdated Show resolved Hide resolved

pkg/output/stdout.go Show resolved Hide resolved

CONTRIBUTORS.md Outdated Show resolved Hide resolved

justinsteven added 3 commits January 23, 2022 09:47

Merge branch 'master' into feature/json_output

ebc0ce5

sort

6b6fcc2

Clear terminal line via STDERR foreach JSON result

83d35ef

For each JSON result being printed, prepend it with a TERMINAL_CLEAR_LINE via STDERR. This clears the progress line (which is also being emitted via STDERR) and leaves us with a clean stream of JSON lines in the terminal.

joohoi requested changes Jan 26, 2022

View reviewed changes

Merge branch 'master' into feature/json_output

e8ce805

joohoi approved these changes Mar 6, 2022

View reviewed changes

joohoi merged commit 4c1a754 into ffuf:master Mar 6, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Add -json option #509

Add -json option #509

Uh oh!

justinsteven commented Jan 17, 2022

Uh oh!

joohoi left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

joohoi left a comment

Uh oh!

joohoi commented Feb 7, 2022

Uh oh!

justinsteven commented Feb 7, 2022

Uh oh!

joohoi left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Add -json option #509

Add -json option #509

Uh oh!

Conversation

justinsteven commented Jan 17, 2022

Description

Demo

Additonally

Uh oh!

joohoi left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

joohoi left a comment

Choose a reason for hiding this comment

Uh oh!

joohoi commented Feb 7, 2022

Uh oh!

justinsteven commented Feb 7, 2022

Uh oh!

joohoi left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants