# Katana | A Next-Generation Crawling And Spidering Framework

**URL:** <https://onehack.st/t/katana-a-next-generation-crawling-and-spidering-framework/238622>\
**Category:** Tools & Scripts\
**Tags:** tools, programming, freebies\
**Created:** [April 19, 2023, 10:34pm UTC](https://onehack.st/t/katana-a-next-generation-crawling-and-spidering-framework/238622 "2023-04-19T22:34:56Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![WhiteHat](https://onehack.st/user_avatar/onehack.st/whitehat/32/162514_2.png) [@WhiteHat](https://onehack.st/u/WhiteHat)\
**Post date:** [April 19, 2023, 10:34pm UTC](https://onehack.st/t/katana-a-next-generation-crawling-and-spidering-framework/238622/1 "2023-04-19T22:34:56Z")

</div>

[![](https://user-images.githubusercontent.com/8293321/196779266-421c79d4-643a-4f73-9b54-3da379bbac09.png) ](https://user-images.githubusercontent.com/8293321/196779266-421c79d4-643a-4f73-9b54-3da379bbac09.png)

#### A next-generation crawling and spidering framework

[![](https://camo.githubusercontent.com/009e04dede3b8440ee6c4dc7df833e9ec1084cb4a92e22127ab8c536e1e67efc/68747470733a2f2f676f7265706f7274636172642e636f6d2f62616467652f6769746875622e636f6d2f70726f6a656374646973636f766572792f6b6174616e61)](https://goreportcard.com/report/github.com/projectdiscovery/katana) [![](https://camo.githubusercontent.com/f5054ffcd4245c10d3ec85ef059e07aacf787b560f83ad4aec2236364437d097/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f636f6e747269627574696f6e732d77656c636f6d652d627269676874677265656e2e7376673f7374796c653d666c6174)](https://github.com/projectdiscovery/katana/issues) [![](https://camo.githubusercontent.com/4833927e0bbdc62937abdd77de377db147e3f7857b4e920f09b644850d568af1/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f72656c656173652f70726f6a656374646973636f766572792f6b6174616e61)](https://github.com/projectdiscovery/katana/releases) [![](https://camo.githubusercontent.com/28c331d7899d563f2523eca1d20dcfed9faa70bcaece05a709bdda3919aa4dff/68747470733a2f2f696d672e736869656c64732e696f2f747769747465722f666f6c6c6f772f70646973636f76657279696f2e7376673f6c6f676f3d74776974746572)](https://twitter.com/pdiscoveryio) [![](https://camo.githubusercontent.com/7da3e7a5401dcf958535f08c2d8cbffe8a5f61de7bcb7de9a6b3b097c1cd141e/68747470733a2f2f696d672e736869656c64732e696f2f646973636f72642f3639353634353233373431383133313530372e7376673f6c6f676f3d646973636f7264)](https://discord.gg/projectdiscovery)

[Features](https://github.com/projectdiscovery/katana#features) • [Installation](https://github.com/projectdiscovery/katana#installation) • [Usage](https://github.com/projectdiscovery/katana#usage) • [Scope](https://github.com/projectdiscovery/katana#scope-control) • [Config](https://github.com/projectdiscovery/katana#crawler-configuration) • [Filters](https://github.com/projectdiscovery/katana#filters) • [Join Discord](https://discord.gg/projectdiscovery)

# Features

[![image](https://user-images.githubusercontent.com/8293321/199371558-daba03b6-bf9c-4883-8506-76497c6c3a44.png)](https://user-images.githubusercontent.com/8293321/199371558-daba03b6-bf9c-4883-8506-76497c6c3a44.png)

- Fast And fully configurable web crawling
- **Standard** and **Headless** mode support
- **JavaScript** parsing / crawling
- Customizable **automatic form filling**
- **Scope control** - Preconfigured field / Regex
- **Customizable output** - Preconfigured fields
- INPUT - **STDIN** , **URL** and **LIST**
- OUTPUT - **STDOUT** , **FILE** and **JSON**

## Installation

katana requires **Go 1.18** to install successfully. To install, just run the below command or download pre-compiled binary from [release page](https://github.com/projectdiscovery/katana/releases).

```auto
go install github.com/projectdiscovery/katana/cmd/katana@latest

```

**More options to install / run katana-**

## Usage

```auto
katana -h

```

This will display help for the tool. Here are all the switches it supports.

```auto
Katana is a fast crawler focused on execution in automation
pipelines offering both headless and non-headless crawling.

Usage:
  ./katana [flags]

Flags:
INPUT:
   -u, -list string[] target url / list to crawl

CONFIGURATION:
   -r, -resolvers string[] list of custom resolver (file or comma separated)
   -d, -depth int maximum depth to crawl (default 3)
   -jc, -js-crawl enable endpoint parsing / crawling in javascript file
   -ct, -crawl-duration int maximum duration to crawl the target for
   -kf, -known-files string enable crawling of known files (all,robotstxt,sitemapxml)
   -mrs, -max-response-size int maximum response size to read (default 9223372036854775807)
   -timeout int time to wait for request in seconds (default 10)
   -aff, -automatic-form-fill enable automatic form filling (experimental)
   -retry int number of times to retry the request (default 1)
   -proxy string http/socks5 proxy to use
   -H, -headers string[] custom header/cookie to include in request
   -config string path to the katana configuration file
   -fc, -form-config string path to custom form configuration file
   -flc, -field-config string path to custom field configuration file
   -s, -strategy string Visit strategy (depth-first, breadth-first) (default "depth-first")
   -iqp, -ignore-query-params Ignore crawling same path with different query-param values

DEBUG:
   -health-check, -hc run diagnostic check up
   -elog, -error-log string file to write sent requests error log

HEADLESS:
   -hl, -headless enable headless hybrid crawling (experimental)
   -sc, -system-chrome use local installed chrome browser instead of katana installed
   -sb, -show-browser show the browser on the screen with headless mode
   -ho, -headless-options string[] start headless chrome with additional options
   -nos, -no-sandbox start headless chrome in --no-sandbox mode
   -cdd, -chrome-data-dir string path to store chrome browser data
   -scp, -system-chrome-path string use specified chrome browser for headless crawling
   -noi, -no-incognito start headless chrome without incognito mode

SCOPE:
   -cs, -crawl-scope string[] in scope url regex to be followed by crawler
   -cos, -crawl-out-scope string[] out of scope url regex to be excluded by crawler
   -fs, -field-scope string pre-defined scope field (dn,rdn,fqdn) (default "rdn")
   -ns, -no-scope disables host based default scope
   -do, -display-out-scope display external endpoint from scoped crawling

FILTER:
   -mr, -match-regex string[] regex or list of regex to match on output url (cli, file)
   -fr, -filter-regex string[] regex or list of regex to filter on output url (cli, file)
   -f, -field string field to display in output (url,path,fqdn,rdn,rurl,qurl,qpath,file,ufile,key,value,kv,dir,udir)
   -sf, -store-field string field to store in per-host output (url,path,fqdn,rdn,rurl,qurl,qpath,file,ufile,key,value,kv,dir,udir)
   -em, -extension-match string[] match output for given extension (eg, -em php,html,js)
   -ef, -extension-filter string[] filter output for given extension (eg, -ef png,css)

RATE-LIMIT:
   -c, -concurrency int number of concurrent fetchers to use (default 10)
   -p, -parallelism int number of concurrent inputs to process (default 10)
   -rd, -delay int request delay between each request in seconds
   -rl, -rate-limit int maximum requests to send per second (default 150)
   -rlm, -rate-limit-minute int maximum number of requests to send per minute

UPDATE:
   -up, -update update katana to latest version
   -duc, -disable-update-check disable automatic katana update check

OUTPUT:
   -o, -output string file to write output to
   -sr, -store-response store http requests/responses
   -srd, -store-response-dir string store http requests/responses to custom directory
   -j, -json write output in JSONL(ines) format
   -nc, -no-color disable output content coloring (ANSI escape codes)
   -silent display output only
   -v, -verbose display verbose output
   -debug display debug output
   -version display project version

```

## Running Katana

### Input for katana

**katana** requires **url** or **endpoint** to crawl and accepts single or multiple inputs.

Input URL can be provided using `-u` option, and multiple values can be provided using comma-separated input, similarly **file** input is supported using `-list` option and additionally piped input (stdin) is also supported.

#### URL Input

katana -u [https://tesla.com](https://tesla.com)

#### Multiple URL Input (comma-separated)

katana -u [https://tesla.com](https://tesla.com),[https://google.com](https://google.com)

#### List Input

$ cat url\_list.txt [https://tesla.com](https://tesla.com) [https://google.com](https://google.com)

```auto
katana -list url_list.txt

```

#### STDIN (piped) Input

echo [https://tesla.com](https://tesla.com) | katana

cat domains | httpx | katana

Example running katana -

```auto
katana -u https://youtube.com

   ____                
  / / ______/ /____  _______ _
 / '_/ _ / __/ _ / _ \/ _ /
/_/\_\\_,_/\__/\_,_/_//_/\_,_/ v0.0.1                     

      projectdiscovery.io

[WRN] Use with caution. You are responsible for your actions.
[WRN] Developers assume no liability and are not responsible for any misuse or damage.
https://www.youtube.com/
https://www.youtube.com/about/
https://www.youtube.com/about/press/
https://www.youtube.com/about/copyright/
https://www.youtube.com/t/contact_us/
https://www.youtube.com/creators/
https://www.youtube.com/ads/
https://www.youtube.com/t/terms
https://www.youtube.com/t/privacy
https://www.youtube.com/about/policies/
https://www.youtube.com/howyoutubeworks?utm_campaign=ytgen&utm_source=ythp&utm_medium=LeftNav&utm_content=txt&u=https%3A%2F%2Fwww.youtube.com%2Fhowyoutubeworks%3Futm_source%3Dythp%26utm_medium%3DLeftNav%26utm_campaign%3Dytgen
https://www.youtube.com/new
https://m.youtube.com/
https://www.youtube.com/s/desktop/4965577f/jsbin/desktop_polymer.vflset/desktop_polymer.js
https://www.youtube.com/s/desktop/4965577f/cssbin/www-main-desktop-home-page-skeleton.css
https://www.youtube.com/s/desktop/4965577f/cssbin/www-onepick.css
https://www.youtube.com/s/_/ytmainappweb/_/ss/k=ytmainappweb.kevlar_base.0Zo5FUcPkCg.L.B1.O/am=gAE/d=0/rs=AGKMywG5nh5Qp-BGPbOaI1evhF5BVGRZGA
https://www.youtube.com/opensearch?locale=en_GB
https://www.youtube.com/manifest.webmanifest
https://www.youtube.com/s/desktop/4965577f/cssbin/www-main-desktop-watch-page-skeleton.css
https://www.youtube.com/s/desktop/4965577f/jsbin/web-animations-next-lite.min.vflset/web-animations-next-lite.min.js
https://www.youtube.com/s/desktop/4965577f/jsbin/custom-elements-es5-adapter.vflset/custom-elements-es5-adapter.js
https://www.youtube.com/s/desktop/4965577f/jsbin/webcomponents-sd.vflset/webcomponents-sd.js
https://www.youtube.com/s/desktop/4965577f/jsbin/intersection-observer.min.vflset/intersection-observer.min.js
https://www.youtube.com/s/desktop/4965577f/jsbin/scheduler.vflset/scheduler.js
https://www.youtube.com/s/desktop/4965577f/jsbin/www-i18n-constants-en_GB.vflset/www-i18n-constants.js
https://www.youtube.com/s/desktop/4965577f/jsbin/www-tampering.vflset/www-tampering.js
https://www.youtube.com/s/desktop/4965577f/jsbin/spf.vflset/spf.js
https://www.youtube.com/s/desktop/4965577f/jsbin/network.vflset/network.js
https://www.youtube.com/howyoutubeworks/
https://www.youtube.com/trends/
https://www.youtube.com/jobs/
https://www.youtube.com/kids/

```

## Crawling Mode

### Standard Mode

Standard crawling modality uses the standard go http library under the hood to handle HTTP requests/responses. This modality is much faster as it doesn’t have the browser overhead. Still, it analyzes HTTP responses body as is, without any javascript or DOM rendering, potentially missing post-dom-rendered endpoints or asynchronous endpoint calls that might happen in complex web applications depending, for example, on browser-specific events.

### Headless Mode

Headless mode hooks internal headless calls to handle HTTP requests/responses directly within the browser context. This offers two advantages:

- The HTTP fingerprint (TLS and user agent) fully identify the client as a legitimate browser
- Better coverage since the endpoints are discovered analyzing the standard raw response, as in the previous modality, and also the browser-rendered one with javascript enabled.

Headless crawling is optional and can be enabled using `-headless` option.

Here are other headless CLI options -

katana -h headless Flags: HEADLESS: -hl, -headless enable headless hybrid crawling (experimental) -sc, -system-chrome use local installed chrome browser instead of katana installed -sb, -show-browser show the browser on the screen with headless mode -ho, -headless-options string start headless chrome with additional options -nos, -no-sandbox start headless chrome in --no-sandbox mode -cdd, -chrome-data-dir string path to store chrome browser data -scp, -system-chrome-path string use specified chrome browser for headless crawling -noi, -no-incognito start headless chrome without incognito mode

## _`-no-sandbox`_

Runs headless chrome browser with **no-sandbox** option, useful when running as root user.

katana -u [https://tesla.com](https://tesla.com) -headless -no-sandbox

## _`-no-incognito`_

Runs headless chrome browser without incognito mode, useful when using the local browser.

katana -u [https://tesla.com](https://tesla.com) -headless -no-incognito

## _`-headless-options`_

When crawling in headless mode, additional chrome options can be specified using `-headless-options`, for example -

katana -u [https://tesla.com](https://tesla.com) -headless -system-chrome -headless-options --disable-gpu,proxy-server=[http://127.0.0.1:8080](http://127.0.0.1:8080)

## Scope Control

Crawling can be endless if not scoped, as such katana comes with multiple support to define the crawl scope.

## _`-field-scope`_

Most handy option to define scope with predefined field name, `rdn` being default option for field scope.

- `rdn` - crawling scoped to root domain name and all subdomains (e.g. `*example.com`) (default)
- `fqdn` - crawling scoped to given sub(domain) (e.g. `www.example.com` or `api.example.com`)
- `dn` - crawling scoped to domain name keyword (e.g. `example`)

katana -u [https://tesla.com](https://tesla.com) -fs dn

## _`-crawl-scope`_

For advanced scope control, `-cs` option can be used that comes with **regex** support.

katana -u [https://tesla.com](https://tesla.com) -cs login

For multiple in scope rules, file input with multiline string / regex can be passed.

$ cat in\_scope.txt login/ admin/ app/ wordpress/

katana -u [https://tesla.com](https://tesla.com) -cs in\_scope.txt

## _`-crawl-out-scope`_

For defining what not to crawl, `-cos` option can be used and also support **regex** input.

katana -u [https://tesla.com](https://tesla.com) -cos logout

For multiple out of scope rules, file input with multiline string / regex can be passed.

$ cat out\_of\_scope.txt /logout /log\_out

katana -u [https://tesla.com](https://tesla.com) -cos out\_of\_scope.txt

## _`-no-scope`_

Katana is default to scope `*.domain`, to disable this `-ns` option can be used and also to crawl the internet.

katana -u [https://tesla.com](https://tesla.com) -ns

## _`-display-out-scope`_

As default, when scope option is used, it also applies for the links to display as output, as such **external URLs are default to exclude** and to overwrite this behavior, `-do` option can be used to display all the external URLs that exist in targets scoped URL / Endpoint.

```auto
katana -u https://tesla.com -do

```

Here is all the CLI options for the scope control -

katana -h scope Flags: SCOPE: -cs, -crawl-scope string in scope url regex to be followed by crawler -cos, -crawl-out-scope string out of scope url regex to be excluded by crawler -fs, -field-scope string pre-defined scope field (dn,rdn,fqdn) (default “rdn”) -ns, -no-scope disables host based default scope -do, -display-out-scope display external endpoint from scoped crawling

## Crawler Configuration

Katana comes with multiple options to configure and control the crawl as the way we want.

## _`-depth`_

Option to define the `depth` to follow the urls for crawling, the more depth the more number of endpoint being crawled + time for crawl.

```auto
katana -u https://tesla.com -d 5

```

## _`-js-crawl`_

Option to enable JavaScript file parsing + crawling the endpoints discovered in JavaScript files, disabled as default.

```auto
katana -u https://tesla.com -jc

```

## _`-crawl-duration`_

Option to predefined crawl duration, disabled as default.

```auto
katana -u https://tesla.com -ct 2

```

## _`-known-files`_

Option to enable crawling `robots.txt` and `sitemap.xml` file, disabled as default.

```auto
katana -u https://tesla.com -kf robotstxt,sitemapxml

```

## _`-automatic-form-fill`_

Option to enable automatic form filling for known / unknown fields, known field values can be customized as needed by updating form config file at `$HOME/.config/katana/form-config.yaml`.

Automatic form filling is experimental feature.

```auto
katana -u https://tesla.com -aff

```

There are more options to configure when needed, here is all the config related CLI options -

katana -h config Flags: CONFIGURATION: -r, -resolvers string list of custom resolver (file or comma separated) -d, -depth int maximum depth to crawl (default 3) -jc, -js-crawl enable endpoint parsing / crawling in javascript file -ct, -crawl-duration int maximum duration to crawl the target for -kf, -known-files string enable crawling of known files (all,robotstxt,sitemapxml) -mrs, -max-response-size int maximum response size to read (default 9223372036854775807) -timeout int time to wait for request in seconds (default 10) -aff, -automatic-form-fill enable automatic form filling (experimental) -retry int number of times to retry the request (default 1) -proxy string http/socks5 proxy to use -H, -headers string custom header/cookie to include in request -config string path to the katana configuration file -fc, -form-config string path to custom form configuration file -flc, -field-config string path to custom field configuration file -s, -strategy string Visit strategy (depth-first, breadth-first) (default “depth-first”)

## Filters

## _`-field`_

Katana comes with built in fields that can be used to filter the output for the desired information, `-f` option can be used to specify any of the available fields.

```auto
   -f, -field string field to display in output (url,path,fqdn,rdn,rurl,qurl,qpath,file,key,value,kv,dir,udir)

```

Here is a table with examples of each field and expected output when used -

| FIELD | DESCRIPTION | EXAMPLE |
| --- | --- | --- |
| `url` | URL Endpoint | `https://admin.projectdiscovery.io/admin/login?user=admin&password=admin` |
| `qurl` | URL including query param | `https://admin.projectdiscovery.io/admin/login.php?user=admin&password=admin` |
| `qpath` | Path including query param | `/login?user=admin&password=admin` |
| `path` | URL Path | `https://admin.projectdiscovery.io/admin/login` |
| `fqdn` | Fully Qualified Domain name | `admin.projectdiscovery.io` |
| `rdn` | Root Domain name | `projectdiscovery.io` |
| `rurl` | Root URL | `https://admin.projectdiscovery.io` |
| `ufile` | URL with File | `https://admin.projectdiscovery.io/login.js` |
| `file` | Filename in URL | `login.php` |
| `key` | Parameter keys in URL | `user,password` |
| `value` | Parameter values in URL | `admin,admin` |
| `kv` | Keys=Values in URL | `user=admin&password=admin` |
| `dir` | URL Directory name | `/admin/` |
| `udir` | URL with Directory | `https://admin.projectdiscovery.io/admin/` |

Here is an example of using field option to only display all the urls with query parameter in it -

```auto
katana -u https://tesla.com -f qurl -silent

https://shop.tesla.com/en_au?redirect=no
https://shop.tesla.com/en_nz?redirect=no
https://shop.tesla.com/product/men_s-raven-lightweight-zip-up-bomber-jacket?sku=1740250-00-A
https://shop.tesla.com/product/tesla-shop-gift-card?sku=1767247-00-A
https://shop.tesla.com/product/men_s-chill-crew-neck-sweatshirt?sku=1740176-00-A
https://www.tesla.com/about?redirect=no
https://www.tesla.com/about/legal?redirect=no
https://www.tesla.com/findus/list?redirect=no

```

### Custom Fields

You can create custom fields to extract and store specific information from page responses using regex rules. These custom fields are defined using a YAML config file and are loaded from the default location at `$HOME/.config/katana/field-config.yaml`. Alternatively, you can use the `-flc` option to load a custom field config file from a different location. Here is example custom field.

- name: email type: regex regex: - ‘([a-zA-Z0-9._-]+@[a-zA-Z0-9._-]+.[a-zA-Z0-9\_-]+)’ - ‘([a-zA-Z0-9+._-]+@[a-zA-Z0-9._-]+.[a-zA-Z0-9\_-]+)’ - name: phone type: regex regex: - ‘\d{3}-\d{8}|\d{4}-\d{7}’

When defining custom fields, following attributes are supported:

- **name** (required)

> The value of **name** attribute is used as the `-field` cli option value.

- **type** (required)

> The type of custom attribute, currenly supported option - `regex`

- **part** (optional)

> The part of the response to extract the information from. The default value is `response`, which includes both the header and body. Other possible values are `header` and `body`.

- group (optional)

> You can use this attribute to select a specific matched group in regex, for example: `group: 1`

#### Running katana using custom field:

katana -u [https://tesla.com](https://tesla.com) -f email,phone

## _`-store-field`_

To compliment `field` option which is useful to filter output at run time, there is `-sf, -store-fields` option which works exactly like field option except instead of filtering, it stores all the information on the disk under `katana_field` directory sorted by target url.

```auto
katana -u https://tesla.com -sf key,fqdn,qurl -silent

```

$ ls katana\_field/ https\_www.tesla.com\_fqdn.txt https\_www.tesla.com\_key.txt https\_www.tesla.com\_qurl.txt

The `-store-field` option can be useful for collecting information to build a targeted wordlist for various purposes, including but not limited to:

- Identifying the most commonly used parameters
- Discovering frequently used paths
- Finding commonly used files
- Identifying related or unknown subdomains

## _`-extension-match`_

Crawl output can be easily matched for specific extension using `-em` option to ensure to display only output containing given extension.

```auto
katana -u https://tesla.com -silent -em js,jsp,json

```

## _`-extension-filter`_

Crawl output can be easily filtered for specific extension using `-ef` option which ensure to remove all the urls containing given extension.

```auto
katana -u https://tesla.com -silent -ef css,txt,md

```

## _`-match-regex`_

The `-match-regex` or `-mr` flag allows you to filter output URLs using regular expressions. When using this flag, only URLs that match the specified regular expression will be printed in the output.

```auto
katana -u https://tesla.com -mr 'https://shop\.tesla\.com/*' -silent

```

## _`-filter-regex`_

The `-filter-regex` or `-fr` flag allows you to filter output URLs using regular expressions. When using this flag, it will skip the URLs that are match the specified regular expression.

```auto
katana -u https://tesla.com -fr 'https://www\.tesla\.com/*' -silent

```

Here are additional filter options -

katana -h filter Flags: FILTER: -mr, -match-regex string regex or list of regex to match on output url (cli, file) -fr, -filter-regex string regex or list of regex to filter on output url (cli, file) -f, -field string field to display in output (url,path,fqdn,rdn,rurl,qurl,qpath,file,ufile,key,value,kv,dir,udir) -sf, -store-field string field to store in per-host output (url,path,fqdn,rdn,rurl,qurl,qpath,file,ufile,key,value,kv,dir,udir) -em, -extension-match string match output for given extension (eg, -em php,html,js) -ef, -extension-filter string filter output for given extension (eg, -ef png,css)

## Rate Limit

It’s easy to get blocked / banned while crawling if not following target websites limits, katana comes with multiple option to tune the crawl to go as fast / slow we want.

## _`-delay`_

option to introduce a delay in seconds between each new request katana makes while crawling, disabled as default.

```auto
katana -u https://tesla.com -delay 20

```

## _`-concurrency`_

option to control the number of urls per target to fetch at the same time.

```auto
katana -u https://tesla.com -c 20

```

## _`-parallelism`_

option to define number of target to process at same time from list input.

```auto
katana -u https://tesla.com -p 20

```

## _`-rate-limit`_

option to use to define max number of request can go out per second.

```auto
katana -u https://tesla.com -rl 100

```

## _`-rate-limit-minute`_

option to use to define max number of request can go out per minute.

```auto
katana -u https://tesla.com -rlm 500

```

Here is all long / short CLI options for rate limit control -

katana -h rate-limit Flags: RATE-LIMIT: -c, -concurrency int number of concurrent fetchers to use (default 10) -p, -parallelism int number of concurrent inputs to process (default 10) -rd, -delay int request delay between each request in seconds -rl, -rate-limit int maximum requests to send per second (default 150) -rlm, -rate-limit-minute int maximum number of requests to send per minute

## Output

Katana support both file output in plain text format as well as JSON which includes additional information like, `source`, `tag`, and `attribute` name to co-related the discovered endpoint.

_`-output`_

By default, katana outputs the crawled endpoints in plain text format. The results can be written to a file by using the -output option.

katana -u [https://example.com](https://example.com) -no-scope -output example\_endpoints.txt

## _`-json`_

katana -u [https://example.com](https://example.com) -json | jq .

{ “timestamp”: “2023-03-20T16:23:58.027559+05:30”, “request”: { “method”: “GET”, “endpoint”: “[https://example.com](https://example.com)”, “raw”: “GET / HTTP/1.1\r\nHost: example.com\r\nUser-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 11\_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36\r\nAccept-Encoding: gzip\r\n\r\n” }, “response”: { “status\_code”: 200, “headers”: { “accept\_ranges”: “bytes”, “expires”: “Mon, 27 Mar 2023 10:53:58 GMT”, “last\_modified”: “Thu, 17 Oct 2019 07:18:26 GMT”, “content\_type”: “text/html; charset=UTF-8”, “server”: “ECS (dcb/7EA3)”, “vary”: “Accept-Encoding”, “etag”: “"3147526947"”, “cache\_control”: “max-age=604800”, “x\_cache”: “HIT”, “date”: “Mon, 20 Mar 2023 10:53:58 GMT”, “age”: “331239” }, “body”: “\n\n\n Example Domain\n\n \<meta charset="utf-8" /\>\n \<meta http-equiv="Content-type" content="text/html; charset=utf-8" /\>\n \<meta name="viewport" content="width=device-width, initial-scale=1" /\>\n \<style type="text/css"\>\n body {\n background-color: #f0f0f2;\n margin: 0;\n padding: 0;\n font-family: -apple-system, system-ui, BlinkMacSystemFont, "Segoe UI", "Open Sans", "Helvetica Neue", Helvetica, Arial, sans-serif;\n \n }\n div {\n width: 600px;\n margin: 5em auto;\n padding: 2em;\n background-color: #fdfdff;\n border-radius: 0.5em;\n box-shadow: 2px 3px 7px 2px rgba(0,0,0,0.02);\n }\n a:link, a:visited {\n color: #38488f;\n text-decoration: none;\n }\n @media (max-width: 700px) {\n div {\n margin: 0 auto;\n width: auto;\n }\n }\n \n\n\n\n

\n 
# Example Domain
\n 

This domain is for use in illustrative examples in documents. You may use this\n domain in literature without prior coordination or asking for permission.

\n 

\<a href="[https://www.iana.org/domains/example\](https://www.iana.org/domains/example%5C)”\>More information…

\n
\n\n\n", “technologies”: [“Azure”, “Amazon ECS”, “Amazon Web Services”, “Docker”, “Azure CDN”], “raw”: “HTTP/1.1 200 OK\r\nContent-Length: 1256\r\nAccept-Ranges: bytes\r\nAge: 331239\r\nCache-Control: max-age=604800\r\nContent-Type: text/html; charset=UTF-8\r\nDate: Mon, 20 Mar 2023 10:53:58 GMT\r\nEtag: "3147526947"\r\nExpires: Mon, 27 Mar 2023 10:53:58 GMT\r\nLast-Modified: Thu, 17 Oct 2019 07:18:26 GMT\r\nServer: ECS (dcb/7EA3)\r\nVary: Accept-Encoding\r\nX-Cache: HIT\r\n\r\n\n\n\n Example Domain\n\n \<meta charset="utf-8" /\>\n \<meta http-equiv="Content-type" content="text/html; charset=utf-8" /\>\n \<meta name="viewport" content="width=device-width, initial-scale=1" /\>\n \<style type="text/css"\>\n body {\n background-color: #f0f0f2;\n margin: 0;\n padding: 0;\n font-family: -apple-system, system-ui, BlinkMacSystemFont, "Segoe UI", "Open Sans", "Helvetica Neue", Helvetica, Arial, sans-serif;\n \n }\n div {\n width: 600px;\n margin: 5em auto;\n padding: 2em;\n background-color: #fdfdff;\n border-radius: 0.5em;\n box-shadow: 2px 3px 7px 2px rgba(0,0,0,0.02);\n }\n a:link, a:visited {\n color: #38488f;\n text-decoration: none;\n }\n @media (max-width: 700px) {\n div {\n margin: 0 auto;\n width: auto;\n }\n }\n \n\n\n\n
\n 
# Example Domain
\n 

This domain is for use in illustrative examples in documents. You may use this\n domain in literature without prior coordination or asking for permission.

\n 

\<a href="[https://www.iana.org/domains/example\](https://www.iana.org/domains/example%5C)”\>More information…

\n
\n\n\n" } }

## _`-store-response`_

The `-store-response` option allows for writing all crawled endpoint requests and responses to a text file. When this option is used, text files including the request and response will be written to the **katana\_response** directory. If you would like to specify a custom directory, you can use the `-store-response-dir` option.

katana -u [https://example.com](https://example.com) -no-scope -store-response

$ cat katana\_response/index.txt katana\_response/example.com/327c3fda87ce286848a574982ddd0b7c7487f816.txt [https://example.com](https://example.com) (200 OK) katana\_response/www.iana.org/bfc096e6dd93b993ca8918bf4c08fdc707a70723.txt [http://www.iana.org/domains/reserved](http://www.iana.org/domains/reserved) (200 OK)

**Note:**

_`-store-response` option is not supported in `-headless` mode._

Here are additional CLI options related to output -

katana -h output OUTPUT: -o, -output string file to write output to -sr, -store-response store http requests/responses -srd, -store-response-dir string store http requests/responses to custom directory -j, -json write output in JSONL(ines) format -nc, -no-color disable output content coloring (ANSI escape codes) -silent display output only -v, -verbose display verbose output -version display project version

## Katana as a library

`katana` can be used as a library by creating an instance of the `Option` struct and populating it with the same options that would be specified via CLI. Using the options you can create `crawlerOptions` and so standard or hybrid `crawler`. `crawler.Crawl` method should be called to crawl the input.

```auto
package main

import (
	"github.com/projectdiscovery/gologger"
	"github.com/projectdiscovery/katana/pkg/engine/standard"
	"github.com/projectdiscovery/katana/pkg/output"
	"github.com/projectdiscovery/katana/pkg/types"
)

func main() {
	options := &types.Options{
		MaxDepth: 1, // Maximum depth to crawl
		FieldScope: "rdn", // Crawling Scope Field
		BodyReadSize: 2 * 1024 * 1024, // Maximum response size to read
		RateLimit: 150, // Maximum requests to send per second
		Strategy: "depth-first", // Visit strategy (depth-first, breadth-first)
		OnResult: func(result output.Result) { // Callback function to execute for result
			gologger.Info().Msg(result.Request.URL)
		},
	}
	crawlerOptions, err := types.NewCrawlerOptions(options)
	if err != nil {
		gologger.Fatal().Msg(err.Error())
	}
	defer crawlerOptions.Close()
	crawler, err := standard.New(crawlerOptions)
	if err != nil {
		gologger.Fatal().Msg(err.Error())
	}
	defer crawler.Close()
	var input = "https://tesla.com"
	err = crawler.Crawl(input)
	if err != nil {
		gologger.Warning().Msgf("Could not crawl %s: %s", input, err.Error())
	}
}

```

* * *

katana is made with ❤ by the [projectdiscovery](https://projectdiscovery.io/) team and distributed under [MIT License](https://github.com/projectdiscovery/katana/blob/main/LICENSE.md).

## GitHub:

[https://github.com/projectdiscovery/katana](https://github.com/projectdiscovery/katana)