# Xurlfind3R | A CLI Utility To Find Domain'S Known URLs From Curated Passive Online Sources

**URL:** <https://onehack.st/t/xurlfind3r-a-cli-utility-to-find-domains-known-urls-from-curated-passive-online-sources/253441>\
**Category:** Tools & Scripts\
**Tags:** tools, programming, freebies, earn-money\
**Created:** [August 9, 2023, 8:40pm UTC](https://onehack.st/t/xurlfind3r-a-cli-utility-to-find-domains-known-urls-from-curated-passive-online-sources/253441 "2023-08-09T20:40:51Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![WhiteHat](https://onehack.st/user_avatar/onehack.st/whitehat/32/162514_2.png) [@WhiteHat](https://onehack.st/u/WhiteHat)\
**Post date:** [August 9, 2023, 8:40pm UTC](https://onehack.st/t/xurlfind3r-a-cli-utility-to-find-domains-known-urls-from-curated-passive-online-sources/253441/1 "2023-08-09T20:40:51Z")

</div>

![image](https://onehack.st/uploads/default/original/3X/f/e/fe8b387b7867c8a25b83ba715ef0d4c109503dd9.jpeg)

# xurlfind3r

[![made with go](https://camo.githubusercontent.com/b0fd323d66f8edfa2b9e9cf058a9afdbc831ef5f0532c2861ed928785bef104a/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6d616465253230776974682d476f2d3030303046462e737667)](https://camo.githubusercontent.com/b0fd323d66f8edfa2b9e9cf058a9afdbc831ef5f0532c2861ed928785bef104a/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6d616465253230776974682d476f2d3030303046462e737667) [![release](https://camo.githubusercontent.com/7da9e39466263d4d48f8a2cdb6694579bc496772c2a4b3bf88b9fa7a8a8d1c52/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f72656c656173652f6875657269737469712f7875726c66696e6433723f7374796c653d666c617426636f6c6f723d303030304646)](https://github.com/hueristiq/xurlfind3r/releases) [![license](https://camo.githubusercontent.com/69da6b1f066fc213dd5d871bf5c8304a570031cec17647a5a428d20d452aef90/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d4d49542d677261792e7376673f636f6c6f723d303030304646)](https://github.com/hueristiq/xurlfind3r/blob/master/LICENSE) [![maintenance](https://camo.githubusercontent.com/103f45167e2b9f337c39b95e637d38e51f435f415f2122a09f5a5e809eac7758/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6d61696e7461696e65642533462d7965732d3030303046462e737667)](https://camo.githubusercontent.com/103f45167e2b9f337c39b95e637d38e51f435f415f2122a09f5a5e809eac7758/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6d61696e7461696e65642533462d7965732d3030303046462e737667) [![open issues](https://camo.githubusercontent.com/91b2c9011807b14159417951c304128df797bdec8c64e9f8f9fb2f778e8ea783/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f6973737565732d7261772f6875657269737469712f7875726c66696e6433722e7376673f7374796c653d666c617426636f6c6f723d303030304646)](https://github.com/hueristiq/xurlfind3r/issues?q=is:issue+is:open) [![closed issues](https://camo.githubusercontent.com/538c240cc240977391f9595d32207b5b17c29cd81f22d02811fa3a94c24c226d/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f6973737565732d636c6f7365642d7261772f6875657269737469712f7875726c66696e6433722e7376673f7374796c653d666c617426636f6c6f723d303030304646)](https://github.com/hueristiq/xurlfind3r/issues?q=is:issue+is:closed) [![contribution](https://camo.githubusercontent.com/77d98406f9a479d112349a5322cac5d78904a4a8e519a97822826d9c1c702d8c/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f636f6e747269627574696f6e732d77656c636f6d652d3030303046462e737667)](https://github.com/hueristiq/xurlfind3r/blob/master/CONTRIBUTING.md)

`xurlfind3r` is a command-line interface (CLI) utility to find domain’s known URLs from curated passive online sources.

## Resource

- [Features](https://github.com/hueristiq/xurlfind3r#features)
- [Installation](https://github.com/hueristiq/xurlfind3r#installation)
  - [Install release binaries (Without Go Installed)](https://github.com/hueristiq/xurlfind3r#install-release-binaries-without-go-installed)
  - [Install source (With Go Installed)](https://github.com/hueristiq/xurlfind3r#install-source-with-go-installed)
    - [`go install ...`](https://github.com/hueristiq/xurlfind3r#go-install)
    - [`go build ...` the development Version](https://github.com/hueristiq/xurlfind3r#go-build--the-development-version)

- [Post Installation](https://github.com/hueristiq/xurlfind3r#post-installation)
- [Usage](https://github.com/hueristiq/xurlfind3r#usage)
  - [Basic](https://github.com/hueristiq/xurlfind3r#basic)
  - [Filter Regex](https://github.com/hueristiq/xurlfind3r#filter-regex)
  - [Match Regex](https://github.com/hueristiq/xurlfind3r#match-regex)

- [Contributing](https://github.com/hueristiq/xurlfind3r#contributing)
- [Licensing](https://github.com/hueristiq/xurlfind3r#licensing)

## Features

- Fetches URLs from curated passive sources to maximize results.
  - **[AlienVault’s OTX](https://otx.alienvault.com/)** ◇ **[BeVigil](https://bevigil.com/)** ◇ **[Common Crawl](https://commoncrawl.org/)** ◇ **[Github](https://github.com/)** ◇ **[Intelligence X](https://intelx.io/)** ◇ **[URLScan](https://urlscan.io/)** ◇ **[Wayback Machine](https://archive.org/web/)**

- Parses URLs from wayback webpages and `robots.txt` snapshots.
- Supports URLs matching and filtering.
- Supports `stdin` and `stdout` for easy integration into workflows.
- Cross-Platform (Windows, Linux & macOS).

## Installation

### Install release binaries (Without Go Installed)

Visit the [releases page](https://github.com/hueristiq/xurlfind3r/releases) and find the appropriate archive for your operating system and architecture. Download the archive from your browser or copy its URL and retrieve it with `wget` or `curl`:

- …with `wget`:

wget [https://github.com/hueristiq/xurlfind3r/releases/download/v](https://github.com/hueristiq/xurlfind3r/releases/download/v)/xurlfind3r--linux-amd64.tar.gz

- …or, with `curl`:

curl -OL [https://github.com/hueristiq/xurlfind3r/releases/download/v](https://github.com/hueristiq/xurlfind3r/releases/download/v)/xurlfind3r--linux-amd64.tar.gz

…then, extract the binary:

tar xf xurlfind3r--linux-amd64.tar.gz

> **TIP:** The above steps, download and extract, can be combined into a single step with this onliner
> 
> curl -sL [https://github.com/hueristiq/xurlfind3r/releases/download/v](https://github.com/hueristiq/xurlfind3r/releases/download/v)/xurlfind3r--linux-amd64.tar.gz | tar -xzv

**NOTE:** On Windows systems, you should be able to double-click the zip archive to extract the `xurlfind3r` executable.

…move the `xurlfind3r` binary to somewhere in your `PATH`. For example, on GNU/Linux and OS X systems:

sudo mv xurlfind3r /usr/local/bin/

**NOTE:** Windows users can follow [How to: Add Tool Locations to the PATH Environment Variable](https://msdn.microsoft.com/en-us/library/office/ee537574(v=office.14).aspx) in order to add `xurlfind3r` to their `PATH`.

### Install source (With Go Installed)

Before you install from source, you need to make sure that Go is installed on your system. You can install Go by following the official instructions for your operating system. For this, we will assume that Go is already installed.

#### `go install ...`

go install -v [github.com/hueristiq/xurlfind3r/cmd/xurlfind3r@latest](http://github.com/hueristiq/xurlfind3r/cmd/xurlfind3r@latest)

#### `go build ...` the development Version

- Clone the repository

git clone [https://github.com/hueristiq/xurlfind3r.git](https://github.com/hueristiq/xurlfind3r.git)

- Build the utility

cd xurlfind3r/cmd/xurlfind3r && \ go build .

- Move the `xurlfind3r` binary to somewhere in your `PATH`. For example, on GNU/Linux and OS X systems:

sudo mv xurlfind3r /usr/local/bin/

**NOTE:** Windows users can follow [How to: Add Tool Locations to the PATH Environment Variable](https://msdn.microsoft.com/en-us/library/office/ee537574(v=office.14).aspx) in order to add `xurlfind3r` to their `PATH`.

**NOTE:** While the development version is a good way to take a peek at `xurlfind3r`’s latest features before they get released, be aware that it may have bugs. Officially released versions will generally be more stable.

## Post Installation

`xurlfind3r` will work right after [installation](https://github.com/hueristiq/xurlfind3r#installation). However, **[BeVigil](https://bevigil.com/)**, **[Github](https://github.com/)** and **[Intelligence X](https://intelx.io/)** require API keys to work, **[URLScan](https://urlscan.io/)** supports API key but not required. The API keys are stored in the `$HOME/.hueristiq/xurlfind3r/config.yaml` file - created upon first run - and uses the YAML format. Multiple API keys can be specified for each of these source from which one of them will be used.

Example `config.yaml`:

version: 0.3.0 sources: - bevigil - commoncrawl - github - intelx - otx - urlscan - wayback keys: bevigil: - awA5nvpKU3N8ygkZ github: - d23a554bbc1aabb208c9acfbd2dd41ce7fc9db39 - asdsd54bbc1aabb208c9acfbd2dd41ce7fc9db39 intelx: - 2.intelx.io:00000000-0000-0000-0000-000000000000 urlscan: - d4c85d34-e425-446e-d4ab-f5a3412acbe8

## Usage

To display help message for `xurlfind3r` use the `-h` flag:

xurlfind3r -h

help message:

```auto
                 _ ___ ______      
_____ _ _ __| |/ _(_)___  __| |___ / _ __ 
\ \/ / | | | ' __| | |_| | '_ \ / _` | |_ \| '__ |
 > <| |_| | | | | _| | | | | (_| |___) | |
/_/\_\\ __,_|_| |_|_| |_|_| |_|\__ ,_| ____ /|_| v0.3.0

USAGE:
  xurlfind3r [OPTIONS]

INPUT:
 -d, --domain string[] target domains
 -l, --list string target domains' list file path

SCOPE:
     --include-subdomains bool match subdomain's URLs

SOURCES:
      --sources bool list supported sources
 -u, --use-sources string[] comma(,) separated sources to use
 -e, --exclude-sources string[] comma(,) separated sources to exclude
      --parse-wayback-robots bool with wayback, parse robots.txt snapshots
      --parse-wayback-source bool with wayback, parse source code snapshots

FILTER & MATCH:
 -f, --filter string regex to filter URLs
 -m, --match string regex to match URLs

OUTPUT:
     --no-color bool disable colored output
 -o, --output string output URLs file path
 -O, --output-directory string output URLs directory path
 -v, --verbosity string debug, info, warning, error, fatal or silent (default: info)

CONFIGURATION:
 -c, --configuration string configuration file path (default: ~/.hueristiq/xurlfind3r/config.yaml)

```

### Examples

#### Basic

xurlfind3r -d [hackerone.com](http://hackerone.com) --include-subdomains

#### Filter Regex

# filter images xurlfind3r -d [hackerone.com](http://hackerone.com) --include-subdomains -f ‘`^https?://[^/]*?/.*\.(jpg|jpeg|png|gif|bmp)(\?[^\s]*)?$`’

#### Match Regex

# match js URLs xurlfind3r -d [hackerone.com](http://hackerone.com) --include-subdomains -m ‘^https?://[^/]_?/._.js(?[^\s]\*)?$’

## Contributing

[Issues](https://github.com/hueristiq/xurlfind3r/issues) and [Pull Requests](https://github.com/hueristiq/xurlfind3r/pulls) are welcome! **Check out the [contribution guidelines](https://github.com/hueristiq/xurlfind3r/blob/master/CONTRIBUTING.md).**

## Licensing

This utility is distributed under the [MIT license](https://github.com/hueristiq/xurlfind3r/blob/master/LICENSE).

## GitHub:

> **[GitHub - hueristiq/xurlfind3r: A command-line utility designed to discover URLs...](https://github.com/hueristiq/xurlfind3r)**
>
> A command-line utility designed to discover URLs for a given domain in a simple, efficient way. It works by gathering information from a variety of passive sources, meaning it doesn't interact directly with the target but instead gathers data that is already publicly available.