# OnionSearch | A Script That Scrapes Urls On Different .Onion Search Engines

**URL:** <https://onehack.st/t/onionsearch-a-script-that-scrapes-urls-on-different-onion-search-engines/193565>\
**Category:** Tools & Scripts\
**Tags:** seo, programming\
**Created:** [July 12, 2021, 10:04am UTC](https://onehack.st/t/onionsearch-a-script-that-scrapes-urls-on-different-onion-search-engines/193565 "2021-07-12T10:04:25Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![TheStrength](https://onehack.st/user_avatar/onehack.st/thestrength/32/46753_2.png) [@TheStrength](https://onehack.st/u/TheStrength)\
**Post date:** [July 12, 2021, 10:04am UTC](https://onehack.st/t/onionsearch-a-script-that-scrapes-urls-on-different-onion-search-engines/193565/1 "2021-07-12T10:04:25Z")

</div>

# OnionSearch

## Educational purposes only

[![forthebadge made-with-python](https://camo.githubusercontent.com/38f5db5524ba43e7262dfbca1f7d3631ba127fb1596785dfd707d5fc671821c9/687474703a2f2f466f7254686542616467652e636f6d2f696d616765732f6261646765732f6d6164652d776974682d707974686f6e2e737667)](https://www.python.org/)

OnionSearch is a Python3 script that scrapes urls on different “.onion” search engines.

[![](https://camo.githubusercontent.com/b5ff15b7d8670790776be311207f442e59a5993357aed39e2b4e4457a688c061/68747470733a2f2f66696c65732e636174626f782e6d6f652f7667757931652e706e67)](https://camo.githubusercontent.com/b5ff15b7d8670790776be311207f442e59a5993357aed39e2b4e4457a688c061/68747470733a2f2f66696c65732e636174626f782e6d6f652f7667757931652e706e67)

### Demo

[![](https://github.com/megadose/gif-demo/raw/master/onionsearch.gif)](https://github.com/megadose/gif-demo/raw/master/onionsearch.gif)

## 💡 Prerequisite

[Python 3](https://www.python.org/download/releases/3.0/)

## 📚 Currently supported Search engines

- ahmia
- darksearchio
- onionland
- notevil
- darksearchenginer
- phobos
- onionsearchserver
- torgle
- onionsearchengine
- tordex
- tor66
- tormax
- haystack
- multivac
- evosearch
- deeplink

## 🛠 Installation

### With PyPI

`pip3 install onionsearch`

### With Github

git clone [https://github.com/megadose/OnionSearch.git](https://github.com/megadose/OnionSearch.git) cd OnionSearch/ python3 setup.py install

## 📈 Usage

Help:

```auto
usage: onionsearch [-h] [--proxy PROXY] [--output OUTPUT]
                  [--continuous_write CONTINUOUS_WRITE] [--limit LIMIT]
                  [--engines [ENGINES [ENGINES ...]]]
                  [--exclude [EXCLUDE [EXCLUDE ...]]]
                  [--fields [FIELDS [FIELDS ...]]]
                  [--field_delimiter FIELD_DELIMITER] [--mp_units MP_UNITS]
                  search

positional arguments:
  search The search string or phrase

optional arguments:
  -h, --help show this help message and exit
  --proxy PROXY Set Tor proxy (default: 127.0.0.1:9050)
  --output OUTPUT Output File (default: output_$SEARCH_$DATE.txt), where $SEARCH is replaced by the first chars of the search string and $DATE is replaced by the datetime
  --continuous_write CONTINUOUS_WRITE
                        Write progressively to output file (default: False)
  --limit LIMIT Set a max number of pages per engine to load
  --engines [ENGINES [ENGINES ...]]
                        Engines to request (default: full list)
  --exclude [EXCLUDE [EXCLUDE ...]]
                        Engines to exclude (default: none)
  --fields [FIELDS [FIELDS ...]]
                        Fields to output to csv file (default: engine name link), available fields are shown below
  --field_delimiter FIELD_DELIMITER
                        Delimiter for the CSV fields
  --mp_units MP_UNITS Number of processing units (default: core number minus 1)

[...]

```

### Multi-processing behaviour

By default, the script will run with the parameter `mp_units = cpu_count() - 1`. It means if you have a machine with 4 cores, it will run 3 scraping functions in parallel. You can force `mp_units` to any value but it is recommended to leave to default. You may want to set it to 1 to run all requests sequentially (disabling multi-processing feature).

Please note that continuous writing to csv file has not been _heavily_ tested with multiprocessing feature and therefore may not work as expected.

Please also note that the progress bars may not be properly displayed when `mp_units` is greater than 1. **It does not affect the results** , so don’t worry.

### Examples

To request all the engines for the word “computer”:

```auto
onionsearch "computer"

```

To request all the engines excepted “Ahmia” and “Candle” for the word “computer”:

```auto
onionsearch "computer" --exclude ahmia candle

```

To request only “Tor66”, “DeepLink” and “Phobos” for the word “computer”:

```auto
onionsearch "computer" --engines tor66 deeplink phobos

```

The same as previously but limiting to 3 the number of pages to load per engine:

```auto
onionsearch "computer" --engines tor66 deeplink phobos --limit 3

```

Please kindly note that the list of supported engines (and their keys) is given in the script help (-h).

### Output

#### Default output

By default, the file is written at the end of the process. The file will be csv formatted, containing the following columns:

```auto
"engine","name of the link","url"

```

#### Customizing the output fields

You can customize what will be flush in the output file by using the parameters `--fields` and `--field_delimiter`.

`--fields` allows you to add, remove, re-order the output fields. The default mode is show just below. Instead, you can for instance choose to output:

```auto
"engine","name of the link","url","domain"

```

by setting `--fields engine name link domain`.

Or even, you can choose to output:

```auto
"engine","domain"

```

by setting `--fields engine domain`.

These are examples but there are many possibilities.

Finally, you can also choose to modify the CSV delimiter (comma by default), for instance: `--field_delimiter ";"`.

#### Changing filename

The filename will be set by default to `output_$DATE_$SEARCH.txt`, where $DATE represents the current datetime and $SEARCH the first characters of the search string.

You can modify this filename by using `--output` when running the script, for instance:

```auto
onionsearch "computer" --output "\$DATE.csv"
onionsearch "computer" --output output.txt
onionsearch "computer" --output "\$DATE_\$SEARCH.csv"
...

```

(Note that it might be necessary to escape the dollar character.)

In the csv file produced, the name and url strings are sanitized as much as possible, but there might still be some problems…

#### Write progressively

You can choose to progressively write to the output (instead of everything at the end, which would prevent losing the results if something goes wrong). To do so you have to use `--continuous_write True`, just as is:

```auto
onionsearch "computer" --continuous_write True

```

You can then use the `tail -f` (tail follow) Unix command to actively watch or monitor the results of the scraping.

## Thank you to [Gobarigo](https://github.com/Gobarigo)

## Thank you [mxrch](https://github.com/mxrch) for this logo

## 📝 License

[GNU General Public License v3.0](https://www.gnu.org/licenses/gpl-3.0.fr.html)

## GitHub:

> **[GitHub - megadose/OnionSearch: OnionSearch is a script that scrapes urls on...](https://github.com/megadose/OnionSearch)**
>
> OnionSearch is a script that scrapes urls on different .onion search engines.

---

<div class="post-metadata">

**Author:** ![D4663R](https://onehack.st/user_avatar/onehack.st/d4663r/32/96087_2.png) [@D4663R](https://onehack.st/u/D4663R)\
**Post date:** [July 24, 2021, 1:49am UTC](https://onehack.st/t/onionsearch-a-script-that-scrapes-urls-on-different-onion-search-engines/193565/2 "2021-07-24T01:49:44Z")

</div>

hey im new to this stuff, so do i have to install python 3.0.0 or can i install the latest one? well actually ive already tried both and whenever i try to open it it will show the black window then quickly exit out. whenever i spam it, it will stay opened till i try to type something. am i doing something wrong?