# Pagodo | Automate Google Hacking Database Scraping And Searching **URL:** \ **Category:** Tools & Scripts\ **Tags:** tools, programming, tips-tricks\ **Created:** [August 20, 2020, 2:49pm UTC](https://onehack.st/t/pagodo-automate-google-hacking-database-scraping-and-searching/102157 "2020-08-20T14:49:12Z")\ **Posts on this page:** 2\ **Page:** 1
**Author:** ![NoBody](https://onehack.st/user_avatar/onehack.st/nobody/32/163880_2.png) [@NoBody](https://onehack.st/u/NoBody)\ **Post date:** [August 20, 2020, 2:49pm UTC](https://onehack.st/t/pagodo-automate-google-hacking-database-scraping-and-searching/102157/1 "2020-08-20T14:49:12Z")
![image](https://onehack.st/uploads/default/original/3X/3/2/32c61b36029624a320feaba594b3b47fab4771b6.png) The goal of this project was to develop a passive Google dork script to collect potentially vulnerable web pages and applications on the Internet. There are 2 parts. The first is `ghdb_scraper.py` that retrieves Google Dorks and the second portion is `pagodo.py` that leverages the information gathered by `ghdb_scraper.py` . **What are Google Dorks?** The awesome folks at Offensive Security maintain the Google Hacking Database (GHDB) found here: [https://www.exploit-db.com/google-hacking-database](https://www.exploit-db.com/google-hacking-database). It is a collection of Google searches, called dorks, that can be used to find potentially vulnerable boxes or other juicy info that is picked up by Google’s search bots. **Installation** Scripts are written for Python 3.6+. Clone the git repository and install the requirements. ```auto git clone https://github.com/opsdisk/pagodo.gitcd pagodovirtualenv -p python3 .venv # If using a virtual environment.source .venv/bin/activate # If using a virtual environment.pip install -r requirements.txt ``` **Google is blocking me!** If you start getting HTTP 503 errors, Google has rightfully detected you as a bot and will block your IP for a set period of time. The solution is to use proxychains and a bank of proxies to round robin the lookups. Install proxychains4 ```auto apt install proxychains4 -y ``` Edit the `/etc/proxychains4.conf` configuration file to round robin the look ups through different proxy servers. In the example below, 2 different dynamic socks proxies have been set up with different local listening ports (9050 and 9051). Don’t know how to utilize SSH and dynamic socks proxies? Do yourself a favor and pick up a copy of [The Cyber Plumber’s Handbook](https://cph.opsdisk.com/) to learn all about Secure Shell (SSH) tunneling, port redirection, and bending traffic like a boss. ```auto vim /etc/proxychains4.conf ``` ```auto round_robinchain_len = 1proxy_dnsremote_dns_subnet 224tcp_read_time_out 15000tcp_connect_time_out 8000[ProxyList]socks4 127.0.0.1 9050socks4 127.0.0.1 9051 ``` Throw `proxychains4` in front of the Python script and each lookup will go through a different proxy (and thus source from a different IP). You could even tune down the `-e` delay time because you will be leveraging different proxy boxes. ```auto proxychains4 python3 pagodo.py -g ALL_dorks.txt -s -e 17.0 -l 700 -j 1.1 ``` **ghdb\_scraper.py** To start off, `pagodo.py` needs a list of all the current Google dorks. A datetimestamped file with the Google dorks and the indididual dork category dorks are also provided in the repo. Fortunately, the entire database can be pulled back with 1 GET request using `ghdb_scraper.py` . You can dump all dorks to a file, the individual dork categories to separate dork files, or the entire json blob if you want more contextual data about the dork. To retrieve all dorks ```auto python3 ghdb_scraper.py -j -s ``` To retrieve all dorks and write them to individual categories: ```auto python3 ghdb_scraper.py -i ``` Dork categories: ```auto categories = { 1: "Footholds", 2: "File Containing Usernames", 3: "Sensitives Directories", 4: "Web Server Detection", 5: "Vulnerable Files", 6: "Vulnerable Servers", 7: "Error Messages", 8: "File Containing Juicy Info", 9: "File Containing Passwords", 10: "Sensitive Online Shopping Info", 11: "Network or Vulnerability Data", 12: "Pages Containing Login Portals", 13: "Various Online devices", 14: "Advisories and Vulnerabilities", } ``` **pagodo.py** Now that a file with the most recent Google dorks exists, it can be fed into `pagodo.py` using the `-g` switch to start collecting potentially vulnerable public applications. `pagodo.py` leverages the `google` python library to search Google for sites with the Google dork, such as: ```auto intitle:"ListMail Login" admin -demo ``` The `-d` switch can be used to specify a domain and functions as the Google search operator: ```auto site:example.com ``` Performing ~4600 search requests to Google as fast as possible will simply not work. Google will rightfully detect it as a bot and block your IP for a set period of time. In order to make the search queries appear more human, a couple of enhancements have been made. A pull request was made and accepted by the maintainer of the Python `google` module to allow for User-Agent randomization in the Google search queries. This feature is available in [1.9.3](https://pypi.python.org/pypi/google) and allows you to randomize the different user agents used for each search. This emulates the different browsers used in a large corporate environment. The second enhancement focuses on randomizing the time between search queries. A minimum delay is specified using the `-e` option and a jitter factor is used to add time on to the minimum delay number. A list of 50 jitter times is created and one is randomly appended to the minimum delay time for each Google dork search. ```auto categories = { 1: "Footholds", 2: "File Containing Usernames", 3: "Sensitives Directories", 4: "Web Server Detection", 5: "Vulnerable Files", 6: "Vulnerable Servers", 7: "Error Messages", 8: "File Containing Juicy Info", 9: "File Containing Passwords", 10: "Sensitive Online Shopping Info", 11: "Network or Vulnerability Data", 12: "Pages Containing Login Portals", 13: "Various Online devices", 14: "Advisories and Vulnerabilities",} ``` Latter in the script, a random time is selected from the jitter array and added to the delay. ```auto intitle:"ListMail Login" admin -demo ``` Experiment with the values, but the defaults successfully worked without Google blocking my IP. Note that it could take a few days (3 on average) to run so be sure you have the time. To run it: ```auto site:example.com ``` **Conclusion** Comments, suggestions, and improvements are always welcome. Be sure to follow [@opsdisk](https://twitter.com/opsdisk) on Twitter for the latest updates. ## GitHub: > **[GitHub - opsdisk/pagodo: pagodo (Passive Google Dork) - Automate Google...](https://github.com/opsdisk/pagodo)** > > pagodo (Passive Google Dork) - Automate Google Hacking Database scraping and searching ---
**Author:** ![RTF](https://onehack.st/user_avatar/onehack.st/rtf/32/54011_2.png) [@RTF](https://onehack.st/u/RTF)\ **Post date:** [August 20, 2020, 2:52pm UTC](https://onehack.st/t/pagodo-automate-google-hacking-database-scraping-and-searching/102157/2 "2020-08-20T14:52:56Z")
this something juicy information. Thanks bro!