# MetaDetective | Unleash Metadata Intelligence with MetaDetective **URL:** \ **Category:** Tools & Scripts\ **Tags:** hacking, data-recovery\ **Created:** [June 13, 2024, 7:29pm UTC](https://onehack.st/t/metadetective-unleash-metadata-intelligence-with-metadetective/284513 "2024-06-13T19:29:00Z")\ **Posts on this page:** 1\ **Page:** 1
**Author:** ![NoBody](https://onehack.st/user_avatar/onehack.st/nobody/32/163880_2.png) [@NoBody](https://onehack.st/u/NoBody)\ **Post date:** [June 13, 2024, 7:29pm UTC](https://onehack.st/t/metadetective-unleash-metadata-intelligence-with-metadetective/284513/1 "2024-06-13T19:29:00Z")
[![Contributors](https://camo.githubusercontent.com/c17ac958a5e2efb0c7aa7eb5a5c941def08c615837257fac7da3777894352615/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f636f6e7472696275746f72732f6672616e636b6665726d616e2f4d6574614465746563746976652e7376673f7374796c653d666f722d7468652d6261646765)](https://github.com/franckferman/MetaDetective/graphs/contributors) [![Forks](https://camo.githubusercontent.com/8d69c91a46f67cd5512caba60eb0f36f13187072483486b73511f54892d35b48/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f666f726b732f6672616e636b6665726d616e2f4d6574614465746563746976652e7376673f7374796c653d666f722d7468652d6261646765)](https://github.com/franckferman/MetaDetective/network/members) [![Stargazers](https://camo.githubusercontent.com/a8531126fb8ea542c090fbc6e484e5220f8100effe24597996109054a4ecf1e1/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f73746172732f6672616e636b6665726d616e2f4d6574614465746563746976652e7376673f7374796c653d666f722d7468652d6261646765)](https://github.com/franckferman/MetaDetective/stargazers) [![Issues](https://camo.githubusercontent.com/61f508941172f1865d8ccd6f3923a416de313e2e53a7d91dbc7e2237d262ecd6/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f6973737565732f6672616e636b6665726d616e2f4d6574614465746563746976652e7376673f7374796c653d666f722d7468652d6261646765)](https://github.com/franckferman/MetaDetective/issues) [![License](https://camo.githubusercontent.com/29334c861615f5a106f6cba5c719001ebdf9ff71bb4a3aa708bfe2c9c48298d0/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f6c6963656e73652f6672616e636b6665726d616e2f4d6574614465746563746976652e7376673f7374796c653d666f722d7468652d6261646765)](https://github.com/franckferman/MetaDetective/blob/stable/LICENSE) [![GitHub unittest Workflow Status](https://camo.githubusercontent.com/c97885b60e795db685102769f3acce979b31d1acfb2a9a84d67aa5a0ae11ae17/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f616374696f6e732f776f726b666c6f772f7374617475732f6672616e636b6665726d616e2f4d6574614465746563746976652f756e6974746573742e796d6c3f7374796c653d666f722d7468652d6261646765)](https://github.com/franckferman/MetaDetective/actions/workflows/unittest.yml) [![MetaDetective Logo|autoxauto](https://raw.githubusercontent.com/franckferman/MetaDetective/stable/docs/github/graphical_resources/Logo-Without_background-MetaDetective.png)](https://github.com/franckferman/MetaDetective) ### 🕵️‍♂️ MetaDetective _Unleash Metadata Intelligence with MetaDetective._ Bridging the chasm in metadata extraction and analysis. [**📘 Explore the full documentation**](https://github.com/franckferman/MetaDetective/blob/stable/README.md) · [🎥 View Demo](https://asciinema.org/a/55mEbe7GFVfIJ6OSfjOaDeYLv) · [🐞 Report Bug](https://github.com/franckferman/MetaDetective/issues) · [🛠 Request Feature](https://github.com/franckferman/MetaDetective/issues) MetaDetective\_promotional\_trailer.mp4 ## 📜 Table of Contents **Click to collapse/expand** 1. [📖 About](https://github.com/franckferman/MetaDetective#-about) 2. [🛠 Installation](https://github.com/franckferman/MetaDetective#-installation) 3. [🎮 Usage](https://github.com/franckferman/MetaDetective#-usage) 4. [❗ Troubleshooting](https://github.com/franckferman/MetaDetective#-troubleshooting) 5. [🤝 Contributing](https://github.com/franckferman/MetaDetective#-contributing) 6. [⚖ Legal Disclaimer](https://github.com/franckferman/MetaDetective#%EF%B8%8F-legal-disclaimer) 7. [🌠 Star Evolution](https://github.com/franckferman/MetaDetective#-star-evolution) 8. [📜 License](https://github.com/franckferman/MetaDetective#-license) 9. [📞 Contact](https://github.com/franckferman/MetaDetective#-contact) ## 📖 About **MetaDetective:** _Advanced metadata analysis and web scraping._ Metadata, in the realm of cybersecurity, is more than just embedded information; it’s a gateway to insightful perspectives, often unveiling crucial leads in OSINT and pentesting. As key tools like Metagoofil on Kali Linux shifted their trajectory away from pure metadata analysis, the exigency for a robust alternative took center stage. Enter **MetaDetective**. ### 🧠 Tailored Metadata Analysis Drawing inspiration from the foundational tools like Metagoofil, MetaDetective emerges as a revitalized and improved iteration, dedicated to providing efficient metadata extraction and presentation. It stands out as a comprehensive Python 3 tool, purposely designed to bridge the existing gaps in metadata analysis. ### 📊 Streamlined Data Presentation Beyond mere extraction, MetaDetective prides itself on its capability to meticulously categorize and showcase metadata. Whether dealing with an individual file or an array of them, the tool ensures users grasp the entire spectrum of data, both in its breadth and depth. [![MetaDetective Demo Screenshot|autoxauto](https://raw.githubusercontent.com/franckferman/MetaDetective/stable/docs/github/graphical_resources/Screenshot-MetaDetective_Demo.png)](https://raw.githubusercontent.com/franckferman/MetaDetective/stable/docs/github/graphical_resources/Screenshot-MetaDetective_Demo.png) ### 🌐 Web Scraping While Metagoofil once leaned on Google searches—a method riddled with IP restrictions and the labyrinth of proxy workarounds—MetaDetective pioneers a path with direct web scraping. By targeting sites directly, it sidesteps disruptions, delivering a dataset that’s not just richer, but also more precise, spotlighting potential data leaks. [![MetaDetective Scraping Demo Screenshot|autoxauto](https://raw.githubusercontent.com/franckferman/MetaDetective/stable/docs/github/graphical_resources/Screenshot-MetaDetective_Scraping_Demo.png)](https://raw.githubusercontent.com/franckferman/MetaDetective/stable/docs/github/graphical_resources/Screenshot-MetaDetective_Scraping_Demo.png) ### 🔍 Complementary Utility for OSINT and Pentesting Although it is now independent and offers its own functions, including scraping, MetaDetective isn’t just a standalone behemoth. It’s crafted for seamless integration and synergy with tools like Metagoofil. A quintessential addition to every pentester’s and OSINT researcher’s toolkit, MetaDetective magnifies data acquisition prowess and broadens the horizons of analysis. ([🔼 Back to top](https://github.com/franckferman/MetaDetective#top)) ## 🚀 Installation Before diving into the installation process, ensure you meet the following prerequisites. ### Prerequisites 1. **Python 3** : Ensure Python 3 is installed on your system before initiating the installation process. 2. **Exiftool** : Given its simplicity, MetaDetective doesn’t rely on any external dependencies or libraries. However, it does necessitate exiftool. Ensure you have exiftool set up on your system. > ⚠ **Note** : MetaDetective has been rigorously tested with Python 3.11.4 on Linux alongside exiftool version 12.56. While it may function with other versions, compatibility is guaranteed only with these specific configurations. ### Installation methods 1. **Git clone the repository** : git clone [https://github.com/franckferman/MetaDetective.git](https://github.com/franckferman/MetaDetective.git) 1. **Direct download** : To skip cloning and directly download the script (designed for simplicity and flexibility, it doesn’t depend on any external packages, so if you only need the script, you can also directly download it): curl -O [https://raw.githubusercontent.com/franckferman/MetaDetective/stable/src/MetaDetective/MetaDetective.py](https://raw.githubusercontent.com/franckferman/MetaDetective/stable/src/MetaDetective/MetaDetective.py) 1. **Pip Installation** : - Create & Activate a Virtual Environment: python3 -m venv MetaDetectiveEnv source MetaDetectiveEnv/bin/activate - Install MetaDetective: pip install MetaDetective 1. **Docker integration** : For a Docker-based setup, refer to our Docker-specific guide: [MetaDetective Docker Setup](https://github.com/franckferman/MetaDetective/blob/stable/docker/README.md). ([🔼 Back to top](https://github.com/franckferman/MetaDetective#top)) ## 🎮 Usage Ensure you adapt your command according to how you’ve set up `MetaDetective`. ### **Getting started** Kick off with the built-in help to explore MetaDetective’s functionalities: python3 src/MetaDetective/MetaDetective.py -h ### **Command examples** #### 🕵 File analysis: | Task | Command | | --- | --- | | Analyze all files in directory | `python3 src/MetaDetective/MetaDetective.py -d examples/` | | Specific types & ignore patterns | `python3 src/MetaDetective/MetaDetective.py -d examples/ -i ^admin anonymous -t doc pdf` | | Display all results for each file | `python3 src/MetaDetective/MetaDetective.py -d examples/ -t all --display all` | #### 🔎 Export function: | Task | Command | | --- | --- | | Default export (HTML) | `python3 src/MetaDetective/MetaDetective.py -d examples/ --export` | | Formatted display, txt export | `python3 src/MetaDetective/MetaDetective.py -d examples ---format formatted -e txt -o ~/` | #### 🌐 Web Scraping: | Task | Command | | --- | --- | | Scan without downloading | `python3 src/MetaDetective/MetaDetective.py --scraping --scan --url https://example.com/` | | Scan without downloading PDF files only | `python3 src/MetaDetective/MetaDetective.py --scraping --scan --url https://example.com/ --extensions pdf` | | Download to specified directory | `python3 src/MetaDetective/MetaDetective.py --scraping --download-dir ~ --url https://example.com/` | | Download with set depth | `python3 src/MetaDetective/MetaDetective.py --scraping --depth 1 --download-dir ~ --url https://example.com/` | ### **Additional parameters** #### 🌐 Web Scraping: To initiate the web scraping mode, use the `--scraping` flag. Remember, this option doesn’t function independently. It requires either a scanning or downloading parameter. - **Activating web scraping mode** : python3 src/MetaDetective/MetaDetective.py --scraping - **Scanning and displaying statistics** : Ensure both the URL and `--scan` flags are used. python3 src/MetaDetective/MetaDetective.py --scraping --scan --url [https://example.com](https://example.com) - **Scans for specific file types and displays related statistics** : Ensure both the URL and `--scan` flags are used and use `--extensions` followed by a list of desired file types (e.g., pdf docx png). python3 src/MetaDetective/MetaDetective.py --scraping --scan --url [https://example.com](https://example.com) --extensions pdf docx xlsx pptx - **Downloading web content** : Indicate the desired directory using `--download-dir` and provide the target URL. python3 src/MetaDetective/MetaDetective.py --scraping --download-dir ~ --url [https://example.com](https://example.com) - **Downloads web content of specific file types** : Indicate the desired directory using `--download-dir`, provide the target URL and desired file types with `--extensions`. python3 src/MetaDetective/MetaDetective.py --scraping --download-dir ~ --url [https://example.com](https://example.com) --extensions pdf docx xlsx pptx - **Adjusting scraping depth** : Use the `--depth` flag to specify how deeply the scraper should navigate through links. python3 src/MetaDetective/MetaDetective.py --scraping --scan --url [https://example.com](https://example.com) --depth 1 ##### **Additional Flags** : - **External link tracking** : Use `--follow-extern` to allow tracking of external links (those outside the base URL). Typically not advised, but might be useful in certain contexts. - **Thread management** : Use `--threads` to specify the number of threads for concurrent operations. - **Rate limiting** : Use `--rate` to control the maximum number of requests per second. #### 🕵 File analysis & Metadata Analyzer: ##### **Basic Commands** : To begin analyzing files, you’ll use either the `-d` or `-f` flag. - `-d` or `--directory`: Select a directory containing one or multiple files. - `-f` or `--files`: Choose a single or multiple specific files. Analyze the contents of a directory. python3 src/MetaDetective/MetaDetective.py -d examples Analyze the contents of a file. python3 src/MetaDetective/MetaDetective.py -f examples/MetaDetective.docx Analyze the contents of multiple files. python3 src/MetaDetective/MetaDetective.py -f examples/MetaDetective-APTX\_4869\_report.pdf examples/MetaDetective-Kogoro\_s\_Choice.pdf ##### **Specifying data type** You can filter to analyze specific file types: | Task | Command | | --- | --- | | Specify a data type | `python3 src/MetaDetective/MetaDetective.py -d directory -t pdf` | | Add multiple data types | `python3 src/MetaDetective/MetaDetective.py -d directory -t pdf doc` | | Include all types | `python3 src/MetaDetective/MetaDetective.py -d directory -t all` | ##### **Ignoring specific results** : If you want to omit specific keywords from the displayed metadata, use the `-i` or `--ignore` flag. For instance, you might want to exclude common usernames like “admin” during the reconnaissance phase of your pentest. Regex patterns are supported, e.g., `^BeginBy`. | Task | Command | | --- | --- | | Exclude specific results | `python3 src/MetaDetective/MetaDetective.py -d directory -i anonymous` | | Exclude multiple terms | `python3 src/MetaDetective/MetaDetective.py -d directory -i anonymous admin administrateur` | | Regex exclusions | `python3 src/MetaDetective/MetaDetective.py -d directory -i anonymous ^admin` | ##### **Display options** Adapt the display of your results to suit your preferences: | Task | Command | | --- | --- | | Show each file’s metadata | `python3 src/MetaDetective/MetaDetective.py --display all` | | Singular results without duplicates | `python3 src/MetaDetective/MetaDetective.py --display singular` | ##### **Format options** Modify your display further with these: | Task | Command | | --- | --- | | Stylish display | `python3 src/MetaDetective/MetaDetective.py --display all --format formatted` | | Simpler look | `python3 src/MetaDetective/MetaDetective.py --display all --format concise` | #### 🔎 **Export options** MetaDetective provides flexibility in exporting analysis results. By default, using the `--export` or `-e` option will save your results in an HTML format. This design ensures a visually appealing report for your analysis. If you prefer a `.txt` format, that’s possible too. Switch between formats using the `-e` or `--export` flag followed by the desired format: `-e txt` or `-e pdf`. The export will, by default, use a predefined name appended with a timestamp. To customize this name, you can append a suffix using the `-c` or `--custom` flag. Further, the `--out` or `-o` argument lets you specify the directory path for your exported data. **Be aware** : The `display` and `format` options, as previously discussed, will influence the presentation of your exported document, whether in HTML or TXT format. Data representation might differ between the two formats. | Task | Description | Command | | --- | --- | --- | | HTML Export (Default) | Produces an HTML file named: `MetaDetective_Export-.html`. | `python3 src/MetaDetective/MetaDetective.py -d directory -e` | | TXT Format Export | Save results in TXT format. | `python3 src/MetaDetective/MetaDetective.py -d directory --export txt` | | Custom Filename Suffix | Add a custom suffix to the filename. | `python3 src/MetaDetective/MetaDetective.py -d directory -e --custom Pentest-MD_2` | | Specify Output Directory | Define the directory for data export. | `python3 src/MetaDetective/MetaDetective.py -d directory -e -o directory` | [![MetaDetective HTML Export Demo Screenshot|autoxauto](https://raw.githubusercontent.com/franckferman/MetaDetective/stable/docs/github/graphical_resources/Screenshot-MetaDetective_HTML_Export_Demo.png)](https://raw.githubusercontent.com/franckferman/MetaDetective/stable/docs/github/graphical_resources/Screenshot-MetaDetective_HTML_Export_Demo.png) **Note** : The export format can greatly affect data presentation and accessibility. Opt for the format that aligns with your requirements. ([🔼 Back to top](https://github.com/franckferman/MetaDetective#top)) ## 🔧 Troubleshooting Encountering issues? Don’t worry. If you come across any problems or have questions, please don’t hesitate to submit a ticket for assistance: [Submit an issue on GitHub](https://github.com/franckferman/MetaDetective/issues) ([🔼 Back to top](https://github.com/franckferman/MetaDetective#top)) ## 🤝 Contributing We truly appreciate and welcome community involvement. Your contributions, feedback, and suggestions play a crucial role in improving the project for everyone. If you’re interested in contributing or have ideas for enhancements, please feel free to open an issue or submit a pull request on our GitHub repository. Every contribution, no matter how big or small, is highly valued and greatly appreciated! ([🔼 Back to top](https://github.com/franckferman/MetaDetective#top)) ## ⚖ Legal Disclaimer Please be aware that the use of the `MetaDetective` tool may be subject to specific laws and regulations in your country or region. Before using this tool, it is your responsibility to ensure that its use is in compliance with the laws applicable in your jurisdiction. The creator of `MetaDetective` cannot be held responsible for any improper or illegal use of this tool. By using `MetaDetective`, you acknowledge and agree that you are solely responsible for complying with local laws and regulations. In some instances, the use of `MetaDetective` might be justified for legitimate purposes such as protection against authoritarian regimes or for survival matters. However, it is strongly advised to take all necessary measures to protect yourself and to abide by the law. By using `MetaDetective`, you acknowledge having read this disclaimer and agree to act accordingly. The creator of `MetaDetective` does not in any way endorse the use of this tool for illegal or harmful purposes. ([🔼 Back to top](https://github.com/franckferman/MetaDetective#top)) ## 🌠 Star Evolution Explore the star history of this project and see how it has evolved over time: [![Star History Chart](https://camo.githubusercontent.com/11500ae89942770c9e0082db0c387c4c42d3f82315b10fb9037a4be7658e315a/68747470733a2f2f6170692e737461722d686973746f72792e636f6d2f7376673f7265706f733d6672616e636b6665726d616e2f4d65746144657465637469766526747970653d54696d656c696e65)](https://star-history.com/#franckferman/MetaDetective&Timeline) Your support is greatly appreciated. We’re grateful for every star! Your backing fuels our passion. ✨ ## 📚 License This project is licensed under the GNU Affero General Public License, Version 3.0. For more details, please refer to the LICENSE file in the repository: [Read the license on GitHub](https://github.com/franckferman/MetaDetective/blob/stable/LICENSE) ## GitHub: [https://github.com/franckferman/MetaDetective](https://github.com/franckferman/MetaDetective)