# Recursive GoIndex Downloader: Download all files in folders and sub-folders for GoIndex Website **URL:** \ **Category:** Tools & Scripts\ **Created:** [April 14, 2020, 10:33pm UTC](https://onehack.st/t/recursive-goindex-downloader-download-all-files-in-folders-and-sub-folders-for-goindex-website/79823 "2020-04-14T22:33:29Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![atlonxp](https://onehack.st/user_avatar/onehack.st/atlonxp/32/38549_2.png) [@atlonxp](https://onehack.st/u/atlonxp)\ **Post date:** [April 14, 2020, 10:33pm UTC](https://onehack.st/t/recursive-goindex-downloader-download-all-files-in-folders-and-sub-folders-for-goindex-website/79823/1 "2020-04-14T22:33:29Z")
# Description This source code was developed and improved based on the script posted by _pankaj260_ (Thanks to _pankaj260_ [this post](https://onehack.st/t/script-download-all-the-files-in-a-directory-from-goindex-websites/79800)) The main features are that - Recursive crawler (atlonxp) - Download all folders and files in a given url (atlonxp) - Adaptive delay in fetching url (atlonxp) - Store folders/files directly to your Google Drive (pankaj260) - Folders and files exclusion filters # Instructions/Installation/Usage > **[GitHub - atlonxp/recursive-goIndex-downloader: Simple GoIndex Downloader](https://github.com/atlonxp/recursive-goIndex-downloader)** > > Simple GoIndex Downloader Go to my colab: [https://colab.research.google.com/drive/1VpAOxP\_sM6bTmRM8xmrmbowznZP-mX4u](https://colab.research.google.com/drive/1VpAOxP_sM6bTmRM8xmrmbowznZP-mX4u) > # Disclaimer > > This post is provided to help you and for personal use only. Sharing the content of your subscribed materials and the other purchased content is strictly prohibited under 1Hack Terms of Use. > > By using this provided material the website [1Hack.us](http://1Hack.us) is not responsible for any law infringement caused by the users of this material. ---
**Author:** ![Waga](https://onehack.st/user_avatar/onehack.st/waga/32/27787_2.png) [@Waga](https://onehack.st/u/Waga)\ **Post date:** [April 14, 2020, 11:50pm UTC](https://onehack.st/t/recursive-goindex-downloader-download-all-files-in-folders-and-sub-folders-for-goindex-website/79823/2 "2020-04-14T23:50:10Z")
Great work guys. Keep 'em coming. πŸ‘Œ πŸ‘ Cheers!!! > **… And Over Every Possessor of Knowledge, There is (Some) One (Else) More Knowledgeable.** ---
**Author:** ![atlonxp](https://onehack.st/user_avatar/onehack.st/atlonxp/32/38549_2.png) [@atlonxp](https://onehack.st/u/atlonxp)\ **Post date:** [April 15, 2020, 11:04am UTC](https://onehack.st/t/recursive-goindex-downloader-download-all-files-in-folders-and-sub-folders-for-goindex-website/79823/3 "2020-04-15T11:04:50Z")
# General instruction 1. open my code in Google Colab 2. run Section 1, it will ask you to connect to you Google Drive, and enter an authorization code 3. login with you google drive, you will get the authorization code 4. copy the authorization code, and paste in Step 2 5. in the left sidebar, open the file structure (folder icon), you will see your Google Drive here 6. find your folder to you want put all downloading fils e.g. β€œ/content/drive/Shared drives/{your team drive}” 7. right click on folder you want and copy path 8. In Section 6, edit destination = β€œ{from Step 7}” 9. In Section 6, edit download\_url = β€œ{the GoIndex website}” 10. run Section 2-6 11. done # Note This code is just for β€œedu.tuts.workers.dev”. It is just a concept, you will need to work a little bit if you want to use this on other websites. ---
**Author:** ![atlonxp](https://onehack.st/user_avatar/onehack.st/atlonxp/32/38549_2.png) [@atlonxp](https://onehack.st/u/atlonxp)\ **Post date:** [April 15, 2020, 6:21pm UTC](https://onehack.st/t/recursive-goindex-downloader-download-all-files-in-folders-and-sub-folders-for-goindex-website/79823/4 "2020-04-15T18:21:37Z")
# Update **New features** - Download queue supported - Auto-domain URL detection **Coming soon** - TDQM multiple/paralelled downloader - Aria2 integration Please use this link instead [https://github.com/atlonxp/recursive-goIndex-downloader](https://github.com/atlonxp/recursive-goIndex-downloader) ---
**Author:** ![cool\_ahmed](https://onehack.st/user_avatar/onehack.st/cool_ahmed/32/18291_2.png) [@cool\_ahmed](https://onehack.st/u/cool_ahmed)\ **Post date:** [April 16, 2020, 7:22am UTC](https://onehack.st/t/recursive-goindex-downloader-download-all-files-in-folders-and-sub-folders-for-goindex-website/79823/5 "2020-04-16T07:22:55Z")
how do we download a Udacity course with this tool? could you please guide us. ---
**Author:** ![atlonxp](https://onehack.st/user_avatar/onehack.st/atlonxp/32/38549_2.png) [@atlonxp](https://onehack.st/u/atlonxp)\ **Post date:** [April 16, 2020, 10:35am UTC](https://onehack.st/t/recursive-goindex-downloader-download-all-files-in-folders-and-sub-folders-for-goindex-website/79823/6 "2020-04-16T10:35:45Z")
I believe the instruction is given, please take some effort ---
**Author:** ![atlonxp](https://onehack.st/user_avatar/onehack.st/atlonxp/32/38549_2.png) [@atlonxp](https://onehack.st/u/atlonxp)\ **Post date:** [April 16, 2020, 6:54pm UTC](https://onehack.st/t/recursive-goindex-downloader-download-all-files-in-folders-and-sub-folders-for-goindex-website/79823/7 "2020-04-16T18:54:18Z")
Version 2 is just added > **[Google Colab](https://colab.research.google.com/drive/1P9R82sEpOuOK27ZGcZ93x2JnfdC35Eif)** If you wanna help me out improving this code, go to my Github [https://github.com/atlonxp/recursive-goIndex-downloader](https://github.com/atlonxp/recursive-goIndex-downloader) **Features** - Recursive crawler (atlonxp) - Download all folders and files in a given url (atlonxp) - Download all folders and files in in sub-folders (atlonxp) - Adaptive delay in fetching url (atlonxp) - Store folders/files directly to your Google Drive (pankaj260) - Folders and files exclusion filters - Download queue supported - Auto-domain URL detection - API-based GoIndex crawler - Parallel/Multiple files downloader **Version 2** : ``` 16 April 2020 + crawler_v2: * API-based GoIndex crawler * Collecting all urls to be downloaded + parallel downloader * TDQM progress bar ``` ---
**Author:** ![MikeBlueberry](https://onehack.st/user_avatar/onehack.st/mikeblueberry/32/34516_2.png) [@MikeBlueberry](https://onehack.st/u/MikeBlueberry)\ **Post date:** [April 16, 2020, 7:13pm UTC](https://onehack.st/t/recursive-goindex-downloader-download-all-files-in-folders-and-sub-folders-for-goindex-website/79823/9 "2020-04-16T19:13:46Z")
May the blood of 1000 virgins bless upon you and your descendants. ---
**Author:** ![ShivamSharma](https://onehack.st/user_avatar/onehack.st/shivamsharma/32/35959_2.png) [@ShivamSharma](https://onehack.st/u/ShivamSharma)\ **Post date:** [April 22, 2020, 8:55am UTC](https://onehack.st/t/recursive-goindex-downloader-download-all-files-in-folders-and-sub-folders-for-goindex-website/79823/11 "2020-04-22T08:55:28Z")
**Can we use this script to copy shared folder and its content of google drive (not goindex) to our own google drive?** ---
**Author:** ![atlonxp](https://onehack.st/user_avatar/onehack.st/atlonxp/32/38549_2.png) [@atlonxp](https://onehack.st/u/atlonxp)\ **Post date:** [April 22, 2020, 10:42am UTC](https://onehack.st/t/recursive-goindex-downloader-download-all-files-in-folders-and-sub-folders-for-goindex-website/79823/12 "2020-04-22T10:42:16Z")
It is possibility. if you look at the script carefully you will fine the solution, there is an implication if you use some codes here, you can do that πŸ˜ƒ ---
**Author:** ![Piyush\_Raut](https://onehack.st/user_avatar/onehack.st/piyush_raut/32/36598_2.png) [@Piyush\_Raut](https://onehack.st/u/Piyush_Raut)\ **Post date:** [April 22, 2020, 11:20am UTC](https://onehack.st/t/recursive-goindex-downloader-download-all-files-in-folders-and-sub-folders-for-goindex-website/79823/13 "2020-04-22T11:20:39Z")
I too have created similar script to download complete site or any course from go index. > **[GitHub - piyush2947/GoIndex-Downloader: Google Colab Project to download GoIndex...](https://github.com/piyush2947/GoIndex-Downloader)** > > Google Colab Project to download GoIndex Repository ---
**Author:** ![atlonxp](https://onehack.st/user_avatar/onehack.st/atlonxp/32/38549_2.png) [@atlonxp](https://onehack.st/u/atlonxp)\ **Post date:** [April 23, 2020, 7:23pm UTC](https://onehack.st/t/recursive-goindex-downloader-download-all-files-in-folders-and-sub-folders-for-goindex-website/79823/14 "2020-04-23T19:23:20Z")
[https://github.com/atlonxp/recursive-goIndex-downloader](https://github.com/atlonxp/recursive-goIndex-downloader) ```auto 22 April 2020 (v2.3.2) --------------------- + added summary + added Exception when file is unable to download 21 Aprial 2020 (v2.3.1) --------------------- While crawling, fetching might cause errors sometime due to some quick requests or server is busy. This problem has caused the eror in getting a json, so we re-fetch the url again (up to MAX_RETRY_CRAWLING) or until we found key "files" in the return response. Once retries is reached the maximum and the key "files" is not found, so we ignore this link (return []) At the end, if you find there is failure, just re-run the download section again. Unless you set OVERWITE = TRUE, all files will be re-downloaded + added MAX_RETRY_CRAWLING (v2.3) + fixed FILE_EXISTING_CHECK (stupid) bug + added failure-links download task 20 Aprial 2020 (v2.2) --------------------- Some sub-folders may be password-protected which will cause the error while crawling, so we skip this folder + added auto-skip password-protected folder 17 April 2020 (v2.1) --------------------- + fixed URL duplicated when crawling + added search 'files' key for some websites do not have proper files structure. So, we search it\ 16 April 2020 (v2.0) --------------------- + crawler_v2: * API-based GoIndex crawler * Collecting all urls to be downloaded + parallel downloader * TDQM progress bar ``` ---
**Author:** ![pheno](https://onehack.st/user_avatar/onehack.st/pheno/32/20142_2.png) [@pheno](https://onehack.st/u/pheno)\ **Post date:** [April 24, 2020, 9:33am UTC](https://onehack.st/t/recursive-goindex-downloader-download-all-files-in-folders-and-sub-folders-for-goindex-website/79823/15 "2020-04-24T09:33:55Z")
Thank you but where are those goindex site? ---
**Author:** ![atlonxp](https://onehack.st/user_avatar/onehack.st/atlonxp/32/38549_2.png) [@atlonxp](https://onehack.st/u/atlonxp)\ **Post date:** [April 24, 2020, 9:56am UTC](https://onehack.st/t/recursive-goindex-downloader-download-all-files-in-folders-and-sub-folders-for-goindex-website/79823/16 "2020-04-24T09:56:43Z")
interesting! so where are they? πŸ€” ---
**Author:** ![pheno](https://onehack.st/user_avatar/onehack.st/pheno/32/20142_2.png) [@pheno](https://onehack.st/u/pheno)\ **Post date:** [April 24, 2020, 10:30am UTC](https://onehack.st/t/recursive-goindex-downloader-download-all-files-in-folders-and-sub-folders-for-goindex-website/79823/17 "2020-04-24T10:30:13Z")
look like your laughting about my zero level πŸ˜” The problem is that i m not english native. I don’t know very well english. So please tell me the goindex site are where ? by googling this is a gh link [https://index.gd.workers.dev/](https://index.gd.workers.dev/) but they ask for password ---
**Author:** ![atlonxp](https://onehack.st/user_avatar/onehack.st/atlonxp/32/38549_2.png) [@atlonxp](https://onehack.st/u/atlonxp)\ **Post date:** [April 24, 2020, 10:59am UTC](https://onehack.st/t/recursive-goindex-downloader-download-all-files-in-folders-and-sub-folders-for-goindex-website/79823/18 "2020-04-24T10:59:32Z")
> look like your laughting about my zero level My apology if you think I was laughing, actually I am not laughing. It is my way to teach you guys. Every time I replied, I always convince you guys to - do some research yourself at the beginning before asking or bothering the others. - guide or teach, not to give solutions. (these 2 have completely definitions) > Give a man a fish, and you feed him for a day. Teach a man to fish, and you feed him for a lifetime. > interesting! so where are they? πŸ€” If you look carefully in my script, you would find out immediately what is the GoIndex site. FYI, once I joined this community I had no idea as well what is the GoIndex website. > by googling this is a gh link [https://index.gd.workers.dev/](https://index.gd.workers.dev/) See, you find the answer yourself πŸ˜ƒ πŸ‘ > but they ask for password Not as always. Some GoIndex website (hint _free courses_ or in my script) are not password-protected . Note that if you want to download files from password-protected folders, you need to use different solution (probably version 1 + amendments + password from localstorage). ---
**Author:** ![pheno](https://onehack.st/user_avatar/onehack.st/pheno/32/20142_2.png) [@pheno](https://onehack.st/u/pheno)\ **Post date:** [April 27, 2020, 9:12am UTC](https://onehack.st/t/recursive-goindex-downloader-download-all-files-in-folders-and-sub-folders-for-goindex-website/79823/19 "2020-04-27T09:12:09Z")
Thank you dear. I was in a website called coursehunter and after a moment they change all their rules. there were no more free courses after a month. and you have to pay for subscription. Now I see lot of their course in google shared drive on 1h by some members …by googling i found a site webpremium and when i try to download it s impossible and just stream on the url like lol.coronavirus.worker.dev/coursestitle or worker.dev/coursetitle. When I compare those addresses with the one in your script i make conclusion that goindex is in relation with free access to paied services cause of the covid-19 pandemic ---
**Author:** ![atlonxp](https://onehack.st/user_avatar/onehack.st/atlonxp/32/38549_2.png) [@atlonxp](https://onehack.st/u/atlonxp)\ **Post date:** [April 27, 2020, 12:01pm UTC](https://onehack.st/t/recursive-goindex-downloader-download-all-files-in-folders-and-sub-folders-for-goindex-website/79823/20 "2020-04-27T12:01:04Z")
Well-done! There so many people downloaded and uploaded those online. FYI, one has created Coursehunter downloader > [@\[GET\] CourseHunters Batch Downloader \[v 1.1\] \[Beta Testing\]](https://onehack.st/t/get-coursehunters-batch-downloader-v-1-1-beta-testing/13921): > > CourseHunters Batch Downloader This script will allow you to download courses from coursehunters in a way you always wanted. It will structure all lessons and give them names and then you can download. Screens [image] [image] Features Download all lesson from any course. Download Using IDM Structures course and filters generic words. Renames videos to lesson names and number them. Courtesy of @indi_crackerThis is just a way I wanted to thank FTU. Video Tutorial for Newbies credits @Dra … ---
**Author:** ![pheno](https://onehack.st/user_avatar/onehack.st/pheno/32/20142_2.png) [@pheno](https://onehack.st/u/pheno)\ **Post date:** [April 27, 2020, 4:31pm UTC](https://onehack.st/t/recursive-goindex-downloader-download-all-files-in-folders-and-sub-folders-for-goindex-website/79823/21 "2020-04-27T16:31:26Z")
it’s for free course i think ---
**Author:** ![Kumar\_P](https://onehack.st/letter_avatar_proxy/v4/letter/k/8c91f0/32.png) [@Kumar\_P](https://onehack.st/u/Kumar_P)\ **Post date:** [May 30, 2020, 9:59am UTC](https://onehack.st/t/recursive-goindex-downloader-download-all-files-in-folders-and-sub-folders-for-goindex-website/79823/22 "2020-05-30T09:59:38Z")
MAX\_DOWNLOAD\_TASKS = 32 pool = multiprocessing.Pool(processes=MAX\_DOWNLOAD\_TASKS) # Num of CPUs Hi brother, I have a doubt The google colab free version has only 2 cores by default but in this code it is set 32, could you explain why you chose 32 for the MAX\_DOWNLOAD\_TASKS ? [Next page](https://onehack.st/t/recursive-goindex-downloader-download-all-files-in-folders-and-sub-folders-for-goindex-website/79823.md?page=2)