# Goscrape | A Web Scraper Built With Golang

**URL:** <https://onehack.st/t/goscrape-a-web-scraper-built-with-golang/159563>\
**Category:** Tools & Scripts\
**Tags:** tools, programming\
**Created:** [October 20, 2020, 12:36pm UTC](https://onehack.st/t/goscrape-a-web-scraper-built-with-golang/159563 "2020-10-20T12:36:27Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![Prometheus](https://onehack.st/user_avatar/onehack.st/prometheus/32/34950_2.png) [@Prometheus](https://onehack.st/u/Prometheus)\
**Post date:** [October 20, 2020, 12:36pm UTC](https://onehack.st/t/goscrape-a-web-scraper-built-with-golang/159563/1 "2020-10-20T12:36:27Z")

</div>

![image](https://onehack.st/uploads/default/original/3X/f/1/f142d8905369206b2474441cdec0b1e777fb31d7.jpeg)

# goscrape [![Build Status](https://camo.githubusercontent.com/31fca82c39a4fafb094240af0cdbf739dff15e00/68747470733a2f2f7472617669732d63692e6f72672f636f726e656c6b2f676f7363726170652e7376673f6272616e63683d6d6173746572)](https://travis-ci.org/cornelk/goscrape) [![GoDoc](https://camo.githubusercontent.com/584f94f4191cc6879b05e1e2fd768e2365120606/68747470733a2f2f676f646f632e6f72672f6769746875622e636f6d2f636f726e656c6b2f676f7363726170653f7374617475732e737667)](https://godoc.org/github.com/cornelk/goscrape) [![Go Report Card](https://camo.githubusercontent.com/dc5c35dc321988f14acd138db56316e8b2d3a829/68747470733a2f2f676f7265706f7274636172642e636f6d2f62616467652f636f726e656c6b2f676f736372617065)](https://goreportcard.com/report/github.com/cornelk/goscrape) [![codecov](https://camo.githubusercontent.com/c40a950264dd652606c6a0be7ea94e9bdea31558/68747470733a2f2f636f6465636f762e696f2f67682f636f726e656c6b2f676f7363726170652f6272616e63682f6d61737465722f67726170682f62616467652e737667)](https://codecov.io/gh/cornelk/goscrape)

A web scraper built with Golang. It downloads the content of a website or blog and allows you to read it offline.

Features and advantages over existing tools like wget, httrack, Teleport Pro:

- Free and open source
- Available for all platforms that Golang supports
- JPEG and PNG images can be converted down in quality to save disk space
- Excluded URLS will not be fetched (unlike [wget](https://savannah.gnu.org/bugs/?20808))
- No incomplete temp files are left on disk
- Downloaded asset files are skipped in a new scraper run
- Assets from external domains are downloaded automatically
- Sane default values

Limitations:

- No GUI version, console only

## Installation

You need to have Golang installed, otherwise follow the guide at [https://golang.org/doc/install](https://golang.org/doc/install).

```auto
go get github.com/cornelk/goscrape

```

## Usage

```auto
goscrape http://website.com

```

## Options

```auto
Scrape a website and create an offline browsable version on the disk

Usage:
  goscrape http://website.com [flags]

Flags:
      --config string config file (default is $HOME/.goscrape.yaml)
  -d, --depth uint download depth, 0 for unlimited (default 10)
  -x, --exclude stringArray exclude URLs with PERL Regular Expressions support
  -h, --help help for goscrape
  -i, --imagequality int image quality, 0 to disable reencoding
  -n, --include stringArray only include URLs with PERL Regular Expressions support
  -o, --output string output directory to write files to
  -t, --timeout uint time limit in seconds for each http request to connect and read the request body
  -u, --user string user[:password] to use for authentication
  -v, --verbose verbose output

```

## Dependencies

- [github.com/gorilla/css](https://github.com/gorilla/css) css file tokenizer
- [github.com/h2non/filetype](https://github.com/h2non/filetype) image format identification
- [github.com/hashicorp/go-multierror](https://github.com/hashicorp/go-multierror) multi error wrapping
- [github.com/headzoo/surf](https://github.com/headzoo/surf) virtual web browser
- [github.com/PuerkitoBio/goquery](https://github.com/PuerkitoBio/goquery) HTML document traversal
- [github.com/spf13/cobra](https://github.com/spf13/cobra) command line handling
- [github.com/spf13/viper](https://github.com/spf13/viper) configuration
- [go.uber.org/zap](https://go.uber.org/zap) logging

## GitHub:

> **[GitHub - cornelk/goscrape: Web scraper that can create an offline readable...](https://github.com/cornelk/goscrape)**
>
> Web scraper that can create an offline readable version of a website

---

<div class="post-metadata">

**Author:** ![TelB](https://onehack.st/user_avatar/onehack.st/telb/32/75749_2.png) [@TelB](https://onehack.st/u/TelB)\
**Post date:** [October 20, 2020, 1:15pm UTC](https://onehack.st/t/goscrape-a-web-scraper-built-with-golang/159563/2 "2020-10-20T13:15:07Z")

</div>

The problem is with https sites which are now the default.  
In wget we have the --no-check-certificate option but nevertheless it remains ineffective for most https sites. From the options provided above, am I correct in assuming that Golang has the same difficulty?  
Cheers  
TelB