# How To Extract Wikipedia Data Using Python

**URL:** <https://onehack.st/t/how-to-extract-wikipedia-data-using-python/83579>\
**Category:** Tutorials & Methods\
**Tags:** ebook, programming\
**Created:** [April 30, 2020, 12:57pm UTC](https://onehack.st/t/how-to-extract-wikipedia-data-using-python/83579 "2020-04-30T12:57:03Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![NoBody](https://onehack.st/user_avatar/onehack.st/nobody/32/163880_2.png) [@NoBody](https://onehack.st/u/NoBody)\
**Post date:** [April 30, 2020, 12:57pm UTC](https://onehack.st/t/how-to-extract-wikipedia-data-using-python/83579/1 "2020-04-30T12:57:03Z")

</div>

I need to mention that we are not going to web scrape Wikipedia pages manually, [wikipedia](https://pypi.org/project/wikipedia/) module already did the tough work for us. Let’s install it:

```auto
pip3 install wikipedia

```

Open up a Python interactive shell or an empty file and follow along.

Let’s get the summary of what Python programming language is:

```auto
import wikipedia
# print the summary of what python is
print(wikipedia.summary("Python Programming Language"))

```

This will extract the summary from [this wikipedia page](https://en.wikipedia.org/wiki/Python_(programming_language)?ref=hackernoon.com). More specifically, it will print some first sentences, we can specify the number of sentences to extract:

```auto
In [2]: wikipedia.summary("Python programming languag", sentences=2)
Out[2]: "Python is an interpreted, high-level, general-purpose programming language. Created by Guido van Rossum and first released in 1991, Python's design philosophy emphasizes code readability with its notable use of significant whitespace."

```

Notice that I misspelled the query intentionally, it still gives me an accurate result.

Search for a term in wikipedia search:

```auto
In [3]: result = wikipedia.search("Neural networks")
In [4]: print(result)
['Neural network', 'Artificial neural network', 'Convolutional neural network', 'Recurrent neural network', 'Rectifier (neural networks)', 'Feedforward neural network', 'Neural circuit', 'Quantum neural network', 'Dropout (neural networks)', 'Types of artificial neural networks']

```

This returned a list of related page titles, let’s get the whole page for “Neural network” which is “result[0]”:

```auto
# get the page: Neural network
page = wikipedia.page(result[0])

```

Extracting the title:

```auto
# get the title of the page
title = page.title

```

Getting all the categories of that Wikipedia page:

```auto
# get the categories of the page
categories = page.categories

```

Extracting the text after removing all HTML tags (this is done automatically):

```auto
# get the whole wikipedia page text (content)
content = page.content

```

All links:

```auto
# get all the links in the page
links = page.links

```

The references:

```auto
# get the page references
references = page.references

```

Finally, the summary:

```auto
# summary
summary = page.summary

```

Let’s print them out:

```auto
# print info
print("Page content:\n", content, "\n")
print("Page title:", title, "\n")
print("Categories:", categories, "\n")
print("Links:", links, "\n")
print("References:", references, "\n")
print("Summary:", summary, "\n")

```

**Try it out!**

Alright, we are done, this was a brief introduction on how you can extract information from Wikipedia in Python. This can be helpful if you want to automatically collect data for language models, make a question answering chatbot, making a wrapper application around this, and much more! The possibilities are endless. _Source: hackernoon._

Here’s the Visual Edition of this Tutorial:

[![](https://img.youtube.com/vi/DVQlxBxp2zc/maxresdefault.jpg "How to Web Scrape Wikipedia - Be a WIKI POWER USER") ](https://www.youtube.com/watch?v=DVQlxBxp2zc)

enjoy!