# Load external data: Google Drive, Sheets, and Cloud Storage

**URL:** <https://onehack.st/t/load-external-data-google-drive-sheets-and-cloud-storage/73232>\
**Category:** Tools & Scripts\
**Created:** [March 13, 2020, 3:44pm UTC](https://onehack.st/t/load-external-data-google-drive-sheets-and-cloud-storage/73232 "2020-03-13T15:44:55Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![jmpinto](https://onehack.st/user_avatar/onehack.st/jmpinto/32/26546_2.png) [@jmpinto](https://onehack.st/u/jmpinto)\
**Post date:** [March 13, 2020, 3:44pm UTC](https://onehack.st/t/load-external-data-google-drive-sheets-and-cloud-storage/73232/1 "2020-03-13T15:44:55Z")

</div>

a. Load data from local file system  
Uploading files from the computer local file system

files.uploadreturn a dictionary of the files which were uploaded. The dictionary is keyed by the file name, the value is the data which was uploaded.  
from  
google.colab  
import  
files

uploaded = files.upload()

for  
fn  
in  
uploaded.keys():  
print(  
‘User uploaded file “{name}” with length {length} bytes’  
.format(name=fn, length=len(uploaded[fn])))

Resultado da execução: a sintaxe files.upload () fornece o upload de arquivos para o ambiente de execução da Colab

![](https://mmchiou.gitbooks.io/ai_challenge_taiwan_2018-private/content/assets/colab_load01_upload.png)

Downloading files to the computer local file system  
files.downloadwill invoke a browser downloaded of the file to user’s local computer.  
from  
google.colab  
import  
files

with  
open(  
‘example.txt’  
,  
‘w’  
)  
as  
f:  
f.write(  
‘some content’  
)

files.download(  
‘example.txt’  
)

Resultado da execução: Abra o arquivo “example.txt” e escreva uma linha “algum conteúdo” durante a execução. Por fim, baixe o arquivo no seu próprio computador.

![](https://mmchiou.gitbooks.io/ai_challenge_taiwan_2018-private/content/assets/colab_load02_download.png)

## b. Load data from Google Drive

Access files in Google Drive using the[native REST API](https://developers.google.com/drive/v3/web/about-sdk)or a wrapper like[PyDrive](https://github.com/gsuitedevs/PyDrive).

### PyDrive

The example below shows #1 authenticate, #2 file upload, and #3 file download from Google Drive. More examples are available in the[PyDrive documentation](https://gsuitedevs.github.io/PyDrive/docs/build/html/index.htmll)

!pip install -U -q PyDrive

from

pydrive.auth

import

GoogleAuth

from

pydrive.drive

import

GoogleDrive

from

google.colab

import

auth

from

oauth2client.client

import

GoogleCredentials

# 1. Authenticate and create the PyDrive client.

auth.authenticate\_user()

gauth = GoogleAuth()

gauth.credentials = GoogleCredentials.get\_application\_default()

drive = GoogleDrive(gauth)

# PyDrive reference:

# [https://gsuitedevs.github.io/PyDrive/docs/build/html/index.html](https://gsuitedevs.github.io/PyDrive/docs/build/html/index.html)

# 2. Create

&

upload a file text file.

uploaded = drive.CreateFile({

‘title’

:

‘Sample upload.txt’

})

uploaded.SetContentString(

‘Sample upload file content’

)

uploaded.Upload()

print(

‘Uploaded file with ID {}’

.format(uploaded.get(

‘id’

)))

# 3. Load a file by ID and print its contents.

downloaded = drive.CreateFile({

‘id’

: uploaded.get(

‘id’

)})

print(

‘Downloaded content “{}”’

.format(downloaded.GetContentString()))

Resultado da execução: durante o processo de execução, a conta do Google a ser usada será selecionada, o código de autenticação da conta será obtido de acordo com as orientações e será colado no programa

![](https://mmchiou.gitbooks.io/ai_challenge_taiwan_2018-private/content/assets/colab_load03_auth.png)

**Resultado da execução: gere um novo arquivo “Amostra upload.txt” no Google Drive**

![](https://mmchiou.gitbooks.io/ai_challenge_taiwan_2018-private/content/assets/colab_load03_auth_result.png)

**Resultado da execução: arquivo “Exemplo de upload.txt” no Google Drive**

![](https://mmchiou.gitbooks.io/ai_challenge_taiwan_2018-private/content/assets/colab_load04_new_file_drive.png)

### Drive REST API

Authentication is the first step

from

google.colab

import

auth

auth.authenticate\_user()

Then construct a Drive API client.

from

googleapiclient.discovery

import

build

drive\_service = build(

‘drive’

,

‘v3’

)

When client created, can use any of functions in the[Google Drive API reference](https://developers.google.com/drive/v3/reference/).

Create a new Google Drive file with data from Python

# Create a local file.

with

open(

‘/tmp/to\_upload.txt’

,

‘w’

)

as

f:

f.write(

‘my sample file’

)

# print out the file content

print(

‘/tmp/to\_upload.txt contains:’

)

!cat /tmp/to\_upload.txt

Resultados da implementação:

![](https://mmchiou.gitbooks.io/ai_challenge_taiwan_2018-private/content/assets/colab_load05_write_upload.png)

After executing the code above, a new file named “Sample file” will appear in[drive.google drive file list.

```auto
from
 googleapiclient.discovery 
import
 build
drive_service = build(
'drive'
, 
'v3'
)

# Upload the file to Drive. See:
#
# https://developers.google.com/drive/v3/reference/files/create
# https://developers.google.com/drive/v3/web/manage-uploads
from
 googleapiclient.http 
import
 MediaFileUpload

file_metadata = {

'name'
: 
'Sample file'
,

'mimeType'
: 
'text/plain'

}
media = MediaFileUpload(
'/tmp/to_upload.txt'
, 
                        mimetype=
'text/plain'
,
                        resumable=
True
)
created = drive_service.files().create(body=file_metadata,
                                       media_body=media,
                                       fields=
'id'
).execute()
print(
'File ID: {}'
.format(created.get(
'id'
)))

```

The file ID in execution result will different with the example code above.

Resultado de execução

![](https://mmchiou.gitbooks.io/ai_challenge_taiwan_2018-private/content/assets/colab_load06_create_file_to_drive.png)

### Downloading data from a Google Drive file into Python

Before execute the code below, change thefile\_idfield by id with the file in Google Drive.

# Download the file we just uploaded.

# 

# Replace the assignment below with your file ID

# to download a different file.

# 

# A file ID looks like: 1uBtlaggVyWshwcyP6kEI-y\_W3P8D26sz

file\_id =

‘target\_file\_id’

import

io

from

googleapiclient.http

import

MediaIoBaseDownload

request = drive\_service.files().get\_media(fileId=file\_id)

downloaded = io.BytesIO()

downloader = MediaIoBaseDownload(downloaded, request)

done =

False

while

done

is

False

:

# \_ is a placeholder for a progress object that we ignore.

# (Our file is small, so we skip reporting progress.)

\_, done = downloader.next\_chunk()

downloaded.seek(

0

)

print(

‘Downloaded file contents are: {}’

.format(downloaded.read()))

Resultados da implementação:

![](https://mmchiou.gitbooks.io/ai_challenge_taiwan_2018-private/content/assets/colab_load07_load_file_by_id.png)

c. Google Sheets  
d. Google Cloud Storages (GCS)  
Reference

> **[Google Colab](https://colab.research.google.com/notebook#fileId=/v2/external/notebooks/io.ipynb&scrollTo=KHeruhacFpSU)**

---

<div class="post-metadata">

**Author:** ![jmpinto](https://onehack.st/user_avatar/onehack.st/jmpinto/32/26546_2.png) [@jmpinto](https://onehack.st/u/jmpinto)\
**Post date:** [March 13, 2020, 5:44pm UTC](https://onehack.st/t/load-external-data-google-drive-sheets-and-cloud-storage/73232/2 "2020-03-13T17:44:23Z")

</div>

Hi, sorry all  
The links was working but I did not check them today, anyway, I already fixed dead links