Downloading a Billion Files in Python
[Pages:150]Downloading a Billion Files in Python
A case study in multi-threading, multi-processing, and asyncio
James Saryerwinnie
@jsaryer
Our Task
Our Task
There is a remote server that stores files
Our Task
There is a remote server that stores files The files can be accessed through a REST API
Our Task
There is a remote server that stores files The files can be accessed through a REST API Our task is to download all the files on the remote server to our client machine
Our Task (the details)
Our Task (the details)
What client machine will this run on?
Our Task (the details)
What client machine will this run on? We have one machine we can use, 16 cores, 64GB memory
................
................
In order to avoid copyright disputes, this page is only a partial summary.
To fulfill the demand for quickly locating and searching documents.
It is intelligent file search solution for home and business.
Related searches
- python import files in folder
- python get files in folder
- python find files in directory
- python read all files in dir
- python read all files in directory
- python get all files in dir
- python read files in directory
- python read files in dir
- python list all files in dir
- python remove files in folder
- python list of files in dir
- python list files in subdirectory