Go to ...

Techpository

A Better Technical Repository

RSS Feed

Linux: Download from a list of URLs


Downloading a list of files is very easy using wget.
1. Save your url list to a text file on in a folder of your choice
2. Type: wget –content-disposition –trust-server-names -i yoururllist.txt

You can download your files even faster using parallel processing.
1. Save your url list to a text file on in a folder of your choice
2. sudo apt-get parallel
3. parallel –gnu -a yoururllist.txt wget –content-disposition –trust-server-names

This will download as many of the files as it can at one time using all of your computers processor cores.
If you have a File Manager open it is pretty cool to see the files appearing at the same time.
If the server you are connecting to allows multiple connections from the same source, it is an incredibly fast way to download a list of files.

Credit: Tim Conrad