July 20

Linux: Download from a list of URLs

Downloading a list of files is very easy using wget.
1. Save your url list to a text file on in a folder of your choice
2. Type: wget –content-disposition –trust-server-names -i yoururllist.txt

You can download your files even faster using parallel processing.
1. Save your url list to a text file on in a folder of your choice
2. sudo apt-get parallel
3. parallel –gnu -a yoururllist.txt wget –content-disposition –trust-server-names

This will download as many of the files as it can at one time using all of your computers processor cores.
If you have a File Manager open it is pretty cool to see the files appearing at the same time.
If the server you are connecting to allows multiple connections from the same source, it is an incredibly fast way to download a list of files.

Credit: Tim Conrad


Copyright 2021. All rights reserved.

Posted July 20, 2016 by Timothy Conrad in category "Linux

About the Author

If I were to describe myself with one word it would be, creative. I am interested in almost everything which keeps me rather busy. Here you will find some of my technical musings. Securely email me using - PGP: 4CB8 91EB 0C0A A530 3BE9 6D76 B076 96F1 6135 0A1B