How to download an entire directory and subdirectories using wget?

When using the wget command to download entire directories and subdirectories, leverage its recursive download functionality. Here are specific steps and examples:

1. Ensure you have permissions: Before proceeding, verify that you have access permissions to the target website directory.

2. Use the -r or --recursive option: This option enables wget to recursively download the directory, including all contents of the specified URL and its subdirectories.

3. Limit the download depth: If you do not wish to download multiple levels of subdirectories, limit the recursion depth using the -l or --level parameter. For example, -l 2 restricts wget to download only two levels of subdirectories under the target URL.

4. Use the -np or --no-parent option: This option prevents wget from navigating up to the parent directory to search for files.

5. Specify the local directory for downloaded files: Use the -P or --directory-prefix parameter to specify the directory where downloaded files will be stored.

Example Command

Assume you want to download a specific directory of a website along with all its subdirectories. Use the following command:

bash
wget -r -np -P /path/to/local/directory http://example.com/directory/

Here:

• -r indicates recursive download.
• -np prevents wget from navigating up to the parent directory.
• -P /path/to/local/directory specifies that the downloaded content will be stored in the local /path/to/local/directory directory.

Important Notes

• Ensure sufficient disk space is available, as recursive downloads may involve large amounts of content.
• Check the website's robots.txt file to confirm that the site permits such downloads.
• Consider using the -w (wait time) option to avoid excessive server load.

This command will help you efficiently download the website directory and its subdirectories to the specified local location.