Page 1 of 1

download with wget serie of files

Posted: Thu Jan 27, 2022 3:29 am America/New_York
by sudre
Dear helpdesk,
After several years of not having any data on your site. I need to retrieve the data from MODIS-Aqua L3m_DAY_CHL_chlor_a_4km.nc. For that, I used the wget command:

wget --post-data="sensor=modisa&sdate=2019-01-01&edate=2022-01-01&dtype=L3m&addurl=1&results_as_file=1&search=A*L3m_DAY_CHL_chlor_a_4km.nc" -O - https://oceandata.sci.gsfc.nasa.gov/api/file_search.cgi | wget --user=XXX --password=YYY -i -
(where XXX is my login and YYY my password).

I tried yesterday to run this command again but the files I get are empty. Here are the responses to the command

--2022-01-27 09:21:44-- [link=]//oceandata.sci.gsfc.nasa.gov/api/file_search.cgi
Resolving proxy.legos.obs-mip.fr (proxy.legos.obs-mip.fr)... 10.10.0.1
Connecting to proxy.legos.obs-mip.fr (proxy.legos.obs-mip.fr)|10.10.0.1|:3128... connected.
Proxy request sent, awaiting response... 200 OK
Length: 92147 (90K) [text/plain]
Saving to: ‘STDOUT’

- 100%[==========================================================================================>] 89.99K 341KB/s in 0.3s

2022-01-27 09:21:46 (341 KB/s) - written to stdout [92147/92147]

--2022-01-27 09:21:46-- [url]https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/A2019001.L3m_DAY_CHL_chlor_a_4km.nc[/url]
Resolving proxy.legos.obs-mip.fr (proxy.legos.obs-mip.fr)... 10.10.0.1
Connecting to proxy.legos.obs-mip.fr (proxy.legos.obs-mip.fr)|10.10.0.1|:3128... connected.
Proxy request sent, awaiting response... 302 Found
Location: /ob/getfile/A2019001.L3m_DAY_CHL_chlor_a_4km.nc [following]
--2022-01-27 09:21:46-- https://oceandata.sci.gsfc.nasa.gov/ob/getfile/A2019001.L3m_DAY_CHL_chlor_a_4km.nc
Reusing existing connection to oceandata.sci.gsfc.nasa.gov:443.
Proxy request sent, awaiting response... 302 Found
Location: https://urs.earthdata.nasa.gov/oauth/authorize?client_id=Z0u-MdLNypXBjiDREZ3roA&redirect_uri=https%3A%2F%2Foceandata.sci.gsfc.nasa.gov%2Fob%2Fgetfile%2Frestrict&response_type=code [following]
--2022-01-27 09:21:46-- https://urs.earthdata.nasa.gov/oauth/authorize?client_id=Z0u-MdLNypXBjiDREZ3roA&redirect_uri=https%3A%2F%2Foceandata.sci.gsfc.nasa.gov%2Fob%2Fgetfile%2Frestrict&response_type=code
Connecting to proxy.legos.obs-mip.fr (proxy.legos.obs-mip.fr)|10.10.0.1|:3128... connected.
Proxy request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: ‘A2019001.L3m_DAY_CHL_chlor_a_4km.nc’

A2019001.L3m_DAY_CHL_chlor_a_4km.nc [ <=> ] 11.99K --.-KB/s in 0s

2022-01-27 09:21:47 (61.0 MB/s) - ‘A2019001.L3m_DAY_CHL_chlor_a_4km.nc’ saved [12275]

--2022-01-27 09:21:47-- https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/A2019002.L3m_DAY_CHL_chlor_a_4km.nc
...

could you please tell me what changes to make to my command to get the 3 years of data back.
thank you

Joel

Re: download with wget serie of files

Posted: Thu Jan 27, 2022 10:37 am America/New_York
by gnwiii
Have you reviewed the current https://oceancolor.gsfc.nasa.gov/data/download_methods/?

For me, wget is not reliable, but I have had better luck using the python script. One thing to try with wget is the option to use wget's "--adjust-extension" option to set the extension. If there is an issue with your login credentials (e.g., from your ".netrc" file) you will get a ".html" file containing the Earthdata login page.

In the past, at my work location, about 10% of downloaded files were corrupt (typically, blocks of null characters). I found it important to use the file search page to download checksums so I could detect and replace corrupt files before wasting time trying to process them.

Re: download with wget serie of files

Posted: Thu Feb 03, 2022 6:48 am America/New_York
by sudre
Thank you very much for your answer which helped me a lot!

Here is a command with wget that allows to retrieve a series of files with the new commands:
wget -q --post-data="sensor=modisa&sdate=2002-01-01&edate=2002-12-31&dtype=L3m&addurl=1&results_as_file=1&search=A*L3m.DAY.SST.4km.nc" -O - https://oceandata.sci.gsfc.nasa.gov/api/file_search | wget --user=<your_login> --ask-password --auth-no-challenge=on --wait=0.5 -i -

This works perfectly without download errors!