Unable to batch download with wget or curl
Posted: Mon Sep 19, 2022 4:26 pm America/New_York
Hello DAAC community,
I need to batch download MYD06_L2 and MYD03 files over a specific geographic region. I have done this many times in the past, but it's been about a year since my last attempt. My bearer token is currently valid. I am unable to get *any* scripts (either my own or the templates provided by LAADS DAAC tutorials) to complete a download.
Using the LAADS DAAC tutorial examples:
curl -v -H 'Authorization: Bearer $TOKEN' 'https://ladsweb.modaps.eosdis.nasa.gov/archive/allData/61/MYD03/2011/121/MYD03.A2011121.2100.061.2018029063924.hdf' > ./MYD03.A2011121.2100.061.2018029063924.hdf
or
wget -e robots=off -m -np -R .html,.tmp -nH --cut-dirs=3 "https://ladsweb.modaps.eosdis.nasa.gov/archive/allData/61/MYD03/2011/121/MYD03.A2011121.2100.061.2018029063924.hdf" --header "Authorization: Bearer $TOKEN" -P ./
I either get "504 Gateway Time-out" or "401 Username/Password Authentication failed" errors (only timeouts with curl, both errors with wget - first a 504 then 401). It appears that wget is redirected to an authentication link (https://ladsweb.modaps.eosdis.nasa.gov/oauth/login?redirect=...), so maybe there's an added authentication step that I'm missing? Any help here would be greatly appreciated...
Thanks,
Mikael
I need to batch download MYD06_L2 and MYD03 files over a specific geographic region. I have done this many times in the past, but it's been about a year since my last attempt. My bearer token is currently valid. I am unable to get *any* scripts (either my own or the templates provided by LAADS DAAC tutorials) to complete a download.
Using the LAADS DAAC tutorial examples:
curl -v -H 'Authorization: Bearer $TOKEN' 'https://ladsweb.modaps.eosdis.nasa.gov/archive/allData/61/MYD03/2011/121/MYD03.A2011121.2100.061.2018029063924.hdf' > ./MYD03.A2011121.2100.061.2018029063924.hdf
or
wget -e robots=off -m -np -R .html,.tmp -nH --cut-dirs=3 "https://ladsweb.modaps.eosdis.nasa.gov/archive/allData/61/MYD03/2011/121/MYD03.A2011121.2100.061.2018029063924.hdf" --header "Authorization: Bearer $TOKEN" -P ./
I either get "504 Gateway Time-out" or "401 Username/Password Authentication failed" errors (only timeouts with curl, both errors with wget - first a 504 then 401). It appears that wget is redirected to an authentication link (https://ladsweb.modaps.eosdis.nasa.gov/oauth/login?redirect=...), so maybe there's an added authentication step that I'm missing? Any help here would be greatly appreciated...
Thanks,
Mikael