Earthdata Forum

I have been encountering data download problems since January 15, 2020. I used to be able to download a list of files (up to 5000 I think) without any problem using the following command:
curl --interface 2607:fe50:0:6330::120 --retry 5 --retry-delay 2 --max-time 0 --remote-name-all https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/{$(sed ':a;N;$!ba;s/\n/,/g' filelist_to_download )})

After January 15, I was told to use the following command instead:
curl -O -b ~/.urs_cookies -c ~/.urs_cookies -L -n --interface 2607:fe50:0:6330::120 --retry 5 --retry-delay 2 --max-time 0 --remote-name-all https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/{$(sed ':a;N;$!ba;s/\n/,/g' filelist_to_download )})

However, the download was successful for only the first N files, after which no file was downloaded. Here "N" changes in different trials.

Any help please? Chuanmin Hu at University of South Florida

Chuanmin,

Your command (sans the --interface option) works for me, but admittedly I only pulled down 30 files.
To eliminate a redirect (which *may* be causing issues), you should change "cgi" to "ob". So, https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/ should be https://oceandata.sci.gsfc.nasa.gov/ob/getfile/

Sean

Good morning,

I ordered SST data and tried to download following other posts and https://oceancolor.gsfc.nasa.gov/data/download_methods/.
But I couldn't get it. I think I did something wrong. Could anyone help me to get the following order?

  "https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/http_manifest.txt?h=ocdist203&p=/data1/d0c0c0708421e85c"

Thank you for your help, Ali

Try using "ob" in place of "cgi":
"https://oceandata.sci.gsfc.nasa.gov/ob/getfile/http_manifest.txt?h=ocdist203&p=/data1/d0c0c0708421e85c"


Thanks Sean and team. But replacing cgi with ob didn't work. I still had the same problem. The weird thing is, the "N" appears to be proportional to the total number of files in my download list. For example, if I have 5000 files to download, then after 1200 files (N=1200) it failed. I reissued the command for the leftover (3800 files), then N=800. Do it again with the remaining 3000 files, N=500. This appears like an asymptotic problem, where after many attempts I can finally download all 5000 files. But it is super annoying because N is a variable and the problem appears to be intermittent.
I also increased my "--retry 5 --retry-delay 2" to "--retry 10 --retry-delay 10", but again no luck. Chuanmin

Chuanmin,
We've dug a bit into this and are pretty much convinced it's not an issue with anything we've done on this end.  We replicated your curl approach for downloading a large number of files (>9500 in our test case).  While curl would pause on occasion, it always continued without intervention.  Could it be that there is something on your end that is dropping connections if there is a few second pause?

Sean

Sean: thanks so much for diagnosing, and sorry for replying late. I actually have no idea what's going on, but now I wrote a script to check the downloaded files after each round, and the leftovers (compared with the original list) are downloaded the next round - until all files are downloaded. It appeared to be working - it's not a real fix but it works. Cheers, Chuanmin

Chuanmin,
Glad you have a way forward :grin:  It's probably best to have your script with the extra logic to retry anyway...never trust a network :wink:

Sean