wget EarthData Login does not work

Please enter here to ask a question about any NASA Science related topics!
huhuhu
Posts: 14
Joined: Fri Jun 25, 2010 11:46 am America/New_York

wget EarthData Login does not work

by huhuhu » Wed Jan 22, 2020 11:31 am America/New_York

I have been encountering data download problems since January 15, 2020. I used to be able to download a list of files (up to 5000 I think) without any problem using the following command:
curl --interface 2607:fe50:0:6330::120 --retry 5 --retry-delay 2 --max-time 0 --remote-name-all https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/{$(sed ':a;N;$!ba;s/\n/,/g' filelist_to_download )})

After January 15, I was told to use the following command instead:
curl -O -b ~/.urs_cookies -c ~/.urs_cookies -L -n --interface 2607:fe50:0:6330::120 --retry 5 --retry-delay 2 --max-time 0 --remote-name-all https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/{$(sed ':a;N;$!ba;s/\n/,/g' filelist_to_download )})

However, the download was successful for only the first N files, after which no file was downloaded. Here "N" changes in different trials.

Any help please? Chuanmin Hu at University of South Florida

Tags:

OB.DAAC - SeanBailey
User Services
User Services
Posts: 1225
Joined: Wed Sep 18, 2019 6:15 pm America/New_York
Answers: 1

wget EarthData Login does not work

by OB.DAAC - SeanBailey » Wed Jan 22, 2020 6:42 pm America/New_York

Chuanmin,

Your command (sans the --interface option) works for me, but admittedly I only pulled down 30 files.
To eliminate a redirect (which *may* be causing issues), you should change "cgi" to "ob". So, https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/ should be https://oceandata.sci.gsfc.nasa.gov/ob/getfile/

Sean

obdaac_forum_user
Posts: 86
Joined: Wed Jan 27, 2021 1:52 pm America/New_York

wget EarthData Login does not work

by obdaac_forum_user » Thu Jan 23, 2020 9:29 am America/New_York

Good morning,

I ordered SST data and tried to download following other posts and https://oceancolor.gsfc.nasa.gov/data/download_methods/.
But I couldn't get it. I think I did something wrong. Could anyone help me to get the following order?

  "https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/http_manifest.txt?h=ocdist203&p=/data1/d0c0c0708421e85c"

Thank you for your help, Ali

gnwiii
Posts: 642
Joined: Fri Jan 29, 2021 5:51 pm America/New_York
Answers: 2

wget EarthData Login does not work

by gnwiii » Thu Jan 23, 2020 9:42 am America/New_York


huhuhu
Posts: 14
Joined: Fri Jun 25, 2010 11:46 am America/New_York

wget EarthData Login does not work

by huhuhu » Thu Jan 23, 2020 10:45 am America/New_York

Thanks Sean and team. But replacing cgi with ob didn't work. I still had the same problem. The weird thing is, the "N" appears to be proportional to the total number of files in my download list. For example, if I have 5000 files to download, then after 1200 files (N=1200) it failed. I reissued the command for the leftover (3800 files), then N=800. Do it again with the remaining 3000 files, N=500. This appears like an asymptotic problem, where after many attempts I can finally download all 5000 files. But it is super annoying because N is a variable and the problem appears to be intermittent.
I also increased my "--retry 5 --retry-delay 2" to "--retry 10 --retry-delay 10", but again no luck. Chuanmin

OB.DAAC - SeanBailey
User Services
User Services
Posts: 1225
Joined: Wed Sep 18, 2019 6:15 pm America/New_York
Answers: 1

wget EarthData Login does not work

by OB.DAAC - SeanBailey » Tue Jan 28, 2020 6:48 pm America/New_York

Chuanmin,
We've dug a bit into this and are pretty much convinced it's not an issue with anything we've done on this end.  We replicated your curl approach for downloading a large number of files (>9500 in our test case).  While curl would pause on occasion, it always continued without intervention.  Could it be that there is something on your end that is dropping connections if there is a few second pause?

Sean

huhuhu
Posts: 14
Joined: Fri Jun 25, 2010 11:46 am America/New_York

wget EarthData Login does not work

by huhuhu » Thu Feb 06, 2020 8:37 pm America/New_York

Sean: thanks so much for diagnosing, and sorry for replying late. I actually have no idea what's going on, but now I wrote a script to check the downloaded files after each round, and the leftovers (compared with the original list) are downloaded the next round - until all files are downloaded. It appeared to be working - it's not a real fix but it works. Cheers, Chuanmin

OB.DAAC - SeanBailey
User Services
User Services
Posts: 1225
Joined: Wed Sep 18, 2019 6:15 pm America/New_York
Answers: 1

wget EarthData Login does not work

by OB.DAAC - SeanBailey » Fri Feb 07, 2020 9:14 am America/New_York

Chuanmin,
Glad you have a way forward :grin:  It's probably best to have your script with the extra logic to retry anyway...never trust a network :wink:

Sean

Post Reply