wget redirection

Use this Forum to find information on, or ask a question about, NASA Earth Science data.
oo_processing
Posts: 242
Joined: Wed Apr 06, 2005 12:11 pm America/New_York

wget redirection

by oo_processing » Sat May 23, 2020 12:19 pm America/New_York


Tags:

OB.DAAC - amscott
User Services
User Services
Posts: 221
Joined: Mon Jun 22, 2020 5:24 pm America/New_York
Answers: 1
Has thanked: 2 times

wget redirection

by OB.DAAC - amscott » Tue May 26, 2020 1:43 pm America/New_York

We just went through a troubleshooting session where the redirect issue was solved after the user deleted the old cookie file and was able to successfully execute the command with a new cookie file was generated. It is hopefully that simple for everyone else. Please give it a try.

bruce
Posts: 85
Joined: Thu Mar 17, 2005 4:36 pm America/New_York

wget redirection

by bruce » Tue May 26, 2020 1:43 pm America/New_York

Well, for me, nuking the .urs_cookies file seems to have solved the problem.

oo_processing
Posts: 242
Joined: Wed Apr 06, 2005 12:11 pm America/New_York

wget redirection

by oo_processing » Tue May 26, 2020 1:56 pm America/New_York

Bruce,
I create and different .urs_cookies file for each run I do to avoid the issue.
So for instance, today I may create one that is .urs_cookies_viirs_052620 and run something by hand.

Or, because I have several scripts that run that may overlap ( ex. running on different interfaces (machines) but the same user and file system, so the name of the script is used e.g. from my log:

STATUS: $cookiejar: /optics/home/oo_processing/.urs_cookies_get_viirs_h5_gsfc

The file is created at the start of a request and deleted at the end.
I wish it were this simple :eek:

bruce
Posts: 85
Joined: Thu Mar 17, 2005 4:36 pm America/New_York

wget redirection

by bruce » Wed May 27, 2020 8:09 am America/New_York

Well it worked - once.  My overnight job failed again with too many redirections.  Looks like it's time to delete the file on each invocation.

oo_processing
Posts: 242
Joined: Wed Apr 06, 2005 12:11 pm America/New_York

wget redirection

by oo_processing » Tue Jun 02, 2020 12:03 pm America/New_York

Well, I didn't see a response to this, but since your backend is nginx (a load balancer) this may apply in some way:

Problem
You encounter this error message when running the curl command
curl -u admin:changeme -L -k -X GET --header "Accept: application/json" https:///rest/v1.0/projects
The output was: curl: (47) Maximum (50) redirects followed

Solution
This is because the WebServer Loadbalancer VIP was configured to forward the request only to http port 80 of EF webserver. Then in /apache/conf, the EF webserver redirects all requests on port 80 to port 443 of the Loadbalancer.
This goes in a loop and fails after 50 retries.
You can resolve this issue by configuring the WebServer Loadbalancer VIP to forward the request to https port 443 of the EF webservers that it is load balancing.

https://support.cloudbees.com/hc/en-us/articles/360032824192-KBEC-00402-Resolve-curl-47-Maximum-50-redirects-followed-issue

OB.DAAC - SeanBailey
User Services
User Services
Posts: 1292
Joined: Wed Sep 18, 2019 6:15 pm America/New_York
Answers: 1
Been thanked: 1 time

wget redirection

by OB.DAAC - SeanBailey » Tue Jun 02, 2020 2:53 pm America/New_York

Not likely.  Such a misconfiguration would prevent ALL downloads.  I can't speak to the configuration of the Earthdata Login set up, but ours does not support port 80 at all, so all traffic is over 443.
The issue is more likely related to the cookie, as deleting it resolved the problem for Bruce....until it cropped up again, but still likely a cookie issue.  The Earthdata login folks are looking more closely to make sure there isn't an issue on their end.  The cookie is intended to be for a given session, so keeping it around between sessions isn't necessary and may be problematic. 

Sean

oo_processing
Posts: 242
Joined: Wed Apr 06, 2005 12:11 pm America/New_York

wget redirection

by oo_processing » Tue Jun 02, 2020 3:15 pm America/New_York

Sean,

I delete my cookies every time. If its in a script, it checks to see if the cookie file exists. If it does, its deleted prior to the curl call.
I never reuse them after the failure, and it fails every time on a list, so I delete the ~/.cookie_jar_00 for instance, and it is created new with the following command:
If I try to run the command again, using the same cookie file, it starts the redirect failures immediately without downloading any files, so I MUST delete the cookie file.

curl -b ~/.cookie_jar_00 -c ~/.cookie_jar_00 -L -n --interface 2607:fe50:0:6330::100 --retry 5 --retry-delay 2 --max-time 0 --remote-name-all https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/{$(sed ':a;N;$!ba;s/\n/,/g' /shares/cms_optics/virtual_ant/S4P/bin/fa_density/non_fai_rois/CAPE_COD/x00)}

Where the bolded x00 section is a list of about 950 files to download. When it get to 47, it dies with the redirect error.
I can delete the ~/.cookie_jar_00, and remove the files already downloaded from the list, and start with the next files in the x00 list file.
It will fail again at the 47 file d/l mark with the new cookie file ~/.cookie_jar_00 that was created at the start of the run.
I have not seen the cookie file ~/.cookie_jar_00 change (after creation until failure).
It bears the time stamp of the original creation when the command is run.

Thus, I think its unlikely that its on the cookie side locally.

Having said that, if Earthdata has one load balance pointed to port 80 in error somewhere, and that doesn't exist (as you say), then it will fail like described, I think?

Brock

OB.DAAC - SeanBailey
User Services
User Services
Posts: 1292
Joined: Wed Sep 18, 2019 6:15 pm America/New_York
Answers: 1
Been thanked: 1 time

wget redirection

by OB.DAAC - SeanBailey » Tue Jun 02, 2020 4:26 pm America/New_York

Brock,

I cannot get your cURL command to work (appropriately sanitized for my Mac - something not happy with the sed bit), but this one does work for me:

curl -d "sensor=octs&sdate=1996-11-01&edate=1997-01-01&dtype=L3b&addurl=1&results_as_file=1&search=*DAY_CHL*" https://oceandata.sci.gsfc.nasa.gov/api/file_search |grep getfile | cut -d "'" -f 2 | xargs -n 1 curl -LJO -n -c ~/.urs_cookies -b ~/.urs_cookies

It only returns 61 files, but does return them without issue.  As I don't get a redirect failure I cannot explain why you seem to consistently fail after 47 files.
Others with the problem seems to suggest it IS a cookie issue. 

The Earthdata folks live under the same network/IT constraints we do, so I seriously doubt they've got an issue with their load balancing - again, that error would not be intermittent but would deny ALL traffic.

Sean

oo_processing
Posts: 242
Joined: Wed Apr 06, 2005 12:11 pm America/New_York

wget redirection

by oo_processing » Tue Jun 02, 2020 4:32 pm America/New_York

I guess we'll have to see. But, I know that I'm not alone in the magic 47 number.
It pops up on your forums and its what others here, see too.

Hopefully it will be tracked down at some point.

Cheers,
Stay safe!

Post Reply