wget redirection

[oo_processing]

Could it be this simple?
https://support.cloudbees.com/hc/en-us/articles/360032824192-KBEC-00402-Resolve-curl-47-Maximum-50-redirects-followed-issue

[OB.DAAC - amscott]

We just went through a troubleshooting session where the redirect issue was solved after the user deleted the old cookie file and was able to successfully execute the command with a new cookie file was generated. It is hopefully that simple for everyone else. Please give it a try.

[bruce]

Well, for me, nuking the .urs_cookies file seems to have solved the problem.

[oo_processing]

Bruce,
I create and different .urs_cookies file for each run I do to avoid the issue.
So for instance, today I may create one that is .urs_cookies_viirs_052620 and run something by hand.

Or, because I have several scripts that run that may overlap ( ex. running on different interfaces (machines) but the same user and file system, so the name of the script is used e.g. from my log:

STATUS: $cookiejar: /optics/home/oo_processing/.urs_cookies_get_viirs_h5_gsfc

The file is created at the start of a request and deleted at the end.
I wish it were this simple :eek:

[bruce]

Well it worked - once.  My overnight job failed again with too many redirections.  Looks like it's time to delete the file on each invocation.

[oo_processing]

Well, I didn't see a response to this, but since your backend is nginx (a load balancer) this may apply in some way:

Problem
You encounter this error message when running the curl command
curl -u admin:changeme -L -k -X GET --header "Accept: application/json" https:///rest/v1.0/projects
The output was: curl: (47) Maximum (50) redirects followed

Solution
This is because the WebServer Loadbalancer VIP was configured to forward the request only to http port 80 of EF webserver. Then in /apache/conf, the EF webserver redirects all requests on port 80 to port 443 of the Loadbalancer.
This goes in a loop and fails after 50 retries.
You can resolve this issue by configuring the WebServer Loadbalancer VIP to forward the request to https port 443 of the EF webservers that it is load balancing.

https://support.cloudbees.com/hc/en-us/articles/360032824192-KBEC-00402-Resolve-curl-47-Maximum-50-redirects-followed-issue

[OB.DAAC - SeanBailey]

Not likely.  Such a misconfiguration would prevent ALL downloads.  I can't speak to the configuration of the Earthdata Login set up, but ours does not support port 80 at all, so all traffic is over 443.
The issue is more likely related to the cookie, as deleting it resolved the problem for Bruce....until it cropped up again, but still likely a cookie issue.  The Earthdata login folks are looking more closely to make sure there isn't an issue on their end.  The cookie is intended to be for a given session, so keeping it around between sessions isn't necessary and may be problematic. 

Sean

[oo_processing]

Sean,

I delete my cookies every time. If its in a script, it checks to see if the cookie file exists. If it does, its deleted prior to the curl call.
I never reuse them after the failure, and it fails every time on a list, so I delete the ~/.cookie_jar_00 for instance, and it is created new with the following command:
If I try to run the command again, using the same cookie file, it starts the redirect failures immediately without downloading any files, so I MUST delete the cookie file.

curl -b ~/.cookie_jar_00 -c ~/.cookie_jar_00 -L -n --interface 2607:fe50:0:6330::100 --retry 5 --retry-delay 2 --max-time 0 --remote-name-all https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/{$(sed ':a;N;$!ba;s/\n/,/g' /shares/cms_optics/virtual_ant/S4P/bin/fa_density/non_fai_rois/CAPE_COD/x00)}

Where the bolded x00 section is a list of about 950 files to download. When it get to 47, it dies with the redirect error.
I can delete the ~/.cookie_jar_00, and remove the files already downloaded from the list, and start with the next files in the x00 list file.
It will fail again at the 47 file d/l mark with the new cookie file ~/.cookie_jar_00 that was created at the start of the run.
I have not seen the cookie file ~/.cookie_jar_00 change (after creation until failure).
It bears the time stamp of the original creation when the command is run.

Thus, I think its unlikely that its on the cookie side locally.

Having said that, if Earthdata has one load balance pointed to port 80 in error somewhere, and that doesn't exist (as you say), then it will fail like described, I think?

Brock

[OB.DAAC - SeanBailey]

Brock,

I cannot get your cURL command to work (appropriately sanitized for my Mac - something not happy with the sed bit), but this one does work for me:

curl -d "sensor=octs&sdate=1996-11-01&edate=1997-01-01&dtype=L3b&addurl=1&results_as_file=1&search=*DAY_CHL*" https://oceandata.sci.gsfc.nasa.gov/api/file_search |grep getfile | cut -d "'" -f 2 | xargs -n 1 curl -LJO -n -c ~/.urs_cookies -b ~/.urs_cookies

It only returns 61 files, but does return them without issue.  As I don't get a redirect failure I cannot explain why you seem to consistently fail after 47 files.
Others with the problem seems to suggest it IS a cookie issue. 

The Earthdata folks live under the same network/IT constraints we do, so I seriously doubt they've got an issue with their load balancing - again, that error would not be intermittent but would deny ALL traffic.

Sean

[oo_processing]

I guess we'll have to see. But, I know that I'm not alone in the magic 47 number.
It pops up on your forums and its what others here, see too.

Hopefully it will be tracked down at some point.

Cheers,
Stay safe!