wget redirection

Use this Forum to find information on, or ask a question about, NASA Earth Science data.
bruce
Posts: 85
Joined: Thu Mar 17, 2005 4:36 pm America/New_York
Answers: 0

wget redirection

by bruce » Mon May 18, 2020 8:35 am America/New_York

Starting Saturday morning, these 2 commands
wget --post-data="subID=528&cksum=1" -q -O - https://oceandata.sci.gsfc.nasa.gov/api/file_search.cgi

wget --post-data="subID=528&cksum=1" -q -O - https://oceandata.sci.gsfc.nasa.gov/api/file_search.cgi > search.cksums && awk '{print $2}' search.cksums | wget --auth-no-challenge=on --noverbose --no-clobber --base="https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/"  --load-cookies ~/.urs_cookies --save-cookies ~/.urs_cookies --keep-session-cookies --input-file - && sha1sum -c search.cksums


product the following output

3765810dd7c33912924f3aec5aa0ae916af59710  A2020135083500.L1A_LAC.bz2
8d65b5471cf98c7af51c20e0823eb5d603e2aaba  A2020135163000.L1A_LAC.bz2
1dcf679a103819acaecf39f0feca7e4a7a76ef5d  A2020135180500.L1A_LAC.bz2
41797784600f48afdb58ac96fed67684971590ce  A2020135181000.L1A_LAC.bz2
5daf1f26119e63edcca8aed50a06332557f82534  A2020136074000.L1A_LAC.bz2
0dca87f5715ec0a32b1549f458bcc94533aa0370  A2020136153500.L1A_LAC.bz2
90e7fced017a2d2ace0030fa1e0e87917d063277  A2020136171000.L1A_LAC.bz2
780e2b593a377410343c5252a5be280f2d1b23fc  A2020137161500.L1A_LAC.bz2
65a07a1f749df9e21f695d814f980a41ec8a2b62  A2020137175500.L1A_LAC.bz2
20 redirections exceeded.
20 redirections exceeded.
20 redirections exceeded.
20 redirections exceeded.
20 redirections exceeded.
20 redirections exceeded.
20 redirections exceeded.
20 redirections exceeded.
20 redirections exceeded.


with great regularity and repeatability.  I have it on good authority that nothing has changed with our network.
They have worked since the requirement to do the whole authorization stuff.

Bruce

Tags:

OB.DAAC - SeanBailey
User Services
User Services
Posts: 1464
Joined: Wed Sep 18, 2019 6:15 pm America/New_York
Answers: 1
Been thanked: 5 times

wget redirection

by OB.DAAC - SeanBailey » Mon May 18, 2020 8:58 am America/New_York

Bruce,

You're not the first to report such a behavior
Nothing changed on our end either...the trouble is, there is a third party involved.   We're still investigating what might have changed.  As soon as we come up with some information, I'll relay it.

Sean

bruce
Posts: 85
Joined: Thu Mar 17, 2005 4:36 pm America/New_York
Answers: 0

wget redirection

by bruce » Mon May 18, 2020 10:06 am America/New_York

Not that it probably really matters...  It actually goes back to last Thursday (5/14) morning

OB.DAAC - SeanBailey
User Services
User Services
Posts: 1464
Joined: Wed Sep 18, 2019 6:15 pm America/New_York
Answers: 1
Been thanked: 5 times

wget redirection

by OB.DAAC - SeanBailey » Mon May 18, 2020 10:10 am America/New_York

Oh, it matters :grin:  That timing jives with other "oddities" we've experienced - and points more at that third party...

stefan.maier
Posts: 13
Joined: Wed Feb 10, 2021 3:49 pm America/New_York
Answers: 0

wget redirection

by stefan.maier » Mon May 18, 2020 8:43 pm America/New_York

Hi Sean,
I've had frequent error "47 Too many redirects." using curl (V7.29.0) ever since you activated authentication. Here is a quick figure with the number of errors per day. It is not normalised to the number of attempted downloads. The number of attempted downloads is pretty constant, though. So this gives a good indication when things have been changing.
Regards, Stefan

PS: The line is a 7 day running mean.

OB.DAAC - SeanBailey
User Services
User Services
Posts: 1464
Joined: Wed Sep 18, 2019 6:15 pm America/New_York
Answers: 1
Been thanked: 5 times

wget redirection

by OB.DAAC - SeanBailey » Tue May 19, 2020 1:53 pm America/New_York

Thanks.  This is a bit disturbing...if you have any log information that could provide some insight into this, we'd welcome it!
We are investigating this on our end, but have yet to be able to replicate the problem. 
We are also planning to implement an alternative authentication mechanism for non-interactive downloads with the goal of minimizing the need to re-authenticate within a session.

Sean

stefan.maier
Posts: 13
Joined: Wed Feb 10, 2021 3:49 pm America/New_York
Answers: 0

wget redirection

by stefan.maier » Tue May 19, 2020 5:58 pm America/New_York

All I have is:
curl: (47) Maximum (50) redirects followed
I will turn up verbosity and see if I can get more info.
Stefan

stefan.maier
Posts: 13
Joined: Wed Feb 10, 2021 3:49 pm America/New_York
Answers: 0

wget redirection

by stefan.maier » Tue May 19, 2020 6:03 pm America/New_York

... other errors I get only occasionally are:
35 - SSL connect error. The SSL handshaking failed.
51 - The peer's SSL certificate or SSH MD5 fingerprint was not OK.
56 - Failure in receiving network data.
They are not of concern but might help solving the problem.
Stefan

stefan.maier
Posts: 13
Joined: Wed Feb 10, 2021 3:49 pm America/New_York
Answers: 0

wget redirection

by stefan.maier » Thu May 21, 2020 5:44 pm America/New_York

Sean,
I have a more detailed log. I don't want to post the log here as I don't know whether it contains sensitive information. What is the best way to get it to you?
Stefan

oo_processing
Posts: 301
Joined: Wed Apr 06, 2005 12:11 pm America/New_York
Answers: 0
Has thanked: 5 times

wget redirection

by oo_processing » Thu May 21, 2020 7:45 pm America/New_York

Sean,

We cannot d/l anything now. I have many failures (on separate campuses) as do others working in various locations.
See attached file.

The troubling part is
* Re-using existing connection! (#0) with host oceandata.sci.gsfc.nasa.gov                                                                                                                                                                 
* Connected to oceandata.sci.gsfc.nasa.gov (xx.xxx.xx.xx) port 443 (#0)                                                                                                                                                                  
* Server auth using Basic with user 'oo_lab'                                                                                                                                                                                               

> GET /ob/getfile/restrict?code=e60913d028b88e6b82aa14c2e3c4f8cb21a249c9cd180fabbac498fa40da301e HTTP/1.1                                                                                                                                  
> Authorization: Basic b29fbGFiOlpYdzN5eXdDN295RQ==                                                                                                                                                                                        
> User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.44 zlib/1.2.3 libidn/1.18 libssh2/1.4.2                                                                                                                           
> Host: oceandata.sci.gsfc.nasa.gov                                                                                                                                                                                                        
> Accept: */*                                                                                                                                                                                                                              
> Cookie: app-obdaac=2792e4f92c8bc81104c284b64a22f87a7124cdc4                                                                                                                                                                              
>                                                                                                                                                                                                                                          


< HTTP/1.1 403 Forbidden                                                                                                                                                                                                                   
< Server: nginx                                                                                                                                                                                                                            
< Date: Thu, 21 May 2020 23:39:03 GMT                                                                                                                                                                                                      
< Content-Type: text/html; charset=utf-8                                                                                                                                                                                                   
< Content-Length: 665                                                                                                                                                                                                                      
< Connection: keep-alive                                                                                                                                                                                                                   
< Keep-Alive: timeout=60                                                                                                                                                                                                                   
< ETag: "585d4968-299"                                                                                                                                                                                                                     
<
attachment 1

Post Reply