Sentinel-3 L1 files download fails : too many redirections

Use this Forum to find information on, or ask a question about, NASA Earth Science data.
jstum
Posts: 83
Joined: Mon Jul 04, 2005 4:54 am America/New_York
Answers: 0
Been thanked: 1 time

Sentinel-3 L1 files download fails : too many redirections

by jstum » Tue Feb 02, 2021 1:06 pm America/New_York

Hello,

It is impossible this afternoon to download these files using wget (no problem before). The message output is "20 redirections exceeded"I checked my profile : I have the rights.I have seen the same problem mentioned on the forum (e.g., https://oceancolor.gsfc.nasa.gov/forum/ ... irections)

I also tried clicking on the filename on the Direct Data Access web page, but it also fails for the same reasonIs it temporary? What can I do? Thank you for your help,

Jacques

P.S. : my wget command is :

wget $options_wget $line, with options_wget=" --progress=bar:force -np -c -R  -S --timeout=120  -t 2 --user $USER --password $PASSWORD --auth-no-challenge=on", and line="https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/$file",
with
$file=S3B_OL_1_EFR____20190106T151821_20190106T152121_20200111T122801_0180_020_296_2880_MR1_R_NT_002.zip using $file=A2020004003000.L2_LAC_OC.nc works well, in fact it seems that only Sentinel-3 failsI tried with all missions (VIIRS, MODIS,Aquarius, OCTS, GOCI, etc...), it works fine excepted for Sentinel-3 and HICO (but maybe for HICO I don't have the rights)

Tags:

OB.DAAC - SeanBailey
User Services
User Services
Posts: 1464
Joined: Wed Sep 18, 2019 6:15 pm America/New_York
Answers: 1
Been thanked: 4 times

Sentinel-3 L1 files download fails : too many redirections

by OB.DAAC - SeanBailey » Wed Feb 03, 2021 11:29 am America/New_York

Jacques,

If you have accepted the Sentinel 3 EULA (you can verify by looking at your Earthdata Login profile), it's quite odd that it works for all but Sentinel-3
(and HICO - which  should work as well, as there isn't a special EULA for it, it just requires an active Earthdata Login account).

I was able to replicate your redirect error, but if I replace "cgi" with "ob" it worked.  When we added the requirement for authentication for all data, we redirected cgi to ob, for sentinel products that is redirected to "sentinel" (because of the required EULA).  So, either replace cgi with ob (for all downloads), but use sentinel for the Sentinel-3 files.

If you use the  python script  (obdaac_download) we provide (available with SeaDAS or from https://oceancolor.gsfc.nasa.gov/data/download_methods/) you can skip a lot of the silliness required by wget...all it requires is the requests module and a .netrc file with your EDL credentials. 

Sean

jstum
Posts: 83
Joined: Mon Jul 04, 2005 4:54 am America/New_York
Answers: 0
Been thanked: 1 time

Sentinel-3 L1 files download fails : too many redirections

by jstum » Wed Feb 03, 2021 12:37 pm America/New_York

Hello Sean,

Well, cgi/getfile worked well for OLCI in the past days. Now, its doesn't work anymore, I replaced cgi by ob as you suggested and it works, thank you. For VIIRS, cgi/getfile is still functioning well

Jacques

oo_processing
Posts: 301
Joined: Wed Apr 06, 2005 12:11 pm America/New_York
Answers: 0
Has thanked: 5 times

Sentinel-3 L1 files download fails : too many redirections

by oo_processing » Thu Feb 04, 2021 11:35 am America/New_York

Sean,Fails for me though. If I use the cgi/getfile, it fails with the redirect error. cgi/ob works, but gets a ASCII / HTML file returned$ curl -b .urs_cookies_single -c .urs_cookies_single -L -n --interface 2607:fe50:0:6330::105 --retry 5 --retry-delay 2 --max-time 0 --output /cms_zfs/sat_data/olci/l1b/2021/030/S3B_OL_1_EFR____20210130T153242_20210130T153542_20210131T190743_0179_048_282_2160_LN1_O_NT_002.zip https://oceandata.sci.gsfc.nasa.gov/cgi/ob/S3B_OL_1_EFR____20210130T153242_20210130T153542_20210131T190743_0179_048_282_2160_LN1_O_NT_002.ziprm: cannot remove `.urs_cookies_single': No such file or directory  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current                                 Dload  Upload   Total   Spent    Left  Speed102   716  102   716    0     0   2893      0 --:--:-- --:--:-- --:--:-- 12785$ file /cms_zfs/sat_data/olci/l1b/2021/030/S3B_OL_1_EFR____20210130T153242_20210130T153542_20210131T190743_0179_048_282_2160_LN1_O_NT_002.zip/cms_zfs/sat_data/olci/l1b/2021/030/S3B_OL_1_EFR____20210130T153242_20210130T153542_20210131T190743_0179_048_282_2160_LN1_O_NT_002.zip: ASCII HTML document text$ curl -b .urs_cookies_single -c .urs_cookies_single -L -n --interface 2607:fe50:0:6330::102 --retry 5 --retry-delay 2 --max-time 0 --output /cms_zfs/sat_data/olci/l1b/2021/030/S3B_OL_1_EFR____20210130T153242_20210130T153542_20210131T190743_0179_048_282_2160_LN1_O_NT_002.zip https://oceandata.sci.gsfc.nasa.gov/cgi/ob/S3B_OL_1_EFR____20210130T153242_20210130T153542_20210131T190743_0179_048_282_2160_LN1_O_NT_002.zip  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current                                 Dload  Upload   Total   Spent    Left  Speed102   716  102   716    0     0   3057      0 --:--:-- --:--:-- --:--:-- 19351$ file /cms_zfs/sat_data/olci/l1b/2021/030/S3B_OL_1_EFR____20210130T153242_20210130T153542_20210131T190743_0179_048_282_2160_LN1_O_NT_002.zip/cms_zfs/sat_data/olci/l1b/2021/030/S3B_OL_1_EFR____20210130T153242_20210130T153542_20210131T190743_0179_048_282_2160_LN1_O_NT_002.zip: ASCII HTML document text$ curl -4 -b .urs_cookies_single -c .urs_cookies_single -L -n  --retry 5 --retry-delay 2 --max-time 0 --output /cms_zfs/sat_data/olci/l1b/2021/030/S3B_OL_1_EFR____20210130T153242_20210130T153542_20210131T190743_0179_048_282_2160_LN1_O_NT_002.zip https://oceandata.sci.gsfc.nasa.gov/cgi/ob/S3B_OL_1_EFR____20210130T153242_20210130T153542_20210131T190743_0179_048_282_2160_LN1_O_NT_002.zip  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current                                 Dload  Upload   Total   Spent    Left  Speed102   716  102   716    0     0   3062      0 --:--:-- --:--:-- --:--:-- 18842$ file /cms_zfs/sat_data/olci/l1b/2021/030/S3B_OL_1_EFR____20210130T153242_20210130T153542_20210131T190743_0179_048_282_2160_LN1_O_NT_002.zip/cms_zfs/sat_data/olci/l1b/2021/030/S3B_OL_1_EFR____20210130T153242_20210130T153542_20210131T190743_0179_048_282_2160_LN1_O_NT_002.zip: ASCII HTML document textActually I'm sorting through many issues on 2 separate sites.

oo_processing
Posts: 301
Joined: Wed Apr 06, 2005 12:11 pm America/New_York
Answers: 0
Has thanked: 5 times

Sentinel-3 L1 files download fails : too many redirections

by oo_processing » Thu Feb 04, 2021 1:26 pm America/New_York

Sean,Seems that the OLCI data is not found?$ curl -b .urs_cookies_single -c .urs_cookies_single -L -n --retry 5 --retry-delay 2 --max-time 0 --verbose --output ./S3B_OL_1_EFR____20210130T153542_20210130T153842_20210131T190804_0179_048_282_2340_LN1_O_NT_002.zip https://oceandata.sci.gsfc.nasa.gov/cgi/ob/S3B_OL_1_EFR____20210130T153542_20210130T153842_20210131T190804_0179_048_282_2340_LN1_O_NT_002.zip  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current                                 Dload  Upload   Total   Spent    Left  Speed  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* About to connect() to oceandata.sci.gsfc.nasa.gov port 443 (#0)*   Trying xx.xxx.xx.xx...* Connected to oceandata.sci.gsfc.nasa.gov (xx.xxx.xx.xx) port 443 (#0)* Initializing NSS with certpath: sql:/etc/pki/nssdb*   CAfile: /etc/pki/tls/certs/ca-bundle.crt  CApath: none* SSL connection using TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256* Server certificate:*       subject: CN=oceancolor.gsfc.nasa.gov*       start date: Jan 23 20:03:12 2021 GMT*       expire date: Apr 23 20:03:12 2021 GMT*       common name: oceancolor.gsfc.nasa.gov*       issuer: CN=R3,O=Let's Encrypt,C=US* Server auth using Basic with user 'oo_processing'> GET /cgi/ob/S3B_OL_1_EFR____20210130T153542_20210130T153842_20210131T190804_0179_048_282_2340_LN1_O_NT_002.zip HTTP/1.1> Authorization: Basic b29fcHJvY2Vzc2luZzpJZzJpd2tzRkpVaHY=> User-Agent: curl/7.29.0> Host: oceandata.sci.gsfc.nasa.gov> Accept: */*> < HTTP/1.1 404 Not Found< Server: nginx< Date: Thu, 04 Feb 2021 18:23:57 GMT< Content-Type: text/html; charset=utf-8< Content-Length: 716< Connection: keep-alive< { [data not shown]100   716  100   716    0     0   1156      0 --:--:-- --:--:-- --:--:--  1156* Connection #0 to host oceandata.sci.gsfc.nasa.gov left intact[bmurch@dell8 work]$ curl -b .urs_cookies_single -c .urs_cookies_single -L -n --retry 5 --retry-delay 2 --max-time 0 --verbose --output ./S3B_OL_1_EFR____20210201T144320_20210201T144620_20210202T202204_0179_048_310_2340_LN1_O_NT_002.zip  https://oceandata.sci.gsfc.nasa.gov/cgi/ob/S3B_OL_1_EFR____20210201T144320_20210201T144620_20210202T202204_0179_048_310_2340_LN1_O_NT_002.zip  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current                                 Dload  Upload   Total   Spent    Left  Speed  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* About to connect() to oceandata.sci.gsfc.nasa.gov port 443 (#0)*   Trying xx.xxx.xx.xx...* Connected to oceandata.sci.gsfc.nasa.gov (xx.xxx.xx.xx) port 443 (#0)* Initializing NSS with certpath: sql:/etc/pki/nssdb*   CAfile: /etc/pki/tls/certs/ca-bundle.crt  CApath: none* SSL connection using TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256* Server certificate:*       subject: CN=oceancolor.gsfc.nasa.gov*       start date: Jan 23 20:03:12 2021 GMT*       expire date: Apr 23 20:03:12 2021 GMT*       common name: oceancolor.gsfc.nasa.gov*       issuer: CN=R3,O=Let's Encrypt,C=US* Server auth using Basic with user 'oo_processing'> GET /cgi/ob/S3B_OL_1_EFR____20210201T144320_20210201T144620_20210202T202204_0179_048_310_2340_LN1_O_NT_002.zip HTTP/1.1> Authorization: Basic b29fcHJvY2Vzc2luZzpJZzJpd2tzRkpVaHY=> User-Agent: curl/7.29.0> Host: oceandata.sci.gsfc.nasa.gov> Accept: */*> < HTTP/1.1 404 Not Found< Server: nginx< Date: Thu, 04 Feb 2021 18:24:01 GMT< Content-Type: text/html; charset=utf-8< Content-Length: 716< Connection: keep-alive< { [data not shown]100   716  100   716    0     0    966      0 --:--:-- --:--:-- --:--:--   967* Connection #0 to host oceandata.sci.gsfc.nasa.gov left intact

oo_processing
Posts: 301
Joined: Wed Apr 06, 2005 12:11 pm America/New_York
Answers: 0
Has thanked: 5 times

Sentinel-3 L1 files download fails : too many redirections

by oo_processing » Thu Feb 04, 2021 1:58 pm America/New_York

Sean,I think that https://oceandata.sci.gsfc.nasa.gov/sentinel/getfile/used to work as well.No longer.curl -b .urs_cookies_single -c .urs_cookies_single -L -n --retry 5 --retry-delay 2 --max-time 0 --output ./S3A_OL_1_ERR____20210131T021455_20210131T025842_20210201T064939_2627_068_046______LN1_O_NT_002.zip  https://oceandata.sci.gsfc.nasa.gov/sentinel/getfile/S3A_OL_1_ERR____20210131T021455_20210131T025842_20210201T064939_2627_068_046______LN1_O_NT_002.zip  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current                                 Dload  Upload   Total   Spent    Left  Speed  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0...  0     0    0     0    0     0      0      0 --:--:--  0:00:05 --:--:--     0curl: (47) Maximum (50) redirects followedBrock

OB.DAAC - SeanBailey
User Services
User Services
Posts: 1464
Joined: Wed Sep 18, 2019 6:15 pm America/New_York
Answers: 1
Been thanked: 4 times

Sentinel-3 L1 files download fails : too many redirections

by OB.DAAC - SeanBailey » Thu Feb 04, 2021 4:08 pm America/New_York

I've put "top men" on it...I just hope they can find the right box in that warehouse....  If ob/getfile works, for now may just want to not use sentinel/getfile...

Sean

oo_processing
Posts: 301
Joined: Wed Apr 06, 2005 12:11 pm America/New_York
Answers: 0
Has thanked: 5 times

Sentinel-3 L1 files download fails : too many redirections

by oo_processing » Thu Feb 04, 2021 5:06 pm America/New_York

Sean,I think only the cgi/ob works now. It doesn't get the redirect error.$ curl --verbose  -b .urs_cookies_single -c .urs_cookies_single -L -n --retry 5 --retry-delay 2 --max-time 0 --output ./S3A_OL_1_ERR____20210131T021455_20210131T025842_20210201T064939_2627_068_046______LN1_O_NT_002.zip  https://oceandata.sci.gsfc.nasa.gov/cgi/ob/S3A_OL_1_ERR____20210131T021455_20210131T025842_20210201T064939_2627_068_046______LN1_O_NT_002.zip  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current                                 Dload  Upload   Total   Spent    Left  Speed  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* About to connect() to oceandata.sci.gsfc.nasa.gov port 443 (#0)*   Trying xxx.xx.xxx...* Connected to oceandata.sci.gsfc.nasa.gov (xxx.xx.xxx) port 443 (#0)* Initializing NSS with certpath: sql:/etc/pki/nssdb*   CAfile: /etc/pki/tls/certs/ca-bundle.crt  CApath: none* SSL connection using TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256* Server certificate:*       subject: CN=oceancolor.gsfc.nasa.gov*       start date: Jan 23 20:03:12 2021 GMT*       expire date: Apr 23 20:03:12 2021 GMT*       common name: oceancolor.gsfc.nasa.gov*       issuer: CN=R3,O=Let's Encrypt,C=US* Server auth using Basic with user 'oo_processing'> GET /cgi/ob/S3A_OL_1_ERR____20210131T021455_20210131T025842_20210201T064939_2627_068_046______LN1_O_NT_002.zip HTTP/1.1> Authorization: Basic b29fcHJvY2Vzc2luZzpJZzJpd2tzRkpVaHY=> User-Agent: curl/7.29.0> Host: oceandata.sci.gsfc.nasa.gov> Accept: */*> Cookie: plack_session2=31eeb0c9a988b75bc8eb626ceb9664d17edf1935> < HTTP/1.1 404 Not Found< Server: nginx< Date: Thu, 04 Feb 2021 22:03:18 GMT< Content-Type: text/html; charset=utf-8< Content-Length: 716< Connection: keep-alive< { [data not shown]100   716  100   716    0     0   1156      0 --:--:-- --:--:-- --:--:--  1156* Connection #0 to host oceandata.sci.gsfc.nasa.gov left intact[bmurch@dell8 work]$ file /cgi/ob/S3A_OL_1_ERR____20210131T021455_20210131T025842_20210201T064939_2627_068_046______LN1_O_NT_002.zip/cgi/ob/S3A_OL_1_ERR____20210131T021455_20210131T025842_20210201T064939_2627_068_046______LN1_O_NT_002.zip: cannot open (No such file or directory)[bmurch@dell8 work]$ file ./S3A_OL_1_ERR____20210131T021455_20210131T025842_20210201T064939_2627_068_046______LN1_O_NT_002.zip./S3A_OL_1_ERR____20210131T021455_20210131T025842_20210201T064939_2627_068_046______LN1_O_NT_002.zip: HTML document, ASCII text


oo_processing
Posts: 301
Joined: Wed Apr 06, 2005 12:11 pm America/New_York
Answers: 0
Has thanked: 5 times

Sentinel-3 L1 files download fails : too many redirections

by oo_processing » Fri Feb 05, 2021 10:50 am America/New_York

Sean,Thanks. I was able to grab all my sentinel files returned as HTML now.Except for this one:$ curl -b .urs_cookies_single -c .urs_cookies_single -L -n --interface 2607:fe50:0:6330::104 --retry 5 --retry-delay 2 --max-time 0 --output  ./S3A_OL_1_EFR____20210203T024335_20210203T024635_20210203T043623_0179_068_089_2340_MAR_O_NR_002.zip   https://oceandata.sci.gsfc.nasa.gov
/cgi/getfile/S3A_OL_1_EFR____20210203T024335_20210203T024635_20210203T043623_0179_068_089_2340_MAR_O_NR_002.zip  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current                                 Dload  Upload   Total   Spent    Left  Speed102   716  102   716    0     0    129      0  0:00:05  0:00:05 --:--:--   171[bmurch@optics0 034]$ cat ./S3A_OL_1_EFR____20210203T024335_20210203T024635_20210203T043623_0179_068_089_2340_MAR_O_NR_002.zip<!doctype html><html lang='en'><head><meta charset='utf-8'><title>ERROR @ OceanColor Biology Processing Group (OBPG)</title><meta name='viewport' content='width=device-width, initial-scale=1'></head><body link=white vlink=white alink=white style='background-color:#1C4960; color:white; font-size:175%'><br><hr color= #666699><center><h1><b>.:. ERROR .:.</b></h1><h2>OceanColor Biology Processing Group (OBPG)</h2><blockquote>We are sorry, but an error has occurred. Please use the back button to return to the previous page</blockquote><hr color= #666699></body></html><!-- IE needs 512+ bytes: [http]://blogs.msdn.com/b/ieinternals/archive/2010/08/19/http-error-pages-in-internet-explorer.aspx -->

Post Reply