Sentinel-3 L1 files download fails : too many redirections

[jstum]

Hello,

It is impossible this afternoon to download these files using wget (no problem before). The message output is "20 redirections exceeded"I checked my profile : I have the rights.I have seen the same problem mentioned on the forum (e.g., https://oceancolor.gsfc.nasa.gov/forum/ ... irections)

I also tried clicking on the filename on the Direct Data Access web page, but it also fails for the same reasonIs it temporary? What can I do? Thank you for your help,

Jacques

P.S. : my wget command is :

wget $options_wget $line, with options_wget=" --progress=bar:force -np -c -R  -S --timeout=120  -t 2 --user $USER --password $PASSWORD --auth-no-challenge=on", and line="https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/$file",
with
$file=S3B_OL_1_EFR____20190106T151821_20190106T152121_20200111T122801_0180_020_296_2880_MR1_R_NT_002.zip using $file=A2020004003000.L2_LAC_OC.nc works well, in fact it seems that only Sentinel-3 failsI tried with all missions (VIIRS, MODIS,Aquarius, OCTS, GOCI, etc...), it works fine excepted for Sentinel-3 and HICO (but maybe for HICO I don't have the rights)

[OB.DAACx - SeanBailey]

Jacques,

If you have accepted the Sentinel 3 EULA (you can verify by looking at your Earthdata Login profile), it's quite odd that it works for all but Sentinel-3
(and HICO - which  should work as well, as there isn't a special EULA for it, it just requires an active Earthdata Login account).

I was able to replicate your redirect error, but if I replace "cgi" with "ob" it worked.  When we added the requirement for authentication for all data, we redirected cgi to ob, for sentinel products that is redirected to "sentinel" (because of the required EULA).  So, either replace cgi with ob (for all downloads), but use sentinel for the Sentinel-3 files.

If you use the  python script  (obdaac_download) we provide (available with SeaDAS or from https://oceancolor.gsfc.nasa.gov/data/download_methods/) you can skip a lot of the silliness required by wget...all it requires is the requests module and a .netrc file with your EDL credentials. 

Sean

[jstum]

Hello Sean,

Well, cgi/getfile worked well for OLCI in the past days. Now, its doesn't work anymore, I replaced cgi by ob as you suggested and it works, thank you. For VIIRS, cgi/getfile is still functioning well

Jacques

[oo_processing]

Sean,Fails for me though. If I use the cgi/getfile, it fails with the redirect error. cgi/ob works, but gets a ASCII / HTML file returned$ curl -b .urs_cookies_single -c .urs_cookies_single -L -n --interface 2607:fe50:0:6330::105 --retry 5 --retry-delay 2 --max-time 0 --output /cms_zfs/sat_data/olci/l1b/2021/030/S3B_OL_1_EFR____20210130T153242_20210130T153542_20210131T190743_0179_048_282_2160_LN1_O_NT_002.zip https://oceandata.sci.gsfc.nasa.gov/cgi/ob/S3B_OL_1_EFR____20210130T153242_20210130T153542_20210131T190743_0179_048_282_2160_LN1_O_NT_002.ziprm: cannot remove `.urs_cookies_single': No such file or directory  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current                                 Dload  Upload   Total   Spent    Left  Speed102   716  102   716    0     0   2893      0 --:--:-- --:--:-- --:--:-- 12785$ file /cms_zfs/sat_data/olci/l1b/2021/030/S3B_OL_1_EFR____20210130T153242_20210130T153542_20210131T190743_0179_048_282_2160_LN1_O_NT_002.zip/cms_zfs/sat_data/olci/l1b/2021/030/S3B_OL_1_EFR____20210130T153242_20210130T153542_20210131T190743_0179_048_282_2160_LN1_O_NT_002.zip: ASCII HTML document text$ curl -b .urs_cookies_single -c .urs_cookies_single -L -n --interface 2607:fe50:0:6330::102 --retry 5 --retry-delay 2 --max-time 0 --output /cms_zfs/sat_data/olci/l1b/2021/030/S3B_OL_1_EFR____20210130T153242_20210130T153542_20210131T190743_0179_048_282_2160_LN1_O_NT_002.zip https://oceandata.sci.gsfc.nasa.gov/cgi/ob/S3B_OL_1_EFR____20210130T153242_20210130T153542_20210131T190743_0179_048_282_2160_LN1_O_NT_002.zip  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current                                 Dload  Upload   Total   Spent    Left  Speed102   716  102   716    0     0   3057      0 --:--:-- --:--:-- --:--:-- 19351$ file /cms_zfs/sat_data/olci/l1b/2021/030/S3B_OL_1_EFR____20210130T153242_20210130T153542_20210131T190743_0179_048_282_2160_LN1_O_NT_002.zip/cms_zfs/sat_data/olci/l1b/2021/030/S3B_OL_1_EFR____20210130T153242_20210130T153542_20210131T190743_0179_048_282_2160_LN1_O_NT_002.zip: ASCII HTML document text$ curl -4 -b .urs_cookies_single -c .urs_cookies_single -L -n  --retry 5 --retry-delay 2 --max-time 0 --output /cms_zfs/sat_data/olci/l1b/2021/030/S3B_OL_1_EFR____20210130T153242_20210130T153542_20210131T190743_0179_048_282_2160_LN1_O_NT_002.zip https://oceandata.sci.gsfc.nasa.gov/cgi/ob/S3B_OL_1_EFR____20210130T153242_20210130T153542_20210131T190743_0179_048_282_2160_LN1_O_NT_002.zip  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current                                 Dload  Upload   Total   Spent    Left  Speed102   716  102   716    0     0   3062      0 --:--:-- --:--:-- --:--:-- 18842$ file /cms_zfs/sat_data/olci/l1b/2021/030/S3B_OL_1_EFR____20210130T153242_20210130T153542_20210131T190743_0179_048_282_2160_LN1_O_NT_002.zip/cms_zfs/sat_data/olci/l1b/2021/030/S3B_OL_1_EFR____20210130T153242_20210130T153542_20210131T190743_0179_048_282_2160_LN1_O_NT_002.zip: ASCII HTML document textActually I'm sorting through many issues on 2 separate sites.

[oo_processing]

Sean,Seems that the OLCI data is not found?$ curl -b .urs_cookies_single -c .urs_cookies_single -L -n --retry 5 --retry-delay 2 --max-time 0 --verbose --output ./S3B_OL_1_EFR____20210130T153542_20210130T153842_20210131T190804_0179_048_282_2340_LN1_O_NT_002.zip https://oceandata.sci.gsfc.nasa.gov/cgi/ob/S3B_OL_1_EFR____20210130T153542_20210130T153842_20210131T190804_0179_048_282_2340_LN1_O_NT_002.zip  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current                                 Dload  Upload   Total   Spent    Left  Speed  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* About to connect() to oceandata.sci.gsfc.nasa.gov port 443 (#0)*   Trying xx.xxx.xx.xx...* Connected to oceandata.sci.gsfc.nasa.gov (xx.xxx.xx.xx) port 443 (#0)* Initializing NSS with certpath: sql:/etc/pki/nssdb*   CAfile: /etc/pki/tls/certs/ca-bundle.crt  CApath: none* SSL connection using TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256* Server certificate:*       subject: CN=oceancolor.gsfc.nasa.gov*       start date: Jan 23 20:03:12 2021 GMT*       expire date: Apr 23 20:03:12 2021 GMT*       common name: oceancolor.gsfc.nasa.gov*       issuer: CN=R3,O=Let's Encrypt,C=US* Server auth using Basic with user 'oo_processing'> GET /cgi/ob/S3B_OL_1_EFR____20210130T153542_20210130T153842_20210131T190804_0179_048_282_2340_LN1_O_NT_002.zip HTTP/1.1> Authorization: Basic b29fcHJvY2Vzc2luZzpJZzJpd2tzRkpVaHY=> User-Agent: curl/7.29.0> Host: oceandata.sci.gsfc.nasa.gov> Accept: */*> < HTTP/1.1 404 Not Found< Server: nginx< Date: Thu, 04 Feb 2021 18:23:57 GMT< Content-Type: text/html; charset=utf-8< Content-Length: 716< Connection: keep-alive< { [data not shown]100   716  100   716    0     0   1156      0 --:--:-- --:--:-- --:--:--  1156* Connection #0 to host oceandata.sci.gsfc.nasa.gov left intact[bmurch@dell8 work]$ curl -b .urs_cookies_single -c .urs_cookies_single -L -n --retry 5 --retry-delay 2 --max-time 0 --verbose --output ./S3B_OL_1_EFR____20210201T144320_20210201T144620_20210202T202204_0179_048_310_2340_LN1_O_NT_002.zip  https://oceandata.sci.gsfc.nasa.gov/cgi/ob/S3B_OL_1_EFR____20210201T144320_20210201T144620_20210202T202204_0179_048_310_2340_LN1_O_NT_002.zip  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current                                 Dload  Upload   Total   Spent    Left  Speed  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* About to connect() to oceandata.sci.gsfc.nasa.gov port 443 (#0)*   Trying xx.xxx.xx.xx...* Connected to oceandata.sci.gsfc.nasa.gov (xx.xxx.xx.xx) port 443 (#0)* Initializing NSS with certpath: sql:/etc/pki/nssdb*   CAfile: /etc/pki/tls/certs/ca-bundle.crt  CApath: none* SSL connection using TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256* Server certificate:*       subject: CN=oceancolor.gsfc.nasa.gov*       start date: Jan 23 20:03:12 2021 GMT*       expire date: Apr 23 20:03:12 2021 GMT*       common name: oceancolor.gsfc.nasa.gov*       issuer: CN=R3,O=Let's Encrypt,C=US* Server auth using Basic with user 'oo_processing'> GET /cgi/ob/S3B_OL_1_EFR____20210201T144320_20210201T144620_20210202T202204_0179_048_310_2340_LN1_O_NT_002.zip HTTP/1.1> Authorization: Basic b29fcHJvY2Vzc2luZzpJZzJpd2tzRkpVaHY=> User-Agent: curl/7.29.0> Host: oceandata.sci.gsfc.nasa.gov> Accept: */*> < HTTP/1.1 404 Not Found< Server: nginx< Date: Thu, 04 Feb 2021 18:24:01 GMT< Content-Type: text/html; charset=utf-8< Content-Length: 716< Connection: keep-alive< { [data not shown]100   716  100   716    0     0    966      0 --:--:-- --:--:-- --:--:--   967* Connection #0 to host oceandata.sci.gsfc.nasa.gov left intact

[oo_processing]

Sean,I think that https://oceandata.sci.gsfc.nasa.gov/sentinel/getfile/used to work as well.No longer.curl -b .urs_cookies_single -c .urs_cookies_single -L -n --retry 5 --retry-delay 2 --max-time 0 --output ./S3A_OL_1_ERR____20210131T021455_20210131T025842_20210201T064939_2627_068_046______LN1_O_NT_002.zip  https://oceandata.sci.gsfc.nasa.gov/sentinel/getfile/S3A_OL_1_ERR____20210131T021455_20210131T025842_20210201T064939_2627_068_046______LN1_O_NT_002.zip  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current                                 Dload  Upload   Total   Spent    Left  Speed  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0...  0     0    0     0    0     0      0      0 --:--:--  0:00:05 --:--:--     0curl: (47) Maximum (50) redirects followedBrock

[OB.DAACx - SeanBailey]

I've put "top men" on it...I just hope they can find the right box in that warehouse....  If ob/getfile works, for now may just want to not use sentinel/getfile...

Sean

[oo_processing]

Sean,I think only the cgi/ob works now. It doesn't get the redirect error.$ curl --verbose  -b .urs_cookies_single -c .urs_cookies_single -L -n --retry 5 --retry-delay 2 --max-time 0 --output ./S3A_OL_1_ERR____20210131T021455_20210131T025842_20210201T064939_2627_068_046______LN1_O_NT_002.zip  https://oceandata.sci.gsfc.nasa.gov/cgi/ob/S3A_OL_1_ERR____20210131T021455_20210131T025842_20210201T064939_2627_068_046______LN1_O_NT_002.zip  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current                                 Dload  Upload   Total   Spent    Left  Speed  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* About to connect() to oceandata.sci.gsfc.nasa.gov port 443 (#0)*   Trying xxx.xx.xxx...* Connected to oceandata.sci.gsfc.nasa.gov (xxx.xx.xxx) port 443 (#0)* Initializing NSS with certpath: sql:/etc/pki/nssdb*   CAfile: /etc/pki/tls/certs/ca-bundle.crt  CApath: none* SSL connection using TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256* Server certificate:*       subject: CN=oceancolor.gsfc.nasa.gov*       start date: Jan 23 20:03:12 2021 GMT*       expire date: Apr 23 20:03:12 2021 GMT*       common name: oceancolor.gsfc.nasa.gov*       issuer: CN=R3,O=Let's Encrypt,C=US* Server auth using Basic with user 'oo_processing'> GET /cgi/ob/S3A_OL_1_ERR____20210131T021455_20210131T025842_20210201T064939_2627_068_046______LN1_O_NT_002.zip HTTP/1.1> Authorization: Basic b29fcHJvY2Vzc2luZzpJZzJpd2tzRkpVaHY=> User-Agent: curl/7.29.0> Host: oceandata.sci.gsfc.nasa.gov> Accept: */*> Cookie: plack_session2=31eeb0c9a988b75bc8eb626ceb9664d17edf1935> < HTTP/1.1 404 Not Found< Server: nginx< Date: Thu, 04 Feb 2021 22:03:18 GMT< Content-Type: text/html; charset=utf-8< Content-Length: 716< Connection: keep-alive< { [data not shown]100   716  100   716    0     0   1156      0 --:--:-- --:--:-- --:--:--  1156* Connection #0 to host oceandata.sci.gsfc.nasa.gov left intact[bmurch@dell8 work]$ file /cgi/ob/S3A_OL_1_ERR____20210131T021455_20210131T025842_20210201T064939_2627_068_046______LN1_O_NT_002.zip/cgi/ob/S3A_OL_1_ERR____20210131T021455_20210131T025842_20210201T064939_2627_068_046______LN1_O_NT_002.zip: cannot open (No such file or directory)[bmurch@dell8 work]$ file ./S3A_OL_1_ERR____20210131T021455_20210131T025842_20210201T064939_2627_068_046______LN1_O_NT_002.zip./S3A_OL_1_ERR____20210131T021455_20210131T025842_20210201T064939_2627_068_046______LN1_O_NT_002.zip: HTML document, ASCII text

[dana.r.wilson]

The issue have been resolved. Both url's are valid.

https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/S3A_OL_1_EFR____20210105T235944_20210106T000244_20210107T033447_0180_067_073_3060_LN1_O_NT_002.zip

https://oceandata.sci.gsfc.nasa.gov/sentinel/getfile/S3A_OL_1_EFR____20210105T235944_20210106T000244_20210107T033447_0180_067_073_3060_LN1_O_NT_002.zip

[oo_processing]

Sean,Thanks. I was able to grab all my sentinel files returned as HTML now.Except for this one:$ curl -b .urs_cookies_single -c .urs_cookies_single -L -n --interface 2607:fe50:0:6330::104 --retry 5 --retry-delay 2 --max-time 0 --output  ./S3A_OL_1_EFR____20210203T024335_20210203T024635_20210203T043623_0179_068_089_2340_MAR_O_NR_002.zip   https://oceandata.sci.gsfc.nasa.gov
/cgi/getfile/S3A_OL_1_EFR____20210203T024335_20210203T024635_20210203T043623_0179_068_089_2340_MAR_O_NR_002.zip  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current                                 Dload  Upload   Total   Spent    Left  Speed102   716  102   716    0     0    129      0  0:00:05  0:00:05 --:--:--   171[bmurch@optics0 034]$ cat ./S3A_OL_1_EFR____20210203T024335_20210203T024635_20210203T043623_0179_068_089_2340_MAR_O_NR_002.zip<!doctype html><html lang='en'><head><meta charset='utf-8'><title>ERROR @ OceanColor Biology Processing Group (OBPG)</title><meta name='viewport' content='width=device-width, initial-scale=1'></head><body link=white vlink=white alink=white style='background-color:#1C4960; color:white; font-size:175%'><br><hr color= #666699><center><h1><b>.:. ERROR .:.</b></h1><h2>OceanColor Biology Processing Group (OBPG)</h2><blockquote>We are sorry, but an error has occurred. Please use the back button to return to the previous page</blockquote><hr color= #666699></body></html><!-- IE needs 512+ bytes: [http]://blogs.msdn.com/b/ieinternals/archive/2010/08/19/http-error-pages-in-internet-explorer.aspx -->