problems with MODIS / VIIRS Data access some anc data d/ls fine

oo_processing · by **oo_processing** » Thu Jan 21, 2021 11:15 am America/New_York

Lo all,Do you have problems going on with various data (MODIS / VIIRS) Data access?I can't seem to get files across several installations home/saint petersburg/tampa but I can get some ancillary data like N2021020_SST_OIV2AV_24h.nc?I can get this:$ curl -b ~/.urs_cookies -c ~/.urs_cookies -L -n --retry 5 --retry-delay 2 --max-time 0 --output ./N2021020_SST_OIV2AV_24h.nc https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/N2021020_SST_OIV2AV_24h.nc % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0100 1619k 100 1619k 0 0 708k 0 0:00:02 0:00:02 --:--:-- 2890kBut not this:$ curl -b .urs_cookies_single -c .urs_cookies_single -L -n --output ./PM1EPHND_NRT.A2021020.1555.06 https://oceandata.sci.gsfc.nasa.gov (/cgi/PM1EPHND_NRT.A2021020.1555.06) % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed100 665 100 665 0 0 960 0 --:--:-- --:--:-- --:--:-- 959Or any MODIS/VIIRS$ curl -b ~/.urs_cookies -c ~/.urs_cookies -L -n --retry 5 --retry-delay 2 --max-time 0 --output ./MOD00.P2021021.0725_1.PDS.bz2 https://oceandata.sci.gsfc.nasa.gov (/cgi/MOD00.P2021021.0725_1.PDS.bz2) % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed100 665 100 665 0 0 1205 0 --:--:-- --:--:-- --:--:-- 1206$ curl -b ~/.urs_cookies -c ~/.urs_cookies -L -n --retry 5 --retry-delay 2 --max-time 0 --output ./MOD00.A2021020.1850_1.PDS.bz2 https://oceandata.sci.gsfc.nasa.gov (/cgi/MOD00.A2021020.1850_1.PDS.bz2) % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed100 665 100 665 0 0 1203 0 --:--:-- --:--:-- --:--:-- 1202$ curl -b .urs_cookies_single -c .urs_cookies_single -L -n --output ./V2021020045400.L1A_SNPP.nc https://oceandata.sci.gsfc.nasa.gov (/cgi/V2021020045400.L1A_SNPP.nc) % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed100 665 100 665 0 0 1180 0 --:--:-- --:--:-- --:--:-- 1181$ file *MOD00.A2021020.1850_1.PDS.bz2: HTML document, ASCII text, with very long linesMOD00.P2021021.0725_1.PDS.bz2: HTML document, ASCII text, with very long linesN2021020_SST_OIV2AV_24h.nc: bzip2 compressed data, block size = 100kPM1EPHND_NRT.A2021020.1555.06: HTML document, ASCII text, with very long linesV2021020045400.L1A_SNPP.nc: HTML document, ASCII text, with very long lines$ bunzip2 N2021020_SST_OIV2AV_24h.ncbunzip2: Can't guess original name for N2021020_SST_OIV2AV_24h.nc -- using N2021020_SST_OIV2AV_24h.nc.out$ mv N2021020_SST_OIV2AV_24h.nc.out N2021020_SST_OIV2AV_24h.nc$ file N2021020_SST_OIV2AV_24h.ncN2021020_SST_OIV2AV_24h.nc: Hierarchical Data Format (version 5) data

by **OB ODPS - jgwilding** » Thu Jan 21, 2021 11:27 am America/New_York

The URLs on your failed examples seem to be missing the /getfile/ element of the URL. The first successful case has it .../cgi/getfile/name-of-file but the others have .../cgi/name-of-file

john

oo_processing · by **oo_processing** » Thu Jan 21, 2021 11:48 am America/New_York

John,

I think it was a copy paste thing again but here not on the command line again:

From home worked twice:
[bmurch@dell8 test]$ curl -b ./.urs_cookies_get_viirs -c ./.urs_cookies_get_viirs -L -n --retry 5 --retry-delay 2 --max-time 0 --output ./V2020153180600.L1A_SNPP.nc https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/V2020153180600.L1A_SNPP.nc
% Total % Received % Xferd Average Speed   Time Time    Time Current
   Dload Upload   Total   Spent Left Speed
0    0 0    0 0    0 0 0 --:--:-- --:--:-- --:--:--    0
100 451M 100 451M 0    0 24.6M 0 0:00:18 0:00:18 --:--:-- 29.6M
[bmurch@dell8 test]$ curl -b ~/.urs_cookies -c ~/.urs_cookies -L -n --retry 5 --retry-delay 2 --max-time 0 --output ./MOD00.P2021021.0725_1.PDS.bz2 https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/MOD00.P2021021.0725_1.PDS.bz2
% Total % Received % Xferd Average Speed   Time Time    Time Current
   Dload Upload   Total   Spent Left Speed
0    0 0    0 0    0 0 0 --:--:-- --:--:-- --:--:--    0
100 71.0M 100 71.0M 0    0 15.0M 0 0:00:04 0:00:04 --:--:-- 25.2M

But not this one:
[bmurch@dell8 test]$ curl -b ~/.urs_cookies -c ~/.urs_cookies -L -n --retry 5 --retry-delay 2 --max-time 0 --output ./MOD00.A2021021.0835_1.PDS.bz2 https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/MOD00.A2021021.0835_1.PDS.bz2
% Total % Received % Xferd Average Speed   Time Time    Time Current
   Dload Upload   Total   Spent Left Speed
100   665 100   665 0    0    16 0 0:00:41 0:00:41 --:--:--   151
Warning: Transient problem: HTTP error Will retry in 2 seconds. 5 retries
Warning: left.
Throwing away 665 bytes
....

Same files from work (xx.xxx.xx.xx)
curl -b ~/.urs_cookies -c ~/.urs_cookies -L -n --retry 5 --retry-delay 2 --max-time 0 --output ./MOD00.P2021021.0725_1.PDS.bz2 https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/MOD00.P2021021.0725_1.PDS.bz2
% Total % Received % Xferd Average Speed   Time Time    Time Current
   Dload Upload   Total   Spent Left Speed
100   665 100   665 0    0    16 0 0:00:41 0:00:41 --:--:--   208
Warning: Transient problem: HTTP error Will retry in 2 seconds. 5 retries
Warning: left.
....

curl -b ./.urs_cookies_get_viirs -c ./.urs_cookies_get_viirs -L -n --retry 5 --retry-delay 2 --max-time 0 --output ./V2020153180600.L1A_SNPP.nc https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/V2020153180600.L1A_SNPP.nc
% Total % Received % Xferd Average Speed   Time Time    Time Current
   Dload Upload   Total   Spent Left Speed
100   665 100   665 0    0    16 0 0:00:41 0:00:41 --:--:--   156
Warning: Transient problem: HTTP error Will retry in 2 seconds. 5 retries
Warning: left.
Throwing away 665 bytes
....

From work 2607:fe50:0:6330::103:
These are coming down now as 665 (and 0 length until they arrive as 665 size html)
-rw-rw-r-- 1 bmurch cms_optics    665 Jan 21 11:38 /cms_zfs/sat_data/modis/l0/2021/021/MOD00.A2021021.0835_1.PDS.bz2
-rw-rw-r-- 1 bmurch cms_optics    665 Jan 21 11:34 /cms_zfs/sat_data/modis/l0/2021/021/MOD00.A2021021.0850_1.PDS.bz2
-rw-rw-r-- 1 bmurch cms_optics    0 Jan 21 11:42 /cms_zfs/sat_data/modis/l0/2021/021/MOD00.A2021021.0930_1.PDS.bz2
-rw-rw-r-- 1 bmurch cms_optics    665 Jan 20 21:00 /cms_zfs/sat_data/modis/l0/2021/021/MOD00.P2021021.0010_1.PDS.bz2

by **OB ODPS - jgwilding** » Thu Jan 21, 2021 12:04 pm America/New_York

Can you view the HTML in a browser to see what it is asking for, such as maybe an Earthdata login prompt? Verify .netrc values? Remove the ./.urs_cookies_get_* files to force a new one?

I can't replicate the problem, so the getfile service is functioning. You're saying it is sporadically failing on two different machines, or always working on one, failing on another?

john

oo_processing · by **oo_processing** » Thu Jan 21, 2021 12:11 pm America/New_York

It might be IPv6 as well:
Here I try with -4 and it works
[bmurch@optics0 ~]$ curl -4 -b ~/.urs_cookies -c ~/.urs_cookies -L -n --retry 5 --retry-delay 2 --max-time 0 --output ./MOD00.A2021021.0835_1.PDS.bz2 https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/MOD00.A2021021.0835_1.PDS.bz2
% Total % Received % Xferd Average Speed   Time Time    Time Current
   Dload Upload   Total   Spent Left Speed
100 291M 100 291M 0    0 61.4M 0 0:00:04 0:00:04 --:--:-- 87.3M

[bmurch@optics0 ~]$ rm ./MOD00.A2021021.0835_1.PDS.bz2

Here I try with -6 and it fails:
[bmurch@optics0 ~]$ curl -6 -b ~/.urs_cookies -c ~/.urs_cookies -L -n --retry 5 --retry-delay 2 --max-time 0 --output ./MOD00.A2021021.0835_1.PDS.bz2 https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/MOD00.A2021021.0835_1.PDS.bz2
% Total % Received % Xferd Average Speed   Time Time    Time Current
   Dload Upload   Total   Spent Left Speed
110   665 110   665 0    0 901 0 --:--:-- --:--:-- --:--:-- 4650
Warning: Transient problem: HTTP error Will retry in 2 seconds. 5 retries
Warning: left.
^C

[bmurch@optics0 ~]$ rm ./MOD00.A2021021.0835_1.PDS.bz2

Here I try again with -4 and it works again:
[bmurch@optics0 ~]$ curl -4 -b ~/.urs_cookies -c ~/.urs_cookies -L -n --retry 5 --retry-delay 2 --max-time 0 --output ./MOD00.A2021021.0835_1.PDS.bz2 https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/MOD00.A2021021.0835_1.PDS.bz2
% Total % Received % Xferd Average Speed   Time Time    Time Current
   Dload Upload   Total   Spent Left Speed
100 291M 100 291M 0    0 59.7M 0 0:00:04 0:00:04 --:--:-- 80.8M

[bmurch@optics0 ~]$ rm ./MOD00.A2021021.0835_1.PDS.bz2

Here I try again with -6 and it fails:
[bmurch@optics0 ~]$ curl -6 -b ~/.urs_cookies -c ~/.urs_cookies -L -n --retry 5 --retry-delay 2 --max-time 0 --output ./MOD00.A2021021.0835_1.PDS.bz2 https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/MOD00.A2021021.0835_1.PDS.bz2
% Total % Received % Xferd Average Speed   Time Time    Time Current
   Dload Upload   Total   Spent Left Speed
110   665 110   665 0    0 498 0 0:00:01 0:00:01 --:--:-- 1324
Warning: Transient problem: HTTP error Will retry in 2 seconds. 5 retries
Warning: left.
Throwing away 665 bytes
0   665 0    0 0    0 0 0 --:--:-- --:--:-- --:--:--    0^C

And fails with the interface specified as well:
$ curl --interface 2607:fe50:0:6330::102 -b ~/.urs_cookies -c ~/.urs_cookies -L -n --retry 5 --retry-delay 2 --max-time 0 --output ./MOD00.A2021021.0835_1.PDS.bz2 https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/MOD00.A2021021.0835_1.PDS.bz2
% Total % Received % Xferd Average Speed   Time Time    Time Current
   Dload Upload   Total   Spent Left Speed
110   665 110   665 0    0 799 0 --:--:-- --:--:-- --:--:-- 2309
Warning: Transient problem: HTTP error Will retry in 2 seconds. 5 retries
Warning: left.
^C

oo_processing · by **oo_processing** » Thu Jan 21, 2021 12:22 pm America/New_York

I was having issues on all machines with sporadic downloads. I just tried again, and it worked:

[bmurch@optics0 ~]$ curl -4 -b ~/.urs_cookies -c ~/.urs_cookies -L -n --retry 5 --retry-delay 2 --max-time 0 --output ./MOD00.A2021021.0835_1.PDS.bz2 https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/MOD00.A2021021.0835_1.PDS.bz2 % Total % Received % Xferd Average Speed   Time Time    Time Current
   Dload Upload   Total   Spent Left Speed
100 291M 100 291M 0    0 63.4M 0 0:00:04 0:00:04 --:--:-- 83.6M
[bmurch@optics0 ~]$ rm ./MOD00.A2021021.0835_1.PDS.bz2
[bmurch@optics0 ~]$ curl -6 -b ~/.urs_cookies -c ~/.urs_cookies -L -n --retry 5 --retry-delay 2 --max-time 0 --output ./MOD00.A2021021.0835_1.PDS.bz2 https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/MOD00.A2021021.0835_1.PDS.bz2
% Total % Received % Xferd Average Speed   Time Time    Time Current
   Dload Upload   Total   Spent Left Speed
100 291M 100 291M 0    0 51.6M 0 0:00:05 0:00:05 --:--:-- 65.1M

oo_processing · by **oo_processing** » Thu Jan 21, 2021 12:39 pm America/New_York

And yet again just now:

Worked:
[bmurch@optics0 l0]$ curl -b .urs_cookies_single -c .urs_cookies_single -L -n --interface 2607:fe50:0:6330::103 --retry 5 --retry-delay 2 --max-time 0 --output /cms_zfs/sat_data/modis/l0/2021/020/MOD00.P2021020.2315_1.PDS.bz2 https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/MOD00.P2021020.2315_1.PDS.bz2
% Total % Received % Xferd Average Speed   Time Time    Time Current
   Dload Upload   Total   Spent Left Speed
100 70.8M 100 70.8M    0    0 22.9M 0 0:00:03 0:00:03 --:--:-- 54.5M
[bmurch@optics0 l0]$ file /cms_zfs/sat_data/modis/l0/2021/021/MOD00.P2021021.0010_1.PDS.bz2

Failed:
[bmurch@optics0 l0]$ curl -b .urs_cookies_single -c .urs_cookies_single -L -n --interface 2607:fe50:0:6330::103 --retry 5 --retry-delay 2 --max-time 0 --output /cms_zfs/sat_data/modis/l0/2021/021/MOD00.P2021021.0010_1.PDS.bz2 https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/MOD00.P2021021.0010_1.PDS.bz2
% Total % Received % Xferd Average Speed   Time Time    Time Current
   Dload Upload   Total   Spent Left Speed
110   665 110   665 0    0 446 0 0:00:01 0:00:01 --:--:-- 1319
Warning: Transient problem: HTTP error Will retry in 2 seconds. 5 retries
Warning: left.
Throwing away 665 bytes

Then I up arrowed and tried again WORKED!:
[bmurch@optics0 l0]$ curl -b .urs_cookies_single -c .urs_cookies_single -L -n --interface 2607:fe50:0:6330::103 --retry 5 --retry-delay 2 --max-time 0 --output /cms_zfs/sat_data/modis/l0/2021/021/MOD00.P2021021.0010_1.PDS.bz2 https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/MOD00.P2021021.0010_1.PDS.bz2
% Total % Received % Xferd Average Speed   Time Time    Time Current
   Dload Upload   Total   Spent Left Speed
100 279M 100 279M 0    0 57.8M 0 0:00:04 0:00:04 --:--:-- 80.2M

oo_processing · by **oo_processing** » Thu Jan 21, 2021 12:48 pm America/New_York

Same problem with OLCI data:
[bmurch@optics0 seadas_l1a_geo_extract_rt]$ file /cms_zfs/sat_data/olci/l1b/2021/020/S3* | grep HTML
/cms_zfs/sat_data/olci/l1b/2021/020/S3A_OL_1_ERR____20210120T070347_20210120T074738_20210121T104835_2631_067_277______LN1_O_NT_002.zip: HTML document text
/cms_zfs/sat_data/olci/l1b/2021/020/S3B_OL_1_ERR____20210120T062420_20210120T070810_20210121T104400_2630_048_134______LN1_O_NT_002.zip: HTML document text
/cms_zfs/sat_data/olci/l1b/2021/020/S3B_OL_1_ERR____20210120T080519_20210120T084909_20210121T122415_2630_048_135______LN1_O_NT_002.zip: HTML document text
/cms_zfs/sat_data/olci/l1b/2021/020/S3B_OL_1_ERR____20210120T094618_20210120T103008_20210121T140610_2630_048_136______LN1_O_NT_002.zip: HTML document text
/cms_zfs/sat_data/olci/l1b/2021/020/S3B_OL_1_ERR____20210120T112717_20210120T121107_20210121T154357_2630_048_137______LN1_O_NT_002.zip: HTML document text
[bmurch@optics0 seadas_l1a_geo_extract_rt]$ ll /cms_zfs/sat_data/olci/l1b/2021/020/S3A_OL_1_ERR____20210120T070347_20210120T074738_20210121T104835_2631_067_277______LN1_O_NT_002.zip
-rw-rw-r-- 1 bmurch cms_optics 665 Jan 21 12:35 /cms_zfs/sat_data/olci/l1b/2021/020/S3A_OL_1_ERR____20210120T070347_20210120T074738_20210121T104835_2631_067_277______LN1_O_NT_002.zip

[bmurch@optics0 l0]$ rm .urs_cookies_single; curl -b .urs_cookies_single -c .urs_cookies_single -L -n --interface 2607:fe50:0:6330::103 --retry 5 --retry-delay 2 --max-time 0 --output /cms_zfs/sat_data/olci/l1b/2021/020/S3B_OL_1_ERR____20210120T062420_20210120T070810_20210121T104400_2630_048_134______LN1_O_NT_002.zip https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/S3B_OL_1_ERR____20210120T062420_20210120T070810_20210121T104400_2630_048_134______LN1_O_NT_002.zip
rm: cannot remove `.urs_cookies_single': No such file or directory
% Total % Received % Xferd Average Speed   Time Time    Time Current
   Dload Upload   Total   Spent Left Speed
110   665 110   665 0    0 738 0 --:--:-- --:--:-- --:--:-- 2309
Warning: Transient problem: HTTP error Will retry in 2 seconds. 5 retries
Warning: left.
^C
And then this one worked when I up arroed and hit enter:
[bmurch@optics0 l0]$ rm .urs_cookies_single; curl -b .urs_cookies_single -c .urs_cookies_single -L -n --interface 2607:fe50:0:6330::104 --retry 5 --retry-delay 2 --max-time 0 --output /cms_zfs/sat_data/olci/l1b/2021/020/S3B_OL_1_ERR____20210120T062420_20210120T070810_20210121T104400_2630_048_134______LN1_O_NT_002.zip https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/S3B_OL_1_ERR____20210120T062420_20210120T070810_20210121T104400_2630_048_134______LN1_O_NT_002.zip
rm: cannot remove `.urs_cookies_single': No such file or directory
% Total % Received % Xferd Average Speed   Time Time    Time Current
   Dload Upload   Total   Spent Left Speed
100 681M 100 681M 0    0 73.5M 0 0:00:09 0:00:09 --:--:-- 94.8M

OB.DAACx - SeanBailey · by **OB.DAACx - SeanBailey** » Thu Jan 21, 2021 1:57 pm America/New_York

Brock,

Sometimes hardware behaves badly...one of our backend machines was acting up. Our firewall was occasionally (and in a kinda sticky manner) sending your traffic to that machine. The offending machine has been taken out back and shot....well removed from service until it's memory issues can be resolved.

Sean

oo_processing · by **oo_processing** » Thu Jan 21, 2021 2:06 pm America/New_York

Sean,

I understand hardware issues well.

I'm sending you a bunch of Log reports on another post as they are failing during processing.

Thanks,
Brock