problems with MODIS / VIIRS Data access some anc data d/ls fine

Please enter here to ask a question about any NASA Science related topics!
oo_processing
Posts: 211
Joined: Wed Apr 06, 2005 12:11 pm America/New_York

problems with MODIS / VIIRS Data access some anc data d/ls fine

by oo_processing » Thu Jan 21, 2021 11:15 am America/New_York

Lo all,

Do you have problems going on with various data (MODIS / VIIRS) Data access?

I can't seem to get files across several installations home/saint petersburg/tampa but I can get some ancillary data like N2021020_SST_OIV2AV_24h.nc?

I can get this:
$ curl -b ~/.urs_cookies -c ~/.urs_cookies -L -n --retry 5 --retry-delay 2 --max-time 0 --output ./N2021020_SST_OIV2AV_24h.nc https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/N2021020_SST_OIV2AV_24h.nc
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100 1619k  100 1619k    0     0   708k      0  0:00:02  0:00:02 --:--:-- 2890k

But not this:
$ curl -b .urs_cookies_single -c .urs_cookies_single -L -n --output ./PM1EPHND_NRT.A2021020.1555.06 https://oceandata.sci.gsfc.nasa.gov/cgi/PM1EPHND_NRT.A2021020.1555.06
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   665  100   665    0     0    960      0 --:--:-- --:--:-- --:--:--   959

Or any MODIS/VIIRS

$ curl -b ~/.urs_cookies -c ~/.urs_cookies -L -n --retry 5 --retry-delay 2 --max-time 0 --output ./MOD00.P2021021.0725_1.PDS.bz2 https://oceandata.sci.gsfc.nasa.gov/cgi/MOD00.P2021021.0725_1.PDS.bz2
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   665  100   665    0     0   1205      0 --:--:-- --:--:-- --:--:--  1206
$ curl -b ~/.urs_cookies -c ~/.urs_cookies -L -n --retry 5 --retry-delay 2 --max-time 0 --output ./MOD00.A2021020.1850_1.PDS.bz2 https://oceandata.sci.gsfc.nasa.gov/cgi/MOD00.A2021020.1850_1.PDS.bz2
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   665  100   665    0     0   1203      0 --:--:-- --:--:-- --:--:--  1202

$ curl -b .urs_cookies_single -c .urs_cookies_single -L -n --output ./V2021020045400.L1A_SNPP.nc https://oceandata.sci.gsfc.nasa.gov/cgi/V2021020045400.L1A_SNPP.nc
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   665  100   665    0     0   1180      0 --:--:-- --:--:-- --:--:--  1181

$ file *
MOD00.A2021020.1850_1.PDS.bz2: HTML document, ASCII text, with very long lines
MOD00.P2021021.0725_1.PDS.bz2: HTML document, ASCII text, with very long lines
N2021020_SST_OIV2AV_24h.nc:    bzip2 compressed data, block size = 100k
PM1EPHND_NRT.A2021020.1555.06: HTML document, ASCII text, with very long lines
V2021020045400.L1A_SNPP.nc: HTML document, ASCII text, with very long lines
$ bunzip2 N2021020_SST_OIV2AV_24h.nc
bunzip2: Can't guess original name for N2021020_SST_OIV2AV_24h.nc -- using N2021020_SST_OIV2AV_24h.nc.out
$ mv N2021020_SST_OIV2AV_24h.nc.out N2021020_SST_OIV2AV_24h.nc
$ file N2021020_SST_OIV2AV_24h.nc
N2021020_SST_OIV2AV_24h.nc: Hierarchical Data Format (version 5) data

Tags:

OB ODPS - jgwilding
Subject Matter Expert
Subject Matter Expert
Posts: 110
Joined: Fri Feb 19, 2021 1:09 pm America/New_York

problems with MODIS / VIIRS Data access some anc data d/ls fine

by OB ODPS - jgwilding » Thu Jan 21, 2021 11:27 am America/New_York

The URLs on your failed examples seem to be missing the /getfile/ element of the URL.  The first successful case has it .../cgi/getfile/name-of-file but the others have .../cgi/name-of-file

john

oo_processing
Posts: 211
Joined: Wed Apr 06, 2005 12:11 pm America/New_York

problems with MODIS / VIIRS Data access some anc data d/ls fine

by oo_processing » Thu Jan 21, 2021 11:48 am America/New_York

John,

I think it was a copy paste thing again but here not on the command line again:

From home worked twice:
[bmurch@dell8 test]$ curl -b ./.urs_cookies_get_viirs -c ./.urs_cookies_get_viirs -L -n --retry 5 --retry-delay 2 --max-time 0 --output ./V2020153180600.L1A_SNPP.nc https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/V2020153180600.L1A_SNPP.nc
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100  451M  100  451M    0     0  24.6M      0  0:00:18  0:00:18 --:--:-- 29.6M
[bmurch@dell8 test]$ curl -b ~/.urs_cookies -c ~/.urs_cookies -L -n --retry 5 --retry-delay 2 --max-time 0 --output ./MOD00.P2021021.0725_1.PDS.bz2 https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/MOD00.P2021021.0725_1.PDS.bz2
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100 71.0M  100 71.0M    0     0  15.0M      0  0:00:04  0:00:04 --:--:-- 25.2M

But not this one:
[bmurch@dell8 test]$ curl -b ~/.urs_cookies -c ~/.urs_cookies -L -n --retry 5 --retry-delay 2 --max-time 0 --output ./MOD00.A2021021.0835_1.PDS.bz2 https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/MOD00.A2021021.0835_1.PDS.bz2
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   665  100   665    0     0     16      0  0:00:41  0:00:41 --:--:--   151
Warning: Transient problem: HTTP error Will retry in 2 seconds. 5 retries
Warning: left.
Throwing away 665 bytes
....

Same files from work (xx.xxx.xx.xx)
curl -b ~/.urs_cookies -c ~/.urs_cookies -L -n --retry 5 --retry-delay 2 --max-time 0 --output ./MOD00.P2021021.0725_1.PDS.bz2 https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/MOD00.P2021021.0725_1.PDS.bz2
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   665  100   665    0     0     16      0  0:00:41  0:00:41 --:--:--   208
Warning: Transient problem: HTTP error Will retry in 2 seconds. 5 retries
Warning: left.
....

curl -b ./.urs_cookies_get_viirs -c ./.urs_cookies_get_viirs -L -n --retry 5 --retry-delay 2 --max-time 0 --output ./V2020153180600.L1A_SNPP.nc https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/V2020153180600.L1A_SNPP.nc
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   665  100   665    0     0     16      0  0:00:41  0:00:41 --:--:--   156
Warning: Transient problem: HTTP error Will retry in 2 seconds. 5 retries
Warning: left.
Throwing away 665 bytes
....

From work 2607:fe50:0:6330::103:
These are coming down now as 665 (and 0 length until they arrive as 665 size html)
-rw-rw-r-- 1 bmurch cms_optics       665 Jan 21 11:38 /cms_zfs/sat_data/modis/l0/2021/021/MOD00.A2021021.0835_1.PDS.bz2
-rw-rw-r-- 1 bmurch cms_optics       665 Jan 21 11:34 /cms_zfs/sat_data/modis/l0/2021/021/MOD00.A2021021.0850_1.PDS.bz2
-rw-rw-r-- 1 bmurch cms_optics         0 Jan 21 11:42 /cms_zfs/sat_data/modis/l0/2021/021/MOD00.A2021021.0930_1.PDS.bz2
-rw-rw-r-- 1 bmurch cms_optics       665 Jan 20 21:00 /cms_zfs/sat_data/modis/l0/2021/021/MOD00.P2021021.0010_1.PDS.bz2

OB ODPS - jgwilding
Subject Matter Expert
Subject Matter Expert
Posts: 110
Joined: Fri Feb 19, 2021 1:09 pm America/New_York

problems with MODIS / VIIRS Data access some anc data d/ls fine

by OB ODPS - jgwilding » Thu Jan 21, 2021 12:04 pm America/New_York

Can you view the HTML in a browser to see what it is asking for, such as maybe an Earthdata login prompt?  Verify .netrc values?  Remove the ./.urs_cookies_get_* files to force a new one?

I can't replicate the problem, so the getfile service is functioning.  You're saying it is sporadically failing on two different machines, or always working on one, failing on another?

john

oo_processing
Posts: 211
Joined: Wed Apr 06, 2005 12:11 pm America/New_York

problems with MODIS / VIIRS Data access some anc data d/ls fine

by oo_processing » Thu Jan 21, 2021 12:11 pm America/New_York

It might be IPv6 as well:
Here I try with -4 and it works
[bmurch@optics0 ~]$ curl -4 -b ~/.urs_cookies -c ~/.urs_cookies -L -n --retry 5 --retry-delay 2 --max-time 0 --output ./MOD00.A2021021.0835_1.PDS.bz2 https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/MOD00.A2021021.0835_1.PDS.bz2
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  291M  100  291M    0     0  61.4M      0  0:00:04  0:00:04 --:--:-- 87.3M

[bmurch@optics0 ~]$ rm ./MOD00.A2021021.0835_1.PDS.bz2

Here I try with -6 and it fails:
[bmurch@optics0 ~]$ curl -6 -b ~/.urs_cookies -c ~/.urs_cookies -L -n --retry 5 --retry-delay 2 --max-time 0 --output ./MOD00.A2021021.0835_1.PDS.bz2 https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/MOD00.A2021021.0835_1.PDS.bz2
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
110   665  110   665    0     0    901      0 --:--:-- --:--:-- --:--:--  4650
Warning: Transient problem: HTTP error Will retry in 2 seconds. 5 retries
Warning: left.
^C

[bmurch@optics0 ~]$ rm ./MOD00.A2021021.0835_1.PDS.bz2

Here I try again with -4 and it works again:
[bmurch@optics0 ~]$ curl -4 -b ~/.urs_cookies -c ~/.urs_cookies -L -n --retry 5 --retry-delay 2 --max-time 0 --output ./MOD00.A2021021.0835_1.PDS.bz2 https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/MOD00.A2021021.0835_1.PDS.bz2
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  291M  100  291M    0     0  59.7M      0  0:00:04  0:00:04 --:--:-- 80.8M

[bmurch@optics0 ~]$ rm ./MOD00.A2021021.0835_1.PDS.bz2

Here I try again with -6 and it fails:
[bmurch@optics0 ~]$ curl -6 -b ~/.urs_cookies -c ~/.urs_cookies -L -n --retry 5 --retry-delay 2 --max-time 0 --output ./MOD00.A2021021.0835_1.PDS.bz2 https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/MOD00.A2021021.0835_1.PDS.bz2
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
110   665  110   665    0     0    498      0  0:00:01  0:00:01 --:--:--  1324
Warning: Transient problem: HTTP error Will retry in 2 seconds. 5 retries
Warning: left.
Throwing away 665 bytes
  0   665    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0^C

And fails with the interface specified as well:
$ curl --interface 2607:fe50:0:6330::102 -b ~/.urs_cookies -c ~/.urs_cookies -L -n --retry 5 --retry-delay 2 --max-time 0 --output ./MOD00.A2021021.0835_1.PDS.bz2 https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/MOD00.A2021021.0835_1.PDS.bz2
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
110   665  110   665    0     0    799      0 --:--:-- --:--:-- --:--:--  2309
Warning: Transient problem: HTTP error Will retry in 2 seconds. 5 retries
Warning: left.
^C

oo_processing
Posts: 211
Joined: Wed Apr 06, 2005 12:11 pm America/New_York

problems with MODIS / VIIRS Data access some anc data d/ls fine

by oo_processing » Thu Jan 21, 2021 12:22 pm America/New_York

I was having issues on all machines with sporadic downloads. I just tried again, and it worked:

[bmurch@optics0 ~]$ curl -4 -b ~/.urs_cookies -c ~/.urs_cookies -L -n --retry 5 --retry-delay 2 --max-time 0 --output ./MOD00.A2021021.0835_1.PDS.bz2 https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/MOD00.A2021021.0835_1.PDS.bz2  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  291M  100  291M    0     0  63.4M      0  0:00:04  0:00:04 --:--:-- 83.6M
[bmurch@optics0 ~]$ rm ./MOD00.A2021021.0835_1.PDS.bz2
[bmurch@optics0 ~]$ curl -6 -b ~/.urs_cookies -c ~/.urs_cookies -L -n --retry 5 --retry-delay 2 --max-time 0 --output ./MOD00.A2021021.0835_1.PDS.bz2 https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/MOD00.A2021021.0835_1.PDS.bz2
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  291M  100  291M    0     0  51.6M      0  0:00:05  0:00:05 --:--:-- 65.1M

oo_processing
Posts: 211
Joined: Wed Apr 06, 2005 12:11 pm America/New_York

problems with MODIS / VIIRS Data access some anc data d/ls fine

by oo_processing » Thu Jan 21, 2021 12:39 pm America/New_York

And yet again just now:

Worked:
[bmurch@optics0 l0]$ curl -b .urs_cookies_single -c .urs_cookies_single -L -n --interface 2607:fe50:0:6330::103 --retry 5 --retry-delay 2 --max-time 0  --output /cms_zfs/sat_data/modis/l0/2021/020/MOD00.P2021020.2315_1.PDS.bz2 https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/MOD00.P2021020.2315_1.PDS.bz2
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 70.8M 100 70.8M    0     0  22.9M      0  0:00:03  0:00:03 --:--:-- 54.5M
[bmurch@optics0 l0]$ file /cms_zfs/sat_data/modis/l0/2021/021/MOD00.P2021021.0010_1.PDS.bz2

Failed:
[bmurch@optics0 l0]$ curl -b .urs_cookies_single -c .urs_cookies_single -L -n --interface 2607:fe50:0:6330::103 --retry 5 --retry-delay 2 --max-time 0  --output /cms_zfs/sat_data/modis/l0/2021/021/MOD00.P2021021.0010_1.PDS.bz2 https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/MOD00.P2021021.0010_1.PDS.bz2
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
110   665  110   665    0     0    446      0  0:00:01  0:00:01 --:--:--  1319
Warning: Transient problem: HTTP error Will retry in 2 seconds. 5 retries
Warning: left.
Throwing away 665 bytes

Then I up arrowed and tried again WORKED!:
[bmurch@optics0 l0]$ curl -b .urs_cookies_single -c .urs_cookies_single -L -n --interface 2607:fe50:0:6330::103 --retry 5 --retry-delay 2 --max-time 0  --output /cms_zfs/sat_data/modis/l0/2021/021/MOD00.P2021021.0010_1.PDS.bz2 https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/MOD00.P2021021.0010_1.PDS.bz2
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  279M  100  279M    0     0  57.8M      0  0:00:04  0:00:04 --:--:-- 80.2M

oo_processing
Posts: 211
Joined: Wed Apr 06, 2005 12:11 pm America/New_York

problems with MODIS / VIIRS Data access some anc data d/ls fine

by oo_processing » Thu Jan 21, 2021 12:48 pm America/New_York

Same problem with OLCI data:
[bmurch@optics0 seadas_l1a_geo_extract_rt]$ file /cms_zfs/sat_data/olci/l1b/2021/020/S3* | grep HTML
/cms_zfs/sat_data/olci/l1b/2021/020/S3A_OL_1_ERR____20210120T070347_20210120T074738_20210121T104835_2631_067_277______LN1_O_NT_002.zip: HTML document text
/cms_zfs/sat_data/olci/l1b/2021/020/S3B_OL_1_ERR____20210120T062420_20210120T070810_20210121T104400_2630_048_134______LN1_O_NT_002.zip: HTML document text
/cms_zfs/sat_data/olci/l1b/2021/020/S3B_OL_1_ERR____20210120T080519_20210120T084909_20210121T122415_2630_048_135______LN1_O_NT_002.zip: HTML document text
/cms_zfs/sat_data/olci/l1b/2021/020/S3B_OL_1_ERR____20210120T094618_20210120T103008_20210121T140610_2630_048_136______LN1_O_NT_002.zip: HTML document text
/cms_zfs/sat_data/olci/l1b/2021/020/S3B_OL_1_ERR____20210120T112717_20210120T121107_20210121T154357_2630_048_137______LN1_O_NT_002.zip: HTML document text
[bmurch@optics0 seadas_l1a_geo_extract_rt]$ ll /cms_zfs/sat_data/olci/l1b/2021/020/S3A_OL_1_ERR____20210120T070347_20210120T074738_20210121T104835_2631_067_277______LN1_O_NT_002.zip
-rw-rw-r-- 1 bmurch cms_optics 665 Jan 21 12:35 /cms_zfs/sat_data/olci/l1b/2021/020/S3A_OL_1_ERR____20210120T070347_20210120T074738_20210121T104835_2631_067_277______LN1_O_NT_002.zip

[bmurch@optics0 l0]$ rm .urs_cookies_single;  curl -b .urs_cookies_single -c .urs_cookies_single -L -n --interface 2607:fe50:0:6330::103 --retry 5 --retry-delay 2 --max-time 0  --output /cms_zfs/sat_data/olci/l1b/2021/020/S3B_OL_1_ERR____20210120T062420_20210120T070810_20210121T104400_2630_048_134______LN1_O_NT_002.zip https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/S3B_OL_1_ERR____20210120T062420_20210120T070810_20210121T104400_2630_048_134______LN1_O_NT_002.zip
rm: cannot remove `.urs_cookies_single': No such file or directory
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
110   665  110   665    0     0    738      0 --:--:-- --:--:-- --:--:--  2309
Warning: Transient problem: HTTP error Will retry in 2 seconds. 5 retries
Warning: left.
^C
And then this one worked when I up arroed and hit enter:
[bmurch@optics0 l0]$ rm .urs_cookies_single;  curl -b .urs_cookies_single -c .urs_cookies_single -L -n --interface 2607:fe50:0:6330::104 --retry 5 --retry-delay 2 --max-time 0  --output /cms_zfs/sat_data/olci/l1b/2021/020/S3B_OL_1_ERR____20210120T062420_20210120T070810_20210121T104400_2630_048_134______LN1_O_NT_002.zip https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/S3B_OL_1_ERR____20210120T062420_20210120T070810_20210121T104400_2630_048_134______LN1_O_NT_002.zip
rm: cannot remove `.urs_cookies_single': No such file or directory
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  681M  100  681M    0     0  73.5M      0  0:00:09  0:00:09 --:--:-- 94.8M

OB.DAAC - SeanBailey
User Services
User Services
Posts: 1192
Joined: Wed Sep 18, 2019 6:15 pm America/New_York

problems with MODIS / VIIRS Data access some anc data d/ls fine

by OB.DAAC - SeanBailey » Thu Jan 21, 2021 1:57 pm America/New_York

Brock,

Sometimes hardware behaves badly...one of our backend machines was acting up.  Our firewall was occasionally (and in a kinda sticky manner) sending your traffic to that machine.  The offending machine has been taken out back and shot....well removed from service until it's memory issues can be resolved. 

Sean

oo_processing
Posts: 211
Joined: Wed Apr 06, 2005 12:11 pm America/New_York

problems with MODIS / VIIRS Data access some anc data d/ls fine

by oo_processing » Thu Jan 21, 2021 2:06 pm America/New_York

Sean,

I understand hardware issues well.

I'm sending you a bunch of Log reports on another post as they are failing during processing.

Thanks,
Brock

Post Reply