Earthdata Forum

Lo all,Do you have problems going on with various data (MODIS / VIIRS) Data access?I can't seem to get files across several installations home/saint petersburg/tampa but I can get some ancillary data like N2021020_SST_OIV2AV_24h.nc?I can get this:$ curl -b ~/.urs_cookies -c ~/.urs_cookies -L -n --retry 5 --retry-delay 2 --max-time 0 --output ./N2021020_SST_OIV2AV_24h.nc https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/N2021020_SST_OIV2AV_24h.nc  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current                                 Dload  Upload   Total   Spent    Left  Speed  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0100 1619k  100 1619k    0     0   708k      0  0:00:02  0:00:02 --:--:-- 2890kBut not this:$ curl -b .urs_cookies_single -c .urs_cookies_single -L -n --output ./PM1EPHND_NRT.A2021020.1555.06 https://oceandata.sci.gsfc.nasa.gov (/cgi/PM1EPHND_NRT.A2021020.1555.06)  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current                                 Dload  Upload   Total   Spent    Left  Speed100   665  100   665    0     0    960      0 --:--:-- --:--:-- --:--:--   959Or any MODIS/VIIRS$ curl -b ~/.urs_cookies -c ~/.urs_cookies -L -n --retry 5 --retry-delay 2 --max-time 0 --output ./MOD00.P2021021.0725_1.PDS.bz2 https://oceandata.sci.gsfc.nasa.gov (/cgi/MOD00.P2021021.0725_1.PDS.bz2)  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current                                 Dload  Upload   Total   Spent    Left  Speed100   665  100   665    0     0   1205      0 --:--:-- --:--:-- --:--:--  1206$ curl -b ~/.urs_cookies -c ~/.urs_cookies -L -n --retry 5 --retry-delay 2 --max-time 0 --output ./MOD00.A2021020.1850_1.PDS.bz2 https://oceandata.sci.gsfc.nasa.gov (/cgi/MOD00.A2021020.1850_1.PDS.bz2)  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current                                 Dload  Upload   Total   Spent    Left  Speed100   665  100   665    0     0   1203      0 --:--:-- --:--:-- --:--:--  1202$ curl -b .urs_cookies_single -c .urs_cookies_single -L -n --output ./V2021020045400.L1A_SNPP.nc https://oceandata.sci.gsfc.nasa.gov (/cgi/V2021020045400.L1A_SNPP.nc)  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current                                 Dload  Upload   Total   Spent    Left  Speed100   665  100   665    0     0   1180      0 --:--:-- --:--:-- --:--:--  1181$ file *MOD00.A2021020.1850_1.PDS.bz2: HTML document, ASCII text, with very long linesMOD00.P2021021.0725_1.PDS.bz2: HTML document, ASCII text, with very long linesN2021020_SST_OIV2AV_24h.nc:    bzip2 compressed data, block size = 100kPM1EPHND_NRT.A2021020.1555.06: HTML document, ASCII text, with very long linesV2021020045400.L1A_SNPP.nc: HTML document, ASCII text, with very long lines$ bunzip2 N2021020_SST_OIV2AV_24h.ncbunzip2: Can't guess original name for N2021020_SST_OIV2AV_24h.nc -- using N2021020_SST_OIV2AV_24h.nc.out$ mv N2021020_SST_OIV2AV_24h.nc.out N2021020_SST_OIV2AV_24h.nc$ file N2021020_SST_OIV2AV_24h.ncN2021020_SST_OIV2AV_24h.nc: Hierarchical Data Format (version 5) data

The URLs on your failed examples seem to be missing the /getfile/ element of the URL.  The first successful case has it .../cgi/getfile/name-of-file but the others have .../cgi/name-of-file

john

John,

I think it was a copy paste thing again but here not on the command line again:

From home worked twice:
[bmurch@dell8 test]$ curl -b ./.urs_cookies_get_viirs -c ./.urs_cookies_get_viirs -L -n --retry 5 --retry-delay 2 --max-time 0 --output ./V2020153180600.L1A_SNPP.nc https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/V2020153180600.L1A_SNPP.nc
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100  451M  100  451M    0     0  24.6M      0  0:00:18  0:00:18 --:--:-- 29.6M
[bmurch@dell8 test]$ curl -b ~/.urs_cookies -c ~/.urs_cookies -L -n --retry 5 --retry-delay 2 --max-time 0 --output ./MOD00.P2021021.0725_1.PDS.bz2 https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/MOD00.P2021021.0725_1.PDS.bz2
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100 71.0M  100 71.0M    0     0  15.0M      0  0:00:04  0:00:04 --:--:-- 25.2M

But not this one:
[bmurch@dell8 test]$ curl -b ~/.urs_cookies -c ~/.urs_cookies -L -n --retry 5 --retry-delay 2 --max-time 0 --output ./MOD00.A2021021.0835_1.PDS.bz2 https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/MOD00.A2021021.0835_1.PDS.bz2
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   665  100   665    0     0     16      0  0:00:41  0:00:41 --:--:--   151
Warning: Transient problem: HTTP error Will retry in 2 seconds. 5 retries
Warning: left.
Throwing away 665 bytes
....

Same files from work (xx.xxx.xx.xx)
curl -b ~/.urs_cookies -c ~/.urs_cookies -L -n --retry 5 --retry-delay 2 --max-time 0 --output ./MOD00.P2021021.0725_1.PDS.bz2 https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/MOD00.P2021021.0725_1.PDS.bz2
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   665  100   665    0     0     16      0  0:00:41  0:00:41 --:--:--   208
Warning: Transient problem: HTTP error Will retry in 2 seconds. 5 retries
Warning: left.
....

curl -b ./.urs_cookies_get_viirs -c ./.urs_cookies_get_viirs -L -n --retry 5 --retry-delay 2 --max-time 0 --output ./V2020153180600.L1A_SNPP.nc https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/V2020153180600.L1A_SNPP.nc
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   665  100   665    0     0     16      0  0:00:41  0:00:41 --:--:--   156
Warning: Transient problem: HTTP error Will retry in 2 seconds. 5 retries
Warning: left.
Throwing away 665 bytes
....

From work 2607:fe50:0:6330::103:
These are coming down now as 665 (and 0 length until they arrive as 665 size html)
-rw-rw-r-- 1 bmurch cms_optics       665 Jan 21 11:38 /cms_zfs/sat_data/modis/l0/2021/021/MOD00.A2021021.0835_1.PDS.bz2
-rw-rw-r-- 1 bmurch cms_optics       665 Jan 21 11:34 /cms_zfs/sat_data/modis/l0/2021/021/MOD00.A2021021.0850_1.PDS.bz2
-rw-rw-r-- 1 bmurch cms_optics         0 Jan 21 11:42 /cms_zfs/sat_data/modis/l0/2021/021/MOD00.A2021021.0930_1.PDS.bz2
-rw-rw-r-- 1 bmurch cms_optics       665 Jan 20 21:00 /cms_zfs/sat_data/modis/l0/2021/021/MOD00.P2021021.0010_1.PDS.bz2

Can you view the HTML in a browser to see what it is asking for, such as maybe an Earthdata login prompt?  Verify .netrc values?  Remove the ./.urs_cookies_get_* files to force a new one?

I can't replicate the problem, so the getfile service is functioning.  You're saying it is sporadically failing on two different machines, or always working on one, failing on another?

john

It might be IPv6 as well:
Here I try with -4 and it works
[bmurch@optics0 ~]$ curl -4 -b ~/.urs_cookies -c ~/.urs_cookies -L -n --retry 5 --retry-delay 2 --max-time 0 --output ./MOD00.A2021021.0835_1.PDS.bz2 https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/MOD00.A2021021.0835_1.PDS.bz2
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  291M  100  291M    0     0  61.4M      0  0:00:04  0:00:04 --:--:-- 87.3M

[bmurch@optics0 ~]$ rm ./MOD00.A2021021.0835_1.PDS.bz2

Here I try with -6 and it fails:
[bmurch@optics0 ~]$ curl -6 -b ~/.urs_cookies -c ~/.urs_cookies -L -n --retry 5 --retry-delay 2 --max-time 0 --output ./MOD00.A2021021.0835_1.PDS.bz2 https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/MOD00.A2021021.0835_1.PDS.bz2
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
110   665  110   665    0     0    901      0 --:--:-- --:--:-- --:--:--  4650
Warning: Transient problem: HTTP error Will retry in 2 seconds. 5 retries
Warning: left.
^C

[bmurch@optics0 ~]$ rm ./MOD00.A2021021.0835_1.PDS.bz2

Here I try again with -4 and it works again:
[bmurch@optics0 ~]$ curl -4 -b ~/.urs_cookies -c ~/.urs_cookies -L -n --retry 5 --retry-delay 2 --max-time 0 --output ./MOD00.A2021021.0835_1.PDS.bz2 https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/MOD00.A2021021.0835_1.PDS.bz2
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  291M  100  291M    0     0  59.7M      0  0:00:04  0:00:04 --:--:-- 80.8M

[bmurch@optics0 ~]$ rm ./MOD00.A2021021.0835_1.PDS.bz2

Here I try again with -6 and it fails:
[bmurch@optics0 ~]$ curl -6 -b ~/.urs_cookies -c ~/.urs_cookies -L -n --retry 5 --retry-delay 2 --max-time 0 --output ./MOD00.A2021021.0835_1.PDS.bz2 https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/MOD00.A2021021.0835_1.PDS.bz2
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
110   665  110   665    0     0    498      0  0:00:01  0:00:01 --:--:--  1324
Warning: Transient problem: HTTP error Will retry in 2 seconds. 5 retries
Warning: left.
Throwing away 665 bytes
  0   665    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0^C

And fails with the interface specified as well:
$ curl --interface 2607:fe50:0:6330::102 -b ~/.urs_cookies -c ~/.urs_cookies -L -n --retry 5 --retry-delay 2 --max-time 0 --output ./MOD00.A2021021.0835_1.PDS.bz2 https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/MOD00.A2021021.0835_1.PDS.bz2
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
110   665  110   665    0     0    799      0 --:--:-- --:--:-- --:--:--  2309
Warning: Transient problem: HTTP error Will retry in 2 seconds. 5 retries
Warning: left.
^C

I was having issues on all machines with sporadic downloads. I just tried again, and it worked:

[bmurch@optics0 ~]$ curl -4 -b ~/.urs_cookies -c ~/.urs_cookies -L -n --retry 5 --retry-delay 2 --max-time 0 --output ./MOD00.A2021021.0835_1.PDS.bz2 https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/MOD00.A2021021.0835_1.PDS.bz2  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  291M  100  291M    0     0  63.4M      0  0:00:04  0:00:04 --:--:-- 83.6M
[bmurch@optics0 ~]$ rm ./MOD00.A2021021.0835_1.PDS.bz2
[bmurch@optics0 ~]$ curl -6 -b ~/.urs_cookies -c ~/.urs_cookies -L -n --retry 5 --retry-delay 2 --max-time 0 --output ./MOD00.A2021021.0835_1.PDS.bz2 https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/MOD00.A2021021.0835_1.PDS.bz2
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  291M  100  291M    0     0  51.6M      0  0:00:05  0:00:05 --:--:-- 65.1M

And yet again just now:

Worked:
[bmurch@optics0 l0]$ curl -b .urs_cookies_single -c .urs_cookies_single -L -n --interface 2607:fe50:0:6330::103 --retry 5 --retry-delay 2 --max-time 0  --output /cms_zfs/sat_data/modis/l0/2021/020/MOD00.P2021020.2315_1.PDS.bz2 https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/MOD00.P2021020.2315_1.PDS.bz2
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 70.8M 100 70.8M    0     0  22.9M      0  0:00:03  0:00:03 --:--:-- 54.5M
[bmurch@optics0 l0]$ file /cms_zfs/sat_data/modis/l0/2021/021/MOD00.P2021021.0010_1.PDS.bz2

Failed:
[bmurch@optics0 l0]$ curl -b .urs_cookies_single -c .urs_cookies_single -L -n --interface 2607:fe50:0:6330::103 --retry 5 --retry-delay 2 --max-time 0  --output /cms_zfs/sat_data/modis/l0/2021/021/MOD00.P2021021.0010_1.PDS.bz2 https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/MOD00.P2021021.0010_1.PDS.bz2
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
110   665  110   665    0     0    446      0  0:00:01  0:00:01 --:--:--  1319
Warning: Transient problem: HTTP error Will retry in 2 seconds. 5 retries
Warning: left.
Throwing away 665 bytes

Then I up arrowed and tried again WORKED!:
[bmurch@optics0 l0]$ curl -b .urs_cookies_single -c .urs_cookies_single -L -n --interface 2607:fe50:0:6330::103 --retry 5 --retry-delay 2 --max-time 0  --output /cms_zfs/sat_data/modis/l0/2021/021/MOD00.P2021021.0010_1.PDS.bz2 https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/MOD00.P2021021.0010_1.PDS.bz2
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  279M  100  279M    0     0  57.8M      0  0:00:04  0:00:04 --:--:-- 80.2M

Same problem with OLCI data:
[bmurch@optics0 seadas_l1a_geo_extract_rt]$ file /cms_zfs/sat_data/olci/l1b/2021/020/S3* | grep HTML
/cms_zfs/sat_data/olci/l1b/2021/020/S3A_OL_1_ERR____20210120T070347_20210120T074738_20210121T104835_2631_067_277______LN1_O_NT_002.zip: HTML document text
/cms_zfs/sat_data/olci/l1b/2021/020/S3B_OL_1_ERR____20210120T062420_20210120T070810_20210121T104400_2630_048_134______LN1_O_NT_002.zip: HTML document text
/cms_zfs/sat_data/olci/l1b/2021/020/S3B_OL_1_ERR____20210120T080519_20210120T084909_20210121T122415_2630_048_135______LN1_O_NT_002.zip: HTML document text
/cms_zfs/sat_data/olci/l1b/2021/020/S3B_OL_1_ERR____20210120T094618_20210120T103008_20210121T140610_2630_048_136______LN1_O_NT_002.zip: HTML document text
/cms_zfs/sat_data/olci/l1b/2021/020/S3B_OL_1_ERR____20210120T112717_20210120T121107_20210121T154357_2630_048_137______LN1_O_NT_002.zip: HTML document text
[bmurch@optics0 seadas_l1a_geo_extract_rt]$ ll /cms_zfs/sat_data/olci/l1b/2021/020/S3A_OL_1_ERR____20210120T070347_20210120T074738_20210121T104835_2631_067_277______LN1_O_NT_002.zip
-rw-rw-r-- 1 bmurch cms_optics 665 Jan 21 12:35 /cms_zfs/sat_data/olci/l1b/2021/020/S3A_OL_1_ERR____20210120T070347_20210120T074738_20210121T104835_2631_067_277______LN1_O_NT_002.zip

[bmurch@optics0 l0]$ rm .urs_cookies_single;  curl -b .urs_cookies_single -c .urs_cookies_single -L -n --interface 2607:fe50:0:6330::103 --retry 5 --retry-delay 2 --max-time 0  --output /cms_zfs/sat_data/olci/l1b/2021/020/S3B_OL_1_ERR____20210120T062420_20210120T070810_20210121T104400_2630_048_134______LN1_O_NT_002.zip https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/S3B_OL_1_ERR____20210120T062420_20210120T070810_20210121T104400_2630_048_134______LN1_O_NT_002.zip
rm: cannot remove `.urs_cookies_single': No such file or directory
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
110   665  110   665    0     0    738      0 --:--:-- --:--:-- --:--:--  2309
Warning: Transient problem: HTTP error Will retry in 2 seconds. 5 retries
Warning: left.
^C
And then this one worked when I up arroed and hit enter:
[bmurch@optics0 l0]$ rm .urs_cookies_single;  curl -b .urs_cookies_single -c .urs_cookies_single -L -n --interface 2607:fe50:0:6330::104 --retry 5 --retry-delay 2 --max-time 0  --output /cms_zfs/sat_data/olci/l1b/2021/020/S3B_OL_1_ERR____20210120T062420_20210120T070810_20210121T104400_2630_048_134______LN1_O_NT_002.zip https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/S3B_OL_1_ERR____20210120T062420_20210120T070810_20210121T104400_2630_048_134______LN1_O_NT_002.zip
rm: cannot remove `.urs_cookies_single': No such file or directory
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  681M  100  681M    0     0  73.5M      0  0:00:09  0:00:09 --:--:-- 94.8M

Brock,

Sometimes hardware behaves badly...one of our backend machines was acting up.  Our firewall was occasionally (and in a kinda sticky manner) sending your traffic to that machine.  The offending machine has been taken out back and shot....well removed from service until it's memory issues can be resolved. 

Sean

Sean,

I understand hardware issues well.

I'm sending you a bunch of Log reports on another post as they are failing during processing.

Thanks,
Brock