getfile problem

[alaroy]

Since around 2019-10-23 20:43:16 UTC any attempt to retrieve a file using the getfile cgi has been failing with an "Error 404 not found". Could somebody please investigate the problem? Thank you.

[OB WebDev - norman]

Hi,

Could you please give us some additional information?
You will find some instructions here.

https://oceancolor.gsfc.nasa.gov/forum/oceancolor/topic_show.pl?tid=6447

Thanks,
Norman

[alaroy]

Norman,

Thank you for your response. The problem seems to have been resolved shortly before 10:26 UTC (6:26 EDT).

-Andrew L.

[OB WebDev - norman]

OK. Great.

Norman

[b.detracey]

Having the same problem here:
$ wget -v --content-disposition 'https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/20190101120000-CMC-L4_GHRSST-SSTfnd-CMC0.1deg-GLOB-v02.0-fv03.0.nc'
--2019-10-24 15:04:25--  https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/20190101120000-CMC-L4_GHRSST-SSTfnd-CMC0.1deg-GLOB-v02.0-fv03.0.nc
Resolving oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)... xx.xxx.xx.xx, 2001:4d0:2418:128::84
Connecting to oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)|xx.xxx.xx.xx|:443... connected.
HTTP request sent, awaiting response... 404 Not Found
2019-10-24 15:05:16 ERROR 404: Not Found.

[oo_processing]

I had the same problem last night near midnight where all my VIIRS and MODIS downloaded files were "bad" as in html files.
It happened on machines in both Tampa and in Saint Petersburg.

I was able to d/l the correct files doy 296, today.
However, I am getting the errors again just now (note the 665 file size which is the error message) and where get.txt contains:
[bmurch@optics1] /cms_zfs/sat_data/modis/l0/2019/297 $ cat get.txt
MOD00.A2019297.1015_1.PDS.bz2
MOD00.A2019297.1020_1.PDS.bz2
MOD00.A2019297.1025_1.PDS.bz2
MOD00.A2019297.1150_1.PDS.bz2
MOD00.P2019297.1455_1.PDS.bz2
MOD00.P2019297.1500_1.PDS.bz2
MOD00.P2019297.1540_1.PDS.bz2
MOD00.P2019297.1640_1.PDS.bz2

And I killed it after the three html files were downloaded.

[bmurch@optics1] /cms_zfs/sat_data/modis/l0/2019/297 $ time curl --interface 2607:fe50:0:6330::132 --retry 5 --retry-delay 2 --max-time 0 --remote-name-all https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/{$(sed ':a;N;$!ba;s/\n/,/g' get.txt)}

[1/8]: https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/MOD00.A2019297.1015_1.PDS.bz2 --> MOD00.A2019297.1015_1.PDS.bz2
--_curl_--https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/MOD00.A2019297.1015_1.PDS.bz2
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
110   665 110   665    0     0     99      0  0:00:06  0:00:06 --:--:--   194

[2/8]: https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/MOD00.A2019297.1020_1.PDS.bz2 --> MOD00.A2019297.1020_1.PDS.bz2
--_curl_--https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/MOD00.A2019297.1020_1.PDS.bz2
110   665 110   665    0     0    122      0  0:00:05  0:00:05 --:--:--   150

[3/8]: https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/MOD00.A2019297.1025_1.PDS.bz2 --> MOD00.A2019297.1025_1.PDS.bz2
--_curl_--https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/MOD00.A2019297.1025_1.PDS.bz2
110   665  110   665    0     0    122      0  0:00:05  0:00:05 --:--:--   150

[4/8]: https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/MOD00.A2019297.1150_1.PDS.bz2 --> MOD00.A2019297.1150_1.PDS.bz2
--_curl_--https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/MOD00.A2019297.1150_1.PDS.bz2
  0   665    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0^C

real    0m19.311s
user    0m0.044s
sys     0m0.056s

[oo_processing]

It is working again.
If I had to guess, I suspect that the real-time subscription database and the other one are out of sync.

[dana.r.wilson]

The issues have been fixed.

[oo_processing]

I have had another 404 error "file not found" just now. I wonder if the file is on your servers, as other files can be downloaded now:
wget https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/A2003275023500.L0_LAC.bz2
--2019-10-24 16:09:25--  https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/A2003275023500.L0_LAC.bz2
Resolving oceandata.sci.gsfc.nasa.gov... xx.xxx.xx.xx, 2001:4d0:2418:128::84
Connecting to oceandata.sci.gsfc.nasa.gov|xx.xxx.xx.xx|:443... connected.
HTTP request sent, awaiting response... 404 Not Found
2019-10-24 16:09:31 ERROR 404: Not Found.

[oo_processing]

I have tried the other possible names without success (although Chuanmin Hu tired A2003275023500.L0_LAC.bz2 earlier and it worked fine):

From this page:
https://oceancolor.sci.gsfc.nasa.gov/cgi/browse.pl/filelist.1571948228.txt?sub=filenamelist&id=1571948228.23574&prm=L0
I found MOD00.P2003275.0235_1.PDS
and then tried that as well:

what 117 :/optics1/home1/oo_processing> wget https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/MOD00.P2003275.0235_1.PDS.bz2
--2019-10-24 16:21:53--  https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/MOD00.P2003275.0235_1.PDS.bz2
Resolving oceandata.sci.gsfc.nasa.gov... xx.xxx.xx.xx, 2001:4d0:2418:128::84
Connecting to oceandata.sci.gsfc.nasa.gov|xx.xxx.xx.xx|:443... connected.
HTTP request sent, awaiting response... 404 Not Found
2019-10-24 16:21:59 ERROR 404: Not Found.

what 118 :/optics1/home1/oo_processing> wget https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/MOD00.P2003275.0235_1.PDS
--2019-10-24 16:22:07--  https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/MOD00.P2003275.0235_1.PDS
Resolving oceandata.sci.gsfc.nasa.gov... xx.xxx.xx.xx, 2001:4d0:2418:128::84
Connecting to oceandata.sci.gsfc.nasa.gov|xx.xxx.xx.xx|:443... connected.
HTTP request sent, awaiting response... 404 Not Found
2019-10-24 16:22:13 ERROR 404: Not Found