Impossible to download VIIRS SST files still present on your web page

[jstum]

Hello,On page https://oceandata.sci.gsfc.nasa.gov (/VIIRS-SNPP/L2/2019/086/), clicking on the last files of the day (e.g.V2019086235400.L2_SNPP_SST3.nc), returns an error 404. It looks like the file, though being visible on the page, is not physically present on your server. We have thus experienced a lot of ERROR 404 messages from your site, and we are afraid that this may cause a blocking of our IP address (xx.xxx.xx.xx). This is not the case for now, but...Another subject : we did not have new files on our MODIS L1A subscription 1661 since yersterday (no data from day 086)Thank you for your help,Regards,Jacques

[OB ODPS - towens]

I am able to reproduce what you are seeing.
We did a database server and software upgrade yesterday and are still dealing with some issues.
I expect access to these files and subscriptions will be restored shortly.

Thanks for your patience,
Tommy

[OB ODPS - towens]

The files should now be accessible.
Tommy

[oo_processing]

Tommy,

Would this have affected:

getanc.py MYD021KM.2019085.1515.S_BRAZIL.hdf --ancdb=./ancillary_data.db
Error! could not establish a network connection. Check your network connection.
If you do not find a problem, please try again later

AND

modis_GEO.py MOD00.P2003324.0720.CBAY.L1A_LAC -o MOD00.P2003324.0720.CBAY.GEO --threshold=95 --ancdb=./ancillary_data.db --enable-dem
Error! could not establish a network connection. Check your network connection.
If you do not find a problem, please try again later.

I've had thousands of failures overnight in these two steps. When they happened a couple of days ago, Christopher indicated that they had been doing some firewall updates.

Thanks,
Brock

[OB ODPS - towens]

It's possible that the repeated failures caused by this problem got your IP address blocked.
I'll check with the network admins.
Tommy

[oo_processing]

Tommy,
I had sent them a note, haven't heard back. Some are getting though, others not.
Cheers,
Brock

[pfsmith]

I did a quick check.  Most recent block logged was at 16:06 local time on 3/25. None in place currently for your IPv4 address range.

[oo_processing]

Paul,
Thanks, it from a clustered environment and they all use xx.xxx.xx.xx as the outbound ipaddy. Is this the range you checked?
Brock

[oo_processing]

BTW,

Christopher told me that I should try the curl or wget retries, but I D/L all the sat data before processing, and these failures come from your script calls. My failures started up again just now:

getanc.py /work/b/bmurch/tmp/S4P_MODIS_H5_MODIST.BERMUDA.2004.276_version_0.05_e1s6aoLH/MOD021KM.2004276.1440.BERMUDA.hdf --ancdb=./ancillary_data.db
Error! could not establish a network connection. Check your network connection.
If you do not find a problem, please try again later.
[66173 refs]

modis_GEO.py /work/b/bmurch/tmp/S4P_MODIS_H5_MODIST.GALV.2019.086_version_0.05_Xu5ZmsyZ/MOD00.A2019086.1755.GALV_FULL.L1A_LAC -o /work/b/bmurch/tmp/S4P_MODIS_H5_MODIST.GALV.2019.086_version_0.05_Xu5ZmsyZ/MOD00.A2019086.1755.GALV_FULL.GEO --threshold=95 --ancdb=./ancillary_data.db
Traceback (most recent call last):
  File "/shares/cms_optics/apps/seadas/seadas-7.5/ocssw/scripts/modis_GEO.py", line 187, in <module>
    sys.exit(main())
  File "/shares/cms_optics/apps/seadas/seadas-7.5/ocssw/scripts/modis_GEO.py", line 175, in main
    m.utcleap()
  File "/shares/cms_optics/apps/seadas/seadas-7.5/ocssw/scripts/modules/modis_GEO_utils.py", line 140, in utcleap
    lut.get_luts()
  File "/shares/cms_optics/apps/seadas/seadas-7.5/ocssw/scripts/modules/LutUtils.py", line 132, in get_luts
    regex='^((?!\d+).)*' + suffix, check_times=True)
  File "/shares/cms_optics/apps/seadas/seadas-7.5/ocssw/scripts/modules/JsonUtils.py", line 272, in download_allfiles
    all_links = self.get_links(url, regex=regex)
  File "/shares/cms_optics/apps/seadas/seadas-7.5/ocssw/scripts/modules/JsonUtils.py", line 238, in get_links
    content = response.read()
AttributeError: 'NoneType' object has no attribute 'read'
[68468 refs]