Problem access to ocean color data

cfud · by **cfud** » Thu Dec 10, 2020 5:56 am America/New_York

Since many years we used the Ocean color data and more specifically the actual data of the satellite TERRA. The automated process works on the wget order. If we noticed no significant problems during month, we have for a week problem of acquisition.
Main time connection is established, but sometimes it is not successfull.
When we established a control by direct connection on the web site and analysed on the subscriptions Dashboard the availability of the data, it seems OK. But when it is processed by the routine, we obtain no URL connections.
Did you have any idea of the origin of the problem?
Could it be the results of a slower connection or too many requests of our servers?
In addition this morning, we noticed another problem: during the connection, a "Service Unavailable" error is returned. Perhaps because of too many requests we send last few days?

The following corresponds to our connect orders:
----------------------- 2620 -----------------------
wget --post-data="subID=2620&addurl=1&results_as_file=1" -O - https://oceandata.sci.gsfc.nasa.gov/api/file_search | grep 2020343 | wget --load-cookies ~/.urs_cookies --save-cookies ~/.urs_cookies --auth-no-challenge=on --keep-session-cookies -nv --retry-connrefused -c -P /ftmpdir/DATA/2620/ -i -
wget --post-data="subID=2620&addurl=1&results_as_file=1" -O - https://oceandata.sci.gsfc.nasa.gov/api/file_search | grep 2020342 | wget --load-cookies ~/.urs_cookies --save-cookies ~/.urs_cookies --auth-no-challenge=on --keep-session-cookies -nv --retry-connrefused -c -P /ftmpdir/DATA/2620/ -i â€“

Thank a lot for your help.

by **OB.DAAC - amscott** » Thu Dec 10, 2020 9:52 am America/New_York

Hello,

Regarding this morning's issue: There was a large network issue that took place last night that impacted access to our systems but hopefully, all is well but if not, it will be shortly.

Alicia

oo_processing · by **oo_processing** » Thu Dec 10, 2020 10:46 am America/New_York

Alicia

There are also problems returning anc files. For instance when I process this file (or most others right now):

MOD00.P2020340.1910_1.PDS.bz2

One one att and one eph file is returned. I believe that 2 att and 2 eph files are required to process this completely:

$_: att1=/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/340/PM1ATTNR.P2020340.1800.003
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/340/PM1ATTNR.P2020340.1800.003 was not compressed
STATUS: $compression_test_returns existing file: 1
$_: eph1=/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/340/PM1EPHND.P2020340.1200.003
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/340/PM1EPHND.P2020340.1200.003 was not compressed

$par_entry: a1=/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/340/PM1ATTNR.P2020340.1800.003
$par_entry: e1=/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/340/PM1EPHND.P2020340.1200.003

$_: icefile=/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/340/N202034000_SEAICE_NSIDC_24h.hdf
$_: met1=/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/340/N202034018_MET_NCEPR2_6h.hdf
$_: met2=/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/341/N202034100_MET_NCEPR2_6h.hdf
$_: met3=/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/341/N202034100_MET_NCEPR2_6h.hdf
$_: ozone1=/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/340/N202034000_O3_AURAOMI_24h.hdf
$_: ozone2=/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/341/N202034100_O3_AURAOMI_24h.hdf
$_: ozone3=/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/341/N202034100_O3_AURAOMI_24h.hdf
$_: sstfile=/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/340/N2020340_SST_OIV2AV_24h.nc
STATUS: $l1a_file_w_vdir_path /work/b/bmurch/tmp/S4P_MODIS_H5_MODISA.BIGBEND.2020.340_version_0.10_Yr_RaHr6/MOD00.P2020340.1910.BIGBEND_FULL.L1A_LAC was found
STATUS: $current_directory = /work/b/bmurch/tmp/S4P_MODIS_H5_MODISA.BIGBEND.2020.340_version_0.10_Yr_RaHr6
STATUS: $ocssw_command_2 = modis_GEO.py /work/b/bmurch/tmp/S4P_MODIS_H5_MODISA.BIGBEND.2020.340_version_0.10_Yr_RaHr6/MOD00.P2020340.1910.BIGBEND_FULL.L1A_LAC -o /work/b/bmurch/tmp/S4P_MODIS_H5_MODISA.BIGBEND.2020.340_version_0.10_Yr_RaHr6/MOD00.P2020340.1910.BIGBEND_FULL.GEO --threshold=95 --ancdb=./ancillary_data.db
STATUS: $ocssw_return_2 = MODIS GEO version 6.1.0, built Aug 16 2019 12:46:24
scan: 0 out of 196 Thu Dec 10 10:40:04 2020
...
scan: 190 out of 196 Thu Dec 10 10:41:04 2020
Percent valid data (0.00) is less than threshold (95.00)

by **OB ODPS - jgwilding** » Thu Dec 10, 2020 1:20 pm America/New_York

The only time you need more than one ATT/EPH file is when the start or stop time of the granule being processed aligns with the start or stop time of an ATT/EPH file, which isn't the case for this granule. Its start/stop times are well within the periods of the ATT/EPH files chosen. When using the NRT ATT/EPH files, which are 5-minutes long (same as L0/L1 granules), you always need multiple ATT/EPH files.

Have you verified that the ATT/EPH files are data files and not HTML files that can be downloaded when a server error occurs?

MOD00.P2020340.1910_1.PDS -> A2020340191000.L1A_LAC

process_modis_g|10-Dec-20 13:14:58|I| ATT/EPH files are:
10501|PM1EPHND.P2020340.1200.003|/data1/sdpsoper/vdc/vpu0/workbuf||PM1EPHND.P2020340.1200.003|PM1EPHND.P2020340.1200.003|1
10502|PM1ATTNR.P2020340.1800.003|/data1/sdpsoper/vdc/vpu0/workbuf||PM1ATTNR.P2020340.1800.003|PM1ATTNR.P2020340.1800.003|1

process_modis_g|10-Dec-20 13:14:59|I| geogen_modis
MODIS GEO version 6.1.0, built Nov 30 2020 14:35:29
scan: 0 out of 203 Thu Dec 10 13:14:59 2020
scan: 10 out of 203 Thu Dec 10 13:14:59 2020
scan: 20 out of 203 Thu Dec 10 13:15:00 2020
scan: 30 out of 203 Thu Dec 10 13:15:00 2020
scan: 40 out of 203 Thu Dec 10 13:15:00 2020
scan: 50 out of 203 Thu Dec 10 13:15:01 2020
scan: 60 out of 203 Thu Dec 10 13:15:01 2020
scan: 70 out of 203 Thu Dec 10 13:15:02 2020
scan: 80 out of 203 Thu Dec 10 13:15:02 2020
scan: 90 out of 203 Thu Dec 10 13:15:03 2020
scan: 100 out of 203 Thu Dec 10 13:15:03 2020
scan: 110 out of 203 Thu Dec 10 13:15:03 2020
scan: 120 out of 203 Thu Dec 10 13:15:04 2020
scan: 130 out of 203 Thu Dec 10 13:15:04 2020
scan: 140 out of 203 Thu Dec 10 13:15:05 2020
scan: 150 out of 203 Thu Dec 10 13:15:05 2020
scan: 160 out of 203 Thu Dec 10 13:15:06 2020
scan: 170 out of 203 Thu Dec 10 13:15:06 2020
scan: 180 out of 203 Thu Dec 10 13:15:07 2020
scan: 190 out of 203 Thu Dec 10 13:15:07 2020
scan: 200 out of 203 Thu Dec 10 13:15:07 2020
process_modis_g|10-Dec-20 13:15:08|I| finished (status=0)
process_modis_g|10-Dec-20 13:15:08|I| calling geocheck_modis
Percent Longitude good: 100
Percent Latitude good: 100

john

oo_processing · by **oo_processing** » Thu Dec 10, 2020 1:41 pm America/New_York

It appears that you are correct and the problem is HTML files (I thought I had fixed them, but it was on another installation my bad):

/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/345/AM1ATTN0.A2020345.0000.002:    HTML document text
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/345/AM1ATTN0.A2020345.0200.002:    HTML document text
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/345/AM1ATTN0.A2020345.0600.002:    HTML document text
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/345/AM1EPHN0.A2020345.0000.002:    HTML document text
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/345/AM1EPHN0.A2020345.0200.002:    HTML document text
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/345/AM1EPHN0.A2020345.0600.002:    HTML document text
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/345/PM1ATTNR_NRT.A2020345.0105.061: HTML document text
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/345/PM1ATTNR_NRT.A2020345.0110.061: HTML document text
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/345/PM1ATTNR_NRT.A2020345.0250.061: HTML document text
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/345/PM1ATTNR_NRT.A2020345.0255.061: HTML document text
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/345/PM1ATTNR_NRT.A2020345.0430.061: HTML document text
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/345/PM1ATTNR_NRT.A2020345.0435.061: HTML document text
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/345/PM1ATTNR_NRT.A2020345.0440.061: HTML document text
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/345/PM1ATTNR_NRT.A2020345.0445.061: HTML document text
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/345/PM1ATTNR_NRT.A2020345.0610.061: HTML document text
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/345/PM1ATTNR_NRT.A2020345.0615.061: HTML document text
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/345/PM1ATTNR_NRT.A2020345.0640.061: HTML document text
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/345/PM1ATTNR_NRT.A2020345.0645.061: HTML document text
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/345/PM1ATTNR_NRT.A2020345.0650.061: HTML document text
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/345/PM1ATTNR_NRT.A2020345.0655.061: HTML document text
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/345/PM1ATTNR_NRT.A2020345.0955.061: HTML document text
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/345/PM1ATTNR_NRT.A2020345.1000.061: HTML document text
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/345/PM1EPHND_NRT.A2020345.0105.061: HTML document text
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/345/PM1EPHND_NRT.A2020345.0110.061: HTML document text
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/345/PM1EPHND_NRT.A2020345.0250.061: HTML document text
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/345/PM1EPHND_NRT.A2020345.0255.061: HTML document text
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/345/PM1EPHND_NRT.A2020345.0430.061: HTML document text
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/345/PM1EPHND_NRT.A2020345.0640.061: HTML document text
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/345/PM1EPHND_NRT.A2020345.0645.061: HTML document text
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/345/PM1EPHND_NRT.A2020345.0650.061: HTML document text
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/345/PM1EPHND_NRT.A2020345.0655.061: HTML document text
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/345/PM1EPHND_NRT.A2020345.0955.061: HTML document text
/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/345/PM1EPHND_NRT.A2020345.1000.061: HTML document text