Page 1 of 1 failures

Posted: Thu Oct 14, 2021 5:12 pm America/New_York
by alaroy
Some time late in the UTC day 2021-10-10 or early in the day 2021-10-11 mysteriously stopped working (we hadn't manipulated the system in months). Here's an example of what's happening when I try to process almost any data with it:

Code: Select all

/marine-services/ocssw/scripts/ --threshold=10 -e /data/NC/L1/OBPG/FAI/2021-10-10/S20211010140000-EPH-MRT-OBPG.gbad -a /data/NC/L1/OBPG/FAI/2021-10-10/S20211010140000-ATT-MRT-OBPG.gbad -o /data/NC/L1/OBPG/FAI/2021-10-10/ /tmp/ssscratch/ocssw_GEO-working/
Searching for first requested record .........
MODIS GEO version 6.1.0, built Aug 16 2019 12:46:24
scan: 0 out of 203 Wed Oct 13 02:35:41 2021
scan: 10 out of 203 Wed Oct 13 02:35:41 2021
scan: 200 out of 203 Wed Oct 13 02:35:41 2021
Percent valid data (0.00) is less than threshold (10.00)
ERROR: MODIS geolocation processing failed.
Thinking that it was related to the letsencrypt issue I've tested it on CentOS 7, RHEL 8 and SUSE (I forget the version) - it had the same failure in all three contexts. I also have tried re-retrieving ocssw from git twice (which has previously fixed this issue when we had it before) but no such luck.

Thank you in advance for any advice you can give us on this!
-Andrew L.

Re: failures

Posted: Thu Oct 14, 2021 10:09 pm America/New_York
by alaroy
OK, this is embarrassing :oops:

Our download script was retrieving html error messages instead of gbad files and happily injecting them into the production system as if they were actual gbad files. Needless to say didn't read / process them properly.

Anyway, I'm off to add a trap to the download script to identify and reject html files mascaraing as gbad files...

Re: failures

Posted: Fri Oct 15, 2021 8:43 am America/New_York
by gnwiii
If you are using wget, there is an option to have the file extension reset to match the contents. For bulk downloads, you can use the file search to download a list of files with checksums. At one time I was getting corrupt downloads at a rate of 10%, so using checksums was very helpful.

Re: failures

Posted: Fri Oct 15, 2021 1:00 pm America/New_York
by alaroy
Good tips, thank you!

At least for now I'm testing that the gbad files are are 19968 bytes or more long and start with EOSAM1 or EOSAM2. If I find that's not working I'll come up with plan B.

In the case of netcdf/hdf files I have an automated test that ncdump -h doesn't return an error code. In addition to truncated (or otherwise corrupted) downloads, that also traps the situations where a bad file on the source system is being transmitted correctly (I've run across that a few times, I can't recall of any of them were OBPG's systems or not)