Missing Anc Files?

oo_processing · by **oo_processing** » Thu Jul 02, 2020 10:35 am America/New_York

We were checking files, and found 5223 'bad' HTML anc files from 2011-2019.
We were able to download nearly all of them.
However, I have tested on different machines, and I can’t seem to grab anything but HTML for these 'bad' HTML files.
Please advise on the status of these files:

PM1ATTNR_NRT.A2011226.0920.005
PM1ATTNR_NRT.A2011297.1745.005
PM1ATTNR_NRT.A2011299.0420.005
PM1ATTNR_NRT.A2011299.0555.005
PM1ATTNR_NRT.A2011316.1955.005
PM1ATTNR_NRT.A2012007.2030.005
PM1ATTNR_NRT.A2012071.0225.005
PM1ATTNR_NRT.A2012145.0320.005
PM1ATTNR_NRT.A2012183.0225.005
PM1ATTNR_NRT.A2012200.2120.005
PM1ATTNR_NRT.A2012204.1445.005
PM1EPHND_NRT.A2011042.0205.005
PM1EPHND_NRT.A2011065.2000.005
PM1EPHND_NRT.A2011065.2020.005
PM1EPHND_NRT.A2011066.0425.005
PM1EPHND_NRT.A2011126.1955.005
PM1EPHND_NRT.A2011201.0435.005
PM1EPHND_NRT.A2011225.2340.005
PM1EPHND_NRT.A2011252.0130.005
PM1EPHND_NRT.A2011259..005.005
PM1EPHND_NRT.A2012087.0055.005
PM1EPHND_NRT.A2012162.0045.005
PM1EPHND_NRT.A2012218.2000.005
PM1EPHND_NRT.A2013212.1405.005
PM1EPHND_NRT.A2013245.0135.005
PM1EPHND_NRT.A2013305.0020.005

by **OB ODPS - jgwilding** » Thu Jul 02, 2020 10:59 am America/New_York

These files were condemned to hades (a fictional host name we used after a non-fictional storage unit suffered a catastrophic data loss), represented by the host-id 666, and were not recovered because we already had the optimal non-NRT data for their respective coverage periods. Are you deriving the file names or were the returned via a search? If the latter, the records should be removed from the web-db.

host_id basename
------- ------------------------------
   666 PM1ATTNR_NRT.A2011226.0920.005
   666 PM1ATTNR_NRT.A2011297.1745.005
   666 PM1ATTNR_NRT.A2011299.0420.005
   666 PM1ATTNR_NRT.A2011299.0555.005
   666 PM1ATTNR_NRT.A2011316.1955.005
   666 PM1ATTNR_NRT.A2012007.2030.005
   666 PM1ATTNR_NRT.A2012071.0225.005
   666 PM1ATTNR_NRT.A2012145.0320.005
   666 PM1ATTNR_NRT.A2012183.0225.005
   666 PM1ATTNR_NRT.A2012200.2120.005
   666 PM1ATTNR_NRT.A2012204.1445.005
   666 PM1EPHND_NRT.A2011042.0205.005
   666 PM1EPHND_NRT.A2011065.2000.005
   666 PM1EPHND_NRT.A2011065.2020.005
   666 PM1EPHND_NRT.A2011066.0425.005
   666 PM1EPHND_NRT.A2011126.1955.005
   666 PM1EPHND_NRT.A2011201.0435.005
   666 PM1EPHND_NRT.A2011225.2340.005
   666 PM1EPHND_NRT.A2011252.0130.005
   666 PM1EPHND_NRT.A2012087.0055.005
   666 PM1EPHND_NRT.A2012162.0045.005
   666 PM1EPHND_NRT.A2012218.2000.005
   666 PM1EPHND_NRT.A2013212.1405.005
   666 PM1EPHND_NRT.A2013245.0135.005
   666 PM1EPHND_NRT.A2013305.0020.005

(25 rows affected)

john

by **OB.DAAC - SeanBailey** » Thu Jul 02, 2020 11:05 am America/New_York

These were victims of a RAID doing what RAIDs aren't supposed to do. In the recovery process, we deemed these files unnecessary to recover as they were near-real time ancillary and the definitive ancillary for those time periods was available. The records weren't removed from the database so we would know what happened to them (we set the device record to a sentinel value that allows us to quickly know these are not the droids we're looking for....). The call to getfile for these should return a 404 (the HTML page is probably our 404 error page)

Sean

bbbarnes · by **bbbarnes** » Thu Jul 02, 2020 7:02 pm America/New_York

@john: yes, these are still returned in search (reaffirming Sean's statement):

> modis_atteph.py -d -s 2012204145500 -m aqua

att1=/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2012/204/PM1ATTNR_NRT.A2012204.1450.005
att2=/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2012/204/PM1ATTNR_NRT.A2012204.1455.005
att3=/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2012/204/PM1ATTNR_NRT.A2012204.1500.005
eph1=/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2012/204/PM1EPHND.P2012204.1200.003

Currently, our workflow (to account for the new authentication protocols) is to (1) run modis_atteph.py/getanc.py to populate an anc.db for the files we need to process, then (2) download the att/eph/anc files using curl and put them in their appropriate locations within the ancdir, then (3) process the files (L1A->GEO->L1B->L2), reading the anc.db to provide appropriate att/eph/anc where needed.

The solution, then, is to identify files that return from curl as HTML, and remove them from our ancillary directory structure AND from the anc.db (so that the processors don't try to use them). Is there an easier workaround? It may be helpful to have a full list of these unnecessary files (best analogy would be the battle droids from Phantom Menace - completely useless ... no one is looking for them) - if it's a small list, perhaps these could be run by hand?

Thanks
-brian

by **OB.DAAC - SeanBailey** » Thu Jul 02, 2020 8:12 pm America/New_York

Brian,
NRT attitude and emphereris is fine for near realtime processing. Your hurting yourself by not using the definitive attitude and emphereris files once they are available, which is usually within a day of collection. Generating geolocation for files in 2012 would definitely not need the NRT files.

Run modis_atteph.py and getanc.py with --refrshDB to update the records in the ancillary database for files that had previously been processed as NRT.

Sean

bbbarnes · by **bbbarnes** » Thu Jul 02, 2020 9:23 pm America/New_York

Good info, thanks. For inode considerations, probably easiest for us to just regularly purge NRT att/eph files.
-brian