Ancillary file delay

oo_processing · by **oo_processing** » Wed Feb 12, 2020 1:04 pm America/New_York

And again today:

modis_GEO.py --parfile=./MOD00.P2020042.1325.ANGOLA_FULL.L1A_LAC.atteph.fixed MOD00.P2020042.1325.ANGOLA_FULL.L1A_LAC -o MOD00.P2020042.1325.ANGOLA_FULL.GEO --threshold=95 --ancdb=./ancillary_data.db
STATUS: $ocssw_return_2 = ./MOD00.P2020042.1325.ANGOLA_FULL.L1A_LAC.atteph.fixed
{'geogen': {}}
Missing attitude files!
Missing ephemeris files!

$ocssw_command_2 = modis_GEO.py --parfile=./MOD00.P2020043.0815.WGOM_FULL.L1A_LAC.atteph.fixed MOD00.P2020043.0815.WGOM_FULL.L1A_LAC -o MOD00.P2020043.0815.WGOM_FULL.GEO --threshold=95 --ancdb=./ancillary_data.db
STATUS: $ocssw_return_2 = ./MOD00.P2020043.0815.WGOM_FULL.L1A_LAC.atteph.fixed
{'geogen': {}}
Missing attitude files!
Missing ephemeris files!

by **OB ODPS - jgwilding** » Wed Feb 12, 2020 3:39 pm America/New_York

For these granules, it appears that these would be the ATT/EPH files:

PM1ATTNR.P2020042.1200.003
PM1ATTNR.P2020043.0800.003
PM1EPHND.P2020042.1200.003

These appear to be available via https-getfile. Are you still seeing the problem now?

john

oo_processing · by **oo_processing** » Wed Feb 12, 2020 9:29 pm America/New_York

No, the problem clears up eventually.
I suspect that the subscriptions are filled prior to the anc data being available, or in the database that the scripts query.
But it is an ongoing problem.

Brock

by **OB.DAAC - SeanBailey** » Fri Feb 14, 2020 2:15 pm America/New_York

Brock,

Your suspicion is pretty spot on. It's a long-ish story, the details of which would bore all but those most keen on data management intricacies, but the short version is this. The back-end of the subscription service has been modified. Previously, L0 and L1 subscriptions were not staged until the L2 file had been processed. This was a silly thing. So, now the files are staged sooner.
The consequence is that we may not have received the necessary attitude and ephemeris files before you see the subscription data are available. If you try to process those L0 data before the att/eph are available, you will get an error. The att/eph are usually close behind the L0, so if you do get an error, retry the failed process after a pause - 30 to 60 minutes should suffice.

Sean

oo_processing · by **oo_processing** » Tue Feb 18, 2020 11:32 am America/New_York

Sean,

This becomes very tedious and fast. I have scripts that run every 1/2 hour and if subscription file is stages, I grab it.

And I restart failed jobs every hour, so some go through several failures. And as a result, it appears that the .usr_cookies file is often corrupted

When the subscriptions were set up, there were check boxes asking to wait for anc data to become available.
Can those be toggled now, and it they are, will my babysitting job I now have, end? :eek:

Just curious.
Cheers,
Brock

by **OB.DAAC - SeanBailey** » Tue Feb 18, 2020 12:37 pm America/New_York

Brock,
The cookie is really only needed for a session. You could delete it once the session is done and recreate it for the next. That'll get you past the potential corruption problem.
There is no ancillary data for L0 files, so there is no way to wait for something that doesn't exist. If your subscriptions were for L1 data, then the att/eph would most definitely be availble, as we don't catalog the L1 files until we can generate the geolocation file.

As for the "wait for anc", never was such a thing. There was (and still is) a wait for refined, but that only applies to L2 subscriptions. There was also an "include ancillary files", but that option was eliminated when getanc.py was created.

Sean