Refined subscription NRT mixing

Use this Forum to find information on, or ask a question about, NASA Earth Science data.
Post Reply
Posts: 8
Joined: Sun Jan 03, 2016 11:35 am America/New_York
Answers: 0

Refined subscription NRT mixing

by stgo » Mon Jul 31, 2017 4:59 am America/New_York

Hi,We have refined subscriptions for MODIS L2 and VIIRS L2. On June 11th this year VIIRS started receiving mixed NRT and refined files after starting downloading the list provided, we believe this is because the list is posted before the links have been updated. Should the list be being posted/updated before these files are present to download?If this is unintentional how far back is it likely to effect? I am currently investigating our MODIS archive to confirm.Thanks,Stephen


OB.DAAC - SeanBailey
User Services
User Services
Posts: 1441
Joined: Wed Sep 18, 2019 6:15 pm America/New_York
Answers: 1
Been thanked: 1 time

Refined subscription NRT mixing

by OB.DAAC - SeanBailey » Mon Jul 31, 2017 7:44 pm America/New_York


While scratching our heads over this one, we surmise that there is a race condition involved.  As refined files are generated they are checked to see if they match any active subscriptions.
If so, a record in a table for that subscription is updated.  The new file replaces the old in the file location table and the old (non refined) file is staged for deletion.  The DB the webserver uses polls the production DB for updates every five minutes for file locations and every 12 for subscription updates.  It is possible that a subscription gets a record but the web DB file location is still pointing at the old location (which has yet to be deleted) when you query and pull the file down.  I can't imagine this happening terribly frequently uniess you are polling us faster the once every 12 minutes.  I've modified the subscription update logic to let no subscription record less than 10 minutes old get in.  This *should* prevent the potential race condition as the file location will have been updated first.

As for how far back it is likely to affect...probably as far back as the subscription process has used this DB logic... August 1, 2013.


Post Reply