Refined subscription NRT mixing
Refined subscription NRT mixing
Hi,We have refined subscriptions for MODIS L2 and VIIRS L2. On June 11th this year VIIRS started receiving mixed NRT and refined files after starting downloading the list provided, we believe this is because the list is posted before the links have been updated. Should the list be being posted/updated before these files are present to download?If this is unintentional how far back is it likely to effect? I am currently investigating our MODIS archive to confirm.Thanks,Stephen
Tags:
-
OB.DAAC - SeanBailey
- User Services
- Posts: 1468
- Joined: Wed Sep 18, 2019 6:15 pm America/New_York
- Been thanked: 5 times
Refined subscription NRT mixing
Stephen,
While scratching our heads over this one, we surmise that there is a race condition involved. As refined files are generated they are checked to see if they match any active subscriptions.
If so, a record in a table for that subscription is updated. The new file replaces the old in the file location table and the old (non refined) file is staged for deletion. The DB the webserver uses polls the production DB for updates every five minutes for file locations and every 12 for subscription updates. It is possible that a subscription gets a record but the web DB file location is still pointing at the old location (which has yet to be deleted) when you query and pull the file down. I can't imagine this happening terribly frequently uniess you are polling us faster the once every 12 minutes. I've modified the subscription update logic to let no subscription record less than 10 minutes old get in. This *should* prevent the potential race condition as the file location will have been updated first.
As for how far back it is likely to affect...probably as far back as the subscription process has used this DB logic... August 1, 2013.
Sean
While scratching our heads over this one, we surmise that there is a race condition involved. As refined files are generated they are checked to see if they match any active subscriptions.
If so, a record in a table for that subscription is updated. The new file replaces the old in the file location table and the old (non refined) file is staged for deletion. The DB the webserver uses polls the production DB for updates every five minutes for file locations and every 12 for subscription updates. It is possible that a subscription gets a record but the web DB file location is still pointing at the old location (which has yet to be deleted) when you query and pull the file down. I can't imagine this happening terribly frequently uniess you are polling us faster the once every 12 minutes. I've modified the subscription update logic to let no subscription record less than 10 minutes old get in. This *should* prevent the potential race condition as the file location will have been updated first.
As for how far back it is likely to affect...probably as far back as the subscription process has used this DB logic... August 1, 2013.
Sean