CRON Job and changing file name

Use this Forum to find information on, or ask a question about, NASA Earth Science data.
ermill99
Posts: 19
Joined: Tue Feb 11, 2020 10:00 am America/New_York
Answers: 0

CRON Job and changing file name

by ermill99 » Thu Feb 13, 2020 6:33 pm America/New_York

I am trying to set up a CRON job on my server with WGET to download daily Chloro-a and SST png files from my subscription. 

I am in need of guidance on how to set the job command since the file names change due to the time they are processed.  Any help would be greatly appreciated. 

Thank you

Tags:

OB ODPS - jgwilding
Subject Matter Expert
Subject Matter Expert
Posts: 118
Joined: Fri Feb 19, 2021 1:09 pm America/New_York
Answers: 0

CRON Job and changing file name

by OB ODPS - jgwilding » Thu Feb 13, 2020 8:46 pm America/New_York

Hi.  Can you provide a subscription ID and whether it is extracted or not?

john

ermill99
Posts: 19
Joined: Tue Feb 11, 2020 10:00 am America/New_York
Answers: 0

CRON Job and changing file name

by ermill99 » Thu Feb 13, 2020 10:49 pm America/New_York

Certainly, here is one of my subscription IDs. I have a total of 7.

3472 and it is extracted. 

Thank you

OB ODPS - jgwilding
Subject Matter Expert
Subject Matter Expert
Posts: 118
Joined: Fri Feb 19, 2021 1:09 pm America/New_York
Answers: 0

CRON Job and changing file name

by OB ODPS - jgwilding » Fri Feb 14, 2020 10:34 am America/New_York

So for this subscription, you would use something like this to retrieve the list of files that have matched for the subscription.  Each entry returned will have the URL required to download the file.  So there is no need to try to predict any file names.  For your other subscriptions, just change the subID argument in the URL below.

wget -O - "https://oceandata.sci.gsfc.nasa.gov/api/file_search?subID=3472&subType=2&addurl=1&results_as_file=1"

If you had a shell script that looped through each of your subscriptions, the cron job can just call it periodically, and any new files for each subscription would be downloaded.  You probably want to keep track of the files that you've already downloaded so you don't waste time/bandwidth doing them again.  The --no-clobber option might help as well.

john

ermill99
Posts: 19
Joined: Tue Feb 11, 2020 10:00 am America/New_York
Answers: 0

CRON Job and changing file name

by ermill99 » Fri Feb 14, 2020 11:11 am America/New_York

Thank you very, very much, John!

I'll give this a go!

Eric

OB ODPS - jgwilding
Subject Matter Expert
Subject Matter Expert
Posts: 118
Joined: Fri Feb 19, 2021 1:09 pm America/New_York
Answers: 0

CRON Job and changing file name

by OB ODPS - jgwilding » Fri Feb 14, 2020 11:22 am America/New_York

Happy to help.  Feel free to contact me if you need any more help getting things going.

john

ermill99
Posts: 19
Joined: Tue Feb 11, 2020 10:00 am America/New_York
Answers: 0

CRON Job and changing file name

by ermill99 » Fri Feb 14, 2020 6:32 pm America/New_York

John, I was able to successfully call the file with the links.  I'm trying to automate the daily updated png files and am wondering if there a way to just call the .png files? Maybe I could rename the file with WGET?  I am still trying to figure out how to use the URLs in my HTML but the URL will still change.  I've got some thinking to do on this.

OB ODPS - jgwilding
Subject Matter Expert
Subject Matter Expert
Posts: 118
Joined: Fri Feb 19, 2021 1:09 pm America/New_York
Answers: 0

CRON Job and changing file name

by OB ODPS - jgwilding » Fri Feb 14, 2020 10:46 pm America/New_York

Eric,

If you just want the PNG files, you can add the search argument to the URL:

wget -O - "https://oceandata.sci.gsfc.nasa.gov/api/file_search?subID=3472&subType=2&results_as_file=1&search=A*.png"

john

ermill99
Posts: 19
Joined: Tue Feb 11, 2020 10:00 am America/New_York
Answers: 0

CRON Job and changing file name

by ermill99 » Sun Feb 16, 2020 12:59 pm America/New_York

John, thanks for ALL of your help so far.  I was able to take a few more steps towards reaching my objective. 

I have decided to go with the actual file download and save it to my server because I cannot use the URL as the authentication seems to make it unreliable.  My current command is:

wget -P ~/www/saltwx.com/public_html/MODIS/chloro --user=user--password='password' --auth-no-challenge=on "https://oceandata.sci.gsfc.nasa.gov/ob/getfile/A2020046181000.L2_LAC.S3470.nc.Southeast.chlor_a.png"

Is there a way to predict the "A2020046181000" part of the file name?

OB ODPS - jgwilding
Subject Matter Expert
Subject Matter Expert
Posts: 118
Joined: Fri Feb 19, 2021 1:09 pm America/New_York
Answers: 0

CRON Job and changing file name

by OB ODPS - jgwilding » Sun Feb 16, 2020 3:04 pm America/New_York

I'm not sure i understand why there is a need to try to predict part of the file name when the file_search URL gives you the actual names of the files for the subscription.  Whether you get the names from the file_search or got lucky and predicted them, you would need to use the same URL to download them. 

john

Post Reply