CRON Job and changing file name

[ermill99]

I am trying to set up a CRON job on my server with WGET to download daily Chloro-a and SST png files from my subscription. 

I am in need of guidance on how to set the job command since the file names change due to the time they are processed.  Any help would be greatly appreciated. 

Thank you

[OB ODPS - jgwilding]

Hi.  Can you provide a subscription ID and whether it is extracted or not?

john

[ermill99]

Certainly, here is one of my subscription IDs. I have a total of 7.

3472 and it is extracted. 

Thank you

[OB ODPS - jgwilding]

So for this subscription, you would use something like this to retrieve the list of files that have matched for the subscription.  Each entry returned will have the URL required to download the file.  So there is no need to try to predict any file names.  For your other subscriptions, just change the subID argument in the URL below.

wget -O - "https://oceandata.sci.gsfc.nasa.gov/api/file_search?subID=3472&subType=2&addurl=1&results_as_file=1"

If you had a shell script that looped through each of your subscriptions, the cron job can just call it periodically, and any new files for each subscription would be downloaded.  You probably want to keep track of the files that you've already downloaded so you don't waste time/bandwidth doing them again.  The --no-clobber option might help as well.

john

[ermill99]

Thank you very, very much, John!

I'll give this a go!

Eric

[OB ODPS - jgwilding]

Happy to help.  Feel free to contact me if you need any more help getting things going.

john

[ermill99]

John, I was able to successfully call the file with the links.  I'm trying to automate the daily updated png files and am wondering if there a way to just call the .png files? Maybe I could rename the file with WGET?  I am still trying to figure out how to use the URLs in my HTML but the URL will still change.  I've got some thinking to do on this.

[OB ODPS - jgwilding]

Eric,

If you just want the PNG files, you can add the search argument to the URL:

wget -O - "https://oceandata.sci.gsfc.nasa.gov/api/file_search?subID=3472&subType=2&results_as_file=1&search=A*.png"

john

[ermill99]

John, thanks for ALL of your help so far.  I was able to take a few more steps towards reaching my objective. 

I have decided to go with the actual file download and save it to my server because I cannot use the URL as the authentication seems to make it unreliable.  My current command is:

wget -P ~/www/saltwx.com/public_html/MODIS/chloro --user=user--password='password' --auth-no-challenge=on "https://oceandata.sci.gsfc.nasa.gov/ob/getfile/A2020046181000.L2_LAC.S3470.nc.Southeast.chlor_a.png"

Is there a way to predict the "A2020046181000" part of the file name?

[OB ODPS - jgwilding]

I'm not sure i understand why there is a need to try to predict part of the file name when the file_search URL gives you the actual names of the files for the subscription.  Whether you get the names from the file_search or got lucky and predicted them, you would need to use the same URL to download them. 

john