Page 2 of 3

Re: Incorrect File Size for Aqua, Terra and VIIRS SST and Chl files

Posted: Sat Dec 09, 2023 2:14 pm America/New_York
by OB ODPS - towens
Put curly braces around the list variable and escape the question mark with a backslash

Code: Select all

echo ${list[1]}\?appkey=xxxxxxxxx
AQUA_MODIS.20231207T203500.L2.SST.NRT.nc?appkey=xxxxxxxxx
Tommy

Re: Incorrect File Size for Aqua, Terra and VIIRS SST and Chl files

Posted: Mon Dec 11, 2023 6:37 am America/New_York
by pmlrsg
We are having similar issues here at PML. We are using python requests library to download files. How do we pass the appkey/token to the get request? We are recieving the same 'invalid appkey/token used' when running requests.get('https://oceandata.sci.gsfc.nasa.gov/ob/getfile/JPSS1_VIIRS.20231210T003601.L2.OC.NRT.nc')

Thanks

Re: Incorrect File Size for Aqua, Terra and VIIRS SST and Chl files

Posted: Mon Dec 11, 2023 8:22 am America/New_York
by OB ODPS - towens
Does it not work if append it to the end of the URL?:
requests.get('https://oceandata.sci.gsfc.nasa.gov/ob/getfile/JPSS1_VIIRS.20231210T003601.L2.OC.NRT.nc?appkey=xxxxxxx')

Tommy

Re: Incorrect File Size for Aqua, Terra and VIIRS SST and Chl files

Posted: Mon Dec 11, 2023 8:39 am America/New_York
by pmlrsg
Yes it does, I was wondering if there was a slightly better/more secure method for this. ie. we were using .netrc authentication prior to this.

Re: Incorrect File Size for Aqua, Terra and VIIRS SST and Chl files

Posted: Mon Dec 11, 2023 9:25 am America/New_York
by OB.DAAC - SeanBailey
The obdaac_download script (available via https://oceancolor.gsfc.nasa.gov/data/download_methods/) is a python script that uses the .netrc file for authentication. It also can use the appkey approach. I've just reconfirmed that both methods work. How are you setting up your requests call (it should work with a .netrc file in place...)?

Sean

Re: Incorrect File Size for Aqua, Terra and VIIRS SST and Chl files

Posted: Mon Dec 11, 2023 11:34 am America/New_York
by oceani
Is the appkey approach the new recommended method if using wget to download, or will you be reverting back to how the system functioned prior to Dec 6th?

Thanks,

Mark

Re: Incorrect File Size for Aqua, Terra and VIIRS SST and Chl files

Posted: Mon Dec 11, 2023 11:44 am America/New_York
by gbourdin
Hi all,
I have been having exactly the same issue downloading using requests in python. I also tested wget and had the same problem, but no issue when passing the url into a web browser. I was trying to adapt my python work flow to use obdaac_download but it might be faster to fix how I set my request instead.
So far my code was like this:

Code: Select all

with requests.Session() as s:
    r = s.get('https://oceandata.sci.gsfc.nasa.gov/ob/getfile/A2023227115000.L1A_LAC.bz2', auth=(username, password), stream=True, timeout=900)
    with open('tmp_A2023227115000.L1A_LAC.bz2', "ab") as handle:
        for chunk in r.iter_content(chunk_size=16 * 1024):
            if chunk:
                handle.write(chunk)
But it looks like the username/password method doesn't work anymore. I could use some help to set the request to work again.
Also I recently had issues with incomplete files downloaded, I wanted to add a check to ensure the entire file is downloaded, but I didn't find the size of the file in the requests response.
Best,
Guillaume

Re: Incorrect File Size for Aqua, Terra and VIIRS SST and Chl files

Posted: Mon Dec 11, 2023 1:16 pm America/New_York
by OB.DAAC - SeanBailey
Oh what a tangled web... :shock:

We have identified and corrected the issue that was causing the basic authentication of username/password to fail. All *should* be working again...

Please let us know if you're still having issues.

Sean

Re: Incorrect File Size for Aqua, Terra and VIIRS SST and Chl files

Posted: Mon Dec 11, 2023 1:49 pm America/New_York
by gbourdin
It works perfectly for me thank you so much for your reactivity.
Best,
Guillaume

Re: Incorrect File Size for Aqua, Terra and VIIRS SST and Chl files

Posted: Mon Dec 11, 2023 1:59 pm America/New_York
by oceani
Thank you for taking care of that. Would you recommend one method over the other (i.e. appkey vs. using user name and password)?

-Mark