Incorrect File Size for Aqua, Terra and VIIRS SST and Chl files

Use this Forum to find information on, or ask a question about, NASA Earth Science data.
OB ODPS - towens
Subject Matter Expert
Subject Matter Expert
Posts: 391
Joined: Fri Feb 05, 2021 9:17 am America/New_York
Answers: 0
Been thanked: 5 times

Re: Incorrect File Size for Aqua, Terra and VIIRS SST and Chl files

by OB ODPS - towens » Sat Dec 09, 2023 2:14 pm America/New_York

Put curly braces around the list variable and escape the question mark with a backslash

Code: Select all

echo ${list[1]}\?appkey=xxxxxxxxx
AQUA_MODIS.20231207T203500.L2.SST.NRT.nc?appkey=xxxxxxxxx
Tommy

Filters:

pmlrsg
Posts: 11
Joined: Tue Feb 18, 2020 5:50 pm America/New_York
Answers: 0

Re: Incorrect File Size for Aqua, Terra and VIIRS SST and Chl files

by pmlrsg » Mon Dec 11, 2023 6:37 am America/New_York

We are having similar issues here at PML. We are using python requests library to download files. How do we pass the appkey/token to the get request? We are recieving the same 'invalid appkey/token used' when running requests.get('https://oceandata.sci.gsfc.nasa.gov/ob/getfile/JPSS1_VIIRS.20231210T003601.L2.OC.NRT.nc')

Thanks

OB ODPS - towens
Subject Matter Expert
Subject Matter Expert
Posts: 391
Joined: Fri Feb 05, 2021 9:17 am America/New_York
Answers: 0
Been thanked: 5 times

Re: Incorrect File Size for Aqua, Terra and VIIRS SST and Chl files

by OB ODPS - towens » Mon Dec 11, 2023 8:22 am America/New_York

Does it not work if append it to the end of the URL?:
requests.get('https://oceandata.sci.gsfc.nasa.gov/ob/getfile/JPSS1_VIIRS.20231210T003601.L2.OC.NRT.nc?appkey=xxxxxxx')

Tommy

pmlrsg
Posts: 11
Joined: Tue Feb 18, 2020 5:50 pm America/New_York
Answers: 0

Re: Incorrect File Size for Aqua, Terra and VIIRS SST and Chl files

by pmlrsg » Mon Dec 11, 2023 8:39 am America/New_York

Yes it does, I was wondering if there was a slightly better/more secure method for this. ie. we were using .netrc authentication prior to this.

OB.DAAC - SeanBailey
User Services
User Services
Posts: 1485
Joined: Wed Sep 18, 2019 6:15 pm America/New_York
Answers: 1
Been thanked: 5 times

Re: Incorrect File Size for Aqua, Terra and VIIRS SST and Chl files

by OB.DAAC - SeanBailey » Mon Dec 11, 2023 9:25 am America/New_York

The obdaac_download script (available via https://oceancolor.gsfc.nasa.gov/data/download_methods/) is a python script that uses the .netrc file for authentication. It also can use the appkey approach. I've just reconfirmed that both methods work. How are you setting up your requests call (it should work with a .netrc file in place...)?

Sean

oceani
Posts: 79
Joined: Wed Aug 23, 2006 12:32 pm America/New_York
Answers: 0
Has thanked: 2 times

Re: Incorrect File Size for Aqua, Terra and VIIRS SST and Chl files

by oceani » Mon Dec 11, 2023 11:34 am America/New_York

Is the appkey approach the new recommended method if using wget to download, or will you be reverting back to how the system functioned prior to Dec 6th?

Thanks,

Mark

gbourdin
Posts: 34
Joined: Mon Jan 25, 2021 11:59 am America/New_York
Answers: 0
Has thanked: 1 time

Re: Incorrect File Size for Aqua, Terra and VIIRS SST and Chl files

by gbourdin » Mon Dec 11, 2023 11:44 am America/New_York

Hi all,
I have been having exactly the same issue downloading using requests in python. I also tested wget and had the same problem, but no issue when passing the url into a web browser. I was trying to adapt my python work flow to use obdaac_download but it might be faster to fix how I set my request instead.
So far my code was like this:

Code: Select all

with requests.Session() as s:
    r = s.get('https://oceandata.sci.gsfc.nasa.gov/ob/getfile/A2023227115000.L1A_LAC.bz2', auth=(username, password), stream=True, timeout=900)
    with open('tmp_A2023227115000.L1A_LAC.bz2', "ab") as handle:
        for chunk in r.iter_content(chunk_size=16 * 1024):
            if chunk:
                handle.write(chunk)
But it looks like the username/password method doesn't work anymore. I could use some help to set the request to work again.
Also I recently had issues with incomplete files downloaded, I wanted to add a check to ensure the entire file is downloaded, but I didn't find the size of the file in the requests response.
Best,
Guillaume

OB.DAAC - SeanBailey
User Services
User Services
Posts: 1485
Joined: Wed Sep 18, 2019 6:15 pm America/New_York
Answers: 1
Been thanked: 5 times

Re: Incorrect File Size for Aqua, Terra and VIIRS SST and Chl files

by OB.DAAC - SeanBailey » Mon Dec 11, 2023 1:16 pm America/New_York

Oh what a tangled web... :shock:

We have identified and corrected the issue that was causing the basic authentication of username/password to fail. All *should* be working again...

Please let us know if you're still having issues.

Sean

gbourdin
Posts: 34
Joined: Mon Jan 25, 2021 11:59 am America/New_York
Answers: 0
Has thanked: 1 time

Re: Incorrect File Size for Aqua, Terra and VIIRS SST and Chl files

by gbourdin » Mon Dec 11, 2023 1:49 pm America/New_York

It works perfectly for me thank you so much for your reactivity.
Best,
Guillaume

oceani
Posts: 79
Joined: Wed Aug 23, 2006 12:32 pm America/New_York
Answers: 0
Has thanked: 2 times

Re: Incorrect File Size for Aqua, Terra and VIIRS SST and Chl files

by oceani » Mon Dec 11, 2023 1:59 pm America/New_York

Thank you for taking care of that. Would you recommend one method over the other (i.e. appkey vs. using user name and password)?

-Mark

Post Reply