Python Sen3/OLCI download error

Use this Forum to find information on, or ask a question about, NASA Earth Science data.
Post Reply
slsmit34
Posts: 8
Joined: Thu May 18, 2023 10:29 am America/New_York
Answers: 0

Python Sen3/OLCI download error

by slsmit34 » Thu Nov 30, 2023 11:16 am America/New_York

Hello,
I am encountering an issue with downloaded Sen3/OLCI EFR OC data from OB.DAAC through python with the earthaccess package. I can successfully search for granules and retrieve their url for download (see attached image 'download_script'). At this point, clicking on the download url allows me to successfully download the image. However, when I use earthaccess.download, the downloaded file is always a corrupt 3.9kb file (see attached 'download_file'). I've also tried with a .get function (see attached 'alternate_download_script) and encounter the same issue.
This same script can successfully download EMIT and VIIRS, but does not work with S3A or S3B. Do you have any advice?
Best,
Sami
Attachments
An alternate method for downloadin with earthaccess. The downloaded file is always 9.1kb and contains no data.
An alternate method for downloadin with earthaccess. The downloaded file is always 9.1kb and contains no data.
alternate_download_script.PNG (21.95 KiB) Not viewed yet
The downloaded file is always 3.9kb and contains no data.
The downloaded file is always 3.9kb and contains no data.
download_file.PNG (62.74 KiB) Not viewed yet
The file searched and retrieved should be 116mb.
The file searched and retrieved should be 116mb.
download_script.PNG (51.84 KiB) Not viewed yet

Tags:

OB WebDev - masoudso
Subject Matter Expert
Subject Matter Expert
Posts: 23
Joined: Tue Dec 22, 2020 2:11 pm America/New_York
Answers: 0
Has thanked: 2 times

Re: Python Sen3/OLCI download error

by OB WebDev - masoudso » Mon Dec 04, 2023 2:57 pm America/New_York

Hello,
Would you please try downloading the file directly using the API and let us know if the download is successful? You won't need to have 'cmr' before the filename:
https://oceandata.sci.gsfc.nasa.gov/getfile/S3A_OLCI_EFRNT.20221026T153802.L2.OC.nc

slsmit34
Posts: 8
Joined: Thu May 18, 2023 10:29 am America/New_York
Answers: 0

Re: Python Sen3/OLCI download error

by slsmit34 » Mon Dec 04, 2023 3:02 pm America/New_York

Hi,
By API, do you mean the earthdata search portal (https://search.earthdata.nasa.gov/search)? If so, yes, I can successfully download the files from both the earthdata search portal and clicking directly on link. I only encounter issues when passing the OB.DAAC OLCI/MERIS links through earthaccess.download or similar python commands.
Best,
Sami

OB.DAAC - SeanBailey
User Services
User Services
Posts: 1461
Joined: Wed Sep 18, 2019 6:15 pm America/New_York
Answers: 1
Been thanked: 4 times

Re: Python Sen3/OLCI download error

by OB.DAAC - SeanBailey » Wed Dec 06, 2023 9:40 am America/New_York

The earthaccess code is not maintained by the OB.DAAC. It appears that there is an issue passing the EDL credentials to our getfile API. We are investigating, but may have to defer to the developers of the earthaccess code for a resolution.

Regards,
Sean

OB.DAAC - SeanBailey
User Services
User Services
Posts: 1461
Joined: Wed Sep 18, 2019 6:15 pm America/New_York
Answers: 1
Been thanked: 4 times

Re: Python Sen3/OLCI download error

by OB.DAAC - SeanBailey » Thu Dec 07, 2023 11:08 am America/New_York

Just a follow up...

We believe we have identified the issue and indeed it is related to how earthaccess is passing credentials. The problem is how getfile accepts credentials. It will take a little while to make the appropriate changes to getfile.
Until the fix is in place, you can use the earthaccess api to search for OB.DAAC data, but not directly download it. You can use the obdaac_download script (see https://oceancolor.gsfc.nasa.gov/data/download_methods/) to retrieve the data identified by the earthaccess search results.

Regards,
Sean

slsmit34
Posts: 8
Joined: Thu May 18, 2023 10:29 am America/New_York
Answers: 0

Re: Python Sen3/OLCI download error

by slsmit34 » Thu Dec 07, 2023 4:04 pm America/New_York

Hi Sean,
Thanks! I saw that VIIRS data on the Earthdata portal is .h5, but on the oceancolor.gsfc portal it is .nc format. I prefer .nc, so I am going to use the ocean color site instead.
I am trying to use the obdaac_download script you mentioned to download all viirs/modis,etc. data over a bounding box region during an entire year. I was able to use the oceancolor.gsfc.nasa.gov browser to retrieve a list of all of the relevant file names (i.e. SNPP_VIIRS.20171231T185400.L2.OC.nc, SNPP_VIIRS.20171112T191200.L2.OC.nc, etc.) But am not sure if this is the correct input for '--filelist' in the script? And what the '--http_manifest' should be?
I apologize if this is a simple question as I am new to automating satellite download!

OB.DAAC - SeanBailey
User Services
User Services
Posts: 1461
Joined: Wed Sep 18, 2019 6:15 pm America/New_York
Answers: 1
Been thanked: 4 times

Re: Python Sen3/OLCI download error

by OB.DAAC - SeanBailey » Thu Dec 07, 2023 4:18 pm America/New_York

If you have a list of files, you'd want to use the --filelist option (--http-manifest is to support our data ordering system, which produces such a manifest file). That said, it seems my team was a lot faster implementing the fix than I thought would be possible. You can try your earthaccess approach and it should now work :)

Sean

slsmit34
Posts: 8
Joined: Thu May 18, 2023 10:29 am America/New_York
Answers: 0

Re: Python Sen3/OLCI download error

by slsmit34 » Thu Dec 07, 2023 4:50 pm America/New_York

Hi Sean,
Thanks so much! It works now. I have one last question - is there a way to use earthdata to access the .nc VIIRS collection that is on the oceancolor.gsfc.nasa.gov portal? All of the VIIRS collections I see on Earthdata (i.e. VNP09GA) are in .h5 format and not geolocated/scaled. The .nc format VIIRS ocean color product for the oceancolor.gsfc.nasa.gov is much better. Is there a way to pull this data from earthaccess? Or must it be done with the obdaac_download method?

OB.DAAC - SeanBailey
User Services
User Services
Posts: 1461
Joined: Wed Sep 18, 2019 6:15 pm America/New_York
Answers: 1
Been thanked: 4 times

Re: Python Sen3/OLCI download error

by OB.DAAC - SeanBailey » Mon Dec 11, 2023 7:57 am America/New_York

You can identify the VIIRS data sets we support in Earthdata Search by limiting your search to the 'Ocean Biology Distributed Active Archive Center' under the "Organizations" facet on the left side of the search tool. Select 'VIIRS' under the "Instruments" facet as well and you should see the matching collections. You can further limit to by processing level.

Regards,
Sean

Post Reply