Page 1 of 1

Python Sen3/OLCI download error

Posted: Thu Nov 30, 2023 11:16 am America/New_York
by slsmit34
Hello,
I am encountering an issue with downloaded Sen3/OLCI EFR OC data from OB.DAAC through python with the earthaccess package. I can successfully search for granules and retrieve their url for download (see attached image 'download_script'). At this point, clicking on the download url allows me to successfully download the image. However, when I use earthaccess.download, the downloaded file is always a corrupt 3.9kb file (see attached 'download_file'). I've also tried with a .get function (see attached 'alternate_download_script) and encounter the same issue.
This same script can successfully download EMIT and VIIRS, but does not work with S3A or S3B. Do you have any advice?
Best,
Sami

Re: Python Sen3/OLCI download error

Posted: Mon Dec 04, 2023 2:57 pm America/New_York
by OB WebDev - masoudso
Hello,
Would you please try downloading the file directly using the API and let us know if the download is successful? You won't need to have 'cmr' before the filename:
https://oceandata.sci.gsfc.nasa.gov/getfile/S3A_OLCI_EFRNT.20221026T153802.L2.OC.nc

Re: Python Sen3/OLCI download error

Posted: Mon Dec 04, 2023 3:02 pm America/New_York
by slsmit34
Hi,
By API, do you mean the earthdata search portal (https://search.earthdata.nasa.gov/search)? If so, yes, I can successfully download the files from both the earthdata search portal and clicking directly on link. I only encounter issues when passing the OB.DAAC OLCI/MERIS links through earthaccess.download or similar python commands.
Best,
Sami

Re: Python Sen3/OLCI download error

Posted: Wed Dec 06, 2023 9:40 am America/New_York
by OB.DAAC - SeanBailey
The earthaccess code is not maintained by the OB.DAAC. It appears that there is an issue passing the EDL credentials to our getfile API. We are investigating, but may have to defer to the developers of the earthaccess code for a resolution.

Regards,
Sean

Re: Python Sen3/OLCI download error

Posted: Thu Dec 07, 2023 11:08 am America/New_York
by OB.DAAC - SeanBailey
Just a follow up...

We believe we have identified the issue and indeed it is related to how earthaccess is passing credentials. The problem is how getfile accepts credentials. It will take a little while to make the appropriate changes to getfile.
Until the fix is in place, you can use the earthaccess api to search for OB.DAAC data, but not directly download it. You can use the obdaac_download script (see https://oceancolor.gsfc.nasa.gov/data/download_methods/) to retrieve the data identified by the earthaccess search results.

Regards,
Sean

Re: Python Sen3/OLCI download error

Posted: Thu Dec 07, 2023 4:04 pm America/New_York
by slsmit34
Hi Sean,
Thanks! I saw that VIIRS data on the Earthdata portal is .h5, but on the oceancolor.gsfc portal it is .nc format. I prefer .nc, so I am going to use the ocean color site instead.
I am trying to use the obdaac_download script you mentioned to download all viirs/modis,etc. data over a bounding box region during an entire year. I was able to use the oceancolor.gsfc.nasa.gov browser to retrieve a list of all of the relevant file names (i.e. SNPP_VIIRS.20171231T185400.L2.OC.nc, SNPP_VIIRS.20171112T191200.L2.OC.nc, etc.) But am not sure if this is the correct input for '--filelist' in the script? And what the '--http_manifest' should be?
I apologize if this is a simple question as I am new to automating satellite download!

Re: Python Sen3/OLCI download error

Posted: Thu Dec 07, 2023 4:18 pm America/New_York
by OB.DAAC - SeanBailey
If you have a list of files, you'd want to use the --filelist option (--http-manifest is to support our data ordering system, which produces such a manifest file). That said, it seems my team was a lot faster implementing the fix than I thought would be possible. You can try your earthaccess approach and it should now work :)

Sean

Re: Python Sen3/OLCI download error

Posted: Thu Dec 07, 2023 4:50 pm America/New_York
by slsmit34
Hi Sean,
Thanks so much! It works now. I have one last question - is there a way to use earthdata to access the .nc VIIRS collection that is on the oceancolor.gsfc.nasa.gov portal? All of the VIIRS collections I see on Earthdata (i.e. VNP09GA) are in .h5 format and not geolocated/scaled. The .nc format VIIRS ocean color product for the oceancolor.gsfc.nasa.gov is much better. Is there a way to pull this data from earthaccess? Or must it be done with the obdaac_download method?

Re: Python Sen3/OLCI download error

Posted: Mon Dec 11, 2023 7:57 am America/New_York
by OB.DAAC - SeanBailey
You can identify the VIIRS data sets we support in Earthdata Search by limiting your search to the 'Ocean Biology Distributed Active Archive Center' under the "Organizations" facet on the left side of the search tool. Select 'VIIRS' under the "Instruments" facet as well and you should see the matching collections. You can further limit to by processing level.

Regards,
Sean