Python Sen3/OLCI download error

[slsmit34]

Hello,
I am encountering an issue with downloaded Sen3/OLCI EFR OC data from OB.DAAC through python with the earthaccess package. I can successfully search for granules and retrieve their url for download (see attached image 'download_script'). At this point, clicking on the download url allows me to successfully download the image. However, when I use earthaccess.download, the downloaded file is always a corrupt 3.9kb file (see attached 'download_file'). I've also tried with a .get function (see attached 'alternate_download_script) and encounter the same issue.
This same script can successfully download EMIT and VIIRS, but does not work with S3A or S3B. Do you have any advice?
Best,
Sami

[OB WebDev - masoudso]

Hello,
Would you please try downloading the file directly using the API and let us know if the download is successful? You won't need to have 'cmr' before the filename:
https://oceandata.sci.gsfc.nasa.gov/getfile/S3A_OLCI_EFRNT.20221026T153802.L2.OC.nc

[slsmit34]

Hi,
By API, do you mean the earthdata search portal (https://search.earthdata.nasa.gov/search)? If so, yes, I can successfully download the files from both the earthdata search portal and clicking directly on link. I only encounter issues when passing the OB.DAAC OLCI/MERIS links through earthaccess.download or similar python commands.
Best,
Sami

[OB.DAAC - SeanBailey]

The earthaccess code is not maintained by the OB.DAAC. It appears that there is an issue passing the EDL credentials to our getfile API. We are investigating, but may have to defer to the developers of the earthaccess code for a resolution.

Regards,
Sean

[OB.DAAC - SeanBailey]

Just a follow up...

We believe we have identified the issue and indeed it is related to how earthaccess is passing credentials. The problem is how getfile accepts credentials. It will take a little while to make the appropriate changes to getfile.
Until the fix is in place, you can use the earthaccess api to search for OB.DAAC data, but not directly download it. You can use the obdaac_download script (see https://oceancolor.gsfc.nasa.gov/data/download_methods/) to retrieve the data identified by the earthaccess search results.

Regards,
Sean

[slsmit34]

Hi Sean,
Thanks! I saw that VIIRS data on the Earthdata portal is .h5, but on the oceancolor.gsfc portal it is .nc format. I prefer .nc, so I am going to use the ocean color site instead.
I am trying to use the obdaac_download script you mentioned to download all viirs/modis,etc. data over a bounding box region during an entire year. I was able to use the oceancolor.gsfc.nasa.gov browser to retrieve a list of all of the relevant file names (i.e. SNPP_VIIRS.20171231T185400.L2.OC.nc, SNPP_VIIRS.20171112T191200.L2.OC.nc, etc.) But am not sure if this is the correct input for '--filelist' in the script? And what the '--http_manifest' should be?
I apologize if this is a simple question as I am new to automating satellite download!

[OB.DAAC - SeanBailey]

If you have a list of files, you'd want to use the --filelist option (--http-manifest is to support our data ordering system, which produces such a manifest file). That said, it seems my team was a lot faster implementing the fix than I thought would be possible. You can try your earthaccess approach and it should now work

Sean

[slsmit34]

Hi Sean,
Thanks so much! It works now. I have one last question - is there a way to use earthdata to access the .nc VIIRS collection that is on the oceancolor.gsfc.nasa.gov portal? All of the VIIRS collections I see on Earthdata (i.e. VNP09GA) are in .h5 format and not geolocated/scaled. The .nc format VIIRS ocean color product for the oceancolor.gsfc.nasa.gov is much better. Is there a way to pull this data from earthaccess? Or must it be done with the obdaac_download method?

[OB.DAAC - SeanBailey]

You can identify the VIIRS data sets we support in Earthdata Search by limiting your search to the 'Ocean Biology Distributed Active Archive Center' under the "Organizations" facet on the left side of the search tool. Select 'VIIRS' under the "Instruments" facet as well and you should see the matching collections. You can further limit to by processing level.

Regards,
Sean