Accessing NCEPR2 MET files

[rachel.lazzaro]

Hi there! I am trying to processing Landsat 8 data using the OC-SMART atmospheric correction software, and I'm running into some issues regarding the ancillary data. I am not the author of the software, but from what I can tell from my debugging, it using the MET_NCEPR2_6h.hdf and O3_AURAOMI_24.hdf files for processing. However, I am running into file access issues with the MET files as trying to download them results in 'Status: 403 Forbidden Apparently, you did not have privileges to access the requested filename.' Are these MET files no longer available for download, or are they located somewhere else? If they are no longer available, which files are appropriate for their replacement? Is there any documentation available that could help me to make this transition?

Thanks,
Rachel

[OB.DAACx - amscott]

1) What was your data retrieval method? Please share what you executed that resulted in an error message.
2) Data downloads require authentication with Earthdata login. After you register with Earthdata Login, you have to authorize the appropriate applications. Have you authorized the appropriate applications in Earthdata login?

[rachel.lazzaro]

Thanks for your response!
1.) The software is built in Python and uses the Python requests library. When running the Python script, a file is downloaded but is not a properly formatted hdf as it cannot be opened with an HDF viewer and commands like gdalinfo fail on it. When accessing the requested file directly via a web browser, I get the 403 error. Example link: https://oceandata.sci.gsfc.nasa.gov/getfile/N201317518_MET_NCEPR2_6h.hdf
2.) Yes, I have authorized the OB.DAAC data access with Earthdata prior to making any data download requests.

[OB.DAACx - amscott]

Thanks. That's enough information. We can start looking into this!

[OB ODPS - towens]

The NCEPR2 and AURAOMI have been replaced with GMAO MERRA2 ancillary MET files. Update your code to use these instead.

Tommy

[OB ODPS - towens]

It looks like you can contact kstamnes@stevens.edu for questions or feedback about OC-Smart

We can work with them to sort this out.

Thanks
Tommy

[rachel.lazzaro]

Thank you very much; I have been in contact with the developers and had a few questions. With the MERRA2 files, it seems that they are not available past day 275 of 2023. Are they no longer being processed past this point? Are the GMAO_FP or GMAO_IT files supposed to be used instead? Thank you!
Rachel

[OB ODPS - towens]

The MERRA2 reanalysis files are released by GMAO in monthly batches, usually in the third week of the following month. You can expect a 25–55-day latency.

The current Near Real Time products are the GMAO_FP MET files. GMAO is in the process of replacing them with the GEOS_IT NRT stream. We are providing both streams in parallel to allow for testing before the final transition sometime next year.

We have an API that will provide the best available ancillary data. If they want to contact us, we can help them implement it.

Tommy

[rachel.lazzaro]

Awesome, thanks for your response Tommy! Would you be able to point me in the direction of the API for this? I would love to take a look at integrating it as well. Thanks!

[OB ODPS - towens]

if you install SeaDAS you can run getanc

For a date in October 2023 it gets the GEOS_FP NRT ancillary
For a Date in September 2023 it gets the MERRA2 refined ancillary

Code: Select all

SeaDAS/ocssw/bin/getanc -m aqua -s 2023291220001 -v
ancillary_data.db
Searching database: SeaDAS/ocssw/var/log/ancillary_data.db

Input file: None
Sensor    : aqua
Start time: 2023-10-18T22:00:01
End time  : 2023-10-18T22:05:01

OBPG session started
Downloading 2023-10-18T22:00:01.anc.server (    0.00 MBs)
[==================================================]
...Done
Downloading 'GMAO_FP.20231018T210000.MET.NRT.nc' to SeaDAS/ocssw//var/anc/2023/291
Downloading GMAO_FP.20231018T210000.MET.NRT.nc (    2.23 MBs)
[==================================================]
...Done
Downloading 'GMAO_FP.20231019T000000.MET.NRT.nc' to SeaDAS/ocssw//var/anc/2023/292
Downloading GMAO_FP.20231019T000000.MET.NRT.nc (    2.23 MBs)
[==================================================]
...Done
Downloading '20231018120000-CMC-L4_GHRSST-SSTfnd-CMC0.1deg-GLOB-v02.0-fv03.0.nc' to SeaDAS/ocssw//var/anc/2023/291
Downloading 20231018120000-CMC-L4_GHRSST-SSTfnd-CMC0.1deg-GLOB-v02.0-fv03.0.nc (    6.71 MBs)
[==================================================]
...Done

Created '2023-10-18T22:00:01.anc' l2gen parameter text file:

icefile=SeaDAS/ocssw//var/anc/2023/291/20231018120000-CMC-L4_GHRSST-SSTfnd-CMC0.1deg-GLOB-v02.0-fv03.0.nc
met1=SeaDAS/ocssw//var/anc/2023/291/GMAO_FP.20231018T210000.MET.NRT.nc
met2=SeaDAS/ocssw//var/anc/2023/292/GMAO_FP.20231019T000000.MET.NRT.nc
met3=SeaDAS/ocssw//var/anc/2023/292/GMAO_FP.20231019T000000.MET.NRT.nc
ozone1=SeaDAS/ocssw//var/anc/2023/291/GMAO_FP.20231018T210000.MET.NRT.nc
ozone2=SeaDAS/ocssw//var/anc/2023/292/GMAO_FP.20231019T000000.MET.NRT.nc
ozone3=SeaDAS/ocssw//var/anc/2023/292/GMAO_FP.20231019T000000.MET.NRT.nc
sstfile=SeaDAS/ocssw//var/anc/2023/291/20231018120000-CMC-L4_GHRSST-SSTfnd-CMC0.1deg-GLOB-v02.0-fv03.0.nc


- All optimal ancillary data files were determined and downloaded. -
[17:14:15] towens@ayu {70}% SeaDAS/ocssw/bin/getanc -m aqua -s 2023261220001 -v
ancillary_data.db
Searching database: SeaDAS/ocssw/var/log/ancillary_data.db

Input file: None
Sensor    : aqua
Start time: 2023-09-18T22:00:01
End time  : None

OBPG session started
Downloading 2023-09-18T22:00:01.anc.server (    0.00 MBs)
[==================================================]
...Done
Downloading 'GMAO_MERRA2.20230918T220000.MET.nc' to SeaDAS/ocssw//var/anc/2023/261
Downloading GMAO_MERRA2.20230918T220000.MET.nc (    2.64 MBs)
[==================================================]
...Done
Downloading 'GMAO_MERRA2.20230918T230000.MET.nc' to SeaDAS/ocssw//var/anc/2023/261
Downloading GMAO_MERRA2.20230918T230000.MET.nc (    2.67 MBs)
[==================================================]
...Done
Downloading '20230918120000-CMC-L4_GHRSST-SSTfnd-CMC0.1deg-GLOB-v02.0-fv03.0.nc' to SeaDAS/ocssw//var/anc/2023/261
Downloading 20230918120000-CMC-L4_GHRSST-SSTfnd-CMC0.1deg-GLOB-v02.0-fv03.0.nc (    6.88 MBs)
[==================================================]
...Done

Created '2023-09-18T22:00:01.anc' l2gen parameter text file:

icefile=SeaDAS/ocssw//var/anc/2023/261/20230918120000-CMC-L4_GHRSST-SSTfnd-CMC0.1deg-GLOB-v02.0-fv03.0.nc
met1=SeaDAS/ocssw//var/anc/2023/261/GMAO_MERRA2.20230918T220000.MET.nc
met2=SeaDAS/ocssw//var/anc/2023/261/GMAO_MERRA2.20230918T230000.MET.nc
met3=SeaDAS/ocssw//var/anc/2023/261/GMAO_MERRA2.20230918T230000.MET.nc
ozone1=SeaDAS/ocssw//var/anc/2023/261/GMAO_MERRA2.20230918T220000.MET.nc
ozone2=SeaDAS/ocssw//var/anc/2023/261/GMAO_MERRA2.20230918T230000.MET.nc
ozone3=SeaDAS/ocssw//var/anc/2023/261/GMAO_MERRA2.20230918T230000.MET.nc
sstfile=SeaDAS/ocssw//var/anc/2023/261/20230918120000-CMC-L4_GHRSST-SSTfnd-CMC0.1deg-GLOB-v02.0-fv03.0.nc


- All optimal ancillary data files were determined and downloaded. -