Accessing NCEPR2 MET files

Use this Forum to find information on, or ask a question about, NASA Earth Science data.
Post Reply
rachel.lazzaro
Posts: 4
Joined: Mon Oct 23, 2023 11:23 am America/New_York
Answers: 0

Accessing NCEPR2 MET files

by rachel.lazzaro » Mon Oct 23, 2023 11:37 am America/New_York

Hi there! I am trying to processing Landsat 8 data using the OC-SMART atmospheric correction software, and I'm running into some issues regarding the ancillary data. I am not the author of the software, but from what I can tell from my debugging, it using the MET_NCEPR2_6h.hdf and O3_AURAOMI_24.hdf files for processing. However, I am running into file access issues with the MET files as trying to download them results in 'Status: 403 Forbidden Apparently, you did not have privileges to access the requested filename.' Are these MET files no longer available for download, or are they located somewhere else? If they are no longer available, which files are appropriate for their replacement? Is there any documentation available that could help me to make this transition?

Thanks,
Rachel

Tags:

OB.DAAC - amscott
User Services
User Services
Posts: 352
Joined: Mon Jun 22, 2020 5:24 pm America/New_York
Answers: 1
Has thanked: 8 times
Been thanked: 3 times

Re: Accessing NCEPR2 MET files

by OB.DAAC - amscott » Mon Oct 23, 2023 1:37 pm America/New_York

1) What was your data retrieval method? Please share what you executed that resulted in an error message.
2) Data downloads require authentication with Earthdata login. After you register with Earthdata Login, you have to authorize the appropriate applications. Have you authorized the appropriate applications in Earthdata login?

rachel.lazzaro
Posts: 4
Joined: Mon Oct 23, 2023 11:23 am America/New_York
Answers: 0

Re: Accessing NCEPR2 MET files

by rachel.lazzaro » Mon Oct 23, 2023 2:52 pm America/New_York

Thanks for your response!
1.) The software is built in Python and uses the Python requests library. When running the Python script, a file is downloaded but is not a properly formatted hdf as it cannot be opened with an HDF viewer and commands like gdalinfo fail on it. When accessing the requested file directly via a web browser, I get the 403 error. Example link: https://oceandata.sci.gsfc.nasa.gov/getfile/N201317518_MET_NCEPR2_6h.hdf
2.) Yes, I have authorized the OB.DAAC data access with Earthdata prior to making any data download requests.

OB.DAAC - amscott
User Services
User Services
Posts: 352
Joined: Mon Jun 22, 2020 5:24 pm America/New_York
Answers: 1
Has thanked: 8 times
Been thanked: 3 times

Re: Accessing NCEPR2 MET files

by OB.DAAC - amscott » Mon Oct 23, 2023 3:21 pm America/New_York

Thanks. That's enough information. We can start looking into this!

OB ODPS - towens
Subject Matter Expert
Subject Matter Expert
Posts: 378
Joined: Fri Feb 05, 2021 9:17 am America/New_York
Answers: 0
Been thanked: 4 times

Re: Accessing NCEPR2 MET files

by OB ODPS - towens » Mon Oct 23, 2023 6:46 pm America/New_York

The NCEPR2 and AURAOMI have been replaced with GMAO MERRA2 ancillary MET files. Update your code to use these instead.

Tommy

OB ODPS - towens
Subject Matter Expert
Subject Matter Expert
Posts: 378
Joined: Fri Feb 05, 2021 9:17 am America/New_York
Answers: 0
Been thanked: 4 times

Re: Accessing NCEPR2 MET files

by OB ODPS - towens » Tue Oct 24, 2023 11:46 am America/New_York

It looks like you can contact kstamnes@stevens.edu for questions or feedback about OC-Smart

We can work with them to sort this out.

Thanks
Tommy

rachel.lazzaro
Posts: 4
Joined: Mon Oct 23, 2023 11:23 am America/New_York
Answers: 0

Re: Accessing NCEPR2 MET files

by rachel.lazzaro » Mon Oct 30, 2023 5:39 pm America/New_York

Thank you very much; I have been in contact with the developers and had a few questions. With the MERRA2 files, it seems that they are not available past day 275 of 2023. Are they no longer being processed past this point? Are the GMAO_FP or GMAO_IT files supposed to be used instead? Thank you!
Rachel

OB ODPS - towens
Subject Matter Expert
Subject Matter Expert
Posts: 378
Joined: Fri Feb 05, 2021 9:17 am America/New_York
Answers: 0
Been thanked: 4 times

Re: Accessing NCEPR2 MET files

by OB ODPS - towens » Tue Oct 31, 2023 7:28 am America/New_York

The MERRA2 reanalysis files are released by GMAO in monthly batches, usually in the third week of the following month. You can expect a 25–55-day latency.

The current Near Real Time products are the GMAO_FP MET files. GMAO is in the process of replacing them with the GEOS_IT NRT stream. We are providing both streams in parallel to allow for testing before the final transition sometime next year.

We have an API that will provide the best available ancillary data. If they want to contact us, we can help them implement it.

Tommy

rachel.lazzaro
Posts: 4
Joined: Mon Oct 23, 2023 11:23 am America/New_York
Answers: 0

Re: Accessing NCEPR2 MET files

by rachel.lazzaro » Tue Oct 31, 2023 9:44 am America/New_York

Awesome, thanks for your response Tommy! Would you be able to point me in the direction of the API for this? I would love to take a look at integrating it as well. Thanks!

OB ODPS - towens
Subject Matter Expert
Subject Matter Expert
Posts: 378
Joined: Fri Feb 05, 2021 9:17 am America/New_York
Answers: 0
Been thanked: 4 times

Re: Accessing NCEPR2 MET files

by OB ODPS - towens » Wed Nov 01, 2023 1:19 pm America/New_York

if you install SeaDAS you can run getanc

For a date in October 2023 it gets the GEOS_FP NRT ancillary
For a Date in September 2023 it gets the MERRA2 refined ancillary

Code: Select all

SeaDAS/ocssw/bin/getanc -m aqua -s 2023291220001 -v
ancillary_data.db
Searching database: SeaDAS/ocssw/var/log/ancillary_data.db

Input file: None
Sensor    : aqua
Start time: 2023-10-18T22:00:01
End time  : 2023-10-18T22:05:01

OBPG session started
Downloading 2023-10-18T22:00:01.anc.server (    0.00 MBs)
[==================================================]
...Done
Downloading 'GMAO_FP.20231018T210000.MET.NRT.nc' to SeaDAS/ocssw//var/anc/2023/291
Downloading GMAO_FP.20231018T210000.MET.NRT.nc (    2.23 MBs)
[==================================================]
...Done
Downloading 'GMAO_FP.20231019T000000.MET.NRT.nc' to SeaDAS/ocssw//var/anc/2023/292
Downloading GMAO_FP.20231019T000000.MET.NRT.nc (    2.23 MBs)
[==================================================]
...Done
Downloading '20231018120000-CMC-L4_GHRSST-SSTfnd-CMC0.1deg-GLOB-v02.0-fv03.0.nc' to SeaDAS/ocssw//var/anc/2023/291
Downloading 20231018120000-CMC-L4_GHRSST-SSTfnd-CMC0.1deg-GLOB-v02.0-fv03.0.nc (    6.71 MBs)
[==================================================]
...Done

Created '2023-10-18T22:00:01.anc' l2gen parameter text file:

icefile=SeaDAS/ocssw//var/anc/2023/291/20231018120000-CMC-L4_GHRSST-SSTfnd-CMC0.1deg-GLOB-v02.0-fv03.0.nc
met1=SeaDAS/ocssw//var/anc/2023/291/GMAO_FP.20231018T210000.MET.NRT.nc
met2=SeaDAS/ocssw//var/anc/2023/292/GMAO_FP.20231019T000000.MET.NRT.nc
met3=SeaDAS/ocssw//var/anc/2023/292/GMAO_FP.20231019T000000.MET.NRT.nc
ozone1=SeaDAS/ocssw//var/anc/2023/291/GMAO_FP.20231018T210000.MET.NRT.nc
ozone2=SeaDAS/ocssw//var/anc/2023/292/GMAO_FP.20231019T000000.MET.NRT.nc
ozone3=SeaDAS/ocssw//var/anc/2023/292/GMAO_FP.20231019T000000.MET.NRT.nc
sstfile=SeaDAS/ocssw//var/anc/2023/291/20231018120000-CMC-L4_GHRSST-SSTfnd-CMC0.1deg-GLOB-v02.0-fv03.0.nc


- All optimal ancillary data files were determined and downloaded. -
[17:14:15] towens@ayu {70}% SeaDAS/ocssw/bin/getanc -m aqua -s 2023261220001 -v
ancillary_data.db
Searching database: SeaDAS/ocssw/var/log/ancillary_data.db

Input file: None
Sensor    : aqua
Start time: 2023-09-18T22:00:01
End time  : None

OBPG session started
Downloading 2023-09-18T22:00:01.anc.server (    0.00 MBs)
[==================================================]
...Done
Downloading 'GMAO_MERRA2.20230918T220000.MET.nc' to SeaDAS/ocssw//var/anc/2023/261
Downloading GMAO_MERRA2.20230918T220000.MET.nc (    2.64 MBs)
[==================================================]
...Done
Downloading 'GMAO_MERRA2.20230918T230000.MET.nc' to SeaDAS/ocssw//var/anc/2023/261
Downloading GMAO_MERRA2.20230918T230000.MET.nc (    2.67 MBs)
[==================================================]
...Done
Downloading '20230918120000-CMC-L4_GHRSST-SSTfnd-CMC0.1deg-GLOB-v02.0-fv03.0.nc' to SeaDAS/ocssw//var/anc/2023/261
Downloading 20230918120000-CMC-L4_GHRSST-SSTfnd-CMC0.1deg-GLOB-v02.0-fv03.0.nc (    6.88 MBs)
[==================================================]
...Done

Created '2023-09-18T22:00:01.anc' l2gen parameter text file:

icefile=SeaDAS/ocssw//var/anc/2023/261/20230918120000-CMC-L4_GHRSST-SSTfnd-CMC0.1deg-GLOB-v02.0-fv03.0.nc
met1=SeaDAS/ocssw//var/anc/2023/261/GMAO_MERRA2.20230918T220000.MET.nc
met2=SeaDAS/ocssw//var/anc/2023/261/GMAO_MERRA2.20230918T230000.MET.nc
met3=SeaDAS/ocssw//var/anc/2023/261/GMAO_MERRA2.20230918T230000.MET.nc
ozone1=SeaDAS/ocssw//var/anc/2023/261/GMAO_MERRA2.20230918T220000.MET.nc
ozone2=SeaDAS/ocssw//var/anc/2023/261/GMAO_MERRA2.20230918T230000.MET.nc
ozone3=SeaDAS/ocssw//var/anc/2023/261/GMAO_MERRA2.20230918T230000.MET.nc
sstfile=SeaDAS/ocssw//var/anc/2023/261/20230918120000-CMC-L4_GHRSST-SSTfnd-CMC0.1deg-GLOB-v02.0-fv03.0.nc


- All optimal ancillary data files were determined and downloaded. -

Post Reply