Download subset of VNP02IMG data.

Use this Forum to find information on, or ask a question about, NASA Earth Science data.
Post Reply
b.coerver
Posts: 3
Joined: Tue Mar 12, 2024 8:09 am America/New_York
Answers: 0
Been thanked: 2 times

Download subset of VNP02IMG data.

by b.coerver » Tue Mar 12, 2024 8:27 am America/New_York

Hello,

I'm using VNP02IMG data for a project, and more specifically only the following three subdatasets:

- /observation_data/I05_quality_flags
- /observation_data/I05
- /observation_data/I05_brightness_temperature_lut

Until recently, I could download subsets of this dataset through S3, but apparently this was a loophole which is now closed (see here for more info: https://github.com/nsidc/earthaccess/discussions/489).

So I think there are two other options left for me to access this data (https://cmr.earthdata.nasa.gov/search/concepts/C2105091501-LAADS.html):

(1) Using a direct download (https://ladsweb.modaps.eosdis.nasa.gov/archive/allData/5200/VNP02IMG/)
(2) Using OpenDAP (https://ladsweb.modaps.eosdis.nasa.gov/opendap/RemoteResources/laads/allData/5200/VNP02IMG/contents.html)

As far as I know it's impossible to download a subset of the data through option (1). For example when I run:

gdalinfo "/vsicurl/https://ladsweb.modaps.eosdis.nasa.gov/archive/allData/5200/VNP02IMG/2016/015/VNP02IMG.A2016015.0030.002.2021082161921.nc"

I get a message saying "ERROR 1: Range downloading not supported by this server!"

So my last resort is option (2), unfortunately, none of the subdatasets that are included in the netcdf files are listed in the opendap. I guess opendap doesnt work with subdatasets? Or is this a bug? Or am I doing something wrong?

Is it really only possible to work with this data by downloading the entire file (from which I only need about 20%)?

Thanks a lot for any help or insights!

Regards,
Bert

Filters:

LAADS_UserServices_M
User Services
User Services
Posts: 294
Joined: Mon Sep 30, 2019 8:33 am America/New_York
Answers: 1
Has thanked: 3 times

Re: Download subset of VNP02IMG data.

by LAADS_UserServices_M » Tue Mar 12, 2024 10:12 am America/New_York

Dear Bert:
You can use LAADS post processing tool to perform spatial and/or parameter subset on files you need. This is available at the last step (REVIEW & ORDER) of the ordering process using LAADS DAAC's "Find Data" interface at:
https://ladsweb.modaps.eosdis.nasa.gov/search/order/5/VNP02IMG--5200/

This way you can order only the parameters (SDS) you need. If this is the first time you are using post processing tool, you can consult “How to Use LAADS DAAC Post-Processing Tools” tutorial available at:
https://ladsweb.modaps.eosdis.nasa.gov/learn

For OPeNDAP, you can consult various examples provided at:
https://ladsweb.modaps.eosdis.nasa.gov/tools-and-services#opendap
Regards,
LAADS User Services

To receive news from LAADS DAAC direct to your inbox, email laadsdaac-join@lists.nasa.gov with “subscribe” in the subject line.

b.coerver
Posts: 3
Joined: Tue Mar 12, 2024 8:09 am America/New_York
Answers: 0
Been thanked: 2 times

Re: Download subset of VNP02IMG data.

by b.coerver » Tue Mar 12, 2024 10:42 am America/New_York

Hi,

Thanks for your reply.

- About the Post-Processing tool: I saw that, but I don't think it's possible to access this through an API? Also there would be queues and waiting times involved, which is not ideal.

- About openDAP: I've seen the examples and used openDAP before. But I don't think those examples apply here (perhaps because the data is stored inside subdatasets?). If you go to the link here (https://ladsweb.modaps.eosdis.nasa.gov/opendap/RemoteResources/laads/allData/5200/VNP02IMG/2018/009/VNP02IMG.A2018009.0036.002.2021078103914.nc.dmr.html) for example, there is nothing listed under "variables" and "global dimensions". Also when I click on the "Get As Netcdf4" download button, I get a ~11kb file that only contains the global metadata but not the actual data. (even though in the file listing here [https://ladsweb.modaps.eosdis.nasa.gov/opendap/RemoteResources/laads/allData/5200/VNP02IMG/2018/009/contents.html], its says the file has a size of 271169961 bytes).

So if someone knows how to access those subdatasets through opendap, i'd be very happy to know!

Bert

LAADS_UserServices_M
User Services
User Services
Posts: 294
Joined: Mon Sep 30, 2019 8:33 am America/New_York
Answers: 1
Has thanked: 3 times

Re: Download subset of VNP02IMG data.

by LAADS_UserServices_M » Wed Mar 13, 2024 12:25 pm America/New_York

For OPeNDAP, you must be following discussion with Nathan Potter on your ticket to support-laads.
So, while the OPeNDAP issue is worked on and get resolved you can explore LAADS "LWS Classic" API option. This is available at:
https://ladsweb.modaps.eosdis.nasa.gov/tools-and-services/#lwsclassic

Here under "LWS Classic" / "Quick Start Guide" you can find information on how to write script to search and order files with examples of perl scripts?
Regards,
LAADS User Services

To receive news from LAADS DAAC direct to your inbox, email laadsdaac-join@lists.nasa.gov with “subscribe” in the subject line.

b.coerver
Posts: 3
Joined: Tue Mar 12, 2024 8:09 am America/New_York
Answers: 0
Been thanked: 2 times

Re: Download subset of VNP02IMG data.

by b.coerver » Thu Mar 14, 2024 7:52 am America/New_York

Hello,

The OPeNDAP issue is resolved. OPeNDAP now works for LAADS datasets that have variables stored inside "groups" (I tested a few of them).

Thanks a lot for your help!

Regards,
Bert

Post Reply