CMR has no entries for MOD13Q1.061 files at LP DAAC

Use this Forum to find information on, or ask a question about, NASA Earth Science data.
Post Reply
earthengine_urs
Posts: 48
Joined: Mon Jan 27, 2020 10:36 am America/New_York
Answers: 0
Been thanked: 1 time

CMR has no entries for MOD13Q1.061 files at LP DAAC

by earthengine_urs » Tue Oct 25, 2022 5:54 pm America/New_York

I noticed at least one change in MODIS 6.1 CMR ids - the MOD13Q1.061 collections used to be C1621383370-LPDAAC_ECS, but now queries (https://cmr.earthdata.nasa.gov/search/granules.json?online_only=true&collection_concept_id[]=C1621383370-LPDAAC_ECS&page_size=1000&temporal[]=2022-06-26T00:00:00/2022-06-27T00:00:00&page_num=1) for granules in that collection return nothing. Open search returns a new collection id C1748066515-LPCLOUD, and its files (https://cmr.earthdata.nasa.gov/search/granules.json?online_only=true&collection_concept_id[]=C1748066515-LPCLOUD&page_size=1&temporal[]=2022-06-26T00:00:00/2022-06-27T00:00:00&page_num=1) are no longer at LP DAAC - the URLs are
https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/MOD13Q1.061/MOD13Q1.A2022177.h02v10.061.2022195120340/MOD13Q1.A2022177.h02v10.061.2022195120340.hdf
and
s3://lp-prod-protected/MOD13Q1.061/MOD13Q1.A2022177.h02v10.061.2022195120340/MOD13Q1.A2022177.h02v10.061.2022195120340.hdf"

Is this expected? I still see new files at LP DAAC at https://e4ftl01.cr.usgs.gov/MOLT/MOD13Q1.061

Thanks,
Simon

Tags:

LP DAAC - dgolon
User Services
User Services
Posts: 68
Joined: Mon Sep 30, 2019 10:00 am America/New_York
Answers: 0
Has thanked: 3 times

Re: CMR has no entries for MOD13Q1.061 files at LP DAAC

by LP DAAC - dgolon » Fri Oct 28, 2022 10:57 am America/New_York

Hi Simon,

I spoke with the Science Data Lifecycle Lead. He mentioned this is related to the cloud migration efforts. As we flip the ACL's to the new cloud version the concept IDs will change. Please update your scripts to reflect the new concept IDs. Currently we have some data that are both in the LP DAAC's Data Pool and on the cloud. Earthdata Search and CMR are both pulling cloud available data from the cloud, thus you are seeing the concept id for the cloud collection, instead of the on-prem Data Pool collection. The on-prem collection for data that are available in the cloud is not available in Earthdata Search or CMR anymore. We are currently working on migrating all LP DAAC data to the cloud, so you will see this with our other datasets in Earthdata Search and CMR as well, as they complete the migration.

Thanks,
LP DAAC
Subscribe to the LP DAAC listserv by sending a blank email to lpdaac-join@lists.nasa.gov.

Sign up for the Landsat listserv to receive the most up to date information about Landsat data: https://public.govdelivery.com/accounts/USDOIGS/subscriber/new#tab1.

zhan.li
Posts: 2
Joined: Tue Jun 28, 2022 8:01 am America/New_York
Answers: 0

Re: CMR has no entries for MOD13Q1.061 files at LP DAAC

by zhan.li » Mon Nov 07, 2022 5:00 pm America/New_York

Hi @LP DAAC - dgolon,

I'm still facing the same problem today. Multiple V061 data products of MODIS are still missing from the CMR, such as MYD11A1, MOD11A1 and MCD43A3, https://cmr.earthdata.nasa.gov/search/site/collections/directory/LPDAAC_ECS/gov.nasa.eosdis

You mention that this is due to the data migration to the cloud storage. However, the NASA CMR Cloud STAC endpoint currently is not returning these collections of data products (or any collection at all...) https://cmr.earthdata.nasa.gov/cloudstac

Can you give some pointer on how to programmatically query MODIS V061 products at the moment? When do we expect to see the MODIS V061 products appear on the CMR list?

Thanks,

Zhan.

LP DAAC - dgolon
User Services
User Services
Posts: 68
Joined: Mon Sep 30, 2019 10:00 am America/New_York
Answers: 0
Has thanked: 3 times

Re: CMR has no entries for MOD13Q1.061 files at LP DAAC

by LP DAAC - dgolon » Tue Nov 08, 2022 9:07 am America/New_York

Hi @zhan.li We will look into this and get back to you. Thanks!
Subscribe to the LP DAAC listserv by sending a blank email to lpdaac-join@lists.nasa.gov.

Sign up for the Landsat listserv to receive the most up to date information about Landsat data: https://public.govdelivery.com/accounts/USDOIGS/subscriber/new#tab1.

LP DAAC - afriesz
Subject Matter Expert
Subject Matter Expert
Posts: 33
Joined: Tue Nov 12, 2019 4:02 pm America/New_York
Answers: 0

Re: CMR has no entries for MOD13Q1.061 files at LP DAAC

by LP DAAC - afriesz » Tue Nov 08, 2022 12:53 pm America/New_York

@zhan.li & @earthengine_urs ,

As was mentioned before, the LP DAAC is in a state where on-prem data is actively being moved to the NASA Earthdata Cloud. When those on-prem (i.e., provider=LPDAAC_ECS) collections are moved to the cloud (i.e., provider=LPCLOUD) they receive a new concept ID, AND the on-prem collections are made unavailable. We don’t have a mechanism in place to let users know when those collections are now available in the cloud. We apologize for the inconvenience.

Regarding work arounds for the issues you’ve described here…I recommend a change to the CMR API search query you are using. I would recommend searching against both the “LPCLOUD” and “LPDAAC_ECS” providers. I would also query for a collection using the “short_name” parameter and “version”, rather than the “collection_concept_id” parameter. Below are example request queries:

Collection Search:
https://cmr.earthdata.nasa.gov/search/collections.json?provider=LPCLOUD&provider=LPDAAC_ECS&short_name=MYD11A1&version=061

Granules Search:
https://cmr.earthdata.nasa.gov/search/granules.json?provider=LPCLOUD&temporal=2022-06-26T00%3A00%3A00Z,2022-06-27T23%3A59%3A59Z&provider=LPDAAC_ECS&short_name=MYD11A1&version=061

earthengine_urs
Posts: 48
Joined: Mon Jan 27, 2020 10:36 am America/New_York
Answers: 0
Been thanked: 1 time

Re: CMR has no entries for MOD13Q1.061 files at LP DAAC

by earthengine_urs » Tue Nov 08, 2022 8:50 pm America/New_York

Thank you for the response. Can all datasets in CMR be addressed by a short name + version query? I suppose that would work for me too, thought it's nice to deal with a single concept id string.

LP DAAC - dgolon
User Services
User Services
Posts: 68
Joined: Mon Sep 30, 2019 10:00 am America/New_York
Answers: 0
Has thanked: 3 times

Re: CMR has no entries for MOD13Q1.061 files at LP DAAC

by LP DAAC - dgolon » Wed Nov 09, 2022 9:34 am America/New_York

@earthengine_urs All LP DAAC datasets in CMR can be addressed by a short name + version query, I believe that would be the same case for all non-lp related datasets in CMR too.
Subscribe to the LP DAAC listserv by sending a blank email to lpdaac-join@lists.nasa.gov.

Sign up for the Landsat listserv to receive the most up to date information about Landsat data: https://public.govdelivery.com/accounts/USDOIGS/subscriber/new#tab1.

Post Reply