Subsetting MUR-JPL-L4-GLOB-v4.1 data?

Use this Forum to find information on, or ask a question about, NASA Earth Science data.
negri
Posts: 23
Joined: Sat Jul 10, 2021 3:42 pm America/New_York
Answers: 0

Re: Subsetting MUR-JPL-L4-GLOB-v4.1 data?

by negri » Tue Nov 21, 2023 7:45 pm America/New_York

Thanks for the last information. It worked.
but know, I have another issue.

In order to contextualize, I'm following this post (https://github.com/podaac/tutorials/blob/master/notebooks/opendap/MUR-OPeNDAP.ipynb)

1) my request is OK --> manager.add_password(None, endpoint, username, password)
r = requests.get('https://cmr.earthdata.nasa.gov/search/granules.umm_json?collection_concept_id=C1996881146-POCLOUD&temporal=2019-01-01T10:00:00Z,2019-01-05T00:00:00Z&pageSize=365')
r.status_code
Out[140]: <Response [200]>

2) Next, I got the opendap product...
'https://opendap.earthdata.nasa.gov/providers/POCLOUD/collections/GHRSST%20Level%204%20MUR%20Global%20Foundation%20Sea%20Surface%20Temperature%20Analysis%20(v4.1)/granules/20190101090000-JPL-L4_GHRSST-SSTfnd-MUR-GLOB-v02.0-fv04.1'

3) and... I added the extension '.dap.nc4?dap4'
'https://opendap.earthdata.nasa.gov/providers/POCLOUD/collections/GHRSST%20Level%204%20MUR%20Global%20Foundation%20Sea%20Surface%20Temperature%20Analysis%20(v4.1)/granules/20190101090000-JPL-L4_GHRSST-SSTfnd-MUR-GLOB-v02.0-fv04.1.dap.nc4?dap4'

4) next...
required_variables = {'analysed_sst[0:1:0][000:1:9000][000:1:9000]',
'analysis_error[0:1:0][000:1:9000][000:1:9000]',
'lat[000:1:9000]',
'lon[000:1:9000]',
'time[0:1:0]'}

5) and finally (here I have the issue)

basename = os.path.basename(data_url)
request_params = {'dap4.ce': ';'.join(required_variables)}
response = requests.get(data_url, params=request_params, headers={'Accept-Encoding': 'identity'})

response
<Response [401]>
response.text
Out[154]: 'HTTP Basic: Access denied.\n'

Please, some sugestion ?

Filters:

PODAAC - jmcnelis
Subject Matter Expert
Subject Matter Expert
Posts: 16
Joined: Tue Mar 14, 2023 1:41 pm America/New_York
Answers: 0

Re: Subsetting MUR-JPL-L4-GLOB-v4.1 data?

by PODAAC - jmcnelis » Tue Feb 13, 2024 5:31 pm America/New_York

ran252_unsw wrote: Tue Nov 21, 2023 2:32 am America/New_York Yes, First stage I used the following command
pip install podaac-data-subscriber. But it doest work to crop a domain.

Then I used
pip install podaac-data-subscriber==1.15.0a3

and trying to crop data as follows:
podaac-data-downloader -c MUR-JPL-L4-GLOB-v4.1 -d ./data --start-date 2003-01-01T09:00:00Z --end-date 2022-12-31T09:00:00Z
-b="135,-50,175,-10" --subset

Still, the global data is downloading but not getting the cropping domain as specified.

Thanks for your further instructions.
Hi @ran252_unsw,

You need to give the --subset flag before the -b option, like this:

"""
podaac-data-downloader -c MUR-JPL-L4-GLOB-v4.1 -d ./data --start-date 2003-01-01T09:00:00Z --end-date 2022-12-31T09:00:00Z --subset -b="135,-50,175,-10"
"""

I recommend setting smaller timeframes for your subset orders, and downloading them in batches. The podaac-data-downloader will need to wait for an enormous number of subsetting operations to complete when requesting 20 years of data.

Jack

xcancel
Posts: 1
Joined: Wed Apr 24, 2024 9:53 pm America/New_York
Answers: 0

Re: Subsetting MUR-JPL-L4-GLOB-v4.1 data?

by xcancel » Wed Apr 24, 2024 10:05 pm America/New_York

Apparently PODAAC Downloader through Harmony subsetting is broken for older GHRSST MUR data. I also had problems when trying to run the mentioned script for downloading 2002/2003 MUR files. The example provided by Wenhao Li using files from 2022 works fine, but if we try to apply it for 2002 files (for example), Harmony subsetting brokes and we receive an "Error processing harmony subsetting request".

Initially I thought it was a problem caused by trying to download multiple files at once, but even subsetting one by one is impossible. I recommend the PODAAC team to check why this is happening.

Edit: I just tried to subset download through wget using OPeNDAP granules and is working fine. Not sure why PODAAC Downloader can't access these older GHRSST MUR data.

Thank you very much.
Last edited by xcancel on Thu Apr 25, 2024 2:19 am America/New_York, edited 1 time in total.

PODAAC - wenhaoli
Subject Matter Expert
Subject Matter Expert
Posts: 35
Joined: Tue May 11, 2021 12:58 pm America/New_York
Answers: 0

Re: Subsetting MUR-JPL-L4-GLOB-v4.1 data?

by PODAAC - wenhaoli » Thu Apr 25, 2024 2:48 pm America/New_York

Thank you for finding this issue. We will investigate and let you know once we have the answer.

PODAAC - wenhaoli
Subject Matter Expert
Subject Matter Expert
Posts: 35
Joined: Tue May 11, 2021 12:58 pm America/New_York
Answers: 0

Re: Subsetting MUR-JPL-L4-GLOB-v4.1 data?

by PODAAC - wenhaoli » Mon Apr 29, 2024 4:54 pm America/New_York

Hi,

PO.DAAC team has discovered the issue, which is related with old OPeNDAP dmrpp files. Since late of 2023, the most recent MUR granules has been updated with the OPeNDAP dmrpp version 3.21.0-40, which has resolved this issue. So, we have to update dmrpp files for all MUR granules. Most likely, it will take about two week for the updating. Sorry for the inconvenience.

Thanks,

Wenhao

PODAAC - celiaoued
Subject Matter Expert
Subject Matter Expert
Posts: 19
Joined: Fri May 28, 2021 1:30 pm America/New_York
Answers: 0

Re: Subsetting MUR-JPL-L4-GLOB-v4.1 data?

by PODAAC - celiaoued » Tue Apr 30, 2024 7:41 pm America/New_York

Hello,

OPeNDAP should work now for the older MUR files. Before trying again, you should manually delete a hidden .harmony file in the download folder you were using for the Data Subscriber command:

rm ./data/.harmony

xcancel
Posts: 1
Joined: Wed Apr 24, 2024 9:53 pm America/New_York
Answers: 0

Re: Subsetting MUR-JPL-L4-GLOB-v4.1 data?

by xcancel » Fri May 17, 2024 3:23 am America/New_York

Dear PODAAC Team,

Thank you very much for your support. The PODAAC Downloader seems to be working fine for older GHRSST MUR files now.
However, can you please check the following granules?

20210220090000-JPL-L4_GHRSST-SSTfnd-MUR-GLOB-v02.0-fv04.1 (2021-02-20)
20210221090000-JPL-L4_GHRSST-SSTfnd-MUR-GLOB-v02.0-fv04.1 (2021-02-21)
20221109090000-JPL-L4_GHRSST-SSTfnd-MUR-GLOB-v02.0-fv04.1 (2022-11-09)
20230317090000-JPL-L4_GHRSST-SSTfnd-MUR-GLOB-v02.0-fv04.1 (2023-03-17)

I've been trying to download them, but both wget and PODAAC Downloader method returns me a 0 bytes file or no file at all.

Best,
Gabriel
Last edited by xcancel on Fri May 17, 2024 3:24 am America/New_York, edited 1 time in total.

PODAAC - edward.m.armstrong
Subject Matter Expert
Subject Matter Expert
Posts: 8
Joined: Thu May 27, 2021 2:52 pm America/New_York
Answers: 0

Re: Subsetting MUR-JPL-L4-GLOB-v4.1 data?

by PODAAC - edward.m.armstrong » Mon May 20, 2024 4:51 pm America/New_York

Hi Gabriel,

I did confirm we have missing granules for Feb 20-21, 2021. We will try to recreate those. Investigating the other dates.

regards,
Ed

PODAAC - edward.m.armstrong
Subject Matter Expert
Subject Matter Expert
Posts: 8
Joined: Thu May 27, 2021 2:52 pm America/New_York
Answers: 0

Re: Subsetting MUR-JPL-L4-GLOB-v4.1 data?

by PODAAC - edward.m.armstrong » Mon May 20, 2024 4:58 pm America/New_York

Also this granule from 2023 has no OPeNDAP link which is the service used for subsetting, so it will not work with the downloader. Investigating.

https://cmr.earthdata.nasa.gov/virtual-directory/collections/C1996881146-POCLOUD/temporal/2023/03/16

Post Reply