LPdaac opendap server 400 errors for MOD13Q1.061 data

Use this Forum to find information on, or ask a question about, NASA Earth Science data.
Post Reply
timothy.w.hilton
Posts: 6
Joined: Sun Dec 10, 2023 9:59 pm America/New_York
Answers: 0

LPdaac opendap server 400 errors for MOD13Q1.061 data

by timothy.w.hilton » Sun Dec 10, 2023 10:26 pm America/New_York

I'm trying to download MOD13Q1.061 data for several tiles from the LPDAAC opendap server. I'm getting a lot of 400 errors trying to adapt the opendap quick start guide (https://opendap.github.io/documentation/QuickStart.html) to my situation.

I can access the html forms at the data URLs (for example, tile h29v13): https://opendap.cr.usgs.gov/opendap/hyrax/MOD13Q1.061/h29v13.ncml

I can then access the DDS for those data by appending ".dds" to the URL. So far, so good: https://opendap.cr.usgs.gov/opendap/hyrax/MOD13Q1.061/h29v13.ncml.dds

Everything else I've tried gives me a 400 error. For example, trying to access a subset of time values by appending ".ascii" to the URL and applying a constraint: https://opendap.cr.usgs.gov/opendap/hyrax/MOD13Q1.061/h29v13.ncml.ascii?time[2:1:10]

Trying to apply a constraint by obtaining a data URL with constraints from the webform similarly produces 400 errors unless I use the GUI buttons to obtain the data. I need a few tens of files, so I'd like to figure out how to form the requests in a program.

Any help much appreciated.

Tags:

LP DAAC - dgolon
User Services
User Services
Posts: 292
Joined: Mon Sep 30, 2019 10:00 am America/New_York
Answers: 0
Has thanked: 18 times
Been thanked: 2 times
Contact:

Re: LPdaac opendap server 400 errors for MOD13Q1.061 data

by LP DAAC - dgolon » Tue Dec 19, 2023 10:05 am America/New_York

Hi @timothy.w.hilton Apologies for the delay in this response. Our developers are looking into your question. We'll respond back when we have additional information. Thanks -- Danielle
Subscribe to the LP DAAC listserv by sending a blank email to lpdaac-join@lists.nasa.gov.

Sign up for the Landsat listserv to receive the most up to date information about Landsat data: https://public.govdelivery.com/accounts/USDOIGS/subscriber/new#tab1.

LP DAAC - afriesz
Subject Matter Expert
Subject Matter Expert
Posts: 59
Joined: Tue Nov 12, 2019 4:02 pm America/New_York
Answers: 2
Been thanked: 2 times

Re: LPdaac opendap server 400 errors for MOD13Q1.061 data

by LP DAAC - afriesz » Tue Dec 19, 2023 11:11 am America/New_York

@timothy.w.hilton,

When I paste the URL you provided directly into the browser, I get a 400 error as well. This is because the characters in the parameters you are passing (everything after the ?) need to be encoded. Try this URL > https://opendap.cr.usgs.gov/opendap/hyrax/MOD13Q1.061/h29v13.ncml.ascii?time%5B2:1:10%5D

Note that the brackets around the time start:step:end are now using HTML URL encoding.

If you were using a client of some sort, this would be done for you. I'm assuming you are pasting the URL directly into the browser.

Another think to note with the OPeNDAP endpoint you are hitting. The endpoint provides access to the full timeseries from one endpoint, but you are not able to request the full timeseries in a single request. If you request the full timeseries you will receive an error resulting from a timeout that occurs when you make such a request. We will need to consider submitting multiple requests for the timeseries. I've had some success with about 100 time steps within a request, but you'll need to play around with that to see what consistently works for your workflow.

timothy.w.hilton
Posts: 6
Joined: Sun Dec 10, 2023 9:59 pm America/New_York
Answers: 0

Re: LPdaac opendap server 400 errors for MOD13Q1.061 data

by timothy.w.hilton » Wed Dec 20, 2023 8:38 pm America/New_York

LP DAAC - afriesz wrote:
> @timothy.w.hilton,
>
> When I paste the URL you provided directly into the browser, I get a 400 error as
> well. This is because the characters in the parameters you are passing (everything
> after the ?) need to be encoded. Try this URL >
> https://opendap.cr.usgs.gov/opendap/hyrax/MOD13Q1.061/h29v13.ncml.ascii?time%5B2:1:10%5D
>
> Note that the brackets around the time start:step:end are now using HTML URL
> encoding.

OK, this sheds some light on it. Many thanks for this piece of information.

> If you were using a client of some sort, this would be done for you. I'm assuming you
> are pasting the URL directly into the browser.

I tried both wget (`wget --user timothy.w.hilton --ask-password https://opendap.cr.usgs.gov/opendap/hyrax/MOD13Q1.061/h29v13.ncml.ascii?time[2:1:10]`) as well as pasting into a browser, and got 400 from both methods. I notice now that the URL with the brackets works using the Python requests module; it must substitute the characters internally.

> Another think to note with the OPeNDAP endpoint you are hitting. The endpoint
> provides access to the full timeseries from one endpoint, but you are not able to
> request the full timeseries in a single request. If you request the full timeseries
> you will receive an error resulting from a timeout that occurs when you make such a
> request. We will need to consider submitting multiple requests for the timeseries.
> I've had some success with about 100 time steps within a request, but you'll need to
> play around with that to see what consistently works for your workflow.

Noted, thanks. I'm breaking up the files for the actual data downloads. The URL above was simply requesting the timestamps so I could figure out which files I needed.

Post Reply