I'm trying to download MOD13Q1.061 data for several tiles from the LPDAAC opendap server. I'm getting a lot of 400 errors trying to adapt the opendap quick start guide (https://opendap.github.io/documentation/QuickStart.html) to my situation.
I can access the html forms at the data URLs (for example, tile h29v13): https://opendap.cr.usgs.gov/opendap/hyrax/MOD13Q1.061/h29v13.ncml
I can then access the DDS for those data by appending ".dds" to the URL. So far, so good: https://opendap.cr.usgs.gov/opendap/hyrax/MOD13Q1.061/h29v13.ncml.dds
Everything else I've tried gives me a 400 error. For example, trying to access a subset of time values by appending ".ascii" to the URL and applying a constraint: https://opendap.cr.usgs.gov/opendap/hyrax/MOD13Q1.061/h29v13.ncml.ascii?time[2:1:10]
Trying to apply a constraint by obtaining a data URL with constraints from the webform similarly produces 400 errors unless I use the GUI buttons to obtain the data. I need a few tens of files, so I'd like to figure out how to form the requests in a program.
Any help much appreciated.
LPdaac opendap server 400 errors for MOD13Q1.061 data
-
timothy.w.hilton
- Posts: 8
- Joined: Sun Dec 10, 2023 9:59 pm America/New_York
-
LP DAACx - dgolon
- User Services
- Posts: 420
- Joined: Mon Sep 30, 2019 10:00 am America/New_York
- Has thanked: 30 times
- Been thanked: 8 times
- Contact:
Re: LPdaac opendap server 400 errors for MOD13Q1.061 data
Hi @timothy.w.hilton Apologies for the delay in this response. Our developers are looking into your question. We'll respond back when we have additional information. Thanks -- Danielle
Subscribe to the LP DAAC listserv by sending a blank email to lpdaac-join@lists.nasa.gov.
Sign up for the Landsat listserv to receive the most up to date information about Landsat data: https://public.govdelivery.com/accounts/USDOIGS/subscriber/new#tab1.
Sign up for the Landsat listserv to receive the most up to date information about Landsat data: https://public.govdelivery.com/accounts/USDOIGS/subscriber/new#tab1.
-
LP DAAC - afriesz
- Subject Matter Expert
- Posts: 71
- Joined: Tue Nov 12, 2019 4:02 pm America/New_York
- Been thanked: 3 times
Re: LPdaac opendap server 400 errors for MOD13Q1.061 data
@timothy.w.hilton,
When I paste the URL you provided directly into the browser, I get a 400 error as well. This is because the characters in the parameters you are passing (everything after the ?) need to be encoded. Try this URL > https://opendap.cr.usgs.gov/opendap/hyrax/MOD13Q1.061/h29v13.ncml.ascii?time%5B2:1:10%5D
Note that the brackets around the time start:step:end are now using HTML URL encoding.
If you were using a client of some sort, this would be done for you. I'm assuming you are pasting the URL directly into the browser.
Another think to note with the OPeNDAP endpoint you are hitting. The endpoint provides access to the full timeseries from one endpoint, but you are not able to request the full timeseries in a single request. If you request the full timeseries you will receive an error resulting from a timeout that occurs when you make such a request. We will need to consider submitting multiple requests for the timeseries. I've had some success with about 100 time steps within a request, but you'll need to play around with that to see what consistently works for your workflow.
When I paste the URL you provided directly into the browser, I get a 400 error as well. This is because the characters in the parameters you are passing (everything after the ?) need to be encoded. Try this URL > https://opendap.cr.usgs.gov/opendap/hyrax/MOD13Q1.061/h29v13.ncml.ascii?time%5B2:1:10%5D
Note that the brackets around the time start:step:end are now using HTML URL encoding.
If you were using a client of some sort, this would be done for you. I'm assuming you are pasting the URL directly into the browser.
Another think to note with the OPeNDAP endpoint you are hitting. The endpoint provides access to the full timeseries from one endpoint, but you are not able to request the full timeseries in a single request. If you request the full timeseries you will receive an error resulting from a timeout that occurs when you make such a request. We will need to consider submitting multiple requests for the timeseries. I've had some success with about 100 time steps within a request, but you'll need to play around with that to see what consistently works for your workflow.
-
timothy.w.hilton
- Posts: 8
- Joined: Sun Dec 10, 2023 9:59 pm America/New_York
Re: LPdaac opendap server 400 errors for MOD13Q1.061 data
LP DAAC - afriesz wrote:
> @timothy.w.hilton,
>
> When I paste the URL you provided directly into the browser, I get a 400 error as
> well. This is because the characters in the parameters you are passing (everything
> after the ?) need to be encoded. Try this URL >
> https://opendap.cr.usgs.gov/opendap/hyrax/MOD13Q1.061/h29v13.ncml.ascii?time%5B2:1:10%5D
>
> Note that the brackets around the time start:step:end are now using HTML URL
> encoding.
OK, this sheds some light on it. Many thanks for this piece of information.
> If you were using a client of some sort, this would be done for you. I'm assuming you
> are pasting the URL directly into the browser.
I tried both wget (`wget --user timothy.w.hilton --ask-password https://opendap.cr.usgs.gov/opendap/hyrax/MOD13Q1.061/h29v13.ncml.ascii?time[2:1:10]`) as well as pasting into a browser, and got 400 from both methods. I notice now that the URL with the brackets works using the Python requests module; it must substitute the characters internally.
> Another think to note with the OPeNDAP endpoint you are hitting. The endpoint
> provides access to the full timeseries from one endpoint, but you are not able to
> request the full timeseries in a single request. If you request the full timeseries
> you will receive an error resulting from a timeout that occurs when you make such a
> request. We will need to consider submitting multiple requests for the timeseries.
> I've had some success with about 100 time steps within a request, but you'll need to
> play around with that to see what consistently works for your workflow.
Noted, thanks. I'm breaking up the files for the actual data downloads. The URL above was simply requesting the timestamps so I could figure out which files I needed.
> @timothy.w.hilton,
>
> When I paste the URL you provided directly into the browser, I get a 400 error as
> well. This is because the characters in the parameters you are passing (everything
> after the ?) need to be encoded. Try this URL >
> https://opendap.cr.usgs.gov/opendap/hyrax/MOD13Q1.061/h29v13.ncml.ascii?time%5B2:1:10%5D
>
> Note that the brackets around the time start:step:end are now using HTML URL
> encoding.
OK, this sheds some light on it. Many thanks for this piece of information.
> If you were using a client of some sort, this would be done for you. I'm assuming you
> are pasting the URL directly into the browser.
I tried both wget (`wget --user timothy.w.hilton --ask-password https://opendap.cr.usgs.gov/opendap/hyrax/MOD13Q1.061/h29v13.ncml.ascii?time[2:1:10]`) as well as pasting into a browser, and got 400 from both methods. I notice now that the URL with the brackets works using the Python requests module; it must substitute the characters internally.
> Another think to note with the OPeNDAP endpoint you are hitting. The endpoint
> provides access to the full timeseries from one endpoint, but you are not able to
> request the full timeseries in a single request. If you request the full timeseries
> you will receive an error resulting from a timeout that occurs when you make such a
> request. We will need to consider submitting multiple requests for the timeseries.
> I've had some success with about 100 time steps within a request, but you'll need to
> play around with that to see what consistently works for your workflow.
Noted, thanks. I'm breaking up the files for the actual data downloads. The URL above was simply requesting the timestamps so I could figure out which files I needed.