curl download with incorrect credentials writes HTML content to HDF

Use this Forum to find information on, or ask a question about, NASA Earth Science data.
Post Reply
ei_mr_flo
Posts: 1
Joined: Mon Aug 29, 2022 3:45 am America/New_York
Answers: 0

curl download with incorrect credentials writes HTML content to HDF

by ei_mr_flo » Wed Jan 04, 2023 4:09 am America/New_York

Hi,

I am trying to download a MODIS .hdf file from LAADS using curl with deliberately incorrect credentials, meaning the below username and password are mere placeholders. Running the following command

curl \
-u "sad_boyd:Lc557Gv$" \
--fail \
-sS \
-L \
-O https://ladsweb.modaps.eosdis.nasa.gov/archive/allData/61/MCD64A1/2019/335/MCD64A1.A2019335.h32v11.061.2021309110404.hdf

creates a file MCD64A1.A2019335.h32v11.061.2021309110404.hdf in my current directory, but the file content is actually the "please login" HTML web page. By contrast, running the LP DAAC analog

curl \
-u "sad_boyd:Lc557Gv$" \
--fail \
-sS \
-L \
-O https://e4ftl01.cr.usgs.gov/MOTA/MCD64A1.061/2019.12.01/MCD64A1.A2019335.h32v11.061.2021309110404.hdf

raises an authentication error "curl: (22) The requested URL returned error: 401" and does not create any file, which I think is the more error-prone strategy.

I am not entirely certain, but didn't the above curl command for LAADS raise an error until a while ago, just like it still does for LP DAAC? Is it somehow possible to force curl to raise an error instead of downloading the HTML web content from LAADS when authentication fails? For the record, the same happens when using wget instead of curl.

Tags:

yuvipanda
Posts: 1
Joined: Wed Jan 04, 2023 6:32 pm America/New_York
Answers: 0

Re: curl download with incorrect credentials writes HTML content to HDF

by yuvipanda » Wed Jan 04, 2023 6:34 pm America/New_York

Adding a `-v` flag to see what happens, it looks like the first URL returns a '200 OK' status with the error HTML, rather than the correct '401' response code. I don't think you can have curl fail easily here, as the server is returning the wrong code.

You can use the `file` command on the downloaded file to see if it really is hdf5. If it's HTML, file will return something like `MCD64A1.A2019335.h32v11.061.2021309110404.hdf: HTML document text, ASCII text, with very long lines (460)` instead

gnwiii
Posts: 713
Joined: Fri Jan 29, 2021 5:51 pm America/New_York
Answers: 2
Has thanked: 1 time

Re: curl download with incorrect credentials writes HTML content to HDF

by gnwiii » Thu Jan 05, 2023 8:34 pm America/New_York

You can use the `file` command on the downloaded file to see if it really is hdf5
Or you can use

Code: Select all

wget —adjust-extension
to have “.html” appended to the file name if the content is html.

Post Reply