LAADS DAAC MODIS downloads reporting 500.

Use this Forum to find information on, or ask a question about, NASA Earth Science data.
Post Reply
sharkins
Posts: 17
Joined: Wed Jun 16, 2021 1:31 pm America/New_York
Answers: 0

LAADS DAAC MODIS downloads reporting 500.

by sharkins » Sat Jun 19, 2021 4:04 pm America/New_York

After several days of intermittent instability all LAADS DAAC MODIS downloads are now reporting 500. Using the provided wget command

wget -e robots=off -m -np -R .html,.tmp -nH --cut-dirs=3 "https://ladsweb.modaps.eosdis.nasa.gov/ ... /2021/167/" --header "Authorization: Bearer redacted" -P .

Reports

Resolving urs.earthdata.nasa.gov (urs.earthdata.nasa.gov)... 198.118.243.33
Connecting to urs.earthdata.nasa.gov (urs.earthdata.nasa.gov)|198.118.243.33|:443... connected.
HTTP request sent, awaiting response... 500 Internal Server Error
2021-06-19 14:57:39 ERROR 500: Internal Server Error.

I have not seen any response to my related question from earlier this week viewtopic.php?f=7&t=2389&sid=f5566a36b3 ... 1ca4a5ec53. Is there a direct contact at LAADS DAAC who would be best to contact about these ongoing issues?

Tags:

ASDC - ingridgs
Subject Matter Expert
Subject Matter Expert
Posts: 142
Joined: Fri Apr 23, 2021 9:14 am America/New_York
Answers: 1
Has thanked: 17 times
Been thanked: 7 times

Re: LAADS DAAC MODIS downloads reporting 500.

by ASDC - ingridgs » Sun Jun 20, 2021 3:09 pm America/New_York

Hello!
A Subject Matter expert has been notified and will answer your question shortly.
Please stand by.

LAADS_UserServices_M
User Services
User Services
Posts: 279
Joined: Mon Sep 30, 2019 8:33 am America/New_York
Answers: 1
Has thanked: 3 times

Re: LAADS DAAC MODIS downloads reporting 500.

by LAADS_UserServices_M » Mon Jun 21, 2021 9:57 am America/New_York

I tried your wget command with my token and it worked without any issue (see below). Please try again after generating a new token from ladsweb (https://ladsweb.modaps.eosdis.nasa.gov/).

wget -e robots=off -m -np -R .html,.tmp -nH --cut-dirs=3 "https://ladsweb.modaps.eosdis.nasa.gov/ ... /2021/167/" --header "Authorization: Bearer YXNhZHVsbGFo.........TBlY2VhM2UzZjE3N2Rl" -P C:\temp
--2021-06-21 09:46:59-- https://ladsweb.modaps.eosdis.nasa.gov/ ... /2021/167/
Resolving ladsweb.modaps.eosdis.nasa.gov (ladsweb.modaps.eosdis.nasa.gov)... 198.118.194.40
Connecting to ladsweb.modaps.eosdis.nasa.gov (ladsweb.modaps.eosdis.nasa.gov)|198.118.194.40|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: 'C:/temp/MYD09CMA/2021/167/index.html.tmp'

MYD09CMA/2021/167/index.html. [ <=> ] 32.38K --.-KB/s in 0.005s

Last-modified header missing -- time-stamps turned off.
2021-06-21 09:47:03 (6.37 MB/s) - 'C:/temp/MYD09CMA/2021/167/index.html.tmp' saved [33156]

Removing C:/temp/MYD09CMA/2021/167/index.html.tmp since it should be rejected.

--2021-06-21 09:47:03-- https://ladsweb.modaps.eosdis.nasa.gov/ ... 034349.hdf
Reusing existing connection to ladsweb.modaps.eosdis.nasa.gov:443.
HTTP request sent, awaiting response... 200 OK
Length: 387071146 (369M) [application/x-hdf]
Saving to: 'C:/temp/MYD09CMA/2021/167/MYD09CMA.A2021167.006.2021169034349.hdf'
Regards,
LAADS User Services

To receive news from LAADS DAAC direct to your inbox, email laadsdaac-join@lists.nasa.gov with “subscribe” in the subject line.

sharkins
Posts: 17
Joined: Wed Jun 16, 2021 1:31 pm America/New_York
Answers: 0

Re: LAADS DAAC MODIS downloads reporting 500.

by sharkins » Mon Jun 21, 2021 10:59 am America/New_York

Generating a new token seemed to work. Is there a reason why the previous token (which was generated approx 1 week ago) suddenly began failing? We use these persistent tokens in our production system which is dependent on daily Aqua and Terra downloads, is there a new token expiration strategy which is not documented? How can we debug why using the previous token to download was failing?

LAADS_UserServices_M
User Services
User Services
Posts: 279
Joined: Mon Sep 30, 2019 8:33 am America/New_York
Answers: 1
Has thanked: 3 times

Re: LAADS DAAC MODIS downloads reporting 500.

by LAADS_UserServices_M » Tue Jun 22, 2021 2:01 pm America/New_York

There is no expiration for tokens. However, LAADS infrastructure team implemented a cross-site token compatibility last night that might have caused it. The cross-site token compatibility makes it possible for token generated on one LAADS site to work on all sites.
Regards,
LAADS User Services

To receive news from LAADS DAAC direct to your inbox, email laadsdaac-join@lists.nasa.gov with “subscribe” in the subject line.

sharkins
Posts: 17
Joined: Wed Jun 16, 2021 1:31 pm America/New_York
Answers: 0

Re: LAADS DAAC MODIS downloads reporting 500.

by sharkins » Tue Jun 22, 2021 2:37 pm America/New_York

Can you provide a specific datetime when this system configuration change was made? There have been several reports of ongoing download issues as far back as June 1 including viewtopic.php?f=7&t=2386&sid=1fae28c0fb ... 89b997671d viewtopic.php?f=7&t=2389&sid=6f4bc9ecc2 ... fa7f2bee6a and potentially viewtopic.php?f=7&t=2396&sid=85da4953bf ... 10fc2e48e9. All of these issues seem related to some ongoing configuration changes. If there is a listing of recent system updates that can be made publicly available it might assist in diagnosing these issues. Thanks.

LAADS_UserServices_M
User Services
User Services
Posts: 279
Joined: Mon Sep 30, 2019 8:33 am America/New_York
Answers: 1
Has thanked: 3 times

Re: LAADS DAAC MODIS downloads reporting 500.

by LAADS_UserServices_M » Wed Jun 23, 2021 4:11 pm America/New_York

The latest change was made between 6/21 and 6/22, however old style app key authentication was decommissioned on June 1. Please refer to:
https://ladsweb.modaps.eosdis.nasa.gov/ ... /?id=63313
Regards,
LAADS User Services

To receive news from LAADS DAAC direct to your inbox, email laadsdaac-join@lists.nasa.gov with “subscribe” in the subject line.

Post Reply