LAADS DAAC MODIS downloads reporting 500.

Please enter here to ask a question about any NASA Science related topics!
Post Reply
sharkins
Posts: 4
Joined: Wed Jun 16, 2021 1:31 pm America/New_York

LAADS DAAC MODIS downloads reporting 500.

by sharkins » Sat Jun 19, 2021 4:04 pm America/New_York

After several days of intermittent instability all LAADS DAAC MODIS downloads are now reporting 500. Using the provided wget command

wget -e robots=off -m -np -R .html,.tmp -nH --cut-dirs=3 "https://ladsweb.modaps.eosdis.nasa.gov/ ... /2021/167/" --header "Authorization: Bearer redacted" -P .

Reports

Resolving urs.earthdata.nasa.gov (urs.earthdata.nasa.gov)... 198.118.243.33
Connecting to urs.earthdata.nasa.gov (urs.earthdata.nasa.gov)|198.118.243.33|:443... connected.
HTTP request sent, awaiting response... 500 Internal Server Error
2021-06-19 14:57:39 ERROR 500: Internal Server Error.

I have not seen any response to my related question from earlier this week viewtopic.php?f=7&t=2389&sid=f5566a36b3 ... 1ca4a5ec53. Is there a direct contact at LAADS DAAC who would be best to contact about these ongoing issues?

Tags:

ASDC - ingridgs
User Services
User Services
Posts: 13
Joined: Fri Apr 23, 2021 9:14 am America/New_York
Answers: 1

Re: LAADS DAAC MODIS downloads reporting 500.

by ASDC - ingridgs » Sun Jun 20, 2021 3:09 pm America/New_York

Hello!
A Subject Matter expert has been notified and will answer your question shortly.
Please stand by.

LAADS_UserServices_M
User Services
User Services
Posts: 29
Joined: Mon Sep 30, 2019 8:33 am America/New_York

Re: LAADS DAAC MODIS downloads reporting 500.

by LAADS_UserServices_M » Mon Jun 21, 2021 9:57 am America/New_York

I tried your wget command with my token and it worked without any issue (see below). Please try again after generating a new token from ladsweb (https://ladsweb.modaps.eosdis.nasa.gov/).

wget -e robots=off -m -np -R .html,.tmp -nH --cut-dirs=3 "https://ladsweb.modaps.eosdis.nasa.gov/ ... /2021/167/" --header "Authorization: Bearer YXNhZHVsbGFo.........TBlY2VhM2UzZjE3N2Rl" -P C:\temp
--2021-06-21 09:46:59-- https://ladsweb.modaps.eosdis.nasa.gov/ ... /2021/167/
Resolving ladsweb.modaps.eosdis.nasa.gov (ladsweb.modaps.eosdis.nasa.gov)... 198.118.194.40
Connecting to ladsweb.modaps.eosdis.nasa.gov (ladsweb.modaps.eosdis.nasa.gov)|198.118.194.40|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: 'C:/temp/MYD09CMA/2021/167/index.html.tmp'

MYD09CMA/2021/167/index.html. [ <=> ] 32.38K --.-KB/s in 0.005s

Last-modified header missing -- time-stamps turned off.
2021-06-21 09:47:03 (6.37 MB/s) - 'C:/temp/MYD09CMA/2021/167/index.html.tmp' saved [33156]

Removing C:/temp/MYD09CMA/2021/167/index.html.tmp since it should be rejected.

--2021-06-21 09:47:03-- https://ladsweb.modaps.eosdis.nasa.gov/ ... 034349.hdf
Reusing existing connection to ladsweb.modaps.eosdis.nasa.gov:443.
HTTP request sent, awaiting response... 200 OK
Length: 387071146 (369M) [application/x-hdf]
Saving to: 'C:/temp/MYD09CMA/2021/167/MYD09CMA.A2021167.006.2021169034349.hdf'

sharkins
Posts: 4
Joined: Wed Jun 16, 2021 1:31 pm America/New_York

Re: LAADS DAAC MODIS downloads reporting 500.

by sharkins » Mon Jun 21, 2021 10:59 am America/New_York

Generating a new token seemed to work. Is there a reason why the previous token (which was generated approx 1 week ago) suddenly began failing? We use these persistent tokens in our production system which is dependent on daily Aqua and Terra downloads, is there a new token expiration strategy which is not documented? How can we debug why using the previous token to download was failing?

LAADS_UserServices_M
User Services
User Services
Posts: 29
Joined: Mon Sep 30, 2019 8:33 am America/New_York

Re: LAADS DAAC MODIS downloads reporting 500.

by LAADS_UserServices_M » Tue Jun 22, 2021 2:01 pm America/New_York

There is no expiration for tokens. However, LAADS infrastructure team implemented a cross-site token compatibility last night that might have caused it. The cross-site token compatibility makes it possible for token generated on one LAADS site to work on all sites.

sharkins
Posts: 4
Joined: Wed Jun 16, 2021 1:31 pm America/New_York

Re: LAADS DAAC MODIS downloads reporting 500.

by sharkins » Tue Jun 22, 2021 2:37 pm America/New_York

Can you provide a specific datetime when this system configuration change was made? There have been several reports of ongoing download issues as far back as June 1 including viewtopic.php?f=7&t=2386&sid=1fae28c0fb ... 89b997671d viewtopic.php?f=7&t=2389&sid=6f4bc9ecc2 ... fa7f2bee6a and potentially viewtopic.php?f=7&t=2396&sid=85da4953bf ... 10fc2e48e9. All of these issues seem related to some ongoing configuration changes. If there is a listing of recent system updates that can be made publicly available it might assist in diagnosing these issues. Thanks.

LAADS_UserServices_M
User Services
User Services
Posts: 29
Joined: Mon Sep 30, 2019 8:33 am America/New_York

Re: LAADS DAAC MODIS downloads reporting 500.

by LAADS_UserServices_M » Wed Jun 23, 2021 4:11 pm America/New_York

The latest change was made between 6/21 and 6/22, however old style app key authentication was decommissioned on June 1. Please refer to:
https://ladsweb.modaps.eosdis.nasa.gov/ ... /?id=63313

Post Reply