MODIS download with token key

Please enter here to ask a question about any NASA Science related topics!
Post Reply
niglas_cas
Posts: 5
Joined: Sun Jul 11, 2021 1:26 pm America/New_York

MODIS download with token key

by niglas_cas » Sun Jul 11, 2021 1:38 pm America/New_York

Hi,

I am trying to download MODIS L1A data with a token key using python, but I received the HTML as following instead of MODIS data.
Could anyone help me?

Best,
Ming



<!DOCTYPE html>
<!--[if lt IE 7]><html class="no-js lt-ie9 lt-ie8 lt-ie7"> <![endif]-->
<!--[if IE 7]><html class="no-js lt-ie9 lt-ie8"> <![endif]-->
<!--[if IE 8]><html class="no-js lt-ie9"> <![endif]-->
<!--[if gt IE 8]><!--><html lang="en" class="no-js"><!--<![endif]-->
<head>
<meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1">
<title>Earthdata Login</title>
<meta name="description" content="Earthdata Login">
<meta name="viewport" content="width=device-width, initial-scale=1.0">

<!-- Google Tag Manager -->
<script>(function(w,d,s,l,i){w[l]=w[l]||[];w[l].push(

{'gtm.start': new Date().getTime(),event:'gtm.js'}

);var f=d.getElementsByTagName(s)[0],
j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src=
'https://www.googletagmanager.com/gtm.js ... efore(j,f);
})(window,document,'script','dataLayer','GTM-WNP7MLF');</script>
<!-- End Google Tag Manager -->

<link href="https://cdn.earthdata.nasa.gov/eui/1.1. ... cation.css" rel="stylesheet" />
<link rel="stylesheet" media="all" href="/assets/application-2f6d12daf5fc75b8f8a4872f4c11305dcc01a22100f2d40d5ffd9fc637ff789f.css" />
<!--[if IE 7]>
<link rel="stylesheet" href="/assets/font-awesome-ie7.min.css">
<![endif]-->
<link href="//netdna.bootstrapcdn.com/font-awesome/4.3.0/css/font-awesome.min.css" rel="stylesheet">
<link href='https://fonts.googleapis.com/css?family ... ro:300,700' rel='stylesheet' type='text/css'>
<meta name="csrf-param" content="authenticity_token" />
<meta name="csrf-token" content="E9ZvV6vWgJTBNLwJ27vye8cchxBKcKuBJtNw+a0kQbmgkbBCa0iFQCgzP1vEio3CSYZbZoW9iM6qz3dfJU/NUQ==" />


<!-- Grid background: http://subtlepatterns.com/graphy/ -->
</head>
<body class="oauth authorize" data-turbolinks-eval=false>

<!-- Google Tag Manager (noscript) -->
<noscript>
<iframe src="https://www.googletagmanager.com/ns.html?id=GTM-WNP7MLF"
height="0" width="0" style="display:none;visibility:hidden"></iframe>
</noscript>
<!-- End Google Tag Manager (noscript) -->

<header id="earthdata-tophat2" style="height: 32px;"></header>
<!--[if lt IE 7]>
<p class="chromeframe">You are using an <strong>outdated</strong> browser. Please <a href="http://browsehappy.com/">upgrade your browser</a> or <a href="http://www.google.com/chromeframe/?redi ... ">activate Google Chrome Frame</a> to improve your experience.</p>
<![endif]-->
<div class="container">
<header role="banner">
<div id="masthead-logo">
<h1><a class="ir" href="/">Earthdata Login</a></h1>
<span class="eui-badge badge daac">Earthdata Login</span>
</div>
<a id="hamburger" href="#"><img title="Mobile Menu" alt="Three horizontal lines stacked" src="/assets/hamburger-68c8505066427f3e3f6ee40b24cfd3c9f7c0fe93ee298b9046564637262115fa.png" /></a>
<nav role="navigation" class="masthead">
<div id="hide">
<ul>
</ul>
</div>
</nav>
</header>








<section id="callout-login">
<div class="client-login">

<br>
<h3 class="client-description">

</h3>

</div>
<form id="login" action="/login" accept-charset="UTF-8" method="post"><input name="utf8" type="hidden" value="&#x2713;" /><input type="hidden" name="authenticity_token" value="jUfJ/YewSB3Lj1Zsm04pcxc2Tw4HLsOnwSIEogO/5uU+ABboRy5NySKI1T6Ef1bKmayTeMjj4OhNPgMEi9RqDQ==" />
<p><label for="username">Username</label><i class="fa fa-question-circle fa-question-circle--blue user-name" title="Login using either your Username or Email Address"></i><input type="text" name="username" id="username" autofocus="autofocus" class="default" /></p>
<p><label for="password">Password</label><br /><input type="password" name="password" id="password" /></p>

<p><input type="hidden" name="client_id" id="client_id" value="Z0u-MdLNypXBjiDREZ3roA" /></p>
<p><input type="hidden" name="redirect_uri" id="redirect_uri" value="https://oceandata.sci.gsfc.nasa.gov/ob/getfile/restrict" /></p> <p><input type="hidden" name="response_type" id="response_type" value="code" /></p>
<p><input type="hidden" name="state" id="state" /></p>
<p><input type="checkbox" name="stay_in" id="stay_in" value="1" checked="checked" /> <label for="stay_in">Stay signed in (this is a private workstation)</label></p>

<p class="button-with-notes">
<input type="submit" name="commit" value="Log in" class="eui-btn--round eui-btn--green" data-disable-with="Log in" />
<a class="eui-btn--round eui-btn--blue" href="/users/new?client_id=Z0u-MdLNypXBjiDREZ3roA&amp;redirect_uri=https%3A%2F%2Foceandata.sci.gsfc.nasa.gov%2Fob%2Fgetfile%2Frestrict&amp;response_type=code">Register</a>
</p>
<p class="form-instructions">
<em class="icon-question-sign"></em>
<a class="" href="/retrieve_info">I don&rsquo;t remember my username</a>
<br /><em class="icon-question-sign"></em>
<a class="" href="/reset_passwords/new">I don&rsquo;t remember my password</a>
<br />
<em class="icon-question-sign"></em>
<a href="javascript:feedback.showForm();" title = 'Need Help? Click on the Feedback button to request help'>Help</a>
</p>
</form>
<aside class="govt-msg">
<div class="nasa-logo"></div>
<p><strong>Why must I register?</strong></p>
<p>
The Earthdata Login provides a single mechanism for user registration and profile management for all EOSDIS system components (DAACs, Tools, Services).
Your Earthdata login also helps the EOSDIS program better understand the usage of EOSDIS services to improve user experience through customization of tools and improvement of services.
EOSDIS data are openly available to all and free of charge except where governed by international agreements.
</p>
</aside>

</section>
<section id="cta">
<h3>Get single sign-on access to all your favorite EOSDIS sites</h3>
<a class="eui-btn--round eui-btn--blue" href="/users/new?client_id=Z0u-MdLNypXBjiDREZ3roA&amp;redirect_uri=https%3A%2F%2Foceandata.sci.gsfc.nasa.gov%2Fob%2Fgetfile%2Frestrict&amp;response_type=code">Register for a Profile</a>
</section>
<div class="govt-warning eui-info-box">
<div class="warning-desktop">
<p><strong>By accessing and using this information system, you acknowledge and consent to the following:</strong></p>
You are accessing a U.S. Government information system, which includes:
(1) this computer;
(2) this computer network;
(3) all computers connected to this network including end user systems;
(4) all devices and storage media attached to this network or to any computer on this network; and
(5) cloud and remote information services. This information system is provided for U.S. Government-authorized use only.
You have no reasonable expectation of privacy regarding any communication transmitted through or data stored on this
information system. At any time, and for any lawful purpose, the U.S. Government may monitor, intercept, search, and
seize any communication or data transiting, stored on, or traveling to or from this information system. You are
NOT authorized to process classified information on this information system. Unauthorized or improper use of this
system may result in suspension or loss of access privileges, disciplinary action, and civil and/or criminal penalties.
</div>
<div class="warning-mobile">
<p><strong>By accessing and using this information system, you acknowledge and consent to the following:</strong></p>
You are accessing a U.S. Government information system, which includes:
(1) this computer;
(2) this computer network;
(3) all computers connected to this network including end user systems;
(4) all devices and storage media attached to this network or to any computer on this network; and
(5) cloud and remote information services. This information system is provided for U.S. Government-authorized use only.
Unauthorized or improper use of this system may result in suspension or loss of access privileges, disciplinary action,
and civil and/or criminal penalties. By using this information system, you acknowledge and consent to the terms
and conditions established in NASA policy and regulatory guidance for NASA IT Systems.
</div>
<div class="warning-mobile-mini">
<strong>
US Govt Property. Unauthorized use subject to prosecution. Use subject to monitoring per
<a href="https://nodis3.gsfc.nasa.gov/displayDir ... NPD2810</a>.
</strong>
</div>
</div>


</div>
<footer role="contentinfo">
<h3>For questions regarding the EOSDIS Earthdata Login, please contact <a href="javascript:feedback.showForm();" title="Earthdata Support form">Earthdata Support</a></h3>
<ul>
<li class="version badge eui-badge--md">V 4.134
</li>
<li><a href="/">Home</a></li>
<li><a href="/users/new">Register</a></li>
<li><a href="/documentation">Documentation</a></li>
<li><a title="NASA Home" href="http://www.nasa.gov">NASA</a></li>
</ul>
<p>NASA Official: Stephen Berrick</p>
</footer>

<script src="/assets/application-fb42de9f82d354d4dff22da9ba9c2e1b5ee46eb3fecbd6f6419d4c9d17561a80.js"></script>
<script type="text/javascript">
$(window).scroll(function(e){
parallax();
});
function parallax(){
var scrolled = $(window).scrollTop();
$('#content').css('background-position', 'right ' + -(scrolled*0.25)+'px ');
}
</script>
<script src="https://cdn.earthdata.nasa.gov/tophat2/tophat2.js" id="earthdata-tophat-script" data-show-fbm="true" data-show-status="true" data-status-api-url="https://status.earthdata.nasa.gov/api/v ... "></script>
<script type="text/javascript" src="https://fbm.earthdata.nasa.gov/for/URS4 ... "></script>
<script type="text/javascript">
feedback.init();
</script>
<script type="text/javascript">
setTimeout(function()
{var a=document.createElement("script"); var b=document.getElementsByTagName("script")[0];
a.src=document.location.protocol+"//dnn506yrbagrg.cloudfront.net/pages/scripts/0013/2090.js?"+Math.floor(new Date().getTime()/3600000);
a.async=true;a.type="text/javascript";b.parentNode.insertBefore(a,b)}
, 1);
</script>

<!-- BEGIN: DAP Google Analytics -->
<script language="javascript" id="_fed_an_ua_tag" src="https://dap.digitalgov.gov/Universal-Fe ... "></script>
<!-- END: DAP Google Analytics -->


</body>
</html>
Attachments
html.pdf
(158.67 KiB) Downloaded 32 times

Tags:

OB ODPS - jgwilding
Subject Matter Expert
Subject Matter Expert
Posts: 110
Joined: Fri Feb 19, 2021 1:09 pm America/New_York

Re: MODIS download with token key

by OB ODPS - jgwilding » Wed Jul 14, 2021 8:19 am America/New_York

Hi. I just want to be clear that you are trying to download data from OBDAAC and not LAADS. You mention a token key, which is what LAADS offers for data downloads. OBDAAC uses the name app key. They are not interchangeable. You would use the token key obtained from LAADS to download data from them, and the app key to download data from OBDAAC. According to our database, you have not created an app key, so if you are attempting to download data from OBDAAC using either a LAADS token or an old OBDAAC app key, your attempt would be redirected to the Earthdata login page. If you are trying to download data from OBDAAC, you should visit the OBDAAC web site https://oceandata.sci.gsfc.nasa.gov/appkey/ and generate one.

I also do not know if you are using the Python download script provided by OBDAAC available at https://oceancolor.gsfc.nasa.gov/data/ ... _methods/ or a script that you wrote yourself. If you are not using the one provided by OBDAAC, I would recommend that you try that one (after you have generated an app key), to see if that works.

Regards,
john

niglas_cas
Posts: 5
Joined: Sun Jul 11, 2021 1:26 pm America/New_York

Re: MODIS download with token key

by niglas_cas » Wed Jul 14, 2021 9:59 am America/New_York

Hi John,

Many thanks for your reply.

I don't know why but it seems that because the Earthdata login page was blocked when my own script tries to log in. However, I successfully downloaded the data using a VPN. Moreover, I find that data from OBDAAC are available for both the app key from OBDAAC and the token key from LAADS^-^.

I also try the python script at https://oceancolor.gsfc.nasa.gov/data/download_methods/, but it sometimes works but sometimes failed with a 404 error, no matter whether I used the VPN. (Of course, it may be something wrong with my network.)

Also, I find if the '--force' argument is not specified, the following error will occur. (command "obdaac_download.py -v --appkey my_appkey S3A_OL_1_EFR____20160425T113314_20160425T113421_20180201T184243_0067_003_237_1260_LR2_R_NT_002.zip"; Python 3.7.1 in windows)

Traceback (most recent call last):
File "obdaac_download.py", line 281, in <module>
appkey=args.appkey)
File "obdaac_download.py", line 212, in retrieveURL
return httpdl(server, netpath, localpath=localpath, uncompress=uncompress, verbose=verbose,force_download=force_download)
File "obdaac_download.py", line 84, in httpdl
ofile = localpath / rpath.name
AttributeError: 'str' object has no attribute 'name'


Thanks again.

Best wishes,
Ming

OB ODPS - jgwilding
Subject Matter Expert
Subject Matter Expert
Posts: 110
Joined: Fri Feb 19, 2021 1:09 pm America/New_York

Re: MODIS download with token key

by OB ODPS - jgwilding » Wed Jul 14, 2021 4:43 pm America/New_York

Hi Ming.

Did you set up the .netrc file in your home directory as shown in the download_methods page? If so, I'm wondering if that might be why it appears that the LAADS token would work for OBDAAC. Otherwise that would surprise me because the APIs are different and how the token is specified for LAADS is completely different from how the app-key is specified for OBDAAC. I do see that you did generate an app-key on our site.

My other question is about the file you're trying to download in your example, an OLCI file. These require accepting the ESA end user license agreement (EULA), so if you haven't done that, you might be getting redirected to the page to do that. You wouldn't get that for MODIS or VIIRS files. To check, go to https://urs.earthdata.nasa.gov/profile , click on Applications -> Accepted EULAs. If you don't see the Sentinel EULA listed as an accepted one, you might need to initiate an interactive download of a file through a web browser, and then when you are presented with the EULA, you can accept it.

I believe the --force argument just causes the python script to download the file even if it already exists locally.

Do you have access to wget on your system? It might be easier to use that for debugging until we can get a download to work.

john

niglas_cas
Posts: 5
Joined: Sun Jul 11, 2021 1:26 pm America/New_York

Re: MODIS download with token key

by niglas_cas » Wed Jul 14, 2021 11:31 pm America/New_York

Hi John,

Yes, I have set up the .netrc file. And I have successfully download the data with a VPN. I think it must be something wrong with my network before.

Many thanks,
Ming

Post Reply