Trouble using Wget for Downloading dataset

Use this Forum to find information on, or ask a question about, NASA Earth Science data.
Post Reply
ariesds
Posts: 8
Joined: Sat Sep 25, 2021 2:10 pm America/New_York
Answers: 0

Trouble using Wget for Downloading dataset

by ariesds » Tue Apr 26, 2022 6:49 am America/New_York

Dear All,

I need help to solve my problem when using wget for downloading datasets.

The dataset will download:
https://oceandata.sci.gsfc.nasa.gov (/directaccess/MODIS-Aqua/L3SMI/2010/)

The syntax used:
wget -q -O - https://oceandata.sci.gsfc.nasa.gov (/directaccess/MODIS-Aqua/L3SMI/2010/) wget --user=USERNAME --ask-password --auth-no-challenge=on --base https://oceandata.sci.gsfc.nasa.gov (/ -N --wait=0.5 --random-wait --force-html -i -)

The result is the file html (namely index.htm)

Thank you for your help.
Best
Aries

Tags:

OB.DAAC - amscott
User Services
User Services
Posts: 350
Joined: Mon Jun 22, 2020 5:24 pm America/New_York
Answers: 1
Has thanked: 8 times
Been thanked: 3 times

Re: Trouble using Wget for Downloading dataset

by OB.DAAC - amscott » Tue Apr 26, 2022 11:59 am America/New_York

Try adding a day directory to your URL. Instead of https://oceandata.sci.gsfc.nasa.gov (/directaccess/MODIS-Aqua/L3SMI/2010/) try https://oceandata.sci.gsfc.nasa.gov (/directaccess/MODIS-Aqua/L3SMI/2010/033/)

then add a grep statement to select the type of files you want from that directory:
grep CHL

So the command becomes:
wget -q -O - https://oceandata.sci.gsfc.nasa.gov (/directaccess/MODIS-Aqua/L3SMI/2010/033/) |grep CHL| wget --user=<username> --ask-password --auth-no-challenge=on --base https://oceandata.sci.gsfc.nasa.gov (/ -N --wait=0.5 --random-wait --force-html -i -)

Make sure to replace <username> with your Earthdata login username.

ariesds
Posts: 8
Joined: Sat Sep 25, 2021 2:10 pm America/New_York
Answers: 0

Re: Trouble using Wget for Downloading dataset

by ariesds » Tue Apr 26, 2022 4:10 pm America/New_York

Dear OB.DAAC - amscott,

Thank you for your response and explanation to solve the problem.

I already followed your instruction, including changing the <username> with my account.

I got the error message :
'grep' is not recognized as an internal or external command,
operable program or batch file.

When I did not use 'grep', the following result below:
<!DOCTYPE html>
<html lang="en">
<head>

<script src="/globalassets/static/js/jquery.min.js"></script>

<!-- Google Tag Manager -->
<script>
var dataLayer = window.dataLayer = window.dataLayer || [];
$.get('URL://oceancolor.gsfc.nasa.gov/get_ip/', function(data) {
dataLayer.push({
'event':'ipAddress',
'ipAddress':data
});
});

(function(w,d,s,l,i){w[l]=w[l]||[];w[l].push(

{'gtm.start': new Date().getTime(),event:'gtm.js'}

);var f=d.getElementsByTagName(s)[0],
j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src=
[URL: googletagmanager.com /gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);
})(window,document,'script','dataLayer','GTM-WNP7MLF');</script>
<!-- End Google Tag Manager -->]

<meta name="MobileOptimized" content="width">
<meta name="HandheldFriendly" content="true">
<meta name="viewport" content="width=device-width">
<!--
<script src="/js/jquery-ui-1.11.4.custom/jquery-ui.min.js"></script>
<link href="/js/jquery-ui-1.11.4.custom/jquery-ui.css" rel="stylesheet" />
-->
<script src="/globalassets/static/js/jquery-ui-1.13.0/jquery-ui.js"></script>
<link href="/globalassets/static/js/jquery-ui-1.13.0/jquery-ui.css" rel="stylesheet" />

<!--
<link href="/css/theme/smoothness_jquery-ui.css" rel="stylesheet" />
-->
<link href="/globalassets/static/css/theme/jquery-ui.min.smoothness.css" rel="stylesheet" />
<link href="/css/localstyle.css" rel="stylesheet" />



<link href="/globalassets/static/css/theme/eui/application.css" rel="stylesheet" />
<link href="/globalassets/static/css/theme/eui/application-1.0.0.css" rel="stylesheet" />
<link href="/globalassets/static/css/theme/font-awesome-4.5.0/css/font-awesome.min.css" rel="stylesheet" />
<link href="/globalassets/static/css/w3.css" rel="stylesheet" />
<link href="/globalassets/static/css/theme/styles.css" rel="stylesheet" />
<link href="/globalassets/static/css/theme/footer.css" rel="stylesheet" />

<link href="/css/progress_bar.css" rel="stylesheet">
<link href="/css/oceandata.css" rel="stylesheet">

<link href="/globalassets/static/js/subscriptions/create.css" rel="stylesheet" />

<title>MODIS-Aqua/L3SMI/2010/033</title>


</head>
<body>
<!-- Google Tag Manager (noscript) -->
<noscript><iframe src=. [URL: googletagmanager.com /ns.html?id=GTM-WNP7MLF"
height="0" width="0" style="display:none;visibility:hidden"></iframe></noscript>
<!-- End Google Tag Manager (noscript) -->]


<p id="skip-link"><a href="#main" class="element-invisible element-focusable">Jump to navigation</a></p>
<div id="page-admin">
<header class="header" id="header">
<a href="/" title="OceanData Home" rel="home" class="header__logo" id="logo"> <img src="/img/oceancolor_data.png" alt="Home" class="header__logo-image" /></a>

</header>
<div id="main">
<div id="content" class="column" role="main">

<!--div class="eui-banner-danger">
<p class="eui-banner__message">
</p>
</div-->
<a id="main-content"></a>

<section class='breadcrumbs_direct_data_access'>
<a href='/'>OceanData Home</a><span class="breadcrumb_sep"> Γû╕ </span><a href='/directaccess/'>directaccess</a><span class="breadcrumb_sep"> Γû╕ </span><a href='/directaccess/MODIS-Aqua/'>MODIS-Aqua</a><span class="breadcrumb_sep"> Γû╕ </span><a href='/directaccess/MODIS-Aqua/L3SMI/'>L3SMI</a><span class="breadcrumb_sep"> Γû╕ </span><a href='/directaccess/MODIS-Aqua/L3SMI/2010/'>2010</a><span class="breadcrumb_sep"> Γû╕ </span><a href='/directaccess/MODIS-Aqua/L3SMI/2010/033/'>033</a>
</section>

<table>
<thead>
<tr>
<th>Filename</th>
<th>Last Modified</th><th>Size</th>
</tr>
</thead>
<tbody>
<tr>
<td><a href= //oceandata.sci.gsfc.nasa.gov/ob/getfile/A2010033.L3m_DAY_CHL_chlor_a_4km.nc'>A2010033.L3m_DAY_CHL_chlor_a_4km.nc</a> &nbsp;</td>
<td>2017-12-31 05:09:40</td>
<td>9796180</td>
</tr>
<tr>
<td><a href= //oceandata.sci.gsfc.nasa.gov/ob/getfile/A2010033.L3m_DAY_CHL_chlor_a_9km.nc'>A2010033.L3m_DAY_CHL_chlor_a_9km.nc</a> &nbsp;</td>
<td>2017-12-31 05:09:40</td>
<td>3707471</td>
</tr>
<tr>
<td><a href= //oceandata.sci.gsfc.nasa.gov/ob/getfile/A2010033.L3m_DAY_CHL_chl_ocx_4km.nc'>A2010033.L3m_DAY_CHL_chl_ocx_4km.nc</a> &nbsp;</td>
<td>2017-12-31 05:09:40</td>
<td>9867111</td>
</tr>
<tr>
<td><a href= //oceandata.sci.gsfc.nasa.gov/ob/getfile/A2010033.L3m_DAY_CHL_chl_ocx_9km.nc'>A2010033.L3m_DAY_CHL_chl_ocx_9km.nc</a> &nbsp;</td>
<td>2017-12-31 05:09:40</td>
<td>3721728</td>
</tr>
<tr>
<td><a href= //oceandata.sci.gsfc.nasa.gov/ob/getfile/A2010033.L3m_DAY_FLH_ipar_4km.nc'>A2010033.L3m_DAY_FLH_ipar_4km.nc</a> &nbsp;</td>
<td>2017-12-31 12:01:59</td>
<td>3204854</td>
</tr>
<tr>
........
...........
...........
etc......




</div><!--content-->
<nav id="navigation">
<div class="topnav" id="myTopnav">
<a href="/" class="home" aria-label="home"><i class="fa fa-home"></i></a>
<div class="dropdown">
<button type="button" class="dropbtn">ABOUT</button>
<div id="about" class="dropdown-content">
<a href="//oceancolor.gsfc.nasa.gov/about/">What We Do</a>
<a href="//oceancolor.gsfc.nasa.gov/missions/">Supported Missions</a>
<a href="//oceancolor.gsfc.nasa.gov/staff/">Staff Directory</a>
<h6>Forums</h6>
<a href="app.php/tag/OBDAAC/AND?" title="EarthData Forum">Ask a Question</a>
<a href="//oceancolor.gsfc.nasa.gov/forum/oceancolor/forum_show.pl" title="OceanColor Forum">Search Forum Archive</a>
</div>
</div>
<div class="dropdown">
<button type="button" class="dropbtn">MISSIONS</button>
<div id="missions" class="dropdown-content">
<a href="//oceancolor.gsfc.nasa.gov/data/aquarius/">Aquarius</a>
<a href="//oceancolor.gsfc.nasa.gov/data/czcs/">CZCS</a>
<a href="//oceancolor.gsfc.nasa.gov/data/goci/">GOCI</a>
<a href="//oceancolor.gsfc.nasa.gov/data/hawkeye/">HawkEye</a>
<a href="//oceancolor.gsfc.nasa.gov/data/hico/">HICO</a>
<a href="//oceancolor.gsfc.nasa.gov/data/meris/">MERIS</a>
<a href="//oceancolor.gsfc.nasa.gov/data/aqua/">MODIS-Aqua</a>
<a href="//oceancolor.gsfc.nasa.gov/data/terra/">MODIS-Terra</a>
<a href="//oceancolor.gsfc.nasa.gov/data/octs/">OCTS</a>
<a href="//oceancolor.gsfc.nasa.gov/data/olci-s3a/">OLCI-S3A</a>
<a href="//oceancolor.gsfc.nasa.gov/data/olci-s3b/">OLCI-S3B</a>
<a href="//oceancolor.gsfc.nasa.gov/data/pace/">PACE</a>
<a href="//oceancolor.gsfc.nasa.gov/data/seawifs/">SeaWiFS</a>
<a href="//oceancolor.gsfc.nasa.gov/data/viirs-j1/">VIIRS-JPSS1</a>
<a href="//oceancolor.gsfc.nasa.gov/data/viirs-snpp/">VIIRS-SNPP</a>
<h6>Projects</h6>
<a href="/coral_browser/">PRISM-CORAL</a>
<a href="//oceancolor.gsfc.nasa.gov/projects/cyan/">CyAN</a>
<a href="//oceancolor.gsfc.nasa.gov/projects/inlandwaters/">Inland Waters</a>
<h6>Forums</h6>
<a href="app.php/tag/OBDAAC/AND?" title="EarthData Forum">Ask a Question</a>
<a href="//oceancolor.gsfc.nasa.gov/forum/oceancolor/forum_show.pl" title="OceanColor Forum">Search Forum Archive</a>
</div>
</div>
<div class="dropdown">
<button type="button" class="dropbtn">DATA</button>
<div id="data" class="dropdown-content">
<a href="//oceancolor.gsfc.nasa.gov/data/overview/">Data Overview</a>
<a href="/">Ocean Data Home</a>
<a href="//seabass.gsfc.nasa.gov/"> SeaBASS</a>

<h6>Get Data</h6>
<a href="/directaccess/">Direct Data Access</a>
<a href="//oceandata.sci.gsfc.nasa.gov/api/file_search/">File Search</a>
<a href="//seabass.gsfc.nasa.gov/search/"> Field Data</a>
<a href="/opendap/">OPeNDAP</a>

<h6>How To</h6>
<a href="//oceancolor.gsfc.nasa.gov/data/download_methods/">Search &amp; Download</a>
<a href="//oceancolor.gsfc.nasa.gov/citations/">Cite Data</a>

<!--h6>Projects</h6-->
<!--a href="/projects/">Projects</a-->

<h6>Browse Data</h6>
<a href="//oceancolor.gsfc.nasa.gov/cgi/browse.pl?sen=amod">Level 1&amp;2 Browser<br><img src="//oceancolor.gsfc.nasa.gov/icons/l1.png" alt=""></a>
<a href="//oceancolor.gsfc.nasa.gov/l3/">Level 3 Browser<br><img src="//oceancolor.gsfc.nasa.gov/icons/l3.png" alt=""></a>

<h6>Trending & Analysis</h6>
<a href="//oceancolor.gsfc.nasa.gov/cgi/l3bts">Level-3 Time Series Plotter</a>
<a href="/overpass_pred/">Overpass Predictor</a>

<h6>Quality Assessment</h6>
<a href="//seabass.gsfc.nasa.gov/search#val">Product Validation</a>
<a href="//oceancolor.gsfc.nasa.gov/analysis/global/">Global L3 Trends</a>
<a href="//oceancolor.gsfc.nasa.gov/cgi/mission_quality_monitor/">Mission Quality Monitor</a>

<h6>Forums</h6>
<a href="app.php/tag/OBDAAC/AND?" title="EarthData Forum">Ask a Question</a>
<a href="//oceancolor.gsfc.nasa.gov/forum/oceancolor/forum_show.pl" title="OceanColor Forum">Search Forum Archive</a>
</div>
</div>
<div class="dropdown">
<button type="button" class="dropbtn">DOCS</button>
<div id="docs" class="dropdown-content">
<h6>Technical Docs</h6>
<a href="//oceancolor.gsfc.nasa.gov/docs/technical/#TM">NASA Technical Reports</a>
<a href="//oceancolor.gsfc.nasa.gov/docs/technical/#UG">Users Guides</a>
<a href="//oceancolor.gsfc.nasa.gov/docs/technical/#WP">White Papers</a>
<a href="//oceancolor.gsfc.nasa.gov/docs/technical/#AT">MODIS Docs (historical)</a>
<a href="//oceancolor.gsfc.nasa.gov/docs/technical/#SPEC">Sensor Spectral Information</a>
<h6>Data</h6>
<a href="//oceancolor.gsfc.nasa.gov/docs/filenaming-convention/">Filenaming Convention</a>
<a href="//oceancolor.gsfc.nasa.gov/citations/">How to Cite</a>
<a href="//oceancolor.gsfc.nasa.gov/reprocessing/">Processing History</a>
<h6>Community Engagement</h6>
<a href="//oceancolor.gsfc.nasa.gov/meetings/">Meetings/Workshops</a>
<a href="//oceancolor.gsfc.nasa.gov/outreach/">Outreach &amp; Education</a>
<h6>Products</h6>
<a href="//oceancolor.gsfc.nasa.gov/products/">Level 1, 2, &amp; 3 Definitions</a>
<a href="//oceancolor.gsfc.nasa.gov/docs/format/">Format Specifications</a>
<a href="//oceancolor.gsfc.nasa.gov/product_status/">Product Status by Mission</a>
<a href="//oceancolor.gsfc.nasa.gov/atbd/">Algorithm Descriptions</a>
<a href="//oceancolor.gsfc.nasa.gov/docs/ancillary/">Ancillary Sources</a>
<h6>Forums</h6>
<a href="app.php/tag/OBDAAC/AND?" title="EarthData Forum">Ask a Question</a>
<a href="//oceancolor.gsfc.nasa.gov/forum/oceancolor/forum_show.pl" title="OceanColor Forum">Search Forum Archive</a>
</div>
</div>
<div class="dropdown">
<button type="button" class="dropbtn">SOFTWARE & TOOLS</button>
<div id="tools" class="dropdown-content">
<h6>Visualization</h6>
<a href="//seadas.gsfc.nasa.gov/">SeaDAS</a>
<a href="//seadas.gsfc.nasa.gov/tutorials/">SeaDAS Documentation</a>
<a href="https://www.youtube.com/user/nasaoceancolor" class="ext">SeaDAS Informational Videos</a>
<a href="/ocssw/"> OCSSW Software</a>
<a href="//oceancolor.gsfc.nasa.gov/docs/ocssw/"> OCSSW Documentation</a>
<h6>Online Tools</h6>
<a href="//oceancolor.gsfc.nasa.gov/cgi/l3bts">Level-3 Time Series Plotter</a>
<a href="//oceancolor.gsfc.nasa.gov/otherresources/">Other Resources</a>
<!--a href="//disc.sci.gsfc.nasa.gov/giovanni/" class="ext">Giovanni</a-->
<h6>Forums</h6>
<a href="app.php/tag/OBDAAC/AND?" title="EarthData Forum">Ask a Question</a>
<a href="//oceancolor.gsfc.nasa.gov/forum/oceancolor/forum_show.pl" title="OceanColor Forum">Search Forum Archive</a>
</div>
</div>
<div class="dropdown">
<button type="button" class="dropbtn">SERVICES</button>
<div id="services" class="dropdown-content">
<h6>Sign Up</h6>
<a href="//oceancolor.gsfc.nasa.gov/registration/">User Registration</a>
<a href="//oceancolor.sci.gsfc.nasa.gov/mailman/listinfo">Mailing Lists</a>
<h6>Online Tools</h6>
<a href="/overpass_pred/">Overpass Predictor</a>
<h6>Subscriptions/Orders</h6>
<a href="/data_dashboard/">Data Dashboard</a>
<h6>More...</h6>
<a href="//oceancolor.gsfc.nasa.gov/fsg/hplc/">HPLC Pigments</a>
<h6>Forums</h6>
<a href="app.php/tag/OBDAAC/AND?" title="EarthData Forum">Ask a Question</a>
<a href="//oceancolor.gsfc.nasa.gov/forum/oceancolor/forum_show.pl" title="OceanColor Forum">Search Forum Archive</a>
</div>
</div>
<a id="ig" href="//oceancolor.gsfc.nasa.gov/gallery/">GALLERY</a>
<a id="ocf" href="app.php/tag/OBDAAC/AND?">FORUM</a>
<a href="javascript:void(0);" class="icon" onclick="myFunction()">&#9776;</a>
<!--login button-->
<div class="block block-search last even" id="block-search-form"><div class="container-inline">
<form action="/login/">
<button class='btn search-home ui-button ui-widget ui-state-default ui-corner-all ui-button-text-only' type='submit' title="Login/Logout"><span>Login</span><i class="fa fa-sign-in"></i> </button>
</form>
</div>
</div><!--end login button-->
</div><!--close navbar div-->
</nav><!--close nav wrapper-->
</div><!--main-->
</div><!--page-admin-->

<footer id="footer" class="region region-footer">
<div>
Responsible NASA Official: <a href= ://science.gsfc.nasa.gov/sed/bio/sean.w.bailey">Sean Bailey</a><br>
Curator: <a href="mailto:webadmin@oceancolor.gsfc.nasa.gov">OceanColor Webmaster</a><br>
</div>
<div class="socialmedia">
<a id="fb" title="Follow us on Facebook!" alt="Find us @NASAOcean on Facebook" href="https://www.facebook.com/NASAOcean" target="blank"><i class="fa fa-facebook-square fa fa-inverse"></i></a>
<a id="tw" title="Follow us on Twitter!" alt="Find us @NASAOcean on Twitter" href="https://www.twitter.com/NASAOcean" target="blank"><i class="fa fa-twitter fa fa-inverse"></i></a>
<a id="yt" title="Subscribe to our YouTube Channel!" alt="Subscribe to NASA's OceanColor YouTube Channel" href="https://www.youtube.com/user/nasaoceancolor" target="blank"><i class="fa fa-youtube-square fa-inverse"></i></a>
<a id="gram" title="Follow @nasaocean on Instagram" alt="Follow us @nasaocean on Instagram" ><i class="fa fa-instagram fa-inverse"></i></a>
<!--a href="app.php/tag/OB%20DAAC/AND?" title="Ask a Question." alt="Find solutions on the OceanColor Forum" target="blank"><i class="fa fa-comments fa-2x fa-inverse fa-fw"></i></a-->
</div>
<div class="compliance">
<ul>
<li><a class="ext" href="https://www.nasa.gov. (/about/highlights/HP_Privacy.html)">Web Privacy Policy</a></li>
<li><a class="ext" href="https://science.nasa.gov (/earth-science/earth-science-data/data-information-policy/) ">Data &amp; Information Policy</a></li>
<li><a class="ext" href="https://www.nasa.gov. (/audience/formedia/features/communication_policy.html)">Communications Policy</a></li>
<li><a class="ext" href="https://www.nasa.gov. (/FOIA/index.html) ">Freedom of Information Act</a></li>
<li><a class="ext" href=" (https://).usa.gov/">USA.gov</a></li>
</ul>
</div>
</footer>


<script src="/css/theme/eui/eui.js"></script>
<script src="https://cdn.earthdata.nasa.gov/tophat2/tophat2.js"
id="earthdata-tophat-script"
data-show-fbm="false"
data-show-status="false"
data-status-polling-interval=5
data-current-daac="OB.DAAC"></script>
<script id="_fed_an_ua_tag" src="https://dap.digitalgov.gov/Universal-Federated-Analytics-Min.js?agency=NASA&subagency=GSFC&yt=true&dclink=true"></script>

<script>
jQuery(document).ready(
function($) {
$('.banner-dismissible').on('click', 'a.dismiss', function() {
$(this).parents('.banner').remove();
});
});
/* Toggle between adding and removing the "responsive" class to topnav when the user clicks on the icon */
function myFunction() {
var x = document.getElementById("myTopnav");
if (x.className === "topnav") {
x.className += " responsive";
}
else {
x.className = "topnav";
}
}

</script>
</body>
</html>

Could you please help me?

Thank you.

Best,
Aries

OB.DAAC - amscott
User Services
User Services
Posts: 350
Joined: Mon Jun 22, 2020 5:24 pm America/New_York
Answers: 1
Has thanked: 8 times
Been thanked: 3 times

Re: Trouble using Wget for Downloading dataset

by OB.DAAC - amscott » Wed Apr 27, 2022 3:55 pm America/New_York

Are you using a Windows machine?

I found this: https://www.windows-commandline.com/findstr-command-examples-regular/ which says you may be able to accomplish the same goal using the findstr command instead of grep.

Otherwise try this:
wget -q --post-data="sensor=aqua&sdate=2010-01-01&edate=2010-12-31&dtype=L3b&addurl=1&results_as_file=1&search=*DAY_CHL*" -O - https://oceandata.sci.gsfc.nasa.gov/api/file_search

changing the 'search' parameter as needed to indicate which type of files you want. You should get back a list of URLs for downloading each data file.

ariesds
Posts: 8
Joined: Sat Sep 25, 2021 2:10 pm America/New_York
Answers: 0

Re: Trouble using Wget for Downloading dataset

by ariesds » Thu Apr 28, 2022 2:43 am America/New_York

Yes, I use Windows machine.

I have tried the example that you mentioned. The result shows the list of the datasets based on the type is looking for.

Then, I need to download each piece of data. That is very time-consuming.

I still figure out the other way.

Thank you for your suggestion.

OB.DAAC - amscott
User Services
User Services
Posts: 350
Joined: Mon Jun 22, 2020 5:24 pm America/New_York
Answers: 1
Has thanked: 8 times
Been thanked: 3 times

Re: Trouble using Wget for Downloading dataset

by OB.DAAC - amscott » Thu Apr 28, 2022 1:21 pm America/New_York

If you would like to try a bulk download method, we show an example on how to do this using python on the Search and Download Data page, as well.

ariesds
Posts: 8
Joined: Sat Sep 25, 2021 2:10 pm America/New_York
Answers: 0

Re: Trouble using Wget for Downloading dataset

by ariesds » Wed May 04, 2022 5:15 pm America/New_York

I already tried to use python as you mention (https://oceancolor.gsfc.nasa.gov/data/download_methods/), but I did not know which part should be changed related to the username and URL for the datasets.

Post Reply