wget issues

Use this Forum to find information on, or ask a question about, NASA Earth Science data.
Post Reply
dbotis
Posts: 40
Joined: Mon Nov 21, 2011 7:44 pm America/New_York
Answers: 0

wget issues

by dbotis » Thu Jan 05, 2017 1:23 pm America/New_York

Hi,

I just ordered some data and got the normal email back saying files were ready. Here are the pertinent lines:

Manifest URL:
         http://oceandata.sci.gsfc.nasa.gov/cgi/getfile/http_manifest.txt?h=ocdist101&p=/data1/a070a060e766895f

You can then use wget to download the files in your manifest file:
         wget --content-disposition -i manifest-file-name

You can also pipe wget commands together to initiate the download:
         wget -O - 'manifest-url-from-above' | wget --content-disposition -i -

Note that the URL has "http" and not "https". When I inserted the URL into the wget line as usual with "http", it could not connect, so I changed the URL to "https". It was able to connect, but the transfer of files did not occur. Here is what I got:
seashell 184 :/home1/scratch/epa/satellite/modis/GOM/L2_OC> wget -O - 'https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/http_manifest.txt?h=ocdist101&p=/data1/a070a060e766895f' | wget --content-disposition -i -
--2017-01-05 18:08:24--  https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/http_manifest.txt?h=ocdist101&p=/data1/a070a060e766895f
Resolving oceandata.sci.gsfc.nasa.gov... xx.xxx.xx.xx, 2001:4d0:2418:128::84
Connecting to oceandata.sci.gsfc.nasa.gov|xx.xxx.xx.xx|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1479 (1.4K) [application/octet-stream]
Saving to: âSTDOUTâ

After this, that was it. No transfer of files. This is on a UNIX server. I suspect this has to do with the recent change to https, since I order data often and never have had issues in the past. I looked in the manifest file and the URLs in there also start with "http". Any ideas are appreciated. Thanks,

Dan

Filters:

OB.DAAC - SeanBailey
User Services
User Services
Posts: 1485
Joined: Wed Sep 18, 2019 6:15 pm America/New_York
Answers: 1
Been thanked: 5 times

wget issues

by OB.DAAC - SeanBailey » Thu Jan 05, 2017 5:38 pm America/New_York

Dan,

Yes, 'tis related.  We're still finding places where the URL needs to change :sad:
This is one of them...
The manifest file has http links in it as well.  You can either download and edit the manifest file,
or you can add a step in the middle to translate the http to https:

wget -O - 'https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/http_manifest.txt?h=ocdist101&p=/data1/a070a060e766895f' |sed -e 's/http:/https:/'| wget --content-disposition -i -

Sean

dbotis
Posts: 40
Joined: Mon Nov 21, 2011 7:44 pm America/New_York
Answers: 0

wget issues

by dbotis » Fri Jan 06, 2017 10:09 am America/New_York

Thanks, Sean

That works just fine.

Post Reply