Page 1 of 1

wget issues

Posted: Thu Jan 05, 2017 1:23 pm America/New_York
by dbotis
Hi,

I just ordered some data and got the normal email back saying files were ready. Here are the pertinent lines:

Manifest URL:
         http://oceandata.sci.gsfc.nasa.gov/cgi/getfile/http_manifest.txt?h=ocdist101&p=/data1/a070a060e766895f

You can then use wget to download the files in your manifest file:
         wget --content-disposition -i manifest-file-name

You can also pipe wget commands together to initiate the download:
         wget -O - 'manifest-url-from-above' | wget --content-disposition -i -

Note that the URL has "http" and not "https". When I inserted the URL into the wget line as usual with "http", it could not connect, so I changed the URL to "https". It was able to connect, but the transfer of files did not occur. Here is what I got:
seashell 184 :/home1/scratch/epa/satellite/modis/GOM/L2_OC> wget -O - 'https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/http_manifest.txt?h=ocdist101&p=/data1/a070a060e766895f' | wget --content-disposition -i -
--2017-01-05 18:08:24--  https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/http_manifest.txt?h=ocdist101&p=/data1/a070a060e766895f
Resolving oceandata.sci.gsfc.nasa.gov... xx.xxx.xx.xx, 2001:4d0:2418:128::84
Connecting to oceandata.sci.gsfc.nasa.gov|xx.xxx.xx.xx|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1479 (1.4K) [application/octet-stream]
Saving to: âSTDOUTâ

After this, that was it. No transfer of files. This is on a UNIX server. I suspect this has to do with the recent change to https, since I order data often and never have had issues in the past. I looked in the manifest file and the URLs in there also start with "http". Any ideas are appreciated. Thanks,

Dan

wget issues

Posted: Thu Jan 05, 2017 5:38 pm America/New_York
by OB.DAAC - SeanBailey
Dan,

Yes, 'tis related.  We're still finding places where the URL needs to change :sad:
This is one of them...
The manifest file has http links in it as well.  You can either download and edit the manifest file,
or you can add a step in the middle to translate the http to https:

wget -O - 'https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/http_manifest.txt?h=ocdist101&p=/data1/a070a060e766895f' |sed -e 's/http:/https:/'| wget --content-disposition -i -

Sean

wget issues

Posted: Fri Jan 06, 2017 10:09 am America/New_York
by dbotis
Thanks, Sean

That works just fine.