Page 1 of 1

HICO data download

Posted: Sat May 18, 2019 12:08 pm America/New_York
by mkahru
I am trying to download HICO level-2 data using a list of file links that I got from the Level1&2 browser but I get files with 0 length.
My scripts work with regular http_manifest.txt files but not for the list I get for HICO.
Can anybody give a script using either Curl or Wget for Windows?
Thanks!
Mati

HICO data download

Posted: Mon May 20, 2019 10:39 am America/New_York
by OB WebDev - norman
Hi Mati,

Have you specifically registered to download HICO data?

https://oceancolor.gsfc.nasa.gov/registration/

If so, can you share a specific URL that is not working for you?

Regards,
Norman

HICO data download

Posted: Fri May 24, 2019 11:31 am America/New_York
by mkahru
Hi Norman,A sample link is: https://oceandata.sci.gsfc.nasa.gov [/hico/getfile/H2014252205906.L2_ISS_OC.ncI ]can download the NC file when I manually copy this address to my browser. I have over 100 or these links and it would take a long time to do it manually. I cannot use wget or curl to download from a link like that. Can you give an example?Thanks!

HICO data download

Posted: Sat May 25, 2019 7:24 am America/New_York
by gnwiii
Downloading data from GES DISC has instructions for using wget in windows.

HICO data download

Posted: Sat May 25, 2019 4:53 pm America/New_York
by mkahru
I am using wget 1.18 and I am downloading all kinds of gibberish (css, js, html, etc) but NOT the needed netCDF files.

HICO data download

Posted: Sat May 25, 2019 5:48 pm America/New_York
by gnwiii
The "gibberish" may provide details for the failure to retrieve your requested file.   It may become readable if you can load it in a browser (maybe rename to .html).   Wget has a number of debugging options that may be helpful.   When NASA moved to https there were problems with wget and curl built with older libraries that didn't handle the redirects or didn't support the (more secure) https configuration NASA is using.   At that time, many of the binaries you could download didn't work.  The current versions of wget from Msys2 or Cygwin should be OK.

Google found 64-bit wget version 1.20.3 zip archives at Windows binaries of GNU Wget

C:UsersseadasDesktop>wget --version
GNU Wget 1.20.3 built on mingw32.

-cares +digest +gpgme +https +ipv6 +iri +large-file +metalink -nls
+ntlm +opie +psl +ssl/openssl

Wgetrc:
    C:Usersgwhiteeosdis.wgetrc (env)
    C:Usersgwhite/.wgetrc (user)
    /win32dev/misc/wget/out64/etc/wgetrc (system)
Compile:
    x86_64-w64-mingw32-gcc -DHAVE_CONFIG_H
    -DSYSTEM_WGETRC="/win32dev/misc/wget/out64/etc/wgetrc"
    -DLOCALEDIR="/win32dev/misc/wget/out64/share/locale" -I. -I../lib
    -I../lib -I/win32dev/misc/wget/out64/include
    -I/win32dev/misc/wget/out64/include
    -I/win32dev/misc/wget/out64/include -DPCRE2_STATIC
    -I/win32dev/misc/wget/out64/include
    -I/win32dev/misc/wget/out64/include -DHAVE_LIBSSL
    -I/win32dev/misc/wget/out64/include
    -I/win32dev/misc/wget/out64/include -DNDEBUG -g -O2
Link:
    x86_64-w64-mingw32-gcc -I/win32dev/misc/wget/out64/include
    -I/win32dev/misc/wget/out64/include -DPCRE2_STATIC
    -I/win32dev/misc/wget/out64/include
    -I/win32dev/misc/wget/out64/include -DHAVE_LIBSSL
    -I/win32dev/misc/wget/out64/include
    -I/win32dev/misc/wget/out64/include -DNDEBUG -g -O2
    -L/win32dev/misc/wget/out64/lib -L/win32dev/misc/wget/out64/lib
    -lmetalink -lunistring -liconv -L/win32dev/misc/wget/out64/lib
    -lpcre2-8 -lidn2 -L/win32dev/misc/wget/out64/lib -lssl -lcrypto
    -L/win32dev/misc/wget/out64/lib -lz -L/win32dev/misc/wget/out64/lib
    -lpsl -lws2_32 -lunistring -lws2_32 -lole32 -lcrypt32 -lexpat
    -L/win32dev/misc/wget/out64/lib -lgpgme -lassuan -lws2_32
    -lgpg-error ftp-opie.o mswindows.o openssl.o http-ntlm.o
    ../lib/libgnu.a -lws2_32 -lws2_32 -lws2_32 -lws2_32 -lws2_32
    /win32dev/misc/wget/out64/lib/libiconv.a
    /win32dev/misc/wget/out64/lib/libunistring.a -lws2_32

Copyright (C) 2015 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
<http://www.gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Originally written by Hrvoje Niksic <hniksic@xemacs.org>.
Please send bug reports and questions to <bug-wget@gnu.org>.


When I tried the EOSDIS incantation it gave a prompt for the password, but didn't appear to take the input so gave html for the EOSDIS registration page.   To avoid entering the password, create a file "eosdis.wgetrc" with the user name and password:
user=<EOSDIS user>
password=<EOSDIS password>


Then, omitting the user and password from the previous command line and setting WGETRC the download succeeds:
C:UsersseadasDesktop>set WGETRC=C:Usersseadaseosdis.wgetrc
C:UsersseadasDesktop>wget --load-cookies C:Usersseadas.urs_cookies --save-cookies C:Usersseadas.urs_cookies --auth-no-challenge=on --keep-session-cookies --content-disposition https://oceandata.sci.gsfc.nasa.gov/hico/getfile/H2014252205906.L2_ISS_OC.nc
--2019-05-26 06:07:05--  https://oceandata.sci.gsfc.nasa.gov/hico/getfile/H2014252205906.L2_ISS_OC.nc
Resolving oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)... xx.xxx.xx.xx
Connecting to oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)|xx.xxx.xx.xx|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://urs.earthdata.nasa.gov/oauth/authorize?redirect_uri=https%3A%2F%2Foceandata.sci.gsfc.nasa.gov%2Fhico%2Fgetfile%2Frestrict&response_type=code&client_id=Nwx7SywSELFPjkOk6Ifsjw [following]
--2019-05-26 06:07:05--  https://urs.earthdata.nasa.gov/oauth/authorize?redirect_uri=https%3A%2F%2Foceandata.sci.gsfc.nasa.gov%2Fhico%2Fgetfile%2Frestrict&response_type=code&client_id=Nwx7SywSELFPjkOk6Ifsjw
Resolving urs.earthdata.nasa.gov (urs.earthdata.nasa.gov)... xx.xxx.xx.xx
Connecting to urs.earthdata.nasa.gov (urs.earthdata.nasa.gov)|xx.xxx.xx.xx|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://oceandata.sci.gsfc.nasa.gov/hico/getfile/restrict?code=390b6a1a3881dc2a0c3574e0eff6307ad92e71dd4d3d99efc35cabb90c00adec [following]
--2019-05-26 06:07:06--  https://oceandata.sci.gsfc.nasa.gov/hico/getfile/restrict?code=390b6a1a3881dc2a0c3574e0eff6307ad92e71dd4d3d99efc35cabb90c00adec
Connecting to oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)|xx.xxx.xx.xx|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: /hico/getfile/H2014252205906.L2_ISS_OC.nc [following]
--2019-05-26 06:07:06--  https://oceandata.sci.gsfc.nasa.gov/hico/getfile/H2014252205906.L2_ISS_OC.nc
Reusing existing connection to oceandata.sci.gsfc.nasa.gov:443.
HTTP request sent, awaiting response... 200 OK
Length: 50612245 (48M) [application/octet-stream]
Saving to: 'H2014252205906.L2_ISS_OC.nc'

H2014252205906.L2_ISS_OC.nc   100%[=================================================>]  48.27M  1.82MB/s    in 11s

2019-05-26 06:08:19 (4.59 MB/s) - 'H2014252205906.L2_ISS_OC.nc' saved [50612245/50612245]


This version of wget "worked for me" on both Windows 7 and Windows 10.