HICO data download

Please enter here to ask a question about any NASA Science related topics!
Post Reply
mkahru
Posts: 14
Joined: Wed Apr 27, 2005 1:20 pm America/New_York

HICO data download

by mkahru » Sat May 18, 2019 12:08 pm America/New_York

I am trying to download HICO level-2 data using a list of file links that I got from the Level1&2 browser but I get files with 0 length.
My scripts work with regular http_manifest.txt files but not for the list I get for HICO.
Can anybody give a script using either Curl or Wget for Windows?
Thanks!
Mati

Tags:

OB WebDev - norman
Subject Matter Expert
Subject Matter Expert
Posts: 136
Joined: Tue Feb 09, 2021 8:19 am America/New_York

HICO data download

by OB WebDev - norman » Mon May 20, 2019 10:39 am America/New_York

Hi Mati,

Have you specifically registered to download HICO data?

https://oceancolor.gsfc.nasa.gov/registration/

If so, can you share a specific URL that is not working for you?

Regards,
Norman

mkahru
Posts: 14
Joined: Wed Apr 27, 2005 1:20 pm America/New_York

HICO data download

by mkahru » Fri May 24, 2019 11:31 am America/New_York

Hi Norman,
A sample link is: https://oceandata.sci.gsfc.nasa.gov/hico/getfile/H2014252205906.L2_ISS_OC.nc
I can download the NC file when I manually copy this address to my browser. I have over 100 or these links and it would take a long time to do it manually.
I cannot use wget or curl to download from a link like that. Can you give an example?
Thanks!

gnwiii
Posts: 612
Joined: Fri Jan 29, 2021 5:51 pm America/New_York
Answers: 1

HICO data download

by gnwiii » Sat May 25, 2019 7:24 am America/New_York

Downloading data from GES DISC has instructions for using wget in windows.

mkahru
Posts: 14
Joined: Wed Apr 27, 2005 1:20 pm America/New_York

HICO data download

by mkahru » Sat May 25, 2019 4:53 pm America/New_York

I am using wget 1.18 and I am downloading all kinds of gibberish (css, js, html, etc) but NOT the needed netCDF files.

gnwiii
Posts: 612
Joined: Fri Jan 29, 2021 5:51 pm America/New_York
Answers: 1

HICO data download

by gnwiii » Sat May 25, 2019 5:48 pm America/New_York

The "gibberish" may provide details for the failure to retrieve your requested file.   It may become readable if you can load it in a browser (maybe rename to .html).   Wget has a number of debugging options that may be helpful.   When NASA moved to https there were problems with wget and curl built with older libraries that didn't handle the redirects or didn't support the (more secure) https configuration NASA is using.   At that time, many of the binaries you could download didn't work.  The current versions of wget from Msys2 or Cygwin should be OK.

Google found 64-bit wget version 1.20.3 zip archives at Windows binaries of GNU Wget

C:UsersseadasDesktop>wget --version
GNU Wget 1.20.3 built on mingw32.

-cares +digest +gpgme +https +ipv6 +iri +large-file +metalink -nls
+ntlm +opie +psl +ssl/openssl

Wgetrc:
    C:Usersgwhiteeosdis.wgetrc (env)
    C:Usersgwhite/.wgetrc (user)
    /win32dev/misc/wget/out64/etc/wgetrc (system)
Compile:
    x86_64-w64-mingw32-gcc -DHAVE_CONFIG_H
    -DSYSTEM_WGETRC="/win32dev/misc/wget/out64/etc/wgetrc"
    -DLOCALEDIR="/win32dev/misc/wget/out64/share/locale" -I. -I../lib
    -I../lib -I/win32dev/misc/wget/out64/include
    -I/win32dev/misc/wget/out64/include
    -I/win32dev/misc/wget/out64/include -DPCRE2_STATIC
    -I/win32dev/misc/wget/out64/include
    -I/win32dev/misc/wget/out64/include -DHAVE_LIBSSL
    -I/win32dev/misc/wget/out64/include
    -I/win32dev/misc/wget/out64/include -DNDEBUG -g -O2
Link:
    x86_64-w64-mingw32-gcc -I/win32dev/misc/wget/out64/include
    -I/win32dev/misc/wget/out64/include -DPCRE2_STATIC
    -I/win32dev/misc/wget/out64/include
    -I/win32dev/misc/wget/out64/include -DHAVE_LIBSSL
    -I/win32dev/misc/wget/out64/include
    -I/win32dev/misc/wget/out64/include -DNDEBUG -g -O2
    -L/win32dev/misc/wget/out64/lib -L/win32dev/misc/wget/out64/lib
    -lmetalink -lunistring -liconv -L/win32dev/misc/wget/out64/lib
    -lpcre2-8 -lidn2 -L/win32dev/misc/wget/out64/lib -lssl -lcrypto
    -L/win32dev/misc/wget/out64/lib -lz -L/win32dev/misc/wget/out64/lib
    -lpsl -lws2_32 -lunistring -lws2_32 -lole32 -lcrypt32 -lexpat
    -L/win32dev/misc/wget/out64/lib -lgpgme -lassuan -lws2_32
    -lgpg-error ftp-opie.o mswindows.o openssl.o http-ntlm.o
    ../lib/libgnu.a -lws2_32 -lws2_32 -lws2_32 -lws2_32 -lws2_32
    /win32dev/misc/wget/out64/lib/libiconv.a
    /win32dev/misc/wget/out64/lib/libunistring.a -lws2_32

Copyright (C) 2015 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
<http://www.gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Originally written by Hrvoje Niksic <hniksic@xemacs.org>.
Please send bug reports and questions to <bug-wget@gnu.org>.


When I tried the EOSDIS incantation it gave a prompt for the password, but didn't appear to take the input so gave html for the EOSDIS registration page.   To avoid entering the password, create a file "eosdis.wgetrc" with the user name and password:
user=<EOSDIS user>
password=<EOSDIS password>


Then, omitting the user and password from the previous command line and setting WGETRC the download succeeds:
C:UsersseadasDesktop>set WGETRC=C:Usersseadaseosdis.wgetrc
C:UsersseadasDesktop>wget --load-cookies C:Usersseadas.urs_cookies --save-cookies C:Usersseadas.urs_cookies --auth-no-challenge=on --keep-session-cookies --content-disposition https://oceandata.sci.gsfc.nasa.gov/hico/getfile/H2014252205906.L2_ISS_OC.nc
--2019-05-26 06:07:05--  https://oceandata.sci.gsfc.nasa.gov/hico/getfile/H2014252205906.L2_ISS_OC.nc
Resolving oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)... xx.xxx.xx.xx
Connecting to oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)|xx.xxx.xx.xx|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://urs.earthdata.nasa.gov/oauth/authorize?redirect_uri=https%3A%2F%2Foceandata.sci.gsfc.nasa.gov%2Fhico%2Fgetfile%2Frestrict&response_type=code&client_id=Nwx7SywSELFPjkOk6Ifsjw [following]
--2019-05-26 06:07:05--  https://urs.earthdata.nasa.gov/oauth/authorize?redirect_uri=https%3A%2F%2Foceandata.sci.gsfc.nasa.gov%2Fhico%2Fgetfile%2Frestrict&response_type=code&client_id=Nwx7SywSELFPjkOk6Ifsjw
Resolving urs.earthdata.nasa.gov (urs.earthdata.nasa.gov)... xx.xxx.xx.xx
Connecting to urs.earthdata.nasa.gov (urs.earthdata.nasa.gov)|xx.xxx.xx.xx|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://oceandata.sci.gsfc.nasa.gov/hico/getfile/restrict?code=390b6a1a3881dc2a0c3574e0eff6307ad92e71dd4d3d99efc35cabb90c00adec [following]
--2019-05-26 06:07:06--  https://oceandata.sci.gsfc.nasa.gov/hico/getfile/restrict?code=390b6a1a3881dc2a0c3574e0eff6307ad92e71dd4d3d99efc35cabb90c00adec
Connecting to oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)|xx.xxx.xx.xx|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: /hico/getfile/H2014252205906.L2_ISS_OC.nc [following]
--2019-05-26 06:07:06--  https://oceandata.sci.gsfc.nasa.gov/hico/getfile/H2014252205906.L2_ISS_OC.nc
Reusing existing connection to oceandata.sci.gsfc.nasa.gov:443.
HTTP request sent, awaiting response... 200 OK
Length: 50612245 (48M) [application/octet-stream]
Saving to: 'H2014252205906.L2_ISS_OC.nc'

H2014252205906.L2_ISS_OC.nc   100%[=================================================>]  48.27M  1.82MB/s    in 11s

2019-05-26 06:08:19 (4.59 MB/s) - 'H2014252205906.L2_ISS_OC.nc' saved [50612245/50612245]


This version of wget "worked for me" on both Windows 7 and Windows 10.

Post Reply