wget only downloading partial file

Use this Forum to find information on, or ask a question about, NASA Earth Science data.
ermill99
Posts: 19
Joined: Tue Feb 11, 2020 10:00 am America/New_York
Answers: 0

wget only downloading partial file

by ermill99 » Sun Feb 16, 2020 11:15 pm America/New_York

I'm using WGET to download L2 files to my server.  The files are showing up but seem to be capped at 13 kb.  Any idea why?

WGET Command
wget -P ~/www/saltwx.com/public_html/MODIS/sst/ --user=my-user--password='my-pw' --auth-no-challenge=on "https://oceandata.sci.gsfc.nasa.gov/ob/getfile/A2020046181000.L2_LAC.S3470.nc.Southeast.sst.png"

Result

--2020-02-17 02:51:01--  https://oceandata.sci.gsfc.nasa.gov/ob/getfile/A2020046181000.L2_LAC.S3470.nc.Southeast.sst.png
Resolving oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)... xx.xxx.xx.xx, 2001:4d0:2418:128::84
Connecting to oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)|xx.xxx.xx.xx|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://urs.earthdata.nasa.gov/oauth/authorize?redirect_uri=https%3A%2F%2Foceandata.sci.gsfc.nasa.gov%2Fob%2Fgetfile%2Frestrict&response_type=code&client_id=Z0u-MdLNypXBjiDREZ3roA [following]
--2020-02-17 02:51:01--  https://urs.earthdata.nasa.gov/oauth/authorize?redirect_uri=https%3A%2F%2Foceandata.sci.gsfc.nasa.gov%2Fob%2Fgetfile%2Frestrict&response_type=code&client_id=Z0u-MdLNypXBjiDREZ3roA
Resolving urs.earthdata.nasa.gov (urs.earthdata.nasa.gov)... xx.xxx.xx.xx, 2001:4d0:241a:4081::89
Connecting to urs.earthdata.nasa.gov (urs.earthdata.nasa.gov)|xx.xxx.xx.xx|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: '/home/my_key/www/saltwx.com/public_html/MODIS/sst/A2020046181000.L2_LAC.S3470.nc.Southeast.sst.png'

     0K .......... ..                                           381K=0.03s

2020-02-17 02:51:01 (381 KB/s) - '/home/my_key/www/saltwx.com/public_html/MODIS/sst/A2020046181000.L2_LAC.S3470.nc.Southeast.sst.png'
saved [12671] <----This appears to be the total file size downloaded.

Tags:

OB ODPS - jgwilding
Subject Matter Expert
Subject Matter Expert
Posts: 139
Joined: Fri Feb 19, 2021 1:09 pm America/New_York
Answers: 0
Been thanked: 1 time

wget only downloading partial file

by OB ODPS - jgwilding » Mon Feb 17, 2020 9:45 am America/New_York

This usually indicates that an HTML file was downloaded.  If you're on a Linux platform, you can try the 'file' command on the saved file.  If it is HTML, you should be able to view it with a text viewer or in a browser.  A lot of times this ends up being the Earthdata login page and might mean your login credentials are not being accepted.

john

gnwiii
Posts: 713
Joined: Fri Jan 29, 2021 5:51 pm America/New_York
Answers: 2
Has thanked: 1 time

wget only downloading partial file

by gnwiii » Mon Feb 17, 2020 9:54 am America/New_York

Rather than a partial download you may getting an html document you can view in a browser.   Try adding the wget --adjust-extension option.   You may need the --load-cookies ~/.urs_cookies --save-cookies ~/.urs_cookies --keep-session-cookies options.

ermill99
Posts: 19
Joined: Tue Feb 11, 2020 10:00 am America/New_York
Answers: 0

wget only downloading partial file

by ermill99 » Mon Feb 17, 2020 11:55 am America/New_York

So, like this?

wget -P ~/www/saltwx.com/public_html/MODIS/sst/ --adjust-extension --user=user--password='password' --auth-no-challenge=on "https://oceandata.sci.gsfc.nasa.gov/ob/getfile/A2020046181000.L2_LAC.S3470.nc.Southeast.sst.png"

OR

wget -P ~/www/saltwx.com/public_html/MODIS/sst/ --adjust-extension --load-cookies ~/.urs_cookies --save-cookies ~/.urs_cookies --keep-session-cookies "https://oceandata.sci.gsfc.nasa.gov/ob/getfile/A2020046181000.L2_LAC.S3470.nc.Southeast.sst.png"

Postscript: I able was to open the file and it is indeed the login page.  I'm beginning to feel that there is no way to simply download a PNG file with WGET.

ermill99
Posts: 19
Joined: Tue Feb 11, 2020 10:00 am America/New_York
Answers: 0

wget only downloading partial file

by ermill99 » Mon Feb 17, 2020 1:35 pm America/New_York

I've tried all suggestions and just cannot get the single file I would like, to actually download.   

Are there any other suggestions to how to get this file : A2020046181000.L2_LAC.S3470.nc.Southeast.sst.png to download non-interactively?

All that I am getting is the login page html downloaded to my server.

gnwiii
Posts: 713
Joined: Fri Jan 29, 2021 5:51 pm America/New_York
Answers: 2
Has thanked: 1 time

wget only downloading partial file

by gnwiii » Mon Feb 17, 2020 1:44 pm America/New_York

Wget (as well as wget2) also fails for me, but downloads succeed using a browser or the following curl incantation (from Download Methods):

curl -O -b ~/.urs_cookies -c ~/.urs_cookies -L -n https://oceandata.sci.gsfc.nasa.gov/ob/getfile/A2020046181000.L2_LAC.S3470.nc.Southeast.sst.png

ermill99
Posts: 19
Joined: Tue Feb 11, 2020 10:00 am America/New_York
Answers: 0

wget only downloading partial file

by ermill99 » Mon Feb 17, 2020 2:02 pm America/New_York

here are the results of both methods and there appears to be no difference in outcome.  Still only getting the HTML file

1
Command
wget -P ~/www/saltwx.com/public_html/MODIS/sst/ --user=user--password='PW' --auth-no-challenge=on "https://oceandata.sci.gsfc.nasa.gov/ob/getfile/A2020046181000.L2_LAC.S3470.nc.Southeast.sst.png"

Result
--2020-02-17 18:50:01--  https://oceandata.sci.gsfc.nasa.gov/ob/getfile/A2020046181000.L2_LAC.S3470.nc.Southeast.sst.png
Resolving oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)... xx.xxx.xx.xx, 2001:4d0:2418:128::84
Connecting to oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)|xx.xxx.xx.xx|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://urs.earthdata.nasa.gov/oauth/authorize?redirect_uri=https%3A%2F%2Foceandata.sci.gsfc.nasa.gov%2Fob%2Fgetfile%2Frestrict&response_type=code&client_id=Z0u-MdLNypXBjiDREZ3roA&required_scope=study_area,country [following]
--2020-02-17 18:50:02--  https://urs.earthdata.nasa.gov/oauth/authorize?redirect_uri=https%3A%2F%2Foceandata.sci.gsfc.nasa.gov%2Fob%2Fgetfile%2Frestrict&response_type=code&client_id=Z0u-MdLNypXBjiDREZ3roA&required_scope=study_area,country
Resolving urs.earthdata.nasa.gov (urs.earthdata.nasa.gov)... xx.xxx.xx.xx, 2001:4d0:241a:4081::89
Connecting to urs.earthdata.nasa.gov (urs.earthdata.nasa.gov)|xx.xxx.xx.xx|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: '/home/u250-zd3apyn5m53e/www/saltwx.com/public_html/MODIS/sst/A2020046181000.L2_LAC.S3470.nc.Southeast.sst.png.2'

     0K .......... ..                                          41.1M=0s

2020-02-17 18:50:02 (41.1 MB/s) - '/home/u250-zd3apyn5m53e/www/saltwx.com/public_html/MODIS/sst/A2020046181000.L2_LAC.S3470.nc.Southeast.sst.png.2' saved [12859]

2
Command
wget -P ~/www/saltwx.com/public_html/MODIS/sst/ --user=user--password='pw' --load-cookies ~/.urs_cookies --save-cookies ~/.urs_cookies --keep-session-cookies --auth-no-challenge=on "https://oceandata.sci.gsfc.nasa.gov/ob/getfile/A2020046181000.L2_LAC.S3470.nc.Southeast.sst.png"

Result
--2020-02-17 18:40:01--  https://oceandata.sci.gsfc.nasa.gov/ob/getfile/A2020046181000.L2_LAC.S3470.nc.Southeast.sst.png
Resolving oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)... xx.xxx.xx.xx, 2001:4d0:2418:128::84
Connecting to oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)|xx.xxx.xx.xx|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://urs.earthdata.nasa.gov/oauth/authorize?redirect_uri=https%3A%2F%2Foceandata.sci.gsfc.nasa.gov%2Fob%2Fgetfile%2Frestrict&response_type=code&client_id=Z0u-MdLNypXBjiDREZ3roA&required_scope=study_area,country [following]
--2020-02-17 18:40:08--  https://urs.earthdata.nasa.gov/oauth/authorize?redirect_uri=https%3A%2F%2Foceandata.sci.gsfc.nasa.gov%2Fob%2Fgetfile%2Frestrict&response_type=code&client_id=Z0u-MdLNypXBjiDREZ3roA&required_scope=study_area,country
Resolving urs.earthdata.nasa.gov (urs.earthdata.nasa.gov)... xx.xxx.xx.xx, 2001:4d0:241a:4081::89
Connecting to urs.earthdata.nasa.gov (urs.earthdata.nasa.gov)|xx.xxx.xx.xx|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: '/home/u250-zd3apyn5m53e/www/saltwx.com/public_html/MODIS/sst/A2020046181000.L2_LAC.S3470.nc.Southeast.sst.png.1'

     0K .......... ..                                          56.3M=0s

2020-02-17 18:40:08 (56.3 MB/s) - '/home/u250-zd3apyn5m53e/www/saltwx.com/public_html/MODIS/sst/A2020046181000.L2_LAC.S3470.nc.Southeast.sst.png.1' saved [12859]

ermill99
Posts: 19
Joined: Tue Feb 11, 2020 10:00 am America/New_York
Answers: 0

wget only downloading partial file

by ermill99 » Mon Feb 17, 2020 2:37 pm America/New_York

OK, the cURL seemed to download but I think it is still only delivering the HTML page.

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0

100 12859    0 12859    0     0  38157      0 --:--:-- --:--:-- --:--:-- 38157

ermill99
Posts: 19
Joined: Tue Feb 11, 2020 10:00 am America/New_York
Answers: 0

wget only downloading partial file

by ermill99 » Mon Feb 17, 2020 3:04 pm America/New_York

Confirmed that each method, WGET and cURL, result in a 12.859 KB file download. 

This authentication process is ridiculous.

OB.DAAC - SeanBailey
User Services
User Services
Posts: 1464
Joined: Wed Sep 18, 2019 6:15 pm America/New_York
Answers: 1
Been thanked: 5 times

wget only downloading partial file

by OB.DAAC - SeanBailey » Mon Feb 17, 2020 3:32 pm America/New_York

The cURL method requires a .netrc file (wget can also use this file, but you will need the --auth-no-challenge=on option) that contains your login Earthdata credentials.
Verify you have one, and it is correct, or generate one ala:

echo "machine urs.earthdata.nasa.gov login USERNAME password PASSWD" > ~/.netrc
chmod  0600 ~/.netrc


https://oceancolor.gsfc.nasa.gov/data/download_methods/ describes the process
Sean

Post Reply