wget only downloading partial file

[ermill99]

Thanks Sean.  Just to clarify I'm doing the following:

1) saving my .netrc file to the my server where I am writing the file to 

2) Running the wget command

wget -P ~/www/saltwx.com/public_html/MODIS/sst/ --load-cookies ~/.urs_cookies --save-cookies ~/.urs_cookies --keep-session-cookies --auth-no-challenge=on "https://oceandata.sci.gsfc.nasa.gov/ob/getfile/A2020046181000.L2_LAC.S3470.nc.Southeast.sst.png"

OR

the cURL command
curl -O -b ~/.urs_cookies -c ~/.urs_cookies -L -n https://oceandata.sci.gsfc.nasa.gov/ob/getfile/A2020046181000.L2_LAC.S3470.nc.Southeast.sst.png

[OB.DAAC - SeanBailey]

Yes, as long as the .netrc file is in the home directory for the account on the server that will be running the wget/curl commands, it should work.
If you still have issues getting it to work,  run the commands with verbose -v or debugging on (-d, --debug for wget , --trace) to see what's going on.

Sean

[gnwiii]

Some software won't use a "~/.netrc" if it has incorrect permissions, so good practice is "chmod 600 ~/.netrc".   If "~/.netrc" was created with an overly helpful editor it may contain Unicode look-alike characters in place of quotes, dashes, etc.  EarthData WIKI:  Troubleshooting Wget mentions a perl script to create a "~/.netrc".

[ermill99]

Do I need ~/.netrc or hmod 600 ~/.netrc in my curl command? 

I am still only getting small size files (27 b) to download to my folder.

I have my .netrc file in the destination folder plus the home directory. Heck, I think I have it every possible folder.  Still no complete file is downloading. 

I am using the following command

curl https://oceandata.sci.gsfc.nasa.gov/ob/getfile/A2020046181000.L2_LAC.S3470.nc.Southeast.sst.png -b ~/.urs_cookies -c ~/.urs_cookies -L -n -o ~/www/saltwx.com/public_html/MODIS/sst/A2020046181000.L2_LAC.S3470.nc.Southeast.sst.png

[OB.DAAC - amscott]

wget -P ~/www/saltwx.com/public_html/MODIS/sst/ --load-cookies ~/.urs_cookies --save-cookies ~/.urs_cookies --keep-session-cookies --auth-no-challenge=on https://oceandata.sci.gsfc.nasa.gov/ob/getfile/A2020046181000.L2_LAC.S3470.nc.Southeast.sst.png

works fine. You need to remove the quotes around the URL.

[OB.DAAC - amscott]

Do I need ~/.netrc or hmod 600 ~/.netrc in my curl command? NO

With the curl command:
curl https://oceandata.sci.gsfc.nasa.gov/ob/getfile/A2020046181000.L2_LAC.S3470.nc.Southeast.sst.png -b ~/.urs_cookies -c ~/.urs_cookies -L -n -o ~/www/saltwx.com/public_html/MODIS/sst/A2020046181000.L2_LAC.S3470.nc.Southeast.sst.png

Create the directory first, then it will put the file there. --create-dirs should work, but it didn't for me.

[ermill99]

I keep getting Username/Password Authentication Failed.  when using Wget

[gnwiii]

"~/.netrc" or "chmod 600 ~/.netrc are not curl options.   The "~" is a bash shorthand for the user's HOME directory.  Curl and wget both check for "~/.netrc".  Problems with "~/.netrc" file will not affect downloads with credentials provided on the command line.   Since the contents are sensitive, you should delete all the other copies.   Both wget and curl have options to provide more details of the rather complicated 9 step download process.  
Try setting verbose output and dumping the http headers to, e.g. /tmp/headers:

curl https://oceandata.sci.gsfc.nasa.gov/ob/getfile/A2020046181000.L2_LAC.S3470.nc.Southeast.sst.png -D /tmp/headers -v -b ~/.urs_cookies -c ~/.urs_cookies -L -n -o ~/www/saltwx.com/public_html/MODIS/sst/A2020046181000.L2_LAC.S3470.nc.Southeast.sst.png

For me, this gives a lot of detail and a successful download (using /tmp in place of the above output directory).  The sequence of headers was:
$ grep '^HTTP' /tmp/headers
HTTP/2 302
HTTP/1.1 302 Found
HTTP/2 302
HTTP/2 200

The output will have messages like:
* Couldn't find host oceandata.sci.gsfc.nasa.gov in the .netrc file; using defaults which you can ignore since the login is done on ura.earthdata.nasa.gov.

If you get something different there is a lot more information in the output, but also some sensitive information.

If you have the OCSSW processing software, you could try getanc.py.   If that works, you could adapt the second python script from  How to Access Data with Python.  This uses python3 and the requests library (generally available as a distro package for linux and may already be installed on your system). You will need to edit the script to provide your credentials and replace the URL with the one you want.  I used this example as the basis for a python script that gets credentials from "~/.netrc".   Unlike wget, it has never failed for me in tests on several platforms (Windows cygwin64, macOS, and 3 linux distros).

[OB.DAAC - SeanBailey]

did you try the suggestions I offered earlier to help diagnose the problem?
https://oceancolor.gsfc.nasa.gov/forum/oceancolor/topic_show.pl?pid=52248#pid52248