SeaDAS 7.5 ancillary data downloads via getanc.py are failing

[liamgumley]

I have installed a fresh copy of SeaDAS 7.5 on CentOS 7.7.

I am trying to download ancillary data using getanc.py, and it is consistently failing as shown below. I understand that NASA Earthdata login is now required to download data. Is there a way to provide my login credentials to getanc.py (assuming that is the problem)?

Liam.

getanc.py -v -s 2020016150000
Searching database: /home/gumley/dbvm/ocssw_build/tarfiles/test/ocssw/var/ancillary_data.db
()
Input file: None
Sensor    : None
Start time: 2020016150000
End time  : None
()
Downloading 'N202001606_MET_NCEP_6h.hdf' to /home/gumley/dbvm/ocssw_build/tarfiles/test/ocssw/var/anc/2020/016
Connection interrupted, retrying up to 5 more time(s)
Connection interrupted, retrying up to 4 more time(s)
Connection interrupted, retrying up to 3 more time(s)
^CWell, the server did not like this...reports: Found
*** ERROR: The HTTP transfer failed with status code 302.
*** Please check your network connection and for the existence of the remote file:
*** oceandata.sci.gsfc.nasa.gov/cgi/getfile/N202001606_MET_NCEP_6h.hdf
***
*** Also check to make sure you have write permissions under the directory:
*** /home/gumley/dbvm/ocssw_build/tarfiles/test/ocssw/var/anc/2020/016

[OB.DAAC - SeanBailey]

Liam,

Not yet...

Until we get a version that will, a work around would be to call getanc.py with the -d option (--disable-download) to get the list of files and use wget or cURL to get the files.

Sean

[woodbri]

Sean,

I wrote the follow python replacement for getanc.py, rename the original getanc.py to getanc-real.py and put this in getanc.py
I offer it as is, here with public domain license, in the hope that it will help others. (EDIT: If you got an earlier post this has some fixes in it)

#!/usr/bin/env python3

import subprocess
import os.path
import sys
import os

def fetchAndStore(fname):

    (fpath, fn) = os.path.split(fname)
    os.makedirs(fpath, exist_ok=True)

    url = 'https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/' + fn

    wget = ['wget', '-q', '--load-cookies', '~/.urs_cookies', '--save-cookies', '~/.urs_cookies', '--auth-no-challenge=on', '--keep-session-cookies', '--content-disposition', url]

    out = subprocess.run(wget, stdout= subprocess.PIPE, stderr=subprocess.STDOUT)
    if os.path.exists(fn):
        mv = ['mv', fn, fname]
        subprocess.run(mv)
    elif os.path.exists(fn + '.bz2'):
        mv = ['mv', fn + '.bz2', fname + '.bz2']
        subprocess.run(mv)
        unzip = ['bunzip2', fname + '.bz2']
        subprocess.run(unzip)
    else:
        print('ERROR: failed to fetch: ' + url + "\n")
        print(out.stdout.decode('utf-8'))

getanc = '/u/oceandata/scripts/getanc-real.py';

cmd = [getanc, '-d'] + sys.argv[1:]

out = subprocess.run(cmd, stdout= subprocess.PIPE, stderr=subprocess.STDOUT)
lines = out.stdout.decode('utf-8').split()

for line in lines:
    line = line.strip()
    if "icefile=" in line:
        icefile = line[8:]
        print(icefile + "\n")
        if not os.path.exists(icefile):
            fetchAndStore(icefile)
    elif "met" in line:
        metfile = line[5:]
        print(metfile + "\n")
        if not os.path.exists(metfile):
            fetchAndStore(metfile)
    elif "sstfile=" in line:
        sstfile = line[8:]
        print(sstfile + "\n")
        if not os.path.exists(sstfile):
            fetchAndStore(sstfile)