Earthdata Forum

Hi!

This is my attempt to make hot fix for login problem.

I am not enough good with Python and internet protocols, so i decided to use external python library requests.

We are using Calculate Linux, so i don't know what the exactly package name in Ubuntu, RHL, or other non-gentoo-based distributions.
But i think it also can be installed though pip or pipenv

So this is last version of my fix. You also can donload it from Pastebin

diff --git a/modules/ProcUtils.py b/modules/ProcUtils.py
index edceb41..7fb44f2 100644
--- a/modules/ProcUtils.py
+++ b/modules/ProcUtils.py
@@ -6,7 +6,7 @@ SeaDAS library for commonly used functions within other python scripts
from __future__ import print_function

import sys
-
+import requests

#  ------------------ DANGER -------------------
#
@@ -251,9 +251,50 @@ def _httpdl(url, request, localpath='.', outputfilename=None, ntries=5,
def httpdl(url, request, localpath='.', outputfilename=None, ntries=5,
            uncompress=False, timeout=30., reqHeaders={}, verbose=False,
            reuseConn=False, urlConn=None, chunk_size=DEFAULT_CHUNK_SIZE):
-    urlConn, status = _httpdl(url, request, localpath, outputfilename, ntries,
-                              uncompress, timeout, reqHeaders, verbose,
-                              reuseConn, urlConn, chunk_size)
+
+    import requests
+    import os
+    import re
+
+    if not os.path.exists(localpath):
+        os.umask(0o02)
+        os.makedirs(localpath, mode=0o2775)
+
+    r = requests.get("https://%s%s" % (url,request), stream=True)
+
+    if r.status_code == 200 or r.status_code == 206:
+
+        if outputfilename:
+            ofile = os.path.join(localpath, outputfilename)
+        else:
+            if 'content-disposition' in r.headers:
+                ofile = os.path.join(localpath, r.headers.get('content-disposition').split('filename=')[1] )
+            else:
+                ofile = os.path.join(localpath, os.path.basename(request))
+
+        if r.status_code == 200:
+            f = open(ofile, 'wb')
+        else:
+            f = open(ofile, 'ab')
+
+        for chunk in r.iter_content(chunk_size=chunk_size):
+            if chunk: # filter out keep-alive new chunks
+                f.write(chunk)
+
+        f.flush()
+
+        if re.search(".(Z|gz|bz2)$", ofile) and uncompress:
+            compressStatus = uncompressFile(ofile)
+            if compressStatus:
+                status = compressStatus
+        else:
+            status = 0
+    else:
+        status = r.status_code
+
+#    urlConn, status = _httpdl(url, request, localpath, outputfilename, ntries,
+#                              uncompress, timeout, reqHeaders, verbose,
+#                              reuseConn, urlConn, chunk_size)
     if reuseConn:
         return (urlConn, status)
     else:


How to use it:
1. Download it (or copy from this post) into some file. For example /tmp/seadas.diff
2. You should have installed GNU patch utility in your system.
3. Use command patch -d ${OCSSWROOT}/scripts -p1 </tmp/seadas.diff
4. ...
5. PROFIT!

In additional you can create local branch in git to prevent future conflicts with updates.

Any comments or improvements are welcome.

UPD. Made some changes for fix `UnboundLocalError: local variable 'status' referenced before assignment`
See in attachment.
attachment 1

attachment 2

Dear Pavel, thanks for the fix. Good job. But I am still facing some problem..

For my Ubuntu system I installed pip & requests by
sudo apt-get install pip
sudo apt-get install python3-pip
sudo easy_install requests

Then

I modified ProcUtils.py as you mentioned
------------------------------------------------------------------------------
#  ------------------ DANGER -------------------
# See comment above
def httpdl(url, request, localpath='.', outputfilename=None, ntries=5,
           uncompress=False, timeout=30., reqHeaders={}, verbose=False,
           reuseConn=False, urlConn=None, chunk_size=DEFAULT_CHUNK_SIZE):
    import requests
    import os
    import re

    if not os.path.exists(localpath):
        os.umask(0o02)
        os.makedirs(localpath, mode=0o2775)

    r = requests.get("https://%s%s" % (url,request), stream=True)

    if r.status_code == 200 or r.status_code == 206:

        if outputfilename:
            ofile = os.path.join(localpath, outputfilename)
        else:
            if 'content-disposition' in r.headers:
                ofile = os.path.join(localpath, r.headers.get('content-disposition').split('filename=')[1] )
            else:
                ofile = os.path.join(localpath, os.path.basename(request))

        if r.status_code == 200:
            f = open(ofile, 'wb')
        else:
            f = open(ofile, 'ab')

        for chunk in r.iter_content(chunk_size=chunk_size):
            if chunk: # filter out keep-alive new chunks
                f.write(chunk)

        f.flush()

        if re.search(".(Z|gz|bz2)$", ofile) and uncompress:
            compressStatus = uncompressFile(ofile)
            if compressStatus:
                status = compressStatus
        else:
            status = 0
    else:
        status = r.status_code

#    urlConn, status = _httpdl(url, request, localpath, outputfilename, ntries,
#                              uncompress, timeout, reqHeaders, verbose,
#                              reuseConn, urlConn, chunk_size)

    if reuseConn:
        return (urlConn, status)
    else:
        return status
------------------------------------------------------------------------------

Got this problem

Searching database: /home/o-mc/SeaDAS7.5/ocssw/var/ancillary_data.db
Determining pass start and end times...
Aqua
()
Input file: A2020021062500.L1B_LAC
Sensor    : aqua
Start time: 2020021062500
End time  : 2020021063000
()
Downloading 'N202002112_MET_NCEP_6h.hdf' to /home/o-mc/SeaDAS7.5/ocssw/var/anc/2020/021
Downloading 'N202002106_MET_NCEP_6h.hdf' to /home/o-mc/SeaDAS7.5/ocssw/var/anc/2020/021
Downloading 'N202002000_O3_AURAOMI_24h.hdf' to /home/o-mc/SeaDAS7.5/ocssw/var/anc/2020/020
Downloading 'N2020020_SST_OIV2AV_24h.nc' to /home/o-mc/SeaDAS7.5/ocssw/var/anc/2020/020
Traceback (most recent call last):
  File "/home/o-mc/SeaDAS7.5/ocssw/scripts/getanc.py", line 196, in <module>
    exit(main())
  File "/home/o-mc/SeaDAS7.5/ocssw/scripts/getanc.py", line 190, in main
    g.locate(forcedl=force)
  File "/home/o-mc/SeaDAS7.5/ocssw/scripts/modules/anc_utils.py", line 623, in locate
    reuseConn=True, urlConn=urlConn, verbose=self.verbose)
  File "/home/o-mc/SeaDAS7.5/ocssw/scripts/modules/ProcUtils.py", line 298, in httpdl
    return (urlConn, status)
UnboundLocalError: local variable 'status' referenced before assignment
-E- clo_readFile: Can't open parameter file - A2020021062500.L1B_LAC.anc
-E- clo_readFile: Can't open parameter file - A2020021062500.L1B_LAC.anc

Finished Processing L1B to L2 for file: A2020021062500.L1A_LAC at: 22-Jan-2020 17:37:35

And when I remove # from  urlConn i.e.
urlConn, status = _httpdl(url, request, localpath, outputfilename, ntries,
                              uncompress, timeout, reqHeaders, verbose,
                              reuseConn, urlConn, chunk_size)

Then I got this problem

o-mc@o-mc:~/Desktop/Test_Anc_Dload/Patch/021/L1$ /home/o-mc/SeaDAS7.5/ocssw/scripts/getanc.py A2020021062500.L1B_LAC -v
Searching database: /home/o-mc/SeaDAS7.5/ocssw/var/ancillary_data.db
Determining pass start and end times...
Aqua
()
Input file: A2020021062500.L1B_LAC
Sensor    : aqua
Start time: 2020021062500
End time  : 2020021063000
()
  Found: /home/o-mc/SeaDAS7.5/ocssw/var/anc/2020/021/N202002112_MET_NCEP_6h.hdf
  Found: /home/o-mc/SeaDAS7.5/ocssw/var/anc/2020/021/N202002106_MET_NCEP_6h.hdf
  Found: /home/o-mc/SeaDAS7.5/ocssw/var/anc/2020/020/N202002000_O3_AURAOMI_24h.hdf
  Found: /home/o-mc/SeaDAS7.5/ocssw/var/anc/2020/020/N2020020_SST_OIV2AV_24h.nc
Downloading 'N202002000_SEAICE_NSIDC_24h.hdf' to /home/o-mc/SeaDAS7.5/ocssw/var/anc/2020/020
Connection interrupted, retrying up to 5 more time(s)
Connection interrupted, retrying up to 4 more time(s)
Connection interrupted, retrying up to 3 more time(s)
Connection interrupted, retrying up to 2 more time(s)
Connection interrupted, retrying up to 1 more time(s)
We failed to reach a server.
Please retry this request at a later time.
URL attempted: oceandata.sci.gsfc.nasa.gov
HTTP Error: 302 - Found
*** ERROR: The HTTP transfer failed with status code 302.
*** Please check your network connection and for the existence of the remote file:
*** oceandata.sci.gsfc.nasa.gov/cgi/getfile/N202002000_SEAICE_NSIDC_24h.hdf
***
*** Also check to make sure you have write permissions under the directory:
*** /home/o-mc/SeaDAS7.5/ocssw/var/anc/2020/020
()

I guess you can figure out what went wrong. I am not fluent in python yet. :confused:

I ran into the below issue when trying to batch download many files.  The hot fix patch would work once, then would get the following error the next file I tried to download:

Traceback (most recent call last):
  File "./Yukon_MODIS_l2gen.py", line 266, in <module>
    raise Exception('Error in getanc.py call: {:}'.format(s4.stderr))
Exception: Error in getanc.py call: Traceback (most recent call last):
  File "/Users/jbclark8/ocssw/scripts/getanc.py", line 196, in <module>
    exit(main())
  File "/Users/jbclark8/ocssw/scripts/getanc.py", line 190, in main
    g.locate(forcedl=force)
  File "/Users/jbclark8/ocssw/scripts/modules/anc_utils.py", line 623, in locate
    reuseConn=True, urlConn=urlConn, verbose=self.verbose)
  File "/Users/jbclark8/ocssw/scripts/modules/ProcUtils.py", line 299, in httpdl
    return (urlConn, status)
UnboundLocalError: local variable 'status' referenced before assignment

This indicated to me that 'status' for some reason on call back wasn't being assigned, so I just gave it the value of 0 in the loop where statement the error was occurring:

298     if reuseConn:
299         status=0
300         return (urlConn, status)
301     else:
302         status=0

It has been working now for the last few hours, so seems to be ok
Be aware though, in some applications the "status" variable may be important, it is unclear to me at this point.  It works for me, but may not work for somebody else.

Thanks to Pavel for the fix

Blake

> And when I remove # from  urlConn i.e.

You should not uncomment this block.
This is old code, i just forgot to remove it.

About previous error `UnboundLocalError: local variable 'status' referenced before assignment`
This is very strange, because `status` variable should be initialized in any case.

Anyway, I made some changes and attached both the patch and the corrected file ProcUtils.py.
See update on start message.

This is very strange that `status` has not been initialized. May be after applying patch formatting was broken and behavior of function changed?

Anyway, your changes are bad.
Because in your case function always return 0 as status code.

Right solution is to initialize variable `status` at start of function, right after `import re`.
See update on start post.

I attached new variant of patch and patched file.

Thank You Pavel.. The new script works like charm..:grin: