HotFix for SeaDAS 7.5 login problem

Use this Forum to find information on, or ask a question about, NASA Earth Science data.
Post Reply
pavel_babyak
Posts: 12
Joined: Wed Sep 04, 2019 10:46 am America/New_York
Answers: 0

HotFix for SeaDAS 7.5 login problem

by pavel_babyak » Wed Jan 22, 2020 1:24 am America/New_York

Hi!

This is my attempt to make hot fix for login problem.

I am not enough good with Python and internet protocols, so i decided to use external python library requests.

We are using Calculate Linux, so i don't know what the exactly package name in Ubuntu, RHL, or other non-gentoo-based distributions.
But i think it also can be installed though pip or pipenv

So this is last version of my fix. You also can donload it from Pastebin

diff --git a/modules/ProcUtils.py b/modules/ProcUtils.py
index edceb41..7fb44f2 100644
--- a/modules/ProcUtils.py
+++ b/modules/ProcUtils.py
@@ -6,7 +6,7 @@ SeaDAS library for commonly used functions within other python scripts
from __future__ import print_function

import sys
-
+import requests

#  ------------------ DANGER -------------------
#
@@ -251,9 +251,50 @@ def _httpdl(url, request, localpath='.', outputfilename=None, ntries=5,
def httpdl(url, request, localpath='.', outputfilename=None, ntries=5,
            uncompress=False, timeout=30., reqHeaders={}, verbose=False,
            reuseConn=False, urlConn=None, chunk_size=DEFAULT_CHUNK_SIZE):
-    urlConn, status = _httpdl(url, request, localpath, outputfilename, ntries,
-                              uncompress, timeout, reqHeaders, verbose,
-                              reuseConn, urlConn, chunk_size)
+
+    import requests
+    import os
+    import re
+
+    if not os.path.exists(localpath):
+        os.umask(0o02)
+        os.makedirs(localpath, mode=0o2775)
+
+    r = requests.get("https://%s%s" % (url,request), stream=True)
+
+    if r.status_code == 200 or r.status_code == 206:
+
+        if outputfilename:
+            ofile = os.path.join(localpath, outputfilename)
+        else:
+            if 'content-disposition' in r.headers:
+                ofile = os.path.join(localpath, r.headers.get('content-disposition').split('filename=')[1] )
+            else:
+                ofile = os.path.join(localpath, os.path.basename(request))
+
+        if r.status_code == 200:
+            f = open(ofile, 'wb')
+        else:
+            f = open(ofile, 'ab')
+
+        for chunk in r.iter_content(chunk_size=chunk_size):
+            if chunk: # filter out keep-alive new chunks
+                f.write(chunk)
+
+        f.flush()
+
+        if re.search(".(Z|gz|bz2)$", ofile) and uncompress:
+            compressStatus = uncompressFile(ofile)
+            if compressStatus:
+                status = compressStatus
+        else:
+            status = 0
+    else:
+        status = r.status_code
+
+#    urlConn, status = _httpdl(url, request, localpath, outputfilename, ntries,
+#                              uncompress, timeout, reqHeaders, verbose,
+#                              reuseConn, urlConn, chunk_size)
     if reuseConn:
         return (urlConn, status)
     else:


How to use it:
1. Download it (or copy from this post) into some file. For example /tmp/seadas.diff
2. You should have installed GNU patch utility in your system.
3. Use command patch -d ${OCSSWROOT}/scripts -p1 </tmp/seadas.diff
4. ...
5. PROFIT!

In additional you can create local branch in git to prevent future conflicts with updates.

Any comments or improvements are welcome.

UPD. Made some changes for fix `UnboundLocalError: local variable 'status' referenced before assignment`
See in attachment.
attachment 1

attachment 2

Filters:

alakes
Posts: 26
Joined: Fri Aug 26, 2016 7:46 am America/New_York
Answers: 0

HotFix for SeaDAS 7.5 login problem

by alakes » Wed Jan 22, 2020 7:33 am America/New_York

Dear Pavel, thanks for the fix. Good job. But I am still facing some problem..

For my Ubuntu system I installed pip & requests by
sudo apt-get install pip
sudo apt-get install python3-pip
sudo easy_install requests

Then

I modified ProcUtils.py as you mentioned
------------------------------------------------------------------------------
#  ------------------ DANGER -------------------
# See comment above
def httpdl(url, request, localpath='.', outputfilename=None, ntries=5,
           uncompress=False, timeout=30., reqHeaders={}, verbose=False,
           reuseConn=False, urlConn=None, chunk_size=DEFAULT_CHUNK_SIZE):
    import requests
    import os
    import re

    if not os.path.exists(localpath):
        os.umask(0o02)
        os.makedirs(localpath, mode=0o2775)

    r = requests.get("https://%s%s" % (url,request), stream=True)

    if r.status_code == 200 or r.status_code == 206:

        if outputfilename:
            ofile = os.path.join(localpath, outputfilename)
        else:
            if 'content-disposition' in r.headers:
                ofile = os.path.join(localpath, r.headers.get('content-disposition').split('filename=')[1] )
            else:
                ofile = os.path.join(localpath, os.path.basename(request))

        if r.status_code == 200:
            f = open(ofile, 'wb')
        else:
            f = open(ofile, 'ab')

        for chunk in r.iter_content(chunk_size=chunk_size):
            if chunk: # filter out keep-alive new chunks
                f.write(chunk)

        f.flush()

        if re.search(".(Z|gz|bz2)$", ofile) and uncompress:
            compressStatus = uncompressFile(ofile)
            if compressStatus:
                status = compressStatus
        else:
            status = 0
    else:
        status = r.status_code

#    urlConn, status = _httpdl(url, request, localpath, outputfilename, ntries,
#                              uncompress, timeout, reqHeaders, verbose,
#                              reuseConn, urlConn, chunk_size)

    if reuseConn:
        return (urlConn, status)
    else:
        return status
------------------------------------------------------------------------------

Got this problem

Searching database: /home/o-mc/SeaDAS7.5/ocssw/var/ancillary_data.db
Determining pass start and end times...
Aqua
()
Input file: A2020021062500.L1B_LAC
Sensor    : aqua
Start time: 2020021062500
End time  : 2020021063000
()
Downloading 'N202002112_MET_NCEP_6h.hdf' to /home/o-mc/SeaDAS7.5/ocssw/var/anc/2020/021
Downloading 'N202002106_MET_NCEP_6h.hdf' to /home/o-mc/SeaDAS7.5/ocssw/var/anc/2020/021
Downloading 'N202002000_O3_AURAOMI_24h.hdf' to /home/o-mc/SeaDAS7.5/ocssw/var/anc/2020/020
Downloading 'N2020020_SST_OIV2AV_24h.nc' to /home/o-mc/SeaDAS7.5/ocssw/var/anc/2020/020
Traceback (most recent call last):
  File "/home/o-mc/SeaDAS7.5/ocssw/scripts/getanc.py", line 196, in <module>
    exit(main())
  File "/home/o-mc/SeaDAS7.5/ocssw/scripts/getanc.py", line 190, in main
    g.locate(forcedl=force)
  File "/home/o-mc/SeaDAS7.5/ocssw/scripts/modules/anc_utils.py", line 623, in locate
    reuseConn=True, urlConn=urlConn, verbose=self.verbose)
  File "/home/o-mc/SeaDAS7.5/ocssw/scripts/modules/ProcUtils.py", line 298, in httpdl
    return (urlConn, status)
UnboundLocalError: local variable 'status' referenced before assignment
-E- clo_readFile: Can't open parameter file - A2020021062500.L1B_LAC.anc
-E- clo_readFile: Can't open parameter file - A2020021062500.L1B_LAC.anc

Finished Processing L1B to L2 for file: A2020021062500.L1A_LAC at: 22-Jan-2020 17:37:35

And when I remove # from  urlConn i.e.
urlConn, status = _httpdl(url, request, localpath, outputfilename, ntries,
                              uncompress, timeout, reqHeaders, verbose,
                              reuseConn, urlConn, chunk_size)

Then I got this problem

o-mc@o-mc:~/Desktop/Test_Anc_Dload/Patch/021/L1$ /home/o-mc/SeaDAS7.5/ocssw/scripts/getanc.py A2020021062500.L1B_LAC -v
Searching database: /home/o-mc/SeaDAS7.5/ocssw/var/ancillary_data.db
Determining pass start and end times...
Aqua
()
Input file: A2020021062500.L1B_LAC
Sensor    : aqua
Start time: 2020021062500
End time  : 2020021063000
()
  Found: /home/o-mc/SeaDAS7.5/ocssw/var/anc/2020/021/N202002112_MET_NCEP_6h.hdf
  Found: /home/o-mc/SeaDAS7.5/ocssw/var/anc/2020/021/N202002106_MET_NCEP_6h.hdf
  Found: /home/o-mc/SeaDAS7.5/ocssw/var/anc/2020/020/N202002000_O3_AURAOMI_24h.hdf
  Found: /home/o-mc/SeaDAS7.5/ocssw/var/anc/2020/020/N2020020_SST_OIV2AV_24h.nc
Downloading 'N202002000_SEAICE_NSIDC_24h.hdf' to /home/o-mc/SeaDAS7.5/ocssw/var/anc/2020/020
Connection interrupted, retrying up to 5 more time(s)
Connection interrupted, retrying up to 4 more time(s)
Connection interrupted, retrying up to 3 more time(s)
Connection interrupted, retrying up to 2 more time(s)
Connection interrupted, retrying up to 1 more time(s)
We failed to reach a server.
Please retry this request at a later time.
URL attempted: oceandata.sci.gsfc.nasa.gov
HTTP Error: 302 - Found
*** ERROR: The HTTP transfer failed with status code 302.
*** Please check your network connection and for the existence of the remote file:
*** oceandata.sci.gsfc.nasa.gov/cgi/getfile/N202002000_SEAICE_NSIDC_24h.hdf
***
*** Also check to make sure you have write permissions under the directory:
*** /home/o-mc/SeaDAS7.5/ocssw/var/anc/2020/020
()

I guess you can figure out what went wrong. I am not fluent in python yet. :confused:

blakeclark
Posts: 1
Joined: Wed Jan 22, 2020 10:57 am America/New_York
Answers: 0

HotFix for SeaDAS 7.5 login problem

by blakeclark » Wed Jan 22, 2020 11:10 am America/New_York

I ran into the below issue when trying to batch download many files.  The hot fix patch would work once, then would get the following error the next file I tried to download:

Traceback (most recent call last):
  File "./Yukon_MODIS_l2gen.py", line 266, in <module>
    raise Exception('Error in getanc.py call: {:}'.format(s4.stderr))
Exception: Error in getanc.py call: Traceback (most recent call last):
  File "/Users/jbclark8/ocssw/scripts/getanc.py", line 196, in <module>
    exit(main())
  File "/Users/jbclark8/ocssw/scripts/getanc.py", line 190, in main
    g.locate(forcedl=force)
  File "/Users/jbclark8/ocssw/scripts/modules/anc_utils.py", line 623, in locate
    reuseConn=True, urlConn=urlConn, verbose=self.verbose)
  File "/Users/jbclark8/ocssw/scripts/modules/ProcUtils.py", line 299, in httpdl
    return (urlConn, status)
UnboundLocalError: local variable 'status' referenced before assignment

This indicated to me that 'status' for some reason on call back wasn't being assigned, so I just gave it the value of 0 in the loop where statement the error was occurring:

298     if reuseConn:
299         status=0
300         return (urlConn, status)
301     else:
302         status=0

It has been working now for the last few hours, so seems to be ok
Be aware though, in some applications the "status" variable may be important, it is unclear to me at this point.  It works for me, but may not work for somebody else.

Thanks to Pavel for the fix

Blake

pavel_babyak
Posts: 12
Joined: Wed Sep 04, 2019 10:46 am America/New_York
Answers: 0

HotFix for SeaDAS 7.5 login problem

by pavel_babyak » Wed Jan 22, 2020 11:01 pm America/New_York

> And when I remove # from  urlConn i.e.


You should not uncomment this block.
This is old code, i just forgot to remove it.

About previous error `UnboundLocalError: local variable 'status' referenced before assignment`
This is very strange, because `status` variable should be initialized in any case.

Anyway, I made some changes and attached both the patch and the corrected file ProcUtils.py.
See update on start message.

pavel_babyak
Posts: 12
Joined: Wed Sep 04, 2019 10:46 am America/New_York
Answers: 0

HotFix for SeaDAS 7.5 login problem

by pavel_babyak » Wed Jan 22, 2020 11:14 pm America/New_York

This is very strange that `status` has not been initialized. May be after applying patch formatting was broken and behavior of function changed?

Anyway, your changes are bad.
Because in your case function always return 0 as status code.

Right solution is to initialize variable `status` at start of function, right after `import re`.
See update on start post.

I attached new variant of patch and patched file.

alakes
Posts: 26
Joined: Fri Aug 26, 2016 7:46 am America/New_York
Answers: 0

HotFix for SeaDAS 7.5 login problem

by alakes » Thu Jan 23, 2020 2:33 am America/New_York

Thank You Pavel.. The new script works like charm..:grin:

Post Reply