problem with https

Use this Forum to find information on, or ask a question about, NASA Earth Science data.
Post Reply
melinfr
Posts: 8
Joined: Mon Aug 08, 2005 12:43 pm America/New_York

problem with https

by melinfr » Wed Jan 11, 2017 9:12 am America/New_York

Dear colleagues,
after the switch to https in december we encountered various problems, including in running the updates that could bring solutions - importantly we're operating behind a corporate http proxy. So this post might be related to the post https://oceancolor.gsfc.nasa.gov/forum/oceancolor/topic_show.pl?tid=6431 .
Below is an example with a proposed work-around.

Particularly we encountered problems downloading the 'luts' files.
The error on command "./install_ocssw.py --install-dir=/opt/seadas-7.3.2/ocssw --viirsn"  was:
....
Installing viirsn-luts (10 of 11)
Error! could not establish a network connection. Check your network connection.
If you do not find a problem, please try again later.
Error - Could not install luts for viirsn

To work around this issue we did the following changes to the file /run/scripts/modules/ProcUtils.py :

run/scripts/modules/ProcUtils.py:68
#    if proxy is None:
#        full_request = request
#    else:
#       full_request = ''.join(['http://', url, request])
    if proxy is not None:
        urlConn.set_tunnel(url, 443)

    try:
#       req = urlConn.request('GET', full_request, headers=reqHeaders)
        req = urlConn.request('GET', request, headers=reqHeaders)
    except:
        err_msg = '\n'.join(['Error! could not establish a network connection. Check your network connection.',
                             'If you do not find a problem, please try again later.'])
        sys.exit(err_msg)

With these changes we are able to download all the luts successfully. But of course any feedback is very much welcome.
Cheers,
F.

Tags:

ana_dogliotti
Posts: 28
Joined: Mon Sep 12, 2005 1:16 pm America/New_York

problem with https

by ana_dogliotti » Fri Jan 13, 2017 9:50 am America/New_York

Hello,

I have exactly the same problem, but I tried this work-around, but didn't work for me. I removed the proxy and had the same error...
Could be there any other problem?
Thanks

Ana

OB.DAAC - SeanBailey
User Services
User Services
Posts: 1314
Joined: Wed Sep 18, 2019 6:15 pm America/New_York
Answers: 1
Been thanked: 1 time

problem with https

by OB.DAAC - SeanBailey » Tue Jan 17, 2017 7:10 pm America/New_York

Thanks :smile:
Another code modification was previously proposed and will be included in a soon-to-be-released update to SeaDAS.

Regards,
Sean

mpenalver
Posts: 6
Joined: Thu Mar 07, 2019 7:47 pm America/New_York

problem with https

by mpenalver » Mon Mar 11, 2019 7:25 am America/New_York

The installation script for SeaDAS up to the current version 7.5.3 uses the Python 2 httplib (or Python 3 http.client) library to download ancillary files via HTTPS (or HTTP). First, an HTTPS connection is created to the proxy with httplib.HTTPSConnection, and then a GET request for the relevant file located on the endpoint server is sent through that connection with httplib.HTTPSConnection.request.

In earlier versions of the library, the request had to include the URL of the endpoint, which was passed on to the proxy. However, in the version included in Python 2.7 (and 3.2) the new method httplib.HTTPSConnection.set_tunnel was introduced with which the endpoint must be specified for the connection and doesn't need to be included in every request sent through it. This change has not yet been incorporated into the SeaDAS installation code, which prevents the successful setup of proxied connections. Below, the required modifications to seadas-7.5.3/ocssw/scripts/modulesProcUtils.py.

httpinit:
        #=======================================================================
        # elif proxy.scheme == 'https':
        #     urlConn = hclient.HTTPSConnection(proxy.hostname, proxy.port,
        #                                      timeout=timeout)
        # else:
        #     urlConn = hclient.HTTPConnection(proxy.hostname, proxy.port,
        #                                      timeout=timeout)
        #=======================================================================
        else:
            if proxy.scheme == 'https':
                urlConn = hclient.HTTPSConnection(proxy.hostname, proxy.port,
                                                  timeout=timeout)
            else:
                urlConn = hclient.HTTPConnection(proxy.hostname, proxy.port,
                                                 timeout=timeout)
            urlConn.set_tunnel(url)

_httpdl:
#=======================================================================
#    if proxy is None:
#        full_request = request
#    else:
#        full_request = ''.join(['https://', url, request])
#
#    try:
#        req = urlConn.request('GET', full_request, headers=reqHeaders)
#=======================================================================
    try:
        req = urlConn.request('GET', request, headers=reqHeaders)

OB.DAAC - SeanBailey
User Services
User Services
Posts: 1314
Joined: Wed Sep 18, 2019 6:15 pm America/New_York
Answers: 1
Been thanked: 1 time

problem with https

by OB.DAAC - SeanBailey » Thu Mar 14, 2019 10:38 am America/New_York

Thanks :grin:
Looks like your changes are fine.  We will incorporate them in a future update.

Sean

zhigang
Posts: 72
Joined: Tue Nov 10, 2020 8:03 pm America/New_York

problem with https

by zhigang » Mon Mar 18, 2019 9:04 pm America/New_York

Exactly, the set_tunnel is very useful. I have been using this function to access the https connection.

Zhigang

Post Reply