getanc socket.timeout

Use this Forum to find information on, or ask a question about, NASA Earth Science data.
Post Reply
oo_processing
Posts: 301
Joined: Wed Apr 06, 2005 12:11 pm America/New_York
Answers: 0
Has thanked: 5 times

getanc socket.timeout

by oo_processing » Fri Dec 02, 2022 9:01 am America/New_York

Dear Support,

Although it's occasional, this error is disturbing. Excuse me to copy & paste all the error output here, ocssw version is V2022.2, please advice the possible reason:
$ getanc --disable-download -m 'olcis3a' -s '2022335095607' -e '2022335095607' --ancdb='/tmp/junk.db'
Traceback (most recent call last):
File "/usr/lib64/python3.6/site-packages/urllib3/connectionpool.py", line 445, in _make_request
six.raise_from(e, None)
File "<string>", line 3, in raise_from
File "/usr/lib64/python3.6/site-packages/urllib3/connectionpool.py", line 440, in _make_request
httplib_response = conn.getresponse()
File "/usr/lib64/python3.6/http/client.py", line 1346, in getresponse
response.begin()
File "/usr/lib64/python3.6/http/client.py", line 307, in begin
version, status, reason = self._read_status()
File "/usr/lib64/python3.6/http/client.py", line 268, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "/usr/lib64/python3.6/socket.py", line 586, in readinto
return self._sock.recv_into(b)
File "/usr/lib64/python3.6/ssl.py", line 971, in recv_into
return self.read(nbytes, buffer)
File "/usr/lib64/python3.6/ssl.py", line 833, in read
return self._sslobj.read(len, buffer)
File "/usr/lib64/python3.6/ssl.py", line 590, in read
v = self._sslobj.read(len, buffer)
socket.timeout: The read operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/lib64/python3.6/site-packages/urllib3/connectionpool.py", line 706, in urlopen
chunked=chunked,
File "/usr/lib64/python3.6/site-packages/urllib3/connectionpool.py", line 447, in _make_request
self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
File "/usr/lib64/python3.6/site-packages/urllib3/connectionpool.py", line 337, in _raise_timeout
self, url, "Read timed out. (read timeout=%s)" % timeout_value
urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='oceandata.sci.gsfc.nasa.gov', port=443): Read timed out. (read timeout=10.0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/lib64/python3.6/site-packages/requests/adapters.py", line 449, in send
timeout=timeout
File "/usr/lib64/python3.6/site-packages/urllib3/connectionpool.py", line 796, in urlopen
**response_kw
File "/usr/lib64/python3.6/site-packages/urllib3/connectionpool.py", line 796, in urlopen
**response_kw
File "/usr/lib64/python3.6/site-packages/urllib3/connectionpool.py", line 796, in urlopen
**response_kw
[Previous line repeated 2 more times]
File "/usr/lib64/python3.6/site-packages/urllib3/connectionpool.py", line 756, in urlopen
method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
File "/usr/lib64/python3.6/site-packages/urllib3/util/retry.py", line 574, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='oceandata.sci.gsfc.nasa.gov', port=443): Max retries exceeded with url: /api/anc_data_api/?&m=29&s=2022-12-01T09:56:07&e=2022-12-01T09:56:07&missing_tags=1 (Caused by ReadTimeoutError("HTTPSConnectionPool(host='oceandata.sci.gsfc.nasa.gov', port=443): Read timed out. (read timeout=10.0)",))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/shares/cms_optics/apps/seadas/ocssw/V2022.2/bin/getanc", line 149, in <module>
exit(main())
File "/shares/cms_optics/apps/seadas/ocssw/V2022.2/bin/getanc", line 142, in main
g.findweb()
File "/shares/cms_optics/apps/seadas/ocssw/V2022.2/bin/seadasutils/anc_utils.py", line 526, in findweb
verbose=self.verbose
File "/shares/cms_optics/apps/seadas/ocssw/V2022.2/bin/seadasutils/ProcUtils.py", line 105, in httpdl
with obpgSession.get(urlStr, stream=True, timeout=timeout, headers=headers) as req:
File "/usr/lib64/python3.6/site-packages/requests/sessions.py", line 555, in get
return self.request('GET', url, **kwargs)
File "/usr/lib64/python3.6/site-packages/requests/sessions.py", line 542, in request
resp = self.send(prep, **send_kwargs)
File "/usr/lib64/python3.6/site-packages/requests/sessions.py", line 655, in send
r = adapter.send(request, **kwargs)
File "/usr/lib64/python3.6/site-packages/requests/adapters.py", line 516, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='oceandata.sci.gsfc.nasa.gov', port=443): Max retries exceeded with url: /api/anc_data_api/?&m=29&s=2022-12-01T09:56:07&e=2022-12-01T09:56:07&missing_tags=1 (Caused by ReadTimeoutError("HTTPSConnectionPool(host='oceandata.sci.gsfc.nasa.gov', port=443): Read timed out. (read timeout=10.0)",))

-Yuyuan

Tags:

OB.DAAC - SeanBailey
User Services
User Services
Posts: 1464
Joined: Wed Sep 18, 2019 6:15 pm America/New_York
Answers: 1
Been thanked: 4 times

Re: getanc socket.timeout

by OB.DAAC - SeanBailey » Fri Dec 02, 2022 9:32 am America/New_York

In network traffic, timeouts happen. There is no specific reason we can point to without watching both ends of the connection during the transaction. The client (in this case getanc) didn't get a response from the server within 10 seconds and so it bailed. Our web log shows that our server closed the connection because it didn't get a response from the client.
So, something went awry with the packets between us. There is nothing to do to "fix" this situation, except that you should try again when this situation arises. If it is a consistent and persistent problem, then it'll be obvious (and may be the result of hardware failures).

Sean

oo_processing
Posts: 301
Joined: Wed Apr 06, 2005 12:11 pm America/New_York
Answers: 0
Has thanked: 5 times

Re: getanc socket.timeout

by oo_processing » Fri Dec 02, 2022 9:40 am America/New_York

Thanks Sean,

I will keep monitoring the activities on my end.

-Yuyuan

OB.DAAC - SeanBailey
User Services
User Services
Posts: 1464
Joined: Wed Sep 18, 2019 6:15 pm America/New_York
Answers: 1
Been thanked: 4 times

Re: getanc socket.timeout

by OB.DAAC - SeanBailey » Fri Dec 02, 2022 11:19 am America/New_York

Just a quick follow up...I asked our network folks to take a peek and it seems the "client closed connection" issue with getanc requests over the past few days is predominately from your IP address. Today was particularly egregious with about 10% of your requests affected...but your requests were only 10% of the total getanc requests we've served - while your IP address wasn't the only to report the "client closed connection", it was 75% of the total...one other IP accounted for the remaining 25% (so the total errors seen was only 1% of the total requests made and two IPs accounted for them). This suggests it is not an issue with the getanc service, but something in the air between us and you.

Sean

oo_processing
Posts: 301
Joined: Wed Apr 06, 2005 12:11 pm America/New_York
Answers: 0
Has thanked: 5 times

Re: getanc socket.timeout

by oo_processing » Wed Dec 07, 2022 8:30 am America/New_York

Thanks, Sean.
It seems that error has gone.

Cheers,
Yuyuan

gbourdin
Posts: 34
Joined: Mon Jan 25, 2021 11:59 am America/New_York
Answers: 0
Has thanked: 1 time

Re: getanc socket.timeout

by gbourdin » Mon Mar 27, 2023 9:01 am America/New_York

Hi,
I am having the same error when I query ancillary data for several satellites scenes simultaneously with several instances running in parallel but I never had any problem when I request ancillary data for one scene at a time.
Best,
Guillaume

Post Reply