weird getanc error for OLCI NT subscriptions only

[oo_processing]

Dear Support,

My cronjobs read MODIS/VIIRS/OLCI subscriptions and download new files staged, then run 'getanc' command to get ancillary files. However, I encountered the weird thing, the cronjobs run very well except for OLCI NT subscriptions. The error message is posted in the end.
But it doesn't make sense, because there was no error for the same subscriptions but of OLCI NR (though NR came about a day before NT), and I'm running 'getanc' command with starttime and endtime provided rather than letting the program to read info from level-1B files. I have tried hard to compare the cronjobs for NT and NR to see if I accidentally set different python or ocssw environments for them, but nothing found as yet. I just wonder if you happen to know the reason.

Thanks,
Yuyuan

Traceback (most recent call last):
File "/usr/lib64/python3.6/site-packages/urllib3/connectionpool.py", line 445, in _make_request
six.raise_from(e, None)
File "<string>", line 3, in raise_from
File "/usr/lib64/python3.6/site-packages/urllib3/connectionpool.py", line 440, in _make_request
httplib_response = conn.getresponse()
File "/usr/lib64/python3.6/http/client.py", line 1346, in getresponse
response.begin()
File "/usr/lib64/python3.6/http/client.py", line 307, in begin
version, status, reason = self._read_status()
File "/usr/lib64/python3.6/http/client.py", line 268, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "/usr/lib64/python3.6/socket.py", line 586, in readinto
return self._sock.recv_into(b)
File "/usr/lib64/python3.6/ssl.py", line 971, in recv_into
return self.read(nbytes, buffer)
File "/usr/lib64/python3.6/ssl.py", line 833, in read
return self._sslobj.read(len, buffer)
File "/usr/lib64/python3.6/ssl.py", line 590, in read
v = self._sslobj.read(len, buffer)
socket.timeout: The read operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/lib64/python3.6/site-packages/urllib3/connectionpool.py", line 706, in urlopen
chunked=chunked,
File "/usr/lib64/python3.6/site-packages/urllib3/connectionpool.py", line 447, in _make_request
self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
File "/usr/lib64/python3.6/site-packages/urllib3/connectionpool.py", line 337, in _raise_timeout
self, url, "Read timed out. (read timeout=%s)" % timeout_value
urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='oceandata.sci.gsfc.nasa.gov', port=443): Read timed out. (read timeout=10.0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/lib64/python3.6/site-packages/requests/adapters.py", line 449, in send
timeout=timeout
File "/usr/lib64/python3.6/site-packages/urllib3/connectionpool.py", line 796, in urlopen
**response_kw
File "/usr/lib64/python3.6/site-packages/urllib3/connectionpool.py", line 796, in urlopen
**response_kw
File "/usr/lib64/python3.6/site-packages/urllib3/connectionpool.py", line 796, in urlopen
**response_kw
[Previous line repeated 2 more times]
File "/usr/lib64/python3.6/site-packages/urllib3/connectionpool.py", line 756, in urlopen
method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
File "/usr/lib64/python3.6/site-packages/urllib3/util/retry.py", line 574, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='oceandata.sci.gsfc.nasa.gov', port=443): Max retries exceeded with url: /api/anc_data_api/?&m=29&s=2023-06-27T00:45:26&e=2023-06-27T00:45:26&missing_tags=1 (Caused by ReadTimeoutError("HTTPSConnectionPool(host='oceandata.sci.gsfc.nasa.gov', port=443): Read timed out. (read timeout=10.0)",))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/shares/cms_optics/apps/seadas/ocssw/V2023.2/bin/getanc", line 149, in <module>
exit(main())
File "/shares/cms_optics/apps/seadas/ocssw/V2023.2/bin/getanc", line 142, in main
g.findweb()
File "/shares/cms_optics/apps/seadas/ocssw/V2023.2/bin/seadasutils/anc_utils.py", line 529, in findweb
verbose=self.verbose
File "/shares/cms_optics/apps/seadas/ocssw/V2023.2/bin/seadasutils/ProcUtils.py", line 105, in httpdl
with obpgSession.get(urlStr, stream=True, timeout=timeout, headers=headers) as req:
File "/usr/lib64/python3.6/site-packages/requests/sessions.py", line 555, in get
return self.request('GET', url, **kwargs)
File "/usr/lib64/python3.6/site-packages/requests/sessions.py", line 542, in request
resp = self.send(prep, **send_kwargs)
File "/usr/lib64/python3.6/site-packages/requests/sessions.py", line 655, in send
r = adapter.send(request, **kwargs)
File "/usr/lib64/python3.6/site-packages/requests/adapters.py", line 516, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='oceandata.sci.gsfc.nasa.gov', port=443): Max retries exceeded with url: /api/anc_data_api/?&m=29&s=2023-06-27T00:45:26&e=2023-06-27T00:45:26&missing_tags=1 (Caused by ReadTimeoutError("HTTPSConnectionPool(host='oceandata.sci.gsfc.nasa.gov', port=443): Read timed out. (read timeout=10.0)",))

[oo_processing]

Add:
One cronjob may call 'getanc' multiple times depending on how many new files staged. The number can be 20+ or even 40+, not all of them failed, sometimes worse, sometimes better. It dumps all error messages making the cornlog very long, it's annoying. I added a catch exception mechanism to run 'getanc' again if it failed, have seen some of them gone through without the error. That's the best I got so far.

[OB ODPS - towens]

The intermittent errors seem to be due to a server timeout:
host='oceandata.sci.gsfc.nasa.gov', port=443): Read timed out. (read timeout=10.0)
Is your internet connection stable?

If you run both queries on the command line does NR work and NT fail.

Tommy

[oo_processing]

Thanks Tommy,

My internet is stable. Because my cronjobs are running back-to-back, the OLCI-NT cronjob is followed by a MODIS one, and no error for the MODIS. The command lines for NT and NR of the same subscription (area) are literally the same, but I call them in different time (NR good, NT failed), if the time is the factor.

Yuyuan

[OB ODPS - towens]

can you provide both commands that you run? and do they repeatably fail for NT and succeed for NRT?
My point is that the network problem could have been intermittent and coincidentally happened at the the time you were pulling ancillary for the NT files.

Tommy

[oo_processing]

Sorry, I was away from my computer.

Right now, I'm running these commands:
For NT, it failed with the same connection issue:
getanc --disable-download -m 'olcis3a' -s '2023177154049' -e '2023177154049' --ancdb='/work/o/oo_processing/tmp/junk.db'
For NR, it succeed immediately:
getanc --disable-download -m 'olcis3a' -s '2023178165537' -e '2023178165537' --ancdb='/work/o/oo_processing/tmp/junk.db'

My guess is when getanc is pulling yesterday's ancillary files there was no response from server then it looks like a connection issue.

Yuyuan

[OB ODPS - towens]

I am able to replicate the error at my desk which is in the same building as the data server.
I'll pass this to the SeaDAS developers.

Tommy

[OB SeaDAS - xuanyang02]

Normally, the query https://oceandata.sci.gsfc.nasa.gov/api/anc_data_api/?&m=29&s=2023-06-26T15:40:49&e=2023-06-26T15:45:49&missing_tags=1 should be very quick, but we have seen the above query takes 20 to 50 seconds, which causes the timeout error you see running getanc.

The work around is to run getanc with the --timeout option

getanc --disable-download -m 'olcis3a' -s '2023177154049' -e '2023177154049' --ancdb='/work/o/oo_processing/tmp/junk.db' --timeout 60.0

[oo_processing]

It's working, many thanks!

Yuyuan