getanc socket.timeout

Use this Forum to find information on, or ask a question about, NASA Earth Science data.
Post Reply
tlin2_cuny.edu
Posts: 13
Joined: Tue Feb 28, 2023 2:16 pm America/New_York
Answers: 0

getanc socket.timeout

by tlin2_cuny.edu » Tue Dec 05, 2023 2:46 pm America/New_York

Dear Support,

I got issue when using L2gen getanc -v. My SeaDAS version is V2022.3. Now it is ok to run L2gen if I use the existing ancfile as par, but I can not use getanc command.

Sensor : "OLI_TIRS"
Start time: 2023-08-09T15:33:08
End time : 2023-08-09T15:33:08

OBPG session started
Traceback (most recent call last):
File "/Users/tonglin/anaconda3/lib/python3.9/site-packages/urllib3/connection.py", line 174, in _new_conn
conn = connection.create_connection(
File "/Users/tonglin/anaconda3/lib/python3.9/site-packages/urllib3/util/connection.py", line 95, in create_connection
raise err
File "/Users/tonglin/anaconda3/lib/python3.9/site-packages/urllib3/util/connection.py", line 85, in create_connection
sock.connect(sa)
socket.timeout: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/Users/tonglin/anaconda3/lib/python3.9/site-packages/urllib3/connectionpool.py", line 714, in urlopen
httplib_response = self._make_request(
File "/Users/tonglin/anaconda3/lib/python3.9/site-packages/urllib3/connectionpool.py", line 403, in _make_request
self._validate_conn(conn)
File "/Users/tonglin/anaconda3/lib/python3.9/site-packages/urllib3/connectionpool.py", line 1053, in _validate_conn
conn.connect()
File "/Users/tonglin/anaconda3/lib/python3.9/site-packages/urllib3/connection.py", line 363, in connect
self.sock = conn = self._new_conn()
File "/Users/tonglin/anaconda3/lib/python3.9/site-packages/urllib3/connection.py", line 179, in _new_conn
raise ConnectTimeoutError(
urllib3.exceptions.ConnectTimeoutError: (<urllib3.connection.HTTPSConnection object at 0x7fbfa179f340>, 'Connection to oceandata.sci.gsfc.nasa.gov timed out. (connect timeout=10.0)')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/Users/tonglin/anaconda3/lib/python3.9/site-packages/requests/adapters.py", line 487, in send
resp = conn.urlopen(
File "/Users/tonglin/anaconda3/lib/python3.9/site-packages/urllib3/connectionpool.py", line 826, in urlopen
return self.urlopen(
File "/Users/tonglin/anaconda3/lib/python3.9/site-packages/urllib3/connectionpool.py", line 826, in urlopen
return self.urlopen(
File "/Users/tonglin/anaconda3/lib/python3.9/site-packages/urllib3/connectionpool.py", line 826, in urlopen
return self.urlopen(
[Previous line repeated 2 more times]
File "/Users/tonglin/anaconda3/lib/python3.9/site-packages/urllib3/connectionpool.py", line 798, in urlopen
retries = retries.increment(
File "/Users/tonglin/anaconda3/lib/python3.9/site-packages/urllib3/util/retry.py", line 592, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='oceandata.sci.gsfc.nasa.gov', port=443): Max retries exceeded with url: /api/anc_data_api/?&m=0&s=2023-08-09T15:33:08&e=2023-08-09T15:33:08&missing_tags=1 (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7fbfa179f340>, 'Connection to oceandata.sci.gsfc.nasa.gov timed out. (connect timeout=10.0)'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/Users/tonglin/ocssw/bin/getanc", line 149, in <module>
exit(main())
File "/Users/tonglin/ocssw/bin/getanc", line 142, in main
g.findweb()
File "/Users/tonglin/ocssw/bin/seadasutils/anc_utils.py", line 521, in findweb
dlstat = ProcUtils.httpdl(self.query_site,
File "/Users/tonglin/ocssw/bin/seadasutils/ProcUtils.py", line 105, in httpdl
with obpgSession.get(urlStr, stream=True, timeout=timeout, headers=headers) as req:
File "/Users/tonglin/anaconda3/lib/python3.9/site-packages/requests/sessions.py", line 600, in get
return self.request("GET", url, **kwargs)
File "/Users/tonglin/anaconda3/lib/python3.9/site-packages/requests/sessions.py", line 587, in request
resp = self.send(prep, **send_kwargs)
File "/Users/tonglin/anaconda3/lib/python3.9/site-packages/requests/sessions.py", line 701, in send
r = adapter.send(request, **kwargs)
File "/Users/tonglin/anaconda3/lib/python3.9/site-packages/requests/adapters.py", line 508, in send
raise ConnectTimeout(e, request=request)
requests.exceptions.ConnectTimeout: HTTPSConnectionPool(host='oceandata.sci.gsfc.nasa.gov', port=443): Max retries exceeded with url: /api/anc_data_api/?&m=0&s=2023-08-09T15:33:08&e=2023-08-09T15:33:08&missing_tags=1 (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7fbfa179f340>, 'Connection to oceandata.sci.gsfc.nasa.gov timed out. (connect timeout=10.0)'))
-E- clo_readFile: Can't open parameter file - /Volumes/Tlin/LIS/Landsat8/Collection2/2023/0809/LC08_L1TP_013031_20230809_20230812_02_T1_MTL.txt.anc
ancillary_data.db
Searching database: /Users/tonglin/ocssw/var/log/ancillary_data.db
*** ERROR: Input file doesn't exist and mission not set...bailing out...
-E- clo_readFile: Can't open parameter file - /Volumes/Tlin/LIS/Landsat8/Collection2/2023/0809/LC08_L1TP_013032_20230809_20230812_02_T1_MTL.txt.anc_MTL.txt.anc

Best,
Tong

Filters:

haag
Posts: 33
Joined: Wed Jun 08, 2016 7:06 am America/New_York
Answers: 0

Re: getanc socket.timeout

by haag » Wed Dec 06, 2023 8:52 pm America/New_York

Just to confirm, I am seeing the same. We have been routinely processing (a simple run of multilevel_processor to get L1b) using the latest downloaded L1A files on a daily basis. It failed on Dec 5. A manual run of modis_GEO today (6th) gave the same timeout.

OB General Science - guoqingw
Subject Matter Expert
Subject Matter Expert
Posts: 79
Joined: Fri Jun 03, 2022 10:54 am America/New_York
Answers: 0
Location: NASA GSFC
Been thanked: 1 time
Contact:

Re: getanc socket.timeout

by OB General Science - guoqingw » Thu Dec 07, 2023 10:06 am America/New_York

@tlin2_cuny.edu @haag
Please email "connection_problems@oceancolor.gsfc.nasa.gov" with the public ip address you are connection from, the time you tried to connect and a short description of the network issue you are experiencing. The NASA OBPG network team will assist you.

haag
Posts: 33
Joined: Wed Jun 08, 2016 7:06 am America/New_York
Answers: 0

Re: getanc socket.timeout

by haag » Thu Dec 07, 2023 11:16 am America/New_York

I don't mean to hijack the OP's issue; but figured it was related.

For what it's worth, my auto processing worked fine for today (yesterday's data) so I reran data from the 5th and 6th and (some of) the data from the 5th still fails, so it suggests the issue is not connectivity, but that the ancillary data is missing?

In particular, modis_GEO fails on these specific granules for me:

T2023339145500.L1A_LAC
T2023339150000.L1A_LAC
T2023339163500.L1A_LAC
T2023339164000.L1A_LAC
A2023339191000.L1A_LAC
A2023339191500.L1A_LAC

There may be others, but I'm focussed on the Gulf of Mexico.

I'll try again in a few days in case the ancillary data gets back-filled.

Many thanks!
Alaric

OB ODPS - jgwilding
Subject Matter Expert
Subject Matter Expert
Posts: 139
Joined: Fri Feb 19, 2021 1:09 pm America/New_York
Answers: 0
Been thanked: 1 time

Re: getanc socket.timeout

by OB ODPS - jgwilding » Thu Dec 07, 2023 2:55 pm America/New_York

We made the GEO products for these files on Dec 5, so I don't believe this is an issue with ancillary data.
Are you able to download the files below using the getfile method described here?

basename=TERRA_MODIS.20231205T145500.GEO.NRT.hdf
create_date=Dec 5 2023 21:38:13.426
basename=TERRA_MODIS.20231205T150000.GEO.NRT.hdf
create_date=Dec 5 2023 21:44:08.440
basename=TERRA_MODIS.20231205T163500.GEO.NRT.hdf
create_date=Dec 5 2023 23:44:04.233
basename=TERRA_MODIS.20231205T164000.GEO.NRT.hdf
create_date=Dec 5 2023 23:52:09.116
basename=AQUA_MODIS.20231205T191001.GEO.NRT.hdf
create_date=Dec 5 2023 20:15:54.193
basename=AQUA_MODIS.20231205T191501.GEO.NRT.hdf
create_date=Dec 5 2023 20:14:56.256


john

haag
Posts: 33
Joined: Wed Jun 08, 2016 7:06 am America/New_York
Answers: 0

Re: getanc socket.timeout

by haag » Tue Dec 12, 2023 11:55 am America/New_York

Thank you John,

After more digging, and upgrading SeaDAS from 8.3 to 8.4.1, I am certain this is not about ancillary data, nor about downloading configuration. I will start a new thread outlining what I'm seeing.

And I wholeheartedly apologize to the OP for taking over this thread!

OB SeaDAS - xuanyang02
Subject Matter Expert
Subject Matter Expert
Posts: 657
Joined: Tue Feb 09, 2021 5:42 pm America/New_York
Answers: 1
Been thanked: 2 times

Re: getanc socket.timeout

by OB SeaDAS - xuanyang02 » Tue Dec 12, 2023 4:50 pm America/New_York

It takes longer for some files to getanc. If you can prolong the timeout, it might work --

% getanc A2023339191000.L1A_LAC --timeout 45
Running debug binaries...
/Users/*/ocssw/bin
ancillary_data.db
Aqua
icefile=/Users/*/ocssw/var/anc/2023/339/20231205120000-CMC-L4_GHRSST-SSTfnd-CMC0.1deg-GLOB-v02.0-fv03.0.nc
met1=/Users/*/ocssw/var/anc/2023/339/GMAO_FP.20231205T180000.MET.NRT.nc
met2=/Users/*/ocssw/var/anc/2023/339/GMAO_FP.20231205T210000.MET.NRT.nc
met3=/Users/*/ocssw/var/anc/2023/339/GMAO_FP.20231205T210000.MET.NRT.nc
ozone1=/Users/*/ocssw/var/anc/2023/339/GMAO_FP.20231205T180000.MET.NRT.nc
ozone2=/Users/*/ocssw/var/anc/2023/339/GMAO_FP.20231205T210000.MET.NRT.nc
ozone3=/Users/*/ocssw/var/anc/2023/339/GMAO_FP.20231205T210000.MET.NRT.nc
sstfile=/Users/*/ocssw/var/anc/2023/339/20231205120000-CMC-L4_GHRSST-SSTfnd-CMC0.1deg-GLOB-v02.0-fv03.0.nc

Post Reply