GESDISC Server Netcdf Error NLDAS Access

[leahstaub4]

I have been accessing NLDAS data from the GESDISC Server through a jupyter notebook for a couple of months now. Every couple of weeks a strange error occurs, we are thinking it has to do with netcdf and this server interacting. Maybe a firewall issue? or a caching issue? We have adjusted netcdf versions in the past to get past this error but had no luck this time.

Here is the python code:
import warnings
import xarray as xr
# from netrc import netrc
from subprocess import Popen
from getpass import getpass
import platform
import os
import shutil

run_authentication = True
if run_authentication:
urs = 'urs.earthdata.nasa.gov' # Earthdata URL to call for authentication
prompts = ['Enter NASA Earthdata Login Username \n(or create an account at urs.earthdata.nasa.gov): ',
'Enter NASA Earthdata Login Password: ']

homeDir = os.path.expanduser("~") + os.sep
print(homeDir)

with open(f'{homeDir}.netrc', 'w') as file:
file.write('machine {} login {} password {}'.format(urs, getpass(prompt=prompts[0]), getpass(prompt=prompts[1])))
file.close()
with open(f'{homeDir}.urs_cookies', 'w') as file:
file.write('')
file.close()
with open(f'{homeDir}.dodsrc', 'w') as file:
file.write(f'HTTP.COOKIEJAR={homeDir}.urs_cookies\n')
file.write(f'HTTP.NETRC={homeDir}.netrc')
file.close()

print('Saved .netrc, .urs_cookies, and .dodsrc to:', homeDir)

# Set appropriate permissions for Linux/macOS
if platform.system() != "Windows":
Popen('chmod og-rw ~/.netrc', shell=True)
shutil.copy2(f'{homeDir}.dodsrc', os.getcwd())
print('Copied .dodsrc to:', os.getcwd())
else:
# Copy dodsrc to working directory in Windows
shutil.copy2(f'{homeDir}.dodsrc', os.getcwd())
print('Copied .dodsrc to:', os.getcwd())

!chmod 0600 ~/.netrc

tmpds = xr.open_dataset('https://hydro1.gesdisc.eosdis.nasa.gov/dods/NLDAS_NOAH0125_H.002/')
tmpds

Here is the error:

syntax error, unexpected WORD_WORD, expecting SCAN_ATTR or SCAN_DATASET or SCAN_ERROR
context: HTTP^ Basic: Access denied.
syntax error, unexpected WORD_WORD, expecting SCAN_ATTR or SCAN_DATASET or SCAN_ERROR
context: HTTP^ Basic: Access denied.
syntax error, unexpected WORD_WORD, expecting SCAN_ATTR or SCAN_DATASET or SCAN_ERROR
context: HTTP^ Basic: Access denied.
syntax error, unexpected WORD_WORD, expecting SCAN_ATTR or SCAN_DATASET or SCAN_ERROR
context: HTTP^ Basic: Access denied.
syntax error, unexpected WORD_WORD, expecting SCAN_ATTR or SCAN_DATASET or SCAN_ERROR
context: HTTP^ Basic: Access denied.
syntax error, unexpected WORD_WORD, expecting SCAN_ATTR or SCAN_DATASET or SCAN_ERROR
context: HTTP^ Basic: Access denied.
syntax error, unexpected WORD_WORD, expecting SCAN_ATTR or SCAN_DATASET or SCAN_ERROR
context: HTTP^ Basic: Access denied.

---------------------------------------------------------------------------
RuntimeError Traceback (most recent call last)
File ~/miniconda3/envs/gesdisc/lib/python3.9/site-packages/xarray/conventions.py:431, in decode_cf_variables(variables, attributes, concat_characters, mask_and_scale, decode_times, decode_coords, drop_variables, use_cftime, decode_timedelta)
430 try:
--> 431 new_vars[k] = decode_cf_variable(
432 k,
433 v,
434 concat_characters=concat_characters,
435 mask_and_scale=mask_and_scale,
436 decode_times=decode_times,
437 stack_char_dim=stack_char_dim,
438 use_cftime=use_cftime,
439 decode_timedelta=decode_timedelta,
440 )
441 except Exception as e:

File ~/miniconda3/envs/gesdisc/lib/python3.9/site-packages/xarray/conventions.py:281, in decode_cf_variable(name, var, concat_characters, mask_and_scale, decode_times, decode_endianness, stack_char_dim, use_cftime, decode_timedelta)
280 if decode_times:
--> 281 var = times.CFDatetimeCoder(use_cftime=use_cftime).decode(var, name=name)
283 if decode_endianness and not var.dtype.isnative:

File ~/miniconda3/envs/gesdisc/lib/python3.9/site-packages/xarray/coding/times.py:724, in CFDatetimeCoder.decode(self, variable, name)
723 calendar = pop_to(attrs, encoding, "calendar")
--> 724 dtype = _decode_cf_datetime_dtype(data, units, calendar, self.use_cftime)
725 transform = partial(
726 decode_cf_datetime,
727 units=units,
728 calendar=calendar,
729 use_cftime=self.use_cftime,
730 )

File ~/miniconda3/envs/gesdisc/lib/python3.9/site-packages/xarray/coding/times.py:182, in _decode_cf_datetime_dtype(data, units, calendar, use_cftime)
180 values = indexing.ImplicitToExplicitIndexingAdapter(indexing.as_indexable(data))
181 example_value = np.concatenate(
--> 182 [first_n_items(values, 1) or [0], last_item(values) or [0]]
183 )
185 try:

File ~/miniconda3/envs/gesdisc/lib/python3.9/site-packages/xarray/core/formatting.py:77, in first_n_items(array, n_desired)
76 array = array[indexer]
---> 77 return np.asarray(array).flat[:n_desired]

File ~/miniconda3/envs/gesdisc/lib/python3.9/site-packages/xarray/core/indexing.py:484, in ImplicitToExplicitIndexingAdapter.__array__(self, dtype)
483 def __array__(self, dtype: np.typing.DTypeLike = None) -> np.ndarray:
--> 484 return np.asarray(self.get_duck_array(), dtype=dtype)

File ~/miniconda3/envs/gesdisc/lib/python3.9/site-packages/xarray/core/indexing.py:487, in ImplicitToExplicitIndexingAdapter.get_duck_array(self)
486 def get_duck_array(self):
--> 487 return self.array.get_duck_array()

File ~/miniconda3/envs/gesdisc/lib/python3.9/site-packages/xarray/core/indexing.py:551, in LazilyIndexedArray.get_duck_array(self)
550 def get_duck_array(self):
--> 551 array = self.array[self.key]
552 # self.array[self.key] is now a numpy array when
553 # self.array is a BackendArray subclass
554 # and self.key is BasicIndexer((slice(None, None, None),))
555 # so we need the explicit check for ExplicitlyIndexed

File ~/miniconda3/envs/gesdisc/lib/python3.9/site-packages/xarray/backends/netCDF4_.py:100, in NetCDF4ArrayWrapper.__getitem__(self, key)
99 def __getitem__(self, key):
--> 100 return indexing.explicit_indexing_adapter(
101 key, self.shape, indexing.IndexingSupport.OUTER, self._getitem
102 )

File ~/miniconda3/envs/gesdisc/lib/python3.9/site-packages/xarray/core/indexing.py:858, in explicit_indexing_adapter(key, shape, indexing_support, raw_indexing_method)
857 raw_key, numpy_indices = decompose_indexer(key, shape, indexing_support)
--> 858 result = raw_indexing_method(raw_key.tuple)
859 if numpy_indices.tuple:
860 # index the loaded np.ndarray

File ~/miniconda3/envs/gesdisc/lib/python3.9/site-packages/xarray/backends/netCDF4_.py:113, in NetCDF4ArrayWrapper._getitem(self, key)
112 original_array = self.get_array(needs_lock=False)
--> 113 array = getitem(original_array, key)
114 except IndexError:
115 # Catch IndexError in netCDF4 and return a more informative
116 # error message. This is most often called when an unsorted
117 # indexer is used before the data is loaded from disk.

File ~/miniconda3/envs/gesdisc/lib/python3.9/site-packages/xarray/backends/common.py:151, in robust_getitem(array, key, catch, max_retries, initial_delay)
150 try:
--> 151 return array[key]
152 except catch:

File src/netCDF4/_netCDF4.pyx:4958, in netCDF4._netCDF4.Variable.__getitem__()

File src/netCDF4/_netCDF4.pyx:5916, in netCDF4._netCDF4.Variable._get()

File src/netCDF4/_netCDF4.pyx:2029, in netCDF4._netCDF4._ensure_nc_success()

RuntimeError: NetCDF: Access failure

During handling of the above exception, another exception occurred:

RuntimeError Traceback (most recent call last)
Cell In[3], line 1
----> 1 tmpds = xr.open_dataset('https://hydro1.gesdisc.eosdis.nasa.gov/dods/NLDAS_NOAH0125_H.002/')
2 tmpds

File ~/miniconda3/envs/gesdisc/lib/python3.9/site-packages/xarray/backends/api.py:570, in open_dataset(filename_or_obj, engine, chunks, cache, decode_cf, mask_and_scale, decode_times, decode_timedelta, use_cftime, concat_characters, decode_coords, drop_variables, inline_array, chunked_array_type, from_array_kwargs, backend_kwargs, **kwargs)
558 decoders = _resolve_decoders_kwargs(
559 decode_cf,
560 open_backend_dataset_parameters=backend.open_dataset_parameters,
(...)
566 decode_coords=decode_coords,
567 )
569 overwrite_encoded_chunks = kwargs.pop("overwrite_encoded_chunks", None)
--> 570 backend_ds = backend.open_dataset(
571 filename_or_obj,
572 drop_variables=drop_variables,
573 **decoders,
574 **kwargs,
575 )
576 ds = _dataset_from_backend_dataset(
577 backend_ds,
578 filename_or_obj,
(...)
588 **kwargs,
589 )
590 return ds

File ~/miniconda3/envs/gesdisc/lib/python3.9/site-packages/xarray/backends/netCDF4_.py:616, in NetCDF4BackendEntrypoint.open_dataset(self, filename_or_obj, mask_and_scale, decode_times, concat_characters, decode_coords, drop_variables, use_cftime, decode_timedelta, group, mode, format, clobber, diskless, persist, lock, autoclose)
614 store_entrypoint = StoreBackendEntrypoint()
615 with close_on_error(store):
--> 616 ds = store_entrypoint.open_dataset(
617 store,
618 mask_and_scale=mask_and_scale,
619 decode_times=decode_times,
620 concat_characters=concat_characters,
621 decode_coords=decode_coords,
622 drop_variables=drop_variables,
623 use_cftime=use_cftime,
624 decode_timedelta=decode_timedelta,
625 )
626 return ds

File ~/miniconda3/envs/gesdisc/lib/python3.9/site-packages/xarray/backends/store.py:46, in StoreBackendEntrypoint.open_dataset(self, filename_or_obj, mask_and_scale, decode_times, concat_characters, decode_coords, drop_variables, use_cftime, decode_timedelta)
43 vars, attrs = filename_or_obj.load()
44 encoding = filename_or_obj.get_encoding()
---> 46 vars, attrs, coord_names = conventions.decode_cf_variables(
47 vars,
48 attrs,
49 mask_and_scale=mask_and_scale,
50 decode_times=decode_times,
51 concat_characters=concat_characters,
52 decode_coords=decode_coords,
53 drop_variables=drop_variables,
54 use_cftime=use_cftime,
55 decode_timedelta=decode_timedelta,
56 )
58 ds = Dataset(vars, attrs=attrs)
59 ds = ds.set_coords(coord_names.intersection(vars))

File ~/miniconda3/envs/gesdisc/lib/python3.9/site-packages/xarray/conventions.py:442, in decode_cf_variables(variables, attributes, concat_characters, mask_and_scale, decode_times, decode_coords, drop_variables, use_cftime, decode_timedelta)
431 new_vars[k] = decode_cf_variable(
432 k,
433 v,
(...)
439 decode_timedelta=decode_timedelta,
440 )
441 except Exception as e:
--> 442 raise type(e)(f"Failed to decode variable {k!r}: {e}")
443 if decode_coords in [True, "coordinates", "all"]:
444 var_attrs = new_vars[k].attrs

RuntimeError: Failed to decode variable 'time': NetCDF: Access failure

[cbattisto]

Hello Leah,

Thank you for reaching out to us! We also received your original query through the helpdesk and were working on resolving your issue.

Please try your request again, but be aware that although the server virtually aggregates granules across a long time span (1979-2023), it is also configured to limit subset sizes to 500MB max, and requests will time out after 100 seconds. Requests for larger subsets will fail.

In your original code, please try a smaller ending increment time instead and let us know if your issue is resolved.

Thank you!
Chris Battisto

[GES DISC - mgreene]

Thank you for reporting this issue. Our science team will take a look at what you are seeing. We'll report back when we have more information.

[GES DISC - mgreene]

There has been a period of recent netCDF software releases that are not working well with OPeNDAP. These are the buggy versions of the python packages:

libnetcdf 4.9.2
netcdf4 1.6.4

I am able to open the URL with xarray:

ds=xarray.open_dataset(https://hydro1.gesdisc.eosdis.nasa.gov/dods/NLDAS_NOAH0125_H.2.0/).sel(time=slice("2023-08-20", "2023-08-26"))

when using the following (older) version combination:
libnetcdf 4.8.1
netcdf4 1.6.2