Error loading into memory HLS data

[cbourque17]

Hello, I am following a python tutorial to learn how to set a script to use HLS data for climate change research purposes. When it comes to loading the HLS scenes into memory I constantly get this error code. I have tried 2 different tutorials, and I always get that error at the same step in the code.

```
hls_cloud_free = hls_masked.resample(time="1M").median(skipna=True).compute()
```
RuntimeError: Error opening 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T15UWP.2022184T170901.v2.0/HLS.S30.T15UWP.2022184T170901.v2.0.B04.tif': RasterioIOError("'/vsicurl/https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T15UWP.2022184T170901.v2.0/HLS.S30.T15UWP.2022184T170901.v2.0.B04.tif' not recognized as a supported file format.")

I have a netcr file in my home directory with my username and password and passed those configurations with GDAL as well.

gdal.SetConfigOption('GDAL_HTTP_COOKIEFILE','~/cookies.txt')
gdal.SetConfigOption('GDAL_HTTP_COOKIEJAR', '~/cookies.txt')
gdal.SetConfigOption('GDAL_DISABLE_READDIR_ON_OPEN','EMPTY_DIR')
gdal.SetConfigOption('CPL_VSIL_CURL_ALLOWED_EXTENSIONS','TIF')

Another person in my lab tried running those codes and get the same error message. Any thoughts?

Thank you

[LP DAACx - dgolon]

Hello @cbourque17 Thank you for writing in. We have reported the issue to our SMEs. They are looking into it and we will report back when we have more information.

[LP DAACx - lien]

Hi Charlie,
We've updated the HLS_Tutorial.ipynb to include a try loop around reading of the HLS files via https. There seems to be some sort of network issue that can cause that error, and it appears to be independent of the code being executed in the notebook. We would recommend adding a retry loop around reading of the files within your script. An example from the updated tutorial:

Use vsicurl to load the data directly into memory (be patient, may take a few seconds)
chunk_size = dict(band=1, x=512, y=512) # Tiles have 1 band and are divided into 512x512 pixel chunks
Sometimes a vsi curl error occurs so we need to retry if it does
max_retries = 10
for e in evi_band_links:
print(e)
# Try Loop
for _i in range(max_retries):
try:
# Open and build datasets
if e.rsplit('.', 2)[-2] == evi_bands[0]: # NIR index
nir = rxr.open_rasterio(e, chunks=chunk_size, masked=True).squeeze('band', drop=True)
nir.attrs['scale_factor'] = 0.0001 # hard coded the scale_factor attribute
elif e.rsplit('.', 2)[-2] == evi_bands[1]: # red index
red = rxr.open_rasterio(e, chunks=chunk_size, masked=True).squeeze('band', drop=True)
red.attrs['scale_factor'] = 0.0001 # hard coded the scale_factor attribute
elif e.rsplit('.', 2)[-2] == evi_bands[2]: # blue index
blue = rxr.open_rasterio(e, chunks=chunk_size, masked=True).squeeze('band', drop=True)
blue.attrs['scale_factor'] = 0.0001 # hard coded the scale_factor attribute
break # Break out of the retry loop
except Exception as ex:
print(f"vsi curl error: {ex}. Retrying...")
else:
print(f"Failed to process {e} after {max_retries} retries. Please check to see you're authenticated with earthaccess.")
print("The COGs have been loaded into memory!")

Hope this helps.

Regards,
LP DAAC User Services

[cbourque17]

Hello,
As an update, I have now transferred to using a remote desktop with Linux and do not get the error anymore. So, it was indeed a problem related to my authentication file.
I am still unable to run the notebook on my Windows machine, but as mentioned above, something isn't working well with my authentication .netrc file.

[LP DAACx - dgolon]

We are glad to hear you were able to access the data using your remote desktop. Thanks for reporting back. Since both you and your colleague in the lab were seeing the same error it could be related to the VPN you use, your lab's network, or settings on your window machines.

[LP DAACx - dgolon]

You might also be interested in trying to login using the earthaccess.login(), although noting VPN can impact it as well. For more information on earthaccess check out https://earthaccess.readthedocs.io/en/latest/