Error loading into memory HLS data

Use this Forum to find information on, or ask a question about, NASA Earth Science data.
Post Reply
cbourque17
Posts: 6
Joined: Mon Jan 15, 2024 11:05 am America/New_York
Answers: 0

Error loading into memory HLS data

by cbourque17 » Mon Jan 15, 2024 11:20 am America/New_York

Hello, I am following a python tutorial to learn how to set a script to use HLS data for climate change research purposes. When it comes to loading the HLS scenes into memory I constantly get this error code. I have tried 2 different tutorials, and I always get that error at the same step in the code.

```
hls_cloud_free = hls_masked.resample(time="1M").median(skipna=True).compute()
```
RuntimeError: Error opening 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T15UWP.2022184T170901.v2.0/HLS.S30.T15UWP.2022184T170901.v2.0.B04.tif': RasterioIOError("'/vsicurl/https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T15UWP.2022184T170901.v2.0/HLS.S30.T15UWP.2022184T170901.v2.0.B04.tif' not recognized as a supported file format.")

I have a netcr file in my home directory with my username and password and passed those configurations with GDAL as well.

gdal.SetConfigOption('GDAL_HTTP_COOKIEFILE','~/cookies.txt')
gdal.SetConfigOption('GDAL_HTTP_COOKIEJAR', '~/cookies.txt')
gdal.SetConfigOption('GDAL_DISABLE_READDIR_ON_OPEN','EMPTY_DIR')
gdal.SetConfigOption('CPL_VSIL_CURL_ALLOWED_EXTENSIONS','TIF')

Another person in my lab tried running those codes and get the same error message. Any thoughts?

Thank you

Filters:

LP DAAC - dgolon
User Services
User Services
Posts: 308
Joined: Mon Sep 30, 2019 10:00 am America/New_York
Answers: 0
Has thanked: 22 times
Been thanked: 2 times
Contact:

Re: Error loading into memory HLS data

by LP DAAC - dgolon » Tue Jan 16, 2024 9:41 am America/New_York

Hello @cbourque17 Thank you for writing in. We have reported the issue to our SMEs. They are looking into it and we will report back when we have more information.
Subscribe to the LP DAAC listserv by sending a blank email to lpdaac-join@lists.nasa.gov.

Sign up for the Landsat listserv to receive the most up to date information about Landsat data: https://public.govdelivery.com/accounts/USDOIGS/subscriber/new#tab1.

LP DAAC - lien
User Services
User Services
Posts: 185
Joined: Thu Jun 25, 2020 9:51 am America/New_York
Answers: 1
Been thanked: 6 times

Re: Error loading into memory HLS data

by LP DAAC - lien » Mon Jan 22, 2024 10:22 am America/New_York

Hi Charlie,
We've updated the HLS_Tutorial.ipynb to include a try loop around reading of the HLS files via https. There seems to be some sort of network issue that can cause that error, and it appears to be independent of the code being executed in the notebook. We would recommend adding a retry loop around reading of the files within your script. An example from the updated tutorial:

Use vsicurl to load the data directly into memory (be patient, may take a few seconds)
chunk_size = dict(band=1, x=512, y=512) # Tiles have 1 band and are divided into 512x512 pixel chunks
Sometimes a vsi curl error occurs so we need to retry if it does
max_retries = 10
for e in evi_band_links:
print(e)
# Try Loop
for _i in range(max_retries):
try:
# Open and build datasets
if e.rsplit('.', 2)[-2] == evi_bands[0]: # NIR index
nir = rxr.open_rasterio(e, chunks=chunk_size, masked=True).squeeze('band', drop=True)
nir.attrs['scale_factor'] = 0.0001 # hard coded the scale_factor attribute
elif e.rsplit('.', 2)[-2] == evi_bands[1]: # red index
red = rxr.open_rasterio(e, chunks=chunk_size, masked=True).squeeze('band', drop=True)
red.attrs['scale_factor'] = 0.0001 # hard coded the scale_factor attribute
elif e.rsplit('.', 2)[-2] == evi_bands[2]: # blue index
blue = rxr.open_rasterio(e, chunks=chunk_size, masked=True).squeeze('band', drop=True)
blue.attrs['scale_factor'] = 0.0001 # hard coded the scale_factor attribute
break # Break out of the retry loop
except Exception as ex:
print(f"vsi curl error: {ex}. Retrying...")
else:
print(f"Failed to process {e} after {max_retries} retries. Please check to see you're authenticated with earthaccess.")
print("The COGs have been loaded into memory!")

Hope this helps.

Regards,
LP DAAC User Services

cbourque17
Posts: 6
Joined: Mon Jan 15, 2024 11:05 am America/New_York
Answers: 0

Re: Error loading into memory HLS data

by cbourque17 » Wed Feb 28, 2024 3:28 pm America/New_York

Hello,
As an update, I have now transferred to using a remote desktop with Linux and do not get the error anymore. So, it was indeed a problem related to my authentication file.
I am still unable to run the notebook on my Windows machine, but as mentioned above, something isn't working well with my authentication .netrc file.

LP DAAC - dgolon
User Services
User Services
Posts: 308
Joined: Mon Sep 30, 2019 10:00 am America/New_York
Answers: 0
Has thanked: 22 times
Been thanked: 2 times
Contact:

Re: Error loading into memory HLS data

by LP DAAC - dgolon » Fri Mar 15, 2024 2:15 pm America/New_York

We are glad to hear you were able to access the data using your remote desktop. Thanks for reporting back. Since both you and your colleague in the lab were seeing the same error it could be related to the VPN you use, your lab's network, or settings on your window machines.
Subscribe to the LP DAAC listserv by sending a blank email to lpdaac-join@lists.nasa.gov.

Sign up for the Landsat listserv to receive the most up to date information about Landsat data: https://public.govdelivery.com/accounts/USDOIGS/subscriber/new#tab1.

LP DAAC - dgolon
User Services
User Services
Posts: 308
Joined: Mon Sep 30, 2019 10:00 am America/New_York
Answers: 0
Has thanked: 22 times
Been thanked: 2 times
Contact:

Re: Error loading into memory HLS data

by LP DAAC - dgolon » Fri Mar 15, 2024 2:22 pm America/New_York

You might also be interested in trying to login using the earthaccess.login(), although noting VPN can impact it as well. For more information on earthaccess check out https://earthaccess.readthedocs.io/en/latest/
Subscribe to the LP DAAC listserv by sending a blank email to lpdaac-join@lists.nasa.gov.

Sign up for the Landsat listserv to receive the most up to date information about Landsat data: https://public.govdelivery.com/accounts/USDOIGS/subscriber/new#tab1.

Post Reply