Hello,
I have a set of code that builds an HLS data cube over a given location and time-interval, then runs further processing. As part of that, this data cube needs to be loaded into memory with persist().
The code works well in most scenarios. However, I have found that one particular tif is unable to be accessed, triggering an error and preventing the HLS data cube from being loaded into memory.
The relevant portions of the error:
CPLE_OpenFailedError: '/vsicurl/https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T10UDU.2014082T190147.v2.0/HLS.L30.T10UDU.2014082T190147.v2.0.B06.tif' does not exist in the file system, and is not recognized as a supported dataset name.
RuntimeError: Error opening 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T10UDU.2014082T190147.v2.0/HLS.L30.T10UDU.2014082T190147.v2.0.B06.tif': RasterioIOError("'/vsicurl/https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T10UDU.2014082T190147.v2.0/HLS.L30.T10UDU.2014082T190147.v2.0.B06.tif' does not exist in the file system, and is not recognized as a supported dataset name.")
Note that it is always the same tif link that is provided by the error, and retrying the function triggers the same error.
Any help would be appreciated!
HLSL30 - Cannot load certain tif into memory
-
LP DAAC - dgolon
- User Services
- Posts: 237
- Joined: Mon Sep 30, 2019 10:00 am America/New_York
- Has thanked: 12 times
- Been thanked: 1 time
- Contact:
Re: HLSL30 - Cannot load certain tif into memory
Hi @mitchbon I've passed your question along to our developers. We will reach back on on this post once we have an answer or if we need additional information from you. Thanks!
Subscribe to the LP DAAC listserv by sending a blank email to lpdaac-join@lists.nasa.gov.
Sign up for the Landsat listserv to receive the most up to date information about Landsat data: https://public.govdelivery.com/accounts/USDOIGS/subscriber/new#tab1.
Sign up for the Landsat listserv to receive the most up to date information about Landsat data: https://public.govdelivery.com/accounts/USDOIGS/subscriber/new#tab1.
Re: HLSL30 - Cannot load certain tif into memory
I ran the same code again today and it was able to successfully find and load that problematic tif into memory. I guess that means this is some sort of server issue?
Last edited by mitchbon on Mon Mar 13, 2023 12:28 pm America/New_York, edited 1 time in total.
-
LP DAAC - jwilson
- User Services
- Posts: 252
- Joined: Mon Sep 30, 2019 12:39 pm America/New_York
- Has thanked: 9 times
Re: HLSL30 - Cannot load certain tif into memory
@mitchbon
If you are unable to load EarthData assets from https URLs via vsicurl and rasterio in Python, there are 3 common solutions:
1. Ensure you have a properly configured .netrc file. Instructions can be found [here|https://github.com/nasa/LPDAAC-Data-Resources/blob/main/notebooks/Earthdata_Authentication__Create_netrc_file.ipynb].
2.Ensure that you have set the necessary gdal configurations to access data using vsicurl. The below code can be used to set these:
gdal.SetConfigOption('GDAL_HTTP_COOKIEFILE','~/cookies.txt')
gdal.SetConfigOption('GDAL_HTTP_COOKIEJAR', '~/cookies.txt')
gdal.SetConfigOption('GDAL_DISABLE_READDIR_ON_OPEN','EMPTY_DIR')
gdal.SetConfigOption('CPL_VSIL_CURL_ALLOWED_EXTENSIONS','TIF')
3. Sometimes cached information can cause an issue, to resolve this, try restarting your Python Kernel.
If you are unable to load EarthData assets from https URLs via vsicurl and rasterio in Python, there are 3 common solutions:
1. Ensure you have a properly configured .netrc file. Instructions can be found [here|https://github.com/nasa/LPDAAC-Data-Resources/blob/main/notebooks/Earthdata_Authentication__Create_netrc_file.ipynb].
2.Ensure that you have set the necessary gdal configurations to access data using vsicurl. The below code can be used to set these:
gdal.SetConfigOption('GDAL_HTTP_COOKIEFILE','~/cookies.txt')
gdal.SetConfigOption('GDAL_HTTP_COOKIEJAR', '~/cookies.txt')
gdal.SetConfigOption('GDAL_DISABLE_READDIR_ON_OPEN','EMPTY_DIR')
gdal.SetConfigOption('CPL_VSIL_CURL_ALLOWED_EXTENSIONS','TIF')
3. Sometimes cached information can cause an issue, to resolve this, try restarting your Python Kernel.