L1A data downloads stop
L1A data downloads stop
Hello
We regularly download L1A data from MODIS and VIIRS, in order to provide a marine monitoring system for the local governemet. Starting from few days ago, I am experiencing problems related to data download. The batch download, that used to work, currently stops. Sometimes the file download restarts, but others not. Maybe could be something related to file size because the problem prevented the VIIRS complete dowloads, however MODIS ones can by accomplished (not always). Please see below the output messages from wget, curl and obdaac_download.py downloads:
------------ Using wget ----------------------
wget --load-cookies ~/.urs_cookies --save-cookies ~/.urs_cookies --auth-no-challenge=on --content-disposition --keep-session-cookies https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/SNPP_VIIRS.20231002T135400.L1A.nc
wget --load-cookies ~/.urs_cookies --save-cookies ~/.urs_cookies --auth-no-challenge=on --content-disposition --keep-session-cookies https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/SNPP_VIIRS.20231002T135400.L1A.nc
--2023-10-04 14:26:57-- https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/SNPP_VIIRS.20231002T135400.L1A.nc
Resolviendo oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)... 169.154.128.84, 2001:4d0128::84
Conectando con oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)[169.154.128.84]:443... conectado.
Petición HTTP enviada, esperando respuesta... 301 Moved Permanently
Ubicación: https://oceandata.sci.gsfc.nasa.gov/getfile/SNPP_VIIRS.20231002T135400.L1A.nc [siguiente]
--2023-10-04 14:26:57-- https://oceandata.sci.gsfc.nasa.gov/getfile/SNPP_VIIRS.20231002T135400.L1A.nc
Reutilizando la conexión con oceandata.sci.gsfc.nasa.gov:443.
Petición HTTP enviada, esperando respuesta... 302 FOUND
Ubicación: https://oceandata.sci.gsfc.nasa.gov/getfile/urs/?next=http%3A%2F%2Flocalhost%3A5007%2Fgetfile%2FSNPP_VIIRS.20231002T135400.L1A.nc [siguiente]
--2023-10-04 14:26:57-- https://oceandata.sci.gsfc.nasa.gov/getfile/urs/?next=http%3A%2F%2Flocalhost%3A5007%2Fgetfile%2FSNPP_VIIRS.20231002T135400.L1A.nc
Reutilizando la conexión con oceandata.sci.gsfc.nasa.gov:443.
Petición HTTP enviada, esperando respuesta... 302 FOUND
Ubicación: https://urs.earthdata.nasa.gov//oauth/authorize?client_id=pDPu0awH156XLrK6VV0Y0w&response_type=code&redirect_uri=https://oceandata.sci.gsfc.nasa.gov/getfile/urs/ [siguiente]
--2023-10-04 14:26:57-- https://urs.earthdata.nasa.gov//oauth/authorize?client_id=pDPu0awH156XLrK6VV0Y0w&response_type=code&redirect_uri=https://oceandata.sci.gsfc.nasa.gov/getfile/urs/
Resolviendo urs.earthdata.nasa.gov (urs.earthdata.nasa.gov)... 198.118.243.33, 2001:4d04081::89
Conectando con urs.earthdata.nasa.gov (urs.earthdata.nasa.gov)[198.118.243.33]:443... conectado.
Petición HTTP enviada, esperando respuesta... 302 Found
Ubicación: https://oceandata.sci.gsfc.nasa.gov/getfile/urs/?code=da1a2479a27a8bc0240914bd142b3af7972d5ec2e7fd76cd4ca394431c1b0c9d [siguiente]
--2023-10-04 14:26:58-- https://oceandata.sci.gsfc.nasa.gov/getfile/urs/?code=da1a2479a27a8bc0240914bd142b3af7972d5ec2e7fd76cd4ca394431c1b0c9d
Conectando con oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)[169.154.128.84]:443... conectado.
Petición HTTP enviada, esperando respuesta... 302 FOUND
Ubicación: https://oceandata.sci.gsfc.nasa.gov/getfile/SNPP_VIIRS.20231002T135400.L1A.nc [siguiente]
--2023-10-04 14:26:59-- https://oceandata.sci.gsfc.nasa.gov/getfile/SNPP_VIIRS.20231002T135400.L1A.nc
Reutilizando la conexión con oceandata.sci.gsfc.nasa.gov:443.
Petición HTTP enviada, esperando respuesta... 200 OK
Longitud: 452945750 (432M) [application/octet-stream]
Guardando como: ‘SNPP_VIIRS.20231002T135400.L1A.nc’
SNPP_VIIRS.20231002T135400.L1A.nc 6%[======> ] 29,94M 1,01MB/s en 30s
2023-10-04 14:27:29 (1,01 MB/s) - Conexión cerrada en el byte 31399712. Reintentando.
--2023-10-04 14:27:30-- (intento: 2) https://oceandata.sci.gsfc.nasa.gov/getfile/SNPP_VIIRS.20231002T135400.L1A.nc
Conectando con oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)[169.154.128.84]:443... conectado.
Petición HTTP enviada, esperando respuesta... 200 OK
Longitud: 452945750 (432M) [application/octet-stream]
Guardando como: ‘SNPP_VIIRS.20231002T135400.L1A.nc’
SNPP_VIIRS.20231002T135400.L1A.nc 9%[=========> ] 42,26M 1,02MB/s eta 5m 56s ^SNPP_VIIRS.20231002T135400.L1A.nc 42%[===========================================> ] 183,52M 1,19MB/s en 2m 45s
2023-10-04 14:30:15 (1,12 MB/s) - Conexión cerrada en el byte 192434216. Reintentando.
--2023-10-04 14:30:17-- (intento: 3) https://oceandata.sci.gsfc.nasa.gov/getfile/SNPP_VIIRS.20231002T135400.L1A.nc
Conectando con oceandata.sci.gsfc.nasa.gov (Hello We regularly download L1A data from MODIS and VIIRS, in order to provide a marine monitoring system for the local governemet. Starting from few days ago, I am experiencing problems related to data download. The batch download, that used to work, currently stops. Sometimes the file download restarts, but others not. Maybe could be something related to file size because the problem prevented the VIIRS complete dowloads, however MODIS ones can by accomplished (not always). Please see below the output messages from wget, curl and obdaac_download.py downloads.oceandata.sci.gsfc.nasa.gov)[169.154.128.84]:443... conectado.
Petición HTTP enviada, esperando respuesta... 200 OK
Longitud: 452945750 (432M) [application/octet-stream]
Guardando como: ‘SNPP_VIIRS.20231002T135400.L1A.nc’
SNPP_VIIRS.20231002T135400.L1A.nc 9%[========> ] 41,01M 975KB/s en 38s
2023-10-04 14:30:56 (1,08 MB/s) - Conexión cerrada en el byte 192434216. Reintentando.
--2023-10-04 14:30:59-- (intento: 4) https://oceandata.sci.gsfc.nasa.gov/getfile/SNPP_VIIRS.20231002T135400.L1A.nc
Conectando con oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)[169.154.128.84]:443... conectado.
Petición HTTP enviada, esperando respuesta... 200 OK
Longitud: 452945750 (432M) [application/octet-stream]
Guardando como: ‘SNPP_VIIRS.20231002T135400.L1A.nc’
SNPP_VIIRS.20231002T135400.L1A.nc 36%[=====================================> ] 157,86M 1,04MB/s en 2m 19s
2023-10-04 14:33:19 (1,13 MB/s) - Conexión cerrada en el byte 192434216. Reintentando.
.....
------------ Using curl ----------------------
curl -O -b ~/.urs_cookies -c ~/.urs_cookies -L -n https://oceandata.sci.gsfc.nasa.gov/ob/getfile/SNPP_VIIRS.20231002T135400.L1A.nc
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 162 0 162 0 0 419 0 --:--:-- --:--:-- --:--:-- 419
100 395 0 395 0 0 764 0 --:--:-- --:--:-- --:--:-- 764
100 545 0 545 0 0 837 0 --:--:-- --:--:-- --:--:-- 0
100 184 0 184 0 0 111 0 --:--:-- 0:00:01 --:--:-- 0
100 333 0 333 0 0 133 0 --:--:-- 0:00:02 --:--:-- 133
19 431M 19 84.9M 0 0 1119k 0 0:06:35 0:01:17 0:05:18 1202k
curl: (92) HTTP/2 stream 0 was not closed cleanly: INTERNAL_ERROR (err 2)
---------- Using obdaac_download.py -----------------------------------------
./obdaac_download.py -v --checksum --filelist to_download.txt
Output directory: /home/josep/Descargas
Retrieving SNPP_VIIRS.20231002T135400.L1A.nc
Downloading SNPP_VIIRS.20231002T135400.L1A.nc ( 431.96 MBs)
[========= ]
...Done
The file SNPP_VIIRS.20231002T135400.L1A.nc failed checksum test
There was a problem retrieving SNPP_VIIRS.20231002T135400.L1A.nc (received status 1)
Retrieving SNPP_VIIRS.20231002T140000.L1A.nc
Downloading SNPP_VIIRS.20231002T140000.L1A.nc ( 461.45 MBs)
[== ]
...Done
The file SNPP_VIIRS.20231002T140000.L1A.nc failed checksum test
There was a problem retrieving SNPP_VIIRS.20231002T140000.L1A.nc (received status 1)
Retrieving SNPP_VIIRS.20231002T140600.L1A.nc
Downloading SNPP_VIIRS.20231002T140600.L1A.nc ( 434.29 MBs)
[===== ]
...Done
The file SNPP_VIIRS.20231002T140600.L1A.nc failed checksum test
There was a problem retrieving SNPP_VIIRS.20231002T140600.L1A.nc (received status 1)
Best regards
Josep
We regularly download L1A data from MODIS and VIIRS, in order to provide a marine monitoring system for the local governemet. Starting from few days ago, I am experiencing problems related to data download. The batch download, that used to work, currently stops. Sometimes the file download restarts, but others not. Maybe could be something related to file size because the problem prevented the VIIRS complete dowloads, however MODIS ones can by accomplished (not always). Please see below the output messages from wget, curl and obdaac_download.py downloads:
------------ Using wget ----------------------
wget --load-cookies ~/.urs_cookies --save-cookies ~/.urs_cookies --auth-no-challenge=on --content-disposition --keep-session-cookies https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/SNPP_VIIRS.20231002T135400.L1A.nc
wget --load-cookies ~/.urs_cookies --save-cookies ~/.urs_cookies --auth-no-challenge=on --content-disposition --keep-session-cookies https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/SNPP_VIIRS.20231002T135400.L1A.nc
--2023-10-04 14:26:57-- https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/SNPP_VIIRS.20231002T135400.L1A.nc
Resolviendo oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)... 169.154.128.84, 2001:4d0128::84
Conectando con oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)[169.154.128.84]:443... conectado.
Petición HTTP enviada, esperando respuesta... 301 Moved Permanently
Ubicación: https://oceandata.sci.gsfc.nasa.gov/getfile/SNPP_VIIRS.20231002T135400.L1A.nc [siguiente]
--2023-10-04 14:26:57-- https://oceandata.sci.gsfc.nasa.gov/getfile/SNPP_VIIRS.20231002T135400.L1A.nc
Reutilizando la conexión con oceandata.sci.gsfc.nasa.gov:443.
Petición HTTP enviada, esperando respuesta... 302 FOUND
Ubicación: https://oceandata.sci.gsfc.nasa.gov/getfile/urs/?next=http%3A%2F%2Flocalhost%3A5007%2Fgetfile%2FSNPP_VIIRS.20231002T135400.L1A.nc [siguiente]
--2023-10-04 14:26:57-- https://oceandata.sci.gsfc.nasa.gov/getfile/urs/?next=http%3A%2F%2Flocalhost%3A5007%2Fgetfile%2FSNPP_VIIRS.20231002T135400.L1A.nc
Reutilizando la conexión con oceandata.sci.gsfc.nasa.gov:443.
Petición HTTP enviada, esperando respuesta... 302 FOUND
Ubicación: https://urs.earthdata.nasa.gov//oauth/authorize?client_id=pDPu0awH156XLrK6VV0Y0w&response_type=code&redirect_uri=https://oceandata.sci.gsfc.nasa.gov/getfile/urs/ [siguiente]
--2023-10-04 14:26:57-- https://urs.earthdata.nasa.gov//oauth/authorize?client_id=pDPu0awH156XLrK6VV0Y0w&response_type=code&redirect_uri=https://oceandata.sci.gsfc.nasa.gov/getfile/urs/
Resolviendo urs.earthdata.nasa.gov (urs.earthdata.nasa.gov)... 198.118.243.33, 2001:4d04081::89
Conectando con urs.earthdata.nasa.gov (urs.earthdata.nasa.gov)[198.118.243.33]:443... conectado.
Petición HTTP enviada, esperando respuesta... 302 Found
Ubicación: https://oceandata.sci.gsfc.nasa.gov/getfile/urs/?code=da1a2479a27a8bc0240914bd142b3af7972d5ec2e7fd76cd4ca394431c1b0c9d [siguiente]
--2023-10-04 14:26:58-- https://oceandata.sci.gsfc.nasa.gov/getfile/urs/?code=da1a2479a27a8bc0240914bd142b3af7972d5ec2e7fd76cd4ca394431c1b0c9d
Conectando con oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)[169.154.128.84]:443... conectado.
Petición HTTP enviada, esperando respuesta... 302 FOUND
Ubicación: https://oceandata.sci.gsfc.nasa.gov/getfile/SNPP_VIIRS.20231002T135400.L1A.nc [siguiente]
--2023-10-04 14:26:59-- https://oceandata.sci.gsfc.nasa.gov/getfile/SNPP_VIIRS.20231002T135400.L1A.nc
Reutilizando la conexión con oceandata.sci.gsfc.nasa.gov:443.
Petición HTTP enviada, esperando respuesta... 200 OK
Longitud: 452945750 (432M) [application/octet-stream]
Guardando como: ‘SNPP_VIIRS.20231002T135400.L1A.nc’
SNPP_VIIRS.20231002T135400.L1A.nc 6%[======> ] 29,94M 1,01MB/s en 30s
2023-10-04 14:27:29 (1,01 MB/s) - Conexión cerrada en el byte 31399712. Reintentando.
--2023-10-04 14:27:30-- (intento: 2) https://oceandata.sci.gsfc.nasa.gov/getfile/SNPP_VIIRS.20231002T135400.L1A.nc
Conectando con oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)[169.154.128.84]:443... conectado.
Petición HTTP enviada, esperando respuesta... 200 OK
Longitud: 452945750 (432M) [application/octet-stream]
Guardando como: ‘SNPP_VIIRS.20231002T135400.L1A.nc’
SNPP_VIIRS.20231002T135400.L1A.nc 9%[=========> ] 42,26M 1,02MB/s eta 5m 56s ^SNPP_VIIRS.20231002T135400.L1A.nc 42%[===========================================> ] 183,52M 1,19MB/s en 2m 45s
2023-10-04 14:30:15 (1,12 MB/s) - Conexión cerrada en el byte 192434216. Reintentando.
--2023-10-04 14:30:17-- (intento: 3) https://oceandata.sci.gsfc.nasa.gov/getfile/SNPP_VIIRS.20231002T135400.L1A.nc
Conectando con oceandata.sci.gsfc.nasa.gov (Hello We regularly download L1A data from MODIS and VIIRS, in order to provide a marine monitoring system for the local governemet. Starting from few days ago, I am experiencing problems related to data download. The batch download, that used to work, currently stops. Sometimes the file download restarts, but others not. Maybe could be something related to file size because the problem prevented the VIIRS complete dowloads, however MODIS ones can by accomplished (not always). Please see below the output messages from wget, curl and obdaac_download.py downloads.oceandata.sci.gsfc.nasa.gov)[169.154.128.84]:443... conectado.
Petición HTTP enviada, esperando respuesta... 200 OK
Longitud: 452945750 (432M) [application/octet-stream]
Guardando como: ‘SNPP_VIIRS.20231002T135400.L1A.nc’
SNPP_VIIRS.20231002T135400.L1A.nc 9%[========> ] 41,01M 975KB/s en 38s
2023-10-04 14:30:56 (1,08 MB/s) - Conexión cerrada en el byte 192434216. Reintentando.
--2023-10-04 14:30:59-- (intento: 4) https://oceandata.sci.gsfc.nasa.gov/getfile/SNPP_VIIRS.20231002T135400.L1A.nc
Conectando con oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)[169.154.128.84]:443... conectado.
Petición HTTP enviada, esperando respuesta... 200 OK
Longitud: 452945750 (432M) [application/octet-stream]
Guardando como: ‘SNPP_VIIRS.20231002T135400.L1A.nc’
SNPP_VIIRS.20231002T135400.L1A.nc 36%[=====================================> ] 157,86M 1,04MB/s en 2m 19s
2023-10-04 14:33:19 (1,13 MB/s) - Conexión cerrada en el byte 192434216. Reintentando.
.....
------------ Using curl ----------------------
curl -O -b ~/.urs_cookies -c ~/.urs_cookies -L -n https://oceandata.sci.gsfc.nasa.gov/ob/getfile/SNPP_VIIRS.20231002T135400.L1A.nc
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 162 0 162 0 0 419 0 --:--:-- --:--:-- --:--:-- 419
100 395 0 395 0 0 764 0 --:--:-- --:--:-- --:--:-- 764
100 545 0 545 0 0 837 0 --:--:-- --:--:-- --:--:-- 0
100 184 0 184 0 0 111 0 --:--:-- 0:00:01 --:--:-- 0
100 333 0 333 0 0 133 0 --:--:-- 0:00:02 --:--:-- 133
19 431M 19 84.9M 0 0 1119k 0 0:06:35 0:01:17 0:05:18 1202k
curl: (92) HTTP/2 stream 0 was not closed cleanly: INTERNAL_ERROR (err 2)
---------- Using obdaac_download.py -----------------------------------------
./obdaac_download.py -v --checksum --filelist to_download.txt
Output directory: /home/josep/Descargas
Retrieving SNPP_VIIRS.20231002T135400.L1A.nc
Downloading SNPP_VIIRS.20231002T135400.L1A.nc ( 431.96 MBs)
[========= ]
...Done
The file SNPP_VIIRS.20231002T135400.L1A.nc failed checksum test
There was a problem retrieving SNPP_VIIRS.20231002T135400.L1A.nc (received status 1)
Retrieving SNPP_VIIRS.20231002T140000.L1A.nc
Downloading SNPP_VIIRS.20231002T140000.L1A.nc ( 461.45 MBs)
[== ]
...Done
The file SNPP_VIIRS.20231002T140000.L1A.nc failed checksum test
There was a problem retrieving SNPP_VIIRS.20231002T140000.L1A.nc (received status 1)
Retrieving SNPP_VIIRS.20231002T140600.L1A.nc
Downloading SNPP_VIIRS.20231002T140600.L1A.nc ( 434.29 MBs)
[===== ]
...Done
The file SNPP_VIIRS.20231002T140600.L1A.nc failed checksum test
There was a problem retrieving SNPP_VIIRS.20231002T140600.L1A.nc (received status 1)
Best regards
Josep
Tags:
-
- Subject Matter Expert
- Posts: 79
- Joined: Fri Jun 03, 2022 10:54 am America/New_York
- Location: NASA GSFC
- Been thanked: 1 time
- Contact:
Re: L1A data downloads stop
Hi Josep,
It seems the files were located on our server but the downloading was incomplete. Do you mind trying to use the app key and let us know how it works? You can find more details on this page: https://oceancolor.gsfc.nasa.gov/data/download_methods/
It seems the files were located on our server but the downloading was incomplete. Do you mind trying to use the app key and let us know how it works? You can find more details on this page: https://oceancolor.gsfc.nasa.gov/data/download_methods/
-
- Subject Matter Expert
- Posts: 378
- Joined: Fri Feb 05, 2021 9:17 am America/New_York
- Been thanked: 4 times
Re: L1A data downloads stop
Hello Josep,
I am not able to reproduce the error.
But I noticed that your consecutive downloads failed at the exact same byte:
2023-10-04 14:30:15 (1,12 MB/s) - Conexión cerrada en el byte 192434216. Reintentando.
2023-10-04 14:30:56 (1,08 MB/s) - Conexión cerrada en el byte 192434216. Reintentando.
2023-10-04 14:33:19 (1,13 MB/s) - Conexión cerrada en el byte 192434216. Reintentando.
Are you sure that you are not filling a disk locally?
Tommy
I am not able to reproduce the error.
But I noticed that your consecutive downloads failed at the exact same byte:
2023-10-04 14:30:15 (1,12 MB/s) - Conexión cerrada en el byte 192434216. Reintentando.
2023-10-04 14:30:56 (1,08 MB/s) - Conexión cerrada en el byte 192434216. Reintentando.
2023-10-04 14:33:19 (1,13 MB/s) - Conexión cerrada en el byte 192434216. Reintentando.
Are you sure that you are not filling a disk locally?
Tommy
Re: L1A data downloads stop
Hello Tommy
It is not a disk space problem. Today I have changed the cron schedule and the download has improved, although it continue being slow,. At the end ,today they have been successful. Sometimes the downloads have stopped, but they restarted (sometimes after several minutes).
Josep
It is not a disk space problem. Today I have changed the cron schedule and the download has improved, although it continue being slow,. At the end ,today they have been successful. Sometimes the downloads have stopped, but they restarted (sometimes after several minutes).
Josep
-
- Subject Matter Expert
- Posts: 378
- Joined: Fri Feb 05, 2021 9:17 am America/New_York
- Been thanked: 4 times
Re: L1A data downloads stop
It's strange that the download fails at the exact same byte each time. There is a bottleneck somewhere in your route.
When I download the same file to an off-premises computer, I see 15-20 MB/s transfer rates.
Tommy
When I download the same file to an off-premises computer, I see 15-20 MB/s transfer rates.
Tommy
Re: L1A data downloads stop
Hello I have tried the downloads at home during the weekend and they work properly. I have asked again to the network responsible, and there were a misunderstanding, the bottleneck is from our side.
I am very sorry for the time that you have spent with issue
I apologize for hat
I apologize for that
Josep
I am very sorry for the time that you have spent with issue
I apologize for hat
I apologize for that
Josep
-
- Subject Matter Expert
- Posts: 378
- Joined: Fri Feb 05, 2021 9:17 am America/New_York
- Been thanked: 4 times
Re: L1A data downloads stop
No worries Josep, problems also happen on our side, so we appreciate the reports.
Glad we were able to help track this down.
Tommy
Glad we were able to help track this down.
Tommy