We regularly download L1A data from MODIS and VIIRS, in order to provide a marine monitoring system for the local governemet. Starting from few days ago, I am experiencing problems related to data download. The batch download, that used to work, currently stops. Sometimes the file download restarts, but others not. Maybe could be something related to file size because the problem prevented the VIIRS complete dowloads, however MODIS ones can by accomplished (not always). Please see below the output messages from wget, curl and obdaac_download.py downloads:
------------ Using wget ----------------------
wget --load-cookies ~/.urs_cookies --save-cookies ~/.urs_cookies --auth-no-challenge=on --content-disposition --keep-session-cookies https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/SNPP_VIIRS.20231002T135400.L1A.nc
wget --load-cookies ~/.urs_cookies --save-cookies ~/.urs_cookies --auth-no-challenge=on --content-disposition --keep-session-cookies https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/SNPP_VIIRS.20231002T135400.L1A.nc
--2023-10-04 14:26:57-- https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/SNPP_VIIRS.20231002T135400.L1A.nc
Resolviendo oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)... 169.154.128.84, 2001:4d0
Conectando con oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)[169.154.128.84]:443... conectado.
Petición HTTP enviada, esperando respuesta... 301 Moved Permanently
Ubicación: https://oceandata.sci.gsfc.nasa.gov/getfile/SNPP_VIIRS.20231002T135400.L1A.nc [siguiente]
--2023-10-04 14:26:57-- https://oceandata.sci.gsfc.nasa.gov/getfile/SNPP_VIIRS.20231002T135400.L1A.nc
Reutilizando la conexión con oceandata.sci.gsfc.nasa.gov:443.
Petición HTTP enviada, esperando respuesta... 302 FOUND
Ubicación: https://oceandata.sci.gsfc.nasa.gov/getfile/urs/?next=http%3A%2F%2Flocalhost%3A5007%2Fgetfile%2FSNPP_VIIRS.20231002T135400.L1A.nc [siguiente]
--2023-10-04 14:26:57-- https://oceandata.sci.gsfc.nasa.gov/getfile/urs/?next=http%3A%2F%2Flocalhost%3A5007%2Fgetfile%2FSNPP_VIIRS.20231002T135400.L1A.nc
Reutilizando la conexión con oceandata.sci.gsfc.nasa.gov:443.
Petición HTTP enviada, esperando respuesta... 302 FOUND
Ubicación: https://urs.earthdata.nasa.gov//oauth/authorize?client_id=pDPu0awH156XLrK6VV0Y0w&response_type=code&redirect_uri=https://oceandata.sci.gsfc.nasa.gov/getfile/urs/ [siguiente]
--2023-10-04 14:26:57-- https://urs.earthdata.nasa.gov//oauth/authorize?client_id=pDPu0awH156XLrK6VV0Y0w&response_type=code&redirect_uri=https://oceandata.sci.gsfc.nasa.gov/getfile/urs/
Resolviendo urs.earthdata.nasa.gov (urs.earthdata.nasa.gov)... 198.118.243.33, 2001:4d0
Conectando con urs.earthdata.nasa.gov (urs.earthdata.nasa.gov)[198.118.243.33]:443... conectado.
Petición HTTP enviada, esperando respuesta... 302 Found
Ubicación: https://oceandata.sci.gsfc.nasa.gov/getfile/urs/?code=da1a2479a27a8bc0240914bd142b3af7972d5ec2e7fd76cd4ca394431c1b0c9d [siguiente]
--2023-10-04 14:26:58-- https://oceandata.sci.gsfc.nasa.gov/getfile/urs/?code=da1a2479a27a8bc0240914bd142b3af7972d5ec2e7fd76cd4ca394431c1b0c9d
Conectando con oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)[169.154.128.84]:443... conectado.
Petición HTTP enviada, esperando respuesta... 302 FOUND
Ubicación: https://oceandata.sci.gsfc.nasa.gov/getfile/SNPP_VIIRS.20231002T135400.L1A.nc [siguiente]
--2023-10-04 14:26:59-- https://oceandata.sci.gsfc.nasa.gov/getfile/SNPP_VIIRS.20231002T135400.L1A.nc
Reutilizando la conexión con oceandata.sci.gsfc.nasa.gov:443.
Petición HTTP enviada, esperando respuesta... 200 OK
Longitud: 452945750 (432M) [application/octet-stream]
Guardando como: ‘SNPP_VIIRS.20231002T135400.L1A.nc’
SNPP_VIIRS.20231002T135400.L1A.nc 6%[======> ] 29,94M 1,01MB/s en 30s
2023-10-04 14:27:29 (1,01 MB/s) - Conexión cerrada en el byte 31399712. Reintentando.
--2023-10-04 14:27:30-- (intento: 2) https://oceandata.sci.gsfc.nasa.gov/getfile/SNPP_VIIRS.20231002T135400.L1A.nc
Conectando con oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)[169.154.128.84]:443... conectado.
Petición HTTP enviada, esperando respuesta... 200 OK
Longitud: 452945750 (432M) [application/octet-stream]
Guardando como: ‘SNPP_VIIRS.20231002T135400.L1A.nc’
SNPP_VIIRS.20231002T135400.L1A.nc 9%[=========> ] 42,26M 1,02MB/s eta 5m 56s ^SNPP_VIIRS.20231002T135400.L1A.nc 42%[===========================================> ] 183,52M 1,19MB/s en 2m 45s
2023-10-04 14:30:15 (1,12 MB/s) - Conexión cerrada en el byte 192434216. Reintentando.
--2023-10-04 14:30:17-- (intento: 3) https://oceandata.sci.gsfc.nasa.gov/getfile/SNPP_VIIRS.20231002T135400.L1A.nc
Conectando con oceandata.sci.gsfc.nasa.gov (Hello We regularly download L1A data from MODIS and VIIRS, in order to provide a marine monitoring system for the local governemet. Starting from few days ago, I am experiencing problems related to data download. The batch download, that used to work, currently stops. Sometimes the file download restarts, but others not. Maybe could be something related to file size because the problem prevented the VIIRS complete dowloads, however MODIS ones can by accomplished (not always). Please see below the output messages from wget, curl and obdaac_download.py downloads.oceandata.sci.gsfc.nasa.gov)[169.154.128.84]:443... conectado.
Petición HTTP enviada, esperando respuesta... 200 OK
Longitud: 452945750 (432M) [application/octet-stream]
Guardando como: ‘SNPP_VIIRS.20231002T135400.L1A.nc’
SNPP_VIIRS.20231002T135400.L1A.nc 9%[========> ] 41,01M 975KB/s en 38s
2023-10-04 14:30:56 (1,08 MB/s) - Conexión cerrada en el byte 192434216. Reintentando.
--2023-10-04 14:30:59-- (intento: 4) https://oceandata.sci.gsfc.nasa.gov/getfile/SNPP_VIIRS.20231002T135400.L1A.nc
Conectando con oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)[169.154.128.84]:443... conectado.
Petición HTTP enviada, esperando respuesta... 200 OK
Longitud: 452945750 (432M) [application/octet-stream]
Guardando como: ‘SNPP_VIIRS.20231002T135400.L1A.nc’
SNPP_VIIRS.20231002T135400.L1A.nc 36%[=====================================> ] 157,86M 1,04MB/s en 2m 19s
2023-10-04 14:33:19 (1,13 MB/s) - Conexión cerrada en el byte 192434216. Reintentando.
.....
------------ Using curl ----------------------
curl -O -b ~/.urs_cookies -c ~/.urs_cookies -L -n https://oceandata.sci.gsfc.nasa.gov/ob/getfile/SNPP_VIIRS.20231002T135400.L1A.nc
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 162 0 162 0 0 419 0 --:--:-- --:--:-- --:--:-- 419
100 395 0 395 0 0 764 0 --:--:-- --:--:-- --:--:-- 764
100 545 0 545 0 0 837 0 --:--:-- --:--:-- --:--:-- 0
100 184 0 184 0 0 111 0 --:--:-- 0:00:01 --:--:-- 0
100 333 0 333 0 0 133 0 --:--:-- 0:00:02 --:--:-- 133
19 431M 19 84.9M 0 0 1119k 0 0:06:35 0:01:17 0:05:18 1202k
curl: (92) HTTP/2 stream 0 was not closed cleanly: INTERNAL_ERROR (err 2)
---------- Using obdaac_download.py -----------------------------------------
./obdaac_download.py -v --checksum --filelist to_download.txt
Output directory: /home/josep/Descargas
Retrieving SNPP_VIIRS.20231002T135400.L1A.nc
Downloading SNPP_VIIRS.20231002T135400.L1A.nc ( 431.96 MBs)
[========= ]
...Done
The file SNPP_VIIRS.20231002T135400.L1A.nc failed checksum test
There was a problem retrieving SNPP_VIIRS.20231002T135400.L1A.nc (received status 1)
Retrieving SNPP_VIIRS.20231002T140000.L1A.nc
Downloading SNPP_VIIRS.20231002T140000.L1A.nc ( 461.45 MBs)
[== ]
...Done
The file SNPP_VIIRS.20231002T140000.L1A.nc failed checksum test
There was a problem retrieving SNPP_VIIRS.20231002T140000.L1A.nc (received status 1)
Retrieving SNPP_VIIRS.20231002T140600.L1A.nc
Downloading SNPP_VIIRS.20231002T140600.L1A.nc ( 434.29 MBs)
[===== ]
...Done
The file SNPP_VIIRS.20231002T140600.L1A.nc failed checksum test
There was a problem retrieving SNPP_VIIRS.20231002T140600.L1A.nc (received status 1)
Best regards
Josep