L1A data downloads stop

[jcoca]

Hello

We regularly download L1A data from MODIS and VIIRS, in order to provide a marine monitoring system for the local governemet. Starting from few days ago, I am experiencing problems related to data download. The batch download, that used to work, currently stops. Sometimes the file download restarts, but others not. Maybe could be something related to file size because the problem prevented the VIIRS complete dowloads, however MODIS ones can by accomplished (not always). Please see below the output messages from wget, curl and obdaac_download.py downloads:

------------ Using wget ----------------------

wget --load-cookies ~/.urs_cookies --save-cookies ~/.urs_cookies --auth-no-challenge=on --content-disposition --keep-session-cookies https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/SNPP_VIIRS.20231002T135400.L1A.nc

wget --load-cookies ~/.urs_cookies --save-cookies ~/.urs_cookies --auth-no-challenge=on --content-disposition --keep-session-cookies https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/SNPP_VIIRS.20231002T135400.L1A.nc
--2023-10-04 14:26:57-- https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/SNPP_VIIRS.20231002T135400.L1A.nc
Resolviendo oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)... 169.154.128.84, 2001:4d0128::84
Conectando con oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)[169.154.128.84]:443... conectado.
Petición HTTP enviada, esperando respuesta... 301 Moved Permanently
Ubicación: https://oceandata.sci.gsfc.nasa.gov/getfile/SNPP_VIIRS.20231002T135400.L1A.nc [siguiente]
--2023-10-04 14:26:57-- https://oceandata.sci.gsfc.nasa.gov/getfile/SNPP_VIIRS.20231002T135400.L1A.nc
Reutilizando la conexión con oceandata.sci.gsfc.nasa.gov:443.
Petición HTTP enviada, esperando respuesta... 302 FOUND
Ubicación: https://oceandata.sci.gsfc.nasa.gov/getfile/urs/?next=http%3A%2F%2Flocalhost%3A5007%2Fgetfile%2FSNPP_VIIRS.20231002T135400.L1A.nc [siguiente]
--2023-10-04 14:26:57-- https://oceandata.sci.gsfc.nasa.gov/getfile/urs/?next=http%3A%2F%2Flocalhost%3A5007%2Fgetfile%2FSNPP_VIIRS.20231002T135400.L1A.nc
Reutilizando la conexión con oceandata.sci.gsfc.nasa.gov:443.
Petición HTTP enviada, esperando respuesta... 302 FOUND
Ubicación: https://urs.earthdata.nasa.gov//oauth/authorize?client_id=pDPu0awH156XLrK6VV0Y0w&response_type=code&redirect_uri=https://oceandata.sci.gsfc.nasa.gov/getfile/urs/ [siguiente]
--2023-10-04 14:26:57-- https://urs.earthdata.nasa.gov//oauth/authorize?client_id=pDPu0awH156XLrK6VV0Y0w&response_type=code&redirect_uri=https://oceandata.sci.gsfc.nasa.gov/getfile/urs/
Resolviendo urs.earthdata.nasa.gov (urs.earthdata.nasa.gov)... 198.118.243.33, 2001:4d04081::89
Conectando con urs.earthdata.nasa.gov (urs.earthdata.nasa.gov)[198.118.243.33]:443... conectado.
Petición HTTP enviada, esperando respuesta... 302 Found
Ubicación: https://oceandata.sci.gsfc.nasa.gov/getfile/urs/?code=da1a2479a27a8bc0240914bd142b3af7972d5ec2e7fd76cd4ca394431c1b0c9d [siguiente]
--2023-10-04 14:26:58-- https://oceandata.sci.gsfc.nasa.gov/getfile/urs/?code=da1a2479a27a8bc0240914bd142b3af7972d5ec2e7fd76cd4ca394431c1b0c9d
Conectando con oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)[169.154.128.84]:443... conectado.
Petición HTTP enviada, esperando respuesta... 302 FOUND
Ubicación: https://oceandata.sci.gsfc.nasa.gov/getfile/SNPP_VIIRS.20231002T135400.L1A.nc [siguiente]
--2023-10-04 14:26:59-- https://oceandata.sci.gsfc.nasa.gov/getfile/SNPP_VIIRS.20231002T135400.L1A.nc
Reutilizando la conexión con oceandata.sci.gsfc.nasa.gov:443.
Petición HTTP enviada, esperando respuesta... 200 OK
Longitud: 452945750 (432M) [application/octet-stream]
Guardando como: ‘SNPP_VIIRS.20231002T135400.L1A.nc’

SNPP_VIIRS.20231002T135400.L1A.nc 6%[======> ] 29,94M 1,01MB/s en 30s

2023-10-04 14:27:29 (1,01 MB/s) - Conexión cerrada en el byte 31399712. Reintentando.

--2023-10-04 14:27:30-- (intento: 2) https://oceandata.sci.gsfc.nasa.gov/getfile/SNPP_VIIRS.20231002T135400.L1A.nc
Conectando con oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)[169.154.128.84]:443... conectado.
Petición HTTP enviada, esperando respuesta... 200 OK
Longitud: 452945750 (432M) [application/octet-stream]
Guardando como: ‘SNPP_VIIRS.20231002T135400.L1A.nc’

SNPP_VIIRS.20231002T135400.L1A.nc 9%[=========> ] 42,26M 1,02MB/s eta 5m 56s ^SNPP_VIIRS.20231002T135400.L1A.nc 42%[===========================================> ] 183,52M 1,19MB/s en 2m 45s

2023-10-04 14:30:15 (1,12 MB/s) - Conexión cerrada en el byte 192434216. Reintentando.

--2023-10-04 14:30:17-- (intento: 3) https://oceandata.sci.gsfc.nasa.gov/getfile/SNPP_VIIRS.20231002T135400.L1A.nc
Conectando con oceandata.sci.gsfc.nasa.gov (Hello We regularly download L1A data from MODIS and VIIRS, in order to provide a marine monitoring system for the local governemet. Starting from few days ago, I am experiencing problems related to data download. The batch download, that used to work, currently stops. Sometimes the file download restarts, but others not. Maybe could be something related to file size because the problem prevented the VIIRS complete dowloads, however MODIS ones can by accomplished (not always). Please see below the output messages from wget, curl and obdaac_download.py downloads.oceandata.sci.gsfc.nasa.gov)[169.154.128.84]:443... conectado.
Petición HTTP enviada, esperando respuesta... 200 OK
Longitud: 452945750 (432M) [application/octet-stream]
Guardando como: ‘SNPP_VIIRS.20231002T135400.L1A.nc’

SNPP_VIIRS.20231002T135400.L1A.nc 9%[========> ] 41,01M 975KB/s en 38s

2023-10-04 14:30:56 (1,08 MB/s) - Conexión cerrada en el byte 192434216. Reintentando.

--2023-10-04 14:30:59-- (intento: 4) https://oceandata.sci.gsfc.nasa.gov/getfile/SNPP_VIIRS.20231002T135400.L1A.nc
Conectando con oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)[169.154.128.84]:443... conectado.
Petición HTTP enviada, esperando respuesta... 200 OK
Longitud: 452945750 (432M) [application/octet-stream]
Guardando como: ‘SNPP_VIIRS.20231002T135400.L1A.nc’

SNPP_VIIRS.20231002T135400.L1A.nc 36%[=====================================> ] 157,86M 1,04MB/s en 2m 19s

2023-10-04 14:33:19 (1,13 MB/s) - Conexión cerrada en el byte 192434216. Reintentando.

.....




------------ Using curl ----------------------

curl -O -b ~/.urs_cookies -c ~/.urs_cookies -L -n https://oceandata.sci.gsfc.nasa.gov/ob/getfile/SNPP_VIIRS.20231002T135400.L1A.nc
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 162 0 162 0 0 419 0 --:--:-- --:--:-- --:--:-- 419
100 395 0 395 0 0 764 0 --:--:-- --:--:-- --:--:-- 764
100 545 0 545 0 0 837 0 --:--:-- --:--:-- --:--:-- 0
100 184 0 184 0 0 111 0 --:--:-- 0:00:01 --:--:-- 0
100 333 0 333 0 0 133 0 --:--:-- 0:00:02 --:--:-- 133
19 431M 19 84.9M 0 0 1119k 0 0:06:35 0:01:17 0:05:18 1202k
curl: (92) HTTP/2 stream 0 was not closed cleanly: INTERNAL_ERROR (err 2)


---------- Using obdaac_download.py -----------------------------------------

./obdaac_download.py -v --checksum --filelist to_download.txt
Output directory: /home/josep/Descargas
Retrieving SNPP_VIIRS.20231002T135400.L1A.nc
Downloading SNPP_VIIRS.20231002T135400.L1A.nc ( 431.96 MBs)
[========= ]
...Done
The file SNPP_VIIRS.20231002T135400.L1A.nc failed checksum test
There was a problem retrieving SNPP_VIIRS.20231002T135400.L1A.nc (received status 1)
Retrieving SNPP_VIIRS.20231002T140000.L1A.nc
Downloading SNPP_VIIRS.20231002T140000.L1A.nc ( 461.45 MBs)
[== ]
...Done
The file SNPP_VIIRS.20231002T140000.L1A.nc failed checksum test
There was a problem retrieving SNPP_VIIRS.20231002T140000.L1A.nc (received status 1)
Retrieving SNPP_VIIRS.20231002T140600.L1A.nc
Downloading SNPP_VIIRS.20231002T140600.L1A.nc ( 434.29 MBs)
[===== ]
...Done
The file SNPP_VIIRS.20231002T140600.L1A.nc failed checksum test
There was a problem retrieving SNPP_VIIRS.20231002T140600.L1A.nc (received status 1)

Best regards
Josep

[OB General Science - guoqingw]

Hi Josep,

It seems the files were located on our server but the downloading was incomplete. Do you mind trying to use the app key and let us know how it works? You can find more details on this page: https://oceancolor.gsfc.nasa.gov/data/download_methods/

[OB ODPS - towens]

Hello Josep,
I am not able to reproduce the error.

But I noticed that your consecutive downloads failed at the exact same byte:
2023-10-04 14:30:15 (1,12 MB/s) - Conexión cerrada en el byte 192434216. Reintentando.
2023-10-04 14:30:56 (1,08 MB/s) - Conexión cerrada en el byte 192434216. Reintentando.
2023-10-04 14:33:19 (1,13 MB/s) - Conexión cerrada en el byte 192434216. Reintentando.

Are you sure that you are not filling a disk locally?

Tommy

[jcoca]

Hello Tommy
It is not a disk space problem. Today I have changed the cron schedule and the download has improved, although it continue being slow,. At the end ,today they have been successful. Sometimes the downloads have stopped, but they restarted (sometimes after several minutes).

Josep

[OB ODPS - towens]

It's strange that the download fails at the exact same byte each time. There is a bottleneck somewhere in your route.
When I download the same file to an off-premises computer, I see 15-20 MB/s transfer rates.

Tommy

[jcoca]

Hello I have tried the downloads at home during the weekend and they work properly. I have asked again to the network responsible, and there were a misunderstanding, the bottleneck is from our side.
I am very sorry for the time that you have spent with issue
I apologize for hat

I apologize for that
Josep

[OB ODPS - towens]

No worries Josep, problems also happen on our side, so we appreciate the reports.
Glad we were able to help track this down.

Tommy