L1A data downloads stop

Use this Forum to find information on, or ask a question about, NASA Earth Science data.
Post Reply
jcoca
Posts: 20
Joined: Thu Feb 11, 2021 6:51 am America/New_York
Answers: 0

L1A data downloads stop

by jcoca » Thu Oct 05, 2023 6:20 am America/New_York

Hello

We regularly download L1A data from MODIS and VIIRS, in order to provide a marine monitoring system for the local governemet. Starting from few days ago, I am experiencing problems related to data download. The batch download, that used to work, currently stops. Sometimes the file download restarts, but others not. Maybe could be something related to file size because the problem prevented the VIIRS complete dowloads, however MODIS ones can by accomplished (not always). Please see below the output messages from wget, curl and obdaac_download.py downloads:

------------ Using wget ----------------------

wget --load-cookies ~/.urs_cookies --save-cookies ~/.urs_cookies --auth-no-challenge=on --content-disposition --keep-session-cookies https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/SNPP_VIIRS.20231002T135400.L1A.nc

wget --load-cookies ~/.urs_cookies --save-cookies ~/.urs_cookies --auth-no-challenge=on --content-disposition --keep-session-cookies https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/SNPP_VIIRS.20231002T135400.L1A.nc
--2023-10-04 14:26:57-- https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/SNPP_VIIRS.20231002T135400.L1A.nc
Resolviendo oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)... 169.154.128.84, 2001:4d0:2418:128::84
Conectando con oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)[169.154.128.84]:443... conectado.
Petición HTTP enviada, esperando respuesta... 301 Moved Permanently
Ubicación: https://oceandata.sci.gsfc.nasa.gov/getfile/SNPP_VIIRS.20231002T135400.L1A.nc [siguiente]
--2023-10-04 14:26:57-- https://oceandata.sci.gsfc.nasa.gov/getfile/SNPP_VIIRS.20231002T135400.L1A.nc
Reutilizando la conexión con oceandata.sci.gsfc.nasa.gov:443.
Petición HTTP enviada, esperando respuesta... 302 FOUND
Ubicación: https://oceandata.sci.gsfc.nasa.gov/getfile/urs/?next=http%3A%2F%2Flocalhost%3A5007%2Fgetfile%2FSNPP_VIIRS.20231002T135400.L1A.nc [siguiente]
--2023-10-04 14:26:57-- https://oceandata.sci.gsfc.nasa.gov/getfile/urs/?next=http%3A%2F%2Flocalhost%3A5007%2Fgetfile%2FSNPP_VIIRS.20231002T135400.L1A.nc
Reutilizando la conexión con oceandata.sci.gsfc.nasa.gov:443.
Petición HTTP enviada, esperando respuesta... 302 FOUND
Ubicación: https://urs.earthdata.nasa.gov//oauth/authorize?client_id=pDPu0awH156XLrK6VV0Y0w&response_type=code&redirect_uri=https://oceandata.sci.gsfc.nasa.gov/getfile/urs/ [siguiente]
--2023-10-04 14:26:57-- https://urs.earthdata.nasa.gov//oauth/authorize?client_id=pDPu0awH156XLrK6VV0Y0w&response_type=code&redirect_uri=https://oceandata.sci.gsfc.nasa.gov/getfile/urs/
Resolviendo urs.earthdata.nasa.gov (urs.earthdata.nasa.gov)... 198.118.243.33, 2001:4d0:241a:4081::89
Conectando con urs.earthdata.nasa.gov (urs.earthdata.nasa.gov)[198.118.243.33]:443... conectado.
Petición HTTP enviada, esperando respuesta... 302 Found
Ubicación: https://oceandata.sci.gsfc.nasa.gov/getfile/urs/?code=da1a2479a27a8bc0240914bd142b3af7972d5ec2e7fd76cd4ca394431c1b0c9d [siguiente]
--2023-10-04 14:26:58-- https://oceandata.sci.gsfc.nasa.gov/getfile/urs/?code=da1a2479a27a8bc0240914bd142b3af7972d5ec2e7fd76cd4ca394431c1b0c9d
Conectando con oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)[169.154.128.84]:443... conectado.
Petición HTTP enviada, esperando respuesta... 302 FOUND
Ubicación: https://oceandata.sci.gsfc.nasa.gov/getfile/SNPP_VIIRS.20231002T135400.L1A.nc [siguiente]
--2023-10-04 14:26:59-- https://oceandata.sci.gsfc.nasa.gov/getfile/SNPP_VIIRS.20231002T135400.L1A.nc
Reutilizando la conexión con oceandata.sci.gsfc.nasa.gov:443.
Petición HTTP enviada, esperando respuesta... 200 OK
Longitud: 452945750 (432M) [application/octet-stream]
Guardando como: ‘SNPP_VIIRS.20231002T135400.L1A.nc’

SNPP_VIIRS.20231002T135400.L1A.nc 6%[======> ] 29,94M 1,01MB/s en 30s

2023-10-04 14:27:29 (1,01 MB/s) - Conexión cerrada en el byte 31399712. Reintentando.

--2023-10-04 14:27:30-- (intento: 2) https://oceandata.sci.gsfc.nasa.gov/getfile/SNPP_VIIRS.20231002T135400.L1A.nc
Conectando con oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)[169.154.128.84]:443... conectado.
Petición HTTP enviada, esperando respuesta... 200 OK
Longitud: 452945750 (432M) [application/octet-stream]
Guardando como: ‘SNPP_VIIRS.20231002T135400.L1A.nc’

SNPP_VIIRS.20231002T135400.L1A.nc 9%[=========> ] 42,26M 1,02MB/s eta 5m 56s ^SNPP_VIIRS.20231002T135400.L1A.nc 42%[===========================================> ] 183,52M 1,19MB/s en 2m 45s

2023-10-04 14:30:15 (1,12 MB/s) - Conexión cerrada en el byte 192434216. Reintentando.

--2023-10-04 14:30:17-- (intento: 3) https://oceandata.sci.gsfc.nasa.gov/getfile/SNPP_VIIRS.20231002T135400.L1A.nc
Conectando con oceandata.sci.gsfc.nasa.gov (Hello We regularly download L1A data from MODIS and VIIRS, in order to provide a marine monitoring system for the local governemet. Starting from few days ago, I am experiencing problems related to data download. The batch download, that used to work, currently stops. Sometimes the file download restarts, but others not. Maybe could be something related to file size because the problem prevented the VIIRS complete dowloads, however MODIS ones can by accomplished (not always). Please see below the output messages from wget, curl and obdaac_download.py downloads.oceandata.sci.gsfc.nasa.gov)[169.154.128.84]:443... conectado.
Petición HTTP enviada, esperando respuesta... 200 OK
Longitud: 452945750 (432M) [application/octet-stream]
Guardando como: ‘SNPP_VIIRS.20231002T135400.L1A.nc’

SNPP_VIIRS.20231002T135400.L1A.nc 9%[========> ] 41,01M 975KB/s en 38s

2023-10-04 14:30:56 (1,08 MB/s) - Conexión cerrada en el byte 192434216. Reintentando.

--2023-10-04 14:30:59-- (intento: 4) https://oceandata.sci.gsfc.nasa.gov/getfile/SNPP_VIIRS.20231002T135400.L1A.nc
Conectando con oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)[169.154.128.84]:443... conectado.
Petición HTTP enviada, esperando respuesta... 200 OK
Longitud: 452945750 (432M) [application/octet-stream]
Guardando como: ‘SNPP_VIIRS.20231002T135400.L1A.nc’

SNPP_VIIRS.20231002T135400.L1A.nc 36%[=====================================> ] 157,86M 1,04MB/s en 2m 19s

2023-10-04 14:33:19 (1,13 MB/s) - Conexión cerrada en el byte 192434216. Reintentando.

.....




------------ Using curl ----------------------

curl -O -b ~/.urs_cookies -c ~/.urs_cookies -L -n https://oceandata.sci.gsfc.nasa.gov/ob/getfile/SNPP_VIIRS.20231002T135400.L1A.nc
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 162 0 162 0 0 419 0 --:--:-- --:--:-- --:--:-- 419
100 395 0 395 0 0 764 0 --:--:-- --:--:-- --:--:-- 764
100 545 0 545 0 0 837 0 --:--:-- --:--:-- --:--:-- 0
100 184 0 184 0 0 111 0 --:--:-- 0:00:01 --:--:-- 0
100 333 0 333 0 0 133 0 --:--:-- 0:00:02 --:--:-- 133
19 431M 19 84.9M 0 0 1119k 0 0:06:35 0:01:17 0:05:18 1202k
curl: (92) HTTP/2 stream 0 was not closed cleanly: INTERNAL_ERROR (err 2)


---------- Using obdaac_download.py -----------------------------------------

./obdaac_download.py -v --checksum --filelist to_download.txt
Output directory: /home/josep/Descargas
Retrieving SNPP_VIIRS.20231002T135400.L1A.nc
Downloading SNPP_VIIRS.20231002T135400.L1A.nc ( 431.96 MBs)
[========= ]
...Done
The file SNPP_VIIRS.20231002T135400.L1A.nc failed checksum test
There was a problem retrieving SNPP_VIIRS.20231002T135400.L1A.nc (received status 1)
Retrieving SNPP_VIIRS.20231002T140000.L1A.nc
Downloading SNPP_VIIRS.20231002T140000.L1A.nc ( 461.45 MBs)
[== ]
...Done
The file SNPP_VIIRS.20231002T140000.L1A.nc failed checksum test
There was a problem retrieving SNPP_VIIRS.20231002T140000.L1A.nc (received status 1)
Retrieving SNPP_VIIRS.20231002T140600.L1A.nc
Downloading SNPP_VIIRS.20231002T140600.L1A.nc ( 434.29 MBs)
[===== ]
...Done
The file SNPP_VIIRS.20231002T140600.L1A.nc failed checksum test
There was a problem retrieving SNPP_VIIRS.20231002T140600.L1A.nc (received status 1)

Best regards
Josep

Filters:

OB General Science - guoqingw
Subject Matter Expert
Subject Matter Expert
Posts: 79
Joined: Fri Jun 03, 2022 10:54 am America/New_York
Answers: 0
Location: NASA GSFC
Been thanked: 1 time
Contact:

Re: L1A data downloads stop

by OB General Science - guoqingw » Thu Oct 05, 2023 2:15 pm America/New_York

Hi Josep,

It seems the files were located on our server but the downloading was incomplete. Do you mind trying to use the app key and let us know how it works? You can find more details on this page: https://oceancolor.gsfc.nasa.gov/data/download_methods/

OB ODPS - towens
Subject Matter Expert
Subject Matter Expert
Posts: 381
Joined: Fri Feb 05, 2021 9:17 am America/New_York
Answers: 0
Been thanked: 4 times

Re: L1A data downloads stop

by OB ODPS - towens » Thu Oct 05, 2023 2:30 pm America/New_York

Hello Josep,
I am not able to reproduce the error.

But I noticed that your consecutive downloads failed at the exact same byte:
2023-10-04 14:30:15 (1,12 MB/s) - Conexión cerrada en el byte 192434216. Reintentando.
2023-10-04 14:30:56 (1,08 MB/s) - Conexión cerrada en el byte 192434216. Reintentando.
2023-10-04 14:33:19 (1,13 MB/s) - Conexión cerrada en el byte 192434216. Reintentando.

Are you sure that you are not filling a disk locally?

Tommy

jcoca
Posts: 20
Joined: Thu Feb 11, 2021 6:51 am America/New_York
Answers: 0

Re: L1A data downloads stop

by jcoca » Fri Oct 06, 2023 9:19 am America/New_York

Hello Tommy
It is not a disk space problem. Today I have changed the cron schedule and the download has improved, although it continue being slow,. At the end ,today they have been successful. Sometimes the downloads have stopped, but they restarted (sometimes after several minutes).

Josep

OB ODPS - towens
Subject Matter Expert
Subject Matter Expert
Posts: 381
Joined: Fri Feb 05, 2021 9:17 am America/New_York
Answers: 0
Been thanked: 4 times

Re: L1A data downloads stop

by OB ODPS - towens » Fri Oct 06, 2023 11:09 am America/New_York

It's strange that the download fails at the exact same byte each time. There is a bottleneck somewhere in your route.
When I download the same file to an off-premises computer, I see 15-20 MB/s transfer rates.

Tommy

jcoca
Posts: 20
Joined: Thu Feb 11, 2021 6:51 am America/New_York
Answers: 0

Re: L1A data downloads stop

by jcoca » Mon Oct 09, 2023 8:19 am America/New_York

Hello I have tried the downloads at home during the weekend and they work properly. I have asked again to the network responsible, and there were a misunderstanding, the bottleneck is from our side.
I am very sorry for the time that you have spent with issue
I apologize for hat

I apologize for that
Josep

OB ODPS - towens
Subject Matter Expert
Subject Matter Expert
Posts: 381
Joined: Fri Feb 05, 2021 9:17 am America/New_York
Answers: 0
Been thanked: 4 times

Re: L1A data downloads stop

by OB ODPS - towens » Mon Oct 09, 2023 8:22 am America/New_York

No worries Josep, problems also happen on our side, so we appreciate the reports.
Glad we were able to help track this down.

Tommy

Post Reply