Bulk downloading encounters connection errors

Use this Forum to find information on, or ask a question about, NASA Earth Science data.
Post Reply
vds_downloader
Posts: 4
Joined: Thu Jul 14, 2022 5:58 am America/New_York
Answers: 0

Bulk downloading encounters connection errors

by vds_downloader » Fri Aug 19, 2022 3:52 am America/New_York

I am bulk downloading some GPM SSMI data from GES DISC with wget and am experiencing connection timeouts:

Connecting to gpm1.gesdisc.eosdis.nasa.gov (gpm1.gesdisc.eosdis.nasa.gov)|198.118.197.50|:443... failed: Connection timed out.
Connecting to gpm1.gesdisc.eosdis.nasa.gov (gpm1.gesdisc.eosdis.nasa.gov)|2001:4d0:241a:4041::50|:443... failed: Network is unreachable.

Is this because of a limit on the number of sequential (or concurrent) requests in a certain period of time? And if so, can this limit be increased?

Tags:

GES DISC - mgreene
User Services
User Services
Posts: 186
Joined: Wed Sep 25, 2019 9:35 am America/New_York
Answers: 2
Been thanked: 1 time

Re: Bulk downloading encounters connection errors

by GES DISC - mgreene » Fri Aug 19, 2022 10:55 am America/New_York

Thank you for your interest in using GPM data.
In regards to the error you received, the system may issue service temporarily
unavailable for more than one concurrent connection in certain circumstances.
Please review/reduce the number of concurrent connections you are submitting,
and try again.

vds_downloader
Posts: 4
Joined: Thu Jul 14, 2022 5:58 am America/New_York
Answers: 0

Re: Bulk downloading encounters connection errors

by vds_downloader » Mon Aug 22, 2022 7:23 am America/New_York

Thank you for you reaction. Indeed when I decrease the number of wget processes running in parallel I get much less Connection timeouts. But when I have only one wget running I still get the Connection timeouts. It runs for 30 seconds or so and downloads a couple of files but then stalls more than a minute.

GES DISC - mgreene
User Services
User Services
Posts: 186
Joined: Wed Sep 25, 2019 9:35 am America/New_York
Answers: 2
Been thanked: 1 time

Re: Bulk downloading encounters connection errors

by GES DISC - mgreene » Tue Aug 23, 2022 1:21 pm America/New_York

Our staff are working on this issue. Once I hear back from them I will be in contact with you.
Thanks

Post Reply