slow downloads

[stefan.maier]

Hi there,

since Jan 17 my downloads have become extremely slow (from what took about 5min to 1.5h). I'm accessing MODIS PDS files. The downloads are still successful. So my switch to https seems to have been successful.

Any ideas what might be causing this?

Thanks
Stefan

[OB.DAAC - SeanBailey]

Stefan,

There was a change to the firewall on Jan20th that may improve your connection.  If you are still seeing unacceptable slowness, please see the recent "access issue" post and follow the suggestions contained.

Sean

[stefan.maier]

Hi Sean,

I still see these issues. Download speeds were back to normal from about 2017-01-21 20:20 UTC to 2017-01-21 23:26 UTC. Now, they are very slow again.

My external IP is: xx.xxx.xx.xx

traceroute to oceandata.sci.gsfc.nasa.gov (xx.xxx.xx.xx), 30 hops max, 60 byte packets
1  router (xx.xxx.xx.xx)  0.893 ms  1.190 ms  1.337 ms
2  lo0.bng1.drw1.on.ii.net (xx.xxx.xx.xx)  5.078 ms  5.093 ms  5.208 ms
3  aeXX.cr1.drw1.on.ii.net (xx.xxx.xx.xx)  5.084 ms  5.195 ms  5.225 ms
4  xe-10-2-5.cr1.adl6.on.ii.net (xx.xxx.xx.xx)  203.276 ms  204.646 ms  204.657 ms
5  ae4.br1.syd7.on.ii.net (xx.xxx.xx.xx)  279.215 ms  279.246 ms  279.273 ms
6  ae0.br1.syd4.on.ii.net (xx.xxx.xx.xx)  63.242 ms  55.166 ms  60.539 ms
7  te0-1-1-2.br1.lax1.on.ii.net (xx.xxx.xx.xx)  202.317 ms te-0-2-1-2.br1.lax1.on.ii.net (xx.xxx.xx.xx)  202.194 ms  202.288 ms
8  los-brdr-01.inet.qwest.net (xx.xxx.xx.xx)  204.158 ms  204.252 ms  204.144 ms
9  * * *
10  xx.xxx.xx.xx (xx.xxx.xx.xx)  1439.869 ms  1492.838 ms  1492.558 ms
11  * * *
12  * * *
13  * * *
14  xx.xxx.xx.xx (xx.xxx.xx.xx)  1480.850 ms  1428.751 ms  1430.325 ms
15  rtr-s28-hecn-test.sci.gsfc.nasa.gov (xx.xxx.xx.xx)  1420.206 ms  1429.191 ms *
16  * * *
17  * * *
18  * * *
19  * * *
20  * * *
21  * * *
22  * * *
23  * * *
24  * * *
25  * * *
26  * * *
27  * * *
28  * * *
29  * * *
30  * * *

wget -d https://oceandata.sci.gsfc.nasa.gov
DEBUG output created by Wget 1.14 on linux-gnu.

URI encoding = ‘UTF-8’
--2017-01-24 18:46:34--  https://oceandata.sci.gsfc.nasa.gov/
Resolving oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)... xx.xxx.xx.xx, 2001:4d0:2418:128::84
Caching oceandata.sci.gsfc.nasa.gov => xx.xxx.xx.xx 2001:4d0:2418:128::84
Connecting to oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)|xx.xxx.xx.xx|:443... connected.
Created socket 3.
Releasing 0x000000000148fc00 (new refcount 1).
Initiating SSL handshake.
Handshake successful; connected socket 3 to SSL handle 0x00000000014a9650
certificate:
  subject: /OU=Domain Control Validated/OU=PositiveSSL Multi-Domain/CN=oceancolor.sci.gsfc.nasa.gov
  issuer:  /C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO ECC Domain Validation Secure Server CA
X509 certificate successfully verified and matches host oceandata.sci.gsfc.nasa.gov

---request begin---
GET / HTTP/1.1
User-Agent: Wget/1.14 (linux-gnu)
Accept: */*
Host: oceandata.sci.gsfc.nasa.gov
Connection: Keep-Alive

---request end---
HTTP request sent, awaiting response...
---response begin---
HTTP/1.1 200 OK
Server: nginx
Date: Tue, 24 Jan 2017 09:16:39 GMT
Content-Type: text/html;charset=UTF-8
Content-Length: 11911
Connection: keep-alive
Set-Cookie: app-obdaac=244f36c2fc34cf6920bf845ab9b12f1217123849--9ee888a228006583bcd00bd61b3aeee24b03621d; expires=Tue, 24 Jan 2017 10:16:39 GMT; path=/; secure; HttpOnly
Strict-Transport-Security: max-age=31536000; includeSubDomains; preload

---response end---
200 OK

Stored cookie oceandata.sci.gsfc.nasa.gov -1 (ANY) / <permanent> <secure> [expiry 2017-01-24 19:46:39] app-obdaac 244f36c2fc34cf6920bf845ab9b12f1217123849--9ee888a228006583bcd00bd61b3aeee24b03621d
Registered socket 3 for persistent reuse.
URI content encoding = ‘UTF-8’
Length: 11911 (12K) [text/html]
Saving to: ‘index.html’

100%[================================================>] 11,911      --.-K/s   in 0s     

2017-01-24 18:46:40 (148 MB/s) - ‘index.html’ saved [11911/11911]

In case it is not a network issue but rather a data server issue. https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/MOD00.A2017023.0325_1.PDS.bz2 is one of the files I'm trying to access.

Thanks for your help
Stefan

[OB.DAAC - SeanBailey]

Stefan,

There is nothing on our side that would limit your connection speed.  I did see that your IP is doing about 3 times as many file_search requests as file downloads.
If you're using file_search in a script, is it because you have a subscription that you're querying?  If so, you probably can query less frequently...

What is your script doing?  Perhaps knowing that may help us figure out if there's anything we're missing as to the cause of your slowness.

Sean

[stefan.maier]

Sean,

all is back to normal speed again. Let's hope it stays this way!

My script uses file_search to determine whether certain (usually current) MODIS PDS granules are available, yet. If they are it uses getfile to download them.

I've tried a manual download using getfile and it was very slow as well. I've also checked my connection speed using speedtest and a server in the US. And it all seemed fine.

However, as mentioned it is all back to normal at the moment.

Thanks
Stefan

[OB.DAAC - SeanBailey]

Glad things are back to normal :grin:

As for querying for data, you could set up a subscription for the data. 
The file_search utility can then return the files that match you subscription(s). 

Sean

[stefan.maier]

I'm back to slow speeds.

To test I've turned off all access to oceandata.sci.gsfc.nasa.gov. An hour later I manually ran: wget https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/MOD00.A2017026.0040_1.PDS.bz2

Initially, speed was above 1MB/s but soon dropped to below 200kB/s. It felt like something was throttling the speed. Repeating the same a few minutes later revealed speeds of below 100kB/s only. Transfer from a machine on a completely different network was fine (5.4MB/s). So I think I have to take this up with my ISP.

However, before I do this I will do some more tests with speedtest. Do you know what the speedtest.net server "closest" to you is?

Thanks
Stefan

[OB.DAAC - SeanBailey]

Does sound like something between your machine and our servers.  We don't throttle connection bandwidth.    Goddard is near Washington, D.C.

Sean