since Jan 17 my downloads have become extremely slow (from what took about 5min to 1.5h). I'm accessing MODIS PDS files. The downloads are still successful. So my switch to https seems to have been successful.
Any ideas what might be causing this?
I still see these issues. Download speeds were back to normal from about 2017-01-21 20:20 UTC to 2017-01-21 23:26 UTC. Now, they are very slow again.
My external IP is: xx.xxx.xx.xx
traceroute to oceandata.sci.gsfc.nasa.gov (xx.xxx.xx.xx), 30 hops max, 60 byte packets
1 router (xx.xxx.xx.xx) 0.893 ms 1.190 ms 1.337 ms
2 lo0.bng1.drw1.on.ii.net (xx.xxx.xx.xx) 5.078 ms 5.093 ms 5.208 ms
3 aeXX.cr1.drw1.on.ii.net (xx.xxx.xx.xx) 5.084 ms 5.195 ms 5.225 ms
4 xe-10-2-5.cr1.adl6.on.ii.net (xx.xxx.xx.xx) 203.276 ms 204.646 ms 204.657 ms
5 ae4.br1.syd7.on.ii.net (xx.xxx.xx.xx) 279.215 ms 279.246 ms 279.273 ms
6 ae0.br1.syd4.on.ii.net (xx.xxx.xx.xx) 63.242 ms 55.166 ms 60.539 ms
7 te0-1-1-2.br1.lax1.on.ii.net (xx.xxx.xx.xx) 202.317 ms te-0-2-1-2.br1.lax1.on.ii.net (xx.xxx.xx.xx) 202.194 ms 202.288 ms
8 los-brdr-01.inet.qwest.net (xx.xxx.xx.xx) 204.158 ms 204.252 ms 204.144 ms
9 * * *
10 xx.xxx.xx.xx (xx.xxx.xx.xx) 1439.869 ms 1492.838 ms 1492.558 ms
11 * * *
12 * * *
13 * * *
14 xx.xxx.xx.xx (xx.xxx.xx.xx) 1480.850 ms 1428.751 ms 1430.325 ms
15 rtr-s28-hecn-test.sci.gsfc.nasa.gov (xx.xxx.xx.xx) 1420.206 ms 1429.191 ms *
16 * * *
17 * * *
18 * * *
19 * * *
20 * * *
21 * * *
22 * * *
23 * * *
24 * * *
25 * * *
26 * * *
27 * * *
28 * * *
29 * * *
30 * * *
wget -d https://oceandata.sci.gsfc.nasa.gov
DEBUG output created by Wget 1.14 on linux-gnu.
URI encoding = â€˜UTF-8â€™
--2017-01-24 18:46:34-- https://oceandata.sci.gsfc.nasa.gov/
Resolving oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)... xx.xxx.xx.xx, 2001:4d0:2418:128::84
Caching oceandata.sci.gsfc.nasa.gov => xx.xxx.xx.xx 2001:4d0:2418:128::84
Connecting to oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)|xx.xxx.xx.xx|:443... connected.
Created socket 3.
Releasing 0x000000000148fc00 (new refcount 1).
Initiating SSL handshake.
Handshake successful; connected socket 3 to SSL handle 0x00000000014a9650
subject: /OU=Domain Control Validated/OU=PositiveSSL Multi-Domain/CN=oceancolor.sci.gsfc.nasa.gov
issuer: /C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO ECC Domain Validation Secure Server CA
X509 certificate successfully verified and matches host oceandata.sci.gsfc.nasa.gov
GET / HTTP/1.1
User-Agent: Wget/1.14 (linux-gnu)
HTTP request sent, awaiting response...
HTTP/1.1 200 OK
Date: Tue, 24 Jan 2017 09:16:39 GMT
Set-Cookie: app-obdaac=244f36c2fc34cf6920bf845ab9b12f1217123849--9ee888a228006583bcd00bd61b3aeee24b03621d; expires=Tue, 24 Jan 2017 10:16:39 GMT; path=/; secure; HttpOnly
Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
Stored cookie oceandata.sci.gsfc.nasa.gov -1 (ANY) / <permanent> <secure> [expiry 2017-01-24 19:46:39] app-obdaac 244f36c2fc34cf6920bf845ab9b12f1217123849--9ee888a228006583bcd00bd61b3aeee24b03621d
Registered socket 3 for persistent reuse.
URI content encoding = â€˜UTF-8â€™
Length: 11911 (12K) [text/html]
Saving to: â€˜index.htmlâ€™
100%[================================================>] 11,911 --.-K/s in 0s
2017-01-24 18:46:40 (148 MB/s) - â€˜index.htmlâ€™ saved [11911/11911]
In case it is not a network issue but rather a data server issue. https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/MOD00.A2017023.0325_1.PDS.bz2 is one of the files I'm trying to access.
Thanks for your help
There is nothing on our side that would limit your connection speed. I did see that your IP is doing about 3 times as many file_search requests as file downloads.
If you're using file_search in a script, is it because you have a subscription that you're querying? If so, you probably can query less frequently...
What is your script doing? Perhaps knowing that may help us figure out if there's anything we're missing as to the cause of your slowness.
all is back to normal speed again. Let's hope it stays this way!
My script uses file_search to determine whether certain (usually current) MODIS PDS granules are available, yet. If they are it uses getfile to download them.
I've tried a manual download using getfile and it was very slow as well. I've also checked my connection speed using speedtest and a server in the US. And it all seemed fine.
However, as mentioned it is all back to normal at the moment.
To test I've turned off all access to oceandata.sci.gsfc.nasa.gov. An hour later I manually ran: wget https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/MOD00.A2017026.0040_1.PDS.bz2
Initially, speed was above 1MB/s but soon dropped to below 200kB/s. It felt like something was throttling the speed. Repeating the same a few minutes later revealed speeds of below 100kB/s only. Transfer from a machine on a completely different network was fine (5.4MB/s). So I think I have to take this up with my ISP.
However, before I do this I will do some more tests with speedtest. Do you know what the speedtest.net server "closest" to you is?