Page 1 of 1

curl redirect max out

Posted: Wed Mar 29, 2017 8:30 pm America/New_York
by oo_processing
So I'm having odd issues with curl. This command is run on various machines:

curl -b ~/.urs_cookies -c ~/.urs_cookies -L -n --output /tmp/G20161220016.L1B_COMS.he5.gz https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/COMS_GOCI_L1B_GA_20150501001642.he5.gz

I am always successful with CentOS 6 machines. I have had a success or two with 3 different CentOS 7 machines.
The failures look like this:
searobin 182 :/tmp/test> curl -b ~/.urs_cookies -c ~/.urs_cookies -L -n --output /tmp/G20161220016.L1B_COMS.he5.gz https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/COMS_GOCI_L1B_GA_20150501001642.he5.gz
  % Total    % Received % Xferd  Average Speed   Time    Time Time  Current
                                 Dload  Upload   Total   Spent Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  ...
  0     0    0     0    0     0      0      0 --:--:--  0:00:13 --:--:--     0
  0     0    0   193    0     0     13      0 --:--:--  0:00:14 --:--:--    13
curl: (47) Maximum (50) redirects followed

The next time I happened to catch it correctly somehow (which i cntl-C out of when the data started to d/l)
searobin 183 :/tmp/test> curl --ipv4 -b ~/.urs_cookies -c ~/.urs_cookies -L -n --output /tmp/G20161220016.L1B_COMS.he5.gz https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/COMS_GOCI_L1B_GA_20150501001642.he5.gz
  % Total    % Received % Xferd  Average Speed   Time    Time Time  Current
                                 Dload  Upload   Total   Spent Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0   193    0     0    143      0 --:--:--  0:00:01 --:--:--   143
  0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0
  2  734M    2 16.2M    0     0  4431k      0  0:02:49  0:00:03 0:02:46 11.6M^C

One one CentOS machine there is no joy ever...
seahawk 125 :/optics1/home1/oo_processing> curl --verbose -b ~/.urs_cookies -c ~/.urs_cookies -L -n --output /tmp/G20161220016.L1B_COMS.he5.gz https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/COMS_GOCI_L1B_GA_20150501001642.he5.gz
* Couldn't find host oceandata.sci.gsfc.nasa.gov in the .netrc file; using defaults
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* About to connect() to oceandata.sci.gsfc.nasa.gov port 443 (#0)
*   Trying xx.xxx.xx.xx...
  0     0    0     0    0     0      0      0 --:--:--  0:02:07 --:--:--     0* Connection timed out
*   Trying 2001:4d0:2418:128::84...
* Failed to connect to 2001:4d0:2418:128::84: Network is unreachable
* Failed connect to oceandata.sci.gsfc.nasa.gov:443; Network is unreachable
* Closing connection 0
curl: (7) Failed to connect to 2001:4d0:2418:128::84: Network is unreachable

I notice that the machine seems to be trying ipv6? Why would this one machine be resolving to an ipv6 number?

So I forced it to use ipv4:
seahawk 126 :/optics1/home1/oo_processing> curl -ipv4 --verbose -b ~/.urs_cookies -c ~/.urs_cookies -L -n --output /tmp/G20161220016.L1B_COMS.he5.gz https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/COMS_GOCI_L1B_GA_20150501001642.he5.gz
* Couldn't find host oceandata.sci.gsfc.nasa.gov in the .netrc file; using defaults
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* About to connect() to oceandata.sci.gsfc.nasa.gov port 443 (#0)
*   Trying xx.xxx.xx.xx...
  0     0    0     0    0     0      0      0 --:--:--  0:02:07 --:--:--     0* Connection timed out
* Failed connect to oceandata.sci.gsfc.nasa.gov:443; Connection timed out
* Closing connection 0
curl: (7) Failed connect to oceandata.sci.gsfc.nasa.gov:443; Connection timed out

And yet this machine can reach google just fine:

seahawk 127 :/optics1/home1/oo_processing> ping google.com
PING google.com (xx.xxx.xx.xx) 56(84) bytes of data.
64 bytes from xx.xxx.xx.xx (xx.xxx.xx.xx): icmp_seq=1 ttl=59 time=6.81 ms
64 bytes from xx.xxx.xx.xx (xx.xxx.xx.xx): icmp_seq=2 ttl=59 time=6.80 ms
64 bytes from xx.xxx.xx.xx (xx.xxx.xx.xx): icmp_seq=3 ttl=59 time=6.82 ms
^C
--- google.com ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 6.806/6.812/6.820/0.067 ms

And curl seems to be working just fine!
seahawk 128 :/optics1/home1/oo_processing> curl --verbose --output installing.html http://matplotlib.org/basemap/users/installing.html
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* About to connect() to matplotlib.org port 80 (#0)
*   Trying xx.xxx.xx.xx...
* Connected to matplotlib.org (xx.xxx.xx.xx) port 80 (#0)

> GET /basemap/users/installing.html HTTP/1.1
> User-Agent: curl/7.29.0
> Host: matplotlib.org
> Accept: */*
>


< HTTP/1.1 200 OK
< Server: GitHub.com
< Date: Thu, 30 Mar 2017 00:41:17 GMT
< Content-Type: text/html; charset=utf-8
< Content-Length: 10012
< Last-Modified: Fri, 14 Feb 2014 19:25:23 GMT
< Access-Control-Allow-Origin: *
< Expires: Thu, 30 Mar 2017 00:51:17 GMT
< Cache-Control: max-age=600
< Accept-Ranges: bytes
< X-GitHub-Request-Id: CB46:5914:44A2DB:5DDD5B:58DC542D
<
{ [data not shown]
100 10012  100 10012    0     0   118k      0 --:--:-- --:--:-- --:--:--  119k
* Connection #0 to host matplotlib.org left intact

seahawk 129 :/optics1/home1/oo_processing> ll -h installing.html
-rw-rw-r-- 1 oo_processing oo_processing 9.8K Mar 29 20:41 installing.html

Ideas? I have a feeling it has to do with the down time today with your servers? I am attaching the redirect errors from a "curl --trace" and a "curl --trace-ascii" version of the above.

Thanks,
Brock
attachment 1

attachment 2

curl redirect max out

Posted: Thu Mar 30, 2017 10:36 am America/New_York
by oo_processing
It appears that something has changed again, as this issue seems resolved?
seahawk 135 :/optics1/home1/oo_processing> curl  -b ~/.urs_cookies -c ~/.urs_cookies -L -n --output /tmp/G20161220016.L1B_COMS.he5.gz https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/COMS_GOCI_L1B_GA_20150501001642.he5.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0   193    0     0    165      0 --:--:--  0:00:01 --:--:--     0
  0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0
  2  734M    2 21.0M    0     0  3596k      0  0:03:29  0:00:05  0:03:24 5442k^C