Sean,
Yes, crawl-????.googlebot.com sounds right. If it's impossible to unblock it, I'll try CMR.
Search found 85 matches
- Wed Jan 29, 2020 12:13 pm America/New_York
- Forum: Home
- Question: Issues with wget authentication
- Replies: 23
- Tue Jan 28, 2020 7:54 pm America/New_York
- Forum: Home
- Question: Issues with wget authentication
- Replies: 23
Issues with wget authentication
I'm able to repeat the request with the exact same headers without issues from my desktop, so I'm kinda stumped. Could you look in nginx's error logs to see if there are any details? The URL I tried was:https://oceandata.sci.gsfc.nasa.gov/api/file_search?search=A2020014.L3m_DAY_CHL_chlor...
- Tue Jan 28, 2020 5:48 pm America/New_York
- Forum: Home
- Question: Issues with wget authentication
- Replies: 23
Issues with wget authentication
Thanks, Sean. This looks promising, but I'm hitting a weird snag. Using wget on such URLs works fine, but fetching them using our internal system returns a 403 and this:<!DOCTYPE html><html lang="en-US"><head><meta http-equiv="content-type" content="text/html;charset=utf...
- Mon Jan 27, 2020 10:57 pm America/New_York
- Forum: Home
- Question: Issues with wget authentication
- Replies: 23
Issues with wget authentication
Sean,Thank you, but the problem is I actually can't use neither wget nor curl, as my binaries do not have access to the Internet. Instead, we have an internal system that proxies HTTP requests, and I don't think I'd be able to use plaintext auth there. I was hoping to use this system for...
- Mon Jan 27, 2020 10:52 am America/New_York
- Forum: Home
- Question: Issues with wget authentication
- Replies: 23
Issues with wget authentication
Hi,I'm having the same problems as above - wget only works --auth-no-challenge=on, but since this method sends plaintext password, it's not great. LPDAAC downloads (eg, https://e4ftl01.cr.usgs.gov/VIIRS/VNP13A1.001/2016.08.28/VNP13A1.A2016241.h24v06.001.2018162005736.h5) also require authent...