cannot connect to OBPG server to retrieve ancil file list -- connection refused!

[slwood]

Hi,

I am trying to reprocess lots (many years worth) of MODIS/aqua data on our cluster but keep getting an error from modis_GEO.py when it tries to get the atteph list from your server (findweb() in anc_utils.py, called from modis_GEO_utils.py).  The error message I get is

Error! could not establish a network connection. Check your network connection.
If you do not find a problem, please try again later.

suggesting the problem is the initial 'GET' in ProcUtils.httpdl().

When I point a browser (Google Chrome on a Mac) at https://oceandata.sci.gsfc.nasa.gov I get connection refused (ERR_CONNECTION_REFUSED).

Is there something going on with your server?

The problem does seem intermittent -- last week was mostly OK with only the occasional failure such that I could cope by retrying the failed jobs manually, but since yesterday it is pretty much all the time (nearly all processing over last night failed -- few thousand jobs running between about 10:00 to 19:00 Z, first 800 were OK then all rest failed (from about 12:30 Z onwards)).

I am running 20 jobs in parallel from a slurm task array (where a job is modis_GEO then MODIS_L1B on a single L1A file).  The requests would be appearing from IP address xx.xxx.xx.xx.  Do you have some maximum connection restriction or similar?  Recently introduced?  It has always worked fine in the past (although perhaps not 20 at a time ;-)).

Please advise how I can get the server to not refuse my connections.

regards
Simon

[OB.DAAC - SeanBailey]

Simon,

Your IP is not on our block list (although it may have hit our temporary block, which would have cleared before I had the chance to investigate).
It is also (and perhaps more) likely that you were unlucky enough to get routed to one of our backend servers that was cranky following our power outage on Saturday.
The cranky machines are all happy again, and we're seeing traffic from your IP,  although looking for files in the future...you probably don't want to not do that, as that is
one way to get blocked.

Sean

[slwood]

Thanks Sean,

> although looking for files in the future...you probably don't want to not do that, as that is
> one way to get blocked.

Can you clarify this last bit for me -- the double negative is a bit confusing -- what should I not do if I don't want to get blocked?

If the answer is that making (lots of) concurrent requests from multiple parallel sessions will get me on the (temporary?) block list, can you suggest an alternative strategy that will enable me to achieve the same throughput / time to solution?  Surely since you coded the scripts to always make this request for every file this sort of behaviour is expected?

More details:
I have already pre-downloaded all the att/eph files (we run a daily cron to mirror your archive) -- I just allow the modis_GEO.py script to do it's normal thing and call finddb() / findweb() to determine the best files to use.  I point the script at a central persistent ancil DB but it turns out that since I run with --disable-download nothing is ever written to the ancil DB in findweb(), so finddb() always fails next time and a new web query is required.  I'm tempted to modify findweb() at line 490 as follows, so that at least the DB gets populated for next time:

scripts $ git diff modules/anc_utils.py
diff --git a/modules/anc_utils.py b/modules/anc_utils.py
index e9c627a..b545957 100644
--- a/modules/anc_utils.py
+++ b/modules/anc_utils.py
@@ -487,7 +487,8 @@ Using current working directory for storing the ancillary database file: %s''' %
             if self.files[anctype] == 'missing':
                 missing.append(anctype)
                 continue
-            if self.file and self.dl:
+#           if self.file and self.dl:           ### NIWA: write to database even if not downloading
+            if self.file:
                 year = self.files[anctype][1:5]
                 day = self.files[anctype][5:8]
                 path = self.dirs['anc']

But obviously this won't help with getting blocked due to multiple concurrent requests... Possibly I could script up something that would go through our archive and for each file just make the webdb() request in order to populate the DB without actually doing any processing.  I could then run that from a single thread, which would result in your server seeing thousands of sequential requests in short succession but no concurrent ones -- would this be OK or is it still likely to confuse this as an attack?

S

[OB.DAAC - SeanBailey]

Simon,

Sorry for the double negative fingers typing on their own....:grin:

Multiple concurrent requests will not necessarily get you blocked, but it is possible. 
Your IP was (and still is) making requests for files we do not have because they
have not been produced yet.  This results in a 404 error.  Too many of those will get
you blocked. Although the number of 404s from your IP is not excessive right now,
asking for files we don't have is not a good idea.

The scripts were written with a "typical researcher" in mind - locally processing a small
amount of data serially.  The scripts are not intended for an "operational" use.  That said,
the code is open, and you can modify them as you see fit to suite your needs.

A while back (OK, a long while) I toyed with a version of the ancDB that used mysql instead
of sqlite, with the intent to provide users who wanted to run multiple threads with an option
that avoided the inevitable write locking you get with sqlite.  It might still work :grin:
Checkout $OCSSWROOT/scripts/modules/ancDBmsql.py - modify as necessary.  I offer no
warranty :razz:

Sean

[slwood]

Sean,

Thanks for the clarification.  I guess I'd better look at our download script again -- I just ask for everything that should exist up to 'now' and expect 404s for the most recent files that don't actually exist yet, then repeat hourly... at the time it seemed like a -good- easy way to get the latest files with minimum delay (well easier than parsing the html for the page to see what's already there).  But not good if risks getting us blocked... (I only do this for met/oz files since these are needed for our near-real-time level 2 product generation.)

WRT the 'not intended for "operational" use' bit I think I've now convinced myself what, in hindsight, should have been blindingly obvious: that we really need to implement the 'which ancil files should I use' logic ourselves in our own processing scripts and not call out to a convenience service on your server every time... to reprocess 50,000 files... we've already downloaded all the actual files after all, we should be able to figure out which ones to use ;-)

S

[OB.DAAC - SeanBailey]

Simon,

The getanc.py script does not hit our servers every time, only the first.  Once the ancDB is populated, it uses that to look up the files.  That's the reason it creates the local sqlite DB in the first place :grin:  The "which ancillary files to use" logic is, in fact, quite convoluted...which is why we provide the service.  Unless you're finding you're being blocked (BTW, you were NOT blocked last week, it was a hardware issue on our end that was precipitated by our power outage), I would suggest you keep the status quo - although you may want to work up an approach that doesn't guess filenames to download, perhaps use our file search utility:

wget --post-data="psdate=2018-08-22&addurl=1&results_as_file=1&search=N*MET_NCEP_*h.hdf" -O - https://oceandata.sci.gsfc.nasa.gov/api/file_search
wget -q --post-data="psdate=2018-08-22&addurl=1&results_as_file=1&search=N*O3_AURAOMI_24h.hdf" -O - https://oceandata.sci.gsfc.nasa.gov/api/file_search
wget -q --post-data="psdate=2018-08-22&addurl=1&results_as_file=1&search=N*OZONE" -O - https://oceandata.sci.gsfc.nasa.gov/api/file_search
wget -q --post-data="psdate=2018-08-22&addurl=1&results_as_file=1&search=N*SST*" -O - https://oceandata.sci.gsfc.nasa.gov/api/file_search
wget -q --post-data="psdate=2018-08-22&addurl=1&results_as_file=1&search=N*SEAICE*" -O - https://oceandata.sci.gsfc.nasa.gov/api/file_search


The "psdate" is the processed date start time to search based on the date that the file was flowed through the system.  It's a way to get the new stuff :wink:
There's a corresponding pedate for the stop time to search.  These are not the dates of the data contained in the files - those would be sdate and edate (without the 'p').
Right now, sdate and edate don't work for ancillary files, an issue I just discovered.  This will be fixed soon, but for your purposes the "psdate" should do nicely.

Sean
Edit:  s/edate issue fixed.