NASA Harmony WorkItem Failed on MUR SST

[ascherer97]

I'm trying to download the entire time series of NASA MUR v4.2 data (ccid C2036880657-POCLOUD) in a small spatial region (-126E to -124E, 44N to 46N). I don't have access to an AWS instance in the US West 2 region. I've tried several other ways and they all seem to be running into very confusing issues.

For example, I've been trying to use the harmony subsetter, both through harmony-py but also directly from the provided URL on the PO.DAAC page. Here's an example API request that I tried:

https://harmony.earthdata.nasa.gov/C2036880657-POCLOUD/ogc-api-coverages/1.0.0/collections/all/coverage/rangeset?subset=lat(44:46)&subset=lon(-126:-124)&format=application%2Fx-netcdf4

When I run this, inevitably, many items will start to fail with:

"WorkItem failed: nasa/harmony-opendap-subsetter:1.2.4: Subsetter failed with error: 502 error retrieving: https://opendap.earthdata.nasa.gov/collections/C2036880657-POCLOUD/granules/20030217090000-JPL-L4_GHRSST-SSTfnd-MUR25-GLOB-v02.0-fv04.2.dmr.xml"

For what it's worth, I've also ran into similar server errors trying to subset directly with OPeNDAP as well. I'm leaving out those details for brevity here, but I'm wondering if they're related. I basically can't find a method that works for accessing this dataset (I'd like to avoid GUI approaches, since this is ultimately intended for a reusable script for collaborators).

If someone can either point out what I'm doing wrong or let me know if something is going on with the server, that would be amazing. I've been fighting with this one for a few days now.

[PODAAC - scosic]

Hi ascherer97,

Your request was correct, and so is the way you're calling Harmony. The failures come from the OPeNDAP server that Harmony's subsetter reads each granule through and OPeNDAP most likely starts returning 502s under large load requests. Not having an AWS instance in us-west-2 isn't a factor: Harmony does the processing in AWS and serves the results over HTTPS.

Root cause:

• The granule in your error (2003-02-17) loads fine from OPeNDAP on its own, and a direct OPeNDAP subset of your box for that day also worked.

• I ran your exact subset (same box and format) as smaller jobs. About 300 granules went through with no failures: all of 2002, and half of 2003.

• Your original request covers the whole series in one job, about 8,800 granules. Harmony turns that into thousands of OPeNDAP requests at once, and OPeNDAP most likely starts returning 502s under that load. This matches what you saw: things start fine, then failures pile up.

Quick fix: Use one job per year instead of one job for the whole series

• Pick a year by granule name. Add "&granuleName=2003*" to your URL, or pass granule_name=['2003*'] in harmony-py. Granule names start with the date, so this selects exactly that year with no overlap into the next. A date range would overlap, because each file covers 21:00 to 21:00 UTC.

• Add "&skipPreview=true " (skip_preview=True in harmony-py). Otherwise Harmony pauses large jobs after a preview batch. harmony-py's wait then returns early, and download_all() fetches only the preview files.

• Run the years one after another, not all at once, so you don't recreate the same load. In my tests a full year of 2002 (122 granules) took about 10 minutes, and the whole series should take roughly 12–14 hours.

• Resubmit any granules that still fail, by name. The job's error list names them (client.status(job_id)['errors'] in harmony-py). Waiting a few minutes before resubmitting gives OPeNDAP time to recover.

Each day comes back as an 8×8 grid of 0.25° cells, about 66 KB, with all five variables. If you still see 502s with yearly jobs, send us the job IDs and we'll pass them to the OPeNDAP and Harmony teams.


PO.DAAC Team