l2gen in batch environment

[alaroy]

Good Morning :).

I'm trying to process L1A data to L2 data in an environment where a brand new virtual machine is started for each run (and unfortunately is destroyed after the run completes, making debugging that much more ...interesting...). I'm seeing two issues that I hope you guys might have some insight into.

The first issue is that getanc.py seems to be running more-or-less normally when it is called before l2gen, but when l2gen is called it uses climatology. The getanc output looks like so:

-I- calling cd /tmp/ssscratch/ocssw_l1a_to_l2-working/;getanc.py -s 2019111193000
icefile=/marine-services/ocssw/var/anc/2019/111/N201911100_SEAICE_NSIDC_24h.hdf
met1=/marine-services/ocssw/var/anc/2019/111/N201911118_MET_NCEP_6h.hdf
met2=/marine-services/ocssw/var/anc/2019/112/N201911200_MET_NCEP_6h.hdf
met3=/marine-services/ocssw/var/anc/2019/112/N201911200_MET_NCEP_6h.hdf
ozone1=/marine-services/ocssw/var/anc/2019/111/N201911100_O3_AURAOMI_24h.hdf
ozone2=/marine-services/ocssw/var/anc/2019/111/N201911100_O3_AURAOMI_24h.hdf
ozone3=/marine-services/ocssw/var/anc/2019/111/N201911100_O3_AURAOMI_24h.hdf
sstfile=/marine-services/ocssw/var/anc/2019/111/N2019111_SST_OIV2AV_24h.nc

*** WARNING: The following ancillary data types were missing or are not optimal:  OZONE
*** Beware that certain MET and OZONE files just chosen by this program are not optimal.
*** For near real-time processing the remaining files may become available soon.

And the relevant part of the l2gen output looks like:

Opening meteorological files.
  met1   = /marine-services/ocssw/share/common/met_climatology_v2014.hdf
  met2   =
  met3   =
  ozone1 = /marine-services/ocssw/share/common/ozone_climatology_v2014.hdf
  ozone2 =
  ozone3 =
  no2    = /marine-services/ocssw/share/common/no2_climatology_v2013.hdf


The second issue is that (under some conditions I don't seem to be able to repeat) l2gen will be happily humming along then suddenly seg fault with the message:

Processing scan # 750 (751 of 4060) after 761 seconds
Processing scan # 800 (801 of 4060) after 800 seconds
-E- /home/seadas/ocssw/oel_hdf4/libdfutils/wrapper.c 491: NetCDF: HDF error for longitude

Any hints you have as to what is going on to cause these problems would be much appreciated!

Thank You, Andrew L.

[gfireman]

> The first issue is that getanc.py seems to be running more-or-less normally when it is called before l2gen, but when l2gen is called it uses climatology.

Running "getanc.py -s 2019111193000" will create a file called "2019111193000.anc".

The contents of this file must be passed to l2gen:
l2gen par=2019111193000.anc

[alaroy]

That helped a lot, thank you!

I think (but can't be sure) that the other errors are the file system getting full.

[gnwiii]

I don't know how tightly controlled your platform is, but in general, POSIX systems such as linux allow you to "wrap" an binary program in a shell script that can provide information to help understand misbehaviours.   Batch systems often provide a minimal "/bin/sh" shell so some constructs used with bash may not be available.  It may also help if you can find a "test" system running linux (even Window 10 with WSL should work) so you can run problematic jobs in less restrictive environment.

[OB SeaDAS - dshea]

make sure your "ofile" is being written into a directory that has enough space.  Sometimes /tmp is pretty small.

don