l2bin error

Please enter here to ask a question about any NASA Science related topics!
obdaac_forum_user
Posts: 86
Joined: Wed Jan 27, 2021 1:52 pm America/New_York

l2bin error

by obdaac_forum_user » Wed May 17, 2017 4:54 pm America/New_York

Greetings
I am trying to process a number of L2 oc (MODIS Aqua) files I ordered and downloaded.

I am running l2bin on individual wavelengths, starting with 207 original scenes (files).  The line of code that gets submitted via command line is...

dmwarner$ /Users/dmwarner/Documents/MODIS/scripts/l2bin_lake.sh MODIS Michigan Rrs_443 2016

I am not getting that 207 L3bin files.  And I am getting different numbers of L3 bin files depending on the wavelength I process.

I noticed that during the processing that there was the following result multiple times.  This happens for each wavelength, some more than others, but none of them provide me with the full 207 L3 binned scenes.  This is causing some problems for my processing chain and I would like to understand this behavior.

Working on A2016321182500.L2_LAC_OC.x.nc
Defined A2016321182500.L2_LAC_OC.x.nc
L2BIN 4.0.9 (Apr 27 2016 10:22:37)
INTERP parameter disabled.
Single HDF input
Averaging: standard
Resolution: 1
Max Qual Allowed: 2
prod_avg:
flagusemask: 33559043
required: 0
NetCDF4 input file
-W- /Users/seadas/ocssw/build/src/l2bin/dataday.c line 294: Insufficient valid latitudes found
Consider QC fail for file: A2016321182500.L2_LAC_OC.x.nc
...look at the file though, as I might be lying...
/Users/dmwarner/Documents/MODIS/scripts/l2bin_lake.sh: line 121: 22150 Segmentation fault: 11  l2bin infile=${INFILE} ofile=${ofile} prodtype=${prodtype} l3bprod=${3} resolve=${resolve} flaguse=${flaguse} noext=${noext}

When I open this file in SeaDAS there are clearly lots of valid latitudes (though some data are masked).  Does anyone know why this is happening and if it is the cause of the Segmentation fault:  11, whatever that is?  Is it possible that I am running out of memory or something?  This seems unlikely as the file is 3.6 mb.

I am using a Mac running OSX 10.11, 16GB ram, SeaDAS v7.4.  I've attached a png of the scene for reference and the script I am using for reference.

I have determined that processing the example file alone using the GUI is successful.  I have also determined, by using a different Mac with 24GB of ram, that this is not a memory issue (in my opinion).  Finally, it happens with flaguse ="NONE" or with a number of flags.

Thanks,
Dave

attachment 2

Tags:

OB SeaDAS - dshea
Subject Matter Expert
Subject Matter Expert
Posts: 219
Joined: Thu Mar 05, 2009 10:25 am America/New_York

l2bin error

by OB SeaDAS - dshea » Thu May 18, 2017 10:56 am America/New_York

Does l2bin fail the same way if you uncomment these lines in your script?

#export OCSSWROOT=/Applications/seadas-7.4/ocssw
#source ${OCSSWROOT}/OCSSW_bash.env
#echo ${OCSSWROOT}

This is how the GUI runs the processing programs.  I am assuming that SeaDAS 7.4 is installed in "/Applications/seadas-7.4"

obdaac_forum_user
Posts: 86
Joined: Wed Jan 27, 2021 1:52 pm America/New_York

l2bin error

by obdaac_forum_user » Thu May 18, 2017 10:58 am America/New_York

Let me try that!

obdaac_forum_user
Posts: 86
Joined: Wed Jan 27, 2021 1:52 pm America/New_York

l2bin error

by obdaac_forum_user » Thu May 18, 2017 11:44 am America/New_York

Yes, it fails the same way.  I made it through Rrs_412 without a Segmentation fault but there were still only 201 (of 207) scenes binned to L3 for this wavelength, and when I did Rrs_443, I got a single scene with a Segmentation fault and only 200 scenes binned.  Then I deleted the output for Rrs_443 and re-ran this wavelength and only got 199 binned files.

I can see getting a result that says "total_filled_bins: 0" for some scenes and NOT getting L3 output, but I just don't get why the number of scenes with this result would vary from wavelength to wavelength.  I also can't see why the results would change in repeated runs for the same wavelength. 

Perhaps there is something in the source code to help, but I have no idea how to read the source code.
Dave

gnwiii
Posts: 642
Joined: Fri Jan 29, 2021 5:51 pm America/New_York
Answers: 2

l2bin error

by gnwiii » Thu May 18, 2017 12:56 pm America/New_York

Like you, I was having network connection problems when installing the OCSSW Processing System on macOS El Capitan.  After installing, programs were failing with Segmentation fault: 11.  

In fact, if $OCSSWROOT/run/var/modis/leapsec.dat is missing, I get:
$ l2bin infile=$HOME/ocssw/benchmark/A2006167181000.L2_OC.nc ofile=A2006167181000.L3b_space.nc l3bprod="chlor_a" resolve=2 prodtype=regional
L2BIN 4.0.9 (Apr 27 2016 10:22:37)
INTERP parameter disabled.
Single HDF input
Averaging: standard
Resolution: 2
Max Qual Allowed: 2
prod_avg:
flagusemask: 107599675
required: 0
NetCDF4 input file
/Users/seadas/ocssw/benchmark/A2006167181000.L2_OC.nc   brk:    0   2030 167  65407
total number of bins: 95046854
1970001 2038018 270
krow:     0 out of   8640  (-90.00 to -84.38) Thu May 18 13:46:20 2017
[...]
krow:  5670 out of   8640  ( 28.12 to  33.75) Thu May 18 13:46:20 2017
Segmentation fault: 11


After running update_luts.py manually, the errors stopped.   I have also seen Segmentation fault errors where a user had multiple ocssw directories from failed or partial installs, e.g., after a failed install using the GUI (which defaults to /Applications/seadas-7.4/ocssw) they tried a command-line install which went to $HOME/ocssw.

obdaac_forum_user
Posts: 86
Joined: Wed Jan 27, 2021 1:52 pm America/New_York

l2bin error

by obdaac_forum_user » Thu May 18, 2017 2:17 pm America/New_York

Thanks gnwiii for yet another effort to help! 
I have

/Users/dmwarner/Applications/seadas-7.4/ocssw/run/var/modis/leapsec.dat

My script tells to use SeaDAS 7.4 and where it is and I checked for another instance of ocssw and don't see one.  I guess that is a good thing.

I ran update_luts.py aqua but still got Segmentation fault :11 AND different numbers of processing results for Rrs_412 and Rrs_443. 

I did not go back to running benchmark as that was the first thing I did and it seemed to work.  Maybe I should.
Dave

gnwiii
Posts: 642
Joined: Fri Jan 29, 2021 5:51 pm America/New_York
Answers: 2

l2bin error

by gnwiii » Fri May 19, 2017 10:15 am America/New_York

Running the benchmark is helpful when you suspect a problem with your ocssw configuration.    The benchmark failed for me today  (at l1agen) on one El Capitan box after running update_luts.py, but started working after I ran the installer again (install_ocssw.py -v -b v7.4 --aqua).  My benchmark is modified to use the OC suite, So I ran the following stripped down version of your script:
#!/bin/bash
# mwe -- minimal working example for https://oceancolor.gsfc.nasa.gov/forum/oceancolor/topic_show.pl?pid=28692
#
export OCSSWROOT=${OCSSWROOT:-/Applications/seadas-7.4/ocssw}
echo ${OCSSWROOT}
source ${OCSSWROOT}/OCSSW_bash.env

resolve=1
flaguse="LAND,CLDICE,HISOLZEN,NAVFAIL,ATMFAIL"
prodtype=“regional”
noext=1
qual_max=1

infile=$OCSSWROOT/benchmark/A2006167181000.L2_OC.nc
for l3bprod in Rrs_412 Rrs_443 Rrs_469 Rrs_488 Rrs_531 Rrs_547 Rrs_555 Rrs_645 Rrs_667 Rrs_678 Rrs_748 chlor_a sst Kd_490 ; do
  ofile=A2006167181000.L3b_OC_$l3bprod.nc
  l2bin infile=${infile} ofile=${ofile} prodtype=${prodtype} l3bprod=${l3bprod} resolve=${resolve} flaguse=${flaguse} noext=${noext}
#  echo `rm -f $INFILE`
done


The "sst" product failed since it wasn't in the OC suite, but the rest appear to have worked.

OB SeaDAS - dshea
Subject Matter Expert
Subject Matter Expert
Posts: 219
Joined: Thu Mar 05, 2009 10:25 am America/New_York

l2bin error

by OB SeaDAS - dshea » Fri May 19, 2017 10:24 am America/New_York

Find out which file with which band fails. Make sure that executing l2bin (without the script) fails.  Give me the input file and your exact command line arguments that fail and I will see what is going on.

obdaac_forum_user
Posts: 86
Joined: Wed Jan 27, 2021 1:52 pm America/New_York

l2bin error

by obdaac_forum_user » Fri May 19, 2017 10:26 am America/New_York

gnwii
Thanks! 
I was and am working on creating a reproducible working example myself with just a subset of some of the files.  I should have done this before posting.  Thanks for not bashing me on this!

Not sure if this is part of the issue but even though I installed bash 4 through homebrew and thought I had it symlinked as my default shell, that seems not to have been successful as $ bash --version still tells me GNU bash, version 3.2.57(1)-release (x86_64-apple-darwin15).  Appears that I am having permission issues related to not having admin privileges.  Seems that if this were an issue I would not be able to use the gui for the task at hand.

obdaac_forum_user
Posts: 86
Joined: Wed Jan 27, 2021 1:52 pm America/New_York

l2bin error

by obdaac_forum_user » Fri May 19, 2017 11:01 am America/New_York

dshea it seems I can't get l2bin to fail without the script on a file/band that generated a Segmentation fault :11 WITH the script.

I am not a programming wiz but I would say that means that there is something being caused by my script.

As a result I will attempt to rewrite the script as per suggestions from gnwiii.

Thanks for the effort to help!

Dave

Post Reply