Page 1 of 1

Multilevel_processor issues

Posted: Wed Feb 02, 2022 5:44 pm America/New_York
by wspeiser
I am a PhD student, new to using SeaDas, and am having some issues with the Multilevel_processor tool. I am trying to get Rrs for ~16 years of daily data from MODIS L1A products. Whenever I manually do an individual image through the modis_GEO -> modis_L1B -> l2gen process it works out great. But when I use the same input settings I use for the above chain in the multilevel_processor so I can batch process files, the outputs are either filled with nan or are projected to a completely different location than the swath. Any recommendations or tutorials for the Multilevel_processor tool would be great! Thank you so much for your time.

Re: Multilevel_processor issues

Posted: Thu Feb 03, 2022 10:05 am America/New_York
by OB SeaDAS - xuanyang02
Please let us know a few things so we can replicate the error:

1. what version of SeaDAS and what tag of OCSSW you are using.
2. the name of an L1A file you processed
3. the input settings for the processors if not default
4. command line or GUI

Re: Multilevel_processor issues

Posted: Thu Feb 03, 2022 3:26 pm America/New_York
by wspeiser
Thanks for the reply! Makes sense. Here they are:

1. SeaDas:8.1.0, OCSSW : V2021.2
2. All are named as such (L1A from the OC website): A2004006210000.L1A_LAC.x.hdf
4. GUI
3. Settings:
[main]
ifile=/home/will/Seadas_Tests/_inputFiles.lst
overwrite=1
ocproc_getanc=1

[l2gen]
aer_opt=-9
l2prod=Rrs
maskland=0
proc_land=0
proc_sst=0
resolution=500
keepfiles=1

[geo]
--refreshDB

Re: Multilevel_processor issues

Posted: Thu Feb 03, 2022 5:54 pm America/New_York
by OB SeaDAS - xuanyang02
I did modis_GEO, modis_L1B and l2gen step by step on A2004006210000.L1A_LAC and created A2004006210000step.L2_LAC_OC

And I tried multilevel_processor GUI with your setting(with A2004006210000.L1A_LAC being the context of _inputFiles.lst), and created A2004006210000mlp.L2_LAC_OC. This is Rrs_469 in A2004006210000mlp.L2_LAC_OC look like, but I can't tell the difference between this image and the one in A2004006210000step.L2_LAC_OC
Screen Shot 2022-02-03 at 5.45.06 PM.png
Screen Shot 2022-02-03 at 5.45.06 PM.png (269.93 KiB) Not viewed yet
This is the help page for multilevel_processor -- https://seadas.gsfc.nasa.gov/help-8.1.0/processors/ProcessMultilevel_processor.html

Please also do SeaDAS-OCSSW -> SeaDAS/System Info and paste the result here

Re: Multilevel_processor issues

Posted: Thu Feb 03, 2022 7:07 pm America/New_York
by wspeiser
Sorry, perhaps that was a bad example... Attached is an image of output L2 files projected to incorrect locations and correct locations (like the file you listed). When I use the reprojection tool they stay in the same locality. Here is the requested system information:

Main Application Platform:
Application Version: SeaDAS 8.1.0
Installation Directory: /home/will/Desktop/Seadas
Data Directory: /home/will/.seadas8
Configuration: /home/will/Desktop/Seadas/etc/snap.properties
WARNING!! File '/home/will/Desktop/Seadas/etc/snap.properties' does not exist
Configuration2: /home/will/.seadas8/etc/snap.properties
VM Configuration: /home/will/Desktop/Seadas/bin/pconvert.vmoptions
WARNING!! File '/home/will/Desktop/Seadas/bin/pconvert.vmoptions' does not exist
WARNING!! File '/home/will/Desktop/Seadas/bin/gpt.vmoptions' does not exist
Desktop Specification Version: 8.0.3
Engine Specification Version: 8.0.3
JRE: Java(TM) SE Runtime Environment 1.8.0_201-b09
JVM: Java HotSpot(TM) 64-Bit Server VM by Oracle Corporation
Memory: 3641 MiB

OCSSWROOT (Java Env): null

Python3 Directory:
/home/will/anaconda3/bin/python3

SeaDAS Toolbox:
SeaDAS Toolbox Specification Version: 1.1.0
Configuration: /home/will/.seadas8/etc/seadas.properties
OCSSW Root Directory: /home/will/Desktop/Seadas/ocssw
OCSSW Log Directory: /home/will/Desktop/Seadas/bin
OCSSW Location: local
Environment {$OCSSWROOT} (external): null

-----------------------------------------------

NASA Science Processing (OCSSW):
Environment {$OCSSWROOT}: /home/will/Desktop/Seadas/ocssw
Environment {$OCDATAROOT}: /home/will/Desktop/Seadas/ocssw/share
l2gen version: msl12 9.5.1-V2021.2 (Jun 18 2021 11:22:53)
l2bin version: 7.0.1
l3bin version: 5.13
l3mapgen version: l3mapgen 2.3.0-V2021.2 (Jun 18 2021 11:24:42)
Installed Missions: ['aqua', 'terra', 'ocm1', 'ocm2']

General System and Software:
Operating system: Pop!_OS 21.10
Java version: 11.0.13
Python3 version: 3.8.3
Python3 Path: /home/will/anaconda3/bin/python3
Env: ASAP_PARMS=/home/will/Desktop/Seadas/ocssw/share/common/nav
CC=gcc
_CE_CONDA=
_CE_M=
COLORTERM=truecolor
CONDA_DEFAULT_ENV=base
CONDA_EXE=/home/will/anaconda3/bin/conda
CONDA_PREFIX=/home/will/anaconda3
CONDA_PROMPT_MODIFIER=(base)
CONDA_PYTHON_EXE=/home/will/anaconda3/bin/python
CONDA_SHLVL=1
CXX=g++
DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus
DEFAULTS_PATH=/usr/share/gconf/pop.default.path
DESKTOP_SESSION=pop
DISPLAY=:1
ELEMENTS=/home/will/Desktop/Seadas/ocssw/var/seawifs
EOS_LIB_PREFIX=linux64
FC=gfortran
GCC_TUNE=
GDMSESSION=pop
G_ENABLE_DIAGNOSTIC=0
GNOME_DESKTOP_SESSION_ID=this-is-deprecated
GNOME_SHELL_SESSION_MODE=pop
GNOME_TERMINAL_SCREEN=/org/gnome/Terminal/screen/3d803fc9_7536_491d_b3b1_e3f87eea2234
GNOME_TERMINAL_SERVICE=:1.146
GPG_AGENT_INFO=/run/user/1000/gnupg/S.gpg-agent:0:1
GTK_IM_MODULE=ibus
GTK_MODULES=appmenu-gtk-module
HDFEOS_LIB=/home/will/Desktop/Seadas/ocssw/opt/EOS/lib/linux64
HOME=/home/will
_=/home/will/Desktop/Seadas/ocssw/bin/seadas_info
HRPT_STATION_IDENTIFICATION_FILE=/home/will/Desktop/Seadas/ocssw/share/seawifs/station/station_id.dat
INVOCATION_ID=6904b63d814d495e8440e2b79bb43088
J2D_PIXMAPS=shared
JOURNAL_STREAM=8:109666
L2GEN_ANC=/home/will/Desktop/Seadas/ocssw/var/anc
LANG=en_US.UTF-8
LANGUAGE=en_US:en
LC_ADDRESS=en_US.UTF-8
LC_IDENTIFICATION=en_US.UTF-8
LC_MEASUREMENT=en_US.UTF-8
LC_MONETARY=en_US.UTF-8
LC_NAME=en_US.UTF-8
LC_NUMERIC=en_US.UTF-8
LC_PAPER=en_US.UTF-8
LC_TELEPHONE=en_US.UTF-8
LC_TIME=en_US.UTF-8
LD_LIBRARY_PATH=/home/will/Desktop/Seadas/jre1.8.0_201/jre/lib/amd64:/home/will/Desktop/Seadas/jre1.8.0_201/jre/lib/i386:
LESSCLOSE=/usr/bin/lesspipe %s %s
LESSOPEN=| /usr/bin/lesspipe %s
LIB3_BIN=/home/will/Desktop/Seadas/ocssw/opt/bin
LIB3_CHECK=0
LIB3_DIR=/home/will/Desktop/Seadas/ocssw/opt
LIB3_INC=/home/will/Desktop/Seadas/ocssw/opt/include
LIB3_LIB=/home/will/Desktop/Seadas/ocssw/opt/lib
LOGNAME=will
LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.webp=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:
MANAGERPID=3798
MANDATORY_PATH=/usr/share/gconf/pop.mandatory.path
NAVCTL=/home/will/Desktop/Seadas/ocssw/share/seawifs/nav
NAVQC=/home/will/Desktop/Seadas/ocssw/share/seawifs/nav
OCDATAROOT=/home/will/Desktop/Seadas/ocssw/share
OCSSW_ARCH=linux_64
OCSSW_BIN=/home/will/Desktop/Seadas/ocssw/bin
OCSSW_DEBUG=0
OCSSWROOT=/home/will/Desktop/Seadas/ocssw
OCSSW_SVN=localhost
OCTS_REGISTRATION_TABLES=/home/will/Desktop/Seadas/ocssw/share/octs/nav
OCVARROOT=/home/will/Desktop/Seadas/ocssw/var
ORBCTL=/home/will/Desktop/Seadas/ocssw/share/seawifs/nav
PAPERSIZE=letter
PATH=/home/will/Desktop/Seadas/ocssw/opt/bin:/home/will/Desktop/Seadas/ocssw/bin:/home/will/bin:/home/will/anaconda3/bin:/home/will/anaconda3/condabin:/home/will/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
PGSINC=/home/will/Desktop/Seadas/ocssw/opt/EOS/include
PGSLIB=/home/will/Desktop/Seadas/ocssw/opt/EOS/lib/linux64
PROJ_LIB=/home/will/Desktop/Seadas/ocssw/opt/share/proj
PWD=/home/will/Desktop/Seadas/bin
QT_ACCESSIBILITY=1
QT_IM_MODULE=ibus
SESSION_MANAGER=local/pop-os:@/tmp/.ICE-unix/4437,unix/pop-os:/tmp/.ICE-unix/4437
SHELL=/bin/bash
SHLVL=2
SSH_AGENT_LAUNCHER=gnome-keyring
SSH_AUTH_SOCK=/run/user/1000/keyring/ssh
SWFTBL=/home/will/Desktop/Seadas/ocssw/share/seawifs/qc
SWTBL=/home/will/Desktop/Seadas/ocssw/share/seawifs
SYSTEMD_EXEC_PID=8169
TERM=xterm-256color
USERNAME=will
USER=will
VTE_VERSION=6402
WINDOWPATH=2
XAUTHORITY=/run/user/1000/gdm/Xauthority
XDG_CONFIG_DIRS=/etc/xdg/xdg-pop:/etc/xdg
XDG_CURRENT_DESKTOP=pop:GNOME
XDG_DATA_DIRS=/usr/share/pop:/home/will/.local/share/flatpak/exports/share:/var/lib/flatpak/exports/share:/usr/local/share/:/usr/share/
XDG_MENU_PREFIX=gnome-
XDG_RUNTIME_DIR=/run/user/1000
XDG_SESSION_CLASS=user
XDG_SESSION_DESKTOP=pop
XDG_SESSION_TYPE=x11
XMODIFIERS=@im=ibus

Re: Multilevel_processor issues

Posted: Fri Feb 04, 2022 8:21 am America/New_York
by gnwiii
I see that
OB SeaDAS - xuanyang02 wrote: Thu Feb 03, 2022 5:54 pm America/New_York by OB SeaDAS - xuanyang02 » Thu Feb 03, 2022 6:54 pm America/Anguilla

And I tried multilevel_processor GUI with your setting(with A2004006210000.L1A_LAC being the context of _inputFiles.lst), and created A2004006210000mlp.L2_LAC_OC. This is Rrs_469 in A2004006210000mlp.L2_LAC_OC look like, but I can't tell the difference between this image and the one in A2004006210000step.L2_LAC_OC
didn't reproduce your problem.

There have been a number of instances where non-reproducible problems were due to the use of an overly helpful editor to create a script or configuration file. Some editors will insert Unicode versions of look-alike dashes, quote marks, or even spaces. These can be hard to recognize, so there are "find non-ASCII" tools, and even an option in the MS Code Editor.

https://www.datagenx.net/2015/12/notepad-tip-find-out-non-ascii.html
[link] https://www.baeldung.com/linux/find-non-ascii-chars
https://stackoverflow.com/questions/61792387/powershell-find-non-ascii-characters-in-text-file#61798294

Linux and macOS sometimes have problems with scripts edited using a Windows editor that puts `<CR><LF>` at line ends. Use `cat -v <filename>` and check for `^M` at line ends.

Re: Multilevel_processor issues

Posted: Thu Feb 10, 2022 1:52 pm America/New_York
by OB SeaDAS - xuanyang02
I tried multilevel_processor GUI with your setting, with the content of _inputFiles.lst as following -- corresponding to the files 119, 120, 121 and 122 on your attached image:
/Users/***/Scenes/MODIS_AQUA/forum/A2004291221000.L1A_LAC
/Users/***/Scenes/MODIS_AQUA/forum/A2004292211500.L1A_LAC
/Users/***/Scenes/MODIS_AQUA/forum/A2004003203000.L1A_LAC
/Users/***/Scenes/MODIS_AQUA/forum/A2004001204500.L1A_LAC

Here is the world map image of the 4 L2 files produced through multilevel_processor
Screen Shot 2022-02-10 at 1.52.22 PM.png
Screen Shot 2022-02-10 at 1.52.22 PM.png (29.18 KiB) Not viewed yet

Re: Multilevel_processor issues

Posted: Thu Feb 10, 2022 2:03 pm America/New_York
by OB SeaDAS - xuanyang02
I also attached the image of the 4 corresponding L2 files downloaded from https://oceandata.sci.gsfc.nasa.gov

A2004291221000.L2_LAC_OC.nc
A2004292211500.L2_LAC_OC.nc
A2004003203000.L2_LAC_OC.nc
A2004001204500.L2_LAC_OC.nc

The geographic area looks consistent with the L2 files generated by multilevel_processor
Screen Shot 2022-02-10 at 1.57.08 PM.png
Screen Shot 2022-02-10 at 1.57.08 PM.png (30.97 KiB) Not viewed yet

Re: Multilevel_processor issues

Posted: Thu Feb 10, 2022 2:16 pm America/New_York
by OB SeaDAS - xuanyang02
Can you show me the image of the 4 L2 files you created through the modis_GEO -> modis_L1B -> l2gen step by step process instead of through multilevel_processor?

A2004291221000.L2_LAC_OC.x.hdf
A2004292211500.L2_LAC_OC.x.hdf
A2004003203000.L2_LAC_OC.x.hdf
A2004001204500.L2_LAC_OC.x.hdf

Here is the help page of multilevel_processor -- https://seadas.gsfc.nasa.gov/help-8.1.0/processors/ProcessMultilevel_processor.html