Batch processing in seadas

[arnabp893]

Hello xuanyang,

Find the statement of both the file below.
I redownloaded the 2023 file and the file you wanted me to download

(base) Euricos-iMac:bin gulfcarbon$ /Users/gulfcarbon/ocssw/bin/obpg_file_type /Users/gulfcarbon/Downloads/A2022023192500.L1A_LAC
A2022023192500.L1A_LAC: unknown: unknown
(base) Euricos-iMac:bin gulfcarbon$ /Users/gulfcarbon/ocssw/bin/obpg_file_type /Users/gulfcarbon/Downloads/A2010283180500.L1A_LAC
A2010283180500.L1A_LAC: unknown: unknown

I took a random file and did multilevel_processing and it executed although could not select the OC suite even after writing it in par file. It is in the attachement. but I opened with obpg_file_type, the result is
(base) Euricos-iMac:bin gulfcarbon$ /Users/gulfcarbon/ocssw/bin/obpg_file_type /Users/gulfcarbon/Downloads/MODIS/2022/L1A/A2022241193000.L1A_LAC
A2022241193000.L1A_LAC: unknown: unknown
I tried another system of my lab but the result came same.

Hello gnwii,

I did as you said. I got
(base) Euricos-iMac:L1A gulfcarbon$ file A2022241193000.L1A_LAC
A2022241193000.L1A_LAC: Hierarchical Data Format (version 4) data

[arnabp893]

Find the attachment here

[arnabp893]

Sorry for missing seadas_info result
Here it is

NASA Science Processing (OCSSW):
Environment {$OCSSWROOT}: /Users/gulfcarbon/ocssw
Environment {$OCDATAROOT}: Not Installed
l2gen version: -E- /Users/gulfcarbon/ocssw/bin/l2gen: Error parsing input parameters.
l2bin version: 7.0.3
l3bin version: 5.13
l3mapgen version: -E- clo_getOptionString: option=ifile needs to be set
Installed Missions: []

General System and Software:
Operating system: MacOS 10.16
Java version: 16.0.1
Python3 version: 3.8.8
Python3 Path: /opt/anaconda3/bin/python3
Env: CONDA_DEFAULT_ENV=base
CONDA_EXE=/opt/anaconda3/bin/conda
CONDA_PREFIX=/opt/anaconda3
CONDA_PROMPT_MODIFIER=(base)
CONDA_PYTHON_EXE=/opt/anaconda3/bin/python
CONDA_SHLVL=1
HOME=/Users/gulfcarbon
LANG=en_US.UTF-8
LOGNAME=gulfcarbon
LaunchInstanceID=6A70E9C9-97E9-4A69-95B8-6618CAD2B6B6
OCSSWROOT=/Users/gulfcarbon/ocssw
PATH=/Users/gulfcarbon/bin:/opt/anaconda3/bin:/opt/anaconda3/condabin:/Library/Frameworks/Python.framework/Versions/3.9/bin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin
PWD=/Users/gulfcarbon/ocssw/bin
SECURITYSESSIONID=186a3
SHELL=/bin/bash
SHLVL=2
SSH_AUTH_SOCK=/private/tmp/com.apple.launchd.pYNiVAWkmU/Listeners
TERM=xterm-256color
TERM_PROGRAM=Apple_Terminal
TERM_PROGRAM_VERSION=447
TERM_SESSION_ID=B652D4ED-4834-43E2-8465-EF5FE90DEC19
TMPDIR=/var/folders/gl/fqyzcg2n5dj7f_8cbw11_y7r0000gp/T/
USER=gulfcarbon
XPC_FLAGS=0x0
XPC_SERVICE_NAME=0
_=/usr/bin/env
_CE_CONDA=
_CE_M=
__CFBundleIdentifier=com.apple.Terminal

[arnabp893]

I have saved the OCDATAROOT and the installed mission is now ['aqua']. Yet, I am getting same error. I also tried the same method of multilevel processing in another system of my lab. The result came same. Is it something with the installation process?

[OB SeaDAS - xuanyang02]

seadas_info is supposed to give us the info of your OCSSW tag like this --
OCSSWROOT=/Users/*/ocssw
OCDATAROOT=/Users/*/ocssw/share
l2gen version: msl12 9.6.0-V2023.1 (Apr 4 2023 11:28:21)
l2bin version: 7.0.4
l3bin version: 5.13
l3mapgen version: l3mapgen 2.3.0-V2023.1 (Apr 4 2023 11:27:05)
Installed Missions: ['aqua', 'terra', 'olil8', 'olil9', 'viirsn', 'viirsj1', 'viirsj2', 'viirsdem']

The error
"Environment {$OCDATAROOT}: Not Installed
l2gen version: -E- /Users/gulfcarbon/ocssw/bin/l2gen: Error parsing input parameters."
makes me think there might be issues with your installation.

I am also puzzled how you can finish the multilevel_processor process on a single L1A file and
1. you didn't get complaint like this --"Warning: Unable to determine a type for file /Users/gulfcarbon/Downloads/MODIS/2022/L1A/A2022023192500.L1A_LAC. It will not be processed."
2. you are able to do geo, level 1b and l2gen without the "aqua" mission installed. Can you, in SeaDAS, open the L2 file created by multilevel_processor, and see if you get a reasonable chlor_a image?

Can you do

Code: Select all

l2gen -h 

Code: Select all

which l2gen

?

Did you install Science Processors (OCSSW) by GUI or command line? If by GUI, were you able to choose a tag? If by command line, what tag did you give to the --tag option? Could it be possible that you didn't finish the install process?

If you decide to reinstall OCSSW, you can delete /Users/gulfcarbon/ocssw/ and install OCSSW again either by GUI or command line. -- https://seadas.gsfc.nasa.gov/downloads/

If by command line, make sure you command has -- tag 2023.2, and --seadas --modisa like this --

Code: Select all

$  ./install_ocssw --install_dir $HOME/ocssw --tag V2023.2 --seadas --modisa --seawifs

Also add these lines to your .bashrc file --
export OCSSWROOT=[SeaDAS_install_dir]/ocssw
source $OCSSWROOT/OCSSW_bash.env

If by GUI, make sure you choose tag 2023.2, and at least check aqua for missions.

[arnabp893]

Seems like this line actually worked without uninstalling seadas. I have got the processing started.
source $OCSSWROOT/OCSSW_bash.env.
Thanks to you (and one of my friends). But can you kindly explain why this line was needed.
Also, after processing of few scenes I am getting

Running level 1b on file /Users/gulfcarbon/Downloads/MODIS/2005/L1A/A2005017191500.L1A_LAC.
Running level 1b on file /Users/gulfcarbon/Downloads/MODIS/2005/L1A/A2005021185000.L1A_LAC.
Running level 1b on file /Users/gulfcarbon/Downloads/MODIS/2005/L1A/A2005042191000.L1A_LAC.
Running level 1b on file /Users/gulfcarbon/Downloads/MODIS/2005/L1A/A2005047192500.L1A_LAC.
Running level 1b on file /Users/gulfcarbon/Downloads/MODIS/2005/L1A/A2005049191500.L1A_LAC.
Running level 1b on file /Users/gulfcarbon/Downloads/MODIS/2005/L1A/A2005070193000.L1A_LAC.
Running level 1b on file /Users/gulfcarbon/Downloads/MODIS/2005/L1A/A2005093194000.L1A_LAC.
Running level 1b on file /Users/gulfcarbon/Downloads/MODIS/2005/L1A/A2005111192500.L1A_LAC.
Running level 1b on file /Users/gulfcarbon/Downloads/MODIS/2005/L1A/A2005117185000.L1A_LAC.
Running level 1b on file /Users/gulfcarbon/Downloads/MODIS/2005/L1A/A2005127192500.L1A_LAC.
Unrecoverable error encountered in processing.

and I deleted the scene A2005127192500.L1A_LAC but every 4th or 5th image is show this error. I have enough space. I am also sharing the warning log report.


****************************************
BEGIN_PGE: Fri Jun 30 13:55:46 2023
MSG_TAG: 11
FILE: /Users/gulfcarbon/LogReport.AQUA_MODIS.20050507T192500.L1B.hdf
LOGGING: status message logging enabled
TRACE_LEVEL: tracing disabled
PID_LOGGING: disabled
DISABLED_LEVELS: none
DISABLED_SEEDS: none
DISABLED_CODES: none
THREAD-SAFE MODE: disabled
TOOLKIT_VERSION: SCF TK5.2.20
****************************************

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!! W A R N I N G !!
!! The Toolkit version found in the PCF does not !!
!! match the current Toolkit version. The PCF in use !!
!! should be replaced with a PCF constructed from the !!
!! template PCF delivered with THIS version of the !!
!! Toolkit (see TOOLKIT_VERSION in banner, above). !!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Fri Jun 30 13:55:46 2023

Check_For_Moon_in_SV_KOB():MODIS_F_OUT_OF_RANGE:295817221
Number of scans in GEO file does not match the current L1A file.
File LUN: 600000
File type: MOD03 input granule
The PCF may have the wrong names.
==========================================================

Fri Jun 30 13:55:46 2023

Out of range running...MOD_PR02. TIME:Fri Jun 30 13:55:46 2023
Operator Actions:
Check previous messages to identify LUN of the file and other possible causes.
Contact MCST.
Also contact SDST if the LUN corresponds to a MOD01 or MOD03 granule.

==========================================================

Thanks a tonne for your help.

[OB SeaDAS - xuanyang02]

multilevel_processor has a little trouble to match an L1A file with the right geo file. The solution is to skip [geo] in parfile (mlp knows to create a geo file if it does not find one). So your parfile text area should look like this:

[main]
ifile=/Users/gulfcarbon/Downloads/new2022l1a.txt
odir=/Users/gulfcarbon/Downloads/MODIS/2022

[level 1b]

[l2gen]
l2prod=Kd_490 Rrs Rrs_1240 Rrs_1640 Rrs_2130 Rrs_748 Rrs_859 Rrs_869 Rrs_vvv aer_model_max aer_model_min aer_model_ratio angstrom aot_869 chlor_a humidity ipar nLw nLw_1240 nLw_1640 nLw_2130 nLw_748 nLw_859 nLw_869 nLw_vvv nflh ozone par pic poc rhos rhos_1240 rhos_1640 rhos_2130 rhos_748 rhos_859 rhos_869 rhos_vvv chl_ocx
aer_opt=-3
suite=OC

[l2bin]

[l3mapgen]

Here is a similar post -- viewtopic.php?t=3685&hilit=multilevel_processor#p13295

[gnwiii]

Seems like this line actually worked without uninstalling seadas. I have got the processing started.

Code: Select all

source $OCSSWROOT/OCSSW_bash.env

.
Thanks to you (and one of my friends). But can you kindly explain why this line was needed.

[Note: added code </> formatting]

Before retiring I worked in a group that did many ocean colour workshops. We found it very helpful to spend the first 2 afternoon of a 2-week workshop introducing the POSIX command-line as used in linux and macOS. Students reported they found it helpful with other aspects of their "research" (some of which probably involved recreational activities). There are many good online resources such as https://linuxcommand.org and distro specific manuals. Ubuntu includes some Debian manuals that have sections on command-line basics as well as system administration.

The

Code: Select all

OCSSW_bash.env

script sets shell (bash or zsh) variables that are needed by the various programs to find data directories, names of compilers, type of system, path elements used for Python scripts and the OCSSW programs, etc. Many linux programs do something similar, but the OCSSW package is high on the spectrum of complexity because there are many programs that need to work together and because it supports both macOS and linux. This complexity combined with the many different macOS and linux versions means there can be glitches which are easily fixed with basic knowledge of the POSIX command-line.

[arnabp893]

Thanks a bunch Xuanyang,
processing is going smooth so far. Probably gonna redo the the files which are causing trouble.
Thanks gnwii,
I will look into the link that you shared for linux learning. All this terminal functions are very new to me and I will try to gain knowledge as mush as possible.