V2023.0 -- MODIS L1A processing failed

Use this Forum to find information on, or ask a question about, NASA Earth Science data.
Post Reply
oo_processing
Posts: 301
Joined: Wed Apr 06, 2005 12:11 pm America/New_York
Answers: 0
Has thanked: 5 times

V2023.0 -- MODIS L1A processing failed

by oo_processing » Tue Mar 28, 2023 11:02 pm America/New_York

Dear Support,

First, $OCSSWROOT/benchmark/seadas_benchmark.bash ran successfully.
But it failed with any recent L0 files:
$ modis_L1A -o out.L1A MOD00.A2023087.1520_1.PDS
MODIS L1A version 6.0.6, built Nov 28 2022 12:23:38
modis_l1a: ERROR: MODIS L1A processing failed.
Please examine the LogStatus and LogUser files for more information.
Following is the content of LogStatus.out.L1A
****************************************
BEGIN_PGE: Tue Mar 28 22:50:47 2023
MSG_TAG: 11
FILE: /work/o/oo_processing/tmp/junk3/LogStatus.out.L1A
LOGGING: status message logging enabled
TRACE_LEVEL: tracing disabled
PID_LOGGING: disabled
DISABLED_LEVELS: none
DISABLED_SEEDS: none
DISABLED_CODES: none
THREAD-SAFE MODE: disabled
TOOLKIT_VERSION: SCF TK5.2.20
****************************************

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!! W A R N I N G !!
!! The Toolkit version found in the PCF does not !!
!! match the current Toolkit version. The PCF in use !!
!! should be replaced with a PCF constructed from the !!
!! template PCF delivered with THIS version of the !!
!! Toolkit (see TOOLKIT_VERSION in banner, above). !!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

level1a():MODIS_U_L1A_BEGIN:286763081
The L1A process started: L1A version 6.0.6 built on Nov 28 2022, at 12:23:38

PGS_PC_GetPCSDataRetrieveData():PGSPC_W_NO_CONFIG_VALUE:76803
No configuration value exists for logical

PGS_PC_GetPCSData():PGSPC_W_NO_CONFIG_VALUE:76803
No configuration value exists for logical

PGS_PC_GetConfigData():PGSPC_W_NO_CONFIG_FOR_ID:76810
The Product ID does not contain a configuration value. Problematic ID is 599004.

get_pcf_config_data():MODIS_E_GETCONFIG_FAILED:286764563
A non-success message was returned while trying to retrieve data from the PCF file: The MOD_PR01 LUT RCS Revision number could not be retrieved

initialize_level1a():MODIS_F_NO_PCF_CONFIG_DATA:286765122
All configuration data could not be retrieved from the PCF file: error occured in trying to get PCF config data values Granule Length or Scan Rate

level1a():MODIS_E_INITIALIZATION_FAILED:286764566
Program failed initialization: Routine initialize_level1a failed

PGS_IO_L0_ManageTable():PGSIO_E_L0_UNUSED_TABLE_ENTRY:11825
File table lookup index points to unused table entry

PGS_IO_L0_Close():PGSIO_E_L0_VIRTUAL_DS_NOT_OPEN:11806
Virtual data set is not open

close_processing_run():MODIS_E_PGS_IO_LO_CLOSE:286764589
PGS_IO_L0_Close failed to close the L0 file successfully: Unable to close the L0 file successfully LUN: 0

level1a():MODIS_U_L1A_END:286763082
The L1A process ended: L1A return code = 1


Please advice!
Yuyuan

Tags:

OB SeaDAS - xuanyang02
Subject Matter Expert
Subject Matter Expert
Posts: 644
Joined: Tue Feb 09, 2021 5:42 pm America/New_York
Answers: 1
Been thanked: 1 time

Re: V2023.0 -- MODIS L1A processing failed

by OB SeaDAS - xuanyang02 » Wed Mar 29, 2023 5:11 pm America/New_York

Yes, V2023.0 has issues with both modis_L1A and modis_GEO. Please use V2022.3

OB.DAAC - SeanBailey
User Services
User Services
Posts: 1464
Joined: Wed Sep 18, 2019 6:15 pm America/New_York
Answers: 1
Been thanked: 5 times

Re: V2023.0 -- MODIS L1A processing failed

by OB.DAAC - SeanBailey » Fri Mar 31, 2023 11:12 am America/New_York

The issue is that the LUT updates required for the update MODIS L1A and GEO codes failed to be included in the V2023.0 release. This will be corrected soon (and you'll see V2023.1). The proper LUTs are available in the T2023.7 test tag.

Sean

daurin
Posts: 42
Joined: Thu Apr 22, 2010 8:06 am America/New_York
Answers: 0
Has thanked: 1 time

Re: V2023.0 -- MODIS L1A processing failed

by daurin » Wed Jul 26, 2023 4:54 pm America/New_York

I am running into a similar problem despite have just updated ("clean install") to OCSSW V2023.2

Incidentally, this version now throws a red warning "OCSSW Tag: WARNING!!! latest tag is 2022.3" that I don't know what to make of.

Here is the LogReport when I try to batch modis_L1B (mysteriously suffixed with ".hdf" despite being ascii; identical to LogStatus, and to the first block of LogUser):
****************************************
BEGIN_PGE: Wed Jul 26 16:06:27 2023
MSG_TAG: 11
FILE: /Users/daurin/GitRepos/batch_SeaDAS/LogReport.AQUA_MODIS.20200822T110500.L1B.hdf
LOGGING: status message logging enabled
TRACE_LEVEL: tracing disabled
PID_LOGGING: disabled
DISABLED_LEVELS: none
DISABLED_SEEDS: none
DISABLED_CODES: none
THREAD-SAFE MODE: disabled
TOOLKIT_VERSION: SCF TK5.2.20
****************************************

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!! W A R N I N G !!
!! The Toolkit version found in the PCF does not !!
!! match the current Toolkit version. The PCF in use !!
!! should be replaced with a PCF constructed from the !!
!! template PCF delivered with THIS version of the !!
!! Toolkit (see TOOLKIT_VERSION in banner, above). !!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Wed Jul 26 16:06:28 2023

Read_Overlap_OBCEng():MODIS_S_OK:295813648
WARNING: Could not retrieve leading granule file name from PCF.
Call to PGS_PC_GetReference() failed.
File LUN: 500000
File type: previous MOD01 input granule
The leading MOD01 granule will be treated as missing.
The resulting L1B output will be valid if no other errors occur.
==========================================================

Wed Jul 26 16:06:28 2023

Read_Overlap_OBCEng():MODIS_S_OK:295813648
WARNING: Could not retrieve trailing granule file name from PCF.
Call to PGS_PC_GetReference() failed.
File LUN: 500002
File type: following MOD01 input granule
The trailing MOD01 granule will be treated as missing.
The resulting L1B output will be valid if no other errors occur.
==========================================================


Here's the system report:
-----------------------------------------------------------
Main Application Platform:
-----------------------------------------------------------
Application Version: SeaDAS 8.3.10
SNAP Engine Version: 10.0.0.0* (SeaDAS Platform modified)
SNAP Desktop Version: 10.0.0.0* (SeaDAS Platform modified)
SNAP Engine Build Date: 202306140444
SNAP Desktop Build Date: 202306140448
Installation Directory: /Users/daurin/Applications/seadas_8.3.10
Data Directory: /Users/daurin/.seadas8
Configuration: /Users/daurin/Applications/seadas_8.3.10/etc/snap.properties
VM Configuration: /Users/daurin/Applications/seadas_8.3.10/etc/seadas.conf
WARNING!! File '/Users/daurin/Applications/seadas_8.3.10/etc/seadas.conf' does not exist
VM Configuration: /Users/daurin/Applications/seadas_8.3.10/etc/snap.conf
WARNING!! File '/Users/daurin/Applications/seadas_8.3.10/etc/snap.conf' does not exist
VM Configuration (gpt): /Users/daurin/Applications/seadas_8.3.10/bin/gpt.vmoptions
VM Configuration (pconvert): /Users/daurin/Applications/seadas_8.3.10/bin/pconvert.vmoptions
Runtime Configuration: /Users/daurin/.seadas8/etc/snap.properties
Runtime Configuration (SeaDAS Toolbox): /Users/daurin/.seadas8/etc/seadas.properties
JRE: OpenJDK Runtime Environment 11.0.19+7
JVM: OpenJDK 64-Bit Server VM by Eclipse Adoptium
Memory: 4096 MiB
OCSSWROOT (Java Env): /Users/daurin/Applications/ocssw
Python3 Directory: /Users/daurin/anaconda3/bin/python3


-----------------------------------------------------------
SeaDAS Toolbox:
-----------------------------------------------------------
SeaDAS Toolbox Version: 1.3.0
SeaDAS Toolbox Build Date: 202306140456
Configuration: /Users/daurin/.seadas8/etc/seadas.properties
OCSSW Root Directory: /Users/daurin/Applications/ocssw
OCSSW Log Directory: /
OCSSW Location: local
Environment {$OCSSWROOT} (external): /Users/daurin/Applications/ocssw


-----------------------------------------------------------
NASA Science Processing (OCSSW):
-----------------------------------------------------------
OCSSWROOT=/Users/daurin/Applications/ocssw
OCDATAROOT=/Users/daurin/Applications/ocssw/share
l2gen version: Reason: tried: '/usr/local/opt/gcc@9/lib/gcc/9/libgfortran.5.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/usr/local/opt/gcc@9/lib/gcc/9/libgfortran.5.dylib' (no such file), '/usr/local/opt/gcc@9/lib/gcc/9/libgfortran.5.dylib' (no such file), '/usr/local/lib/libgfortran.5.dylib' (no such file), '/usr/lib/libgfortran.5.dylib' (no such file, not in dyld cache)
l2bin version: Reason: tried: '/usr/local/opt/gcc@9/lib/gcc/9/libgfortran.5.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/usr/local/opt/gcc@9/lib/gcc/9/libgfortran.5.dylib' (no such file), '/usr/local/opt/gcc@9/lib/gcc/9/libgfortran.5.dylib' (no such file), '/usr/local/lib/libgfortran.5.dylib' (no such file), '/usr/lib/libgfortran.5.dylib' (no such file, not in dyld cache)
l3bin version: Reason: tried: '/usr/local/opt/gcc@9/lib/gcc/9/libgfortran.5.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/usr/local/opt/gcc@9/lib/gcc/9/libgfortran.5.dylib' (no such file), '/usr/local/opt/gcc@9/lib/gcc/9/libgfortran.5.dylib' (no such file), '/usr/local/lib/libgfortran.5.dylib' (no such file), '/usr/lib/libgfortran.5.dylib' (no such file, not in dyld cache)
l3mapgen version: Reason: tried: '/usr/local/opt/gcc@9/lib/gcc/9/libgfortran.5.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/usr/local/opt/gcc@9/lib/gcc/9/libgfortran.5.dylib' (no such file), '/usr/local/opt/gcc@9/lib/gcc/9/libgfortran.5.dylib' (no such file), '/usr/local/lib/libgfortran.5.dylib' (no such file), '/usr/lib/libgfortran.5.dylib' (no such file, not in dyld cache)
Installed Missions: ['hawkeye', 'aqua', 'terra', 'olcis3a', 'olcis3b', 'viirsn', 'viirsj1', 'viirsj2']


-----------------------------------------------------------
General System and Software:
-----------------------------------------------------------
Operating system: MacOS 13.4.1
Java version:
Python3 version: 3.8.16
Python3 Path: /Users/daurin/anaconda3/bin/python3
Env:
CC=gcc
CXX=g++
ELEMENTS=/Users/daurin/Applications/ocssw/var/seawifs
EOS_LIB_PREFIX=macintel64
FC=gfortran
GCC_TUNE=
HDFEOS_LIB=/Users/daurin/Applications/ocssw/opt/EOS/lib/macintel64
HRPT_STATION_IDENTIFICATION_FILE=/Users/daurin/Applications/ocssw/share/seawifs/station/station_id.dat
L2GEN_ANC=/Users/daurin/Applications/ocssw/var/anc
LIB3_BIN=/Users/daurin/Applications/ocssw/opt/bin
LIB3_CHECK=0
LIB3_DIR=/Users/daurin/Applications/ocssw/opt
LIB3_INC=/Users/daurin/Applications/ocssw/opt/include
LIB3_LIB=/Users/daurin/Applications/ocssw/opt/lib
NAVCTL=/Users/daurin/Applications/ocssw/share/seawifs/nav
NAVQC=/Users/daurin/Applications/ocssw/share/seawifs/nav
OCSSW_ARCH=macosx_intel
OCSSW_BIN=/Users/daurin/Applications/ocssw/bin
OCSSW_DEBUG=0
OCSSW_MODIS=1
OCTS_REGISTRATION_TABLES=/Users/daurin/Applications/ocssw/share/octs/nav
OCVARROOT=/Users/daurin/Applications/ocssw/var
ORBCTL=/Users/daurin/Applications/ocssw/share/seawifs/nav
PGSINC=/Users/daurin/Applications/ocssw/opt/EOS/include
PGSLIB=/Users/daurin/Applications/ocssw/opt/EOS/lib/macintel64
PROJ_DATA=/Users/daurin/Applications/ocssw/opt/share/proj
PROJ_LIB=/Users/daurin/Applications/ocssw/opt/share/proj
SWFTBL=/Users/daurin/Applications/ocssw/share/seawifs/qc
SWTBL=/Users/daurin/Applications/ocssw/share/seawifs

daurin
Posts: 42
Joined: Thu Apr 22, 2010 8:06 am America/New_York
Answers: 0
Has thanked: 1 time

Re: V2023.0 -- MODIS L1A processing failed

by daurin » Wed Jul 26, 2023 5:28 pm America/New_York

I should add, the call (often successful in the past, if perhaps unorthodox) made to generate the L1B (different image, same problem) is along the lines of:

python `which modis_L1B` -v --log /Volumes/Megalodon/MODIS/WaveCIS/L1/A2022125074000.L1A_LAC /Volumes/Megalodon/MODIS/WaveCIS/L1/A2022125074000.GEO

Results in the calling directory with:
-rw-r--r-- 1 daurin staff 137M Jul 26 17:17 AQUA_MODIS.20220505T074000.L1B.hdf
-rw-r--r-- 1 daurin staff 21K Jul 26 17:17 AQUA_MODIS.20220505T074000.L1B.hdf.pcf

...but decidedly not
/Volumes/Megalodon/MODIS/WaveCIS/L1/A2022125074000.L1B_*

...and completes with standard output:
LUT directory: /Users/daurin/Applications/ocssw/var/modisa/cal/OPER
LUT version: 6.2.1.86_OC_v2.11
Reflective LUT: MYD02_Reflective_LUTs.V6.2.1.86_OC_v2.11.hdf
Emissive LUT: MYD02_Emissive_LUTs.V6.2.1.86_OC_v2.11.hdf
QA LUT: MYD02_QA_LUTs.V6.2.1.86_OC_v2.11.hdf

Processing MODIS L1A file to L1B...
MODIS Aqua L1B version 6.2.1_obpg, built Apr 4 2023 11:27:06
scan: 0 out of 204 Wed Jul 26 17:17:50 2023
scan: 10 out of 204 Wed Jul 26 17:17:51 2023
scan: 20 out of 204 Wed Jul 26 17:17:51 2023
scan: 30 out of 204 Wed Jul 26 17:17:51 2023
scan: 40 out of 204 Wed Jul 26 17:17:51 2023
scan: 50 out of 204 Wed Jul 26 17:17:51 2023
scan: 60 out of 204 Wed Jul 26 17:17:51 2023
scan: 70 out of 204 Wed Jul 26 17:17:51 2023
scan: 80 out of 204 Wed Jul 26 17:17:51 2023
scan: 90 out of 204 Wed Jul 26 17:17:52 2023
scan: 100 out of 204 Wed Jul 26 17:17:52 2023
scan: 110 out of 204 Wed Jul 26 17:17:52 2023
scan: 120 out of 204 Wed Jul 26 17:17:52 2023
scan: 130 out of 204 Wed Jul 26 17:17:52 2023
scan: 140 out of 204 Wed Jul 26 17:17:52 2023
scan: 150 out of 204 Wed Jul 26 17:17:52 2023
scan: 160 out of 204 Wed Jul 26 17:17:52 2023
scan: 170 out of 204 Wed Jul 26 17:17:53 2023
scan: 180 out of 204 Wed Jul 26 17:17:53 2023
scan: 190 out of 204 Wed Jul 26 17:17:53 2023
scan: 200 out of 204 Wed Jul 26 17:17:53 2023
/Users/daurin/Applications/ocssw/bin/l1bgen_modisa exit status: 0
MODIS L1B processing complete.


-Dirk

PS. Now that I look more closely at the original post error messages, I see that the resemblance to my problem is actually quite superficial, so apologies for littering the wrong forum post. I can start a new one for it if preferred.
Last edited by daurin on Wed Jul 26, 2023 7:29 pm America/New_York, edited 1 time in total.

OB SeaDAS - xuanyang02
Subject Matter Expert
Subject Matter Expert
Posts: 644
Joined: Tue Feb 09, 2021 5:42 pm America/New_York
Answers: 1
Been thanked: 1 time

Re: V2023.0 -- MODIS L1A processing failed

by OB SeaDAS - xuanyang02 » Wed Jul 26, 2023 6:01 pm America/New_York

There is something not right with your installation of V2023.2. I don't know what happened to clean install. The version of l2gen/l2bin/l3bin/l3mapgen should look like this --

l2gen version: msl12 9.6.0-V2023.1 (Apr 4 2023 11:28:21)
l2bin version: 7.0.4
l3bin version: 5.13
l3mapgen version: l3mapgen 2.3.0-V2023.1 (Apr 4 2023 11:27:05)

Could you please delete /Users/daurin/Applications/ocssw and install V2023.2 again?

BTW, don't worry about the red warning "OCSSW Tag: WARNING!!! latest tag is 2022.3". When SeaDAS 8.3.0 was released, V2022.3 was the latest version of the Science Processor (OCSSW) at the time. 8.3.10 is an emergency release for Mac Ventura users and it still thinks V2022.3 is the latest version.

The name LogReport.AQUA_MODIS.20200822T110500.L1B.hdf means it's a log report on "AQUA_MODIS.20200822T110500.L1B.hdf". It didn't mean it's an hdf file.

Could you please run modis_L1B again once you reinstalled V2023.2?

Bing

daurin
Posts: 42
Joined: Thu Apr 22, 2010 8:06 am America/New_York
Answers: 0
Has thanked: 1 time

Re: V2023.0 -- MODIS L1A processing failed

by daurin » Thu Jul 27, 2023 10:27 am America/New_York

Yes, there is something very wrong indeed. I tried getting versions of those, but while modis_L1B and modis_GEO both reported versions 1.1, l2gen, l2bin, l3bin, and l3mapgen all had errors similar to this one (from l2gen):

Code: Select all

daurin@MacBook-Pro:~/GitRepos/batch_SeaDAS$ l2gen --version
dyld[19128]: Library not loaded: /usr/local/opt/gcc@9/lib/gcc/9/libgfortran.5.dylib
  Referenced from: <5C959932-E6C6-3693-8C6C-0833BD48E46A> /Users/daurin/Applications/ocssw/bin/l2gen
  Reason: tried: '/usr/local/opt/gcc@9/lib/gcc/9/libgfortran.5.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/usr/local/opt/gcc@9/lib/gcc/9/libgfortran.5.dylib' (no such file), '/usr/local/opt/gcc@9/lib/gcc/9/libgfortran.5.dylib' (no such file), '/Users/daurin/Applications/ocssw/opt/lib/libgfortran.5.dylib' (no such file), '/usr/local/lib/libgfortran.5.dylib' (no such file), '/usr/lib/libgfortran.5.dylib' (no such file, not in dyld cache)
Abort trap: 6
I deleted the contents of ocssw and ran a clean install of 2023.2 for a third time, but the result is the same. These problems with fortran libraries appear to be reflected in the OCSSW portion of system status report posted above.

daurin
Posts: 42
Joined: Thu Apr 22, 2010 8:06 am America/New_York
Answers: 0
Has thanked: 1 time

Re: V2023.0 -- MODIS L1A processing failed

by daurin » Thu Jul 27, 2023 12:26 pm America/New_York

Thanks for your help, Bing. Problems are ongoing, but unrelated to this original post, so I am going to start a new thread.

-Dirk

Post Reply