Issue with Naming Conventions in Multilevel_Processor

Use this Forum to find information on, or ask a question about, NASA Earth Science data.
wspeiser
Posts: 11
Joined: Wed Feb 02, 2022 5:38 pm America/New_York
Answers: 0

Re: Issue with Naming Conventions in Multilevel_Processor

by wspeiser » Tue Dec 06, 2022 3:57 pm America/New_York

Hi! So this does work on small batches of images (has worked on 5-10 files in the ifile text), but when I try to do many (>20) the processor inevitably runs into an error and stops working/exits processing. Is there something I can do to remedy the issue? Either change an option so its more well suited to multiple files or just to skip the errored files? Again trying to solve the error by looking at other forum posts but having issues locating a similar one... Below are the processor options and below that is the error message..


[main]
ifile=/home/williamcoast/Desktop/Projects/NearshrTemp/Data/Raw/ifilelist.txt
odir=/home/williamcoast/Desktop/Projects/NearshrTemp/Data/Raw
overwrite=1
deletefiles=1
use_ancillary=1

[level 1b]

[l2gen]
atmocor=1
eval=16
maskglint=1
maskland=0
resolution=250
suite=SST



execution exception: java.io.IOException: multilevel_processor failed with exit code 1.
Check log for more details.
Running geo on file /home/williamcoast/Desktop/Projects/NearshrTemp/Data/Raw/A2002185205500.L1A_LAC.x.hdf.
Running level 1b on file /home/williamcoast/Desktop/Projects/NearshrTemp/Data/Raw/A2002185205500.L1A_LAC.x.hdf.
Running geo on file /home/williamcoast/Desktop/Projects/NearshrTemp/Data/Raw/A2002206211500.L1A_LAC.x.hdf.
Running level 1b on file /home/williamcoast/Desktop/Projects/NearshrTemp/Data/Raw/A2002206211500.L1A_LAC.x.hdf.
Running geo on file /home/williamcoast/Desktop/Projects/NearshrTemp/Data/Raw/A2002219204500.L1A_LAC.x.hdf.
Running level 1b on file /home/williamcoast/Desktop/Projects/NearshrTemp/Data/Raw/A2002219204500.L1A_LAC.x.hdf.
Running geo on file /home/williamcoast/Desktop/Projects/NearshrTemp/Data/Raw/A2002233205500.L1A_LAC.x.hdf.
Running level 1b on file /home/williamcoast/Desktop/Projects/NearshrTemp/Data/Raw/A2002233205500.L1A_LAC.x.hdf.
Running geo on file /home/williamcoast/Desktop/Projects/NearshrTemp/Data/Raw/A2002241214500.L1A_LAC.x.hdf.
Running level 1b on file /home/williamcoast/Desktop/Projects/NearshrTemp/Data/Raw/A2002241214500.L1A_LAC.x.hdf.
Running geo on file /home/williamcoast/Desktop/Projects/NearshrTemp/Data/Raw/A2002255215500.L1A_LAC.x.hdf.
Running level 1b on file /home/williamcoast/Desktop/Projects/NearshrTemp/Data/Raw/A2002255215500.L1A_LAC.x.hdf.
Running geo on file /home/williamcoast/Desktop/Projects/NearshrTemp/Data/Raw/A2011222212000.L1A_LAC.x.hdf.
Running level 1b on file /home/williamcoast/Desktop/Projects/NearshrTemp/Data/Raw/A2011222212000.L1A_LAC.x.hdf.
Running geo on file /home/williamcoast/Desktop/Projects/NearshrTemp/Data/Raw/A2011223220500.L1A_LAC.x.hdf.
Running level 1b on file /home/williamcoast/Desktop/Projects/NearshrTemp/Data/Raw/A2011223220500.L1A_LAC.x.hdf.
Running geo on file /home/williamcoast/Desktop/Projects/NearshrTemp/Data/Raw/A2011224211000.L1A_LAC.x.hdf.
Running level 1b on file /home/williamcoast/Desktop/Projects/NearshrTemp/Data/Raw/A2011224211000.L1A_LAC.x.hdf.
Running geo on file /home/williamcoast/Desktop/Projects/NearshrTemp/Data/Raw/A2011227214000.L1A_LAC.x.hdf.
Running level 1b on file /home/williamcoast/Desktop/Projects/NearshrTemp/Data/Raw/A2011227214000.L1A_LAC.x.hdf.
Running geo on file /home/williamcoast/Desktop/Projects/NearshrTemp/Data/Raw/A2011228204500.L1A_LAC.x.hdf.
Running level 1b on file /home/williamcoast/Desktop/Projects/NearshrTemp/Data/Raw/A2011228204500.L1A_LAC.x.hdf.
Running geo on file /home/williamcoast/Desktop/Projects/NearshrTemp/Data/Raw/A2011230221000.L1A_LAC.x.hdf.
Running level 1b on file /home/williamcoast/Desktop/Projects/NearshrTemp/Data/Raw/A2011230221000.L1A_LAC.x.hdf.
Running geo on file /home/williamcoast/Desktop/Projects/NearshrTemp/Data/Raw/A2011231211500.L1A_LAC.x.hdf.
Running level 1b on file /home/williamcoast/Desktop/Projects/NearshrTemp/Data/Raw/A2011231211500.L1A_LAC.x.hdf.
Running geo on file /home/williamcoast/Desktop/Projects/NearshrTemp/Data/Raw/A2011232220000.L1A_LAC.x.hdf.
Running level 1b on file /home/williamcoast/Desktop/Projects/NearshrTemp/Data/Raw/A2011232220000.L1A_LAC.x.hdf.
Running geo on file /home/williamcoast/Desktop/Projects/NearshrTemp/Data/Raw/A2011233210500.L1A_LAC.x.hdf.
Running level 1b on file /home/williamcoast/Desktop/Projects/NearshrTemp/Data/Raw/A2011233210500.L1A_LAC.x.hdf.
Running geo on file /home/williamcoast/Desktop/Projects/NearshrTemp/Data/Raw/A2011234214500.L1A_LAC.x.hdf.
Running level 1b on file /home/williamcoast/Desktop/Projects/NearshrTemp/Data/Raw/A2011234214500.L1A_LAC.x.hdf.
Running geo on file /home/williamcoast/Desktop/Projects/NearshrTemp/Data/Raw/A2011235205000.L1A_LAC.x.hdf.
Running level 1b on file /home/williamcoast/Desktop/Projects/NearshrTemp/Data/Raw/A2011235205000.L1A_LAC.x.hdf.
Running geo on file /home/williamcoast/Desktop/Projects/NearshrTemp/Data/Raw/A2011236213500.L1A_LAC.x.hdf.
Running level 1b on file /home/williamcoast/Desktop/Projects/NearshrTemp/Data/Raw/A2011236213500.L1A_LAC.x.hdf.
Running geo on file /home/williamcoast/Desktop/Projects/NearshrTemp/Data/Raw/A2011237204000.L1A_LAC.x.hdf.
Running level 1b on file /home/williamcoast/Desktop/Projects/NearshrTemp/Data/Raw/A2011237204000.L1A_LAC.x.hdf.
Running geo on file /home/williamcoast/Desktop/Projects/NearshrTemp/Data/Raw/A2011237221500.L1A_LAC.x.hdf.
Running level 1b on file /home/williamcoast/Desktop/Projects/NearshrTemp/Data/Raw/A2011237221500.L1A_LAC.x.hdf.
Running geo on file /home/williamcoast/Desktop/Projects/NearshrTemp/Data/Raw/A2011238212000.L1A_LAC.x.hdf.
Running level 1b on file /home/williamcoast/Desktop/Projects/NearshrTemp/Data/Raw/A2011238212000.L1A_LAC.x.hdf.
Running geo on file /home/williamcoast/Desktop/Projects/NearshrTemp/Data/Raw/A2011239220500.L1A_LAC.x.hdf.
Running level 1b on file /home/williamcoast/Desktop/Projects/NearshrTemp/Data/Raw/A2011239220500.L1A_LAC.x.hdf.
Running geo on file /home/williamcoast/Desktop/Projects/NearshrTemp/Data/Raw/A2011240211000.L1A_LAC.x.hdf.
Running level 1b on file /home/williamcoast/Desktop/Projects/NearshrTemp/Data/Raw/A2011240211000.L1A_LAC.x.hdf.
Running geo on file /home/williamcoast/Desktop/Projects/NearshrTemp/Data/Raw/A2011242205500.L1A_LAC.x.hdf.
Running level 1b on file /home/williamcoast/Desktop/Projects/NearshrTemp/Data/Raw/A2011242205500.L1A_LAC.x.hdf.
Running geo on file /home/williamcoast/Desktop/Projects/NearshrTemp/Data/Raw/A2011243214000.L1A_LAC.x.hdf.
Running level 1b on file /home/williamcoast/Desktop/Projects/NearshrTemp/Data/Raw/A2011243214000.L1A_LAC.x.hdf.
Running geo on file /home/williamcoast/Desktop/Projects/NearshrTemp/Data/Raw/A2011244204500.L1A_LAC.x.hdf.
Running level 1b on file /home/williamcoast/Desktop/Projects/NearshrTemp/Data/Raw/A2011244204500.L1A_LAC.x.hdf.
Running geo on file /home/williamcoast/Desktop/Projects/NearshrTemp/Data/Raw/A2011246221000.L1A_LAC.x.hdf.
Running level 1b on file /home/williamcoast/Desktop/Projects/NearshrTemp/Data/Raw/A2011246221000.L1A_LAC.x.hdf.
Running l2gen on file /home/williamcoast/Desktop/Projects/NearshrTemp/Data/Raw/AQUA_MODIS.20020704T205500.L1B.hdf.
Running l2gen on file /home/williamcoast/Desktop/Projects/NearshrTemp/Data/Raw/AQUA_MODIS.20020725T211500.L1B.hdf.
Running l2gen on file /home/williamcoast/Desktop/Projects/NearshrTemp/Data/Raw/AQUA_MODIS.20020807T204500.L1B.hdf.
Running l2gen on file /home/williamcoast/Desktop/Projects/NearshrTemp/Data/Raw/AQUA_MODIS.20020821T205500.L1B.hdf.
Running l2gen on file /home/williamcoast/Desktop/Projects/NearshrTemp/Data/Raw/AQUA_MODIS.20020829T214500.L1B.hdf.
Running l2gen on file /home/williamcoast/Desktop/Projects/NearshrTemp/Data/Raw/AQUA_MODIS.20020912T215603.L1B.hdf.
Unrecoverable error encountered in processing.

Tags:

OB SeaDAS - xuanyang02
Subject Matter Expert
Subject Matter Expert
Posts: 439
Joined: Tue Feb 09, 2021 5:42 pm America/New_York
Answers: 1
Been thanked: 1 time

Re: Issue with Naming Conventions in Multilevel_Processor

by OB SeaDAS - xuanyang02 » Fri Dec 09, 2022 1:04 pm America/New_York

What's in your ifilelist.txt?

Do you have a corresponding geo file for /home/williamcoast/Desktop/Projects/NearshrTemp/Data/Raw/AQUA_MODIS.20020912T215603.L1B.hdf in the same directory?

Please also make sure that you have enough disk space to save all the geo/L1B/L2 files.

wspeiser
Posts: 11
Joined: Wed Feb 02, 2022 5:38 pm America/New_York
Answers: 0

Re: Issue with Naming Conventions in Multilevel_Processor

by wspeiser » Mon Jan 16, 2023 2:37 pm America/New_York

I do have a corresponding GEO file but it looks like (unlike for the other L1B files) the multilevel processor did not create HKM or QKM files. Is there a reason why it would create them for other dates and not this one? Is there a way to automatically skip files that it does this to, or do I need to manually remove them from the ifilelist? I have enough file on my disk to save all of the files. Below is the content of the ifilelist.txt:

/home/williamcoast/Desktop/Projects/NearshrTemp/Data/Raw/A2002185205500.L1A_LAC.x.hdf
/home/williamcoast/Desktop/Projects/NearshrTemp/Data/Raw/A2002206211500.L1A_LAC.x.hdf
/home/williamcoast/Desktop/Projects/NearshrTemp/Data/Raw/A2002219204500.L1A_LAC.x.hdf
/home/williamcoast/Desktop/Projects/NearshrTemp/Data/Raw/A2002233205500.L1A_LAC.x.hdf
/home/williamcoast/Desktop/Projects/NearshrTemp/Data/Raw/A2002241214500.L1A_LAC.x.hdf
/home/williamcoast/Desktop/Projects/NearshrTemp/Data/Raw/A2002255215500.L1A_LAC.x.hdf
/home/williamcoast/Desktop/Projects/NearshrTemp/Data/Raw/A2011222212000.L1A_LAC.x.hdf
/home/williamcoast/Desktop/Projects/NearshrTemp/Data/Raw/A2011223220500.L1A_LAC.x.hdf
/home/williamcoast/Desktop/Projects/NearshrTemp/Data/Raw/A2011224211000.L1A_LAC.x.hdf
/home/williamcoast/Desktop/Projects/NearshrTemp/Data/Raw/A2011227214000.L1A_LAC.x.hdf
/home/williamcoast/Desktop/Projects/NearshrTemp/Data/Raw/A2011228204500.L1A_LAC.x.hdf
/home/williamcoast/Desktop/Projects/NearshrTemp/Data/Raw/A2011230221000.L1A_LAC.x.hdf
/home/williamcoast/Desktop/Projects/NearshrTemp/Data/Raw/A2011231211500.L1A_LAC.x.hdf
/home/williamcoast/Desktop/Projects/NearshrTemp/Data/Raw/A2011232220000.L1A_LAC.x.hdf
/home/williamcoast/Desktop/Projects/NearshrTemp/Data/Raw/A2011233210500.L1A_LAC.x.hdf
/home/williamcoast/Desktop/Projects/NearshrTemp/Data/Raw/A2011234214500.L1A_LAC.x.hdf
/home/williamcoast/Desktop/Projects/NearshrTemp/Data/Raw/A2011235205000.L1A_LAC.x.hdf
/home/williamcoast/Desktop/Projects/NearshrTemp/Data/Raw/A2011236213500.L1A_LAC.x.hdf
/home/williamcoast/Desktop/Projects/NearshrTemp/Data/Raw/A2011237204000.L1A_LAC.x.hdf
/home/williamcoast/Desktop/Projects/NearshrTemp/Data/Raw/A2011237221500.L1A_LAC.x.hdf
/home/williamcoast/Desktop/Projects/NearshrTemp/Data/Raw/A2011238212000.L1A_LAC.x.hdf
/home/williamcoast/Desktop/Projects/NearshrTemp/Data/Raw/A2011239220500.L1A_LAC.x.hdf
/home/williamcoast/Desktop/Projects/NearshrTemp/Data/Raw/A2011240211000.L1A_LAC.x.hdf
/home/williamcoast/Desktop/Projects/NearshrTemp/Data/Raw/A2011242205500.L1A_LAC.x.hdf
/home/williamcoast/Desktop/Projects/NearshrTemp/Data/Raw/A2011243214000.L1A_LAC.x.hdf
/home/williamcoast/Desktop/Projects/NearshrTemp/Data/Raw/A2011244204500.L1A_LAC.x.hdf
/home/williamcoast/Desktop/Projects/NearshrTemp/Data/Raw/A2011246221000.L1A_LAC.x.hdf

OB SeaDAS - xuanyang02
Subject Matter Expert
Subject Matter Expert
Posts: 439
Joined: Tue Feb 09, 2021 5:42 pm America/New_York
Answers: 1
Been thanked: 1 time

Re: Issue with Naming Conventions in Multilevel_Processor

by OB SeaDAS - xuanyang02 » Tue Jan 17, 2023 4:29 pm America/New_York

Since the start time of A2002255215500.L1A_LAC is 2002-09-12T21:56:03, the corresponding GEO/L1B/L2 files for A2002255215500.L1A_LAC are
AQUA_MODIS.20020912T215603.GEO.hdf
AQUA_MODIS.20020912T215603.L1B.hdf
AQUA_MODIS_HKM.20020912T215603.L1B.hdf
AQUA_MODIS_QKM.20020912T215603.L1B.hdf
AQUA_MODIS.20020912T215603.L2.SST.nc
I was able to finish the MLP process creating the above GEO/L1B/L2 files.

I'd like to know if the error is caused by A2002255215500.L1A_LAC.x.hdf or 20+ files.

Can you remove /home/williamcoast/Desktop/Projects/NearshrTemp/Data/Raw/A2002255215500.L1A_LAC.x.hdf from your ifilelist.txt and try again?

Can you also try 5 files in ifilelist.txt with the following files --
/home/williamcoast/Desktop/Projects/NearshrTemp/Data/Raw/A2002219204500.L1A_LAC.x.hdf
/home/williamcoast/Desktop/Projects/NearshrTemp/Data/Raw/A2002233205500.L1A_LAC.x.hdf
/home/williamcoast/Desktop/Projects/NearshrTemp/Data/Raw/A2002241214500.L1A_LAC.x.hdf
/home/williamcoast/Desktop/Projects/NearshrTemp/Data/Raw/A2002255215500.L1A_LAC.x.hdf
/home/williamcoast/Desktop/Projects/NearshrTemp/Data/Raw/A2011222212000.L1A_LAC.x.hdf

MLP (V2022.1) will exit when there is an error in any of the process in the process chain. We made some changes so MLP will continue to the next file in ifilelist.txt if any of the processes fails. The change will be in the next operational tag. For now, you need to manually remove /home/williamcoast/Desktop/Projects/NearshrTemp/Data/Raw/A2002255215500.L1A_LAC.x.hdf from your ifilelist.txt if we confirm A2002255215500.L1A_LAC.x.hdf is causing the error

wspeiser
Posts: 11
Joined: Wed Feb 02, 2022 5:38 pm America/New_York
Answers: 0

Re: Issue with Naming Conventions in Multilevel_Processor

by wspeiser » Tue Jan 17, 2023 4:33 pm America/New_York

I removed the file from the ifilelist.txt and the MLP ran successfully on the rest of the files including the ones listed in your suggestion. So since time of posting I have just manually deleted problem files, although this has only happened on two other files out of >900. Thank you for your help and work on updates!

Post Reply