Batch processing in seadas

Use this Forum to find information on, or ask a question about, NASA Earth Science data.
arnabp893
Posts: 39
Joined: Tue Jun 21, 2022 1:35 pm America/New_York
Answers: 0

Batch processing in seadas

by arnabp893 » Wed Jun 21, 2023 11:28 am America/New_York

Hi all, I am working on time series analysis using MODIS data. My starting point is L1A data and for starters I want to generate geo files. I couldn't be able to process multiple images in Seadas 8.3.0. GUI. So, I am using my Mac terminal.
I wrote this underlying code for one file.

$ ./l2gen ifile=/Users/gulfcarbon/Downloads/MODIS/2003/L1A/A2003003192000.L1A_LAC ofile=output_file.GEO.hdf l2prod=l1a_geo
The error came as:
-E- /Users/dshea/focs/oel_hdf4/filetype/filetype.c Line 752: Unrecognized HDF-EOS file /Users/gulfcarbon/Downloads/MODIS/2003/L1A/A2003003192000.L1A_LAC
-E- /Users/dshea/focs/oel_hdf4/libl1/l1_options.c Line 207: Could not find type for file /Users/gulfcarbon/Downloads/MODIS/2003/L1A/A2003003192000.L1A_LAC.

The L1A.LAC files are hdf format. So this error must not come. Any help is appreciated.

Filters:

OB SeaDAS - xuanyang02
Subject Matter Expert
Subject Matter Expert
Posts: 659
Joined: Tue Feb 09, 2021 5:42 pm America/New_York
Answers: 1
Been thanked: 2 times

Re: Batch processing in seadas

by OB SeaDAS - xuanyang02 » Wed Jun 21, 2023 4:50 pm America/New_York

l2gen on modis needs L1B + GEO as input.

Here is the workflow:
modis_GEO: L1A -> GEO
modis_L1B: L1A + GEO -> L1B
l2gen: L1B + GEO -> L2

You can try multilevel-processor to process L1A to L2. If you put multiple files in ifiles.txt in plain text (use vi or emacs), multilevel-processor can take ifiles.txt as input, therefore do batch processing --https://seadas.gsfc.nasa.gov/help-8.3.0/processors/ProcessMultilevel_processor.html

arnabp893
Posts: 39
Joined: Tue Jun 21, 2022 1:35 pm America/New_York
Answers: 0

Re: Batch processing in seadas

by arnabp893 » Thu Jun 22, 2023 12:43 pm America/New_York

I can do multilevel processing but it will allow single file execution. Is there any other way I can do batch processing? cause I have almost 500 images. Thank you for your response

OB SeaDAS - xuanyang02
Subject Matter Expert
Subject Matter Expert
Posts: 659
Joined: Tue Feb 09, 2021 5:42 pm America/New_York
Answers: 1
Been thanked: 2 times

Re: Batch processing in seadas

by OB SeaDAS - xuanyang02 » Thu Jun 22, 2023 1:46 pm America/New_York

Edit an l1files.txt with plain text (vi or emacs), like this --
/Users/*/Scenes/MODIS_AQUA/forum/A2023051011000.L1A_LAC
/Users/*/Scenes/MODIS_AQUA/forum/A2023131213500.L1A_LAC

Edit a par file (mlp-aqua-l2gen.par) like this --
# Sample par file for multilevel_processor.
[main]
ifile=/Users/*/Scenes/MODIS_AQUA/forum/l1files.txt
[l2gen]
l2prod=chlor_a
# final processing level

and then

Code: Select all

multilevel_processor mlp-aqua-l2gen.par
Another post regarding batch processing -- viewtopic.php?p=13670&hilit=multilevel#p13670

arnabp893
Posts: 39
Joined: Tue Jun 21, 2022 1:35 pm America/New_York
Answers: 0

Re: Batch processing in seadas

by arnabp893 » Mon Jun 26, 2023 4:02 pm America/New_York

I have started the processing. This is my file list new2022l1a.txt
/Users/gulfcarbon/Downloads/MODIS/2022/L1A/A2022023192500.L1A_LAC
/Users/gulfcarbon/Downloads/MODIS/2022/L1A/A2022014193000.L1A_LAC
/Users/gulfcarbon/Downloads/MODIS/2022/L1A/A2022018190500.L1A_LAC
...so on

This is my .par file
[main]
ifile=/Users/gulfcarbon/Downloads/new2022l1a.txt
odir=/Users/gulfcarbon/Downloads/MODIS/2022

[geo]


[level 1b]


[l2gen]
l2prod=Kd_490 Rrs Rrs_1240 Rrs_1640 Rrs_2130 Rrs_748 Rrs_859 Rrs_869 Rrs_vvv aer_model_max aer_model_min aer_model_ratio angstrom aot_869 chlor_a humidity ipar nLw nLw_1240 nLw_1640 nLw_2130 nLw_748 nLw_859 nLw_869 nLw_vvv nflh ozone par pic poc rhos rhos_1240 rhos_1640 rhos_2130 rhos_748 rhos_859 rhos_869 rhos_vvv chl_ocx
aer_opt=-3
suite=OC

after running
bin gulfcarbon$ multilevel_processor /Users/gulfcarbon/Downloads/MODIS/mlp_aqua_l2gen_bfw.par

The outcome is
Warning: Unable to determine a type for file /Users/gulfcarbon/Downloads/MODIS/2022/L1A/A2022023192500.L1A_LAC. It will not be processed.
Warning: Unable to determine a type for file /Users/gulfcarbon/Downloads/MODIS/2022/L1A/A2022014193000.L1A_LAC. It will not be processed.
Warning: Unable to determine a type for file /Users/gulfcarbon/Downloads/MODIS/2022/L1A/A2022018190500.L1A_LAC. It will not be processed.
...
I also tried with this extension for a sample file as give in the link viewtopic.php?p=13670&hilit=multilevel#p13670
A2022014193000.L1A_LAC.x.hdf

the error is
(base) Euricos-iMac:L1A gulfcarbon$ multilevel_processor /Users/gulfcarbon/Downloads/MODIS/mlp_aqua_l2gen_bfw.par
Warning: Unable to determine a type for file /Users/gulfcarbon/Downloads/MODIS/2022/L1A/x.hdf/A2022014193000.L1A_LAC.x.hdf. It will not be processed.
No valid data files were specified for processing.
Is there something I am missing?

OB SeaDAS - xuanyang02
Subject Matter Expert
Subject Matter Expert
Posts: 659
Joined: Tue Feb 09, 2021 5:42 pm America/New_York
Answers: 1
Been thanked: 2 times

Re: Batch processing in seadas

by OB SeaDAS - xuanyang02 » Tue Jun 27, 2023 4:07 pm America/New_York

I can run multilevel_Processor with the par file similar to yours and new2022l1a.txt as following.

/Users/*/L1A/A2022023192500.L1A_LAC
/Users/*/L1A/A2022014193000.L1A_LAC

Most likely, your new2022l1a.txt is not plain text. See a similar post that addresses the issue --
viewtopic.php?p=11391&hilit=emacs#p11385

Could you please do

Code: Select all

obpg_file_type /Users/gulfcarbon/Downloads/MODIS/2022/L1A/A2022023192500.L1A_LAC
and post the result here?

BTW, you can add l2bin and l3mapgen to the processor chain in your par file if desired --

[main]
ifile=/Users/gulfcarbon/Downloads/new2022l1a.txt
odir=/Users/gulfcarbon/Downloads/MODIS/2022

[geo]


[level 1b]


[l2gen]
l2prod=Kd_490 Rrs Rrs_1240 Rrs_1640 Rrs_2130 Rrs_748 Rrs_859 Rrs_869 Rrs_vvv aer_model_max aer_model_min aer_model_ratio angstrom aot_869 chlor_a humidity ipar nLw nLw_1240 nLw_1640 nLw_2130 nLw_748 nLw_859 nLw_869 nLw_vvv nflh ozone par pic poc rhos rhos_1240 rhos_1640 rhos_2130 rhos_748 rhos_859 rhos_869 rhos_vvv chl_ocx
aer_opt=-3
suite=OC

[l2bin]

[l3mapgen]

arnabp893
Posts: 39
Joined: Tue Jun 21, 2022 1:35 pm America/New_York
Answers: 0

Re: Batch processing in seadas

by arnabp893 » Tue Jun 27, 2023 4:41 pm America/New_York

Hello xuanyang,

(base) Euricos-iMac:bin gulfcarbon$ /Users/gulfcarbon/ocssw/bin/obpg_file_type /Users/gulfcarbon/Downloads/MODIS/2022/L1A/A2022023192500.L1A_LAC
A2022023192500.L1A_LAC: unknown: unknown
I don't know why only obpg_file_type was not working even in the bin folder. I have attached the vi result of the text files of 2003 images
Attachments
Screenshot 2023-06-27 at 3.37.06 PM.png
Screenshot 2023-06-27 at 3.37.06 PM.png (475.83 KiB) Not viewed yet

OB SeaDAS - xuanyang02
Subject Matter Expert
Subject Matter Expert
Posts: 659
Joined: Tue Feb 09, 2021 5:42 pm America/New_York
Answers: 1
Been thanked: 2 times

Re: Batch processing in seadas

by OB SeaDAS - xuanyang02 » Tue Jun 27, 2023 8:04 pm America/New_York

Can you do

Code: Select all

seadas_info
Last edited by OB SeaDAS - xuanyang02 on Tue Jun 27, 2023 8:11 pm America/New_York, edited 1 time in total.

OB SeaDAS - xuanyang02
Subject Matter Expert
Subject Matter Expert
Posts: 659
Joined: Tue Feb 09, 2021 5:42 pm America/New_York
Answers: 1
Been thanked: 2 times

Re: Batch processing in seadas

by OB SeaDAS - xuanyang02 » Tue Jun 27, 2023 8:11 pm America/New_York

It's very odd that obpg_file_type returns unknow:unknown.

Mine shows
$ obpg_file_type A2022023192500.L1A_LAC
A2022023192500.L1A_LAC: MODIS Aqua: Level 1A

Can you download the file A2022023192500.L1A_LAC again and try obpg_file_type on it? Also try to download and obpg_file_type the file A2010283180500.L1A_LAC? Could the unzip process corrupt the file?

Did you say you can do multilevel_processor on a single file? Can you do obpg_file_type on that file?

gnwiii
Posts: 713
Joined: Fri Jan 29, 2021 5:51 pm America/New_York
Answers: 2
Has thanked: 1 time

Re: Batch processing in seadas

by gnwiii » Tue Jun 27, 2023 8:37 pm America/New_York

I have encountered several types of corruption with downloaded files. These days, many "enterprise" sites have malware filters. Some filters refuse to deal with NASA OBDAAC files as "unsupported file type", other flag files "at random" when they happen to have a byte sequence found in malware. Usually files flagged as malware are replaced with some ASCII or html text that may explain how to report a false detection or get a file out of quarantine. You can use the linux "file" command to check the file type. As a result of these problems I generally request checksums when doing batch downloads.

Sometimes it can be expedient to download files blocked by "enterprise" filters onto a portable drive at home.

Post Reply