Batch processing in seadas

[arnabp893]

Hi all, I am working on time series analysis using MODIS data. My starting point is L1A data and for starters I want to generate geo files. I couldn't be able to process multiple images in Seadas 8.3.0. GUI. So, I am using my Mac terminal.
I wrote this underlying code for one file.

$ ./l2gen ifile=/Users/gulfcarbon/Downloads/MODIS/2003/L1A/A2003003192000.L1A_LAC ofile=output_file.GEO.hdf l2prod=l1a_geo
The error came as:
-E- /Users/dshea/focs/oel_hdf4/filetype/filetype.c Line 752: Unrecognized HDF-EOS file /Users/gulfcarbon/Downloads/MODIS/2003/L1A/A2003003192000.L1A_LAC
-E- /Users/dshea/focs/oel_hdf4/libl1/l1_options.c Line 207: Could not find type for file /Users/gulfcarbon/Downloads/MODIS/2003/L1A/A2003003192000.L1A_LAC.

The L1A.LAC files are hdf format. So this error must not come. Any help is appreciated.

[OB SeaDAS - xuanyang02]

l2gen on modis needs L1B + GEO as input.

Here is the workflow:
modis_GEO: L1A -> GEO
modis_L1B: L1A + GEO -> L1B
l2gen: L1B + GEO -> L2

You can try multilevel-processor to process L1A to L2. If you put multiple files in ifiles.txt in plain text (use vi or emacs), multilevel-processor can take ifiles.txt as input, therefore do batch processing --https://seadas.gsfc.nasa.gov/help-8.3.0/processors/ProcessMultilevel_processor.html

[arnabp893]

I can do multilevel processing but it will allow single file execution. Is there any other way I can do batch processing? cause I have almost 500 images. Thank you for your response

[OB SeaDAS - xuanyang02]

Edit an l1files.txt with plain text (vi or emacs), like this --
/Users/*/Scenes/MODIS_AQUA/forum/A2023051011000.L1A_LAC
/Users/*/Scenes/MODIS_AQUA/forum/A2023131213500.L1A_LAC

Edit a par file (mlp-aqua-l2gen.par) like this --
# Sample par file for multilevel_processor.
[main]
ifile=/Users/*/Scenes/MODIS_AQUA/forum/l1files.txt
[l2gen]
l2prod=chlor_a
# final processing level

and then

Code: Select all

multilevel_processor mlp-aqua-l2gen.par

Another post regarding batch processing -- viewtopic.php?p=13670&hilit=multilevel#p13670

[arnabp893]

I have started the processing. This is my file list new2022l1a.txt
/Users/gulfcarbon/Downloads/MODIS/2022/L1A/A2022023192500.L1A_LAC
/Users/gulfcarbon/Downloads/MODIS/2022/L1A/A2022014193000.L1A_LAC
/Users/gulfcarbon/Downloads/MODIS/2022/L1A/A2022018190500.L1A_LAC
...so on

This is my .par file
[main]
ifile=/Users/gulfcarbon/Downloads/new2022l1a.txt
odir=/Users/gulfcarbon/Downloads/MODIS/2022

[geo]


[level 1b]


[l2gen]
l2prod=Kd_490 Rrs Rrs_1240 Rrs_1640 Rrs_2130 Rrs_748 Rrs_859 Rrs_869 Rrs_vvv aer_model_max aer_model_min aer_model_ratio angstrom aot_869 chlor_a humidity ipar nLw nLw_1240 nLw_1640 nLw_2130 nLw_748 nLw_859 nLw_869 nLw_vvv nflh ozone par pic poc rhos rhos_1240 rhos_1640 rhos_2130 rhos_748 rhos_859 rhos_869 rhos_vvv chl_ocx
aer_opt=-3
suite=OC

after running
bin gulfcarbon$ multilevel_processor /Users/gulfcarbon/Downloads/MODIS/mlp_aqua_l2gen_bfw.par

The outcome is
Warning: Unable to determine a type for file /Users/gulfcarbon/Downloads/MODIS/2022/L1A/A2022023192500.L1A_LAC. It will not be processed.
Warning: Unable to determine a type for file /Users/gulfcarbon/Downloads/MODIS/2022/L1A/A2022014193000.L1A_LAC. It will not be processed.
Warning: Unable to determine a type for file /Users/gulfcarbon/Downloads/MODIS/2022/L1A/A2022018190500.L1A_LAC. It will not be processed.
...
I also tried with this extension for a sample file as give in the link viewtopic.php?p=13670&hilit=multilevel#p13670
A2022014193000.L1A_LAC.x.hdf

the error is
(base) Euricos-iMac:L1A gulfcarbon$ multilevel_processor /Users/gulfcarbon/Downloads/MODIS/mlp_aqua_l2gen_bfw.par
Warning: Unable to determine a type for file /Users/gulfcarbon/Downloads/MODIS/2022/L1A/x.hdf/A2022014193000.L1A_LAC.x.hdf. It will not be processed.
No valid data files were specified for processing.
Is there something I am missing?

[OB SeaDAS - xuanyang02]

I can run multilevel_Processor with the par file similar to yours and new2022l1a.txt as following.

/Users/*/L1A/A2022023192500.L1A_LAC
/Users/*/L1A/A2022014193000.L1A_LAC

Most likely, your new2022l1a.txt is not plain text. See a similar post that addresses the issue --
viewtopic.php?p=11391&hilit=emacs#p11385

Could you please do

Code: Select all

obpg_file_type /Users/gulfcarbon/Downloads/MODIS/2022/L1A/A2022023192500.L1A_LAC

and post the result here?

BTW, you can add l2bin and l3mapgen to the processor chain in your par file if desired --

[main]
ifile=/Users/gulfcarbon/Downloads/new2022l1a.txt
odir=/Users/gulfcarbon/Downloads/MODIS/2022

[geo]


[level 1b]


[l2gen]
l2prod=Kd_490 Rrs Rrs_1240 Rrs_1640 Rrs_2130 Rrs_748 Rrs_859 Rrs_869 Rrs_vvv aer_model_max aer_model_min aer_model_ratio angstrom aot_869 chlor_a humidity ipar nLw nLw_1240 nLw_1640 nLw_2130 nLw_748 nLw_859 nLw_869 nLw_vvv nflh ozone par pic poc rhos rhos_1240 rhos_1640 rhos_2130 rhos_748 rhos_859 rhos_869 rhos_vvv chl_ocx
aer_opt=-3
suite=OC

[l2bin]

[l3mapgen]

[arnabp893]

Hello xuanyang,

(base) Euricos-iMac:bin gulfcarbon$ /Users/gulfcarbon/ocssw/bin/obpg_file_type /Users/gulfcarbon/Downloads/MODIS/2022/L1A/A2022023192500.L1A_LAC
A2022023192500.L1A_LAC: unknown: unknown
I don't know why only obpg_file_type was not working even in the bin folder. I have attached the vi result of the text files of 2003 images

[OB SeaDAS - xuanyang02]

Can you do

Code: Select all

seadas_info

[OB SeaDAS - xuanyang02]

It's very odd that obpg_file_type returns unknow:unknown.

Mine shows
$ obpg_file_type A2022023192500.L1A_LAC
A2022023192500.L1A_LAC: MODIS Aqua: Level 1A

Can you download the file A2022023192500.L1A_LAC again and try obpg_file_type on it? Also try to download and obpg_file_type the file A2010283180500.L1A_LAC? Could the unzip process corrupt the file?

Did you say you can do multilevel_processor on a single file? Can you do obpg_file_type on that file?

[gnwiii]

I have encountered several types of corruption with downloaded files. These days, many "enterprise" sites have malware filters. Some filters refuse to deal with NASA OBDAAC files as "unsupported file type", other flag files "at random" when they happen to have a byte sequence found in malware. Usually files flagged as malware are replaced with some ASCII or html text that may explain how to report a false detection or get a file out of quarantine. You can use the linux "file" command to check the file type. As a result of these problems I generally request checksums when doing batch downloads.

Sometimes it can be expedient to download files blocked by "enterprise" filters onto a portable drive at home.