Batch Script 8D

Use this Forum to find information on, or ask a question about, NASA Earth Science data.
Post Reply
r.fernandes
Posts: 27
Joined: Wed Jun 06, 2012 6:55 pm America/New_York

Batch Script 8D

by r.fernandes » Mon Feb 12, 2018 10:14 am America/New_York

Hi,

I need to process a whole year of daily L2 to 8D averages. I'm using the mutilevel_processor.py and i'm able to create 8D averages although i wanted to rewrite the script for it to run through the file list doing the 8D automatically. I've been reading about creating loops in the scripts but i'm having troubles understanding how to do it.
This are the parameters i'm using:

[main]
ifile=/User/Desktop/inputfiles2003.txt

[l2bin]
resolve=1
suite=CHL
prodtype=8-day
verbose=1

[l3bin]
loneast=0
lonwest=60
latnorth=60
latsouth=0
verbose=1

[l3mapgen]
projection=smi
resolution=1km
interp=area
north=60
south=0
east=0
west=60


I'm running:

multilevel_processor.py scripteste  --ifile=/Users/Mogli/Desktop/inputfiles2003.txt --output_dir=/Users/Mogli/Desktop/teste


but if i run it it will use all the images on the .txt and create only one image. Which arguments do i need to add for it to create the 45/46 8D instead of only one?

Thanks,
Ricardo

Tags:

melliott
Posts: 11
Joined: Tue Jun 08, 2010 2:23 pm America/New_York

Batch Script 8D

by melliott » Mon Feb 12, 2018 5:58 pm America/New_York

Firstly, it's great that you're using the multilevel processor (mlp for short)!

The mlp doesn't have a built-in ability to break up the input data files like that. In order to do what you want, it will be necessary to create a different text ifile for each 8 day period. Then you'd have a script (in your language of choice, although we usually recommend Python) with a loop that runs the mlp for each different 8 day ifile.  The script could create the 8 day ifiles and then run the mlp within the same loop.

That said, we already distribute 8 day chlorophyll data products. Would they work for you?

-Matt

r.fernandes
Posts: 27
Joined: Wed Jun 06, 2012 6:55 pm America/New_York

Batch Script 8D

by r.fernandes » Tue Feb 13, 2018 8:31 am America/New_York

Hi Matt thanks for the help.

I need to work with 1km res and i want to be able to process the images by myself too.

My workaround was the following:

1) I created a inputfiles2003.txt (with all the paths for the L2 files)
2) created 46 par files with the different sday and eday (covering the whole year)
example of Par_1
[main]
ifile=/Users/Mogli/Desktop/inputfiles2003.txt

[l2bin]
resolve=1
sday=2003001
eday=2003008
suite=CHL
prodtype=8-day
verbose=1

[l3mapgen]
projection=smi
resolution=1km
interp=area
north=60
south=0
east=0
west=-60
fudge=1.5

3) in the parameters is where i though to be better to add the loop so that mlp runs the same inputfiles2003.txt but everytime with a different par_*. I did that with a script
while read input
do
multilevel_processor.py --ifile=/Users/Mogli/Desktop/inputfiles2003.txt --output_dir=/Users/Mogli/Desktop/teste --overwrite $input 

done < /Users/Mogli/Desktop/inputs.txt

with inputs.txt being the list of the paths to all the par_* files (par_1, par_2, etc)
4) chmod +x for the script to run

5) run it

it runs but l2bin is creating  A20030012003080.L3b_CU_CHL, while the l3mapgen is giving the output with the correct dates in the name atleast.
Is it normal for l2bin to name the file with the first and last file on the list or is it doing something wrong? I thought i would create a *L3b_CU_CHL for each par_*, am i thinking wrong ? (it only goes from 001 to 080 because i'm only running 10 par at a time so its faster)
Also from 10 supposed outputs im getting 6 or 7 1km files only and lots of .log files. In the log of the missing 1km the error usually is
-W- /Users/seadas/ocssw/build/src/l2bin/dataday.c line 294: Insufficient valid latitudes found
Consider QC fail for file: /Users/Mogli/Desktop/MANUAL/2003/A2003041134500.L2_LAC_OC.x.nc
...look at the file though, as I might be lying...


I've been reading a lot trying to figure out whats wrong but i fell a little bit lost. Can you understand what is wrong here? I'm i missing any parameter for the mlp?

thank you in advance!
Ricardo

gnwiii
Posts: 665
Joined: Fri Jan 29, 2021 5:51 pm America/New_York
Answers: 2

Batch Script 8D

by gnwiii » Tue Feb 13, 2018 11:21 am America/New_York

A common workflow is to bin all the files for a given day using l2bin with 365 or 366 input files (but you only need one of these files at a time), dealing with the QC fail problems when you encounter them, and using a single par file with l2bin.  You can then generate 46 lists of daily binned files and a multilevel_processor.py par file that has l3bin and l3mapgen sections.

MacOS and linux are both based on Unix.  The creators of Unix were programmers who realized they were spending most of their time doing text processing, and so designed Unix to make that easy.  They also designed the system so it would not bog down when working with many small files.   You can generate list of all the input files for a given day (l2) or interval (daily l3b). For example,
$ ls -1 V2012005*.L2_SNPP_OC.extNA.nc
V2012005153600.L2_SNPP_OC.extNA.nc
V2012005154200.L2_SNPP_OC.extNA.nc
V2012005171800.L2_SNPP_OC.extNA.nc
V2012005172400.L2_SNPP_OC.extNA.nc
V2012005190000.L2_SNPP_OC.extNA.nc

or, as a list:

$ ls -1 V2012005*.L2_SNPP_OC.extNA.nc > V2012005.L2_SNPP_OC.extNA.list
$ cat V2012005.L2_SNPP_OC.extNA.list
[...]


For level-3 files, the names become predictable so generating a list from a simple program (or spreadsheet) may be easier than selecting files from a list.

OB SeaDAS - knowles
Subject Matter Expert
Subject Matter Expert
Posts: 245
Joined: Mon Apr 07, 2008 4:40 pm America/New_York

Batch Script 8D

by OB SeaDAS - knowles » Tue Feb 13, 2018 11:24 am America/New_York


r.fernandes
Posts: 27
Joined: Wed Jun 06, 2012 6:55 pm America/New_York

Batch Script 8D

by r.fernandes » Tue Feb 13, 2018 3:13 pm America/New_York

Thank you! I used your suggestion and it looks way easier that what i was trying.

But the l3bin is giving and error that wasnt giving before.

DEBUG:root:Processing for l3bin:
INFO:root:Error! Cannot find source files for l3bin.


I used one of the input_list that the mlp is using on the SeaDAS GUI and i was able to process the images, it seems the error is coming from the mlp. Any idea?

r.fernandes
Posts: 27
Joined: Wed Jun 06, 2012 6:55 pm America/New_York

Batch Script 8D

by r.fernandes » Wed Feb 14, 2018 9:22 am America/New_York

Thank you danny, i missed your post yesterday.

I ended up writing a scrip to run l2bin with different input_files(8-D) and l3mapgen leaving the l3bin out because of the error i mentioned before.

Thank you all for the help!

gnwiii
Posts: 665
Joined: Fri Jan 29, 2021 5:51 pm America/New_York
Answers: 2

Batch Script 8D

by gnwiii » Wed Feb 14, 2018 10:58 am America/New_York

I think you need to provide more details.  Can you show us the command-line you used, and attach logs from running the command along with the multi-level processor script you used? 

Your previously posted multilevel parameter file refers to l3mapgen, but the multilevel_processor.py in branch 7.4 uses smigen, so you may want to use the multilevel processor to get the 8-day binned files and then run l3mapgen 46 times.  This can be done iin a simple bash shell loop, something like:

for f in $(cat list_of_8day_binned_files) ; do echo l3mapgen par=<parfilename> ifile=$f ofile=${f}.tif ; done

(if you run this with the "echo" left in, it will show you the command that would be run when the "echo" is removed).

Post Reply