Welcome to the Earthdata Forum! Here, the scientific user community and subject matter experts from NASA Distributed Active Archive Centers (DAACs), and other contributors, discuss research needs, data, and data applications.
by r.fernandes » Mon Feb 12, 2018 10:14 am America/New_York
Hi,
I need to process a whole year of daily L2 to 8D averages. I'm using the mutilevel_processor.py and i'm able to create 8D averages although i wanted to rewrite the script for it to run through the file list doing the 8D automatically. I've been reading about creating loops in the scripts but i'm having troubles understanding how to do it. This are the parameters i'm using:
but if i run it it will use all the images on the .txt and create only one image. Which arguments do i need to add for it to create the 45/46 8D instead of only one?
by melliott » Mon Feb 12, 2018 5:58 pm America/New_York
Firstly, it's great that you're using the multilevel processor (mlp for short)!
The mlp doesn't have a built-in ability to break up the input data files like that. In order to do what you want, it will be necessary to create a different text ifile for each 8 day period. Then you'd have a script (in your language of choice, although we usually recommend Python) with a loop that runs the mlp for each different 8 day ifile. The script could create the 8 day ifiles and then run the mlp within the same loop.
That said, we already distribute 8 day chlorophyll data products. Would they work for you?
by r.fernandes » Tue Feb 13, 2018 8:31 am America/New_York
Hi Matt thanks for the help.
I need to work with 1km res and i want to be able to process the images by myself too.
My workaround was the following:
1) I created a inputfiles2003.txt (with all the paths for the L2 files) 2) created 46 par files with the different sday and eday (covering the whole year) example of Par_1 [main] ifile=/Users/Mogli/Desktop/inputfiles2003.txt
3) in the parameters is where i though to be better to add the loop so that mlp runs the same inputfiles2003.txt but everytime with a different par_*. I did that with a script while read input do multilevel_processor.py --ifile=/Users/Mogli/Desktop/inputfiles2003.txt --output_dir=/Users/Mogli/Desktop/teste --overwrite $input
done < /Users/Mogli/Desktop/inputs.txt with inputs.txt being the list of the paths to all the par_* files (par_1, par_2, etc) 4) chmod +x for the script to run
5) run it
it runs but l2bin is creating A20030012003080.L3b_CU_CHL, while the l3mapgen is giving the output with the correct dates in the name atleast. Is it normal for l2bin to name the file with the first and last file on the list or is it doing something wrong? I thought i would create a *L3b_CU_CHL for each par_*, am i thinking wrong ? (it only goes from 001 to 080 because i'm only running 10 par at a time so its faster) Also from 10 supposed outputs im getting 6 or 7 1km files only and lots of .log files. In the log of the missing 1km the error usually is -W- /Users/seadas/ocssw/build/src/l2bin/dataday.c line 294: Insufficient valid latitudes found Consider QC fail for file: /Users/Mogli/Desktop/MANUAL/2003/A2003041134500.L2_LAC_OC.x.nc ...look at the file though, as I might be lying...
I've been reading a lot trying to figure out whats wrong but i fell a little bit lost. Can you understand what is wrong here? I'm i missing any parameter for the mlp?
by gnwiii » Tue Feb 13, 2018 11:21 am America/New_York
A common workflow is to bin all the files for a given day using l2bin with 365 or 366 input files (but you only need one of these files at a time), dealing with the QC fail problems when you encounter them, and using a single par file with l2bin. You can then generate 46 lists of daily binned files and a multilevel_processor.py par file that has l3bin and l3mapgen sections.
MacOS and linux are both based on Unix. The creators of Unix were programmers who realized they were spending most of their time doing text processing, and so designed Unix to make that easy. They also designed the system so it would not bog down when working with many small files. You can generate list of all the input files for a given day (l2) or interval (daily l3b). For example, $ ls -1 V2012005*.L2_SNPP_OC.extNA.nc V2012005153600.L2_SNPP_OC.extNA.nc V2012005154200.L2_SNPP_OC.extNA.nc V2012005171800.L2_SNPP_OC.extNA.nc V2012005172400.L2_SNPP_OC.extNA.nc V2012005190000.L2_SNPP_OC.extNA.nc or, as a list:
$ ls -1 V2012005*.L2_SNPP_OC.extNA.nc > V2012005.L2_SNPP_OC.extNA.list $ cat V2012005.L2_SNPP_OC.extNA.list [...]
For level-3 files, the names become predictable so generating a list from a simple program (or spreadsheet) may be easier than selecting files from a list.
by r.fernandes » Tue Feb 13, 2018 3:13 pm America/New_York
Thank you! I used your suggestion and it looks way easier that what i was trying.
But the l3bin is giving and error that wasnt giving before.
DEBUG:root:Processing for l3bin: INFO:root:Error! Cannot find source files for l3bin.
I used one of the input_list that the mlp is using on the SeaDAS GUI and i was able to process the images, it seems the error is coming from the mlp. Any idea?
by gnwiii » Wed Feb 14, 2018 10:58 am America/New_York
I think you need to provide more details. Can you show us the command-line you used, and attach logs from running the command along with the multi-level processor script you used?
Your previously posted multilevel parameter file refers to l3mapgen, but the multilevel_processor.py in branch 7.4 uses smigen, so you may want to use the multilevel processor to get the 8-day binned files and then run l3mapgen 46 times. This can be done iin a simple bash shell loop, something like:
for f in $(cat list_of_8day_binned_files) ; do echo l3mapgen par=<parfilename> ifile=$f ofile=${f}.tif ; done
(if you run this with the "echo" left in, it will show you the command that would be run when the "echo" is removed).
