Runing GEO and L1B processed in parallel

Use this Forum to find information on, or ask a question about, NASA Earth Science data.
Post Reply
jvaldezch
Posts: 27
Joined: Fri Aug 24, 2018 2:52 pm America/New_York
Answers: 0

Runing GEO and L1B processed in parallel

by jvaldezch » Wed Sep 09, 2020 11:31 am America/New_York

Hi forum,

I was wodering if it is possible to run processes in parallel. I'm currently processing L2 products to obtain AFAI, but when running processes like modis_GEO.py and modis_L1B.py I realize that both scripts uses current work directory and creates two files with a non unique name "ShmMem" and "GetAttr.temp" and this provoques other processes runing over same directory to crash.

Do you have any suggestion?

Thanks

Tags:

OB SeaDAS - dshea
Subject Matter Expert
Subject Matter Expert
Posts: 258
Joined: Thu Mar 05, 2009 10:25 am America/New_York
Answers: 0
Been thanked: 2 times

Runing GEO and L1B processed in parallel

by OB SeaDAS - dshea » Wed Sep 09, 2020 12:31 pm America/New_York

I guess I would take the easy way out.  Make a directory for each parallel process and move a set of files to each directory.  Process each directory with one process.  When done move all the files back to the original directory.  Moving a file just changes a pointer to the actual data in the filesystem, so it is fast regardless of the size of the file.  Assuming you are moving the file within a physical storage device.

We might revisit those scripts and make process unique temp filenames, but don't hold your breath.

don

jvaldezch
Posts: 27
Joined: Fri Aug 24, 2018 2:52 pm America/New_York
Answers: 0

Runing GEO and L1B processed in parallel

by jvaldezch » Wed Sep 09, 2020 12:39 pm America/New_York

Thanks Don! I will modify my scripts to do as suggested.

gnwiii
Posts: 713
Joined: Fri Jan 29, 2021 5:51 pm America/New_York
Answers: 2
Has thanked: 1 time

Runing GEO and L1B processed in parallel

by gnwiii » Wed Sep 09, 2020 3:03 pm America/New_York

The level-2 processing is the one that can benefit the most from parallel processing.  The GEO and L1B processing are dominated by I/O.  In the past (on a system "borrowed" from numerical modellers that had one data disk but 24 cores) it made sense run the lower level processing steps serially and use GNU parallel for the level-2 processing.  There were diminishing returns and heat problems using all 24 cores, so in the end l2gen was given fewer cores.

Post Reply