Running simultaneous instances of multilevel_processor.py

Please enter here to ask a question about any NASA Science related topics!
Post Reply
tconroy
Posts: 1
Joined: Mon May 10, 2021 9:37 pm America/New_York

Running simultaneous instances of multilevel_processor.py

by tconroy » Tue May 11, 2021 5:16 pm America/New_York

I am trying to run multilevel_processor as a job array on a cluster, to simultaneously run the code on multiple MODIS L1A files at once. I have followed the advice of past forum posts, here for example:

https://oceancolor.gsfc.nasa.gov/forum/ ... l?tid=5751

Where I have run get_anc.py and modis_atteph.py for each L1A file (in the file “create_anc_files.qsub”) prior to running multilevel_processor, which I do in the file “file_array.qsub”. In my parameter file “nir_settings.par” code I specify the --disable-download option and the --ancdir and --ancdb directories in the geo and l1a_extract_modis sections.

However, when I run this code for an L1A file directory with two files in it, only one of the files completes and the other fails at GEO (log files attached, and they have the naming convention of Seadas_NIR.o). For the file that was successful, an .anc file was created in the directory that I ran the code in, which makes me think that there is an error with my setup and the --disable download isn’t working as I am intending.

I have put all the files I have referred to here (as I wasn't able to attach them):
https://www.dropbox.com/sh/k79zgfxblqje ... FAoda?dl=0

Any advice would be appreciated.

Thanks,
Ted

Tags:

OB.DAAC - SeanBailey
User Services
User Services
Posts: 1164
Joined: Wed Sep 18, 2019 6:15 pm America/New_York

Re: Running simultaneous instances of multilevel_processor.py

by OB.DAAC - SeanBailey » Thu May 13, 2021 10:57 am America/New_York

Ted,

You've stumbled upon an issue that's been a thorn for a long time. The problem is that the MODIS scripts use non-unique temporary files, so running more than one instance at a time will result in disaster....well disaster may be too strong, but at least a not nice situation. We have it on our todo list to correct this issue, and hopefully with an upcoming release it will be resolved. Until then, a workaround would be for your controlling process to create temporary working directories for each instance to avoid one process stepping on the toes of another.

Sean

Post Reply