Batch Processing Sentinel-2 in l2gen

Please enter here to ask a question about any NASA Science related topics!
Post Reply
Posts: 33
Joined: Tue Oct 11, 2016 11:49 am America/New_York

Batch Processing Sentinel-2 in l2gen

by dogun » Wed Dec 05, 2018 12:57 am America/New_York

Wondering if there's a way to batch process multiple Sentinel-2 scenes of different areas in l2gen? Also, is it possible to make l2gen with Sentinel-2 scenes run faster (multiprocessor?).



Posts: 606
Joined: Fri Jan 29, 2021 5:51 pm America/New_York
Answers: 1

Batch Processing Sentinel-2 in l2gen

by gnwiii » Wed Dec 05, 2018 7:31 am America/New_York

If you have many files to process and have a suitable multi-processor system you can run multiple jobs.   GNU parallel is designed for just this sort of task:

"GNU parallel is a shell tool for executing jobs in parallel using one or more computers. A job can be a single command or a small script that has to be run for each of the lines in the input. The typical input is a list of files, a list of hosts, a list of users, a list of URLs, or a list of tables. A job can also be a command that reads from a pipe. GNU parallel can then split the input and pipe it into commands in parallel."

Note that l2gen processing is quite I/O intensive and many modern muti-core CPU's can saturate disk IO.  You may find it helpful to write output to a separate filesystem.
A common strategy to determine how many jobs can be run in parallel is to start with 2 simultaneous, then 4, 8, ... until thruput falls off.

Post Reply