L2Gen for 500m resolution too slow

[jellyking217]

**Title:** MODIS `l2gen` is very slow at 500 m resolution from existing L1B+GEO files — any recommended parameters or workflow improvements?

Hello SeaDAS / OCSSW team,

I am processing **MODIS Aqua** data with **existing L1B and GEO files**, and generating L2 products at **500 m resolution** using `l2gen`.

My main issue is that **`l2gen` is extremely slow** at 500 m resolution, and each individual process uses only a small fraction of one CPU core. I would like to ask whether this is expected, whether there are any recommended parameters for improving performance, or whether there may be something wrong in my workflow.

## Environment

* Platform: Linux (AutoDL server)
* OCSSW root: `/root/autodl-fs/ocssw`
* Python: `/root/miniconda3/bin/python3`
* Processing mode: existing **L1B + GEO -> L2**
* Resolution: **500**
* Parallel strategy:

* `getanc.py` runs serially
* `l2gen` runs in parallel across many scenes

## What I observed

For one scene, `l2gen` processing is very slow.
Example progress output looks like this:

```text
Processing scan # 50 (51 of 4060) after 311 seconds
Processing scan # 100 (101 of 4060) after 577 seconds
Processing scan # 150 (151 of 4060) after 837 seconds
Processing scan # 200 (201 of 4060) after 1086 seconds
Processing scan # 250 (251 of 4060) after 1344 seconds
Processing scan # 300 (301 of 4060) after 1594 seconds
Processing scan # 350 (351 of 4060) after 1850 seconds
Processing scan # 400 (401 of 4060) after 2100 seconds
Processing scan # 450 (451 of 4060) after 2354 seconds
Processing scan # 500 (501 of 4060) after 2599 seconds
```

This is roughly about **5 seconds per scan**, so one full scene can take many hours.

At the same time:

* each `l2gen` process uses only about **20%–30% of one CPU core**
* total system CPU usage remains low
* disk I/O is also low
* memory is sufficient

So this does **not** appear to be a CPU saturation or disk bottleneck problem.

## What I already checked

I monitored the server with `top` and `iostat`.

### CPU observations

* CPU idle is often around **90%**
* iowait is near **0%**
* memory is sufficient

### I/O observations

Typical `iostat -x 1` output shows:

* `%util` is very low (roughly 0–3%)
* `await` is very low
* no obvious storage bottleneck

This suggests the main limitation is **inside `l2gen` itself**, or perhaps some aspect of 500 m processing for MODIS.

## Current command / parameters

I am using a Python pipeline that:

1. scans existing `L1B` and `GEO`
2. runs `getanc.py` serially
3. runs `l2gen` in parallel

The main `l2gen` call is conceptually like this:

```bash
l2gen \
ifile=<L1B file> \
geofile=<GEO file> \
ofile=<output L2 file> \
par=<anc file> \
l2prod=rhos_469,rhos_555,rhos_645,rhos_859,rhos_1240,l2_flags \
resolution=500 \
oformat=netcdf4 \
proc_ocean=1 \
maskland=0 \
deflate=0
```

## Questions

1. Is this level of slowness for **MODIS `l2gen` at 500 m** expected?
2. Are there any recommended parameters to improve performance for this case?
3. Are there known performance issues in recent OCSSW / `l2gen` versions for MODIS 500 m processing?
4. Is there a recommended workflow for better throughput besides launching many separate `l2gen` processes?
5. Does `resolution=500` require any additional considerations regarding HKM companion files that could affect performance?

## Code structure

My script is a stable pipeline with:

* serial `getanc.py`
* recovery of `.anc` from log if needed
* parallel `l2gen`
* automatic HKM link repair for 500 m

If helpful, I can also provide the full Python script.

Any advice would be greatly appreciated. Thank you very much.

[OB SeaDAS - dshea]

What version of OCSSW are you running?
You might want to try installing the "--arch=odps" if your linux kernel is new enough to see if that helps.
When I run a MODIS granule at resolution=500, I get mostly CPU saturation.
If you are not CPU or IO limited and the memory is only about 1.2G, I don't know where to look for a bottle neck.

Here is my test setup.

My system is a decent Ubuntu 20.04 server
OCSSW V2026.0 arch=odps (Ubuntu 20.04 build)
(install_ocssw --arch=odps ...)

I ran the first 500 lines using this par file:

ifile=AQUA_MODIS.20251105T003500.L1B.hdf
geofile=AQUA_MODIS.20251105T003500.GEO.hdf
ofile=AQUA_MODIS.20251105T003500.L2.500.hdf
par=AQUA_MODIS.20251105T003500.L1B.hdf.anc
l2prod=rhos_469,rhos_555,rhos_645,rhos_859,rhos_1240,l2_flags
resolution=500
oformat=netcdf4
proc_ocean=1
maskland=0
deflate=0
eline=500

Output looks like this:

Processing scan # 50 (51 of 500) after 9 seconds
Processing scan # 100 (101 of 500) after 16 seconds
Processing scan # 150 (151 of 500) after 23 seconds
Processing scan # 200 (201 of 500) after 30 seconds
Processing scan # 250 (251 of 500) after 36 seconds
Processing scan # 300 (301 of 500) after 43 seconds
Processing scan # 350 (351 of 500) after 51 seconds
Processing scan # 400 (401 of 500) after 58 seconds
Processing scan # 450 (451 of 500) after 67 seconds
...
End MSl12 processing at 2026085141738000
Processing Rate = 6.666667 scans/sec

output if "time -v l2gen ..."

User time (seconds): 74.19
System time (seconds): 0.88
Percent of CPU this job got: 99%
Elapsed (wall clock) time (h:mm:ss or m:ss): 1:15.08
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 1221348
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 379189
Voluntary context switches: 193
Involuntary context switches: 7325
Swaps: 0
File system inputs: 2945298
File system outputs: 173470
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0

Also note that my MacBook Pro 2023 is quite a bit faster than this Linux server running l2gen.

don

[jellyking217]

Hi Don,

Thank you for the detailed reply and the benchmark result.

I checked my environment and found that I am currently running:

* `l2gen 9.11.0-V2025.5`
* Ubuntu 22.04.3
* Linux kernel `5.15.0-86-generic`

So my setup is different from yours, since you tested with `OCSSW V2026.0 arch=odps`.

I am now going to install a separate `V2026.0` environment with `--arch=odps` and repeat the test with the same input style, including a short `eline=500` benchmark, to compare performance and CPU usage.

My current issue is that `l2gen` at 500 m is still extremely slow and does not come close to saturating one CPU core, so I agree it makes sense to first rule out a version / build difference.

I will update the thread after I finish the comparison.

Thanks again.