l2gen segmentation fault

Please enter here to ask a question about any NASA Science related topics!
gbourdin
Posts: 12
Joined: Mon Jan 25, 2021 11:59 am America/New_York

l2gen segmentation fault

by gbourdin » Sun Oct 03, 2021 8:29 pm America/New_York

Hello,
I am facing two different errors related to a segmentation fault running l2gen on ~50 VIIRS-SNPP L1A images.
I copy below two examples but all ~50 images fail during l2gen:

1) V2016258230600.L1A_SNPP.nc gives me the following error:

Begin l2gen Version 9.5.1-V2021.2 Processing
Sensor is VIIRSN
Sensor ID is 11
Sensor has 10 reflective bands
Sensor has 5 emissive bands
Number of along-track detectors per band is 16
Number of input pixels per scan is 3200
Processing pixels 1 to 3200 by 1
Processing scans 1 to 3248 by 1
Ocean processing enabled
Land processing enabled
Atmospheric correction disabled

Begin MSl12 processing at 2021276183912000

Allocated 3824040 bytes in L1 record.
Allocated 3824040 bytes in L1 record.
Allocated 3824040 bytes in L1 record.
Allocated 3824040 bytes in L1 record.
Allocated 3824040 bytes in L1 record.
Allocated 3824040 bytes in L1 record.
Allocated 3824040 bytes in L1 record.
*** Error in `l2gen': corrupted double-linked list: 0x000000001df48d70 ***


2) V2016249205400.L1A_SNPP.nc gives me the following error:

Begin l2gen Version 9.5.1-V2021.2 Processing
Sensor is VIIRSN
Sensor ID is 11
Sensor has 10 reflective bands
Sensor has 5 emissive bands
Number of along-track detectors per band is 16
Number of input pixels per scan is 3200
Processing pixels 1 to 3200 by 1
Processing scans 1 to 3232 by 1
Ocean processing enabled
Land processing enabled
Atmospheric correction disabled

Begin MSl12 processing at 2021276184343000

Allocated 3824040 bytes in L1 record.
Allocated 3824040 bytes in L1 record.
Allocated 3824040 bytes in L1 record.
Allocated 3824040 bytes in L1 record.
Allocated 3824040 bytes in L1 record.
Allocated 3824040 bytes in L1 record.
Allocated 3824040 bytes in L1 record.
*** Error in `l2gen': corrupted size vs. prev_size: 0x0000000003d09570 ***

This same code was working a month ago with OCSSW up to date.
Any idea where this is coming from?

Best,
Guillaume

Tags:

gbourdin
Posts: 12
Joined: Mon Jan 25, 2021 11:59 am America/New_York

Re: l2gen segmentation fault

by gbourdin » Tue Oct 19, 2021 11:33 am America/New_York

It seems that the subsetting causes l2gen to crash but only with VIIRS-SNPP and VIIRS-JPSS1, MODIS-Aqua and MODIS-Terra work fine. It was working on a previous version of OCSSW and I think I updated OCSSW just recently, I am now using the tag V2021.2.
I have several thousands images to process so I really need to subset each image to my zone of interest.
Anyone knows how to fix this?
Thank you,
Guillaume

OB SeaDAS - dshea
Subject Matter Expert
Subject Matter Expert
Posts: 220
Joined: Thu Mar 05, 2009 10:25 am America/New_York

Re: l2gen segmentation fault

by OB SeaDAS - dshea » Tue Oct 19, 2021 12:40 pm America/New_York

Can you give me the exact command lines you are using to go from V2016258230600.L1A_SNPP.nc to the subsetted files that makes l2gen crash? I will try to reproduce the crash here.

don

gbourdin
Posts: 12
Joined: Mon Jan 25, 2021 11:59 am America/New_York

Re: l2gen segmentation fault

by gbourdin » Tue Oct 19, 2021 4:36 pm America/New_York

This is not specific to one image, all VIIRS images crash but only when I subset. I have tried the same settings from the GUI and every image crashes as well when I subset, some right at the start, but others in the middle or at the end of the process.

I call this command from python:
cmd = [OCSSW_RUNNER, '--ocsswroot', PATH_OCSSW, 'l2gen',
'suite='OC',
'ifile=V2016248211200.L1A_SNPP.nc',
'geofile=V2016248211200.GEO-M_SNPP.nc',
'ofile=V2016248211200.L1C_SNPP.nc',
'maskland=1',
'maskhilt=0',
'north=-23.78',
'south=-30.45',
'east=-106.03',
'west=-112.70',
'gain=[1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0]',
'maskcloud=0',
'aer_opt=-99',
'brdf_opt=0',
'atmocor=0',
'l2prod=rhot_nnn,polcor_nnn,sena,senz,sola,solz,latitude,longitude']
if get_anc:
foo = filter(lambda x: x != "", ancil_list.split('<>')) # remove empty string
anc_key = list(foo)
for key in anc_key:
cmd.append(key) # append each ancillary file to cmd
check_call(cmd)

Thank you,
Guillaume

OB SeaDAS - dshea
Subject Matter Expert
Subject Matter Expert
Posts: 220
Joined: Thu Mar 05, 2009 10:25 am America/New_York

Re: l2gen segmentation fault

by OB SeaDAS - dshea » Wed Oct 20, 2021 11:49 am America/New_York

I can reproduce the seg fault on my Linux machine. Seems to work fine on my mac, but that is how these memory issues usually behave. There is an easy work around that you can put into your processing stream. l2gen works fine if you use an L1B file as the ifile instead of an L1A.

Using the example file you gave me, first produce the L1B:

Code: Select all

calibrate_viirs ifile=V2016248211200.L1A_SNPP.nc l1bfile_mod=V2016248211200.L1B-M_SNPP.nc
Next, pass the L1B file into l2gen:

Code: Select all

l2gen suite='OC' ifile=V2016248211200.L1B-M_SNPP.nc geofile=V2016248211200.GEO-M_SNPP.nc ofile=V2016248211200.L1C_SNPP.nc maskland=1 maskhilt=0 north=-23.78 south=-30.45 east=-106.03 west=-112.70 'gain=[1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0]' maskcloud=0 aer_opt=-99 brdf_opt=0 atmocor=0 l2prod=rhot_nnn,polcor_nnn,sena,senz,sola,solz,latitude,longitude
Let me know if this works for you.

This is how VIIRS L2 files are produced in our production system at the moment, which is why this bug was not discovered by our regression tests. I will let you know when I get this fixed.

Thank You for finding this bug.

don

gbourdin
Posts: 12
Joined: Mon Jan 25, 2021 11:59 am America/New_York

Re: l2gen segmentation fault

by gbourdin » Wed Oct 20, 2021 1:30 pm America/New_York

Thank you Don for looking into this.
The process is running with calibrate_viirs and then l2gen but it performs the atmospheric correction on the whole image even when include north, south, east and west boundaries.
It is definitely too slow to process, even on the university HPC, and the output files are too large.
Do you know any way how to make the subset work?
Thank you,
Guillaume

OB SeaDAS - dshea
Subject Matter Expert
Subject Matter Expert
Posts: 220
Joined: Thu Mar 05, 2009 10:25 am America/New_York

Re: l2gen segmentation fault

by OB SeaDAS - dshea » Thu Oct 21, 2021 1:04 pm America/New_York

OK, looks like the n,s,e,w params for l2gen processing VIIRS has a nasty bug in it. Here is what I suggest for your processing stream.

- run lonlat2pixline with the GEO file and n,s,e,w to get start/end, line/pixel for region of interest
- run l2gen with L1A and GEO file and start/end, line/pixel params instead of n,s,e,w

Here are the steps and the timing results run on my desktop Linux machine:

Code: Select all

dshea@gs616-crab:~/data/tmp$ time lonlat2pixline V2016248211200.GEO-M_SNPP.nc -112.70 -30.45 -106.03 -23.78
# 777 1595 2713 3248
sline=2713
eline=3248
spixl=777
epixl=1595

real    0m0.378s
user    0m0.366s
sys     0m0.012s

dshea@gs616-crab:~/data/tmp$ time l2gen suite=OC ifile=V2016248211200.L1A_SNPP.nc geofile=V2016248211200.GEO-M_SNPP.nc ofile=V2016248211200.L1C_SNPP.nc maskland=1 maskhilt=0 sline=2713 eline=3248 spixl=777 epixl=1595 'gain=[1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0]' maskcloud=0 aer_opt=-99 brdf_opt=0 atmocor=0 l2prod=rhot_nnn,polcor_nnn,sena,senz,sola,solz,latitude,longitude

real    0m18.696s
user    0m18.355s
sys     0m0.340s
Processing time is down to 18 seconds. The full l2gen run was taking over 4 minutes and the size of the L2 file is much smaller now:

Code: Select all

dshea@gs616-crab:~/data/tmp$ ls -l *L1C*
-rw-rw-r-- 1 dshea dshea 453937253 Oct 21 15:24 V2016248211200.L1C_SNPP.full.nc
-rw-rw-r-- 1 dshea dshea  19293308 Oct 21 16:30 V2016248211200.L1C_SNPP.nc
Let me know if you have any questions.

don

gbourdin
Posts: 12
Joined: Mon Jan 25, 2021 11:59 am America/New_York

Re: l2gen segmentation fault

by gbourdin » Thu Oct 21, 2021 7:15 pm America/New_York

It is running well when I use lonlat2pixline thank you Don.
I hope the subsetting tool on l2gen will be fixed though, it is much easier to use.
Best,
Guillaume

OB SeaDAS - dshea
Subject Matter Expert
Subject Matter Expert
Posts: 220
Joined: Thu Mar 05, 2009 10:25 am America/New_York

Re: l2gen segmentation fault

by OB SeaDAS - dshea » Fri Oct 22, 2021 8:49 am America/New_York

Yes, the n,s,e,w params for l2gen are more convenient than grabbing the output of lonlat2pixline. I am almost done fixing the bug. Expect the fix in the next release.

Thanks again for finding this.

don

gbourdin
Posts: 12
Joined: Mon Jan 25, 2021 11:59 am America/New_York

Re: l2gen segmentation fault

by gbourdin » Mon Oct 25, 2021 11:06 am America/New_York

Hi Don,
Do you have any time estimate for the next release, please?
I really need to finish this analysis soon and I am running into problems using the lonlat2pixline. It was working fine on images on the eastern part of the Pacific Ocean, East of the prime meridian, but it is crashing on the Western side of the Pacific Ocean, West of the prime meridian. I am batch processing a lot of images so I haven't checked all files one by one but it seems to crash only when all longitudes are positives. I am not sure it includes all pixels it should when the image is right on top of the prime meridian but there is definitely an issue worth investigating.

An example that work:
lonlat2pixline V2016317230600.GEO-M_SNPP.nc -154.83 -144.83 -22.53 -12.53

An example that runs but I am not sure of the output:
lonlat2pixline V2017003013600.GEO-M_SNPP.nc 175.78 -177.55 -11.84 -5.18

Three examples that crash:
lonlat2pixline V2016362020600.GEO-M_SNPP.nc 168.25 177.58 -2.88 6.46
lonlat2pixline V2017035032400.GEO-M_SNPP.nc 140.48 148.81 9.14 17.47
lonlat2pixline V2019048014800.GEO-M_SNPP.nc 165.96 -170.70 -29.89 -6.56

I have tried swapping east and west longitude, just in case, but it doesn't change anything.

Best,
Guillaume

Post Reply