Empty files after processing

Please enter here to ask a question about any NASA Science related topics!
Post Reply
madjidh
Posts: 25
Joined: Wed Feb 27, 2019 10:05 am America/New_York

Empty files after processing

by madjidh » Mon Sep 07, 2020 11:34 am America/New_York

Hello,

I processed some MODIS-A files with the standard corrections applied, using a shell script, from L1 to L3.
Sometimes, an image is empty at the end (masks/clouds) so the final file is not saved (l3map).
I wonder :
1) How many pixels / percentage of pixels are required for a file to be saved ?
2) Is there a way to force a file to be saved ?
3) When a file is empty and then deleted after the processing, can we get the information by doing something ? So I can remove it from the list later and avoid processing it again if I need to reprocess the data. I could check if the file exists after the processing, but then I would miss real failure of the processing, that is why I ask.
4) Bonus : How can we change the size of the straylight mask ?

Regards

Tags:

gnwiii
Posts: 605
Joined: Fri Jan 29, 2021 5:51 pm America/New_York
Answers: 1

Empty files after processing

by gnwiii » Mon Sep 07, 2020 3:31 pm America/New_York

l3mapgen -h
l3mapgen 2.2.0-V2019.3 (Aug 16 2019 12:46:16)
Usage: l3mapgen argument-list

  This program takes a product (or products if netCDF output) from an L3 bin
  or SMI file, reprojects the data using proj.4 and writes a mapped file in
  the requested output format.

  Return values
    0 = All Good
    1 = Error
    110 = No valid data to map
[...]
   threshold (float) (default=0) = minimum percentage of filled pixels before
        an image is generated
   num_cache (int) (default=500) = number of rows to cache in memory.
   mask_land (boolean) (default=no) = set land pixels to pixel value 254
   land (ifile) (default=$OCDATAROOT/common/landmask_GMT15ARC.nc) = land mask file
   full_latlon (boolean) (default=yes) = write full latitude and longitude arrays (except for SMI)


The return value of 110 should allow you to separate processing errors from files with no filled pixels.

madjidh
Posts: 25
Joined: Wed Feb 27, 2019 10:05 am America/New_York

Empty files after processing

by madjidh » Tue Sep 08, 2020 5:03 am America/New_York

Thanks George, any chance to see an example ?

gnwiii
Posts: 605
Joined: Fri Jan 29, 2021 5:51 pm America/New_York
Answers: 1

Empty files after processing

by gnwiii » Tue Sep 08, 2020 7:35 am America/New_York

If you are using bash or a similar shell (dash or zsh), the return value is stored in "$?".   For l3mapgen, you can adapt the following sample script:
#! /bin/bash
#l3mapgen ...
./retval.sh $1
case $? in
0) echo All good from l3mapgen
    ;;
1) echo l3mapgen: error
    echo "insert error processing here ..."
    ;;
110) echo l3mapgen: No valid data to map
   echo "insert no data handling here ... "
   ;;
*) echo Novel return value from l3mapgen.
    echo "Insert novel return value handling ..."
    ;;
esac

To test this script, create retval.sh:#! /bin/bash
exit $1

Make these two scripts executable and try them out:
$ ./retval.sh 0; echo $?
0
$ ./retval.sh 110; echo $?
110
$ ./runner.sh 110
l3mapgen: No valid data to map
insert no data handling here ...
$ ./runner.sh 99
Novel return value from l3mapgen.
Insert novel return value handling ...

madjidh
Posts: 25
Joined: Wed Feb 27, 2019 10:05 am America/New_York

Empty files after processing

by madjidh » Tue Sep 08, 2020 10:23 am America/New_York

Fantastic, thanks again !

Post Reply