l2gen products

Use this Forum to find information on, or ask a question about, NASA Earth Science data.
gnwiii
Posts: 713
Joined: Fri Jan 29, 2021 5:51 pm America/New_York
Answers: 2
Has thanked: 1 time

l2gen products

by gnwiii » Mon Apr 27, 2020 1:09 pm America/New_York

Java performance is sensitive to memory allocations.  Both the GUI and gpt.sh use the same Graph Processing Framework (GPF) libraries, so "Proper Parameters Prevent PXXX Poor Performance".  Java memory allocation is a tricky topic, but for starters, you should compare the values of the "-Xmx" option used by the SeaDAS 7 GUI from seadas.vmoptions with the value in your .  On a system used for small teaching examples:

% grep Xmx seadas.vmoptions gpt.sh
seadas.vmoptions:# -Xmx512m
seadas.vmoptions:-Xmx1024M
gpt.sh:    -Xmx1024M \


Here, the hash (#) indicates a line that has been commented out.   These can be considered starting values.   If you encounter performance problems, try doubling the values until performance is acceptable without exceeding 60% of the system memory.   If your system has minimal RAM (4GB for linux) you may need to add RAM.   Reducing the size of your level-2 files will lower memory demands for GPF processing.  

If your gpt.sh script already has the same setting as your GUI, you may be attempting to use more memory than is installed on the system.  Linux can use swapping to temporarily stores areas of memory to disk, but this is very slow.   On lower spec machines you may not be able to use the GUI and gpt.sh efficiently at the same time.   There are many tools that can be used to monitor system memory usage.   You can use the free command in a terminal window:
$ free -m
              total        used        free      shared  buff/cache   available
Mem:           6256          63        3344           0        2849        5906
Swap:          2048           0        2048


Consult the references for your distribution for GUI tools.

Tags:

OB SeaDAS - knowles
Subject Matter Expert
Subject Matter Expert
Posts: 271
Joined: Mon Apr 07, 2008 4:40 pm America/New_York
Answers: 0

l2gen products

by OB SeaDAS - knowles » Mon Apr 27, 2020 2:13 pm America/New_York

In addition to possible memory allocation differences between the GUI and command line gpt run of Reprojection, the GUI run of Reprojection can be extremely fast if you choose to NOT save the target file.  In this GUI case, the target file is "virtual" and reprojection does not actually occur until a band of the target is actually viewed in the GUI, and even then reprojection only occurs in the target band being viewed, not the full file.

When running gpt, you might also choose to close the SeaDAS GUI application during the run for possible better use of the memory so as to not have 2 simultaneous instances of seadas going.

Danny

avmehta
Posts: 184
Joined: Mon Feb 03, 2020 10:27 am America/New_York
Answers: 0
Been thanked: 1 time

l2gen products

by avmehta » Mon Apr 27, 2020 4:04 pm America/New_York

Thanks Danny.
I am running the GUI and gpt on two different computers.
I am note sure that only rasters reprojected are the ones you view! I open an L2 file (.nc) in GUI created by l2gen on my Linux machine via a batch job, just select the file and choose raster-->reproject! There is an option to save the reprojected file that I use and I do get a 'reprojected' file with all the rasters!

Thanks.
Amita

gnwiii
Posts: 713
Joined: Fri Jan 29, 2021 5:51 pm America/New_York
Answers: 2
Has thanked: 1 time

l2gen products

by gnwiii » Mon Apr 27, 2020 5:20 pm America/New_York

Are both computers processing files on local disks or is one using files shared on a network drive?   Does one system have a rotating magentic media disk and the other a modern solid-state disk?   Perhaps the computer you use for gpt is just older and slower than the one you use for the GUI.  Even with identical hardware, newer Java versions may be faster at some operations.   Also, Java on Windows may use runtime libraries better matched to the hardware.  On linux, Java may be built to run on a wider range of hardware and miss optimizations for recent processor models.

OB.DAAC - SeanBailey
User Services
User Services
Posts: 1469
Joined: Wed Sep 18, 2019 6:15 pm America/New_York
Answers: 1
Been thanked: 5 times

l2gen products

by OB.DAAC - SeanBailey » Wed Apr 29, 2020 8:21 am America/New_York

The L2 file format is the same for extracts (subset) as for non-extracted data, so to the reprojecton tool they are the same.  There is no issue reprojecting  data subsetted at the L2 step.

Sean

avmehta
Posts: 184
Joined: Mon Feb 03, 2020 10:27 am America/New_York
Answers: 0
Been thanked: 1 time

l2gen products

by avmehta » Wed Apr 29, 2020 8:58 am America/New_York

Great! Thanks for clarifying this, Sean.
Amita

OB SeaDAS - knowles
Subject Matter Expert
Subject Matter Expert
Posts: 271
Joined: Mon Apr 07, 2008 4:40 pm America/New_York
Answers: 0

l2gen products

by OB SeaDAS - knowles » Wed Apr 29, 2020 11:46 am America/New_York

This is correct Amita.  If you save the file then the reprojection occurs for all bands.  As I mentioned, if you create a "virtual" file (not saved but just in the GUI) then the reprojection does not occur until you view the band.

thanks
Danny

swathy
Posts: 12
Joined: Wed Jun 22, 2016 11:00 am America/New_York
Answers: 0

l2gen products

by swathy » Wed May 20, 2020 6:57 pm America/New_York

I was trying to reproject L2 SST files downloaded from Oceacolor website. It is showing the following error. It will be great if any of you could help.
OS: Windows
Command line code:FOR %X in (I:\DINEOF\BOB\test\*.nc) DO (gpt reproject.xml -Ssource="%X" -f 'GeoTIFF' -Ptarget="I:\DINEOF\BOB\test\test2\%~nX.tif")
Error :
I:\DINEOF\BOB\test>(gpt reproject.xml -Ssource="I:\DINEOF\BOB\test\AQUA_MODIS.20
071229T071507.L2.SST.x.nc" -f 'GeoTIFF' -Ptarget="I:\DINEOF\BOB\test\test2\AQUA_
MODIS.20071229T071507.L2.SST.x.tif" )
04:11:29.614 [main] DEBUG ucar.nc2.NetcdfFile - Using IOSP ucar.nc2.iosp.hdf5.H5
iosp
04:11:29.678 [main] DEBUG ucar.nc2.NetcdfFile - Using IOSP ucar.nc2.iosp.hdf5.H5
iosp
04:11:29.712 [main] DEBUG ucar.nc2.NetcdfFile - Using IOSP ucar.nc2.iosp.hdf5.H5
iosp
04:11:29.739 [main] DEBUG ucar.nc2.NetcdfFile - Using IOSP ucar.nc2.iosp.hdf5.H5
iosp
04:11:29.759 [main] DEBUG ucar.nc2.NetcdfFile - Using IOSP ucar.nc2.iosp.hdf5.H5
iosp
04:11:29.781 [main] DEBUG ucar.nc2.NetcdfFile - Using IOSP ucar.nc2.iosp.hdf5.H5
iosp

Error: java.lang.NullPointerException

I:\DINEOF\BOB\test>(gpt reproject.xml -Ssource="I:\DINEOF\BOB\test\AQUA_MODIS.20
071230T080006.L2.SST.x.nc" -f 'GeoTIFF' -Ptarget="I:\DINEOF\BOB\test\test2\AQUA_
MODIS.20071230T080006.L2.SST.x.tif" )
04:11:31.687 [main] DEBUG ucar.nc2.NetcdfFile - Using IOSP ucar.nc2.iosp.hdf5.H5
iosp
04:11:31.752 [main] DEBUG ucar.nc2.NetcdfFile - Using IOSP ucar.nc2.iosp.hdf5.H5
iosp
04:11:31.787 [main] DEBUG ucar.nc2.NetcdfFile - Using IOSP ucar.nc2.iosp.hdf5.H5
iosp
04:11:31.814 [main] DEBUG ucar.nc2.NetcdfFile - Using IOSP ucar.nc2.iosp.hdf5.H5
iosp
04:11:31.834 [main] DEBUG ucar.nc2.NetcdfFile - Using IOSP ucar.nc2.iosp.hdf5.H5
iosp

Error: java.lang.NullPointerException

XML file used :

<graph id="ReprojectionGraph">
  <version>1.0</version>
  <node id="reprojectNode">
    <operator>Reproject</operator>
    <sources>
    <source>${source}</source>
   </sources>
    <parameters>
    <wktFile/>
    <crs>GEOGCS["WGS 84",
      DATUM["World Geodetic System 1984",
      SPHEROID["WGS 84", 6378137.0, 298.257223563, AUTHORITY["EPSG","7030"]],
      AUTHORITY["EPSG","6326"]],
      PRIMEM["Greenwich", 0.0, AUTHORITY["EPSG","8901"]],
      UNIT["degree", 0.017453292519943295],
      AXIS["Geodetic longitude", EAST],
      AXIS["Geodetic latitude", NORTH],
      AUTHORITY["EPSG","4326"]]</crs>
    <resampling>Nearest</resampling>
    <orthorectify>false</orthorectify>
    <noDataValue>NaN</noDataValue>
    <includeTiePointGrids>true</includeTiePointGrids>
    <addDeltaBands>false</addDeltaBands>
  </parameters>
  </node>
</graph>

Thanking you in advance

OB SeaDAS - knowles
Subject Matter Expert
Subject Matter Expert
Posts: 271
Joined: Mon Apr 07, 2008 4:40 pm America/New_York
Answers: 0

l2gen products

by OB SeaDAS - knowles » Fri May 22, 2020 11:01 am America/New_York

I downloaded the file (https://oceandata.sci.gsfc.nasa.gov/ob/getfile/AQUA_MODIS.20071229T071507.L2.SST.nc) and ran it on my mac with no problem.

I used your graph file and the command line I used is:
gpt reproject.xml -Ssource=AQUA_MODIS.20071229T071507.L2.SST.nc -f GeoTIFF -Ptarget=AQUA_MODIS.20071229T071507.L2.SST_reprojected.tif

I do see an "x" in your file name likely indicating an extract.  I don't believe this is the issue but to rule it out perhaps try running it on the unextracted file.

thanks
Danny

swathy
Posts: 12
Joined: Wed Jun 22, 2016 11:00 am America/New_York
Answers: 0

l2gen products

by swathy » Fri May 22, 2020 1:17 pm America/New_York

Hi Danny,
Thank you for your response. Unfortunately, it didn't work for me. I downloaded the file from your link and tried with your command line with the same graph file I posted earlier. However I got the following error.

I:\DINEOF\BOB\test>gpt reproject.xml -Ssource=AQUA_MODIS.20071229T071507.L2.SST.
nc -f GeoTIFF -Ptarget=AQUA_MODIS.20071229T071507.L2.SST_reprojected.tif
22:48:15.019 [main] DEBUG ucar.nc2.NetcdfFile - Using IOSP ucar.nc2.iosp.hdf5.H5
iosp

Error: null

Post Reply