SeaDAS 7.5 RedHat varients now useless?

[oo_processing]

Sean,
Since one cannot update GLIBC without breaking the OS, are we now at a point where all Red Hat variants are no longer supported? You are supporting Macs and Windows now, but not the largest market share linux?

Both EL7 and EL6 have GLIBC < the required GLIBC-2.23, so is it the case that the hundreds of computers running RHEL6/RHEL7 variants are no longer able to use SeaDAS? (our university has a support agreement with Red Hat, and all our cluster operations (100s of machines), use the most current Red Hat EL6. These machines have literally hundreds of specialized custom compiled programs used by many many different research groups. The university cannot update to another distribution without expending enormous resources including breaking contracts, etc.  And this affects all  Red Hat, CentOS, Scientific linux and many other distros shown here:
https://en.wikipedia.org/wiki/Red_Hat_Enterprise_Linux_derivatives
And Red Hat is the market leader of Linux distributions, with an estimated market share of 67%!!! (Not sure if that includes variants).

Comments suggested that it can be recompiled from source, however, VIIRS is not open source code, and thus cannot be recompiled from source as none exists for distribution. If I use 7.4 to process VIIRS data, am I correct that I will not get updated LUTS? Have/will the LUTS changed for VIIRS in 7.5? If I don't update the code, I can never get new LUTS for terra modis regardless. And if I do recompile, I cannot process VIIRS which is a huge part of our business.

You mentioned in one place:
"Since you're in a CentOS world, you may be able to just build on the oldest system and run those binaries on all the machines."

However, it looks that is never going to work for VIIRS. We were able to use RHEL until now. Is it not possible that the SeaDAS group compiles SeaDAS 7.5 with compat libraries that can be used on the Red Hat variants (And others)? There isn't really an alternative that I can see right now. Our cluster has been down this week for updates to the GPFS file system alone (people from IBM are here now) so I would imagine, that changing everything for one program isn't going to happen. It would take compiling hundreds of programs.
https://en.wikipedia.org/wiki/IBM_General_Parallel_File_System

If I am missing something, please let me know.
Brock

[treble]

Looks like I am in the same boat.  I am on CentOS 7, with GLIBC 2.17....  I tried compiling from source as well.  No luck.  Stuck at Step 4 at https://seadas.gsfc.nasa.gov/build_ocssw/


[...]
Scanning dependencies of target H5test_FortranHavekind
[  0%] Built target zip_perf_files
[  0%] Building Fortran object fortran/src/CMakeFiles/H5test_FortranHavekind-shared.dir/H5test_kind_STORAGE_SIZE.f90.o
[  0%] Building Fortran object fortran/src/CMakeFiles/H5test_FortranHavekind.dir/H5test_kind_STORAGE_SIZE.f90.o
[  0%] Building C object src/CMakeFiles/H5make_libsettings.dir/H5make_libsettings.c.o
[  0%] Building C object src/CMakeFiles/H5detect.dir/H5detect.c.o
[  0%] Linking C executable ../bin/H5make_libsettings
[  0%] Linking Fortran executable ../../bin/H5test_FortranHavekind-shared
[  0%] Linking Fortran executable ../../bin/H5test_FortranHavekind
[  0%] Built target H5make_libsettings
[  0%] Built target H5test_FortranHavekind-shared
[  0%] Built target H5test_FortranHavekind
[  0%] Generating ../H5fortran_detect.f90
/opt/seadas/seadas-7.5/ocssw/opt/src/hdf5/hdf5-1.8.18/build/bin/H5test_FortranHavekind: /lib64/libm.so.6: version `GLIBC_2.23' not found (required by /opt/seadas/seadas-7.5/ocssw/opt/lib/libquadmath.so.0)
make[2]: *** [fortran/H5fortran_detect.f90] Error 1
make[2]: *** Deleting file `fortran/H5fortran_detect.f90'
make[1]: *** [fortran/src/CMakeFiles/H5fortran_detect.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
[  0%] Linking C executable ../bin/H5detect
[  0%] Built target H5detect
make: *** [all] Error 2
***** make failed *****
***** BuildIt (hdf5) failed *****


[OB.DAACx - SeanBailey]

If  you have gcc-4; remove $OCSSWROOT/opt/lib/libquadmath.so.0 and try the binaries we distribute.
Sean

[OB.DAACx - SeanBailey]

Brock,

We are under no obligation to make sure users can compile the source code we distribute.
We do not support building OCSSW under Windows - you'll notice our example uses Ubuntu
as a VM (or WSL option)
macOS is easy since it's generally up-to-date and we have the OS here to build against.

VIIRS is another ball of wax, a legal one we've been struggling with for 3 years - and continue to do so.
I still hold out hope that we can release the code to the public, but for now we cannot.

It is not my business what your business is, nor is it my business to ensure you can do your business. 
The binaries we distribute support VIIRS, and are supported under freely available OSes. 
All OSes? No.  But ensuring it works on all OSes is not our business, nor do we have the resources to
make it our business.

While I understand your frustration, the reality is that our resources are limited and while we will help as we are able,
you get what you pay for.

BTW, another user, with Scientific Linux,  was able - with very few minor modifications -  to get the code to compile
...and his patches were included in the distribution this morning (oh, and those changes had nothing to do with GLIBC)
So, it's NOT all RH based OSes.

Sean

[OB SeaDAS - dshea]

I made a Virtual machine for CentOS 7 last night.  Here is what I did.


- Install CentOS 7
    glibc version is 2.17
- $ yum group install "Development Tools"
    This gives me gcc 4.8.5
- download https://oceandata.sci.gsfc.nasa.gov/ocssw/install_ocssw.py
- $ chmod +x install_ocssw.py
- $ install_ocssw.py -i ocssw -s --aqua
- $ export $OCSSWROOT=$HOME/ocssw
- $ source $OCSSWROOT/OCSSW_bash.env
- $ rm $OCSSWROOT/opt/lib/libquadmath.so.0
    The "Development Tools" supplies a compatable version
- grab an AQUA L1A file
- $ modis_GEO.py A2018101191500.L1A_LAC
- $ modis_L1B.py A2018101191500.L1A_LAC
- $ l2gen ifile=A2018101191500L1B_LAC geofile=A2018101191500.GEO ofile=A2018101191500.L2


Works fine without recompiling anything.

I am in the process of recompiling everything on CentOS 7.  I'll get back to you with the results of that.

Hope that helps.

don

[oo_processing]

Don,
Thanks. I was able to follow Sean's advice and:
rm $OCSSWROOT/opt/lib/libquadmath.so.0
But I'm happy that you tell me:
"The "Development Tools" supplies a compatable version"
And this worked in RHEL7, however, this seems not to help at all for RHEL6 machines:

[oo_processing@seadog ~]$ which l2gen
/optics1/software/seadas/seadas-7.5/ocssw/bin/l2gen
[oo_processing@seadog ~]$ l2gen -h
l2gen: /lib64/libc.so.6: version `GLIBC_2.17' not found (required by l2gen)
l2gen: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by l2gen)
l2gen: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by /optics1/software/seadas/seadas-7.5/ocssw/bin/../opt/lib/libmfhdf.so.0)
l2gen: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by /optics1/software/seadas/seadas-7.5/ocssw/bin/../opt/lib/libdf.so.0)
l2gen: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by /optics1/software/seadas/seadas-7.5/ocssw/bin/../opt/lib/libnetcdf.so.11)
l2gen: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by /optics1/software/seadas/seadas-7.5/ocssw/bin/../opt/lib/libhdf5.so.10.2.1)
l2gen: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by /optics1/software/seadas/seadas-7.5/ocssw/bin/../opt/lib/libgsl.so.19)
l2gen: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by /optics1/software/seadas/seadas-7.5/ocssw/bin/../opt/lib/libhdf5_hl.so.10.2.1)
l2gen: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by /optics1/software/seadas/seadas-7.5/ocssw/bin/../opt/lib/libproj.so.9)
l2gen: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by /optics1/software/seadas/seadas-7.5/ocssw/bin/../opt/lib/libgeotiff.so.2)
l2gen: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by /optics1/software/seadas/seadas-7.5/ocssw/bin/../opt/lib/libtiff.so.5)
l2gen: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by /optics1/software/seadas/seadas-7.5/ocssw/bin/../opt/lib/libpugixml.so.1)
l2gen: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by /optics1/software/seadas/seadas-7.5/ocssw/bin/../opt/lib/libfann.so.2)
l2gen: /lib64/libm.so.6: version `GLIBC_2.15' not found (required by /optics1/software/seadas/seadas-7.5/ocssw/bin/../opt/lib/libopenjp2.so.7)
l2gen: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by /optics1/software/seadas/seadas-7.5/ocssw/bin/../opt/lib/libopenjp2.so.7)
l2gen: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by /optics1/software/seadas/seadas-7.5/ocssw/bin/../opt/lib/libnetcdf_c++4.so.1)
l2gen: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by /optics1/software/seadas/seadas-7.5/ocssw/bin/../opt/lib/libnetcdff.so.6)
l2gen: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by /optics1/software/seadas/seadas-7.5/ocssw/bin/../opt/lib/libgfortran.so.3)
l2gen: /lib64/libc.so.6: version `GLIBC_2.17' not found (required by /optics1/software/seadas/seadas-7.5/ocssw/bin/../opt/lib/libgfortran.so.3)
l2gen: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by /optics1/software/seadas/seadas-7.5/ocssw/bin/../opt/lib/../lib/libxdr.so.0)
l2gen: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by /optics1/software/seadas/seadas-7.5/ocssw/bin/../opt/lib/../lib/libjpeg.so.9)

WHERE:
gcc -v
Using built-in specs.
Target: x86_64-redhat-linux
Configured with: ../configure --prefix=/usr --mandir=/usr/share/man --infodir=/usr/share/info --with-bugurl=http://bugzilla.redhat.com/bugzilla --enable-bootstrap --enable-shared --enable-threads=posix --enable-checking=release --with-system-zlib --enable-__cxa_atexit --disable-libunwind-exceptions --enable-gnu-unique-object --enable-languages=c,c++,objc,obj-c++,java,fortran,ada --enable-java-awt=gtk --disable-dssi --with-java-home=/usr/lib/jvm/java-1.5.0-gcj-xx.xxx.xx.xx/jre --enable-libgcj-multifile --enable-java-maintainer-mode --with-ecj-jar=/usr/share/java/eclipse-ecj.jar --disable-libjava-multilib --with-ppl --with-cloog --with-tune=generic --with-arch_32=i686 --build=x86_64-redhat-linux
Thread model: posix
gcc version 4.4.7 20120313 (Red Hat 4.4.7-18) (GCC)

Further, if I understand Sean correctly, if you recompile everything on CentOS7, you will not have the ability to use VIIRS processing.

The biggest issue I face is that I need to run the OCSSW code on a huge cluster with hundreds of machines, but they are all RHEL6. Is there any way you can compile the VIIRS in a CentOS6 virtual machine?

Brock

[OB SeaDAS - dshea]

OK, I compiled everything...

Using Virtual Machine with CentOS 7 above

- grab latest cmake binaries, install and get the new cmake into your PATH
- $ yum install libX11-devl.x86_64
- $ cd ocssw/opt/src
- $ ./BuildIt
- $ cd ../../ocssw-src
- $ mkdir build; cd build; cmake ..
- $ make -j 20 install

all works fine.

[OB SeaDAS - dshea]

Actually, we put precompiled VIIRS libs in ocssw/opt/lib, so you can compile and still have VIIRS functionality.  You need to use gcc-4.x since the C++ ABI changed at gcc-5.x

you need to edit ocssw/ocssw-src/CMakeLists.txt line 29

OPTION(BUILD_VIIRS_L1 "Build VIIRS L1 code" ON)

[oo_processing]

Note that my RHEL6 machines run gcc version 4.4.7 20120313 (Red Hat 4.4.7-18) (GCC)
Will this work since you state:
"You need to use gcc-4.x since the C++ ABI changed at gcc-5.x"
BTW, how did you get around the hdf5 require cmake3?

[OB SeaDAS - dshea]

We'll see how this plays out on CentOS 6.  Before the release we made sure everything worked on CentOS 7.

I just grabbed the latest binaries from cmake.org and installed them in my home dir.

don