Building on Ubuntu 18.04

bruce · by **bruce** » Wed Jun 20, 2018 10:49 am America/New_York

problem 1 (easy to solve) I was missing libx11-dev.

Problem 2 (tail end only)

[ 26%] Building Fortran object src/l2gen/CMakeFiles/l1mapgen.dir/atrem_bndprms.f.o
[ 26%] Building Fortran object src/l2gen/CMakeFiles/l1mapgen.dir/atrem_tpvmr_init.f.o
[ 26%] Building C object src/l2gen/CMakeFiles/l1mapgen.dir/numerical.c.o
[ 26%] Building C object src/l2gen/CMakeFiles/l1mapgen.dir/get_habs.c.o
[ 26%] Linking CXX executable l1mapgen
/usr/bin/x86_64-linux-gnu-ld: /seadas/lcl/ocssw/opt/lib/libViirsCal.a(Calibrate_Viirs_Connector.cpp.o): relocation R_X86_64_32 against `.rodata.str1.1' can not be used when making a PIE object; recompile with -fPIC
/usr/bin/x86_64-linux-gnu-ld: /seadas/lcl/ocssw/opt/lib/libViirsCal.a(VcstViirsCal.cpp.o): relocation R_X86_64_32 against symbol `__pthread_key_create@@GLIBC_2.2.5' can not be used when making a PIE object; recompile with -fPIC
/usr/bin/x86_64-linux-gnu-ld: /seadas/lcl/ocssw/opt/lib/libViirsCal.a(VcstObc.cpp.o): relocation R_X86_64_32 against symbol `__pthread_key_create@@GLIBC_2.2.5' can not be used when making a PIE object; recompile with -fPIC
/usr/bin/x86_64-linux-gnu-ld: /seadas/lcl/ocssw/opt/lib/libViirsCal.a(VcstViirsThermal.cpp.o): relocation R_X86_64_32S against symbol `_ZN7VcstObc9radRange_E' can not be used when making a PIE object; recompile with -fPIC
/usr/bin/x86_64-linux-gnu-ld: /seadas/lcl/ocssw/opt/lib/libViirsCal.a(VcstCalNetcdfInputItem.cpp.o): relocation R_X86_64_32S against symbol `_ZTV22VcstCalNetcdfInputItem' can not be used when making a PIE object; recompile with -fPIC
/usr/bin/x86_64-linux-gnu-ld: /seadas/lcl/ocssw/opt/lib/libViirsCal.a(VcstCalLutInputItem.cpp.o): relocation R_X86_64_32S against symbol `_ZTV19VcstCalLutInputItem' can not be used when making a PIE object; recompile with -fPIC
/usr/bin/x86_64-linux-gnu-ld: /seadas/lcl/ocssw/opt/lib/libViirsCal.a(VcstViirsBand.cpp.o): relocation R_X86_64_32S against symbol `_ZTV9ViirsBand' can not be used when making a PIE object; recompile with -fPIC
/usr/bin/x86_64-linux-gnu-ld: /seadas/lcl/ocssw/opt/lib/libViirsCal.a(VcstViirsBandDnb.cpp.o): relocation R_X86_64_32S against symbol `_ZTV12ViirsBandDnb' can not be used when making a PIE object; recompile with -fPIC
/usr/bin/x86_64-linux-gnu-ld: /seadas/lcl/ocssw/opt/lib/libViirsCal.a(VcstViirsBandImgRsb.cpp.o): relocation R_X86_64_32S against symbol `_ZTV15ViirsBandImgRsb' can not be used when making a PIE object; recompile with -fPIC
/usr/bin/x86_64-linux-gnu-ld: /seadas/lcl/ocssw/opt/lib/libViirsCal.a(VcstViirsBandImgTeb.cpp.o): relocation R_X86_64_32S against symbol `_ZTV15ViirsBandImgTeb' can not be used when making a PIE object; recompile with -fPIC
/usr/bin/x86_64-linux-gnu-ld: /seadas/lcl/ocssw/opt/lib/libViirsCal.a(VcstViirsBandModDGRsb.cpp.o): relocation R_X86_64_32S against symbol `_ZTV17ViirsBandModDGRsb' can not be used when making a PIE object; recompile with -fPIC
/usr/bin/x86_64-linux-gnu-ld: /seadas/lcl/ocssw/opt/lib/libViirsCal.a(VcstViirsBandModDGTeb.cpp.o): relocation R_X86_64_32S against symbol `_ZTV17ViirsBandModDGTeb' can not be used when making a PIE object; recompile with -fPIC
/usr/bin/x86_64-linux-gnu-ld: /seadas/lcl/ocssw/opt/lib/libViirsCal.a(VcstViirsBandModSGRsb.cpp.o): relocation R_X86_64_32S against symbol `_ZTV17ViirsBandModSGRsb' can not be used when making a PIE object; recompile with -fPIC
/usr/bin/x86_64-linux-gnu-ld: /seadas/lcl/ocssw/opt/lib/libViirsCal.a(VcstViirsBandModSGTeb.cpp.o): relocation R_X86_64_32S against symbol `_ZTV17ViirsBandModSGTeb' can not be used when making a PIE object; recompile with -fPIC
/usr/bin/x86_64-linux-gnu-ld: /seadas/lcl/ocssw/opt/lib/libViirsCal.a(VcstViirsBandImg.cpp.o): relocation R_X86_64_32S against symbol `_ZTV12ViirsBandImg' can not be used when making a PIE object; recompile with -fPIC
/usr/bin/x86_64-linux-gnu-ld: /seadas/lcl/ocssw/opt/lib/libViirsCal.a(VcstViirsBandMod.cpp.o): relocation R_X86_64_32S against symbol `_ZN9ViirsBand14processAtNightE' can not be used when making a PIE object; recompile with -fPIC
/usr/bin/x86_64-linux-gnu-ld: /seadas/lcl/ocssw/opt/lib/libViirsCal.a(VcstViirsBandModDG.cpp.o): relocation R_X86_64_32S against symbol `_ZTV14ViirsBandModDG' can not be used when making a PIE object; recompile with -fPIC
/usr/bin/x86_64-linux-gnu-ld: /seadas/lcl/ocssw/opt/lib/libViirsCmn.a(VcstCmnGeo.cpp.o): relocation R_X86_64_32 against symbol `__pthread_key_create@@GLIBC_2.2.5' can not be used when making a PIE object; recompile with -fPIC
/usr/bin/x86_64-linux-gnu-ld: /seadas/lcl/ocssw/opt/lib/libViirsCmn.a(VcstCmnLutInputItem.cpp.o): relocation R_X86_64_32S against symbol `_ZTV19VcstCmnLutInputItem' can not be used when making a PIE object; recompile with -fPIC
/usr/bin/x86_64-linux-gnu-ld: /seadas/lcl/ocssw/opt/lib/libViirsCmn.a(VcstLutInputItem.cpp.o): relocation R_X86_64_32S against symbol `_ZTV16VcstLutInputItem' can not be used when making a PIE object; recompile with -fPIC
/usr/bin/x86_64-linux-gnu-ld: /seadas/lcl/ocssw/opt/lib/libViirsCmn.a(VcstMath.cpp.o): relocation R_X86_64_32 against `.rodata.str1.1' can not be used when making a PIE object; recompile with -fPIC
/usr/bin/x86_64-linux-gnu-ld: /seadas/lcl/ocssw/opt/lib/libViirsCmn.a(VcstParamsReader.cpp.o): relocation R_X86_64_32 against `.rodata.str1.1' can not be used when making a PIE object; recompile with -fPIC
/usr/bin/x86_64-linux-gnu-ld: /seadas/lcl/ocssw/opt/lib/libViirsCmn.a(VcstPolarWanderItem.cpp.o): relocation R_X86_64_32S against symbol `_ZTV19VcstPolarWanderItem' can not be used when making a PIE object; recompile with -fPIC
/usr/bin/x86_64-linux-gnu-ld: /seadas/lcl/ocssw/opt/lib/libViirsCmn.a(VcstTime.cpp.o): relocation R_X86_64_32 against `.rodata.str1.8' can not be used when making a PIE object; recompile with -fPIC
/usr/bin/x86_64-linux-gnu-ld: /seadas/lcl/ocssw/opt/lib/libnovas.a(jplint.cpp.o): relocation R_X86_64_32S against `.bss' can not be used when making a PIE object; recompile with -fPIC
/usr/bin/x86_64-linux-gnu-ld: /seadas/lcl/ocssw/opt/lib/libnovas.a(novas.cpp.o): relocation R_X86_64_32 against `.rodata' can not be used when making a PIE object; recompile with -fPIC
/usr/bin/x86_64-linux-gnu-ld: final link failed: Nonrepresentable section on output
collect2: error: ld returned 1 exit status
src/l2gen/CMakeFiles/l1mapgen.dir/build.make:2813: recipe for target 'src/l2gen/l1mapgen' failed
make[2]: *** [src/l2gen/l1mapgen] Error 1
CMakeFiles/Makefile2:2241: recipe for target 'src/l2gen/CMakeFiles/l1mapgen.dir/all' failed
make[1]: *** [src/l2gen/CMakeFiles/l1mapgen.dir/all] Error 2
Makefile:140: recipe for target 'all' failed
make: *** [all] Error 2

gfireman · by **gfireman** » Wed Jun 20, 2018 11:16 am America/New_York

This suggests using the -no-pie compile option:
https://stackoverflow.com/questions/50213089/linking-error-with-gcc-g-7-3-0-on-ubuntu-18-04

You could edit $OCSSWROOT/CMakeLists.txt

   # standard Linux bits...
-  set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -std=c99 -D_XOPEN_SOURCE=600 -D_DEFAULT_SOURCE")
+  set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -std=c99 -no-pie -D_XOPEN_SOURCE=600 -D_DEFAULT_SOURCE")

However:
It's becoming obvious that Ubuntu 18.04 is not just 16.04 with a better UI. While we will likely support that OS someday, it will take some time to work out all the details. I strongly suggest reverting to Ubuntu 16.04 - that's what we use for development.

jasonlefler · by **jasonlefler** » Wed Jun 20, 2018 11:25 pm America/New_York

A bit of some good news/bad news here. Good news, OCSSW builds just fine on 18.04, with a single minor patch. Bad news, I couldn't reproduce any of the problems shown, except the xdr package problem. (Sidebar: the code in xdr that the compiler complains about worries me, as well; it does look like they're missing some brackets and I can't say with confidence if it's a typo or a style choice.) Setting the CFLAGS variable at the top of opt/src/BuildIt.env to "-Wno-misleading-indentation" got xdr through compilation and everything else went smoothly. My install is a fresh install of 18.04 from the Live CD, x64, using the Minimal Install option when it came up and only the following packages installed via apt:

csh tcsh unzip git cmake bison flex zlib1g zlib1g-dev libx11-dev subversion gfortran gcc g++

(gfortran, gcc, and g++ are all version 7.3.0.)

Where did you download the opt/ src tree from? What version does sdptk say it is? I guess I'd try commenting out the unzip (tar) command in sdptk/BuildIt, manually unzipping it, and modifying the last handful of lines in TOOLKIT/bin/INSTALL with a metric ton of debug commands, such as `env`, `echo $script_dir`, `ls $script_dir` and `ls $LIB3_DIR/src/sdptk/TOOLKIT/bin`. (I'm only assuming that INSTALL script is the right one.)

bruce · by **bruce** » Thu Jun 21, 2018 8:59 am America/New_York

I'll try again "later". I got the opt/src tree by choosing the "source code" option on the GUI that appears after "Data Processing > Install OC processors"

bruce · by **bruce** » Thu Jun 21, 2018 9:19 am America/New_York

I'm starting to get frustrated (well, more like "really frustrated", I was already frustrated).

I've created a new VM with 16.04, downloaded Seadas 7.5, installed the source package (from the GUI), installed all the prereqs and followed the instructions here - https://seadas.gsfc.nasa.gov/build_ocssw/.

When it gets to HDF5, I get...

What the H am I doing wrong?

Bruce

PS downloaded opt src package using wget (after blowing away what was already there) and the result is the same
attachment 1

bruce · by **bruce** » Thu Jun 21, 2018 10:18 am America/New_York

perhaps you could do an "apt list --installed" on one of your systems where it installs properly and I can see what packages you have installed that I don't. Feel free to email it rather than post it here if you prefer.

gfireman · by **gfireman** » Thu Jun 21, 2018 11:01 am America/New_York

Here's mine:

gs616-shark$ uname -a; lsb_release -a
Linux gs616-shark 4.4.0-128-generic #154-Ubuntu SMP Fri May 25 14:15:18 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 16.04.4 LTS
Release: 16.04
Codename: xenial
attachment 1

gfireman · by **gfireman** » Thu Jun 21, 2018 11:09 am America/New_York

A clue from your error log:

/seadas/lcl/ocssw/opt/src/hdf5/hdf5-1.8.18/build/bin/H5test_FortranHavekind: /seadas/lcl/ocssw/opt/lib/libgfortran.so.3: version `GFORTRAN_1.4' not found

Why is your compiler looking for gfortran in $LIB3_LIB?

gfireman · by **gfireman** » Thu Jun 21, 2018 11:33 am America/New_York

I overlooked the first error:
./BuildIt: 35: [: linux_64: unexpected operator

The hdf5 BuildIt script hasn't changed for some time. Please see if you can duplicate:

ls -l $LIB3_DIR/src/hdf5/BuildIt; wc $LIB3_DIR/src/hdf5/BuildIt
-rwxrwxr-x 1 gfireman gfireman 1899 2017-07-18 17:43 /accounts/gfireman/ocssw/opt/src/hdf5/BuildIt
  82  242 1899 /accounts/gfireman/ocssw/opt/src/hdf5/BuildIt

That's in our git repo; the same content in the public distribution has a later date:
-rwxrwxr-x 1 gfireman gfireman 1899 2017-10-12 14:32 ocssw-src/opt/src/hdf5/BuildIt

bruce · by **bruce** » Thu Jun 21, 2018 1:35 pm America/New_York

bcb@ubuntu:/seadas/lcl/ocssw/opt/src/hdf5$ ls -l BuildIt
-rwxrwxr-x 1 bcb bcb 1899 Oct 12  2017 BuildIt
bcb@ubuntu:/seadas/lcl/ocssw/opt/src/hdf5$ wc BuildIt
  82  242 1899 BuildIt
bcb@ubuntu:/seadas/lcl/ocssw/opt/src/hdf5$

So I edited the hdf5 BuildIt to fix this

#if [ "$OCSSW_ARCH" == "macosx_intel" ]; then
#    fortranSwitch="-DHDF5_BUILD_FORTRAN=0"
#else
    fortranSwitch="-DHDF5_BUILD_FORTRAN=1"
#fi

(OCSSW_ARCH was set to linux_64, as I would have expected)

and it looks like it's working...