Error initializing l2gen and l3gen (SEVERE: productInfo.xml not found!)

[augustine]

Hello gnwiii,

thank you for the response. I have re-posted this as a new topic.

I have below some information about my configuration files.

If there is something else I need to provide, please let me know.

I am new to Linux and learning on the go.

1) I am using the original WSL

2) Ubuntu 16.04 LTS

3) Windows 10 Education, Version 1903, OS build 18362.418

4) java version "1.8.0_221"
   Java(TM) SE Runtime Environment (build 1.8.0_221-b11)
   Java HotSpot(TM) 64-Bit Server VM (build 25.221-b11, mixed mode)

5) ocsswserver config
######################
ocsswrest.version=1.0
baseUriPortNumber=6400
ocsswroot=/home/ladoga/ocssw
serverWorkingDirectory=/mnt/c/Users/gbagir/seadasClientServerShared
clientServerSharedDir=true
keepIntermediateFilesOnServer=false
processInputStreamPortNumber=6402
processErrorStreamPortNumber=6403
#########################

6) seadas.config
#########################

# set location of OCSSWROOT
seadas.ocssw.root =/home/ladoga/ocssw
#seadas.ocssw.location = local

# settings used for the remote OCSSW server
#
# the seadas.ocssw.location property should be assigned one of these three values: "local", "virtualMachine",
# or the remote server address.
seadas.ocssw.location = virtualMachine
seadas.ocssw.port=6400
seadas.ocssw.sharedDir=${user.home}/seadasClientServerShared
seadas.client.id=${user.name}
seadas.ocssw.keepFilesOnServer=false
seadas.ocssw.processInputStreamPort=6402
seadas.ocssw.processErrorStreamPort=6403

###############################################################
7) running ls -l /mnt/c/Users/gbagir/seadasClientServerShared/productInfo.xml
#####################################
-rwxrwxrwx 1 ladoga ladoga 157077 Oct 31 15:42 /mnt/c/Users/gbagir/seadasClientServerShared/productInfo.xml
###################################
8) Running l2gen and l3gen from GUI
##################################
#output l2gen
server ocssw root path: /home/ladoga/ladoga/ocssw/scripts
Debugging:  programName = l2gen
Debugging:  element = prodxmlfile_OFILE
Debugging: command array element = prodxmlfile=/mnt/c/Users/gbagir/seadasClientServerShared/productInfo.xml
Debugging: commandArrayPrefix: ladoga/ocssw/scripts/ocssw_runner
Debugging: commandArrayPrefix: --ocsswroot
Debugging: commandArrayPrefix: ladoga/ocssw
Debugging: command array content: ladoga/ocssw/scripts/ocssw_runner --ocsswroot ladoga/ocssw l2gen prodxmlfile=/mnt/c/Users/gbagir/seadasClientServerShared/productInfo.xml
Debugging: process exited! exit value = 0
Debugging: java.lang.UNIXProcess$ProcessPipeInputStream@34899cb3
process status: 0

#output l3gen
thesame as l2gen above.

9) On my local pc there are two paths to SeaDAS (see figure, seadas.path). I do not understand why the installation created these.

Thanks for the support

Augustine
attachment 1

attachment 2

attachment 3

[gnwiii]

You should be able to cut and paste text from the WSL terminal, which would make it easier to read your post (particularly as the images aren't available while writing a reply).    You should edit item 4) to mention the java version used for the server (as it may be helpful to others in the future). 

Item 5) the server config looks OK

Item 6) the client seadas.config the line seadas.ocssw.root =/home/ladoga/ocssw should not be needed -- try adding a
"#" at the front of the line. 

Item 7) it appears that the first step in l2gen initialization worked.  The xml file contains the information needed to populate the
l2gen menu, so the question is why the GUI can't find the file.  The size is the same as mine.  It should be ASCII text, so you can view it in an editor to check that the contents are accessible.  The first few lines should be:
<productList>
    <product name="bbp">
        <algorithm name="mgiop">
            <prefix>bbp_</prefix>
            <suffix>_mgiop</suffix>
            <parameterType>All</parameterType>
            <dataType>INT16</dataType>
            <units>m^-1</units>
            <description>Particulate backscattering at 0 nm, mGIOP algorithm</description>


Assuming the file is readable in Windows, this points to a problem with your seadas.config.   Try replacing the "${..}" constructs with the actual value.

Item 8) indicates that the l2gen task was successful on the server, so points to a problem with the client finding the result

Item 9) I don't have those multiple directories.  They might be a symptom of a configuration glitch.

[augustine]

Hello gnwiii,

thanks for the response.

The java version I used on the server is:

"java version "1.8.0_221"
   Java(TM) SE Runtime Environment (build 1.8.0_221-b11)
   Java HotSpot(TM) 64-Bit Server VM (build 25.221-b11, mixed mode)"

I have tried your suggestions but I still have thesame error.

1) Here are the first few lines contained in the xml file (thesame as yours) in my seadasClientShared folder:

<productList>
    <product name="bbp">
        <algorithm name="mgiop">
            <prefix>bbp_</prefix>
            <suffix>_mgiop</suffix>
            <parameterType>All</parameterType>
            <dataType>INT16</dataType>
            <units>m^-1</units>
            <description>Particulate backscattering at 0 nm, mGIOP algorithm</description>

2) Here is how my seadas.config file looks like now:

# set location of OCSSWROOT
#seadas.ocssw.root =home/ladoga/ocssw
#seadas.ocssw.location = local

# settings used for the remote OCSSW server
#
# the seadas.ocssw.location property should be assigned one of these three values: "local", "virtualMachine",
# or the remote server address.
seadas.ocssw.location = virtualMachine
seadas.ocssw.port=6400
seadas.ocssw.sharedDir=C:/Users/gbagir/seadasClientShared
seadas.client.id=gbagir
seadas.ocssw.keepFilesOnServer=false
seadas.ocssw.processInputStreamPort=6402
seadas.ocssw.processErrorStreamPort=6403

3) The location of my files are as follows:

C:/Users/gbagir/seadas-7.5.3/config/seadas.config
C:/Users/gbagir/seadasClientServerShared/productInfo.xml
ocsswroot=/home/ladoga/ocssw

4) Starting the OCSSW server

I have tested starting the server in different locations to see if the problem is resolved but not successful (server starts and stops correctly).
a) from linux home
 
b) from the shared folder
    /mnt/c/Users/gbagir/seadasClientServerShared/

5) I also installed seadas on Ubuntu 18.04 using two different java versions on the linux server but I still have thesame error

SEVERE: productInfo.xml not found!

  a) :~$ java -version

java version "1.8.0_221"
Java(TM) SE Runtime Environment (build 1.8.0_221-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.221-b11, mixed mode)

b) ~$ java -version

java version "1.8.0_231"
Java(TM) SE Runtime Environment (build 1.8.0_231-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.231-b11, mixed mode)

I appreciate your efforts to help me resolve this issue.

Best regards

Augustine

[gnwiii]

Try using double backslashes as in this post, e.g., replace

seadas.ocssw.sharedDir=C:/Users/gbagir/seadasClientShared
with
seadas.ocssw.sharedDir=C:\\Users\\gbagir\\seadasClientServerShared

[augustine]

Hello gnwiii,

the error still persists. I modified seadas.config as shown in (A).
In addition, also installed seadas on a VM on cPouta cloud service but it did not work either.
In this case the start the server on the VM before the GUI but I get the error that server is down.
I installed everything exactly thesame as I did on WSL using Ubuntu 16.04 LTS.
On cPouta, the user name for Ubuntu 16.04  is cloud-user
The modified ocsswser.config and seadas.config files are shown in (B).

(A)
# set location of OCSSWROOT
#seadas.ocssw.root =home/ladoga/ocssw
#seadas.ocssw.location = local

# settings used for the remote OCSSW server
#
# the seadas.ocssw.location property should be assigned one of these three values: "local", "virtualMachine",
# or the remote server address.
seadas.ocssw.location = virtualMachine
seadas.ocssw.port=6400
seadas.ocssw.sharedDir=C:\Users\gbagir\seadasClientShared
seadas.client.id=gbagir
seadas.ocssw.keepFilesOnServer=false
seadas.ocssw.processInputStreamPort=6402
seadas.ocssw.processErrorStreamPort=6403

(B)

#seadas.config

seadas.ocssw.location = xx.xxx.xx.xx
seadas.ocssw.port=6400
seadas.client.id=gbagir
seadas.ocssw.keepFilesOnServer=false
seadas.ocssw.processInputStreamPort=6402
seadas.ocssw.processErrorStreamPort=6403

#ocsswserver.config
ocsswrest.version=1.0
baseUriPortNumber=6400
ocsswroot=/home/cloud-user/ocssw
serverWorkingDirectory=/home/cloud-user/seadas/seadasClientServerShared
clientServerSharedDir=false
keepIntermediateFilesOnServer=false
processInputStreamPortNumber=6402
processErrorStreamPortNumber=6403

[gnwiii]

Can you run the GUI on the cloud service?  That would allow you to bypass the server.  Using the SeaDAS OCSSW server on a cloud service may require first getting an internet IP assigned to your VM and then setting up an ssh tunnel for the ports used by the server (6400, 6402, 6403), so adds layers of complexity.    With new installation, how did you determine that the server is working?   You can check that the URL shown in the last few lines of the server output, e.g., INFO: [HttpServer] Started.Jersey new app started with WADL available at ://xx.xxx.xx.xx:6400/ocsswws/application.wadlPress 'Ctrl' + 'C'  to stop it...provides data:$ wget ://xx.xxx.xx.xx:6400/ocsswws/application.wadl--2019-11-08 15:20:16--  ://xx.xxx.xx.xx:6400/ocsswws/application.wadlConnecting to xx.xxx.xx.xx:6400... connected.HTTP request sent, awaiting response... 200 OKLength: unspecified [application/vnd.sun.wadl+xml]Saving to: ‘application.wadl’application.wadl        [ <=>                ]  24.40K  --.-KB/s    in 0sLast-modified header invalid -- time-stamp ignored.2019-11-08 15:20:16 (180 MB/s) - ‘application.wadl’ saved [24983]Then run:$ head application.wadl<?xml version="1.0" encoding="UTF-8" standalone="yes"?><application xmlns="http://wadl.dev.java.net [/2009/02]">    <doc xmlns:jersey="http://jersey.java.net/" jersey:generatedBy="Jersey: 2.26 2017-09-05 11:50:34"/>    <doc xmlns:jersey="http://jersey.java.net/" jersey:hint="This is simplified WADL with user and core resources only. To get full WADL with extended resources use the query parameter detail. Link: ://xx.xxx.xx.xx:6400/ocsswws/application.wadl?detail=true"/>    <grammars>        <include href="application.wadl/xsd0.xsd">            <doc title="Generated" xml:lang="en"/>        </include>    </grammars>    <resources base="://xx.xxx.xx.xx:6400/ocsswws/">If that works, try the same download on your Windows system.  Some browsers may refuse to connect, so you may have to install wget or a simple text mode browser like links or lynx.

[augustine]

Hello gnwiii,

I have tried running the GUI on cloud service but does not work.

I modified ssh tunnels for my VM and now I can start the server and OCSSW processors are now available.

The l2gen and l3gen now works.

However, I now have these issues:

(i) When I try to update processors via the GUI, the interface to select and update processors does not open.
   I can only update via command line.

(ii) I tested the l2gen using Landsat 8 . I imput the Metafiles and selct run but I recieve the error:

execution exception: java.io.IOException: l2gen failed with exit code -1.
Check log for more details.

With the additional information displayed in the bash shell:

2gen ifile=/mnt/c/Users/gbagir/seadasClientServerShared/gbagir/LC08_L1TP_181017_                   20140606_20170422_01_T1_MTL.txt ofile=/mnt/c/Users/gbagir/seadasClientServerShar                   ed/gbagir/LC08_L1TP_181017_20140606_20170422_01_T1_MTL_TEST.txt Debugging:
command array content ended
process status on server = -100
Debugging: process is alive:
server process observer: 6402
server process observer: 6403
process status: -1
process status: -1
process status: -1

(iii)  I ran the following code on my cloud VM

wget http://xx.xxx.xx.xx:6400/ocsswws/application.wadl

but gives the error below even though my server:

Connecting to xx.xxx.xx.xx:6400... failed: Connection refused.

Thanks for your help.

Augustine

[gnwiii]

Running a GUI on a cloud service: some services are "headless", but many do support some form of GUI access, but a) it may be more expensive, and b) you have to configure the recommended client and ssh tunnels.

I do most of my work with the command line.  I tend to use the GUI for constructing an example command-line that I can then edit for use in a shell script on the remote server. 

I'm glad you are making progress, and not surprised that some operations give problems.   This a very complex software and there have been many changes so there are bound to be glitches waiting to be discovered by early adopters.   We don't have to buy the OCSSW software, but in return we have to expect rough edges.

(i) I don't normally update from the GUI, but just tried it without success.

(ii) I often get a failure running l2gen.   In some cases, the results have appeared in the Windows directory, and in others there was a glitch in the processing which can be fixed after trying to run the command-line via a terminal on the server system.   Your command line:

l2gen ifile=/mnt/c/Users/gbagir/seadasClientServerShared/gbagir/LC08_L1TP_181017_20140606_20170422_01_T1_MTL.txt ofile=/mnt/c/Users/gbagir/seadasClientServerShared/gbagir/LC08_L1TP_181017_20140606_20170422_01_T1_MTL_TEST.txt
seems to be for a server running on WSL.   I see paths to the data directory on my remote server.    

(iii) The wget failure indicates the server isn't running.   I normally run the server in the background with "nohup java  ... &" so I can continue to use the command line while the OCSSW server is running, in particular, when something fails I can check that it is still working using wget.  

Common mistakes include running wget in the wrong terminal  or entering <ctrl>-Z to suspend a process and forgetting to use bg to keep it running in background.  If you are using linux on a cloud service you will find it helpful to read up on linux job control.

[augustine]

Hello gnwiii,

I have not been able to resolve this issue, so I have installed a linux OS (Ubuntu 18.04 LTS) on my pc.
I will come to this issue a bit later and if I find a solution I will post it.

With the installed linux OS, the errors I have are a bit different. I will make a new post regarding that and hopefully someone will help me out.

Thanks

Augustine

[gnwiii]

Many users have years of experience running the OCSSW software in linux, so you have a much better chance that someone will be able to help.   I used to teach practical sessions in ocean remote sensing workshops.  We found it worth dedicating a full half-day to introductory linux material.  This was useful for the remote sensing workshop and we found that the participants were better prepared for topics like data analysis and numerical modelling.

A good place to start is LinuxCommand.org for an introduction to the linux shell (bash command line).   Since are using Ubuntu, you can install the official Ubuntu package for the Debian Reference Manual (available in many translations for those whose native language is not English).

There are good distro-specific forums that can help with issues such as getting linux working with your particular hardware, but watch out for web sites that get high Google rankings but are only designed to capture advertising clicks. They typically claim to solve one of the most common linux issues, but don't support user feedback that would explain flaws in the advice they provide.