python calls shell to l2gen error: l2gen not found. How to set env in python script?

[lwk1542]

the l2gen can run in terminal. but dont in python. the error is:sh: 1: l2gen: not found.
according to google searching result, the environment variable need to be set again in python script. the method is:
     new_env=os.env.copy()
     new_env['OCSSWROOT'] = '/usr/local/seadas-7.5.3/ocssw'
     returnCode = subprocess.call(l2genshell , env=new_env,shell=True)
there is a problem that environment variables in OCSSW_bash.env can not be add to new_env by using SOURCE command line.
But there is few question/answer to demonstrate how to parse the xxx.env file (OCSSW_bash.env) in python.
How should i add this environment variable in python script? THANKS

[gnwiii]

The OCSSW software is commonly used with bash shell scripts -- using python means few OCSSW users will have relevant experience.  It would be helpful if you could tell us what you are trying to accomplish using python in case there is a way to do the same task with bash scripts. 

If you can't implement your processing tasks using bash scripts, the $OCSSWROOT/scripts/ocssw_runner shell script addresses your problems setting the environment for OCSSW programs from python.  This script sets OCSSWROOT, sources $OCSSWROOT/OCSSW_bash.env, and then runs the supplied OCSSW program and arguments.   You might find plumbum helpful.

[lwk1542]

sorry for my unclear question!
the detail is:
   my os and software environment:  VMware workstation Pro 14, ubuntu16.04 X64, seadas7.5.3, anaconda/python3.7.3, pycharm education 2019
   my scheme is that using “l2gen” to process MODIS L1B data (MOD021KM.**hdf) by python script.
   (1) The following coding can work in bash shell terminal: 
==========================the command lines=========================================
l2gen ifile=/mnt/hgfs/share/test/MOD021KM.A2009033.0810.061.2017295060325.hdf geofile=/mnt/hgfs/share/test/MOD03.A2009033.0810.061.2017294135803.hdf ofile=/mnt/hgfs/share/test/MOD021KM.A2009033.0810.061._seadas.hdf
===============================================================================
     the perfect result file "ofile" is available.

    (2) But similar coding cannot work in python script?
==========================the python script: MODIS-l2gen====================================
import subprocess,os,glob
if __name__=='__main__':
    ifiles=glob.glob('/mnt/hgfs/share/test/MOD021KM*.hdf')
    with open('/mnt/hgfs/share/test/processing_log.txt','w') as f:
        for i,ifile in enumerate(ifiles):
            dir=os.path.dirname(ifile)
            name=os.path.basename(ifile)
            ID=name[9:27]
            #print(dir+'/'+'MOD03.'+ID+'*hdf')
            geofile=glob.glob(dir+'/'+'MOD03.'+ID+'*hdf')[0]

            ofile=dir+'/'+name[0:27]+'_seadas.hdf'
            maskland = 1
            cloudland = 1
            slat=17
            slon=117
            elat=19
            elon=118
            l2prod = 'La_748 La_859 La_869 La_vvv Rrs_vvv angstrom aot_nnn brdf brdf_748 brdf_859 brdf_869\
                     brdf_vvv chlor_a'
            l2gencmd1='l2gen'+' ifile='+ ifile +' geofile='+ geofile +' ofile='+ ofile +' slat='+ \
                     str(slat)+' slon='+str(slon)+' elat='+str(elat)+' elon='+str(elon)+\
                     ' l2prod='+str(l2prod)
            l2gencmd2 = 'l2gen' + ' ifile=' + ifile + ' geofile=' + geofile + ' ofile=' + ofile
            print(l2gencmd2)
            l2cmdtest = 'l2gen -h'
            try:
                #new_env = os.environ.copy()
                #new_env['OCSSWROOT'] = '/usr/local/seadas-7.5.3/ocssw'
                returnCodetest = subprocess.call(l2cmdtest, shell=True)
                returnCode2 = subprocess.call(l2gencmd2, shell=True)
                returnCode1 = subprocess.call(l2gencmd1 ,shell=True)
            except:
                f.write(ifile+'\n')
===========================================================================
the error:
/bin/sh: 1: l2gen: not found

  (3) the question:
     I don't know why the python script did not can work. I have set environment according https://seadas.gsfc.nasa.gov/downloads/

[zhigang]

Try to use source ~/.bashrc and test the l2gen in terminal, if it is okay, python should work as well.

[gnwiii]

The handling of linux environment variables is not simple, and differs between different distros and has changed over time in some distros.  You will find it very worthwhile to spend a few hours learning about Ubuntu linux environment variables.  Note that "Python environments" are not the same thing, and IDE's like pyCharm generally need extra steps to use environment variable settings.

Did you put the OCSSW environment variable settings in your ~/.bashrc?   After making changes to ~/.bashrc it is best to log out and back to ensure all the running programs have consistent environment variable settings.  Even with the settings in your ~/.bashrc they may not be available when Python is started outside a terminal (e.g., using an IDE like pyCharm).  In your python IDE try:

import os
print(os.environ['OCSSWROOT'])
print(os.environ['PATH'])


If these aren't correct then you could try the ocssw_runner script to run your processing commands.

[lwk1542]

thanks, Prof.:
     I have solved this problem. It is an environment variable setting problem in python script. Because the "source $OCSSWROOT/OCSSW_bash.env" is a temporary setting in my Linux OS (i do not know the reason). Therefore, I have to set this environment variable group in the python script. the following code is to solve this problem.
=================================python script=====================================================           
                import subprocess
                commands = 'source $OCSSWROOT/OCSSW_bash.env;l2gen -h'
                process = subprocess.Popen(commands,executable='/bin/bash', stdin=subprocess.PIPE, stdout=subprocess.PIPE,shell=True)
                out, err = process.communicate()
==============================================================================================
    where: commands includes two command lines: 'source $OCSSWROOT/OCSSW_bash.env' and 'l2gen -h'. Note that executable is '/bin/bash'.
Thank you very much!
Thank Seadas developing group and ocean color community!

[lwk1542]

thank you very much! I have solved this problem. The detail was replied to gnwiii?