Problem Files?

oo_processing · by **oo_processing** » Wed May 20, 2020 12:41 pm America/New_York

Is there a problem with these 2 files:

-rw-r--r-- 1 oo_processing oo_processing 275M May 11 06:50 /optics1/scratch/data/roi/l0/2020/132/MOD00.P2020132.0915_1.PDS.bz2
-rw-r--r-- 1 oo_processing oo_processing 305M May 11 10:18 /optics1/scratch/data/roi/l0/2020/132/MOD00.P2020132.0920_1.PDS.bz2
-rw-r--r-- 1 oo_processing oo_processing 165M May 11 11:17 /optics1/scratch/data/roi/l0/2020/132/MOD00.P2020132.0940_1.PDS.bz2
-rw-rw-r-- 1 oo_processing oo_processing 73M May 20 12:15 /optics1/scratch/data/roi/l0/2020/132/MOD00.P2020132.0945_1.PDS.bz2
-rw-r--r-- 1 oo_processing oo_processing 293M May 11 11:19 /optics1/scratch/data/roi/l0/2020/132/MOD00.P2020132.1055_1.PDS.bz2

Every thing that I process that relies on them has failed, with geo-location errors, and did try to download the MOD00.P2020132.0945_1.PDS.bz2 again but its the same file.

STATUS: $ocssw_return_2: MODIS GEO version 6.1.0, built Aug 16 2019 12:46:24
scan: 0 out of 197 Wed May 20 12:01:51 2020
scan: 10 out of 197 Wed May 20 12:01:51 2020
....
scan: 180 out of 197 Wed May 20 12:01:53 2020
scan: 190 out of 197 Wed May 20 12:01:53 2020
Percent valid data (0.00) is less than threshold (95.00)

Thanks,
Brock

oo_processing · by **oo_processing** » Sat May 23, 2020 12:17 pm America/New_York

I am still having no joy with these files. Fails like this:
FILE0 = /optics1/scratch/data/roi/l0/2020/132/MOD00.P2020132.0945_1.PDS.bz2
ORIGINAL_WO = /optics1/scratch/data/roi/wo_backup_h5/2020/132/PRI1.DO.SEADAS_L1A_GEO_EXTRACT_H5.GLAKES.MODISA.MOD00.P2020132.0945_1.wo
ROI = GLAKES
SAT = MODISA
WO_EXISTED = 0
YEAR = 2020
STATUS: $sleep_cmd: sleep 7
STATUS: /optics1/scratch/data/roi/l0/2020/132/MOD00.P2020132.0945_1.PDS.bz2
STATUS: $new_name = MOD00.P2020132.0945_1.PDS
STATUS: $no_underscore_hour_min = 0945
STATUS: $extracted_l1a_file_w_vdir_path = /tmp/S4P_ROI_H5_MODISA.GLAKES.2020.132_version_0.07_S4hORrir/MOD00.P2020132.0945.GLAKES.L1A_LAC
STATUS: $extracted_geo_file_w_vdir_path = /tmp/S4P_ROI_H5_MODISA.GLAKES.2020.132_version_0.07_S4hORrir/MOD00.P2020132.0945.GLAKES.GEO
STATUS: $start_time_initial = MOD00.P2020132.0945.GLAKES_FULL.L1A_LAC
STATUS: $start_time = 0945
STATUS: $start_time = 0945
STATUS: $color_pass_start = 1500
STATUS: $color_pass_end = 2100
STATUS: $colorpass is FALSE
STATUS: $ocssw_command_1 = modis_L1A.py /tmp/S4P_ROI_H5_MODISA.GLAKES.2020.132_version_0.07_S4hORrir/MOD00.P2020132.0945.PDS -o /tmp/S4P_ROI_H5_MODISA.GLAKES.2020.132_version_0.07_S4hORrir/MOD00.P2020132.0945.GLAKES_FULL.L1A_LAC --startnudge=0 --stopnudge=10
$return_bunzip =
STATUS: $ocssw_return_1 = MODIS L1A version 6.0.6, built Aug 16 2019 12:46:26
scan: 0 out of 201 Sat May 23 12:10:41 2020
scan: 10 out of 201 Sat May 23 12:10:41 2020
scan: 20 out of 201 Sat May 23 12:10:41 2020
scan: 30 out of 201 Sat May 23 12:10:41 2020
scan: 40 out of 201 Sat May 23 12:10:41 2020
scan: 50 out of 201 Sat May 23 12:10:41 2020
scan: 60 out of 201 Sat May 23 12:10:41 2020
scan: 70 out of 201 Sat May 23 12:10:42 2020
scan: 80 out of 201 Sat May 23 12:10:42 2020
scan: 90 out of 201 Sat May 23 12:10:42 2020
scan: 100 out of 201 Sat May 23 12:10:42 2020
scan: 110 out of 201 Sat May 23 12:10:42 2020
scan: 120 out of 201 Sat May 23 12:10:42 2020
scan: 130 out of 201 Sat May 23 12:10:42 2020
scan: 140 out of 201 Sat May 23 12:10:42 2020
scan: 150 out of 201 Sat May 23 12:10:42 2020
scan: 160 out of 201 Sat May 23 12:10:42 2020
scan: 170 out of 201 Sat May 23 12:10:42 2020
scan: 180 out of 201 Sat May 23 12:10:42 2020
scan: 190 out of 201 Sat May 23 12:10:42 2020
STATUS: $modis_atteph_cmd = modis_atteph.py -d --verbose /tmp/S4P_ROI_H5_MODISA.GLAKES.2020.132_version_0.07_S4hORrir/MOD00.P2020132.0945.GLAKES_FULL.L1A_LAC
$_: att1=/optics1/software/seadas/seadas-7.5.3/ocssw/var/anc/2020/132/PM1ATTNR.P2020132.0800.003
STATUS: $compression_test_returns existing file: 1
/optics1/software/seadas/seadas-7.5.3/ocssw/var/anc/2020/132/PM1ATTNR.P2020132.0800.003 was not compressed
$_: eph1=/optics1/software/seadas/seadas-7.5.3/ocssw/var/anc/2020/131/PM1EPHND.P2020131.1200.003
STATUS: $compression_test_returns existing file: 1
STATUS: $getanc_cmd = getanc.py -d --verbose /tmp/S4P_ROI_H5_MODISA.GLAKES.2020.132_version_0.07_S4hORrir/MOD00.P2020132.0945.GLAKES_FULL.L1A_LAC
/optics1/software/seadas/seadas-7.5.3/ocssw/var/anc/2020/131/PM1EPHND.P2020131.1200.003 was not compressed
$_: icefile=/optics1/software/seadas/seadas-7.5.3/ocssw/var/anc/2020/131/N202013100_SEAICE_NSIDC_24h.hdf
STATUS: $compression_test_returns existing file: 1
/optics1/software/seadas/seadas-7.5.3/ocssw/var/anc/2020/131/N202013100_SEAICE_NSIDC_24h.hdf was not compressed
$_: met1=/optics1/software/seadas/seadas-7.5.3/ocssw/var/anc/2020/132/N202013206_MET_NCEPR2_6h.hdf
STATUS: $compression_test_returns existing file: 1
/optics1/software/seadas/seadas-7.5.3/ocssw/var/anc/2020/132/N202013206_MET_NCEPR2_6h.hdf was not compressed
$_: met2=/optics1/software/seadas/seadas-7.5.3/ocssw/var/anc/2020/132/N202013212_MET_NCEPR2_6h.hdf
STATUS: $compression_test_returns existing file: 1
/optics1/software/seadas/seadas-7.5.3/ocssw/var/anc/2020/132/N202013212_MET_NCEPR2_6h.hdf was not compressed
$_: met3=/optics1/software/seadas/seadas-7.5.3/ocssw/var/anc/2020/132/N202013212_MET_NCEPR2_6h.hdf
STATUS: $compression_test_returns existing file: 1
/optics1/software/seadas/seadas-7.5.3/ocssw/var/anc/2020/132/N202013212_MET_NCEPR2_6h.hdf was not compressed
$_: ozone1=/optics1/software/seadas/seadas-7.5.3/ocssw/var/anc/2020/131/N202013100_O3_AURAOMI_24h.hdf
STATUS: $compression_test_returns existing file: 1
/optics1/software/seadas/seadas-7.5.3/ocssw/var/anc/2020/131/N202013100_O3_AURAOMI_24h.hdf was not compressed
$_: ozone2=/optics1/software/seadas/seadas-7.5.3/ocssw/var/anc/2020/132/N202013200_O3_AURAOMI_24h.hdf
STATUS: $compression_test_returns existing file: 1
/optics1/software/seadas/seadas-7.5.3/ocssw/var/anc/2020/132/N202013200_O3_AURAOMI_24h.hdf was not compressed
$_: ozone3=/optics1/software/seadas/seadas-7.5.3/ocssw/var/anc/2020/132/N202013200_O3_AURAOMI_24h.hdf
STATUS: $compression_test_returns existing file: 1
/optics1/software/seadas/seadas-7.5.3/ocssw/var/anc/2020/132/N202013200_O3_AURAOMI_24h.hdf was not compressed
$_: sstfile=/optics1/software/seadas/seadas-7.5.3/ocssw/var/anc/2020/131/N2020131_SST_OIV2AV_24h.nc
STATUS: $compression_test_returns existing file: 1
STATUS: $l1a_file_w_vdir_path /tmp/S4P_ROI_H5_MODISA.GLAKES.2020.132_version_0.07_S4hORrir/MOD00.P2020132.0945.GLAKES_FULL.L1A_LAC was found
STATUS: $current_directory = /tmp/S4P_ROI_H5_MODISA.GLAKES.2020.132_version_0.07_S4hORrir
STATUS: $ocssw_command_2: modis_GEO.py /tmp/S4P_ROI_H5_MODISA.GLAKES.2020.132_version_0.07_S4hORrir/MOD00.P2020132.0945.GLAKES_FULL.L1A_LAC -o /tmp/S4P_ROI_H5_MODISA.GLAKES.2020.132_version_0.07_S4hORrir/MOD00.P2020132.0945.GLAKES_FULL.GEO --threshold=95 --ancdb=./ancillary_data.db
/optics1/software/seadas/seadas-7.5.3/ocssw/var/anc/2020/131/N2020131_SST_OIV2AV_24h.nc was not compressed
STATUS: $ocssw_return_2: MODIS GEO version 6.1.0, built Aug 16 2019 12:46:24
scan: 0 out of 197 Sat May 23 12:10:43 2020
scan: 10 out of 197 Sat May 23 12:10:43 2020
scan: 20 out of 197 Sat May 23 12:10:44 2020
scan: 30 out of 197 Sat May 23 12:10:44 2020
scan: 40 out of 197 Sat May 23 12:10:44 2020
scan: 50 out of 197 Sat May 23 12:10:44 2020
scan: 60 out of 197 Sat May 23 12:10:44 2020
scan: 70 out of 197 Sat May 23 12:10:44 2020
scan: 80 out of 197 Sat May 23 12:10:44 2020
scan: 90 out of 197 Sat May 23 12:10:45 2020
scan: 100 out of 197 Sat May 23 12:10:45 2020
scan: 110 out of 197 Sat May 23 12:10:45 2020
scan: 120 out of 197 Sat May 23 12:10:45 2020
scan: 130 out of 197 Sat May 23 12:10:45 2020
scan: 140 out of 197 Sat May 23 12:10:45 2020
scan: 150 out of 197 Sat May 23 12:10:45 2020
scan: 160 out of 197 Sat May 23 12:10:46 2020
scan: 170 out of 197 Sat May 23 12:10:46 2020
scan: 180 out of 197 Sat May 23 12:10:46 2020
scan: 190 out of 197 Sat May 23 12:10:46 2020
Percent valid data (0.00) is less than threshold (95.00)
ERROR: MODIS geolocation processing failed.
ERROR: The command: modis_GEO.py /tmp/S4P_ROI_H5_MODISA.GLAKES.2020.132_version_0.07_S4hORrir/MOD00.P2020132.0945.GLAKES_FULL.L1A_LAC -o /tmp/S4P_ROI_H5_MODISA.GLAKES.2020.132_version_0.07_S4hORrir/MOD00.P2020132.0945.GLAKES_FULL.GEO --threshold=95 --ancdb=./ancillary_data.db
returned an error code 'ERROR:'

gfireman · by **gfireman** » Tue May 26, 2020 4:54 pm America/New_York

Brock -

The "refined" attitude file PM1ATTNR.P2020132.0800.003 seems to be corrupted on the provider's end. The granule in question processes geolocation successfully with the NRT attitude file PM1ATTNR_NRT.A2020132.0940.061. Unless and until the refined file is corrected, you should use the NRT for that granule (and a few others following).

- Gwyn

oo_processing · by **oo_processing** » Tue May 26, 2020 5:30 pm America/New_York

Gwyn,

If the file is expected to be processed, I'll wait.
It has already been processed by the PM1ATTNR_NRT.
Do you know it they will correct the error?

Thanks,
Brock

gfireman · by **gfireman** » Tue May 26, 2020 5:34 pm America/New_York

I hope so. IIUC, our production staff will let the atteph production people know that there's a problem, and they'll take it from there.

oo_processing · by **oo_processing** » Thu May 28, 2020 10:19 pm America/New_York

Still nothing yet I guess:

seahorse 109 :/optics1/home1/oo_processing> ll -ltrh /optics1/software/seadas/anc/2020/132/PM1ATTNR.P2020132.0800.003.old /optics1/software/seadas/anc/2020/132/PM1ATTNR.P2020132.0800.003
-rw-rw-rw- 1 oo_processing oo_processing 452K May 11 15:10 /optics1/software/seadas/anc/2020/132/PM1ATTNR.P2020132.0800.003.old
-rw-rw-r-- 1 oo_processing oo_processing 452K May 28 22:17 /optics1/software/seadas/anc/2020/132/PM1ATTNR.P2020132.0800.003
seahorse 110 :/optics1/home1/oo_processing> diff /optics1/software/seadas/anc/2020/132/PM1ATTNR.P2020132.0800.003.old /optics1/software/seadas/anc/2020/132/PM1ATTNR.P2020132.0800.003
seahorse 111 :/optics1/home1/oo_processing>

oo_processing · by **oo_processing** » Tue Jun 02, 2020 11:52 am America/New_York

Still no change:

ll /optics1/software/seadas/anc/2020/132/PM1ATTNR.P2020132.0800.003* -
rw-rw-rw- 1 oo_processing oo_processing 462848 Jun 2 11:49 /optics1/software/seadas/anc/2020/132/PM1ATTNR.P2020132.0800.003
-rw-rw-rw- 1 oo_processing oo_processing 462848 May 11 15:10 /optics1/software/seadas/anc/2020/132/PM1ATTNR.P2020132.0800.003.old
searobin 230 :> diff /optics1/software/seadas/anc/2020/132/PM1ATTNR.P2020132.0800.003 /optics1/software/seadas/anc/2020/132/PM1ATTNR.P2020132.0800.003.old
searobin 231 :>