modis_GEO taking an inordinately long time

bruce · by **bruce** » Fri Jan 17, 2020 12:48 pm America/New_York

I've made the changes to getanc.py and modis_atteph.py suggested elsewhere in the forum and they seem to work, but the actual modis_geogen call is taking FOREVER on a relatively lightly loaded system (ubuntu 16.04LTS, running geogen and top along with other baseline stuff.

Here's some sample output.

MODIS GEO version 6.1.0, built Aug 16 2019 12:46:24
scan: 0 out of 203 Fri Jan 17 10:07:42 2020
scan: 10 out of 203 Fri Jan 17 10:14:44 2020
scan: 20 out of 203 Fri Jan 17 10:22:26 2020
scan: 30 out of 203 Fri Jan 17 10:30:07 2020
scan: 40 out of 203 Fri Jan 17 10:37:49 2020
scan: 50 out of 203 Fri Jan 17 10:45:31 2020
scan: 60 out of 203 Fri Jan 17 10:53:14 2020
scan: 70 out of 203 Fri Jan 17 11:00:57 2020
scan: 80 out of 203 Fri Jan 17 11:08:57 2020
scan: 90 out of 203 Fri Jan 17 11:16:52 2020
scan: 100 out of 203 Fri Jan 17 11:24:43 2020
scan: 110 out of 203 Fri Jan 17 11:32:30 2020
scan: 120 out of 203 Fri Jan 17 11:40:11 2020
scan: 130 out of 203 Fri Jan 17 11:47:52 2020
scan: 140 out of 203 Fri Jan 17 11:55:33 2020
scan: 150 out of 203 Fri Jan 17 12:03:14 2020
scan: 160 out of 203 Fri Jan 17 12:10:58 2020
scan: 170 out of 203 Fri Jan 17 12:18:38 2020
scan: 180 out of 203 Fri Jan 17 12:26:18 2020
scan: 190 out of 203 Fri Jan 17 12:33:58 2020

more than 2 hours to process 190 scans??? Really??? Any idea why it's taking so long?

I've attached LogReport and LogUser but not LogStatus since it's ~440MB long. It has lots of

/home/seadas/ocssw/src/geogen_modis/GEO_check_ea_headers.c, GEO_check_ea_headers():MODIS_E_GEO:288779784
Fri Jan 17 10:07:42 2020
Error returned by function PGS_EPH_getEphemHeaders()

PGS_IO_Gen_Open():PGS_E_UNIX:17943
No such file or directory

PGS_IO_Gen_Open():PGS_E_UNIX:3848
No entry for file with LID 10502. UNIX error

PGS_EPH_getAttitHeaders():PGSEPH_E_NO_SC_EPHEM_FILE:44547
unable to open spacecraft attitude file

PGS_IO_Gen_Open():PGS_E_UNIX:17943
No such file or directory

PGS_IO_Gen_Open():PGS_E_UNIX:3848
No entry for file with LID 10502. UNIX error

PGS_EPH_getAttitHeaders():PGSEPH_E_NO_SC_EPHEM_FILE:44547
unable to open spacecraft attitude file

PGS_IO_Gen_Open():PGSIO_E_GEN_FILE_NOEXIST:11779
No entry for file with LID 10502. File does not exist, or cannot be created.

PGS_EPH_getAttitHeaders():PGSEPH_E_NO_SC_EPHEM_FILE:44547
unable to open spacecraft attitude file

PGS_EPH_getAttitHeaders():PGSEPH_E_NO_SC_EPHEM_FILE:44547
No valid attitude files found.

/home/seadas/ocssw/src/geogen_modis/GEO_check_ea_headers.c, GEO_check_ea_headers():MODIS_E_GEO:288779784
Fri Jan 17 10:07:42 2020
Error returned by function PGS_EPH_getAttitHeaders()

/home/seadas/ocssw/src/geogen_modis/GEO_ephem_attit.c, GEO_interp_ephemeris_attitude():MODIS_E_GEO:288779784
Fri Jan 17 10:07:42 2020
Error returned by function GEO_check_ea_headers(2020-01-12T16:45:00.674647Z)

PGS_PC_GetReference():PGSPC_W_NO_REFERENCE_FOUND:76809
No reference was found matching Product ID (10501) and Version number (3).

PGS_EPH_EphemAttit():PGSPC_W_NO_REFERENCE_FOUND:76809
Error getting reference for eph files

/home/seadas/ocssw/src/geogen_modis/GEO_ephem_attit.c, GEO_interp_ephemeris_attitude():MODIS_E_GEO:288779784
Fri Jan 17 10:07:42 2020
Error returned by function PGS_EPH_EphemAttit(2020-01-12T16:45:00.674647Z)

/home/seadas/ocssw/src/geogen_modis/GEO_interp_ECR.c, GEO_interp_ECR():MODIS_E_GEO:288779784
Fri Jan 17 10:07:42 2020
Error returned by function GEO_interp_ephemeris_attitude(2020-01-12T16:45:00.674647Z)

/home/seadas/ocssw/src/geogen_modis/GEO_locate_one_scan.c, GEO_locate_one_scan():MODIS_E_GEO:288779784
Fri Jan 17 10:07:42 2020
Error returned by function GEO_interp_ECR(853001110.674647, 2709)

/home/seadas/ocssw/src/geogen_modis/GEO_ephem_attit.c, GEO_interp_ephemeris_attitude():MODIS_E_GEO:288779784
Fri Jan 17 10:07:42 2020
Error returned by function GEO_check_ea_headers(2020-01-12T16:45:00.674647Z)

PGS_IO_Gen_Open():PGS_E_UNIX:17943
No such file or directory

PGS_IO_Gen_Open():PGS_E_UNIX:3848
No entry for file with LID 10501. UNIX error

PGS_EPH_getEphemHeaders():PGSEPH_E_NO_SC_EPHEM_FILE:44547
unable to open spacecraft ephemeris file

PGS_IO_Gen_Open():PGS_E_UNIX:17943
No such file or directory

PGS_IO_Gen_Open():PGS_E_UNIX:3848
No entry for file with LID 10501. UNIX error

PGS_EPH_getEphemHeaders():PGSEPH_E_NO_SC_EPHEM_FILE:44547
unable to open spacecraft ephemeris file

PGS_IO_Gen_Open():PGSIO_E_GEN_FILE_NOEXIST:11779
No entry for file with LID 10501. File does not exist, or cannot be created.
attachment 1

attachment 2

by **OB.DAAC - SeanBailey** » Fri Jan 17, 2020 1:59 pm America/New_York

It's taking so long because it was writing a 440MB error log. The inputs are bad. Specifically (or at least) the ephemeris file.

Sean

gnwiii · by **gnwiii** » Fri Jan 17, 2020 5:32 pm America/New_York

It might be useful to know that failed downloads often result in a much smaller file with the name of the desired download file but containing html, which might be a login page (or, for people behind enterprise firewalls, a page directing you to contact IT to explain why you need to connect to the EOSDIS server). The "file" command should tell you if your download is html. If your download is a bzip2 compressed file you can check that the download is not corrupt by running "bzip2 -t <filename>". Finally, the file search page can provide a "checksum text file" that can be used to verify that a download file is not corrupt (or has not been replaced by some html document).