Incorrect File Size for Aqua, Terra and VIIRS SST and Chl files

[oceani]

I imagine this might be related to your Dec. 6th maintenance as the issue started yesterday around midday, but just in case it is an independent problem, all of the subscription data files that we are downloading using wget are coming in at exactly 18,663 bytes. This includes all of our subscriptions for level 2 SST and chlorophyll data from Aqua, Terra and VIIRS. A few examples of what we are seeing for all download attempts looks like:

AQUA_MODIS.20231207T190001.L2.SST.NRT.nc
--2023-12-08 10:36:17-- https://oceandata.sci.gsfc.nasa.gov/getfile/AQUA_MODIS.20231207T190001.L2.SST.NRT.nc
Resolving oceandata.sci.gsfc.nasa.gov... 169.154.128.84
Connecting to oceandata.sci.gsfc.nasa.gov|169.154.128.84|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: “AQUA_MODIS.20231207T190001.L2.SST.NRT.nc”
Connecting to oceandata.sci.gsfc.nasa.gov|169.154.128.84|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: “AQUA_MODIS.20231207T190001.L2.SST.NRT.nc”

[ <=> ] 18,663 --.-K/s in 0.07s

2023-12-08 10:36:18 (260 KB/s) - “AQUA_MODIS.20231207T190001.L2.SST.NRT.nc” saved [18663]


AQUA_MODIS.20231207T203500.L2.SST.NRT.nc
--2023-12-08 10:36:18-- https://oceandata.sci.gsfc.nasa.gov/getfile/AQUA_MODIS.20231207T203500.L2
DOWNLOADING GULF_OF_MEXICO NETCDF SST FILES
AQUA_MODIS.20231207T203500.L2.SST.NRT.nc
--2023-12-08 10:36:18-- https://oceandata.sci.gsfc.nasa.gov/getfile/AQUA_MODIS.20231207T203500.L2.SST.NRT.nc
Resolving oceandata.sci.gsfc.nasa.gov... 169.154.128.84
Connecting to oceandata.sci.gsfc.nasa.gov|169.154.128.84|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: “AQUA_MODIS.20231207T203500.L2.SST.NRT.nc”

[ <=> ] 18,663 --.-K/s in 0.06s

2023-12-08 10:36:18 (289 KB/s) - “AQUA_MODIS.20231207T203500.L2.SST.NRT.nc” saved [18663]

Example subscription numbers would be 1139 and 1993, however this is happening for every file from all of our subscriptions.

Thanks in advance for your assistance,

Mark

[OB ODPS - towens]

Run file on them to see if they are HTML. If so, open with a browser to see what the error message is.

Tommy

[oceani]

They are all indeed HTML files (even though our scripts are requesting NetCDF (.nc) files and they are being saved as such. When I open the files in either Internet Explorer or Chrome those browsers just show a large page of HTML code. If I open using Firefox, I get a "Problem loading page message and then this page displays:

EARTHDATA
Other DAACs▾
OceanColor Home
What We Do Supported Missions Staff Directory
Forums
Ask a Question Search Forum Archive
Aquarius CZCS GOCI HawkEye HICO MERIS MODIS-Aqua MODIS-Terra OCTS OLCI-S3A OLCI-S3B PACE SeaWiFS VIIRS-JPSS1 VIIRS-SNPP
Projects
PRISM-CORAL CyAN Inland Waters
Forums
Ask a Question Search Forum Archive
Data Overview Ocean Data Home
Get Data
Direct Data Access File Search OPeNDAP
How To
Search & Download Cite Data
Browse Data
Level 1&2 Browser
Level 3 Browser
Trending & Analysis
Overpass Predictor
Quality Assessment
Product Validation Global L3 Trends Mission Quality Monitor
Forums
Ask a Question Search Forum Archive
Technical Docs
NASA Technical Reports Users Guides White Papers MODIS Docs (historical) Sensor Spectral Information
Data
Filenaming Convention How to Cite Processing History
Community Engagement
Meetings/Workshops Outreach & Education
Products
Level 1, 2, & 3 Definitions Format Specifications Product Status by Mission Algorithm Descriptions Ancillary Sources
Forums
Ask a Question Search Forum Archive
Visualization
SeaDAS SeaDAS Documentation SeaDAS Informational Videos OCSSW Software OCSSW Documentation
Online Tools
Other Resources
Forums
Ask a Question Search Forum Archive
Sign Up
User Registration Mailing Lists
Online Tools
Overpass Predictor
Subscriptions/Orders
Data Dashboard Request a HawkEye Image
More...
HPLC Pigments
Forums
Ask a Question Search Forum Archive
GALLERY FORUM ☰
The GetFile Application
Invalid appkey/token used.
Responsible NASA Official: Sean Bailey
Curator: OceanColor Webmaster

Web Privacy Policy
Data & Information Policy
Communications Policy
Freedom of Information Act
USA.gov

It should be noted that the wget commands, URLs and scripts that retrieve these data have been in place for many, many years and nothing on our end has changed. I tried uploading a sample file for you, however, even when I changed the .nc extension of the file name to several different variations, your system would not allow it (gave an incorrect extension error).

-Mark

[OB ODPS - towens]

Try putting the URL you are fetching into a fresh browser and see if your username and password are accepted. They are rolling out a 60-day expiration policy on Earthdata logins.

Tommy

[OB WebDev - masoudso]

Hi Mark,
I see a 'Invalid appkey/token used.' in the HTML file you put here. Please make sure the APPKEY or EDL token that you're using is valid.

Thanks,
Masoud

[oceani]

I am not entirely sure what you mean by "Please make sure the APPKEY or EDL token that you're using is valid".

When our script use the wget command: wget --user=<my-username> --password=<my-password> --no-check-certificate --post-data="subID=$2557&results_as_file=1" -O - https://oceandata.sci.gsfc.nasa.gov/api/file_search to get the list of available files to download, it returns the list as expected.

When the script uses the wget command: wget --user=<my-username> --password=<my-password> --auth-no-challenge=on --no-check-certificate https://oceandata.sci.gsfc.nasa.gov/getfile/JPSS1_VIIRS.20231207T171201.L2.OC.NRT.nc (as an example), it downloads the file JPSS1_VIIRS.20231207T171201.L2.OC.NRT.nc, however the file is actually a .html format file with that name with the size of 18,663 bytes.

If I put the URL https://oceandata.sci.gsfc.nasa.gov/getfile/JPSS1_VIIRS.20231207T171201.L2.OC.NRT.nc into a browser, it downloads the NetCDF file fine as JPSS1_VIIRS.20231207T171201.L2.OC.NRT.nc with the correct size of 82,522KB.

I see no errors about invalid username, password, APPKEY or EDL token in response to the wget command on my Linux system nor when I put that URL into the browser address using Chrome or Firefox.

Mark

[OB ODPS - towens]

Thanks for fixing the post. We are looking into it.

Tommy

[jeffg]

FWIW, I'm getting exactly the same thing that Mark is reporting. All the files downloaded are actually the same html page that he posted.

[OB ODPS - towens]

It looks like a workaround is using an appkey for authentication instead of the username, password combination.

You can generate one here: https://oceandata.sci.gsfc.nasa.gov/appkey/

Then add it to your URL:

Code: Select all

https://oceandata.sci.gsfc.nasa.gov/getfile/AQUA_MODIS.20231207T190001.L2.SST.NRT.nc?appkey=xxxxxx

You can use the -O option to wget to specify the output file name.

Tommy

[oceani]

The appkey following the URL seems to work as a one-off within a web browser, but not in my CShell script since I need to use variables for the file names to be retrieved. The ?appkey={appkey} causes the wget line to fail with the variable usage.

The command line without the user name and password info looks like:
wget https://oceandata.sci.gsfc.nasa.gov/getfile/$listfiles[1]?appkey=82277XXXXXXX

The variable $listfiles[1] is part of a 'while' loop that holds each of the many file names for a particular subscription number and it does not like the ?appkey at the end.

Since we download hundreds of files from your server each day, using this work around 'manually' will not be possible.

Any other suggestions on how to incorporate the appkey into the wget command line would be much apprecaited.

Thanks,

Mark