Python script to download VIIRS data doesn't work after Jan 2023

Use this Forum to find information on, or ask a question about, NASA Earth Science data.
tahersima
Posts: 6
Joined: Wed Mar 02, 2022 11:27 am America/New_York
Answers: 0

Python script to download VIIRS data doesn't work after Jan 2023

by tahersima » Wed Jan 25, 2023 3:07 pm America/New_York

Hello,

I have a python code (the second code from https://urs.earthdata.nasa.gov/documentation/for_users/data_access/python) to configure a connection and download data from Earthdata. In 2022 I could download thousands of URLs using that scripts. But starting January 2023 the code hasn't been working for me and I cannot figure out what is wrong.

Once I execute the code, it runs and downloads empty files - they are just 20KB and do not contain imagery data. If I copy the URL in browser, it donwloads the imagery and the files are ~150MB.
An example URL is: "https://ladsweb.modaps.eosdis.nasa.gov/archive/allData/5200/VNP02MOD/2022/365/VNP02MOD.A2022365.2142.002.2023001063801.nc"

Something is wrong and I can't figure out what it is. Any help would be much appreciated.

Tags:

OB ODPS - towens
Subject Matter Expert
Subject Matter Expert
Posts: 368
Joined: Fri Feb 05, 2021 9:17 am America/New_York
Answers: 0
Been thanked: 4 times

Re: Python script to download VIIRS data doesn't work after Jan 2023

by OB ODPS - towens » Thu Jan 26, 2023 6:58 am America/New_York

Run the file command on the short files.
They are likely HTML files. If so, open one in a browser and there will likely be a message about an authentication error when using the script.

Tommy

tahersima
Posts: 6
Joined: Wed Mar 02, 2022 11:27 am America/New_York
Answers: 0

Re: Python script to download VIIRS data doesn't work after Jan 2023

by tahersima » Thu Jan 26, 2023 9:34 am America/New_York

You are absolutely right. Although they have .nc at the end of file name, they are just html files telling me to "Please copy and save the token below for use with any supported LAADS download utilities or scripts."

I know how to generate tokens from NRT, but don't know how to use them in a Python script.
The python examples I've found on earthdata use username/password. Like the ones here:
https://wiki.earthdata.nasa.gov/display/EL/How+To+Access+Data+With+Python

So far, we know this is an authentication issue that is happening to python codes after January 2023.

LAADS_UserServices_M
User Services
User Services
Posts: 279
Joined: Mon Sep 30, 2019 8:33 am America/New_York
Answers: 1
Has thanked: 3 times

Re: Python script to download VIIRS data doesn't work after Jan 2023

by LAADS_UserServices_M » Mon Jan 30, 2023 11:30 am America/New_York

Python script example for LAADS DAAC data download can be found at:
https://ladsweb.modaps.eosdis.nasa.gov/tools-and-services/data-download-scripts/#python
Regards,
LAADS User Services

To receive news from LAADS DAAC direct to your inbox, email laadsdaac-join@lists.nasa.gov with “subscribe” in the subject line.

tahersima
Posts: 6
Joined: Wed Mar 02, 2022 11:27 am America/New_York
Answers: 0

Re: Python script to download VIIRS data doesn't work after Jan 2023

by tahersima » Thu Feb 02, 2023 3:25 pm America/New_York

Thank you for referring me to the python script example for LAADS DAAC data.

I configured the arguments with the URL and a token. But it throws the following error: 'utf-8' codec can't decode byte 0x89 in position 0: invalid start byte.
This happens at this line 'return fh.read().decode('utf-8')'. I have tried several approaches that I found on the Internet, but nothing seems to work for this particular case.

Any ideas how to decode this correctly?
I am running the code in Python 3 and Windows.

LAADS_UserServices_M
User Services
User Services
Posts: 279
Joined: Mon Sep 30, 2019 8:33 am America/New_York
Answers: 1
Has thanked: 3 times

Re: Python script to download VIIRS data doesn't work after Jan 2023

by LAADS_UserServices_M » Fri Mar 03, 2023 8:25 am America/New_York

See if the suggestions made for another user at viewtopic.php?t=3979 resolve your issue.
Regards,
LAADS User Services

To receive news from LAADS DAAC direct to your inbox, email laadsdaac-join@lists.nasa.gov with “subscribe” in the subject line.

tahersima
Posts: 6
Joined: Wed Mar 02, 2022 11:27 am America/New_York
Answers: 0

Re: Python script to download VIIRS data doesn't work after Jan 2023

by tahersima » Fri Mar 03, 2023 8:58 am America/New_York

Unfortunately, it didn't resolve the issue. The error remains exactly the same:

in geturl
return fh.read().decode('utf-8')

UnicodeDecodeError: 'utf-8' codec can't decode byte 0x89 in position 0: invalid start byte

LAADS_UserServices_M
User Services
User Services
Posts: 279
Joined: Mon Sep 30, 2019 8:33 am America/New_York
Answers: 1
Has thanked: 3 times

Re: Python script to download VIIRS data doesn't work after Jan 2023

by LAADS_UserServices_M » Tue Mar 07, 2023 3:14 pm America/New_York

@tahersima, It would help to know what URL you are trying to download. We have not seen that error in all our testing, but you might be downloading different types of files than we have tested. Also it would help to know if you are using Python2 or Python3.
Regards,
LAADS User Services

To receive news from LAADS DAAC direct to your inbox, email laadsdaac-join@lists.nasa.gov with “subscribe” in the subject line.

tahersima
Posts: 6
Joined: Wed Mar 02, 2022 11:27 am America/New_York
Answers: 0

Re: Python script to download VIIRS data doesn't work after Jan 2023

by tahersima » Wed Mar 08, 2023 12:16 pm America/New_York

I also thought it could be because of file type but couldn't figure out what.
Example of the URLs that I am trying to download is:
https://ladsweb.modaps.eosdis.nasa.gov/archive/allData/5200/VNP02MOD/2022/365/VNP02MOD.A2022365.2142.002.2023001063801.nc
The file contains NetCDF4 scenes captured by S-NPP VIIRS.

I am using Python 3 in Windows. I am also using Anaconda for environment management and run my codes in Spyder.

LAADS_UserServices_M
User Services
User Services
Posts: 279
Joined: Mon Sep 30, 2019 8:33 am America/New_York
Answers: 1
Has thanked: 3 times

Re: Python script to download VIIRS data doesn't work after Jan 2023

by LAADS_UserServices_M » Fri Mar 10, 2023 7:37 am America/New_York

We think it is because you are trying to use the script to download a single data file. The script is meant for downloading directory trees by getting a directory listing in either json or csv format first. Since you are asking for a netCDF formatted data file, the script doesn’t know what to do with it or how to parse it for directory information. We can modify the python script to be smarter and handle the single file case, but that will take some time to implement the change and test it. You can download just that single file with cURL, like

curl --fail --header "Authorization: Bearer YOUR_DOWNLOAD_TOKEN" -sS -L -b session --get https://ladsweb.modaps.eosdis.nasa.gov/archive/allData/5200/VNP02MOD/2022/365/VNP02MOD.A2022365.2142.002.2023001063801.nc > VNP02MOD.A2022365.2142.002.2023001063801.nc

or use wget. The LAADSWeb archive pages have examples of how to use wget.
Regards,
LAADS User Services

To receive news from LAADS DAAC direct to your inbox, email laadsdaac-join@lists.nasa.gov with “subscribe” in the subject line.

Post Reply