Slow directory listings

Use this Forum to find information on, or ask a question about, NASA Earth Science data.
Post Reply
satellitedataservices
Posts: 1
Joined: Fri Mar 08, 2024 2:45 pm America/New_York
Answers: 0

Slow directory listings

by satellitedataservices » Fri Mar 08, 2024 2:54 pm America/New_York

We are downloading modis terra and aqua data. We are using python.
URL: https://nrt4.modaps.eosdis.nasa.gov:/api/v2/content/archives/allData/61/MOD021KM/2024/068
Initially we get a listing for today's and yesterday's data. This requires 4 request 2 for aqua and 2 for terra. The directory listings take rather long time.

We compare the output of the list to files we currently have locally and download if necessary. Is there a more efficient way. I was hoping there might be a single file detailing the directory contents. The actual data downloads on the order of seconds. Code snipit
for sat in ['terra', 'aqua']:

if 'terra' in sat:
remote_dir = 'api/v2/content/archives/allData/61/MOD021KM'
else:
remote_dir = 'api/v2/content/archives/allData/61/MYD021KM'

url_base = 'https://nrt4.modaps.eosdis.nasa.gov'

for get_day in [yesterday, today]:

day_dir = os.path.join(remote_dir, get_day)
url = os.path.join(url_base, day_dir)

# Headers with Authorization token
headers = {
"Authorization": "Bearer 79149322-EC68-11E8-B5CB-6AE3680D89FB"
}

# Send a GET request to retrieve the directory listing
response = requests.get(url, headers=headers)

# Check if the request was successful
if response.status_code == 200:
# Parse the HTML using BeautifulSoup
soup = BeautifulSoup(response.content, 'html.parser')

# Find all <a> tags
links = soup.find_all('a')

# Extract href attribute from each <a> tag
urls = [link.get('href') for link in links]

if sat == 'terra':
terra_urls.extend(urls)
elif sat == 'aqua':
aqua_urls.extend(urls)

LOG.debug('got directory listing from nrt4')

Filters:

LAADS_UserServices_M
User Services
User Services
Posts: 295
Joined: Mon Sep 30, 2019 8:33 am America/New_York
Answers: 1
Has thanked: 3 times

Re: Slow directory listings

by LAADS_UserServices_M » Mon Mar 11, 2024 11:24 am America/New_York

You may want to user nrt3, the primary nrt server. Also, you can try code samples provided on nrt3 webpage under "Help" >> "Automating NRT Downloads" (top right of the page):
https://nrt3.modaps.eosdis.nasa.gov/
Regards,
LAADS User Services

To receive news from LAADS DAAC direct to your inbox, email laadsdaac-join@lists.nasa.gov with “subscribe” in the subject line.

Post Reply