Slow directory listings

[satellitedataservices]

We are downloading modis terra and aqua data. We are using python.
URL: https://nrt4.modaps.eosdis.nasa.gov:/api/v2/content/archives/allData/61/MOD021KM/2024/068
Initially we get a listing for today's and yesterday's data. This requires 4 request 2 for aqua and 2 for terra. The directory listings take rather long time.

We compare the output of the list to files we currently have locally and download if necessary. Is there a more efficient way. I was hoping there might be a single file detailing the directory contents. The actual data downloads on the order of seconds. Code snipit
for sat in ['terra', 'aqua']:

if 'terra' in sat:
remote_dir = 'api/v2/content/archives/allData/61/MOD021KM'
else:
remote_dir = 'api/v2/content/archives/allData/61/MYD021KM'

url_base = 'https://nrt4.modaps.eosdis.nasa.gov'

for get_day in [yesterday, today]:

day_dir = os.path.join(remote_dir, get_day)
url = os.path.join(url_base, day_dir)

# Headers with Authorization token
headers = {
"Authorization": "Bearer 79149322-EC68-11E8-B5CB-6AE3680D89FB"
}

# Send a GET request to retrieve the directory listing
response = requests.get(url, headers=headers)

# Check if the request was successful
if response.status_code == 200:
# Parse the HTML using BeautifulSoup
soup = BeautifulSoup(response.content, 'html.parser')

# Find all <a> tags
links = soup.find_all('a')

# Extract href attribute from each <a> tag
urls = [link.get('href') for link in links]

if sat == 'terra':
terra_urls.extend(urls)
elif sat == 'aqua':
aqua_urls.extend(urls)

LOG.debug('got directory listing from nrt4')

[LAADS_UserServices_M]

You may want to user nrt3, the primary nrt server. Also, you can try code samples provided on nrt3 webpage under "Help" >> "Automating NRT Downloads" (top right of the page):
https://nrt3.modaps.eosdis.nasa.gov/