Page 1 of 1

Slow directory listings

Posted: Fri Mar 08, 2024 2:54 pm America/New_York
by satellitedataservices
We are downloading modis terra and aqua data. We are using python.
Initially we get a listing for today's and yesterday's data. This requires 4 request 2 for aqua and 2 for terra. The directory listings take rather long time.

We compare the output of the list to files we currently have locally and download if necessary. Is there a more efficient way. I was hoping there might be a single file detailing the directory contents. The actual data downloads on the order of seconds. Code snipit
for sat in ['terra', 'aqua']:

if 'terra' in sat:
remote_dir = 'api/v2/content/archives/allData/61/MOD021KM'
remote_dir = 'api/v2/content/archives/allData/61/MYD021KM'

url_base = ''

for get_day in [yesterday, today]:

day_dir = os.path.join(remote_dir, get_day)
url = os.path.join(url_base, day_dir)

# Headers with Authorization token
headers = {
"Authorization": "Bearer 79149322-EC68-11E8-B5CB-6AE3680D89FB"

# Send a GET request to retrieve the directory listing
response = requests.get(url, headers=headers)

# Check if the request was successful
if response.status_code == 200:
# Parse the HTML using BeautifulSoup
soup = BeautifulSoup(response.content, 'html.parser')

# Find all <a> tags
links = soup.find_all('a')

# Extract href attribute from each <a> tag
urls = [link.get('href') for link in links]

if sat == 'terra':
elif sat == 'aqua':

LOG.debug('got directory listing from nrt4')

Re: Slow directory listings

Posted: Mon Mar 11, 2024 11:24 am America/New_York
by LAADS_UserServices_M
You may want to user nrt3, the primary nrt server. Also, you can try code samples provided on nrt3 webpage under "Help" >> "Automating NRT Downloads" (top right of the page):