We are downloading modis terra and aqua data. We are using python.
URL: https://nrt4.modaps.eosdis.nasa.gov:/api/v2/content/archives/allData/61/MOD021KM/2024/068
Initially we get a listing for today's and yesterday's data. This requires 4 request 2 for aqua and 2 for terra. The directory listings take rather long time.
We compare the output of the list to files we currently have locally and download if necessary. Is there a more efficient way. I was hoping there might be a single file detailing the directory contents. The actual data downloads on the order of seconds. Code snipit
for sat in ['terra', 'aqua']:
if 'terra' in sat:
remote_dir = 'api/v2/content/archives/allData/61/MOD021KM'
else:
remote_dir = 'api/v2/content/archives/allData/61/MYD021KM'
url_base = 'https://nrt4.modaps.eosdis.nasa.gov'
for get_day in [yesterday, today]:
day_dir = os.path.join(remote_dir, get_day)
url = os.path.join(url_base, day_dir)
# Headers with Authorization token
headers = {
"Authorization": "Bearer 79149322-EC68-11E8-B5CB-6AE3680D89FB"
}
# Send a GET request to retrieve the directory listing
response = requests.get(url, headers=headers)
# Check if the request was successful
if response.status_code == 200:
# Parse the HTML using BeautifulSoup
soup = BeautifulSoup(response.content, 'html.parser')
# Find all <a> tags
links = soup.find_all('a')
# Extract href attribute from each <a> tag
urls = [link.get('href') for link in links]
if sat == 'terra':
terra_urls.extend(urls)
elif sat == 'aqua':
aqua_urls.extend(urls)
LOG.debug('got directory listing from nrt4')
Slow directory listings
-
- Posts: 1
- Joined: Fri Mar 08, 2024 2:45 pm America/New_York
-
- User Services
- Posts: 289
- Joined: Mon Sep 30, 2019 8:33 am America/New_York
- Has thanked: 3 times
Re: Slow directory listings
You may want to user nrt3, the primary nrt server. Also, you can try code samples provided on nrt3 webpage under "Help" >> "Automating NRT Downloads" (top right of the page):
https://nrt3.modaps.eosdis.nasa.gov/
https://nrt3.modaps.eosdis.nasa.gov/
Regards,
LAADS User Services
To receive news from LAADS DAAC direct to your inbox, email laadsdaac-join@lists.nasa.gov with “subscribe” in the subject line.
LAADS User Services
To receive news from LAADS DAAC direct to your inbox, email laadsdaac-join@lists.nasa.gov with “subscribe” in the subject line.