Search engines are answer machines. Search engines are answer machines. They help searchers find, understand, and organize information on the internet to provide the best results for their questions. Search engines must first see your content before you can show up in search results. This is the most crucial piece of SEO: Without your site being found, you won’t appear in the SERPs (Search Engine Result Page).
What is the working principle of search engines?
Three primary functions are the basis of search engines:
- Crawling: Look for content on the Internet and examine the code/content of each URL.
- Indexing: Organize and store the crawled content. Once a page has been added to the index, it will be available for display as a result of relevant queries.
- Ranking: You should provide the best content to answer a searcher’s query. This means that results will be sorted by most relevant to least relevant.
What is search engine crawling?
Crawling refers to the process by which search engines send out a group of robots (known collectively as crawlers or spiders), in order to discover new and updated content. It can be a webpage, image, video, PDF, or another type of content. Regardless of format, links are used to discover content.
What does that word actually mean?
Are you having trouble understanding any of these definitions? To help you stay on top of SEO terminology, our glossary contains chapter-specific definitions.
Googlebot begins by fetching web pages. Then it follows links from those pages to discover new URLs. The crawler can then follow the links to find new content. They add the URL to their index Caffeine, which is a huge database of URLs that they have discovered. This allows the crawler to retrieve the information later when a searcher needs it.
What is a search engine-indexed website?
Search engines store and process information found in an index. This is a massive database that contains all the content they have discovered and feel good enough to offer searchers.
Ranking search engines
Search engines search their index to find highly relevant content. They then order that content to solve the query. Ranking is the process of ranking search results based on relevance. It is generally believed that the higher ranking a website is, the more relevant it is to the query.
Search engine crawlers can block a portion of your site, or tell search engines not to store certain pages in their index. Although there are reasons to do this, it is not necessary. Search engines will only index content that is accessible to them. It’s just as good as invisible if it isn’t.
Search engines in SEO are not all created equal
Many people are unsure about the relative importance and relevance of search engines. While most people are aware that Google holds the largest market share in search engines, many don’t realize how important optimizing for Bing, Yahoo, and other search engines is. The truth is, despite the existence of more than 30 web search engines The SEO community is only interested in Google.
Google is the most popular search engine for web searches. Google Images, Google Maps, and YouTube (a Google property) are all included. Google is home to nearly 90 percent of all web searches — more than Yahoo and Bing combined.
Search engines can find your pages by crawling
You’ve learned that crawling and indexing your website is essential for it to appear in the SERPs. It might be worthwhile to check how many pages you have in the index if you already own a website. This will give you great insight into Google’s crawling capabilities and reveal if it is finding the pages you want.
You can check your index pages by using “site:yourdomain.com”, an advanced search operator. Go to Google and type site:yourdomain.com into the search box. Google will then return the results for the site you have specified in its index. Although the number of results Google displays is not exact, it gives you an idea of which pages on your site are indexed and how they appear in search results.
Monitor and use Google Search Console’s Index Coverage report to get more precise results. If you do not have a Google Search console account, you can register for one. This tool allows you to submit sitemaps and track how many pages have been added to Google’s search index.
There are several reasons you might not be appearing in search results.
- Your site is new and has not been crawled yet.
- External websites are not linked to your site.
- It is difficult for robots to crawl your site because of its navigation.
- Search engines are being blocked by crawler directives, which is a basic code that your site has.
- Google has penalized your site for using spammy tactics
Search engines can learn how to crawl your site
Google Search Console and the “site.domain.com” advanced operator revealed that certain pages of your most important pages were not included in the index. If this is the case, you have options to direct Googlebot as to how you would like your web content crawled. You can tell search engines how to crawl your website, which will give you more control over what ends up in the index.
While most people focus on making Google find important pages, it is easy to forget there are likely to be pages that you don’t want Googlebot to find. These could include URLs with thin content, duplicate URLs (such as e-commerce sort-and filter parameters), special promo codes pages, staging pages or test pages.
Use robots.txt to direct Googlebot away from certain pages or sections of your website.
Robots.txt files can be found in the root directory for websites (ex. Yourdomain.com/robots.txt files are located in the root directory of websites (ex.
Googlebot’s treatment of robots.txt files
- Googlebot will crawl a site if it can’t locate a robots.txt files.
- Googlebot will crawl a site if it finds a robots.txt site file.
- Googlebot will not crawl a site if it encounters an error when trying to access its robots.txt files.
Optimize your crawl budget
Crawl budget refers to the number of URLs Googlebot will crawl your site before leaving. Optimizing the crawl budget ensures that Googlebot doesn’t waste time crawling your pages and isn’t neglecting your most important pages. Crawl budget is especially important for large sites that have tens or thousands of URLs. However, it’s not a bad idea to block crawlers’ access to content you don’t want. You should not block crawlers from pages that you have added additional directives to, such as noindex or canonical tags. Googlebot won’t see instructions on pages if it is blocked.
Robots.txt is not the only protocol that web robots must follow. Bots created by people with malicious intent (e.g., email address scrapers) don’t adhere to this protocol. Some bad actors actually use robots.txt file to locate your private content. It might seem obvious to prevent crawlers from accessing private pages like login or administration pages, but this also makes it easier for malicious intent to find those URLs. These pages should be NoIndexed and secured behind login forms.
GSC URL Parameters Definition
Sites that use e-commerce often make the same content available at multiple URLs. This is done by adding certain parameters to URLs. You’ve probably used filters to narrow down your search if you’ve ever shop online. You might search on Amazon for shoes. Then refine your search by style, color, and size. The URL changes slightly each time you refine your search:
Google doesn’t know what version of a URL it should serve searchers. Google does a good job of determining the representative URL. However, you can use Google Search Console’s URL Parameters feature to tell Google how you want your pages to be treated. This feature will tell Googlebot to “crawl all URLs with the ____ parameter”, which can cause Googlebot to remove those pages from search results. This is what you want if the parameters create duplicate pages. However, it’s not ideal if these pages are to be indexed.
Are crawlers able to find all of your most important content?
Now that you have some strategies to ensure search engine crawlers don’t find unimportant content, let us learn more about optimizations that Googlebot can use to help you find the most important pages.
Sometimes, a search engine can crawl a portion of your site but not other pages or sections. Search engines should be able to find all content that you wish to index, not just your homepage.
Ask yourself: Can the bot crawl to your site, and not just to it?”
Are login forms hiding your content?
Search engines will not see protected pages if users have to sign in, fill out forms or answer surveys. Search engines will not log in to crawlers.
Do you rely on search engines?
Search forms are not available to robots. Some people believe that search engines can find all the information that visitors search for if they have a search box.
Is text being hidden in non-textual content?
Images, videos, GIFs, and other non-text media (images, GIFs, GIFs, etc. You should not use them to display text you want to index. Although search engines are becoming more adept at recognizing images, it is not certain that they will be able to read and understand them yet. It is always best to include text in the markup of your website.
Search engines can follow your site navigation
A crawler must find your site through links from other websites. It also needs links on your site to help it navigate from page to page. It’s possible for search engines to miss a page that you have created, but they won’t find it if it isn’t linked from other pages. Many websites make the fatal mistake of making their navigation difficult for search engines. This can hinder their ability to be listed in search results.
Common navigation errors that could prevent crawlers from viewing all of your website:
- A mobile navigation system that displays different results from your desktop navigation
- A search engine crawler might think personalization, which is showing unique navigation to one type of visitor, is cloaking.
- You should link to the primary page of your website via your navigation. Remember, links are the path crawlers use to reach new pages.
It is important that your website has clear navigation and useful URL folder structures.
Are you able to create a clean information architecture?
Information architecture refers to the organization and labeling of content on a website in order to make it more user-friendly and easier to find. Information architecture that is intuitive means users don’t need to think too hard in order to navigate your website and find what they are looking for.
Are sitemaps being used?
Sitemaps are exactly what they sound like. They are a list of URLs that your site allows crawlers to use to index and discover your content. Create a file that conforms to Google’s standards, and submit it to Google Search Console. This will ensure Google finds your most important pages. Although submitting a sitemap does not replace good site navigation, crawlers can follow a path to your most important pages by submitting it through Google Search Console.
Make sure you only include URLs you want search engines to index. Also, be consistent with your directions. If you have blocked a URL via robots.txt, or if URLs are duplicates of the canonical version, do not include them in your sitemap.
You might still be able to get your site indexed even if it doesn’t have other sites linking to you. To submit your XML sitemap to Google Search Console. Although they may not include your URL in their index, it is worth the effort.
Are crawlers making errors trying to access your URLs
A crawler might encounter errors while crawling URLs on your website. To find URLs that might be affected by crawl errors, you can visit Google Search Console’s “Crawl errors” report. This report will display both server errors and not-found errors. This information can also be found in server log files. However, this is an advanced technique and we won’t go into detail about it here.
Before you can make any meaningful use of the crawl error report, it is important to understand server errors as well as “not found” errors.
4xx Codes: Search engine crawlers cannot access your content because of a client error
4xx errors refer to client errors. This means that the URL requested does not meet the requirements. The “404 – not found” error is one of the most frequent 4xx errors. These errors can be caused by a URL typo or a deleted page. Search engines can’t access URLs if they hit a “404” error. Users can become frustrated when they reach a 404 and then leave.
5xx Codes: Search engine crawlers cannot access your content because of a server error
5xx errors refer to server errors. This means that the server where the page is located has failed to satisfy the request of the searcher. These errors can be found in Google Search Console’s “Crawl error” report. These errors are usually caused by Googlebot abandoning the request. To learn more about server connectivity issues, visit Google’s documentation.
There is an easy way to let search engines and search engines know that your page has changed — the permanent 301 redirect.
Make your own 404 pages
You can personalize your 404 page with links to other pages, site search features, contact information, and links to other important pages. Visitors will be less likely to leave your site if they reach a 404 page.
Let’s say you want to move a page from www.young-dogs.com to www.puppies.com. Users and search engines need a bridge that allows them to move from the old URL into the new. This bridge is called a 301 redirect.
|Implementing a 301 is easy.||If you don’t implement the 301:|
|Link Equity||Transfer link equity from the old URL to the new URL||The authority of the previous URL cannot be transferred to the new URL without a 301.|
|Indexing||Google helps you find and index the latest version of your page.||While 404 errors alone won’t affect search performance, ranking/trafficked pages 404 may cause them to drop out of the index. Rankings and traffic can also be affected.|
|User Experience||This ensures that users can find the page they are looking for.||Your visitors may click on dead links, which will redirect them to an error page instead of the intended page. This can lead to frustration.|
The 301 status code means that the page has moved permanently to a new location. Avoid redirecting URLs URLs to pages with irrelevant content — URLs where the old URL’s actual content isn’t live. If a page ranks for a query, and you 301 it with different content to another URL, it may drop in rank because the content that made it relevant is no longer there. 301s can be powerful. Please use caution when moving URLs.
The option to redirect a page via 302 is also available. However, this should only be used for temporary moves or in cases where passing on link equity is not an issue. 302 redirects can be thought of as a detour. Temporarily, you are siphoning traffic along a route. But it won’t last forever.
Indexing: How search engines interpret and store pages?
After you have ensured that your site was crawled, it is time to ensure it can be indexed. It’s true, just because your site is found and crawled by search engines doesn’t mean it will be included in their index. We discussed in the section on crawling how search engines find your web pages. Your discovered pages are stored in the index. The index is where a crawler discovers a page. It renders it exactly like a browser. The search engine analyzes the contents of the page to determine if it is suitable for its purpose. It stores all of this information in its index.
Continue reading to find out how indexing works, and how you can ensure your site is included in this important database.
What can I do to see what a Googlebot crawler sees on my pages?
Google caches web pages at different frequencies. Sites that are more well-known and frequent, such as https://www.nytimes.com, will be crawled less often than the lesser-known website for WebJIVE, https://www.web-jive.com
Click the drop-down icon next to the URL and choose “Cached” to view the cached version.
To determine if important content on your site is being cached or crawled, you can also see the text-only version.
Is it possible to remove pages from the index?
Yes, you can remove pages from the index. A URL may be removed for the following reasons:
- The URL returned a “not found” error (4XXX or server error (5XXX). This could be accidental (the site was moved and no 301 redirect was set up) or it could be intentional (the page was deleted to remove it from the index and 404ed).
- The URL was given a noindex meta tag. Site owners can add this tag to instruct search engines to remove the page from their index.
- The URL was manually penalized for violating search engine’s Webmaster Guidelines. It was then removed from the index.
- Visitors can only access the URL if they have a password.
You can use the URL Inspection tool if you suspect that a page from your website is not showing up in Google’s index. Or, use Fetch as Google, which has a “Request Indexing” feature that allows you to submit URLs to the index. You can also use the GSC’s “fetch” tool to render your page and see if Google is misinterpreting it.
Search engines can be told how to index your site
Robots meta directives
Meta directives, also known as “meta tags”, are instructions that you can give search engines about how you want your website to be treated.
Search engine crawlers can be told to not index this page in search results or to not pass link equity on to other pages. These are the instructions Robots Meta Tags are used to execute these commands in theYour HTML pages (most frequently used) or the X-Robots Tag in the HTTP header.
Robots meta tag
You can use the robots meta tag within the HTML code of your webpage. It can be used to exclude specific search engines or all. These are the most popular meta directives. You can also see what situations they might be used in.
index/noindexThis tells search engines whether to crawl the page and keep it in an index that can be retrieved. By choosing “noindex”, you are telling search engines that you don’t want the page to be included in search results. Search engines assume that they can index all pages by default so the “index” value should not be used.
- When might you use:If you want to remove pages from Google’s index (ex. user-generated profile pages), you might choose to mark the page “noindex”. However, you still want them to be accessible to visitors.
Follow/nofollowSearch engines can tell if links should be followed or not. Bots will follow the links on your page, passing link equity to them if they click “Follow”. You can also choose to use “nofollow” to make search engines not follow the pages and pass link equity to them. All pages will default have the “follow” attribute.
- When might you use: Nofollow is often used in conjunction with noindex to stop a page being indexed and prevent crawlers from following links.
- NoarchiveThis is used to prevent search engines from saving a cached version of the page. Search engines will automatically keep visible copies of all pages that they have indexed. These copies can be accessed by searchers via the cached link in search results.
- When might you use: You might use the noarchive tag if you have an e-commerce website and your prices are changing frequently. This will prevent searchers from seeing obsolete pricing.
Meta directives impact indexing but not crawling
Googlebot must crawl your page to view its meta directives. If you want to stop crawlers from accessing specific pages, meta directives will not work. To be considered respectable, robots tags must be crawled.
If you wish to block search engines at large, the x-robots tags is included in your URL’s HTTP header. It provides more flexibility and functionality than meta tags. You can also use regular expressions and block non-HTML files and apply noindex tags sitewide.
You could, for example, exclude whole folders or file types (like moz.com/nobake/old_recipes-to_noindex).
Header set XRobots-Tag "noindex" and "nofollow"
An X-Robots tag can use the derivatives from a robots meta-tag.
Or, you can choose specific file types (e.g. PDFs).
Header set XRobots-Tag "noindex" and "nofollow"
For more information about Meta Robot Tags, visit Google’s Robots Meta Tag Specifications.
Make sure that the “Search Engine Visibility” box in Dashboard > Settings > Reading is and not. This will prevent search engines from visiting your site via robots.txt!
You can learn how you can influence indexing and crawling to help you avoid common pitfalls that could prevent important pages from being found.
Ranking: How search engines rank URLs
How can search engines make sure that searchers get the most relevant results when they type a query in the search bar? This is called ranking. It refers to the order of search results according to relevance.
Search engines use algorithms to determine relevancy. This is a method or formula that retrieves stored information and arranges it in meaningful ways. Over the years, these algorithms have undergone many modifications to improve search results quality. Google makes algorithm changes every day, some minor quality tweaks and others core/broad algorithm updates to address a specific issue. For example, Penguin is used to combat link spam. For a complete list of confirmed and unconfirmed Google updates, see our Google Algorithm History.
Why is the algorithm changing so frequently? Google is just trying to keep us awake at night? Although Google isn’t always clear about why they do certain things, we know that Google’s goal when making algorithm adjustments to improve search quality is to increase overall search quality. Google will respond to queries about algorithm updates with the following: “We make quality updates every day.” This means that if your site has suffered from an algorithm adjustment, you can compare it with Google’s Quality Guidelines and Search Quality Rater Guidelines. Both are extremely telling about what search engines want.
What are search engines looking for?
Search engines have always aimed for the same thing: to answer searcher’s queries in the most useful formats. If this is true, then it makes sense that SEO has changed over the years.
It’s like someone learning a new tongue.
Their understanding of the language at first is limited. “See Spot Run” is the best example. As they get more practice, their understanding improves and they begin to understand semantics, the meaning behind words and the relationships between them. With enough practice, the student will eventually be able to understand the language and provide answers to any questions that are not clear or complete.
It was easier to manipulate search engines when they were still learning our language. This made it much easier to use tricks and tactics that went against quality guidelines. Keyword stuffing is one example. To rank for a keyword such as “funny humor”, you could add these words to your page a lot of times and make them bold. This will increase your chances of ranking for the term.
This strategy led to terrible user experiences. Instead of laughing at funny jokes, people were bombarded with annoying, difficult-to-read text. This tactic may have worked in the past but it is not what search engines want.
Linking is an important part of SEO
Two things could be meant by links when we refer to them. Backlinks, also known as “inbound” links, are links that point to your website from other websites. While internal links refer to links that point to other pages on your site (on the same website),
Linking has always played an important role in SEO. Search engines required assistance in determining which URLs were more trustworthy to determine how they rank search results. This was done by calculating the number of links that point to a given site.
Backlinks are very similar to real-life WoM referrals. Let’s use Jenny’s Coffee as an example.
- Referrals by others = sign of authority
- Example: Many people will tell you Jenny’s Coffee is the best coffee in town.
- Referring from yourself = biased. This is not a sign of authority.
- Example: Jenny says Jenny’s Coffee has the best coffee in town
- Referrals from low-quality or irrelevant sources are not a sign of authority. You could be flagged as spam if you do.
- Example: Jenny spent money to have people who had never been to her coffee shop tell other people how great it is.
- No referrals = unclear authority
- Example: Jenny’s Coffee may be great, but you haven’t been able to find anyone with an opinion so you don’t know.
This is why PageRank was invented. PageRank, part of Google’s core algorithm, is a link analysis algorithm that was named after Larry Page, one of Google founders. PageRank measures the number and quality of links that point to a page in order to determine its importance. It is assumed that the more trustworthy, relevant, and important a page is, the higher the number of links it has earned.
Your chances of ranking higher in search results are greater if you have more natural backlinks from trusted websites.
SEO Content plays a major role
Links would not be useful if they weren’t directing searchers to the right thing. Content is that something! Content is not just words. It’s everything that searchers can consume, including video, image, and text. Search engines can be thought of as answer machines. Content is what the engines use to deliver the answers.
There are many possible results when someone searches for something. Search engines can help decide which pages they will find most valuable. How well your content matches the query’s intent is a major factor in determining the rank of your page for a particular query. This means that the page must match the query’s intent and provide the information needed to complete the task.
This emphasis on user satisfaction and task achievement means that there are no set guidelines for how long your content should have, how many keywords it should include, or what header tags you should use. While all these factors can impact how well a page ranks in search results, the main focus should be on the people who will be reading it.
Today there are hundreds, if not thousands, of ranking signals. The top three remain consistent: links to your site (which act as a third party credibility signal), on-page content (quality content that fulfills a user’s searcher’s intent), RankBrain.
What is RankBrain?
RankBrain, the machine-learning component of Google’s core algorithm, is called RankBrain. Machine learning is a computer program that continuously improves its predictions through training data and new observations. It’s constantly learning. Search results should always improve because of this.
If RankBrain detects a lower ranking URL that provides a better result for users than the higher ranking URLs you can bet that RankBrain adjusts those results, moving the most relevant result higher, and degrading the less relevant pages as a side effect.
We don’t know what RankBrain is like most things in the search engine. However, the folks at Google do.
What does this all mean for SEO?
Google will continue to use RankBrain for the promotion of relevant and helpful content. We need to be more focused on satisfying searcher intent because Google will continue to leverage RankBrain. You’ve already taken a major step towards achieving high performance in a RankBrain world by providing the best information and experience possible to searchers who may land on your page.
Google has made it clear that click data is used to modify SERPs for specific queries, even though they haven’t used the term “direct rank signal”.
Click data can affect the ranking. We can see that for a query, 82% click on #2, and 10% click on #1. After a while, we realize that #2 is probably the one people are looking for, so we’ll change it.
A second comment by Edmond Lau, an ex-Google engineer, supports this:
It is obvious that any reasonable search engine would use click information on their results to improve ranking and quality. Although the exact mechanics of click data are often secretive, Google clearly states that click data is used with its patents for systems such as rank-adjusted contents items.
Google must maintain and improve search quality. It seems obvious that engagement metrics will be more than correlation. However, Google does not consider engagement metrics to be a ranking signal. These metrics are used to improve search quality and rank individual URLs.
What have the experts said?
Multiple tests have shown that Google will alter the order of SERPs in response to user engagement.
- Rand Fishkin’s 2014 test resulted at #7, moving up to #1 after getting about 200 people to click the URL from the SERP. It was interesting to note that ranking improvements were not dependent on the geographical location of people who clicked on the link. In the US, many participants saw a rise in rank, while it was lower in Google Canada and Google Australia.
- Larry Kim’s comparison top pages and average dwell time pre and post-RankBrain seemed like it indicated that Google’s machine-learning algorithm lowers the rank of pages people spend less time on.
- Darren Shaw’s testing showed user behavior’s effect on local search results and map pack results.
User engagement metrics are used to adjust SERPs quality and rank position changes. It’s obvious that SEOs should optimize for engagement. While engagement doesn’t affect the quality of your page, it does increase your value relative to other search results. If searchers like pages other than yours, then it is possible for it to drop in ranking.
Engagement metrics are a fact-checker when it comes to ranking web pages. Google uses engagement metrics to help them adjust if they don’t get it right.
Search results evolution
Search engines were not as sophisticated back then, so the term “10 Blue Links” was created to describe the flat structure in the SERP. Google would return 10 organic results for a search, each with the same format.
SEO’s holy grail was holding the #1 spot in this search landscape. Then, something happened. Google started adding new formats to search results pages. These are called SERP features. These SERP features include:
- Paid advertising
- Snippets featured
- People also ask boxes
- Pack Local (map)
- Knowledge panel
Google keeps adding new results all the time. They even tried “zero-result SERPs,” a phenomenon in which only one result from Knowledge Graph was displayed on the SERP, with no other results below it. There was also an option to “view more”
These features created panic initially for two reasons. One, these features caused organic search results to drop further down the SERP. A second result is that searchers click less on organic results because more queries are being answered directly on the SERP.
Google did this because it was easy. All of it comes down to search experience. Different content formats may be more effective in answering certain queries, according to user behavior. You can see how different types SERP features correspond to different types of query intents.
|Query Intent||Possible SERP Feature Triggered|
|Informational with only one answer||Knowledge graph / Instant answer|
Google, a search engine that uses local business listings as its index, creates local search results.
Local SEO should be done for businesses that have a physical location that customers can visit (ex. dentist), or for businesses that travel to see their customers (ex. plumber). Make sure you claim, verify and optimize a Google My Business Listing.
Google has three major factors that determine the ranking of localized search results:
Relevance refers to how closely a local business matches the searcher’s requirements. You can ensure that your business is relevant to searchers by making sure all information is correct and complete.
Google uses your location to provide you with better local results. Local search results are sensitive to proximity. This refers to the searcher’s location and/or the query location (if one was included).
Although organic search results can be sensitive to the location of a searcher, they are not as prominent as local pack results.
Google considers prominence to be a factor and will reward well-known businesses in the real world. Google considers other factors than the offline reputation of a business to determine its local ranking. These include:
Local businesses rank higher in local search results if they have received a lot of Google reviews.
A “business citation” (or “business listing”) is a web-based reference for a local business’ NAP (name, address, and phone number) on a localized website (Yelp.com, Acxiom.com, YP.infogroup.localeze.com, etc.
Local rankings are affected by the consistency and number of local business citations. Google continually creates its local business index by pulling data from many sources. Google increases its “trust” in the validity and reliability of data by locating consistent references to a business name, address, and phone number. Google can then show the business with greater confidence. Google also makes use of information from other websites, such as articles and links.
Local SEO is also a good practice, as Google considers the website’s position within organic search results when determining its local ranking.
The next chapter will cover on-page best practices to help Google and other users understand your content better.
While engagement is not yet a Google ranking factor for local search, it will continue to grow in importance. Google continues to enrich local search results by including real-world data such as popular times to visit and average visits. Local results are being influenced more than ever by real-world data. This interactivity allows searchers to interact with local businesses and provide feedback, as opposed to static information such as citations.
Google wants to provide the most relevant local businesses to their users. Therefore, they should use real-time engagement metrics to assess quality and relevance.