There are different type of robots for different function in Search Engine, some robots index the HTML Titles, or the first few paragraphs, or index all words, with giving more weight on HTML constructs, etc. Some parse the META tag, or a hidden tags, while some decide to index or not that based on the attribute of your site. However if an indexing robot know your site document it may decide to parse it by recognizing words and phrases in text, and insert information into its database, that depends on the robot as each one robot uses different strategy. However all robots are going to read your robot.txt file if present.

The popularity of a website is extremely important while all search engines can find out how many links are going to and from your page. If you have good number of outgoing and inbound link there are more chances that the indexing robots can consider for deep crawling.