Dignity nobility excellent state highness standing class caste standing superiority priority manner distinction dominance preferment primacy nobleness gentleness gentility preeminence loftiness station grandness txt file is then parsed and may instruct the robotic as to which web pages are usually not to generally be crawled. Like a internet search engine https://www.youtube.com/watch?v=0USI3vTZ0aw