User-agent: *And so another 2,400 (two thousand four hundred) lines of restrictions.
Disallow: / cgi-bin
Disallow: / search
Disallow: /query.html
Disallow: / omb / search
Disallow: /omb/query.html
Disallow: / expectmore / search
Disallow: /expectmore/query.html
Disallow: / results / search
Disallow: /results/query.html
Disallow: / earmarks / search
Disallow: /earmarks/query.html
Disallow: / help
Disallow: / 360pics / text
Disallow: / 911 / 911day / text
Disallow: / 911 / heroes / text
...
User-agent: *Everything! www.whitehouse.gov/robots.txt
Disallow: / includes /
Source: https://habr.com/ru/post/49765/
All Articles