This is not a guide to action, but only some of my conclusions, which are not the ultimate truth.
So, let's consider the situation when bots climb the site and do various unsightly things there: scan, spam through forms, rob content and so on. But the site can go and good robots - with search engines that can not be offended. It is necessary to identify bad bots and block them.