Content filtering: why and how to do it
Hi, Habr! Today we will talk about filtering Internet content. Three years ago, the federal law 139- came into force, supplementing the already adopted 436- “On protecting children from information harmful to their health and development”. In accordance with Russian legislation , access to the Internet in schools is possible only “subject to the application of administrative and organizational measures, technical, software and hardware tools to protect children from information that is harmful to their health and (or) development”. In other words, the law requires mandatory filtering of Internet content. Welcome under cat.
Perhaps someone thinks that this is an exclusively Russian innovation. Not at all. The practice of content filtering has been around for a long time in many countries; this is done in different ways. For example, in France, the Ministry of Public Education launched automated and centralized content filtering in schools on the basis of two “black lists”: in the first list - pornographic resources, in the second - racist and anti-Semitic sites. It was compiled in accordance with the pan-European project to develop a safe Internet (Safer Internet Action Plan).
In the United States, the “Act on the Protection of Children from the Internet” was adopted in 2001. Filtering uses commercial filtering software packages, and in some states, blocking IP addresses at the provider level.
')
In Canada, as part of the “Clean Communication” project, since 2006, providers who voluntarily participate in the program block the transition from the “black list” links, which is formed by analysts of the Canadian Center for Child Protection (Canadian Center for Child Protection). Providers themselves decide how to block content — by IP address or domain name, while Sasktel BellCanada and Telus only block links in principle to avoid accidentally blocking resources that do not contain prohibited content.
Most of the search engines in Germany - Google, Lycos Europe, MSN Deutschland, AOL Deutschland, Yahoo !, T-Online and T-info - have joined the “Voluntary Self-Control for Multimedia Service Providers” agreement. They filter Internet sites on the basis of a list that is determined by the Federal Department of Media Resources Harmful to Young People.
The most stringent measures in Europe are established in the UK. Prohibited Internet content is blocked at the carrier level based on the Home Office Taskforce on Child Protection on the Internet standard. In addition, British law obliges providers to report on whether they are taking measures to curb access to prohibited Internet sites. Internet providers even transmit information to the Internet Watch Foundation (IWF) and the police about suspicious users and network conferences. However, this only applies to the spread of child pornography, which in England is filtered at the level of providers for all. Or at least try.
Content and url filtering is necessary not only in libraries, schools and universities, where it is necessary to do so.
For a long time, most companies are trying to close access to entertainment resources and social networks for their employees. No need to explain why. On the other hand, this must be done wisely. After all, access to Facebook and LinkedIn for HR, PR and sales staff is needed for everyday work. Yes, it is impossible to completely block access to resources that are considered undesirable to visit in the company. You can bypass these prohibitions by going online from your smartphone or tablet. But at least not through the corporate network.
For simplicity, consider the content filtering settings on the example of a regular school. Configuring the NetPolice module and user rules by type, group, and category for any other organization is similar.
What do we need to do? Forbid for all access to sites from the list of Rosreestr, configure schoolchildren access only in the permitted categories, and for teachers - access in all categories, except those forbidden.
1. Start by creating user groups. In our case, these are the Teacher and Schoolchildren groups. Of course, the company will need to create more groups: "Managers", "Employees", "PR", "HR" and so on. The principle of creating user groups is the same as in our example.
To create a user group, go to the “Users and Groups” management console section. In the “Users and groups” block in the “Actions” tab click on the “Add group” link:
2. First, create the group "Teachers":
3. Now we create rules for users using the NetPolice module. In the management console, go to the section "Expansion Modules - NetPolice - Rules" and add it:
4. We call the rule “Prohibition by categories (teachers)” and choose the categories of prohibition:
5. create a user rule and select the “Deny access” rule type:
6. In the rule setting, select the “Teachers” group. This completes the setting for this group:
7. Now we are going to create rules for the “Schoolchildren” group. First (let it not seem strange) we need to deny access to all resources:
7. Add a rule allowing users to work with DNS (port 53) so that Internet access is possible. To do this, create a “DNS client” rule, select TCP / UDP protocols and change the range of destination ports to port 53:
8. Now add a custom rule allowing students to browse sites:
9. Unlike the “Teachers” group, which is allowed to visit any resources other than those prohibited, users from the “Schoolchildren” group can only access certain Internet resources:
10. Create a new category and confirm the automatic creation of a new rule for permission:
11. And finally, we set up a rule for working with the Shkolniki group.
12. We return to the settings of user groups, because we still need to ensure that all traffic passes through a proxy server and is blocked when requests are by proxy.
13. First on the list are the rules for permissions. For the “Teachers” group, the “Ban on categories (teachers)” rule is added automatically:
It remains for us to only add users to certain groups and check the correctness of the settings by going to www.smart-soft.ru/ru/solutions/check-federal-law/ .
These features Traffic Inspector are not limited. In addition to content filtering, we can, by creating a so-called “black list”, deny access to certain sites that may not be included in categories that are already prohibited. For example, in this way, a company may prohibit access to social networks and entertainment services for certain categories of employees.
Read more about black list settings here .
If our readers know other interesting solutions to this problem, as always, we invite you to discuss.
Perhaps someone thinks that this is an exclusively Russian innovation. Not at all. The practice of content filtering has been around for a long time in many countries; this is done in different ways. For example, in France, the Ministry of Public Education launched automated and centralized content filtering in schools on the basis of two “black lists”: in the first list - pornographic resources, in the second - racist and anti-Semitic sites. It was compiled in accordance with the pan-European project to develop a safe Internet (Safer Internet Action Plan).
In the United States, the “Act on the Protection of Children from the Internet” was adopted in 2001. Filtering uses commercial filtering software packages, and in some states, blocking IP addresses at the provider level.
')
In Canada, as part of the “Clean Communication” project, since 2006, providers who voluntarily participate in the program block the transition from the “black list” links, which is formed by analysts of the Canadian Center for Child Protection (Canadian Center for Child Protection). Providers themselves decide how to block content — by IP address or domain name, while Sasktel BellCanada and Telus only block links in principle to avoid accidentally blocking resources that do not contain prohibited content.
Most of the search engines in Germany - Google, Lycos Europe, MSN Deutschland, AOL Deutschland, Yahoo !, T-Online and T-info - have joined the “Voluntary Self-Control for Multimedia Service Providers” agreement. They filter Internet sites on the basis of a list that is determined by the Federal Department of Media Resources Harmful to Young People.
The most stringent measures in Europe are established in the UK. Prohibited Internet content is blocked at the carrier level based on the Home Office Taskforce on Child Protection on the Internet standard. In addition, British law obliges providers to report on whether they are taking measures to curb access to prohibited Internet sites. Internet providers even transmit information to the Internet Watch Foundation (IWF) and the police about suspicious users and network conferences. However, this only applies to the spread of child pornography, which in England is filtered at the level of providers for all. Or at least try.
Content and url filtering is necessary not only in libraries, schools and universities, where it is necessary to do so.
For a long time, most companies are trying to close access to entertainment resources and social networks for their employees. No need to explain why. On the other hand, this must be done wisely. After all, access to Facebook and LinkedIn for HR, PR and sales staff is needed for everyday work. Yes, it is impossible to completely block access to resources that are considered undesirable to visit in the company. You can bypass these prohibitions by going online from your smartphone or tablet. But at least not through the corporate network.
For simplicity, consider the content filtering settings on the example of a regular school. Configuring the NetPolice module and user rules by type, group, and category for any other organization is similar.
What do we need to do? Forbid for all access to sites from the list of Rosreestr, configure schoolchildren access only in the permitted categories, and for teachers - access in all categories, except those forbidden.
1. Start by creating user groups. In our case, these are the Teacher and Schoolchildren groups. Of course, the company will need to create more groups: "Managers", "Employees", "PR", "HR" and so on. The principle of creating user groups is the same as in our example.
To create a user group, go to the “Users and Groups” management console section. In the “Users and groups” block in the “Actions” tab click on the “Add group” link:
2. First, create the group "Teachers":
3. Now we create rules for users using the NetPolice module. In the management console, go to the section "Expansion Modules - NetPolice - Rules" and add it:
4. We call the rule “Prohibition by categories (teachers)” and choose the categories of prohibition:
5. create a user rule and select the “Deny access” rule type:
6. In the rule setting, select the “Teachers” group. This completes the setting for this group:
7. Now we are going to create rules for the “Schoolchildren” group. First (let it not seem strange) we need to deny access to all resources:
7. Add a rule allowing users to work with DNS (port 53) so that Internet access is possible. To do this, create a “DNS client” rule, select TCP / UDP protocols and change the range of destination ports to port 53:
8. Now add a custom rule allowing students to browse sites:
9. Unlike the “Teachers” group, which is allowed to visit any resources other than those prohibited, users from the “Schoolchildren” group can only access certain Internet resources:
10. Create a new category and confirm the automatic creation of a new rule for permission:
11. And finally, we set up a rule for working with the Shkolniki group.
12. We return to the settings of user groups, because we still need to ensure that all traffic passes through a proxy server and is blocked when requests are by proxy.
13. First on the list are the rules for permissions. For the “Teachers” group, the “Ban on categories (teachers)” rule is added automatically:
It remains for us to only add users to certain groups and check the correctness of the settings by going to www.smart-soft.ru/ru/solutions/check-federal-law/ .
These features Traffic Inspector are not limited. In addition to content filtering, we can, by creating a so-called “black list”, deny access to certain sites that may not be included in categories that are already prohibited. For example, in this way, a company may prohibit access to social networks and entertainment services for certain categories of employees.
Read more about black list settings here .
If our readers know other interesting solutions to this problem, as always, we invite you to discuss.
Source: https://habr.com/ru/post/273095/
All Articles