How to win a click into Ya. Direct and AdWords for 600 thousand rubles a month
Over the past six months, we have managed to win the “clickthrough” of our contextual advertising with a budget of 1 million rubles a month.
The key to defeating the fraud was the per-minute monitoring of traffic with notifications of abnormal changes and disabling problematic API announcements, and a series of reports that reflect the situation in real time.
Figure 1. Visitor number chart by keywords for
')
One of the first signs of “skipping” advertising will be an increase in the percentage of funds returned for fraud in Direct and AdWords.
In AdWords, you can turn on the "invalid clicks" level on the "columns" tab:
Figure 2. AdWords configured “invalid click” columns
In our case, with an average level of “ invalid clicks ” in Yandex.Direct ≈ 10%, Yandex suddenly began to return 40% of the advertising budget, and after a month it altogether 54%.
The next sign of fraud is unreasonable growth in the number of transitions and strong changes in behavioral indicators for a number of ad groups.
We noticed that in a number of keywords for which there were never more than 200 visitors per day, suddenly there were bursts of up to 3,000 visitors. In fact, the budget, in the days of such activities, could go on one advertising campaign, if it was not stopped in time.
Figure 3. Unjustified traffic growth for a specific ad group in Direct
It is enough to read the discussion of the fraud level in contextual advertising in the official Direct Club to make sure that many advertisers lose money due to fraud.
Google officially recognizes the mistakes of its security system and provides advertisers with “Refund Claims” (refund of the spent budget). According to statistics from the service ClickSease, Google Ads on average returns 12% of the advertising budget.
Yandex, on the other hand, does not recognize the vulnerability of its own protection filters and complaints of obvious fraud cases. It sends a sample response that the problem lies with the advertiser and his website.
In our case, the percentage of “invalid clicks” in the report by Ya. Direct has never risen above the cherished 50% in any of the advertising campaigns, even on the days of the most violent bursts, when 80% of the budget “merged” into the usually not very popular ad group without calls and applications.
Figure 4. The “Invalid Clicks” Level in One of the Campaigns in Direct
Fraudulent traffic went to four business lines in two cities. When new advertising campaigns were launched or when they were launched for a long time, they were redistributed to them within a few hours.
Click Fraud was not time bound, and the redistribution of the budget, for example, for the night, did not give any effect: we still received our own volume of traffic. "Clicking" occurred equally actively both in YAN, and on Yandex search, and on Google’s GMS.
On the site, bots simulated user behavior, navigated through sections of the site, selected text, naturally scrolled, and moved the cursor.
All click-protection services have only one direct fight tool - blocking suspicious IP addresses or placement sites in Ya.Direct and AdWords campaigns.
If dynamic IPs are used against you, any anti-click fraud service will always be one step behind the fraudsters in blocking them by IP: the bot has already made a few clicks on your ad by the time the service blacklists the IP and stop advertising on it. In addition, after the fraudulent software will not be able to see your advertisement, it will simply change the IP address, Hardware-ID and continue its actions.
When attacking advertising in the CCM or YAN, various placements are usually used, and automatic protection systems cannot detect suspicious sites for blocking.
Let's go back to blocking by IP - here we come to the most interesting part - if AdWords allows blocking up to 500 IP addresses, then Yandex Direct can block only 25 unique IP addresses for one advertising campaign! Such a small black list of IP addresses is no longer relevant, since now you can safely purchase 500 IPv4 addresses for 10 thousand rubles and bypass this restriction.
You can protect yourself from “clipping” performed at a high level in just two ways:
If you learn not to show ads click on bots or fraudulent users, then they will not be able to harm.
You can always track similar behaviors and patterns, for example, that fraud usually runs on Windows 7 from 5:00 to 9:00 in Moscow, and set a rate adjustment of -100% for a similar audience in all advertising campaigns attacked. The functionality of bid adjustments in AdWords is quite extensive, which cannot be said about Yandex Direct adjustments .
In order to have an idea of how exactly we are attacked, and to manually trace patterns in the fraudulent traffic, we have connected the Russian service from fraud ClickFrog . The product is a well-known, popular among CPA and so on.
ClickFrog quickly proved complete incapacity:
Then we installed the code of the American service ClickSease , aimed at AdWords, and not yet working with Direct. The service, by the way, has, in contrast to ClickFrog, a free test period of 2 weeks.
ClickSease was more useful: he began to catch 300-400 unique fraudulent IP per day. For each blocked IP service gives statistics:
From the ClickSease report, we were able to identify patterns in frade:
However, even such obvious patterns were not enough to reduce the harm from fraud, and I didn’t want to disable advertising on mobile. Services are usually able to only give ideas on identifying similar patterns in the frode, and then it is necessary to detect fraud in the Metric (in the case of an attack on Direct) and select it in separate Yandex Audience segment for subsequent analysis and blocking.
Figure 5. An example of analyzing traffic by age group in the Metric to find fraud patterns
Traffic slices to help fraud patterns:
In the case of AdWords, the countermeasure is clear:
In Yandex Direct, combat mechanics are more complex and are divided into two options:
a) you managed to find an obvious fraud pattern related to gender, age or mobility:
b) no obvious patterns were found:
Frode always gives rise to obvious pockets and spikes , whether it is abstruse software with imitation of the behavior of a real user or a group of freelancers performing a technical task.
Figure 6. Visitor Number Chart by Decker Order
Frode occurs unevenly for several reasons:
If you learn to quickly find and eliminate foci, you can significantly reduce the harm from fraud. In our case, the obvious sign was the anomalous increase in the number of clicks on contextual advertising in specific 10 minutes or one minute for some keywords.
Google Data Studio is best suited for visualization, as it is only Analytics that is able to correctly collect data divided into 1 and 10 minutes, and Metric gives incorrect results when building reports on the panels.
In order not to keep track of all fraud cases manually, I made a report in Google Spreadsheets, which updates the data every minute and notifies you that fraud starts.
Google Spreadsheets support the Core Reporting API , which can be accessed through the Script Editor in Spreadsheets.
Step 1. Go to the script editor to access Analytics
Figure 7. Script editor for accessing the Analytics Core Reporting API via Google Tables
Step 2. We register the API request to Analytics in order to obtain data on the necessary indicators (for example, the number of users who have switched on paid advertising in each minute of the day, as in our case).
Step 3. Set the trigger to update the data every minute:
Figure 8. We request fresh data every minute to respond quickly to fraud.
Step 4. We create a pivot table from a sheet that is updated with the necessary data once a minute, and we analyze these indicators to set up triggers for email notifications or to disable ad groups using the Ya.Direct or AdWords API.
Figure 9. An example of setting up formulas for anomaly notifications
Counter-click fraud can be divided into three groups:
a) Proactive action:
b) Preventive actions:
c) Post factum actions:
Figure 10. Protection against click fraud
Useful links:
Those who are also faced with the click-through of contextual advertising - write in the comments, we will try to help each other!
The key to defeating the fraud was the per-minute monitoring of traffic with notifications of abnormal changes and disabling problematic API announcements, and a series of reports that reflect the situation in real time.
Figure 1. Visitor number chart by keywords for
')
How to find out that you are being attacked?
One of the first signs of “skipping” advertising will be an increase in the percentage of funds returned for fraud in Direct and AdWords.
“In Yandex Direct, the fraud expenses are automatically returned to the balance of the advertising campaign. The number of clicks eliminated by the fraud protection system is displayed in the “daily statistics” reports, “general statistics” in the “invalid clicks for the entire selected period.”- Help Ya. Direct "invalid clicks . "
In AdWords, you can turn on the "invalid clicks" level on the "columns" tab:
Figure 2. AdWords configured “invalid click” columns
In our case, with an average level of “ invalid clicks ” in Yandex.Direct ≈ 10%, Yandex suddenly began to return 40% of the advertising budget, and after a month it altogether 54%.
The next sign of fraud is unreasonable growth in the number of transitions and strong changes in behavioral indicators for a number of ad groups.
We noticed that in a number of keywords for which there were never more than 200 visitors per day, suddenly there were bursts of up to 3,000 visitors. In fact, the budget, in the days of such activities, could go on one advertising campaign, if it was not stopped in time.
Figure 3. Unjustified traffic growth for a specific ad group in Direct
Yandex and Google do not protect against fraud
It is enough to read the discussion of the fraud level in contextual advertising in the official Direct Club to make sure that many advertisers lose money due to fraud.
Google officially recognizes the mistakes of its security system and provides advertisers with “Refund Claims” (refund of the spent budget). According to statistics from the service ClickSease, Google Ads on average returns 12% of the advertising budget.
How to return part of the money for the "sklik" through AdWords
You must send a complaint to the "invalid clicks" in Advords, after consideration of which you will be reimbursed ≈ 12% of the money spent. Also, the service ClickSease automatically sends similar requests to Google for a refund every 2 months.
In our case, Google AdWords first recognized 18% of our traffic as “ invalid ” and returned the money for them, and when we sent a refund claims complaint, Google reimbursed another 13% of the budget.
In our case, Google AdWords first recognized 18% of our traffic as “ invalid ” and returned the money for them, and when we sent a refund claims complaint, Google reimbursed another 13% of the budget.
Yandex, on the other hand, does not recognize the vulnerability of its own protection filters and complaints of obvious fraud cases. It sends a sample response that the problem lies with the advertiser and his website.
In our case, the percentage of “invalid clicks” in the report by Ya. Direct has never risen above the cherished 50% in any of the advertising campaigns, even on the days of the most violent bursts, when 80% of the budget “merged” into the usually not very popular ad group without calls and applications.
Figure 4. The “Invalid Clicks” Level in One of the Campaigns in Direct
What level of attack are we facing
Fraudulent traffic went to four business lines in two cities. When new advertising campaigns were launched or when they were launched for a long time, they were redistributed to them within a few hours.
Click Fraud was not time bound, and the redistribution of the budget, for example, for the night, did not give any effect: we still received our own volume of traffic. "Clicking" occurred equally actively both in YAN, and on Yandex search, and on Google’s GMS.
On the site, bots simulated user behavior, navigated through sections of the site, selected text, naturally scrolled, and moved the cursor.
Are there any ready-made remedies?
All click-protection services have only one direct fight tool - blocking suspicious IP addresses or placement sites in Ya.Direct and AdWords campaigns.
If dynamic IPs are used against you, any anti-click fraud service will always be one step behind the fraudsters in blocking them by IP: the bot has already made a few clicks on your ad by the time the service blacklists the IP and stop advertising on it. In addition, after the fraudulent software will not be able to see your advertisement, it will simply change the IP address, Hardware-ID and continue its actions.
When attacking advertising in the CCM or YAN, various placements are usually used, and automatic protection systems cannot detect suspicious sites for blocking.
Let's go back to blocking by IP - here we come to the most interesting part - if AdWords allows blocking up to 500 IP addresses, then Yandex Direct can block only 25 unique IP addresses for one advertising campaign! Such a small black list of IP addresses is no longer relevant, since now you can safely purchase 500 IPv4 addresses for 10 thousand rubles and bypass this restriction.
You can protect yourself from “clipping” performed at a high level in just two ways:
- learn not to show advertisements to fraudulent users or bots, for which you need to find certain "patterns" in their behavior and characteristics;
- temporarily stop specific ad groups and keywords that are being attacked.
Cut off part of the audience to keep most of it.
If you learn not to show ads click on bots or fraudulent users, then they will not be able to harm.
You can always track similar behaviors and patterns, for example, that fraud usually runs on Windows 7 from 5:00 to 9:00 in Moscow, and set a rate adjustment of -100% for a similar audience in all advertising campaigns attacked. The functionality of bid adjustments in AdWords is quite extensive, which cannot be said about Yandex Direct adjustments .
We are looking for patterns in frode through protection services
In order to have an idea of how exactly we are attacked, and to manually trace patterns in the fraudulent traffic, we have connected the Russian service from fraud ClickFrog . The product is a well-known, popular among CPA and so on.
ClickFrog quickly proved complete incapacity:
- allocated no more than 40 suspicious IP addresses per day, with traffic from Yandex.Direct of 3,000 thousand hops per day, and, even recognized by Yandex, 1,300 left-click clicks per day;
- The main tool for protecting the service is blocking by IP address, the command about which is sent to Ya.Direct by API. However, as soon as the black list of 25 IP addresses is filled, you must manually delete the last few IP addresses in each advertising campaign and wait for the next list to be filled, and so in a circle.
Then we installed the code of the American service ClickSease , aimed at AdWords, and not yet working with Direct. The service, by the way, has, in contrast to ClickFrog, a free test period of 2 weeks.
ClickSease was more useful: he began to catch 300-400 unique fraudulent IP per day. For each blocked IP service gives statistics:
- internet provider;
- platform from which the transition was made;
- operating system;
- unique device ID;
- the time of the first and last transition;
- region.
From the ClickSease report, we were able to identify patterns in frade:
- in 81% of cases, the device simulates a mobile OS: Android or iOS;
- in 59% of cases, geolocation of an IP address does not apply to Moscow, with a ford aimed at Moscow.
We are looking for patterns in Frode manually
However, even such obvious patterns were not enough to reduce the harm from fraud, and I didn’t want to disable advertising on mobile. Services are usually able to only give ideas on identifying similar patterns in the frode, and then it is necessary to detect fraud in the Metric (in the case of an attack on Direct) and select it in separate Yandex Audience segment for subsequent analysis and blocking.
Figure 5. An example of analyzing traffic by age group in the Metric to find fraud patterns
Traffic slices to help fraud patterns:
- audience dynamics by age groups;
- the dynamics of the long-term interests of users;
- device dynamics and OS.
In the case of AdWords, the countermeasure is clear:
- determine the audience segment "infected" fraud;
- set rate adjustment -100% for the selected segment;
- we track changes in indicators: conversion, time on the site, depth of viewing, bounce rate.
In Yandex Direct, combat mechanics are more complex and are divided into two options:
a) you managed to find an obvious fraud pattern related to gender, age or mobility:
- set the rate adjustment -50% or -100% for the selected segment;
- we trace change of key indicators.
b) no obvious patterns were found:
- We allocate frod traffic to a separate segment of Yandex. Audience (for example, you knew for sure that from October 1 to October 20 there could not be 5,000 clicks on an ad group, which always had no more than 30 visits per day)
- we create a segment of users similar to our fraud through the Yandex look-alike ;
- set up a rate adjustment of -100% for a manually created audience segment;
- We carefully test the reduction in advertising rates for the segments created by Yandex.
Build charts that show fraud
Frode always gives rise to obvious pockets and spikes , whether it is abstruse software with imitation of the behavior of a real user or a group of freelancers performing a technical task.
Figure 6. Visitor Number Chart by Decker Order
Frode occurs unevenly for several reasons:
- in order to make the attack “smoothed out” you need to possess confidential information and know who, when and how many transitions you make in your advertisement;
- the software acts in spurts, and on the minute, 10 minute, and sometimes on the hourly chart, its actions will be evident;
- even if “schoolchildren” are working against you from the bulletin boards, then they also act on a specific task with an algorithm, and the anomalies generated by them will be easily traced.
If you learn to quickly find and eliminate foci, you can significantly reduce the harm from fraud. In our case, the obvious sign was the anomalous increase in the number of clicks on contextual advertising in specific 10 minutes or one minute for some keywords.
Google Data Studio is best suited for visualization, as it is only Analytics that is able to correctly collect data divided into 1 and 10 minutes, and Metric gives incorrect results when building reports on the panels.
How to build charts in 10 minutes, not in an hour, in Google Data Studio
By default, in Analytics or Data Studio it is impossible to build graphs by the minute or 10 minutes, but this can be done as follows in the Date Studio:
Step 1. Open the edit fields
Step 2. Create copies of the following fields: Year, Month of the year, Day of the month, Hour, Minute, and call them, for example, Year (day), Month of the year (day) and so on. Also in the copied fields it is necessary to change the Type from the time and date format to “number” as shown in the figure.
Step 2. Change the type of the copied field from “date” to “number”
Step 3. Create a new field in which we write the following formula: Year (number) * 10,000,000 + Month of the year (number) * 100,000 + Day of the month (date) * 1000 + Hour (number) * 10 + FLOOR (Minute (date) / ten)
Step 3. Create a calculated field "Time for 10 minutes"
Step 4. Save the created field, then go back to the list of all the fields and find our new field “Time for 10 minutes (window)”. You need to change its type from “Number” to “Date and time” as shown in the figure, and then assign the type “Number” back to this field.
Step 4. Create a calculated field "Time for 10 minutes"
Step 5. Create a “Combined Chart” and set our new field “Time to 10 minutes” as a parameter, as shown in the figure. Is done.
Step 5. Create a “combined chart”
Step 1. Open the edit fields
Step 2. Create copies of the following fields: Year, Month of the year, Day of the month, Hour, Minute, and call them, for example, Year (day), Month of the year (day) and so on. Also in the copied fields it is necessary to change the Type from the time and date format to “number” as shown in the figure.
Step 2. Change the type of the copied field from “date” to “number”
Step 3. Create a new field in which we write the following formula: Year (number) * 10,000,000 + Month of the year (number) * 100,000 + Day of the month (date) * 1000 + Hour (number) * 10 + FLOOR (Minute (date) / ten)
Step 3. Create a calculated field "Time for 10 minutes"
Step 4. Save the created field, then go back to the list of all the fields and find our new field “Time for 10 minutes (window)”. You need to change its type from “Number” to “Date and time” as shown in the figure, and then assign the type “Number” back to this field.
Step 4. Create a calculated field "Time for 10 minutes"
Step 5. Create a “Combined Chart” and set our new field “Time to 10 minutes” as a parameter, as shown in the figure. Is done.
Step 5. Create a “combined chart”
Set up notifications for fraud sites
In order not to keep track of all fraud cases manually, I made a report in Google Spreadsheets, which updates the data every minute and notifies you that fraud starts.
Google Spreadsheets support the Core Reporting API , which can be accessed through the Script Editor in Spreadsheets.
Step 1. Go to the script editor to access Analytics
Figure 7. Script editor for accessing the Analytics Core Reporting API via Google Tables
Step 2. We register the API request to Analytics in order to obtain data on the necessary indicators (for example, the number of users who have switched on paid advertising in each minute of the day, as in our case).
Google Script code to request any data from Analytics to Google Spreadsheets
function runDemo() { try { var firstProfile = getFirstProfile(); var results = getReportDataForProfile(firstProfile); outputToSpreadsheet(results); } catch(error) { Browser.msgBox(error.message); } } function getFirstProfile() { var accounts = Analytics.Management.Accounts.list(); if (accounts.getItems()) { var firstAccountId = accounts.getItems()[0].getId(); var webProperties = Analytics.Management.Webproperties.list(firstAccountId); if (webProperties.getItems()) { var firstWebPropertyId = webProperties.getItems()[0].getId(); var profiles = Analytics.Management.Profiles.list(firstAccountId, firstWebPropertyId); if (profiles.getItems()) { var firstProfile = profiles.getItems()[0]; return firstProfile; } else { throw new Error('No views (profiles) found.'); } } else { throw new Error('No webproperties found.'); } } else { throw new Error('No accounts found.'); } } function getReportDataForProfile(firstProfile) { var profileId = firstProfile.getId(); var tableId = 'ga:' + profileId; var startDate = "today"; // getLastNdays(14) 2 weeks (a fortnight) ago. var endDate = "today"; //getLastNdays(0) Today. var optArgs = { 'dimensions': 'ga:date,ga:hour,ga:minute,ga:sourceMedium', // Comma separated list of dimensions. 'sort': 'ga:date,ga:hour,ga:minute', // Sort by sessions descending, then keyword. //'segment': 'dynamic::ga:isMobile==Yes', // Process only mobile traffic. 'filters': 'ga:sourceMedium==yandex / cpc', 'start-index': '1', 'max-results': '10000' // Display the first 250 results. }; // Make a request to the API. var results = Analytics.Data.Ga.get( tableId, // Table id (format ga:xxxxxx). startDate, // Start-date (format yyyy-MM-dd). endDate, // End-date (format yyyy-MM-dd). 'ga:users', // Comma seperated list of metrics. optArgs); if (results.getRows()) { return results; } else { throw new Error('No views (profiles) found'); } } function getLastNdays(nDaysAgo) { var today = new Date(); var before = new Date(); before.setDate(today.getDate() - nDaysAgo); return Utilities.formatDate(before, 'GMT', 'yyyy-MM-dd'); } function outputToSpreadsheet(results) { var sheets = SpreadsheetApp.getActiveSpreadsheet(); var sheet = sheets.getSheetByName("coeff1"); var range = sheet.getRange('A:E'); range.clear(); // Print the headers. var headerNames = []; for (var i = 0, header; header = results.getColumnHeaders()[i]; ++i) { headerNames.push(header.getName()); } sheet.getRange(1, 1, 1, headerNames.length) .setValues([headerNames]); // Print the rows of data. sheet.getRange(2, 1, results.getRows().length, headerNames.length) .setValues(results.getRows()); } Step 3. Set the trigger to update the data every minute:
Figure 8. We request fresh data every minute to respond quickly to fraud.
Step 4. We create a pivot table from a sheet that is updated with the necessary data once a minute, and we analyze these indicators to set up triggers for email notifications or to disable ad groups using the Ya.Direct or AdWords API.
Figure 9. An example of setting up formulas for anomaly notificationsSample my Google Script code for email notifications
function myFunction() { var ss = SpreadsheetApp.getActiveSpreadsheet(); var sheet = ss.getSheetByName("notification"); var range = sheet.getRange("D2:E4"); // The row and column here are relative to the range // getCell(1,1) in this code returns the cell at B2, B2 var cell = range.getCell(1, 2); Logger.log(cell.getValue()); if (cell.getValue() !== "no") { MailApp.sendEmail("your_email@yandex.ru", "Fraud notification "+cell.getValue(), "Check me "+range.getCell(1, 1).getValue()); } else { } var cell2 = range.getCell(2, 2); Logger.log(cell2.getValue()); if (cell2.getValue() !== "no") { MailApp.sendEmail("your_email@yandex.ru", "Fraud notification "+cell2.getValue(), "Check me "+range.getCell(2, 1).getValue()); } else { } var cell3 = range.getCell(3, 2); Logger.log(cell3.getValue()); if (cell3.getValue() !== "no") { MailApp.sendEmail("your_email@yandex.ru", "Fraud notification "+cell3.getValue(), "Check me "+range.getCell(3, 1).getValue()); } else { } } Results: how to win the click
Counter-click fraud can be divided into three groups:
a) Proactive action:
- shutdown of "polluted sites";
- Disabling ad serving for an audience with signs that are fraudy for you, for example, for people on tablets from St. Petersburg (more advanced parameters for blocking can be used through AdWords lists and Metrics segments);
- adjusting bids for audience segments that are similar to fraud segments (“look-alike” segments are created in I. Audiences and Google Lists);
- fraud blocking by IP network masks (available only in AdWords).
b) Preventive actions:
- Send complaints about the return of the budget in AdWords and Direct;
- an investigation of "who ordered the attack on you";
- grouping suspicious and frequently attacked ad groups into a single advertising campaign;
- "Traps" for the simplest bots, namely the hidden buttons on the site, which are visible only to the bot and when clicked, it falls into the list.
c) Post factum actions:
- blocking by IP addresses;
- quick disconnection of click points: keywords, ad groups, advertising campaigns, audience segments.
Figure 10. Protection against click fraud
Useful links:
- Core Reporting API library ;
- Query Explorer for easy testing API commands;
- We write scripts to automate the work with Google applications ;
- Automate Google Sheets ;
- Analytics Intelligence , to ask the Analytics bot: - “But have there been any anomalies in the paid traffic over the last year?” And get a clear answer.
How to find out who ordered the attack on your ad
Any adequate competitor will minimize their damage when attacking others:
In our case, the competitor also began to click fraud in 4 directions in two cities, so it was not difficult to calculate it.
To make it easier to analyze competitors with whom you intersect, you can watch all the included ads of competitors for each keyword in the Yandex.Direct area:
Figure 9. All competitor ads by keyword
- firstly, the attacker will try not to show his ad in the directions in which the attack is taking place at the moment, so as not to merge his CTR and increase his cost per click;
- secondly, an unscrupulous competitor will select such keywords for an attack, on which he can stop displaying his ads without much harm to himself.
In our case, the competitor also began to click fraud in 4 directions in two cities, so it was not difficult to calculate it.
To make it easier to analyze competitors with whom you intersect, you can watch all the included ads of competitors for each keyword in the Yandex.Direct area:
Figure 9. All competitor ads by keyword
Those who are also faced with the click-through of contextual advertising - write in the comments, we will try to help each other!
Source: https://habr.com/ru/post/423335/
All Articles