Logging hits (payload) Google Analytics to Google Sheets via Google Tag Manager
Task: determine which events exceed the payload size of Google Analytics
Solution: logging Google Analytics hits (including payload size) to Google Sheets using Google Tag Manager, without the participation of developers
If you have been able to implement Enhanced Ecommerce for Google Analytics (GA) via Google Tag Manager (GTM) and then debug this business using the Google Analytics debugger , then you have probably come across the fact that some events "for some reason" do not reach GA and Error: Payload size is to large (9000). Max allowed is 8192
')
The fact is that the analytics.js library does not accept hits of more than 8192 bytes . If the hit size is larger, then it will not reach GA and will be empty in the event reports.
Example situation:
The web analyst asks the developer to push (or shoves) all product impressions in one hit. As a result, the hit does not reach. in the listing on one page more than 50 products. Or 50+ different products are added to the basket, as a result of which there are problems with the events of checkout step and transaction.
Try to always optimize the data structure of the hit (for example, send hits as the product appears in the user's field of view, do not push unnecessary variables into the hit (variant, category, brand), do not use long product names, etc.) This, firstly, will increase the speed of sending the hit; secondly, it will allow to avoid problems with payload size.
If it is impossible to optimize, then there are several ways to get around these limitations:
Before we start cutting hits and optimizing the content, we will determine which events exceed the payload size.
Create a new table.
In the header (1 line), write the names of the parameters that we want to extract from the hit (be careful, the names indicate exactly the same ones that you will then use in the JS script in GTM). As an example, we extract the following data:
Example of mapping columns for logging payload in Google Sheet
The order of the parameters in the columns is unimportant (except for the timestamp - it must be first). Columns with cid and ti parameters should be formatted as Plain text (Format> Number> Plain text), in order to avoid errors with auto-formatting.
If you wish, you can add / remove the necessary parameters (do not forget to change the list of variables in the JS script in GTM). List of possible fields and parameters analytics.js
Next, open the script editor and add the code (the original script belongs to Martin Hawksey https://gist.github.com/mhawksey/1276293 ):
Deploy the script as a web application (Publish> Deploy as web app ...). Permissions - anyone, even anonymous. We publish.
Google Script editor settings
In the future, we will need the URL of our web application (Current web app URL), so do not close the tab yet.
Create 2 custom JavaScript variables:
JavaScript variable example in GTM
The first JS variable , to determine the time of sending the hit (timestamp). This variable will be used in the second JS variable.
The second JS variable , for catching the necessary hits and transferring them to Google Sheet (the code from this material is taken as the basis).
What you need to configure:
Now, we find in our container a tag (tags) responsible for sending Enhanced Ecommerce events to GA and create a copy of them with binding to the corresponding triggers.
In the copy of the tags, we change the GA ID to any (test / fake) and add the sendHitTask field and the name JS variable (v_EE_mimic GA payload) to the Fields to Set.
sendHitTask, in this case, is modified i.e. we send hit data to the test GA (you can not send it, for this, delete in the code the value 'https://www.google-analytics.com/collect', ) and in Google Sheet (if it exceeds the specified payload size).
A copy of the tags with a test GA ID is needed for security, so as not to touch the original Enhanced Ecommerce tag. You can modify sendHitTask in the original tag (without creating copies), but then you cannot use customTask (you have to modify it by integrating sendHitTask into it) and there is a risk that the hit will not reach GA (I haven’t seen this, but better insure).
Enhanced Ecommerce tag settings in GTM
Save the tag, publish the new version of the GTM container.
Now, when the Enhance Ecommerce tag is triggered, the v_EE_mimic GA payload script will also work. If at the specified payload settings exceeds its values, this hit will be recorded in Google Sheet.
By collecting logs by hits, you can determine which specific event did not reach GA, where it happened, in which browser, etc. (see the list of possible fields and analytics.js parameters ).
Thanks for attention!
I hope this material will be useful to you and make life easier with the Google Analytics debug.
PS Who can / knows how to automate the verification of GTM tags (autotest for tags), please respond! Once upon a time, Simo wrote an article about this www.simoahava.com/analytics/automated-tests-for-google-tag-managers-datalayer but couldn’t understand it, can anyone try / know other ways. I would be very grateful for advice and assistance in this matter.
Solution: logging Google Analytics hits (including payload size) to Google Sheets using Google Tag Manager, without the participation of developers
The essence of the problem
If you have been able to implement Enhanced Ecommerce for Google Analytics (GA) via Google Tag Manager (GTM) and then debug this business using the Google Analytics debugger , then you have probably come across the fact that some events "for some reason" do not reach GA and Error: Payload size is to large (9000). Max allowed is 8192
')
Why it happens?
The fact is that the analytics.js library does not accept hits of more than 8192 bytes . If the hit size is larger, then it will not reach GA and will be empty in the event reports.
Example situation:
The web analyst asks the developer to push (or shoves) all product impressions in one hit. As a result, the hit does not reach. in the listing on one page more than 50 products. Or 50+ different products are added to the basket, as a result of which there are problems with the events of checkout step and transaction.
What do we have to do
Try to always optimize the data structure of the hit (for example, send hits as the product appears in the user's field of view, do not push unnecessary variables into the hit (variant, category, brand), do not use long product names, etc.) This, firstly, will increase the speed of sending the hit; secondly, it will allow to avoid problems with payload size.
If it is impossible to optimize, then there are several ways to get around these limitations:
- A beautiful solution to the problem with customTask. One by one, it removes unnecessary parameters from payload until it is less than 8192. You can customize what to throw out specifically.
- Cut the hit into several pieces and send it into pieces.
- GA data import (we push only product id into hit, the rest is loaded in the form of a database through data import).
Preparation for action
Before we start cutting hits and optimizing the content, we will determine which events exceed the payload size.
Step 1. Setting up Google Sheets
Create a new table.
In the header (1 line), write the names of the parameters that we want to extract from the hit (be careful, the names indicate exactly the same ones that you will then use in the JS script in GTM). As an example, we extract the following data:
- timestamp
- payLoadLength
- tid (tracking id)
- cid (client id)
- uid (user id)
- t (type of hit)
- pa (product action)
- ni (non interaction)
- dl (document location)
- dp (document path)
- dt (document title)
- ec (event category)
- ea (event action)
- el (event label)
- ti (transaction id)
- tr (transaction revenue)
Example of mapping columns for logging payload in Google Sheet
The order of the parameters in the columns is unimportant (except for the timestamp - it must be first). Columns with cid and ti parameters should be formatted as Plain text (Format> Number> Plain text), in order to avoid errors with auto-formatting.
If you wish, you can add / remove the necessary parameters (do not forget to change the list of variables in the JS script in GTM). List of possible fields and parameters analytics.js
Next, open the script editor and add the code (the original script belongs to Martin Hawksey https://gist.github.com/mhawksey/1276293 ):
function doGet(e){ return handleResponse(e); } function doPost(e){ return handleResponse(e); } function handleResponse(e) { var lock = LockService.getPublicLock(); lock.waitLock(30000); // wait 30 seconds before conceding defeat. try { // next set where we write the data - you could write to multiple/alternate destinations var ss = SpreadsheetApp.getActiveSpreadsheet(); var sheet = ss.getSheets()[0]; // we'll assume header is in row 1 but you can override with header_row in GET/POST data var headRow = e.parameter.header_row || 1; var headers = sheet.getRange(1, 1, 1, sheet.getLastColumn()).getValues()[0]; var nextRow = sheet.getLastRow()+1; // get next row var row = []; // loop through the header columns for (i in headers){ if (headers[i] == "timestamp"){ // special case if you include a 'timestamp' column row.push(Utilities.formatDate(new Date(), Session.getScriptTimeZone(), "MMM d yyyy HH:mm:ss")); } else { // else use header name to get data row.push(e.parameter[headers[i]]); } } // more efficient to set values as [][] array than individually sheet.getRange(nextRow, 1, 1, row.length).setValues([row]); // return json success results return ContentService .createTextOutput(JSON.stringify({"result":"success", "row": nextRow})) .setMimeType(ContentService.MimeType.JSON); } catch(e){ // if error return this return ContentService .createTextOutput(JSON.stringify({"result":"error", "error": e})) .setMimeType(ContentService.MimeType.JSON); } finally { //release lock lock.releaseLock(); } } Deploy the script as a web application (Publish> Deploy as web app ...). Permissions - anyone, even anonymous. We publish.
Google Script editor settings
In the future, we will need the URL of our web application (Current web app URL), so do not close the tab yet.
Step 2. Configure GTM
Create 2 custom JavaScript variables:
- v_EE_timestamp
- v_EE_mimic GA payload
JavaScript variable example in GTM
The first JS variable , to determine the time of sending the hit (timestamp). This variable will be used in the second JS variable.
function() { // Get local time as ISO string with offset at the end var now = new Date(); var tzo = -now.getTimezoneOffset(); var dif = tzo >= 0 ? ' Timezone: +' : ' Timezone: -'; var pad = function(num) { var norm = Math.abs(Math.floor(num)); return (norm < 10 ? '0' : '') + norm; }; return now.getFullYear() + '-' + pad(now.getMonth()+1) + '-' + pad(now.getDate()) + ' Time' + pad(now.getHours()) + ':' + pad(now.getMinutes()) + ':' + pad(now.getSeconds()) + dif + pad(tzo / 60) + ':' + pad(tzo % 60); } The second JS variable , for catching the necessary hits and transferring them to Google Sheet (the code from this material is taken as the basis).
function sendHitTask(){ return function(model) { var payLoad = model.get('hitPayload'); var trackingBaseUrls = ['https://www.google-analytics.com/collect', 'https://script.google.com/macros/s/AKfycbxJLy3eYBLpPu_S_eNccxzn_GwHXkZWr-93feMuBaAZelk3fj01yB/exec']; for (i = 0; i < trackingBaseUrls.length; i++) { var baseUrl = trackingBaseUrls[i]; if (trackingBaseUrls[i].indexOf('collect') > -1) { var req = new XMLHttpRequest(); req.open('POST', baseUrl, true); req.send(payLoad); } else if (payLoad.length > 7500){ var payLoadExtract = payLoad.split('&'); var payLoadArray = {}; // Push values to array for later access for (i = 0; i < payLoadExtract.length; i++){ var splitArray = payLoadExtract[i].split('='); payLoadArray[splitArray[0].trim()] = splitArray[1].trim(); } // Specify values to be sent to Google Sheets from array var tid = 'tid=' + payLoadArray.tid, cid = '&cid=' + payLoadArray.cid, uid = '&uid=' + payLoadArray.uid, t = '&t=' + payLoadArray.t, pa = '&pa=' + payLoadArray.pa, ni = '&ni=' + payLoadArray.ni, dl = '&dl=' + payLoadArray.dl, dp = '&dp=' + payLoadArray.dp, dt = '&dt=' + payLoadArray.dt, ec = '&ec=' + payLoadArray.ec, ea = '&ea=' + payLoadArray.ea, el = '&el=' + payLoadArray.el, ti = '&ti=' + payLoadArray.ti, tr = '&tr=' + payLoadArray.tr, timestamp = '×tamp=' + {{v_EE_timestamp}}, payLoadLength = '&payLoadLength=' + payLoad.length; var collectPayLoad = tid + cid + uid + t + pa + ni + dl + dp + dt + ec + ea + el + ti + tr + timestamp + payLoadLength; // Send Values to Google Sheets var collectUrl = baseUrl +'?'+ collectPayLoad; var myImage = new Image(); myImage.src = collectUrl; } } } } What you need to configure:
- We insert the url of your web application (created in step 1) and the url google analytics into the trackingBaseUrls variable.
- In payLoad.length determine the size of the hit that you want to catch. In the example,> 7500 is set. we catch everything that could potentially exceed the threshold of 8192 bytes. You can put less, if the whole log is interesting.
- In req.open , the hit sending method is defined (it can be POST or GET, depending on the hit size; Google recommends using POST since this allows you to send a larger payload). By default, GA events are sent via GET. For Enhanced Ecommerce I recommend changing the way to send to POST (done by adding to the Enhance Ecommerce tag in Fields to set: transport - beakon )
Now, we find in our container a tag (tags) responsible for sending Enhanced Ecommerce events to GA and create a copy of them with binding to the corresponding triggers.
In the copy of the tags, we change the GA ID to any (test / fake) and add the sendHitTask field and the name JS variable (v_EE_mimic GA payload) to the Fields to Set.
sendHitTask, in this case, is modified i.e. we send hit data to the test GA (you can not send it, for this, delete in the code the value 'https://www.google-analytics.com/collect', ) and in Google Sheet (if it exceeds the specified payload size).
A copy of the tags with a test GA ID is needed for security, so as not to touch the original Enhanced Ecommerce tag. You can modify sendHitTask in the original tag (without creating copies), but then you cannot use customTask (you have to modify it by integrating sendHitTask into it) and there is a risk that the hit will not reach GA (I haven’t seen this, but better insure).
Enhanced Ecommerce tag settings in GTM
Save the tag, publish the new version of the GTM container.
Total
Now, when the Enhance Ecommerce tag is triggered, the v_EE_mimic GA payload script will also work. If at the specified payload settings exceeds its values, this hit will be recorded in Google Sheet.
By collecting logs by hits, you can determine which specific event did not reach GA, where it happened, in which browser, etc. (see the list of possible fields and analytics.js parameters ).
Thanks for attention!
I hope this material will be useful to you and make life easier with the Google Analytics debug.
PS Who can / knows how to automate the verification of GTM tags (autotest for tags), please respond! Once upon a time, Simo wrote an article about this www.simoahava.com/analytics/automated-tests-for-google-tag-managers-datalayer but couldn’t understand it, can anyone try / know other ways. I would be very grateful for advice and assistance in this matter.
Source: https://habr.com/ru/post/358754/
All Articles