📜 ⬆️ ⬇️

Nuances and algorithms of programming engine for online marketing research

Good day, dear habravchane. It has long been tempting me to write such a manual, and now, I decided to force myself to sit down and write it - to share some experience that I gained during my programming research in the field of marketing and about some algorithms embedded in the engine, on which more than one project was implemented.

It would seem that what could be simpler than the TZ in the form of a spiked Word questionnaire for 30-40 questions, which the customer wants to see in the programmed form in the window of his browser, and he also gives four whole days to everything? As my friend used to say, until he began working with me:

- Yes, there is something for about four hours! I sat down in the morning, and by the evening everything was ready and tested. I do not understand why you are kolupaetsya ...
')
But everything, unfortunately, is far from being as simple as he imagined, and he programmed his first questionnaire in the next job for two days (followed by a twist for two or three days), constantly receiving new and new comments by phone and email. And the point is not in reassessing one's capabilities or professional insolvency (and comrade, I must say, with a rich experience of website builder), but in the field itself, which is very specific and has its pitfalls and subtle points. Just about some of these pitfalls and fine points, as well as how to avoid them, I would like to tell.

“I want red, but blue” or code modifiability


I don’t know why it happened, but for some reason in marketing, customers like to change their application requirements several times a day. Apparently, this is due to the internal organization, where the left hand does not always know what the right hand is doing and, at times, wants a completely different one. But this is not the essence, but the essence is that it all turns into a programmer’s headache (by the way, there was a case not so long ago when the final version of the CATI questionnaire was listed as “V.13.3_final_final”, and this is after the approval “ final "TZ).

The idea of ​​creating an engine that would ease the work, shortened the development time and minimized the influence of the human factor on possible errors, came somewhere after the fourth questionnaire and it was decided to program the engine immediately and for centuries. The basis was a naive idea of ​​the possibility of “programming” questionnaires without knowing this very programming and the ease of modifying existing projects.

The idea was, in essence, not so bad, but the key role was played by the lack of resources for implementation (man-hours) and the wrong approach: it was assumed that the questionnaire would be an Excel file whose columns would be responsible for certain commands for module constructor:

Resolved - done. But the first pancake, as usual, lumpy ...

Failure theory


The engine was tested on small projects, but after the sixth or seventh, it became clear that the chosen path of development was a dead end due to the immensely growing functionality. It got to the point that I and my partner, the developers, themselves have already become confused about what functionality is laid. The reasons:

There were other, but secondary reasons.

The solution was to create a wizard, which would form a template of a future questionnaire in the form of a “survey-response”, and the template would later be brought to mind in the built-in editor. But all this is a new man-clock, endless code modifications, constant updates and ... as a result, a kind of Frankenstein monster. So it was decided to abandon this approach.

The ideas of writing an internal scripting language, programming a C ++ daemon (due to wider possibilities), a parser for Word documents (written in a certain format), and many others were considered. But all of them were swept away either because of their absurdity, or because of the complexity of implementation. It was necessary to find some simple, but at the same time, a flexible solution ...

Problem statement and solution


A competent formulation of the problem is, if not a pledge, then a good help on the way to the successful realization of the idea.

Having already had enough experience in programming online research and faced with a bunch of pitfalls and nuances in this business, I identified several key (in my opinion) points that the system had to answer:

Also, in order to minimize possible difficulties, it was decided to follow a few simple rules:

As a result, the output was not a monolithic engine, but a complex from a module that displays questions to the user, as well as js and php libraries of functions.

It is time to paint in more detail some of the points I mentioned above:

Ability to connect various designs and change them "on the fly"

Customers often require the use of their own design in marketing research, even if they order the research itself from a third-party company. This is especially true for brands or when a link to the questionnaire is posted on the client’s site (to create the impression that the user has not left the site).

Because the overwhelming majority of polls saw one common point - information changes only in one area, the solution was to use two php functions: initialize the main variables when loading the questionnaire and display the necessary data in a div. It looks like this:
 <?php session_start(); include('./php/funcs.php'); //    if (!$_SESSION['user_ID']){ //         $user_id = md5(date('ymd H:i:s').rand()); $_SESSION['user_ID'] = $user_id; $_SESSION['user_position'] = setUserPosition($user_id); //      ,            ( "start") } else { $_SESSION['user_position'] = setUserPosition($_SESSION['user_ID']); //   ,        ,        } ?> ... // html- <div id="workDiv" class="workDiv" > <?php showQuest($_SESSION['user_position']); // ,      div' ?> </div> ... // html-    ("", "", "") <input type="button" style="width: 75px;" id="backBtn" onclick="request('back');" value=""> <input type="submit" style="width: 75px;" id="submitBtn" onclick="request('forvard');" value=""> <input type="submit" style="width: 75px; display: none;" id="endBtn" onclick="endOfSurvey();" value=""> 
<?php session_start(); include('./php/funcs.php'); // if (!$_SESSION['user_ID']){ // $user_id = md5(date('ymd H:i:s').rand()); $_SESSION['user_ID'] = $user_id; $_SESSION['user_position'] = setUserPosition($user_id); // , ( "start") } else { $_SESSION['user_position'] = setUserPosition($_SESSION['user_ID']); // , , } ?> ... // html- <div id="workDiv" class="workDiv" > <?php showQuest($_SESSION['user_position']); // , div' ?> </div> ... // html- ("", "", "") <input type="button" style="width: 75px;" id="backBtn" onclick="request('back');" value=""> <input type="submit" style="width: 75px;" id="submitBtn" onclick="request('forvard');" value=""> <input type="submit" style="width: 75px; display: none;" id="endBtn" onclick="endOfSurvey();" value="">

This is enough to meet the requirements of 3/4 of all possible variations of polls. For the remaining 1/4, nothing prevents you from adding a couple of tags / functions.

The problem with the design change is solved.

Maximum acceleration of the programming process

Due to the fact that everything is implemented in the form of a set of functions and the main tasks of data exchange with the server and their interpretation are solved and are on the engine, programming the questionnaires themselves is reduced to writing a file with logic, writing a js-script of correspondences to which question what kind of check and semi-automatically create question templates in the following form:
 <form name="form" method="post"> <b>  :</b> <br /><br /> <input type="radio" name="Z0" id="Z0" value="1" /> <div class="textAssociationDiv" >  </div> <br /> <input type="radio" name="Z0" id="Z0" value="2" /> <div class="textAssociationDiv" >  </div> <br /> </form> 
<form name="form" method="post"> <b> :</b> <br /><br /> <input type="radio" name="Z0" id="Z0" value="1" /> <div class="textAssociationDiv" > </div> <br /> <input type="radio" name="Z0" id="Z0" value="2" /> <div class="textAssociationDiv" > </div> <br /> </form>


There are several divs in the code with the class “textAssociationDiv” - this is used to associate a click on the text, like on a certain checkbox or radio button. If anyone is interested, here is the code of the js-function that does this:
 function associateTextToElement(){ $(".textAssociationDiv").click(function(){document.forms['form'].elements[$(".textAssociationDiv").index($(this)[0])].click();}); } 
function associateTextToElement(){ $(".textAssociationDiv").click(function(){document.forms['form'].elements[$(".textAssociationDiv").index($(this)[0])].click();}); }

Also, the code uses the same id, since This simplifies the analysis of page elements if more than one question is displayed on it. If it is necessary to display a question of the type of small-set (multiple choice), then the names and id take the following form: S9.1, S9.2, S9.3, S9.4, S9.5, S9.97, where the digit after the point - response code.

When the user clicks the “Next” or “Back” buttons, javascript collects all the entered data and Ajax sends them to the php script in the following format: variable_code = value. The variable code is taken from the id of the element, because they are the same in single-questions, and different in multi-sets - it is so easy. The script, receiving them by the POST method (it is important to hide such information from the user), writes them to a log file with the name stored in $ _SESSION ['user_ID'].

Using the file system to store data

There were several reasons for refusing to use the database, the main of which was the need to quickly add / remove variables in the survey process. Also, this approach solved the difficulties with the number of fields in the user response table when using MySQL (the total number of different variables in some studies sometimes reached one and a half to two thousand).

When using the same file system, all difficulties disappeared, because Absolutely all the data that was sent to the server was recorded in the corresponding file log. And in order to add several new answers, simply add a couple of lines with the necessary input'ami to the appropriate question file and ... wahl - all the data is saved, and the time spent on it was only a few seconds. For ease of interpretation of logs, both computer and human, they have the following format:

userID = 1cb3761177ec8846fe8ae35392017e36
userIP = 192.168.1.34
startTime = 04/19/10 12:50:21
S4 = 33
S5 = 3
S9_8 = 8 // so the small-set sets are recorded (S9.8, S9.10, etc.)

Bringing logic to a separate file

The problem with the recording of logic was solved by itself, when it was decided that the programmer was programming the questionnaire, and the manager only gave the TK — that is, you do not need to “simplify” someone’s life with a visual editor / wizard (complicating the life of a programmer), and the person with an idea of ​​js and php will work with the engine and libraries. Even it was not necessary to reinvent the wheel - when analyzing tons of implemented projects, it was noted that “direct” logic can satisfy all customer needs and to implement branching of any complexity, just a couple of functions, two variables, and an if operator are enough:



If we take into account the fact that in online questionnaires the polling vector always goes in one direction, and there are only two possible branching options from the computer point of view (either a condition has passed or not), then all the logic is simplified to the limit and reduces to the following format (php ):
 if ($_SESSION['user_position'] == 'S9' && checkAnswer('S9',1)){ //    S9     1-  $question_contents = file_get_contents('../qst/Z6.php'); echo iconv("cp1251", "UTF-8", $question_contents); //include_once('../qst/Z6.php'); //echo iconv("cp1251", "UTF-8", showQuest()); $_SESSION['user_position'] = 'Z6'; exit; } //   ,      php (,            ) if ($_SESSION['user_position'] == 'A1'){ //    A1 //$question_contents = file_get_contents('../qst/A3.php'); //echo iconv("cp1251", "UTF-8", $question_contents); include_once('../qst/A3.php'); echo iconv("cp1251", "UTF-8", showQuest()); $_SESSION['user_position'] = 'A3'; exit; } 
if ($_SESSION['user_position'] == 'S9' && checkAnswer('S9',1)){ // S9 1- $question_contents = file_get_contents('../qst/Z6.php'); echo iconv("cp1251", "UTF-8", $question_contents); //include_once('../qst/Z6.php'); //echo iconv("cp1251", "UTF-8", showQuest()); $_SESSION['user_position'] = 'Z6'; exit; } // , php (, ) if ($_SESSION['user_position'] == 'A1'){ // A1 //$question_contents = file_get_contents('../qst/A3.php'); //echo iconv("cp1251", "UTF-8", $question_contents); include_once('../qst/A3.php'); echo iconv("cp1251", "UTF-8", showQuest()); $_SESSION['user_position'] = 'A3'; exit; }

As can be seen from the above code, both options are almost identical and the conditions for all other checks look similar. Proceeding from this, we simply load the list of questions into the corresponding script in the order in which they are followed, and at the output we get the generated logic, which we will only have to correct a little with our hands (uncomment the relevant blocks in the conditions and add, if necessary, some additional functions and / or conditions). If you need to put a condition on a whole block of questions - this is also solved by if (for example, open question A1, and close A8).

Having such a file, we simply add to its beginning the function of recording the data transferred from js to the log and that's it. The logic is ready and the questions are replaced one by one. And the thing is that for the sake of simplicity, the request from js is sent straight to this script, which writes data, and then just continues to be executed until it meets a suitable condition, and the script will send the necessary content to js and complete the work. Js, in turn, inserts the received data into the aforementioned div with id = “workDiv” and the cycle repeats: the user marks the answers, js sends them to the logic file, php writes data to the log and returns the contents of the next question, js replaces the contents of the working div with the received data .

After receiving data from the server and replacing the contents of the working div, js performs another function — it calls the binding function to check the validity of the question.

Js-check filling questions

Most of the questions that are used in online research can be divided into several types:

Based on this, a library of functions was written that automatically bind the corresponding checks on each of the input elements.

When you first enter the questionnaire, as well as updating the block with a question, the questsChecks () js function is called, the essence of which, in brief, is as follows: block the Next button (this is a new question and the data has not yet been entered) From the php script, the value of $ _SESSION ['user_position'] (i.e. the current question), run the switch:
 switch(quest){ case 'S2': disableNext(); radioMultCheck(); progressBar(0); break; case 'S4': disableNext(); checkAgeText(10, 99); progressBar(7); break; case 'S12': disableNext(); radioTextCheck(); progressBar(14); break; // ...    default: break; } 
switch(quest){ case 'S2': disableNext(); radioMultCheck(); progressBar(0); break; case 'S4': disableNext(); checkAgeText(10, 99); progressBar(7); break; case 'S12': disableNext(); radioTextCheck(); progressBar(14); break; // ... default: break; }


Depending on the code of the question, the corresponding library function is called for us, which hangs up checks. Here is an example of one of them (multiple choice + text) and two related ones:
 // ----------------------------------------------------------------------------- function checkboxText(check,text){ disableNext(); if (!check){ $("div[id='workDiv'] input[type='checkbox']").click(function(){checkboxText('true');}); $("div[id='workDiv'] input[type='text']").keyup(function(){checkboxText('true',this);}); } if (check){ if (!text && $("div[id='workDiv'] input[type='text'][value!='']").length > 0) enableNext(); if (!text && $("div[id='workDiv'] input[type='text'][value!='']").length == 0) checkCheckbox(); if (text && !checkCheckbox()){ checkText(text); } } } // ----------------------------------------------------------------------------- function checkCheckbox(uniqCodes){ if (!uniqCodes){ uniqText = getUniqTextObj(); if ($("div[id='workDiv'] input[type='checkbox'][name!='toggleOther']:checked").length > 0){ enableNext(); return true; } if ($("div[id='workDiv'] input[type='checkbox']:checked").length == 0 && !uniqText) disableNext(); if ($("div[id='workDiv'] input[type='checkbox']:checked").length == 0 && uniqText) checkText(uniqText); } if (uniqCodes){ for (var key in uniqCodes){ $(document.getElementById(getQuest()+'.'+uniqCodes[key])).attr("checked", ""); } checkCheckbox(); } } // ----------------------------------------------------------------------------- function checkText(text,uniqCodes){ if (!uniqCodes){ if (text.value) enableNext(); else disableNext(); } if (uniqCodes){ for (var key in uniqCodes){ $(document.getElementById(getQuest()+'.'+uniqCodes[key])).attr("checked", ""); } if (!checkCheckbox()) checkText(text); } } 
// ----------------------------------------------------------------------------- function checkboxText(check,text){ disableNext(); if (!check){ $("div[id='workDiv'] input[type='checkbox']").click(function(){checkboxText('true');}); $("div[id='workDiv'] input[type='text']").keyup(function(){checkboxText('true',this);}); } if (check){ if (!text && $("div[id='workDiv'] input[type='text'][value!='']").length > 0) enableNext(); if (!text && $("div[id='workDiv'] input[type='text'][value!='']").length == 0) checkCheckbox(); if (text && !checkCheckbox()){ checkText(text); } } } // ----------------------------------------------------------------------------- function checkCheckbox(uniqCodes){ if (!uniqCodes){ uniqText = getUniqTextObj(); if ($("div[id='workDiv'] input[type='checkbox'][name!='toggleOther']:checked").length > 0){ enableNext(); return true; } if ($("div[id='workDiv'] input[type='checkbox']:checked").length == 0 && !uniqText) disableNext(); if ($("div[id='workDiv'] input[type='checkbox']:checked").length == 0 && uniqText) checkText(uniqText); } if (uniqCodes){ for (var key in uniqCodes){ $(document.getElementById(getQuest()+'.'+uniqCodes[key])).attr("checked", ""); } checkCheckbox(); } } // ----------------------------------------------------------------------------- function checkText(text,uniqCodes){ if (!uniqCodes){ if (text.value) enableNext(); else disableNext(); } if (uniqCodes){ for (var key in uniqCodes){ $(document.getElementById(getQuest()+'.'+uniqCodes[key])).attr("checked", ""); } if (!checkCheckbox()) checkText(text); } }

I will not comment on each line, because this sweeps up too much text, but I will briefly explain the algorithm: we attach a call to the same function for each element of type input at the click () or keyup () event, but with the “check” parameter and object transfer (for text fields). When calling a check (when clicking), the function runs over all elements within the working div in accordance with a predefined algorithm (which is based on the type of question), collects field values, analyzes them and, if required, calls for more specialized check functions.

In this example, the multi-choice validation function with text may trigger the launch of the simple multiple-choice validation function, since simple multiple choice is a simplified version of the multiple choice with the text and is included in it. All verification functions of primitive types (these include: multiple choice, single choice, open-ended question) are designed in such a way that only types of primitive fields are checked. Thus, by automatically sorting out complex types of questions into simple ones and performing checks on them, we get the desired result - true or false - whether the test passed or not. And, if so, remove the lock from the Next button, if not, we do nothing, or we block it if it is available.

If the check is successful, a new cycle begins: sent, received, filled, checked, sent, etc.

Using structured programming

Using bare functions, instead of classes and objects, in this case (in my opinion) is more preferable in terms of performance. With a large flow of users, creating an object with a bunch of properties just for the sake of using one specific method is, at least, not reasonable. Manipulations with five to ten objects are more costly in terms of resources than similar actions with functions.

Besides, due to the simplicity of the engine architecture, in the OOP, I simply don’t see the point.

Summing up



This engine architecture worked out with a bang in one of the studies, in which the number of users who received invitations to participate was about 120-130 thousand people, with loyal users being customers of the customer, and the mailing itself was on behalf of the customer - while about 22k successful interviews, and the total number of visits was about 45k.

Smaller studies have been carried out a great many.

Statistics say that this architecture is reliable and highly productive. In addition, it is also convenient (in my opinion) and quite flexible.

If there is enough free time, I may still return to the revision of this engine and, again, maybe I will put all the sources in the public domain, along with a detailed description of the functionality and all sorts of buns. But all this is only “possible” ...

At one time, when I was just beginning to dive into this area, the lack of information on the topic was catastrophic and all the tasks had to be solved, often with a trial and error method.

I hope that this manual and my personal experience, which I reflected in it, somehow helped you.

And I also hope that this material will somehow fill the informational vacuum around a topic such as programming online questionnaires in marketing research.

But, unfortunately, such topics as quota programming, conditioner, or programming of questionnaires and / or the engine for CATI (actually, as well as the organization of the technical base of CATI-studio itself) are not covered. And much more…

But these are all topics for individual FAQs and manuals.

Thanks for attention.

Source: https://habr.com/ru/post/96343/


All Articles