Standard exchange 1C-Bitrix on BASH: Detailed parsing of the script of incremental unloading
To ensure synchronization of the product catalog between the 1C system and the site on the Bitrix management system, the standard for Bitrix is the XML file exchange protocol in CommerceML format, based on the transfer from 1C to the Bitrix HTTP GET and POST requests with certain parameters, and receiving standard responses containing operations, denoting the result of its implementation.
In the first article of this series , the rationale is given for the possibility of using a separate script that takes XML files generated by 1C or another system or program, and transmits them to Bitrix using a standard protocol.
In this article I will give detailed comments on each line of the script. This will simplify its modification for your needs.
')
The script is written in BASH and is one of several scripts providing various exchanges through the same standard that 1C-Bitrix offers for transferring goods from 1C and downloading orders from a site built on the basis of the Bitrix management system.
Next comes the text of the script with comments. A brief view of the script, containing only the code, is given in the previous article , and is available on github of the bitrexchange project.
In order to prevent the transfer of empty or broken files, to prevent data corruption, if any error we terminate the script.
This is such an interesting method of switching to the current directory, and at the same time saving the full current path to the variable
Usually, 1C uploads files to some shared folder on the server on the network, which we will pre-mount. But it can be a local directory for you, which you share on the network.
The name of the zip file in which the xml files will be packed. Instead, you can use just a temporary name. But this hard-coded file name is used to save copies of previously transferred files. Very convenient: you can see at any time what day a product appeared and at what price, etc.
The names of the files that we will transfer. These files form 1C, they usually have such names, but there may be others! Need to check it out! Files are listed through a space.
Administrator mail to which error messages are sent.
Determine the current time:
This script is a special case of all the numerous types of downloads. In addition to uploading the catalog, which we are doing now, there are other exchanges for which there are separate scripts. But within the framework of this unloading - unloading of the catalog, two types are defined: the so-called incremental unloading, that is, unloading changes, and full unloading. In format, they differ from each other only in one parameter: attribute value
- it is either true or false. Speaking about the contents of XML files, we can say that incremental uploading can contain a full set of products, the difference is different: if the bitrix receives a full download, it completely erases the entire existing catalog of goods and fills it again. All product pictures are also lost, so full uploading necessarily includes all images, while incremental may not include. And of course, if it is important to save any bindings you create in the site database, then it is better not to delete the existing directory. In general, one way or another, there is a huge, decisive difference, which type of unloading to use. Therefore, we will definitely check, by no means trusting 1C, that this upload is really incremental, and check it in both files. At the same time, we will verify that these files exist at all, and that there are two of them. Sometimes, for example, 1C for some reason only unloads the number plate, and it may happen that the ball fell off and the files did not upload at all. Therefore, in case of a problem, we inform ourselves about it and exit.
We start the actual work: we pack the files and make the archive of the previous file with the current date. Please note that this command packs all xml files in the directory.
Set the HTTP request headers. Actually, this is, firstly, completely optional, and secondly, you can specify some of your own, for example, the user-agent, by which you can then catch these requests in the Apache logs on the receiving side.
Username and password of the user under which the download will be logged. This is usually 1c_import or import. These username and password are set in the Bitrix control panel! For the test, you can use your administrator account, admin.
In order not to clutter up the script, we will place all requests in a separate variable base url. usually this is your domain / bitrix / admin / 1c_exchange.php
Since the documentation is very (!) Superficial, and the specific methods of using the protocol differ from the 1C configuration, the exchange method used is verified and verified experimentally and works without changes on several working projects, which confirms the possibility of its mass use.
According to the exchange protocol, requests of several types are transmitted sequentially. The type of request is determined by the parameter that is added to the base url, which we have already specified above as baseurl. Two variables in the url specify the type of request. The first, type, takes the values either sale or catalog. You can try to change it, at least for the first query, and always use catalog. The second, more important for us, as it defines the stage of exchange, in the variable mode.
First request, mode = checkauth
The second request, mode = init
The third request, mode = file + POST zip file.
Subsequent requests, mode = import
We login using HTTP protocol authentication! Usually, this method is included in the bitrix, but if not, check the server settings. As far as I know, there are no settings for turning this method on or off in the bitrix, but authentication can be turned off programmatically in many ways. The so-called “proactive protection” can also interfere, if you cannot login in any way, try disabling it for the duration of the test.
According to the protocol, in case of a successful entry, the bitrix gives the answer in the form of three lines in the body of the HTTP response:
The first is the success string.
The second is a session variable, usually PHPSESSID
The third is the session value, a string.
If sussecc is received, save the session variable and its value for further use, if not, exit the script. At this moment in case of an error, we can get an HTML page that you can save and somehow view, and, most likely, you will find in it a message that you need to login. At this stage - most likely - check the username and password of the user, try, first of all, to log in with the username and password into the control panel of the bitrix.
The second request is Init. When you receive this request, the / upload / 1c_catalog / directory is completely cleared (!) Be careful if you need any data from the previous upload. For example, it will be important if you transfer pictures of goods. 1C version 8.3, as it was discovered, for some reason this request does not transmit, but goes straight to the file request. At this stage, an error may occur, which will be described later in the appendix; see possible errors (1).
In the third request, you need to send a zip file with a POST request. mode should be = file, but the file name is the filename parameter — as I understand it, it can be any, because it does not appear anywhere else in the request, and specifies only the name under which the transmitted data is saved as a file (the file is transmitted via A POST request and, therefore, in the context of HTTP, this is not a file but just some piece of data that can be anything).
At this step in the upload / 1c_catalog directory is the import.zip file, unpacked. Interestingly, no further requests for it need to be sent. As soon as there is a first import request, it will be unpacked.
Further, according to the exchange protocol 1C-Bitrix, you need to send requests with mode = file, until the success string is received. In fact, you need to check not success, but the fact that every request has a response progress, and if it is, then you can continue the import, and if something else came, then the import ended for some reason, one of which maybe success. One of the many undocumented moments is that such a cycle of requests must be repeated for each unpacked file, and the name of the zip file should not be specified anywhere else, since it is unpacked immediately upon receipt. See also appendix, possible errors (2).
After this cycle, you need to check that the last message = success, and otherwise, notify yourself again. This can be done within the framework of this script, or in some other way, for example, check the log.
According to the documentation, the last request should be type = success, 1C sends this request, but this script does without it.
On the second request, an “failure” error is issued. The component should output the word “failure” in error and further the description of the error. But, apparently, can not determine the language when requesting, and therefore simply failure without detailing the error is given. Although there must be some kind of default language. Perhaps the reason is something else, but the fact is that there is a failure and that's all. The source of the error cannot be determined. Solved, as always, by climbing into the code.
You can disable the check of this parameter directly in the code, you can set it to N:
Settings - Tools - PHP command line.
An obvious mistake: at each stage two lines are given: the first is progress, and the second is a description of what is being done. So, there must necessarily be progress blocks. Processed XXXX from YYYYY elements.
- This is "processed so many of so many elements" - this is the import. if not, then the import is not done. such cases were in certain configurations.
Standard exchange 1C-Bitrix for BASH: The first article in a series of publications .
In the first article of this series , the rationale is given for the possibility of using a separate script that takes XML files generated by 1C or another system or program, and transmits them to Bitrix using a standard protocol.
In this article I will give detailed comments on each line of the script. This will simplify its modification for your needs.
')
The script is written in BASH and is one of several scripts providing various exchanges through the same standard that 1C-Bitrix offers for transferring goods from 1C and downloading orders from a site built on the basis of the Bitrix management system.
Next comes the text of the script with comments. A brief view of the script, containing only the code, is given in the previous article , and is available on github of the bitrexchange project.
Script and local file settings
In order to prevent the transfer of empty or broken files, to prevent data corruption, if any error we terminate the script.
set -e This is such an interesting method of switching to the current directory, and at the same time saving the full current path to the variable
cd $(dirname $0) cdir=$(pwd)"/" Usually, 1C uploads files to some shared folder on the server on the network, which we will pre-mount. But it can be a local directory for you, which you share on the network.
remote_dir="/mnt/localwinserver_fs/import/webdata/" The name of the zip file in which the xml files will be packed. Instead, you can use just a temporary name. But this hard-coded file name is used to save copies of previously transferred files. Very convenient: you can see at any time what day a product appeared and at what price, etc.
zip_fname="catalogue.zip" The names of the files that we will transfer. These files form 1C, they usually have such names, but there may be others! Need to check it out! Files are listed through a space.
xml_files="import0_1.xml offers0_1.xml" Administrator mail to which error messages are sent.
email1="admin@yourinternetshop.com"; email2="alert@yourinternetshop.com"; Determine the current time:
ctime=$(date +%Y-%m-%d-%H%M) Check and prepare files
This script is a special case of all the numerous types of downloads. In addition to uploading the catalog, which we are doing now, there are other exchanges for which there are separate scripts. But within the framework of this unloading - unloading of the catalog, two types are defined: the so-called incremental unloading, that is, unloading changes, and full unloading. In format, they differ from each other only in one parameter: attribute value
- it is either true or false. Speaking about the contents of XML files, we can say that incremental uploading can contain a full set of products, the difference is different: if the bitrix receives a full download, it completely erases the entire existing catalog of goods and fills it again. All product pictures are also lost, so full uploading necessarily includes all images, while incremental may not include. And of course, if it is important to save any bindings you create in the site database, then it is better not to delete the existing directory. In general, one way or another, there is a huge, decisive difference, which type of unloading to use. Therefore, we will definitely check, by no means trusting 1C, that this upload is really incremental, and check it in both files. At the same time, we will verify that these files exist at all, and that there are two of them. Sometimes, for example, 1C for some reason only unloads the number plate, and it may happen that the ball fell off and the files did not upload at all. Therefore, in case of a problem, we inform ourselves about it and exit.
chan=$(grep -e "=\"true" ${remote_dir}*.xml | wc -l) if test "$chan" != "2"; then echo "Error: XMLs are not in 'changes only' mode or file(s) are missing" mail -s " " -a "From: bitrexchange <${email1}>" $email1,$email2 <<< " ." exit 1 else echo "OK: Format of XMLs are 'changes only'" fi We start the actual work: we pack the files and make the archive of the previous file with the current date. Please note that this command packs all xml files in the directory.
if [ -f $zip_fname ]; then mv $zip_fname "${zip_fname}.${ctime}"; fi /usr/bin/zip -9j "$zip_fname" ${remote_dir}*.xml HTTP request settings
Set the HTTP request headers. Actually, this is, firstly, completely optional, and secondly, you can specify some of your own, for example, the user-agent, by which you can then catch these requests in the Apache logs on the receiving side.
headers="--header=\"User-Agent: 1C+Enterprise/8.2\" --header=\"Accept-Encoding: deflate\"" Username and password of the user under which the download will be logged. This is usually 1c_import or import. These username and password are set in the Bitrix control panel! For the test, you can use your administrator account, admin.
login="import" password="yourpasswordonbitrix" In order not to clutter up the script, we will place all requests in a separate variable base url. usually this is your domain / bitrix / admin / 1c_exchange.php
baseurl="http://yourinternetshop.com/bitrix/admin/1c_exchange.php" Proper exchange
Since the documentation is very (!) Superficial, and the specific methods of using the protocol differ from the 1C configuration, the exchange method used is verified and verified experimentally and works without changes on several working projects, which confirms the possibility of its mass use.
According to the exchange protocol, requests of several types are transmitted sequentially. The type of request is determined by the parameter that is added to the base url, which we have already specified above as baseurl. Two variables in the url specify the type of request. The first, type, takes the values either sale or catalog. You can try to change it, at least for the first query, and always use catalog. The second, more important for us, as it defines the stage of exchange, in the variable mode.
First request, mode = checkauth
The second request, mode = init
The third request, mode = file + POST zip file.
Subsequent requests, mode = import
First request
We login using HTTP protocol authentication! Usually, this method is included in the bitrix, but if not, check the server settings. As far as I know, there are no settings for turning this method on or off in the bitrix, but authentication can be turned off programmatically in many ways. The so-called “proactive protection” can also interfere, if you cannot login in any way, try disabling it for the duration of the test.
ret_line=$( wget $headers --user=${login} --password=${password} --auth-no-challenge -O - -q "${baseurl}?type=sale&mode=checkauth" ) read -a ret_ar <<< $ret_line According to the protocol, in case of a successful entry, the bitrix gives the answer in the form of three lines in the body of the HTTP response:
The first is the success string.
The second is a session variable, usually PHPSESSID
The third is the session value, a string.
If sussecc is received, save the session variable and its value for further use, if not, exit the script. At this moment in case of an error, we can get an HTML page that you can save and somehow view, and, most likely, you will find in it a message that you need to login. At this stage - most likely - check the username and password of the user, try, first of all, to log in with the username and password into the control panel of the bitrix.
if [ ${ret_ar[0]} != "success" ]; then echo "Login error\r\n"; exit -1; fi sessvar=${ret_ar[1]} sessid=${ret_ar[2]} echo sessid=$sessid Second request
The second request is Init. When you receive this request, the / upload / 1c_catalog / directory is completely cleared (!) Be careful if you need any data from the previous upload. For example, it will be important if you transfer pictures of goods. 1C version 8.3, as it was discovered, for some reason this request does not transmit, but goes straight to the file request. At this stage, an error may occur, which will be described later in the appendix; see possible errors (1).
ret=$(wget $headers --header="Cookie: ${sessvar}=${sessid}" -O - -q "${baseurl}?type=catalog&mode=init"); echo $ret Third request
In the third request, you need to send a zip file with a POST request. mode should be = file, but the file name is the filename parameter — as I understand it, it can be any, because it does not appear anywhere else in the request, and specifies only the name under which the transmitted data is saved as a file (the file is transmitted via A POST request and, therefore, in the context of HTTP, this is not a file but just some piece of data that can be anything).
At this step in the upload / 1c_catalog directory is the import.zip file, unpacked. Interestingly, no further requests for it need to be sent. As soon as there is a first import request, it will be unpacked.
ret=$(wget $headers --post-file ${zip_fname} --header="Cookie: ${sessvar}=${sessid}" -O - -q "${baseurl}?type=catalog&mode=file&filename=import.zip"); echo $ret Import requests
Further, according to the exchange protocol 1C-Bitrix, you need to send requests with mode = file, until the success string is received. In fact, you need to check not success, but the fact that every request has a response progress, and if it is, then you can continue the import, and if something else came, then the import ended for some reason, one of which maybe success. One of the many undocumented moments is that such a cycle of requests must be repeated for each unpacked file, and the name of the zip file should not be specified anywhere else, since it is unpacked immediately upon receipt. See also appendix, possible errors (2).
for fname in $xml_files; do st="progress"; while [ "$st" = "progress" ]; do ret=$(wget $headers --header="Cookie: ${sessvar}=${sessid}" -O - -q "${baseurl}?type=catalog&mode=import&filename=${fname}"); st=$( <<< "$ret" head -n1 | cut -c1-8); echo "$ret" | iconv -f cp1251 -t utf-8 done done After this cycle, you need to check that the last message = success, and otherwise, notify yourself again. This can be done within the framework of this script, or in some other way, for example, check the log.
Last request
According to the documentation, the last request should be type = success, 1C sends this request, but this script does without it.
Application:
Possible mistakes (1)
On the second request, an “failure” error is issued. The component should output the word “failure” in error and further the description of the error. But, apparently, can not determine the language when requesting, and therefore simply failure without detailing the error is given. Although there must be some kind of default language. Perhaps the reason is something else, but the fact is that there is a failure and that's all. The source of the error cannot be determined. Solved, as always, by climbing into the code.
COption::GetOptionString("sale", "secure_1c_exchange", "N") == "Y" You can disable the check of this parameter directly in the code, you can set it to N:
Settings - Tools - PHP command line.
COption::SetOptionString("catalog", "DEFAULT_SKIP_SOURCE_CHECK", "Y" ); COption::SetOptionString("sale", "secure_1c_exchange", "N" ); Possible mistakes (2)
An obvious mistake: at each stage two lines are given: the first is progress, and the second is a description of what is being done. So, there must necessarily be progress blocks. Processed XXXX from YYYYY elements.
- This is "processed so many of so many elements" - this is the import. if not, then the import is not done. such cases were in certain configurations.
Standard exchange 1C-Bitrix for BASH: The first article in a series of publications .
Source: https://habr.com/ru/post/321266/
All Articles