Access to large files and various external dynamic data is often a very important part of a decentralized application. At the same time, Ethereum does not provide for the mechanism of circulation to the outside itself - smart contracts can read and write only within the framework of the blockchain itself. In this article we will consider Oraclize, which just gives the opportunity to interact with the outside world by querying virtually any Internet resources. A related topic is IPFS, briefly mentioning it.
IPFS
IPFS is a distributed file system with content addressing. This means that the content of any file added there is considered a unique hash. The same hash is then used to search and retrieve this content from the network.
The main information is already described in
this article and in several others, so there is no point in repeating.
Why use IPFS in conjunction with Ethereum?
Any volumetric content stored on the blockchain is too expensive and harmful to the network. Therefore, the best option is to save any link to the file that is located in the off-line repository, not necessarily IPFS. But IPFS has several advantages:
')
- A file reference is a hash that is unique for a specific file content, so if we put this hash on the blockchain, then we can be sure that the file received from it is the one that was originally added, the file cannot be changed.
- Distributed system insures against the inaccessibility of a specific server (due to blocking or other reasons)
- The link to the file and the hash confirmation are combined into one line, which means you can write less to the blockchain and save gas
Among the shortcomings we can mention that since there is no central server, then for the availability of the files it is necessary that at least one of these files be “distributed”. But if you have a specific file, then it is easy to connect to the distributors — launch the ipfs daemon and add the file via
ipfs add
.
The technology is very well suited to the ideology of decentralization; therefore, considering Oraclize now, we will more than once encounter the use of IPFS in different oracle mechanisms.
Oraclize
To perform almost any useful work, a smart contract needs to receive new data. However, there is no built-in ability to execute a request from the blockchain to the outside world. You can of course add everything that is required by transactions manually, but it is impossible to verify where this data came from and their accuracy. Plus, you may need to organize additional infrastructure for the rapid updating of dynamic data, such as exchange rates. And updates with a fixed interval will lead to gas overspending.
Therefore, the service provided by
Oraclize comes in handy: in a smart contract you can send a request to almost any API or resource on the Internet, be sure that the data came from the specified resource unchanged, and use the result in the same smart contract.
Oraclize is not only an Ethereum service, similar functionality is provided to other blockchains, but we will only describe a bundle with Ethereum.
Beginning of work
All you need to get started is to add yourself to the project one of the oraclizeAPI files from the
repository . You only need to choose the appropriate compiler for your version (solc): oraclizeAPI_0.5.sol for versions starting at 0.4.18, oraclizeAPI_0.4.sol for versions from 0.4.1, oraclizeAPI_pre0.4.sol for everything older, support This version has already been discontinued. If you use truffle, do not forget to rename the file to usingOraclize - it requires that the name of the file and the contract match.
By including the appropriate file in your project, inherit the contract from
usingOraclize
. And you can start using Oracle, which boils down to two main things: sending a request using the
oraclize_query
helper, and then processing the result in the
__callback
function. The simplest smart contract (to get the current price of the broadcast in dollars) may look like this:
pragma solidity 0.4.23; import "./usingOraclize.sol"; contract ExampleContract is usingOraclize { string public ETHUSD; event updatedPrice(string price); event newOraclizeQuery(string description); function ExampleContract() payable { updatePrice(); } function __callback(bytes32 myid, string result) { require (msg.sender == oraclize_cbAddress()); ETHUSD = result; updatedPrice(result); } function updatePrice() payable { if (oraclize_getPrice("URL") > this.balance) { newOraclizeQuery("Oraclize query was NOT sent, please add some ETH to cover for the query fee"); } else { newOraclizeQuery("Oraclize query was sent, standing by for the answer.."); oraclize_query("URL", "json(https://api.coinmarketcap.com/v1/ticker/ethereum/?convert=USD).0.price_usd"); } } }
The function that sends the request is
updatePrice
. You can see that first there is a check that
oraclize_getPrice(“URL”)
greater than the current balance of the contract. This is done because the call to
oraclize_query
has to be paid, the price is calculated as the sum of the fixed commission and the payment for the gas to call the callback.
“URL”
is the designation of one of the types of data sources, in this case it is a simple https request, then we will consider other options. Answers on request can be pre-parsed as json (as in the example) and in several other ways (we will look further). In
__callback
, a string with the answer is returned. At the very beginning, it is checked that the call passed from the trusted address oraclize
All options for using oraclize are based on the same scheme, only data sources and the ability to add authentication to
__callback
. Therefore, in future examples, we will cite only significant differences.
Price of use
As already mentioned, additional air is paid for oraclize requests, and it is removed from the balance of the contract, and not the calling address. The only exception is the first request from each new contract, it is provided free of charge. It is also interesting that in test networks the same mechanics is preserved, but the payment goes on the air of the corresponding network, that is, in testnets, the requests are actually free.
It has already been mentioned that the request price is made up of two quantities: a fixed commission and a callback call charge. The fixed commission is determined in dollars, and the amount of air is calculated from the current rate. The Commission depends on the source of the data and additional supporting mechanisms, which we will dwell on. The current price table looks like this:
As you can see, the price per URL request is a few cents. Is it a lot or a little? To do this, let's look at how much the second part costs - gas for a call to a callback.
This works according to the following scheme: the amount of air needed to pay for a fixed amount of gas at a fixed price is transferred in advance from the contract together with the request. This amount should be enough to fulfill the callback, and the price should be adequate to the market, otherwise the transaction will not work or will hang for a very long time. At the same time, it is clear that it is not always possible to know the amount of gas in advance, and therefore the fee must be in stock (the stock is not returned). The default values ​​are 200 thousand gas limit at 20 gwei. This is enough for an average callback with several entries and some kind of logic. Although the price of 20 gwei may seem too high at the moment (at the time of writing, the average is 4 gwei), but at the moments of the influx of transactions, the market price may suddenly jump and be even higher, therefore, in general, these values ​​are close to the actual ones. So, with such values ​​and the price of air in the region of $ 500, payment for gas will approach $ 2, so we can say that the fixed commission takes an insignificant part.
If you know what you are doing, then there is an option to change the limit and price of gas, thus significantly saving on requests.
The gas price can be set by a separate function -
oraclize_setCustomGasPrice(< wei>)
. After the call, the price is saved and used in all subsequent requests.
The limit can be set in the
oraclize_query
request
oraclize_query
, specifying it with the last argument, for example:
oraclize_query("URL", "<>", 50000);
If you have complicated logic in
__callback
and gas is spent more than 200k, then you will definitely need to set a limit that covers the worst case of gas consumption. Otherwise, if the limit is exceeded,
__callback
simply roll back.
By the way, recently, oraclize has information that requests can be paid outside the blockchain, which will allow you not to spend the entire limit or return the balance (and the payment is not from the contract). We have not had to use it yet, but oraclize offers to contact them at info@oraclize.it, if this option is interesting. So keep in mind.
How does it work
Why, inheriting from the usual smart contract, we get functionality that was not originally supported by the blockchain mechanisms? Actually service OrakLayz consists not only of contracts with functions-helpers. The main job of getting data is external service. Smart contracts form applications for access to external data and put them in the blockchain. External service - monitors new blocks of the blockchain, and if it detects an application - executes it. Schematically, this can be represented as:
Data sources
In addition to the considered
URL
, oraclize provides 4 more options (which you saw in the section on prices):
WolframAlpha
,
IPFS
,
random
and
computation
. Consider each one of them.
1. URL
The example already considered uses this data source. This is the source for HTTP requests to various APIs. In the example was the following:
oraclize_query("URL", "json(https://api.coinmarketcap.com/v1/ticker/ethereum/?convert=USD).0.price_usd");
This is the receipt of the price of the broadcast, and since api provides the json string with the data set, the request is wrapped in a json parser and returns only the field we need. In this case, this is GET, but the source URL also supports POST requests. The type of request is automatically determined by the optional argument. If there is a valid json there as in this example:
oraclize_query("URL", "json(https://shapeshift.io/sendamount).success.deposit", '{"pair":"eth_btc","amount":"1","withdrawal":"1AAcCo21EUc1jbocjssSQDzLna9Vem2UN5"}')
then the request is processed as POST (the api used is described
here , if interested)
2. WolframAlpha
This data source allows you to access the
WolframAlpha service, which can provide answers to various requests for facts or calculations, for example
oraclize_query(“WolframAlpha”, “president of Russia”)
will return
Vladimir Putin
, and the request
oraclize_query(“WolframAlpha”, “solve x^2-4”)
returns
x = 2
.
As you can see, the result was incomplete, because the symbol ± was lost. Therefore, before using this source, you need to check that the value of a specific request can be used in a smart contract. In addition, authentication is not supported for answers, therefore oraclize themselves recommend using this source only for testing.
3. IPFS
As you can guess, allows you to get the contents of the file in IPFS on the multihash. The timeout for receiving content is 20 seconds.
oraclize_query(“IPFS”, “QmTL5xNq9PPmwvM1RhxuhiYqoTJcmnaztMz6PQpGxmALkP”)
will return
Hello, Habr!
(if the file with such contents is still available)
4. random
Random number generation works in the same way as other sources, but if you use
oraclize_query
, you need time-consuming preparation of arguments. To avoid this, you can use the helper function
oraclize_newRandomDSQuery(delay, nbytes, customGasLimit)
, specifying only the execution delay (in seconds), the number of bytes generated and the throttle limit for the
__callback
call.
Using
random
has a couple of features to keep in mind:
- To confirm that the number is actually random, a special type of verification, Ledger, is used, which can be performed on the blockchain (unlike all the others, but more on that later). This means that in the constructor of a smart contract you need to set this method of verification by function:
oraclize_setProof(proofType_Ledger);
And at the beginning of the callback there should be the check itself:
function __callback(bytes32 _queryId, string _result, bytes _proof) { require (oraclize_randomDS_proofVerify__returnCode(_queryId, _result, _proof) == 0) ); <...>
This check requires a real network and will not work on ganache, so for local testing you can temporarily remove this line. By the way, the third argument in __callback
here is the additional parameter _proof
. It is required whenever one of the confirmation types is used. - If you use a random number for critical moments, for example, to determine the winner in the lottery, fix the user input before sending newRandomDSQuery. Otherwise, this situation may occur: oraclize calls _callback and the transaction is visible to everyone in the pending list. Along with this, the random number itself is visible. If users can continue, roughly speaking, to make bets, then they will be able to indicate the price of gas more, and push through their bet before the _callback is executed, knowing in advance that it will be advantageous.
5. computation
It is the most flexible of the sources. It allows you to write your own scripts and use them as a data source. Calculations take place on AWS. For execution, you need to describe the Dockerfile and put it together with arbitrary additional files in a zip-archive, and upload the archive to IPFS. Execution must meet the following conditions:
- Write the answer that you need to return, the last line in stdout
- The answer must be no more than 2500 characters.
- Initialization and execution should not go longer than 5 minutes in total
For an example of how this is done, consider how to perform the simplest join of the passed strings and return the result.
Dockerfile:
FROM ubuntu:16.04 MAINTAINER "info@rubyruby.ru" CMD echo "$ARG0 $ARG1 $ARG2 $ARG3"
Environment Variables
ARG0
,
ARG1
, etc. - These are the parameters passed along with the request.
Add pre-file to the archive, start the ipfs server and add this archive there
$ zip concatenation.zip Dockerfile $ ipfs daemon & $ ipfs add concatenation.zip QmWbnw4BBFDsh7yTXhZaTGQnPVCNY9ZDuPBoSwB9A4JNJD
The resulting hash is used to send the request via
oraclize_query
in the smart contract:
oraclize_query("computation", ["QmVAS9TNKGqV49WTEWv55aMCTNyfd4qcGFFfgyz7BYHLdD", "s1", "s2", "s3", "s4"]);
The argument is an array, in which the first element is the multi-cache archive, and all the rest are parameters that fall into the environment variables.
If you wait for the query, then in
__callback
will come the result
s1 s2 s3 s4
.
Helper parsers and subqueries
From the response returned by any source, you can pre-select only the required information using a number of helpers, such as:
1. JSON parser
You saw this method in the very first example, where from the result that coinmarketcap returns, only the price was returned:
json(https://api.coinmarketcap.com/v1/ticker/ethereum/?convert=USD).0.price_usd
The use case is pretty obvious, going back to the example:
[ { "id": "ethereum", "name": "Ethereum", "symbol": "ETH", "rank": "2", "price_usd": "462.857", "price_btc": "0.0621573", "24h_volume_usd": "1993200000.0", "market_cap_usd": "46656433775.0", "available_supply": "100800968.0", "total_supply": "100800968.0", "max_supply": null, "percent_change_1h": "-0.5", "percent_change_24h": "-3.02", "percent_change_7d": "5.93", "last_updated": "1532064934" } ]
Since this is an array, we take the element
0
, and from it the
price_usd
field
2. XML
Use similar to JSON, for example:
xml(https://informer.kovalut.ru/webmaster/getxml.php?kod=7701).Exchange_Rates.Central_Bank_RF.USD.New.Exch_Rate
3. HTML
You can parse XHTML with XPath. For example, get a market cap with etherscan:
html(https://etherscan.io/).xpath(string(//*[contains(@href, '/stat/supply')]/font))
We
MARKET CAP OF $46.148 BillionB
4. Binary helper
Allows you to cut pieces from raw data using the slice (offset, length) function. That is, for example, we have a file with the contents of “abc”:
echo "abc" > example.bin
Put it on IPFS:
$ ipfs add example.bin added Qme4u9HfFqYUhH4i34ZFBKi1ZsW7z4MYHtLxScQGndhgKE
And now let's cut out 1 character from the middle:
binary(Qme4u9HfFqYUhH4i34ZFBKi1ZsW7z4MYHtLxScQGndhgKE).slice(1, 1)
In the answer we get
b
As you may have noticed, in the case of the binary helper, IPFS was used, not the URL source. In fact, parsers can be applied to any sources, let's say it is not necessary to apply JSON to what the URL returns, you can add such content to the file:
{ "one":"1", "two":"2" }
Add it to IPFS:
$ ipfs add test.json added QmZinLwAq5fy4imz8ZNgupWeNFTneUqHjPiTPX9tuR7Vxp
And then disassemble like this:
json(QmZinLwAq5fy4imz8ZNgupWeNFTneUqHjPiTPX9tuR7Vxp).one
We get
1
And a particularly interesting use case is to combine any data sources and any parsers in one query. This is possible using a separate
nested
data source. We use the newly created file in a more complex query (the addition of values ​​in two fields):
[WolframAlpha] add ${[IPFS] json(QmZinLwAq5fy4imz8ZNgupWeNFTneUqHjPiTPX9tuR7Vxp).one} to ${[IPFS] json(QmZinLwAq5fy4imz8ZNgupWeNFTneUqHjPiTPX9tuR7Vxp).two}
We get
3
The query is formed as follows: specify the
nested
data source, then for each query add the source name in front of it in square brackets, and further enclose all subqueries in
${..}
.
Testing
Oraclize provides a
useful query validation
service without the need for smart contracts. Just go, choose a data source, a verification method and see what will return in __ callback if you send the corresponding requests
For local verification in conjunction with a smart contract, you can use a
special version of the Remix IDE that supports oraclize requests.
And to check locally with ganache, you will need an
ethereum bridge , which will deploy smart oraclize contracts into your testnet. To test, first add the following line to your contract constructor:
OAR = OraclizeAddrResolverI(0x6f485C8BF6fc43eA212E93BBF8ce046C7f1cb475);
run
ganache-cli
Then
node bridge --dev
Wait for the contracts to go through and be tested. In the output of the
node bridge
it will be possible to see the sent requests and the received responses.
Another help not only for testing, but also for real use - the ability to monitor requests
here . If you request a public network, you can use the hash of the transaction in which the request is executed. If you use authentication, then keep in mind that they are guaranteed to be sent only to mainnet, for other networks it can arrive 0. If the request was on the local network, then you can use the id of the request, which is returned by
oraclize_query
. By the way, this id is always recommended to be saved, for example in a similar mapping:
mapping(bytes32=>bool) validIds;
During the request, mark the id sent as
true
:
bytes32 queryId = oraclize_query(<...>); validIds[queryId] = true;
And then in
__callback
to check that the request with such id was not processed yet:
function __callback(bytes32 myid, string result) { require(validIds[myid] != bytes32(0)); require(msg.sender == oraclize_cbAddress()); validIds[myid] = bytes32(0); <...>
This is necessary because the
__callback
per request can be called more than once due to the peculiarities of the operation of the Oraclize mechanisms.
Authentication
In the table with sources you could see that different sources can support different types of confirmations, and different commissions may be charged. This is a very important part of oraclize, but a detailed description of these mechanisms is a separate topic.
The most commonly used mechanism, at least by us, is
TLSNotary with storage in IPFS. Storage in IPFS is more efficient because the
__callback
does not return the proof itself (maybe around 4-5 kilobytes), but a much smaller multi-cache. To set this type, add a line in the constructor:
oraclize_setProof(proofType_TLSNotary | proofStorage_IPFS);
We can only say that this type, roughly speaking, protects us from the unreliability of data obtained from Oraclize. But Oraclize uses Amazon servers, which act as an auditor, so they only have to trust.
Read more
here .
Conclusion
Oraclize provides tools that significantly increase the number of use cases for smart contracts, as well as IPFS, which can be seen in several variations of Oracle requests. The main problem is that we again use external data that is subject to the threats from which the blockchain was supposed to protect: centralization, blocking capabilities, code changes, result substitution. But while this is all inevitable, and the option of obtaining data is very useful and viable, you just need to be aware of why the use of the blockchain was introduced into the project and whether the appeal reduces external unreliable sources to zero advantage.
If you are interested in some topics on the development of Ethereum not yet disclosed in these articles - write in the comments, perhaps we will reveal in the following.
Ethereum development:
Part 1: introductionPart 2: Web3.js and gasPart 3: user applicationPart 4: warmth and debug in truffle, ganache, infura