There are often contradictions between developers and products. The first ones are closer to the hardware and are responsible for it, the second ones are for the user interface. Backend-developers do not want to load the system once again, store deliberately false unverified data, waste processor time and disk space. In their worldview, a priori, all requests, all users are intruders who just want to score all the memory and disable the system. If not for vandalism, then for the sake of competitors of similar services.

For example, there are no problems to take and register a user in the database using the βemail and passwordβ combination. If we do not send any mailings, do not work with monetary systems and user identification by his email is required solely for his convenience, then we still have a question about the hardware resources of our systems. By running an infinite loop with requests for registration, we can drive all the disks, and the system will fail. Therefore, we have to force users to make such inconvenient steps in the form of confirming their email. The same thing happens with reference to phone numbers. The same is true for linking social network accounts that have been delegated user identification tasks.
There will always be those who are able to raise their own mail server. Creating virtual mailboxes, clicking on confirmation links in them is trivially automated, which again can damage server resources. However, in reality this happens so rarely that against the general statistical background of all requests it will be clearly traceable and the consequences are preventable. A rarity: due to the high threshold of entry for registration in the system. It is the entry threshold that, in its majority, eliminates the attempts of intruders.
')
The threshold of entry of mere mortals
The ideal world of product managers does not contain any complex technical systems in the form of postal systems, referrals and other things. The ideal end result for them is when the user, or rather his software client (a program in the browser, among other things), identifies and authenticates himself as a unique entity. The user connects to the service, and he immediately recognizes it without unnecessary gestures in the form of SMS-ok and links. In addition to unnecessary actions, this requires the user and binding to third-party services, for which he, too, will have to spend time.
The ideal solution to the forehead is when the program simply generates a random large number (random enough so that it does not coincide with anyone else) and will use it as an identifier. Each user is his own service and authentication tool. We assume that it has a reliable secure long-term storage of this identifier and all communication with us occurs over an encrypted trusted channel (
TLS ).
There remains one βbutβ: the threshold for entering such software solutions is almost zero for registration. A software client can be both a good person and forgotten about the
laws of Azimov as a robot.
As a solution, you can enter something that would distinguish a robot from a human:
CAPTCHA . Quite simply implemented and working effectively. At a minimum, it gives a sufficient entry threshold to weed out many vandals (but not competitors who buy Chinese discriminator farms). However, such a decision is strongly disliked both by users breaking eyes and nerves, and, accordingly, by products.
Prove that worked
One of the solutions we use is a good old
proof-of-work (PoW) system, known since the beginning of email spam, which (whether it is a robot or not) did not have enough data to identify. Once in our context, we want to secure the waste of our resources, since sending data by the client is a much cheaper operation, then PoW is a means of restoring equity because the user will have to work out their right to process the request.
Proof-of-work is a technology in which it is necessary to expend the minimum established work in order to perform an action. This work is resource-intensive, but verification of the success of its implementation is cheap. If someone wants to spend our resources, then with a hundredfold difference between the check and the solution to the work, the attacker will have to spend a hundred times more resources than necessary. As a rule, it will be economically disadvantageous.
The user does not have to enter any CAPTCHA or depend on third-party services. It only has to warm up its processor a bit, transparently for the interface.
The client-server operation scheme is simple:
βββββββ ββββββββ
βCustomerβ βServerβ
βββ¬ββββ ββββ¬ββββ
β JSON-RPC REQUEST β
βββββββββββββββββββββββββββββ>
β β
β TASK β
β <βββββββββββββββββββββββββββββ
β β
βββββ β
β β Solve (TASK)
β <ββββ β
β β
β REQUEST, TASK, SOLUTION β
βββββββββββββββββββββββββββββ>
β β
ββββββ
β β β Check (TASK, SOLUTION)
β β <ββββ
β β
β Result of the request
β <βββββββββββββββββββββββββββββ
The data bound to the task conditions are not stored in the database and therefore must be authenticated, for example, using
cryptographic signatures . And in order to avoid the possibility of repeating the request, then at least, for example, tie to the time and give clear deadlines for the task.
If the client's request is a JSON-structure
REQ , then the client in the response from the server receives:
{"req": REQ, "bestbefore": 1234, "task": TASK, "sign": "SIGN"}
The client sends in response:
{"req": REQ, "bestbefore": 1234, "task": TASK, "sign": "SIGN", "answer": RESULT}
where
bestbefore is the time until which a given query can be executed, TASK is the initial conditions of the problem, and
SIGN is the cryptographic signature of the entire serialized dictionary.
RESULT is a solution to a problem.
If the client tries to change his original request, the signature will be invalid. If he tries to do it again, after some time (for example, after the comment is published), the server, having checked with his watch, invalidates the request. If the client tries to substitute a previously solved task, the signature will also be invalid.
Instead of time binding, you can issue one-time tokens that are randomly generated. For the sake of simplicity, you can keep it in
Redis and make a limited lifetime. Upon request, we read 16 bytes (128 bits) from
/ dev / urandom and put this line in Redis for ten seconds. This token should appear in the client's response. If it is still in Redis, then we delete and process the request. When reused, the server no longer knows such a token. Their short time of life guarantees us that it will not be possible to score and disable Redis, except that only with large communication channels and capacities.
It would be even better to add some unique identifier to the request, leaving the
bestbefore . If the task is successfully verified, we will save this identifier in Redis with the remaining lifetime. Thus, when issuing tasks, we do not save anything. The user will have to solve the problem to try to repeat the action, but only within the remaining time he has the opportunity to commit them, while it is really
bestbefore . For a server, this translates into a cheap PoW check and a key existence check in Redis.
TASK_SIGN_KEY = "ff7555ded9526ecf03b1617a61514d30".decode("hex") TASK_LIFETIME = 60 def task_generate(req): result = { "req": req, "bestbefore": datetime.utcnow() + timedelta(seconds=TASK_LIFETIME), "task": task_new(), "id": urandom(16).encode("hex"), } result["sign"] = sign(TASK_SIGN_KEY, serialize(result)) return result def task_validate(req): if req["bestbefore"] < datetime.utcnow(): return False signature = req.pop("sign") answer = req.pop("answer") if not constant_time_compare(signature, sign(TASK_SIGN_KEY, req)): return False if redis.exists(req["id"]): return False redis.setex(req["id"], TASK_LIFETIME) if not task_verify(req["task"], answer): return False return True
Do not forget that any strings used in cryptography and requiring comparison should not be compared simply out of the box with native language tools. Comparison of "==" will be optimal in performance and, roughly speaking, the operation will be completed at the first unmatched byte: as a result, the test time is not constant and because of this you can make powerful attacks in the case of TLS that are able to decrypt the entire message. Frequent practice (though not optimal in terms of performance) is a comparison through HMACs:
hmac(compare_key, string1) == hmac(compare_key, string2)
At the same time, the comparison time is not constant, but randomized, which will prevent attacks aimed at leaking string comparison time information.
Tasks
What can these tasks represent? At the first approach to the projectile, we decided to use a part already from the 1970s of the famous
Merkle puzzles . Officially, this can be considered the first asymmetric key exchange system, looking at which the
Diffie-Hellman algorithm was invented. Each puzzle is an encrypted line and a part of the decryption key. If this is the
DES encryption algorithm, then a full brute-force search takes an average of 2 ^ 55 operations. We can give 5 bytes (40 bits) of this key, and then it will be necessary to do 2 ^ 15 operations, that is, more than 32 thousand decryption operations. The result of the decision will be the rest of the key: the missing two bytes. Checking the solution on the server is a single decryption operation. Previously it is possible to agree that the line βYOUBROKEβ is encrypted.
It is not very difficult to estimate what the parameters of such tasks should be. We take the lowest-performing device supported, which can be found in large quantities among users. We assume that there is old software with not very optimal implementations of algorithms (for example, it is JavaScript in the browser). We set the maximum timeout for the user and see how many iterations can be performed. We understand how many bits of the key not to disclose.
What encryption algorithm do you prefer? The choice is huge. It would be desirable to be legally clean when using it, not to pay, so that it is both fast and cryptographically secure (although we are not secrets to transmit). Most importantly, at the same time its implementations are on all platforms. Among the platforms supported by ivi are Web browsers, iOS, Android, SmartTV and others. Everywhere its developers, who, of course, would not want to write a lot of code. JavaScript does not have any built-in cryptographic tools. If
AES library implementation was everywhere, then the choice would be obvious - the path of least resistance (if you donβt get involved with the state, then GOST is not dictated).
The choice fell on
Blowfish - the block cipher known since 1993, developed by the famous cryptographer
Bruce Schneier . It fully satisfies the set conditions, has completely free and open implementations. Yes, there are certainly more technically advanced algorithms, but their code is either more complicated, or the algorithm is not so well known, or there are not so many implementations in various languages. Ciphers such as
Arcfour and
Salsa20 are simpler, but they are streamed, and this limits the scope of application, which will be
discussed below. In general, the
XTEA cipher is also excellent, but human subjectivity is everywhere present and Schneier's weight was decisive.
In addition, in all the tasks below, this cipher is used only in encryption mode, without requiring code to decrypt. Since this is
a Feistel network , decryption is simply an inversion of the course of operations, but it also takes its share of unused code.
Birthdays
The task of finding the decryption key has a serious drawback: there is too much difference in performance between user devices. The JavaScript code in the browser on the ARM processor is several orders of magnitude slower than what will be running on a modern PC with implementation on C. The quantitative difference becomes qualitative: either the user of mobile devices waits too long for a PoW solution, or the complexity on the PC becomes insignificant and sending traffic It will be much more expensive.
The available resource available on all platforms except the processor is RAM. The tasks of the processors can be implemented in hardware in the form of ASIC and FPGA solutions with higher performance per currency unit. But the difference in memory performance on various platforms available to users is much less significant.
A well-known algorithm for storing passwords
scrypt uses random access memory to speed up calculations. If everything is done on the processor, the calculations will take a very long time. The difference is so great that ASIC / FPGA or powerful Intel Xeon processors will play absolutely no role if there is no memory.
The solution adopted by ivi is the use of memory as a cache in the problem of searching for collisions between the results of the
PRP function (pseudo random permutation is a pseudo-random permutation).
In probability theory there is a
paradox of birthdays : if a group of 23 or more people is given, then the probability that at least two of them have birthdays (number and month) coincide, exceeds 50%. If we take a
hash function with a 128-bit output, then the probability that two different inputs match the hash value will be one per 2 ^ 64.
As a certain hash value, we use part of an encrypted piece of data. For example, we take only three bytes from the encrypted data (we have 24-bit similarity to the hash function). The server sends the key to be used for encryption. In the client, rules for searching / collision searching are pre-protected: for example, these will be two line spaces with the length equal to the size of the cipher block used by the algorithm. One row space is βXXXXXXX1β, βXXXXXXX2β, βXXXXXXX3β and so on. The second is βYYYYYYY1β, βYYYYYYY2β, βYYYYYYY3β and so on. Each element of this space is encrypted and the last 3 bytes are taken from the result. The entire space of the first sequence is stored in RAM. During the calculation of the second sequence, we look for matching elements with the first sequence and remember these collisions. For example, for some given key, the element with sequence number 123 will have the same value of our hash with the element of the second sequence with sequence number 5678: that is, the last three bytes of the encrypted line βXXXXX123β will match the last three bytes of the encrypted line βYYYY5678β. The stored hashes of the first sequence are our cache. The required cache size (so that, for example, at least 60% of collisions fall) depends on the size of the hash values.
The task itself is:
{"key": "KEY", "size": 1048576, "count": 4096}
where
KEY is a random encryption key,
size is the size of the sequence to cache, and
count is the minimum number of collisions to find. The solution is a list of pairs of consecutive sequence numbers, where the collision was found (in our case, one of the pairs is 123 and 5678).
def solve(key, size, count): bf = Blowfish(key) cache = {} for i in xrange(size): val = str(i) val = "X" * (8 - len(val)) val = bf.encrypt(val) cache[val[-3:]] = i collisions = [] for i in xrange(size): val = str(i) val = "Y" * (8 - len(val)) val = bf.encrypt(val) found = cache.get(val[-3:]) if found: collisions.append((found, i)) if len(collisions) == count * 2: return collisions
For our task from the server from the example, the client needs to perform 1048576 + 4096 encryption operations (minimum) or 1048576 + 1048576 (maximum) and three megabytes of RAM at least (in practice, a little more to store the list of hashes of the first sequence in the form of a tree to quickly find the presence of element in it). To check, you need to perform 4096 + 4096 encryption operations and have a little more than 4 kilobytes of RAM (overhead for the work of Blowfish).
It is highly desirable that the required number of elements be such that in the vast majority of cases, force the client to go through the entire second sequence (estimate how many collisions there are at a given cache size and take a number around this). It is also very important to estimate the size of the cache: it can be small enough to fit completely into the cache of the processor of the second or even the first level, which will give a significant acceleration of the calculations.
Davier mayer
As mentioned above, platforms that only have JavaScript do not have ready-made cryptographic tools, such as even hash functions. There are often situations where the use of hashes is constantly required due to convenience.
Known common functions such as MD5, SHA1, SHA2 for such platforms will have to implement (or use someone else's implementation) independently. For political reasons, the choice of licensing restrictions can be drastically reduced. More code means more likelihood of errors.
Since we already have a working block cipher, it can be easily applied to data hashing tasks. Most of the hash functions are based on
one-way compression functions . The function has two inputs and one output, all of the same size. Upon exiting the function, it is impossible to recover and assume what was at the inputs. A regular block cipher almost has this property and therefore it can be used as a function of compression. This structure is called the
Davier-Meier function.
ββββββββββββββββββββ
β β¨
ββββ βββββ βββββββ
ββ> β H β ββ> β E β ββ> β XOR ββ>
ββββ βββββ βββββββ
β§
β
β
m
The message is divided into parts of
m (the length of each is equal to the length of the cipher block), each part is encrypted. Ciphertext
H of the previous block is used as the encryption key. The very first iteration uses predefined constants.
def hasher(data): data += "\x80" if len(data) < 8: pad_length = 8 - len(data) else: pad_length = 8 - len(data) % 8 data += "\x00" * pad_length prev = "1aec98c401022e7c".decode("hex") for i in xrange(0, len(data), 8): bf = Blowfish.new(data[i:i + 8]) prev = strxor(bf.encrypt(prev), prev) return prev
,
(padding) . :
- ,
- Β«1Β», ( MD5 SHA1 )
- , (ANSI X.923)
- , (ISO 10126)
- , (PKCS 7)
- . Blowfish : 64 . , ( ), 64 ( 128 ). TLS. , , .
- - , . , , - , , .
- SHA2 , AES? - . . : Blowfish . Blowfish - ,
S- , DES, .
- . - Arcfour Salsa20 . Salsa20
BLAKE -, Salsa20 .
, -,
, .
PKI TLS. TLS. ( , curl, ), , TLS , .
, , - JavaScript, , ,
Ed25519 ,
padding RSA β . , .
: β . : ivi , . , , (,
CHAP ) , .
MAC (message authentication code) . MAC
HMAC ( ,
Poly1305 ).
- ( ). , .
HMAC , . MAC , -. , MD5 , - , HMAC-MD5 . MAC,
CBC-MAC ,
CBC- MAC.
«» CBC-MAC : . , - CBC-, MAC-, , . : . . .
JSON, . . MAC.
def mac(k, data): data = struct.pack("!I", len(data)) + data if len(data) < 8: pad_length = 8 - len(data) else: pad_length = 8 - len(data) % 8 data += "\x00" * pad_length k = Blowfish.new(k) prev = "\x00" * 8 for i in xrange(0, len(data), 8): prev = bf.encrypt(strxor(data[i:i + 8], prev)) return prev
MAC PoW , , , Blowfish.
findings
- , , , .
- . Blowfish , . RSA , .
- . , . . , .
- , . .
- . , , , cookie. , / , .
All the best, do not switch!
Β© , Python Go ivi.ru