Lazy map on Qt

Qt has the ability to execute your function in parallel for each member of a sequence - QtConcurrent :: mapped () and its friends.

The only problem is that the results are saved in QVector. Therefore, when I needed to perform my function for 65 million pieces of data, I did not torture the RAM, but wrote about the same thing, only more lazy, that is, the new values ​​will be calculated only if the old ones have already been used.

Code: bitbucket.org/cblp/threadpool

Under the hood

ThreadPool inherits QThreadPool. The ThreadPool :: map () function takes a Java-style iterator at the input and a conversion function, returns a Java-style iterator with the results.
')
Your function is wrapped in QRunnable and in the number of M = maxThreadCount pieces (as a rule, this is the number of processors) is thrown into QThreadPool.

FutureIterator provides hasNext () and next () methods that return the next result, if any, or are blocked until the next result is ready. If M current tasks are completed, and no one has come for the results, the computers are blocked.

Example

Read from the standard input line, from each count N-times MD5 with salt, the result is written to the standard output. (This is a heavily modified real-life example.)

const uint N = 10000; const QString Salt = "5417"; QByteArray md5_N_times(QString line) { QByteArray data = line.toUtf8(); for ( uint i = 0; i < N; ++i ) data = QCryptographicHash::hash(Salt.toUtf8() + data, QCryptographicHash::Md5); return data; } int main() { QTextStream input(stdin); QTextStream output(stdout); ThreadPool pool; FutureIterator<QByteArray> results = pool.map( QTextStreamIterator(input), md5_N_times ); while ( results.hasNext() ) output << results.next().toHex() << endl; return 0; } 

All this is still quite damp, for example, you cannot run several tasks in one pool so that they do not interfere with each other. You can offer your own improvements.