Geekly Articles each Day
📜 ⬆️ ⬇️

We write our BitTorrent client based on the Bt library

Hello! In this article, I will talk about what the Bt library is, how good it is, and why you should keep it in mind if you plan to use BitTorrent to solve some problems. Then, as a demonstration of the basic functions and the main API, we will implement the simplest console torrent client.


Distinctive features of Bt


Bt is a modern fully functional implementation of the BitTorrent protocol in Java 8 [1] . Compared with existing open-source counterparts [2] [3] , Bt has the following advantages:



Required options for a serious BitTorrent client that are supported in Bt include:



Creating the simplest command-line client


For the sake of minimizing gestures and in order to avoid unnecessary mistakes, I recommend not trying to reproduce the project code in the text of the article, but download the finished project right away from github [11] .


Project configuration


For ease of use, the client's executable file will be a fat jar : the classes and resources of the application and its dependencies will be compiled into a single archive. Let's create a new Maven project, in pom.xml of which we will declare a class containing the main() method, the name of the executable file, external dependencies and a couple of plugins.


 <?xml version="1.0" encoding="UTF-8"?> <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> <modelVersion>4.0.0</modelVersion> <groupId>com.github.atomashpolskiy</groupId> <artifactId>bt-cli-demo</artifactId> <version>1.0-SNAPSHOT</version> <name>Bt CLI Launcher</name> <description>Command line BitTorrent client</description> <properties> <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding> <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding> <compiler.source>1.8</compiler.source> <compiler.target>1.8</compiler.target> <main.class>bt.cli.CliClient</main.class> <bt-version>1.7</bt-version> <jopts-version>5.0.2</jopts-version> <slf4j-version>1.7.21</slf4j-version> <log4j-version>2.4.1</log4j-version> </properties> <build> <finalName>bt-launcher</finalName> ... </build> <dependencies> <dependency> <groupId>com.github.atomashpolskiy</groupId> <artifactId>bt-core</artifactId> <version>${bt-version}</version> </dependency> <dependency> <groupId>com.github.atomashpolskiy</groupId> <artifactId>bt-http-tracker-client</artifactId> <version>${bt-version}</version> </dependency> <dependency> <groupId>com.github.atomashpolskiy</groupId> <artifactId>bt-dht</artifactId> <version>${bt-version}</version> </dependency> <dependency> <groupId>net.sf.jopt-simple</groupId> <artifactId>jopt-simple</artifactId> <version>${jopts-version}</version> </dependency> <dependency> <groupId>org.apache.logging.log4j</groupId> <artifactId>log4j-core</artifactId> <version>${log4j-version}</version> </dependency> <dependency> <groupId>org.apache.logging.log4j</groupId> <artifactId>log4j-slf4j-impl</artifactId> <version>${log4j-version}</version> </dependency> </dependencies> </project>

We have specified three standard Bt modules as dependencies:



We also need Log4J (including the bridge to SLF4J) and the beloved by many JOpt Simple library [12] , which makes working with command line arguments simple and enjoyable.


We will immediately add the configuration to the log4j2.xml logging configuration , but here we will not give its text. We only say that the application will log into two files: bt.log and bt-dht.log . The second file will contain events and messages related to the work of DHT, which in most cases do not represent much interest for the user.


Source


Okay, with the setup of the project is over, proceed to writing the code. In the next section, we will devote some time to writing the necessary “strapping” for processing program arguments and setting up the JRE. The most impatient can skip this section and go directly to writing the code of the torrent client.


Options, Modes and Parameters JRE


Even such a small application as a torrent client can have a large number of configuration parameters, depending on the purpose of use and the current environment. Define a list of options that I would like to provide to the user of our application:



We will hide the list of options and parsing of arguments in a separate Options class, and here we will give only the initial version of the main method of the bt.cli.CliClient.main() program.


 package bt.cli;  import joptsimple.OptionException; public class CliClient { private static final Logger LOGGER = LoggerFactory.getLogger(CliClient.class); public static void main(String[] args) { Options options; try { options = Options.parse(args); } catch (OptionException e) { Options.printHelp(System.out); return; } } }

Now our application is able to display nice help!


 Option (* = required) Description --------------------- ----------- -?, -h, --help -S, --sequential Download sequentially -a, --all Download all files (file selection will be disabled) * -d, --dir <File> Target download location --dhtport <Integer> Listen on specific port for DHT messages -e, --encrypted Enforce encryption for all connections -f, --file <File> Torrent metainfo file -i, --inetaddr Use specific network address (possible values include IP address literal or hostname) -m, --magnet Magnet URI -p, --port <Integer> Listen on specific port for incoming connections -s, --seed Continue to seed when download is complete --trace Enable trace logging -v, --verbose Enable more verbose logging

With almost all options processing, all that remains is to set up Log4J and set JRE parameters for obfuscation to work correctly. Add a call to several utility methods in main() .


 configureLogging(options.getLogLevel()); configureSecurity(); registerLog4jShutdownHook();

The methods associated with setting up Log4J are not of great interest to us now; you can see them here and here . The configureSecurity() method will touch on a little more.


The point is that the obfuscation protocol uses encryption using session keys, the minimum recommended size of which is 160 bits. According to US laws that regulate software distribution (which means that Oracle JDK is also inevitably related), the maximum permissible key size for encryption by default cannot exceed 128 bits. Using larger keys is not prohibited, but the user must make the necessary settings and “unlock” this feature. The Bt configuration allows you to set the size of the keys from 128 to 4096 bits, but in this case we would like to leave the optimal value set by default and configure the same JRE. Up to the Oracle JRE version 8u152, for this, it was necessary to download the jar file from the Oracle website [13] and replace the eponymous file in the installed distribution. Starting from version 8u152, the same effect can be achieved simply by setting the environment variable crypto.policy=unlimited [14] . This is exactly what the configureSecurity() method does.


 private static void configureSecurity() { // Starting with JDK 8u152 this is a way // to programmatically allow unlimited encryption // See http://www.oracle.com/technetwork/java/javase/8u152-relnotes-3850503.html String key = "crypto.policy"; String value = "unlimited"; try { Security.setProperty(key, value); } catch (Exception e) { LOGGER.error(String.format( "Failed to set security property '%s' to '%s'", key, value), e); } }

Thus, we choose a compromise option:



Whew! No wonder Java is famous for its verbosity and confusion. Getting down to the final part ...


Integration with Bt


The torrent client code will consist of a constructor, several helper methods, and a method to launch. To begin, consider the constructor.


 private final Options options; private final SessionStatePrinter printer; private final BtClient client; public CliClient(Options options) { this.options = options; this.printer = new SessionStatePrinter(); Config config = buildConfig(options); BtRuntime runtime = BtRuntime.builder(config) .module(buildDHTModule(options)) .autoLoadModules() .build(); Storage storage = new FileSystemStorage(options.getTargetDirectory().toPath()); PieceSelector selector = options.downloadSequentially() ? SequentialSelector.sequential() : RarestFirstSelector.randomizedRarest(); BtClientBuilder clientBuilder = Bt.client(runtime) .storage(storage) .selector(selector); if (!options.shouldDownloadAllFiles()) { CliFileSelector fileSelector = new CliFileSelector(); clientBuilder.fileSelector(fileSelector); runtime.service(IRuntimeLifecycleBinder.class) .onShutdown(fileSelector::shutdown); } clientBuilder.afterTorrentFetched(printer::onTorrentFetched); clientBuilder.afterFilesChosen(printer::onFilesChosen); if (options.getMetainfoFile() != null) { clientBuilder = clientBuilder.torrent(toUrl(options.getMetainfoFile())); } else if (options.getMagnetUri() != null) { clientBuilder = clientBuilder.magnet(options.getMagnetUri()); } else { throw new IllegalStateException("Torrent file or magnet URI is required"); } this.client = clientBuilder.build(); }

Go through the code, making the necessary explanations.


 Config config = buildConfig(options);

The buildConfig() method creates the Bt runtime configuration. Rantaim is a container for clients , each of whom performs processing of their torrent. The main functions of runtime:



A single client is a small, lightweight wrapper over several torrent-specific objects ( context ). Its task is to sequentially execute the processing stages of a specific type of torrent ( .torrent file or magnet link) and provide the user with an API to start and stop processing.


Accordingly, the Bt setting is performed on two levels:



Consider runtime configuration creation code.


 private static Config buildConfig(Options options) { Optional<InetAddress> acceptorAddressOverride = getAcceptorAddressOverride(options); Optional<Integer> portOverride = tryGetPort(options.getPort()); return new Config() { @Override public InetAddress getAcceptorAddress() { return acceptorAddressOverride.orElseGet(super::getAcceptorAddress); } @Override public int getAcceptorPort() { return portOverride.orElseGet(super::getAcceptorPort); } @Override public int getNumOfHashingThreads() { return Runtime.getRuntime().availableProcessors(); } @Override public EncryptionPolicy getEncryptionPolicy() { return options.enforceEncryption()? EncryptionPolicy.REQUIRE_ENCRYPTED : EncryptionPolicy.PREFER_PLAINTEXT; } }; } private static Optional<Integer> tryGetPort(Integer port) { if (port == null) { return Optional.empty(); } else if (port < 1024 || port > 65535) { throw new IllegalArgumentException("Invalid port: " + port + "; expected 1024..65535"); } return Optional.of(port); } private static Optional<InetAddress> getAcceptorAddressOverride(Options options) { String inetAddress = options.getInetAddress(); if (inetAddress == null) { return Optional.empty(); } try { return Optional.of(InetAddress.getByName(inetAddress)); } catch (UnknownHostException e) { throw new IllegalArgumentException( "Failed to parse the acceptor's internet address", e); } }

Here, based on user-specified parameters, we create a new instance of the bt.runtime.Config class, in which we override a number of methods so that they return the value specified by the user, if one exists, or the default value otherwise.


It is worth paying attention to two parameters.


The first is the numOfHashingThreads , or the number of threads that will perform the initial verification of the data already downloaded ("hashing" in the common jargon; it is necessary when restarting the client). By default, Bt uses only one stream, but the verification procedure is perfectly parallelizable, so it makes sense to use several streams. The optimal number of threads is in the interval [ ; * 2] [ ; * 2] , because individual threads may be idle while waiting for the next I / O read operation to complete.


The second parameter is the traffic obfuscation policy. Policies are used in the protocol for establishing a connection with peers, and there are only four of them [15] :


1) Do not apply obfuscation and do not establish connections with feasts that require the use of obfuscation.
2) By default, offer the feast to use plaintext, but agree to obfuscation if the feast requires it.
3) By default, offer a feast to use obfuscation, but agree to plaintext if a feast requires it.
4) Always apply obfuscation and not establish connections with peers that require the use of plaintext.


In our application, we choose between the two most common policies: forced obfuscation or obfuscation at the request of a feast. The first policy is suitable for paranoid users (and unscrupulous ISPs), while the second allows connections with the maximum number of peers.


 BtRuntime runtime = BtRuntime.builder(config) .module(buildDHTModule(options)) .autoLoadModules() .build();

Runtime build usually comes down to two things:



In addition to the main module, there are two extension modules in our application: an integration module with HTTP trackers and an integration module with Mainline DHT. The first module will be found and loaded automatically thanks to the autoLoadModules() call, and for the second module we want to specify a non-standard configuration and therefore override it manually.


 private static Module buildDHTModule(Options options) { Optional<Integer> dhtPortOverride = tryGetPort(options.getDhtPort()); return new DHTModule(new DHTConfig() { @Override public int getListeningPort() { return dhtPortOverride.orElseGet(super::getListeningPort); } @Override public boolean shouldUseRouterBootstrap() { return true; } }); }

We override two parameters:



 Storage storage = new FileSystemStorage(options.getTargetDirectory().toPath()); PieceSelector selector = options.downloadSequentially() ? SequentialSelector.sequential() : RarestFirstSelector.randomizedRarest(); BtClientBuilder clientBuilder = Bt.client(runtime) .storage(storage) .selector(selector);

The next step is to specify the directory to save the downloaded files. As mentioned above, the bt.data.file.FileSystemStorage class bt.data.file.FileSystemStorage takes a parameter of type java.nio.file.Path , which allows its use in combination with in-memory file systems, such as JimFS. In the library itself, this feature is used in integration tests (in order to save execution time on file I / O), but, hypothetically, there may be more exotic uses [16] , for example:



 if (!options.shouldDownloadAllFiles()) { CliFileSelector fileSelector = new CliFileSelector(); clientBuilder.fileSelector(fileSelector); runtime.service(IRuntimeLifecycleBinder.class) .onShutdown(fileSelector::shutdown); }

We continue to collect the client. We gave the user the opportunity to skip the step of selecting files in case there are too many of them, or if the user wants to download the entire distribution. If this option is not specified, then we provide Bt with its own implementation of the file selector, oriented to work in the conditions of the terminal interface.


 public class CliFileSelector extends TorrentFileSelector { private static final String PROMPT_MESSAGE_FORMAT = "Download '%s'? (hit <Enter> or type 'y' to confirm or type 'n' to skip)"; private static final String ILLEGAL_KEYPRESS_WARNING = "*** Invalid key pressed. Please, use only <Enter>, 'y' or 'n' ***"; private AtomicReference<Thread> currentThread; private AtomicBoolean shutdown; public CliFileSelector() { this.currentThread = new AtomicReference<>(null); this.shutdown = new AtomicBoolean(false); } @Override protected SelectionResult select(TorrentFile file) { while (!shutdown.get()) { System.out.println(getPromptMessage(file)); try { switch (readNextCommand(new Scanner(System.in))) { case "": case "y": case "Y": { return SelectionResult.select().build(); } case "n": case "N": { System.out.println("Skipping..."); return SelectionResult.skip(); } default: { System.out.println(ILLEGAL_KEYPRESS_WARNING); } } } catch (IOException e) { throw new RuntimeException(e); } } throw new IllegalStateException("Shutdown"); } private static String getPromptMessage(TorrentFile file) { return String.format(PROMPT_MESSAGE_FORMAT, String.join("/", file.getPathElements())); } private String readNextCommand(Scanner scanner) throws IOException { currentThread.set(Thread.currentThread()); try { return scanner.nextLine().trim(); } finally { currentThread.set(null); } } public void shutdown() { this.shutdown.set(true); Thread currentThread = this.currentThread.get(); if (currentThread != null) { currentThread.interrupt(); } } }

We are required to implement a class with one method, which for each individual file determines how to proceed: download or skip. In this case, we interactively request the desired action from the user. If the user presses Enter or enters “y”, the file will be downloaded, and if the user enters “n”, the file will be skipped.


 clientBuilder.afterTorrentFetched(printer::onTorrentFetched); clientBuilder.afterFilesChosen(printer::onFilesChosen);

, , :



UI , .


, , , stopWhenDownloaded() , . , .. , .


 if (options.getMetainfoFile() != null) { clientBuilder = clientBuilder.torrent(toUrl(options.getMetainfoFile())); } else if (options.getMagnetUri() != null) { clientBuilder = clientBuilder.magnet(options.getMagnetUri()); } else { throw new IllegalStateException("Torrent file or magnet URI is required"); } this.client = clientBuilder.build();

, .torrent , magnet -. java.util.function.Supplier<Torrent> , (, ).


: ! main() , .


 public static void main(String[] args) throws IOException { Options options; try { options = Options.parse(args); } catch (OptionException e) { Options.printHelp(System.out); return; } configureLogging(options.getLogLevel()); configureSecurity(); registerLog4jShutdownHook(); CliClient client = new CliClient(options); client.start(); } //   ... private void start() { printer.start(); client.startAsync(state -> { boolean complete = (state.getPiecesRemaining() == 0); if (complete) { if (options.shouldSeedAfterDownloaded()) { printer.onDownloadComplete(); } else { printer.stop(); client.stop(); } } printer.updateState(state); }, 1000).join(); }

start() , startAsync() . , , — , , — -, ( — 1 ).


:



, .. , , -.


, . SessionStatePrinter , UI. , , .


,


, , , .


image


System.out InterruptedException, , .


- , :


magnet:?xt=urn:btih:985BAD472E60E763F5C77B13CBE41AE2892604B6


, , public domain. 12 1959-1961 .


Conclusion


, Bt. API BitTorrent , uTorrent, Transmission, Deluge ., , , ..


, : 2 . , . BitTorrent , [17] , Bt.


, BitTorrent . , , , . , Java BitTorrent , .. , Android [18] .


, ! .


Links


[1] https://github.com/atomashpolskiy/bt
[2] https://github.com/mpetazzoni/ttorrent
[3] https://github.com/bitletorg/bitlet
[4] https://github.com/google/jimfs
[5] http://atomashpolskiy.imtqy.com/bt/extension-protocol/
[6] http://atomashpolskiy.imtqy.com/bt/javadoc/latest/bt/runtime/Config.html
[7] https://en.wikipedia.org/wiki/Mainline_DHT
[8] https://en.wikipedia.org/wiki/Local_Peer_Discovery
[9] https://en.wikipedia.org/wiki/Peer_exchange
[10] https://en.wikipedia.org/wiki/BitTorrent_protocol_encryption
[11] https://github.com/atomashpolskiy/bt-cli-demo
[12] https://pholser.imtqy.com/jopt-simple/
[13] http://www.oracle.com/technetwork/java/javase/downloads/jce8-download-2133166.html
[14] http://www.oracle.com/technetwork/java/javase/8u152-relnotes-3850503.html
[15] http://atomashpolskiy.imtqy.com/bt/encryption/
[16] https://github.com/atomashpolskiy/bt/issues/9
[17] http://bittorrent.org/beps/bep_0052.html
[18] https://www.makeuseof.com/tag/8-legal-uses-for-bittorrent-youd-be-surprised/


')

Source: https://habr.com/ru/post/350076/

More articles:


All Articles