We collect the Shark under iOS and OSX

Yes, this title is not very meaningful, so I will give a few explanations for those who still think "to go under the cat or not?".

Shark is a set of C ++ machine learning libraries (Machine Learning), namely linear and nonlinear optimization, neural networks, and with and without teacher training, evolutionary algorithms, and much more. A more detailed description can be found on the project website .

If you are interested in the answers to the following questions

• How to build a static library for iOS?
• How to create a framework for iOS?
• How to publish iOS and OSX framework via CocoaPods?

')
Then you under the cat.
.

What for?

One could simply answer, “For more libraries to be good and useful for iOS,” but I had another reason. While working on a board game under iOS, I wanted to use machine learning to develop Artificial Intelligence (AI) of a computer opponent.

The plan was simple, first teach the AI, driving the neural network through tens of thousands of games, running them on powerful hardware under Mac OS, then take the trained network and use it in the application on the iPhone / iPad. The first part of the plan was relatively easy to implement, if only because Shark version 2.3.4 can be installed on the poppy using brew install shark .

It’s time to point out that the article deals with version 2.3.4 of the library. The project itself is already far from 2.3.4, beta version 3.0 is currently available. The code for the new version has been significantly reworked, using boost as a third-party library, so this is a completely different story, especially in the context of iOS.

Let's return to the second part of the plan. After training, I have a file with neural network coefficients in my hands. The format of this file is not described anywhere (or I was looking badly), so the least expensive way to read and use it is with the same Shark, which means you need to port the library under iOS.

How?

The short answer is to take the sources and compile, the detailed answer is the article on Habré.

Understandably, before writing on Habr, I first compiled the library and designed the whole thing as a repository on GitHub . Therefore, for clarity, I will spice up the post with pieces of code from the build script.

The whole process can be broken down into the following steps.

• Download source code
• Patch source code
• Configure and build
• Make a "thick" library
• Pack in a framework
• Publish to CocoaPods

Yes, precisely, we will not just build a “fat” (aka “fat” or fat) static library for developing for iOS devices and a simulator (and separately for OSX), we will also design everything as a framework and make it available through CocoaPods .

Well, let's get started, step by step.

Download source code

Everything is simple, I will not go into much detail so that it does not work out too chewed, download and unpack the archive.

Patch source code

Initially, the code was developed using gcc 4.2, but we are trying to compile it using clang version 5.0. Since the release of gcc 4.2, compilers and the C ++ standard have gone far ahead, so there is nothing surprising in the fact that some pieces of code will not like clang. Nothing remains but to fix, i.e. patch the problem code.

Default constructor

The first time the compiler stumbles on line 78 in the ReClaM/EarlyStopping.cpp .

 EarlyStopping::EarlyStopping(unsigned sl = 5) 

The constructor of EarlyStopping has a single argument ( unsigned sl ) and the default value ( 5 ) is specified for this argument. Thus, this constructor becomes the default constructor, “so be it,” gcc 4.2 says, but clang has a completely different opinion on this. On StackOverflow, you can find a discussion of this error.

We will correct “in the forehead”, i.e. just remove the default value.

 EarlyStopping::EarlyStopping(unsigned sl) 

Wait a second! - you say - but what if this constructor is used somewhere in the library code without an argument?
Absolutely fair question. Fortunately, this constructor is not invoked anywhere else in the code (both explicitly and implicitly), so there will be no harm from such a change, but you need to be careful when calling it in your code and be sure to pass the value for sl .

finite is not available for iOS

The next error is that the finite(x) function is not available (not included) for iOS, either for devices or for a simulator. On the network, you can find references to this problem, for example here .

The solution is to use the isfinite(x) function. Override finite(x) as isfinite(x) using preprocessing macros, to do this, add the following code to SharkDefs.h


The logic here is the following, if we compile for one of Apple's operating systems, i.e. __APPLE__ and __MACH__ , then we include the header file TargetConditionals.h , then we check the values TARGET_IPHONE_SIMULATOR , TARGET_OS_IPHONE and TARGET_OS_MAC , and redefine the finite(x) for the iOS device and simulator.

Exactly the same problem with the drem(x, y) function, which needs to be replaced with remainder(x, y) . However, Shark drem(x, y) does not use, so this is just a note of information.

FileUtil fixes

The following errors are related to the FileUtil module, namely the FileUtil.h file.

At first, clang cannot find iotype type iotype and SetDefault , ScanFrom and PrintTo . Let's give the compiler a hint using the namespace, i.e. perform a simple replacement where needed

 iotype -> FileUtil::iotype SetDefault -> FileUtil::SetDefault ScanFrom -> FileUtil::ScanFrom PrintTo -> FileUtil::PrintTo 

The last problem in this file is the io_strict function, which calls the scanFrom_strict and printTo_strict functions declared later.
Decision? Just move io_strict to the end of the file and there is no problem.

double erf (double) throw () ;

Compiling Mixture/MixtureOfGaussians.cpp mark the following code as invalid

 extern "C" double erf(double) throw(); 

extern "C" declaration does not match the original prototype.
In this case, the problem is solved by removing throw() .

 extern "C" double erf(double); 

I am aware that for some corrections I do not give a detailed explanation, such as removing throw() . I would be happy to comment in the comments.

RandomVector this-> p

The next in line is the file Mixture/RandomVector.h , line 72. The compiler has no idea how to Mixture/RandomVector.h with p . Also alarming is the comment to the code in this line ( // !!! ).

 for (unsigned k = x.dim(0); k--;) { l += log(Shark::max(p(x[ k ]), 1e-100)); // !!! } 

After analyzing the code, I found this very p . RandomVector inherits RandomVar , which declared the "lost" method p .


The construction of this->p .

 for (unsigned k = x.dim(0); k--;) { l += log(Shark::max(this->p(x[ k ]), 1e-100)); // !!! } 

CMakeLists.txt

The last patch has nothing to do with compilation errors.
Since we want to get a library for iOS, this library should be static, but the project will build a dynamic library by default. To get the desired result, replace one line in the CMakeLists.txt file.

 ADD_LIBRARY( shark SHARED ${SRCS} ) #   ADD_LIBRARY( shark STATIC ${SRCS} ) 

In principle, you can not replace the line with “SHARED”, but simply add another one with “STATIC”, then the dynamic and static libraries will be assembled as a result.

Apply patch

Of course, if you decide to repeat this process, it makes no sense to pick the code manually. To save time and nerves there is a ready-made patch .
If you're interested, this patch is obtained using the diff utility.

 diff -crB shark_orig shark_patched > shark.patch 

And to apply this patch, you need to use the utility, attention, patch

 SRC_DIR=src # --- patchSource() { echo Patching source code... patch -d $SRC_DIR/Shark -p1 --forward -r - -i ../../shark.patch doneSection } 

Configure and build

Finally, the patches are over, you can proceed to the assembly directly.
We will build the project with make , it remains only to configure everything correctly for each platform, this will be helped by the utility cmake (configure make), which will create a Makefile and other files needed by make . We will have to configure and build the library 3 times, for the following platforms and architectures

• iOS devices - armv7 , armv7s , arm64 architectures
• iOS simulator - i386 and x86_64
• OSX - x86_64

Yes, an important point, we will use Xcode 5. Support for the architecture of armv6 officially removed from Xcode 5, so we don’t consider it at all.

Libraries for iOS devices and simulators, we will later merge into one “thick” library, which can be used simultaneously for development on devices and simulators without unnecessary gestures.

Basics of cmake

So, cmake . For each platform, we need to properly configure the environment using the following settings

• C ++ compiler ( CMAKE_CXX_COMPILER )
  The default is /usr/bin/c++ , we also need to use clang++ from the Xcode toolchain.
• C ++ compiler flags ( CMAKE_CXX_FLAGS )
  It is with these flags that we will install the necessary architectures and other important compilation parameters.
• C compiler ( CMAKE_C_COMPILER )
  Despite the lack of pure C code, it is still necessary to properly configure this parameter.
• Root system directory ( CMAKE_OSX_SYSROOT )
  With this setting, we will tell the compiler where to look for all the standard system libraries for the selected platform.
• Installation prefix ( CMAKE_INSTALL_PREFIX )
  This parameter is optional. It is used by the make install command, but we will not do this.
• Assembly file generator ( -G flag)
  With this flag, we can choose one of many build systems. In our case it will be “Unix Makefiles”, i.e. cmake generate the familiar Makefile . It would be logical to use the Xcode project, but I was not able to collect everything in this way.

Xcode Toolkit

Maybe not the most successful translation of the term Toolchain, but certainly better than the "chain of tools." Xcode contains all the tools we need (compilers, libraries for iPhone OS SDK and iPhone Simulator SDK, etc.) Clearly, Xcode should be installed, you also need to install the very same toolkit (Xcode Command Line Tools), you can do it in the settings of Xcode itself or using the xcode-select --install .

Now use xcode-select to find all the necessary tools and directories.


In the case of the Mac OS X SDK, cmake is smart enough to find the root system directory on its own, without further instructions.

Next for each platform in order.

iOS devices (iPhone OS SDK)
We will collect in a separate folder, say build/ios .

We already found compilers, now we will decide on flags for the C ++ compiler. Since we will build for armv7 , armv7s and arm64 architectures, we will use the special -arch flag for clang++ .

All necessary system libraries for ARM architectures are located in the SDKs/iPhoneOS7.0.sdk subdirectory of XCODE_ARM_ROOT

 CXX_FLAGS="-arch armv7 -arch armv7s -arch arm64" SYSTEM_ROOT=${XCODE_ARM_ROOT}/SDKs/iPhoneOS7.0.sdk 

The cmake call will look like this.

 mkdir -p build/ios cd build/ios cmake \ -DCMAKE_CXX_COMPILER=$CXX_COMPILER \ -DCMAKE_OSX_SYSROOT="$SYSTEM_ROOT" \ -DCMAKE_C_COMPILER=$C_COMPILER \ -DCMAKE_CXX_FLAGS="$CXX_FLAGS" \ -G "Unix Makefiles" \ ../../src/Shark 

By the way, cmake will do a lot of extra work for us. We only need to specify C and C ++ compilers, cmake will find all other utilities from the same toolkit, for example, ranlib , lipo , ld , etc.

iOS simulator (iPhone Simulator SDK)
Same compilers.

The architecture we need at this time is i386 and x86_64 , the latter is needed for testing on a simulator of devices with 64-bit architecture, for example, “iPhone Retina (4-inch 64-bit)”.

The required libraries are in ${XCODE_SIM_ROOT}/SDKs/iPhoneSimulator7.0.sdk .

And another very important point is that the code is collected for the simulator, and not OS X with the same architecture, the compiler should be given an additional prompt using the -mios-simulator-version-min=7.0 flag

 mkdir -p build/sim cd build/sim CXX_FLAGS="-arch i386 -arch x86_64 -mios-simulator-version-min=7.0" SYSTEM_ROOT=${XCODE_SIM_ROOT}/SDKs/iPhoneSimulator7.0.sdk cmake \ -DCMAKE_CXX_COMPILER=$CXX_COMPILER \ -DCMAKE_OSX_SYSROOT="$SYSTEM_ROOT" \ -DCMAKE_C_COMPILER=$C_COMPILER \ -DCMAKE_CXX_FLAGS="$CXX_FLAGS" \ -G "Unix Makefiles" \ ../../src/Shark 

Mac os x
Finally, Mac OS X.
At this time, it suffices to specify only the C and C ++ compilers, cmake will find everything else itself.

 mkdir -p build/osx cd build/osx cmake \ -DCMAKE_CXX_COMPILER=$CXX_COMPILER \ -DCMAKE_C_COMPILER=$C_COMPILER \ -G "Unix Makefiles" \ ../../src/Shark 

Tricks cmake

Well now let's do make !, - you think, but no.
cmake has a couple of features to consider.

Compiler test
First, the very first run cmake runs a test for C and C ++ compilers. Oddly enough, clang and clang ++ successfully fail this test for iOS platforms. On the Internet, there are several recommendations on how to bypass this test, for example, in the CMakeLists.txt file, add NONE to the project description, but this did not help me.

 PROJECT( shark NONE ) 

The way that worked for me is to first run cmake without specifying clang and clang ++ compilers and other options. cmake generate a CMakeCache.txt file and a CMakeFiles directory. If you now run cmake with different settings, the compilers test will not be executed this time.

Configuration change
So, you need to run cmake at least 2 times - you will think again.
And again - no.

If important parameters, such as C and C ++ compilers, were changed during the next start, the changes will not take effect immediately. cmake will issue a warning and advise you to run it again. In fact, cmake will not say anything like that, but the ccmake utility ccmake provide more information. ccmake is a console graphic interface for cmake .

Generally

• Once cmake , without parameters to pass the test
• Two cmake , with the necessary parameters
• Three cmake to make changes take effect.

make

Yes, you can now perform the coveted make . To make things go faster, you can parallelize the assembly using the -j flag.

 make -j16 

This process will not take much time, as a result we will have the static library libshark.a in our hands. For every fireman, with the help of the file utility, make sure that the resulting libraries support all the necessary architectures.

Make a "thick" library

Libraries for iOS devices and a simulator need to be combined into one “thick” library.

It's all quite simple, the lipo utility will cope with this task lipo

 mkdir -p lib/ios $XCODE_TOOLCHAIN_BIN/lipo -create build/ios/libshark.a build/sim/libshark.a -o lib/ios/libshark.a 

Just in case, check the resulting library ( file lib/ios/libshark.a ) to make sure that all 5 architectures are in place.

Pack in a framework

The time has come to carefully pack the library in the form of a framework. The framework directory is often referred to as a “bundle”.

Create a bundle

At this stage, we will create the Shark.framework directory and the necessary folder structure inside, with all the necessary symbolic links, using mkdir and ln for this.

 Shark.framework/ ├── Documentation -> Versions/Current/Documentation ├── Headers -> Versions/Current/Headers ├── Resources -> Versions/Current/Resources └── Versions ├── A │ ├── Documentation │ ├── Headers │ └── Resources └── Current -> A 

Copy the library

Now we will copy the static library inside the bundle, while renaming it to Shark .

 cp build/ios/libshark.a Shark.framework/Versions/A/Shark 

Copy Header Files

Next, copy all .h files from the src/Shark/include to the Headers folder inside the bundle. Then remove the unnecessary statistics.h . This file is not needed at least because there is no corresponding INSTALL command in the CMakeLists.txt file.

 cp -r src/Shark/include/* Shark.framework/Headers/ rm Shark.framework/Headers/statistics.h 

Patch header files
It would seem that heders copied everything, but again everything is not so obvious. If you try to use the framework right now, it will pop up several obscure errors related to the paths to the header files. And if in the case of the dynamic library under OS X I managed to get around the problem using the Header Search Path, then in the case of the framework, everything is not so simple.

Since the Shark developers have not taken care of the correct paths for the #include directives, we will have to do this work for them.

In search of the right approach, I turned my attention to an example of a well-organized library, namely - boost. All paths in the #include directives for boost components start with boost/ , for example

 #include "boost/config.hpp" #include <boost/type_traits/remove_reference.hpp> 

Use the sed editor and patch files using the following rules for all #include directives

• Add the missing space in "#include <something>" and "#include" something "" (yes, there were such)
• SharkDefs.h with Shark/SharkDefs.h
• Add Shark/ in the path for all components of the library

Components here mean subdirectories of the Headers folder, i.e. Array, Rng, LinAlg, FileUtil, EALib, MOO-EALib, ReClaM, Mixture, TimeSeries, Fuzzy .

Create an Info.plist

The last step in creating the framework is to create the appropriate Info.plist. Important properties are the name and version of the framework.

Before the release of iOS 7, namely, devices with the arm64 architecture, it was possible to create universal static libraries, and therefore frameworks, simultaneously for iOS and OSX, including a simulator. Nothing prevented packing armv6, armv7, armv7s, i386 and x86_64 into one bold library. Devices with 64-bit architecture confused all the cards. In a “thick” library, there cannot be two layers with the same architecture, which means you cannot get x86_64 for iOS simulator of 64-bit devices and x86_64 for OS X.

Publish to CocoaPods

Right now, you can take a ready-made framework and safely add it to your projects, but I want to automate this stage too. This is exactly what CocoaPods is for . If for some reason you are behind the train and are not using CocoaPods yet, it's time to pay attention to this project.

Creating a specification for the hearth is another story. Despite the fact that I almost painted every little thing up to this point, here I refer to the finished result.


And an example of use.

Results

So, if you suddenly come up with yourself some kind of task, the result of which, to put it mildly, is not guaranteed, try to do something anyway. You never know what new skills will be able to pump as a result.