As we struggle with dynamic libraries in Swift. Yandex experience
Honestly, when we started to work on restarting Yandex.Map, I couldn’t imagine how many problems we would end up with in Swift. If you started writing on Swift quite recently, then believe me, you missed all the fun. Some two years ago there was no incremental compilation in principle (even a space in any file led to a complete reassembly), the compiler itself constantly crashed with Segmentation Fault on quite harmless things like triple nesting of types or inheritance from Generic, the project indexation lasted an unimaginably long time, autocompletion worked once and so on and so forth, all misfortunes do not count. Such moments undoubtedly complicate the life of programmers, but are gradually solved with each Xcode update. However, there are more significant problems affecting not only the development, but also the quality of the application: the prohibition of compiling static libraries from SWIFT code, and the lack of Swift support at the iOS level.
Initially, it was not obvious that the use of Swift and dynamic libraries leads to an increase in launch time. We did not compare the launch time with the previous version and took the long load as a given. Yes, and diagnostic tools that actually happens at the stage of loading the application, in general, was not. But one day, Apple developers added the ability to profile the work of the system loader. It turned out that loading dynamic libraries takes a very long time compared to other stages. Of course, with our code, not everything was perfect either, but perhaps these are particular features of a separate application and not everyone will be interested in reading about them. But the struggle with dynamic libraries is a common theme for all developers using Swift. It is about this issue that will be discussed.
The application is loaded in two stages. Prior to launching main, the system loader does the job of preparing the application image in memory:
')
1. loads dynamic libraries,
2. puts addresses to external pointers (bind) and base addresses to internal pointers (rebase),
3. creates an Objective-C context,
4. calls constructors of global C ++ variables and + load methods in Objective-C classes.
Only after that the application code starts to run.
Measuring pre-main is not a trivial task, since this stage is performed by the system and cannot be pledged as user code. Fortunately, at WWDC 2016: Optimizing App Startup Time talked about the DYLD_PRINT_STATISTICS environment variable , which, when turned on, displays the loader operation statistics in stages. For example, for an empty Swift application when running on iPhone 5, the statistics are as follows:
The loading of system libraries is optimized - just make sure of this by creating an empty project on Objective-C and running it with DYLD_PRINT_LIBRARIES & DYLD_PRINT_STATISTICS:
Before you start working on reducing the number of dynamic libraries, you need to determine their minimum set in the application using Swift. Obviously, these will be dynamic libraries linked to an empty project. To see them, you need to go to the assembled bundle after the build (via “Show in Finder” in the context menu) and go to the Frameworks folder:
These are the so-called Swift standard libraries (swift runtime). If at least one * .swift file is added to the project, Xcode copies them to a bundle and links to the binary file. What are they needed for? It's all about the youth of the language. Swift continues to evolve and does not support binary compatibility . If the swift runtime had been made part of the system (as it had long been done for Objective-C), then with the next iOS update, the old programs could not work on the new version of the system and vice versa. Therefore, applications contain a copy of the swift runtime in the Frameworks folder, and the system treats them as custom, hence the long load. Such is the charge for using a dynamically developing language.
Let us turn to a more complex example. Let some application:
- uses CocoaPods to connect dependencies, with some dependencies coming in ready-made dynamic libraries,
- divided into several targets,
- uses CoreLocation, MapKit, AVFoundation.
Inside his bundle, in the Frameworks folder, are the following libraries:
Download statistics for this application on iPhone 5 looks like this:
As you can see, in this example, five swift runtime libraries are larger than in an empty project. If there is an import CoreLocation in any * .swift file, or #import <CoreLocation / CoreLocation.h> is in the bridging header, then Xcode adds libswiftCoreLocation.dylib to the bundle. However, using #import <CoreLocation / CoreLocation.h> in Objective-C code does not add this library. This suggests a solution - to make Objective-C wrappers over the necessary parts of CoreLocation and use only them in the application. An example of wrappers can be found here .
Unfortunately, this may not be enough due to transitive dependencies. Using import MapKit in any * .swift file results in adding libswiftmapkit.dylib and libswiftCoreLocation.dylib, using import AVFoundation to adding libswiftAVFoundation.dylib, libswiftCoreAudio.dylib, libswiftCoreMedia.dylib. Therefore, the necessary parts of MapKit and AVFoundation also have to wrap. And libswiftCoreLocation.dylib is added if there is #import <CoreLocation / CoreLocation.h> in any header file on which the bridging header transitively depends. If this #import is in a library, then it will also need to be wrapped. All this sounds unpleasant, but the result is justified - you can achieve the same set of Swift standard libraries as in an empty application.
The next mass source of dynamic libraries is scams collected into dynamic frameworks when specifying! Use_frameworks in a Podfile. The! Use_frameworks flag is required for connecting dependencies written in Swift, since Xcode does not allow using Swift in static frameworks - throws out the error "Swift is not supported for static libraries".
In fact, this does not mean that you cannot create and use static libraries with Swift code, since a static library is just an archive of object files. The Swift compiler for each source file generates the usual Mach-O object files. With ar or libtool, you can archive them into a static library and substitute the result in a link command:
- Let SomeLib module consist of two files: SomeClass.swift and SomeOtherClass.swift. SomeLib can be built with Xcode 8.3.1 into a static library and linked to main.swift with the following commands:
Fortunately, in Xcode 9 beta 4, the ban on using Swift in static frameworks has been removed. You can wait for the release of Xcode and the corresponding edits in cocoapods (so that static frameworks are compiled), and the problem will disappear by itself. But for those who do not plan or can not switch to Xcode 9, it is worth mentioning the existing rather simple solution - cocoapods-amimono . The idea is simple - after building each subteam in a separate build folder, build artifacts, including object files, remain. Instead of linking to dynamic libraries, you can link directly to the object files from which they were collected. Cocoapods-amimono:
- adds a build phase that executes the script that makes up the LinkFileList from the object files in the build folders,
- linking with the framework framework replaces linking with LinkFileList,
- removes embedding frameworks in the application bundle.
The solution works: the framework frameworks disappear from the Frameworks folder, and you can use module import, that is, the application code does not change.
In the same way, you can get rid of dynamic frameworks collected from user targets: either wait for Xcode 9 (and use static frameworks), or link object books directly into a binary application file, as cocoapods-amimono does. For this you need:
- leave the target in the dependencies of the main target,
- do not embed the framework in the bundle and do not link with it,
- add build phase, component LinkFileList, by analogy with cocoapods-amimono:
With ready-made dynamic frameworks, it is more difficult, since a dynamic library cannot be converted into a static library - this is essentially an executable file that allows only dynamic linking. If these are the core-framework of the application and its symbols are needed right at the start - nothing can be done, its use will inevitably increase the launch time. But if the framework is not used when starting the program, then you can load it lazily through dlopen. And it is lazy to load via dlopen + dlsym only an Objective-C compatible part of the interface, since the module import into Swift links the library automatically. If everything you need is available from Objective-C, then you need:
1. Remove the linking of the library with the main target. If the dependency is connected via cocoapods, then the linking can be removed by adding a fake target (to which the problem file will be attached) or via post_install in the Podfile:
- Download library.
As a result, only the swift runtime libraries and vendored frameworks remain, loaded as lazily as possible. Moreover, the swift runtime library set is the same as for an empty application. Statistics pre-main now looks like this:
There are a couple of simple suggestions how not to spoil the achieved results with the next update. The first is to add the execution of the script to the build-phases, which checks after building a list of libraries and frameworks in the Frameworks folder of the application bundle - whether something new has appeared.
The listed problems are entirely generated by the youth of Swift. Some of them will disappear with the release of Xcode 9, which allowed static libraries on Swift, which will get rid of crutches like cocoapods-amimono. But finally, the problem of increasing the size of the bundle and the application launch time will be solved only when the swift runtime becomes part of iOS. Moreover, for some time after this, applications will have to carry it with them in order to maintain previous versions of the system. The development of Swift 5 is aimed at stabilizing the Swift standard library binary interface. The binary interface was planned to stabilize in Swift 4, but Xcode 9 still copies the swift runtime into the application bundle with the deployment target iOS 11, which means that Swift is still not part of iOS.
Initially, it was not obvious that the use of Swift and dynamic libraries leads to an increase in launch time. We did not compare the launch time with the previous version and took the long load as a given. Yes, and diagnostic tools that actually happens at the stage of loading the application, in general, was not. But one day, Apple developers added the ability to profile the work of the system loader. It turned out that loading dynamic libraries takes a very long time compared to other stages. Of course, with our code, not everything was perfect either, but perhaps these are particular features of a separate application and not everyone will be interested in reading about them. But the struggle with dynamic libraries is a common theme for all developers using Swift. It is about this issue that will be discussed.
Pre-main
The application is loaded in two stages. Prior to launching main, the system loader does the job of preparing the application image in memory:
')
1. loads dynamic libraries,
2. puts addresses to external pointers (bind) and base addresses to internal pointers (rebase),
3. creates an Objective-C context,
4. calls constructors of global C ++ variables and + load methods in Objective-C classes.
Only after that the application code starts to run.
Measuring pre-main is not a trivial task, since this stage is performed by the system and cannot be pledged as user code. Fortunately, at WWDC 2016: Optimizing App Startup Time talked about the DYLD_PRINT_STATISTICS environment variable , which, when turned on, displays the loader operation statistics in stages. For example, for an empty Swift application when running on iPhone 5, the statistics are as follows:
Total pre-main time: 1.0 seconds (100.0%)
dylib loading time: 975.17 milliseconds (95.8%)
rebase/binding time: 14.39 milliseconds (1.4%)
ObjC setup time: 12.46 milliseconds (1.2%)
initializer time: 15.27 milliseconds (1.6%)The loading of system libraries is optimized - just make sure of this by creating an empty project on Objective-C and running it with DYLD_PRINT_LIBRARIES & DYLD_PRINT_STATISTICS:
dyld: loaded: /var/containers/Bundle/Application/6232DEDA-1E38-44B9-8CE8-01E244711306/Test.app/Test
...
dyld: loaded: /System/Library/Frameworks/JavaScriptCore.framework/JavaScriptCore
dyld: loaded: /System/Library/Frameworks/AudioToolbox.framework/AudioToolbox
dyld: loaded: /System/Library/PrivateFrameworks/TCC.framework/TCC
Total pre-main time: 19.65 milliseconds (100.0%)
dylib loading time: 1.32 milliseconds (6.7%)
rebase/binding time: 1.30 milliseconds (6.6%)
ObjC setup time: 5.11 milliseconds (26.0%)
initializer time: 11.90 milliseconds (60.5%)Minimum set of dynamic libraries
Before you start working on reducing the number of dynamic libraries, you need to determine their minimum set in the application using Swift. Obviously, these will be dynamic libraries linked to an empty project. To see them, you need to go to the assembled bundle after the build (via “Show in Finder” in the context menu) and go to the Frameworks folder:
These are the so-called Swift standard libraries (swift runtime). If at least one * .swift file is added to the project, Xcode copies them to a bundle and links to the binary file. What are they needed for? It's all about the youth of the language. Swift continues to evolve and does not support binary compatibility . If the swift runtime had been made part of the system (as it had long been done for Objective-C), then with the next iOS update, the old programs could not work on the new version of the system and vice versa. Therefore, applications contain a copy of the swift runtime in the Frameworks folder, and the system treats them as custom, hence the long load. Such is the charge for using a dynamically developing language.
Fighting dynamic libraries
Let us turn to a more complex example. Let some application:
- uses CocoaPods to connect dependencies, with some dependencies coming in ready-made dynamic libraries,
- divided into several targets,
- uses CoreLocation, MapKit, AVFoundation.
Inside his bundle, in the Frameworks folder, are the following libraries:
Download statistics for this application on iPhone 5 looks like this:
Total pre-main time: 3.6 seconds (100.0%)
dylib loading time: 3.5 seconds (95.3%)
rebase/binding time: 50.04 milliseconds (1.3%)
ObjC setup time: 59.78 milliseconds (1.6%)
initializer time: 60.02 milliseconds (1.8%)Reducing Swift standard libraries
As you can see, in this example, five swift runtime libraries are larger than in an empty project. If there is an import CoreLocation in any * .swift file, or #import <CoreLocation / CoreLocation.h> is in the bridging header, then Xcode adds libswiftCoreLocation.dylib to the bundle. However, using #import <CoreLocation / CoreLocation.h> in Objective-C code does not add this library. This suggests a solution - to make Objective-C wrappers over the necessary parts of CoreLocation and use only them in the application. An example of wrappers can be found here .
Unfortunately, this may not be enough due to transitive dependencies. Using import MapKit in any * .swift file results in adding libswiftmapkit.dylib and libswiftCoreLocation.dylib, using import AVFoundation to adding libswiftAVFoundation.dylib, libswiftCoreAudio.dylib, libswiftCoreMedia.dylib. Therefore, the necessary parts of MapKit and AVFoundation also have to wrap. And libswiftCoreLocation.dylib is added if there is #import <CoreLocation / CoreLocation.h> in any header file on which the bridging header transitively depends. If this #import is in a library, then it will also need to be wrapped. All this sounds unpleasant, but the result is justified - you can achieve the same set of Swift standard libraries as in an empty application.
Static linking of source files
The next mass source of dynamic libraries is scams collected into dynamic frameworks when specifying! Use_frameworks in a Podfile. The! Use_frameworks flag is required for connecting dependencies written in Swift, since Xcode does not allow using Swift in static frameworks - throws out the error "Swift is not supported for static libraries".
In fact, this does not mean that you cannot create and use static libraries with Swift code, since a static library is just an archive of object files. The Swift compiler for each source file generates the usual Mach-O object files. With ar or libtool, you can archive them into a static library and substitute the result in a link command:
- Let SomeLib module consist of two files: SomeClass.swift and SomeOtherClass.swift. SomeLib can be built with Xcode 8.3.1 into a static library and linked to main.swift with the following commands:
DEVELOPER_DIR=/Applications/Xcode8.3.1.app/Contents/Developer/
SWIFTC=$DEVELOPER_DIR/Toolchains/XcodeDefault.xctoolchain/usr/bin/swiftc
SDK=$DEVELOPER_DIR/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS10.3.sdk
#
$SWIFTC -target armv7-apple-ios8.0 -sdk $SDK -module-name SomeLib \
SomeClass.swift SomeOtherClass.swift -c
# swiftmodule, main.swift
$SWIFTC -target armv7-apple-ios8.0 -sdk $SDK -module-name SomeLib \
SomeClass.swift SomeOtherClass.swift -emit-module
#
$libtool -static -o libSomeLib.a SomeClass.o SomeOtherClass.o
# main.swift libSomeLib.a
$SWIFTC -target armv7-apple-ios8.0 -sdk $SDK -I . -L . main.swift -lSomeLibFortunately, in Xcode 9 beta 4, the ban on using Swift in static frameworks has been removed. You can wait for the release of Xcode and the corresponding edits in cocoapods (so that static frameworks are compiled), and the problem will disappear by itself. But for those who do not plan or can not switch to Xcode 9, it is worth mentioning the existing rather simple solution - cocoapods-amimono . The idea is simple - after building each subteam in a separate build folder, build artifacts, including object files, remain. Instead of linking to dynamic libraries, you can link directly to the object files from which they were collected. Cocoapods-amimono:
- adds a build phase that executes the script that makes up the LinkFileList from the object files in the build folders,
- linking with the framework framework replaces linking with LinkFileList,
- removes embedding frameworks in the application bundle.
The solution works: the framework frameworks disappear from the Frameworks folder, and you can use module import, that is, the application code does not change.
Static linking of own targets
In the same way, you can get rid of dynamic frameworks collected from user targets: either wait for Xcode 9 (and use static frameworks), or link object books directly into a binary application file, as cocoapods-amimono does. For this you need:
- leave the target in the dependencies of the main target,
- do not embed the framework in the bundle and do not link with it,
- add build phase, component LinkFileList, by analogy with cocoapods-amimono:
# ,
DEPENDENCIES=('SomeTarget' 'SomeOtherTarget');
ARCHS_LIST=($ARCHS)
# ,
for ARCH in ${ARCHS[@]}; do
DIR=$OBJECT_FILE_DIR_normal/$ARCH
# LinkFileList
FILE_PATH=$DIR/$TARGET_NAME.Dependencies.LinkFileList
FILE_LIST=""
#
for DEPENDENCY in "${DEPENDENCIES[@]}"; do
# , -
PATH=$CONFIGURATION_TEMP_DIR/${DEPENDENCY}.build/Objects-normal/$ARCH
#
SEARCH_EXP="$PATH/*.o"
# , SEARCH_EXP
for OBJ_FILE in $SEARCH_EXP; do
# FILE_LIST
FILE_LIST+="${OBJ_FILE}\n"
done
done
FILE_LIST=${FILE_LIST%$'\n'}
# FILE_LIST FILE_PATH
echo -n -e $FILE_LIST > $FILE_PATH
done-filelist "${OBJECT_FILE_DIR_normal}/${CURRENT_ARCH}/${TARGET_NAME}.Dependencies.LinkFileList"Lazy loading of dynamic libraries
With ready-made dynamic frameworks, it is more difficult, since a dynamic library cannot be converted into a static library - this is essentially an executable file that allows only dynamic linking. If these are the core-framework of the application and its symbols are needed right at the start - nothing can be done, its use will inevitably increase the launch time. But if the framework is not used when starting the program, then you can load it lazily through dlopen. And it is lazy to load via dlopen + dlsym only an Objective-C compatible part of the interface, since the module import into Swift links the library automatically. If everything you need is available from Objective-C, then you need:
1. Remove the linking of the library with the main target. If the dependency is connected via cocoapods, then the linking can be removed by adding a fake target (to which the problem file will be attached) or via post_install in the Podfile:
post_install do | installer |
# Amimono::Patcher.patch!(installer), amimono
# ,
installer.aggregate_targets.each do |aggregate_target|
# xcconfig-,
target_xcconfigs = aggregate_target.xcconfigs
# - xcconfig
aggregate_target.user_build_configurations.each do |config_name,_|
# xcconfig
path = aggregate_target.xcconfig_path(config_name)
#
xcconfig = Xcodeproj::Config.new(path)
#
xcconfig.frameworks.delete("SomeFramework")
#
xcconfig.save_as(path)
end
end
end- Download library.
#import <dlfcn.h>
NSString *frameworksPath = [[NSBundle mainBundle] privateFrameworksPath];
NSString *dyLib = @"DynamicLib.framework/DynamicLib";
//
NSString *path = [NSString stringWithFormat:@"%@/%@", frameworksPath, dyLib];
const char *pathPtr = [path cStringUsingEncoding:NSASCIIStringEncoding]
//
void *handle = dlopen(pathPtr, RTLD_LAZY);$nm -gU $BUNDLE_PATH/Frameworks/DynamicLib.framework/DynamicLib
DynamicLib (for architecture armv7):
00007ef0 S _DynamicLibVersionNumber
00007ec8 S _DynamicLibVersionString
0000837c S _OBJC_CLASS_$__TtC10DynamicLib16SomeClass
00008408 D _OBJC_METACLASS_$__TtC10DynamicLib16SomeClass
...
00004b98 T _someGlobalFunc
000083f8 D _someGlobalStringVar
000083f4 D _someGlobalVar
...// dlsym .
//
int (*someGlobalFuncPtr)(int) = dlsym(handle, "someGlobalFunc");
//
someGlobalFuncPtr(5);
//
int *someGlobalVarPtr = (int *)dlsym(handle, "someGlobalVar");
NSLog(@"%@", *someGlobalVarPtr);
//
NSString *__autoreleasing *someGlobalStringVarPtr =
(NSString *__autoreleasing *)dlsym(handle, "someGlobalStringVar");
NSLog(@"%@", *someGlobalStringVarPtr);
*someGlobalStringVar = @"newValue";#import <DynamicLib/SomeClass.h>
//dlsym Class
Class class = (__bridge Class)dlsym(handle,
"OBJC_CLASS_$__TtC10DynamicLib16SomeClass")
// class-
[class someClassFunc];
//
SomeClass *obj = [(Somelass *)[class alloc] init];
//
NSLog(@"%@", obj.someVar)
[obj someMethod];Optimization result
As a result, only the swift runtime libraries and vendored frameworks remain, loaded as lazily as possible. Moreover, the swift runtime library set is the same as for an empty application. Statistics pre-main now looks like this:
Total pre-main time: 1.0 seconds (100.0%)
dylib loading time: 963.68 milliseconds (90.0%)
rebase/binding time: 35.65 milliseconds (3.3%)
ObjC setup time: 29.08 milliseconds (2.7%)
initializer time: 41.35 milliseconds (4.0%)Saving result
There are a couple of simple suggestions how not to spoil the achieved results with the next update. The first is to add the execution of the script to the build-phases, which checks after building a list of libraries and frameworks in the Frameworks folder of the application bundle - whether something new has appeared.
FRAMEWORKS_DIR="${BUILT_PRODUCTS_DIR}/${FRAMEWORKS_FOLDER_PATH}"
FRAMEWORKS_SEARCH_PATTERN="${FRAMEWORKS_DIR}/*"
# , FRAMEWORKS_SEARCH_PATTERN
FRAMEWORKS=($FRAMEWORKS_SEARCH_PATTERN)
# Frameworks
ALLOWED_FRAMEWORKS=(libswiftFoundation.dylib SomeFramework.framework)
for FRAMEWORK in ${ALLOWED_FRAMEWORKS[@]}
do
PATTERN="*${FRAMEWORK}"
# , PATTERN
FRAMEWORKS=(${FRAMEWORKS[@]/${PATTERN}/})
done
echo ${FRAMEWORKS[@]}
# FRAMEWORKS
#
exit ${#FRAMEWORKS[@]}var count: UInt32 = 0
//
let imagesPathsPointer: UnsafeMutablePointer<UnsafePointer?>! =
objc_copyImageNames(&count)
//
let expectedImages: Set = ["libswiftCore.dylib"]
//
let frameworksPath = Bundle.main.privateFrameworksPath ?? "none"
for i in 0..<count {
let pathPointer = imagesPathsPointer.advanced(by: Int(i)).pointee
let path = pathPointer.flatMap { String(cString: $0) } ?? ""
//
guard path.contains(frameworksPath) else { continue }
let name = (path as NSString).lastPathComponent
assert(expectedImages.contains(name))
}Conclusion
The listed problems are entirely generated by the youth of Swift. Some of them will disappear with the release of Xcode 9, which allowed static libraries on Swift, which will get rid of crutches like cocoapods-amimono. But finally, the problem of increasing the size of the bundle and the application launch time will be solved only when the swift runtime becomes part of iOS. Moreover, for some time after this, applications will have to carry it with them in order to maintain previous versions of the system. The development of Swift 5 is aimed at stabilizing the Swift standard library binary interface. The binary interface was planned to stabilize in Swift 4, but Xcode 9 still copies the swift runtime into the application bundle with the deployment target iOS 11, which means that Swift is still not part of iOS.
Source: https://habr.com/ru/post/335768/
All Articles