KTV. Rake on the way to marshaling

I wrote about KTV , but one thing is to come up with something incomprehensible, another is to try using it. In addition to the S2 style system, I plan to use KTV to work with the server instead of JSON. I have no plans to conquer the world, but I want to figure out whether it was more convenient or not. In order to communicate easily, you need to be able to parse objects from ktv-files, and serialize them back to them.

Swift, for which I am writing this, at the moment (Swift 2.x), is not intended for dynamic parsing at all, not at all, in general. So I had to think of something a bit strange and unusual. After that, this strange and non-standard needed to be implemented.

In the process, there was an incalculable number of rakes, about which I will tell. Maybe someone will laugh at the insensible me, maybe they will help someone to avoid similar things - I don't know. I was helpful to understand.
')
If anyone sees how easy or better to solve these problems, write. I am pleased to learn more options, since all that are listed below, to one degree or another, are crutches. Suddenly there is something more pleasant.

How to get the structure of the object?

The first task that arises if we want to convert what has come across the network into a native object (in my case of the Swift language) is to deal with the structure of the object. Recalling the Java experience (where two hundred libraries have already been written for each sneeze), I went through several methods.

Introspection of language objects

Reflection, runtime or at least something similar. In Swift, there are two directions that develop in this direction:

• Mirror class . This is the thing that the debugger or Playgrounds use to display information about objects. Accordingly, the information that they need: type, field name, value, generic. There are no access parameters (private / public), there are no annotations, and the method itself is not defined for all objects.
• The function reflect , which returns exactly the same data in a slightly different format.

Most likely, both of these methods converge somewhere in one, and therefore the result is similar. This method is the coolest if you can use it. Alas, it now works only for reading, there is no record in any form. We are waiting for the expansion of "mirrors", go to the next method.

Source code parsing Sourcekit

The next way is to parse the source itself, in which, obviously, everything that is possible is indicated. It would be extremely difficult if Apple did not provide SourceKit. What is SourceKit? This is a framework (sourcekitd.framework) that can perform queries like "proparsi, please, this file, and tell what you see there in the form of a syntax tree (Abstract Syntax Tree AST)." It is extremely useful that SourceKit is also able to parse documentation to elements of a language, and not just identifiers and types.

In addition to the framework itself, SourceKitten lives on Github , which provides an interface on Swift to sourcekitd.framework. Using it is very simple:

let sourceKitFile = File(path:classFileName) //   let structure = Structure(file:sourceKitFile!).dictionary //  AST 

True, in order to connect both sourcekitd.framework and SourceKitten, I had to get involved. It turned out something like this:

• download, source SourceKitten source code
• Do not forget the dependency, SWXMLHash.swift
• connect sourcekitd.framework and libclang.dylyb
• Write a Bridging-Header in which to include the necessary headers:
  

   #import "Index.h" #import "Documentation.h" #import "CXCompilationDatabase.h" #import "sourcekitd.h" 

After that, the lines above will work. After receiving the AST, the job is only to interpret it correctly (for this I wrote a class and structure with different fields and ran it, checked what SourceKit would give me. I managed to pull out the object types (though only if they are spelled explicitly, inferrence is not supported), and access modifiers, and understand where constants are, where they are not, and take generics from ordinary and associative arrays.

In addition, SourceKit also provides documentation for the code element. Here, however, is also not without a rake:

• for some reason, my Structure(file:String) does not show documentation (and there are not enough types inside the documentation structure), to see it, you need to execute another command:
  let structureWithDocs = SwiftDocs (file: sourceKitFile! arguments: [classFileName]) ?. docsDictionary
• behavior is not very deterministic. For example, if you @name in the comment, it disappears from AST. Maybe there are some other secret words, I don't know.

Well, got the structure, what to do next?

Work with structure

There are three ways to work with the structure:

• generate code at the compilation stage, which will marshal objects.
• write a common code, which (usually again using reflex or other introspection method) directly in the process of execution puts the necessary values ​​into the fields of the objects and takes them, producing a format for transmission.
• generate marshaling code right in the execution process.

The second method disappears because the mirrors in Swift do not know how to write anything to objects. The third method is very cool, but with no chance at all (unlike Java, where you can generate bytecode on the fly), only the first remains. That is, according to the structure, we need to create methods that will receive KTV or JSON as input, producing the necessary, filled object, or vice versa, receiving text in KTV or JSON formats from the object.

It is not necessary to generate the code in any way (just took and assigned values ​​to the fields):

• you need to work with all sorts of checks (including optional types, for example),
• be able to work with nested objects (class hierarchies)
• be able to customize (at least I'll fix it on a field with a different name)
• be able to use custom parsers / serializers (for example, if a date comes in a non-standard format)
• be able to generate code that is accessible from Objective-C, and not just from Swift.

There are many tasks, and in order to solve them all, I had to be seriously confused.

Stage 1. Just assignments

Before analyzing the code, let's see what we actually do. We have a model class that should create a bunch of model objects using data from a KTV file. The diagram represents the participants. Scheme as work a little more complicated.


How to do it in the simplest case? Let's start by learning how to pull data from a KTV object. Read the file - we have already read and saved to the KTV object, which in its structure is a bit like the associative array returned by NSJSONSerialization .

So, you need to write a function (or functions) that would pull a value out of KTV by key, and then assign it to property. The only difficulty was that the values ​​are Optional, and there are many different types. We analyze the steps.

First, we write a method that pulls a KTV value using an arbitrary key. In this case, the links are resolved, mixins are taken into account. I got something like that. I also use this method to get the link text in S2, so we return a tuple.

 private func valueAndReferenceForKey(key:String, resolveReferences:Bool = true) throws -> (value:KTVValue?, reference:String?) { var result:KTVValue? = properties[key] var reference:String? = nil if let result_ = result { switch result_ { case .reference(let link): if resolveReferences || link.hasPrefix("~") { result = try findObjectByReference(link) } else { result = nil reference = link .stringByReplacingOccurrencesOfString("@", withString:"") } default: break } } return (result, reference) } 

After that, I thought it was worth writing a generalized method that I can call to get values ​​of different types.

 private func specificValueForKey<T>(key:String, defaultValue:T?, resolveReferences:Bool, valueResolver:(value:KTVValue) throws -> T?) throws -> (value:T?, reference:String?) { let (resultValue, reference) = try valueAndReferenceForKey(key, resolveReferences:resolveReferences) var result = defaultValue if let result_ = resultValue { result = try valueResolver(value:result_) } return (result, reference) } 

Here the only interesting part is the resolver, which is able to get a value of a particular type from the KTV value. When this method is written, you can either use it directly, or write a few wrappers for standard types, to make it easier to use.

 public func string(key key:String, defaultValue:String? = "") throws -> String? { return try specificValueForKey(key, defaultValue:defaultValue, resolveReferences:true, valueResolver:KTVValue.stringResolver).value } 

I use another generalized method to work uniformly with optional types.

 private func deoptionizeValue<T>(value:T?, defaultValue:T) -> T { if let value_ = value { return value_ } else { return defaultValue } } 

As a result, the parser consists of very simple blocks, each of which is dedicated to one property.

 stringProperty = deoptionizeValue(value:string(key:"key"), defaultValue:"") 

Objects have to be dealt with a little less conveniently, but from the point of view of tricks, it is quite obvious. I did something like this:

 if let object_ = getGeneralObject(name:"object", ktv:ktv) { object = ChildObject(ktvLenient:object_) } else { object = nil } 

Stage 2. Dates, study

The next task that required a solution was date formatters. There may also be several solutions.

Use special model class methods to format fields. I used this option in Objective-C version of the same, and it worked well. The problem is only in the separation of the declaration of the field and its parameters. It is inconvenient, you constantly forget, or you forget to correct, if the name has been changed to peperti. Plus, there is still a problem with structures.

The fact is that the structures have an automatically created initializer with the names of all the fields. This is convenient, and, given the semantics of this type, it is economical. If we want to use some methods of an object or class, then we need to use self , which requires initialization before the call. Therefore, the structure needs the (empty) default init() . This is an extra code that needs to be written in each model class / in each structure (it cannot be generated, it should be in the main class) and which will also always be forgotten.

Use special classes, or stupid, as the types of property model class. That is, not String, but MappedString. This allows them to be configured (right at creation), for example, like this: var property:MappedDate = MappedDate(format:'dd-MM-yyyy') , and you can use their methods for serializing / parsing values. You can use tuples instead of a class, it looks absolutely monstrous, but it also works. This solution has a lot of minuses, the main one - to access the perperty, you will need to somehow dodge ( object.property.value , for example). Well, recording wildly.

Use mappers. Having tried the above options, I came to just that.

Stage 3. Custom mappers

Mapper, I call a class that knows how a certain type is converted from / to KTV. You can send KTV-value to it so that it returns the real type ( String , NSDate , ...), and vice versa, you can return the type so that it returns KTV-value (for serialization).

Normal mapper we have already implemented a couple of sections back, for the line. In principle, you can wrap this code in a class, and then choose classes depending on the type of property. At this point, it turns out an interesting feature of Swift.

The fact is that in the end, we need some kind of factory , which by type (by line or by class) will produce the desired object. It would be nice if this factory would issue a generalized object. Then there will be no ghosts, and the call logic will be as simple as possible, somehow

 let stringMapper:Mapper<String> = mapper.getFor("String") var value:String = stringMapper.valueFromKTV(ktvValue) 

The problem is how to make this method func getFor<T>(className:String) -> Mapper<T> . Due to the peculiarities of generics in the protocols (more precisely, their absence, associative types are used instead), for example, it is impossible to make an array of generalized objects. That is, it cannot be this way (it doesn’t matter whether you specify the generic type or not) var mappers:\[Mapper] = \[] . The array must already have specific types.

As a result, I had to cheat a little. The array of mappers inside the factory had to be made up of some kind of common non-generic protocol — the parent of all mappers. One could just make an AnyObject array, but somehow it does not work out well.

 public protocol KTVModelAnyMapper {} public class KTVModelMapper<MappingType>: KTVModelAnyMapper { ... } public class KTVModelMapperFactory { private var _mappersByType = [String:KTVModelAnyMapper]() public func mapper<T>(type:String, propertyName:String) throws -> KTVModelMapper<T> { if let mapper = _mappersByType[propertyName] as? KTVModelMapper<T> { return mapper } else { throw KTVModelObjectParseableError.CantFindTypeMapper } } } 

I did not invent a solution without a cast. Maybe someone from the readers tell?

As a result, the parser code was like this:

 let mappers = KTVModelMapperFactory() do { _stringOrNil = try mappers .mapper("String", propertyName:mappers.ktvNameFor("stringOrNil")) .parseOptionalValue(ktv["stringOrNil"], defaultValue:_stringOrNil) } catch { errors["stringOrNil"] = error } 

The beauty of this solution is that for each model class you can substitute your factory, and thus control how the ktv values ​​are mapped into the object.


The question remains, how to set this custom factory for a model object? I came up with two options:

• Use the custom protocol. In the name of the protocol, encode the name of the custom factory, and in the process of creating the parser extension, pull out the name by the SourceKit, from there the name of the factory, and connect when parsing. The solution is working, I checked, but terribly crooked.
• No less crooked decision (I don’t know any good ones at all), but at least beautiful - the use of comments.

Stage 4. Annotations in the comments

The fact is that, in addition to information about classes and structures, SourceKit also provides comments tied to elements. The ones from which documentation is then obtained, starting with /// or /** */ . Thus, there you can stuff anything, parse it anything as you like, and do what we want. It is clear that no typing here and does not smell, writing - solely on the conscience of the developer, but, having tried all of the above methods (and a couple more very nightmarish), it turns out that this is the most adequate.

It looks like a comment with a custom factory for a model class like this:

 /// @mapper = RootObjectMapperFactory public struct RootObject: KTVModelObject { ... } 

And so, for example, you can control the name of the property in KTV:

 /// @ktv = __date__ var date:NSDate 

However, in this place the possibilities are endless, since we are moving away from Swift and starting to just parse arbitrary text. You can set the date formats, you can - an arbitrary code that will then fit into the parser or serializer.

Result, conclusions

Swift is still a bad language for “magic” libraries. That is, those in which you put something simple, it is brewed there and given another simple and beautiful. At the same time, from the developer’s side, “nothing needs to be done” (supposedly). This type of library is always the most difficult, but it shows the power of the platform. This is Hibernate, this is Rails, this is CoreData, and so on. On Swift, writing this now is insanely difficult and only SourceKit reduces the complexity to acceptable, it would not exist, we would have to parse the classes with our hands, which, to put it mildly, is ungrateful.

However, as charging for the mind, this code turned out to be great. I did not find so many rakes in one place for a long time. You can touch what happened here: https://github.com/bealex/KTV This is a very live, non-production code where I learn how to work with Swift, so please treat it as well.

I hope you will be interested. If suddenly there are questions - ask!