How we developed the AR-application for the review of historical places

Recently, we have combined technology with the old technology of modern, what came of it, read under the cut.

Augmented Reality

Applications with augmented reality as guides to cities - a theme well-known and implemented by many developers. This direction of using AR appeared one of the first because it allows you to use all the obvious possibilities of augmented reality: to show users information about buildings, to give information about the work of the institution and to get acquainted with the sights. At the last hackathon, which was carried out within the company, several projects were presented using augmented reality, and we had the idea to create an AR application that would show what the landmark or historical place looked like in the past. To do this - combine modern augmented reality technology with vintage photos. For example, facing the St. Isaac’s Cathedral, it will be possible to point the camera of a smartphone on it and see its first wooden building, which was dismantled in 1715.

The mechanics of operation are as follows: the application displays specified historical places and sights of the city on a map, displays brief information about them, notifies the user with the help of notifications that it is located near an interesting point. When a person approaches a historical monument at a distance of 40 meters, an AR mode becomes available. This opens the camera, and brief information about the objects is displayed directly in the surrounding user space. The latter has the ability to interact with virtual objects: by touching the card of a historical place, you can go to view the album with images.
')
It would seem that the application is very simple, however, it was not without pitfalls. I will not bore you with a story about the implementation of trivial things like loading data from a server or displaying points on a map, I’ll go straight to the functions that caused the problems.

Problem 1. Floating points

So, first of all, it was necessary to place markers in the space in accordance with the real location of historical places relative to the current location and direction of the user's gaze.

To begin with, we decided to use the already ready library for iOS: ARKit-CoreLocation The project lies on GitHub in free access, contains in addition to the code of the main classes, examples of integration and allows you to perform the task of interest to us in a couple of hours. It is only necessary to feed the library the coordinates of the points and the image used as a marker.

It is no wonder that for this ease it was necessary to pay. Marker points constantly floated in space: now they climbed to the ceiling, then they were drawn somewhere under their feet. Not every user would agree to catch the AR object in focus for a few minutes in order to get acquainted with the information of interest.

As it turned out, many people encountered this library bug, but no solution has yet been found. The code on GitHub, unfortunately, has not been updated for more than six months, so I had to go around.

We tried to use altitude in the coordinates instead of a fixed height above sea level, which the LocationManager returned for the current position of the user. However, this did not completely eliminate the problem. The data coming from the Location Manager, began to jump with a range of up to 60 meters, it was enough to twist the device in his hands. As a result, the picture was unstable, which, of course, did not suit us again.

As a result, it was decided to abandon the ARKit-CoreLocation library and place points in space on their own. The ARKit and CoreLocation article written by Christopher Web-Orenstein helped greatly in this. I had to spend a little more time and brush up on some mathematical aspects, but the result was worth it: AR-objects finally found themselves in their places. After that, it remains only to scatter them along the Y axis, so that the inscriptions and points are easier to read, and to put the correspondence between the distance from the current position to the point and the Z coordinate of the AR object, so that information about the nearest historical places is in the foreground.

It was necessary to calculate the new position of SCNNode in space, focusing on the coordinates:

let place = PlaceNode() let locationTransform = MatrixHelper.transformMatrix(for: matrix_identity_float4x4, originLocation: curUserLocation, location: nodeLocation, yPosition: pin.yPos, shouldScaleByDistance: false) let nodeAnchor = ARAnchor(transform: locationTransform) scene.session.add(anchor: nodeAnchor) scene.scene.rootNode.addChildNode(place) 

In the MatrixHelper class the auxiliary functions were rendered:

 class MatrixHelper { static func transformMatrix(for matrix: simd_float4x4, originLocation: CLLocation, location: CLLocation, yPosition: Float) -> simd_float4x4 { let distanceToPoint = Float(location.distance(from: originLocation)) let distanceToNode = (10 + distanceToPoint/1000.0) let bearing = GLKMathDegreesToRadians(Float(originLocation.coordinate.direction(to: location.coordinate))) let position = vector_float4(0.0, yPosition, -distanceToNode, 0.0) let translationMatrix = MatrixHelper.translationMatrix(with: matrix_identity_float4x4, for: position) let rotationMatrix = MatrixHelper.rotateAroundY(with: matrix_identity_float4x4, for: bearing) let transformMatrix = simd_mul(rotationMatrix, translationMatrix) return simd_mul(matrix, transformMatrix) } static func translationMatrix(with matrix: matrix_float4x4, for translation : vector_float4) -> matrix_float4x4 { var matrix = matrix matrix.columns.3 = translation return matrix } static func rotateAroundY(with matrix: matrix_float4x4, for degrees: Float) -> matrix_float4x4 { var matrix : matrix_float4x4 = matrix matrix.columns.0.x = cos(degrees) matrix.columns.0.z = -sin(degrees) matrix.columns.2.x = sin(degrees) matrix.columns.2.z = cos(degrees) return matrix.inverse } } 

To calculate the azimuth added extension CLLocationCoordinate2D

 extension CLLocationCoordinate2D { func calculateBearing(to coordinate: CLLocationCoordinate2D) -> Double { let a = sin(coordinate.longitude.toRadians() - longitude.toRadians()) * cos(coordinate.latitude.toRadians()) let b = cos(latitude.toRadians()) * sin(coordinate.latitude.toRadians()) - sin(latitude.toRadians()) * cos(coordinate.latitude.toRadians()) * cos(coordinate.longitude.toRadians() - longitude.toRadians()) return atan2(a, b) } func direction(to coordinate: CLLocationCoordinate2D) -> CLLocationDirection { return self.calculateBearing(to: coordinate).toDegrees() } } 

Problem 2. Excess AR Objects

The next problem we encountered was a huge number of AR objects. There are a lot of historical places and sights in our city, so the dies with information merged and crawled one over the other. The user with great difficulty would be able to disassemble some of the inscriptions, and this could make a repulsive impression. After consulting, they decided to limit the number of simultaneously displayed AR-objects, leaving only points within a radius of 500 meters from the current location.

However, in some areas the concentration of the points was still too high. Therefore, to increase clarity, we decided to use clustering. On the map screen, this feature is available by default due to the logic embedded in MapKit, but in the AR mode it was necessary to implement it manually.

The clustering was based on the distance from the current position to the target. Thus, if a point fell into a zone with a radius equal to half the distance between the user and the previous landmark from the list, it simply hid and was part of the cluster. When the user approached it, the distance decreased, the radius of the cluster zone decreased accordingly, so the sights located nearby did not merge into clusters. In order to visually distinguish clusters from single points, we decided to change the color of the marker and, instead of the place name, display the number of objects in AR.



To ensure the interactivity of AR-objects on ARSCNView, UITapGestureRecognizer was hung and in the handler using the hitTest method, it was checked which of the SCNNode objects the user clicked. If it was a photograph of a nearby landmark, the application opened the corresponding album in full screen.

Problem 3. Radar

During the implementation of the application, it was necessary to show points on a small radar. In theory, there should be no misunderstandings with this, because we have already calculated the azimuth and distance to the point, even managed to convert them into 3D coordinates. It remained only to place points in two-dimensional space on the screen.



In order not to reinvent the wheel, turned to the Radar library, the open source code of which is published on GitHub. Bright previews and flexible example settings were encouraging, but in fact the points were shifted relative to the true location in space. Having spent some time trying to correct the formulas, turned to a less beautiful, but more reliable option, described in the iPhone application Augmented Reality Toolkit :

 func place(dot: Dot) { var y: CGFloat = 0.0 var x: CGFloat = 0.0 if degree < 0 { degree += 360 } let bearing = dot.bearing.toRadians() let radius: CGFloat = 60.0 // radius of the radar view if (bearing > 0 && bearing < .pi / 2) { //the 1 quadrant of the radar x = radius + CGFloat(cosf(Float((.pi / 2) - bearing)) * Float(dot.distance)) y = radius - CGFloat(sinf(Float((.pi / 2) - bearing)) * Float(dot.distance)) } else if (bearing > .pi / 2.0 && bearing < .pi) { //the 2 quadrant of the radar x = radius + CGFloat(cosf(Float(bearing - (.pi / 2))) * Float(dot.distance)) y = radius + CGFloat(sinf(Float(bearing - (.pi / 2))) * Float(dot.distance)) } else if (bearing > .pi && bearing < (3 * .pi / 2)) { //the 3 quadrant of the radar x = radius - CGFloat(cosf(Float((3 * .pi / 2) - bearing)) * Float(dot.distance)) y = radius + CGFloat(sinf(Float((3 * .pi / 2) - bearing)) * Float(dot.distance)) } else if (bearing > (3 * .pi / 2.0) && bearing < (2 * .pi)) { //the 4 quadrant of the radar x = radius - CGFloat(cosf(Float(bearing - (3 * .pi / 2))) * Float(dot.distance)) y = radius - CGFloat(sinf(Float(bearing - (3 * .pi / 2))) * Float(dot.distance)) } else if (bearing == 0) { x = radius y = radius - CGFloat(dot.distance) } else if (bearing == .pi / 2) { x = radius + CGFloat(dot.distance) y = radius } else if (bearing == .pi) { x = radius y = radius + CGFloat(dot.distance) } else if (bearing == 3 * .pi / 2) { x = radius - CGFloat(dot.distance) y = radius } else { x = radius y = radius - CGFloat(dot.distance) } let newPosition = CGPoint(x: x, y: y) dot.layer.position = newPosition 

Backend

It remains to solve the problem of storing points and photos. For these purposes, it was decided to use Contentful, and in the current implementation of the project, he completely satisfied us.


At the time of mobile application development, all backenders were busy on commercial projects, and contentful allowed to provide for several hours:

• mobile developer - convenient backend
• content manager - convenient admin panel for filling data

A similar implementation of the backend was originally used by the teams that participated in the hackathon (mentioned at the beginning of the article), which once again proves that such things as hackathons make it possible to distract from solving their immediate problems on projects, provide an opportunity to create something and try new

Conclusion

Developing an AR-application was very interesting, in the process we tried several ready-made libraries, but we also had to remember mathematics and write a lot of things ourselves.

Simple at first glance, the project required a lot of working hours for the implementation and fine-tuning of the algorithms, despite the fact that we used the standard SDK from Apple.

Recently, we posted the application in the AppStore . Here is how it looks in work.


So far, we have points in the database only for Taganrog, however, everyone can participate in the expansion of the “coverage area”.