Integration of CoreSpotlight on the example of "Rambler. Mail"

In iOS 9, Apple added a spotlight search for third-party applications. This is especially important for applications with the same type of content: news applications, email clients, event posters. The iOS development department of the Rambler & Co holding has already integrated search into some of its applications.

This article will highlight the integration of CoreSpotlight into an existing project and talk about the problems encountered and their solutions.

Search API

To begin with, in order to present a complete picture of the current functionality and the places of its application, it is worth clarifying what content search capabilities in third-party applications are currently available in iOS 9.
Currently, the Search API consists of the following components:
')
NSUserActivity is a class that preserves and restores the displayed content and user activity. This class was introduced in iOS 8 for the implementation of Handoff (the ability to continue working in the same application on another device.). In iOS 9, this class is also used when opening an application from search results. It is a model that contains:

information to restore the state of the interface;
a description of how to interact with this information;
ways to store this information in the system.

It is possible to make this activity public, that is, it will appear in the search results not only of this device, but also among other users. It is worth noting that this opportunity will begin to work only in a few months.

CoreSpotlight is a framework that allows developers to index application content for later display in the spotlight search results. It presents two models:

attributes to display in the search;
properties to identify content.

CoreSpotlight also includes methods for adding and removing objects from the search.

Web Markup is a technology that allows an Apple bot to index the site content (special meta tags) for displaying it in search results and opening it in an application.

You can learn more about Web Markup and NSUserActivity from WWDC 2015, the documentation and links at the end of the article.

Formulation of the problem

We consider the integration of CoreSpotlight on the example of the application "Rambler. Mail". The application adds a search for folders, letters, attachments and contacts that are stored in the database (CoreData). By the time of the update, this database will already be filled. Based on this, the first requirement will be the primary indexation of all stored data .

Since all these objects can be created, deleted and changed, it is required to maintain consistency between the main database and the indexed objects . Since there can be a lot of data change operations and they can overlap each other, the system should be implemented in such a way that it processes change sets.

Based on the requirements described above, it is necessary to “collapse” the number of operations per indexing . With several changes of one object, it is enough to process only one update operation. When deleting an object, there is no point in processing requests for indexation of changes that already exist for it.

In the future, it is planned to integrate into other projects, so it is necessary that the indexing system be as independent as possible, able to index all types of objects , as well as easily integrate, modify and expand .

Since indexing can be long (especially primary), and can be interrupted, you need to keep its state. That is, you need to save all the unprocessed changes , and run their indexing on restarting the application. It is necessary to implement the system so as to minimize the number of calls to the database , and the number of auxiliary objects for storing changes was less than the number of objects being changed.

Structure

After setting the task, we can divide our system into several modules, each of which will be responsible for one of the functional tasks. In this case, modules that work with indexed objects will be closed by protocols, so that, if necessary, you can write your own implementation of the module, and the system would continue to work.

The data source is <ChangeProvider> . The class implementing this protocol provides the monitor with data about changes in objects. For each entity that needs to be indexed, there must be a provider. For convenience, a base class is implemented, which is initialized by NSFetchedResultsController, exposes itself as a delegate and proxies all events about changes in IndexerMonitor (it is described further on).

@protocol RamblerChangeProvider <NSObject> @property (weak, nonatomic) id<RamblerChangeProviderDelegate> delegate; @end

 @protocol RamblerChangeProviderDelegate <NSObject> - (void)changeProvider:(id<RamblerChangeProvider>)changeProvider didChangeObject:(id)object changeType:(RamblerChangeType)changeType; - (void)processChanges; - (NSArray *)obtainObjectsForInitialIndexing; @end

 @interface RamblerFetchedResultsControllerChangeProvider : NSObject <RamblerChangeProvider> + (instancetype)changeProviderWithFetchedResultsController:(NSFetchedResultsController *)controller; @end

Indexer - <Indexer> . The protocol of the object that is engaged in indexing data from the database. For each entity that needs to be indexed, its own indexer object is added. He is able to handle a set of changes and give a unique identifier for the object passed to him and the object by identifier. For convenience, a base class has also been implemented, from which to inherit. The indexer creates the batch processing operation and returns it to the monitor.

 @protocol RamblerIndexer <NSObject> - (NSOperation *)operationForIndexBatch:(RamblerIndexTransactionBatch *)batch withCompletionBlock:(RamblerErrorBlock)block; - (BOOL)canIndexObjectWithIdentifier:(NSString *)identifier; - (NSString *)identifierForObject:(id)object; - (id)objectForIdentifier:(NSString *)object; @end

 @interface RamblerIndexerBase : NSObject <RamblerIndexer> @property (strong, nonatomic) CSSearchableIndex *searchableIndex; - (CSSearchableItem *)searchableItemForObject:(id)object; - (BOOL)canIndexObjectWithType:(NSString *)objectType; @end

To work, you must override the two methods from RamblerIndexerBase and the last three methods from the RamblerIndexer protocol.

Identifier Generator - IndexIdentifierFormatter - an auxiliary object for generating and breaking an identifier into “meaningful parts” (parts of the string that are needed for recovery) to search for the original object. For example, the identifier for the letter would look like this: " RCMMessage_129_Inbox ". "RCMMessage" is the object type, " 129 " is the letter identifier in the folder, "Inbox" is the name of the folder in which the message is located. With this information, you can uniquely determine the type of object and find it.

 @protocol RamblerIndexIdentifierFormatter <NSObject> - (NSString *)identifierForObject:(id)object; - (BOOL)isCorrectIdentifier:(NSString *)identifier; @end

 @interface RamblerMessageIndexIdentifierFormatter : NSObject - (NSNumber *)messageUIDFromIdentifier:(NSString *)identifier; - (NSString *)folderNameFromIdentifier:(NSString *)identifier; @end

Change store - StateStorage - the module responsible for saving changes for their subsequent processing. He receives a transaction at the input that stores the type of change, the type of the changed object and its identifier, and then saves it to the database. For each type of object being indexed, there is one entry in this database. Each object contains OrderSet identifiers of various types of changes. This is necessary so that the identifiers for one type of object change are not repeated.

 @interface RamblerIndexerStateStorage : NSObject - (void)insertTransaction:(RamblerIndexTransaction *)transaction; - (void)insertTransactionsArray:(NSArray<NSArray *> *)transactionsArray                    changeType:(RamblerChangeType)changeType; - (RamblerIndexTransactionBatch *)obtainTransactionBatch; - (void)removeProcessedBatch:(RamblerIndexTransactionBatch *)batch - (BOOL)shouldPerformInitialIndexing; @end

System kernel (Monitor) - IndexerMonitor - this module connects all other parts of the system . The monitor contains:

a set of indexers and providers for each type of object;
StateStorage for saving changes coming from providers;
the queue in which the operation is put for indexing a set of changes;
CSSearchableIndex, and acts as its delegate.

 @interface RamblerIndexerMonitor : NSObject - (void)startMonitor; - (void)stopMonitor; - (void)addIndexer:(id<RamblerIndexer>)indexer withChangeProvider:(id<RamblerChangeProvider>)changeProvider; @end

Work algorithm

The whole process can be divided into two logical periods:

Saving changes - at this stage events about changes are collected and saved to the database. The stages of this period are marked on the chart with red markers.
Indexing Changes — retrieving a set of changes and indexing them. Stages of the period are marked with purple markers.

Stages of saving changes:

Using the delegate method NSFetchedResultsController informs the provider (<ChangeProvider>) about a change in the object in the source database.

 - (void)controller:(NSFetchedResultsController *)controller didChangeObject:(NSManagedObject *)anObject atIndexPath:(NSIndexPath *)indexPath forChangeType:(NSFetchedResultsChangeType)type newIndexPath:(NSIndexPath *)newIndexPath;

The provider calls the delegate method on the monitor, in which it transfers itself, the object and the type of change.

 - (void)changeProvider:(id<RamblerChangeProvider>)changeProvider didChangeObject:(id)object changeType:(RamblerChangeType)changeType;

The monitor identifies the indexer associated with it by the provider and asks for its identifier for this object. The indexer uses the IndexIdentifierFormatter for this. The monitor then forms the transaction from the identifier and the type of change, and then sends it to the StateStorage.
StateStorage queries or creates an IndexState (NSManagedObject) for the current object type and populates its OrderSets (insertIdentifiers, updateIdentifiers, deleteIdentifiers). Sets the modification date needed to determine the relevance of changes. And writes changes to the database.

Stages of indexing changes:

On a specific event, for example, calling the delegate method of the indexer, starting monitoring or exiting the background, the monitor understands that you need to start indexing if it is not currently being performed.
The monitor queries the StateStorage for a set of changes in the form of an IndexTransactionBatch object, which contains changes for only one type of object (created from IndexState).
Monitor using the method
```
 - (BOOL)canIndexObjectWithType:(NSString *)objectType 
```
finds the first indexer that can process a change set with a particular type of object, and passes that set to it to create an indexing operation.
The indexer creates an operation and returns it to the monitor.
The monitor adds an operation to the processing queue. By calling the end of the operation block, if the indexing was successful, the batch will be passed to the StateStorage to remove all processed identifiers from the database. After that, you can handle the following changes.

Operation:

combines insertIdentifiers and updateIdentifiers, leaving only unique identifiers, and then deletes identifiers from deleteIdentifiers from the resulting OrderSet;
queries the indexer for indexed objects;
for the resulting objects, it requests the CSSearchableItem array and submits them to the CSSearchableIndex indexing;
when done, passes the deleteIdentifiers to CSSearchableIndex to remove from the search results;
after that the completion block is called, which is passed to the method.

Primary indexing

When our system first starts, the monitor asks the StateStorage for initial indexing. StateStorage, in turn, looks to see if at least one IndexState exists. If there is no record, then primary indexing has not yet occurred. The monitor requests from all providers a list of objects for primary indexing, creating an array of transaction arrays from them (one array for each type of object). All this is passed on to save in StateStorage.

StateStorage saves all these changes for one performBlockAndWait , which guarantees us atomicity. As a result, either all changes will be saved, or none (if the application is turned off), in which case the primary indexing will be performed anew.

Total

We received a system that can be easily expanded and built in for indexing an object of any type, and also satisfies all the other requirements set by us at the beginning of the article.

For the integration, we will need to define several of our own indexers that will inherit the basic functionality of the RamblerIndexerBase class, as well as several providers if the objects are not stored in the database. Otherwise, you can use the RamblerFetchedResultsControllerChangeProvider class. After that, the entire system is assembled together, either manually or using a DI framework, for example, Typhoon .

Some features of the work:

The monitor must process the delegate methods of CSSearchableIndex to re-index objects. You also need to know that in iOS there is a limit on the number of indexed objects: these are 10K objects and approximately 130 MB of the total size of indexing.
Indexing should be done simultaneously in only one stream . That is why we use a queue of operations .
Do not use spaces for identifiers.
To increase the relevance of search results, you should implement the NSUserActivity and use the same identifier and the same attributes as in the CoreSpotlight implementation. To do this, you can use the already prepared indexer.
It is recommended to use from five to ten keywords .

The system presented in this article will soon be published on GitHub , we will inform Rambler.iOS on Twitter, and also add a link to this article.

The processing of the discovery of search results will be covered in the next section. In addition, we will touch on such topics as spacing logic for various application launch options (by push-notification, from search results, normal launch), processing input parameters when opening (userInfo from notification, NSUserActivity), ways to go to the desired screen (for example, manually picking up a navigation stack with several storyboards).

Source: https://habr.com/ru/post/268257/

All Articles