Full text search on App Engine now

All of course are looking forward to when the full-text search will appear in the App Engine , but so far it is not even in the roadmap . However, for GAE / Java, full-text search can be done independently now.



You'd be surprised, but this is Lucene . No, no, do not rush to leave in frustration, sighing, "Well, this is how much you have to mess around." Making Lucene friends with the App Engine datastore is probably not five minutes, but if you are using JDO, then the compass-project developers in compass 2.3 (now in beta) have already done this for you.
')
Compass can turn your JDO objects into indexed documents and return them as search results. The code of the simplest JDO object with indexing and search, and comments to interesting places in it, see below. In order for this to compile and work, you need to add several libraries to war/WEB-INF/lib : lucene-core, commons-logging compass-2.3.0-beta1 . They all live in the distribution kit of the Compass night build, which is buried deep in the jungle of their continuous build system. At the time of writing, the last successful build lives here: http://build.compass-project.org/download/CMPTRK-NIGHTLY/artifacts/build-786/Release . I admit that I did not check the performance of this particular build.

@PersistenceCapable(identityType = IdentityType.APPLICATION)
@Searchable // [1]
public class GreetingServiceUser {
@PrimaryKey
@Persistent(valueStrategy = IdGeneratorStrategy.IDENTITY)
@SearchableId
private Long id; // [2]

@Persistent
private String name;

public GreetingServiceUser( String name) {
this .name = name;
}

@SearchableProperty // [3]
public String getName() {
return name;
}

private static final PersistenceManagerFactory factory =
JDOHelper.getPersistenceManagerFactory( "transactions-optional" );
private static Compass compass;
private static CompassGps compassGps;

static {
// [4]
compass = new CompassConfiguration()
.setConnection( "gae://index" )
// [5]
.setSetting(CompassEnvironment.ExecutorManager.EXECUTOR_MANAGER_TYPE,
"disabled" )
.addScan( "compass_test" )
.buildCompass();
compassGps = new SingleCompassGps(compass);
Jdo2GpsDevice jdo2GpsDevice = new Jdo2GpsDevice( "appengine" , factory);
// [6]
jdo2GpsDevice.setMirrorDataChanges( false );
compassGps.addGpsDevice(jdo2GpsDevice);
// [7]
// if (compass.getSearchEngineIndexManager().isLocked()) {
// compass.getSearchEngineIndexManager().releaseLocks();
// }
compassGps.start();
}

public void save() {
PersistenceManager pm = factory.getPersistenceManager();
CompassIndexSession indexSession = null ;
try {
pm.makePersistent( this );

// [8]
// compass.getSearchEngineIndexManager().releaseLocks();

// [9]
indexSession = compass.openIndexSession();
indexSession.save( this );
indexSession.commit();
}
catch (Throwable e) {
e.printStackTrace();
}
finally {
pm.close();
if (indexSession != null ) {
indexSession.close();
}
}
}

public static List <GreetingServiceUser> search( String query) {
CompassSearchSession searchSession = compass.openSearchSession();

CompassHits hits = searchSession.find(query);
List <GreetingServiceUser> results =
new ArrayList <GreetingServiceUser>(hits.getLength());
PersistenceManager pm = factory.getPersistenceManager();
try {
for ( int i=0; i<hits.length(); i++) {
// [10]
results.add(pm.getObjectById(
GreetingServiceUser. class ,
Long.valueOf(hits.resource(i).getId())));
}
}
catch (JDOObjectNotFoundException e) {
e.printStackTrace();
}
finally {
pm.close();
}
return results;
}
}

* This source code was highlighted with Source Code Highlighter .

[1] Annotations mark JDO classes to be indexed ...
[2] ... properties identifying documents, ...
[3] ... and finally properties whose values ​​will be indexed.
[4] When constructing a Compass instance, you specify which packages to scan for such annotations.
[5] This is an App Engine specificity. Compass generally works in several threads, but you cannot create threads on App Engine, so you need to disable threads.
[6] This is also a small hack relevant for App Engine. Compass is generally able to update the index automatically when something changes in the persistent data interesting to it. But when working on App Engine, it may not fit into the allotted 30 seconds. Calling setMirrorDataChanges(false) will disable automatic updating and you will need to manually update the index. This is not painful, and as compensation it can be done, for example, in a separate task in the task queue.
[7, 8] App Engine support is still damp, and sometimes Compass seems to forget to remove the index locks. I have not yet understood the reasons and the correct way of correction, but an explicit request to release the locks helps. We must, of course, understand that this may not be very safe. If anyone takes the time to catch the problem and win, let me know.
[9] Actually, the object is indexed manually.
[10] The result of the search is the CompassHits collection. It is theoretically possible to extract the objects found ( GreetingServiceUser in our case) by the data(int) method, but in my case they were obtained with empty values. Therefore, I use the Resource object that corresponds to the document, extract the document id from it and look for the corresponding JDO object using JDO / AppEngine.