The remote proxy class is not (very) painful

(Dynamic dispatch to the rescue)

After several articles about MapReduce, we found it necessary to once again step aside and talk about the infrastructure that will help facilitate the construction of the MapReduce solution. We are still talking about InterSystems Caché , and, still, we are trying to build a MapReduce system based on the available materials in the system.

At a certain stage of writing the system, such as MapReduce , there is the task of conveniently calling remote methods and procedures (for example, sending control messages from the controller to the side of the controlled nodes). In the Caché environment, there are several simple, but not very convenient methods to achieve this goal, whereas I would like to get just a convenient one .

I want to take a simple and consistent code, here and there calling the methods of an object or class, and with a magic wave of the hand to make it work with methods already removed. Of course, the degree of "remoteness" may be different, for example, we can simply call methods in another process of the same node, the essence will not change much - we need to get a convenient way to marshal the calls "to the other side" outside the current process, working in the current area .

After several false starts, the author suddenly realized that in Caché ObjectScript there is a very simple mechanism that allows you to hide all the low-level details under a convenient, high-level shell - this is the mechanism of dynamic dispatching of methods and properties .

If you look back (far) back, you can see that since Caché 5.2 (and this is just a minute since 2007) there are several predefined methods in the base class %RegisteredObject inherited by every object in the system that are called when trying to call an unknown during compilation method or property (at the moment these methods have moved to the %Library.SystemBase interface, but this has not changed much).

For the simplicity of the experiment, we will use only functions responsible for invoking unknown methods and scalar properties. In the product environment, at a certain stage you may need to override all or most of the methods described, incl. Be carefull.

At first easier - logging proxy object

Recall that from the time of the “King of Peas” in the standard library CACHELIB there were standard methods and classes for working with the projection of JavaScript objects in XEN - %ZEN.proxyObject , it allowed to manipulate dynamic properties even in the times when there was no work on the document base DocumentDB (do not ask) and moreover there was no native support for JSON objects in the Caché core environment.

Let's try to create, for priming, a simple, logging all calls, proxy object ? Where we wrap all calls through dynamic dispatching with preservation of the protocol about each event that occurred. [Very similar to mocking in other language environments.]
[[How does this translate into Russian? "weeping"?]]

As an example, take the strongly simplified class Sample.SimplePerson (by a strange coincidence, it is very similar to Sample.Person from the SAMPLES area in the standard package: wink:)

 DEVLATEST:15:23:32:MAPREDUCE>set p = ##class(Sample.SimplePerson).%OpenId(2) DEVLATEST:15:23:34:MAPREDUCE>zw p p=<OBJECT REFERENCE>[1@Sample.SimplePerson] +----------------- general information --------------- | oref value: 1 | class name: Sample.SimplePerson | %%OID: $lb("2","Sample.SimplePerson") | reference count: 2 +----------------- attribute values ------------------ | %Concurrency = 1 <Set> | Age = 9 | Contacts = 23 | Name = "Waal,Nataliya Q." +----------------------------------------------------- 

Those. we have a persistent class - with 3 simple properties: Age, Contacts and Name. Wrap access to all properties of this class and call all its methods in its class Sample.LoggingProxy , and we will log every such call or access to a property ... somewhere.

 ///    : Class Sample.LoggingProxy Extends %RegisteredObject { ///      Parameter LoggingGlobal As %String = "^Sample.LoggingProxy"; ///      Property OpenedObject As %RegisteredObject; ///         ClassMethod Log(Value As %String) { #dim gloRef = ..#LoggingGlobal set @gloRef@($sequence(@gloRef)) = Value } ///        ClassMethod LogArgs(prefix As %String, args...) { #dim S as %String = $get(prefix) _ ": " _ $get(args(1)) #dim i as %Integer for i=2:1:$get(args) { set S = S_","_args(i) } do ..Log(S) } ///       %ID ClassMethod %CreateInstance(className As %String, %ID As %String) As Sample.LoggingProxy { #dim wrapper = ..%New() set wrapper.OpenedObject = $classmethod(className, "%OpenId", %ID) return wrapper } ///          Method %DispatchMethod(methodName As %String, args...) { do ..LogArgs(methodName, args...) return $method(..OpenedObject, methodName, args...) } ///          Method %DispatchGetProperty(Property As %String) { #dim Value as %String = $property(..OpenedObject, Property) do ..LogArgs(Property, Value) return Value } ///          /// log arguments and then dispatch dynamically property access to the proxy object Method %DispatchSetProperty(Property, Value As %String) { do ..LogArgs(Property, Value) set $property(..OpenedObject, Property) = Value } } 

1. The #LoggingGlobal class #LoggingGlobal sets the name of the global where we will store the log (in this case, the global named ^Sample.LogginGlobal );

   
   

2. There are two simple methods Log(Arg) and LogArgs(prefix, args...) that write the protocol to the global specified by the property above;

   
   

3. %DispatchMethod , %DispatchGetProperty and %DispatchSetProperty process the corresponding scripts with calls to an unknown method or property call. They log each access case via LogArgs , and then directly call the method or property of the object from the link ..%OpenedObject ;

   
   
4. Also, there is set the method of "class factory" %CreateInstance , which opens an instance of the specified class by its ID %ID . The created object is "wrapped" in the Sample.LogginProxy object, the link to which is returned from this class method.

No shamanism, nothing special, but in these 70 lines of Caché ObjectScript we tried to show a pattern of calling a method / property with a side effect (a more useful example of such a pattern will be shown below).

Let's see how our “logging proxy object” behaves:

 DEVLATEST:15:25:11:MAPREDUCE>set w = ## class(Sample.LoggingProxy).%CreateInstance("Sample.SimplePerson", 2) DEVLATEST:15:25:32:MAPREDUCE>zw w w=<OBJECT REFERENCE>[1@Sample.LoggingProxy] +----------------- general information --------------- | oref value: 1 | class name: Sample.LoggingProxy | reference count: 2 +----------------- attribute values ------------------ | (none) +----------------- swizzled references --------------- | i%OpenedObject = "" | r%OpenedObject = [2@Sample.SimplePerson](mailto:2@MR.Sample.AgeAverage.Person) +----------------------------------------------------- DEVLATEST:15:25:34:MAPREDUCE>w w.Age 9 DEVLATEST:15:25:41:MAPREDUCE>w w.Contacts 23 DEVLATEST:15:25:49:MAPREDUCE>w w.Name Waal,Nataliya Q. DEVLATEST:15:26:16:MAPREDUCE>zw ^Sample.LoggingProxy ^Sample.LoggingProxy=4 ^Sample.LoggingProxy(1)="Age: 9" ^Sample.LoggingProxy(2)="Contacts: 23" ^Sample.LoggingProxy(3)="Name: Waal,Nataliya Q." 

We obtained the state of the instance of the Sample.SimplePerson class accessible through a proxy, and the logging results stored in the global when accessing the properties of the proxy object. All as expected.

Remote object proxy

The attentive reader still has to remember why we are here - we need all these exercises to implement a simple proxy object that displays an object on a remote cluster node. In fact, the class with the relevant functionality in Caché is %Net.RemoteConnection . What could be wrong with him?

Much (and the fact that the class is officially marked as "deprecated" is not on the list of our complaints, we have questions of a different kind).

As many know, the %Net.RemoteConnection class uses c-binding to call remote Caché methods, which, in turn, are a wrapper over cpp-binding . If you know the address of the system, the area with which you want to work, and know the username and password, then you have everything to remotely call a method in this area of ​​this node. The problem with this API from %Net.RemoteConnection is that it is very cumbersome and verbose:

 Class MR.Sample.TestRemoteConnection Extends %RegisteredObject { ClassMethod TestMethod(Arg As %String) As %String { quit $zu(5)_"^"_##class(%SYS.System).GetInstanceName()_"^"_ $i(^MR.Sample.TestRemoteConnectionD) } ClassMethod TestLocal() { #dim connection As %Net.RemoteConnection = ##class(%Net.RemoteConnection).%New() #dim status As %Status = connection.Connect("127.0.0.1",$zu(5),^%SYS("SSPort"),"_SYSTEM","SYS") set status = connection.ResetArguments() set status = connection.AddArgument("Hello", 0 /*by ref*/, $$$cbindStringId) #dim rVal As %String = "" set status = connection.InvokeClassMethod(..%ClassName(1), "TestMethod", .rVal, 1 /*has return*/, $$$cbindStringId) zw rVal do connection.Disconnect() } ... } 

After creating the connection, and before calling the class method, you have to take care of passing the argument list starting with the ResetArguments call, and then passing each following argument through the AddArgument call, not forgetting a bunch of unclear, low-level parameters describing the argument (for example, its type in cpp-binding nomenclature, argument type, input or output, and more).

Also, personally, I was very frustrated that it was impossible to simply return the value after calling the remote method (since the return value of the InvokeClassMethod is just a status code, and to simply return the scalar value from the function, you yourself had to take care of the appropriate type argument when passing a long list of parameters).

I'm too old for such wordy and long preliminary games!

Ideally, I wanted to get a short and simple method of passing parameters to a remote function running on another machine or in another area.

Remember in Caché ObjectScript there is a method for passing a variable number of parameters through an array of args... in the function arguments? Why not search for such a mechanism to hide all these dirty details of the low-level interface, leaving us just the name and the list of arguments? And so that the engine did everything by itself (having guessed the type of the transmitted data, for example)?

 ///      %Net.RemoteConnection Class Sample.RemoteProxy Extends %RegisteredObject { Property RemoteConnection As %Net.RemoteConnection [Internal ]; Property LastStatus As %Status [InitialExpression = {$$$OK}]; Method %OnNew() As %Status { set ..RemoteConnection = ##class(%Net.RemoteConnection).%New() return $$$OK } ///     Method %CreateInstance(className As %String) As Sample.RemoteProxy.Object { #dim instanceProxy As Sample.RemoteProxy.Object = ##class(Sample.RemoteProxy.Object).%New($this) return instanceProxy.%CreateInstance(className) } ///       %ID Method %OpenObjectId(className As %String, Id As %String) As Sample.RemoteProxy.Object { #dim instanceProxy As Sample.RemoteProxy.Object = ##class(Sample.RemoteProxy.Object).%New($this) return instanceProxy.%OpenObjectId(className, Id) } ///       /// { "IP": IP, "Namespace" : Namespace, ... } Method %Connect(Config As %Object) As Sample.RemoteProxy { #dim sIP As %String = Config.IP #dim sNamespace As %String = Config.Namespace #dim sPort As %String = Config.Port #dim sUsername As %String = Config.Username #dim sPassword As %String = Config.Password #dim sClientIP As %String = Config.ClientIP #dim sClientPort As %String = Config.ClientPort if sIP = "" { set sIP = "127.0.0.1" } if sPort = "" { set sPort = ^%SYS("SSPort") } set ..LastStatus = ..RemoteConnection.Connect(sIP, sNamespace, sPort, sUsername, sPassword, sClientIP, sClientPort) return $this } ClassMethod ApparentlyClassName(CompoundName As %String, Output ClassName As %String, Output MethodName As %String) As %Boolean [Internal ] { #dim returnValue As %Boolean = 0 if $length(CompoundName, "::") > 1 { set ClassName = $piece(CompoundName, "::", 1) set MethodName = $piece(CompoundName, "::", 2, *) return 1 } elseif $length(CompoundName, "'") > 1 { set ClassName = $piece(CompoundName, "'", 1) set MethodName = $piece(CompoundName, "'", 2, *) return 1 } return 0 } ///    ()   Method %DispatchMethod(methodName As %String, args...) { #dim className as %String = "" if ..ApparentlyClassName(methodName, .className, .methodName) { return ..InvokeClassMethod(className, methodName, args...) } return 1 } ///         Method InvokeClassMethod(ClassName As %String, MethodName As %String, args...) { #dim returnValue = "" #dim i as %Integer do ..RemoteConnection.ResetArguments() for i=1:1:$get(args) { set ..LastStatus = ..RemoteConnection.AddArgument(args(i), 0) } set ..LastStatus = ..RemoteConnection.InvokeClassMethod(ClassName, MethodName, .returnValue, $quit) return returnValue } } 

When designing this interface, we introduced several fashionable idioms that were supposed to simplify the interaction interface, reduce the size of the code to be written, and, if possible, increase the stability of the interface.

The first such simplification that we tried to make to the interface is to use the configurator object to pass named arguments into the function instead of a long list. Caché ObjectScript does not have (yet) a built-in way to pass named arguments , and if, for example, you only need to pass the last two arguments from a long list of function parameters, then you need to carefully count the commas of parameters that you are not interested in and pass at the end what you want. Frankly, a very fragile design.

On the other hand, more recently, ObjectScript has built-in support for JSON objects that can be created on the fly, inside the expression {} . We can follow Perl’s example, try to reuse such dynamically created objects (in the case of Perl, it was a hash) to pass the named arguments to the function. The dynamic object configurator can contain only those value keys that interest us.

 DEVLATEST:16:27:18:MAPREDUCE>set w = ##class(Sample.RemoteProxy).%New().%Connect({"Namespace":"SAMPLES", "Username":"_SYSTEM", "Password":"SYS"}) DEVLATEST:16:27:39:MAPREDUCE>zw w w=<OBJECT REFERENCE>[1@Sample.RemoteProxy] +----------------- general information --------------- | oref value: 1 | class name: Sample.RemoteProxy | reference count: 2 +----------------- attribute values ------------------ | LastStatus = 1 +----------------- swizzled references --------------- | i%Config = "" | r%Config = "" | i%RemoteConnection = "" | r%RemoteConnection = 2@%Net.RemoteConnection +----------------------------------------------------- 

Yes, I agree, the construction is still not as transparent as it was in Perl, because there are still additional curly braces framing the object, but this is already the way in the right direction

The second modern idiom introduced in this example is call cascading. Where it was only possible, we returned from the methods a link to the current %this object, which made it possible to call several methods of the same class as a cascade.

We write less code - we sleep better.

The problem of calling class method

The wrapper object we created, encapsulating the functionality of Net.RemoteConnection , in its current state cannot do much that we want. If there is no place to store the context of the object being created ... (Not yet, We will solve this problem later, in another class) The only thing we can try to do now, at the current level of abstraction and with the current shell design, is to simplify the method of calling class methods called without reference to the object instance.

We can try to redefine %DispatchClassMethod , but in our case this will not help much - we want to write a generalized proxy class that would work for any remote class. In the case of a simple 1: 1 ratio, when a certain specialized shell on our side corresponds to a certain class on that side, this approach with redefining %DispatchClassMethod works quite well, but ... not for a generic class.

In general, we will need to come up with something else, but, preferably, still, simple, that would work with any connection and any target class.

We will give our rather elegant solution to this problem below, but for now let's step aside and see what can be used in Caché as an identifier for a method or property. Not everyone knows (I, at least, found out about this a couple of years ago) that the names of identifiers in ObjectScript can consist not only of Latin letters and numbers, but also of any "alphabet characters" specified by the current locale (for example, not only Latin letters A -Za-z and Arabic numerals 0-9, but also Cyrillic letters A-Ya-i, with the Russian locale installed ). [ this problem was discussed in passing in this discussion on StackOverflow ] Moreover, if you continue to be perverted, you can insert any emoji characters as a separator in the identifier name if you create and activate a locale where emoji would be considered literal characters in the current language. In general, it still seems that any trick sensitive to the installed locale will not fly very far and is not suitable as a generalized solution, t.ch. let's stop.

... On the other hand, the idea of ​​using a certain separator character inside the name of a method (class) seems quite reasonable and promising. We could hide the delimiter processing inside the %DispatchMethod special implementation, where we would separate the class name from the method name, and accordingly transfer control to the desired class method, hiding all implementation details.

So, perhaps, we will.

Returning to the syntax of valid method names, an even less well-known fact is that you can write anything you want into a class method if you put such a name inside double quotes "". Combining the name “quoting” and a special delimiter for the class name, I could, for example, call the LogicalToDisplay class method from the Cinema.Duration class using the following, at first glance, unusual syntax:

 DEVLATEST:16:27:41:MAPREDUCE>set w = ##class(Sample.RemoteProxy).%New().%Connect({"Namespace":"SAMPLES", "Username":"_SYSTEM", "Password":"SYS"}) DEVLATEST:16:51:39:MAPREDUCE>write w."Cinema.Duration::LogicalToDisplay"(200) 3h20m 

It looks a bit strange, but extremely simple and compact, is not it?

Special name processing and delimiter recognition takes place in the ApparentlyClassName function, where we look for special characters as a separator between the class name and the class method name — such delimiters were "::" (double colon like in C ++) or "" "(single quotation mark as in Ada or original Perl (e).

Note that you should not try to display something on the screen with this remote class method - the entire output of the output will be lost (ignored), since The cpp-binding protocol does not intercept the output to the screen, and does not return it back to the caller.

The cpp-binding protocol returns scalar data, not side effects .

 DEVLATEST:16:51:47:MAPREDUCE>do w."Sample.Person::PrintPersons"(1) 

Remote proxy objects

So far, we have not done a lot of useful things in the Sample.RemoteProxy code above: we just created the connection and forwarded calls to the class methods.

If you need to create remote instances of classes, or, as expected, open objects by their% ID, then you can use the services of our other shell class %Sample.RemoteProxy.Object .

 Class Sample.RemoteProxy.Object Extends %RegisteredObject { ///         Property OpenedObject As %Binary; Property Owner As Sample.RemoteProxy [ Internal ]; Property LastStatus As %Status [ InitialExpression = {$$$OK}, Internal ]; Method RemoteConnection() As %Net.RemoteConnection [ CodeMode = expression ] { ..Owner.RemoteConnection } Method %OnNew(owner As Sample.RemoteProxy) As %Status { set ..Owner = owner return $$$OK } ///      Method %CreateInstance(className As %String) As Sample.RemoteProxy.Object { #dim pObject As %RegisteredObject = "" set ..LastStatus = ..RemoteConnection().CreateInstance(className, .pObject) set ..OpenedObject = "" if $$$ISOK(..LastStatus) { set ..OpenedObject = pObject } return $this } ///       %ID Method %OpenObjectId(className As %String, Id As %String) As Sample.RemoteProxy.Object { #dim pObject As %RegisteredObject = "" set ..LastStatus = ..RemoteConnection().OpenObjectId(className, Id, .pObject) set ..OpenedObject = "" if $$$ISOK(..LastStatus) { set ..OpenedObject = pObject } return $this } ///        Method InvokeMethod(MethodName As %String, args...) [ Internal ] { #dim returnValue = "" #dim i as %Integer #dim remoteConnection = ..RemoteConnection() do remoteConnection.ResetArguments() for i=1:1:$get(args) { set ..LastStatus = remoteConnection.AddArgument(args(i), 0) } set ..LastStatus = remoteConnection.InvokeInstanceMethod(..OpenedObject, MethodName, .returnValue, $quit) return returnValue } ///    ()   Method %DispatchMethod(methodName As %String, args...) { //do ..LogArgs(methodName, args...) return ..InvokeMethod(methodName, args...) } ///       Method %DispatchGetProperty(Property As %String) { #dim value = "" set ..LastStatus = ..RemoteConnection().GetProperty(..OpenedObject, Property, .value) return value } ///        Method %DispatchSetProperty(Property, Value As %String) As %Status { set ..LastStatus = ..RemoteConnection().SetProperty(..OpenedObject, Property, Value) return ..LastStatus } } 

, Sample.RemoteProxy , cpp-binding ( %Net.RemoteConnection ). Sample.RemoteProxy.Object , ..Owner Sample.RemoteProxy . ( %OnNew ).

() InvokeMethod , , %Net.RemoteConnection (, .. %Net.RemoteConnection ResetArguments , , AddArgument , %NetRemoteConnection::InvokeInstanceMethod " ")

 DEVLATEST:19:23:54:MAPREDUCE>set w = ##class(Sample.RemoteProxy).%New().%Connect({"Namespace":"SAMPLES", "Username":"_SYSTEM", "Password":"SYS"}) … DEVLATEST:19:23:56:MAPREDUCE>set p = w.%OpenObjectId("Sample.Person",1) DEVLATEST:19:24:05:MAPREDUCE>write p.Name Quince,Maria B. DEVLATEST:19:24:11:MAPREDUCE>write p.SSN 369-27-1697 DEVLATEST:19:24:17:MAPREDUCE>write p.Addition(1,2) 3 

"SAMPLES", Sample.Person 1, (Name, SSN) (Addition).

, , , , , .

Instead of conclusion

, , ( , , , , , . , , .

MapReduce ...

, , gist .