DSL on Scala for working with HTML forms

Surely many of you are familiar with the process of creating and processing HTML forms. It may be trivial for a typical web application, but if you work in the corporate sector, the situation is a little different. Forms for creating or editing clients, documents and more become a daily routine. Developing Java frameworks, they offer more and more convenient API and components for working with them. But even in spite of this, many probably wondered if it was possible to make work with forms a little more convenient.
First of all, of course, I would like the framework to make the following tasks as easy as possible:

• defining properties of form fields, such as type, title, or validity;
• processing the form data after it is sent;
• rendering form.

Moreover, it is desirable that many errors would be detected at the compilation stage.

In this article, I will describe the process of creating your own DSL in the Scala language, and then I will show how to use the new way to describe forms in the context of the Play Framework 2.

A little about the terminology.

In this article, I am talking about internal DSL. Internal DSL is not a new language, but only a convenient way of describing a subject area using the syntax of the main (host) programming language. True, if the host-language syntax is sufficiently flexible, the internal DSL may look like it is a new language intended for a given area. The advantages of this option include the fact that development environments understand the internal DSL, highlight the syntax, and offer options for auto-completion. For comparison, external DSL is really a new language that needs its own parser.
')
The DSL described in the article was originally created to solve problems that arose due to some limitations of the form engine in the Play Framework 2. But now it is independent of Play and can be used with any JVM framework if you implement the appropriate adapters.

How to start developing DSL?

First of all, decide what you want to get as a result. We need to dream up on the topic: “what an ideal DSL should look like in order to solve this problem,” forgetting for a while that we are limited by the host-language syntax.
As an example, take the registration form with the following fields:

• mail - text, mandatory, validation of the postal address;
• Name - text, mandatory;
• date of birth is an optional date.

To describe it, you can write this entry in pseudocode:

form( string(email, required, validate(EmailAddress)) string(name, required) date(birthDate) ) 

Now it's time to think about how to implement it. First consider the general description of the form, namely the set of fields and their type. Scala provides us with several approaches to create such a description:

1. Builder pattern and method chaining

Signature factory method for the form:

 def form(builderFoo: FormBuilder => FormBuilder) 

Using:

 form(_ .string(...) .string(...) .extend(commonFields) .date(...) ) 

Here we use the builder and method chaining patterns. This gives us the following benefits:

• You can build a form description using the abstract builder interface - this allows you to change a specific implementation if necessary;
• A part of the call chain can be put into a separate function and used in different forms;
• need to write less auxiliary code;
• the closure type is derived from the form (...) signature.

Looking ahead, I will say that this option was chosen .

2. As a function with a variable number of parameters

Factory Method:

 def form(fields: FormBuilder => FormBuilder*) 

Using:

 form( _.string(...), _.string(...), _.date(...), commonFields:_* ) 

There are several drawbacks to this option:

• the need to write more auxiliary code;
• additional fields can be added only at the end of the list of arguments.

3. Block code with a sequence of calls

Factory Method:

 def form(fields: => ()) 

 : form{ string(...) date(...) commonFields() } 

This method is bad because:

• at the time of form determination, we are tied to the specific implementation of the string, date, etc. methods.
• to implement it, we would have to work with a changeable state, which we would like to avoid;
• There is no typification of transmitted expressions.

So, a preliminary analysis showed that the best option is using a known design pattern. Surprise.

Field description

For a detailed description of the fields also selected method chaining and builder, for the same reasons.
The signature of the field definition method:

 def string(fieldFoo: FieldBuilder => FieldBuilder): FormBuilder 

Using:

 .string(_.name("email").required.validate(EmailAddress)) 

A sketch of the implementation of the builder

The case class with its automatically defined copy method is ideal for implementing Scala in Scala. This method copies the current object, modifying the desired attributes. A builder for a particular field might look like this:

 case class FieldBuilderImpl(fieldType: String, label: String, isRequired: Boolean, validators: Seq[Validator]) extends FieldBuilder { def fieldType(t: String): FieldBuilder = copy(fieldType = t) def label(l: String): FieldBuilder = copy(label = l) def required: FieldBuilder = copy(isRequired = true) def validate(vs: Validator*): FieldBuilder = copy(validators = validators ++ vs) def build: FieldDescription = ??? } 

The main form parameter is a set of its fields, so for it the builder will look slightly different:

 case class FormBuilderImpl(fields: Map[String, FieldDescription]) extends FormBuilder { def field[F](name: String)(foo: FieldBuilder[F] => FieldBuilder[F]) = copy(fields = fields ++ Map(name -> foo(newFieldBuilder[F]).asInstanceOf[FieldBuilderImpl[F]].build)) def string(name: String)(foo: FieldBuilder[String] => FieldBuilder[String]) = field[String](name)(foo andThen (_.fieldType("string"))) def newFieldBuilder[F]: FieldBuilder[F] = ??? def build: FormDescription = ??? } 

It is worth noting that in this case we still need to specify the field ID manually. At the same time, if we accidentally specify an incorrect identifier (for example, having sealed it), we will learn about the error only during the execution of the program.

Form Data Presentation

After the user submits the form, the web framework used receives the request data and converts it to its own framework-specific representation. One of the tasks of integration with the framework is to transform its internal form representation to a convenient form (this process is briefly described at the end of the article with reference to the Play Framework)

It is convenient to present the form data as an instance of the case class . For brevity, we will call it “form data object” or simply “form data”. Form fields will be represented by typed fields of this class. In this case, we can identify the form fields with a closure — by calling the getter of the desired field of the form data object (for example, a closure (_.someField) will correspond to a form field called "someField" ).

Sending a closure instead of a string has an important advantage - it allows us to control the type of the form field, because we know the type of the closure. For example, by the type UserFormData => String we can determine that the form field should be only string.

Getting the field name using reflection

There are at least two ways to get the field name in Scala: the first, the traditional one - with the help of reflection, the second - with the help of macros. Reflection is good because it is a long-established, rather unpretentious mechanism, while macros are a new feature that is in the status of an experiment, and there are some limitations to its use. But they allow you to determine the name of the field at compile time, which, of course, is a significant advantage. In this article, we will limit ourselves to reflection (we'll get to the macro next time).
So, we are interested in the ability to define a field in the following way:

 form[FormData](_.string( _.someField )(_.someProperty)) 

those. on the _.someField closure get the name "someField" .
A function to do this would have the following signature:
fieldName[T:Manifest](fieldFoo: T => Any): String
The [T:Manifest] entry means that an additional, implicit parameter list will be added to the method signature, expecting a Manifest [T] argument from the compiler. In fact, this is nothing more than a crutch to overcome such a phenomenon in the JVM as Type Erasure . With the manifest, we get the opportunity to get type parameter (similar to the type variable from the Generic class in Java) at run time.
Having received a class from the manifest, we construct from it a dynamic proxy proxyObject : an object that satisfies the type that we transferred, the only task of which is to communicate the name of the first method called. If the closure (_: T).someField is passed as fieldFoo: T => Any , then a call to fieldFoo(proxyObject) will give us the string "someField" . These actions were moved to the scala-reflective-tools library.
Its use looks like this:

 case class MyClass(fieldA: String) import FieldNameGetter._ assertTrue $[MyClass](_.fieldA) == "fieldA" 

Now we can rewrite the form builder like this:

 case class FormBuilderImpl[T:Manifest](fields: Map[String, FieldDescription]) extends FormBuilder[T] with FieldNameGetter { def field[F](fieldFoo: T => F)(foo: FieldBuilder[F] => FieldBuilder[F]) = copy(fields = fields ++ Map($[T](fieldFoo) -> foo(newFieldBuilder[F]).asInstanceOf[FieldBuilderImpl[F]].build)) def string(fieldFoo: T => String)(foo: FieldBuilder[String] => FieldBuilder[String]) = field[String](fieldFoo)(foo andThen (_.fieldType("string"))) ... } 

Form fields can be required and optional. We can reflect the optional property of a field in the case class, by giving the corresponding field the type Option [...].
For example, for our form, the case class might look like this:

 case class RegistrationFormData( name: String, surname: Option[String], email: String, birthDate: Option[Date] ) 

The getter of the name field is of type RegistrationFormData => String , and the getter surname is of type RegistrationFormData => Option[String] . Accordingly, we must have two methods for defining fields.
For mandatory:
(def string(fieldFoo: T => String)(foo: FieldBuilder[String] => FieldBuilder[String])
For optional:
def stringOpt(fieldFoo: T => Option[String])(foo: FieldBuilder[String] => FieldBuilder[String])
For each field type (except boolean) both options are required. We customize the string method by declaring the required property in it. With the help of currying it was possible to get a rather compact code:

 case class FormBuilderImpl[T: Manifest](fields: Map[String, FieldDescription]) extends FormBuilder[T] with FieldNameGetter { //    : type FieldFoo[F] = FieldBuilder[F] => FieldBuilder[F] type FormField[F] = FieldFoo[F] => FormBuilder[T] //    def string(fieldFoo: T => String): FormField[String] = fieldBase[String](fieldFoo)(_.required) //    def stringOpt(fieldFoo: T => Option[String]): FormField[String] = fieldBase[String](fieldFoo)(identity) def field[F](fieldName: String)(foo: FieldFoo[F]) = copy(fields = fields ++ Map(fieldName -> foo(newFieldBuilder[F]).asInstanceOf[FieldBuilderImpl[F]].build)) def fieldBase[F: Manifest](fieldFoo: T => Any) (innerConfigFoo: FieldFoo[F]) (userConfigFoo: FieldFoo[F]) = field($[T](fieldFoo))(innerConfigFoo andThen (_.fieldType(fieldTypeBy[F])) andThen userConfigFoo) ... } 

In addition, here we define the fieldType property using the type parameter.

DSL extension

At the moment, with our FieldBuilder we can only specify the label and required properties, and this is not enough. Developers should be able to extend the set of field properties if required by the task. In addition, fields of different types need to specify different properties.
To solve this problem, we will use implicit transforms and the Pimp My Library pattern.

Add the following method to FieldBuilder :
def addProperty(key: String, value: Any): FieldBuilder[T]
Further we will set all field configuration through it. Users of our DSL will not access it directly, the user API will be composed of implicit converters of the form:

 object FieldDslExtenders { import FieldAttributes._ implicit class StringFieldBuilderExtender(val fb: FieldBuilder[String]) extends AnyVal { def minLength(length: Int) = fb.addProperty(MinLength, length) def maxLength(length: Int) = fb.addProperty(MaxLength, length) } implicit class SeqFieldBuilderExtender[A](val fb: FieldBuilder[Seq[A]]) extends AnyVal { ... } implicit class DateFieldBuilderExtender(val fb: FieldBuilder[Date]) extends AnyVal { def format(datePattern: String) = fb.addProperty(DatePattern, datePattern) } } 

Note that implicit classes are inherited from AnyVal . This is necessary so that at run time an object of this wrapper class is not instantiated, but instead a static method is called on the companion object, which will be implicitly created by the compiler.

Not everything is perfect in our FormBuilder : with its help, so far you can add a rather limited set of field types. It is clear that adding all the most common views to it is not difficult, but what if users need a new field type that is not in FormBuilder ? This problem can be solved with the help of the same implicit converters that access the base fieldBase method. With their help, we can extend FormBuilder - for example, we could add a method to create a field with a date:

 object FormDslExtensions extends FieldNameGetter { val defaultDateFormat = new SimpleDateFormat("dd.MM.yyyy").toPattern import FieldDslExtenders._ implicit class DateFormExtension[T: Manifest](val fb: FormBuilder[T]) extends AnyVal { def dateOpt(fieldFoo: T => Option[Date]) = fb.fieldBase[Date](fieldFoo)(_.format(defaultDateFormat)) _ def date(fieldFoo: T => Date) = fb.fieldBase[Date](fieldFoo)(_.required.format(defaultDateFormat)) _ } } 

Internal form presentation

As a result of the work of our builders, we get a FormDescription description object containing the String -> FieldDescription . In turn, the FieldDescription contains the map String -> Any . Thus, we can set the fields for any attributes that may be required. The resulting description of the form, in my opinion, is quite framework-agnostic, i.e. can be used with various frameworks. All that is needed is to implement the conversion into the form representation of the framework used. Next we look at how this was done for the Play Framework.

Play integration

Play already has a mechanism that is convenient for not very complex forms. But it has the following limitations:

• a limit of 18 fields;
• the ability to specify only the type of fields and validation;
• the absence of fields with multiple choice (at the time of writing, should soon appear).

When submitting a form, Play receives an HTTP request, deserializes it and presents the request parameters as Map[String, Seq[String]] . Further, they are expected to be validated and converted into a suitable representation for custom code. Built-in Play allows you to convert this data either into a Tuple or into an arbitrary object - for this, however, you must provide the function of its creation. If the form has enough fields, it can lead to code that is potentially rich in errors. Imagine: you need to make sure that the 18 function arguments are in the correct order.

To validate and convert raw data, Play uses the Mapping class.
Form mappings built into Play receive field mappings as separate constructor arguments (for example, ObjectMapping9 ), so a form can only have a strictly fixed set of fields specified at the time of its definition. Our class, which we call FormMapping , can work with an arbitrary number of fields. On the other hand, the fields passed to it lose their typing, but this is not terrible, since FormMapping not intended to work with fields manually. Field typing is guaranteed by DSL, and the conversion to a form data object and vice versa occurs automatically using reflection.

Forms on Play are represented using case class play.api.data.Form , and fields are play.api.data.Field . Our implementations of forms and fields will be inherited from these classes, since we need to achieve compatibility with the old API. The fields of the form will have a new field of attributes - with its help additional parameters will be passed.

Usage example

Form Definition:

  PlayFormFactory.form[FormData](_.string(_.name)(_.label("").someAttribute(42)) 

Template with the form:

 @(form: com.naumen.scala.forms.play.ExtendedForm[RegistrationFormData]) <div> .... @myComponent(form(_.name)) .... </div> 

Template component myComponent:

 @(customField: com.naumen.scala.forms.play.ExtendedField) <div> .... @customField.ext.attrs("someAttribute") .... </div> 

The ability to set arbitrary field parameters at the form definition stage allows you to concentrate in the template on the layout and put the definition of parameters not related to the layout to the controller.