Development of a mechanism for extracting DTO from a database using LINQ

Formulation of the problem

In this article I will describe the mechanism for creating a DTO, implemented in one of our company's projects. The project consists of a server part and several clients (Silverlight, Outlook, iPad). The server is a series of services implemented on WCF. If there are services, then you need to exchange some data with them. The option when clients know about the domain domain entities and get them from the server dropped right away for several reasons:

1. Not all clients are implemented on .NET
2. Possible problems with the serialization of complex object graphs
3. Data redundancy

In principle, all these shortcomings have long been known and to eliminate them, smart people have invented the Data Transfer Object (DTO) pattern. That is, the domain domain entity classes are known only to the server, while clients operate with DTO classes and instances of the same classes are exchanged with services. In theory, everything is fine, but in practice, among others, there are issues of creating DTO and writing data from entities into them. In small projects the operator "=" will perfectly cope with this work. But, when the size of the project begins to grow and the requirements for code performance and maintenance increase, the need arises for a more flexible solution. Below I will describe the evolution of the mechanism that we use to create and populate the DTO.

Example domain model

For a more visual illustration, we define a test domain model. Suppose our application keeps records of meteorites. We have the following main types:

/// <summary> ///    . /// </summary> public class Entity { /// <summary> /// . /// </summary> public Guid Id { get; set; } } /// <summary> ///   . /// </summary> public abstract class Meteor: Entity { /// <summary> /// . /// </summary> public string Name { get; set; } /// <summary> /// . /// </summary> public double Weight { get; set; } /// <summary> ///     . /// </summary> public Material Material { get; set; } /// <summary> ///   . /// </summary> public double DistanceToEarth { get; set; } /// <summary> ///   . /// </summary> public RiskLevel RiskLevel { get; set; } } /// <summary> ///  . /// </summary> public class SpaceMeteor: Meteor { /// <summary> /// / . /// </summary> public DateTime DetectedAt { get; set; } /// <summary> ///  . /// </summary> public Person DetectingPerson { get; set; } /// <summary> /// ,   . /// </summary> public Galaxy PlaceOfOrigin { get; set; } } /// <summary> ///   . /// </summary> public class ArtificialMeteor: Meteor { /// <summary> /// -. /// </summary> public Country Country { get; set; } /// <summary> /// -. /// </summary> public SecretFactory Maker { get; set; } /// <summary> ///  . /// </summary> public string SerialNumber { get; set; } /// <summary> ///  . /// </summary> public Person QualityEngineer { get; set; } } 

For the corresponding domain classes we create DTO classes. It should be noted that in our project we use completely flat DTO.
')

 /// <summary> ///    DTO. /// </summary> public class BaseDto { public Guid Id { get; set; } } /// <summary> ///   DTO . /// </summary> public abstract class MeteorDto: BaseDto { public string Name { get; set; } public double Weight { get; set; } public string MaterialName { get; set; } public Guid? MaterialId { get; set; } public double DistanceToEarth { get; set; } public string RiskLevelName { get; set; } public Guid RiskLevelId { get; set; } } /// <summary> /// DTO  . /// </summary> public class SpaceMeteorDto: MeteorDto { public DateTime DetectedAt { get; set; } public string DetectingPersonName { get; set; } public Guid DetectingPersonId { get; set; } public string PlaceOfOriginName { get; set; } public Guid? PlaceOfOriginId { get; set; } } /// <summary> /// DTO   . /// </summary> public class ArtificialMeteorDto: MeteorDto { public string CountryName { get; set; } public Guid CountryId { get; set; } public string MakerName { get; set; } public string MakerAddress { get; set; } public string MakerDirectorName { get; set; } public Guid MakerId { get; set; } public string SerialNumber { get; set; } public string QualityEngineerName { get; set; } public Guid QualityEngineerId { get; set; } } 

Mechanism # 1 - an entity from the database, then a DTO from the entity

The first approach to the problem resulted in the creation of the interface

 interface IDtoMapper<TEntity, TDto> { IEnumerable<TDto> Map(IEnumerable<TEntity> entities); } 

For each entity-DTO pair, a class was created that implements an interface that is closed by the appropriate types. Mapping was implemented through Automapper , which allowed us to get rid of the routine assignment of properties and explicitly describe only those cases where the naming of properties did not fall under the agreement used by Automapper. How it worked:

1. Client calls WCF operation to get dataset
2. Entities according to certain criteria are loaded from the database and transferred to IDtoMapper
3. IDtoMapper creates DTO instances and copies the data from the entities into the DTO.
4. The DTO collection is returned to the client.

This mechanism worked and we were completely satisfied until the performance problems appeared with increasing load. Measurements showed that one of the perpetrators was the old IDtoMapper. (We will not scold him much, as he, in the best traditions of Agile, helped us quickly launch a product, better understand it in the process, and then rewrite parts into account of the experience gained.) The problem was that more data was retrieved from it was necessary to fill the DTO. Associations and aggregations between objects led to a large number of joines, which negatively affected the speed of the system.

We considered 2 possible solutions to the problem: conjure with data extraction strategies (lazy or eager loading, etc.), or directly retrieve DTO from the database. The second path was chosen as the most simple, productive and flexible. Here it should be noted that we use NHibernate as the ORM, queries to the database are made using LINQ. All of the following will also work in the Entity Framework.

Mechanism №2 - DTO from DB

The following interface has been created:

 interface IDtoFetcher<TEntity, TDto> { IEnumerable<TDto> Fetch(IQueryable<TEntity> query, Paging paging, FetchAim fetchAim); } 

Now the method takes 3 parameters instead of one:

1. query - LINQ query by entity type
2. paging - information about the page being retrieved
3. fetchAim - extraction target

Since the DTO mapping mechanism completely corresponded, it was decided to carry out optimization in several directions at once. One of these was the notion of an extraction target. For different forms on the client, DTO must have different properties set. For example, to select from a drop-down list, you only need to have a name and an identifier. To display in the registry must be set to text properties. A different data set is required for the card. Therefore, the Fetch method takes a parameter by which the target of the DTO extraction is determined and, accordingly, only the required fields are loaded from the database.

 /// <summary> ///   DTO. /// </summary> public enum FetchAim { /// <summary> ///    /// </summary> None, /// <summary> ///  /// </summary> Card, /// <summary> ///  /// </summary> List, /// <summary> ///  /// </summary> Index } 

Abstraction identified, it is the turn of its implementation. For selective sampling of fields in SQL, projection is used (projection), for example:

 SELECT (id, name) FROM meteors 

In LINQ, projections are implemented using the Select () method. The SQL query above will be generated when executing the following LINQ query:

 IQueryable<Meteor> meteorQuery = _meteorRepository.Query(); IEnumerable<MeteorDto> meteors = meteorQuery .Select(m => new MeteorDto { Id = m.Id, Name = m.Name }) .ToList(); 

After learning about this LINQ ability, we began to diligently create specific IDtoFetcher implementations:

 class SpaceMeteorDtoFetcher: IDtoFetcher<SpaceMeteor, SpaceMeteorDto> { public IEnumerable<SpaceMeteorDto> Fetch(IQueryable<SpaceMeteor> query, Page page, FetchAim fetchAim) { if (fetchAim == FetchAim.Index) { return query .Select(m => new SpaceMeteorDto { Id = m.Id, Name = m.Name }) .Page(page) .ToList(); } else if (fetchAim == FetchAim.List) { // ... } // ... } } 

But after the second class, there was a sudden onset of laziness (and the realization that this approach would lead to large-scale code duplication and considerable difficulties in further maintaining the system and adding new entities). For example, when mapping the heirs of the base class in all of them will have to repeat the line with the initialization of common properties. Also duplication will occur when mapping one entity for different extraction purposes. And here, in my head, simple Russian words appeared involuntarily: expression trees ...

Since the LINQ query is an expression tree, which is then parsed and the SQL query is generated, it was decided to create a declarative mechanism to describe the mapping of the properties of the entities to the DTO properties and build the necessary LINQ query using this information.

Implementation

The source code of the project with the implementation (.NET 4.0, NHibernate 3.3.2, Visual Studio 2012) is here .

To understand why it was necessary to enter into an unequal struggle with expression trees, I’ll give an example of how the configuration of the fetcher for a particular class is now done.

 /// <summary> ///  DTO  . /// </summary> public class SpaceMeteorDtoFetcher: BaseMeteorDtoFetcher<SpaceMeteor, SpaceMeteorDto> { static SpaceMeteorDtoFetcher() { CreateMapForIndex(); CreateMapForList(); CreateMapForCard(); } private static void CreateMapForIndex() { var map = CreateFetchMap(FetchAim.Index); //       MapBaseForIndex(map); } private static void CreateMapForList() { var map = CreateFetchMap(FetchAim.List); //       MapBaseForList(map); MapSpecificForList(map); } /// <summary> ///       . /// </summary> /// <param name="map">   </param> private static void MapSpecificForList(IFetchMap<SpaceMeteor, SpaceMeteorDto> map) { map.Map(d => d.DetectedAt, e => e.DetectedAt) .Map(d => d.DetectingPersonName, e => e.DetectingPerson.FullName) .Map(d => d.PlaceOfOriginName, e => e.PlaceOfOrigin.Name); } private static void CreateMapForCard() { var map = CreateFetchMap(FetchAim.Card); MapBaseForCard(map); MapSpecificForCard(map); } /// <summary> ///       . /// </summary> /// <param name="map">   </param> private static void MapSpecificForCard(IFetchMap<SpaceMeteor, SpaceMeteorDto> map) { map.Map(d => d.DetectedAt, e => e.DetectedAt) .Map(d => d.DetectingPersonId, e => e.DetectingPerson.Id) .Map(d => d.PlaceOfOriginId, e => e.PlaceOfOrigin.Id); } public SpaceMeteorDtoFetcher(IRepository repository) : base(repository) { } } 

To configure the fetcher, use the mapping abstraction.

 public interface IFetchMap<TSource, TTarget> where TSource : Entity where TTarget : BaseDto { /// <summary> ///          DTO. /// </summary> IFetchMap<TSource, TTarget> Map<TProperty>( Expression<Func<TTarget, TProperty>> targetProperty, Expression<Func<TSource, TProperty>> sourceProperty); /// <summary> ///       DTO,      . /// </summary> IFetchMap<TSource, TTarget> CustomMap(Action<IQueryable<TSource>, IEnumerable<TTarget>> fetchOperation); } 

How the configuration happens: create a mapping object for a specific extraction target, call Map methods to specify which properties should be loaded from the database. The CustomMap method is used as an extension point — in the delegate passed there, we can prescribe the logic for manually loading data from the database and writing them to the extracted DTO.

BaseMeteorDtoFetcher Base Meteorite DTO Fetcher class provides methods for mapping base class properties - this way we avoid duplication and speed up the creation of fetchers for new types of meteorites. BaseMeteorDtoFetcher itself, in turn, is inherited from BaseDtoFetcher, which stores a collection of created objects of type IFetchMap and uses them to retrieve DTO.

A new abstraction has been added and, according to the established tradition, we need its implementation. (Actually, in life everything was the other way around - first a specific class appeared, and then an interface was extracted from it.) The implementation is represented by the FetchMap class. That is where the entire logic of working with expression trees is located. The article is quite large, so here I will not step by step analyze the implementation of FetchMap. You can see it in the attached project . For understanding, you need to have some idea of ​​expression trees.

Conclusion

Thus, at the moment we have a mechanism that allows the optimal extraction of DTO from the database and has a declarative syntax for customizing mappings, which allows to simplify their creation. It suits us in terms of speed and ease of expansion for new entities and DTO.

Hopefully, the experience described above will allow some readers to bypass those rakes that we successfully collected during the project. And if someone knows how all this can be implemented easier and more flexible, I will be glad to hear about it. Thank you for attention!