Online Database, data structure service, dynamic object-relational projection (Dynamic ORM)
This article will focus on creating a working prototype of an on-line database. And considered some of the services analogues from Google and Yahoo.
In recent times, services for working with online databases have become more active. Such services include such platforms as GData (Google Data API) and Yahoo! Query Language (YQL).
These online database access services allow you to better control and operate on data obtained from different platform services, and the most important thing is ready-made services for the use of which it suffices to study the documentation and examples.
Google’s online data access service allows
')
- request, enter and manage data programmatically to create applications and mashups;
- allows you to view and update the contents of spreadsheets in the form of Google data feeds (“GData”);
- the client can request a list of user tables, as well as present, edit or delete content in an existing table;
- apply autofiltering and other functions of the tables in the application.
For convenient work with the above services, Google has developed its own SQL-like query language Google Visualization API Query Language.
Query language to retrieve data from Yahoo! Data arrays .. An SQL-like query language for working with web services from Yahoo! .. Query results can be obtained in XML or JSON formats.
Here is what Jonathan Trevor, head of development at YQL Execute, says:
“Developers can now call multiple services at the same time and operate on multiple data sources, making their applications more powerful and diverse. Received from Yahoo! data can be real-time recoded from one format to another. ”
YQL makes it easy to handle data using SQL with a similar syntax. For example, you want to select photos in Paris, France that have a Creative Commons Attribution (4) license:
select * from flickr.photos.info where photo_id in (select id from flickr.photos.search where woe_id in (select woeid from geo.places where text = 'paris, france') and license = 4)
Data tables can be formed and stored using the “Open Table” service. The service is quite interesting, but the description is still very small.
The above services will allow access and online manipulation of data tables, which is vital for many applications. After analyzing the above services and a number of other similar services, we implemented a similar service for working with data tables. However, it has a specific distinguishing feature. Let's take a closer look at our development right from the inside.
API documentation www.hivext.ru/index.php/Structures
One of the extremely necessary for flexible and convenient development of applications based on the platform is service structures. A distinctive feature of our development is the object approach in the implementation of such a service. In order to provide all the advantages of an object-oriented approach to building applications that has been proven over the years, the service was designed with a maximum “tilt” in the direction of the OOP (Object Oriented Design). Due to this, the structure service allows developers to create their own applications in the OOP style, allows you to manipulate and control the types and objects of the developed applications.
The structure service is intended for defining types and creating instances of types (objects). The approach to creating your objects is quite simple and familiar. To create a collection, a set of any objects, it is necessary to determine the type of future objects. Defining a type means defining fields that future objects must contain. In turn, the definition of a type field means specifying the name of the field and the simple base type of this field. There are predefined simple types in the structure service, based on which developers can create their more complex types.
List of simple types that are predefined in the structure service
Based on the above types, developers can create their own more complex types. Let's look at an example. Suppose we need to create a book online store. One of the required objects of our application will be a product-book, which we sell. We define the type for our book objects. To define a type, use the DefineType method (appid, session, type, fields)
Regarding our books, we will call the method with the following parameters.
DefineType (appid, session, “book”, {price: ”float”, name: ”string”, description: ”string (500)”, author: ”string (100)”})
Thus, we have defined the type book , which has price fields — the price of the book, name — the name of the book, description — description of the book, author — author of the book.
Then we can safely create instances of this type, i.e. objects are books. To do this, use the method
CreateObject (appid, session, type, data)
Regarding our book type, call the method with the following parameters
CreateObject (appid, session, “book”, {price: 10.5, name: “Web Application Development”, description: ”about designing and developing complex one-page application sites”, author: “Know-it-all I.”})
Upon successful creation of the object, in the response we get the identifier of the new object.
Now we have the first object of type book . Further, in a similar way, we create the set of objects necessary for our online store.
CreateObject (appid, session, “book”, {price: 8, name: "Desktop Development", description: "about designing and developing simple desktop applications", author: "EV News"})
CreateObject (appid, session, “book”, {price: 12, name: "Mobile Application Development", description: "about designing and developing applications for mobile phones", author: "Know-it-all I.N.}}
And so on. It's pretty simple!
Further, it is necessary to display formatted lists of books available in our store to users. This is made even easier. To select a set of objects, two methods of structure service are used.
GetObjects (appid, session, type, from, count)
GetObjectsByCriteria (appid, session, type, criteria, from, count)
The GetObjects method is intended for rectilinear sequential selection of objects. It returns lists of objects that are selected according to their creation sequence. In other words, select lists of objects sorted in ascending order of identifiers (id). The parameters from and count indicate from which index to take and how much, these are not required parameters.
Consider an example. Call the method with the following parameters GetObjects (appid, session, “book”)
As a result, we get the following answer.
{response: 0, book: [
{id: 1, price: 10.5, name: ”Web Application Development”, description: ”about designing and developing complex one-page application sites”, author: ”I. Know-it-all”},
{id: 2, price: 8, name: ”Development of desktop applications”, description: ”about design and development of simple desktop applications”, author: ”EVT News”},
{id: 3, {price: 12, name: ”Development of mobile applications”, description: ”about design and development of applications for mobile phones”, author: ”Know-it-all I.N”}
]}
The operation of the GetObjects method is fairly simple and transparent.
Consider the work of the GetObjectsByCriteria method. A distinctive feature of this method relative to the previous method is the ability to set search criteria for objects. This method allows you to flexibly form lists of displayed objects.
You can create search criteria using two different syntaxes. Either using SQL similar commands or using JSONQL syntax. You do not need to explicitly specify which syntax to form a criterion you use. The method automatically determines which syntax the developer uses. Let's look at an example of the same query in SQL and JSONQL syntaxes. For example, we get a list of books, the price of which is less than 11.
SQL search criteria: price <11
GetObjectsByCriteria (appid, session, “book”, “price <11”)
JSONQL search term: {price: {'<': 11}}
GetObjectsByCriteria (appid, session, “book”, “{price: {'<': 11}}”)
Get a list of books whose authors are Nerdy or Dunno.
SQL search criteria: author author like '% Nerdy%' or author like '% Dunno%'
GetObjectsByCriteria (appid, session, “book”, “author like '% Nerdy%' or author like '% Dunno%'”)
JSONQL search term: {author: {like: ['% Nerdy%', '% Dunno%']}}
GetObjectsByCriteria (appid, session, “book”, {author: {like: ['% Nerdy%', '% Dunno%']}})
You can use both SQL and JSONQL syntax. Choose what you prefer. If everything is clear and familiar with SQL syntax, then JSONQL is a unique development that was invented when developing the Hivext platform. Let's dwell a little more on JSONQL. The syntax of forming criteria using JSONQL is very flexible and has several variations that are similar in their action. For a more complete understanding of the syntax of such criteria, consider in detail the principles of the formation of these search criteria. Since the structure service is implemented on the basis of the Hibernate library, the criteria for searching objects were based on the Hibertnate criteria. Let's take a look at the implementation of the last example, namely, we will get a list of books whose authors are Nerdy or Dunno. Hibertnate uses the following syntax to form a criterion.
Criterion criterion = Restrictions.or (Restrictions.like (“author”, ”% Nerdy%”), Restrictions.like (“author”, "% Dunno%"));
Transform the specified expression in JSONQL syntax. The join operation according to the OR logic is defined by the array [..., ...]. In this way
[{like: {author: '% Nerdy%'}}, {like: {author: '% Dunno%'}}]
Everything is quite simple and transparent. Record almost repeats Hibernate syntax. Let's consider similar in action possible options for the formation of the criterion.
{like: [{author: '% Nerdy%'}, {author: '% Dunno%'}]}
Another option
{like: {author: ['% Nerdy%', '% Dunno%']}}
As you can see, the last option is simplified by the number of characters and readable. However, there is another option that, according to the author, is even more readable. Just swap the condition and field name
{author: [{like: '% Nerdy%'}, {like: '% Dunno%'}]}
{author: [{like: '% Nerdy%'}, {like: '% Dunno%'}]}
{author: {like: ['% Nerdy%', '% Dunno%']}}
The last option is the most readable, however, all 6 of the last options examined are working and perform the same action. Such freedom in shaping the search criteria gives the formation of the criterion great flexibility. Let's consider the variant with the AND operation.
Get a list of books whose authors are Nerdy and Dunno. Hibertnate syntax next
Criterion criterion = Restrictions.and (Restrictions.like (“author”, ”% Nerdy%”), Restrictions.like (“author”, "% Dunno%"));
In JSONQL, the AND join operation is defined by a sequence of single-level objects {...}, {...}
Thus, we get JSONQL syntax:
{author: {like: '% Nerdy%'}}, {author: {like: '% Dunno%'}}
{author: {{like: '% Nerdy%'}, {like: '% Dunno%'}}}
Having such an extensive set of possible JSONQL options, as well as the ability to query using standard SQL, we get a very flexible method for querying objects using criteria.
Also in the service of structures there are other methods for managing types and objects.
All methods of service structures can be found in the documentation: hivext.ru/index.php/Structures
For convenient development and testing of service methods, there is a developer console code.hivext.ru/development/APIConsole/#Data.Base for testing query criteria .
The console has a list of all existing service methods. The console can be tried in real work when creating its types and objects.
One of the main points in the server implementation of the service structures is the dynamic mapping of the created objects into the database tables. At definition of types, at change of structure of types on the server automatic generation of classes which describe types defined by the developer is made. This allows for simple mapping of objects into tables, as well as for inter-linking objects. All the principles of the implementation and construction of service structures cannot be described in one article. I will add that the service also has caching, transactions, and “lazy” (lazy) loading of objects, and much more ...
Service structures will allow you to conveniently and flexibly determine the object-oriented structure of the developed application. The specified service has a complete set of methods for creating, editing, deleting and selecting the necessary types and instances of these types (objects). Also, the service of structures, thanks to the used ORM, allows to abstract from the DBMS and conveniently work with objects. In combination with the service of access rights, it allows defining the OO application architecture and defining clear access rights for subjects of access to objects.
In the next article it is planned to consider the service of access rights, which is also one of the main and critical services for developing full-fledged applications.
We hope that the habrasoobshchestvo with understanding will approach to the analysis of the specified services. Constructive criticism regarding the implementation of such services and the syntax of working with services is welcomed.
Documentation for the service structures hivext.ru/index.php/Structures
In recent times, services for working with online databases have become more active. Such services include such platforms as GData (Google Data API) and Yahoo! Query Language (YQL).
These online database access services allow you to better control and operate on data obtained from different platform services, and the most important thing is ready-made services for the use of which it suffices to study the documentation and examples.
Google Base API + Google Data Table API
Google’s online data access service allows
')
- request, enter and manage data programmatically to create applications and mashups;
- allows you to view and update the contents of spreadsheets in the form of Google data feeds (“GData”);
- the client can request a list of user tables, as well as present, edit or delete content in an existing table;
- apply autofiltering and other functions of the tables in the application.
For convenient work with the above services, Google has developed its own SQL-like query language Google Visualization API Query Language.
Yahoo! Query Language
Query language to retrieve data from Yahoo! Data arrays .. An SQL-like query language for working with web services from Yahoo! .. Query results can be obtained in XML or JSON formats.
Here is what Jonathan Trevor, head of development at YQL Execute, says:
“Developers can now call multiple services at the same time and operate on multiple data sources, making their applications more powerful and diverse. Received from Yahoo! data can be real-time recoded from one format to another. ”
YQL makes it easy to handle data using SQL with a similar syntax. For example, you want to select photos in Paris, France that have a Creative Commons Attribution (4) license:
select * from flickr.photos.info where photo_id in (select id from flickr.photos.search where woe_id in (select woeid from geo.places where text = 'paris, france') and license = 4)
Data tables can be formed and stored using the “Open Table” service. The service is quite interesting, but the description is still very small.
The above services will allow access and online manipulation of data tables, which is vital for many applications. After analyzing the above services and a number of other similar services, we implemented a similar service for working with data tables. However, it has a specific distinguishing feature. Let's take a closer look at our development right from the inside.
Hivext structure service
API documentation www.hivext.ru/index.php/Structures
One of the extremely necessary for flexible and convenient development of applications based on the platform is service structures. A distinctive feature of our development is the object approach in the implementation of such a service. In order to provide all the advantages of an object-oriented approach to building applications that has been proven over the years, the service was designed with a maximum “tilt” in the direction of the OOP (Object Oriented Design). Due to this, the structure service allows developers to create their own applications in the OOP style, allows you to manipulate and control the types and objects of the developed applications.
General characteristics of the service
- Service and data are available on any platform and device where HTTP requests are supported (directly to the database can also be accessed from Javascript, Actionscript and other client programming languages);
- Object relational projection (ORM) of the created objects in the database tables;
- Service and data are available on any platform and device where HTTP requests are supported;
- Unlimited database volumes for applications;
- Work with the database at the level of types and objects, flexible data management through service methods.
The structure service is intended for defining types and creating instances of types (objects). The approach to creating your objects is quite simple and familiar. To create a collection, a set of any objects, it is necessary to determine the type of future objects. Defining a type means defining fields that future objects must contain. In turn, the definition of a type field means specifying the name of the field and the simple base type of this field. There are predefined simple types in the structure service, based on which developers can create their more complex types.
List of simple types that are predefined in the structure service
- string / String - length 255 characters (default type)
- string (N) / String (N) - N any integer (from 1 to 10240 characters)
- text / Text is a text data type, can contain a maximum of 65000 characters.
- mtext / Mtext is a text data type, can contain a maximum of 16000000 characters.
- short / short
- int / int / integer / integer
- long / Long
- bool / Bool / boolean / Boolean
- float / float
- double / Double
- char / char
- byte / byte
Based on the above types, developers can create their own more complex types. Let's look at an example. Suppose we need to create a book online store. One of the required objects of our application will be a product-book, which we sell. We define the type for our book objects. To define a type, use the DefineType method (appid, session, type, fields)
Regarding our books, we will call the method with the following parameters.
DefineType (appid, session, “book”, {price: ”float”, name: ”string”, description: ”string (500)”, author: ”string (100)”})
Thus, we have defined the type book , which has price fields — the price of the book, name — the name of the book, description — description of the book, author — author of the book.
Then we can safely create instances of this type, i.e. objects are books. To do this, use the method
CreateObject (appid, session, type, data)
Regarding our book type, call the method with the following parameters
CreateObject (appid, session, “book”, {price: 10.5, name: “Web Application Development”, description: ”about designing and developing complex one-page application sites”, author: “Know-it-all I.”})
Upon successful creation of the object, in the response we get the identifier of the new object.
Now we have the first object of type book . Further, in a similar way, we create the set of objects necessary for our online store.
CreateObject (appid, session, “book”, {price: 8, name: "Desktop Development", description: "about designing and developing simple desktop applications", author: "EV News"})
CreateObject (appid, session, “book”, {price: 12, name: "Mobile Application Development", description: "about designing and developing applications for mobile phones", author: "Know-it-all I.N.}}
And so on. It's pretty simple!
Further, it is necessary to display formatted lists of books available in our store to users. This is made even easier. To select a set of objects, two methods of structure service are used.
GetObjects (appid, session, type, from, count)
GetObjectsByCriteria (appid, session, type, criteria, from, count)
The GetObjects method is intended for rectilinear sequential selection of objects. It returns lists of objects that are selected according to their creation sequence. In other words, select lists of objects sorted in ascending order of identifiers (id). The parameters from and count indicate from which index to take and how much, these are not required parameters.
Consider an example. Call the method with the following parameters GetObjects (appid, session, “book”)
As a result, we get the following answer.
{response: 0, book: [
{id: 1, price: 10.5, name: ”Web Application Development”, description: ”about designing and developing complex one-page application sites”, author: ”I. Know-it-all”},
{id: 2, price: 8, name: ”Development of desktop applications”, description: ”about design and development of simple desktop applications”, author: ”EVT News”},
{id: 3, {price: 12, name: ”Development of mobile applications”, description: ”about design and development of applications for mobile phones”, author: ”Know-it-all I.N”}
]}
The operation of the GetObjects method is fairly simple and transparent.
Consider the work of the GetObjectsByCriteria method. A distinctive feature of this method relative to the previous method is the ability to set search criteria for objects. This method allows you to flexibly form lists of displayed objects.
You can create search criteria using two different syntaxes. Either using SQL similar commands or using JSONQL syntax. You do not need to explicitly specify which syntax to form a criterion you use. The method automatically determines which syntax the developer uses. Let's look at an example of the same query in SQL and JSONQL syntaxes. For example, we get a list of books, the price of which is less than 11.
SQL search criteria: price <11
GetObjectsByCriteria (appid, session, “book”, “price <11”)
JSONQL search term: {price: {'<': 11}}
GetObjectsByCriteria (appid, session, “book”, “{price: {'<': 11}}”)
Get a list of books whose authors are Nerdy or Dunno.
SQL search criteria: author author like '% Nerdy%' or author like '% Dunno%'
GetObjectsByCriteria (appid, session, “book”, “author like '% Nerdy%' or author like '% Dunno%'”)
JSONQL search term: {author: {like: ['% Nerdy%', '% Dunno%']}}
GetObjectsByCriteria (appid, session, “book”, {author: {like: ['% Nerdy%', '% Dunno%']}})
You can use both SQL and JSONQL syntax. Choose what you prefer. If everything is clear and familiar with SQL syntax, then JSONQL is a unique development that was invented when developing the Hivext platform. Let's dwell a little more on JSONQL. The syntax of forming criteria using JSONQL is very flexible and has several variations that are similar in their action. For a more complete understanding of the syntax of such criteria, consider in detail the principles of the formation of these search criteria. Since the structure service is implemented on the basis of the Hibernate library, the criteria for searching objects were based on the Hibertnate criteria. Let's take a look at the implementation of the last example, namely, we will get a list of books whose authors are Nerdy or Dunno. Hibertnate uses the following syntax to form a criterion.
Criterion criterion = Restrictions.or (Restrictions.like (“author”, ”% Nerdy%”), Restrictions.like (“author”, "% Dunno%"));
Transform the specified expression in JSONQL syntax. The join operation according to the OR logic is defined by the array [..., ...]. In this way
[{like: {author: '% Nerdy%'}}, {like: {author: '% Dunno%'}}]
Everything is quite simple and transparent. Record almost repeats Hibernate syntax. Let's consider similar in action possible options for the formation of the criterion.
{like: [{author: '% Nerdy%'}, {author: '% Dunno%'}]}
Another option
{like: {author: ['% Nerdy%', '% Dunno%']}}
As you can see, the last option is simplified by the number of characters and readable. However, there is another option that, according to the author, is even more readable. Just swap the condition and field name
{author: [{like: '% Nerdy%'}, {like: '% Dunno%'}]}
{author: [{like: '% Nerdy%'}, {like: '% Dunno%'}]}
{author: {like: ['% Nerdy%', '% Dunno%']}}
The last option is the most readable, however, all 6 of the last options examined are working and perform the same action. Such freedom in shaping the search criteria gives the formation of the criterion great flexibility. Let's consider the variant with the AND operation.
Get a list of books whose authors are Nerdy and Dunno. Hibertnate syntax next
Criterion criterion = Restrictions.and (Restrictions.like (“author”, ”% Nerdy%”), Restrictions.like (“author”, "% Dunno%"));
In JSONQL, the AND join operation is defined by a sequence of single-level objects {...}, {...}
Thus, we get JSONQL syntax:
{author: {like: '% Nerdy%'}}, {author: {like: '% Dunno%'}}
{author: {{like: '% Nerdy%'}, {like: '% Dunno%'}}}
Having such an extensive set of possible JSONQL options, as well as the ability to query using standard SQL, we get a very flexible method for querying objects using criteria.
Also in the service of structures there are other methods for managing types and objects.
All methods of service structures can be found in the documentation: hivext.ru/index.php/Structures
Console
For convenient development and testing of service methods, there is a developer console code.hivext.ru/development/APIConsole/#Data.Base for testing query criteria .
The console has a list of all existing service methods. The console can be tried in real work when creating its types and objects.
One of the main points in the server implementation of the service structures is the dynamic mapping of the created objects into the database tables. At definition of types, at change of structure of types on the server automatic generation of classes which describe types defined by the developer is made. This allows for simple mapping of objects into tables, as well as for inter-linking objects. All the principles of the implementation and construction of service structures cannot be described in one article. I will add that the service also has caching, transactions, and “lazy” (lazy) loading of objects, and much more ...
Service structures will allow you to conveniently and flexibly determine the object-oriented structure of the developed application. The specified service has a complete set of methods for creating, editing, deleting and selecting the necessary types and instances of these types (objects). Also, the service of structures, thanks to the used ORM, allows to abstract from the DBMS and conveniently work with objects. In combination with the service of access rights, it allows defining the OO application architecture and defining clear access rights for subjects of access to objects.
In the next article it is planned to consider the service of access rights, which is also one of the main and critical services for developing full-fledged applications.
We hope that the habrasoobshchestvo with understanding will approach to the analysis of the specified services. Constructive criticism regarding the implementation of such services and the syntax of working with services is welcomed.
Documentation for the service structures hivext.ru/index.php/Structures
Source: https://habr.com/ru/post/59365/
All Articles