Bottom-up development and database.

Paul Graham in his essays often deals with the topic of bottom-up development. This method is mentioned by him when he writes about software development, about the way of doing business, about the advantages of open source and blogs. Below I will describe why, with this model of development, interaction with the database rises across the throat, and propose a solution that is appropriate in some cases.

It cannot be said that the bottom-up method is better than any other method, just in more cases it is more viable. Indeed, a person opening, for example, a store or creating a DE will start small and will gradually develop it; but if he has already developed his little shop into a chain of stores comparable to Auchan, or wrote KDE, then the next time he will not start small again, but he will spend a long time in plans and then he will implement a large project right away. The first time a person receives feedback during the development, the second time, too, though he is already developing a store, not a store, and the response is received from a previously developed project.

In large companies, development is often top-down. Unfortunately, simple performers because of this technique begin to dislike their work, as they lose the element of creativity. Paul, in his essay, What business can learn from open source, proposed an interesting development model when a software company invests money in teams of programmers who are already developing in the field that interests the company. Thus, the hierarch boss and slave disappears and the developers work with more enthusiasm, and the company deals with marketing, distribution, and does not contain its own staff of programmers. Of the examples that are known to the readers of Habr is the F # language, on which Don Syme worked for a long time and which microsoft decided to launch into the mainstream in the next studio release.
')
In his essays, Paul focuses on the prevalence of practice over planning, speed of writing, and ease of change. Unfortunately, many tools for developers were created without regard for these principles and were aimed at those companies that adhere to the established boss-slave relationship. Often, when working in XP style, interaction with the database rises like a bone in the throat: difficulties in testing; the heterogeneity of the code - you have to think about it on a HLM, but on declarative SQL, meaning the nuances of mapping one to another; a huge amount of unnecessary code and unnecessary abstractions that are responsible for interacting with the base, and the solutions that are designed to fix this ugliness are, in fact, kodogeneration with all the ensuing problems.

The first to realize this mess was people who wrote in dynamic languages ​​- and tried to fix this problem. So couchdb, strokedb and simpledb appeared. This solution is suitable for small projects, but I think Yandex-level companies will not keep their bases on subd, which have not been tested by time.

Imagine that in the courtyard in '95, you woke up and decided to create Yandex, that is, to create a web search engine. Obviously, the most interesting thing in the task is the analysis of information, and not the way it is stored, so you immediately pounced on this task. Partially decided it and want to make the first prototype that would find an investor. It is clear that you can’t do without data storage here, but on the other hand, you understand that you didn’t choose any structure of the database, then by the time it was launched, it would change a hundred times, so from the prospect of writing a hundred times workaround to work with the database you are sick .

Thus, you need a solution that will not hold back development in the future, it should easily change when requirements change and have the property that at any moment this solution can be quickly transferred to relational databases. Wanting to also get rid of heterogeneity and facilitate testing, we conclude that a good solution would be to write your database engine in the language in which the development is being carried out, and bring the interaction methods to the interface and work only with the interface, thus the transition to the tested DBMS will be made implementation of interaction with the database, consistent with this interface.

If we want to write our own simple database engine, then it is easy to do using a language that supports tuples, lists and list comprehension, and since the implementation of the database engine is not the main task that we have, then it is desirable that this language was high-level, had a familiar syntax and supported popular paradigms. I think you already guessed that I am writing about nemerle. Below I give a primitive example of the implementation of the database engine.

Suppose the base scheme is as follows:

DROP TABLE IF EXISTS Author;
CREATE TABLE Author (
ID INT AUTO_INCREMENT NOT NULL ,
Firstname VARCHAR (500) NOT NULL ,
Lastname VARCHAR (500) NOT NULL ,
PRIMARY KEY (ID)
);

DROP TABLE IF EXISTS Journal;
CREATE TABLE Journal (
ID INT AUTO_INCREMENT NOT NULL ,
Title VARCHAR (500) NOT NULL ,
PRIMARY KEY (ID)
);

DROP TABLE IF EXISTS Article;
CREATE TABLE Article (
ID INT AUTO_INCREMENT NOT NULL ,
Title VARCHAR (500) NOT NULL ,
JournalID INT NOT NULL ,
AuthorID INT NOT NULL ,
CONSTRAINT `journal_id` FOREIGN KEY (JournalID) REFERENCES Journal(ID) ON DELETE CASCADE ,
CONSTRAINT `author_id` FOREIGN KEY (AuthorID) REFERENCES Author(ID) ON DELETE CASCADE ,
PRIMARY KEY (ID)
);

* This source code was highlighted with Source Code Highlighter .

You can implement this structure and a couple of requests to it in the following way.

public interface IDB
{
AddAuthor(firstName : string , lastName : string ) : int ;
GetAuthor(firstName : string ) : list[ int * string * string ];
}

public class MyDB : IDB
{
public this () { Author = []; Journal = []; Article = []; ID = 0; }
public Author : list[ int * string * string ] { get ; set ; } // (ID, Firstname, Lastname)
public Journal : list[ int * string ] { get ; set ; } // (ID, Title)
public Article : list[ int * string * int * int ] { get ; set ; } // (ID, Title, JournalID, AuthorID)
public ID : int { get ; set ;};

public AddAuthor(firstName : string , lastName : string ) : int
{
ID++;
Author = (ID,firstName,lastName)::Author;
ID
}

public GetAuthor(firstName : string ) : list[ int * string * string ]
{
$[(id,first,last)|(id,first,last) in Author, first==firstName]
}
}

* This source code was highlighted with Source Code Highlighter .

It seems to me that it is convenient to use this approach in tasks where the base structure is not obvious at the time of the start of development, where the main emphasis is placed on something else (as in the example with Yandex) and where the base structure is small.