Flask Mega-Tutorial, Part 8: Subscribers, Contacts and Friends

This is the eighth article in the series where I describe my experience of writing a Python web application using the Flask mic framework.

The purpose of this guide is to develop a fairly functional microblog application, which I decided to call microblog, in the absence of originality.


')

Brief repetition

Our little microblogging is slowly growing, and today we will touch on topics that are necessary for a complete application.

Today we will work a little with our database. Each user of our application should be able to choose the users whom he wants to track, and our database should also store data about who is tracking whom. All social applications have these features in different variations. Some call it Contacts, others Connections, Friends, Pals or Subscribers. Some sites use a similar idea for the Allowed and Ignored Users list. We will call them Subscribers, but we will implement without holding on to the name.

Subscribers feature design

Before we start writing code, let's think about the functionality that we want to get from this feature. Let's start with the most obvious. We want our users to conveniently maintain lists of subscriptions to other users. On the other hand, we want to know the list of subscribers of each user. We also want to be able to find out whether the user has a subscriber or whether he subscribes to other users. Users will click on the Subscribe link in any other user’s profile to start tracking it. In the same way, clicking on the "unsubscribe" link will cancel the subscription to the user. The last requirement is the ability of the user to request from the database all posts of the monitored users.

So, if you thought it would be quick and easy, think again!

Communication within the base

We said that we want to have lists of subscribers and subscriptions for all users. Unfortunately, relational databases do not have the @@ list @@ type, all we have is tables with records and relationships between records. We already have a table in our database representing the users, it remains to come up with dependency relationships that will model the connections of subscribers / subscriptions. This is a good point to sort out three types of relationships in relational databases:

One-to-many

We have already seen the one-to-many relationship in the previous database article:


Two entities related to this relationship are users and posts . We say that the user can have many posts, and the post has only one user. These relationships are used in the database with the foreign key (FK) on the "many" side. In the example above, the external key is the user_id field added to the posts table. This field links each post with the author’s entry in the users table.

It is clear that the user_id field provides direct access to the author of this post, but what about feedback? For the link to be useful, we must be able to get a list of posts written by the user. It turns out that the user_id field in the posts table is enough to answer our question, since the databases have indices that allow you to make such requests as “receive all messages where user_id is X”.

Many-to-many

The many-to-many relationship is a bit more complicated. For example, consider the database in which students and teachers are stored. We can say that a student can have many teachers, and a teacher can have many students. It looks like two partially coincident one-to-many relationships.

For this type of relationship, we need to be able to query the database and get a list of the teachers who teach the student and the list of students in the teacher’s class. It turns out that it is rather difficult to present in the database; such an attitude cannot be modeled by adding foreign keys to already existing tables.

The implementation of the many-to-many relationship requires the use of an auxiliary table, called a pivot table. For example, the database for students and teachers will look like:



Although it may seem difficult, but the summary table can answer many of our questions, such as:

• Who does student S study?
• Who does teacher T teach?
• How many students does teacher T have?
• How many teachers does student S have?
• Teacher T teaches student S?
• Is student S attending teacher class T?

One to one

One-to-one relationships are a special case of one-to-many relationships. The presentation is very similar, but it prohibits the addition of more than one link, so as not to become one-to-many. Although there are cases in which this type of relationship is useful, it does not happen as often as in the case of the other two types, since in a situation where two tables are related by the one-to-one relationship, it may make sense to combine the tables into one.

Subscribers \ Subscriptions Submission

From the relationships above, we can easily determine that the many-to-many data model is right for us, because the user can follow many other users and the user can have many subscribers. But there is a feature. We want to represent users who are subscribed to other users, but we have only one user table. So, what should we use as a second entity in a many-to-many relationship?

Of course, the second entity in the relationship will be the same table of users. Relations in which instances of an entity are associated with other instances of the same entity are called self-referential relationships, and this is exactly what we need.

This is a diagram of our many-to-many relationships:



The followers table is a pivot table. Both foreign keys point to the user table, since we tied the table with ourselves. Each entry in this table represents one relationship between the subscribed user and the person to whom it is subscribed. As in the example of students and teachers, a configuration like this allows our database to answer all the questions about subscribers and their subscriptions that we need. It's pretty simple.

DB Model

Changes in our model will not be very big. We start by adding the @@ followers @@ table (file @@ app / models.py @@):

 followers = db.Table('followers', db.Column('follower_id', db.Integer, db.ForeignKey('user.id')), db.Column('followed_id', db.Integer, db.ForeignKey('user.id')) ) 

This is a live translation of the table links from our diagram. Notice that we did not declare this table as a model, as we did for users and posts . Since this is an auxiliary table that does not have data other than foreign keys, we will use the low-level flask-sqlalchemy API to create the table without creating its model.

Next, we describe the many-to-many relationships in the users table ( app/models.py )

 class User(db.Model): id = db.Column(db.Integer, primary_key = True) nickname = db.Column(db.String(64), unique = True) email = db.Column(db.String(120), index = True, unique = True) role = db.Column(db.SmallInteger, default = ROLE_USER) posts = db.relationship('Post', backref = 'author', lazy = 'dynamic') about_me = db.Column(db.String(140)) last_seen = db.Column(db.DateTime) followed = db.relationship('User', secondary = followers, primaryjoin = (followers.c.follower_id == id), secondaryjoin = (followers.c.followed_id == id), backref = db.backref('followers', lazy = 'dynamic'), lazy = 'dynamic') 

The configuration of the relationship is non-trivial and requires some explanation. We use the db.relationship function to define the relationship between the tables, as we did in the previous article. We will associate a User instance with another User instance, and for the agreement, we say that, in a pair of related users, the left user is subscribed to the right user. As can be seen from the description of the relationship, we called the left side followed , because when we request the left side relationship, we get a list of subscribers. Let's look at all the db.relationship arguments db.relationship by one:

• 'User' right side of the entity in the relationship (the left side is the parent class). When we define self-referencing relationships, we use the same class on both sides.
• secondary indicates the pivot table used in this relationship.
• primaryjoin describes the connection of the left side of an entity with a pivot table. Note that the followers table is not a model, so a slightly weird syntax is used to get to the field name.
• secondaryjoin describes the relationship of the right side with the pivot table.
• backref describes how this relationship will be available on the right side of the entity. We said that for this user request followed returned all users on the right side that are associated with the user on the left. The backlink is called followers and it will return all users on the left, who are associated with the user on the right. The optional argument lazy indicates how this request is executed. This mode indicates that the request will not be executed unless explicitly requested. This is useful for improving performance, and also because we can get this request and change it before executing. We will discuss this in more detail later.
• The lazy parameter is a bit like the parameter with the same name in the backref , but this one refers to the request, not the feedback.

Do not despair if it is difficult to understand. We will see how to use these requests, and then everything will become clearer. Since we made the database update, now we can create a new migration:

 ./db_migrate.py 

This we completed the database changes. It remains quite a bit.

Add and remove subscribers

To support reuse of the code, we will implement the subscription \ subscriber functionality within the User model and will not make it into the view. Thus, we can use this function for the current application (refer from the view) and use it in testing. In principle, it is always better to move the application logic from the view to the model, this greatly simplifies testing. You have to keep the submissions as simple as possible, because they are harder to test automatically.

Below is the code for adding and removing subscribers, defined as methods of the User model (file app/models.py ):

 class User(db.Model): #... def follow(self, user): if not self.is_following(user): self.followed.append(user) return self def unfollow(self, user): if self.is_following(user): self.followed.remove(user) return self def is_following(self, user): return self.followed.filter(followers.c.followed_id == user.id).count() > 0 

Thanks to the power of Alchemy SQL, which does a lot of work, these methods are surprisingly simple. We simply add or remove items, and SQLAlchemy does the rest of the work. The follow and unfollow defined so that they return an object when everything is successful, and None when the operation cannot be completed. When the object returns, it must be added to the database and made a commit.

The is_following method does quite a lot, despite one line of code. We accept a query that returns all pairs (follower, followed) with user input and we filter them by the followed column. From filter() returns the modified request, not yet executed. Thus, we call count() on this query and now this query will be executed and will return the number of records found. If we get at least one, then we will know that there are connections. If we do not receive anything, then we will know that there are no connections.

Testing

Let's write a test for our code (file tests.py ):

 class TestCase(unittest.TestCase): #... def test_follow(self): u1 = User(nickname = 'john', email = 'john@example.com') u2 = User(nickname = 'susan', email = 'susan@example.com') db.session.add(u1) db.session.add(u2) db.session.commit() assert u1.unfollow(u2) == None u = u1.follow(u2) db.session.add(u) db.session.commit() assert u1.follow(u2) == None assert u1.is_following(u2) assert u1.followed.count() == 1 assert u1.followed.first().nickname == 'susan' assert u2.followers.count() == 1 assert u2.followers.first().nickname == 'john' u = u1.unfollow(u2) assert u != None db.session.add(u) db.session.commit() assert u1.is_following(u2) == False assert u1.followed.count() == 0 assert u2.followers.count() == 0 

After adding this test to the test framework, we can start the test suite with the command:

 ./tests.py 

And if everything works, all tests will be successfully passed.

Queries to the database

Our current database model supports most of the requirements we listed at the beginning. What we lack is, in fact, the most difficult to implement. On the main page of the site will be shown messages written by all the people who are monitored by our logged in user, so we need a request that returns all these messages.

The most obvious solution is a query that will give a list of tracked people that we can already do. Then for each of these users, we will execute a request to get his messages. When we have all the messages, we can combine them into a single list and sort them by time. It sounds good? Not really.

This approach has a couple of problems. What happens if a user tracks a thousand people? We will have to perform a thousand queries to the database only to collect messages. And now we have thousands of lists in memory that we need to sort and merge. On our main page page numbers are implemented, so we will not show all available messages, but only the first 50, and the links on which the next 50 can be viewed. If we are going to show messages sorted by date, how do we know which of them are the last 50 messages all users, unless we first receive all the messages and sort them.

In fact, this is a terrible solution that scales very badly. Although this method of collecting and sorting somehow works, it is not effective enough. This is exactly the job in which relational databases succeed. The database contains indexes that allow it to perform queries and sorts much more efficiently than we can do it on our part.

We have to come up with a query that expresses what we want to receive, and the database will calculate how to more effectively extract the information we need.

To dispel the mystery, here is a request that will do what we need. Unfortunately, this is another overloaded one-liner, which we add to the user's model ( app.models.py file):

 class User(db.Model): #... def followed_posts(self): return Post.query.join(followers, (followers.c.followed_id == Post.user_id)).filter(followers.c.follower_id == self.id).order_by(Post.timestamp.desc()) 

Let's try to decrypt this request step by step. Here are 3 parts: join, filter, and order_by.

Joins

To understand what the join operation does, let's look at an example. Suppose we have a User table with the following content:

User
id	nickname
one	john
2	susan
3	mary
four	david

Other table fields are not displayed, so as not to complicate the example.

Let's assume that our summary table says that the user “john” is subscribed to “susan” and “david”, the user “susan” is subscribed to “mary” and “mary” is subscribed to “david”. Then the pivot table will look like this:

followers
follower_id	followed_id
one	2
one	four
2	3
3	four

Finally, our Post table contains one post from each user:

Post
id	text	user_id
one	post from susan	2
2	post from mary	3
3	post from david	four
four	post from john	one

Some fields have also been removed here so as not to complicate the example.

The following is part of our query with join isolated from the rest:

 Post.query.join(followers, (followers.c.followed_id == Post.user_id)) 

The join operation is called on the Post table. There are two arguments, the first one is another table, in our case followers . The second argument specifies which fields to join the table. The join operation will make a temporary table with data from Post and followers merged according to the specified condition.

In this example, we want the followers table fields to correspond to the user_id table user_id fields.

To perform this merge, we take each record from the Post table (the left part of the join) and join the fields from the record in the followers table (the right part of the join) that match the condition. If the record does not meet the condition, then it does not fall into the table.

The result of the join on our example in this temporary table

Post			followers
id	text	user_id	follower_id	followed_id
one	post from susan	2	one	2
2	post from mary	3	2	3
3	post from david	four	one	four
3	post from david	four	3	four

Notice how the message with user_id = 1 was removed from join, because there are no entries in the table of subscribers that there was followed_id = 1. Also note that the message with user_id = 4 appears twice, because the table of subscribers has two entries with followed_id = 4.

Filters

The join operation gave us a list of messages from people that someone is following, without specifying who the subscriber is. We are interested in a subset of this list, in which only those messages that are monitored by one particular user. So we will filter this table by subscriber. The part of the request with the filter will be as follows:

 filter(followers.c.follower_id == self.id) 

Remember that the request is executed in the context of our target user, therefore the self.id method of the User class in this context gives the id of the user that interests us. With this filter, we tell the database that we want to leave only those records from the table created with the help of join in which our user is specified as a subscriber. Continuing our example, if we query users with id = 1, then we will come to another temporary table:

Post			followers
id	text	user_id	follower_id	followed_id
one	post from susan	2	one	2
3	post from david	four	one	four

And this is exactly the posts that we need!

Remember that the query was executed on the Post class, so even if we end up in a temporary table not related to any model, the result will be included in this temporary table, without additional columns added by the join operation.

Sorting

The final step in the process is to sort the results according to our criteria. The part of the query that does this is as follows:

 order_by(Post.timestamp.desc()) 

Here we say that the results should be sorted by timestamp in descending order, so the first post will be the first.

There is only one minor detail that can improve our query. When users read posts that are subscribed to, they may want to see their own posts in the feed, and it would be nice to include them in the query result.

There is an easy way to do this, which does not require any changes! We just simply make sure that each user is added to the database as his own subscriber and this little problem will no longer concern us. At the conclusion of our long discussion of queries, let's write a unit test for our query (file tests.py):

 #... from datetime import datetime, timedelta from app.models import User, Post #... class TestCase(unittest.TestCase): #... def test_follow_posts(self): # make four users u1 = User(nickname = 'john', email = 'john@example.com') u2 = User(nickname = 'susan', email = 'susan@example.com') u3 = User(nickname = 'mary', email = 'mary@example.com') u4 = User(nickname = 'david', email = 'david@example.com') db.session.add(u1) db.session.add(u2) db.session.add(u3) db.session.add(u4) # make four posts utcnow = datetime.utcnow() p1 = Post(body = "post from john", author = u1, timestamp = utcnow + timedelta(seconds = 1)) p2 = Post(body = "post from susan", author = u2, timestamp = utcnow + timedelta(seconds = 2)) p3 = Post(body = "post from mary", author = u3, timestamp = utcnow + timedelta(seconds = 3)) p4 = Post(body = "post from david", author = u4, timestamp = utcnow + timedelta(seconds = 4)) db.session.add(p1) db.session.add(p2) db.session.add(p3) db.session.add(p4) db.session.commit() # setup the followers u1.follow(u1) # john follows himself u1.follow(u2) # john follows susan u1.follow(u4) # john follows david u2.follow(u2) # susan follows herself u2.follow(u3) # susan follows mary u3.follow(u3) # mary follows herself u3.follow(u4) # mary follows david u4.follow(u4) # david follows himself db.session.add(u1) db.session.add(u2) db.session.add(u3) db.session.add(u4) db.session.commit() # check the followed posts of each user f1 = u1.followed_posts().all() f2 = u2.followed_posts().all() f3 = u3.followed_posts().all() f4 = u4.followed_posts().all() assert len(f1) == 3 assert len(f2) == 2 assert len(f3) == 2 assert len(f4) == 1 assert f1 == [p4, p2, p1] assert f2 == [p3, p2] assert f3 == [p4, p3] assert f4 == [p4] 

This test has a lot of pre-tuning code, but the test code itself is rather short. First we check that the number of monitored posts returned for each user is equal to the expected one. Then for each user, we check that the correct posts were returned and they came in the correct order (note that we inserted messages with timestamps guaranteeing always the same order).

Note the use of followed_post () method. This method returns a query object, not a result. It also works lazy = "dynamic" in the relationship DB.

It is always a good idea to return an object instead of a result, because it gives the caller the opportunity to complete the query before executing.

There are several methods in the query object that trigger the query. We have seen that count () executes the query and returns the number of results, discarding the data itself. We also used first () to return the first result in the list and discard the rest. In the test, we used the all () method to get an array with all the results.

Possible improvements

We have now implemented all the necessary functions of subscriptions, but there are several ways to improve our design and make it more flexible. All social networks that we love to hate support similar communication paths of users, but they have more opportunities to manage information. For example, it is not possible to block subscribers. This will add another layer of complexity to our requests, as we now have to not only select users, but also weed out the posts of those users who have blocked us. How to implement it?

The simple way is another self-referencing table with a many-to-many relationship for recording who is blocking whom, and another join + filter in the query that returns tracked posts. Another popular feature of social networks is the ability to group subscribers into lists to share their information with each group. This also requires additional links and adds complexity to the queries.

We will not have these functions in the microblogging, but if it causes enough interest, I will be happy to write an article on this topic. Let me know in the comments!

We put things in order

We are now quite advanced. But although we solved the problems with database configuration and queries, we did not include new functionality in our application. Fortunately for us, this is no problem. We just need to fix the view functions and patterns to call new methods in the User model when necessary. So let's do it.

We make ourselves our own subscriber.
We decided to mark all users subscribed to themselves, so that they could see their posts in the feed.

We are going to do this at the point where users are assigned the first account settings in the after_login handler for OpenID (the file 'app / views.py'):

 @oid.after_login def after_login(resp): if resp.email is None or resp.email == "": flash('Invalid login. Please try again.') return redirect(url_for('login')) user = User.query.filter_by(email = resp.email).first() if user is None: nickname = resp.nickname if nickname is None or nickname == "": nickname = resp.email.split('@')[0] nickname = User.make_unique_nickname(nickname) user = User(nickname = nickname, email = resp.email, role = ROLE_USER) db.session.add(user) db.session.commit() # make the user follow him/herself db.session.add(user.follow(user)) db.session.commit() remember_me = False if 'remember_me' in session: remember_me = session['remember_me'] session.pop('remember_me', None) login_user(user, remember = remember_me) return redirect(request.args.get('next') or url_for('index')) 

Links subscribe and unsubscribe

Next, we define the functions of the subscription and unsubscribe presentation (file app / views.py):

 @app.route('/follow/<nickname>') @login_required def follow(nickname): user = User.query.filter_by(nickname = nickname).first() if user == None: flash('User ' + nickname + ' not found.') return redirect(url_for('index')) if user == g.user: flash('You can\'t follow yourself!') return redirect(url_for('user', nickname = nickname)) u = g.user.follow(user) if u is None: flash('Cannot follow ' + nickname + '.') return redirect(url_for('user', nickname = nickname)) db.session.add(u) db.session.commit() flash('You are now following ' + nickname + '!') return redirect(url_for('user', nickname = nickname)) @app.route('/unfollow/<nickname>') @login_required def unfollow(nickname): user = User.query.filter_by(nickname = nickname).first() if user == None: flash('User ' + nickname + ' not found.') return redirect(url_for('index')) if user == g.user: flash('You can\'t unfollow yourself!') return redirect(url_for('user', nickname = nickname)) u = g.user.unfollow(user) if u is None: flash('Cannot unfollow ' + nickname + '.') return redirect(url_for('user', nickname = nickname)) db.session.add(u) db.session.commit() flash('You have stopped following ' + nickname + '.') return redirect(url_for('user', nickname = nickname)) 

This should be clear, but you should pay attention to the checks in which we try to prevent an error and try to provide a message to the user when a problem has occurred. Now we have view functions, so we can connect them. Links to subscribe or unsubscribe will be available on each user's profile page (file app / templates / user.html):

 <!-- extend base layout --> {% extends "base.html" %} {% block content %} <table> <tr valign="top"> <td><img src=""></td> <td> <h1>User: {{user.nickname}}</h1> {% if user.about_me %}<p>{{user.about_me}}</p>{% endif %} {% if user.last_seen %}<p><i>Last seen on: {{user.last_seen}}</i></p>{% endif %} <p>{{user.followers.count()}} followers | {% if user.id == g.user.id %} <a href="{{url_for('edit')}}">Edit your profile</a> {% elif not g.user.is_following(user) %} <a href="{{url_for('follow', nickname = user.nickname)}}">Follow</a> {% else %} <a href="{{url_for('unfollow', nickname = user.nickname)}}">Unfollow</a> {% endif %} </p> </td> </tr> </table> <hr> {% for post in posts %} {% include 'post.html' %} {% endfor %} {% endblock %} 

In the line in which there was an “Edit” link, we now show the number of subscribers that the user has and one of three possible links:

• if the profile belongs to the logged in user, the button “Edit” will be visible
• otherwise, if not subscribed to Subscribe
• otherwise, the "Unsubscribe" link

, , OpenID .

, , , .

Final words

.
, - , .
, , , .

:

microblog-0.8.zip .

, flask. .

. !
Miguel