Making a modern web application from scratch

So you decided to make a new project. And this project is a web application. How long will it take to create a basic prototype? How hard is it? What should already be able to start a modern website from the start?

In this article, we will try to write a boilerplate of a simplest web application with the following architecture:

What we will cover:
')

docker-compose dev configuration
backend creation on Flask.
Creating frontend on Express.
build js using webpack.
React, Redux and server side rendering.
task queues with RQ.

Introduction

Before development, of course, you first need to decide what we are developing! As a model application for this article, I decided to make a primitive wiki engine. We will have cards drawn up in Markdown; they can be viewed and (sometime in the future) suggest edits. All this we will issue as a one-page application with server-side rendering (which is absolutely necessary for indexing our future terabyte of content).

Let's take a little more detail on the components that we need for this:

Customer Let's make a one-page application (i.e., with transitions between pages using AJAX) on the React + Redux bundle, which is very widespread in the world of frontend.
Frontend . Let's make a simple Express server that will render our React application (requesting all the necessary data in the back end asynchronously) and provide it to the user.
Backend . Master of business logic, our backend will be a small Flask application. We will store the data (our cards) in MongoDB's popular key-value storage, and we will use Redis for the task queue and, possibly, in the future, caching.
Worker . We will run a separate container for tough tasks with the RQ library.

Infrastructure: git

Probably, it was possible not to talk about it, but, of course, we will be developing in the git-repository.

git init git remote add origin git@github.com:Saluev/habr-app-demo.git git commit --allow-empty -m "Initial commit" git push

(Here it is worth immediately filling .gitignore .)

The final project can be viewed on Github . Each section of the article corresponds to one commit (I did quite a lot of rebeasing to achieve this!).

Infrastructure: docker-compose

Let's start with setting the environment. While the component that we have is abundant, the use of docker-compose would be a very logical development decision.

Add the docker-compose.yml file to the repository docker-compose.yml follows:

 version: '3' services: mongo: image: "mongo:latest" redis: image: "redis:alpine" backend: build: context: . dockerfile: ./docker/backend/Dockerfile environment: - APP_ENV=dev depends_on: - mongo - redis ports: - "40001:40001" volumes: - .:/code frontend: build: context: . dockerfile: ./docker/frontend/Dockerfile environment: - APP_ENV=dev - APP_BACKEND_URL=backend:40001 - APP_FRONTEND_PORT=40002 depends_on: - backend ports: - "40002:40002" volumes: - ./frontend:/app/src worker: build: context: . dockerfile: ./docker/worker/Dockerfile environment: - APP_ENV=dev depends_on: - mongo - redis volumes: - .:/code

Let's take a quick look at what is happening here.

A MongoDB container and a Redis container are created.
The container of our backend is created (which we will describe below). The environment variable APP_ENV = dev is passed to it (we will look at it in order to understand which settings to load Flask), and its port 40001 opens (through it our browser client will go to the API).
A container of our front end is being created. Various environment variables are also thrown into it, which will be useful to us later, and port 40002 opens. This is the main port of our web application: in the browser we will go to http: // localhost: 40002 .
The container of our worker is created. It does not need external ports, but only needs access to MongoDB and Redis.

Now let's create dock files. Right now on Habré there is a series of translations of beautiful articles about Docker - for all the details you can safely go there.

Let's start with the backend.

 # docker/backend/Dockerfile FROM python:stretch COPY requirements.txt /tmp/ RUN pip install -r /tmp/requirements.txt ADD . /code WORKDIR /code CMD gunicorn -w 1 -b 0.0.0.0:40001 --worker-class gevent backend.server:app

This implies that we are launching through gunicorn a Flask application hiding under the name app in the module backend.server .

Not less important docker/backend/.dockerignore :

 .git .idea .logs .pytest_cache frontend tests venv *.pyc *.pyo

A worker in general is similar to a backend, but instead of gunicorn, we have the usual launch of a pit module:

 # docker/worker/Dockerfile FROM python:stretch COPY requirements.txt /tmp/ RUN pip install -r /tmp/requirements.txt ADD . /code WORKDIR /code CMD python -m worker

We will do all the work in worker/__main__.py .

.dockerignore worker is completely analogous to the .dockerignore backend.

Finally, the frontend. About him on Habré there is a whole separate article , but judging by the detailed discussion on StackOverflow and comments in the spirit of “Guys, is it already 2018, there is still no normal solution?” Everything is not so simple. I stopped on this version of dokerfile.

 # docker/frontend/Dockerfile FROM node:carbon WORKDIR /app #  package.json  package-lock.json   npm install,   . COPY frontend/package*.json ./ RUN npm install #       , #     PATH. ENV PATH /app/node_modules/.bin:$PATH #      . ADD frontend /app/src WORKDIR /app/src RUN npm run build CMD npm run start

Pros:

everything is cached as expected (on the bottom layer - dependencies, on the top - the build of our application);
docker-compose exec frontend npm install --save newDependency works as docker-compose exec frontend npm install --save newDependency and modifies package.json in our repository (which would not be the case if we used COPY, as many suggest). It would be undesirable to run just npm install --save newDependency outside the container anyway, because some dependencies of the new package may already be present and be assembled for another platform (for the one that is inside the docker, and not for our working macbook) ), and we don’t even want to require the presence of a Node on a development machine. One Docker to rule them all!

And, of course, docker/frontend/.dockerignore :

 .git .idea .logs .pytest_cache backend worker tools node_modules npm-debug tests venv

So, our frame made of containers is ready and can be filled with its contents!

Backend: Frame on Flask

Add flask , flask-cors , gevent and gunicorn to requirements.txt and create a simple Flask application in backend/server.py .

 # backend/server.py import os.path import flask import flask_cors class HabrAppDemo(flask.Flask): def __init__(self, *args, **kwargs): super().__init__(*args, **kwargs) # CORS        #    ,      # (  Access-Control-Origin  ). #   - . flask_cors.CORS(self) app = HabrAppDemo("habr-app-demo") env = os.environ.get("APP_ENV", "dev") print(f"Starting application in {env} mode") app.config.from_object(f"backend.{env}_settings")

We told Flask to pull up the settings from the backend.{env}_settings file backend.{env}_settings , which means we also need to create (at least an empty) backend/dev_settings.py file so that everything takes off.

Now we can officially RISE our backend!

 habr-app-demo$ docker-compose up backend ... backend_1 | [2019-02-23 10:09:03 +0000] [6] [INFO] Starting gunicorn 19.9.0 backend_1 | [2019-02-23 10:09:03 +0000] [6] [INFO] Listening at: http://0.0.0.0:40001 (6) backend_1 | [2019-02-23 10:09:03 +0000] [6] [INFO] Using worker: gevent backend_1 | [2019-02-23 10:09:03 +0000] [9] [INFO] Booting worker with pid: 9

Moving on.

Front End: Frame on Express

Let's start by creating a package. Having created the frontend folder and running npm init in it, after several unsophisticated questions we will get a complete package.json in the spirit of

 { "name": "habr-app-demo", "version": "0.0.1", "description": "This is an app demo for Habr article.", "main": "index.js", "scripts": { "test": "echo \"Error: no test specified\" && exit 1" }, "repository": { "type": "git", "url": "git+https://github.com/Saluev/habr-app-demo.git" }, "author": "Tigran Saluev <tigran@saluev.com>", "license": "MIT", "bugs": { "url": "https://github.com/Saluev/habr-app-demo/issues" }, "homepage": "https://github.com/Saluev/habr-app-demo#readme" }

In the future, we don’t need Node.js at all on the developer’s machine (although we could still dodge and run npm init via Docker, but oh well).

In Dockerfile we mentioned npm run build and npm run start — add the following commands to package.json :

 --- a/frontend/package.json +++ b/frontend/package.json @@ -4,6 +4,8 @@ "description": "This is an app demo for Habr article.", "main": "index.js", "scripts": { + "build": "echo 'build'", + "start": "node index.js", "test": "echo \"Error: no test specified\" && exit 1" }, "repository": {

The build command is not doing anything yet, but it still comes in handy.

Add Express dependencies and create a simple application in index.js :

 --- a/frontend/package.json +++ b/frontend/package.json @@ -17,5 +17,8 @@ "bugs": { "url": "https://github.com/Saluev/habr-app-demo/issues" }, - "homepage": "https://github.com/Saluev/habr-app-demo#readme" + "homepage": "https://github.com/Saluev/habr-app-demo#readme", + "dependencies": { + "express": "^4.16.3" + } }

 // frontend/index.js const express = require("express"); app = express(); app.listen(process.env.APP_FRONTEND_PORT); app.get("*", (req, res) => { res.send("Hello, world!") });

Now docker-compose up frontend raises our frontend! Moreover, http: // localhost: 40002 should already feature the classic “Hello, world”.

Frontend: build with webpack and React application

The time has come to portray something more than plain text in our application. In this section, we will add the simplest React component of the App and configure the assembly.

When programming on React it is very convenient to use JSX - JavaScript dialect, extended with syntactic constructions of the form

 render() { return <MyButton color="blue">{this.props.caption}</MyButton>; }

However, JavaScript engines do not understand it, so an assembly step is usually added to the frontend. Special JavaScript compilers (aha-aha) turn syntactic sugar into ~~ugly~~ classic JavaScript, handle imports, minify, and so on.

year 2014. apt-cache search java

So, the simplest React component looks very simple.

 // frontend/src/components/app.js import React, {Component} from 'react' class App extends Component { render() { return <h1>Hello, world!</h1> } } export default App

It will simply display our greeting with a more convincing pin.

Add the frontend/src/template.js file containing the minimal HTML framework of our future application:

 // frontend/src/template.js export default function template(title) { let page = ` <!DOCTYPE html> <html lang="en"> <head> <meta charset="utf-8"> <title>${title}</title> </head> <body> <div id="app"></div> <script src="/dist/client.js"></script> </body> </html> `; return page; }

Add a client entry point:

 // frontend/src/client.js import React from 'react' import {render} from 'react-dom' import App from './components/app' render( <App/>, document.querySelector('#app') );

To build all this beauty, we need:

webpack is a fashionable youth collector for JS (although I haven’t read any articles on the frontend for three hours, so I’m not sure about fashion);
babel is a compiler for all sorts of gadgets like JSX, and at the same time a supplier of polyfills for all IE cases.

If the previous iteration of the frontend is still running, all you have to do is

 docker-compose exec frontend npm install --save \ react \ react-dom docker-compose exec frontend npm install --save-dev \ webpack \ webpack-cli \ babel-loader \ @babel/core \ @babel/polyfill \ @babel/preset-env \ @babel/preset-react

to install new dependencies. Now configure the webpack:

 // frontend/webpack.config.js const path = require("path"); //  . clientConfig = { mode: "development", entry: { client: ["./src/client.js", "@babel/polyfill"] }, output: { path: path.resolve(__dirname, "../dist"), filename: "[name].js" }, module: { rules: [ { test: /\.js$/, exclude: /node_modules/, loader: "babel-loader" } ] } }; //  .     : // 1. target: "node" -      import path. // 2.   ..,    ../dist --   //    ,   ! serverConfig = { mode: "development", target: "node", entry: { server: ["./index.js", "@babel/polyfill"] }, output: { path: path.resolve(__dirname, ".."), filename: "[name].js" }, module: { rules: [ { test: /\.js$/, exclude: /node_modules/, loader: "babel-loader" } ] } }; module.exports = [clientConfig, serverConfig];

To make babel work, you need to configure frontend/.babelrc :

 { "presets": ["@babel/env", "@babel/react"] }

Finally, let's make our npm run build meaningful:

 // frontend/package.json ... "scripts": { "build": "webpack", "start": "node /app/server.js", "test": "echo \"Error: no test specified\" && exit 1" }, ...

Now our client, along with a bundle of polyfills and all his dependencies, is run through the babel, compiled and put into a monolithic minified file ../dist/client.js . Add the ability to load it as a static file into our Express application, and in the default route we will begin to return our HTML:

 // frontend/index.js // ,    , //  - . import express from 'express' import template from './src/template' let app = express(); app.use('/dist', express.static('../dist')); app.get("*", (req, res) => { res.send(template("Habr demo app")); }); app.listen(process.env.APP_FRONTEND_PORT);

Success! Now, if we run docker-compose up --build frontend , we will see “Hello, world!” In a new, shiny wrapper, and if you have the React Developer Tools ( Chrome , Firefox ) extension installed, then the React component is also in developer tools:

Backend: data in MongoDB

Before moving on and breathing life into our application, you must first breathe it into the backend. It seems we were going to keep the cards marked in Markdown - it's time to do it.

While there is an ORM for MongoDB on python , I find the use of ORM a bad practice and leave the study of the appropriate solutions to you. Instead, we will make a simple class for a card and a companion DAO :

 # backend/storage/card.py import abc from typing import Iterable class Card(object): def __init__(self, id: str = None, slug: str = None, name: str = None, markdown: str = None, html: str = None): self.id = id self.slug = slug #    self.name = name self.markdown = markdown self.html = html class CardDAO(object, metaclass=abc.ABCMeta): @abc.abstractmethod def create(self, card: Card) -> Card: pass @abc.abstractmethod def update(self, card: Card) -> Card: pass @abc.abstractmethod def get_all(self) -> Iterable[Card]: pass @abc.abstractmethod def get_by_id(self, card_id: str) -> Card: pass @abc.abstractmethod def get_by_slug(self, slug: str) -> Card: pass class CardNotFound(Exception): pass

(If you are still not using type annotations in Python, be sure to take a look at these articles !)

Now we CardDAO create an implementation of the CardDAO interface that accepts a Database object from pymongo (yes, it's time to add pymongo to requirements.txt ):

 # backend/storage/card_impl.py from typing import Iterable import bson import bson.errors from pymongo.collection import Collection from pymongo.database import Database from backend.storage.card import Card, CardDAO, CardNotFound class MongoCardDAO(CardDAO): def __init__(self, mongo_database: Database): self.mongo_database = mongo_database # , slug   . self.collection.create_index("slug", unique=True) @property def collection(self) -> Collection: return self.mongo_database["cards"] @classmethod def to_bson(cls, card: Card): # MongoDB     BSON.  #       BSON- #  ,      . result = { k: v for k, v in card.__dict__.items() if v is not None } if "id" in result: result["_id"] = bson.ObjectId(result.pop("id")) return result @classmethod def from_bson(cls, document) -> Card: #   ,     #     ,     #  .    id    # ,   -   . document["id"] = str(document.pop("_id")) return Card(**document) def create(self, card: Card) -> Card: card.id = str(self.collection.insert_one(self.to_bson(card)).inserted_id) return card def update(self, card: Card) -> Card: card_id = bson.ObjectId(card.id) self.collection.update_one({"_id": card_id}, {"$set": self.to_bson(card)}) return card def get_all(self) -> Iterable[Card]: for document in self.collection.find(): yield self.from_bson(document) def get_by_id(self, card_id: str) -> Card: return self._get_by_query({"_id": bson.ObjectId(card_id)}) def get_by_slug(self, slug: str) -> Card: return self._get_by_query({"slug": slug}) def _get_by_query(self, query) -> Card: document = self.collection.find_one(query) if document is None: raise CardNotFound() return self.from_bson(document)

Time to set the Mongi configuration in the backend settings. We simply called our container mongo mongo , so MONGO_HOST = "mongo" :

 --- a/backend/dev_settings.py +++ b/backend/dev_settings.py @@ -0,0 +1,3 @@ +MONGO_HOST = "mongo" +MONGO_PORT = 27017 +MONGO_DATABASE = "core"

Now you need to create a MongoCardDAO and give it to the Flask application. Although we now have a very simple hierarchy of objects (settings → pymongo client → pymongo database → MongoCardDAO ), let's immediately create a centralized component king that does dependency injection (it will come in handy again when we do the worker and tools).

 # backend/wiring.py import os from pymongo import MongoClient from pymongo.database import Database import backend.dev_settings from backend.storage.card import CardDAO from backend.storage.card_impl import MongoCardDAO class Wiring(object): def __init__(self, env=None): if env is None: env = os.environ.get("APP_ENV", "dev") self.settings = { "dev": backend.dev_settings, # (    # ,   !) }[env] #        . #        DI,  . self.mongo_client: MongoClient = MongoClient( host=self.settings.MONGO_HOST, port=self.settings.MONGO_PORT) self.mongo_database: Database = self.mongo_client[self.settings.MONGO_DATABASE] self.card_dao: CardDAO = MongoCardDAO(self.mongo_database)

Time to add a new route to the Flask application and enjoy the view!

 # backend/server.py import os.path import flask import flask_cors from backend.storage.card import CardNotFound from backend.wiring import Wiring env = os.environ.get("APP_ENV", "dev") print(f"Starting application in {env} mode") class HabrAppDemo(flask.Flask): def __init__(self, *args, **kwargs): super().__init__(*args, **kwargs) flask_cors.CORS(self) self.wiring = Wiring(env) self.route("/api/v1/card/<card_id_or_slug>")(self.card) def card(self, card_id_or_slug): try: card = self.wiring.card_dao.get_by_slug(card_id_or_slug) except CardNotFound: try: card = self.wiring.card_dao.get_by_id(card_id_or_slug) except (CardNotFound, ValueError): return flask.abort(404) return flask.jsonify({ k: v for k, v in card.__dict__.items() if v is not None }) app = HabrAppDemo("habr-app-demo") app.config.from_object(f"backend.{env}_settings")

Restart with docker-compose up --build backend :

Oops ... oh, right. We need to add content! Let's open the tools folder and add the script into it, which adds one test card:

 # tools/add_test_content.py from backend.storage.card import Card from backend.wiring import Wiring wiring = Wiring() wiring.card_dao.create(Card( slug="helloworld", name="Hello, world!", markdown=""" This is a hello-world page. """))

The docker-compose exec backend python -m tools.add_test_content fills our monitor with content from inside the docker-compose exec backend python -m tools.add_test_content container.

Success! Now is the time to support this on the front end.

Front End: Redux

Now we want to make a route /card/:id_or_slug , which will open our React-application, load the card data from the API and somehow show it to us. And here, perhaps, the most difficult begins, because we want the server to immediately give us HTML with the contents of the card, suitable for indexing, but at the same time that the application, when navigating between cards, receives all the data in the form of JSON from the API, and the page is not overloaded. And so that all this - without copy-paste!

Let's start by adding Redux. Redux is a JavaScript library for storing state. The idea is that instead of the thousands of implicit states that your components change during user actions and other interesting events, have one centralized state, and make any change through a centralized mechanism of actions. So, if earlier we first included a download gif for navigation, then made a request through AJAX and, finally, registered the update of the necessary parts of the page in the success-callback, in the Redux paradigm we are asked to send an action “change the content to the animation gif” change the global state so that one of your components throws out the previous content and sets up the animation, then make a request, and in its success callback send another action, “change the content to loaded”. In general, we will see it now.

Let's start by installing new dependencies in our container.

 docker-compose exec frontend npm install --save \ redux \ react-redux \ redux-thunk \ redux-devtools-extension

The first is Redux itself, the second is a special library for crossing React and Redux (written by mating experts), the third is a very necessary thing, a necessity that is well-grounded in its README , and finally, the fourth is the library necessary for Redux DevTools Extension .

Let's start with the boilerplate Redux-code: creating a reducer that does nothing and initializing the state.

 // frontend/src/redux/reducers.js export default function root(state = {}, action) { return state; }

 // frontend/src/redux/configureStore.js import {createStore, applyMiddleware} from "redux"; import thunkMiddleware from "redux-thunk"; import {composeWithDevTools} from "redux-devtools-extension"; import rootReducer from "./reducers"; export default function configureStore(initialState) { return createStore( rootReducer, initialState, composeWithDevTools(applyMiddleware(thunkMiddleware)), ); }

Our client is slightly modified, mentally preparing to work with Redux:

 // frontend/src/client.js import React from 'react' import {render} from 'react-dom' import {Provider} from 'react-redux' import App from './components/app' import configureStore from './redux/configureStore' //      ... const store = configureStore(); render( // ...      , //     <Provider store={store}> <App/> </Provider>, document.querySelector('#app') );

Now we can run docker-compose up - build frontend to make sure that nothing is broken, and our primitive state has appeared in Redux DevTools:

Frontend: card page

Before making pages with SSR, you need to make pages without SSR! Let's finally use our brilliant API to access the cards and create a card page on the front end.

Time to use intelligence and design the structure of our state. There are quite a lot of materials on this topic, so I suggest that you do not abuse the intellect and will stop at a simple one. For example, like this:

 { "page": { "type": "card", //     //       type=card: "cardSlug": "...", //     "isFetching": false, //      API "cardData": {...}, //   (  ) // ... }, // ... }

Let's start the “card” component, which accepts the contents of cardData as props (this is actually the content of our mongo card):

 // frontend/src/components/card.js import React, {Component} from 'react'; class Card extends Component { componentDidMount() { document.title = this.props.name } render() { const {name, html} = this.props; return ( <div> <h1>{name}</h1> <!---,  HTML  React  - !--> <div dangerouslySetInnerHTML={{__html: html}}/> </div> ); } } export default Card;

Now let's get the component for the whole page with the card. He will be responsible for retrieving the necessary data from the API and transferring it to the Card. And we will do the data fetching React-Redux way.

To begin with, create a frontend/src/redux/actions.js and create an action that gets the contents of the card from the API, if not yet:

 export function fetchCardIfNeeded() { return (dispatch, getState) => { let state = getState().page; if (state.cardData === undefined || state.cardData.slug !== state.cardSlug) { return dispatch(fetchCard()); } }; }

The fetchCard action, which actually makes fetching, is a bit more complicated:

 function fetchCard() { return (dispatch, getState) => { //    ,    . //     , , //    . dispatch(startFetchingCard()); //    API. let url = apiPath() + "/card/" + getState().page.cardSlug; // , ,   ,  //    . , ,  //    . return fetch(url) .then(response => response.json()) .then(json => dispatch(finishFetchingCard(json))); }; // ,  redux-thunk   //     . } function startFetchingCard() { return { type: START_FETCHING_CARD }; } function finishFetchingCard(json) { return { type: FINISH_FETCHING_CARD, cardData: json }; } function apiPath() { //    .    server-side // rendering,   API     -  //         localhost, //   backend. return "http://localhost:40001/api/v1"; }

, , - ! :

 // frontend/src/redux/reducers.js import { START_FETCHING_CARD, FINISH_FETCHING_CARD } from "./actions"; export default function root(state = {}, action) { switch (action.type) { case START_FETCHING_CARD: return { ...state, page: { ...state.page, isFetching: true } }; case FINISH_FETCHING_CARD: return { ...state, page: { ...state.page, isFetching: false, cardData: action.cardData } } } return state; }

( .)

, Redux actions, CardPage :

 // frontend/src/components/cardPage.js import React, {Component} from 'react'; import {connect} from 'react-redux' import {fetchCardIfNeeded} from '../redux/actions' import Card from './card' class CardPage extends Component { componentWillMount() { //   ,  React  //   .      //   ,    " // "   ,    //  - .    -   //       HTML  // renderToString,      SSR. this.props.dispatch(fetchCardIfNeeded()) } render() { const {isFetching, cardData} = this.props; return ( <div> {isFetching && <h2>Loading...</h2>} {cardData && <Card {...cardData}/>} </div> ); } } //       ,   //  .        //  react-redux.   page    //  dispatch,   . function mapStateToProps(state) { const {page} = state; return page; } export default connect(mapStateToProps)(CardPage);

page.type App:

 // frontend/src/components/app.js import React, {Component} from 'react' import {connect} from "react-redux"; import CardPage from "./cardPage" class App extends Component { render() { const {pageType} = this.props; return ( <div> {pageType === "card" && <CardPage/>} </div> ); } } function mapStateToProps(state) { const {page} = state; const {type} = page; return { pageType: type }; } export default connect(mapStateToProps)(App);

— - page.type page.cardSlug URL .

, . - . . , !

 // frontend/src/client.js import React from 'react' import {render} from 'react-dom' import {Provider} from 'react-redux' import App from './components/app' import configureStore from './redux/configureStore' let initialState = { page: { type: "home" } }; const m = /^\/card\/([^\/]+)$/.exec(location.pathname); if (m !== null) { initialState = { page: { type: "card", cardSlug: m[1] }, } } const store = configureStore(initialState); render( <Provider store={store}> <App/> </Provider>, document.querySelector('#app') );

docker-compose up --build frontend , helloworld …

, … ? , Markdown!

: RQ

Markdown HTML — «» , , , — .

; Redis RQ (Redis Queue), pickle .

, !

 --- a/requirements.txt +++ b/requirements.txt @@ -3,3 +3,5 @@ flask-cors gevent gunicorn pymongo +redis +rq --- a/backend/dev_settings.py +++ b/backend/dev_settings.py @@ -1,3 +1,7 @@ MONGO_HOST = "mongo" MONGO_PORT = 27017 MONGO_DATABASE = "core" +REDIS_HOST = "redis" +REDIS_PORT = 6379 +REDIS_DB = 0 +TASK_QUEUE_NAME = "tasks" --- a/backend/wiring.py +++ b/backend/wiring.py @@ -2,6 +2,8 @@ import os from pymongo import MongoClient from pymongo.database import Database +import redis +import rq import backend.dev_settings from backend.storage.card import CardDAO @@ -21,3 +23,11 @@ class Wiring(object): port=self.settings.MONGO_PORT) self.mongo_database: Database = self.mongo_client[self.settings.MONGO_DATABASE] self.card_dao: CardDAO = MongoCardDAO(self.mongo_database) + + self.redis: redis.Redis = redis.StrictRedis( + host=self.settings.REDIS_HOST, + port=self.settings.REDIS_PORT, + db=self.settings.REDIS_DB) + self.task_queue: rq.Queue = rq.Queue( + name=self.settings.TASK_QUEUE_NAME, + connection=self.redis)

 # worker/__main__.py import argparse import uuid import rq import backend.wiring parser = argparse.ArgumentParser(description="Run worker.") #   ,      #  .  ,       rq. parser.add_argument( "--burst", action="store_const", const=True, default=False, help="enable burst mode") args = parser.parse_args() #       Redis. wiring = backend.wiring.Wiring() with rq.Connection(wiring.redis): w = rq.Worker( queues=[wiring.settings.TASK_QUEUE_NAME], #         # ,    . name=uuid.uuid4().hex) w.work(burst=args.burst)

mistune :

 # backend/tasks/parse.py import mistune from backend.storage.card import CardDAO def parse_card_markup(card_dao: CardDAO, card_id: str): card = card_dao.get_by_id(card_id) card.html = _parse_markdown(card.markdown) card_dao.update(card) _parse_markdown = mistune.Markdown(escape=True, hard_wrap=False)

: CardDAO , . , , pickle — , RQ. - Wiring … :

 --- a/worker/__main__.py +++ b/worker/__main__.py @@ -2,6 +2,7 @@ import argparse import uuid import rq +from rq.job import Job import backend.wiring @@ -16,8 +17,23 @@ args = parser.parse_args() wiring = backend.wiring.Wiring() + +class JobWithWiring(Job): + + @property + def kwargs(self): + result = dict(super().kwargs) + result["wiring"] = backend.wiring.Wiring() + return result + + @kwargs.setter + def kwargs(self, value): + super().kwargs = value + + with rq.Connection(wiring.redis): w = rq.Worker( queues=[wiring.settings.TASK_QUEUE_NAME], - name=uuid.uuid4().hex) + name=uuid.uuid4().hex, + job_class=JobWithWiring) w.work(burst=args.burst)

, kwargs- . ( , , , RQ .) — , — , :

 # backend/tasks/task.py import functools from typing import Callable from backend.wiring import Wiring def task(func: Callable): #    : varnames = func.__code__.co_varnames @functools.wraps(func) def result(*args, **kwargs): #  .  .pop(),     # ,        . wiring: Wiring = kwargs.pop("wiring") wired_objects_by_name = wiring.__dict__ for arg_name in varnames: if arg_name in wired_objects_by_name: kwargs[arg_name] = wired_objects_by_name[arg_name] #          #   ,  -   . return func(*args, **kwargs) return result

 import mistune from backend.storage.card import CardDAO from backend.tasks.task import task @task def parse_card_markup(card_dao: CardDAO, card_id: str): card = card_dao.get_by_id(card_id) card.html = _parse_markdown(card.markdown) card_dao.update(card) _parse_markdown = mistune.Markdown(escape=True, hard_wrap=False)

? , , :

 $ docker-compose up worker ... Creating habr-app-demo_worker_1 ... done Attaching to habr-app-demo_worker_1 worker_1 | 17:21:03 RQ worker 'rq:worker:49a25686acc34cdfa322feb88a780f00' started, version 0.13.0 worker_1 | 17:21:03 *** Listening on tasks... worker_1 | 17:21:03 Cleaning registries for queue: tasks

… ! , !

, , : ) , ( ); ) .

 # tools/add_test_content.py from backend.storage.card import Card, CardNotFound from backend.tasks.parse import parse_card_markup from backend.wiring import Wiring wiring = Wiring() try: card = wiring.card_dao.get_by_slug("helloworld") except CardNotFound: card = wiring.card_dao.create(Card( slug="helloworld", name="Hello, world!", markdown=""" This is a hello-world page. """)) # ,   card_dao.get_or_create,  #      ! wiring.task_queue.enqueue_call( parse_card_markup, kwargs={"card_id": card.id})

backend, worker. , . docker-compose exec worker python -m tools.add_test_content — - !

 worker_1 | 17:34:26 tasks: backend.tasks.parse.parse_card_markup(card_id='5c715dd1e201ce000c6a89fa') (613b53b1-726b-47a4-9c7b-97cad26da1a5) worker_1 | 17:34:27 tasks: Job OK (613b53b1-726b-47a4-9c7b-97cad26da1a5) worker_1 | 17:34:27 Result is kept for 500 seconds

, :

:

, SSR, React - single page application single page. , ( , ! , !) , , .

 # tools/add_test_content.py def create_or_update(card): try: card.id = wiring.card_dao.get_by_slug(card.slug).id card = wiring.card_dao.update(card) except CardNotFound: card = wiring.card_dao.create(card) wiring.task_queue.enqueue_call( parse_card_markup, kwargs={"card_id": card.id}) create_or_update(Card( slug="helloworld", name="Hello, world!", markdown=""" This is a hello-world page. It can't really compete with the [demo page](demo). """)) create_or_update(Card( slug="demo", name="Demo Card!", markdown=""" Hi there, habrovchanin. You've probably got here from the awkward ["Hello, world" card](helloworld). Well, **good news**! Finally you are looking at a **really cool card**! """ ))

, . !

. HTML , React, React- .

 // frontend/src/components/card.js class Card extends Component { componentDidMount() { document.title = this.props.name } navigate(event) { //       .  //      ,    . if (event.target.tagName === 'A' && event.target.hostname === window.location.hostname) { //     event.preventDefault(); //      this.props.dispatch(navigate(event.target)); } } render() { const {name, html} = this.props; return ( <div> <h1>{name}</h1> <div dangerouslySetInnerHTML={{__html: html}} onClick={event => this.navigate(event)} /> </div> ); } }

CardPage , (!) :

 export function navigate(link) { return { type: NAVIGATE, path: link.pathname } }

 // frontend/src/redux/reducers.js import { START_FETCHING_CARD, FINISH_FETCHING_CARD, NAVIGATE } from "./actions"; function navigate(state, path) { //     react-router,    ! // (       SSR.) let m = /^\/card\/([^/]+)$/.exec(path); if (m !== null) { return { ...state, page: { type: "card", cardSlug: m[1], isFetching: true } }; } return state } export default function root(state = {}, action) { switch (action.type) { case START_FETCHING_CARD: return { ...state, page: { ...state.page, isFetching: true } }; case FINISH_FETCHING_CARD: return { ...state, page: { ...state.page, isFetching: false, cardData: action.cardData } }; case NAVIGATE: return navigate(state, action.path) } return state; }

, CardPage componentDidUpdate , componentWillMount . CardPage (, cardSlug ) ( componentWillMount ).

, docker-compose up --build frontend !

, URL — Hello, world- demo-. , - . history, !

, — navigate history.pushState .

 export function navigate(link) { history.pushState(null, "", link.href); return { type: NAVIGATE, path: link.pathname } }

URL . , «» !

, popstate window . , , ( dispatch(navigate(...)) ), navigate « pushState » ( !). , «» , pushState . , ! App:

 // frontend/src/components/app.js class App extends Component { componentDidMount() { //     --   //      "". history.replaceState({ pathname: location.pathname, href: location.href }, ""); //     . window.addEventListener("popstate", event => this.navigate(event)); } navigate(event) { //    "" ,   //        ,    //   (or is it a good thing?..) if (event.state && event.state.pathname) { event.preventDefault(); event.stopPropagation(); //      "  pushState". this.props.dispatch(navigate(event.state, true)); } } render() { // ... } }

— navigate:

 // frontend/src/redux/actions.js export function navigate(link, dontPushState) { if (!dontPushState) { history.pushState({ pathname: link.pathname, href: link.href }, "", link.href); } return { type: NAVIGATE, path: link.pathname } }

.

: navigate , , ? navigate location:

 --- a/frontend/src/client.js +++ b/frontend/src/client.js @@ -3,23 +3,16 @@ import {render} from 'react-dom' import {Provider} from 'react-redux' import App from './components/app' import configureStore from './redux/configureStore' +import {navigate} from "./redux/actions"; let initialState = { page: { type: "home" } }; -const m = /^\/card\/([^\/]+)$/.exec(location.pathname); -if (m !== null) { - initialState = { - page: { - type: "card", - cardSlug: m[1] - }, - } -} const store = configureStore(initialState); +store.dispatch(navigate(location));

: server-side rendering

( ) — SEO-. , React-, React, .

. : HTML- HTML, React- App . HTML ( JS, -). : <script> , - (, window ) , HTML. ( hydrate HTML, DOM tree ).

, HTML .

 // frontend/src/server.js import "@babel/polyfill" import React from 'react' import {renderToString} from 'react-dom/server' import {Provider} from 'react-redux' import App from './components/app' import {navigate} from "./redux/actions"; import configureStore from "./redux/configureStore"; export default function render(initialState, url) { //  store,    . const store = configureStore(initialState); store.dispatch(navigate(url)); let app = ( <Provider store={store}> <App/> </Provider> ); // ,        ! // ,         ? let content = renderToString(app); let preloadedState = store.getState(); return {content, preloadedState}; };

, :

 // frontend/src/template.js function template(title, initialState, content) { let page = ` <!DOCTYPE html> <html lang="en"> <head> <meta charset="utf-8"> <title>${title}</title> </head> <body> <div id="app">${content}</div> <script> window.__STATE__ = ${JSON.stringify(initialState)} </script> <script src="/dist/client.js"></script> </body> </html> `; return page; } module.exports = template;

Express-:

 // frontend/index.js app.get("*", (req, res) => { const initialState = { page: { type: "home" } }; const {content, preloadedState} = render(initialState, {pathname: req.url}); res.send(template("Habr demo app", preloadedState, content)); });

— :

 // frontend/src/client.js import React from 'react' import {hydrate} from 'react-dom' import {Provider} from 'react-redux' import App from './components/app' import configureStore from './redux/configureStore' import {navigate} from "./redux/actions"; //         ! const store = configureStore(window.__STATE__); // render   hydrate. hydrate    // DOM tree,       . hydrate( <Provider store={store}> <App/> </Provider>, document.querySelector('#app') );

«history is not defined». ( ) - utility.js .

 // frontend/src/utility.js export function isServerSide() { //   ,      process, //     -   . return process.env.APP_ENV !== undefined; }

- , ( ). React- , .

! , , …

LOADING? , Google - — LOADING?!

, , . , , React- . , .

. — , , , ( ). . , .

( - !) . , - , , (, , fetch) - . , , .

.

 // frontend/src/redux/actions.js function addPromise(promise) { return { type: ADD_PROMISE, promise: promise }; } function removePromise(promise) { return { type: REMOVE_PROMISE, promise: promise, }; }

, , — .then() .

:

 // frontend/src/redux/reducers.js export default function root(state = {}, action) { switch (action.type) { case ADD_PROMISE: return { ...state, promises: [...state.promises, action.promise] }; case REMOVE_PROMISE: return { ...state, promises: state.promises.filter(p => p !== action.promise) }; ...

fetchCard :

 // frontend/src/redux/actions.js function fetchCard() { return (dispatch, getState) => { dispatch(startFetchingCard()); let url = apiPath() + "/card/" + getState().page.cardSlug; let promise = fetch(url) .then(response => response.json()) .then(json => { dispatch(finishFetchingCard(json)); // " ,  " dispatch(removePromise(promise)); }); // "  ,  " return dispatch(addPromise(promise)); }; }

initialState ! render :

 // frontend/src/server.js function hasPromises(state) { return state.promises.length > 0 } export default async function render(initialState, url) { const store = configureStore(initialState); store.dispatch(navigate(url)); let app = ( <Provider store={store}> <App/> </Provider> ); //  renderToString     // (  ). CardPage     . renderToString(app); // ,   !    - //    (  // , ),     //    . let preloadedState = store.getState(); while (hasPromises(preloadedState)) { await preloadedState.promises[0]; preloadedState = store.getState() } //  renderToString.    HTML. let content = renderToString(app); return {content, preloadedState}; };

render :

 // frontend/index.js app.get("*", (req, res) => { const initialState = { page: { type: "home" }, promises: [] }; render(initialState, {pathname: req.url}).then(result => { const {content, preloadedState} = result; const response = template("Habr demo app", preloadedState, content); res.send(response); }, (reason) => { console.log(reason); res.status(500).send("Server side rendering failed!"); }); });

Et voilà!

Conclusion

, . ! Github , , Docker, .

, ! - , :

, , .
, CI, CD.
.
-.

Thanks for attention!

Source: https://habr.com/ru/post/444446/

All Articles