As we in the django-project js collect + couple of tricks in Gulp
Hello!
This is not a guide, I am sharing the experience of how we in a large Django project from the ugly jQuery scripts have come to build and minify complex frontend applications on AngularJS using gulp and browserify.
There is a large multi-year Django project with a bunch of legacy code, a billion dependencies and a team without an official frontend developer. Somehow it happened that I gradually became more and more engaged in js, was drawn into the frontend and now it takes more than half of my working time.
In the history of the frontend of our project (and, accordingly, of my development as a js-developer), we can distinguish three major stages:
')
It was that period when, having mastered a couple of jQuery methods, having mastered selectors and having learned with animation to show / hide elements on a page, you consider yourself to be a complete frontend developer. All newbies went through this and everyone knows what it looks like: each piece of functionality is a separate file, there are a dozen script connections on large pages, no system — every script is for itself, with all the consequences, as they say. There was no specific place for storage of vendor libraries, each next developer threw a new lib, wherever he wanted. In addition to everything that I wrote myself, there was still a huge pile of old scripts written before me.
There is a need to write more complex interfaces, wizards and other things for the admin. By this time, it was understood that jQuery is not a panacea, and that you need to somehow organize your code. Knockout and RequireJS came to the rescue. RequireJS allowed to break the code into modules, specify dependencies, reuse modules on different pages, build a normal file structure for each application. At least some system appeared: a config file for RequireJS was created with paths to all libraries, it was used on all knockout pages, all vendor libraries settled in one place. Only one problem remains: although now only one script was connected in the template, the rest of the dependencies were already stretched by RequireJS itself, and often the module files were so small that the ping to the server was longer than the download time — meaningless brakes. I often pointed out this problem and offered different solutions, but the answer of the authorities was always the same: “This is admin panel. Here it is not critical. We will not waste time on it. ”
Finally, the hands reached the Customer Area: tricky interfaces, plus the UX requirements. It was impossible to ignore the problem of loading scripts. At that time, I had already gained experience in developing at NodeJS using frontend squeak assembly. Now it was impossible to look without tears at the config file for RequireJS and at the systematized trash bin of vendor libraries.
A little bit about how the project works. Each django application has its own static folder. During development, the Djangov dev server looks for scripts connected to the pages in these folders. During the production deployment, collectstatic is made, which collects all the files in one folder so that the web server can deliver them. Nothing unusual.
I wanted to get the following:
There was a question - from which side would it be screwed to the project in order not to break the usual workflow and not to frighten the authorities with new dependencies in the form of NodeJS (read, like “new language in the team of pythonists”) and its utilities?
It was decided that all manipulations with js-code (build, minification) will be done before the commit, the finished package will be copied into the folder with the statics of the corresponding django application and connected from there. Thus, the deployment process will remain unchanged, plus - no new dependencies in production.
So, the first thing we need:
They should be placed globally in the system, because we need their console utilities. Fortunately, we are developing in Vagrant, so I just added the appropriate chef-recipes to its config. After installation, you need to run npm init and bower init in the project root and set the minimum required parameters. At the output we get package.json and bower.json . The final step in preparing the environment will be to make node_modules / and bower_components / in .gitignore, since the entire assembly will be made directly during development.
When using bower and npm to install packages, do not forget to use the argument --save-dev so that the package information is saved in bower.json and package.json respectively, and other developers can easily bring up the environment by simply running npm install and bower install in project root.
I decided to store the source code for js applications in a separate directory in the project root. At first, I wanted to analyze the directory structure on the fly during the build, but I thought that for each smart analyzer, sooner or later there would be a task that would have to be supported with crutches, so I decided to just create a config in which I would describe all these applications. So at the root of the project the config-spa.js file appeared:
Thus, it is easy to understand which application the scripts belong to. Common modules are placed in directories named common.
It remains the case for small - a description of the tasks for the assembly. In general, it turned out the standard gulpfile, but there are a couple of tricks that can be useful to someone.
Since we have several applications, it was necessary to somehow indicate which particular application you need to build, or indicate that you need to rebuild them all.
Another argument is the flag that cancels the minification of the application so that you can see the normal stack traces when debugging.
What is the trick? First, the fact that I parsed the arguments as a separate task so that it can be specified in the dependencies of other tasks, and, second, the arguments that were parsed once are stored in a global variable, so that when you call some tasks from others, they will work with the same settings.
Task of monitoring changes in js-application files:
The development process looks like this: run the django dev server , launch the gulp watch and write / debug the frontend application. Thus, the development process itself ensures that the actual assembled application will be immediately in the statics folder with any changes, and we no longer need additional steps during the deployment. But the problem is that the development is usually done with the --dev parameter (without minification), and here, a couple of times in the process of parking, the noncommissioned package under the size of 2 megabytes in the production, I thought that I should think up some kind of reminder, and better is automation.
So the following code appeared in the watch watch:
The result is: start gulp watch --app appname --dev , debug the application, press CTRL + C to stop watch and gulp immediately collects the minified version of the package. Easy commit and enjoy the result of their work in production.
We got a system for building js-applications without changes in the process of deployment and without new dependencies on the production. It allowed us to divide the code into modules and receive one compact file at the output. Here you can add js-linter, tests and much more.
In the same way, you can easily translate, for example, Styles to some Stylus and also minify them, but due to some human reasons, we haven’t started to do this yet.
Everyone who read, thank you for your attention.
Gulpfile completely with sample application .
This is not a guide, I am sharing the experience of how we in a large Django project from the ugly jQuery scripts have come to build and minify complex frontend applications on AngularJS using gulp and browserify.
Prehistory
There is a large multi-year Django project with a bunch of legacy code, a billion dependencies and a team without an official frontend developer. Somehow it happened that I gradually became more and more engaged in js, was drawn into the frontend and now it takes more than half of my working time.
In the history of the frontend of our project (and, accordingly, of my development as a js-developer), we can distinguish three major stages:
')
jQuery is our everything
It was that period when, having mastered a couple of jQuery methods, having mastered selectors and having learned with animation to show / hide elements on a page, you consider yourself to be a complete frontend developer. All newbies went through this and everyone knows what it looks like: each piece of functionality is a separate file, there are a dozen script connections on large pages, no system — every script is for itself, with all the consequences, as they say. There was no specific place for storage of vendor libraries, each next developer threw a new lib, wherever he wanted. In addition to everything that I wrote myself, there was still a huge pile of old scripts written before me.
Knockout + RequireJS
There is a need to write more complex interfaces, wizards and other things for the admin. By this time, it was understood that jQuery is not a panacea, and that you need to somehow organize your code. Knockout and RequireJS came to the rescue. RequireJS allowed to break the code into modules, specify dependencies, reuse modules on different pages, build a normal file structure for each application. At least some system appeared: a config file for RequireJS was created with paths to all libraries, it was used on all knockout pages, all vendor libraries settled in one place. Only one problem remains: although now only one script was connected in the template, the rest of the dependencies were already stretched by RequireJS itself, and often the module files were so small that the ping to the server was longer than the download time — meaningless brakes. I often pointed out this problem and offered different solutions, but the answer of the authorities was always the same: “This is admin panel. Here it is not critical. We will not waste time on it. ”
AngularJS + Gulp + Browserify + Uglify
Finally, the hands reached the Customer Area: tricky interfaces, plus the UX requirements. It was impossible to ignore the problem of loading scripts. At that time, I had already gained experience in developing at NodeJS using frontend squeak assembly. Now it was impossible to look without tears at the config file for RequireJS and at the systematized trash bin of vendor libraries.
A little bit about how the project works. Each django application has its own static folder. During development, the Djangov dev server looks for scripts connected to the pages in these folders. During the production deployment, collectstatic is made, which collects all the files in one folder so that the web server can deliver them. Nothing unusual.
I wanted to get the following:
- normal frontend package manager;
- normal design code in the form of reusable-modules;
- build js-application in one file and its minification.
There was a question - from which side would it be screwed to the project in order not to break the usual workflow and not to frighten the authorities with new dependencies in the form of NodeJS (read, like “new language in the team of pythonists”) and its utilities?
It was decided that all manipulations with js-code (build, minification) will be done before the commit, the finished package will be copied into the folder with the statics of the corresponding django application and connected from there. Thus, the deployment process will remain unchanged, plus - no new dependencies in production.
Get on the right path
Environment
So, the first thing we need:
- nodejs ;
- gulp - to describe the assembly tasks;
- npm - to install the packages required for the assembly;
- bower - for installing packages required in the frontend.
They should be placed globally in the system, because we need their console utilities. Fortunately, we are developing in Vagrant, so I just added the appropriate chef-recipes to its config. After installation, you need to run npm init and bower init in the project root and set the minimum required parameters. At the output we get package.json and bower.json . The final step in preparing the environment will be to make node_modules / and bower_components / in .gitignore, since the entire assembly will be made directly during development.
When using bower and npm to install packages, do not forget to use the argument --save-dev so that the package information is saved in bower.json and package.json respectively, and other developers can easily bring up the environment by simply running npm install and bower install in project root.
Directory structure
I decided to store the source code for js applications in a separate directory in the project root. At first, I wanted to analyze the directory structure on the fly during the build, but I thought that for each smart analyzer, sooner or later there would be a task that would have to be supported with crutches, so I decided to just create a config in which I would describe all these applications. So at the root of the project the config-spa.js file appeared:
module.exports = { apps: { 'appname': { // js- main: 'app.js', // path: './spa/dj-app/appname/', // bundle: 'appname.min.js', // dest: './dj-app/static/dj-app/js/', // django- watch: ['./spa/dj-app/appname/**/*.js'] // glob- ( ) }, ... } } - spa / - directory where all js applications will be located
- dj-app - the name of the django application in which the assembled package will be used
Thus, it is easy to understand which application the scripts belong to. Common modules are placed in directories named common.
gulpfile.js
It remains the case for small - a description of the tasks for the assembly. In general, it turned out the standard gulpfile, but there are a couple of tricks that can be useful to someone.
Parsing command line arguments and first trick
Since we have several applications, it was necessary to somehow indicate which particular application you need to build, or indicate that you need to rebuild them all.
Another argument is the flag that cancels the minification of the application so that you can see the normal stack traces when debugging.
What is the trick? First, the fact that I parsed the arguments as a separate task so that it can be specified in the dependencies of other tasks, and, second, the arguments that were parsed once are stored in a global variable, so that when you call some tasks from others, they will work with the same settings.
// , . var config = require('./config-spa'), argv = {parsed: false} gulp.task('parseArgs', function() { // prevent multiple parsing when watching if (argv.parsed) return true // check the process arguments var options = minimist(process.argv) if (_.size(options) === 1) { printArgumentsErrorAndExit() } // , var apps = [] if (options.app && config.apps[options.app]) { apps.push(options.app) } else if (options.all) { apps = _.keys(config.apps) } if (!apps.length) printArgumentsErrorAndExit() argv.apps = apps // dev - , if (options.dev) argv.dev = true argv.parsed = true }) function printArgumentsErrorAndExit() { gutil.log(gutil.colors.red('You must specify the app or'), gutil.colors.yellow('--all')) gutil.log(gutil.colors.red('Available apps:')) _.each(config.apps, function(item, i) { gutil.log(gutil.colors.yellow(' --app ' + i)) }) // break the task on error process.exit() } Build the application
function bundle() { return through.obj(function(file, enc, cb) { var b = browserify({entries: file.path}) file.contents = b.bundle() this.push(file) cb() }) } gulp.task('build', ['parseArgs'], function(cb) { var prefix = gutil.colors.yellow(' ->') async.each(argv.apps, function(app, cb) { gutil.log(prefix, 'Building', gutil.colors.cyan(app), '...') var conf = config.apps[app] if (!conf) return cb(new Error('No conf for app ' + app)) gulp.src(path.join(conf.path, conf.main)) .pipe(bundle()) .pipe(gulpif(!argv.dev, streamify(uglify()))) .pipe(rename(conf.bundle)) .pipe(gulp.dest(conf.dest)) .on('end', function() { cb() }) }, function(err) { cb(err) } ) }) - function bundle () {...} is a self-written wrapper for browserify. Who uses it, he has long known that browserify can work with streams, so the gulp-browserify package has not been used for a long time;
- [parseArgs] - we specify in task dependencies for parsing command line arguments. Thus, we are sure that the argv variable already contains valid settings;
- async.each, cb () - enumeration of the applications specified in the arguments. Why is there asink and zamorochki with kollbekami? The fact is that the build procedure itself (gulp.src (). Pipe () ...) is asynchronous, and the task can be completed before the entire chain is executed, and this, in turn, leads to Tasks dependent on it begin their execution earlier. There are three possible solutions - a callback task, a return from the thread task - return gulp.src () ... and a promise return. We cannot return the flow here, because there are several of them, so I stopped at the callback;
- .pipe (gulp.dest (conf.dest)) - the assembled package is copied to the folder with statics specified in the js-application config, so that with the diststroy collectstatic will do its job without additional gestures.
Recompile with changes in files
Task of monitoring changes in js-application files:
gulp.task('watch', ['build'], function() { var targets = [] _.each(argv.apps, function(app) { var conf = config.apps[app] if (!conf) return if (conf.watch) { if (_.isArray(conf.watch)) { targets = _.union(targets, conf.watch) } else { targets.push(conf.watch) } } }) targets = _.uniq(targets) // start watching files gulp.watch(targets, ['build']) }) - ['build'] - we specify task builds in dependencies. Firstly, it will rebuild the application before starting the observation, secondly, we know that before taksk build we are parsing command line arguments;
- _.each (argv.apps, ...) - we iterate through the applications specified in the arguments, we look at their settings in the config, we collect the target to monitor the changes;
- gulp.watch (targets, ['build']) - we launch observation, the task of build is carried out at changes. There is one drawback - if we launch watch for some applications, then with any changes they will be re-assembled all, but in fact it is unlikely to ever (ever) need to monitor several applications at the same time, so we don’t bother.
Reassemble with minifikatsiya after completion of the watch - the second trick
The development process looks like this: run the django dev server , launch the gulp watch and write / debug the frontend application. Thus, the development process itself ensures that the actual assembled application will be immediately in the statics folder with any changes, and we no longer need additional steps during the deployment. But the problem is that the development is usually done with the --dev parameter (without minification), and here, a couple of times in the process of parking, the noncommissioned package under the size of 2 megabytes in the production, I thought that I should think up some kind of reminder, and better is automation.
So the following code appeared in the watch watch:
// handle Ctrl+C and build a minified version on exit process.on('SIGINT', function() { if (!argv.dev) process.exit() argv.dev = false console.log() gutil.log(gutil.colors.yellow('Building a minified version...')) gulp.stop() gulp.start('build', function() { process.exit() }) }) - we catch CTRL + C;
- if watch was launched with minifikatsii, then we simply complete the process;
- argv.dev = false - cancel the prohibition of minification so that the next build will assemble us a package for production;
- gulp.stop () - completes all current tasks;
- gulp.start ('build', function () {...}) - call the build task and exit after it is completed. Here it is very important that the callback after the build was correctly called in the build task, which I mentioned earlier, otherwise the task will complete before the package is copied to the folder with statics, and the process will exit. The start method is not in the gulp documentation, because in fact it is not his method: it was inherited from Orchestrator.
The result is: start gulp watch --app appname --dev , debug the application, press CTRL + C to stop watch and gulp immediately collects the minified version of the package. Easy commit and enjoy the result of their work in production.
Total
We got a system for building js-applications without changes in the process of deployment and without new dependencies on the production. It allowed us to divide the code into modules and receive one compact file at the output. Here you can add js-linter, tests and much more.
In the same way, you can easily translate, for example, Styles to some Stylus and also minify them, but due to some human reasons, we haven’t started to do this yet.
Everyone who read, thank you for your attention.
Gulpfile completely with sample application .
Source: https://habr.com/ru/post/244641/
All Articles