FlyElephant as a tool for computing in C ++, R, Python or Octave

Greetings to all!

Today I will talk about the capabilities of the FlyElephant platform for scientists and engineers who, in their work, carry out various calculations in C ++, R, Python or Octave. These can be scientific calculations, data analysis, modeling, or other tasks. On January 22, I will hold a webinar “Introduction to FlyElephant” , where I will talk in detail about the FlyElephant platform, and today I will introduce you to it in general terms and show you the process of making calculations.

FlyElephant is a platform that provides scientists with a ready-made computing infrastructure for performing calculations, automates routine tasks and allows you to focus on basic research issues.

As a computing resource, we still use the Azure cloud. But our team continues to work on building its own computing cluster, connecting the Amazon cloud, a number of supercomputers and GRID systems, which will significantly expand our computing power. I will answer one frequently asked question: “And what are you better than clouds?” . Of course, each user can take a cloud, independently deploy the infrastructure, configure it, maintain the system, and change it if necessary. In the use case of FlyElephant, we will take care of all these time-consuming aspects! You get a ready-made infrastructure of specified parameters with the necessary software. In addition, in the future, you will not only be able to perform calculations in the cloud, but use for them supercomputers and other specialized computing resources that are not always publicly available. Also, if you have your own computing power, cluster or cloud, you will be able to integrate with the platform and, if necessary, expand your capacity or, on the contrary, lease those resources that are not used.
')
The FlyElephant platform is not only a tool that helps you perform calculations, but also a community. One of the priorities of our team for this year is to build a strong community around FlyElephant. We want to unite in one place specialists from different areas who need to use large computing powers to solve problems, help them to exchange knowledge and help each other with problem solving, and also to unite to collaborate on projects. While the work on this functionality is in full swing, I invite everyone to our slack-chat .

What languages ​​and tools are already available?

Users can run computational tasks written using C ++ (with OpenMP support), R, Python (v2 & v3), Octave and Scilab on servers with 1 to 32 cores and RAM up to 448 GB . In addition, members of the beta program now help us debug the following tools: Java (v7 & v8), Julia, OpenFOAM, GROMACS, Blender and the Public API . Next in line are tools for visualizing the results of calculations, as well as databases for collecting and storing large amounts of information. We are actively working on expanding our catalog, so if you need any tools, let us know and we will try to quickly integrate them into the platform.

How to run the task?

On the example of programs written using C ++, R, Python or Octave, we now consider the whole process of working with FlyElephant. To get started, we need three things.
First - you need to register.
If you have not already done so, then register here . After registering on your account will be $ 10; using the promotional code 127575063277 and by entering it in your Personal Account, you can replenish your account with another $ 100.
Second, you need a repository with the code of your program.
We have prepared several test repositories that can be taken as a basis:

• github.com/FlyElephantTeam/gcc-example
• github.com/FlyElephantTeam/R-example
• github.com/FlyElephantTeam/python-example

The repository should contain files with the code of your program, as well as two required files, build.sh and run.sh. In the first one, you need to register the commands that must be executed on the server before starting the calculation (it could be some kind of environment pre-configuration, data loading or something else). The second file contains the commands for launching the computational program itself.
Third, download the data.
If for your calculation you need to work with some data that is not in the repository or in an external source, then you need to download it in the Storage section.

When everything is ready, you can run the calculation. To do this, you need to log in and perform the following steps:

• Go to the “Calculations” section and select the language or tool we need.
• Going to it, click the “Run Task” button.
• A form appears in which you must fill in the fields as follows:
  
  • Task name - we give the task a name.
  • Select the data file - if necessary, specify the file that must be uploaded to the server to complete the task. You can complete the task without downloading an additional file; select the file that was uploaded to the "Storage"; select a file that is the result of another task. The file will be uploaded to the server in the root directory of your repository.
  • The name of the result file - specify the name of the file in which the program will save the result. This file will be uploaded to the “Storage” and is available in it, as well as in the details of the task.
  • Repository - specify the address of the repository in which your program is located. Now only work with public repositories is available, but in the next update it will be possible to work with closed ones.
  • Server parameters - select the configuration of the server on which the program will be launched for execution. It is possible to run a task on a server with the number of cores from 1 to 32 and RAM up to 448 GB. In the upcoming releases, there will also be updates related to the possibility of a more flexible configurable computing infrastructure and the ability to run tasks not only in the cloud, but also on the HPC cluster or in the GRID.
• Click the “Run Task” button. The task starts, is automatically executed and becomes available in the “Tasks” section. Here you can see the details of the implementation, the result and logs.

Thus, completing a few steps, you get the finished result.

We continue to work on improving the scheme of work and in the near future we will expand the list of supported tools and computing resources. And to make this list replenished with the necessary tools for you, we invite you to take part in the survey .

Our team is open for cooperation. We will be glad to cooperate with the owners of computing resources and manufacturers of specialized software. We actively support scientists and are ready to provide the necessary amount of resources for conducting your open research free of charge. If you are interested in using FlyElephant, we will be happy to conduct a demo presentation of the platform.

Join our pages in the social. networks ( Facebook , Twitter , LinkedIn ) and follow the latest news of the FlyElephant team.

Any comments, suggestions and feedback are very important to us. We will answer them in the comments to this post or by mail: support@flyelephant.net.