Flexiant Cloud Orchestrator: what it is eaten with

To provide the IaaS (Virtual Data Center) service, we at Rusonyx use the commercial orchestrator Flexiant Cloud Orchestrator (FCO). This solution has a rather unique architecture that distinguishes it from the well-known general public, Openstack and CloudStack.

As compute nodes, KVM, VmWare, Xen, Virtuozzo6 / 7, as well as containers from the same Virtuozzo are supported as hypervisors. The supported storage is local, NFS, Ceph and Virtuozzo Storage.
')
FCO supports creating and managing multiple clusters from a single interface. That is, you can manage a Virtuozzo cluster and a KVM + Ceph cluster by switching between them by a mouse click.

In essence, FCO is a comprehensive solution for cloud providers, which, in addition to orchestration, includes billing, with all settings, payment plugins, invoices, notifications, resellers, tariffs, and so on. However, the billing part is not able to cover all the Russian nuances, so we refused to use it in favor of another solution.

We are very pleased with the flexible system of distribution of rights to all cloud resources: images, disks, products, servers, firewalls - all this can be “fumbled” and granted rights between users, and even between users of different clients. Each client can create several independent data centers in his cloud and manage them from a single control panel.



Architecturally, the FCO consists of several parts, each of which has its own independent code, and some of its own database.

Skyline - admin and user interface
Jade - business logic, billing, task management
Tigerlily - service coordinator, manages and coordinates the exchange of information between business logic and clusters.
XVPManager - management of cluster elements: nodes, storage, network and virtual machines.
XVPAgent - agent installed on nodes for interaction with XVPManager



A detailed story about the architecture of each component, we plan to fit into a series of articles, if, of course, the topic is of interest.

The main advantage of the FCO stems from its “boxiness”. At your service simplicity and minimalism. For a managing node, one virtual machine is allocated on Ubuntu, into which all necessary packages are installed. All settings are made to the configuration files in the form of variable-value:

# cat /etc/extility/config/vars … export LIMIT_MAX_LIST_ADMIN_DEFAULT="30000" export LIMIT_MAX_LIST_USER_DEFAULT="200" export LOGDIR="/var/log/extility" export LOG_FILE="misc.log" export LOG_FILE_LOG4JHOSTBILLMODULE="hostbillmodule.log" export LOG_FILE_LOG4JJADE="jade.log" export LOG_FILE_LOG4JTL="tigerlily.log" export LOG_FILE_LOG4JXVP="xvpmanager.log" export LOG_FILE_VARS="misc.log" … 

The entire configuration is corrected initially in the templates, then the generator starts.
# build-config which will generate the vars file and instruct the services to re-read the config. The user interface is nice and can be easily branded.



As you can see, the interface consists of widgets that are accessible to the user. He can easily add / remove widgets from the page, thereby forming the necessary dashboard.

Despite its closeness, FCO is a very customizable system. It has a huge number of settings and input points for changing workflow:

1. Custom plugins are supported, for example, you can write your own billing method or your own external resource to provide the user with
2. Custom triggers for certain events are supported, for example, adding the first virtual machine to the client when it is created
3. Custom widgets in the interface are supported, for example, to embed youtube videos directly into the user interface.

All customization is written in FDL, which is based on Lua. If you know Lua, there will be no problem with FDL.

Here is an example of one of the simplest triggers that we use. This trigger does not allow users to share their own images with other clients. We do this so that one user cannot create a malicious image for other users.

 function register() return {"pre_user_api_publish"} end function pre_user_api_publish(p) if(p==nil) then return{ ref = "cancelPublishImage", name = "Cancel publishing", description = "Cancel all user's images publishing", triggerType = "PRE_USER_API_CALL", triggerOptions = {"publishResource", "publishImage"}, api = "TRIGGER", version = 1, } end -- Turn publishing off return {exitState = "CANCEL"} end 

The register function will be called by the FCO core. It will return the name of the function to be called. The “p” parameter of this function stores the call test, and at the first call it will be empty (nil). That will allow us to register our trigger. In triggerType, we show that the trigger is called BEFORE the publish operation, and applies only to users. To the system administrators, of course, we allow publishing everything. In triggerOptions, we detail the operations for which the trigger will fire.

And most importantly, return {exitState = “CANCEL”}, then what the trigger was designed for. It will return failure when the user tries to share his image in the control panel.

In the FCO architecture, any object (disk, server, image, network, network adapter, etc.) is represented as a Resource entity, which has common parameters:

• Resource uuid
• resource name
• resource type
• Resource owner uuid
• resource status (active, inactive)
• resource metadata
• resource keys
• UUID of the product to which the resource belongs
• VDC resource

This is very convenient when working on an API, when all resources are handled on the same principle. Products are configured by the provider, and their customer orders. Since our billing is on the sidelines, the customer can order any product from the panel for free. He will be considered later in the billing. The product can be - ip address per hour, additional GB disk per hour or just a server.

Keys can mark certain resources to change the logic of working with them. For example, we can mark three physical nodes with the Weight key, and mark some clients with the same key, thereby allocating these nodes to personally given clients. We use this mechanism for VIP clients who do not like neighbors near their VMs. The very same functionality can be used much more widely.

A licensing model implies a payment for each core of a physical node processor. Also the cost is affected by the number of cluster types. If you plan to use together, for example, KVM and VMware, then the cost of the license will increase.

FCO is a complete product, its functionality is very rich, so we plan to prepare several articles at once with a detailed description of the functioning of the network part.

Having worked with this orchestrator for several years, we can mark it as very suitable. Alas, the product is not without flaws:

• we had to optimize the database, because queries started to slow down as the amount of data in them increased;
• after one accident because of a bug, the recovery mechanism did not work, and the machines of unhappy clients had to be raised by their own set of scripts;
• The mechanism for detecting the inaccessibility of a node is sewn into the code and cannot be customized. That is, we cannot create our own policies for determining the inaccessibility of a node.
• Logging is not always detailed. Sometimes, when you need to go down to a very low level to analyze a particular problem, there is not enough source code for some components to understand the causes;

TOTAL: overall, the product impressions are good. We are in constant contact with the developers of the orchestrator. Guys are disposed to constructive cooperation.

Despite its simplicity, the FCO has broad functionality. In future articles, we plan to delve into the following topics:

• networking in FCO
• providing live-recovery and FQP protocol
• writing your own plugins and widgets
• connection of additional services such as Load Balancer and Acronis
• backup
• unified node configuration and configuration mechanism
• virtual machine metadata processing

PS Write in the comments if other aspects are interesting. Stay tuned!