Sonata import bundle
Until now, one of the best admin panels for symfony is SonataAdminBundle, and for good reason. Simple installation, configuration, many out-of-the-box features and a large community.
The only thing missing is the import of files. Agree, an important function.
The network contains many import implementations for Sonata, but there are minor flaws everywhere - the ability to import text fields only, not entities, does not work with collections, it’s problematic to load huge databases that can be processed for more than one hour ...
')
Today I want to present you my implementation, which I have been using successfully for quite a long time, but only now my hands have come to comb it all and arrange it in a separate bundle.
I will not describe the entire installation and configuration process here. Moreover, unlike many implementations, it is extremely simple. You can read all this in the README.md and wiki on github:
In this article I want to describe only the interesting moments that I encountered during its creation.
For the first time, the idea of ​​implementing this bundle came to me at a time when I needed to transfer a fairly large table of regions, cities, towns, squares, and everything, everything, to the customer’s base (~ 3 million lines). Complicated all the fact that access to its server we did not have.
I tried several ready-made solutions, but I realized that they are designed for small volumes that can be downloaded while waiting for a response from the server.
It was necessary to implement this not through a web server, but through php-cli. Fortunately, symfony has very good tools for working with console commands.
To call there is an excellent class Application :
But this method is also not suitable, because it works through a web server. Only one thing remains: Symfony \ Component \ Process \ Process , since it works directly with the console. We create a simple command (thanks to oxidmod for a more beautiful and correct solution):
Last line for asynchronous operation. And run all this in the background.
Agree that it is hard to wait more than a minute, not understanding what exactly is happening. And if this process takes an hour? Two?
That is why we need some kind of log console command. I usually use text files for logs, but this time, because of the amount of information, I decided to use a database.
An entity is responsible for each line: Doctrs \ SonataImportBundle \ Entity \ ImportLog .
Each entry corresponds to a line from a file and it has everything you need:
It is for this data that we will further monitor the download process, and display the final detailed report.
Since an iterator is used to parse the file, the percentage of completion will not work. We display just the total number of records processed.
Unfortunately, I did not learn to catch FatalError. Therefore, in the case of, for example,
command will drop with FatalError.
Another exception I encountered is an ORMException.
What is so interesting about it? Normal exception when trying to process a request with incorrect data.
Actually, this is exactly what it is meant for, though after throwing out such an exception, the EntityManager closes the connection, and responds to any attempts to query the database:
EntityManager is closed
In my bandle, such an exception is thrown in 2 cases. The first is if the validation of the entity is incorrectly configured (they must be validated before adding to the base of the entity)
And the second is related to the work of the bundle with the fields of type choice and entity . If we essentially have a subsidiary entity (for example, the book has an author. The author’s choice comes from the database), then when importing the book we can specify the author either by using the ID or by using the title. If the field is not numeric, then the system tries to find the entity by the name field. If the entity does not have such a field (for example, the author's name is stored not in name, but in login or username), then we get an ORMException.
In principle, they were quite frequent, so I had to make a small hack to reload the EntityManager, so that after throwing an exception, the system could expose the STATUS_ERROR file, and successfully display all of this in the interface:
By default, Sonata exports only simple fields (text, date, numbers). In order for it to export nested entities, they must be explicitly set in the getExportFields method. In addition, nested entities need to configure the __toString () method; The representation of the entity as a string will be exported.
ImportBundle also uses this method so that the newly imported file can be loaded into the database without any changes. In case you re-create the file, then the table with the corresponding column-field is on the import page.
I have never liked the fact that in order to change a couple of lines in a bundle, it is necessary to make (not that complicated, but not too comfortable) add-on using easy-extends .
Therefore, all that is possible, I brought to the configs. Even the class with which the file is parsed. So, in which case, you can always implement loading and XML and JSON and XLS.
Read more about all configuration options in the wiki.
If you have non-standard fields in the database (for example, in my case, center_point is the coordinates in the database), you must declare a class that will process the data from the file, and bring them to the form in which they will poured into mysql.
For example: the type center_point is the coordinate (MySql type is point ). When added to the database and retrieved from the database, it is an object of the class Point . The Point object has a __toString method.
It is used to import it, and we get beautiful coordinates in the import file. If we try to fill the database with the same x, y, then an ORMException is waiting for us. This is what the mappings array is intended for. In this case, it simply takes the service with the id doctrs.form_format.point , which implements the Doctrs \ SonataImportBundle \ Service \ ImportAbstract interface , and, based on the obtained value, returns the necessary type, which we can fill in the base.
Here is the code of the service itself
Service code doctrs.form_format.city_pa
As you can see, in the mappings parameter we specify not the names of the classes, but the id of the services, which gives us freedom of action. For example, for a type conversion of city_autocomplete I needed container.
I used this bundle for half a year (at that time it was not yet issued and I just pulled it up with bitbucket). Of course there were some uncritical errors, but after registering on packagist.org I try to fix everything so that there are no questions or vague error messages left.
There are small plans to improve this bandla, but let's see if they can reach their hands.
Any comments and comments will be glad.
The only thing missing is the import of files. Agree, an important function.
The network contains many import implementations for Sonata, but there are minor flaws everywhere - the ability to import text fields only, not entities, does not work with collections, it’s problematic to load huge databases that can be processed for more than one hour ...
')
Today I want to present you my implementation, which I have been using successfully for quite a long time, but only now my hands have come to comb it all and arrange it in a separate bundle.
I will not describe the entire installation and configuration process here. Moreover, unlike many implementations, it is extremely simple. You can read all this in the README.md and wiki on github:
In this article I want to describe only the interesting moments that I encountered during its creation.
Large amounts of data
For the first time, the idea of ​​implementing this bundle came to me at a time when I needed to transfer a fairly large table of regions, cities, towns, squares, and everything, everything, to the customer’s base (~ 3 million lines). Complicated all the fact that access to its server we did not have.
I tried several ready-made solutions, but I realized that they are designed for small volumes that can be downloaded while waiting for a response from the server.
Decision
It was necessary to implement this not through a web server, but through php-cli. Fortunately, symfony has very good tools for working with console commands.
To call there is an excellent class Application :
$application = new Application(); // ... register commands $application->run(); But this method is also not suitable, because it works through a web server. Only one thing remains: Symfony \ Component \ Process \ Process , since it works directly with the console. We create a simple command (thanks to oxidmod for a more beautiful and correct solution):
$command = sprintf( '/usr/bin/php %s/console promoatlas:sonata:import %d "%s" "%s" > /dev/null 2>&1 &', $this->get('kernel')->getRootDir(), $fileEntity->getId(), $this->admin->getCode(), $fileEntity->getEncode() ? $fileEntity->getEncode() : 'utf8' ); Last line for asynchronous operation. And run all this in the background.
Reporting
Agree that it is hard to wait more than a minute, not understanding what exactly is happening. And if this process takes an hour? Two?
That is why we need some kind of log console command. I usually use text files for logs, but this time, because of the amount of information, I decided to use a database.
An entity is responsible for each line: Doctrs \ SonataImportBundle \ Entity \ ImportLog .
Each entry corresponds to a line from a file and it has everything you need:
- ts - timestamp,
- status - what happened to the string
- message - error messages
- line - the line number from the file,
- foreignId - entity id
It is for this data that we will further monitor the download process, and display the final detailed report.
Since an iterator is used to parse the file, the percentage of completion will not work. We display just the total number of records processed.
Errors
Unfortunately, I did not learn to catch FatalError. Therefore, in the case of, for example,
function setOwner(Owner $owner); $owner = $em->findOwner(); // , null $entity->setOwner($owner); command will drop with FatalError.
Another exception I encountered is an ORMException.
What is so interesting about it? Normal exception when trying to process a request with incorrect data.
Actually, this is exactly what it is meant for, though after throwing out such an exception, the EntityManager closes the connection, and responds to any attempts to query the database:
EntityManager is closed
In my bandle, such an exception is thrown in 2 cases. The first is if the validation of the entity is incorrectly configured (they must be validated before adding to the base of the entity)
$validator = $this->getContainer()->get('validator'); $errors = $validator->validate($entity); And the second is related to the work of the bundle with the fields of type choice and entity . If we essentially have a subsidiary entity (for example, the book has an author. The author’s choice comes from the database), then when importing the book we can specify the author either by using the ID or by using the title. If the field is not numeric, then the system tries to find the entity by the name field. If the entity does not have such a field (for example, the author's name is stored not in name, but in login or username), then we get an ORMException.
In principle, they were quite frequent, so I had to make a small hack to reload the EntityManager, so that after throwing an exception, the system could expose the STATUS_ERROR file, and successfully display all of this in the interface:
if (!$this->em->isOpen()) { $this->em = $this->em->create( $this->em->getConnection(), $this->em->getConfiguration() ); } Import / Export Setup
By default, Sonata exports only simple fields (text, date, numbers). In order for it to export nested entities, they must be explicitly set in the getExportFields method. In addition, nested entities need to configure the __toString () method; The representation of the entity as a string will be exported.
ImportBundle also uses this method so that the newly imported file can be loaded into the database without any changes. In case you re-create the file, then the table with the corresponding column-field is on the import page.
Extensibility
I have never liked the fact that in order to change a couple of lines in a bundle, it is necessary to make (not that complicated, but not too comfortable) add-on using easy-extends .
Therefore, all that is possible, I brought to the configs. Even the class with which the file is parsed. So, in which case, you can always implement loading and XML and JSON and XLS.
doctrs_sonata_import: mappings: - { name: center_point, class: promaotlas.form_format.point} - { name: city_autocomplete, class: promoatlas.form_format.city_pa} upload_dir: %kernel.root_dir%/../web/uploads class_loader: Doctrs\SonataImportBundle\Loaders\CsvFileLoader encode: default: utf8 list: - cp1251 - utf8 - koir8 Read more about all configuration options in the wiki.
Custom field types
If you have non-standard fields in the database (for example, in my case, center_point is the coordinates in the database), you must declare a class that will process the data from the file, and bring them to the form in which they will poured into mysql.
For example: the type center_point is the coordinate (MySql type is point ). When added to the database and retrieved from the database, it is an object of the class Point . The Point object has a __toString method.
public function __toString(){ retrun $this->x . ', ' . $this->y; } It is used to import it, and we get beautiful coordinates in the import file. If we try to fill the database with the same x, y, then an ORMException is waiting for us. This is what the mappings array is intended for. In this case, it simply takes the service with the id doctrs.form_format.point , which implements the Doctrs \ SonataImportBundle \ Service \ ImportAbstract interface , and, based on the obtained value, returns the necessary type, which we can fill in the base.
Here is the code of the service itself
class Point implements ImportAbstract { public function getFormatValue($value){ $value = explode(',', $value); $point = new \PHPOpenGIS\MainBundle\Geometry\Point($value[0], $value[1] ?? 0); return $point; } } Service code doctrs.form_format.city_pa
class CityPa implements ImportAbstract, ContainerAwareInterface { private $container; public function setContainer(ContainerInterface $container = null) { $this->container = $container; } public function getFormatValue($value){ /** @var ContainerInterface $container */ $container = $this->container; $city = $container->get('promoatlas.city_autocomplete')->byName($value); return $city; } } As you can see, in the mappings parameter we specify not the names of the classes, but the id of the services, which gives us freedom of action. For example, for a type conversion of city_autocomplete I needed container.
Conclusion
I used this bundle for half a year (at that time it was not yet issued and I just pulled it up with bitbucket). Of course there were some uncritical errors, but after registering on packagist.org I try to fix everything so that there are no questions or vague error messages left.
There are small plans to improve this bandla, but let's see if they can reach their hands.
Any comments and comments will be glad.
Source: https://habr.com/ru/post/338986/
All Articles