Geekly Articles each Day
📜 ⬆️ ⬇️

Migrate API and what it eat. On the example of the migration of the forum Drupal 7. Part 1

Migrate API . Drupal 7. 1
This guide is a translation of the article .

I would like to share my experience with the migration of the forum from Drupal 7 to Drupal 8 , as well as talk about the problems I had to face during this process, as well as the tools I used. In addition, I will talk about the pitfalls that have occurred during the migration of the forum and the terms to it.

Instruments


We will carry out the entire migration process using Drush . Now let's define what we need for the migration (I will indicate the versions that will be used at the time of this writing)

Drupal: 8.4.5

Modules for migration:
')

Drush: 9.1.0

The entire migration process took place using


PHP: 7.1.14
MySQL: MariaDB 10.1.31

Note:


  • All paths will be specified relative to the module root (the custom_migration_forum directory, or the name you give to your module).
  • Before starting the migration, disable the rdf module in Drupal 8, since it can cause problems during the execution of Rolling back (when we cancel the migration changes).

Test content


For the relevance of the information, I decided in the course of writing this article to append the module that migrates the forum. I created test content using Devel to generate content. The total number of generated forum topics is 300 pcs., As well as comments to them. It turned out this:

Devel'

Preparing for migration


First, we will deploy a clean site on Drupal 8. On a clean site, we create our own module, or use GitHub .

Create a custom_migration_forum.info.yml file in which we enter basic information about our module and dependencies:

name: Custom Migration Forum description: Custom module for migrating forum from a Drupal 7 site. package: Migrations type: module core: 8.x dependencies: - drupal:migrate - drupal:migrate_drupal - drupal:forum - migrate_plus:migrate_plus (>=4.0-beta2) - migrate_tools:migrate_tools (>=4.0-beta2)

In order to remove the old configs of migrations, when uninstalling the module, it should be described in custom_migration_forum.install . I used to come across situations where a conflict of configs arose due to the fact that the old configs were not deleted when the module was uninstalled. Therefore, in order to protect yourself, it is better to remove them when uninstalling the module.

custom_migration_forum.install

 <?php /** * @file * Contains migrate_forum_drupal8.install. */ /** * Implements hook_uninstall(). * * Removes stale migration configs during uninstall. */ function custom_migration_forum_uninstall() { $query = \Drupal::database()->select('config', 'c'); $query->fields('c', ['name']); $query->condition('name', $query->escapeLike('migrate_plus.') . '%', 'LIKE'); $config_names = $query->execute()->fetchAll(); // Delete each config using configFactory. foreach ($config_names as $config_name) { \Drupal::configFactory()->getEditable($config_name->name)->delete(); } }

Or, you just need to register a dependency on each migration:
 dependencies: enforced: module: - custom_migration_forum

Migration of forum terms


Since the forum in Drupal 7 is, in fact, a node with terms, then for a start, we need to migrate terms. Start by creating plugins for migrating terms and vocabulary.

src / plugin / migrate / source / Vocabulary.php:

 <?php /** * @file * Contains \Drupal\migrate_therasomnia\Plugin\migrate\source\Vocabulary. */ namespace Drupal\custom_migration_forum\Plugin\migrate\source; use Drupal\migrate\Row; use Drupal\migrate\Plugin\migrate\source\SqlBase; /** * Drupal 7 vocabularies source from database. * * @MigrateSource( * id = "custom_migration_forum_vocabulary", * source_provider = "taxonomy" * ) */ class Vocabulary extends SqlBase { /** * {@inheritdoc} */ public function query() { $query = $this->select('taxonomy_vocabulary', 'v') ->fields('v', array( 'vid', 'name', 'description', 'hierarchy', 'module', 'weight', 'machine_name' )); // Filtered out unnecessary dictionaries. $query->condition('machine_name', 'forums'); return $query; } /** * {@inheritdoc} */ public function fields() { return array( 'vid' => $this->t('The vocabulary ID.'), 'name' => $this->t('The name of the vocabulary.'), 'description' => $this->t('The description of the vocabulary.'), 'help' => $this->t('Help text to display for the vocabulary.'), 'relations' => $this->t('Whether or not related terms are enabled within the vocabulary. (0 = disabled, 1 = enabled)'), 'hierarchy' => $this->t('The type of hierarchy allowed within the vocabulary. (0 = disabled, 1 = single, 2 = multiple)'), 'weight' => $this->t('The weight of the vocabulary in relation to other vocabularies.'), 'parents' => $this->t("The Drupal term IDs of the term's parents."), 'node_types' => $this->t('The names of the node types the vocabulary may be used with.'), ); } /** * {@inheritdoc} */ public function getIds() { $ids['vid']['type'] = 'integer'; return $ids; } }

src / plugin / migrate / source / Terms.php:

 <?php /** * @file * Contains \Drupal\migrate_therasomnia\Plugin\migrate\source\Terms. */ namespace Drupal\custom_migration_forum\Plugin\migrate\source; use Drupal\migrate\Row; use Drupal\migrate\Plugin\migrate\source\SqlBase; /** * Drupal 7 taxonomy terms source from database. * * @MigrateSource( * id = "custom_migration_forum_term", * source_provider = "taxonomy" * ) */ class Terms extends SqlBase { /** * {@inheritdoc} */ public function query() { $query = $this->select('taxonomy_term_data', 'td') ->fields('td', ['tid', 'vid', 'name', 'description', 'weight', 'format']) ->fields('tv', ['vid', 'machine_name']) ->distinct(); // Add table for condition on query. $query->innerJoin('taxonomy_vocabulary', 'tv', 'td.vid = tv.vid'); // Filtered out unnecessary dictionaries. $query->condition('tv.machine_name', 'forums'); return $query; } /** * {@inheritdoc} */ public function fields() { return [ 'tid' => $this->t('The term ID.'), 'vid' => $this->t('Existing term VID'), 'name' => $this->t('The name of the term.'), 'description' => $this->t('The term description.'), 'weight' => $this->t('Weight'), 'parent' => $this->t("The Drupal term IDs of the term's parents."), ]; } /** * {@inheritdoc} */ public function prepareRow(Row $row) { // Find parents for this row. $parents = $this->select('taxonomy_term_hierarchy', 'th') ->fields('th', ['parent', 'tid']); $parents->condition('tid', $row->getSourceProperty('tid')); $parents = $parents->execute()->fetchCol(); $row->setSourceProperty('parent', reset($parents)); return parent::prepareRow($row); } /** * {@inheritdoc} */ public function getIds() { $ids['tid']['type'] = 'integer'; return $ids; } }

After creating plugins for migrating terms and dictionaries, you need to configure for our migration. To do this, create a migrate_plus.migration.term.yml and migrate_plus.migration.vocablary.yml .

config / install / migrate_plus.migration.vocablary.yml:

 id: custom_migration_forum_vocabulary label: Taxonomy vocabulary forum migration_group: Custom Migration Forum dependencies: enforced: module: - custom_migration_forum source: plugin: custom_migration_forum_vocabulary target: migrate process: vid: - plugin: machine_name source: machine_name - plugin: make_unique_entity_field entity_type: taxonomy_vocabulary field: vid length: 32 migrated: true label: name name: name description: description hierarchy: hierarchy module: module weight: weight destination: plugin: entity:taxonomy_vocabulary

In the 'source' we specify our plugin custom_migration_forum_vocabulary , which was described in the class Vocabulary .

config / install / migrate_plus.migration.term.yml:

 id: custom_migration_forum_term label: Taxonomy terms forum migration_group: Custom Migration Forum dependencies: enforced: module: - custom_migration_forum source: plugin: custom_migration_forum_term target: migrate process: tid: tid vid: plugin: migration migration: custom_migration_forum_vocabulary source: vid name: name description: description weight: weight parent: parent changed: timestamp destination: plugin: entity:taxonomy_term migration_dependencies: required: - custom_migration_forum_vocabulary

When migrating terms in the vid, we specify the machine name ( custom_migration_forum_vocabulary ), from which we will take the forum dictionary ID.

Forum Migration


I would like to note that in this article we migrate the forum without additional fields. Before you start migrating forums, we need to migrate users, because without them, forums will not have authors. We will do this migration using plugins from the Kernel, but we will add them to our “ Custom Migration Forum ” migration group in order to be able to start the entire migration with one team. We need three migrations:

  1. role migration (custom_migration_forum_user_role)
  2. user migration (custom_migration_forum_user)
  3. Text format migration (custom_migration_forum_filter_format).

Here are our three yml files.

config / install / migrate_plus.migration.user_role.yml:

 id: custom_migration_forum_user_role label: User roles migration_group: Custom Migration Forum dependencies: enforced: module: - custom_migration_forum source: plugin: d7_user_role process: id: - plugin: machine_name source: name - plugin: user_update_8002 label: name permissions: - plugin: static_map source: permissions bypass: true map: 'use PHP for block visibility': 'use PHP for settings' 'administer site-wide contact form': 'administer contact forms' 'post comments without approval': 'skip comment approval' 'edit own blog entries': 'edit own blog content' 'edit any blog entry': 'edit any blog content' 'delete own blog entries': 'delete own blog content' 'delete any blog entry': 'delete any blog content' 'create forum topics': 'create forum content' 'delete any forum topic': 'delete any forum content' 'delete own forum topics': 'delete own forum content' 'edit any forum topic': 'edit any forum content' 'edit own forum topics': 'edit own forum content' - plugin: flatten weight: weight destination: plugin: entity:user_role migration_dependencies: optional: - custom_migration_forum_filter_format

config / install / migrate_plus.migration.user.yml:

 id: custom_migration_forum_user label: User accounts migration_group: Custom Migration Forum dependencies: enforced: module: - custom_migration_forum class: Drupal\user\Plugin\migrate\User source: plugin: d7_user process: uid: uid name: name pass: pass mail: mail created: created access: access login: login status: status timezone: timezone langcode: plugin: user_langcode source: language fallback_to_site_default: false preferred_langcode: plugin: user_langcode source: language fallback_to_site_default: true preferred_admin_langcode: plugin: user_langcode source: language fallback_to_site_default: true init: init roles: plugin: migration_lookup migration: custom_migration_forum_user_role source: roles user_picture: - plugin: default_value source: picture default_value: null - plugin: migration_lookup migration: d7_file destination: plugin: entity:user migration_dependencies: required: - custom_migration_forum_user_role optional: - d7_field_instance - d7_file - language - default_language - user_picture_field_instance - user_picture_entity_display - user_picture_entity_form_display

config / install / migrate_plus.migration.filter_format.yml:

 id: custom_migration_forum_filter_format label: Filter format configuration migration_group: Custom Migration Forum dependencies: enforced: module: - custom_migration_forum source: plugin: d7_filter_format process: format: format name: name cache: cache weight: weight filters: plugin: sub_process source: filters key: '@id' process: id: plugin: filter_id bypass: true source: name map: { } settings: plugin: filter_settings source: settings status: plugin: default_value default_value: true weight: weight destination: plugin: entity:filter_format

Now you can start preparing a plugin for the migration of forum materials .

src / plugin / migrate / source / Forum.php:

 <?php namespace Drupal\custom_migration_forum\Plugin\migrate\source; use Drupal\Core\Extension\ModuleHandlerInterface; use Drupal\migrate\Row; use Drupal\migrate_drupal\Plugin\migrate\source\d7\FieldableEntity; use Drupal\Core\Database\Query\SelectInterface; use Drupal\Core\Entity\EntityManagerInterface; use Drupal\Core\Extension\ModuleHandler; use Drupal\Core\State\StateInterface; use Drupal\migrate\Plugin\MigrationInterface; use Symfony\Component\DependencyInjection\ContainerInterface; /** * Extract forum from Drupal 7 database. * * @MigrateSource( * id = "custom_migration_forum_forum", * ) */ class Forum extends FieldableEntity { /** * The module handler. * * @var \Drupal\Core\Extension\ModuleHandlerInterface */ protected $moduleHandler; /** * {@inheritdoc} */ public function __construct(array $configuration, $plugin_id, $plugin_definition, MigrationInterface $migration, StateInterface $state, EntityManagerInterface $entity_manager, ModuleHandlerInterface $module_handler) { parent::__construct($configuration, $plugin_id, $plugin_definition, $migration, $state, $entity_manager); $this->moduleHandler = $module_handler; } /** * {@inheritdoc} */ public static function create(ContainerInterface $container, array $configuration, $plugin_id, $plugin_definition, MigrationInterface $migration = NULL) { return new static( $configuration, $plugin_id, $plugin_definition, $migration, $container->get('state'), $container->get('entity.manager'), $container->get('module_handler') ); } /** * The join options between the node and the node_revisions table. */ const JOIN = 'n.vid = nr.vid'; /** * {@inheritdoc} */ public function query() { // Select node in its last revision. $query = $this->select('node_revision', 'nr') ->fields('n', [ 'nid', 'type', 'language', 'status', 'created', 'changed', 'comment', 'promote', 'sticky', 'tnid', 'translate', ]) ->fields('nr', [ 'vid', 'title', 'log', 'timestamp', ]) ->fields('fb', [ 'body_value', 'body_format', ]); $query->addField('n', 'uid', 'node_uid'); $query->addField('n', 'type', 'node_type'); $query->addField('nr', 'uid', 'revision_uid'); $query->innerJoin('node', 'n', static::JOIN); $query->innerJoin('field_data_body', 'fb', 'n.nid = fb.entity_id'); // If the content_translation module is enabled, get the source langcode // to fill the content_translation_source field. if ($this->moduleHandler->moduleExists('content_translation')) { $query->leftJoin('node', 'nt', 'n.tnid = nt.nid'); $query->addField('nt', 'language', 'source_langcode'); } $this->handleTranslations($query); // Filtered node type forum. $query->condition('n.type', 'forum'); return $query; } /** * {@inheritdoc} */ public function prepareRow(Row $row) { // Get Field API field values. foreach (array_keys($this->getFields('node', 'forum')) as $field) { $nid = $row->getSourceProperty('nid'); $vid = $row->getSourceProperty('vid'); $row->setSourceProperty($field, $this->getFieldValues('node', $field, $nid, $vid)); } // Make sure we always have a translation set. if ($row->getSourceProperty('tnid') == 0) { $row->setSourceProperty('tnid', $row->getSourceProperty('nid')); } return parent::prepareRow($row); } /** * {@inheritdoc} */ public function fields() { $fields = [ 'nid' => $this->t('Node ID'), 'type' => $this->t('Type'), 'title' => $this->t('Title'), 'body_value' => $this->t('Full text of body'), 'body_format' => $this->t('Format of body'), 'node_uid' => $this->t('Node authored by (uid)'), 'revision_uid' => $this->t('Revision authored by (uid)'), 'created' => $this->t('Created timestamp'), 'changed' => $this->t('Modified timestamp'), 'status' => $this->t('Published'), 'promote' => $this->t('Promoted to front page'), 'sticky' => $this->t('Sticky at top of lists'), 'revision' => $this->t('Create new revision'), 'language' => $this->t('Language (fr, en, ...)'), 'tnid' => $this->t('The translation set id for this node'), 'timestamp' => $this->t('The timestamp the latest revision of this node was created.'), ]; return $fields; } /** * {@inheritdoc} */ public function getIds() { $ids['nid']['type'] = 'integer'; $ids['nid']['alias'] = 'n'; return $ids; } /** * Adapt our query for translations. * * @param \Drupal\Core\Database\Query\SelectInterface $query * The generated query. */ protected function handleTranslations(SelectInterface $query) { // Check whether or not we want translations. if (empty($this->configuration['translations'])) { // No translations: Yield untranslated nodes, or default translations. $query->where('n.tnid = 0 OR n.tnid = n.nid'); } else { // Translations: Yield only non-default translations. $query->where('n.tnid <> 0 AND n.tnid <> n.nid'); } } }

And the yml file to the plugin.

config / install / migrate_plus.migration.forum.yml:

 id: custom_migration_forum_forum label: Custom forum migration migration_group: Custom Migration Forum dependencies: enforced: module: - custom_migration_forum dependencies: enforced: module: - custom_migration_forum source: plugin: custom_migration_forum_forum node_type: forum target: migrate migration_dependencies: required: - custom_migration_forum_term - custom_migration_forum_user - custom_migration_forum_filter_format process: nid: tnid vid: vid langcode: plugin: default_value source: language default_value: 'en' title: title type: plugin: default_value default_value: forum 'body/value': body_value 'body/format': body_format uid: plugin: migration migration: custom_migration_forum_user source: node_uid status: status created: created changed: changed promote: promote sticky: sticky revision_uid: revision_uid revision_log: log revision_timestamp: timestamp taxonomy_forums: plugin: migration migration: custom_migration_forum_term source: taxonomy_forums destination: plugin: entity:node

The last thing left for us is comments on the topics (how can they be without them?).
Create a plugin and describe it in the configuration.

src / plugin / migrate / source / ForumComment.php:

 <?php /** * @file * Contains \Drupal\custom_migration_forum\Plugin\migrate\source\ForumComment. */ namespace Drupal\custom_migration_forum\Plugin\migrate\source; use Drupal\migrate\Row; use Drupal\migrate_drupal\Plugin\migrate\source\d7\FieldableEntity; /** * Drupal 7 comment forum source from database. * * @MigrateSource( * id = "custom_migration_forum_forum_comment", * source_provider = "comment", * ) */ class ForumComment extends FieldableEntity { /** * {@inheritdoc} */ public function query() { $query = $this->select('comment', 'c')->fields('c'); $query->innerJoin('node', 'n', 'c.nid = n.nid'); $query->addField('n', 'type', 'node_type'); $query->addField('n', 'nid'); $query->condition('n.type', 'forum'); $query->orderBy('c.created'); return $query; } /** * {@inheritdoc} */ public function prepareRow(Row $row) { $cid = $row->getSourceProperty('cid'); $node_type = $row->getSourceProperty('node_type'); $comment_type = 'comment_node_' . $node_type; $row->setSourceProperty('comment_type', 'comment_forum'); foreach (array_keys($this->getFields('comment', $comment_type)) as $field) { $row->setSourceProperty($field, $this->getFieldValues('comment', $field, $cid)); } return parent::prepareRow($row); } /** * {@inheritdoc} */ public function fields() { return [ 'cid' => $this->t('Comment ID.'), 'pid' => $this->t('Parent comment ID. If set to 0, this comment is not a reply to an existing comment.'), 'nid' => $this->t('The {node}.nid to which this comment is a reply.'), 'uid' => $this->t('The {users}.uid who authored the comment. If set to 0, this comment was created by an anonymous user.'), 'subject' => $this->t('The comment title.'), 'comment' => $this->t('The comment body.'), 'hostname' => $this->t("The author's host name."), 'created' => $this->t('The time that the comment was created, as a Unix timestamp.'), 'changed' => $this->t('The time that the comment was edited by its author, as a Unix timestamp.'), 'status' => $this->t('The published status of a comment. (0 = Published, 1 = Not Published)'), 'format' => $this->t('The {filter_formats}.format of the comment body.'), 'thread' => $this->t("The vancode representation of the comment's place in a thread."), 'name' => $this->t("The comment author's name. Uses {users}.name if the user is logged in, otherwise uses the value typed into the comment form."), 'mail' => $this->t("The comment author's email address from the comment form, if user is anonymous, and the 'Anonymous users may/must leave their contact information' setting is turned on."), 'homepage' => $this->t("The comment author's home page address from the comment form, if user is anonymous, and the 'Anonymous users may/must leave their contact information' setting is turned on."), 'type' => $this->t("The {node}.type to which this comment is a reply."), ]; } /** * {@inheritdoc} */ public function getIds() { $ids['cid']['type'] = 'integer'; return $ids; } }

config / install / migrate_plus.migration.forum_comment.yml:

 id: custom_migration_forum_forum_comment label: Comments forum migration_group: Custom Migration Forum dependencies: enforced: module: - custom_migration_forum source: plugin: custom_migration_forum_forum_comment target: migrate constants: entity_type: node process: cid: cid pid: plugin: migration_lookup migration: custom_migration_forum_forum_comment source: pid entity_id: nid entity_type: 'constants/entity_type' comment_type: comment_type field_name: comment_type subject: subject uid: uid name: name mail: mail homepage: homepage hostname: hostname created: created changed: changed status: status thread: thread comment_body: comment_body destination: plugin: entity:comment migration_dependencies: required: - custom_migration_forum_forum

Begin migration


Now we are ready for the migration. Add the Drupal 7 database to the configuration file of our site on Drupal 8. We do this in the settings.php file (or in another file that connects your settings. I have this settings.local.php file):

 <?php $databases['migrate']['default'] = array ( 'database' => 'Drupal_7', 'username' => 'root', 'password' => 'root', 'prefix' => '', 'host' => 'localhost', 'port' => '3306', 'namespace' => 'Drupal\Core\Database\Driver\mysql', 'driver' => 'mysql', );

We include the following modules:

 drush en migrate migrate_drupal migrate_plus migrate_tools taxonomy forum -

If you downloaded a module from GitHub , you can simply enable it:

 drush en custom_migration_forum -y

image

Drush Migration


Check the list of available migrations:

 drush ms

image

We start all our migrations with one command:

 drush mim --group="Custom Migration Forum"

image

In order to undo migration changes, there is a drush mr command:

 drush mr --group="Custom Migration Forum"

image

Also, sometimes errors occur during migration, and migration may hang in the status of Importing or Rolling back . In order to reset the migration status, you need to run:

 drush php-eval 'var_dump(Drupal::keyValue("migrate_status")->set('custom_migration_forum_forum', 0))'

Where custom_migration_forum_forum is the migration ID.

Our forum migration is complete. As a result, we got a fully migrated forum with users and comments on topics.

image

Source: https://habr.com/ru/post/349854/

More articles:


All Articles