Drupal 7 to Drupal 8 migration with configuration entities

Since Drupal 8.5.0, the migrate system is stable, so this is a great time to review the migration tools provided by the community.

This post does not cover migration from the UI, it focuses on partially customized migration that are runned with Drush. In most cases, you will probably want to review the content model a bit (you know, that content type or field machine name that does not actually reflect the truth since its creation, ...).

The idea behind is to delegate the heavy lifting to Migrate Upgrade for migration template generation, then apply content model changes if needed.

Prepare your setup

Require dependencies
The needed tools are quite obvious, the only tricky part for the moment is getting the right versions.

# Require Drush 8 https://www.drupal.org/project/migrate_upgrade/issues/2915726
composer require drush/drush:~8.0
composer require drupal/migrate_upgrade:~3.0
composer require drupal/migrate_tools:~4.0
composer require drupal/migrate_plus:~4.0

Then enable them.

drush en migrate_tools migrate_upgrade

Add Drupal 7 database references
Edit your Drupal 8 settings file, so in this example, you will have to import your Drupal 7 SQL dump in a separate d7 database.

// Database entry for `drush migrate-upgrade --configure-only`
$databases['upgrade']['default'] = array (
  'database' => 'd7',
  'username' => 'root',
  'password' => 'root',
  'prefix' => '',
  'host' => 'localhost',
  'port' => '3306',
  'namespace' => 'Drupal\\Core\\Database\\Driver\\mysql',
  'driver' => 'mysql',
);
// Database entry for `drush migrate-import`
$databases['migrate']['default'] = array (
  'database' => 'd7',
  'username' => 'root',
  'password' => 'root',
  'prefix' => '',
  'host' => 'localhost',
  'port' => '3306',
  'namespace' => 'Drupal\\Core\\Database\\Driver\\mysql',
  'driver' => 'mysql',
);
// Database entry for `drush migrate-import`
$databases['drupal_7']['default'] = array (
  'database' => 'd7',
  'username' => 'root',
  'password' => 'root',
  'prefix' => '',
  'host' => 'localhost',
  'port' => '3306',
  'namespace' => 'Drupal\\Core\\Database\\Driver\\mysql',
  'driver' => 'mysql',
);

Get the Drupal 7 files
A good idea would be to import files on your development server with rsync, so you can get the changed files only before rolling a migration update.

Example

# Get files on your local dev environment from the production server
rsync -avzh my_user@drupal_7_server:/var/www/my_d7_site/docroot/sites/default/files/ /var/www/my_d8_site/web/sites/default/files/d7/public_files/docroot/sites/default/files/

Get the migration templates

This step will generate migration configuration entities in the active configuration store.

You can make a database backup first, as it could be needed before enabling your custom migration module (to avoid a PreExistingConfigException). Note that restarting from a backup is the "hacking around" method, you may want to rename each migration template instead, after having exported them in your custom module. See below.

drush sql-dump > my_drupal_8.sql
drush migrate-upgrade --configure-only

Create a custom module and the configuration directory.

# Follow Drupal Console steps, naming the module my_migrate in this example.
drupal generate:module
# Create then the config/install directory in your module.
mkdir -p my_migrate/config/install

Export site configuration generated by drush migrate-upgrade --configure-only

drush config-export --destination=/tmp/migrate

Then copy it in the custom migration directory. Do not include migrate_plus.migration_group.default.yml

cp /tmp/migrate/migrate_plus.migration.* /tmp/migrate/migrate_plus.migration_group.migrate_*.yml /path/to/my_migrate/config/install/

Remove then unnecessary templates (entities that will not be part of your migration). When removing templates, make sure that they are not required in the migration_dependencies section. If you remove dependencies, make sure that they have been migrated in another way.

Example

The migration template migrate_plus.migration.upgrade_d7_node_article.yml has the following requirements:

migration_dependencies:
  required:
    - upgrade_d7_user
    - upgrade_d7_node_type

For some reasons, you could have created content types manually and want to get rid of the upgrade_d7_node_type template.

Optionally customize your migration with machine name changes, source / process plugins or hook_migrate_prepare_row() / hook_migrate_MIGRATION_ID_prepare_row().

Restart then from your Drupal 8 database backup, or just change your migration templates file name, uuid, id and group for each yml file. Just keep in mind that while renaming your migration templates (configuration entities) id, you will have to rename the required and optional dependencies as well.

Run migration with Drush

Once done with your migration templates, enable your new module.

drush en my_migrate

Start to test your migration by id, with dependencies that are required by other entities first (role, user, file, taxonomy term, ...).

# Check the migrate status first, by id
drush ms upgrade_d7_user_role
# Migrate import by id
drush mim upgrade_d7_user_role

Update

Include changes that were made to entities (new or modified ones).

drush mim upgrade_d7_user --update

Limit

drush mim upgrade_d7_file --limit="10 items"

Display migration messages

drush mmsg upgrade_d7_file

Errors will be logged in these messages, so it could be nice to redirect them into a log file.

drush mmsg upgrade_d7_file > sites/default/files/migration_logs/upgrade_d7_file.txt

Rollback

drush mr upgrade_d7_file

Other Drush migrate commands

Examples

Users and roles

# Migrate status of roles and users
drush ms upgrade_d7_user_role
drush ms upgrade_d7_user
# Migrate import for roles then users
drush mim upgrade_d7_user_role
drush mim upgrade_d7_user

Content and translation

# Migrate content type article + i18n translation
drush mim upgrade_d7_node_article
drush mim upgrade_d7_node_translation_article

Files

Public and private files

If you have imported your files in /var/www/my_d8_site/web/sites/default/files/d7/public_files/docroot/sites/default/files/
modify the /my_migrate/config/install/migrate_plus.migration.upgrade_d7_file.yml file by setting the source_base_path to /var/www/my_d8_site/web/sites/default/files/d7/public_files/docroot

# For a legacy site that has /docroot/sites/default/files
source_base_path: /var/www/my_d8_site/web/sites/default/files/d7/public_files/docroot

Do the same in the /my_migrate/config/install/migrate_plus.migration.upgrade_d7_file_private.yml file according to your private files path (ideally outside of the docroot).

Then

drush mim upgrade_d7_file
# And optionally
drush mim upgrade_d7_file_private

Applying changes after having enabled the custom migration module

Use the Configuration Development module or drush config-import --partial
 

Troubleshooting

Before running migration import, and depending on the database size, you may consider increasing memory_limit in your php.ini
 

Documentation

Migrate API

• Chapter Three [06/04/2017] Drupal to Drupal 8 via Migrate API
• Drupalize Me [26/04/2016] Custom Drupal-to-Drupal Migrations with Migrate Tools
• Drupalize Me [31/10/2017] Create Migrations from Core Templates

Drush

• Drush Migrate commands

Users and Roles

• Drupal.org Migrating users
• Deninet [06/07/2017] Building a custom migration in Drupal 8, Part 3: Users and Roles
• Metal Toad [30/11/2014] Drupal 8 Migrations, part 2: Migrating users from a Drupal 7 site
• Jeff Geerling [13/07/2013] Migrating Drupal 7 users from site to site while preserving password hashes

Nodes, merge entities, translation

• Evolving Web on GitHub [01/07/2017] Drupal 8 I18N / Translation Migration Example
• Lullabot [24/07/2017] Merging Entities During a Migration to Drupal 8

File Entities and Media Entities

• Previous Next [24/3/2017] Migrating Drupal 7 File Entities to Drupal 8 Media Entities
• Jeff Geerling [04/10/2016] Migrating 20,000 images, audio clips, and video clips into Drupal 8
• Evolving Web [30/1/2017] Drupal 8 Migration: Migrating Files / Images (Part 3)
• Deninet [13/06/2017] File migrations