Firestore Data migration between projects

Keep reading and learn how to use Cloud Firestore import and export features and migrate project data.

This example demonstrates how to export data from a source project and import that data into the destination project. In this example, the source project is called production-firestore and the destination project is development-firestore.

First things first -  we will need to export production data from Firestore to the Cloud Storage Bucket. Once we have that exported, we need to transfer the exported data to the development storage bucket, after which we will finally be able to import the data.

Before we start transferring data, there are some requirements.

1. You’ll need to enable billing for both projects. Only Google Cloud projects with billing enabled can use the export and import functionality.
2. Make sure your account has the necessary Cloud IAM permission for both projects.

If you are the project owner for both projects, you have the required permissions. Otherwise, the following Cloud IAM roles grant the necessary IAM permissions:

 Owner, Cloud Datastore owner, Cloud Datastore Import Export Admin

3. Access gcloud from the Google Cloud Console using Cloud Shell.

Be careful! Export and import operations are charged for document reads and writes at the rates listed in Cloud Firestore pricing. The costs of export and import operations do not count towards your spending limit. Export or import operations will not trigger your Google Cloud budget alerts until after completion.

Reads and writes performed during these steps are applied to your daily quota after the operation is complete.

Export data from the source project

1. Create a Cloud Storage bucket

Create a bucket in the source project - in our case, we will create a production-bucket.

Your bucket name should meet bucket name requirements.

2. Export data using gcloud

We will use the gcloud firestore export command to export data from your source project.

Make sure that you are in the correct Firestore location (production-firestore

There are 2 options:

1. Export all data

 Replace [SOURCE_BUCKET] with the name of your Cloud Storage bucket

gcloud firestore export gs://[SOURCE_BUCKET] --async

 In our case it will be:

gcloud firestore export gs://production-bucket --async

2. Export specific collections

gcloud firestore export gs://[SOOURCE_BUCKET] --collection-ids=[COLLECTION_ID_1],[COLLECTION_ID_2] --async

For example, in our case, we want to export users and reports collection

gcloud firestore export gs://production-bucket --collection-ids=users,reports --async

Take a note of your export operation’s outputUriPrefix as we will use this later on. By default, Cloud Firestore adds a prefix to our export files based on a timestamp.

outputUriPrefix: gs://production-bucket/2022-01-18T10:58:23_56418

Transfer data from one bucket to another

1. Create a Cloud Storage bucket

Just like we did for the source/production environment, we will need to create a bucket in the destination project - in our case, we will create a development-bucket.

Important: Your bucket has to be in the same location as your Cloud Firestore (destination Firestore Database).

2. Open the Transfer page in the Google Cloud Console

First of all, we will need to create a transfer job.

Choose your source bucket as a source and destination bucket as a destination.

After selecting the correct source and destination buckets, update the other settings as you’d like (you can choose to overwrite or delete data). Finally, schedule the transfer and the data will start migrating between environments. When all data has been transferred, we are ready to import data from the destination bucket to the destination Firestore Database.

Import data into the destination project

Before we can start an import operation, we need to make sure that our destination project can access our Cloud Firestore data files.

1. Make sure that gcloud is configured for the correct project

You can configure gcloud to the correct project with this command:

gcloud config set project [DESTINATION_PROJECT_ID]

In our case, we will set our project to development-firestore:

gcloud config set project development-firestore

2. Use gcloud firestore import

We will use the gcloud firestore import command to import the data into our destination bucket.

gcloud firestore import gs://[SOURCE_BUCKET]/[EXPORT_PREFIX] --async

Our source bucket will be development-bucket and the export prefix will be our export operation’s ouputUriPrefix timestamp (see the last step in exporting data).

In our case, it should look like this:

gcloud firestore import gs://development-bucket/2022-01-18T10:58:23_56418 --async

While the export operation is running, you can check your operation’s progress with this command:

gcloud firestore operations list

And that's it!

We have successfully moved data from the source project to the destination project! For more detailed information, you can check the official Firestore documentation.

Thank you for your the time to read this blog! Feel free to share your thoughts about this topic in the comments or drop us an email at hello@prototyp.digital.