Firestore Data migration between projects
Have you ever wanted to migrate an app to another Firestore project? You only have a production environment, and you want to create a development environment with the same data?
Keep reading and learn how to use Cloud Firestore import and export features and migrate project data.
This example demonstrates how to export data from a source project and import that data into the destination project. In this example, the source project is called production-firestore and the destination project is development-firestore.
First things first - we will need to export production data from Firestore to the Cloud Storage Bucket. Once we have that exported, we need to transfer the exported data to the development storage bucket, after which we will finally be able to import the data.
Before we start transferring data, there are some requirements.
- You’ll need to enable billing for both projects. Only Google Cloud projects with billing enabled can use the export and import functionality.
- Make sure your account has the necessary Cloud IAM permission for both projects.
If you are the project owner for both projects, you have the required permissions. Otherwise, the following Cloud IAM roles grant the necessary IAM permissions:
Owner, Cloud Datastore owner, Cloud Datastore Import Export Admin
3. Access gcloud from the Google Cloud Console using Cloud Shell.
Be careful! Export and import operations are charged for document reads and writes at the rates listed in Cloud Firestore pricing. The costs of export and import operations do not count towards your spending limit. Export or import operations will not trigger your Google Cloud budget alerts until after completion.
Reads and writes performed during these steps are applied to your daily quota after the operation is complete.
Export data from the source project
Create a bucket in the source project - in our case, we will create a production-bucket.
Your bucket name should meet bucket name requirements.
2. Export data using gcloud
We will use the gcloud firestore export command to export data from your source project.
Make sure that you are in the correct Firestore location (production-firestore
There are 2 options:
1. Export all data
Replace [SOURCE_BUCKET] with the name of your Cloud Storage bucket
gcloud firestore export gs://[SOURCE_BUCKET] --async
In our case it will be:
gcloud firestore export gs://production-bucket --async
2. Export specific collections
gcloud firestore export gs://[SOOURCE_BUCKET] --collection-ids=[COLLECTION_ID_1],[COLLECTION_ID_2] --async
For example, in our case, we want to export users and reports collection
gcloud firestore export gs://production-bucket --collection-ids=users,reports --async
Take a note of your export operation’s outputUriPrefix as we will use this later on. By default, Cloud Firestore adds a prefix to our export files based on a timestamp.
Transfer data from one bucket to another
Just like we did for the source/production environment, we will need to create a bucket in the destination project - in our case, we will create a development-bucket.
Important: Your bucket has to be in the same location as your Cloud Firestore (destination Firestore Database).
2. Open the Transfer page in the Google Cloud Console
First of all, we will need to create a transfer job.
Choose your source bucket as a source and destination bucket as a destination.
After selecting the correct source and destination buckets, update the other settings as you’d like (you can choose to overwrite or delete data). Finally, schedule the transfer and the data will start migrating between environments. When all data has been transferred, we are ready to import data from the destination bucket to the destination Firestore Database.
Import data into the destination project
Before we can start an import operation, we need to make sure that our destination project can access our Cloud Firestore data files.
- Make sure that gcloud is configured for the correct project
You can configure gcloud to the correct project with this command:
gcloud config set project [DESTINATION_PROJECT_ID]
In our case, we will set our project to development-firestore:
gcloud config set project development-firestore
2. Use gcloud firestore import
We will use the gcloud firestore import command to import the data into our destination bucket.
gcloud firestore import gs://[SOURCE_BUCKET]/[EXPORT_PREFIX] --async
Our source bucket will be development-bucket and the export prefix will be our export operation’s ouputUriPrefix timestamp (see the last step in exporting data).
In our case, it should look like this:
gcloud firestore import gs://development-bucket/2022-01-18T10:58:23_56418 --async
While the export operation is running, you can check your operation’s progress with this command:
gcloud firestore operations list
And that's it!
We have successfully moved data from the source project to the destination project! For more detailed information, you can check the official Firestore documentation.
Thank you for your the time to read this blog! Feel free to share your thoughts about this topic in the comments or drop us an email at firstname.lastname@example.org.