Migrating a Django application from Heroku to AWS ECS
At Convox we have had many customers make the switch from Heroku. Some want to significantly lower their hosting costs. Some want more control over their environment. Some want to make use of the greater AWS ecosystem. Many of our Enterprise clients make the switch because the Convox Self-hosted Console which makes it much easier for them to achieve PCI or HIPAA compliance. Whatever the reason, we hope to make the process as easy as possible. We have a Heroku Migration Guide that you can refer to, but for this post I will walk you through migrating a simple Django application from Heroku to the Convox platform running on ECS.
First things first, let’s get our Django application running locally using Convox’s local development environment. In order to do this we are going to need a couple of prerequisites: - Install Docker with Kubernetes support - Signup for a Convox account - Install the Convox CLI on our local computer - Install a local rack
Assuming we have completed all of the above we now need to containerize our Django application. This assumes we are currently deploying using Heroku’s build packs and not their container registry. If we have already containerized our app we can of course skip this step. The Dockerfile defines a container image that will be spun up to run our application. For our simple Django app the Dockerfile looks like:
FROM python:3 ENV PYTHONUNBUFFERED 1 WORKDIR /mysite ADD requirements.txt /mysite/ RUN pip install -r requirements.txt ADD . /mysite/
As you can see it’s quite simple but let’s walk through it. First we are building from a public base image. In this case it’s the public Python 3 image which simply gives us a base container running linux with Python 3 installed. If we were running an older version of Django running on Python 2.7 then we could use the Python 2.7 base image. Next we update the environment to be unbuffered which is just a good practice for running python applications because it ensures stdin, stdout, and stderr are unbuffered. Then we define a work directory on the container for our Django app and we copy our requirements.txt from our local machine into the container. Now we run
pip install to install all the requirements on our container and finally we copy over the rest of our app into the work directory.
You might ask why we copy requirements.txt and run pip install as a separate step instead of copying the entire app and then running pip install. The reason we do this is to take advantage of Docker layer caching to speed up our builds. We know that we don’t change our requirements.txt all that often but we change our application source code constantly. By moving the pip install step first Docker will cache that step and only run it if something in requirements.txt changes.
Alright, so now we have a Dockerfile that defines an image to run our Django app. The next thing we need to do is create a convox.yml file which describes our application and all supporting infrastructure so Convox knows how to run our application both locally and on AWS. For our example Django app the convox.yml file looks like this
resources: database: type: postgres services: web: build: . command: gunicorn mysite.wsgi --bind=0.0.0.0:8000 port: 8000 resources: - database environment: - SECRET_KEY=foo
Again this is pretty simple but let’s walk through it. First we define a database resource. In our case we are going to use Postgres as our primary datastore. With this defined, Convox will automatically start up a docker container running postgres when we run locally and Convox will automatically configure an RDS cluster running Postgres when we deploy to production. Next we define our Django service which we are calling web. We instruct Convox to build using the Dockerfile we just created and we define a startup command to run the Gunicorn web server listening on port 8000. You will notice that this command is almost identical to what is in a typical Heroku procfile. Now we just need to tell Convox that the web server needs to be able to talk to the database by listing the database as a resource for the web service. This step will automatically create a
DATABASE_URL environment variable on the Django container and configure all the correct security groups for RDS. Finally we define any necessary default environment variables in this case we only need the Django secret key.
Now we should have everything we need to run locally. Making sure we are in the root directory for our application we simply run
$ convox start
You may see some errors at this point and that’s ok. The issue is we haven’t migrated our database yet. Convox allows you to run one-off commands against your local and production racks. So let’s go ahead and migrate by opening a second terminal window and running
$ convox switch local
to ensure we are running commands against our local rack and now we can run
$ convox run web python manage.py migrate
to run our migrations. Once our migrations have run we should see the errors clear up in our other terminal window and we should be ready to test. Now we can run
$ convox services
to get the hostname for our local app. Finally we can put that hostname in our browser and make sure our app works. If we want to run any other management commands we can do the same as we did with migrations ex:
$ convox run web python mange.py createsuperuser
So awesome, we have our Django app running locally on Convox we are ready to deploy to AWS! The first thing we need to do is connect Convox to our AWS account following the instructions here. Once we have done this we can create a production rack to deploy to . This entire process should take 15-20 minutes but you only need to do it once. Once our production rack is ready to go we need to switch our Convox CLI from our local rack to our production rack. We run
$ convox racks
to grab the name of our production rack. If you don’t see your production rack make sure it is done being created and that your CLI is logged in to your Convox account. Once we have our production rack name we switch to it with
$ convox switch [rack name]
Now we create an application with
$ convox apps create --wait
and finally we can deploy our application with
$ convox deploy --wait
The very first time we run a deployment it might take 15-20 minutes, as RDS instances, Elastic Load Balancers, etc… are created, but subsequent deploys will be much faster. When our deployment is complete we need to run our migrations with same command as before
$ convox run web python manage.py migrate
Once the migrations are complete we can grab the hostname for our app with
$ convox services
and open it in a browser. So now have successfully deployed our application to ECS! All we need to do now is migrate our data and make the final switch.
We will typically perform the data migration twice. Initially as a test to make sure everything works smoothly and finally when we actually make the switch. Before we begin this process we need to make sure we have Postgres Installed on our local computer, we don’t need to have a local postgres database running but we do need the postgres client installed. The first step in migrating our data is to download a snapshot of our Heroku Postgres Database. We do this by running
$ heroku pg:backups:capture
$ heroku pg:backups:download
taking note of the dumpfile name (typically “latest.dump”).
Now that we have our database snapshot we need to open up a proxy connection to our Convox Postgres Database running on RDS so we can import the data. Convox resources are not externally accessible for security reasons but it is possible to open a local proxy to the resource provided you are logged into your Convox account. In order to do this we first grab our resource name and URL with
$ convox resources
In this example our name is
database so we can open a proxy connection with
$ convox resources proxy database
We might receive an error stating that the port is already in use which will happen if we already have a postgres database running on our local computer. We can mitigate this by either shutting down our local postgres server or specifying a different port for our proxy with
Now that we have our proxy open we can load our data into RDS (please note this will overwrite any data that is present in our Convox RDS instance). We will want to open a new terminal window (leaving the window running the proxy open). We will need to parse the URL we got from
convox resources so we can get the username, password and database name for the import command. The format of the URL is
Once we have everything we need, we restore our data with
$ pg_restore --verbose --clean --no-acl --no-owner -h localhost -U [username] -d [database] [dumpfile name]
If you receive an error like
pg_restore: [archiver] unsupported version you may need to upgrade your local postgres to the latest version. Once our restore is complete we can test our app again and we should see all our data from Heroku is now present. Once we are satisfied that everything restored correctly and the app is working properly we are ready to flip the switch and move our production traffic from Heroku to Convox.
The first few steps in preparing for the move we can do a few days ahead of time. The first thing we want to do is make sure that our DNS record for the domain we are migrating has a low TTL (Time To Live) so that our DNS changes propagate quickly. Typically a TTL of 60 seconds is reasonable. You can find more information on this here. Next we will want to issue a certificate for that domain on Convox with
$ convox certs generate [domain]
If there is not already a certificate in our AWS account for that domain AWS will generate one so we need to have our AWS administrator looking out for an approval request email. The last piece of preparation is to add a domain parameter to our convox.yml,set it to the domain we are migrating, and redeploy our Convox app.
No it’s time to flip the switch! This move will involve some small amount of downtime, so if your Heroku app is serving production traffic you may want to do this during off hours. First we will want to disable any Heroku scheduler jobs we have enabled to ensure that nothing is writing to our Heroku database once we put the app in maintenance mode. If you do have Heroku scheduler jobs you can recreate them with timers in Convox. Next we want to put the Heroku app in maintenance mode. At this point our Heroku application will be offline. Finally if we have any worker dynos for things like celery tasks we will want to give them a few seconds to finish their current tasks and then scale them down to zero. At this point nothing should be writing to our Heroku Database and it is safe to repeat our Data migration steps from above to get the latest data into RDS.
Once our data migration is complete we want to make our DNS switch. We can grab the router value for our convox rack using
$ convox rack
and update the CNAME record for our domain to point to that location. Now we just wait a few minutes for DNS to update (we might need to flush our local DNS cache) and we can see that our site is live on Convox with the latest data!
It is very popular for Django apps to use Celery workers to handle asynchronous tasks. With Heroku we would handle this with a worker Dyno and with Convox the process is very similar. We simply add a data store for our Celery broker (like Redis) and add a new service for our Celery worker. A typical convox.yml for this scenario would look like:
resources: database: type: postgres redis: type: redis services: web: build: . command: gunicorn mysite.wsgi --bind=0.0.0.0:8000 port: 8000 resources: - database - redis environment: - SECRET_KEY=foo worker: build: . command: celery -A web worker -l warning resources: - database - redis
In Heroku it’s popular to use the scheduler to run periodic tasks. Convox supports the same thing with timers which are defined in the convox.yml. For example if we want to trigger a task to send emails every five minutes we might modify our convox.yml from above to look like
resources: database: type: postgres redis: type: redis services: web: build: . command: gunicorn mysite.wsgi --bind=0.0.0.0:8000 port: 8000 resources: - database - redis environment: - SECRET_KEY=foo worker: build: . command: celery -A web worker -l warning resources: - database - redis timers: sendemails: schedule: "*/5 * * * ?" command: python manage.py sendemails service: worker
Hopefully this answers all your questions about migrating a Django app from Heroku to AWS using Convox. If you would like to try out Django on Convox you can grab our sample Django app. This just scratches the surface of what you can do with Convox though. In addition to helping you get up and running on AWS easily, Convox also provides: