We all talk about continuous integration, continuous delivery, and continuous deployment, but there is often a misconception of what these terms actually mean and what we need to do to facilitate them
It’s been over three years since we published this post on twelve factor apps. In that time 12 factor has continued to be the dominant philosophy for building scalable, secure, and maintainable web applications. It has even been applied to the rising popularity of serverless applications
In that time Convox has come a long way as well and we have made building 12 factor apps using Docker even easier. It seems like it’s time to revisit the principles of 12 factor apps and how we can follow them when creating containerized applications.
Is your inbox full of messages like this?
AWS Lambda to end-of-life Node.js v0.10 runtime on April 30, 2017. Please migrate to Node.js v4.3 or v6.10 immediately
Well I’m happy to announce that the Convox fully-automated migration to Node.js v4.3 is available in every region.
Pieces of this have been released over the past weeks and months, so many of you don’t need to take additional action.
But if you’re still getting those emails from AWS, you can:
convox rack update
You can do this from the command line with commands like:
$ convox rack update --wait $ convox env set FOO=bar --promote --app myapp1 $ convox resources unlink syslog-1234 --app myapp $ convox resources delete syslog-1234 $ convox resources create syslog --url tcp+tls://logs1.papertrailapp.com:12345 $ convox resources link syslog-4567 --app myapp
You can expect to receive a couple more updates from both AWS and Convox between now and April 30th, to make sure nobody is affected when the Lambda runtime is shut off for good.
Happy automatic infrastructure updates! 🎉🎉
Most Convox users deploy apps that they develop in-house. A team of developers iterates rapidly on new features and fixes, and they deploy that code multiple times a week, if not multiple times a day. This means building images frequently. When using Convox in this fashion, it’s easy to lose sight of one of the delightful benefits of an image-based platform like Convox: the ease of deploying apps other teams are developing, using images that have already been built.
eu-west-2 to our list of supported AWS regions, bringing the total number of supported regions to ten.
One of the advantages of Convox is that it runs entirely in your own AWS account which provides many benefits around security, cost and control. Many of our users also find this advantageous when it comes to compliance and regulations. Some of our customers pointed out that in order to meet their PCI or HIPAA requirements they needed to know exactly what an intruder may or may not have seen during any periods of unauthorized access.
With the recently released AWS integration for Console, we had the opportunity of taking a second look at Rack’s IAM policy. At the beginning rather than trying to lock down a policy for a product that was in continuous development, Rack required administrator access to an AWS account. This wasn’t ideal, but it worked. Now that Rack has stabilized and is being used in production by security-conscious companies and users, and after feedback from said users, we decided to reevaluate the permissions needed.
The goal of the new permissions model is to limit the access Rack has to your AWS account, while still keeping the seamless Convox experience.
From day one, Convox has followed a philosophy of “Integration over Invention”. We believe that everything needed to run apps with impeccable uptime and security already exists through cloud services. The only challenge is integrating all the component services together into a simple and reliable system.
To this end, we’re happy to announce our AWS Integration. Deploying your apps onto modern and private AWS infrastructure services is only a few clicks away.
Convox integrates AWS with your team collaboration services
us-west-1 to our list of supported AWS regions, bringing the total number of supported regions to nine.
At Wave we’re running several Convox racks: production, staging, and multiple “dev” racks. Our production environment obviously needs to be running constantly, but the other environments are mainly used Monday to Friday, between 9am and 8pm.
One of the reasons we’re migrating to AWS is the improved automation that it provides, and we are leveraging that to turn off our dev and staging racks when they’re not in use — saving roughly 66% on our EC2 bills for those environments.
We played around with a few ways to achieve this, but the simplest ended up being AWS Auto Scaling Groups Scheduled Actions. You can set these up using the AWS console, CLI, or another tool like Terraform.
tl;dr: If you’re having trouble creating apps or resources, upgrade your CLI and Rack to the latest versions. Please note that you will not be able to roll back to a previous Rack release older than
20170106053301 after updating.
On the evening of January 5, AWS turned off the ability to create Lambda functions that use the nodejs v0.10 runtime. All future functions must be created on the nodejs 4.3 runtime. More details here.
The CloudFormation template for Convox creates Lambda functions, so the AWS change immediately blocked the ability to install Racks. It also blocked the creation of new apps on existing Racks.
Convox uses Lambda in several ways, including as glue for CloudFormation Custom Resources to provision and manage some AWS resources. Unfortunately, it is not possible to replace these Lambda functions into the new runtime, because of restrictions within CloudFormation.
Going forward, beginning with Rack version
20170106053301, all new Lambda functions use the nodejs 4.3 runtime. Where possible, older Lambda functions will be replaced during Rack updates and app deployments.
We introduced the Lambda runtime as a CloudFormation parameter (CustomTopicRuntime), so we can retain the nodejs 0.10 runtime on the Lambda functions we can’t replace.
Please note that once a Rack has been updated to version
20170106053301 or later it will not be possible to roll it back to versions before 20170106053301. This would require the creation of Lambda functions in the nodejs 0.10 runtime, which is no longer allowed by AWS.
We are working on a plan to get all Convox-managed Lambda functions onto the new runtime. Those changes will be rolled out in future Rack versions.
You don’t want to fight with system dependencies and setup scripts when you have an app to build instead.
If you use Windows, you have it particularly tough. Setting up a good Rails or Django development environment on Windows is harder than on Mac OS or Linux.
To this end, we’re excited to announce a new platform that works with Convox: Windows.
You can now use
convox start on Windows to run your apps.
Linux containers on Windows
I have been experimenting with Amazon’s Elastic File System (EFS) service on a real web app running across multiple instances.
With some effort, I have a file upload and sharing app serving from three web servers sharing a single EFS volume. But due to the performance characteristics of EFS, it wasn’t easy to get going.
EFS does offer shared persistence across many EC2 instances and/or ECS containers
EFS has rigorously documented performance rates and recommended settings for web apps
EFS CloudWatch metrics are very good
SQLite on EFS doesn’t work
Web servers can show high response time and variance due to cold EFS access
Individual file writes have noticable latency
A standard untar operation to EFS is going at a mere 204 kBps
EFS Performance Characteristics
All this leads me to conclude that EFS works ok. I am actually happy with the app deployment after getting it all set up.
However the relatively slow performance could require code changes and an putting a CDN in front of many types of web apps.
Let’s dig into running a horizontally scaled web app on EFS.
Today I’m pleased to announce The Heroku to Convox Guide.
This is a simple, step-by-step guide to migrate a Twelve-Factor Heroku app to Convox.
Many parts of the Heroku platform map directly to Convox. For example Convox offers a
convox deploy command that works just like
git push heroku master.
Other parts are similar, but represent more significant changes to your apps. For example, Convox leverages Docker to package an app dependencies where Heroku uses buildpacks.
This guide explains the differences of the platforms, and walks you through the steps required to migrate an app.
Heroku and Convox Similarities
Start by reading the Introduction.
Werner Vogels re:Invent 2016 Keynote — Code Build Announcement
I’m very excited that AWS filled in this gap in their platform. CodeBuild enables us to further simplify our systems, letting AWS do all the hard work of securing and operating the build step in our software delivery pipeline.
There is a wave of Docker retrospectives out there right now. A couple are negative:
And one is positive:
I’d like to add my positive perspective…
I’m sitting here at Convox looking over thousands of production servers where Docker is a huge part of the success.
How can this be?
HIPAA and PCI both have strict requirements around “encrypting data at rest.”
AWS Key Management Service (KMS), a managed service that offers API access to a Hardware Security Module (HSM), makes encrypting data at rest so easy and cost effective that all systems, not just those with strict compliance needs, should consider using it.
At Convox, we use encryption at rest with KMS for app environment variables, which can contain sensitive secrets like database credentials in a
A KMS key costs $1/month and offers priceless security.
The Twelve Factor App methodology is a series of best practices to follow when building your app to make it easy to deploy to the cloud.
One of the factors is Dev/Prod Parity:
Keep development, staging, and production as similar as possible
It argues that we should aim to reduce gaps between development and production to increase software quality and velocity. Every gap, be it between people, time or computing environments, adds friction.
One way this commonly manifests is that code “works for me!” on your laptop, but doesn’t start on your team mate’s without extra work. The worst case scenario is that this code is deployed and doesn’t work in production, taking the service down.
This friction can add up to significant time over the lifecycle of software. In the above scenario you and your fellow Developers are spending precious time futzing with your laptop environment, and your fellow DevOps engineers are getting paged because the service is down.
Dev/Prod Parity is an aspiration goal in Twelve Factor, but it’s actually possible to achieve with Docker Images and Containers.
The “container era” that recently began is explained by one core belief:
We–the entire community of developers, operators and infrastructure providers–desire “one true way” to package our application workloads.
Packaging is as old as computing itself. The universal
tar command, short for “tape archive,” was introduced in the seventh edition of Unix in 1979 (wikipedia). Many more package formats have been introduced in the subsequent decades.
Analytics for web and mobile rely on events to deliver insights. For chat bots, on the other hand, insights comes from analyzing the content of conversations. Botmetrics accomplishes this with several services—collectors, workers, and a web app for interfacing with the user. These need to be setup separately and configured to work in concert.
Botmetrics Architecture Diagram
Today I’m pleased to announce that we’re adding Workflows, a set of new Convox tools and APIs that enable DevOps best practices like Continuous Delivery and QA Apps.
With Workflows you can now automate the steps between new code and a production deploy. For example, you can build a Continuous Delivery workflow:
Every modern software delivery system has a build step. This is where we:
This artifact is what will be deployed to our production servers.
Building this artifact in an isolated step provides safety. If something goes wrong, say we fail to fetch dependencies from Ruby Gems, the build will fail and there will be no artifact to deploy. This also provides deployment speed. We can build once, which might take a while due to compilation time, then deploy a single artifact to many servers in parallel.
Even though this is a required step, there is no fully-managed build service available on AWS.
To fill in the gap we can design our own simple build service on top on the EC2 Container Service (ECS) and the EC2 Container Registry (ECR).
On Wed. August 16th and Thur. August 17th some users could not access their Racks from the CLI or through Console due to authorization problems. While there were no reports of application downtime during this time this was a total control plane outage for some users.
We apologize for the inconvenience to affected users.
I’d like to explain more what happened and some steps we will be taking to prevent this from happening again.
The biggest change we’d like to implement is the concept of a ‘stable’ and ‘unstable’ release channel. See GitHub Issue #477 – Support Different Endpoints For
convox rack update – to participate in the design of this enhancement and to track progress.
Amazon Web Services (AWS) just announced a new Application Load Balancer (ALB) service.
I spent some time playing with the new service to understand what it offers and to see how it fits into our cloud architecture.
In summary, ALB is a massive improvement over ELB in almost every way.
For $16/month, a single ALB can serve HTTP, HTTP/2 and websockets to up to 10 microservice backends. Now the hardest problems of monitoring and routing traffic containers, which are constantly moving around due to ECS container orchestration, is solved by configuring one ALB correctly.
We no longer need multiple ELBs or internal service discovery software in our microservice application stack to get traffic to our containers. We also no longer need hacks for websockets and HTTP referrers.
ALB has exactly what ECS needs to make our architectures even better.
Support has a massive impact on the total cost of ownership of business systems.
Pick technologies and vendors with little to no support and you will find yourself spending lots of time on the responsibilities of low level problems. This is a big distraction from your business.
Partner with vendors that offer great support and you can delegate problems to them. This will turbo charge your team.
I’d like to share support nightmare we have seen out in the wild and share how the Convox platform and support packages are here to help your team be successful.
A year ago, I described a strategy we use to build Convox: Integration over Invention. This philosophy leads us to build a platform that integrates existing cloud services instead of building custom software.
This strategy has been a resounding success. The Convox platform is trusted by hundreds of companies to run mission critical apps with the strictest requirements for performance, security and scale.
Convox would simply not be this far along if we were building distributed systems software instead of using utility cloud services.
I’d like to re-affirm the big advantages of using services over writing software.
The goal of the cloud is to reduce the time it takes to get an idea, bug fix or security update live.
If you’re using modern DevOps tools like Convox, many of the big time wasters are solved:
An engineer can start and change the app in seconds with
They can ship a patch to GitHub → CircleCI → Production → Slack Notification with Convox Integrations
They can scale the service with
They can apply infrastructure security updates with
convox rack update
They can debug the live app with
They can inspect the app’s private database with
A surprising new challenge appears with this level of agility: knowing who did what for collaboration, management and security purposes.
We just launched the Convox Audit Log to help.
In April 2015 AWS announced a new cloud service: Elastic File System — a Shared File Storage for Amazon EC2. Fast forward to June 28th 2016, more than a year later, and AWS announced that EFS is finally available and production-ready in 3 regions (us-east-1, us-west-2 and eu-west-1).
Let’s give it a try and run a Postgres container on EFS…
The most common question to my Docker For Mac Beta Review is: Why use Docker at all?
The reason to use Docker is for its modern packaging and runtime APIs. The Docker Image and Container APIs have become a de facto standard. Every major computing platform — from OS X to AWS — now has native support for working with Docker Images and Containers.
This happened because Docker presented a modern, API-first approach to distributing and running software at the exact time when we all want cloud providers interoperate better.
This is an unprecedented feat of cooperation across the industry and makes these Docker APIs the obvious best system to target.
We’ve streamlined our SSL configuration and integrated with Amazon Certificate Manager to bring easy, free SSL to everyone. You can generate a free SSL certificate with a single Convox command:
$ convox certs generate *.mydomain.com Requesting certificate... OK, acm-bea0ac39a538
About a year ago I published a blog post called “A Tale of Two Onboardings” that was widely shared. I think people responded to it, because it painted a picture of an easier world where a development machine with Docker installed can boot anything you need for your development workflow with a single command.
This kind of workflow — where you develop inside a running container — has been a promise of Docker for as long as I’ve known about it, but a great developer experience has never been fully realized, until now.
The Convox CLI now has the ability to enhance Docker usage with 2-way code sync. In the latest release,
convox start will instantly sync file changes from your Docker host to your container and from your container to your host. This enables you to:
I feel incredibly lucky to get to use, build and think about open source software every day at Convox. I find it challenging and rewarding building a business around an open source software project.
I’d like to share my thoughts about the open source ecosystem right now, where I see it going in the future, and how Convox navigates the open source and cloud services ecosystem.
Amazon’s EC2 Container Service (ECS) promises a production-ready container management service but setting it up and running apps on it is not without challenges.
I have been working seriously with ECS for close to a year now, building open-source infrastructure automation tools around all things AWS. While building these tools, running my own production services on ECS, and supporting users in doing the same, I am maintaining a running list of the challenges ECS and containers pose.
Through this lens, I can confidently say that ECS is living up to the promise. It offers flexible options for running apps as containers while offering great reliability for service uptime and and formation management. But it is not trivial to get it all working, and it requires a new way of thinking about our infrastructure and how we build our applications.
This release continues the Convox tradition of offering a simple, reliable, private build service.
We are pleased to announce Convox support for private networks inside your Amazon VPCs.
In public mode a Convox Rack launches its instances in a public subnet relying on security groups to keep out unwanted traffic. Outbound traffic from these instances goes directly to the internet.
In private mode a Convox Rack instead launches instances in private subnets and creates NAT Gateways to handle outbound traffic.
Amazon recently made its new ECR (EC2 Container Registry) service generally available in the US East region of AWS. Convox is happy to announce that we’ve integrated with this service. If you’re running the latest release of our Rack software in the US East region, you’ll be using ECR as soon as you deploy your apps. If not, your apps will continue to use the self-hosted registry until ECR becomes generally available in your region.
Today we’re releasing Rack 0.8 with improvements for introspection, privacy, and security. Run
convox update && convox rack update to update your CLI and Rack to the latest version.
We are proud to announce the latest addition to the Convox platform: Grid.
In August we launched Convox Rack which installs a rock-solid, private PaaS into your own AWS account minutes. The response to Rack has been thrilling. People all over the world are using it to deploy and scale their applications. Rack is a powerful tool for sure, but a great platform is about more than just managing apps. It’s also about managing teams and workflows. Enter Grid.
This release includes a new Papertrail service, support for custom SSL certificates, and many other improvements.
This release includes stability improvements, one-off processes, and application visibility. If you’re new to Convox you can follow our Getting Started guide to get up and running in less than 10 minutes. Existing users can upgrade to the latest release with:
convox update && convox rack update
The latest round of development on Convox represents a serious milestone. You can now install Convox in 4 regions around the world, and use the
convox CLI to create apps that are connected to a production-ready RDS database that easily scale to whatever number and amount of memory you desire.
“…as we enjoy great advantages from the inventions of others, we should be glad of an opportunity to serve others by any invention of ours; and this we should do freely and generously.” -Benjamin Franklin
Convox believes in open source. We think that the inner workings of the software you depend on should be knowable. We also believe in sharing our insights with the world. When we leverage each others’ creations we all benefit.
Convox can run and deploy a default Rails app with no manual configuration. Give it a try:
$ rails new myapp && cd myapp $ convox start
Your app will boot and be available on port 5000 of the Docker host.
Joe is a junior developer starting his first real programming job for the photo sharing site SendFace. He arrives at the office Monday morning and meets Susan, his engineering manager. After a quick tour of the office, Susan hands Joe a shiny new Macbook Pro. Joe asks if there are any new developer docs.
“No, unfortunately,” says Susan, “but it’s a pretty standard Rails app. It shouldn’t be too hard to get going. Let me know if you need help.”
Docker is awesome for developing twelve-factor apps.
Docker images and containers offer a contract with the operating system that can be effectively identical between your development environment and production environment.
This is a brief guide for satisfying key factors of twelve-factor with Docker tools. It explains what techniques you can use to develop a archetypical Rails / Postgres / Redis / web / worker application under Docker.