The Convox Blog

The Convox 2.0 Development Platform

As part of the ongoing Convox 2.0 effort, today I’m excited to offer a preview of the brand new Convox development platform.

These new tools turn your development computer into a platform that you interact with exactly like you do a cloud Platform-as-a-Service. This offers true development and production parity, and makes working on microservices easier than ever.

$ cx apps create
creating docs: OK

$ cx start
build   | building: docs
build   | Step 1/4 : FROM golang:1.8.3
build   | Step 2/4 : RUN go get -v github.com/gohugoio/hugo
build   | Step 3/4 : WORKDIR /app
build   | Step 4/4 : COPY . .
build   | running: docker tag 9836064b convox/docs/web:BFGEZTOEXR
build   | build complete
convox  | promoting RIJBGELKDA
convox  | starting: convox.docs.service.web.1
convox  | starting: convox.docs.service.web.2
web     | syncing: . to /app
web     | Listening on port 1313

$ cx services
NAME  ENDPOINT
web   https://web.docs.convox

Here we created, built and deployed an app to our local machine. It is syncing code changes into the web containers and load balancing traffic between them. We can access it at a friendly URL:

It works!It works!

Try it out yourself by following the local development walk through

ALB Is Ready

A load balancer is the most important cloud service. A great load balancing strategy means the front door to your application is always online and routing traffic to backends that are healthy and can scale out to meet the load. No load balancing strategy means a single point of failure and overloaded backends.

Seven months ago I covered the AWS Application Load Balancer (ALB) announcement. The ALB GitHub issue is one of the most popular feature requests for the Convox platform.

It’s clear that the Elastic Load Balancer (ELB) is old and busted, and ALB is the new hotness.

So I’m happy to announce that Convox now has an ALB path. Check out this new Getting Started Guide to learn how to set up a new version of the platform with a single ALB routing traffic to many apps.

This improves deploy speeds. ALB allows us to launch and register many new containers into the load balancer at once, much faster than rolling new containers out one or two at a time behind an ELB.

It also simplifies scaling on all clusters, letting us run our workloads on fewer, bigger instances. This makes autoscaling of clusters easier than ever and could also mean big cost savings by reducing license fees for agents like NewRelic and Datadog.

Ultimately this brings the cost of a small private, production-ready container cluster down to $35/mo for a handful of apps.

Lambda Node.js 0.10 Update Instructions

Is your inbox full of messages like this?

AWS Lambda to end-of-life Node.js v0.10 runtime on April 30, 2017. Please migrate to Node.js v4.3 or v6.10 immediately

Well I’m happy to announce that the Convox fully-automated migration to Node.js v4.3 is available in every region.

Pieces of this have been released over the past weeks and months, so many of you don’t need to take additional action.

But if you’re still getting those emails from AWS, you can:

  1. Update Rack Lambda functions with convox rack update
    (If you are on an older Rack version you might need to run this multiple times to step through any required releases. Continue to update until your Rack is at version 20170330004259 or later.)
  2. Update App Lambda functions by promoting a new release
  3. Update logging Lambda functions by re-creating syslog resources

You can do this from the command line with commands like:

$ convox rack update --wait
$ convox env set FOO=bar --promote --app myapp1
$ convox resources create syslog --url tcp+tls://logs1.papertrailapp.com:12345
$ convox resources delete syslog-1234

You can expect to receive a couple more updates from both AWS and Convox between now and April 30th, to make sure nobody is affected when the Lambda runtime is shut off for good.

Happy automatic infrastructure updates! 🎉🎉

Deploy Bitbucket to AWS in Five Minutes

Most Convox users deploy apps that they develop in-house. A team of developers iterates rapidly on new features and fixes, and they deploy that code multiple times a week, if not multiple times a day. This means building images frequently. When using Convox in this fashion, it’s easy to lose sight of one of the delightful benefits of an image-based platform like Convox: the ease of deploying apps other teams are developing, using images that have already been built.

eu-west-2 now supported by Convox

We’ve added eu-west-2 to our list of supported AWS regions, bringing the total number of supported regions to ten.

Deploying with Buildpacks

Convox now officially supports Heroku buildpacks.

Buildpacks are build scripts maintained over the years by language specialists at Heroku and in the open source community. Buildpacks introspect an application code base, detect what language it is, then run scripts that build an application package with the right language runtime, dependencies and more.

Under the hood, Convox still leverages Docker and a Dockerfile to build images for an app containers. But for apps that don’t have a Dockerfile, Convox now generates one that uses a buildpack. This bypasses all of the work needed to write a good Dockerfile, like picking a base OS, installing the right version of a langage VM, and installing dependencies correctly.

The result is that you can jump straight into deploying an app – no complex “Dockerizing” step necessary.

Let’s give it a try…

Announcing Audit Logs with Session Playback

One of the advantages of Convox is that it runs entirely in your own AWS account which provides many benefits around security, cost and control. Many of our users also find this advantageous when it comes to compliance and regulations. Some of our customers pointed out that in order to meet their PCI or HIPAA requirements they needed to know exactly what an intruder may or may not have seen during any periods of unauthorized access.

IAM Permissions and Rack

With the recently released AWS integration for Console, we had the opportunity of taking a second look at Rack’s IAM policy. At the beginning rather than trying to lock down a policy for a product that was in continuous development, Rack required administrator access to an AWS account. This wasn’t ideal, but it worked. Now that Rack has stabilized and is being used in production by security-conscious companies and users, and after feedback from said users, we decided to reevaluate the permissions needed.

The goal of the new permissions model is to limit the access Rack has to your AWS account, while still keeping the seamless Convox experience.

AWS Integration

From day one, Convox has followed a philosophy of “Integration over Invention”. We believe that everything needed to run apps with impeccable uptime and security already exists through cloud services. The only challenge is integrating all the component services together into a simple and reliable system.

To this end, we’re happy to announce our AWS Integration. Deploying your apps onto modern and private AWS infrastructure services is only a few clicks away.

Convox integrates AWS with your team collaboration servicesConvox integrates AWS with your team collaboration services

us-west-1 now supported by Convox

We’ve added us-west-1 to our list of supported AWS regions, bringing the total number of supported regions to nine.

Convox Tips: Disabling racks on evenings and weekends

At Wave we’re running several Convox racks: production, staging, and multiple “dev” racks. Our production environment obviously needs to be running constantly, but the other environments are mainly used Monday to Friday, between 9am and 8pm.

One of the reasons we’re migrating to AWS is the improved automation that it provides, and we are leveraging that to turn off our dev and staging racks when they’re not in use — saving roughly 66% on our EC2 bills for those environments.

We played around with a few ways to achieve this, but the simplest ended up being AWS Auto Scaling Groups Scheduled Actions. You can set these up using the AWS console, CLI, or another tool like Terraform.

Migrating to the New Lambda Runtime

tl;dr: If you’re having trouble creating apps or resources, upgrade your CLI and Rack to the latest versions. Please note that you will not be able to roll back to a previous Rack release older than 20170106053301 after updating.

The Details

On the evening of January 5, AWS turned off the ability to create Lambda functions that use the nodejs v0.10 runtime. All future functions must be created on the nodejs 4.3 runtime. More details here.

The CloudFormation template for Convox creates Lambda functions, so the AWS change immediately blocked the ability to install Racks. It also blocked the creation of new apps on existing Racks.

Convox uses Lambda in several ways, including as glue for CloudFormation Custom Resources to provision and manage some AWS resources. Unfortunately, it is not possible to replace these Lambda functions into the new runtime, because of restrictions within CloudFormation.

Our Solution

Going forward, beginning with Rack version 20170106053301, all new Lambda functions use the nodejs 4.3 runtime. Where possible, older Lambda functions will be replaced during Rack updates and app deployments.

We introduced the Lambda runtime as a CloudFormation parameter (CustomTopicRuntime), so we can retain the nodejs 0.10 runtime on the Lambda functions we can’t replace.

Please note that once a Rack has been updated to version 20170106053301 or later it will not be possible to roll it back to versions before 20170106053301. This would require the creation of Lambda functions in the nodejs 0.10 runtime, which is no longer allowed by AWS.

We are working on a plan to get all Convox-managed Lambda functions onto the new runtime. Those changes will be rolled out in future Rack versions.

Windows Support

You don’t want to fight with system dependencies and setup scripts when you have an app to build instead.

If you use Windows, you have it particularly tough. Setting up a good Rails or Django development environment on Windows is harder than on Mac OS or Linux.

To this end, we’re excited to announce a new platform that works with Convox: Windows.

You can now use convox start on Windows to run your apps.

Linux containers on WindowsLinux containers on Windows

The Challenges of EFS

I have been experimenting with Amazon’s Elastic File System (EFS) service on a real web app running across multiple instances.

With some effort, I have a file upload and sharing app serving from three web servers sharing a single EFS volume. But due to the performance characteristics of EFS, it wasn’t easy to get going.

The good:

The bad:

  • SQLite on EFS doesn’t work

  • Web servers can show high response time and variance due to cold EFS access

The ugly:

  • Individual file writes have noticable latency

  • A standard untar operation to EFS is going at a mere 204 kBps

EFS Performance CharacteristicsEFS Performance Characteristics

All this leads me to conclude that EFS works ok. I am actually happy with the app deployment after getting it all set up.

However the relatively slow performance could require code changes and an putting a CDN in front of many types of web apps.

Let’s dig into running a horizontally scaled web app on EFS.

The Heroku to Convox Guide

Today I’m pleased to announce The Heroku to Convox Guide.

This is a simple, step-by-step guide to migrate a Twelve-Factor Heroku app to Convox.

Many parts of the Heroku platform map directly to Convox. For example Convox offers a convox deploy command that works just like git push heroku master.

Other parts are similar, but represent more significant changes to your apps. For example, Convox leverages Docker to package an app dependencies where Heroku uses buildpacks.

This guide explains the differences of the platforms, and walks you through the steps required to migrate an app.

Heroku and Convox SimilaritiesHeroku and Convox Similarities

Start by reading the Introduction.

If you have any questions or feedback, reach out on the Convox Slack or GitHub.

AWS CodeBuild

Cheaper, Faster, Safer Builds

A few months ago I wrote AWS Missing Parts: Build Service. At the 2016 re:Invent conference, AWS launched CodeBuild, a service to “build and test code with continuous scaling.”

[Werner Vogels re:Invent 2016 Keynote](https://www.youtube.com/watch?v=ZDScBNahsL4&t=34m48s) — Code Build AnnouncementWerner Vogels re:Invent 2016 Keynote — Code Build Announcement

I’m very excited that AWS filled in this gap in their platform. CodeBuild enables us to further simplify our systems, letting AWS do all the hard work of securing and operating the build step in our software delivery pipeline.

Docker In Production for 18 Months

There is a wave of Docker retrospectives out there right now. A couple are negative:

And one is positive:

I’d like to add my positive perspective…

I’m sitting here at Convox looking over thousands of production servers where Docker is a huge part of the success.

How can this be?

Encryption at Rest

Table Stakes for Compliance

HIPAA and PCI both have strict requirements around “encrypting data at rest.”

AWS Key Management Service (KMS), a managed service that offers API access to a Hardware Security Module (HSM), makes encrypting data at rest so easy and cost effective that all systems, not just those with strict compliance needs, should consider using it.

At Convox, we use encryption at rest with KMS for app environment variables, which can contain sensitive secrets like database credentials in a DATABASE_URL variable.

A KMS key costs $1/month and offers priceless security.

Dev → Test → Prod Parity with Docker

The Twelve Factor App methodology is a series of best practices to follow when building your app to make it easy to deploy to the cloud.

One of the factors is Dev/Prod Parity:

Keep development, staging, and production as similar as possible

It argues that we should aim to reduce gaps between development and production to increase software quality and velocity. Every gap, be it between people, time or computing environments, adds friction.

One way this commonly manifests is that code “works for me!” on your laptop, but doesn’t start on your team mate’s without extra work. The worst case scenario is that this code is deployed and doesn’t work in production, taking the service down.

This friction can add up to significant time over the lifecycle of software. In the above scenario you and your fellow Developers are spending precious time futzing with your laptop environment, and your fellow DevOps engineers are getting paged because the service is down.

Dev/Prod Parity is an aspiration goal in Twelve Factor, but it’s actually possible to achieve with Docker Images and Containers.

Image Workflows

The “container era” that recently began is explained by one core belief:

We–the entire community of developers, operators and infrastructure providers–desire “one true way” to package our application workloads.

Packaging is as old as computing itself. The universal tar command, short for “tape archive,” was introduced in the seventh edition of Unix in 1979 (wikipedia). Many more package formats have been introduced in the subsequent decades.

Dorothy Whitaker works in the National Oceanographic Data Center (NODC) magnetic tape libraryDorothy Whitaker works in the National Oceanographic Data Center (NODC) magnetic tape library

Botmetrics Enterprise Deployment on AWS with Convox

Botmetrics is an open source package and a hosted service for measuring and growing Facebook Messenger, Kik, Slack and in-app messaging bots.

Analytics for web and mobile rely on events to deliver insights. For chat bots, on the other hand, insights comes from analyzing the content of conversations. Botmetrics accomplishes this with several services—collectors, workers, and a web app for interfacing with the user. These need to be setup separately and configured to work in concert.

Botmetrics Architecture DiagramBotmetrics Architecture Diagram

Announcing Workflows

Today I’m pleased to announce that we’re adding Workflows, a set of new Convox tools and APIs that enable DevOps best practices like Continuous Delivery and QA Apps.

With Workflows you can now automate the steps between new code and a production deploy. For example, you can build a Continuous Delivery workflow:

  1. Push code to a GitHub branch
  2. Trigger TravisCI tests to run and update GitHub branch status to “green”
  3. Merge code to GitHub master
  4. Trigger Convox to Build new Docker Images and release it to production

AWS Missing Parts: Build Service

Every modern software delivery system has a build step. This is where we:

  • Clone app source code
  • Download the app dependencies
  • Zip everything together
  • Upload the artifact to a storage service

This artifact is what will be deployed to our production servers.

Building this artifact in an isolated step provides safety. If something goes wrong, say we fail to fetch dependencies from Ruby Gems, the build will fail and there will be no artifact to deploy. This also provides deployment speed. We can build once, which might take a while due to compilation time, then deploy a single artifact to many servers in parallel.

Even though this is a required step, there is no fully-managed build service available on AWS.

To fill in the gap we can design our own simple build service on top on the EC2 Container Service (ECS) and the EC2 Container Registry (ECR).

Rack Authentication Postmortem

On Wed. August 16th and Thur. August 17th some users could not access their Racks from the CLI or through Console due to authorization problems. While there were no reports of application downtime during this time this was a total control plane outage for some users.

We apologize for the inconvenience to affected users.

I’d like to explain more what happened and some steps we will be taking to prevent this from happening again.

The biggest change we’d like to implement is the concept of a ‘stable’ and ‘unstable’ release channel. See GitHub Issue #477Support Different Endpoints For convox rack update – to participate in the design of this enhancement and to track progress.

AWS ALB — The Container and Microservice Load Balancer

Amazon Web Services (AWS) just announced a new Application Load Balancer (ALB) service.

I spent some time playing with the new service to understand what it offers and to see how it fits into our cloud architecture.

In summary, ALB is a massive improvement over ELB in almost every way.

For $16/month, a single ALB can serve HTTP, HTTP/2 and websockets to up to 10 microservice backends. Now the hardest problems of monitoring and routing traffic containers, which are constantly moving around due to ECS container orchestration, is solved by configuring one ALB correctly.

We no longer need multiple ELBs or internal service discovery software in our microservice application stack to get traffic to our containers. We also no longer need hacks for websockets and HTTP referrers.

ALB has exactly what ECS needs to make our architectures even better.

Who You Gonna Call?

Support has a massive impact on the total cost of ownership of business systems.

Pick technologies and vendors with little to no support and you will find yourself spending lots of time on the responsibilities of low level problems. This is a big distraction from your business.

Partner with vendors that offer great support and you can delegate problems to them. This will turbo charge your team.

I’d like to share support nightmare we have seen out in the wild and share how the Convox platform and support packages are here to help your team be successful.

Services over Software

A year ago, I described a strategy we use to build Convox: Integration over Invention. This philosophy leads us to build a platform that integrates existing cloud services instead of building custom software.

This strategy has been a resounding success. The Convox platform is trusted by hundreds of companies to run mission critical apps with the strictest requirements for performance, security and scale.

Convox would simply not be this far along if we were building distributed systems software instead of using utility cloud services.

I’d like to re-affirm the big advantages of using services over writing software.

Auditing Your Team's Agility

The goal of the cloud is to reduce the time it takes to get an idea, bug fix or security update live.

If you’re using modern DevOps tools like Convox, many of the big time wasters are solved:

  • An engineer can start and change the app in seconds with convox start

  • They can ship a patch to GitHub → CircleCI → Production → Slack Notification with Convox Integrations

  • They can scale the service with convox scale

  • They can apply infrastructure security updates with convox rack update

  • They can debug the live app with convox exec

  • They can inspect the app’s private database with convox proxy

A surprising new challenge appears with this level of agility: knowing who did what for collaboration, management and security purposes.

We just launched the Convox Audit Log to help.

AWS EFS — The Container Friendly File System

In April 2015 AWS announced a new cloud service: Elastic File System — a Shared File Storage for Amazon EC2. Fast forward to June 28th 2016, more than a year later, and AWS announced that EFS is finally available and production-ready in 3 regions (us-east-1, us-west-2 and eu-west-1).

A day later, Convox’s David Dollar opened a pull request integrating EFS into Convox Rack, the open-source cloud management platform built on top of AWS.

Let’s give it a try and run a Postgres container on EFS…

Why Docker? The Image API is Everywhere

The most common question to my Docker For Mac Beta Review is: Why use Docker at all?

The reason to use Docker is for its modern packaging and runtime APIs. The Docker Image and Container APIs have become a de facto standard. Every major computing platform — from OS X to AWS — now has native support for working with Docker Images and Containers.

This happened because Docker presented a modern, API-first approach to distributing and running software at the exact time when we all want cloud providers interoperate better.

This is an unprecedented feat of cooperation across the industry and makes these Docker APIs the obvious best system to target.

DockerCon 2016 Recap

Convox’s Noah Zoschke and David Dollar attended this year’s DockerCon in Seattle, WA.

Easy SSL and Free Certificates

We’ve streamlined our SSL configuration and integrated with Amazon Certificate Manager to bring easy, free SSL to everyone. You can generate a free SSL certificate with a single Convox command:

$ convox certs generate *.mydomain.com
Requesting certificate... OK, acm-bea0ac39a538

True Containerized Development with Convox Code Sync

About a year ago I published a blog post called “A Tale of Two Onboardings” that was widely shared. I think people responded to it, because it painted a picture of an easier world where a development machine with Docker installed can boot anything you need for your development workflow with a single command.

This kind of workflow — where you develop inside a running container — has been a promise of Docker for as long as I’ve known about it, but a great developer experience has never been fully realized, until now.

The Convox CLI now has the ability to enhance Docker usage with 2-way code sync. In the latest release, convox start will instantly sync file changes from your Docker host to your container and from your container to your host. This enables you to:

  • Edit files on your host using your favorite text editor and see file changes reflected in the container immediately.
  • Run commands that generate code changes on the container and have that code appear on your host where you can commit it to version control.

The Open Source Evolution

I feel incredibly lucky to get to use, build and think about open source software every day at Convox. I find it challenging and rewarding building a business around an open source software project.

I’d like to share my thoughts about the open source ecosystem right now, where I see it going in the future, and how Convox navigates the open source and cloud services ecosystem.

The Seven Biggest Challenges of Deployment to ECS

Amazon’s EC2 Container Service (ECS) promises a production-ready container management service but setting it up and running apps on it is not without challenges.

I have been working seriously with ECS for close to a year now, building open-source infrastructure automation tools around all things AWS. While building these tools, running my own production services on ECS, and supporting users in doing the same, I am maintaining a running list of the challenges ECS and containers pose.

Through this lens, I can confidently say that ECS is living up to the promise. It offers flexible options for running apps as containers while offering great reliability for service uptime and and formation management. But it is not trivial to get it all working, and it requires a new way of thinking about our infrastructure and how we build our applications.

Scheduled Tasks

Over the past week the team (and in particular my friend Matt) has been working on a feature I’m excited to share with you. Convox now natively supports Scheduled tasks. This allows you to specify units of work, ran at any interval you desire with out writing any extra code.

Massive Build Improvements

This release continues the Convox tradition of offering a simple, reliable, private build service.

Private Networking

We are pleased to announce Convox support for private networks inside your Amazon VPCs.

In public mode a Convox Rack launches its instances in a public subnet relying on security groups to keep out unwanted traffic. Outbound traffic from these instances goes directly to the internet.

In private mode a Convox Rack instead launches instances in private subnets and creates NAT Gateways to handle outbound traffic.

Convox Integrates with Amazon EC2 Container Registry

Amazon recently made its new ECR (EC2 Container Registry) service generally available in the US East region of AWS. Convox is happy to announce that we’ve integrated with this service. If you’re running the latest release of our Rack software in the US East region, you’ll be using ECR as soon as you deploy your apps. If not, your apps will continue to use the self-hosted registry until ECR becomes generally available in your region.

Rack 0.8: Introspection, Privacy, and Security

Today we’re releasing Rack 0.8 with improvements for introspection, privacy, and security. Run convox update && convox rack update to update your CLI and Rack to the latest version.

Grid: Devops Services for Engineering Teams

We are proud to announce the latest addition to the Convox platform: Grid.

In August we launched Convox Rack which installs a rock-solid, private PaaS into your own AWS account minutes. The response to Rack has been thrilling. People all over the world are using it to deploy and scale their applications. Rack is a powerful tool for sure, but a great platform is about more than just managing apps. It’s also about managing teams and workflows. Enter Grid.

Rack 0.7: Papertrail and SSL

This release includes a new Papertrail service, support for custom SSL certificates, and many other improvements.

Rack 0.6: One-Off Processes, Visibility, and Stability

This release includes stability improvements, one-off processes, and application visibility. If you’re new to Convox you can follow our Getting Started guide to get up and running in less than 10 minutes. Existing users can upgrade to the latest release with: convox update && convox rack update

Convox 0.5: Multi-region VPCs, Process Scaling, and Databases!

The latest round of development on Convox represents a serious milestone. You can now install Convox in 4 regions around the world, and use the convox CLI to create apps that are connected to a production-ready RDS database that easily scale to whatever number and amount of memory you desire.

Integration over Invention

In 10 minutes Convox installs a system that takes over the management of your AWS servers, networking, and data and lets you deploy applications to the Internet with a single command.

Open Source

“…as we enjoy great advantages from the inventions of others, we should be glad of an opportunity to serve others by any invention of ours; and this we should do freely and generously.” -Benjamin Franklin

Convox believes in open source. We think that the inner workings of the software you depend on should be knowable. We also believe in sharing our insights with the world. When we leverage each others’ creations we all benefit.

Zero Configuration Rails

Convox can run and deploy a default Rails app with no manual configuration. Give it a try:

$ rails new myapp && cd myapp  
$ convox start

Your app will boot and be available on port 5000 of the Docker host.

A Tale of Two Onboardings

Joe is a junior developer starting his first real programming job for the photo sharing site SendFace. He arrives at the office Monday morning and meets Susan, his engineering manager. After a quick tour of the office, Susan hands Joe a shiny new Macbook Pro. Joe asks if there are any new developer docs.

“No, unfortunately,” says Susan, “but it’s a pretty standard Rails app. It shouldn’t be too hard to get going. Let me know if you need help.”

Modern Twelve-Factor Apps with Docker

Docker is awesome for developing twelve-factor apps.

Dockerfile and docker-compose.yml are emerging standards for declaring everything about your service in the codebase: dependencies, environment, ports, multiple process types, and backing services.

Docker images and containers offer a contract with the operating system that can be effectively identical between your development environment and production environment.

This is a brief guide for satisfying key factors of twelve-factor with Docker tools. It explains what techniques you can use to develop a archetypical Rails / Postgres / Redis / web / worker application under Docker.