How B12 Hosts Websites at Scale

At B12, we offer customers a new way to create and manage their website. Customers provide us with some basic business information, and in 60 seconds, our algorithms automatically draft a new website from content and aesthetic preferences that we crawl and classify from their existing web presence. Using an open-sourced project we built, Orchestra, we recruit a design team from our expert network to add nuance to the algorithmically generated website and follow up with personalized monthly update recommendations. In this post, we describe the infrastructure we recently unveiled to deploy and host customer websites at high scale and low latency worldwide.

22 May 2017 · 9 min read

by Josh Blum and Adam Marcus


Goals for our new infrastructure

In building our new hosting infrastructure, we had several goals in mind:

  • Infrastructure Reliability. While the availability requirements for all of our services are ambitious, the expectations for our customers’ websites is that they only go down when significant portions of the web are affected. To accomplish this, our hosting infrastructure relies on two relatively bulletproof pieces of internet architecture: Amazon’s Simple Storage Service (S3) and Fastly’s Content Delivery Network (CDN).

  • Globally Distributed Low Latency. Our customers’ customers come from all over the world. We use Fastly’s CDN to cache our customers’ websites in several geographically distributed locations, dramatically decreasing page load times.

  • Low Update Latency. There are two updates to customer websites that should appear near-instantaneously. First, if a customer or designer updates a website, those changes should appear the next time they view their website. Second, if a customer or designer adds a new domain to their website, we should route requests from that domain to the appropriate website as quickly as possible. A fine-grained use of Fastly’s cache invalidation mechanisms allows us to make these updates visible to customers with sub-second latency.

  • SSL Support. Being a good citizen of the web means supporting SSL certificates on all customer websites.

  • Scale. While you can never predict where your scalability bottlenecks will be across orders of magnitude, we’re pretty confident our infrastructure will scale to hundreds of thousands of customer websites without any significant re-architecture.


Architecture Overview

We host all of our customer websites through a single Fastly service. Customers add a CNAME record to dns.b12.io (which, in turn, has a CNAME to the service that Fastly runs for us at pin-b12.map.fastly.net). All records with a CNAME record to dns.b12.io are routed by Fastly to our service, which is configured to retrieve website content from our hosting storage in S3. Fastly initially retrieves data from S3 directly and subsequently caches it in its geographically distributed edge cache servers.

Fastly allows you to programmatically specify the behavior of your service using the Varnish Cache Language (VCL). We take advantage of VCL restarts to first query our configuration API, providing our Fastly service with the metadata necessary to find the storage location of a customer’s website. The request then restarts with this configuration information, looking up and returning the requested page content to the end user. Below we outline the path of a request from a user’s browser through our infrastructure:


Configuration API on S3

When our Fastly service receives a request, we have to retrieve configuration information about the customer associated with the domain, such as the storage location in S3, if SSL is supported on the domain, and any customer-specific redirects. We store this data in a single S3 bucket, which stores key-value pairs as custom HTTP headers for our Fastly service to read. This means that the API has the availability and durability guarantees (and sadly, the consistency guarantees) that S3 provides, isolating our customers from any failures in the rest of our infrastructure.

The S3 bucket for the configuration API has the following layout to store configurations for our customers. We have sub-buckets for our different server environments (such as dev or production) to allow for normal development on the same infrastructure as production. Each server environment sub-bucket also holds different website environments (e.g., our customers’ staging and production environments) since our customers and designers iterate on their designs before publishing them.


config-bucket/
    dev/
        [...]
    production/
        staging/
            [...]
        production/
             joshblum.b12sites.com/
                 <shared-secret> # shh
                     b12-hosting-path: <joshblum-uid>
                     b12-force-ssl: 1
                     b12-redirect-from-0: about.html
                     b12-redirect-to-0: index.html    
                     b12-redirect-from-n: team-page.html
                     b12-redirect-to-n: team.html        
                     [...]
             [...]

In the above example we show the configuration for the domain joshblum.b12sites.com. When a request comes to our Fastly service from joshblum.b12sites.com, we programmatically build the URL to get the configuration API metadata. <shared-secret> is a secret shared between the API and our Fastly service. The data we store isn’t sensitive and is read-only, but we like that the obscure secret makes it harder to explore our domain configuration data ;).

A Fastly service can only read HTTP headers or request metadata during processing (it can’t read the request content), so we encode our configuration data as custom HTTP headers. The b12-hosting-path specifies the path to the website content for the particular customer in our hosting service (described below). b12-force-ssl specifies whether SSL is available for the given domain. We also store a set of redirects (e.g., b12-redirect-from-0, b12-redirect-to-0) if a customer wishes to set up aliases for their pages.

Clothes

Because apex domains (domain.com) cannot point to a CNAME, we have a www-izer service called clothes with a static IP that customers point to with an A record. The service blindly redirects any request from domain.com to www.domain.com, essentially clothing the “naked” apex domain, which has the correct CNAME to dns.b12.io.

Configuration Updates

Whenever a configuration change is made, we must purge stale content from the Fastly cache. We use Surrogate Keys to tag a response in the Fastly cache, allowing us to remove all content with a certain tag. If SSL is added to a single domain, for example, we purge just that domain’s configuration surrogate keys. If a customer’s redirect rules are updated, we purge all domains associated with the customer since this affects all of their domains.

Fastly Service Configuration

Our Fastly service is configured with the following features using custom Fastly VCL. We describe each feature and provide a small code snippet as a toy example for our implementation:

Index Rewriting:

  • At the root of a customer website (/), we automatically redirect the request to index.html.
set req.url = regsub(req.url, "^/$", "/index.html");

Customer-Specific Redirects

  • Customers can specify redirects for their website to internally redirect from one page to another. (Note: We use the Django Template language to write our VCL config for convenience, so the syntax isn’t 100% VCL here. More on that later!)
  
  {% for REDIRECT_FROM_i, REDIRECT_TO_i in B12_REDIRECTS %}
  if (req.http.{{REDIRECT_FROM_i}} && req.http.{{REDIRECT_TO_i}} &&
    req.url == req.http.{{ REDIRECT_FROM_i }}) {
    set req.url = req.http.{{ REDIRECT_TO_i }};
    error 802 "Force Redirect";
  }
  {% endfor %}
  

Force HTTPS

  • For all domains that have a SSL certificate (currently just *.b12sites.com), we force every request to HTTPs.
if (!req.http.Fastly-SSL && req.http. == "1") { 
  error 801 "Force SSL";  
}

Page Rewriting

  • URLs that do not contain a dot in them are rewritten with a .html at the end, so if someone visited the page /about we would automatically rewrite the url to /about.html.
set req.url = regsub(req.url, "^([^\.]+?)([\/]?)$", "\1\.html");

404 Redirect

  • If S3 returns a 404 or 403 for a nonexistent page, we redirect the browser to a customer branded 404.html page.
# If we get a 404 or 403 from the backend, trigger an error so we
# can redirect to a custom 404 page.  
if (beresp.status == 404 || beresp.status == 403) {     
  error 900 "Not Found";  
}

Website Storage & Hosting on S3

All of our customer websites are stored in a single S3 bucket and served through Fastly . For subdomains that we create (e.g., *.b12sites.com), we use Route53 record sets, and support any other DNS provider for customers that control their own custom domains.

S3 Storage

Our customer websites are hosted in a single S3 bucket which contains sub-buckets for our server and website environments. Below is an example of the bucket layout:


hosting-bucket/
    dev/
        [...]
    production/
        staging/
            [...]
        production/
             <joshblum-uid>/
                index.html
                about.html
                img/[...]
             [...]

Content Updates

Whenever new content is pushed to S3, we purge all potentially stale content in Fastly. Similar to configuration changes, each website has a Surrogate Key with which its content is tagged and purged.


Lessons Learned

Building our new hosting infrastructure has been a wild ride. Along the way, we’ve picked up a few takeaways that apply more broadly.

Similar Architecture Across Environments

From the start, we designed our architecture to allow for our normal development workflows, namely working in a dev environment, testing large changes in a staging environment and ultimately releasing new features onto production. It was important that the infrastructure have as few differences as possible between the environments to allow for testing and to minimize production related bugs.

Automated Testing

During the development of our Fastly service, we realized it was key to have automated tests to verify the features we wanted to support. We have a testing service in Fastly that mirrors our production service and allows us to test new features as we develop them without downtime, regressions, or misconfigurations errors for our customers.

Templating Fastly Configs

Our Fastly configuration is written in custom VCL, but we use the Django template language to minimize errors between shared variable names on the external services we use. We also leverage things such as looping to make our lives easier when writing the configuration:

  
    {% for REDIRECT_FROM_i, REDIRECT_TO_i in B12_REDIRECTS %}      
    if (resp.http.{{REDIRECT_FROM_i}} && resp.http.{{REDIRECT_TO_i}}) {        
      set req.http.{{REDIRECT_FROM_i}} = resp.http.{{REDIRECT_FROM_i}};        
      set req.http.{{REDIRECT_TO_i}} = resp.http.{{REDIRECT_TO_i}};      }    
    {% endfor %}
  

Configuration Service Scalability & Resiliency

Our architecture differs from a traditional Fastly setup since we must be able to add and update customers dynamically rather than hosting a fixed number of customers. We looked into several alternatives before settling on a separate configuration API, but no solution would easily reach the scale we expected. We considered creating a pool of Fastly services (normally limited to ~1000 domains each) in conjunction with a chain of Fastly’s Edge Dictionaries (limited to ~1000 entries, with size limits on the key/value lengths) to store customer metadata. Both of these solutions were limited in their scalability without complex application logic, which we wanted to avoid. An external configuration service that could be queried for updates instead of hard-coded values within the service VCL fit our needs better.

We initially hosted the configuration API on our own webservers, but quickly decided for resiliency we should move it to S3, which “never” goes down. Joking aside, having our infrastructure entirely on a highly available, scalable, and relatively low-latency storage service minimizes potential downtime.

Amazon Web Services Gotchas

During the migration to S3, we realized custom HTTP headers on S3 are limited to 2kb so we dropped the human readable names in our examples for single letter headers.

We also found out the hard way that the number of records sets in a Route53 zone is limited to 10k initially during extended automated testing of our algorithmic website generation pipeline.


Next Steps

To make our web hosting solution truly resilient and scalable, we want to work on having multi-region redundancy in S3 or a similar cloud storage service coupled with DNS Failover Routing Policy for both our website storage and configuration API. The folks at Webflow have done a great job outlining steps to reduce downtime of their websites by implementing these features.

Additionally, we plan to integrate with services like Let’s Encrypt to automatically generate SSL certificates for our customers’ custom domains.


Conclusion

Fast and reliable web hosting is a small but crucial piece of the customer experience at B12. We hope that with our new infrastructure, all of our customers and their customers can enjoy websites at blazing fast speeds wherever they are in the world!

Many thanks to the team at Fastly for their helpful support along the way. Thanks also to the folks at Hashicorp, whose blog post on their Fastly setup helped us numerous times. If you find our work interesting, come work with us :).

Read next

See all
Hello World

Hello World

The world of work is changing. Machines are getting smarter and jobs are often defined in minutes, not years. Technology is changing how we work an...

Read now
Introducing Orchestra

Introducing Orchestra

November will mark the 10-year anniversary of Amazon’s Mechanical Turk, which introduced the concept of a microtask to the world.

Read now
Introducing B12

Introducing B12

We are thrilled to share today that B12 (formerly known as Unlimited Labs) has raised a $12.4M Series A. We’ll start by telling you about what we a...

Read now
Introducing Kinase, B12’s Web Content Labeling Framework

Introducing Kinase, B12’s Web Content Labeling Framework

The web contains a variety of content that is easily accessible to humans but ill-formed for machines. Many organizations have spent countless hour...

Read now
This website uses cookies to ensure you get the best browsing experience.  Learn more
I agree