How B12 Hosts Websites at Scale
At B12, we offer customers a new way to create and manage their website. Customers provide us with some basic business information, and in 60 seconds, our algorithms automatically draft a new website from content and aesthetic preferences that we crawl and classify from their existing web presence. Using an open-sourced project we built, Orchestra, we recruit a design team from our expert network to add nuance to the algorithmically generated website and follow up with personalized monthly update recommendations. In this post, we describe the infrastructure we recently unveiled to deploy and host customer websites at high scale and low latency worldwide.
Goals for our new infrastructure
In building our new hosting infrastructure, we had several goals in mind:
-
Infrastructure Reliability. While the availability requirements for all of our services are ambitious, the expectations for our customers’ websites is that they only go down when significant portions of the web are affected. To accomplish this, our hosting infrastructure relies on two relatively bulletproof pieces of internet architecture: Amazon’s Simple Storage Service (S3) and Fastly’s Content Delivery Network (CDN).
-
Globally Distributed Low Latency. Our customers’ customers come from all over the world. We use Fastly’s CDN to cache our customers’ websites in several geographically distributed locations, dramatically decreasing page load times.
-
Low Update Latency. There are two updates to customer websites that should appear near-instantaneously. First, if a customer or designer updates a website, those changes should appear the next time they view their website. Second, if a customer or designer adds a new domain to their website, we should route requests from that domain to the appropriate website as quickly as possible. A fine-grained use of Fastly’s cache invalidation mechanisms allows us to make these updates visible to customers with sub-second latency.
-
SSL Support. Being a good citizen of the web means supporting SSL certificates on all customer websites.
-
Scale. While you can never predict where your scalability bottlenecks will be across orders of magnitude, we’re pretty confident our infrastructure will scale to hundreds of thousands of customer websites without any significant re-architecture.
Architecture Overview
dns.b12.io
(which, in turn, has a CNAME to the service that Fastly runs for us at pin-b12.map.fastly.net
). All records with a CNAME
record to dns.b12.io
are routed by Fastly to our service, which is configured to retrieve website content from our hosting storage in S3. Fastly initially retrieves data from S3 directly and subsequently caches it in its geographically distributed edge cache servers.Configuration API on S3
The S3 bucket for the configuration API has the following layout to store configurations for our customers. We have sub-buckets for our different server environments (such as dev
or production
) to allow for normal development on the same infrastructure as production. Each server environment sub-bucket also holds different website environments (e.g., our customers’ staging
and production
environments) since our customers and designers iterate on their designs before publishing them.
config-bucket/
dev/
[...]
production/
staging/
[...]
production/
joshblum.b12sites.com/
<shared-secret> # shh
b12-hosting-path: <joshblum-uid>
b12-force-ssl: 1
b12-redirect-from-0: about.html
b12-redirect-to-0: index.html
b12-redirect-from-n: team-page.html
b12-redirect-to-n: team.html
[...]
[...]
In the above example we show the configuration for the domain joshblum.b12sites.com
. When a request comes to our Fastly service from joshblum.b12sites.com
, we programmatically build the URL to get the configuration API metadata. <shared-secret>
is a secret shared between the API and our Fastly service. The data we store isn’t sensitive and is read-only, but we like that the obscure secret makes it harder to explore our domain configuration data ;).
A Fastly service can only read HTTP headers or request metadata during processing (it can’t read the request content), so we encode our configuration data as custom HTTP headers. The b12-hosting-path
specifies the path to the website content for the particular customer in our hosting service (described below). b12-force-ssl
specifies whether SSL is available for the given domain. We also store a set of redirects (e.g., b12-redirect-from-0
, b12-redirect-to-0
) if a customer wishes to set up aliases for their pages.
Clothes
domain.com
) cannot point to a CNAME, we have a www-izer service called clothes
with a static IP that customers point to with an A
record. The service blindly redirects any request from domain.com
to www.domain.com
, essentially clothing the “naked” apex domain, which has the correct CNAME
to dns.b12.io
.Configuration Updates
Fastly Service Configuration
Index Rewriting:
- At the root of a customer website (
/
), we automatically redirect the request toindex.html
.
set req.url = regsub(req.url, "^/$", "/index.html");
Customer-Specific Redirects
- Customers can specify redirects for their website to internally redirect from one page to another. (Note: We use the Django Template language to write our VCL config for convenience, so the syntax isn’t 100% VCL here. More on that later!)
{% raw %}
{% for REDIRECT_FROM_i, REDIRECT_TO_i in B12_REDIRECTS %}
if (req.http.{{REDIRECT_FROM_i}} && req.http.{{REDIRECT_TO_i}} &&
req.url == req.http.{{ REDIRECT_FROM_i }}) {
set req.url = req.http.{{ REDIRECT_TO_i }};
error 802 "Force Redirect";
}
{% endfor %}
{% endraw %}
Force HTTPS
- For all domains that have a SSL certificate (currently just
*.b12sites.com
), we force every request to HTTPs.
if (!req.http.Fastly-SSL && req.http.{{ B12_FORCE_SSL }} == "1") {
error 801 "Force SSL";
}
Page Rewriting
- URLs that do not contain a dot in them are rewritten with a
.html
at the end, so if someone visited the page/about
we would automatically rewrite the url to/about.html
.
set req.url = regsub(req.url, "^([^\.]+?)([\/]?)$", "\1\.html");
404 Redirect
- If S3 returns a
404
or403
for a nonexistent page, we redirect the browser to a customer branded404.html
page.
# If we get a 404 or 403 from the backend, trigger an error so we
# can redirect to a custom 404 page.
if (beresp.status == 404 || beresp.status == 403) {
error 900 "Not Found";
}
Website Storage & Hosting on S3
*.b12sites.com
), we use Route53 record sets, and support any other DNS provider for customers that control their own custom domains.S3 Storage
Our customer websites are hosted in a single S3 bucket which contains sub-buckets for our server and website environments. Below is an example of the bucket layout:
hosting-bucket/
dev/
[...]
production/
staging/
[...]
production/
<joshblum-uid>/
index.html
about.html
img/[...]
[...]
Content Updates
Whenever new content is pushed to S3, we purge all potentially stale content in Fastly. Similar to configuration changes, each website has a Surrogate Key with which its content is tagged and purged.
Lessons Learned
Building our new hosting infrastructure has been a wild ride. Along the way, we’ve picked up a few takeaways that apply more broadly.
Similar Architecture Across Environments
From the start, we designed our architecture to allow for our normal development workflows, namely working in a dev environment, testing large changes in a staging environment and ultimately releasing new features onto production. It was important that the infrastructure have as few differences as possible between the environments to allow for testing and to minimize production related bugs.
Automated Testing
During the development of our Fastly service, we realized it was key to have automated tests to verify the features we wanted to support. We have a testing service in Fastly that mirrors our production service and allows us to test new features as we develop them without downtime, regressions, or misconfigurations errors for our customers.
Templating Fastly Configs
{% raw %}
{% for REDIRECT_FROM_i, REDIRECT_TO_i in B12_REDIRECTS %}
if (resp.http.{{REDIRECT_FROM_i}} && resp.http.{{REDIRECT_TO_i}}) {
set req.http.{{REDIRECT_FROM_i}} = resp.http.{{REDIRECT_FROM_i}};
set req.http.{{REDIRECT_TO_i}} = resp.http.{{REDIRECT_TO_i}}; }
{% endfor %}
{% endraw %}
Configuration Service Scalability & Resiliency
Amazon Web Services Gotchas
Next Steps
Conclusion
Fast and reliable web hosting is a small but crucial piece of the customer experience at B12. We hope that with our new infrastructure, all of our customers and their customers can enjoy websites at blazing fast speeds wherever they are in the world!
Read next
See allOpenAI features B12’s Website Generator in the GPT Store
Create and customize your website directly in ChatGPT using DALL-E
Read nowLightning-fast website creation with B12's generative AI
Generate your tailored website in one click. To make edits, regenerate any section instantly.
Read now