LifestyleWe handle 80TB and 5M page views a month...

We handle 80TB and 5M page views a month for under $400

-

- Advertisment -
Note: All stats and figures below are true only at the time of writing, and will certainly change over time.

Running a massively popular website and asset resource while being funded primarily by donations has always been a core challenge of Poly Haven.

It’s easy to throw a bunch of files on Amazon S3 and call it a day, but you’re gonna rack up a bunch of bills that you simply cannot afford to pay.

Serving 80TB of bandwidth from S3 is estimated to cost around $4,000 per month alone, which is coincidentally about the same amount that is being donated to us on Patreon currently.

This obviously leaves zero budget for actually creating assets, nevermind all our other costs.

- Advertisement -

So how the heck do we manage to run a website with 5 million pageviews and 80 terabytes of traffic every month for under $400?

Cloudflare All the Way

This web giant is probably the reason we can do what we do so easily and so affordably. Sure, we could live without them, but it’d be a lot more work.

Cloudflare, for us, is primarily a massive caching layer.

What does that mean? When someone in London downloads a file for example, the file is fetched from our origin server (more on that later) but routed through one of Cloudflare’s nearby edge nodes.

Then, the next time someone in London downloads the same file, Cloudflare has cached it and serves it to them directly, without needing to bother our origin server.

- Advertisement -

We can define how long a particular file or URL should be cached for using a set of rules, so that rarely changing content (like asset files) can stay cached as long as possible, while content that changes every day (e.g. our list of assets) is updated more often.

For asset downloads, these cached downloads make up for about 85% of all traffic.

Cached (orange) vs uncached asset downloads

This doesn’t only apply to file downloads however, it’s also used for everything else on the website too.

Every time you load a page, your browser sends a few dozen requests (think stylesheets, javascript, json data…), most of which is static.

Because of Argo (which I’ll explain below) Cloudflare can cache this kind of traffic even better than the huge asset files, resulting in about a 93% cache ratio.

Traffic for polyhaven.com (everything excluding asset downloads and image requests)
- Advertisement -

All this means that our origin server(s) have a lot less work to do, which means they’re theoretically more affordable.

And what does this amazing caching layer cost us?

$40 per month.

Cloudflare’s Pro membership costs $20 per domain, and we have two domains: polyhaven.com for the main website, and polyhaven.org for the asset downloads.

Backblaze Bandwidth Alliance

So after Cloudflare, we only have to handle about 15% of our 80TB of monthly traffic. That’s still not nothing though, and it would cost us about $650 using Amazon S3.

S3 is not the only player in the game though, there are other more budget-friendly cloud providers.

The one we use is Backblaze – another internet giant that’s more known for its consumer backup service.

Alone, Backblaze’s B2 clone of S3 service would cost us about $130 per month for that last 15% that Cloudflare doesn’t handle.

But, we can get around that because of a partnership between Backblaze and Cloudflare they call the Bandwidth Alliance.

As long as we use both services together, and pay for the $20 cloudflare subscription, we don’t get charged for download traffic at all.

All we have to pay is the storage fee, some upload costs and API requests, which comes to around $11 per month.

Web “Server”

So that covers all of our costs for hosting the asset files themselves so far, but we still need a website to show them on.

polyhaven.com is built with Next.js – a javascript framework created by Vercel.

While you can absolutely run a Next.js application on your own servers or half a dozen other cloud providers, Vercel offers a fairly straight forward and affordable service to deploy your web application with them directly.

This saves us the hassle of managing our own server, while having the added benefit of being a serverless environment which improves performance for users all around the world.

Their base fee is $20 per month, with additional costs based on usage.

Since we use Cloudflare in front of Vercel and are super careful about what can be cached and what can’t (e.g. anything requiring user authentication), we generally don’t go over the included usage limits unless something goes terribly wrong.

Vercel serverless function execution time (free tier is 1000hr)

In fact, we recently acquired sponsorship from Vercel, so they now cover our costs anyway, but for the sake of this article we’ll pretend to have to pay for it 🙂

The Database

An important part of any content website for sure – this is where we store all the information about every asset, lists of every file for each of them, download logs, etc.

Back when we still ran hdrihaven.com and the other sites separately, all on their own manually managed LAMP stacks, the databases were the first to start suffering from performance issues.

To avoid this problem in the future, I decided to splurge a bit and go for a cloud solution where I wouldn’t have to worry about reliability, performance, scaling or integrity ever again: Google Firestore.

This is certainly not the cheapest option, at around $100 per month it accounts for about half of our monthly web budget, but I still think it’s the right way to go.

Firestore usage per day

With Firestore, you pay per usage beyond the free tier. For us, this is primarily database reads (the blue line above).

Every time a bit of data is fetched from the database, we pay a tiny amount. Per read it’s practically insignificant, at $0.0000006. But multiply this by the number of document reads you have to do per page view (e.g. on our library page, that’s one read per asset, 866 currently), multiplied by the number of page views (~5M per month) and it can get very expensive very quickly.

To avoid database reads as much as possible, we cache as much as possible. Our data doesn’t change that often, or when it does (e.g. download counts) it’s not very important to show users the latest information anyway. Cloudflare all the way 🙂

Our API

To give us the most control over caching database reads, and also to avoid racking up bills in Vercel, we have a separate $5 server (yes, seriously) on Vultr that runs our API.

The purpose of this API is to connect our front-end website (on Vercel) to our database. That’s all it does.

And because we cache everything so heavily (in order to reduce database costs), the API server can be extremely basic.

API server usage

It’ll be a long time before this server becomes inadequate for our needs, and before that happens it’ll be trivial to migrate the server snapshot to a beefier server, or load balanced network of servers more likely.

Argo

One of the main reasons our database and API costs are so low (relative to what they could be without Cloudflare) is because of the extremely high cache ratio, 93%.

It’s a direct effect – the more we can get Cloudflare to cache, the less often our API has to fetch things from the database itself, the lower our usage costs.

Argo is an optional extra service from Cloudflare that does two things:

  1. Optimizes DNS routes to improve latency (this helps our site feel faster to users).
  2. Adds an additional layer to the cache, so all global traffic goes through only a handful of their biggest data centers before splitting up to their hundreds of edge nodes.

This second point is what we care about.

Using our example from earlier: With Argo, if someone in London first downloads a file, Cloudflare will now cache that file not just for London, but for the whole of Western Europe.

This results in a much higher cache ratio (going from ~75% to 93% with our configuration), but comes at a cost: You now pay per GB of traffic going through the Argo network.

This is why we have separate .com and .org domains – we have Argo enabled on the .com domain which runs the website with relatively low bandwidth, and then we leave Argo disabled on the .org domain which serves the 80TB of download traffic.

Argo costs us about $160 per month currently, by far the biggest single expense. But remember it saves us a huge chunk of database usage fees, by my back-of-the-napkin math around $250.

Plus we get the benefit of improved latency, less load on our API server, and less Vercel usage.

Image Hosting

Finally, the last piece of the puzzle is mostly separate from everything else.

All of our images shown on the website (thumbnails, renders, previews, etc.) are stored on Bunny.net – another budget CDN.

Bunny.net doesn’t just store our images though, they also have an optimization service which allows us to dynamically resize and compress images for the website. This greatly improves both user experience (faster page loads) and our own workflow (we only need to upload a single PNG and it’s available in any size and format).

This costs us about $27 per month, depending on traffic.

Bunny.net usage

Total

  • DNS, caching & egress: Cloudflare (2 domains)$40
  • Caching: Cloudflare (Argo)$160
  • Asset storage: Backblaze B2$11
  • Web hosting: Vercel$20
  • Database: Firestore$100
  • API: Vultr$5
  • Image hosting & optimization: Bunny.net$27
  • Domains: Cloudflare$4
  • Email fees: MXroute$3

That brings the total to about $370.

A lot of these costs are based on our usage of these services, which is directly affected by traffic on the website.

These costs will likely grow as we grow, which is good. If you’re reading this in the distant future, just take a look at our public finance reports to see how things have changed 🙂

Oh and about those finance reports, the “Web Hosting” category there is a bit higher than $370 as it includes some other servers and services we use to assist in managing our team and uploading assets – those aren’t strictly related to running the polyhaven.com site itself, but rather Poly Haven the company.

Future Thoughts and Contingencies

Our infrastructure is intentionally quite modular, so that we don’t rely on any service too heavily. Even Cloudflare could be replaced if we had to.

Our API server is currently our weakest single point of failure – I don’t have any experience running a public API. We want to allow other people to depend on the API to integrate our assets with their software, so this area needs some research and improvement.

Google Firebase is nice and convenient, but it is quite expensive. We could investigate some other managed database options in future.

One of our biggest cost savings is with the Cloudflare + Backblaze Bandwidth Alliance. The plan, should this cease to exist at some point in the future, would be to set up our own network of load balanced dedicated servers to handle the traffic. This sounds scary, but we actually did this in the past and it worked quite well. It’s a pain to manage for sure, but potentially just as cheap.


Hopefully this monster of a blog post answers some of your questions and helps you figure out the architecture you need for your own project, or was at least interesting to read 🙂

Join the pack! Join 8000+ others registered users, and get chat, make groups, post updates and make friends around the world!
www.knowasiak.com/register/
Read More

- Advertisement -

1 Comment

You might also likeRELATED
Recommended to you

Frame.work laptop now accepting pre-orders from UK, Germany and France

The Framework Laptop is now available for pre-order in the UK, Germany, and France!  We aim to build upgradeable, customizable, repairable products for everyone in the world, and we’re taking the next steps to make that real.  We’re using the same pre-order batch system that we launched with in the US and Canada; a fully…

800,000 Deaths

The number is so enormous that we risk becoming numb to its implications.December 14, 2021, 12:26 PM ETAbout the...

Amazon parents dashboard (U.K.) unavailable

Please Enable Cookies to Continue Use your Amazon account to log in Email or mobile phone number Enter your e-mail address or mobile phone number Password Forgot Password Enter your password Sign-In By signing-in you agree to Amazon's Conditions of Use & Sale. Please see our Privacy Notice, our Cookies Notice and our Interest-Based Ads…
- Advertisement -

Thich Nhat Hanh, Vietnamese Zen Master, Dies at 95

Thich Nhat Hanh at the Plum Village monastery in southern France | Courtesy Plum Village Community of Engaged Buddhism Vietnamese Zen Master Thich Nhat Hanh—a world-renowned spiritual leader, author, poet, and peace activist—died on January 22, 2022 at midnight (ICT) at his root temple, Tu Hien Temple, in Hue, Vietnam. He was 95. “Our beloved…

Before wave of train thefts, Union Pacific laid off some of its police force

News organizations both locally and nationally have been covering the rise of cargo theft in L.A.’s northeast train tracks in the past few days. Anchors on morning news have been quick to point out that there have been over 100 arrests, and even Forbes have been quick to point out the staggering $5 million worth…

Must read

Tokio Console

Today, we, the Tokio team, are announcing the initial release of Tokio Console (Github), enabling Rust developers to gain deeper insight into the runtime behavior of their applications.And just like that, we get to peek under the hood. —niedzejkobTokio Console is a diagnostics and debugging tool for asynchronous Rust programs. It gives you a live,…
- Advertisement -