The Cost of Cloud

This blockchain engineer’s eureka moment about a better approach to infrastructure hosting.

One of the more interesting things that happened to me after starting to work in the blockchain industry was the total obliteration of the prior decade of modus operandi I had developed as it relates to hosting choices.

It had previously been drilled into me by each former employer that we should shift away from hosting our own infrastructure and opt to run everything on a virtual machine. We would select from one of the big three cloud providers, due to their high uptime, extensive toolsets, speed to delivery and accounting benefits (OpEx over CapEx).

This all made sense. I was delivering SQL clusters, web server estates, message queuing clusters… and these platforms made it possible for me to do it quickly and repeatedly. They also offered amazing tooling that guaranteed high availability and disaster recovery capabilities at nearly no cost. Starting my new role of building the infrastructure for a blockchain POS business, I assumed (naively, it turns out) that these benefits would mirror over into the new context. So, I chose to start building infrastructure in GCP, Azure and AWS.

Interestingly, the item which instigated our initial questioning of this model was disk performance. We started to see suboptimal bottlenecks on even high-performance storage options in each provider for our validators—most notably, NEAR. To achieve the next disk performance tier, we had to upgrade the virtual machine specifications themselves which doubled the monthly cost per node. As a design choice, we always run at least one disaster recovery partner for our validator infrastructure, even in a cloud provider. This compounded the issue, as both partners needed the same upgrade to ensure they were both viable as active validators.

Faced with burgeoning cloud costs and a standardized implementation style which has multiple node instances... we decided to look at alternatives.

Enter Bare Metal Cloud

After some research we discovered a few companies that were offering quick and inexpensive access to bare metal infrastructure. That is to say, SSH access into an OS of our choice, installed on a predefined t-shirt size of micro-blade in a shared chassis somewhere.

Get your head out of the clouds.

One major benefit of these machine specifications was their access to locally attached NVME storage drives, the performance of which is pretty unparalleled in a shared storage environment. We decided to give it a whirl… and wow. At roughly 1/6th of the cost than the VM in one of the big three, we had a much more performant node. What’s more, we discovered that most of these bare metal node offerings include egress allowances of 5-15TB per month—so, no egress costs!

1/6th of the cost, better performance… so, what’s the catch? Well, those disks we are running on now aren’t highly redundant like shared storage, and we can’t expect the OS to automigrate like we can with a VM. The solution? Provision more nodes in different locations. The cost reduction enabled us to provision four nodes rather than two whilst still maintaining lower costs. The nodes acting as standbys are more beneficial to us (lower costs than HA storage and VMs), and are better for the networks in which we operate; more copies of the chain database in different locations on different physical nodes.

The Realization

Blockchain technology in combination with amazing open source DevOps tools removes the need for the solutions that cloud providers have built over the past decade or two—the solutions that cater to monolithic application and database infrastructure.

The prices we pay for these cloud providers are inclusive of the cost that they endured while developing all of the services which, frankly, we really don’t need. So why pay them for it?

The Outcome

Here’s my stance now, plain and simple: use lots of different bare metal hosting providers, and run lots of nodes.

Ultimately, the thing we all strive for in this industry is decentralization (at least it should be). It seems that blockchain technology is opening a path for us to at least deconcentrate infrastructure from the big three, and move more into smaller providers. These providers can offer their infrastructure at much lower costs, because they aren’t saddled with decades worth of solutions and overhead catering to problems irrelevant to the blockchain industry. These organizations (like OVHCloud, phoenixNAP, redswitches, and hosthatch, among others) are often more nimble and more open to blockchain as a driving force for their revenue.

It’s a really exciting time to be an infrastructure engineer and experience how these cycles of concentration and proliferation play out. Anyone who declares blockchain isn’t revolutionary just isn’t seeing the full picture. It is unlocking new business models within the infrastructure hosting market, forcing competition onto the big providers.

From this we can see how the viability of decentralization is opening a vision of a different future as far as hosting goes. It will be incredibly interesting to follow whether blockchain unlocks similarly exciting possibilities for the layers underneath the data centers in the future.