On Tuesday, June 13th, Joshua St. James, co-owner and chef at Valiant the Sandwich in Salem, Oregon received some terrible news on his day off. His leads informed him that their online system wasn’t working, and they were having all kinds of problems. Little did they know, Amazon Web Services had crashed, and they were among thousands who were impacted. The outage lasted several hours.
Valiant the Sandwich opened in May 2020, “in the middle of a freaking pandemic,” as St. James puts it. The restaurant features gourmet sandwiches, a rotating menu, and a full bar. Early in the pandemic, St. James worked 14-hour days, prepped and cooked the food himself, while manning the counter. They’ve been popular since the beginning, and online orders have always been a major part of their business. Due to the centralized nature of Amazon Web Services and the vast number of entities that rely on them, an outage from the central hub can have wide sweeping impacts.
“I didn’t know it was an AWS issue,” St. James said. The outage took place right in the middle of lunch time at the local sandwich shop. On the weekend, most orders are in-house, but in the middle of the week about 50% of their lunch orders are online — and they weren’t coming in. Valiant’s website is hosted by Toast, a point of sale (POS) service which is running on AWS.
When St. James looked online, he wasn’t able to place an order. He explained, “Online orders at lunch on a Tuesday are crucial. Half of the orders at the particular time of day are online orders, so the revenue loss is significant. It created a massive issue for that particular day.”
He also could not get in to ensure the third party orders, like DoorDash, GrubHub and UberEats, were coming through. “When I called the service desk at Toast, they couldn’t even look at my account. They had no idea what was happening. They couldn’t see anything on my end or do anything for anybody,” St. James reported. Wondering if others were suffering the same, St. James contacted a few other colleagues around town who also used Toast, and they were all having the same trouble.
“For us on the west coast in the middle of lunch time… there’s thousands and thousands of restaurants that use this service.” (62,000 in the US, as of March 2022.) Even if places had a 10% reduction, as opposed to a 50% reduction [in revenue], that’s a massive amount of loss,” he said.
Restaurants often have pretty slim margins as it is. “They have staffing they have to pay for, and they need every one of those sales. Especially restaurants like ours — we never had a lot of money behind us, and that rolls into payroll….Even $500 makes a difference in a restaurant like ours,” St. James disclosed. The restaurant’s payroll was due to come out that week, as well, adding additional stress.
Toast has redundancies built in for outages, which is great for online orders, but St. James couldn’t even get into the backend to assess or make a plan. He never expected he would have to worry about this kind of downtime. “Those of us that have been doing this for a while oftentimes fix things with the tools that we have, but that eliminated any other tools that we had,” he said.
When asked about choosing a different provider, he said there are no options. “It’s pretty much the industry standard — that particular company, they’re some of the best,” he said. There are other POS processing companies, such as Square, but like Toast, they are attached to AWS. “For them it’s a bigger issue because they’re the ones that really don’t have any options, but that, in turn, hinders what kind of protection they can offer,” St. James concluded.
Compared to large corporate entities, the implications of this sort of disruption are incredibly significant for small businesses. Those are the risks businesses are currently accepting when depending so heavily on a single tech giant. Which is why many are currently seeking decentralized alternatives in the world of Web 3, such as solutions integrating blockchains in the Antelope Coalition, which boast greater security, privacy and control of data.
Antelope coalition chains, Telos, the UX Network, WAX and EOS all have documented TPS (transaction per second) speeds ranging from a minimum of 4000 TPS to over 30,000 TPS. In addition, the Antelope Coalition has the benefit of Trustless IBC (inter-blockchain communication) which can provide unlimited TPS and scalable solutions via horizontal scaling. These are the numbers needed for a POS system to work effectively in the real world. With Trustless IBC, app developers are now building high-performance solutions in the Antelope Coalition.
Because of the fundamental technological nature of Web3, the Amazon crash that negatively impacted so many businesses like Valiant the Sandwich on June 13th, will become a thing of the past for businesses utilizing apps built with technology similar to what is being built by developers in the Antelope Coalition.
GoodBlock Technologies is an app developer and Block Producer candidate on the Telos Blockchain Network, with a focus on 2 of the pillars of Web3.0; Governance (decidevoter.app) and Decentralized Cloud Storage (dstor.cloud). Vote for goodblocktls, and learn more at goodblock.io.
Telos is a cost-effective, energy efficient, fast, and scalable DPoS blockchain that has been operational for over 2 years. The Telos blockchain has leading on-chain governance (Telos Decide), and is built and developed by a core development team using the EOSIO codebase.
GoodBlock Resources and Social Media: