By Alex Karasulu, Bahtiiar Kukanov, and Burak Metin
Just this past month, March 2018, production began on the first engineering units for the Subutai Blockchain Router’s industrial grade edition with our partner LSI-TEC in São Paulo, Brazil. This coincides with the release of a white paper on a mission critical IoT hardware security architecture which summarizes our findings over three years of research.
The Subutai Blockchain Router v2.0 offers advanced P2P cloud router and IoT gateway functions using the world’s most powerful FPGA MSoC, the Xilinx Ultrascale, packed with four 64-bit A53 ARM cores with up to 16GB of DDR4 RAM capacity.
History and Vision
It’s very important to understand the past, the forces shaping our direction, the reasons behind the decisions made and how they shaped our vision for the router. The past and the future are both intertwined together. This is a chronological breakdown of how it all began and why.
Initial Drivers
Back in 2013, while developing the Subutai P2P Cloud Software, now together referred to as the PeerOS, we began leveraging software defined networking in addition to other less appealing precursor technologies, like Universal Plug and Play to try to “coordinate” with the hardware. Various optimizations required working with networking hardware on the edge. Synergy between the networking hardware and the fog software was extremely promising and the emergence of various network virtualization efforts gave us a glimmer of hope.
Our problem was unique however. Because of the sheer diversity of the network hardware devices found on the edge, we did not know what to expect, and realized that at most a common denominator of capabilities could be relied on. This was limiting and especially disappointing when some common functionality like uPnP is essentially insecure and dangerous.
We quickly realized that our grand unified vision for fog computing and IoT demanded more advanced functionality above and beyond what the average router and gateway offered on the edge. For starters cloud multi-tenancy imposed stringent security requirements along with the need for secure boot infrastructure that starts with a trusted platform environment. The multi-tenancy rabbit hole goes deep to include things like network QoS requirements across the environments of tenants running across the nano centers we call peers. On the IoT gateway side, enabling the connectivity of the “Things” requires not only wireless but wired connectivity and that entails being able to attach various peripheral connect buses between them and the gateway. The advanced security measures for mission critical systems also required hardware acceleration to even be remotely feasible.
First Experiments
Around 2014, we decided to make a daughter board for an existing platform, the Parallella, to build an advanced cloud router and IoT gateway reference device upon which we could develop a specification to guide hardware vendors. In 2015, we manufactured a few engineering units of this board that we called the Subutai Liquid Router. Liquid because it was flexible with programmable logic along side a processing system using a Zynq-7000 series SoC.
The images above show the daughter board for the Parallella. Yes the daughter board looks more like a mother board instead, being much larger than the Parallella that attaches to it. Notice the switch and ethernet ports along with the Artix 7 FPGA which communicated with the Parallella through it’s LVDS lines via four Samtec connectors. The Parallella had no viable high speed disk interfaces so we added a eMMC RAID controller and multiple eMMC module slots visible in the upper left hand side of the first board image above.
What is an FPGA? MPSoC?
For those not familiar with FPGA technology, the acronym stands for Field Programmable Gate Array. These chips contain a mesh of electronic parts that can be re-programmed to perform different operations. FPGA devices may even be re-programmed after being purchased and put into the field, hence the “Field Programmable” part of the acronym.
An FPGA MPSoC is hybrid multi-processor chip with an FPGA combined. It usually has 2 or more processors with high speed in chip interfaces like AXI to communicate with circuit designs in the FPGA.
Approaching End of Life
All-in-all we had fun and were able to conduct several tests and experiments using this daughterboard design. Many experiments led us to conclusions that helped mould our security framework which is now being published. The Liquid Router was however severely limited in terms of IO bandwidth and the total amount of resources available. Several flaws resulted from a daughter board design. We knew the Liquid Router did what it could for our research, but we had to move on to a new board design without being dependent on another board as a add on.
Some of our research into hardware security mechanisms required using a MiniITX Zynq-7100 to have enough FPGA resources when a fun side project by our engineers turned into a discovery with vast potential. They essentially implemented a miner in the FPGA to operate on a private Ethereum blockchain where they were able to fit the DAG into the limited 1GB of RAM available on this device. It was able to mine at 3-4MH/s due to bandwidth limits but it only consumed approximately 10-watts of power.
Ultrascale+ Series
At around the same time, we started to see the prices of the next family of Xilinx SoC’s, the Ultrascale+ MPSoC chips, start to drop dramatically. We started designing a new router based on the Ultrascale+ hoping prices would drop considerably to support industrial and commercial applications by the time our design was ready.
Meanwhile working with our partners we realized the availability of more transceivers combined with new memory controller designs using hybrid memory cube serial DRAM with 160GB/s bandwidth which dramatically increases mining hash rates. Tests and theoretical limits now showed hash rates of 20-21 MH/s using HMC memory in the upcoming Residential Edition of the Blockchain Router.
HMC memory is available today, however we’re setting our sights on a new kind of High Bandwidth Memory (HBM) DRAM soon to be integrated directly into the MPSoC family. A standard 8GBs of HBM at 460 GB/s bandwidth is already being packaged inside early FPGA chips themselves. It’s just a short period of time before we see HBM inside the MPSoC family and this could result in hash rates of 58 MH/s.
These possibilities influenced our design directions on both the Industrial and Residential editions of the new router design which we code named the Blockchain Router family of OptDyn products.
Industrial Edition Production Begins
The industrial edition of the Blockchain Router is designed specifically to target high end mission critical IoT applications. The FPGA is used to host hardware security circuitry to augment the expected broadband router functions. We’ve released the security architecture white paper to coincide with the availability of the router for the development of industrial process control applications requiring the highest degree of security and reliability.
Flexibility at its Core
Although more information about the features of the router is available on the specification page, I wanted to stress the compatible expansion headers and shields supported. Specifically 4 PMod headers, a Raspberry Pi header, and an Arduino hat is available.
Almost every detector/sensor, actuator, working with shields for these platforms found at SparkFun Electronics is compatible with the Subutai Blockchain Router. Virtually any new peripheral connect bus could be added to it. In this respect, the Blockchain Router truly surpasses any other IoT gateway on the market. It’s a laboratory and IoT development platform in its own right. Note that this was one of our primary goals when starting out on the design and we’re really excited about it.
Differences Between Editions
The Subutai Blockchain Router’s industrial grade configuration differs from the residential configuration by using FPGA MSoCs operating between extreme temperatures (-40C to 100C) and having a modified memory architecture on the programming logic (PL) side based on HMC DRAM instead of DDR4. The model of the MPSoC’s differs with more transceivers and cross section bandwidth to handle the requirements of Ethereum’s memory hard mining Ethash function. Most importantly though, the FPGA fabric on the industrial grade MPSoC is used for IoT security functions rather than cryptocurrency mining. The Subutai Mission Critical IoT Hardware Security Architecture defines these security functions and their use in mission critical industrial automation applications.
In October 2017, ARM published its Platform Security Architecture which serves as a guidance with primitives to be used. The PSA goes a long way towards fixing the IoT security dilemma coming with the tsunami of devices and data. OptDyn’s framework compliments these primitives and proposes concrete mechanisms for hardware based real time intrusion detection and prevention systems facilitated by IoT gateways like the Subutai Blockchain Router.
Ready for Production Use
OptDyn’s Subutai Blockchain Router takes its place as the high end IoT gateway device in the Caninos Loucos project serving as the foundation of Brazil’s national IoT standard. Professor Zuffo of the University of Sao Paulo spearheads the project,
“We’re really excited about the Subutai Blockchain Router and its industrial configuration with its mission critical security subsystems,” said Dr. Marcelo Zuffo, who spearheads the project at the University of São Paulo, and leads the national IoT effort for Brazil. “The Subutai Blockchain Router, as a secure IoT gateway, has the ability to enable everything from Industry 4.0 to Smart Cities in Brazil. We’re very excited to complete this last part of the puzzle thanks our collaboration with OptDyn, Inc.”
“Mission critical IoT systems don’t have the option of learning from failures where lives and property are at stake,” said Professor Miguel Matrakas, manager of Celtab, one of the Research Centers of the Itaipu Technological Park (PTI; a foundation responsible for promoting scientific, technological and innovation development linked to Itaipu Binacional, the largest clean energy generator in the world), who surveyed the Subutai Blockchain Router v1.0 at Latinoware with Hall and OptDyn CTO and Founder Alex Karasulu. “The Subutai Blockchain Router and its industrial configuration provide the crucial hardware security infrastructure needed for IoT applications in the utility sector.”
“We’re impressed with the innovation and level of performance possible for numerous Smart City IoT projects using the Subutai Blockchain Router,” said Bruno Freitas, Head of the Innovation Unit for the State of São Paulo, Brazil. “The benefits of using this technology —which is being manufactured in São Paulo as part of the Brazilian national IoT program— are apparent. We look forward to considering the Subutai Blockchain Router as a viable option to meet the demands for IoT projects that build an intelligent and prosperous state and cities.”
Residential Edition
The Residential Edition of the Subutai Blockchain Router is slated to go into production later this year after the Industrial Edition enters the market and we’re very excited about it’s ecological impact as an alternative to GPU based mining. It’s coming at just the right time, and offers greater flexibility over hardwired ASIC solutions. We also see the Blockchain Router opening the cryptocurrency mining market to FPGA vendors as well as ISP’s interested in providing them to their customers.
Costs
The primary cost of production for FPGA/SoC manufacturers is the startup costs of R&D and setting up manufacturing facilities. The difference between making 10 million versus 100 million chips is negligible. The high costs of FPGA SoC’s on the market is to control prices to recoup these initial startup costs. With a larger market for their chips FPGA manufacturers will naturally produce more and drop prices to earn more in the end. It’s all about marginal product revenue.
Ecological Impact
The main winner however is the environment. Thankfully only a small minute fraction of the world’s population is mining as we speak. Imagine the strain on the power grid when even 5% of the population starts to mine for cryptocurrency? If we can reduce the power consumption by even 100% the environment wins. We know we can reach higher levels of power efficiency up to 1083% when compared to existing GPU cards.
Tomas Evensen, Xilinx CTO, sees a great future for low power FPGA based mining to prevent the needless waste of electricity from power hungry GPU mining devices, “With the proper FPGA design, custom hash generating cores in the SoC can calculate hashes faster with fewer clock cycles and require much less power than using software running on mainboard CPUs or GPUs in graphics cards,” said Tomas Evensen, CTO Embedded Software at Xilinx. “We’re excited to see the Subutai Blockchain Router use our FPGAs for the consensus layer of distributed ledger technologies —while also protecting the environment.”
Future
We intend to work closely with Xilinx and our partners to develop even more efficient editions of the Subutai Blockchain Router’s Residential Edition with a healthy pricepoint to compete with the costs of ASIC and GPU based mining products. We’re especially looking forward to the introduction of HBM DRAM in the MPSoC family of Xilinx products to raise the hash rates to a jaw dropping 58 MH/s.
What about ASIC Miners?
If you’ve ever felt bad for crumpling a clean sheet of paper after a mistake with indelible ink, then you can imagine the waste for mistakes when designing ASICs. Once burnt onto the silicon there’s no going back. The algorithm design cannot be improved, new blockchain algorithms cannot be supported.
For these and other reasons the process of designing and mass producing ASICs are still very high. Using already manufactured FPGAs does not incur that startup cost while FPGAs can be reprogrammed to improve efficiency even after consumers purchase the product. The first consumers are going to have to absorb the start up costs of these ASICS. We’re already seeing suggested prices of two to three thousand dollars for Bitmain products.
In terms of power efficiency FPGAs and ASICs are comparable. In terms of performance ASICs exhibit 20-50% higher performance. This is offset by the differences in flexibility which is a central theme of the Subutai Blockchain Router.
Also remember there’s no router functionality or multi-purpose function to these devices. The Blockchain Router is a cloud router and IoT gateway first and foremost. Our aim is not to just mine crypto. In fact, it’s never was our primary objective. That’s something the router can do on the side while handling its traditional router and gateway functions. Even if new blockchains with different mining algorithms emerge, the Blockchain Router can be reprogrammed to work with them so consumers will not be stuck with yet another useless electronic device that costs too much. Even if mining capabilities are disabled altogether the router can still use the fabric for its advanced security features.
The Subutai Blockchain Router is based on an open architecture with a FOSSH design available at GitHub https://github.com/subutai-io/blockchain-router
RELATED [white paper]: Open Hardware Security Framework — A MUST for Mission Critical IoT Applications
# # #
Subutai Blog 