by OptDyn Co-CEO/Founder Alex Karasulu, CMO Sally Khudairi, and Subutai Community Manager Felipe Fonseca
Last Friday we were onsite at LSI-TEC (Laboratório de Sistemas Integráveis Tecnológico), our partners in São Paulo, Brazil, who are manufacturing the Industrial Edition of the Subutai Blockchain Router v2.0.
The Subutai Blockchain Router v2.0 is a power-efficient “green” broadband Cloud router and open hardware IoT gateway. As a Subutai appliance, the router comes preinstalled with Subutai’s PeerOS tuned and hardware accelerated.
Aimed at mission-critical IoT applications, the Subutai Blockchain Router Industrial Edition uses higher grade temperature resistant chips that operate between -40°C and 100°C. As opposed to the Residential Edition which mines cryptocurrency, the programmable logic for the Industrial Edition is dedicated to critical IoT security operations. The Blockchain Router’s hardware based IoT security subsystem combined with its interoperability with PMOD, Arduino, and Raspberry Pi shields has brought it to the forefront of Brazil’s national IoT standards, and serves as the high-end IoT gateway for several highly-visible programs, including Itaipu Binacional, the world’s largest generator of renewable energy, and Ilumina SP, the smart city illumination initiative for the State of São Paulo.
Seeing the Subutai Blockchain Router in production feels like Christmas! We’re excited to share our experience with you —
A batch of OptDyn’s Subutai Blockchain Router engineering units are entering the surface mount technology assembly (SMT) line in São Paulo. Here’s a unit going in:
Here’s a test of the positioning of its larger components such as the RJ54 connectors, video and USB ports. The Arduino and Raspberry Pi headers are clearly visible on the lower left hand side.
Here’s the pallette of Xylinx Zynq MPSoC chips (with FPGA and processors) fed into the SMT line. This 3D FPGA and its quad core ARM 64-bit processors power the Subutai Blockchain Router.
After the placement of the Xilinx Zync BGA package:
Placing other components on the board:
Cartridges hold reels of passive components placed onto the board as it goes through the SMT line.
Additional passives being placed and soldered to the board. Note the robotic arm holding the components.
Here’s the entire SMT line. The long deep oven in the back actually heats and cools down the board. The Router’s board is complex with _16-layers_ requiring special thermal conditions.
Our Subutai Community Manager Felipe Fonseca and Arthur Barcellos from LSITEC giving the thumbs up.
Even before Bitmain confirmed the availability of their Ethereum ASIC Miners in July, community developers had already started working on modifications to their Ethash algorithms to render the product dead on arrival.
These ASIC miners will be utterly useless before the first customer takes the device out its packaging, if it even ships at all. This will go down in the annals of crypto-mining history as one of the most expensive blunders of all time. How could Wu and Zhan not see this coming?
GPU Miners Live On
The “Bitmain Stillbirth” will prove, once and for all, that hardware flexibility to adapt to algorithmic changes is just as important as raw hash rates. GPU based miners with the ability to load code that is executed by thousands of SIMD processors live on. The threat of ASIC based mining is finally going to be over. The Monero developers are even planning on mutating their algorithm every six months to prevent the chance of ASIC miners gaining any foothold whatsoever.
Stock market traders may be interested in buying back into NVIDIA and AMD. After Analysts reduced their outlook for GPU producers, their stocks dropped slightly. Once the realization of ASIC miner hopelessness permeates, the stock price of various GPU manufacturers should go back up, but with a vengeance. This does not bode well for the environment since GPU’s are notorious power guzzlers. As the number of enthusiast miners grow, at geometric rates, we’re looking at a serious ecological problem.
The “Middle Way” – FPGA Miners
ASIC miners are fast and power efficient, yet inflexible. GPU miners are slower than ASIC miners and flexible, but they consume much more power: several orders more per hash verification.
If Buddha were around and into crypto, he would probably advise, “Take the Middle Way, use FPGA miners”. An FPGA is essentially an ASIC that can be reprogrammed. Performance wise, there’s a slight penalty for being programmable. Approximately 20-50% performance is sacrificed for this flexibility. In terms of power consumption, ASICs and FPGAs are pretty much the same.
FPGA Feasibility: Hash Rates
Most FPGA platforms possess sufficient computing power. Up until recently, the problem with FPGA miners resided in memory bandwidth. Ethereum’s Ethash algorithm is memory hard, meaning the upper bound of hash calculations require a certain amount and bandwidth of RAM. Then new kinds of memory started to emerge making the endeavor completely feasible.
Hybrid Memory Cube chips emerged three years ago to boast a bandwidth of 160 GB/s. The memory technology let’s an FPGA handle up to 20 MH/s on a single interface with less pins reducing the routing complexity. It’s impressive yet certain problems arise because the chips reside outside of the FPGA. Chip to chip communication requires several gigabit transceivers which increases the cost of the FPGA. Regardless the technology is prevalent and experiments show promising results.
Just about a year ago Xilinx introduced in-chip High Bandwidth Memory on their Virtex FPGA series with a default capacity of 8 GB. HBM has a bandwidth of 460 GB/s and a single FPGA can theoretically support hash rates of up to 58 MH/s. The most beautiful aspect to all this is that it requires no high speed (gigabit) transceivers or even controllers with access mechanisms built into the FPGA.
FPGA Feasibility: Cost
Everyone that knows anything about FPGAs knows they’re not cheap. This is due to a limited market, and the need to recover investment and set up costs (research, development, and manufacturing) to produce them. The materials and operating costs required to produce a single chip is relatively small. Hence making 100 chips costs pretty much the same as producing 1,000 or 10,000 chip batches.
Prices shift when markets grow and marginal product revenue plays a big part in these pricing decisions. If the N-th unit of product produced is of negligible cost, then a producer can lower prices to accommodate a wider consumer base and dramatically increase revenues. As an example, perhaps 10,000 consumers buy the product if it costs $1,200 and pays for itself in 10 months. If the price drops to $320 and the product pays for itself in 2 months, then you could have 10,000,000 or more consumers buying the product. These numbers are completely feasible and the difference is $12M vs. $3.2B in revenue. Our relationships and partnerships show the right price points will come with the emergence of FPGA mining thanks to the Subutai Blockchain Router.
The Subutai Blockchain Router v2.0 offers advanced P2P cloud router and IoT gateway functions using the world’s most powerful FPGA MSoC, the Xilinx Ultrascale, packed with four 64-bit A53 ARM cores with up to 16GB of DDR4 RAM capacity.
History and Vision
It’s very important to understand the past, the forces shaping our direction, the reasons behind the decisions made and how they shaped our vision for the router. The past and the future are both intertwined together. This is a chronological breakdown of how it all began and why.
Back in 2013, while developing the Subutai P2P Cloud Software, now together referred to as the PeerOS, we began leveraging software defined networking in addition to other less appealing precursor technologies, like Universal Plug and Play to try to “coordinate” with the hardware. Various optimizations required working with networking hardware on the edge. Synergy between the networking hardware and the fog software was extremely promising and the emergence of various network virtualization efforts gave us a glimmer of hope.
Our problem was unique however. Because of the sheer diversity of the network hardware devices found on the edge, we did not know what to expect, and realized that at most a common denominator of capabilities could be relied on. This was limiting and especially disappointing when some common functionality like uPnP is essentially insecure and dangerous.
We quickly realized that our grand unified vision for fog computing and IoT demanded more advanced functionality above and beyond what the average router and gateway offered on the edge. For starters cloud multi-tenancy imposed stringent security requirements along with the need for secure boot infrastructure that starts with a trusted platform environment. The multi-tenancy rabbit hole goes deep to include things like network QoS requirements across the environments of tenants running across the nano centers we call peers. On the IoT gateway side, enabling the connectivity of the “Things” requires not only wireless but wired connectivity and that entails being able to attach various peripheral connect buses between them and the gateway. The advanced security measures for mission critical systems also required hardware acceleration to even be remotely feasible.
Around 2014, we decided to make a daughter board for an existing platform, the Parallella, to build an advanced cloud router and IoT gateway reference device upon which we could develop a specification to guide hardware vendors. In 2015, we manufactured a few engineering units of this board that we called the Subutai Liquid Router. Liquid because it was flexible with programmable logic along side a processing system using a Zynq-7000 series SoC.
The images above show the daughter board for the Parallella. Yes the daughter board looks more like a mother board instead, being much larger than the Parallella that attaches to it. Notice the switch and ethernet ports along with the Artix 7 FPGA which communicated with the Parallella through it’s LVDS lines via four Samtec connectors. The Parallella had no viable high speed disk interfaces so we added a eMMC RAID controller and multiple eMMC module slots visible in the upper left hand side of the first board image above.
What is an FPGA? MPSoC?
For those not familiar with FPGA technology, the acronym stands for Field Programmable Gate Array. These chips contain a mesh of electronic parts that can be re-programmed to perform different operations. FPGA devices may even be re-programmed after being purchased and put into the field, hence the “Field Programmable” part of the acronym.
An FPGA MPSoC is hybrid multi-processor chip with an FPGA combined. It usually has 2 or more processors with high speed in chip interfaces like AXI to communicate with circuit designs in the FPGA.
Approaching End of Life
All-in-all we had fun and were able to conduct several tests and experiments using this daughterboard design. Many experiments led us to conclusions that helped mould our security framework which is now being published. The Liquid Router was however severely limited in terms of IO bandwidth and the total amount of resources available. Several flaws resulted from a daughter board design. We knew the Liquid Router did what it could for our research, but we had to move on to a new board design without being dependent on another board as a add on.
Some of our research into hardware security mechanisms required using a MiniITX Zynq-7100 to have enough FPGA resources when a fun side project by our engineers turned into a discovery with vast potential. They essentially implemented a miner in the FPGA to operate on a private Ethereum blockchain where they were able to fit the DAG into the limited 1GB of RAM available on this device. It was able to mine at 3-4MH/s due to bandwidth limits but it only consumed approximately 10-watts of power.
At around the same time, we started to see the prices of the next family of Xilinx SoC’s, the Ultrascale+ MPSoC chips, start to drop dramatically. We started designing a new router based on the Ultrascale+ hoping prices would drop considerably to support industrial and commercial applications by the time our design was ready.
Meanwhile working with our partners we realized the availability of more transceivers combined with new memory controller designs using hybrid memory cube serial DRAM with 160GB/s bandwidth which dramatically increases mining hash rates. Tests and theoretical limits now showed hash rates of 20-21 MH/s using HMC memory in the upcoming Residential Edition of the Blockchain Router.
HMC memory is available today, however we’re setting our sights on a new kind of High Bandwidth Memory (HBM) DRAM soon to be integrated directly into the MPSoC family. A standard 8GBs of HBM at 460 GB/s bandwidth is already being packaged inside early FPGA chips themselves. It’s just a short period of time before we see HBM inside the MPSoC family and this could result in hash rates of 58 MH/s.
These possibilities influenced our design directions on both the Industrial and Residential editions of the new router design which we code named the Blockchain Router family of OptDyn products.
Industrial Edition Production Begins
The industrial edition of the Blockchain Router is designed specifically to target high end mission critical IoT applications. The FPGA is used to host hardware security circuitry to augment the expected broadband router functions. We’ve released the security architecture white paper to coincide with the availability of the router for the development of industrial process control applications requiring the highest degree of security and reliability.
Flexibility at its Core
Although more information about the features of the router is available on the specification page, I wanted to stress the compatible expansion headers and shields supported. Specifically 4 PMod headers, a Raspberry Pi header, and an Arduino hat is available.
Almost every detector/sensor, actuator, working with shields for these platforms found at SparkFun Electronics is compatible with the Subutai Blockchain Router. Virtually any new peripheral connect bus could be added to it. In this respect, the Blockchain Router truly surpasses any other IoT gateway on the market. It’s a laboratory and IoT development platform in its own right. Note that this was one of our primary goals when starting out on the design and we’re really excited about it.
Differences Between Editions
The Subutai Blockchain Router’s industrial grade configuration differs from the residential configuration by using FPGA MSoCs operating between extreme temperatures (-40C to 100C) and having a modified memory architecture on the programming logic (PL) side based on HMC DRAM instead of DDR4. The model of the MPSoC’s differs with more transceivers and cross section bandwidth to handle the requirements of Ethereum’s memory hard mining Ethash function. Most importantly though, the FPGA fabric on the industrial grade MPSoC is used for IoT security functions rather than cryptocurrency mining. The Subutai Mission Critical IoT Hardware Security Architecture defines these security functions and their use in mission critical industrial automation applications.
In October 2017, ARM published its Platform Security Architecture which serves as a guidance with primitives to be used. The PSA goes a long way towards fixing the IoT security dilemma coming with the tsunami of devices and data. OptDyn’s framework compliments these primitives and proposes concrete mechanisms for hardware based real time intrusion detection and prevention systems facilitated by IoT gateways like the Subutai Blockchain Router.
Ready for Production Use
OptDyn’s Subutai Blockchain Router takes its place as the high end IoT gateway device in the Caninos Loucos project serving as the foundation of Brazil’s national IoT standard. Professor Zuffo of the University of Sao Paulo spearheads the project,
“We’re really excited about the Subutai Blockchain Router and its industrial configuration with its mission critical security subsystems,” said Dr. Marcelo Zuffo, who spearheads the project at the University of São Paulo, and leads the national IoT effort for Brazil. “The Subutai Blockchain Router, as a secure IoT gateway, has the ability to enable everything from Industry 4.0 to Smart Cities in Brazil. We’re very excited to complete this last part of the puzzle thanks our collaboration with OptDyn, Inc.”
“Mission critical IoT systems don’t have the option of learning from failures where lives and property are at stake,” said Professor Miguel Matrakas, manager of Celtab, one of the Research Centers of the Itaipu Technological Park (PTI; a foundation responsible for promoting scientific, technological and innovation development linked to Itaipu Binacional, the largest clean energy generator in the world), who surveyed the Subutai Blockchain Router v1.0 at Latinoware with Hall and OptDyn CTO and Founder Alex Karasulu. “The Subutai Blockchain Router and its industrial configuration provide the crucial hardware security infrastructure needed for IoT applications in the utility sector.”
“We’re impressed with the innovation and level of performance possible for numerous Smart City IoT projects using the Subutai Blockchain Router,” said Bruno Freitas, Head of the Innovation Unit for the State of São Paulo, Brazil. “The benefits of using this technology —which is being manufactured in São Paulo as part of the Brazilian national IoT program— are apparent. We look forward to considering the Subutai Blockchain Router as a viable option to meet the demands for IoT projects that build an intelligent and prosperous state and cities.”
The Residential Edition of the Subutai Blockchain Router is slated to go into production later this year after the Industrial Edition enters the market and we’re very excited about it’s ecological impact as an alternative to GPU based mining. It’s coming at just the right time, and offers greater flexibility over hardwired ASIC solutions. We also see the Blockchain Router opening the cryptocurrency mining market to FPGA vendors as well as ISP’s interested in providing them to their customers.
The primary cost of production for FPGA/SoC manufacturers is the startup costs of R&D and setting up manufacturing facilities. The difference between making 10 million versus 100 million chips is negligible. The high costs of FPGA SoC’s on the market is to control prices to recoup these initial startup costs. With a larger market for their chips FPGA manufacturers will naturally produce more and drop prices to earn more in the end. It’s all about marginal product revenue.
The main winner however is the environment. Thankfully only a small minute fraction of the world’s population is mining as we speak. Imagine the strain on the power grid when even 5% of the population starts to mine for cryptocurrency? If we can reduce the power consumption by even 100% the environment wins. We know we can reach higher levels of power efficiency up to 1083% when compared to existing GPU cards.
Tomas Evensen, Xilinx CTO, sees a great future for low power FPGA based mining to prevent the needless waste of electricity from power hungry GPU mining devices, “With the proper FPGA design, custom hash generating cores in the SoC can calculate hashes faster with fewer clock cycles and require much less power than using software running on mainboard CPUs or GPUs in graphics cards,” said Tomas Evensen, CTO Embedded Software at Xilinx. “We’re excited to see the Subutai Blockchain Router use our FPGAs for the consensus layer of distributed ledger technologies —while also protecting the environment.”
We intend to work closely with Xilinx and our partners to develop even more efficient editions of the Subutai Blockchain Router’s Residential Edition with a healthy pricepoint to compete with the costs of ASIC and GPU based mining products. We’re especially looking forward to the introduction of HBM DRAM in the MPSoC family of Xilinx products to raise the hash rates to a jaw dropping 58 MH/s.
What about ASIC Miners?
If you’ve ever felt bad for crumpling a clean sheet of paper after a mistake with indelible ink, then you can imagine the waste for mistakes when designing ASICs. Once burnt onto the silicon there’s no going back. The algorithm design cannot be improved, new blockchain algorithms cannot be supported.
For these and other reasons the process of designing and mass producing ASICs are still very high. Using already manufactured FPGAs does not incur that startup cost while FPGAs can be reprogrammed to improve efficiency even after consumers purchase the product. The first consumers are going to have to absorb the start up costs of these ASICS. We’re already seeing suggested prices of two to three thousand dollars for Bitmain products.
In terms of power efficiency FPGAs and ASICs are comparable. In terms of performance ASICs exhibit 20-50% higher performance. This is offset by the differences in flexibility which is a central theme of the Subutai Blockchain Router.
Also remember there’s no router functionality or multi-purpose function to these devices. The Blockchain Router is a cloud router and IoT gateway first and foremost. Our aim is not to just mine crypto. In fact, it’s never was our primary objective. That’s something the router can do on the side while handling its traditional router and gateway functions. Even if new blockchains with different mining algorithms emerge, the Blockchain Router can be reprogrammed to work with them so consumers will not be stuck with yet another useless electronic device that costs too much. Even if mining capabilities are disabled altogether the router can still use the fabric for its advanced security features.
For those who are unable to watch Open Source educator/evangelist and OptDyn CEO Jon “maddog” Hall’s “The Grand Slam for Latin America” keynote at Latinoware 2017 https://youtu.be/P4GTwfg_1c4 , below is the full transcript:
I apologize that we don’t have any slides for you to see but there isn’t too much to see on the slides anyway so I will paint the picture for you to see in your mind. Sometimes that’s better anyway.
I am Jon “maddog” Hall. I am Chief Executive Officer of a company called OptDyn. I’ve been in the free software space a very long time.
Down there, aiming the camera, is my Chief Technical Officer, Alex Karasulu: “firstname.lastname@example.org”
Beside being Chief Executive Officer for OptDyn, I am also the Board Chairman of the Linux Professional Institute, of which Cesar, who has been entertaining you, is the Director of Operations here in Latin America. I am also the President of Project Cauã, which is a project to help university students make money and start their own company while they are in university.