Data Center Ecosystem Reconstruction

Recently, Rob Ober, a system architect at LSI, gave a detailed interview to the Chinese CEO & CIO magazine, which raised many interesting questions. I would like to bring to your attention a translation of the key points of this interview, since Rob very clearly describes the future of the industry.

CEO & CIO: In recent years, Internet companies have built highly scalable data centers. Unlike conventional companies, IT market players take on the role of leaders in developing data center technologies. From an industry perspective, tell us which three technologies are key in the data center market? Describe them.

Rob: In the super-large data centers, the software, hardware, and pure engineering industries are facing. Therefore, the number of innovations is so large that it is very difficult to choose only three. I would say that the most important are innovations in hardware and infrastructure, and if you need to choose three, I would point out the following points:
')
Autonomous behavior and control

An architect from Microsoft once said to me: “If we hired administrators for our data centers the way ordinary companies do, we would have to hire all administrators in the world.” Now, Microsoft data centers have about one million servers. Ultra-scalable data centers required for their expansion the development of an automatic, self-managing, and often even self-implementing infrastructure. Ultra-large data centers are pioneers in this area, with the help of them, experts learn from their own mistakes, develop practices that allow to obtain an improved work / dollar ratio. We are talking about highly specialized practices, but more and more players in the IT industry are beginning to adopt them. OpenStack is the best example of how specialized knowledge and skills are “packaged” and distributed widely throughout the industry. At LSI, we work with hyper-scalable and automated solutions to create better autonomous infrastructure.

High availability at the data center or individual machines

As systems become larger, they have more components, more points of failure, and this leads to more complex and expensive support for their availability. With the increase in storage volumes, disks start to fail more often. They are simply used more often. And this is against the background of constant pressure, designed to reduce the cost and complexity. Over time, mega-data centers grew very large, sometimes into hundreds of thousands of servers, often spread across different data centers, which led to the need to create solutions to increase absolute reliability, despite the fact that the individual system components became cheaper, simpler and less reliable. All this allowed the use of low-cost components in the "clouds", turning them into a reliable resource.

These solutions also appeared very timely, because many organizations felt the need to maintain their data completely accessible in different data centers.

The traditional approach, which requires server availability at 99.999%, gives way to a pragmatic approach, which is to maintain high availability at the macro level — across the data center. This approach permits failures of individual systems and components as long as it does not threaten the entire data center. Of course, this approach has not yet been fully developed. LSI works with mega-data centers and OEMs to obtain improved operational efficiency and fault tolerance techniques, which will minimize the damage from the failure of individual components while maintaining a reliable layer of high availability for the entire data center.

Big data

This term is too often used. It is hard to believe that a few years ago it did not exist. Hadoop was a real treat for the industry — open source trying to copy Google MapReduce and the Google File System really changed our world incredibly quickly. Today, Hadoop and other big data applications offer us search, analytics, advertising, scalable, reliable file systems, genetic research, and more, even services like Apple Siri use Hadoop. Big data has changed the concept of analytics from statistical simplification to the analysis of all data. And this has already provided many breakthroughs in research in which patterns and patterns are searched empirically, not theoretically.

In general, I think big data has become one of the most transformative technologies of this century. Big data shifted data center focus from compute to storage. Our hard disk controllers, SAS (Serial Attached SCSI) adapters and RAID controllers are at the center of this evolution. Its next step will be the wide dissemination of graph analytics, which will allow analyzing the relationships between data, and not just the data itself.

CEO & CIO: Due to the prevalence of cloud computing, mobile communications and big data, the traditional IT ecosystem in production is changing. What are the three major changes in the current LSI interaction with the ecosystem? How do LSI see changes in various connections in traditional ecosystems? What new connections are worth attention? Please give examples.

Rob: Cloud computing and mobile data availability has already significantly changed and will continue to change our industry and ecosystem. In fact, the corporate market (customers, OEMs, technologies, applications and applications) remained fairly stable for 10 to 20 years, but as soon as cloud computing became a significant part of the server market, it immediately affected ecosystem participants, such as Lsi.

Time: It is no longer enough to follow Intel's well-established, like a clockwork mechanism, product portfolio. Previously, data center development cycles for data centers ranged from 3 to 5 years. But these cycles are getting shorter. Now the need for solutions is approaching 6 months, forcing suppliers of hardware to work with such short development cycles.

Mega-data centers also need the ability to quickly scale up resources in accordance with customer needs. As a result, it is in the data centers that new architectures, solutions and specifications are introduced without the traditional binding to Intel's roadmap. It also disrupts the ecosystem.

End users: hyper-scalable data centers now play a significant client role in the ecosystem. Sometimes their single order can be up to 5% of the server market. Despite the fact that OEMs are still incredibly important, they no longer perform such large implementations and do not develop so quickly. This leads to the fact that often financially win suppliers of individual components or subsystems, if they are able to offer a unique (or at least effective) solution to a real problem. This leads to the fact that the main profit moves from large OEM to strong, fast innovators. This could potentially lead to a decrease in profits for the entire ecosystem, which would be a threat to the growth rate of innovation and reinvestment.

New players: traditionally, several OEMs and several software suppliers almost solely owned the data center market. However, the supply chains of hyper-scalable cloud companies have changed this. The data centers of the market leaders have developed, specified and even built (as is the case with Google) their own infrastructure, although some of the mega-data centers continue to rely on proven solutions from Dell and HP.

More and more, data centers are built according to specifications from suppliers such as Quanta. New network equipment providers, such as Arista, are increasing their market share. Suppliers of hyper-scalable solutions, such as Nebula, are also growing.

The software has moved significantly towards open source with paid support - the model, originally developed by RedHat, is now adopted by Cloudera, Mirantis, United Stack and others.

Open initiatives: yes, we have already seen Hadoop and derivatives being implemented everywhere, even in traditional industries: oil and gas, pharmaceutical, genetic research, etc. And we watched as open databases crowd traditional solutions (for example, Casandra). But now we are seeing new initiatives, such as Open Compute and Open Stack. Of course, they are useful for hyper-scalable data centers, but they also help smaller companies and universities to deploy infrastructure similar to hyper-scalable, and to get the same level of automated control, efficiency and costs as the "big players" (of course, they do not use level of opportunity, but they are very close to this). In the future, this trend can severely damage traditional OEM business models and software vendors and reshape markets in favor of new players, as we can see with the example of Quanta, TYAN, Foxconn, Wistron and others who are just entering the market with new, open initiatives.

New architectures and algorithms: now there is a clear move towards technologies based on a pool of resources. The development of such solutions was made possible through partnerships between companies like Intel and LSI and the architects of the largest data centers. Traditionally, new approaches in architecture were dictated by OEMs, but lately this is not the case. We can see an increasing distribution of solutions aimed at using scalable rack architecture (RSA): silicon photo-tonics, storage pools, software-defined networks, and soon we will see both RAM pools and new types of non-volatile RAM.

In addition, we can observe how new processor architectures regain their place in the data center: ARM 64 for quiet and cold storage and OpenPower P8 for powerful computing, multi-threaded, multi-tasking processing monsters. Behind all this is very interesting to watch. There is growing interest in accelerating applications: general-purpose computations on video card processors, regular expression processors for real-time flow analysis, etc. Right before our eyes, the first generation of graph analysis tools is unfolding.

Innovation: the pace of innovation is increasing, or simply I am getting older. But with fast income over. On the one hand, data centers need an exponential increase in computing power and storage, they need to work faster from 10 to 1000 times. On the other hand, memory, processor cores, disks, and flash drives do not grow as fast. The only way to fill this gap is innovation. So it is not surprising that a lot of interesting things are happening now with OEMs, software vendors, chip makers and ready-made solutions, as well as in the open source community and startups. That's what makes the present so interesting.

Consumption shift: we see a decline in the supply of PCs and laptops, a decline that has led to a decrease in demand for storage in this segment. Laptops are increasingly moving from HDD to SSD, which is not bad for LSI, as our contribution to mobile HDD was small, while the company plays a big role in the SSD market. Smartphones and tablets have led to an increase in the consumption of cloud content, traffic, and dependence on cloud storage. We see a significant increase in demand for large HDD for cloud solutions, this trend is gaining speed, and we think that the cloud cloud HDD market will feel good, and we will see the emergence of new HDD optimized for cloud clouds that are very different from existing and Designed for quiet storage with low heat dissipation.

In cloud storage, there is a growing demand for PCIe SSD cards, which are used for databases, caches, virtual machines, and other applications that require low latency. Much of what we take for granted would be impossible without these flash products with high capacity and low latency. Very few companies can offer viable, flash-based storage at an affordable price. This opens the way for startups experimenting with various solutions.