Sustainability is a topic I’m interested in both inside and outside of work. I’ve been doing a lot of research lately regarding data centers and energy efficiency so that I can apply my interest to my day job. In particular, while probably the smallest source of power consumption within the data center, I’ve spent most of my time researching the power consumption and energy efficiency of network equipment. This research has led me to represent my company on The Green Grid as a voting member in the Network working group. The article below is a summary of several days of research on the topic of “Green Ethernet”.
Why you should care about sustainability
Data Centers today use about 3% of all energy within the United States. A typical 100MW data center uses the equivalent power of 9,000-34,000 homes (depending on whose metric you read and considering that power consumption is variable). About half of that energy is consumed by server infrastructure with cooling/facilities consuming about 40% and network infrastructure consuming around 12%. So, if your data center power costs are $1M/year, then the cost of powering your network infrastructure is around $120k/year. This may seem like a small number, but even minor changes in more efficient equipment, or operational behavior changes, can reduce power costs 10-20%. At $1M/year data center power costs this may seem insignificant – but at $200M/year data center power costs, with $24M/year going to power network infrastructure, a 10-20% efficiency gain can be several million dollars.
Let’s be honest – data centers are expanding and energy costs are also going up so this problem only gets bigger. While making your network more ‘green’ probably isn’t how you’ll invest your limited time to reduce energy costs, it is still a topic that a few simple behavior changes can make a measurable difference. After all, in 2006 it was estimated that network elements in the U.S. consumed about 3billion kWh of power. That’s close to $500M/year at 16cents/kWh.
What should you do tomorrow?
If you choose to not read the rest of the article and become fully educated on this topic, here are a few simple things you can do tomorrow to start making positive changes towards a greener network:
- Develop your network infrastructure energy metric: There are industry standard formulas for this which I will discuss later. Hopefully your network equipment manufacturer can provide you this metric on a per-platform basis so you don’t have to test it yourself. Using your networks capacity data, along with some of the energy variables outlined in this post, you should be able to determine the energy cost of your network infrastructure. You should also be able to model potential cost savings
- Consider energy costs when doing product selection: Your energy costs are part of your total cost of ownership, after all. W/Gbps is a good comparison metric. In my experience, however, this is negligible when comparing apples to apples equipment (e.g: top of rack switches on Trident 2 chips). This is a better comparison metric when comparing older equipment versus newer equipment. You may find that the ROI of replacing older hardware for more energy efficient equipment can make financial sense
- Consolidate or power down orphaned equipment and/or interfaces: By far, the best way to reduce energy costs is to power down a network device completely. There are incremental gains by powering down interfaces and further down I detail some of the gains possible there
- Reduce power supplies if you have network redundancy: Depending on the specific device, and this is something to consult your manufacturer about, dual power supplies may mean more power consumption. If you have redundancy in your network and system architecture, then it may make sense to run on a single power supply
The rest of this article goes into detail about energy consumption of network infrastructure and industry initiatives to have greener networks.
How much does it cost to power a typical network switch?
Using a 16cent/kWh energy cost and an “always powered on” assumption, the following are example costs for typical home devices and network switches:
24wh ceiling fan ($34/year)
33wh Cisco catalyst 2960 @ idle ($46/year)
45wh for laptop ($63/year)
100wh light bulb ($142/year)
130wh for Cisco 3560E @ idle, 142wh@ peak ($199/year)
143wh for Cisco 3750G @ idle, 161wh@peak ($225/year)
185wh Xbox 360 ($264/year)
200wh for LCD TV ($288/year)
It is also important to note that a 100wh switch emits 596lb of CO2 (carbon emissions) on an annual basis. And that for every W of energy used, another 6-7W is used for transmission, generation, cooling, etc.
What components consume what power?
From the “Climate Savers Computing” guide (see reference below), the breakdown for a typical edge switch power consumption is 45% to ASICs, 25-30% to power supplies, PHY components 10% and memory and cooling each come in around 5%.
What affects power consumption on a network device?
Manufacturers are increasingly making more energy efficient network processors. For example, Broadcom’s Trident 2 chipset claims 2x forwarding performance at half the (CPU) power consumption.
The processor is only a small factor in the overall energy consumption of a network device. Here are a few other variables that affect power consumption modeling a typical 1Gx48 copper ethernet switch with fixed ports.
- 151W – baseline – device powered on and ports down
- 184w – every port up – 20% increase in power consumption by bringing every port up – each port uses generally .7W-2W each
- 195w – full traffic on every port – an ~8% gain in power consumption
Other factor that variably affect power consumption are:
- The interface type – the more bandwidth then the more power that is consumed (e.g: 1G vs 10G vs 40G vs 100G)
- If an optic is plugged into a port cage – the optic requires power
- PoE – Power over Ethernet support – if PoE is not required, then disabling PoE is a simple way to reduce power consumption
Most products will include a power consumption metric on their product datasheets. This is generally the maximum possible power consumption under heaviest load and configuration. It is generally not a good metric for power consumption due to the variations noted above.
What are useful network infrastructure energy metrics?
The two most commonly used metrics for modeling the efficiency of network equipment are:
- W/Gbps: This is a metric that can be simply derived from product data sheets where a vendor will generally always publish their maximum power utilization as well as maximum forwarding capacity. Some may not consider this to be a good real-world metric
- ATIS TEER: The ATIS TEER metric is probably the most real world and industry recognized metric and measures power consumption at different load/utilization scenarios like idle, peak and low utilization.
Some manufacturers, Arista Networks for example, also display/show the current power consumption for their network devices simplifying the task altogether.
Other costs of network infrastructure
If you really want to consider every aspect of a green network, a few other ideas to consider are:
- Shipping and shipping material – just think of the cost of every item in a typical Cisco switch shipment – the cardboard, foam, manuals, cables, etc as well as the shipping costs (cargo, fuel, etc). Facebook reportedly uses shipping containers to bulk ship data center equipment from suppliers
- Sheet metal and component costs, faceplates, LEDs, etc
Energy Efficient Ethernet (EEE)
Most network devices today do not consume power in a way that is proportional to load or utilization – as evidenced by the use cases above. Energy Efficient Ethernet is a solution that attempts to address that.
The general concept of EEE is that a Low Power Idle (LPI) signal is sent to indicate that the link can go to sleep.
EEE currently only works with copper interfaces (it is coming to 40G and 100G standards though) today and requires both ends of a link to support EEE (EEE support is negotiated during interface auto-negotiation). There are also shipping products with the Juniper QFX5100 claiming to support EEE. The estimated energy savings for a 48 port switches can be as much as 15w - which is probably around a 10% reduction.
Green Grid Maturity Model
The Green Grid publishes a maturity model for measuring how well you manage energy efficiency of your network equipment. This is a good starting point for introducing a green network initiative into your company. Below is a snapshot from that maturity model. The full model can be found below:
What else can you do in the data center?
If you are going to invest time into data center energy efficiency – you’ll probably get ‘more bang for your buck’ looking at other approaches. Here are a few ideas:
- Raising the ambient operating temperate within your data center – SAFELY. This can reduce cooling costs.
- Using outside air cooling and eliminating cooling costs entirely
- Using more efficient servers since they are a large part of your energy costs
- Using clean power such as wind, hydroeletric or solar – Green Peace produces a data center “Clean Energy Index” that states Google uses 34% clean energy and Microsoft <20% clean energy
- Consolidating workloads – such as virtualizing the network switch