Can users shape traffic better than ISPs? Some Lessons From The Electric Industry.

A dialog between David Weinberg and Seth Finkelstein on David’s blog raises an interesting question. Dave W argues (as do I) that a network provider is the last person who should engage in such practices, because of the inherent potentials for mischief and the possible conflicts of interest. Seth Finkelstein argues that, as a practical matter in the real world, only the ISP can effectively make a determination on traffic shaping that maximizes the use of the network for everyone, protects time sensitive applications, and prevents a “tragedy of the commons” from a handful of users absorbing all the bandwidth.

David Isenberg (in the comments and in this blog entry) makes the case that we don’t need traffic shaping, just more capacity or, in the alternative, neutral means to reduce packet flow such as throttling all traffic equally or going to metered pricing. Others (including myself) have argued that the problems of “bandwidth hogs” are exaggerated, or that users dissatisfied with the “best efforts” environment of the internet should stick with the network optimized for voice (the phone network) or the network optimized for video (cable, broadcast television) rather than “break” the internet to better accommodate these applications. Neither of these answers, however, is popular in regulatory circles. Further, it is a legitimate argument that we should allow ISPs to choose what product to offer customers. If an ISP wants to offer services optimized for VOIP by retaining the power to shape traffic, why shouldn’t it bring that service to market? This inevitably leads to a debate on market power, availability of choice, switching costs, captive customers etc., etc.

So lets shake things up with something new. I will — for the sake of argument here — accept the proposition that we “need” traffic shaping (like I “need” “scare quotes” so that people will not “quote” me out of context or argue on trivialities). But accepting the need for traffic shaping does not mean ceding all power to the broadband access provider. To the contrary, I argue that we will achieve far better results by giving subscribers the ability to shape their own traffic.

Madness you say? “Tragedy Of The Commons” and all that. Maybe, but the electric industry tells a somewhat different tale. As described in this NYT story, a fair number of folks are taking advantage of pilot projects that allow people to shape their power usage in the same way I propose allowing them to shape their Internet use. Such programs may save $70 Billion in the next few years. Why not see if they can have serious impact on the supposed exaflood of internet traffic that supposedly justifies traffic shaping? Especially when contrasted with the pur privatization model, that gave us the Enron scandal and the California black outs in 2001?

More below . . .

I must begin with an observation made by Jon Peha a last year’s Federal Trade Commission Broadband Task Force hearing on net neutrality. After a discussion of some of the efficiencies of letting broadband access providers prioritize traffic v. the inherent dangers in letting them prioritize traffic (particularly in the current marketplace), Peha concluded (paraphrasing from memory): “no regulatory scheme that protects us from all the dangers will capture all the efficiencies of letting broadband providers manage the traffic. But no system that lets broadband providers have unlimited control can protect us from all the dangers. What we need is a real conversation that recognizes the legitimate issues on both sides and tries to find the best trade off.”

I think that’s a good approach, especially for purposes of this argument here. As I have written extensively before, I think that allowing cable cos and telcos to ration bandwidth by shaping traffic is a disaster for getting new capacity built, because you can make more money rationing bandwidth and reselling it to all the third parties that want to reach subscribers than you can by selling a bigger pipe on the subscriber side. I also have grave concerns about the impact of allowing cable cos and telcos to shape traffic on free speech and other vital but non-economic interests, as well as how it will aggravate the digital divide by creating “virtual redlining”. So I (at least) see real dangers of letting the cable cos and telcos shape traffic.

I also suggest that telcos and cablecos have a strong interest in maximizing their own revenue rather than maximizing user utility, which potentially creates a serious problem. Finally, given the limited choices in the market, the difficulty in switching, and the high cost of building a competitor that would match what I think are the right priorities for traffic shaping, I am extremely reluctant to entrust the decision to the carriers. Comcast may think it is worth trading off BitTorrent for VOIP or IPTV of a sort that Comcast likes, and subscribers may live with that because they have no choice, but such a regime ensures that new innovative uses, such as the next BitTorrent or SlingBox, will have a hard time developing and reaching willing users.

And finally, the problem with a carrier decision is that it is not terribly granular. A decision by Comcast or AT&T to adopt certain practices will, in a stroke, effect tens of millions of customers, warping the rest of the market and shutting off some unknown number of users who would chose differently.

So rather than turn traffic shaping and QoS over to the carriers, or to third parties whose choices will distort the market away from true user preferences, why not turn QoS decisions over to the users themselves?

We have already seen the most primitive forms of this idea in the development of edge-based QoS solutions and metered pricing. Things like caching technology (move the content closer), distributed computing (distribute the work among many more computers), and virtual private networks (control of security and privacy by the communicating machines at the edges) are all ways in which end users of various kinds achieve the quality of service they want. Certainly these are not perfect solutions, and network operators can replicate them. But, rather like the magicians of Pharoh replicating the trick of Moses and Aaron of turning a staff into a snake, the fact that network operators can replicate these technologies is not the point. The point is that these primitive first steps at end-user managed QoS rather than network provided QoS are a sign that the folks on the edge do not need to remain in bondage to the telcos and cable cos in order to enjoy QoS. Let end users go and they will provide for themselves.

Similarly, use of metered pricing and capacity caps are demonstrated (but rather primitive) ways of aligning user incentives with network limitations. Understandably, these are primitive and unpopular in the market place. But network operators have the ability to sell customers a more sophisticated tool kit to customize their own traffic. For example:

1) Tell customers when excess network capacity is available or when there is congestion: If I want to use a high-bandwidth feature, allow me to set an “alert” for when the network has available excess capacity I can use, or have the system warn me when I try to use an application that requires high capacity if the network is reaching a level of congestion where this ap might fail. Customers want their applications to succeed. If I am trying to upload the Bible or download a film, I would prefer to do it when the application will actually work. If I am doing something time sensitive, I can still take my chances.

This approach can be used in combination with metered pricing or capacity caps to align user incentives. For example, instead of a vaguely worded “We can kick you off as a bandwidth hog but we won’t tell you the capacity limit” policy, why not say “you can use this much capacity per month, but you can stretch that by shifting your high capacity uses to times of low congestion.”

2) Sell customers excess capacity, or allow them to trade capacity among themselves: Comcast already does this to some degree with its Powerboost feature. Why not tweak it? Let customers buy excess capacity as available, or swap it among themselves. Customers could say “I’m willing to tolerate degradation of my traffic if you pay me, alert me when someone wants to buy a ”powerboost“ and I’ll drop off the system or accept that my traffic will slow to a crawl.” Or the ISP can sell excess capacity to the customer, or openly bribe customers to take a “back seat” in times of congestion.

Why would I let users buy and trade excess capacity but I won’t allow third parties to pay? As I’ve written in my Whitacre Tiering piece and other writings in support of Network Neutrality, the chief evil and chief source of danger is allowing the network operator to overide user preferences (either for its own purposes or in response to third party payments). So while network neutrality opponents spend much time arguing as if we on the NN side want some kind of socialist utopia in which bits stop moving according to one’s ability to pay and instead move according to one’s need, it ain’t so (at least on my part). For me at least, it is about setting up a system where the network operator cannot assert control over the end user, because allowing a network operator to assert control over the end user produces disastrous results for our economy, our democracy, and our society.

These suggestions all have parallels in other industries, and the technology to implement them is no more difficult or expensive to develop and install than the what the carriers are doing for deep packet inspection and content filtering. And while many users probably won’t find such sophisticated offerings tempting, you don’t need to impact the behavior of many users to have significant impact. Those in favor of allowing ISPs to engage in whatever traffic shaping they wish repeatedly claim that it is only a handful of “power users” — often engaged in illegal download activity — that are creating all the congestion. Isn’t it more efficient overall, and better for all customers, to shape solutions to the “QoS problem” and “congestion problem” by targeting this sophisticated group of users and, if you will excuse me, using market-type mechanisms to align the incentives of these power users with the limitations of the network?

The electric industry offers some useful lessons. In the late 1990s, everyone was absolutely intensely positive that deregulating power generation would have drive down prices, positively influence consumption, and encourage investment in power generation and infrastructure where needed. Those favoring deregulation promised that market forces would drive power producers to compete and thus lower prices to competitive levels. Where demand, prompted a rise in prices, users would moderate their behavior and producers would have economic incentive to build more production facilities. A win-win for everyone and far better than the tired old “natural monopoly” theory with its tightly regulated prices limited to a mere 10%-15% return on investment, scheduled gradual mandatory build out, etc., etc.

So California and a number of other states deregulated. Only it turned out the practice did not work as the theory predicted. Enron and other producers manipulated markets to make quick cash through arbitrage while neglecting investment in facilities. The market did little to moderate prices in times of surplus, as producers simply shut down excess capacity. But prices rose dramatically in response to demand or increases in fuel costs because producers could pass these on to consumers. The power market proved difficult to enter for new comers who might have competed with the existing producer cartel, particularly given the ability of incumbents to use their superior position to prevent new entrants from becoming a serious threat.

Flash forward to now. As states debate various plans to control their energy future or even place home electric use under direct state control, a study by the U.S. Department of Energy shows that consumes in a pilot program liked the idea of making their own decisions on energy use based on technology that empowered them to do so in real time. The result savings had significant impact on the energy grid in times of congestion, and could result in billions of dollars in infrastructure savings while avoiding outages during peak load times.

The cases are not exactly parallel, but they do dispel the myth that consumers left to their own devices will never opt to ration themselves. Given the tools and the incentives, enough people get sophisticated enough about their choices to have real impact. Nor do I calim that my off-the-cuff proposals above represent the perfect solutions. But I am saying we don’t have to accept the false “net neutrality v. QoS” frame pushed by the carriers and their supporters. Given the carrier interest in owning the customer, however, I’m rather doubtful they will embrace these suggestions any more than power companies have proven eager to push conservation. For the same reason, I am suspicious of carrier experts who dismiss such suggestions with nothing better than “Don’t sell the bike shop Orville! I’m an expert so I know it won’t work.”

Which brings me to the final lesson from the power industry. Electric companies never embrace conservation plans unless forced to by government agencies (either by imposing them directly or threatening to do so absent “voluntary” measures). It is no more rational to expect “the market” to force broadband access providers to empower users to shape their own traffic than it was rational to expect power producers to push for voluntary conservation. If we ever expect to see users given a chance to shape their own traffic, we will need something better than a pious hope that the cable cos or the telcos will prefer real solutions to capacity limits over short-term profits.

Stay tuned . . . .