Interconnection for Internet traffic over IP networks operates according to a different set of rules from telephony. However an increasing proportion of telephone traffic is carried over IP-enabled carrier networks. There are now many different operators offering network capacity to send, transit and terminate traffic, ranging from traditional telecom carriers to third party vendors, from Internet Access Providers (retail) to Internet Backbone Access (wholesale) carriers, from content distribution networks (CDNs) to utilities with spare capacity to wholesale. The commercial terms and ways in which interconnection is offered varies considerable. For example, carriers traditionally interconnect at network Points of Interconnection (POIs) whereas CDNs and cloud computing service companies interconnect in data centres, and Internet Access Providers at Internet Exchanges. Nevertheless, despite its origins and the fact that Internet traffic was never subject to the same regulatory regime as telecoms, certain common practices have emerged.
From its beginnings telecoms was a state-regulated industry, often part of a Post & Telecoms Department, later to be incorporated as a state-owned telecom enterprise (SOTE). Interconnection at the international level was mandated, and under ITU guidelines an accounting rate and a settlement rate system between international carriers was established. With market liberalization came competing networks and the need for interconnection. Often regulators required the incumbent to register a ROI (Reference Interconnection Offer) to ensure equal treatment among carriers. No such system has ever existed for Internet traffic.
University research funding from the US Department of Defence in the 1950s and 1960s and early trials by universities to establish a network of peering devices using IP/TCP protocol matured in the 1980s and 1990s into the first commercial services by Internet companies. These early developments connected computer networks. These were later connected indirectly using capacity from carriers to transit traffic between IT devices such as computers and terminals. As Internet services, for example e-mail, became mass market products for business users and residential customers, access was increasingly over telecom systems. The spread of the World Wide Web in the 1990s created a platform for the exchange of documents and then for the development of down-loadable and up-loadable content and applications.
The Internet had become big business, posing ever growing demands for network capacity on the telecom industry. This posed both a threat and an opportunity for carriers, and most of the dominant ISPs that emerged from the competition were subsidiaries of the carriers. Often by charging high wholesale prices to all ISPs, telecom companies could squeeze the profit margins of independent ISPs without breaking any equal access regulations. The market power of the carriers lies in their ownership of the backbone networks over which IP packets have to travel irrespective of the route they take, and although the use of least-cost routing will save some money, that only works if there is a competitive wholesale market.
ICAIS (International Charging Arrangements for Internet Services)
It is slightly ironic that the big dispute over IP interconnection that arose in the 1990s, and which still echoes to this day, for example, it resurfaced at the 2012 ITU WCIT-12 in Dubai, was not between ISPs and telecom companies as such but largely between the telecom companies that own most of the ISPs. Overlaying the dispute was the fact that the Internet originated in the USA. In 1995, the US government decommissioned the US National Science Foundation Network (NSFNET) which had been the backbone for most IP traffic in the US and handed over interconnection to four Network Access Points (NAPs). Since then other entities have arisen, some serving academia as education and research networks, others commercial networks including specialist Internet Backbone Access providers. Academia peering arrangements are not difficult to agree, but for commercial service providers peering arrangements are all about market power. The basic rule is that smaller ISPs either cannot peer with larger ISPs in which case they have to find ways to aggregate their traffic to reach critical mass, or reach special agreements with carriers, or pay premium rates for interconnection.
Internationally the same rules apply, but the larger ISPs have for historical reasons been in the USA, and later to a lesser extent in Europe. There is no accounting rate or settlement rate procedure for ISPs and the de facto position is that the major US carriers have always been free to charge the full cost of the international links to ISPs outside the US. In the 1990s there were intense arguments between carriers and even between states over this apparent inequality. For example, in 2000 Telstra’s Managing Director of Global Wholesale Business claimed that up to “70% of an Australian ISP’s costs are due to the international segment to the US.”* In reality the issue is an old one: regulated rates versus market rates.
The way markets work is that imbalances between supply and demand will be reflected in prices, and high prices should act as an incentive to remove the supply bottlenecks. In this case the bottleneck was outside the USA where domestic IP traffic, in the absence of a local Internet Exchange Point (IXP), had no option but to route through the US. To justify the expense of a local IXP there needs be a critical volume of IP traffic. All markets thrive on liquidity, and in this case the liquidity in the Internet market means traffic volume. Unless there are structural impediments to the growth of local Internet traffic, such as a monopoly provider, the market mechanism should result in more local IXPs. This should result in more balanced flows of international traffic which in turn should allow more ISPs to enter into peering arrangements with their US and European corresponding networks. The spread of IXPs seems to be exactly what is happening as explored in section 3.4.1.
3.4.1 Internet Interconnection and IXPs in Developing Countries
To be cost-effective, interconnection between circuit-switched TDM (Time-Division Multiplexing) telecom networks requires points of interconnection (POI) that minimise route distances. For price arbitrage reasons service providers may choose a more round-about routing of traffic, but technically the more direct the routing the more efficient it is and the less latency involved. In a packet-switched world of Internet Protocol (IP), a different set of principles operate. Because different packets of the same transmission are routed over different networks there is no single POI. ISPs do not always own their own networks and there is no guarantee of the quality of the networks over which the packets will route. So unless the network was ‘managed’ and its quality assured, Internet traffic from its earliest days was only ‘best effort’. Investment in broadband in recent years means network quality has generally improved and with more sophisticated routing algorithms ‘best effort’ is now often of very high quality. For example, over-the-top (OTT) voice and video services like Skype and Yahoo Messenger, Facebook and Google that are transmitted internationally over broadband networks can be crystal clear with minimal latency. In addition, a range of specialist managed Internet networks have arisen such as CDNs that guarantee quality of delivery.
ISPs come in three tiers: Tier One ISPs are usually affiliated with a licensed carrier having direct access to an international network, although some of the larger Internet-based companies have begun to build their own networks. For example, Google is ranked third in the carriage of global traffic behind Level 3 and Global Crossing. Tier Two ISPs own or have direct access to local networks and may serve a regional market but require IP transit for international routing. Tier Three carriers have to lease lines and peer with larger ISPs, in some cases as paid peering, to achieve end-to-end delivery of traffic or IP transit. The larger ISPs also provide the connecting networks for IP transit which are known as Autonomous Systems (AS) and are assigned an Autonomous System Number (ASN). The ASN identifies them as using the appropriate routing protocol for IP transit traffic, also known as the Border Gateway Protocol (BGP). When using IP transit, ISPs provide and receive from each other routings to facilitate traffic to and from the customers of the ISPs involved.
Unless the ISP is affiliated to a licenced carrier there is no guarantee of interconnection. Large carriers such as incumbents may reject interconnection with smaller providers for commercial reasons, not technical or regulatory reasons. Alternatively they may impose draconian interconnection charges or high prices for leased lines resulting in profits squeeze of independent ISPs. The lack of domestic interconnection forces ISPs to route their domestic traffic through Internet Exchange Points (IXPs) or to pay for peering to send traffic overseas. Their traffic becomes transit IP traffic which they have to ‘trombone’, that is send over several different networks before it reaches its destination. Naturally, this adds to its cost and to the latency problem.
In the 1990s, IXPs were typically in the US and it is still the case that many countries route much of their domestic traffic through the US. According to Packet Clearing House (PCH), as of May 2013, about half of the world’s 199 countries are without IXPs’* By contrast, only four European countries are without IXPs, while in Asia Pacific the countries without IXPs are largely Pacific Islands. Most South American countries have IXPs, but there are fewer in Central America and the Caribbean islands. “At present only the British Virgin Islands, Haiti, Grenada, St Maarten, Curacao and Dominica have IXPs. In conjunction with the Caribbean Telecommunications Union, PCH is currently assisting several other Caribbean countries, including Barbados, Jamaica and St Kitts and Nevis in establishing local IXPs.” (See Toolkit Broadband in St Kitts and Nevis: Case Study. ) Mexico is the only OECD member country not to have an IXP. *In Africa there are upwards of 20 or more countries with IXPs, but most are small and serve only very localized markets.*
According to one source, 85% of Africa’s traffic routes through Europe and only 1% stays within the region.* South Africa is the major hub, but at least 16 East African and Southern African countries also use the KIXP in Kenya. (See Box 3.7 and also the Toolkit Broadband in Kenya Case Study.) One report also suggests that Nigeria’s IXPN is preparing to provide peering for West African countries.*In North Africa and the Middle East, Egypt has three IXPs in Cairo, and others countries with IXPs include Lebanon, Israel, the United Arab Emirates (UAE), and a state-run IXP in Saudi Arabia. However, the region remains under-served as does Central Asia where, as of the first half of 2013, only Kazakhstan, Mongolia and Uzbekistan had established IXPs.*
Kenya has two IXPs: the first , known as Kenya IXP (KIXP), opened in Nairobi in 2000 and the second in Mombasa in 2010. They were set up with the assistance aid from CISCO and UNESCO, and are operated by the Telecommunications Service Providers Association of Kenya (TESPOK) which is a non-profit organization representing ISPs and telecom service providers. KIXP operates a Multi-Lateral Peering Agreement (MLPA) whereby ISPs are required to interconnect free of charge, but each pays a usage fee to KIXP.
The success of these IXPs is in evidence from a number of measures. By April 2013, membership of KIXP had reached 30, up from 25 in April 2012 and included the mobile and fixed line operators, an educational network called KENET, the National Bank of Kenya, and government agencies such as the Kenyan Revenue Authority (KRA). Aggregate traffic throughput has jumped from 64kbit/s at opening in 2000 to over 1Gbit/s today. One estimate of the cost savings to Kenyan ISPs from not having to pay the cost of international transit and trombone is $1.44 million per year.* ISPs from other African countries are starting to use Kenya’s IXPs; 56% of the ASNs routed through KIXP in the six months to January 2012 were from 16 foreign countries.
Cache and the Digital Economy
The most important step up in usage came after the installation of a Google Global Cache (GGC) in April 2011. Traffic volumes rose more than ten-fold within the year, the lion’s portion of it was streamed video, for example from YouTube, as latency dropped by 20% on top of the initial fall in latency when the KIXP was first opened from 1,200-2,000 milliseconds (via satellite) to 60-80 milliseconds.* The caching capability builds a foundation for local content generation and distribution at affordable prices. Improved latency also bolsters the IXPs capability to become the driver of growth in cloud computing in Kenya.
Conditions for Success
Despite the success of KIPX, it almost faltered in 2000 when it had to close business pending a decision by the Communications Commission of Kenya (CCK) to grant a licence to operate a telecommunication service. This followed a complaint by Telkom Kenya that its monopoly over international traffic was being violated.* KIPX argued it only directly handled domestic traffic (see the Toolkit Kenya Case Study) and the decision to grant a licence has served Kenya well.BOX 3.7The Success of IXPs in Kenya
3.4.2 The Economics of IXPs and Wholesale Charging
Having to trombone traffic adds to cost and to latency, from between 200 to 900 milliseconds in the case of African ISPs.* In the early years the cost factor was the most important consideration for two reasons. Firstly, the cost of international circuits was high and US carriers in particular required overseas ISPs to pay for the full cost of the circuits. By contrast, among carriers transmitting telecom traffic, including packet data such as frame relay, the ITU-approved accounting rate system was used in splitting the costs 50:50 between transmitting and receiving carriers. The settlement rate system could vary the split ratio in certain cases; for example the split between Hong Kong and Mainland China before Hong Kong returned to Chinese sovereignty in 1997 favoured the Mainland. However, Internet traffic never became part of the accounting rate system, and in the US for regulatory purposes the Internet was defined as an information service rather than a telecommunications service.
The second reason is that in the early years the main Internet service was email for which latency is less of a problem. By contrast, a service such as search is highly sensitive to delays. A study in 2009 found that a two second delay on Microsoft’s Bing search engine caused the number of queries to drop by 1.8% and revenue by 4.3%, and a 400 millisecond slowdown caused a fall of 0.59% of queries through Google.*
Since the collapse of the dot.com bubble in 2000, international circuit costs in submarine cables have dropped to a fraction of their former price, a trend that was reinforced as cable capacity soared following the recovery in financial markets in the mid-2000s. The trend was uneven. In some regions both cable capacity and satellite services remain limited and costs relatively high, such as in the Pacific Islands; in other regions the changes are more recent, as in Africa where new cables are now coming online.* Reduced international prices have had a major impact upon the cost of Internet traffic. An OECD assessment of the voice-equivalent cost of Internet transit traffic in 2013 is “USD 0.0000008 per minute – five orders of magnitude lower than typical voice rates.”* However, as the report also points out, local access charges levied by telecom companies consistently seem to account for between 30%-40% of total international transit costs. At the same time, Internet businesses have undergone a complete transformation to create a digital economy, everything from search to e-commerce, from social media to e-Government, from online video content to online gaming. In 2011, it was estimated that the Internet-based digital economy contributed 3-4% GDP to the G-8 nations plus Brazil, China, India, South Korea and Sweden.*
If a flourishing local Internet can generate so much local economic activity and contribute so much to social welfare, for policy-makers and regulators these statistics are just too important to be ignored. The danger is that smaller ISPs can be easily hindered from reinvesting in their business due to high wholesale prices and profits squeeze and the local Internet economy will suffer. There may be a need for regulatory intervention if wholesale charges are clearly discriminatory against ISPs not affiliated to the telco. However, this can be a difficult policy to pursue because a telco may also squeeze its own ISP so that downstream margins are sacrificed to maintain upstream margins and market dominance. In fact most IXPs have not come about through regulatory intervention but by voluntarily market agreements.
In some cases the state itself establishes an IXP, which can be motivated by the need to address market failure, but the motives could be more political. Much more often IXPs are established either as non-profit entities, sometimes by universities or NGOs or associations of ISPs, or as commercial businesses. Most IXPs in the US are commercial, most in Europe, Latin America and Africa are non-profit and there is more of a mix in Asia. For example most commercial IXPs are in Australia, China (including Hong Kong), Japan and Singapore and non-profit IXPs are mostly in India, Nepal and the Philippines.
The commercial IXPs usually co-locate ISPs in data centres, with various charging schemes including charging for ports or capacity usage, rack space, connection fees and/or a range of management and security services. They can be carrier-related or carrier-neutral, co-location neutral or ISP-specific. The non-profit IXPs are usually dedicated operations which only charge cost-recovery fees and facilitate peering between members, likely to be at no charge between ISPs, although in some cases it can be paid peering if the balance of traffic is too one-sided. A study by the OECD covering 86% of the world’s Internet carriers in 96 countries found that 99.51% of peering agreements were made by “handshake”.* Peering arrangements can be single-hop, bilateral or multilateral, and occasionally the latter can be a condition of joining an IXP, as it is to join Kenya’s KIXP or in Chile where peering is mandatory.*
What is common to all of these arrangements is that, even where peering is mandatory, the terms and conditions are not. Regulators have seen the advantages in leaving developments to voluntary agreements between ISPs and other parts of the Internet ecosystem such as CDNs, major content producers, Internet search and social media companies, OTT service providers, etc. The over-riding reason why this has been the right way to do things is because these are all fast-moving and rapidly developing businesses with a need to innovate and experiment with what works best for them. Regulation that addresses market failure might justify mandatory peering if there is a real fear that the ISP attached to the incumbent can prevent new entrants from entering the market. There is little evidence to suggest that discrimination is sustainable as substitutes emerge for network connectivity and Internet access and for the delivery of apps and content. These substitutes include mobile networks, satellite networks, CDNs, WiFi networks, IP ‘connected’ TVs, etc. In a developing country where these substitutes may still be in a nascent stage, market failure could be a barrier to growth in the digital economy; the work-around market failure is to reform regulations to facilitate new entrants into the Internet ecosystem.
3.4.3 Future Charging Arrangements and Developments of IXPs
When Internet companies such as Yahoo! and Google and major content distribution networks (CDNs) such as Akamai, Amazon and Limelight invest in a local server to cache content from overseas the stimulus to the local IXPs is immediate.*
This is especially significant for developing countries where international connectivity remains a problem, because a cache connected to an IXP reduces latency and thereby encourages local demand and usage. In the example from Kenya (see Box 3.7) following the installation of a Google Global Cache in April 2011 traffic at the KIXP rose from just over 100 Megabits every second to well over 1 Gigabits every second in just over two months, most of this accounted for by YouTube downloads.* This in turn can create a market and act as a stimulus to local developers of apps for entertainment and for practical use, such as financial and locational apps for use on mobile phones, and to producers of content. In this way a digital economy gets built up.
As the digital economy grows, so will the number and type of parties connecting to IXPs. In Europe between 2008 and 2010 the percentage of connections to IXPs from content providers increased from 85% to 96.3%, by VoIP providers from 36.8% to 48.1%, by enterprises such as airlines and banks from 30% to 46.2%, by search engines from 25% to 48% and by governments from 50% to 77.8%.* For developing countries, these are trends to take note of as IXPs have an important role in triggering and accelerating the local digital economy.
P2P, OTT and Cloud
While peering or P2P delivery of Internet traffic was the major growth trend of the late 1990s, it has been overtaken by direct download of apps and video content which comes OTT of the fixed and public land mobile networks (PLMNS). A particularly important part of this development is the growing use of cloud computing. Many email servers, for example, are now based in the cloud where emails are stored and retrieved by users instead of being downloaded and stored in computer hard drives. Cloud computing has given rise to a whole new generation of cloud-based services, such as Software-as-a-Service (SaaS), Platform-as-a-Service (PaaS), Application-as-a-Service (AaaS) and even Infrastructure-as-a-Service (IaaS). Specialist cloud service providers have joined CDNs, content creators, application service providers and others in locating in an ever increasing numbers of data centres and countries now directly compete with each other to host these data centres. Countries with reliable broadband infrastructures and appropriate personal privacy and data protection laws in place to safeguard the international transfer of commercially sensitive data between jurisdictions have a competitive advantage.
Charging in an NGN Internet World
More of all of these services are being accessed by mobile wireless devices such as smartphones and tablet computers over PLMNS and WiFi networks. These trends have important implications for the business models of the traditional telecom providers; their pricing models in particular are being redesigned. The idea of charging by-the-second or by-the-minute of usage does not work in an Internet world. Charging by capacity makes more sense and as voice traffic declines as a revenue earner, and as OTT substitutes such as social media chat services and texting become ever more popular, telecom companies are moving towards bundled voice and data services. Bundles are frequently offered at flat rate charges, sometimes with tiered flat rates: each tier with its own capacity ceiling.
The old model of charging termination fees to networks for the delivery of their traffic also comes into question in an Internet environment, for two reasons. First, because there are now many ways for users to access the Internet. Second, as networks upgrade to broadband, telecom companies may be tempted to charge both sides of the market—the providers of services over the Internet and the users—in what is called a ‘two-sided market’. This goes the nub of the net neutrality issue. (See Module 3.7)
In practice most carriers, fixed line and mobile wireless, will make the transition cautiously, not wanting to cannibalise their existing lines of business, such as call services, too early as long as they continue to generate revenues. But as they invest more in all-IP NGNs their billing arrangements are likely to come closer to the charging mechanisms between ISPs, which are mostly Sender-Keeps-All (SKA) – also called Bill-and-Keep (BAK). Where interconnection charges are levied, for example where the balance of traffic is very uneven, it is likely to be capacity-based charging as the marginal costs of sending a packet are very close to zero.