The following are the outputs of the real-time captioning taken during the Tenth Annual Meeting of the Internet Governance Forum (IGF) in João Pessoa, Brazil, from 10 to 13 November 2015. Although it is largely accurate, in some cases it may be incomplete or inaccurate due to inaudible passages or transcription errors. It is posted as an aid to understanding the proceedings at the event, but should not be treated as an authoritative record.
***
>> MODERATOR: Good morning, everyone. On behalf of the Brazilian Steering Committee, I'd like to thank the workshop participants who have accepted to contribute with our workshop. And I'd like to welcome you in this Friday morning to have come along to participate in this workshop.
While I have ‑‑ before starting, I have just a few reminders.
Today, we have two absences in this workshop because of health problems, which is Milton Kashiwakura from Brazil and Mr. Katsuyasu Toyama from Japan. Mr. Katsuyasu will be phoning us remotely from Japan.
And there is Antonio Moreiras which is ‑‑ works at NIC.br that is replacing Milton in this session.
Another thing is that we should be a round table, but the room that is available for us is a room for panels. So I feel the participants of the workshop are here in the first row.
And, also, last thing is start with the feel ‑‑ talk of each participant. And after that, questions can be posed and comments can be made by the people who are watching the workshop.
So I will pass the microphone for Antonio Moreiras who is the moderator of this workshop.
>> ANTONIO MOREIRAS: Good morning, again. You see two persons with health problem. This issue of IXPs must be very demanding really.
Well, thank you, everyone, to be here to debate this important issue. It is IXP sustainability.
I'm Antonio Moreiras. I'm one of the managers working at NIC.br working to deploy and maintain IX.br.
We have here Jane from Internet Society, Henrique Faulhaber from the Brazilian Internet Steering Committee.
Sebastian from Internet Society, Ariel representing CABASE and the LAC‑IX, and Malcolm from LINX.
Sorry. Very sorry. I don't know your name.
>> BASTIAAN GOSLINGS: Bastiaan ‑‑
>> ANTONIO MOREIRAS: Sebastian ‑‑
>> BASTIAAN GOSLINGS: Bastiaan Goslings from AMS‑IX.
>>ANTONIO MOREIRAS: Very sorry.
Management has been for some time an evolving concern for the Internet community around the globe.
The rapid introduction of new Internet applications and services and the rapid growth of Internet‑connected devices have created new challenges for the regional management of the networks.
Despite the advent of initiatives like the CDNs, content distribution, and increasing of bandwidth capacity, with the rise of new technologies, Internet Exchange Points, IXPs, have become an important strategy not only for enhancing the quality of Internet operation but also for creating an innovation‑enabling environment.
Despite the solid evidence and the well‑known benefits of IXPs, the challenge of effectively involving the autonomous assistance community in the country poses a great challenge to their sustainability.
We have the practically in‑house datum, and there are approximately 450 IXPs distributed in 117 countries around the world. Among those, 300 disclose information about the amount of traffic exchange. Only 3 percent of them exchange more than 500 gigabits. There are still 86 countries without IXPs, most of which are developing countries.
According to some studies, the barriers to establishing IXPs in countries are usually not technical. There are a normal set of barriers that hedge from the lack of understanding of the benefits IXPs can deliver for Internet service providers and resistance from the ISPs which mark the dominance that varies according to specific characteristics present where IXPs have been settled. That is why IXPs are organized differently.
The idea for this workshop is to bring to aware different perspectives on how to build sustainable IXPs concerning different aspects related through the models IXPs are organized.
I will start saying few words about the Brazilian model.
In Brazil, IX.br ‑‑ that should be called PTTMetro ‑‑ is the Internet Exchange Point project of the Brazil Internet Steering Committee, cgi.br.
We have around 3,500 autonomous systems, and approximately 87 percent are ISPs here.
Six ISPs hold something as 70 percent of the Internet connections.
It is a highly concentrated sector but could be very worse without IX.br.
Today, the port at IX.br has no cost for the participants. The initiative is supported with the reserves coming from the domain names .br.
cgi.br acts as providing not only infrastructure but a set of related initiatives including capacity building, learning to promote IXP sustainability.
IX.br was started in 2004 and now holds 25 IXPs in different Brazilian cities, covering the five administrative regions of our country: North, Northeast, Southeast, Middle East, and South.
There are around 1200 participants considering the totality of IXPs.
Sao Paulo's IXP has more than 50 percent of the total participants, while some smaller locations have only three participants, the city of San Carlos, for instance.
One of the challenges we are dealing with currently is to stimulate and support interlocation transport service done by IXP participants, allowing the local traffic to stay in the location and at the same time allowing the participants to appear also at a bigger IXP.
In Brazil, many IXPs located in cities far away from Sao Paulo pay expensive price for the transport to reach the participants and the content presenting Sao Paulo IXP instead of supporting the development of a closer facility.
Other important and related challenge is how to create conditions for the main content providers, and CDNs should be peering at the smaller IXPs.
Finally, yet another challenge is how financing the sustainability and growing of the inactive.
The model of sustaining the IXPs with the research from .br domain names have worked well until now, but maybe we need another approach to support the growing of the initiative.
We would really like to hear ideas on the issue of the financial sustainability for the initiative which have attained the size of IX.br in Brazil.
Thank you very much. I will pass the microphone to Jane from Internet Society.
>> JANE COFFIN: Thank you very much.
Part of what the team has asked me to do is give you a broader picture for resources for Internet Exchange Point information and examples of business models where IXPs have taken them themselves from start‑up mode to the next level.
We call this levelling up or in just any Internet business or start‑up, it's the change from that start‑up phase to the more professional business phase.
So a document that was analyzed in the research paper that cgi.br put together ‑‑ and I'm going to try and speak slowly ‑‑ was the IXP took that the Internet Society and many partners around the world put together.
This is a collaborative document. We actually published it online as a collaborative piece. And it's open. It's alive. We hope to reboot more of it next year and add more to it.
We have participated actively across the Internet Society and with teams for the best practices that were put into this meeting on IXPs.
There are many people in the audience like Wim, Bevil, Mike Blanche, Bastiaan, and others who have put some of the data into not only the IXP tool kit but other studies that have been done around the world.
Yesterday, Bill Woodcock mentioned during a very good interconnection session some of the papers he's been involved in, an OECD paper which talks Internet Exchange Points and economic value, and also a good Canadian study that Bill Woodcock did with another person that looks at the economics.
Because the Canadian IXPs were sending their traffic over the U.S. border to connect with each other which was not economically stable.
So our IXP tool kit, which is a living thing, explains the role of Internet Exchange Points, the patterns of distribution, membership, how they're managed, how to get started, the economics, benchmarking, and case studies.
The key thing for this panel is the question of sustainability from an economic perspective, which is also from a capacity and a community‑based perspective. And it can be internal to the country and external with different countries that have successful IXP models going from this next level up.
I'll give you examples of countries where I've seen significant changes in the business model. Those are France, Ireland, and Serbia.
Different models of IXPs and how they were brought together. And I'm going to do this in about a minute and a half so we can turn to the experts on the panel to hear more about local and global experience.
In France, we know that France‑IX was established and doing very well. They were passing about five to 800 megabits of traffic at one point. And they were not constituted with a board or a business board. They weren't charging as much from what I understand to their customers for cross connects and ports and other related issues.
They brought in a new manager to help professionalize the IX.
I know Bastiaan can tell you more about what they've done at AMS‑IX to level up and become more of a business as well.
But what they did at France‑IX was professionalize themselves more to attract more members to this more professional business.
Many IXPs are start‑up businesses if you think of it that way from an Internet perspective.
You've got community help. You may have donations of equipment. You may be working with the government. They may help you with data center space. But you're trying to run that business with volunteers, maybe some paid staff.
But what France‑IX did was hire more staff and the members of the IXP saw more value in it and they realized they wanted to help with the management.
So they established a board, and that board helped do management. They're passing more traffic. They have more pops, have a stronger business.
There are slides that relate to this that we can put on the website if you want to see more because the CEO of France‑IX was speaking with us in Montenegro last year at a workshop, and he has good models.
We want to put that in the tool kit and provide it to Wim for the best practices put forward here.
And I want to say these are not best, best practices. They're good practices.
As Bill Woodcock said, "You pick and choose from models you see to make it work in your environment."
In Serbia, it's a more commercial model. They, too, had to level up. They had to go more professional. The asked for donations of equipment, and we helped them with that. They became more capable on capacity, and their members saw the value of the IXP.
In Ireland, it was the same thing. It is called the Internet Neutral Exchange. They did the same thing that France‑IX did. Almost. They realized they had to take the business up to a new level. They had to attract more content delivery networks, other businesses. So they went from more of that start‑up phase to a more professional phase where they have people helping advise at the IX. But they also use technology to mechanize more of their business.
They've also created something called IXP Manager, which is a free open source software kit that you can work with to manage your IXP.
So they can remotely address both technical and business issues from this platform.
So you can think of this as a natural IXP business or any Internet business model where you're going from the start‑up phase to the more innovative phase where you've got to level up your business, your marketing. And that's something that a lot of IXPs don't realize is that marketing phase.
It's talking more to people in the community who do business who may not think about connecting, like banks and other webhosting companies.
So I'm going to stop so you can hear from the others, but that's some baseline information for you about other IXPs around the world that have taken themselves from one layer of business up to another.
Thank you.
>> ANTONIO MOREIRAS: Thank you very much.
I would like to pass the microphone to Ariel from CABASE and LAC‑IX.
>> ARIEL GRAIZER: Good morning, everyone. Thank you for inviting me to participate in this panel.
I will share with you part of our experience in Argentina and our history and try to with this set the point where we are in the discussion and the discussion about sustainability of our model.
We start our first IX in Buenos Aires in 1996.
The discussion, 1998, was the first day that the Buenos Aires Internet Exchange Point start operating.
Since that day to 2004, we have been 37 members, all of us ISPs, Internet service providers, during all this period, including the two incumbent telecommunication companies.
Over this period and until now, we are basing the model ‑‑ we are a nonprofit association, and we are ‑‑ our model is based on the idea of recovering cost, operational cost. So every single month, we cover the expenses to run our infrastructure.
The technical team was based on three technicians or engineers that belong to one of different Internet service provider, and they donate their time to set up and administrate the IXP in Buenos Aires.
We have been successful in the model, but the traffic that we exchanged until 2009 was very, very small traffic.
And there was almost no more investment that was done at the time of the setup, because with the switch that we bought at the beginning, we covered all the investment that ‑‑ most the investment that we needed.
In 2009, we started a project to cover all the regions inside Argentina. Argentina is a large country, and we planned to develop several IXPs.
And due to the lack of very good bandwidth in the rest of the areas in our country, we start a research where we have fiber optics to cover the bandwidth needs.
So we developed a model that is based on the interconnection of the IXPs to one central node which is based in Buenos Aires like a star.
And all the participants of the different regional IXPs cover the cost to connect to Buenos Aires node.
And we tried to set the negotiation with the carriers to reduce the cost of this bandwidth, because at the beginning, it was very expensive. The national was very expensive.
We start with number of $60 and $69 per meg just in a land‑to‑land connection without Internet, just the clear channel.
Now, we are covering cost more than ‑‑ no more than $18 per meg in the same model because we grow a lot.
But the main discussion arrives to the model in Buenos Aires which we are now almost 70 members and the issue is that the size of the port where we are connected grows.
So at the beginning, everyone has a 10 meg port.
Then 100 meg port.
Then requires 1 gig port.
Now, everyone is requiring 10 gig ports.
So we need to cover the costs to grow in our switches and to find the way to support switches with more 10 gig ports.
The reality right now that we are facing right now in these days is that we are under requirement of thirty 10 gig ports, and our switches don't have it. So we need to buy or to find other switches. Sometimes ISOC donates, sometimes other companies or projects donate switches to us switches. But switches with this amount of 10 gig port are very high priced.
And several companies or project donate to us switches, but switches with this amount of ten gig price are very expensive.
But we are doing a big success because, as a nonprofit association, we have a transparent model to cover the cost; and everyone is accepting the model to pay the percentage of the cost that they need.
So up to now, we are fine with the model and trying to think how to evolve to the new stages.
Thank you.
>> ANTONIO MOREIRAS: Thank you.
Please, Malcolm.
>> MALCOLM HUTTY: Good morning, everyone. My name is Malcolm Hutty. I'm from LINX, the London Internet Exchange.
The London Internet Exchange is also not ‑‑ it is clearly not an exchange in the developing world; but in terms of sustainability, I believe we are entirely sustainable. We've gone through that levelling‑up process. Just a few basic facts about LINX to show that.
We were founded by seven Internet service providers in 1994 who wished to provide a local means for exchanging traffic so as to avoid what was then going on where traffic between Internet providers in London was passing via New York.
This is essentially a very common or standard reason why IXPs get founded everywhere. It was the same with us. But that was back in 1994 so we're now 21 years old.
We now have over 650 members. We have ‑‑ they have over 14 terabits per second of connective capacity and passing over 3 terabits per second in peak traffic.
Those 650‑plus members come from nearly 70 countries.
And in terms of scaling up through the ports, we have now got to the point where we give a port on each of our peering LANs included with our membership fee at the one gig per second rate.
And we have over 800 member facing 10 gig ports and over 60 member facing 100 gig ports.
Over the course of the next year to 18 months, we expect 100 gig ports to be the most ‑‑ to become the most popular ports that we are supplying to our members.
Our ‑‑ clearly to support that, we have to have professionalized the organization.
We have an organization with a little over 50 staff and a turnover of between $20 million and $25 million U.S. per year.
This is achieved through charging a small membership fee to each member and paying per‑port prices.
So if you have one small port, you pay one small amount for that. And then if you go up to the more expensive ones, you pay different prices for that. So everyone is paying according to the cost that they are ‑‑ we're doing.
Now, in terms of how you get to that level of sustainability, our view is that there are certain principles that we have held to that have allowed us to go through that and to build the support and trust from our membership base that allows them to have confidence in us as an organization that they want to invest in.
We are a membership‑based organization. We know there are other Internet Exchanges in the world not based in membership, and some are successful; but our view is that for an Internet Exchange to become large and well trusted, the most likely model to achieve that success is a membership‑based model and one that cleaves to certain core values.
Now, we have always had these core values at the heart of the way that we have run and developed LINX, and they've been our guiding principles as we have grown, and as we have moved through changes in the way that we organize ourselves internally, in our management and in the professionalism.
But we don't believe that it is necessary to become ‑‑ we believe it is necessary to become commercially minded, but we do not believe it is necessary to become for‑profit in order to achieve a sustainable Internet Exchange that is responsive to its member community. We are not a for‑profit organization. We are organized as a membership organization on a mutual basis.
So the core values that we have found that have guided us through this process, that have allowed us to maintain a membership‑based organization that is responsive to members while achieving that level of scaling up and professionalism are the principles of mutuality, neutrality, openness, transparency, and a commitment to excellence in all things, in particular to technical excellence in delivery.
So for mutuality, our organization is based on the principle that there is an equal say for all members, something that is easy to do when you're small and you just have a handful of members discussing amongst themselves what they should do.
As you move through scaling up, you need to put in place more formal processes to ensure that that principle is sustained.
We believe this is extremely important to maintain the confidence of the participation base that this is an organization that you want to build upon and to come to rely upon.
So we have a board that is elected from our membership. The board is elected on the principle of one member, one vote. No matter how big or small you are, no matter how much money you pay for ports to LINX, you get one vote in the election of the board.
And we also have a high degree of member participation for a company of our size such as approving the budget and strategic plans and such.
It is true that as we have grown and as we have professionalized, the level of formality in that has changed. But we still submit all core decisions and fundamental decisions to our membership for approval.
The second major principle is one of neutrality. That is, we treat all members equally. We don't play favorites. You can't get a special discount at LINX because you're really important.
They've tried. Believe me.
We've had ‑‑ they'll remain nameless, but we've had networks ‑‑ I'm trying to the not to look at anyone in the room. We've had networks come to us and said that it's more important to you that we are a participant on your exchange than the fees that we pay you in port fees so we want free service from you.
And we've said, "Yes, actually it is. You're a really good and valuable exchange with really important prefixes that would be a great value to our members; but you're not more important than the principle of neutrality."
The principle that we treat all our members equally, that we do not play favorites, is fundamentally core to achieving and maintaining the confidence that allowing major businesses that place a core trust in the Exchange as a critical service to them; and we will not that up for anything.
So even large and very important networks have been told, "If you're not willing to play by these rules, fine, you don't have to join."
Sometimes they accept it; sometimes they go away for a couple years and come back later.
But those principles that we cleave to with absolute conviction.
Similarly, openness.
This is the one principle that LINX did not always have.
At the beginning when we were first founded, there was a notion that the Internet Exchange is for proper ISPs. And people that were ‑‑ ought to be customers should be excluded. And at the time, that meant ‑‑ it wasn't entirely clear what that really meant, but there was certainly a sense amongst some that it meant you must have your own international transit capacity, particularly to the United States.
Over time, that became unsustainable. It was a way of excluding content providers from joining the Exchange. We have come to the realization that Internet Exchanges most of all provide the ability to connect eyeballs and content together.
So it was around 2000 that was a couple of real stretch on points of principle, and one was the principle of openness.
Now we have the principle that anyone can join the IX. They have to join on the same terms. They have to join on the same contract, and they have to be capable of peering traffic at the Exchange and must actually peer traffic with at least one other person, not with any particular number, but with at least one other person.
So in practice, this means there are some technical requirements like you have to have an autonomous network.
But, essentially, technical requirements, no commercial favoritism.
Openness. Transparency.
As I say, we want to ensure that our members ‑‑ we are here to serve our members, and we want to ensure that our members are able to be satisfied that we are continuing to serve them and that they are able to participate in our decision making, in our core strategic decision making. That means fundamentally a principle of transparency.
Certainly, there are things that are confidential like particular staff issues and so forth. But, fundamentally, at LINX our principle is if there's any reason why a member should want to know it, there needs to be a reason why they can't know it for us to withhold it.
The full position is always to be transparent, to explain what's going on, to explain even precisely what we're doing wrong when we have a problem so they can see it and provide input.
Always transparency first, unless there's a really good reason why it can't happen.
And, of course, the commitment to excellence, always seeking to improve, always seeking to achieve a better technical delivery and a better service to members.
Now, we believe if you follow those principles, you can win the trust and confidence of even the largest organizations even to critically to depend on you as part of their core requirements. And that has been our experience.
As I say, we have a membership organization so we operate, just are Ariel was saying, on a cost‑recovery basis.
That doesn't mean particularly month to month literally because we have to have the ability to financially plan. But any surplus that we make is reinvested in the growth of the Exchange. And any surplus that is beyond that that is not necessary is returned to the members not in the form of dividend, but in a form of price cuts.
We use any additional surplus to allow us to cut our prices further which builds a sustainable virtual cycle of making it easier for new members to join the Exchange by cutting prices which then stimulates demand and provides further revenues that can then be used to invest in and grow the Exchange through the challenges of new technologies and new levels of investment that are needed, whether that's scaling up in ports or in terms of the changes in the technical architecture that are needed to operate an exchange of this scale.
Thank you.
>> ANTONIO MOREIRAS: Thank you very much.
I'd like to apologize again with Bastiaan and ask to please get the mike.
>> BASTIAAN GOSLINGS: Yes, thank you.
You don't have to apologize. I'm happy to be here and honored even. So thank you very much for that.
My name is Bastiaan Goslings, here on behalf of AMS‑IX, the Amsterdam Internet Exchange.
I'd like to briefly describe what AMS‑IX stands for, what it does, and some of its characteristics and values.
Then I'd like to touch upon a broader context the IXP is part of, what we call the so‑called digital infrastructure. And then I want to make some comments about profit verses nonprofit IXPs. I was asked to do so. And finish off with the suggestion how we can possibly help IXPs in developing countries.
So with regard to AMS‑IX, the Amsterdam Internet Exchange, it facilitates interconnection between autonomous systems, the independently run networks that make up the Internet, the Internet we know as a network of networks is all the autonomous systems together.
AMS‑IX operates in the Amsterdam area in the Netherlands and distributed over 11 locations now, 11 data centers in the Amsterdam area, and being a highly complex and resilient, redundant and scalable infrastructure, the functionality of the platform is basically an Ethernet switch.
So the idea is with one physical interconnection and that might be redundant but as a concept one physical connection to this platform, a network can connect to it and potentially exchange traffic with all the other networks on the same platform.
And by now, there are 750 of those networks on the Amsterdam platform. You can think of ISPs, content providers, the large social media providers, CDNs, card hosting providers, et cetera.
I'm saying potentially exchanging traffic because between the networks, it happens on a voluntary basis, voluntary basis based on considerations as to whether it's beneficial for the parties to exchange traffic with each other. And this is mostly in terms of whether it's a commercial industry for them to do so.
I think the origins of AMS‑IX goes back maybe compared to LINX so it's more than twenty years now. We have seen significant growth over the years. And I think we can safely say AMS‑IX is a very important part of underlying technical Internet infrastructure.
Traffic peak ‑‑ exchange traffic peaks at more than 4 terabits per second. We are at the heart of one of the largest global Internet hubs, a densely interconnected ecosystem of data centers, carriers, and hundreds of networks and customers there gather to do business online with each other.
At the time, more than twenty years ago, AMS‑IX started as an endeavor to keep local traffic local and to avoid the tromboning effect of IP traffic going to U.S. and then coming back to Holland and then to be handed over to the destination network.
At the time, 20 networks got together and set up AMS‑IX association.
The management of the IXP was done by the academic network, and the day‑to‑day technical operations was in hands of university data center.
A couple years after that, in terms of professionalizing the organization, we had in the meantime set up the AMS‑IX company, a limited organization.
The technical operations were incorporated in the company, and from then we had our own staff and our own professional company was formed.
Interesting to learn that when AMS‑IX was set up, the incumbent, the Dutch incumbent, KPN, was supportive of the idea of setting up this exchange.
Let me put it this way, they were not obstructing it.
And I think in hindsight, that they had ‑‑ whether they actually knew themselves or not ‑‑ we tend to say they do, but whether it was the case or not, I don't know ‑‑ but they were right to do so because they managed to sell a lot of connectivity to the Exchange and peering at the Exchange brought them a lot of benefits.
So you can argue a smaller piece of the pie, I think the incumbent ended up losing some of the market share that they traditionally had. It doesn't matter if the pie grows. At the end of the day, the result was everyone was better off, including the incumbent. Might be interesting note to make.
Considering the role of AMS‑IX now, there's a very important international aspect to it.
You can probably imagine that the Netherlands is a small country with a small home market.
Being one of the largest Internet hubs in the world, 80 percent of the networks that use our services come from abroad, and they come specifically to Amsterdam to do online business with each other in Amsterdam.
I recognize because of the size of the Netherlands, we do not run into the challenge of needing to set up additional IXPs in remote and lesser interconnected areas as Antonio described the situation in Brazil to cater for local ASs.
So I look forward to hearing more about that. And if I can add to that discussion, I will be happy to.
So, again, AMS‑IX is a facilitator, a neutral interconnect platform. AMS‑IX has no insight whatsoever into the actual content of the traffic that is exchanged via its platform, nor involved or has any insight in the commercial considerations between the networks that decide to peer with each other or not to peer with each other in the platform.
Basically, anyone can join.
I think probably comparable to the situation at LINX, it's a number of technical criteria we have. And whether you're a ISP or a non‑profit governmental organization or academic or hoster, it doesn't matter. As long as you have an AS number, and that is basically it.
And then anyone can join voluntarily interconnect with each other through the AMS‑IX switching platform.
Independency, neutrality, and therefore ‑‑ at least from our perspective ‑‑ trust is essential for the customers of our platform.
These customers are often each other's competitors.
So creating this environment gives them the feeling that, okay, indeed there is benefit for us to do business together with us and there is no third party in between that actually interferes with this.
The sum of the AMS‑IX values and characteristics. As I mentioned, it's an association. It's a nonprofit organization.
But in the AMS‑IX case, it's not an industry body representing its members and customers. It's not an ISP association.
AMS‑IX is neutral with regard to its customers. Every customer is equal.
With regard to its suppliers as well as with regard to the content of traffic exchange with the platform, we are completely independent. It's for the members, by the members. A trusted facilitator.
And at the same time, there's a strong focus on resilience, redundancy, continuity of service with regard to the platform itself which also indirectly implies that backup connectivity for networks that connect to our platform maybe at other exchanges or private interconnects or maybe in other ways, that is the responsibility of those networks. So that's not something we are involved in. We can suggest to the networks that interconnect to our platform redundantly at different housing sites, we can suggest to them to connect to another exchange as well for backup and resilience purposes, but that is their own responsibility.
And the last thing I want to mention ‑‑ mentioned by LINX as well ‑‑ is that AMS‑IX is completely transparent with regards to its operations.
As referred to in the background document to this session, from our perspective, the challenge of involving ASs to use IXPs can be left to the market.
That does not mean to say that one cannot try to influence this.
Capacity building, education, as well as creating trust are not only important but essential.
But in the Dutch case, I think it's ‑‑ you can see different market offerings can take on this challenge.
Considering the role of an AMS‑IX, over the year, we have been focusing more at the international well‑established players. We have another important Internet Exchange in the Netherlands, NL‑ix ‑‑ you might have heard of ‑‑ that was set up to cater more towards the smaller Dutch networks, basically offering them a one‑stop shop. So not only offering the interconnection facility at the exchange, but also if applicable arrange an AS number for the networks, IP address space, provide them with PGP courses, act as a knowledge base.
I think that was a nice way in the Dutch context that the market approached this particular challenge.
From a regulatory angle, when it comes to IXPs, I think the main thing policy makers or governments can do in terms of intervening is to remove regulatory barriers for an IXPs creation.
In the Dutch case, this did not happen because there were no barriers. Until today, exchange points are still not regulated by any form of government policy.
Obviously, as a company, AMS‑IX, operating in the Netherlands, we are bound by law. We pay our taxes. But there has been no interference whatsoever with the running of the IXP itself.
Legislation in a broader context towards this sector has been quite benign over the years, which we think is an element of the success.
Many folks at the economic aspects and how to support the sector in order for it to grow ‑‑ think of the deregulation of the telecom sector, open markets, fair compensation, and intervention only when necessary. And intervention you can think of in the field of protecting the rights of end users. We have a net neutrality law in the Netherlands. These are the points it might be applicable for government to intervene.
It may be interesting to share with you that in this context, barely a month ago in Dutch Parliament, a motion was accepted to instruct government to recognize digital infrastructure as the third main port in Holland. So we have the harbor in Rotterdam, one of the largest in the world. We have the Schiphol International Airport, but the digital infrastructure is officially considered the third main port.
Its role is to neutrally facilitate the digital economy. So think of data centers, carriers, hosting parties, Internet service providers, Internet Exchange Points, the academic network, those type of underlying facilitating entities, the nuts and bolts of the Internet so you will, the physical part of it, for most people, actually invisible, all of it which is necessary to host, carry, or route the bits across the networks that make up the Internet.
I'm focusing here on this digital infrastructure. As we all know, it is a lot more than just the ISP itself that you need to reap the social and economic benefits of the well‑functioning Internet ecosystem.
From our perspective, this entire infrastructure ‑‑ this foundation on top of which the entire Internet economy runs, all the servers and forms of digital communications depend upon it.
So, briefly, with regard to profit verses nonprofit IXPs, as I said, AMS‑IX is a nonprofit entity. But most certainly, it is very profitable.
Port fees. We charge based on cost price, which are partly based on prognosis of numbers of ports sold.
We, furthermore, need to keep in mind the funding necessary to expand the platform and keep healthy reserves in case of what we would call what‑if scenarios. We include these in the port prices.
At the end of the day, when the reserves are considered too high, a price cut will be suggested; and this is by default approved by the member constituency.
And I think interesting to learn from maybe the Japanese experience ‑‑ and I hope Katsuyasu is listening remotely ‑‑ they actually chose to set up a commercial entity to run the exchange because Japanese ISPs and customers are very sensitive to service quality of the Internet, reliability, stability, accountability, et cetera; and they request a high quality of an IXP operation. They're not satisfied with voluntary or best‑effort level.
I think this is interesting because these elements are exactly the reason why we set up a nonprofit exchange. We have no other goal than to offer a stable, continuously‑running, secure, scalable and neutral switching platform for the best price.
And the fact that it is nonprofit and member based guaranties from our perspective the long‑term continuity of the Exchange while reducing the risk that it might be taken over by a third party.
Maybe to emphasize a bit with regard to the best effort comment, AMS‑IX was the first IXP to offer a carrier grade service level agreement as a service towards the networks that use its platform.
So, finally, how to help IXPs in lesser developed countries. I don't have the answer here I must confess.
As came up in other IXP‑related sessions in the IGF, there is no one‑size‑fits‑all recipe or model or best practice for the setting up and nurturing an IXP in order for it to become successful and sustainable.
Market circumstances will differ. The geographics, cultural differences exist and, therefore, also the regulatory environment. And I'm sure there are numerous other elements you would have to keep in mind. We all know it's not about the technicalities of setting up the IXP obviously.
I want to point to the fact that there is a lot of knowledge available at existing IXPs, and lessons can be learned from them.
And I want to particularly point to those interested to the IXP associations in the different regions ‑‑ Euro‑IX, LAC‑IX, AFIX, and APIX. These associations are focused at knowledge sharing, capacity building, and open accessible manner.
More importantly, this is for IXPs by IXPs themselves.
So I would definitely suggest to approach these organizations if you want to learn more about experiences and what has been learned in practice.
Besides these IXP AIs, obviously, there are organizations. I want to point to two of them, ISOC and PCH, who are doing tremendous work in terms of capacity building as well as technical consultancy and education in the area of IXPs.
So I leave it here. Thank you very much.
>> ANTONIO MOREIRAS: Thank you.
So, Sebastian, are you going to answer that smart look about expensive equipments? No?
>> SEBASTIAN BELLAGAMBA: Okay. Send me the bill.
Thank you.
My name Sebastian Bellagamba. I'm the Regional Director for Latin America and Caribbean Internet Society.
I am kind of caught in the fire here because ‑‑ I mean I'm from Argentina, and I have been there when Arianna was describing how we set up the IXP in Argentina, but Jane already spoke from the ISOC perspective so I will try to manage kind of a regional perspective, if I may.
It is amazing that since these days ‑‑ I mean 1998 when we were working on the first IXP in Latin America and Argentina ‑‑ now we have more than 60 IXPs in the region.
The downside of that is more than half of the 60 are located either in Argentina or Brazil.
So we have a lot of work to do in other countries. Not Argentina and Brazil. Argentina and Brazil are way ahead and good service by IXPs I would say. But many other countries in the region are not. So that's where we are focusing our work. And our work is not just our work. I mean it's our panelship work.
I would like to mention to say hi to Bevil ‑‑ hi, Bevil ‑‑ Bevil from PCH. He's the Caribbean guy with respect to IXPs. So if you need anything in the Caribbean, go to him.
So we work a lot on helping people to set up IXPs in the region, and we consider that crucial for the good development of the Internet.
Many of the things that I've already said we sign off already, I mean about how an IXP should be. I mean about mutuality, neutrality, et cetera. But I would like to go in some others.
You mentioned, Bastiaan, about the technicalities being not the most important part of setting up the IXP. I agree.
There is a phrase around setting up an IXP about 80 percent social engineering and 20 percent technicalities.
I don't know who started that phrase, but it is good.
And there is some discussions about who was the first one to say that, but let's take it as an anonymous phrase and quote it like that.
But I think it's completely true.
I mean the work in setting up the scenario in order to ‑‑ guys to sit down on the table in order to start thinking about setting up an IXP is the most intensive part of the work and extensive part of the work of setting up an IXP.
Once you reach agreements on several key topics, the technicalities are there.
We all know how to set up an IXP from a technical standpoint.
But what has to be done is it's ‑‑ in the process of the social engineering to make important decisions with regards to how you manage your IXP and how you manage ‑‑ imagine your future.
For what I've seen in the field, there is, I would say, four different types of IXPs, none of them better than the other; but they all kind of work somewhere and don't work in some other places.
We have these not‑for‑profit industry associations of IXPs that started some of the IXPs in the region, and in the case in Europe, too, operate on mutual commercial and for‑profit companies. I would say we have a couple of successful ones. Q‑SAUD is one in this category.
University and governmental agencies, those are cases, I remember now, for instance, the University of Costa Rica who manages the local IXP, successfully, too, I fail to think of a successful governmental IXP. But I might go and find out if there is a good example of that.
And there is some informal association of networks. Not so formal. They just start exchanging traffic.
There is also here in the region among the 60‑something IXPs, a couple, I won't name names. But there are, I would say, a funny IXP. It's the local incumbent and dominant companies adopt a private club of gentlemen among friends, which you are not ‑‑ I mean they have little power on who can get into the club, and they call that an IXP.
So we're working in some countries to have an alternative IXP to this kind of IXPs that we have to count among the 60.
I would say that among the decisions that have to be made when we start thinking of setting up a successful IXP, it's about the staff, how you staff your IXP.
At the beginning, as Ariel was saying, I think it's most of the time bottom tier work by the members; but that doesn't last well into the future. You have to get into the professional staff as soon as you can.
But that's a decision ‑‑ some other IXPs have started with professional dedicated staff from the beginning like here in Brazil for instance.
So it's a decision that you have to make that will work into the sustainability into the future.
Either you set up a not‑for‑profit or for‑profit organization. That's something that you have to decide at the beginning.
About the ownership model of the IXP, I think is very important. It's going to be a cooperative, or it's going to be owned by the third party? And that's something that has to be discussed and settled.
Most importantly, where is the IXP is going to be hosted. The neutrality of the place that you decide to host the IXP is crucial for the future of the IXP in our perspective.
Which cost‑recovery method are going to be used? I remember the story in CABASE, but we had to change in Argentina several times the cost‑recovery model because some produce inequalities and the other produces other inequalities. There is no perfect model. You have to choose between the different models.
Is it going to be a layer two and three IXP? That's a good question. Starting over with a layer two ‑‑ layer two is ‑‑ I'm not a technical myself. I'm a political scientist. But I learned that a layer two is ‑‑ you have one router and everyone connects to one router.
Layer three is when every person connected brings their own router and they connect in a switch or something like that.
There's positive and negative things, and they started in layer two, it's much more cheaper and faster, but the decisions you can make are not ‑‑ I mean your hands are a bit tied in order to how you set up your own policies.
In the other scenario, it's more complicated from a technical standpoint, but you have more power on your decisions.
And something that I didn't went into the details, but we learned into Argentina the hard way, if you go for what is called a mandatory multilevel peering agreement or bilateral agreement.
A mandatory multilateral peering agreement is something that everyone connected to the IXP should exchange traffic with everyone else in the IXP mandatorily.
That's the way we started in Argentina, and I would say ‑‑ I was the President of the CABASE, the organization in Argentina who runs the association before Ariel ‑‑ and it basically exploded. It didn't work.
We had a big crisis in Argentina for a period of six months while splitting in two, two networks in Argentina that didn't connect to each other.
I was thinking of a word that I wouldn't be seen written in the transcript. We screwed it up big time. And it's something that you have to consider because there's highs and lows in both sides. So this is the kind of things that we see that you have to decide.
A combination ‑‑ there is not a perfect combination in order to ensure the sustainability of IXP in the future, but it resides on the decisions you make in these questions that I posed.
So happy to help with more experience on how things went well or wrong in answering these questions and in many places in Latin America.
Thank you very much.
>> ANTONIO MOREIRAS: Thank you.
And, please, Henrique.
>> HENRIQUE FAULHABER: Good morning.
Cgi.br has proposed this workshop together with ISOC, and we're glad about the level of discussions here. In fact, the best practice forum that we have this year is IGF was also they produce a very good document. There are some paragraphs on the business models for IXPs on this document.
My hope here is to help to make some questions and comments about what we've heard here.
In fact, this piece of ‑‑ on Holland, London, and the experience of Argentina and Brazil, there are difference, of course, between developed nations, IXPs sustained in developed nations in situations like Brazil and Argentina.
I would like first just to begin the question and answer because you have some time to make it here round table, in fact, in order to receive feedback and response from the audience.
I would start making some questions to Bastiaan and to Malcolm about their model because as I understood the model, it's based on association fee and port fee, how they calculate this because when fees cover the costs, other things is to have money to invest in the next generation when we go from 10 gig spots to 100 gig ports.
So most of the IXPs here are ‑‑ other IXPs here making speech are nonprofit.
But they act as commercial IXP as soon as they have to reinvest the results of the operation to make the investment that are needed to grow.
So I would like to ask a first question to understand how those countries that has the biggest experience on sustainability calculate how much they need to invest in the following years.
The second question is just to put to everybody is about the role and the costs associated to the content people and CDNs. In those models in Argentina, Holland, and London, of course, and other countries that, of course, as Jane knows very well, how these participants contribute to the cost of the IXP and somehow if the money that you ‑‑ could be used to investments in the future.
Okay. So just put those two questions, and I hope you can have a fruitful discussion here.
>> MODERATOR: Before Malcolm and Bastiaan answer the questions posed, I'd like to ask Nathalia to read through a comment that Mr. Katsuyasu sent in remotely.
Could you please go through it.
>> NATHALIA PATRICIO: Hello. Mr. Toyama is in remote participation, and he has sent in this message.
"Hello.
This is Katsuyasu Toyama, working for JPNAP, the largest Internet Exchange this Japan. I would like to briefly explain Japanese history of the IXP.
The first IXP was established approximately twenty years ago. At that time, a strong motivation for making Internet Exchange was to improve the links between Japanese IXPs because some connection between Japanese IXPs were going for over the Pacific also.
Additionally, the first IXP and all other IXPs in Japan has been established by IXPs themselves. Most IXPs have the same issue; therefore, that they got together to solve these issues by establishing IXPs.
This means IXPs was not established by government initiatives but collaboration of IXPs in Japan.
Next the majority of Japanese IXPs are operated by commercial company, profit organization.
Why?
This is because Japanese IXPs and their customers are very sensitive to service equality of telecommunication from the perspectives such as reliability, establish, accountability, and responsibility to customers.
Therefore, they were not satisfied with voluntary or best‑effort operation and required very high quality services to IXPs."
>> MODERATOR: Thank you, Nathalia.
We can open the floor for other participants here in the workshop.
Okay?
Who would like to go first?
>> MALCOLM HUTTY: To some extent, my answer is not going to be very interesting because it's really you do proper business planning to answer these questions.
As I said, we're a $20 to $25 million business. Any business that is operating at that scale or anything that needs to be able to sustain itself past a stage of immediately making one decision after another on a very ad hoc basis, when you get to the level of sustainability, you need ordinary business practices to plan your budget, plan your expected costs for the future, decide what the need is going to be.
Now, I mean we set ourselves the target of being a world leading operator.
That's not intended to set me up in an argument with Bastiaan about which of us is better on a particular thing, but it's said and intended to be a commitment to excellence. And that implies certain things.
One of those things is that when our members ask us for capacity, when they ask us for a port, we should be able to provide it. We shouldn't be in a position where we're saying, "I'm sorry, but we can't deliver that." And that implies forward planning.
It implies having good communications with your membership base so you understand what their needs are going to be over the coming budgeting period, and so that you can plan your capacity, and, also, in the longer range, plan the development of your architecture so as to ensure that you're always in a position to meet the demands that the members have.
And the same is true in resilience. Our members have made it pretty clear that they care more about up time than they do about the port fees.
If there ever were to be a notion that, well, if we could pay a little bit more, we would ensure we'd eliminate some of that, we would ensure almost without limit they want us to be available all the time. And we strive for the very highest standards of availability.
And, again, in order to do that, you have to do planning for what the needs are going to be and how you're going to deliver that. That means setting a budget. But it also means and having a network design and a network architecture plan and a road map for the evolution and development of the network architecture.
And, essentially, we do all of these things; and we would expect any organization that was running a substantial network to do that. The larger you get, the more complex those things are and the more formalized the processes have to be in order to make sure that you haven't missed anything and you've made good decisions.
But, fundamentally, it's a process of running a high quality business.
>> BASTIAAN GOSLINGS: Thank you. Bastiaan here.
I don't think I have very much to add. Just to confirm that it is indeed quite a complex and detailed ongoing process of capacity planning and budgetary planning.
It is a business case. And you combine everything, the demand, traffic growth, port growth, targets. Everything is incorporated in there, and we have very detailed analysis tools to look at and form our developments and to project in different scenarios. And to an extent, it is an estimated guess; but that's a continuously ongoing process.
You know that you can adapt while it's ongoing the way that you approach developments. And that is how, at the end of the day, the cost price is determined, what I refer to including a premium for reinvesting in the platform and building up and keeping healthy reserves.
But to confirm what Malcolm said, it's very careful and very professional business planning. At the end of the day, that's what it is. In this case, it's an exchange; but, indeed, that's what an professional organization just has to do that.
>> ARIEL GRAIZER: I totally agree with Malcolm and Bastiaan, what they said.
But I want to add, in our experience, which is different because we are running 16 IXPs right now and four more under development; and in such ways, we are always starting up because we have new ones. Plus in our model, we have to deal with interconnection between them.
So in some ways, we ‑‑ and, also, I want to share with you, we have in almost $600,000 yearly budget for running this IXP, which for Argentina is a lot.
But the stages where are the different IXPs inside our model. For example, Buenos Aires from the rest are very different. Buenos Aires is very big for them. It has their own budget for a year. It's in a different stage and has a professional team and has ‑‑ we are under discussion, as Sebastian says, and ‑‑ I think it's important to clarify this 2005 when the incumbent said that we don't want the mandatory multilevel agreement no more and they leave us and said, "Okay. We just sell to you transit IP, and we'll not interchange more with yours."
And it was a big question. As Sebastian said, the mandatory model multilateral agreement model is good or not. And for us it was like a shield to give us the possibility to be still alive for a while until we decide to change our policy to be open to others, not only the IXPs, but to the other members.
And our traffic grows, and our models grows; and we ‑‑ when we reach this point, we do not need more the incumbents. And this was the key to success of our model.
And this was the reason why we are able to open more IXPs inside our country, because we don't need the incumbent.
And I guess besides the economic problem to be sustainable, the problem of who are the members that are able to interchange traffic inside an IXP is other key issues to discuss to be success in the model.
>> MODERATOR: Okay. Thank you, Ariel.
And I'd like to open the floor for questions for the audience.
>> AUDIENCE: Hello. Good morning. My name is ‑‑ from Equinix in Brazil. My question is actually to everybody.
Let's say we plan to build an IX where there is already an IX in place, what happens with market? Is there room for more than one IX in a particular region or city?
How do you think? Or what do you think will happen in the near future or in the future related to?
>> SEBASTIAN BELLAGAMBA: I think there might be in the future, but you have to start ‑‑ you have to start with one. In Latin America, in many countries as I said, I was just making the amount out of 60 something, 40 are in Argentina and Brazil.
So we have ‑‑ and we have 30‑something economies including the Caribbean in the region. So many countries are not served, and many countries are underserved.
So for starting, you have to start with one. I mean starting with two would be nonsense.
But in the mid‑term or long‑term, I see no problem in having more than one IXP. I think it makes a lot of sense because they will cater to different needs and different markets. As the traffic grows, then it will form ‑‑ Internet Exchange will grow, too.
>> ARIEL GRAIZER: In fact, if we have more than one ‑‑ in other big cities are more than one, but depends on the market and the model.
>> ANTONIO MOREIRAS: As I am the other IXP in the city, my answer is not to be very trustful (laughter).
But in the past, we saw situations where we had two or more IXPs. Then we had a set of autonomous systems in one of them and another set of autonomous systems in the other one. So the traffic was divided.
And if one IXP wanted to peering with all the others, he would have to connect to the two IXPs. So they would have to pay more for that. So it would end with high costs for the participants of both IXPs in the medium or long‑term.
But I don't know if I have to be trusted in this answer.
>> MALCOLM HUTTY: I would ask why you are you thinking of having a second IXP in the same location. If your reasons are good, then it will work. And if they're not, then it's unlikely to.
In London, there are more than one IXP. There have been other IXPs that have been started to run alongside or in competition with LINX, depending how you wish to describe it. And one of them has succeeded, and others of them have failed; and I would put that down to the reasons why they chose to start them.
The one that succeeded was founded because we ‑‑ certainly, at the time, we had quite high prices. And as we've grown, we have an able to cut our costs significantly, cut the prices to join significantly.
But at the time, we had quite high prices; and we were aiming to operate always ‑‑ we've always been aiming to operate on a highly professional basis.
Now, as you know, there are other models that are voluntarily based. Volunteer equipment, volunteer time, no sort of staff, professional staff, and so forth.
Now, that's a model that gives you a lower cost base and enables you to have essentially no prices. But it has other restrictions in terms of its ability to grow and also the service that it can give, and sometimes that can have an impact on uptime as well.
So, essentially, those two models were meeting different markets and we had the different ‑‑ some members that prized high reliability and the ability to scale capacity, and then there were other small networks that really thought the most important thing for them is that it doesn't cost anything. So those two exchanges grew.
Now, the exchange that has done that, now has some form of fees because it's grown and got to a stage where it needs to do that; but it's still being aimed at something else.
Other exchanges started in London were started on, oh, this looks like a good business. We can start this and make lots of money. We could be profitable here.
That's not based on a goal of serving the members, serving the participants. That's based on the selfish goal of being able to take something out.
They didn't succeed.
Beyond that, I'd say that ‑‑ and, clearly, there was overlap between markets of IXPs that are served.
I can't remember off the top of my head quite what the overlapping membership is between the membership of LINX and the membership of the Amsterdam Internet Exchange, but it's a substantial overlap. I mean probably over 20 percent, maybe 30 percent or more of networks that are members of both those exchanges.
So in one sense, you could say you have two exchanges there that are serving the same market and so forth. But there is differences.
Clearly, Amsterdam and London are not the same place; and there's good reasons why you might want both, and they remembered them. Some of them will be about location and accessibility, relative to your network, not in absolute terms.
One thing that members will get from being members of both exchanges is the further resilience of knowing there's two separate ones and they're technically and organizationally separate.
So that kind of argument could apply also to the question that you're asking if there's ‑‑ if that's the driving factor. But really the question is why are you doing it; and if you're offering those sorts of answers as an answer, is that just what you as the person considering setting it up thinks, or is that an opinion that is shared by those that you are expecting to participate? If the people that you're expecting to participate believe in those reasons and want it to happen, then you're likely to be successful.
If you just think, well, I want to make a business here and I can claim that, then unless you can bring the networks along that you wish to participate, you may find it difficult.
Thank you.
>> JANE COFFIN: Just to highlight a point that Sebastian raised about developing countries and emerging markets, a small market, we've seen this happen in countries like Ghana where there were two IXs that started and both were to failing. They joined forces because of it a much better, as everyone has said, business model, excellence for the customers, technical capacity. They used one as a redundant facility and one as the main switching facility. Obviously, the other was a switching facility, too; but it gave them a redundant pop.
It's important to think about, as Malcolm said, why you're doing what you're doing because you could just kill off a really good model in general in a small place. I think Costa Rica would be another place like that where you've got to build up that business and the content coming into the country and the attractiveness of the market.
So it's a serious business question; but, also, you could be doing more harm than good. Just a thought.
>> MODERATOR: Thank you.
Well, do we have another question from the audience?
>> AUDIENCE: It's not a question. Just, well, a comment that I wanted to make. It was already mentioned the best practice forum that we had this year at the IGF on IXPs.
You already heard ‑‑ I also find it a great document. I have been working on it as a consultant for the IGF Secretariat. The fact that it's so great is because we have been working with content on support and input of a lot of great people.
But the two points I wanted to make here is not into the content of the document, but just the idea of the best practice forum itself.
I think as a personal comment, that it's a great acknowledgment and recognition of the fact that the knowledge on how to run an IXP, how to set up an IXP, is available in the community. And I think that's a great recognition also by the wider community because, otherwise, there would never have been a team as IXPs have been accepted as a best practice forum.
First point.
Second point is the outcome document. It's still till the end of the week so you still have one day online to give additional input. So the final document will be published as an outcome document by the IGF.
And I think that might also be important. Although, as I have mentioned, I have been receiving a lot of comments, a lot of input from Internet Society, IX Federation, PCH; but the outcome document is not a document of one of the three organizations. No, it's something else.
And it might be interesting ‑‑ might be good to have the IGF logo there, to have also the United Nations logo there. It might help to reach out to other people, to people that you don't reach normally with the other documents. So I think it very useful in those two. And I hope that ‑‑ well, very had a great document that year ‑‑ that we can continue to work and also produce another document to follow up next year.
Thanks.
>> MODERATOR: Thank you.
We have room for one more question at the back, and then we go for the final comments.
Okay. So if you'd like to make the final comments, panelists. All right?
>> ANTONIO MOREIRAS: So I'd like to thank you, all of you, to be here and also all of you to participate. And that's it.
Thank you.
(Applause)
(Session concludes at 10:35.)