The following are the outputs of the real-time captioning taken during the Eleventh Annual Meeting of the Internet Governance Forum (IGF) in Jalisco, Mexico, from 5 to 9 December 2016. Although it is largely accurate, in some cases it may be incomplete or inaccurate due to inaudible passages or transcription errors. It is posted as an aid to understanding the proceedings at the event, but should not be treated as an authoritative record.
***
>> MODERATOR: Good morning, everyone. Thanks for coming along on this Thursday morning. We are here to talk about IXPs and CDNs, content delivery alternatives, and I will ask for the panelists to present themselves when they start to talk. And we will start with Jane Coffin. And Jane, you have the floor, please.
>> JANE COFFIN: Good morning, everyone. All right. Awesome. In Africa, if you ask that question and you don't answer back, you're in big trouble. So good morning!
Thank you for joining us today for this fabulous panel. For those on the remote connection, this is workshop 47: Content delivery alternatives... intertwining IXPs and CDNs. Thank you to everyone for putting this panel together. And the team in the region, of course, too. I'm Jane Coffin. I'm going to try to speak slowly for the people that are not in the room. We have remote people listening in, maybe.
So Michael here will help us with some of the logistics. And I think if we have questions from the remote participants, if people could help us with that, we'll go for that. So welcome to the panel. You have an amazing group here, as you can see on the screen. Gracias to everyone doing the technical work, too. The bottom line for me and the Internet Society is we're big champions for the internet exchange points. We help develop them. We donate equipment, but really, it's all about them. So we do what we can from the bottom up to help. By helping, that means we listen to what they have to say because they know better than we do. So one thing that I've known and from the work I've been doing around the world, and I often will come in to do best practices workshops to try to boot up the community and get people together. Because the key thing about any internet exchange point is the community. It's the people who interact and help develop and champion. You've been hearing a lot about community networks here. It's the same with IXPs or any group of people that you're bringing together. It's about them and how they interact and how they want to agree together. So at the internet exchange points around the world, there are different characters and characteristics, but there are the same similar issues right now from a technical perspective.
So before I jump into a content delivery one‑minute moment, I would like to ask who in the room runs an internet exchange point, runs IXPs or has developed an IXP. Raise your hand. Awesome. I see people here from Thailand, the Philippines, the Canadian internet registry authority. I see many that I know here. So if you haven't heard me say your name and you want to say something later, we would really like to hear from you during the question‑and‑answer session as well. So content, critical for an internet exchange point. Content delivery networks, anyone running big content pipes, many like to exchange traffic, put their caches in through the IXPs. It's a great place to meet up with many networks.
The challenge for many small internet exchange points is attracting the content delivery networks. You will hear more from the panelists about the economics, the issues with platforms, sharing platform space. You will hear some innovative approaches from the team here, specifically Antonio, about new ideas for content platform sharing.
With that I'm going to turn it over to the people you're here to hear from. But know that there's a great group of people around the world doing this type of work from the bottom up. They're everything from euro IX, APIX in Asia Pacific region, the regional internet registries are strong, strong advocates as well. Chris is in the room. Wave your hands? We all work together. No one does it alone. We're all partners. You can't exchange traffic with BGP without an ASN, right? So there are very important actors in the room and in the world. And I forgot to mention the Canadian IX association, that's the newest one I know of, CAIX. We have had people come to us and say that someone was advertising the entire routing table, what should we do? Perhaps we can have a workshop and you can help reboot BGP training. When you're an internet service provider, less on the content delivery side, perhaps, you have many strong technical people who may leave and when they leave and if they are your BGP expert, if they're gone, what do you do?
Someone might not know exactly how to do SBGP at the IXP. So we often will go in with teams, the network resource center or the regional internet registries and we will do free training. If you're in a region and you know you need help, talk to your RIR. They do BGP training, routing management, IP routing, they're excellent. So back to content, I'm going to turn over to Bastiaan Goslings from one of the largest IXPs in the world. These IXPs started out small. If you're small, more will come. But you have the challenge of how do I attract the CDN to me if you have so few peers and running maybe a gig of traffic. You go from 300 megs to 1,000 in two weeks. If you're not ready, you can't carry that traffic. There's a lot of information here. So Bastiaan, over to you. We're going to reset the clock up front. Each speaker has about ten minutes, and then we'll have Q&A.
>> BASTIAAN GOSLINGS: Thank you very much, Jane, for the introduction. Good morning, everyone. My name is Bastiaan Goslings, and I work for the Amsterdam internet exchange. It's a great honor to be here. I think it's a very interesting and timely topic, so I hope I will be able to add something to it. Today, I would especially like to share our experiences with getting CDNs on board with popular content. Specifically, I want to share the experience I had with the IXP we set up in the Caribbean. Hopefully there were lessons learned and that can give you some food for thought. I think we need to keep in mind that any place in this particular situation will have its own particular characteristics. That doesn't mean that whatever we did there can immediately be copied and pasted to another situation. But none the less, there were some things that we ran into and overcame, and I hope that's interesting for you guys.
You're probably most familiar with us as being one of the larger internet exchanges in the world. The largest global internet hub, so we're proud of that. Over the years we have been exporting that particular model we had in Amsterdam to other places, not only to increase value for existing members and customers, but adding value for other regions and locations and local networks. We always do that on demand. It's not something that we think everything we touch will turn into gold. It will happen when there's demand from the local partners.
So I would like to talk specifically about the Caribbean. I counted a number of challenges specifically with regard to CDNs, and that's something that I can share here during this particular workshop. At the time when we set up in 2008, we knew it was going to be a relatively small regional exchange. It was a smaller island, 150,000 inhabitants. The reason we set it up at the time is because of a government initiative. They felt an IXP would be an added value and they asked us to build the exchange and manage it remotely. From the start, we knew it was going to be a challenge to get the local IXPs to work with each other. That's not what they did. They got all the content from Miami. There was hardly any local content available. And whatever there was, you know, that would be exchanged and handled, maybe some e‑mails that would be handled in Miami, too. They did not want to pair with each other and they still do all of that in Miami. That's what they're familiar with.
So setting up the IXP and the ISPs not being supportive of it, along with the fact that they were unable to pair with each other, we had to find something that would make it attractive for people to come there. Quite obviously at the time, it was obvious that the engines of the ISP, they like content, especially CDN content. So we felt that was a strong reason to get the CDNs on board there. Hardly any local traffic. So that was another confirmation that we had to get the CDNs on board. It took a while, so we started off in 2008. But in 2010, we got Google cache at the IXP as well as another. Basically, I tried to make it as practical as possible, what we did. We found a way of sharing the costs involved and getting all the stakeholders on board and doing all of the challenge.
So the CDNs themselves, they sent the servers. They carried the physical transportation costs, which is actually quite significant. The IXP provides support free of charge. The data center provided space and power. The two largest ISPs, the largest consumers of the content, they were willing to take care of the cache flow. So one took care of the Google cache, and the other took care of the Akimai cache. Ultimately, we were pairing with all the ISPs on the platform.
The idea was to have this particular set‑up, and everyone agreed upon that, including the CDNs. CDNs are always willing to come as close to the end user as possible. Very happy customers of ISPs and willing to put their traffic in the networks. The way we did it was something that the CDNs could work with. We did agree that we would do it for a limited period of time. It was not set in stone. But until the amount of traffic exchanged would also make it a business case for CDN so that they would then eventually be willing to pay for a port on the IX and take care of themselves and the cache flow. We estimated at the time that it would take a couple of years. We're in the process now of having the first CDNs pay for connectivity themselves. So that gives you like an estimate since 2010 about the period of time it took. That could be different somewhere else. I have to say connecting to the exchange has been a great success. Immediately the exchange of traffic explodes.
Also over the years, we have seen because of this trickling down for end users on the islands in the region, internet speeds going significantly increasing and prices going down and also penetration, the amount of people who have usability significantly increased over the years. 1% in 2007 and now we're speaking of 90% of inhabitants having a broadband connection. So that's been very successful.
In general terms just to refer to a couple of lessons learned, at least for us, in order to keep an IXP sustainable, it's very important that all of the longer term costs of running the IXP are covered. And at the end of the day are covered by port fees. If there is any period necessary of subsidizing a certain set‑up in order to create some critical mass and demonstrate the business case, then it's important that the costs involved with the subsidy are also shared like what we did in the case of connecting the CDNs. I would always advise people to try to make an inventory as specific as possible of all the costs involved. Think of the equipments, reserving funds for re‑investing, expanding the platform, total costs, leasing fiber, personnel cost, etc. Try to make as specific a plan as possible in order to justify the port fees you will charge.
Assume that a port is comparable to the CDN case. At the end of the day, you are running a business, even if it's not a nonprofit one in nature as ours is, and because of that you do need a business case and a budget. And try to be as transparent as possible because that will create trust and make it more likely to convince stakeholders and ISPs to participate and share their goods in order to make it a success.
What I have been told, four points from the Caribbean. It's about creating trust. It might sound a bit cliché and vague, but it is very important. Simultaneously, number two, have a solid financial transparent model about what you're doing. Number three, recognize the entire eco‑system and give everyone sufficient attention by engaging all of them. It's about the IXP, the CDNs, about the members or participants or customers and whatever you want to call them and the data center involved. Try to focus on a common interest, a common ground. That will also help make it easier to create this particular trust.
That's basically it with regard to the CDN case. Looking into the future for the Caribbean, the next steps will be involving new CDNs, connecting them to the exchange. I can name names because everyone is aware of Netflix. And we are encouraging the ISPs to pair locally. They might like content but it's important for them to pair as well. It is indeed very important to have traffic there locally.
That is it for me for now. Thank you very much.
>> JANE COFFIN: Excellent. Thank you so much. I would like to just rest on one point that you made about building trust. That's one of the most critical things we've seen and you've heard from an IX themselves and there are plenty of others in the room. I see you in the back there and others in the room. To help start up and develop and support IXPs. Building that trust is what we can often call the human trust network. It's person to person or eye to eye. It's when we meet each other you're not quite sure. I met Arial and he's a great guy and I see you and I know you and we can work together more. Bringing the content closer to the consumer and getting the caches into the IXs. The cache, some of you pronounce it differently. This is the life blood of the CDN. And up next we have Alejandro. Over to you.
>> ALEJANDRO GUZMAN: I'm really happy to be on this panel because my whole professional life has been between IXP, ISPs and CDNs. I was part of the committee in Colombia. I was part of the first board of directors. And I want to talk a little about what was my job every day and how this was operated. And why is an IXP useful for and what it is not. Okay?
So if we can go to the next slide, please?
We want everyone in the world to be able to access information that is useful for them. Thinking about it in a diplomatic focus, the users are always behind an internet service provider. The users will come behind an ISP. So my work every day is to think about how to get connected to the ISPs and ensure that the content is working well and good quality. We understand that when ISPs are getting the costs lower, then they can do that. So the users can get all services for a better price. That is the work I do every day. I'm responsible for Latin America and the relationship with the big ISPs. So I will start first with some cases where the ISPs are not the solution.
So let's start with the first case. A market where there are only big ISPs, so they have big ports. We are talking about, for example, 100X ports. When the number of ISPs is low, they have big traffic. To connect them through an IXP is not the right solution. We are duplicating ports. We are forcing on the side of the ISP and then against ports on the side of internet service provider. In that case, we are also doing by use of permission, sometimes, they are in the same building. A better flavor to connect them. And maybe IXP looks at what we need to set up to get there. In this case, when we have this kind of market, then the IXP is not the right solution for that. When a market didn't have an ISP, we tried to help the community get together to try to build an IXP to try to help them. Sometimes it's not possible, and sometimes we have to solve the content in another way, trying to connect with everyone there in a separate way. And then an IXP is created. But everybody else is already connecting in other ways. They are connected to different CDNs by other means. At that point of time, to just connect to an IXP is not going to add value and at that point we would decide not to join an IXP.
Okay. When the IXP has policies that are not open and says the interest, we all know that there are some IXPs that work in that way. It was created just for some of the ISPs in the country. There are some countries, for example, where the government is going to try to buy services from an ISP and it is required to be connected to the local IXP. So ISPs around that IXP don't want anybody else to join because of competition. So unfortunately, there are some that block other players to connect. In Peru, many others tried to join, but they couldn't. In that case, as Google, we don't want to increase the bad behavior. So we prefer to help everybody else. Something that is run with good policies and open and goes straight to the market, not just to serve the interest of a few ones.
Other case, when the ISP is not cost effective. The port cost is so high that it's cheaper to go to transit. Or when you put together the port, and the transmission to get to the ISP is so expensive that it's better to connect in other ways. There are cases where this is not the right solution. But there are cases where an IXP is the right solution and they are good to have.
The first one, when we use the cost efficiently. In a well delivered market, where they are well connected to the ISP. In this case we are using the capacity in a better way and we are saving costs because everyone is connecting and getting content in that place. The same happens when the market is not that big. And it is easy just to serve the traffic there. Okay. Big markets, more a well-developed and too many ISPs. So we have like 6,000ISPs in Brazil. So the IXP makes a really good job putting all of them together and benefitting from a single point of connection. There are not many markets like this. Like India, Russia, and others, they have so many ISPs, so this is a direct solution.
When we want to develop the environment. They hope to have the definition point from a few ISPs, but we saw that potential and provided a patch. We knew it was not just the case for that. It was not good for a moment. We provide a cache and that helps in the case, Argentina at the moment. We have been helping them. And the content and traffic was super‑fast. And many ISPs were connecting. From like 80 members a few years ago to now 300 this morning... 400 as of this morning. Okay. It's adding value. We can jump to others.
So in conclusion, IXPs are a good solution for many cases and that's why, as Google, we try to help IXPs to develop, to have the native equipment, money, and everything to make this happen. This is not good in all the cases. We have to focus in the ones that it really adds value. If they are open, cost effective and good for the market.
>> JANE COFFIN: Thank you very much, Alejandro. It's been a long week. Thank you very much. One perspective on content for some of you in the room as well, when you have a non‑competitive market, and small ISPs are trained to create an IXP, it's hard to attract content to the networks. And what you're hearing from Alejandro and what you will hear from Martin, possibly, is there are certain things you can do to attract one to come. But if you're trying to create a local IX, it's the local content development. And you've probably heard this in other panels this week, which is so important. Which also means it's a cyclical issue or eco‑system of local hosting. Do you have the capacity to host? Are your ISPs ready? We have been working to host 10,000 sites back in Ruanda. There's no 24/7 support. Part of the issue is having local ISPs ready to bring the traffic host. Not send out a lot of spam, which happened in one country, to their peers.
And also to think about what it is that you can do to increase local content.
So we're going to turn it over to Martin Levy with Cloudflare. Martin, over to you.
>> MARTIN LEVY: Thank you, Jen. I'll do the quick introduction. And then talk about the subject at hand. I've been building internet backbones and moving bits on the internet now for a long time. I am not the most amount of gray here. But some of it is caused by moving bits, some of it. The concept of internet exchanges is not something to repeat, but I have been involved in connecting to them since the mid '90s, basically. Having moved from building large backbones to now content distribution, you can't reiterate more the important of internet exchanges in that job. So I'm going to give you a slightly different perspective than Google, but complimentary. We have absolutely similar issues all around the globe. We're just at a different scale. So when we look at internet exchanges, the common problems of do we have facilities that are appropriate are the internet exchanges that we want to connect to, are they easy to connect to? Have they actually been implemented in the right buildings? In buildings where an interconnect... and I use the word cost effective as opposed to cheap, always.
So these issues sort of are identical around the globe. I'm going to focus also on the need to fill the content. Content in the content delivery network is always... is 100% somebody else's content. We are an integral part of how the internetworks today, but we bring content from somewhere else. And then distribute it around the globe to make it more efficient, faster, to protect it, and other features like that. So we also have to solve the problem of filling the cache, making the data local, the first step before it's delivered to the eyeballs. We are, as was mentioned, building, and this took quite some time because the exchange was not a problem. That's an easy tick mark. Yes, there's an exchange there. We spent most of the time dealing with the problem of cost effectiveness. In other cases when we're in a large city such as London or Hong Kong or Los Angeles or New York City, etc., the content amount that we're delivering is so high that the cost effectiveness of the whole eco‑system that we build is easy to understand. We can buy a very large amount of band width to fill the cache. We can connect with large amounts of capacity to fill the exchanges. We find multitudes of carriers and ISPs that will take a private interconnect, a point to point fiber connection inside the same data center such that we can take the high band width networks and deliver the bits directly. That eco‑system gets financially strained. Interestingly enough, it's not a technology problem at this point in time. It is far more about the economics about every part of the building block.
I will take a time‑out to point something out to the audience that no one can see. The timer has gone into sleep mode. In theory, I have an infinite amount of time to talk. But I believe those in charge will fix this within seconds. They haven't yet. Anyway. I will continue. The clock has not stopped. There are lots of interesting issues brought to the table. We as a content delivery network partly because of our D‑DOS mitigation network, we have a very, very particular requirement on content flow, which means that we normally can't take a generic internet feed and use it efficiently. We have to do quite a bit of engineering on that. So those points are normally discussed on a point‑to‑point basis. But the effort described, and I won't repeat because I know it has existed in other places around the world, that effort is a very worthy way of spreading the word of multiple eyeball networks to bring content.
I'm going to reiterate a point that was made, which is yes there are many times where we sit and talk with the local providers and find that they are not peering amongst themselves, which is foolish. If they all want to peer with us and our competitors, okay, fine. At least that solves our problem. And if nothing else, a little bit of peering can move on to a lot of peering. There are specific cases and points of which to give an example, as a content delivery network, we have built two facilities which are surprisingly close to each other. The Philippines and manila because we have two massive telcos that will not interconnect with each other. They have actually just solved that problem, and I will talk about that in a minute. There are internet exchanges over the last year or so that has grown in manila in the Philippines. But the mere fact that these two telcos won't get along and, by the way, are causing the non‑beneficiaries of this are the end‑users. We as a content delivery network have had to duplicate our capital expense, duplicate our network in manila just because of the connection issues.
I'll sum up and give some time back to the moderator and the panel and simply say that we are finding that an enormous amount of bits on the internet are being delivered by content delivery networks or very large sources of data, normally on behalf of somebody else. And this trend is not going to change. That is fundamental to our business model. But the reality is that we also want to work in every single part and place that we can. But what we want to do is bring the content close tore the eyeballs and in most cases, internet exchanges are the modus operandus.
>> JANE COFFIN: That was Martin Levy from Cloudflare. Before that, Martin spoke from Google. And then Bastiaan Goslings spoke first. Next up is Henrique from CGI, yeah? He will tell you a little bit about who he is and what he does. He has ten minutes like everyone else.
>> MODERATOR: We require a change in the order.
>> JANE COFFIN: Okay. You're in charge. We're going to Antonio now. Antonio Moreiras will be up.
>> ANTONIO MOREIRAS: I'm an engineer and I'm in charge of the development and the presentation? No presentation? Let's do it without the presentation. Okay. We have 26 IXPs in Brazil. IXP.BR has 26IXPs. But one of them is responsible for 80% of the traffic and about 80% of the interconnected ISPs. So we have a lot of concentration. I was paying close attention to the presentation of Alejandro to see if the other IXPs have the correct approach to be useful to the CDNs. They are true for the users and policies of ISPs that want to connect. I think that a lot of content... don't... didn't solve yet the connected problems in their regions where the IXPs are in. But in fact, the CDN has been the main content providers, not just in Sao Paulo.
So what's the problem in Brazil? The ISPs are very small and have very few participants. I think they don't make very good business case for CDNs and other content providers to connect. Oops. It's not this presentation. Sorry. It's the other one.
>> JANE COFFIN: One moment.
>> ANTONIO MOREIRAS: Well, CDNs, we see that they have basically two models today. One model to bring home, it's the model they use in places where we have a lot of ISPs, and big ISPs. They are present in the big data center. Their other model is when the CDNs put the caches aside the ISPs networks. So it generally works for medium and big ISPs because there is minimal amount of traffic to... should the CDN have a good business case to do that. When, for example, ISP asks Netflix to put a cache server inside their network, Netflix, you ask for 800 megabits per second of traffic to be able to do that. So for a small ISP in the country, probably neither of the solutions are very good. They are not able to connect to the bigger IXP that is far away from where they are. And they are not big enough to be a good business case to the CDN to put the cache inside their network.
So this map shows where the ISPs are connected to IXP from Brazil is a very big country. We have ISPs coming from very, very far away to connect to IXP here because the content is present there and not present at the local IXPs.
In the next area, it's very simple. We are not reinventing the wheel. We see a trend of small ISPs, when they are not big enough to ask for a CDN to put a cache inside their network, they come together and approach the CDN... oh! If you put a cache inside the network, this will be shared between these three or four other ISPs. So they make this agreement to share the caches locally. And it works sometimes. But sometimes it does not work very well because their ISP that are hosting the cache grows a bit, and then the cache is... they have the traffic. They have the cache only for them, and they break the agreement to the others. And bell, we see that the agreements between the ISPs are not very stable. So we are trying to have this kind of arrangement and institutionalize it. So we are creating, we are proposing to create an alternate system that we will help with hosting the space near the local ISP. And we are highering the capacity of the IXP of Sao Paulo. And the users, the participants of the local IXP are going to share the costs of connecting and of the hosting. With the CDNs. We are trying to convince the CDNs of participating, of the sharing costs. So the idea is very simple. It's to have a tiered way of model, where the caches could be shared between a lot of ISPs. And the costs also could be shared between ISPs and IXPs. For now it's a project that we are trying to make it relative in the IX.BR in Salvador. Probably we start to break it in next month. But we have an example of local IXP where an association of ISPs meet the national association of inclusive... inclusion. They made a similar product, and they asked for caches for bigger systems. And they shared this cache inside the local IXP. So they did it and you can see on the graph how the traffic has grown. It was about two gigabits and now it's 15.52.
So that's it. We think that the model for the internet has developed, so the first place we have the tier 1, the core with the ISPs connected. After that, we had the donuts internet where the ISPs start working with each other with the help of IXPs. And now it's very important that we ask the CDNs to go to the model of the donut internet. So the CDNs must be closer to the ISPs and to the end users, because the main contents and the CDNs are responsible for approximately 50 to 80% of the traffic that ISP have to bring to the users. So that's it. Thank you very much.
>> JANE COFFIN: Thank you very much, Antonio. That's a very interesting model and something that I would note that some of you, in many countries there is often a cache war. Many ISPs are trying to grow but if a cache is only in an incumbent's network, it's hard to attract. It's just something to think about. If you're starting up an IXP, it can be complicated, but don't lose faith. Things change over time.
The next up is Henrique, who will tell you a bit more about what he wants to talk about. And he's with CGI.
>> HENRIQUE FAULHABER: Good morning, everyone. I am one of the board members of CGI.br. And we are very pleased to be part of the workshop about different perspectives that people bring to the table here. In fact, as just told, an IXP to take 26 cities, and we enter in the situation that ten years ago, we bring content from Miami and outside the country. It's so big today that it's about 80% of the content and everybody goes to Sao Paulo to take content from the major players. Sao Paulo is now a hub like Miami was ten years ago. We're in the middle of discussion of new projects to empower the IXPs in small cities. In fact, Brazil as a country now, a very big country that has different situations. Sao Paulo is too big and Rio is also big with big IXPs, but you have some states not with them inside of the country.
It's interesting to have heard here, I have a few comments. I believe it's more important to have time for questions and answers from the audience. The material is too rich, and we have a future that we have to decide and make planning in how to do well with this issue of the solution of content to CDNs. It's very interesting to us to hear from Google, from Alejandro, which requirements should be considered to fit the IXP through the cache from Google. I believe those comments are very important to us in Brazil and for all the people involved in the IXP that are considering to make cache and make CDNs available on our own infrastructure. Google accounts for 27% of the traffic, Facebook about 15%, and Netflix about 11%, and Microsoft about 6%. The three biggest host about 50% of the content that goes through to the end users in Brazil. This project, the open city project, is supposed to be starting to run software next month. In fact, we'll allow the other citizens in Sao Paulo to have this infrastructure in the beginning for Google, Netflix and Microsoft available to the region.
The fact of the cost is for the ISPs is too... is very important. When it was said that the content for the ISPs participate on the IXPs are about 60 to 80%, it means that above the three or four big players, other players also participate on the IXP, and the transit cost for the ISPs is, in fact, about 60% less than when they are not in the... in this environment. We proposed the new project at the end, considering the big players are offering a service and infrastructure to put their caches. We can do the research in a way that we can match the criteria that this those players ask for. In Brazil, we have over 5,000ISPs and it's not possible to have cache for everyone. We as the international community and our operational arm, we organized these requirements from the ISPs and bring them the opportunity to use the resources that are offered through the content providers in a way that it would be not possible if we don't have a good aggregation for them.
Those are my first comments. I believe Bastiaan's point is very good. He is a close friend, and some of that work helped very much the community. Last year, one workshop about IXPs focusing on assistance ability. In Brazil, we are going to a new model. We are living in Sao Paulo and here is the model for free ports for the content providers and ISPs, and we're going to a model that we are participate... for the participants. Just to cover the operational costs, not the investment. As you know, the president... funded by the resources.
So, the space that Bastiaan and others brings to us, to the community, about the model of assistance ability of cost in the Caribbean on their project, I believe it's very good information for us in order to make IXPs to make them cost efficient and sustainable.
So that's my first... my few comments here. I will be glad to hear from you and ask... the people here at the table make other comments and address the issues that you can bring to us. Thank you.
>> JANE COFFIN: Thank you very much, Henrique. And just a small round of applause for our panelists. We're going to go to question and answer and open mic. Thank you very much.
[ Applause ]
And just to sum up for the remote moderation and those who entered the room, we heard from Bastiaan, Martin Levy, Antonio, Henrique from CGI.br. So does anyone have questions? Excellent. I'm going to start in the back of the room. Hi, how are you?
>> AUDIENCE MEMBER: Hi. Thank you. Thank you very much for the panelists for great presentation this is a very acute issue for us right now. Kyrgyzstan receives 80 to 90% of our content from the outside, which worked until one decided to increase prices three times. Now the cost is $30 for a gigabit per second. A year ago, it was $100. We have realized a difficult situation we have now. Now the country is looking to reinventing the situation. And the issue of IXPs and CDNs has become very important. And Kyrgyzstan hopes to help the neighboring regions and neighboring countries.
>> JANE COFFIN: Does anyone on the panel have a suggestion and a help? Note that Kyrgyzstan, it's almost double landlocked. China is so big, land mass wise, we're looking at situations talking about being connected and bringing economic models that would work for an IX in Kirgizstan, does anyone have any advice for the team there? It's a different situation with different land masses and border issues. Anyone want to... Martin?
>> MARTIN LEVY: Thank you for bringing up a new country. If you have been wondering why I have been at my keyboard, it's because I have a natural desire to say how much traffic actually goes to Kyrgyzstan? And interesting enough in the last two or three days, I've seen all the traffic disappear from Moscow through public and now being fed out of Frankfurt on a completely different port.
So let's say in theory and practice, that they match 100%. Something has changed in traffic there and somebody changing the pricing by a factor of three upwards, the industry does not go up. It's meant to go down. It has an effect. So interesting double problem there. And I'm just going to add that yes, it shows up in the statistics and routing.
>> JANE COFFIN: I believe I saw a hand up over here? Alejandro from Mexico?
>> AUDIENCE MEMBER: It is thought to be a microphone problem but it's more often a mixer problem, so we always have to wait a few seconds. I'm from Mexico. I congratulate the organizers for this panel. I think this workshop should be mandatory for every attendant to IGF2016. It should be tested before people can lock into next year's. For this reason, this is one of the very few workshops that actually touches on the technical aspects of how regions get connected. The impact of the data you are showing is from everything that is in policy discussions. Particularly on net neutrality discussions. The model of a big central server sending tons of data to networks is absolute mostly thanks to your efforts. Bringing data closer to the user. On the other hand, how money changes hands between what someone pays for Netflix or pays for their ISP and finally gets into a company's coffers. That architecture changes dramatically. I would like to ask how do you analyze the impact of the net neutrality discussion and the discussion of regulating the so-called OTTs, over the top providers. And is there any impact that you could tell us significantly if there's a difference between the idea of content of something that is sent to the users to read or watch more passively and be more active production of content that's actually happening, become services on content. Thank you.
>> MARTIN LEVY: If I can quickly correct you, the content will sit untouched unless a user requests the data. So 180 degrees different from what you just said.
>> JANE COFFIN: On the OTP side, do you want to respond, Alejandro?
>> ALEJANDRO GUTZMAN: I am passing.
>> JANE COFFIN: It is a very good point. OTP are tricky. We were just at a major world conference where this issue is a big topic of debate. And stay tuned for the world telecom development conference in Buenos Ares in October. And I believe there will be a discussion about how to develop the infrastructures in different countries. We're talking basic infrastructure as well. For those of you that may not know that there is fiber involved. It's not just through the air. We're talking cables, accessing from fiber backhaul from the landing station to satellite activity and others are all part of the eco‑system.
Another hand was up in the back there. I can't quite see you. Stand up. We've got a white shirt on. You, excellent. If you can introduce yourself.
>> AUDIENCE MEMBER: Carlos from Ecuador. We are building the network, and we are interconnecting these ISPs all around the country. Because more than 300 small ISP has no way to compete with the big internet providers. The big internet providers have plans with different parities for international and local traffic. They sell only to the final user. They do not sell to the small ISP who works in areas where there are not so much people. So between 200 and 300 small ISP organized and made this network of IXP. I'm going to talk about this in the afternoon in the ISP panel of experience.
But the main problem we face now is how to get content servers. How to approach to put in contact with people like Facebook and Google like Netflix. So we can have more content servers to make the IXP effectively work. Some suggestions in order to ask it to work better with the content providers?
Thank you.
>> ALEJANDRO GUTZMAN: I know we have been working with different models for a long time. I think sometimes we face an issue that is not even about wanting to do things. It's about technical difficulties. I remember a case of an IXP that was presented to us. The total traffic introduced was around like 70 or 80. The smaller cache is more in terms of capacity. If we pull out one, there is one like the 4G capacity, the minimum traffic that makes sense for the area is 400megs because below that the cache will consume more traffic than it delivers. It really needs to make sense for the ISPs. You are not saving money, you are wasting money. That's the point. You need to put more people together to get to the point where technically we can help.
>> JANE COFFIN: Thank you very much, Alejandro, for that comment. 20 years ago, people were sending over to the United States to talk to us. This is a picture that I'm going to make on cross border connectivity. Break down the walls. If you're not sending traffic over the borders at a lower cost, by the way, it's complicated. And your costs are so much higher. The long haul, the traffic goes back to talk to each other in a country? Time and distance equals money. So you're trying to minimize that long all traffic. We're seeing much more build‑out in the world. Islamabad just launched out. Thank you for that point, Carlos. It's an important one. Other questions in the room? Hands? Up here and then Allen after that. Please introduce yourself for the audience and remote participation.
>> Hi. I'm from American university. I have a question. What prevents IXPs to connect to each other in a context as Brazil? We have 26IXPs and Alejandro talked about how it is difficult to install CDNs in different ISPs so we have 6,000 ISPs in Brazil, for example.
>> JANE COFFIN: I'm going to turn to Bastiaan and Antonio. The basic idea is not to compete with your customers as well, the ISPs. So you're generally not sending the traffic back and forth between IXPs, it's the localized traffic exchange. But, let's go over to Bastiaan and then Antonio.
>> BASTIAAN GOSLINGS: Yeah, I cannot comment on Brazil's situation. I'm sure Antonio will. Speaking for the company I represent here, as Jane said, you don't compete with a subset of your customers. That has always been our approach. We leave the interconnectivity between a subset of our customers, which are basically carriers.
I realize that in the Amsterdam metro area and the Netherlands, the area I come from, it's a competitive market and many ISPs to choose from. I know that's not the case everywhere. Our model is not interconnected exchanges, that's not, per se, the way to do it. There are other successful internet exchanges. In Holland, we have a really big one, number five or six. And they have a completely distributed network over Europe. So wherever you plug into, commercially speaking it's very successful. So it's not up to me to comment on whether that's a good or a bad model and whether they are competing with customers. I think they don't feel like waiting and they say that their customers don't feel that way. And whether the additional agency and other things that this generates, whether that's an issue or not is probably up to their customers. It seems to be a successful model. I'm curious to hear what Antonio has to comment.
>> ANTONIO MOREIRAS: The ISPs try to solve the problem of local content and stay local. So we need to attract CDNs and make this content look to feel the cache and maybe this content, look how it's different that is starting with our customers, with telecom providers. I could say that ISPs are interconnected, but not by this. By the telecom providers. They provide the service for a cost. So they are interconnected. If you are a participant on IXP want to buy capacity to contribute, they can do that from a telecom providers. Specifically in Brazil, we would have problems with the telecom regulator. We are not regulated as IXP. We are a part of the internet infrastructure, not of the telecom infrastructure. At least in Brazil, the legal aspect, the regulatory perspectives, the things are separated. So we don't want to mix the two things.
>> JANE COFFIN: Thank you very much. IXPs are not the old telco NOCs, the network operation centers. Not like in the old telecom model. There was another hand up, I believe Allen from the Canadian internet registry authority. You're up.
>> Thank you. I'm Allen from the CCT, CCTLD manager from Canada, and we have been involved in helping to set up from the past few years. I wanted to go back to the first presentation actually from Bastiaan because it mirrored our experience in one or two situations where we had a very small city, 500,000 people. We had some ISPs that were willing. We have a very uncompetitive market structure. And there was a willingness to do it, and it was really the CDN that provided that necessary extra little bit of value added that brought them together. So I just want to reinforce what you're saying. Everyone had to take a haircut, as we say in north America. Everyone had to contribute a little bit. We chose to pivot transit costs for a couple of years. That was the way we created the value for it to start. So I guess I really want to reinforce that message, in particular for anyone who's looking at setting up an IX, start small. So CDNs are absolutely critical to getting them going.
So I have a comment, and I have an unrelated question. We have what I think is a very unusual situation in Canada, and I just wanted to see if the experience of anyone else on the panel or in the room, but Netflix will not put a cache in IXPs in Canada.
>> JANE COFFIN: All right. Look at that. We have a question. Netflix won't put a cache in an IXP in Canada.
>> They put a box in the large incumbents. They get close tore the eyeballs but not the cache. I'm wondering if this is something unique to us or if anyone else has this problem.
>> JANE COFFIN: Bastiaan?
>> BASTIAAN GOSLINGS: I'm not going to speak here on behalf of Netflix, but I have a counter question. What is their argument for not doing it?
>> AUDIENCE MEMBER: They don't answer the question. Our strong suspicion is, it's for tax reasons. I'm a Netflix purchaser in Canada. I do not pay sales tax on that when I purchase, but when I buy the competing service, which I did until it went out of business, they had to pay tax. And the speculation is that they have no presence in Canada, no staff, no buildings, etc., so the speculation is that they do it for tax reasons so they avoid having to pay both local sales taxes as well as coffer tax. Again, that's speculation. I've asked and they have refused to answer.
>> JANE COFFIN: And now someone has a question for Netflix, I'm sure. If they're here, this might be a good time. They're not? Okay. We have about a minute left. We would like to wrap up. First, thank you for being here. Really important to have your participation. Thank you for participating and asking questions, but also just listening. We've got a great team of people up here you can talk to. But also in the room, I want to note again, you have Carlos from Ecuador, Bevel Wooding, Christian O'Flarity who has done a lot of work. I see other guys in the back. Who am I missing? Malcolm.
So on the panel here, of course I want to sum up again, Bastiaan, Alejandro, Martin, Henrique, and Antonio. There's a lot of experience here. Ask them questions. I'm sure they're pretty willing to answer. But you've now just had a very cool panel on CDNs. So we've got content issues, trust issues, Netflix not wanting to peer with the IX, very interesting. And you know, this is a key issue for IXPs as they develop, and also in your countries. This is part of the eco‑systemic impact of policy, regulation, opens, open borders, equipment, and so on. Tax now.
So thank you very much. Give a hand to the panel and to yourselves. We appreciate it.
[ Applause ]
And the last thing I'll say because you don't want to listen to me anymore. You probably want a coffee like I do. I have to make a pitch for there's an IXP best practices session this afternoon. Vin from the team working with IGF staff has done a brilliant job helping cat herding, which means trying to take people who are very busy and bringing them together to work on best practices. Last year best practices in IXP was more about how do you get started. Now this is more what we call leveling up to the next level. Business sustainability, bringing more content and peers to the IX. That will be this afternoon. I always have to carry around my piece of paper. So 1630 to 1800, IXP best practices session, WS room 7. He's keeping me in check here. Room 7. Thank you very much. We'll see some of you there, I hope!
[ Applause ]