IGF 2020 – Day 12 – WS350 Attributing attacks: political, technical & legal dimensions

The following are the outputs of the real-time captioning taken during the virtual Fifteenth Annual Meeting of the Internet Governance Forum (IGF), from 2 to 17 November 2020. Although it is largely accurate, in some cases it may be incomplete or inaccurate due to inaudible passages or transcription errors. It is posted as an aid to understanding the proceedings at the event, but should not be treated as an authoritative record. 




>> MARIETJE SCHAAKE: Hello!  Officially, welcome, everybody!  We heard that the stream may have already been on for a while.  You may have seen some informal preplanning.  Now we are very happy to kick off this IGF session on attributing attacks ‑‑ Attributing attacks: political, technical & legal dimensions.

It is exactly the dimensions that have to come together to get to credible places of attribution and that's up for discussion this session.

In the next hour and a half we have a great set of speakers highlighting different attributions of cyberattack, by, for whom, by whom, and when.

After we hear from these experts, we will make sure to allow for as many questions from participants as possible so feel possible to raise questions with us as well and I'll try to bring cross as many as possible.

First, why do attribution really matters.  Clearly, unfortunately, cyberattacks, in frequency, scale, intensity, they're escalating, we see criminal activity, state actors attacking vaccine labs just to name one example and hospitals and engaging in the ongoing activities of intrusion for corporate espionage, geopolitical strife, crimes with economic objectives and even when many of us imagine that during a pandemic there should be a shared focus to end the suffering to allow for healthcare workers to do their job and also there are those that are cynically exploiting the fact that there are more people online, creating a larger attack surface and working in less protected environments than if they were to log on in their offices with cybersecurity experts close by.

I think it is really important when we think about cyberattacks and the technical terms that come into the discussion when we try to understand what is happening and your privacy the technologies are evolving and the methods are evolving to always keep in mind that the digital is human and that we're not talking about an abstract problem.  When a hospital is attacked, for example, patients suffer or can't even enter the hospital because they can't be registered, because the machines are down, how can we even appreciate that they were to recover?  An IT system can be rebuilt, reset, can be bought new if necessary and the human cost is often final and irreparable and I think that's important to keep in mind, especially because technology is no longer a sector and has become a layer of all sectors and more and more aspects of people's lives, it really is a vast problem that has links to so many aspects of the way people live, work and enter into their daily activities.

On top of that, the growing uptick of artificial intelligence, the Internet of Things, other new technologies, again, it allows for new ways to attack.  As I mentioned, it is a growing problem.  The cyberattacks and I think it is urgent to see that we have a better grasp of what's happening so more fact‑based discussion and that we can walk the path towards attribution and accountability and hopefully leading to more responsible behavior in the digital world.  I think oftentimes attackers exploit the lack of legal protections, legal clarity in the digital world to very deliberately avoid accountability.  There may be a reference to one attack or another in the news, but hardly ever to we hear about the following accountability, it's hard to see a public debate about attacks because people, citizens, many politicians too remain in the dark about what's actually happening and without such a well‑informed public debate, it is impossible to get the kind of political priority, kind of political pressure on leaders to act and therefore too often accountability and justice are delayed or never served and I think this gives attacker, criminal, geopolitical actors a double effect, one, the effect of their immediate activities, but secondarily or as a broader effect, the erosion of trust in the ability of a rules‑based order to deliver upon the promise of protecting people' quality of life and values.

In order to get to a better place, it is important to bring together the various aspects of understanding attribution, the technical, the political, the legal, the policy aspects and these sort of societal impacts.  That's what we'll do today, bringing these different aspects together towards a credible, enforceable mechanism within the international order online.

We have agreed speakers will highlight whatever they think is important about themselves, that he research and their bios but briefly we're very happy to welcome the following speakers today, firstly, Johanna Weaver, a special advisor to the Ambassador for cyber affairs in the Australian Department of Foreign affairs and trade, Camille Francois, Chief Innovations Officer at Graphica and Serge Droz, incident response and security teams first, and then Jens Monrad, director of Mandiad Threat Intelligence at Fire Eye.

Sorry to be long to get us all started.  This gives a bit of a scene setter of what we aim to do today.

Now opening up the panel discussion I wanted to start by asking, anyone can raise their hands to answer, what does attribution really mean?  It is taken from the context of traditional military concepts like deterrence, how must we understand attribution and the importance of it in the cyber context.

Johanna Weaver, maybe you can start.

>> JOHANNA WEAVER: Sure.  Thank you.

It is a great question.

Being the nerd that I am, I Googled it and had a look at what the Oxford English Dictionary says for the word attribution, and that sounds a little bit blasé, but actually it is a common ‑‑ there are so many different understandings on what attributions actually means and interestingly, in some languages, the word attribution doesn't even exist, we can come back to that.  The Oxford English language definition of attribution, it is that it is the act of regarding something as being caused by a person or a thing and interestingly, the examples that it give, it is not the defense‑based examples that we probably are going to, but it is looking at the works of particular artists or authors, for example, so when we look at it in the context of cyber attribution, there are four things that stand out to me and as introductory framing points, the first, it is attribution is not new, we have attributed things to people or organizations for a long time.  It is new perhaps in the cyber context.  It is not new full stop.  Second of all, attribution, it is not all public.

It may be that governments, companies, individuals, attribute behavior, they say that I know that X is responsible for Y but we may or may not say that publicly.

I think that's often lost in the debate.

The third point, it is that attribution, it is very rarely an end.  It is actually a means to an end and I'm sure that we'll discuss throughout this panel what that ‑‑ what those ends might be.

The fourth point that I would make, it is that we often talk about attribution in the context of we need to attribute malicious activity by countries, but we also can attribute to individuals, for example, cyber criminals and attribute to groups, for example, terrorist organizations who might be using the internet to incite violence or recruit, we can also use attribution to draw attention to responsibility for companies as well, so the international reports, looking at the export practices of companies to authoritarian regimes is a good example there.

I suspect our conversation will be more focused on states but, you know, the fact that attribution can also extend to other actors, it is an important point.

Thank you.

>> MARIETJE SCHAAKE: Thank you so much.

Before going on, I'm happy to add to the introductions of our panelist who just managed to join us, John Scott‑Railton, I didn't want to introduce you not knowing whether you would make it.  We're super happy to have you, also because you and the team at the citizen lab at the University of Toronto have been so important in focusing on the private sector on hacking as a service, the surveillance industry.  It is great to have you so that we can also talk about attribution of private entities and the roll that they play in the broader ecosystem of being instrumentalized, providing instruments for attack, whatnot.

Welcome.  Catch your breath.  Good to have you.

I'll now be happy to turn to Camille Francois for reflections and we'll just see whoever wants to join after that of the panelist, just raise your hand and I hope to see you.

>> CAMILLE FRANCOIS: I think John's background is introducing the idea of the flag there and how it plays in the attribution definition, let's get the definitions out of the way and welcome to the complexities of attribution and how gains are made by actors.

Let me ‑‑ allow me to be the resident boring nerd here too and refer to a fantastic scholarly Article by Ben McCann none and Benjamin on cyberattack, it is a fundamental piece on attributing cyberattack.  It presents the model for attribution which has nothing do with this, it predates all of that, but it is essentially making a point that various ‑‑ that it is worth repeating, attribution is not a set of technical hoops to go through.  It is an art, it is not just a set of technical consideration.  It is also a series of strategic, political consideration.

That is sort of the main thing that I would put forth in the definition and related to your question on what is attribution, I think, you know, it boils down to the eternal very simple question of when you look at an incident, attribution is there to answer who is behind it, whose done it.  I think it is interesting because we often think about the importance of attribution for deterrents and we'll talk about that, but there is much more to that, there is transparency, the understanding of everybody, the sophistication or the lack of sophistication of the actors behind the specific incidents, it is fundamental for security we expect organizations to protect themselves against threat and it is important to understand who is behind the threats and, of course, there is enablement, once you have shared more of this information and we see that, citizen lab, for instance, when you share this forensics, when you share more of this information, you enable others to continue building on the work that you have done.  Just a series of attributions, not just a technical concept, attribution, it is not just a deterrents problem, and I think that's a good set of starter points for us to continue to build on.


Serge indicated he wants to jump in next.  Feel free to raise your digital hand as panelists and I'll make sure it is somewhat coordinated.

>> SERGE DROZ: I'm Serge Droz, I'm the Chair of First as already said.  I represent the incident response team, so we often talk to or call ourselves the firefighters of the internet, we're the first ones on the scene of an incident happening and we have used the term attribution a lot but differently from the way it is phrased and framed in here.  I feel it is very, very important that we're extremely clear and I'm grateful to Johanna Weaver that she brought forward a good definition.

For the technical community, often attribution just means figuring out more what was it that happened and has it happened before in a similar way.  That allows us to tie together different types of groups without actually knowing who really is behind it.  We'll say, this is the one group, a fancy group here, an APT5, whatever, they say that without knowing who the persona is behind them.  In social media, you have huge discussions who it really is, but I feel it is very important that as incident responders, for us, for the technical community, attribution means something different there and some would argue we should not call it attribution but maybe fact‑finding because what we do, we find facts and it is this computer connected to that computer, this and that happened.

By the way, why I feel it is important, incident response teams, security people, they do not step into this, because we have heard that attribution, it usually has a purpose, and that's a purpose that's usually telling someone, hey, you did something wrong and this is very important that we do this.  It is also something that kind of kills trust.  If you start to point the finger, people will stay away from you.  As the firefighters, you're not going to be the person that people are afraid of.  You want to be the people that earns trust regardless of where the fire burns and it takes both and it is very, very important to protect both of the roles and that's why I also feel it is very, very important that certain parts in the attribution process are really clearly assigned to different roles.

I find it very, very important that we're absolutely clear on what attribution means for whom.

>> MARIETJE SCHAAKE: It is also important to put it in the context of on the one hand, having more evidence‑based information about cyberattacks so that there can be a better informed policymaking, understanding, whatnot, attribution and then the question of accountability, not even just who should be held responsible but then what should be proper sanction and clearly that's a political question under any circumstances.  The credibility, the trustworthiness of the process, it is essential in making sure that the accountability process has meaning.

I agree.  I think it is very helpful to separate those out.

>> JENS MONRAD: I want to echo what was being said, I can say that from the company I come from, we're on execution, so certainly attribution, it is not just who it is from that perspective, because from a technical, an operational standpoint, it is really also how and what they're using, so we use attribution in many different phases where we're able to not necessarily attribute to a specific response but to a specific cluster of activity, what tools are they using, how are they using it.  Something that's very, very critical for the network that doesn't necessarily care too much about who it is, although I would say when we talk to senior leadership both in governments and in the private sector, the first question we always get asked, it is who are doing it and whereas we talk about the operational era, it is about how we're doing it and what are the user rights.  We use attribution in many aspects of our public, private reporting to help the different layers.

>> MARIETJE SCHAAKE: Yeah, you want to jump in, John, perhaps add about on their standards, Best Practices, mechanisms for attribution that's relied on by technical experts like yourself?

>> JOHANNA WEAVER: Thank you.

I think Serge's point makes sense figuratively to a lot of people, it is important to have a cannon of technical standards and approaches.  It is certainly the case that right now the cybersecurity industry and the intelligence communities in many countries that have fed into that industry have developed a series of practices and language around attribution.  I think the elephants in the room for the entire conversation is the radical disconnect between those languages and the precision and meaning that they contain and the political realities of public conversations and policy conversations around attribution and I can't help but think there is an analogy to climate science, the science is chugging along, getting better and better at certain aspects but at the same time, there is no guaranteed relationship between even quite good attribution work and any kind of consequence or even sort of public consensus of who may be responsible.  I think that that is as much the kind of problem that we grapple with and I would love to hear from other panelists about the political realities of this challenge.

>> MARIETJE SCHAAKE: That's helpful.

You want to jump in.


Just so much good stuff there.  I don't know which part to pick up on.

I think on the point that was made by do we call it attribution when it is the technical communities, what certs, et cetera, the fact‑finding as opposed to attribution.  That is quite interesting because when we ‑‑ when we look at attribution, the process of making a decision as to whether or not the Australian government will make a public attribution we have two attribution assessments and then a decision that's made.  The first attribution, assessment, it is a factual assessment, that factual assessment includes the technical indicators of compromises and the technical information and it includes other factual information so for example what's going on in the world environment, Geopolitics, are we in negotiations with particular countries, et cetera, et cetera.

It is that factual assessment encompassing all of the elements.  Then we do a legal assessment.  The legal assessment, it is looking at the things like the law of responsibility, if you're looking at the perspective of attributed to a state and if it is different assessments if you're looking at criminal actors or non‑state actors.  Those two attribution assessments, they're absolutely key.

There is a decision that's made, usually at the highest levels of government, as to whether or not we will then make a public attribution statement.  Often people will early on in the discussion, we would refer to that as political attribution, and we have now stopped referring to it as political attribution because it implies that it is politically motivated where it is not.  What we're saying there, it is that you have the two attribution assessments which are essentially factual assessments based on an impartial assessment of the information and then there is a decision about whether or not we publicly or privately make an attribution disclosure.

I think that's ‑‑ I think it is quite interesting in terms of the crossover of points that were being made there.

I think the political point that John was making, it is also really very relevant.  I think we have been getting quicker at attributions and we're in the attributing everybody and maybe it is not real list tock expect governments to attribute everything.  The speed, it is increasing, the precision of the public attributions are increase, we have gone from 2017 when the government says we don't comment to public attribution, and now we're attributing to specific agencies in specific government, we're making progress, the question, of course, how to make that faster and to be precise as we go forward.


I think that Camille wanted to build on that.  I wanted to ask a follow‑up question that hopefully can be addressed and others may too, it is the incentives for different stakeholders to participate.  For example, companies with concerns for their reputation, Civil Society, with vulnerabilities for their constituents, et cetera, et cetera.

Camille Francois, please share your thoughts and then maybe we can open it up to why certain stakeholders would actually even want to be part of the attribution process or participate if they think that's the outcome.

>> CAMILLE FRANCOIS: A great question. 

Building on what was being said, they're now ‑‑ there are loads of people, types of institutions that participate in the attribution conversations.  We see research institutions attributing, that's new and that's remarkable.  They tend to follow specific sets of guidelines, they tend to have a higher bar for showing their forensics and they toned qualify, that's a good Best Practice, the attribution with the confidence level that they have, so they may say we attribute this campaign to that actor with a moderate degree of confidence.  We look at the private sector attributing, both independent researcher, large corporations, of course, we now see governments participate in the attribution conversation as pointed out by Johanna Weaver in a more transparent way and sometimes with remarkable speed.  On the U.S. side, we just wrapped a really interesting election cycle where we saw the U.S. government publicly attribute a campaign focused on the election where the Iranian entity was posing as a domestic group to create intimidation and it took something like about 24 hours for the government to come out publicly with attribution and here to demonstrate and to show that attribution and public attribution, also sometimes in a heated cycle, they can have the benefit of restoring calm in the face of an attack.  Particularly when you talk about perception attacks and IO, having a government come out, say, well, actually there is nothing to worry about there, because with analyzing, we have understood where it is coming from and also sometimes can participate in restoring public confidence in the face of attacks that are designed to undermine public confidence.  To the point, which attribution matters for accountability, it is an important question.  The private industry does a lot of these attributions and we see them increasingly do that.

Microsoft has been doing a lot of attributions throughout this election cycle.  We have seen Facebook, Twitter to some extent, Google, also participating in publicly attributing campaigns.  When they do that, I think it does create a sense of accountability from the private sector, more often than not they're saying we see you, we won't let this activity happen on our services but it is different from having a government attribution that can sometimes go hand in hand with a process that's a disincentive for continuing this activity.  For the marriott of people that participate in this contribution, I was going to say circus, whatever is the more up lifting attribution ballet, not all carry the same weight for accountability and it is interesting to see government with different policies on attribution.  The French for instance, they have a longstanding policy that they would not publicly attribute cyberattacks.  This would not be the key to their digital strategy.  Of course, it doesn't mean that the French don't care about cyberspace but it is not what they were anchoring their cyberspace deterrent strategy on.  It is interesting as we see those evolutions to observe governments who are engaging in this debate for radically different perspectives.

>> SERGE DROZ: A thing that's pointed out, in the current attribution game, we have a biased view I would obtain, we have governments that are looking at what others are doing, the big cybercrime groups are doing, then we have commercial cybersecurity tech community mostly looking at what are criminals doing against customers, and we completely are leaving out Civil Society and it turns out that they're under more pressure than we think.  This is really bad for two reasons.  Obviously it is bad because Civil Society doesn't have the means to do proper evolution, and there is a lot of information, we don't have the means and these people, they just ‑‑ they get lost in that.  It is also bad because the states, the threat models are based on the available attributions and if you miss half of this stuff, then you get rescued or biased models and pictures.  There was a great talk in the beginning of the year by Lennart Maschmeyer from Citizen Lab comparing the commercially available intel, he knew from Civil Society and there was not overlap.  We tried to bring together at list in this private industry, with Civil Society, trying to work on this problem, finding out how we can solve this.  Interestingly enough, private industry really was very open to that, supportive, and Civil Society, they were afraid of doing so. 

Again, this is something that we should keep in mind, why is Civil Society so afraid of actually participating in these attributions?  I don't think I have a ready answer, maybe someone else on the panel does.  We found this a challenge to invite these people to start talking to the established tech community about these types of things.

You point to the very important question of trust.  Right.  What incentives there are to trust and what risks there are to share information.  I think that's probably explaining a lot of the lack of better coordination and cooperation.

I see John is nodding his head, I promised to come to Jens first, maybe John will jump in later on what he sees as a phenomena in Civil Society and the paper that I would strongly recommend that was produced by your colleague Leonard was mentioned.

Please, go ahead.

From your point of view, what you're seeing here.

>> JENS MONRAD: I think also because there was a question on its actual value of attribution beside pointing fingers, shaming.  I think what we see from the professional community, from the cyberspace.  It is that it carries an important weight because when we do, for example, when we ‑‑ when our company does public attribution, and we have released this to the greater public and we have done interview, blocks, all of that, we also see that we're making it more expensive for the cyber attackers.  We're not necessarily pointing fingers at the specific nation but the way that they're working towards a compromise, we're trying to make that more expensive, we're forcing them to go back and to redesign their attack model, redesign the way that they're using the tools and certainly especially when talking about state‑sponsored campaigns it is important to recognize that it is a relatively time‑consuming operation so it is much less agile than it is when we talk about cybercriminal attacks.  Whenever we're contributing to making it more expensive for private enterprise, governments, NGOs, the Civil Society being attacked, I think we're contributing to that part, but obviously we are still a private company and we don't do government work.

  We can only layout the way we see it.  I agree with Serge, it is a degree of bias we recollects layout our evidence and then obviously people can question it, it will be from our point of view, what we see, what information we have available.

>> MARIETJE SCHAAKE: Thank you.  John.

>> JOHN SCOTT-RAILTON: I think you're right to Serge's point.

Civil Society cares about attribution and what you're picking up on, the last decade or so there is efforts by private industry players to play with targeted civil organizations, organizations that face threat, but unfortunately, many of the corporate security players that have done that, they go in, look, we'll help with security, name and shame attack viruses and they put an intrusion detection box on a network and writing a fancy report that does not highlight the issues of Civil Society, thank you for the threat intelligence on the Chinese attackers going after you.. 

As said, it is about trust, it is also the case, something that we see day in and day out, for most of the attacks that we work at the citizen lab, it is our targeted attacks against Civil Society and most are coming from nation state players and we don't get to any other player doing serious attribution.  It is very rare.  Very rare that there is this degree of clarity coming from the governments and coming from the private sector on the cases that we work with and as a result some of the cases that have the most impact.  A case like stock net, for example, we had the community of folks that are talking about the importance of building attribution mechanisms because you have the presences of the stakeholders.  A lot of cases we work on, huge human impact, Civil Society groups ripped apart by surveillance, hack and leak and taint operations, these things, they never get to convincing attribution.  If they're mentioned by industry groups, they're mentioned only in passing without naming individuals or individual organizations or really laying out the harm, which is that there is a largely under motivated set of cases.

>> MARIETJE SCHAAKE: That's very helpful.

>> JOHANNA WEAVER: Thank you.  I want to go back just briefly to a comment that was made by Camille Francois on attribution, quick attribution restoring confidence.  I think that's absolutely the case, but we also have front of mind whenever we're making an attribution as a development that quick attribution, that subsequently proves to be wrong, it will destroy confidence and destroy all future confidence in attribution.  There is really a role there for the private sector in terms of being able to move quickly and there is something to be said for the gravity of one government saying you are responsible for this malicious activity and that is not the same as when a private sector organization says that and I understand that private sector organizations have commercial imperatives and they don't seem to be wrong, but also there is ways that that can be balanced and that cooperation between the private sector and government cans actually be quite useful in closing the gap in terms of how long it is taking to do attributions.

Then to come to the points that was made by Serge on who attributes and John as well in terms of which attributions matter, I think these are really key points.  From a government perspective, when we're attributing something, we will be attributing those issues that are most central to national security or that go to the core of what we believe in.  That means that many of those things that John was talking about, that have huge human impact may not necessarily be prioritized and not because they're not atrocious, but because they don't fit within that attribution paradigm.  There's a really important role there for other actors to be stepping up in that.  Also, for governments where the harm is that atrocious to also be recognizing that harm as a harm that's got the potential to impinge upon national security, for example.

I think in terms of the value of attribution, that goes to the last point I made there, there are three real reasons, three primary reasons of why we would join this attribution, and the first, it is because we want to be indicating that behavior is unacceptable and that it violates agreed rules or norms and believe it or not, despite common misperceptions, a lot apply in cyberspace.  I'm the head of Australia's delegation to some Working Groups, the two groups at the UN discussing those rules, and so the first, it is to highlight that the behavior, it is unacceptable.  The second, to draw attention to the fact that the behavior is ongoing.  For example, Australia has made a number of attributions where we have named a government to bring attention to industry, to say that you need to fix this because this is this actor.  For example, we attributed a campaign of router scanning to the Russian government and we had previously advised and put out alerts saying that industry needed to be aware of it, no one paid any attention, so the attribution helps draw attention to it.

The third reason we join in an attribution, this is where it really links to some of the points that John was making, it is to support international partners or to support values that we hold dear.  Here I'm thinking about international rules‑based order, Human Rights, democracy, free and open economy, these are motivations and reasons and in particular that we may join with partners to attribute a particular malicious activity that maybe didn't have an impact in Australia.

I'll stop there.

>> MARIETJE SCHAAKE: John, then a question to the panel and the Q&A are piling on.

I want to make sure we're getting people's questions answered.

>> JOHN SCOTT-RAILTON: Briefly, Johanna Weaver, your way of going through this is helpful for us as a set of categories to think about this, because I care a lot about Civil Society, I want to highlight I think one of the areas where there is the most room for innovation by governments, it is finding that sweet spot where governments sit on tremendous amount, especially fly‑byes, tremendous amount of information that leads in the direction of attribution and could do meaningful social good by making that available, including in cases that may be less costly in terms of sources and methods because the actors are sloppy but where the actors are creating real harm.  I want to highlight that as a tremendous area for potential.

>> MARIETJE SCHAAKE: I want to ask you all if you mention one thing that you would like to change, to have better situation for yourself, towards attribution, what would it be?

>> JOHANNA WEAVER: John, to respond to your point, on the sweet spot, I couldn't agree with you more. 

If I can sort of shuttle it, Citizen Lab does great work on, this there is an onus on Civil Society to be really in the face of government and making it clear that this is something that you should pay attention to.  That helps me get the attention of others.

The more we can be vocal as a community about the fact that this behavior is unacceptable, that type of behavior has that human impact, the more that we can push that message through government as well.

I'll hand over the challenge, I'm not sure if that analogy carries across the entire IGF audience.

In terms of the answer to the question, if I could add one thing, I would add better coordination on responses.  Call it responses, accountability, consequences, whatever your word is and attribution is just a means to an end.  It provides the responsiveness to the malicious activity and some actors for whom attribution in and of itself may be a deterrent but very few.  We feed to make sure that we're matching an attribution with responses be they law enforcement, diplomatic, legal, economic, at the high end, even military responses but they must be proportionate and comply with more ‑‑ if all of our responses.

To move beyond seeing attribution as an end, attribution as a means to an end would be my dream scape.

>> MARIETJE SCHAAKE: I agree completely.  Towards accountability.

>> CAMILLE FRANCOIS: Just a provocative thought.  Sometimes we talk about the role of Civil Society, the need to empower Civil Society as, yes, this is good for our value, this is good for Human Rights.

There is also a very strong case to be made that often Civil Society is looking at places that are gigantic blind spots for people who work in national security.  I think about those most rooted in right now, IO, it is really a good illustration of that.  IO Information Operation, I think of it sometimes as the little center of cyber and this is here a really good illustration of how much Civil Society has been leading the game, attribution, it is difficult, yes, in IO, you have had amazing investigative reporters who went and found threats and attributed them way before governments started to consider it.  If you think about the internet research agency, the major trolls are from behind some of the IO against U.S. election in 2016, it was first exposed by Russian investigative journalist going undercover working with, this detailing the threat, writing about it, sharing all of the forensic, they're also times of threats and areas of concerns in which Civil Society is simply leading and everybody deserves that more of that acknowledgment.  I look forward to seeing IO reshuffle the cards a bit on how we think about attribution, yes, of course, that leads to many people thinking that everybody is a Russian troll on the internet, that's too bad, we have been through this cycle with cyber too, slobby attribution is a fact of what happens when more people join the field.  Creating better standards, better practices for sharing forensics between Civil Society, industry, government, and for publishing about it, it is something that can change security for the better.  That's my two cents.


>> SERGE DROZ: A friend of mine working with the FBI recently told me about, you know, security is a team sport, I think we really have to go a long way of becoming a much better team.  Right now we do a lot of this individual and that's not just states, we have had a couple of states that work together, and incident responders should work together on a global scale.  Again, in the industry, in private industry, we are still too much in secrecy, and this is our ‑‑ we're not going to share any of this and the data sharing, we never really live it.  I think we have a lot better, much better in creating an environment in where we have agreed on goals of what we want and responsible behavior, it is by states, but also by private industry and other players, actually working together towards that means.

For me, this is something that we have a lot of room for improvement.


>> JENS MONRAD: I think I agree, especially with Camille Francois on the IO part.  That's definitely something that we are seeing a lot of.  One thing that I would say, that I hoped for, it is with attribution regardless of whose producing it, we could change the rules of engagement with how we're operating and living in cyberspace.  That certainly is my naive hope.  A personal concern that I have seen, it is that I have seen far more nations that are trying to mature their offensive capabilities but don't necessarily have that operational security, they may use local software, may use stolen tools, tools that are able to obtain by leak, by other offensive operations, and that's really my big concern without any rules of engagement, not necessarily talking about survey lapse, stricter internet usage and we need countries to come together and to form some sort of agreement and you have to factor in that cyber is still a relatively new domain when talking about even warfare, right.  So that's my biggest concern.  I see a lot of aspiring nations today trying to become the next cyber super power and that is really effecting citizens and obviously private organizations and governments all over.

>> MARIETJE SCHAAKE: Thank you. 

Besides changing the dynamics of war, it changes the dynamics of peace.  One of the great challenges facing us is the blurring lines between so many previously more clear notions, like civilian and military, war and peace, private and commercial.

These blurred line, I think they create a whole different landscape, ecosystem that we all have to navigate.

I want to go to question, please participants put them in the Q&A function.  I saw one foundational question that was put in the chat.

How should states distinguish between attribution and criminal investigations and indictments?  Is there a risk that political attributions circumvents the individual Rights of suspects?  The presumption of innocence is my interpretation of what was being asked there in the chat.  It is an important question.

>> JOHANNA WEAVER: It is a good question and why attributions statements by governments are so complex.  A common question that comes up, critiques that come up with attributions that have been done by government, they are not specific enough, not saying what law was breached, what norm was breached, et cetera, and part of that reason, there is a consideration on how that impacts on future prosecutions, there is that dimension that's there.  The process of a criminal indictment is different than a process to make a public attribution that a government is responsible for and action or an activity and the indictments, of course, they're usually sealed which makes coordination quite complex as well because the information, by their nature, it is not made available until it is public so it is a factor that is very front of mind when we are preparing attribution centers.

>> MARIETJE SCHAAKE: We know Civil Society and academia are interested in attributions but efforts to prioritize it is often stopped, what's a way out of this?

>> We're all struggling with this, I can't even count them on both hands (John Scott‑Railton) meetings on the importance of attribution in Civil Society, I think it comes down to something simple, which is capacity, we're in the early days of organizations that are in Civil Society being in a position where they're technologically empowered to do their own work and to be technical partners with other stakeholders, coming from a couple of different polls, building out capacity, a help now, Citizen Lab, around a minute, with the University, we're moving there, but this is an example of why we need many more Civil Society players with resources and ability to tap into expertise.

>> MARIETJE SCHAAKE: Is it time to create principles of attribution, technical, legal, even political level corks this be developed by an independent community, academia with company, for example, then responsible states may embrace it to make attribution to be more trustworthy and transparent and it is built on by saying I know that the CyberPeace Institute, several threat intelligence firms and other academics, including the Internet Governance Project have been involved in an effort to create a transnational attribution Working Group to be independent and neutral attributions.  Do panelists support these efforts?  Who wants to reply?

Maybe that's an answer in and of itself!

>> SOFIE MADDENS: Since I was named, and there is considerations of how we can make attribution more trustworthy, it is one of the problems we have, attribution, it is that one political block attributes, the other one, typically say this is our ‑‑ this is fake, you made it up, in fact, it is really hard in the internet to actually come up with conclusive proof unless you kind of show the incapacities and capabilities that you have.

This is a problem that intelligence services had ever since they existed.

A proposal, it was that maybe it needs to be an independent body or a network of bodies similar to what's happening in the area of biological, chemical Patent, you have different labs that assess or make analysis and peer review these to lend this more credibility and I think that we are going back to what John just said, it is a capacity thing, if done by volunteers, done in particular by Civil Society, why should Civil Society who has very few resources anyway, they start doing, checking attributions things, of cases that you're not really concerned with.  There is a bit of a bootstrapping problem, I maintain that the idea of having independent organizations be they states, private companies actually saying, hey, this is an analysis that was done properly and it has merit and this is really good work, this is something that can lend trust to this but it only goes so far and I would argue it only goes to the fact finding part and we have heard Johanna Weaver say it is the technical part, the technical communities, not doing a legal assessment, but we're pretty good in saying technically this is really solid and making the next step, it is something that typically only states can do, they have only access to signal intelligence.

It sounds like a really cool idea and I kind of still like it.  I think we have work to do.


We will fold in another question that was related, would it be possible for certs to assist in delivering the proof to the governments so others can attribute to it?

>> JENS MONRAD: I like the idea.

I have seen that very early discussion about a Cyber Geneva, you know, convention agreement.

The only challenge I have with it, to this date, it is that it would only be effective if every country agrees to abide by whatever is agreed in such a convention.  We still don't have trials of engagement, not every country agrees what is a state sponsored, you know, campaign, some ads are delegitimatized, it is a duel use organization, it is spying, we're expecting intelligence operations and agencies to conduct these against each other.  Because we don't have common sense agreement on what are states allowed to do, when they should not do it and how can we sanction or pull out whenever they're doing something that's not abiding to the agreement, then I don't believe that we are where we are supposed to be unfortunately.

>> CAMILLE FRANCOIS: Yes a few things come to mind.

We have sometimes a tendency to talk about attribution and all cyberattacks like in the same type of remedy and I think it is untrue to generalize that point, sometimes it takes a long time to attribute and that time is important to take and you don't want to rush.  Sometimes it is for instance a hack in the middle of an election cycle and speed is of the essence.  Not all of these attacks warrant the same type of treatment from an attribution perspective.

The second thing is, from my perspective, it is important to recognize the small ways in which Civil Society here to has been leading by when some of the organizations do attribution, providing an extraordinary amount of detail and providing the forensics for how they come to their conclusion.  That's because the bar is higher for them.  When an institution like Signal Lab comes out with a paper and they're attributing a way of saying you can look at this work, draw others in, attribution as a dialogue, that's something that we see emerging, we see others do the same to the extent that we can on IO, we try to do the same, it is okay for people to have different attribution practices and to the point that I was making, attribution is not just a set of technical oops you run through and there are different standard for attribution specifically on new and emerging types of threats.  In the world of IO, we see institutions have different standards for different degrees of confidence and I think it is okay to maintain those different standards as long as everybody who participated in the conversation does the work of showing where they're coming from and the statements saying that this is why we say what we say, this is how we do what we do and these are the types of indicators and what type of weight we assign to them.

So we can go in many details, but for instance, one of the sour points of IO, it is what some people would consider the total match of content between campaigns to be sufficient for attributing to clusters and the same actor, many others would consider that no, this is too low of a bar and to some extent, that's a very healthy, important discussion.  It is important for a different institutions to participate together in creating more robust collective security practices, some of that, it is important to have in public.  In much of the debate, I think we see Civil Society really leading on the transparency and on creating the bar for sharing forensics.


Going to the next question, directed at Johanna Weaver.

She asks ‑‑ I think she, hopefully, that last year and this year, multiple cyberattacks took place against the Australian parliament, would there have been value in Australia publicly attributing the attacks or are you building on building the cyber offensive capabilities to deter the attacks.

>> JOHANNA WEAVER: That's a good question.  I didn't see it in the chat.  I had no forewarning.  That's great.

Australia made public statements about the attacks on our parliament and the Prime Minister spoke about it on the floor of parliament not long after we became aware of it that it had happened.

We have been very open and transparent about our approach to that and also the steps we have taken for mitigation.

We haven't publicly attributed to a country and as I said, we weren't always publicly attributing to a company, but we may still be publicly signaling that we are aware that that activity is going on and that that activity is unacceptable.  That is a strong message in the context.

I forgot the second part of the question.

Are we focusing our efforts on ‑‑

>> MARIETJE SCHAAKE: You know, would it have been useful to attribute or is Australia focused on developing offensive capabilities?  I put it to the answer session.  This is off the top my head.

>> JOHANNA WEAVER: That's great.

I think again, this is sort of tying in to some of the comments that other people have been making about the focus of countries who are wanting to gain offensive cyber capabilities instead of focusing on cybersecurity.  Australia is very transparent about the fact that we have offensive cyber capabilities and we use the offensive cyber capabilities, and we're one of the few countries that is on the public record about this.  We're also on the public record that we use our capabilities in accordance with the agreed rules of engagement and on that point, I think to Jens' comment, there are rules that have been agreed by countries about what countries should and shouldn't do in cyberspace and obviously we have not done a good job of ensuring that those rules are disseminated and that we are aware that countries have agreed to this, every country in the world has agreed that international ‑‑ that brings with it an entire body of rights and obligations that come with the engagement in the international system.  We have a great 11 norms of responsible set behavior, the problem in my view, it is not that we don't have rules and not that countries haven't agreed to those rules, we have ‑‑ it is that we're ‑‑ that there is not enough adherence to the rules and we come back to accountability and I think to go to milt ton's point, about are we having the same conversation, the conversation has moved on, because we're now talking about accountability and I think that's a very important step forward and quite remarkable progress in the period of time.  We need more and we need it faster and we have moved on.

I'm happy to talk more about Australia's approach to either offensive capability or the work ongoing at the UN because that is the two forums which we are agreeing on those rules of the road and we welcome suggestions in terms of how we can better have ‑‑ how we can ‑‑ how we have developed better understanding that there are these agreed rules, that they do exist, how do we get the information out stronger and better. 

Excuse my poor language.  It is 3:00 in the morning!

>> MARIETJE SCHAAKE: It is an important point, there are laws entirely, it is not a Lawless realm.  I think it is a Spectrum of clear application of existing rules and ambiguity about the application of existing rules and the lack of rules because technologies are so new, because there are new questions on jurisdiction, for example, and so my humble suggestion would be that it is creating that clarity and applying the rules, you know, in action, not just in a theoretical discussion is something that could help create de facto clarity and not just academic clarity.

Any question that was asked that I would like to move on to now?  It is interesting to better understand also how rights apply in modern technology and how it changes, is different evidence required for technical attribution and legal attribution?  How should we deal this?  John, I see you nodding.  Coming to you for this one.

>> JOHN SCOTT-RAILTON: I'll say, you know, I think we had a great early conversation about attribution and ultimately, different communities have different radical standards and the way that certain actors are going to be able to do attribution will only go to a certain place, legal things, different standards, so the short answer to that question, yes, there are many different kinds of attribution and there are norms within each community whether it is legal or cybersecurity research or government about how to state conclusions and how to make directions.

In practice, many of the things that look like legal attribution is really hard for Civil Society actors to get to, because what we see, what we don't, they can be challenging for companies, deciding on their ability to get to it as well, the big problem we have, many of the most toys for attribution in a full some way, meeting people's intuitive standards, fingers on the keyboard, they live really only within states and if they're lucky, with big corporations.  We need to be able to have a conversation that acknowledging that and at the same time accepts that we can have conversations about evidence that don't always have to reach that standard.

>> MARIETJE SCHAAKE: That's very helpful.  Also in terms of the question of how do you bring the two together, it is where many of you started.  We need a more public debate on what we know about attack, methods, and then about actors that maybe even capable to deploy them and if you want to use attribution towards accountability, then the challenges, for example, of such attribution and possibly sanctions, they could come before a court.

You probably need technical attribution to maintain the violation of the legal threshold and vice versa.

I would like to go to the next question just to be sure that we keep them going.

There is a question asking it seems to be missing in the dialogue, small states, developing economies in Africa, Asia‑Pacific which form perhaps the weakest links, any advice or guidance for such and then there is a word that I didn't know, so for such nations possibly.  Any thoughts on that?  How do we make sure that there is capacity, engagement of smaller countries.

>> JOHN SCOTT-RAILTON: Briefly, it is absolutely the case that these smaller countries are often racked both with cybercriminal problems and in some cases with nation state driven activity and ultimately the challenge of getting those countries capacity, it is a really big one and I think fundamentally under addressed.  What's interesting, one area where they do seem to be getting capacity faster than protection, it is in surveillance.  We see the industry reaching out to the smallest of countries like Togo, selling them sophisticated nation state grade surveillance kits,  countries in East Africa even with no ability to protect themselves.  Just an interesting point there.

>> MARIETJE SCHAAKE: I'll come to Serge next, after years of negotiation, there is political agreement on stricter rules for exports of survey systems out of the E.U. to any buyer that may use it to violate Human Rights and I think that the question is how will countries and international entities like the E.U. or others, make sure that they bring together for example development which involves capacity building and these higher standards when it comes to capacity building and also safeguarding people's rights and safety in the countries.  Oftentimes it looks like even within governments there can be complete divorce between the stated goals, for example, Human Rights protection and activities, for example, licensing the export of those systems that John observes on the daily basis that will actually actively harm Human Rights.

I guess sometimes simply aligning stated goals with practices would had do miracles in the right direction ‑‑ would do miracles ‑‑

>> SERGE DROZ: Yes n is more with what you were saying, there is ‑‑ you always talk about this digital divide, the digital gap and it is just a fact that a small nation doesn't have the means to do a full attribution program and in fact, actually operating the infrastructure it takes to bring this.  I think that the idea, the very idea that every nation has its own independent attribution capacity, it is just ‑‑ it would be nice.  It is a bit of a dream.  It is naive.  It is willing that's ‑‑ it is the attack community, they have to take responsibility, to start to understand and understand what are the cultural differences or the different cultural habits in these, and the values of the countries.

About a year ago I was in the Pacific, on the Pacific Islands and it may sound strange, traveling to the Pacific Islands from Europe these days, but I was astonished and talking to all of these Pacific Island nations on the challenges that they have, it is tech coming from the west, and these big tech companies kind of just having no idea of what it is that they're doing there.  Not only an attribution generally in bringing, exporting our Western technology to these places and we have to become a lot better of anticipating that the world is a global village and it is a village with many different neighborhoods and I think this is a big challenge.  I don't think there is a way that a small, not very well seen state can match up technologically to the big super powers or the big tech companies and that's something that brings a lot of responsibility for those that have the power.

>> MARIETJE SCHAAKE: Thank you.  It points to the need to have an international rules‑based order that puts a minimum standard in place and would also help towards accountability.

>> JENS MONRAD: Yes.  I think also to the point that John made earlier, at least from what we're seeing from the threat perspective, it is that there is less focus on defensive capabilities and much more focus on offensive capabilities.  Because we're still talking small states, they might be as John mentioned, they may be purchasing commercially available surveillance software.  They may have a different codex and who they're using it against and they may have a different structure, system, we don't necessarily agree with.  We could be better talking about that.

Also maybe to say why attribution also matters, talking about these activities, regardless of who are doing them, eventually when I look at, you know, all of the offensive operations that I'm aware of, from all over the world, historically, states are doing a terrible job protecting their offensive tools, cyber super weapons if you like, and eventually they'll fall in the hands of, you know, cybercriminal, other adversarial that we'll use them against Civil Society, critical journalists, private organizations and so on.  That whole race to be the next cyber super power, it is where attribution is really important as well.

>> MARIETJE SCHAAKE: I couldn't agree more.

This question of proliferation, technology in one context ending up in another context and sprawling out of control often, it deserves more attention.

>> JOHANNA WEAVER: I think also to link on the question very much related, I think that this idea of militarization, the continuing spiraling of the development of offensive cyber capabilities, it is really important and it is ‑‑ the point that we need to focus on here, it is that if countries are developing offensive cyber capabilities that must go hand in hand with the commitment that they will use them in accordance with international law and the agreed norms.  If countries are not willing to publicly disclose that they have the capabilities and that they will use the capabilities in accordance with the frameworks, then that is concerning.  To go to the point on well, isn't then developing the cyber tools against the purpose of the United Nations, no, because countries develop and have weapons of many different types and we commit to becoming ‑‑ by becoming members of the U.N. to use them in accordance with the agreed rules.  It is no different in cyberspace.  The problem is, there is ‑‑ you can count on your hands the number of countries that have publicly disclosed they have the capabilities.  Transparency in the capabilities must also be a priority as well as with respect to accountability.

>> MARIETJE SCHAAKE: That's great.

One more question, if deterrence in the future or ‑‑ is, the future, the future goal or in the future, one of the goals of attribution, are there analysis on how it fairs with regard to technical attribution or techniques used in an attack and then designing defense mechanisms..

Countries do not agree on norms and terms, does this apply to democracies and less democratic countries or also between the two groups?  The latter would make cooperation even harder.  It is more of a statement also.

Is this a division line that you see, is it a little bit of a simplification?

>> JOHN SCOTT-RAILTON: I was going to say something to the earlier question.  Sorry about that.

One thing.

It is I think we're still learning about what the disruptive power is and isn't of naming and sharing for example.  An area that's promising, it is tracking how threat actor DOS and don't adapt and change their behavior when they're named publicly.  You know, I can think of some examples of certain state actors who largely sort of said damn attention, full speed ahead, other, later on, they have been much more weary of attention.

I think we're still in a very promising place for understanding some of the different directions but it is very much in its infancy and that gets to the question of deterrents and I think ‑‑ you know, it is evoked here before, that there are questions of increasing the cost to attack and I think one of the things that we just don't know right now, it is how much public exposure does in and of itself, when say that, I think take a look at the cases of companies that have threat actor, when not associated with any government engagement.  What you typically see is, changes in behavior, so costs incurred to the attacker, but also the attackers don't just shut shop usually, they come back in different ways and I think this is a very interesting area for more investigation and I want to pivot to that next question if others have it.

>> MARIETJE SCHAAKE: I appreciate that.  With that, I'll do a subtle hand‑over to John Hering, I have to jump on to another session.  I miscalculated time zones.  For the conclusion of the session, the last couple of question, John is your moderator and for me to ‑‑ just a small word of thank you.

>> JOHN HERING: Thank you.  I appreciate having you here for this time and good luck in the next session you're in.

I think I have probably seen most of you in various sessions across IGF, I'm a government affair manager at Microsoft on our digital diplomacy team.  I don't want to interrupt too much.  Johanna Weaver, you had the next answer to the question posed.  Over to you.

>> JOHANNA WEAVER: I'm distracted, I'm simultaneously streaming to meetings at the moment, one of which is the UN meetings that are discussing the rules of the road and my delegation is currently speaking.  I'm trying not to be distracted by that.

I think on the question of is there agreement and is there division, the answer is that there is both.  Countries in 2015 ‑‑ actually 2013 and again in 2015 agreed to these seminal points that existing international law applies in cyberspace and they agreed to 11 norms.  Those norms, they're things like don't attack critical infrastructure using ICTs, they're very high‑level.  There is universal agreement on those.  We're having extensive discussions at the moment and in a group which has 193 countries and another with 25 countries and we're looking to put meat on the bones of what does it mean it say that international law applies in cyberspace.  That's a wonderful high‑level statement but how does it apply?  How does sovereignty apply in cyberspace?  What's it mean when we say cause damage to critical infrastructure.

There is so much that you need to drill down on and to provide more detail on.

The fundamental principle, there is the agreement and even in the fairly robust discussions that we're having, no country is disputing those fundamental agreements.

There is a robust discussion on the how and that's what we're hoping is the outcome of the two sets of negotiations, one which is due to wrap up in March and another which is due to wrap up in May.

Obviously, there have been interruptions by COVID, if you can imagine the pleasurer of a wet WebEx call with 193 countries on it, welcome to my life!.

>> JOHN HERING: Anything to add there?


Essentially what Johanna Weaver told us, this is really hard.  We have to take international law that was designed to norms exploding in the back of our minds which is now cyberspace which is something clean, clinical, there is no bombs going off and maybe it is not true but the other thing that makes it really hard, it is not states that operate this infrastructure, it is not states that design the technology behind it.  When we want to implement the norms on how behave, you have to get the tech companies in there, it is totally different from states, there is a lot of cultural understanding.  I think that we need to build up how you can actually take these decisions from international laws in cyberspace down to what it actually means in the everyday life of an engineer.  Just to give you an example, this is not just theoretical blah, blah, Facebook was named for the first time in history as a private company guilty of helping in genocide in the Myanmar debacle.

Everybody agrees that international law is valid, as everybody thinks security is good and what it actually means on the different levels, it is a really, really big challenge and it goes beyond what states need to do, but it is also up to what big companies need to do and companies do different types of things as we see, the answer to these questions from Facebook, Microsoft, Google, they're all very different and we're in a learning phase.  It is really cool to be part of that.

I feel a certain sense of urgency..

>> JOHN HERING: I want to circle back to one question that I think pulse together a couple of different threads from the open questions that we still have in the Q&A, and also one that I wanted to Mick sure that the Pam had a chance to speak to.  A fairly political one and similarly rooted in the times that we're in, which are increasingly polarized, on the other side of the fairly contention election here in the U.S. in which there is a deep, deep mistrust across various political divides and we exist in a time of such mistrust, of such polarization, does this undermine the potential of these attribution efforts which rely on people trusting that the work being produced is something that they can have confidence in if there is action taken on, especially.

I guess my question, do you see that as being a necessary challenge to address it and how might we go about addressing it.

>> JOHANNA WEAVER: So I think if you want to look for politicization around attribution, look at the drama and the challenges that were faced when information is put out.  That case is instructive, it shows us what happens when good work, that is complicated, relies on, you know, work shown, but analytical steps, it meets political realities.

I think that the same exact thing is truth attribution, we shoo it challenged when we attribute to certain nation states that are responsible for attacks, we think that ‑‑ the thing I can say, the particular problem, it doesn't get solved just in the attribution case, you do the world's best attribution and still get met by that.  It points to the larger issue of politicization, where I think that we're still in the infancy, wondering ‑‑ understanding what happens when like Western and NATO states decide they want to challenge attributions they don't like.  I think that's a giant X factor and we don't know what that looks like or what it will do to the space.  I'm watching out for that.

>> JOHN HERING: Over to you.

>> JOHANNA WEAVER: I wasn't actually requesting the floor believe it or not for once!.

>> CAMILLE FRANCOIS: Also looking at what's been holding in parallel, it is an interesting case that I want to bring here.  As you know, in the U.S., there's a conversation involving around accountability for tech companies and specifically questions asked about decisions that major platforms have done to reduce the sharing of the New York Post Article containing material who's origin people ignored.

It seems that a handful of minutes ago, Twitter said on the record we made a fast decision and we thought that the material could have been hacked and that this could have been a foreign actor campaign, we therefore removed it and we were wrong and we were reversing of that decision.  You know, it made me think about the conversation that you and I had earlier about how some of these incidents do require industry to move at a pace that governments will not keep up with, it makes me think back to when my team exposed information operation operating the U.K. government in November of 2019 and the U.K. government, they came out with an attribution confirming this 8 month later, it is great, I'm glad they did, 8 months later, it is not the timeframe that at times industry needs to operate in to do mitigation and enforcement..  I was reflecting on the different constraints here and the reality is that our attribution conversation needs to acknowledge that many different actors need attribution for different purposes, including industry whose often, you know, more often seen as a producer of attribution and who sometimes also is a consumer of attribution for not only security purposes but for enforcement purposes.  I'm inviting us to draw on our thinking on the many different institutions that do see the attribution statements and for the different purposes, and therefore what you need on which timeline.

That's what I had in mind, you caught me thinking.

>> JOHN HERING: I'm so glad you were nominated.  That was a rich contribution and highlights an important, delicate balance we have touched on a couple of times throughout the conversation which is between speed, making sure that you can address things quickly and also confidence in the accuracy.

Any other final thoughts to that polarizing question at the moment?

>> JOHANNA WEAVER: Just to cycle back again on one of the comments that was made in terms of the role of Civil Society and being able to fill that blind spot of the national security community and that really builds off of what you just said, there really is a balance here.  On the one hand, government, yes, we have ‑‑ some of us, they have great signals of intelligence and amazing resources and in many instances, the works of ‑‑ the work of Civil Society, of the ability to focus, to drive attention, it is really important and it does help us to be able to draw attention to those blind spots through the national security apparatus.

I think to come back to the comment by Serge, it is a team sport, it is something that we need to be doing together and it is something that governments need to be cooperating more with Civil Society and more within industry and this is something that we're prioritizing in Australia and something that we pride our self on and I hope to see more doing the same..

>> JENS MONRAD: A quick comment here.  I think it is important especially when talking about the role of social media and the future of social media, we also need to look at the younger generation, how ‑‑ where are they getting the news today?  Do they actually ‑‑ by nature, do they trust government attribution?  Are they picking up news items on social media, other places?  That's also where attribution will also become a very important question to raise and where we need to look into it.  It is not just on the government side, but it is also how do we present attribution, what sort of communication path are we actually creating to ensure that others are also getting unbiased information and relevant data so that they can form their own opinions.

>> JOHN HERING: Thank you.  I did not expect the virtual IGF to make us feel as close as it has over the last few days.  Thank you.