Bob Kraut Talks Community Design: “It’s Not Enough To Analyse Social Data, We Need To See The Effects of Interventions”

I’m at the 4th International Conference on Weblogs and Social Media (ICWSM 2010) in DC this week, and the first keynote talk was given by Robert Kraut, a professor at the Carnegie Mellon University. The topic of his talk was how to build better online communities by adapting existing social science theories. This is a pretty long post but there are some interesting observations.

For the past seven or so years, Robert and his team have been studying how online groups or communities operate, and how the social sciences can benefit these groups. His theory is that some of the accepted wisdom about how offline groups work can be useful as a basis for understanding and designing online communities, but on the other hand, they are also limited since they weren’t designed for the online world. Looking at a few variables at a time (the norm in organisational analysis) doesn’t suffice for a complex system like an online community, especially if you are trying to intervene in the design of that system, as there are potentially hundreds of variables you are trying to play with.

Online communities and social media in general are both really important phenomena due to the large numbers of people that participate in them. An online community is basically an interconnected collection of people who interact over an extended period of time around some shared purpose or need. Current social media research mainly treats online communities as a natural phenomenon, using standard methods from the social sciences, but ignoring what we need to do to make sure that any interventions we make in online communities are beneficial.

If we intervene in the development of an online community, we are most likely trying to achieve stuff like making more people join the community, allowing people to generate more social capital, or just improving the community in general. In order to achieve these aims, there’s a need to go beyond simply looking at/analysing the data, and to start doing interventions and simulations to see how perturbations in the community will affect the community.

Online communities face many challenges, such as: how to get started when there are no people to provide content or there’s no content to attract people; how to recruit, select and socialise with new members; how to get people to stick around long enough so they can operate in that community; how to regulate the behaviour of outsiders and possible vandals; and how to coordinate activity in general. Features of online communities (in comparison to offline groups) such as anonymity, weak ties, high turnover, and a lack of institutionalisation make these challenges more daunting online. For example, how do you introduce newcomers to Wikipedia so that they know how to coordinate edits, be successful in their contributions, and not antagonise the old-timers who think of it as their own domain.

Commitment is an obvious problem in online communities (i.e. one’s psychological commitment to a group), and if one can examine the behavioural outcomes from different communities with the aim of improving commitment, then we can produce members who are more likely to help serve in the group’s name.

Robert listed some of the problems in online communities such as Usenet, Wikipedia, etc.:

  • In Usenet, 60% of people who post in a month are never seen again.
  • In Wikipedia, a typical editor edits a page only once.
  • In MovieLens, the half-life of newcomers is less than 18 days.
  • A World Of Warcraft guild loses 25% of its members per month.
  • In Second Life, 90% of users leave after registration.
  • In an online cancer group, 85% of those who register never participate.

But there are lots of design levers available that should allow us to encourage commitment. We often take decisions when we set up an online community such as: should there be policies for newcomers; should they be kept in a sandbox; should one create subgroups for a community; should off-topic conversations be moderated; or do we allow anyone to say anything at any time. These sort of decisions that will have an influence on how committed people are to the community. But we don’t have a priori knowledge on how those decisions should be made or how those decisions will interact with each other. When designing online communities, social science research and theory gives us some hints about how to use these design levers and guide these decisions, but it gives no guarantees. For example, it’s not clear how small group theories in organisations apply to online communities where you have a high turnover. It is also not clear if the theories we have from the social sciences, which typically take three or four variables at a time into account (reductionist theory), are sufficiently rich to give us an insight into the design of online communities.

Robert’s approach is to use empirical methods to assess if the methods from the offline world can be extended to explain the differentials and successes of online communities, and to use computational simulations to perform ‘what if’ experiments to see what the effects of a design intervention will be on an online community. His three main theories are:

  1. Social science theory is a useful basis for understanding behaviour in online groups. In line with socialisation theory, the first interactions that newcomers have shape in a very fundamental way the long term behaviour of those newcomers.
  2. Social science theory is a basis for designing online communities. The theories of social identity and liking help indicate how a person becomes more committed to an online group.
  3. We can get some leverage from existing literature but it is somewhat limited and results differ from those observed for online communities, so some new computational simulations are required.

Social science theory is a useful basis for understanding behaviour in online groups

The development of commitment amongst people in a group is a process that occurs over an extended period of time. What goes on as an individual becomes committed to a group changes over time. For the group, they have to decide if it is a benefit to have this new member join, and members themselves will assess the group to see if they want to maintain a relationship: its like two dogs sniffing around each other. Newcomers will stick around in groups where they are treated nicely and are granted membership – its like a prediction of the future where they figure out if they will get benefit by joining. The group is looking at the initial interactions with the newcomer to get a sense of what the individual is like. If both the newcomer and the group perceive this positively, it can be a mutually-reinforcing upward spiral.

Some of the CMU research focusses on the cues that groups give off that newcomers use to figure out if they want to join or not. What is it that individuals give off that allows the group to determine if they want to be nice to the individual? In a survey of 98 Usenet groups, with 28869 newcomers joining over 650 days, the research looked at 221092 messages that they posted. Newcomers – when the group is responsive to them – are much more likely to stick around than if they weren’t (initial interactions make a big difference). The signs of acceptance are more powerful if the group uses inclusionist language, ‘we’ vocabulary terms, and avoids exclusionist language, ‘us’ vs. ‘them’. If groups engage, then newcomers are more committed. There are also things that the newcomer can do to demonstrate commitment – which in turn help the group respond more positively to them – such as displaying visible signs that they are more committed to the group.

Groups will respond to a newcomer’s “membership claims” in a reciprocal vetting process:

  • A message from a newcomer such as “Has anyone tried this?” with no membership claim usually doesn’t get many replies.
  • A message such as “I’ve been here for a while and am ready to jump in, has anyone tried this?” gets a much better response (where the person says that even though they’ve been invisible, they have been a part of this group, and they are now ‘de-lurking’).
  • A message such as “I’m interested this topic because of reason X and I wanted to ask if anyone has tried this?” also gets a good response (where the person is saying they are part of the social category from which this group draws its membership).

Membership claims increase replies, and the group-oriented claim has a significant increase. Their research replicated this fundamental result in Wikipedia projects (where a named group of editors manage a collection of pages on a particular topic). In the first two weeks of interaction after someone added their name to the list of members, where there was a higher degree of interaction between newcomers and old-timers, the more edits the newcomer made over the coming months (12 times more in fact).

Socialisation theories claim that one of the ways you can get newcomers to be more happy and to participate/stick around is to give positive feedback about their behaviour in the group. The same theories say that what doesn’t work is constant criticism. One of the tests from CMU was to find out if the old-timers in the Wikipedia project were taking the best route by giving positive feedback (such as an exceptional newcomer award) or by offering regular constructive criticism (e.g. you need to avoid copying materials from other sources). They found that there is always a slow decline in a person’s participation in a project. However, with criticism, there’s a slower decline than without it. The implications of this research for community design are that offline theories are helpful but not definitive: the nature of interaction is different from conventional organisational behaviours. Wikipedia has welcoming committees, but research found that if these welcomers use templates to interact with newcomers, it drives them away (it’s almost worse than ignoring them), and the committees do exactly that (standardised templates serve as the basis for interaction).

Social science theory is a basis for designing online communities

We can not only use these theories as a basis for understanding but also for design. CMU’s research looked at the commitment of newcomers to groups, illustrating two theories in social psychology about how people are attached to communities:

  1. A person feels committed to the group as a whole and the things they stand for. Therefore, we should be focussing a newcomer’s attention on a group’s or subgroup’s characteristics.
  2. A person likes the people in they group, they don’t care about group as a whole, they just like hanging around and talking to those people after hours. If they all left, the person would leave the group too (Facebook cliques are like this because you like the friends you participate with). Therefore, we need to focus a newcomer’s attention on the individuals who are participating, not on the group as a whole.

The research took the generic MovieLens site, with 75,000 members, and where the average member stays less than 19 days. The aim was to try to see how they could increase how much people stick around, add ratings, etc., by focussing on either groups/subgroups or individuals. They began with a version of MovieLens with a focus on creating group profiles, emphasising group membership with a mission statement, comparing the group with other groups, and allowing newcomers to get information about the group as a whole, but the individuals themselves are submerged (‘identity-based’). The second test created a MovieLens with a focus on individual profiles having a name and picture, a profile of interests, comparisons with other people, etc., making it more like a Facebook profile in order to allow newcomers to get connected to individuals (‘bond-based’).

They randomly assigned newcomers and old-timers to one of three experimental conditions: a “vanilla” MovieLens; an identity-based MovieLens with social identity features focussing on groups; and a bond-based MovieLens with interpersonal features focussing on individuals. The MovieLens with the bond-based design (individual focus) had 11% more logins, but the identity-based design (group focus) had 44% more logins. This shows the strong effects of identity-based features, and the weaker effects of bond-based features. One explanation for this given by Robert is the time course involved: it takes a while to develop friendships, whereas social identity can be shifted instantaneously: give someone a t-shirt and they starting thinking of themselves differently than they did before. (There was another issue involved in MovieLens: interpersonal communication is essential, but they couldn’t get people to talk to each other much in the system).

Some new computational simulations are required

Finally, the researchers tried some agent-based simulations to provide some design insights by integrating separate theories (not one at a time as used by most previous studies). Existing theories use a small number of variables, but real online community design is multidimensional with many outcomes (beyond newcomer commitment) that you are trying to optimise across. Without going into the technical details, this agent-based model is used to explain a member’s motivation to participate in an online group (whereby people will participate if they perceive that they will get various types of benefits from the group). In brief, the model says that:

  • Design interventions (discussion methods, moderation techniques, newcomer socialisations, incentive structures) are based on…
  • Community characteristics (message quantity and quality, the size the group influences, how similar the person is to other members, and what kinds of interactions the person has from the group), which in turn shape…
  • Member benefits (accessing information, sharing information, having an attachment to the group, making friends with particular people, having fun in group, or having some benefit from their good reputation in the group), that then motivate…
  • Resultant actions (reading and posting messages).

As an example with this model, the researchers looked at what was the effect of discussion moderation on turnover rates, community growth and activity. Three types of moderation were considered:

  • No moderation: anyone says anything.
  • Community-level moderation: a human throws out off-topic messages, and there are rules that discourage members from discussing off-topic things.
  • In between these two extremes, there is personalised moderation: a person can create a profile of things that they care about, and anyone can say anything, but the user only sees messages that match their profile.

The research found that personalised moderation dominated in terms of logins per user for larger volume sites and more topically diverse groups. They also looked at what kind of benefits were individuals receiving from these groups with different moderation types. Personalised moderation improved both the information and social (bond) benefits, whereas community-level moderation improved a user’s information benefit, but only in homogenous communities.

Robert Kraut is currently writing a book on this topic called “Evidence-Based Social Design: Using the Social Sciences to Design Online Communities”. 90% of the content is available in draft form for free.

One thought on “Bob Kraut Talks Community Design: “It’s Not Enough To Analyse Social Data, We Need To See The Effects of Interventions”

  1. People are social creatures and require attention, recognition and, to an extent we are often loathe to admit, validation – social proof that we are who we think we are and who we are is OK.Computer systems thrive on automation and the idea so often seen on so many social media sites is that we can programme out the, sometimes, hard work of getting to know someone or become better acquainted with a certain group of people.No one is going to argue about the benefits of such automation but one could argue that in addition, a personal touch is vital.I have just been reading the obituaries of Martin Gardener and one thing that stood out is that he pretty much replied to, as far as I can tell, to everyone who ever wrote to him. I would say that would be a large factor in the adoration expressed for him on his passing.One assumption that may need to be challenged is the idea that social media spaces need to expand into monolithic behemoths like Facebook, Myspace and so on. Real, personal connection is sacrificed to the number counts of electronic connections.This talk seems to suggest that genuine personalization might be key to future endeavours in this space.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s