First posted at Ethnographymatters

On the first day of WikiSym last week, as we started preparing for the open space track and the crowd was being petitioned for new sessions over lunch, I suddenly thought that it might be a good idea for researchers who used ethnographic methods to get together to talk about the challenges we were facing and the successes we were having. So I took the mic and asked how many people used ethnographic methods in their research. After a few raised their hands, I announced that lunch would be spent talking about ethnography for those who were interested. Almost a dozen people – many of whom are big data analysts – came to listen and talk at a small Greek restaurant in the center of Linz. I was impressed that so many quantitative researchers came to listen and try to understand how they might integrate ethnographic methods into their research. It made me excited about the potential of ethnographic research methods in this community, but by the end of the conference, I was worried about the assumptions on which much of the research on Wikipedia is based, and at what this means for the way that we understand Wikipedia in the world. 

WikiSym (Wiki Symposium) is the annual meeting of researchers, practitioners and wiki engineers to talk about everything to do with wikis and open collaboration. Founded by the father of the wiki, Ward Cunningham and others, the conference started off as a place where wiki engineers would gather to advance the field. Seven years later, WikiSym is dominated by big data quantitative analyses of English Wikipedia.

Some participants were worried about the movement away from engineering topics (like designing better wiki platforms), while others were worried about the fact that Wikipedia (and its platform, MediaWiki) dominates the proceedings, leaving other equally valuable sites like Wikia and platforms like TikiWiki under-studied.

So, in the spirit of the times, I drew up a few rough analyses of papers presented.

It would be interesting to look at this for other years to see whether the recent Big Data trend is having an impact on Wikipedia research and whether research related to Wikipedia (rather than other open collaboration communities) is on the rise. One thing I did notice was that the demo track was a lot larger this year than the previous two years. Hopefully that is a good sign for the future because it is here that research is put into practice through the design of alternative tools. A good example is Jodi Schneider’s research on Wikipedia deletions that she then used to conceptualize alternative interfaces  that would simplify the process and help to ensure that each article would be dealt with more fairly.

Talking about ethnography?

I am still intrigued by the fact that so many quantitative analysts wanted to know about ethnography during our open space session. We started the session with those who had done ethnographic work talking about their experiences: Stuart Geiger talked about his ethnographic work on Wikipedia bots, Isis Amelie Hjorth talked about her ethnographic enquiry into Wreckamovie, the collaborative movie outfit from Finland and Paško Bilić discussed how he studied breaking news stories on Wikipedia. Others wanted to know how you even begin to do ethnographic research on Wikipedia when editors are a) anonymous and b) located all around the world. One participant said, “I’m faced with 3 million edits (in my dataset) and I have to say something about them. How do I even begin?”

I think this comment points to the key problem that quantitative researchers are facing. Many are recognizing that after they have done the social network analysis of a particular category of articles or come to the conclusion that a particular type of editor is dominating, they start asking “why?” And understanding the “why?” requires them to delve into a grounded understanding of what is happening at the so-called “small data” level, leaving them flailing in questions about how such a small sample can say anything about a world that they have described with thousands, sometimes millions of datapoints.

As a response to such questions, Stuart Geiger referred participants to George Marcus’s work on “multisited ethnography” that uses a question-based approach to studying a particular community and I told them about Jenna Burrell’s very useful paper, “The Field Site as a Network: A Strategy for Locating Ethnographic Research” but I think the key problem was that researchers often are thinking of qualitative research merely as the same dataset “with interviews”. It was a great conversation with some really interesting comments and questions, even if some participants left saying with a smile: “It was a good discussion but I still believe that numbers tell the truth.”

The sample

If I learned anything at WikiSym this year it was how critical it is to be clear about how the sample is selected. As Jenna wrote, when I sent her an initial draft of this post:

Regardless of research approach, I always think HOW the sample was selected is the most important question…always.  The issue of sampling frame…or even in big data approaches where you aren’t necessarily sampling, what you can say/claim stems from what is inside vs. outside of your dataset (whether its a sample or a complete enumeration).

What worried me about a lot of papers dealing with “culture” and “emotion” and “historical sense making” is the over-reliance on studying only the large language Wikipedias and then using this data to make broad claims about human behavior without taking note of the limitations. Below is a list of the top 15 language editions that were often used:

English, German, French, Dutch, Italian, Polish, Spanish, Russian, Japanese, Portuguese, Swedish, Chinese, Vietnamese, Ukrainian, Catalan

Thus, leaving out more than half the world and two of the world’s continents – admittedly better than many of the papers that analysed only English Wikipedia (as opposed to any of the other 284 language versions) and often didn’t even express that in the title or abstract.

On the bright side, although Brent Hecht has also chosen only the biggest languages for his Omnipedia project, his drive to find an alternative method of comparing different language Wikipedias without using English as the “template” against which all other encyclopedias are judged is a really noteworthy accomplishment.

When I became frustrated at the fact that researchers were making broad claims about “culture” when they were studying only the large language Wikipedias (generally located in the developed world), I heard things like: “This is just the beginning. We’ll get to those in time.” (Or more often: “This is just the beginning. I’m not going to do that research because the datasets are too small but I’m happy that you will be.”) And it frustrated me because whenever “further research” was mentioned, small language encyclopedias were not, nor were questions about minority groups on Wikipedia.

This is not, in the end, an ethical question. It is a question about good research. I don’t believe we can truly understand  Wikipedia’s role in society or make the kind of broad claims that we’re seeing if we only look at a small section of the data. Researchers often complain that they can’t do broader, Wikipedians “in-the-world” kind of research because it’s too hard – Wikipedia doesn’t make users’ locations accessible unless they are anonymous (1) and there is too much data to analyse. But this perspective comes from a very particular way of seeing the world as some researchers feel that they have to analyse the same amount of data qualitatively that they analysed from a quantitative perspective when good qualitative research is much more about selecting the right kind of data, getting a grounded, participant driven perspective on the world, and truly understanding what Wikipedia means in the context of the outside world.

What aren’t we measuring?

I was surprised that no one in the panel entitled “What aren’t we measuring?” mentioned the obvious bias towards English, developing world and large language groups. The panel was interesting, however, especially since it elicited questions about how to measure Wikipedia “activity” and the fact that such activity is often equated with a small sliver of Wikipedia activity, leaving out things like offline Wikipedia activities including training and advocacy. A lot of this seems to have to do with what data is available and what tools we have to analyze it.

Which leads me to the question:

To what extent do you believe that the available tools are influencing what kinds of questions we are asking and how we are choosing to answer those questions?

Surely, if the questions were really about cultural and language differences and different ways of seeing the world we would choose a sample of languages from cultures that seemed very far apart rather than a sample of languages from cultures that are very close together? The answer is usually that the dataset for those “far-away” languages is too small,  too disorganised, too uncategorized. This seems to be a perfect example of the tools determining the research questions rather than the other way around.

In his keynote, Jimmy Wales focused on the need to focus more on African Wikipedias. He said that the Wikimedia Foundation is focusing on India and Brazil in their Global Development Program, but that African countries would be next. I always so appreciate Jimmy’s personal interest in the continent and I know that he is only one person on the Wikimedia Foundation Board, but what concerns me is that, although the Global Development Program does amazing work, very little of it is research. And sound, grounded research to understand more about the country and languages in focus would go a long way to avoiding the kinds of problems that have occurred in past – problems like rampant copyright infringement in the India pilot which might have been helped by a deeper understanding of intellectual property questions in India. A contextual understanding of copyright in developing countries and ways to frame this problem is going to be a really important research topic in the coming years as Wikipedia grows. Without it, and despite the incredible staff in the program who have learned how important it is for them to work with locals and get a better understanding of the context before they start major advocacy work, Wikipedia might be seen as just another Western Aid project and not the global project promoting tolerance and understanding that many would hope it to be. If the research on Wikipedia continues to be done on major languages and developing countries, we won’t get any closer to this kind of understanding.

The value of participant observation

The great thing about WikiSym is that there are Wikipedians in the audience who call researchers out when their findings do not equate with their own observations. Another question that came up in the conference was what it means to base your quantitative analysis in the grounded experience of Wikipedians. Being a participant in Wikipedia is one way of solving this problem: you’re more likely to recognize when your findings are incorrect if you’ve had direct experience in the behaviors that you’re counting. As Stuart said after the ethnography meetup: “We need to collaborate and keep each other accountable!”

So that’s what I’m doing – or at least trying to – with this post. When I’ve been criticized for things I hadn’t noticed or thought about in the past, it’s made my research better. And I hope that my role as critic here is somehow constructive. One researcher caught me after the conference to say that there are plenty of conferences I could go to where it wouldn’t be left to me to keep complaining about these issues. It got me thinking: where does ethnography belong? Or more accurately perhaps: Where do critical approaches that put people at the center rather than the tools belong?

Looking forward

In the closing session last year, I remember saying “I have been to a lot of conferences lately and I don’t feel like I belong. But I feel like I belong here.” People come to WikiSym because it’s the place to be if you’re doing Wikipedia work. In the words of conference chair, Cliff Lampe said, “WikiSym is the place we come where we know we don’t have to explain ourselves. Where people just “get it”.

But at the end of this year’s conference, as we stood in a circle and shared our parting thoughts, I didn’t feel like people were getting it. I said, “I was really happy and really angry and really sad and really excited…” That’s one way of saying that my initial enthusiasm has been tempered this year by my feeling like such an outsider, like the old grouch who keeps thinking that Wikipedia research has a bias towards asking very particular questions using very particular tools without a recognition of the limitations of that lens.

I’m still committed to WikiSym – at least for another year – and next year I hope to organize a panel that reflects on the type of research that gets done for WikiSym and outlines what kind of bias that represents. Because in the end I love what wikis represent: the ability to change something if there are enough people behind you. That’s one thing that has stuck with WikiSym and this year Chris Lampe did an amazing job at putting the wiki way into practice. It’s still one of the only academic conferences that I know of where there is a parallel open space track. And I guess that’s one of the ways that keeps WikiSym-ers like me coming back.

(1) anonymous users’ IP addresses are their usernames so you can look up where they resolve to.
(2) this is an interesting observation since it presumes that ethnography and counting are mutually exclusive when that is often not the case.