Home > News and events > News > Latest news > Text mining to analyse crowdsourcing outputs

Consult with rigour, listen with care: using text mining to analyse crowdsourcing outputs

26 October 2010

 

At ORC International we have been actively tracking developments in crowdsourcing and its implications for effective public engagement and consultation. Earlier this year, our article “Harnessing the power of technology in public engagement” introduced some of our tools, including web scraping and text analysis, for analysing the results of such online activities.

We have now put our technological solutions to the test using the Your Freedom website as a case study. The site was set up to allow the public to tell the Government which laws and regulations should be repealed and why. The website was established consisting of three fora: restoring civil liberties, repealing unnecessary laws and cutting business and third sector regulations. Over 100,000 comments were submitted across each of the three areas.

Clearly this has presented the coalition government with a challenge in terms of how to analyse this scale of comments in a comprehensive and meaningful way. Analysing comments manually is extremely resource intensive and has inherent risks, given the potential for individual error. This analysis problem is particularly salient given the context of budget cuts whilst committing government to actively engaging with communities at all levels. The dilemma of how to deal with high volumes of unstructured public feedback credibly and robustly with little associated resource is a problem that is likely to be faced increasingly often in future.

ORC International has always been at the forefront of developing new research approaches and our solution to exploring the wealth of comments posted on the Your Freedom site was to use our text mining software, PolyAnalyst, to process the comments in a format for easy analysis. Using web scraping software we were able to gather a subset of 25,240 comments across the three fora (representing around 25% of all the comments and ideas submitted) for further review.

Our aim was to provide an enhanced understanding of the comments and ideas, to look in detail at the responses and to identify statistical skews in comments mentioned across the three main fora and at an overall level. This allowed us to explore the strongest relationships between ideas posted and the nature of the comments associated with these ideas: effectively a process in making the qualitative, quantitative.

All of the comments were multi-themed and categorised into a number of headings. The top three categories were crime, health and rights. Each of the themes was then further analysed to create sub-themes. For example, the top areas identified within crime were cannabis legalisation, imprisonment, control orders and drugs generally. Within the health theme, smoking was by far the most discussed topic.

Using text mining to analyse crowdsourcing outputs

Structuring the comments in this way gives scope for additional analysis to be conducted which allows “golden nuggets” – those hidden gems of ideas – to be identified. For example, using a systematic approach we are able to:

  • Compare and contrast themes and sub themes across the three fora.
  • Review the ratings that each idea has been given and provide information on the highest and lowest rated ideas.
  • Explore specific ideas that may be of particular interest. For example, in the case of smoking all comments within that theme could be reviewed to see exactly what issues are being mentioned ie, children's health, civil rights or NHS funds.
  • Provide further breakdowns to explore exactly what people are saying in relation to each theme – ie, how many people are saying the public smoking ban should be repealed versus those saying that it should go further.

We believe that our text mining tools, through providing the scope to analyse an almost limitless amount of qualitative feedback in a cost-effective and rigorous way, offer a perfect solution to a perennial problem faced by many organisations. As leaders in the field of text analysis, we can ensure that when our clients consult, they do so in a robust and inclusive manner that gives every participant the opportunity to be heard.

If you have recently run an online public engagement exercise, are in the process of planning one or have other sources of unstructured, freeform feedback that you need to digest, our solutions will provide you with the analysis tools that you need.