User:Masssly/WIGI Usability Report

wigi.wmflabs.org Usability Study

Introduction

Wikipedia Gender Inequality Index (WIGI) is a Wikimedia Foundation Individual Engagement Grant that seeks to present an ongoing statistical representation of data about Wikipedia's gender gap on a website. The website, WIGI.WMFLABS.ORG (or simply "WIGI") utilizes Wikidata to produces open data-sets about the gender, date of birth, place of birth, ethnicity, occupation, and language of biography articles in all Wikipedias. In its beta development phase, WIGI shows a few demonstrations of what can be done with that information.

A usability test was conducted to determine the extent the interface facilitates a user’s ability to complete routine tasks and make meaning from the visualizations and data presented. One-on-one remote interviews were conducted with study participants using Google Hangout to captured each participant’s navigational choices, task completion rates, comments, overall satisfaction ratings, questions and feedback. Sessions were audio recorded, transcribed and analyzed to identify potential areas for improvement to the website.

Executive Summery

During the week of November 17 - December 1, 2015, WIGI.WMFLABS.ORG (WIGI) was tested among participants pulled from the Wikimedia community, loosely described as anyone who edits Wikipedia-the-encyclopedia, or is a potential reader of any of its language versions. The study was carried out via video conference call with all participants, separately.

Nine (9) participants in all took part in the study.

The main objective was to get feedback from the Wikimedia community about how the WIGI presents snapshot data, how it demonstrates what can be done with the data and how it makes open data-sets available for download. The study assessed participants’ perception of the content along with their understanding of the user interface and went ahead to evaluated the study participants' reading enjoyment and overall user experience.

In general, 67% of participants found the website web site to be uncluttered, straightforward and tidy. Whereas 60% of the participants held that it was fun and easy to use, others thought that they spend too much time trying to figure it out; one respondent described their experience as "burdensome". All Five (5) participants who were video interviewed indicated that WIGI will help them further explore the gender gap in Wikipedia biographies. In their delight, one participant said that "I [now] have a deeper understanding of of the diversity and will make strong arguments backed by statistics". Another participant stated that" this is the tool we've been waiting for.

Key problems identified in the study include:

• Difficulty to distinguish between Recent changes and All time displays
• Inflexible Recent changes
• Lack of connection between between variables on the different pages
• Confusion over interpretation of charts
• Difficulty to gauge extent of measure (eg. percentages without figures, and vice versa)
• Use of presumptuous language and maths symbols
• Lack of data probing questions

This document contains the participant feedback, satisfactions ratings, task completion rates, ease or difficulty of completion ratings, time on task, errors, and recommendations for improvements. A copy of the questionnaires are included in the Attachments’ section.

Methodology

Thirteen (13) email invitations were sent out to respondents from an Initial feedback survey in May 2015 that sought to gather variables of interest from the community towards development of WIGI. Seven (7) people agreed to participate and responded with an appropriate date and time; Six (6) of them did show up for the interviews. One facilitator led each session lasting between an hour-and-half and two hours. Also, three (3) community members who were not part of the earlier survey but offered to provide feedback submitted their input in written form.

Participants were mostly long time members of the Wikimedia community; four (4) of whom were male and three (3) females. Two (2) of the participants who submitted written responses did not specify their gender.

During the Google Hangout session centered around a wide range of questions, the facilitator explained the test session and walked the respondents through all pages of the website while recording their feedback about the difficulties they encountered as they attempted find information on the website and make sense of its content. They were probed about what challenged them most, how differently they would like the site to behave and also about what pleased them the most.

After going through each webpage, the facilitator asked the participant to rate the interface; the chart, the data presented, and the wording of the introductory text, on a 5-point Likert Scale with measures ranging from Strongly Dislike to Strongly Approve. (see Attachment below)

Upon going through all the web pages, the facilitator asked the participant to rate the website overall by using a similar 5-point Likert scale (Strongly Disagree to Strongly Agree) for nine subjective measures including:

• Ease of use (Learn ability) - was it easy for you to learn to use the website
• Usefulness - Do you feel that you have obtained value from using the website?
• Fulfillment - Do you feel satisfied after interacting with the website?
• Enjoyment - Is the experience of being on the website fun and not burdensome?
• Information facilitation – Were you able to quickly find information?
• Look & feel appeal – Did the content make you want to explore the site further?
• Site content – site’s content would keep me coming back
• Site organization - Webpages and navigational features are well organised
• Positive Emotions - Do you feel happy, excited, pleased, etc. when you interact with the site?

In addition, the test facilitator asked the participants the following overall website questions:

• What the participant liked most.
• What the participant liked least.
• Recommendations for improvement. (see Attachment below)

Results and weaknesses

Overall, users felt WIGI was easy to use and would could give them enhanced (further) insights about gender diversity in Wikipedia articles. Some stated the project would guide them to know what to edit on Wikipedia. One respondent noted that "this data served as a wake up call to their work on minimizing the gender gap". Participants greatly appreciated the idea of an easy availability of information about Wikipedia’s gender gap located on one website. Majority were enthused about the possibilities for future interpretations of the data. Specifically, one users said: This is just amazing, don’t stop what you're doing".Another pointed that this project is "extremely the way to go".

Common weaknesses they pointed out was:

1. An unattractive color blend; grey on white text instead of black on white
2. Absence of percentages associated with female numbers, most notable was Gender by Culture
3. No actual numbers associated with female percentages
4. Not easy to recognize (or realize) that there are two sections on each page; Recent changes and All time
5. Display of only ten (10) top/bottom items
6. Ill-defined captions; i.e. Recent changes of what? All time of what?
7. Incomplete data; more than a year down the line since October 2014
8. Out of line display of numbers in the tables
9. No attempts made at explaining anomalous data
10. Unsortable tables
11. Obscured nature of chat navigational features
12. Unavailability of a search button
13. Rigid Recent changes

The rest are outlined explicitly below;

Landing page

Users expressed mixed opinions when asked about their first impression upon landing on the main page. Whiles some felt they did not need further introduction as to what the website was about, some genuinely though WIGI was analyzing data on the gender diversity of Wikipedia contributors. Others simply thought that the main page was not intriguing; one user said, “Nothing sparks me to ask questions”. However, users unanimously felt that the image used in the main page was spot on. One user said, “…it tells it all, it's so sad, it tells what we have been talking about for a long time now”.

Key weaknesses identified on the Landing page are:

1. Insufficient/unclear summary (explanations) of project
2. Absence of a descriptions of why WIGI is useful
3. Pre-emptive introduction
4. Does not pose research questions
5. Lacking a methodology of which 35% of users were curious about
6. Unattributed image

Gender by Culture

Key weaknesses identified on the Gender by Culture page are:

1. No percentages added to female column in the table
2. Y-axis difficult to interpret because of presumptive use of e+ mathematical notation
3. Table displays 9 cultures but not 10 as stated
4. List not numbered
5. Anomalies (e.g. numbers such as -2 for Africa) are not explained
6. Unexplained use of an unusual term; Mechanical Turk

Gender by Country

Key weaknesses identified on the Gender by Country page are:

1. Ambiguous meaning of WIGI Score and Change in WIGI
2. No direction to users to hover on map
3. List in table not numbered
4. unspecified (non defined) color gradient
5. Recent changes show only 2 countries, not 10 as stated

Gender by Date of Birth

Key weaknesses identified on the Gender by Date of Birth page are:

1. Some preferred to read chart with a hover axis-line
2. List in table not numbered

Gender by Language

Key weaknesses identified on the Gender by Language page are:

1. Difficult to perceive percentage in the table without a readable number
2. Absence of a real number besides the percentage for each language in the chart
3. Too few languages to visualize in the chart
4. X and Y planes difficult to interpret
5. No option to view fewer/more languages in chart
6. No option to view languages as measured by some other variable such as culture
7. Absence of a hover axis-line
8. Presence of difficult to understand ᴧ mathematical notation

Data

Key weaknesses identified on the Data page are:

1. Not clear what the Name, Last modified, Size and Description link will do
2. Read Me file contains no help
3. Long pathway to data files
4. Confusion about what is contained in each sub folder; either code, csv, etc
5. Huge downloadable files

Recommendations

Respondents generally recommended that it be made more obvious that there were two sections on each page showing different data for Recent and All time changes. They also noted that captions for both recent and all time changes were ill defined and confusing to understand, recommending that they be modified to reflect what it is they are measuring. The participants indicted their desire to be able to change between different recent changes; among many suggestions, "per monthly" was the most preferred. Furthermore, they were very interested in having a "see more" drop-down button on the tables to display more data beyond top/bottom ten. In addition to the display of data on the tables, they wanted the tables to be sortable so they can compere figures using any column of their choice. This particular request was most recurrent on the Gender by date of birth and death page. It was also suggested by a participant to add an about page to the website, made in the same CSS framework. This page could have information about how many women are being analyzed, a detailed methodology of how the data was analysed and finally a profile of the WIGI project participating members.

Other generalized recommendations or suggestions are

1. Human interpretation of what the number in the tables mean
2. Pose research questions for each chart, an excellent location would be in the graph titles
3. Provide a generic guide to understand any abnormal/unusual results that may be displayed in the chart and tables, eg -2 for Africa
4. Grey text on white background be replaced with black text on white background
5. Scale all graphs to proportion
6. Align numbers in the table to the right
7. Add left-hand nav for same page sections
8. Decrease font size of all X and Y-axis titles
9. Add a link between Top 10 and Bottom 10 that expands and shows all the rows of the table in between
10. Increase space between sections
11. Improve visibility of chart navigational features; change their location, add their text name beside them so one does not have to hover to know what they will do because chances are "I may not be bothered to use them" in the words of one participant.
12. Include a search box to; put together data related to a particular search term, or to direct users to which page that search term can be found
13. Add a loading spinner
14. On page load, a hover state is opened already
15. Turn alignment on in the charts
16. Change links color to pink and make them bottom aligned to WIFI
17. Make navigation bar color black
18. Data set downloads available for each chart

More specific recommended courses of action are specified under high and second priories for each page.
High Priority: Items in this section can significantly improve the usability of WIGI
Second Priority: These recommendations do not greatly enhance the usability of the WIGI for the Beta release. They can be considered if there is time, otherwise they should be reviewed prior to any future development.

Landing page

High Priority

1. Improve summery of background to make it clear; what WIGI is, and What WIGI is not
2. Pose research questions (without answering) to spark an interests in users to want to stay on the site to find answers
3. Mention briefly the methodology that was used

Second Priority

1. Include in the background summery; Why WIGI is useful
2. Add broad hypotheses (without answering) to get users thinking and interested in wanting to know more about the gender diversity in Wikipedia
3. Provide a proper attribution of image
4. Disable WIGI link or "only" considered active on the other pages
5. Consider revising title, suggestions include
   1. Wikipedia Indicators of Gender Inequality
   2. Wikidata Human Gender Indicator (WHGI)
6. Blurb changed to match http://blog.wikimedia.org/2015/05/01/meet-the-inspire-grantees/ "This public website is where we're gathering, automating, graphing and observing gender trends in Wikipedia’s biographical articles over time using Wikidata. You will also find open-data downloads of our dataset above. This project started as a personal research interest, and is now funded by a Wikimedia Foundation Grant"

Gender by Culture

High Priority

1. Include a column in both Recent changes and All time tables showing percentage of females
2. Remove the e+ mathematical notation on the Y-axis and replace them with wholes numbers
3. Fix bug: Instead of ten (10) cultures, the tables display only nine (9).

Second Priority

1. Provide generic explanations for possible outlier values that may pop up, eg negative numbers in the case for Africa
2. Exclude or concisely explain use of the term Mechanical Turk
3. Add a column to number lists on the table

Gender by Country

High Priority

1. Either change decimals to percentage, or specify boldly how they can be read as percentages
2. Define the different color gradients
3. Fix bug: Recent changes show only two (countries) instead of ten (10).

Second Priority

1. Include an option that can allow users see the map (by color categorization) from the perspective of different the cultures. One user stated "that would help me easily make connections"
2. Include a bold direction in the introductory text directing users to hover on map
3. Add a column to number lists on the table

Gender by Date of Birth

High Priority

1. Help users know they can hover, zoom in and reset chart by including a bold direction in the introductory text
2. Add a clickable feature for users to be able to concatenate and view years by their choice, eg. by decades, centuries, 400yrs, etc

Second Priority

1. Add a hover line-axis for ease of interpretation of chart
2. Set default recent change to be consistent with calendar days, ie 30 days
3. Add a column to number lists on the table
4. Consider reducing number of columns in the tables

Gender by Language

High Priority

1. Include a fifth in table to display actual number of female biographies besides the percentages.
2. Add a fourth information, ie an actual figure besides the percentage in the visualization as well
3. Allow users to choose how many languages they want to visualize
4. Exclude mathematical notation from y plane
5. Add percentage sign to x plane values
6. Add drop down list of entire list of languages
7. Make counts in table to begin from One and not Zero

Second Priority

1. Add a hover line-axis to aid visualization
2. Increase connection to other pages: identify common languages (eg European languages) with the same color

Data

High Priority

1. Make it obvious that Name, Last modified, Size and Description are sortable
2. Reduce length of pathway to files in the folders

Second Priority

1. A few users felt that the label content wasn’t written for a general audience.
2. Break huge files into smaller chunks
3. Make obvious the name of format of files contained in the folders. One of the study participants stated that "I think I may hit some code, I don't like code".
4. Include some help text in the Read Me file about how to interpret difficult parts of the charts/tables/data
5. Make it more user-friendly

Conclusion

Most of the participants had vague goals before visiting the site but left feeling accomplished that they had learnt something interesting, reiterating the recommendation that available research questions posed on each page might stimulate their interests even further. They found it somewhat easier to make meaning from the visualizations as compared to the tabulated data. Many indicated that the Date of Birth page was the most interesting to them, next was the Gender by language visualizations. Participants were however less excited about the data-sets available for download. 40% said they spend more time trying to understand the charts than consuming the statistics. Aside a few technical terms and mathematical notations participants found WIGI to be well organised, legible and easily comprehensible. 77% indicated that they enjoyed reading its content. Participants combined user experience rating on Fulfillment, Enjoyment, Positive Emotions and overall website usefulness was above average at 56%. Implementing the recommendations and continuing to work with Wikimedia community members will ensure a continued user-centered website.

Appendix

The raw data .xlsx feedback files are located here: https://drive.google.com/file/d/0BdjCIl911eXacS1HSk01dTBQZTg/view?usp=sharin%20g

Basic stats analysis are located here: https://drive.google.com/file/d/0B0jCsl911eXaejR1ak9pRzd0QTA/view?usp=sharing