Understanding the work of designing and sustaining toolkits

Digital Discernment Lab

Date

Jan-Sept 2023

Role

User Interviews, Coding, Affinity Diagramming, Thematic Analysis, Academic Writing

Team

Ahmer Arif: Assistant Professor, Supervisor

Tamar Wilner: Post Doc Fellow

Akhil Adavi: PhD Student

Sreehana Mandava: Undergraduate Student

Tools

Miro, Atlas.ti, Zoom

Despite their crucial role in sharing expertise and influencing practices, the process of creating and maintaining toolkits remains poorly understood. How do designers know who’s using their toolkit, or whether they’re getting any benefit out of it? How do they make sure their toolkit stays relevant to users over time? This highlights the need to understand the work of creating toolkits from the perspective of toolkit designers.

Problem Statement

Have you ever wanted to learn a new skill set and turned to Google for guidance? If you have, chances are you encountered a “toolkit.” Originally toolkits were physical objects like first-aid kits, but now they are often digital collections of resources. Today, many toolkits are designed to help communities face pressing challenges at the intersection of technology and society, such as misinformation, online harassment and intrusive digital surveillance.

What are toolkits?

We investigated what it means to make a toolkit work in 2 phases

Stage 1: Interviewing toolkit designers

Stage 2: Thematic Analysis (Coding and Affinity diagramming)

First, we interviewed toolkit designers to understand their perspective on the work behind creating a toolkit.

  1. We conducted 20 Zoom interviews with practitioners who designed educational and activism-focused toolkits. Each interview lasted up to 75 minutes.

  2. Our interview protocol involved tracing participants’ experiences of working on specific toolkits as well as designing toolkits more generally.

  3. Our interview targeted three key aspects of toolkit design

    • How their projects began

    • How they collaborated with users

    • What happened once the toolkit was "released"

Post- interview, we thematically analyzed the transcripts

We undertook a process called “coding,” in which we identified common themes in the transcripts. Next, we grouped codes into categories using a digital collaborative platform called Miro, identifying broad themes in the dataset.

A screenshot of the resulting affinity diagram from our research on Miro

We uncovered that the toolkit design process is deeply nuanced in the real world leading to our two key findings

Building a community around a toolkit was a theme that surfaced time and time again throughout our research. Oftentimes, designers saw the users as community partners in building toolkits. This revelation led to the two main findings of our research.

Finding#1: Community building was a way to understand user needs

All participants conducted user consultations, helping them better understand user needs. For instance, the News and Media Literacy 101 toolkit, a set of lesson plans for various school levels, benefited from this approach. A designer explained how focus groups and UX research allowed their team to tailor lesson details. User feedback prompted them to create 15-minute versions of their 45-minute lessons. This example highlights the importance of community collaboration to meet user needs effectively.

“You listen to the teachers. They are saying, I have no time… Well, what are you gonna do? You’re gonna provide them what they can do [in that time].”

Finding#2: Community building was a way to keep a toolkit alive

Building a community around a toolkit had another benefit: keeping the toolkit relevant. Participants saw toolkits as needing regular updates to remain useful. Without maintenance, toolkits can become obsolete. Building a community around the toolkit helps keep it active by allowing users to share their experiences. For instance, the designer of the Library Workers’ Field Guide to Designing and Discovering Restorative Environments holds monthly meetups to discuss, use, and improve the toolkit.

“We discuss the activities [in the toolkit], do them together, iterate upon them, that sort of thing.”

This engagement helps refine the toolkit continually. Similarly, the designer of News and Media Literacy 101 conducted professional development activities with teachers, creating long-term relationships to gather feedback and improve materials.

Implications

  1. Our findings offer valuable insights for toolkit designers, emphasizing the importance of building and maintaining a strong user community to ensure toolkits meet evolving user needs and remain useful over time.

  2. This research lays the groundwork for further studies, providing scholars with a deeper understanding of the complex work involved in creating and sustaining effective toolkits.