First Workshop

Hypotheses to Validate

The first workshop was organized in order to quickly test if the proposed methods helped participants to build conversations. Regarding the improv method, I was testing if participants acted out and iterating on their conversations, they gain a better understanding of the goals and purpose for their conversations. For the Bot Persona method, I was testing if participants filled out personas for their chatbots, they would quickly be able to collaboratively design personalities and identify the right kinds of conversations their chatbots should have. As for the conversation mapping method, I was testing if the mapping framework fit closely enough to participant’s mental models of how conversations can be organized and if, by mapping their dialogue, teams could build effective conversations. Lastly, myself and the team I was collaborating with were testing if participants could easily take their mapped dialogue and easily implement it into our specially formatted chatbot configuration file. Overall, I was looking to see if participants would feel confident designing conversations and that their resulting conversations achieved their conversational goals.

Context for the First Workshop

I recruited three colleagues of mine to test the workshop, all user experience designers with no experience designing conversations and little to no experience programing applications. All were interested in designing and building chatbots, though had no preconceived ideas on where to start. I structured the workshop to last one hour, covering instruction setting up computers with the necessary development environment, completing all of the methods, and testing their chatbots. This aggressive timeline was chosen so I could teach only the core concept of each method. By possibly under communicating, I could listen for what participants found confusing and learn how to teach the methods more effectively in future iterations.

Summary of Events

The workshop began with participants quickly ideating and deciding a character, context, and goal for their chatbots. This was to seed context for their conversations. Using their initial ideas, I asked teams to improv with each other conversations their chatbots may have, while one of their team members takes notes on what worked and what did not work. Next, participants were led on how to install the relevant software and development environment on their computers. At the end of this process, all of the participants had successfully activated their chatbots and were able to have an initial interaction conversation with their chatbots, pre programmed to respond to “Hello”. Afterwords, I introduced the participants to Bot Personas, asking the participants to collaboratively fill out blank worksheets and discuss the personality elements of their chatbots. Working from one of the completed Bot Personas, I illustrated conversation mapping. The participants then mapped one of their own conversations. Once done, participants were instructed how to translate conversation maps to the chatbot configuration files. Finally, participants could test out the conversations with their bots to see if the conversations worked as intended, or if they needed to make changes.

Observations and Feedback

Overall, the workshop achieved its aim of making people confident in designing conversations. At the end of the workshop, one participant exclaimed “This shit is so cool! I had no idea how cool this would be.” However, each of the methods faced fairly substantial critiques.

The improv method was primarily successful because the participants were already comfortable and trusting of each other. Participants were quickly able to assume the roles of their characters and explore possible conversations. Everyone attempted a wide variety of situations in a surprisingly short amount of time. Yet, because the participants were having so much fun, few if any of the attempted improvisational conversations related to the goals they gave to their bots. Similarly, none of the participants remembered to take notes on their experimentation. Only one seemed able to reflect critically on what happened. All of the participants still felt improv helped them empathize with their chatbots. Reflecting on the method, participants requested more explicit instruction how to improv and clarification on what the “user” is supposed to do.

The Bot Persona was the most successful method of the workshop. Participants, previously educated in the Cooper persona method, were easily understand how to fill the respective “Do, Think, and Feel” components. Everyone agreed how filling out the Bot Persona was a clear, concrete process to design a rich and complex personality. Participants had a little more difficulty determining good “Do” behaviors for their bots. To address this, I asked the participants to recall the goal they gave to their characters and to write the goal onto their Bot Personas. This helped provide more context and guidance on what to list as behaviors. An important element to the participants I did not consider was the importance in naming a bot. Participants spent an unexpectedly long time brainstorming names, and adding the names unprompted to the persona. Reflecting on Bot Personas, participants suggested that this method should precede the improv method, as it may provide a better reference point to seed experimentation.

Conversation mapping was mildly successful. Participants easily grasped the different dialogue components “Hear,” “Say,” and “Ask.” They also understood how they could map out their dialogue. However, we had to skip having the participants more fully map out their conversations. Our team was conscious to test the bot configuration file to see participants ability to program their bots and I did not allocate enough time in the workshop for everyone to set up their computers. Yet, the dialogue components and mapping was explained and the configuration file matched the components and mapping. Participants felt that they were sufficiently taught how to translate their intended conversations to a working chatbot. I am cautious to accept this as the conversations built were quite shallow – the most complex conversation written had only two ‘branches’. In these cases, as anticipated, I assume conversation mapping may feel unnecessary.

Reflection and Changes to the Methods

The results of the initial workshop were positive, the participants had a great time, and everyone felt there was promise for this format. I was worried our participants would become frustrated with the programing and would not be able to launch their bot. To the credit of my team members, the configuration file they created was incredibly intuitive, dramatically reducing the barrier to building a chatbot. With this logistical fear successfully addressed, I was better able to focus on adjusting the methods.

I felt that a longer workshop would provide better insight whether the methods could work. The shorter length of the first workshop illustrated that the methods could be completely fairly quickly, though little time was given to process their learnings. This workshop was built with experiential learning theory in mind (Itin, 1999; Kolb and Kolb, 2005), with time spent on doing activities teaching conversation design. A key element that follows the experience of an activity is reflection (Itin, 1999). I knew that, with such a short amount of time, participants had little time to reflect on their experience in groups. For the later workshops, I would need to reserve enough time for reflection.

After reflecting on my observations of the participants, feedback from the participants and my collaborators, and my own experience, I had a few ideas on how to iterate on the methods and the workshop design. Firstly, I changed the order of the the methods, beginning with the Bot Persona, then Improv. This could help provide more constraints and direction for the Improv method. Next, I would teach the Improv method by example, acting out a conversation for others. This could better help illustrate what participants needed to do. Finally, I would try to give much more time for the whole workshop in order to give more time for participants to reflect.

Bibliography

Itin, C.M. (1999) ‘Reasserting the philosophy of experiential education as a vehicle for change in the 21st century’, Journal of Experiential Education, 22(2), pp. 91–98. doi: 10.1177/105382599902200206.

Kolb, A.Y. and Kolb, D.A. (2005) ‘Learning styles and learning spaces: Enhancing experiential learning in higher education’, Academy of Management Learning & Education, 4(2), pp. 193–212. doi: 10.5465/amle.2005.17268566.