‘Hi, can I speak with UNHCR?’
Mobile messaging apps are the fastest-growing digital communications phenomenon to date. Today, more than 2.5 billion people around the world use messaging apps, a figure that is expected to rise to 3.6 billion by 2018 – that’s almost half of the world’s population. The world is concurrently seeing unprecedented numbers of people forcibly displaced from their homes, people who are often separated from the families and communities they once had close to them.
For UNHCR, this is changing the dynamic of how we’re engaging with refugees. With a presence in over 140 countries, UNHCR plays a critical role in supporting and facilitating community building -traditionally engaging with refugees face-to-face on a daily basis. Given this technological and social shift, humanitarian responders, including UNHCR, are realising that using multiple channels – including new technologies – are part and parcel of how we need to approach to communicating with communities moving forward. Through engagement with refugees via digital platforms, humanitarian responders can provide not only relaying critical lifesaving information to refugees, but also establish a dialogue in which refugees can provide their insights, feedback and priorities. In turn, this will result in more effective, better-targeted humanitarian programming.
Often the scale of resources required to adequately deal with, address and refer issues raised through these platforms has limited the level of use, or priority they have been given. Additionally, each request’s own uniqueness, complexity and sensitivity requires that UNHCR invests adequate time to deal with it appropriately – people are looking for more personalised service provision. Nonetheless, the humanitarian community is beginning to realise that as technology is advancing, they represent an opportunity to engage at scale, ensure that data is adequately captured, securely stored and shared with front-line staff, who are currently wading through ad-hoc unstructured requests for support.
One example of this is the I am a Syrian in Lebanon Facebook platform which is a large Facebook group run by volunteers who inform and support the group members. These volunteers have a direct support line to UNHCR staff in Lebanon to ensure they are able to provide up-to-date information and that any serious concerns can be raised to UNHCR or authorities.
Lines are open between 08:00 and 16:00.
The advent of artificial intelligence presents an opportunity. The capacity for technology to navigate human speech and text has evolved to such an extent that it is become ever more possible and plausible to create dialogue and understanding to the level where – at its most powerful – users cannot discern between a human and a machine.
Practically speaking though, the artificial intelligence used in commercial chatbots has a long way to go before it can replace all human aspects of dialogue, or create a seamless interaction with a user. For example research by the International Committee of the Red Cross in their Report Messaging Apps for Humanitarian Futures highlights the potential opportunities and risks of using messaging apps and chatbots in humanitarian response but is relatively muted about the potential of chatbots:
“Despite mixed reviews for the first generation of messaging-app bots, which users often described as glitchy or limited in their ability to understand users’ intent, other major messaging apps are expected to support similar technology by the end of 2016. For the foreseeable future, the landscape is likely to be dominated by simple, functional bots that stick reliably to a pre-defined script.”
And there are others that are claiming that Chatbots are not going to be the next big thing. Nonetheless, we are also seeing reports that chatbots will help businesses save billions in the near future through their automation and efficiency gains vis-a-vis customer service.
Whichever side of the fence you might be on, there remains an opportunity in seeing how artificial intelligence can operate in place of a human to undertake certain simple tasks. Through the effective use of a chatbot, it may be possible to triage and refer requests, saving the human time required to deal with each and every query. In the face of the aforementioned challenges, combined with an increasingly challenging funding environment for humanitarian organisations, chatbots have become an avenue for scale and efficiency in ways difficult to establish with traditional channels of communication.
I didn’t understand your query. Please try again.
Given this context, UNHCR Innovation Service wanted to start experimenting with chatbots to see whether they were a suitable tool for UNHCR operations. UNHCR has had a strong level of collaboration with Facebook on numerous projects throughout the years and Facebook was eager to see how UNHCR could start experimenting with chatbots.
Facebook funded a small pilot test to see what could be established and whether it might be something that can support UNHCR meet the information needs of refugees. Facebook and a sub-contractor worked with UNHCR Innovation Service to determine how to kick this off within UNHCR.
The UNHCR office in Jordan has been operating a call centre for a number of years. Through brokering a partnership with Mobile Network Operator ZAIN, the call centre has become one of the most well-utilised channels of communication between refugees in Jordan and UNHCR. With 14 agents on staff, the centre handles thousands of calls a day. Yet due to the cost, and increased popularity of the hotline, the call centre is seeking new ways to make efficiencies in their operation. Currently, the centre is on the cusp of a technological advancement linking the Interactive Voice Response system that triages call with the refugee assistance database. This allows for simple call requests such as regarding the office opening/consultation times.
Given the conducive environment created by the operation already having protocols for keeping information up-to-date for the call centre staff, UNHCR Innovation Service approached the Regional Office and Branch Office in Jordan and they illustrated their interest in being part of the experiment. Subsequently, UNHCR defined user stories (working out what different users would like to achieve with the tool), high-level requirements and then worked with Facebook and the subcontractors to start developing an experimental bot. We utilised call centre scripts to help develop the content and linked in with different UNHCR staff from technology roles to community-based protection staff to ensure cohesiveness with the ongoing call-centre operations. Further, into the development process UNHCR undertook some scenario-based user testing of the bot from both an end-user and system administrator perspective.
A screenshot of the chatbot UNHCR tested with Facebook.
Simultaneously we carried on doing research as to how other agencies were experimenting with bots. The Mobile Vulnerability Assessment Mapping (MVAM) team at WFP have also been leading the charge with Chatbot development, piloting and testing approaches and sharing their lessons following a hackathon on the topic. They recently released the first iteration of their WFP MVAM Chatbot. We were also part of a hackathon in which many of the participants were utilising chatbots for numerous purposes from triaging requests relating to food security to supporting a radio station
Ultimately, this first experiment will not be rolled out into our operation. Our testing of the bot uncovered too many issues with the bot. We realised we needed to bring some of the development in-house to ensure that it was truly built to our unique and rather complex context, rather than pay for third-party contractors. There were also a number of legal and data protection grounds for this decision. Aside from these technical aspects, there were a number of core lessons learned that are going to be vital for us to take into account for our next chatbot iteration.
Your satisfaction is important to us. Please let us know your feedback so we can improve our services.
- Build from existing efforts
Going through this project it was clear that the opportunity provided by chatbots really resonates with those who are undertaking similar initiatives through more conventional channels. For instance with the call centre in Jordan, the process of engaging with remote queries was a day-to-day reality for the team. This team was also inundated with simple queries that could be automated and the team were already taking technical steps to create the IVR system to reduce the burden on the staff. From even mentioning the goals around using a chatbot, it was immediately clear to the team about the benefits it would bring as it touched upon their daily concerns and frustrations. Leveraging these opportunities and helping solve our colleague’s day-to-day problems goes a long way in making a chatbot become a reality. As soon as a project moves away from this, it is hard to ground it and it is possible that it could lose its way or fail to take off.
- Lead from the field
In synergy with this, the ideas and impetus needs to come from the field. Trying to cookie-cutter a nascent technology onto an operation simply doesn’t work. While our approach meant working closely with some staff on the ground, we didn’t identify in advance how the project could fit into their work plan, within their budgets. Space in their schedules and finances to support a project like this would be needed to bring it to fruition. Since the first experiment, UNHCR Innovation Service has travelled to Jordan and discussed the options and opportunities with the teams on the ground. This has helped bring greater clarity for all involved – including a better understanding of roadblocks and speed bumps from the operational side, and how UNHCR Innovation Service can most adequately support further endeavours.
- Designing for Users
One of the main lessons we learnt through our experiment was around chatbot architecture and the importance of designing for the user. Preconceptions around ‘starting simple’ led us to utilise an approach that was based on Frequently Asked Questions (FAQs). While this made sense on paper, it was a preconception that we hadn’t challenged with end users. We actually ended up determining our user stories at a slightly later stage of the process and recognised that many of the aspects important for the end users wouldn’t necessarily be covered by the FAQ approach, and that maybe a slightly different – perhaps a flow/decision tree approach would have been more suitable at meeting user needs than an FAQ approach. While we only tested one iteration of the FAQ approach, it was clear from the feedback from both end users and administrators during user testing that a lot of aspects such as the raising of red flags, and confidential referral and transfer were a high priority – something perhaps unique to UNHCR’s role in ensuring the forcibly displaced are protected. We also didn’t test the limits of how AI, and in particular Natural Language Processing could support and enhance some of these aspects. While measures need to be put in place to ensure the bot doesn’t turn against us, we felt like these aspects could very much improve the experience when built from the right base.
- Recognise where the intervention adds value
One other issue which became clear from our user testing was that an FAQ approach added little value over what could be provided through a simple website. User stories are fine, but often we might end up trying to shoehorn a solution or product to fit these or explore ways they can. When looking back at these stories, it became clear that maybe a website would have been a more comfortable fit than a chatbot for fulfilling them. This came across in user testing as well: people wanted personality, they wanted answers based on input driven by the user; they felt an FAQ simply replicated what a web portal could provide. From this testing, we gathered a lot of data about what users expect out of a chatbot and these different aspects will be a priority in further iterations. We will try and focus on where chatbots are adding value, looking at aspects of personality and how the dialogue can be used to really try and customise information.
- Agility
Moving forward UNHCR Innovation Service will be looking to iterate and refine our pilots to help them achieve their potential. We’ve learnt a lot by going through this experiment and are constantly learning more from Facebook, WFP and many other as to how we can recognise the potential of the tools and avoid the pitfalls that may lay ahead by constantly reflecting on what users want and adapting to their needs and feedback. We are opting to take a more “DIY” approach, augmented by external expertise, to give us the agility and flexibility we need to pivot when users – whether refugees or our call centre staff – are facing issues with what is being developed.
Thank you for contacting the United Nations Refugee Agency.
UNHCR is going to continue scoping how chatbots can support our operations in providing more timely up-to-date information to refugees and increase efficiencies in our ability to serve our constituents in a timely manner. In Lebanon, UNHCR is leading on a new initiative – the Humanitarian Innovation Lab, which is committed to ensuring that refugees can access better information and receive better services through social media platforms. We plan to iterate on our experiments in this context and work with Facebook and other partners to maximise the potential of the Humanitarian Innovation Lab initiative.
We’re always looking for great stories, ideas, and opinions on innovations that are led by or create impact for refugees. If you have one to share with us send us an email at innovation@unhcr.org
If you’d like to repost this article on your website, please see our reposting policy.