Commonwealth’s Emerging Tech for All (ETA) initiative aims to integrate the needs and aspirations of people living on low to moderate incomes (LMI) into the next generation of emerging technologies. We partner with financial organizations, fintechs, and platform providers implementing new technologies, research the needs and experiences of customers living on LMI, and conduct pilots to produce actionable insights for providers to meet those needs.
As part of the Emerging Tech for All initiative, Commonwealth launched the Financial AI for Good: Guide & Chatbot to include new insights from our 2023 national survey report, Generative AI and Emerging Technology: Actionable Guidance for Financial Service Providers. The Guide & Chatbot also includes results from new field tests with partners boost.ai and the Michigan State University Federal Credit Union, and Posh and the Associated Credit Union of Texas.
With the release of the Guide & Chatbot, Commonwealth launched a custom ChatGPT-based chatbot to make insights discovery easier for practitioners. Adding this chatbot to the site allowed the Commonwealth team to gain more experience with ChatGPT’s technology, test research learnings from the Emerging Technology for All Initiative on our site, and provide an interactive example of how chatbots operate.
Below, we share some findings from our experience creating and deploying the custom chatbot. In this post, the Guide refers to the written content on the page, and the Chatbot refers to the experience and technology of our interactive chat tool.
About the Commonwealth Chatbot
Commonwealth’s Chatbot was built with OpenAI’s ChatGPT 3.5 platform and uses exclusive source material and training data from Commonwealth’s website, research, and reports related to the Emerging Tech for All Initiative. We opted to use ChatGPT for two reasons: 1) its ability to generate new responses rapidly and 2) its prominence and track record in the market.
We created a custom set of source data from our research and uploaded it to the OpenAI platform; the chatbot references only this source data when answering prompts. In addition to having an open prompt field for users, we created and embedded commonly asked questions as prompts throughout the Guide’s content to drive engagement.
The updated Guide & Chatbot combines the latest research from Commonwealth’s Emerging Tech for All project. It provides a complete set of actionable guidance incorporating how people living on LMI view new technologies and what they need most. It builds on an earlier version of an actionable guidance publication, the Tech for Good Toolkit, that had included findings from a 2021 national survey, field tests, and in-depth interviews.
Launching a Chatbot: Initial Learnings
The Chatbot’s strength is its ability to provide quick, easily readable summaries. Users can quickly find answers to specific questions by asking our Chatbot rather than having to read through the entire Guide text or the accompanying reports. We also directed our Chatbot to provide brief answers in bullet-point format to increase readability. Whether users want to search for a particular data point quickly or want to summarize an entire section of research, our Chatbot can deliver relevant information quickly in ways that are easy to engage with and understand.
Establishing the context for use cases and keywords provides more accurate responses. One of ChatGPT’s key benefits is its pre-trained models, which allow it to generate human-sounding, logical responses. However, this training data also initially allowed the Chatbot to give responses beyond the scope of the Emerging Technology for All research.
For example, this discrepancy was noticeable in our report’s usage of the word “accessibility.” In our research, accessibility represents the challenges people living on LMI may face when accessing or opening financial service accounts; those barriers can include language, geography, or technology. Still, the Chatbot’s underlying models led it to attempt to answer questions related to accessibility with information on building for customers with vision or hearing impairment. While these are also important needs to consider, the Chatbot’s recommendations were still sourced from its general large language models and not from the research provided by Commonwealth. To better direct the Chatbot’s responses, we created additional custom training data questions to align the Chatbot’s answers with our research.
Even with closed data sources, the Chatbot needed an additional training period to ensure accurate boundaries for responses. The Financial AI for Good: Guide & Chatbot relies on the Commonwealth website, text for the Guide, the 2023 national survey report, and two research field tests (see here and here) from the Emerging Tech for All Initiative. We also collaborated with internal stakeholders to create an additional document with custom questions and answers to prudently expand the scope of possible responses and, importantly, help our Chatbot continue to learn while still allowing for new questions to be answered. Adding this FAQ-style document with additional potential questions and responses to the training source data greatly improved the reliability of the answers.
Sample prompts drove Chatbot engagement. We created sample Chatbot prompts to nurture first-touch engagement with users using the technology. By simply clicking the pre-written prompt (denoted with a special design) on the page, a user could generate a Chatbot-provided answer to quickly surface summary-style answers. Usage data from the Chatbot showed the first prompt provided in the Guide (“What are the top 3 things I can do to improve our chatbot?”) along with the two example prompts the Chatbot itself provided, which were the most asked questions by site visitors. This supports our insight that financial institutions can drive engagement with their chatbots by providing examples of potential interactions with users.
Organizational Takeaways
The Financial AI for Good: Guide & Chatbot is Commonwealth’s first attempt to provide a more interactive way to navigate specific reports while also giving visitors a real-world example of how a chatbot works. We believe this example offers further clarity on how a financial institution or organization could implement technologies like ChatGPT while putting user trust and accessibility at the center. The lessons learned from our implementation process could benefit financial institutions or providers looking to take a first step into providing a chatbot themselves:
- Choose a specific pilot project to familiarize yourself with chatbot technology. In launching this project, our Commonwealth team had to learn more about writing prompts for ChatGPT, evaluating the quality of responses, and understanding where responses may be coming from. Launching any chatbot in a constrained, project-specific way can help work out potential problems before scaling it to additional research projects.
- Integrate website language that lets users know they are interacting with a chatbot. In Commonwealth’s field test research, users said they wanted clarity on whether they were interacting with a bot or a human and what the bot could and could not accomplish. Setting expectations and providing language on the website that clearly states when there is a new version of a chatbot can build trust with users, especially if they encounter new bugs or challenges. In the Guide & Chatbot, we made the experience explicit with language for “How to Use this Guide & Chatbot,” which included a description of our Chatbot’s source material.
- Seek stakeholder collaboration and agreement on criteria and scope for ideal chatbot responses. Taking the time to write and review responses to frequently asked questions can increase the reliability of the chatbot and ensure that users have a positive experience. This practice also helped us to generate a database of acceptable responses to check against our Chatbot’s output, and allowed for greater and faster quality control.
Conclusion and What’s Next
The growing use of chatbots in financial services presents an opportunity to provide more people living on LMI with access to financial help and information. We are enthusiastic about further research into generative AI and its applications for financial institutions.
We also see the potential in expanding access to our own research at Commonwealth. As we continue to grow our understanding of chatbot design principles, we plan to expand our site’s chatbot across more research.
Our generative AI research, as well as our own experience of building with the technology firsthand, have underscored just how it can impact user trust. With insights like the ones in our Guide & Chatbot and a thoughtful implementation plan, financial institutions can ensure these technologies ultimately support users’ well-being.
Listen to Commonwealth’s Emerging Tech & Ideas podcast episode with Ben Maxim, MSUFCU’s Chief Technology Officer, on how the credit union utilized AI tools with its customer service team.