Utah Online Dispute Resolution Platform: A Usability Evaluation and Report

Overview

This report includes findings and recommendations from usability testing of the State of Utah’s online dispute resolution (ODR) platform, which was conducted by the Innovation for Justice (i4J) Program at the University of Arizona James E. Rogers College of Law. Utah’s ODR platform is a web-based alternative dispute resolution tool that provides parties in small claims debt collection actions with an opportunity to resolve their cases online. The i4J Program partnered with the Utah Administrative Office of the Courts and the Pew Charitable Trusts to conduct observation-based usability testing and to identify how the Utah ODR platform could be improved or enhanced, with a focus on functionality and usability.

Online dispute resolution (ODR) is a method of alternative dispute resolution that aims to resolve civil disputes using court-annexed, internet-based platforms rather than in-person hearings, mediation, or arbitration. ODR is one emerging access to justice strategy, intended to expand the public’s access to their civil legal system through reducing cost and transportation barriers associated with travel to a physical courthouse. Although ODR usage is rapidly expanding across the U.S. and nationally, no research has explored whether the litigants for whom ODR is intended are able to successfully and satisfactorily navigate ODR platforms. This is troubling, because the traditional civil legal system already has a usability problem: the 70% of debt collection defendants who do not respond to their debt collection lawsuits are essentially would-be court users who view court as a product they are not interested in consuming. 

This study used what is widely considered the most effective usability evaluation method -- observation-based usability testing -- as well as participatory action research strategies. When applied to online dispute resolution, these methods help ensure that court users are able to successfully navigate an online judicial system intended to provide them with digital access to their civil legal system. We executed a multi-phase testing process designed to engage the low-income community in the review and redesign of Utah’s ODR platform. All testing involved participants who were screened for demographic characteristics that aligned with national data on non-bank personal loan debtors. Testing included two rounds of observation-based usability testing and three design workshops with representative users to collect community input. This report’s analysis of the qualitative and quantitative data from this multi-phase testing process informs five actionable recommendations for improving the Utah ODR platform’s usability, as well as suggestions for future testing and development to create a more accessible and just online space that meets the needs of all users.

Five Actionable Recommendations

Our baseline test of the Utah ODR platform demonstrated that typical ODR users experienced significant difficulty making the transition from the affidavit and summons to the ODR website, and experienced visibility and accessibility issues once on the homepage. Our redesigned affidavit and summons aims to help participants more easily navigate the transition from paper to the online platform through clear information hierarchy that highlights key information needed for future steps and a QR code that connects to the ODR platform. We recommend a short and simple URL for the ODR platform that uses a familiar domain name and easy path, and is not case-sensitive. In addition, we recommend an official website name on the summons that is discoverable through a web search apart from typing a direct link. The ODR homepage should include the term “ODR” and the phrase “online dispute resolution” on the website homepage. Apply best practices in user interface design and accessible design on the ODR site by adopting a higher contrast color palette; applying a clear, consistent information hierarchy throughout the site; and avoiding text on images.

Sixty-four percent of Utah defendants never log in to the ODR platform, so it is essential that the registration and login process not present unnecessary hardships that inadvertently prevent potential users from successfully engaging with the court. Study participants struggled to navigate the registration and login process on the Utah ODR platform, which included many steps requiring repeated referencing of the paper summons and affidavit, and lacked sufficient error-prevention and recovery measures. Common issues included typing in a name or case number incorrectly, re-entering the password because the participants didn’t notice the system requirements, and failing to understand key terms, such as the distinction between plaintiff and defendant or business and individual. We recommend reducing the number of steps required for registration, giving users a progress bar, displaying system requirements for acceptable passwords, verification codes, and email addresses and providing error-prevention feedback to prevent users from attempting to submit incorrect information. In addition, “tooltips” that open lightboxes can provide visual instruction regarding the location of the requested information on the documents. 

The ability for users to upload, share, preview and sign documents is an essential element of the ODR process. This was one of the most difficult tasks for study participants because the process to upload documents in ODR was not intuitive. Many users struggled to simply locate files on their phone for upload, and could not identify the path to reviewing and signing a settlement agreement. One participant became so frustrated that they opted to leave before completion of the test. We recommend allowing users to share documents directly within the chat feature, simplifying the upload process by using familiar icons and text labels, and adding system status feedback to indicate that a document upload is in progress and has successfully completed. In addition, a simplified “Settlement Agreement Review” process would allow users to clearly review and confirm settlement details on the platform before previewing and signing the document. Users should also have the option of downloading and printing documents before and after signing. 

A majority of participants reported being frustrated with the UTAH ODR platform due to the difficulty of finding information about ODR, including information about how it worked, whether participation was mandatory, and how to contact someone for more assistance. Study participants expressed concern about the low perceived legitimacy of the affidavit, summons and ODR platform. In addition, the baseline required users to successfully complete four steps in order to navigate from the affidavit and summons to online information about how to use the ODR platform. We recommend structuring the information that defendants need to make informed decisions about engaging with ODR so that it is clear and easy to locate, and prominently featuring the State of Utah seal on the affidavit, summons, and homepage. A welcome message on the homepage could introduce the online dispute ODR platform and how it operates. Adding an “FAQ” button to the homepage and main navigation menu would provide users with a consistent method for finding help information at any time. We also recommend an overview guide for first-time visitors and a welcome overview video outlining how ODR works, with closed captions in several languages for accessibility and inclusion. 

Concerns about the lack of access to, or understanding of, legal information were expressed by participants again and again throughout the multi-phase testing process, and thus were shown to be an issue underlying many of the difficulties that participants faced with the ODR experience. We recommend ensuring that the information that defendants need to make informed decisions about their legal rights is clear and easy to locate on the affidavit, summons and throughout the ODR platform. Using a visual information hierarchy to clearly label and highlight defendant options, providing definitions for relevant legal terms used throughout the ODR platform, and adding a claim response tool that helps defendants craft and customize their responses are all strategies designed to connect users with substantive legal information. In addition, clarifying the role of the ODR Facilitator by explaining it in the overview guide, and ensuring that the ODR Facilitator begins each chat with an introductory message explaining their role and the participant’s options, can add further transparency to the online process.

Conclusion

Users are ready for online courts: when asked if they had the option, nearly every study participant said they would prefer to use a website over physically going to a courthouse to resolve a dispute. The challenge—and opportunity—now is to integrate human-centered design into the expansion of ODR so the technologies employed are useful and intuitive for all users. The findings from this research study demonstrate that ODR users want accessible and transparent information about how ODR works and what their legal rights and options are within an ODR platform. They also want a platform that is responsive, simple, and has the look and feel of other mobile-first online consumer spaces they use on a regular basis. With the national trend toward ODR accelerated by the unanticipated shift to remote courts in response to the COVID-19 pandemic, it is more important than ever that courts commit to creating fair, just, and equitable online legal environments that meet the needs of all users.

Read the Full Report