Implementations of the present disclosure are directed to systems, devices, methods, and computer-readable media for passively authenticating a user based on speech data collected from a user during a conversation with a conversational user interface (CUI). A computing device, such as a personal assistant (PA) device, may interact with a user through a session that involves speech interactions. For example, the user may ask the CUI a question (e.g., “what is my checking account balance?”). The PA device can request relevant information from a remote server or, in some instances, retrieve the relevant information from local storage, and the CUI can provide speech output that answers the question (e.g., “two hundred seventeen dollars and 57 cents”). In this way, the user and the CUI may engage in a speech interaction (e.g., a conversation) in which the user requests information and the CUI responds appropriately if it is able to provide the requested information. In some instances, the PA device is a stand-alone computing device with components to receive speech input, provide speech output, process information, retrieve stored information, and/or communicate with other computing device(s) over one or more wired and/or wireless networks. For example, the PA device may be an Amazon Echo?.