Techniques are described for passive authentication based at least partly on collected voice data of a user. During a speech interaction between a user and a personal assistant (PA) device, the user's speech may be analyzed to authenticate the user. The authentication of the user may be a passive authentication, in which the user is not explicitly asked to provide authenticating credentials. Instead, the speech data of the user is collected during the user's interactions with the PA device, and the collected speech data is compared to a previously developed model of the user's speech. The user is successfully authenticated based on determining that there is sufficient correspondence between the collected speech data and the model of the user's speech. After the user is authenticated passively during the conversation, they may be able to access sensitive data or services that may not be otherwise inaccessible without authentication.