The inventors recently developed a system that authenticates and/or identifies a user of an electronic device based on passive factors, which do not require conscious user actions. During operation of the system, in response to a trigger event, the system collects sensor data from one or more sensors in the electronic device, wherein the sensor data includes movement-related sensor data caused by movement of the portable electronic device while the portable electronic device is in control of the user. Next, the system extracts a feature vector from the sensor data, and analyzes the feature vector to authenticate and/or identify the user. During this process, the feature vector is analyzed using a model trained with sensor data previously obtained from the portable electronic device while the user was in control of the portable electronic device.