In accordance with an embodiment of the present invention, a capacitor based resistive processing unit with symmetrical weight updating is described. The resistive processing unit includes a first capacitor that stores a charge corresponding to a weight value. A readout circuit reads the charge stored in the first capacitor to apply a weight to an input value corresponding to an input signal using the weight value to produce an output. An update circuit updates the weight value stored in the first capacitor, including a second capacitor in communication with the first capacitor to transfer an amount of charge to or from the first capacitor according to an error of the output by changing a voltage difference across the first capacitor by a voltage change corresponding to the amount of charge, the voltage difference corresponding to the charge stored in the first capacitor. The error of the output can be calculated by machine learning algorithms such as stochastic gradient descend and back propagation.