In some embodiments, before a page is made exclusive, an invalidation operation is performed to invalidate all other existing copies of the page. This can be used to guarantee evaluation order in existing architectures. The invalidation operation can be performed by sending out messages to all other nodes, requesting that they invalidate their copy of the page. When responses (e.g., acknowledgments) to those requests are received, the receipt of all those responses indicates that there are no other locations at which the page exists. The guest operating system can then start up again and write to the page. When the write is completed, other pages may want to have a copy of the page, and in some embodiments, a snapshot of the page can be taken and used to create new secondaries that have the updated information for the page. Thus, through the use of secondaries, it will appear to the guest operating system that the pages are local.
When a vcpu on a node m (m n) requests access to p from n, if that page is Prime or Exclusive, the page p currently on n is marked Invalid, and a copy of the page is then sent to m which marks p as Prime. In some embodiments, as an optimization, the vcpu on node m may mark the page p as Exclusive if the vcpu on node m knows that is what is needed.
In some embodiments, if a node has a page that is prime or exclusive, then when it receives a request to send prime or exclusive writes to that page it is converted on that node to a secondary. The right to write into that page is then transferred to the node that is requesting the page. This is an optimization that may be performed based on an assumption that the node would not be requesting the page unless the node were going to write to the page. This saves a transaction in the protocol having to be performed, increasing efficiency.