In a typical operating system, an operating system runs on a bare metal machine. As described above, using the techniques described herein, the bare metal machine is replaced with a distributed hyper-kernel, which gives the operating system the impression that it is running on a bare metal piece of hardware. This is supported by second level page tables, which exist in processors on the nodes in the TidalPod. If there is not an entry in the second level page table for a page of memory, a fault will occur when performing hardware address translation. As described above, based, for example, on cost functions, the virtual processor can be moved to the page with the memory, or the memory can be moved to where the virtual processor resides. In some embodiments, when the memory is moved, not only are the contents of the memory copied over, but the second level page table at the destination (node) is also updated. Thus, when a process on the node attempts to access the page, another fault will not occur. This provides, to the guest operating system, the appearance of a single large address space, which does not exist in hardware, but is defined and supported in software using the techniques described herein. If the request for a page is rejected (e.g., based on a message that comes into the cost function), then the processor moves to the location of the page (or both the virtual process and the page are moved to another node—i.e., post-condition, they are co-resident).
In some embodiments, when a page is requested, the page is requested from its primary location. That page is then marked as secondary, while the page sent to the requesting node is marked as exclusive or prime on the requesting node.