In one embodiment, to migrate a virtual processor, the hyper-kernel takes a snapshot of the state of the processor (e.g., a continuation, using 6400 bytes of data, or any other amount as appropriate) and sends it in a message over the dedicated interconnect (e.g., Ethernet) to the chosen destination, where the suspended virtual processor may be restored onto another physical processor (e.g., implemented as a hyperthread of a processor core) at the destination node. Saving and restoring processor state may be implemented using mechanisms provided for processors supporting virtualization. Since the program counter has not advanced, the instruction is then restarted. Since the page and the virtual processor are now co-resident, the virtual processor continues running. It is possible that in some cases the instruction generates additional interrupts to access different non-resident pages, but the mechanism that is used may be the same. When the virtual processor migrates, its updated location is recorded (e.g., in the resource maps described above). However, for reliability, perfect location knowledge is not assumed, as the virtual processor may have subsequently re-migrated.
In the following example of resource migration, suppose an enterprise supercomputer holds a large in-memory database, larger than can fit into a single node. Part of the database is in a first node, “node1.” Suppose one of the cores on a different node, “node2,” is trying to access data that is owned by node1 and that does not reside locally in a cache on node2. The core on node2 will receive a memory access violation because it is trying to access data that it believes it should be able to access (but cannot). As will be described in more detail below, the exception is handled in the hyper-kernel.