Method of defining and performing dynamic user-computer interaction, computer guided navigation, and application integration for any procedure, instructions, instructional manual, or fillable form
地域：
CA
CA San Francisco
摘要
Various embodiments described in this disclosure relate to methods and computer-based systems that implement those methods for overlaying or superimposing computer-user interaction widgets and application interface widgets on top of content blocks contained within digitized documents which represent written procedures or instructions, instructional manuals and fillable forms, in order to support means of providing dynamic user-computer interaction and computer guided navigation, as well as application functions and data integration, driven through structured interaction and integration metadata definitions, as it relates to said content blocks, during job performance.
說明書
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
This application claims the benefit of and priority to U.S. Provisional Patent Application Ser. No. 62/676,197 filed on May 24, 2018 and entitled “Method of Defining and Performing Dynamic Human-Computer Interaction, Computer Guided Navigation, And Application Integration for Any Procedure, Instructions, Instructional Manual, or Fillable Form,” which application is expressly incorporated herein by reference in its entirety.
Across any industry that requires a structured approach to job performance, many jobs performed by people are guided by written procedures and work instructions, which ensures reliable and consistent performance of tasks and minimizes the risk of user errors. Even instructional manuals for goods, requiring setup and maintenance, contain instructions that need to be followed by a person using such goods.
Both instructional manuals as well as written procedures/instructions direct the actions of a person performing the job in a defined sequence, while, optionally, providing other information and documents such as diagrams, drawings, illustrations, images, and fillable forms.
Instructional manuals, written procedures and work instructions are often presented to one who performs a job in the form of paper documents or an equivalent digitized document, such as a PDF, on a computer device.
權利要求
What is claimed is:1. In a computing system designed for having one or more computers configured to implement a method to define user-computer interaction and computer guided navigation for content blocks of a written procedure or instructions, instructional manual or fillable form, with definitions used at job performance time to provide said interaction and navigation, a method comprising:obtaining a digital version of a digitized document, the digitized document comprising a written procedure or instructions, an instructional manual or a fillable form; providing a set of typed dynamic graphical user interface interaction widgets on an interactive computer user interface, where each widget is used to encapsulate and contain a content block, being a logical unit of content or a collection of such, from the digitized document, and where each widget is designed to provide specific user-computer interactions and computer guided navigation at a job performance time; superimposing or overlaying, in the user interface, a first widget from said set of interaction widgets on top of or over a content block of said digitized document thereby encapsulating and containing said content block by capturing contents of the content block; defining an order of appearance and activation of each interaction widget relative to other widgets at said job performance time; configuring attributes of the widgets; saving resulting widget information, comprising information about the content block contained within each widget together with its attributes, on a computer-readable storage medium for execution or interpretation by one or more processors when work associated with an encapsulated content block of the digitized document is being performed by an entity; and wherein an interaction widget associated to a Step content block is placed inside an interaction widget associated to an Instruction section, as its child sub-widget, and linked with other widgets including other Step widgets, with each Step widget having a series of properties and sub-widgets to support one or more of the following:declaring the order of execution of Steps, at job performance time, as sequential, based on widget sequence numbers, or in an arbitrary order; indicating that completion of a step requires confirmation by the user and to provide a property or a sub-widget that may be placed inside a Step widget, for enforcing confirmation; a time-dependent step by automatically providing a timer starting from when a step was entered, at job performance time, and automatically enforcing a time dependency of a content block while providing the ability to pause and resume when applicable; indicating if the completion of a Step, at job performance time, requires a workers initials and providing a sub-widget for the initials that may be used, by placing it inside a Step widget; indicating if a Step is a Critical Step so that it may be rendered in a visually distinct way at job performance time; or indicating if a Step requires two users to be physically present together to implement concurrent verification before a step may be marked as completed, in which case an initial or signature sub-widget is added to a Step widget. 2. The method of claim 1 , further comprising:wherein providing a set of typed dynamic graphical user interface interaction widgets comprises providing a set of resizable interaction widgets corresponding to one or more of sections, subsections, steps, or any content block representing one or more logical units of the digitized document; receiving user input selecting the widget corresponding to a section, subsection, step, or content block of the digitized document, from the set of widgets; receiving user input placing the selected widget on the user interface and resizing and moving it to surround and to contain said section, subsection, step, or content block; adding to a widget data structure for the widget, at least one of the following: i) coordinates of the widget relative to the displayed digitized document such that widgets may be superimposed or overlaid over a copy of the displayed digitized document at job performance time, ii) a snagged image of a content block contained by the widget, such that at job performance time, content blocks contained by widgets can be displayed irrespective of the position of the widgets relative to a copy of the displayed digitized document and without using the digitized document in the background, or iii) contents being contained by the widget corresponding to the contained content block of the displayed digitized document, when the digitized document is of rich-content type, such that the widget data structure may be used later, to represent the corresponding content block of the digitized document in a different display form factor, for different computer devices without relying-on or displaying the underlying digitized document and by displaying the contents associated with the containing widgets based on the position of the containing widgets relative to each other and then overlaying or superimposing the widgets; setting attributes and properties of the widget as it relates to look and feel and interaction choices relevant to the contained content block; defining the order of appearance of widgets, in order to support computer-guided navigation of the widgets at job performance time, by providing means to sequence and link one widget to another, either statically or based on dynamically evaluated expressed conditions and adding such sequencing information to the widget data structures; saving, on a persistent computer data store, a copy of the displayed digitized document together with at least one of the following: i) the position of the overlaid or superimposed widgets relative to the displayed digitized document, ii) the snagged image of the content of the content block contained by each widget, iii) content of the content block contained by each widget, iv) look & feel properties of each widget, v) interaction properties of each widget, or vi) sequencing information of each widget. 3. The method of claim 1 , wherein the set of widgets or their user-computer interaction properties comprise titles or headings corresponding to at least one of: Purpose, Scope, Responsibilities, Precautions and Limitations, Prerequisites, Parts and Materials, Instructions, References, or Attachments with one or more keywords being associated to each title and the corresponding widget type;wherein when an Instruction section is included, a subset of widgets types are configured to associate to subsection titles under the Instructions section, including at least one of, Step, Action Step, Time Dependent Step, Continuous Action Step, Note, Caution, Warning, Conditional Action, Branch, Component Verification, Reference, Bulleted list, Figure, Table, Critical Step, Hold, or Inspection Point, with one or more keywords being associated to each title and corresponding widget type. 4. The method of claim 1 , further comprising displaying a dynamic Table of Contents (TOC) on a portion of a device display representing all the widgets superimposed or overlaid on top of the digitized document in the order defined by the widget relationships or links or sequence numbers and with each entry in the TOC uniquely representing a superimposed or overlaid widget in a manner that the TOC can be used to navigate to a particular widget on a display containing the digitized document. 5. The method of claim 1 , wherein each superimposed or overlaid widget is associated with one or more named and indexed views, through a widget attribute, such that at job performance time, when the user selects one of those views, only the content of the widgets associated to said view are displayed with a TOC containing only the widgets associated to said view with navigation to the content block of the associated widgets. 6. The method of claim 1 , wherein general configurable properties or sub-widgets contained within another widget are provided across all widgets to support specific interaction behavior at job performance time including at least one of:to support typed data entry interface widgets corresponding to basic data types, complex data entry widgets corresponding to tables, widgets for images, drawings, sound, video and other form media, such that all said widgets may be placed inside of interaction widgets, or other data entry interface widgets; to view feedback associated to any select content block of the digitized document contained by a widget, or feedback which were provided at job performance time by persons performing a job; to specify what percentage or how many units of work is associated with each widget relative to the overall work and for each sub-widget relative to the parent widget or the overall work such that at job performance time, the user is enabled to see what percentage of work associated to widgets and sub-widgets is performed in relation to the total work associated to a parent widget or overall work; to require acknowledgment at job performance time, through a sub-widget or property, that the content block contained within a widget is read and conformed with, before the person performing the procedure or instructions can continue to the next content block contained by the next widget; or to cause a widget to appear collapsed or expanded, by default, at job performance time. 7. The method of claim 1 , wherein a widget, including its content block and attributes except for the sequence designator attribute or an attribute used for sequencing, is added to a set of reusable widget templates, grouped by widget type, and a widget instantiated from a reusable widget template is linked with other widgets taking on its sequence attribute in the context of other linked widgets; and, where the reusable widget instance has child widgets, the child widgets' hierarchical sequence designators adapted to its parent widget's sequence attribute. 8. The method of claim 1 , further comprising, providing a set of procedure or instructions authoring interaction widgets having widget types that correspond to the widget types used for superimposing or overlaying interactions on top of existing content blocks, and with the same interaction capabilities as their corresponding types, wherein the procedure or instructions authoring interaction widgets are configured to be used for authoring or defining new content blocks and are configured to be linked to an existing widget. 9. The method of claim 3 , wherein a widget associated with a Conditional Action content block is configured, through at least one of properties or sub-widgets, to express one or more logical conditions based on dynamic input data, where all conditions are configured to be dynamically evaluated at job performance time and result in automatic navigation to widgets associated with different sections and content blocks of the procedure or instructions, with configuration defining one or more of:multiple conditions linked through logical operators; each condition to define the next widget to navigate to when the condition evaluates to true or a different next widget when it evaluates to false; declaring the widget as a Continuous Action, when the expressed condition must follow the form: if at any time [condition], then [Action]. 10. The method of claim 3 , further comprising one or more of the following:a widget associated with a Branch content block is configured to cause a transition to a procedure or instructions step other than a next step in a current procedure or instructions or to another digitized document containing other procedure or instructions without returning to current sections of the digitized document with the widget associated with the Branch content block, at job performance time; a widget associated with a Reference content block is configured to cause a transition to perform actions from another document or a transition to a section in another set of written procedures or instructions and then return to a current digitized document to complete a task or activity, at job performance time; a widget associated with a Component Verification content block is configured to provide interaction means for integration to equipment sensors, barcodes, RFID or similar technologies, or the ability to scan and analyze a component or a tool's tag or ID for purposes of verifying that a correct equipment or tool is being worked-on or used for a procedure or instructions at hand; a widget associated with a Signature or Initials content block is configured to provide means for a user to electronically sign a corresponding section; a widget associated with a Figure content block is configured to include a figure and may be configured to display the figure or to collapse a section containing the figure such that it can be expanded on demand by a user, at job performance time; a widget associated with a Table content block is configured to include sub-widgets designed for data-entry or dynamic data population of a section of the procedure or instructions at job performance time; a widget associated with a Hold Point content block is configured to enforce and support obtaining a signature of another qualified user for verification without requiring a verifier to be present, at job performance time to mark the step as complete; a widget associated with a Bulleted list content block is configured to provide interaction means to perform a contained bulleted instructions in any order while keeping track with a current bullet, having marked bullets that were already performed and enforcing performance of all bullets; or a General Content widget, with configurable interaction capabilities, is associated with a content block of the digitized document to capture the content of an associated section and to provide relevant interaction through configuration of its attributes. 11. The method of claim 3 , further comprising:a Repeat interaction widget is associated with a portion of instructions requiring repeating a series of steps based on dynamically evaluated conditions or a fixed number of iterations where said Repeat widget automatically provides all required navigation, computation, branching and placekeeping interactions as well as a clear view of all iterations performed; a widget associated with a Step is configured to contain nested sections and Steps and their corresponding widgets. 12. The method of claim 1 , further comprising:programmatically reading the digitized document into computer memory; programmatically analyzing the contents of the digitized document; programmatically matching content blocks with associated widgets; programmatically superimposing or overlaying interaction widgets associated with sections and content blocks of the digitized document; programmatically sequencing the widgets; programmatically inferring other attributes; and programmatically saving the resulting structured metadata such that the structured metadata can be displayed for further interaction design or validation. 13. In a computing system designed for having one or more computers configured to implement a method to dynamically integrate data or cause the dispatch of software functions, from external software applications, in the context of select areas of the content blocks of digitized written procedure or instructions, an instructional manual or a fillable form, or within widgets containing said content blocks, a method comprising:obtaining a digital version of a digitized document, the digitized document comprising a written procedure or instructions, instructional manual or fillable form; implicitly or through explicit configuration, associating each possible type of input or output data of software services to a typed interface widget; providing a user interface element in a user interface for the selection of a specific input or output of a software service available where said software service may be used as a means to pass data or bring data or dispatch a function to or from an external application; providing a user interface element in the user interface to superimpose or overlay the widget associated to said input or output on top of an area of a content block of said digitized document or within other widgets that are designed to encapsulate and contain such content block by capturing contents of the content block; providing a user interface element in the user interface to define the order of appearance and activation of each widget relative to other widgets at job performance time; providing a user interface element in the user interface to configure the attributes of the widgets; providing a user interface element in the user interface to save the resulting widget information, including at least one of: information about a content block contained within each widget, a widgets position relative to an associated content block or relative to a containing widget, or a widgets attributes, on a computer-readable storage medium, for execution or interpretation by one or more hardware processors to automatically gather dynamic data for the inputs of the software service through its associated interface widget superimposed or overlaid on selected areas of a said content block, to automatically invoke the service, and to populate widgets associated with the service outputs with output data of the service effectively over the selected areas of a said content block of the digitized document at job performance time; and providing a user interface element to place an interaction widget associated to a Step content block inside an interaction widget associated to an Instruction section, as its child sub-widget, and linked with other widgets including other Step widgets, with each Step widget having a series of properties and sub-widgets to support one or more of the following:declaring the order of execution of Steps, at job performance time, as sequential, based on widget sequence numbers, or in an arbitrary order; indicating that completion of a step requires confirmation by the user and to provide a property or a sub-widget that may be placed inside a Step widget, for enforcing confirmation; a time-dependent step by automatically providing a timer starting from when a step was entered, at job performance time, and automatically enforcing a time dependency of a content block while providing the ability to pause and resume when applicable; indicating if the completion of a Step, at job performance time, requires a workers initials and providing a sub-widget for the initials that may be used, by placing it inside a Step widget; indicating if a Step is a Critical Step so that it may be rendered in a visually distinct way at job performance time; or indicating if a Step requires two users to be physically present together to implement concurrent verification before a step may be marked as completed, in which case an initial or signature sub-widget is added to a Step widget. 14. The method of claim 13 , further comprising:displaying the digitized document in a graphical user interface; providing access to graphical user interface input or output definitions of a software service for describing the inputs or outputs of a service including data types, optional defaults, or restrictions for each input or output field; providing user interface elements to allow a user to select a service input or output field and then automatically display a data widget implicitly, or explicitly through configuration, associated with said input or output field and allowing the user to place the widget on an area of the display over a content block of the digitized document being displayed, or over another widget containing said content block, and accommodating the resizing and relocation of the widget by the user to cause the widget to either overlay or to superimpose over the selected said content block or its containing widget, for capturing dynamic user input data to be automatically mapped to the inputs of an associated software service, or for populating a specific area of said content block with dynamic application data from a service output, at job performance time, after the automatic execution of the associated service; when an interface widget needs to occupy space that is larger than space provided by boundaries of a contained content type, modifying the digitized document to add the required space, extending said widget to occupy the added space, readjusting coordinates of other widgets that are in a location of the display below the extended widget to compensate for the added space and displaying the modified version instead of the original digitized document on the display; providing a user interface element to allow the user to place an interface widget inside of another widget, where the other widget may be used in order to contain or capture the contents of a content block of the digitized document; adding to each interface widget metadata including coordinates of the widget relative to the displayed digitized document such that said widget may be superimposed or overlaid on top of the digitized document at job performance time; when an interface widget is contained in another widget, adding to widget metadata the coordinates of the interface widget relative to the containing widget, such that at job performance time, the containing widget and its captured content block may be displayed first and then the interface widget can be displayed relative to the containing widget irrespective of the position of said interface widget relative to the digitized document; providing a user interface element to allow the user to set the look & feel properties or change the associated default type of the widget to a different widget; storing and referencing within an interface widget, user defined constants with their default values; defining the order of appearance, in order to support computer-guided navigation of the widgets at job performance time, by providing means to sequence and link one widget to another, either statically or based on dynamically evaluated expressed conditions and adding such sequencing information to the widget data structures; saving, on a persistent computer data store each widget's look & feel properties, other user attributes as well as similar metadata of contained and containing widgets and other metadata associated with the widgets, including: i) a copy of the displayed digitized document together with the position of overlaid or superimposed widgets relative to said digitized document or relative to its containing widget, ii) content of content blocks contained by each containing widget, in a structured format, for interpretation at job performance time. 15. The method of claim 14 , wherein one or more of the following interface widget types is provided:a Data Interface widget that is not tied to the inputs or outputs of software services and is used to collect input data dynamically and without invoking a service at job performance time and allow a user and the external application to reference dynamic data within other widgets in order to support conditional navigation and to perform computations; an Equipment interface widget that integrates events and data from specific loT enabled equipment or to cause such equipment to perform a specific action at job performance time; a configurable Notification interface widget used to cause the automatic publication of a notification based on the evaluation of dynamic data to a person performing a job or to other applications through automatic dispatch of a software service associated to said widget. 16. In a computing system designed for having one or more computers configured to implement a method to perform automated software application integration, user-computer interactions, computer guided navigation or automation functionality for user or computerized machines performing a job, with said job relating to content blocks of a digitized document comprising at least one of written procedure or instructions, an instructional manual or a fillable form, with said content blocks being encapsulated within sequenced and linked widgets with specific interaction, integration and navigation behavior, and based on information contained in structured metadata, a method comprising:loading structured metadata in application memory in a structure appropriate for execution or interpretation by various types of processors when a work associated with an encapsulated content block of a corresponding digitized document, the content block comprising captured contents of the digitized document, is being performed by a person, group of people or one or more computerized machines; displaying the digitized document on a device display; activating and performing interactions, automations and integrations of a first widget, associated with a first encapsulated content block of the digitized document being displayed, based on the structured metadata; activating and performing interactions, automations and integrations of relevant sub-widgets; determining a next widget that needs to be activated based on static or conditional information in the metadata that depends on dynamic data generated or gathered during performance of content blocks associated to widgets at job performance time; performing interactions, automations and integrations of the next widget; repeating the acts of determining a next widget performing interactions, automations and integrations of the next widget until a last activated widget is reached; and wherein an interaction widget associated to a Step content block is placed inside an interaction widget associated to an Instruction section, as its child sub-widget, and linked with other widgets including other Step widgets, with each Step widget having a series of properties and sub-widgets to support one or more of the following:declaring the order of execution of Steps, at job performance time, as sequential, based on widget sequence numbers, or in an arbitrary order; indicating that completion of a step requires confirmation by the user and to provide a property or a sub-widget that may be placed inside a Step widget, for enforcing confirmation; a time-dependent step by automatically providing a timer starting from when a step was entered, at job performance time, and automatically enforcing a time dependency of a content block while providing the ability to pause and resume when applicable; indicating if the completion of a Step, at job performance time, requires a workers initials and providing a sub-widget for the initials that may be used, by placing it inside a Step widget; indicating if a Step is a Critical Step so that it may be rendered in a visually distinct way at job performance time; or indicating if a Step requires two users to be physically present together to implement concurrent verification before a step may be marked as completed, in which case an initial or signature sub-widget is added to a Step widget. 17. The method of claim 16 , further comprising:wherein displaying the digitized document on a device display comprises either displaying the digitized document in a graphical user interface, or without relying on the whole of the digitized document, displaying all content blocks associated with each widget on the device display by determining or assigning an appropriate location for a first widget, corresponding to a first content block and then determining locations of all the other linked widgets relative to the first or a previous widget or a containing widget and then displaying a content block contained within each widget inside a location determined for each widget; displaying or activating the first widget associated with the first displayed content block by superimposing or overlaying it over the associated content block based on properties expressed in the widget metadata, and then proceeding to the display of a next widget in a similar fashion, after the interaction of the first widget is completed; repeating the process until all superimposed or overlaid widgets, required for a job performance are displayed or a last section of the digitized document is reached and a last widget is displayed and associated interactions performed; when displaying an interface widget, based on information in the widget metadata, either automatically invoking all software services whose outputs are associated with the widget and then populating or displaying the output data, resulting from the automatic invocation of the service, inside of the associated interface widget; or, prompting the user for input data, using an appropriate widget associated to a type of input and then transferring user supplied inputs to inputs of the associated service and invoking the service to perform a function on another application; and when displaying an interaction widget, and of its child widgets, performing user-computer interaction implied by a specific type of the interface widget and displaying any contained interface widgets, and then advancing to a next relevant linked widget based on information in the structured metadata and displaying a containing widget associated with a next section of procedure or instructions. 18. The method of claim 16 , wherein real-time state, data and the statistics associated with the performance of a procedure or instructions based on the interactions and advancement through widgets is automatically recorded, calculated, inferred and reported; wherein such statistics include one or more of: real-time reporting of a current section or step of procedure or instructions, location of performance of a section based on GPS coordinates, start-time and completion time of each procedure or instructions section or step, or analytics based on aggregated statistics across all job instances associated with a procedure or instructions. 19. The method of claim 17 , wherein widgets associated to some select sections of the procedure or instructions are superimposed or overlaid on top of the digitized document and the performance of some other select sections, including steps within select instructions, are transferred to another device.
意見反饋