\cleardoublepage

\chapter{A Domain Ontology for Task Instructions}\label{chap:uagent}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Introduction} % (1 page)
% Why an ontology? We want to use it as an interchange format.
%%%%%%%%%%%%%%%%%%%
Knowledge graphs facilitate data integration across highly heterogeneous sources in a semantically useful way. Knowledge graphs may be equipped with a schema, frequently an ontology, that combines the associative power of the knowledge graph with the semantics of the ontology. Due to this, they are uniquely suited to support research in cognitive science, where it is often necessary to incorporate information from fields like computer science, psychology, neuroscience, philosophy, and more. 

\emph{Cognitive agents} are a sub-field of cognitive science and an application of the more broad study of cognitive architectures. Cognitive architectures, like ACT-R \cite{anderson2007can} for example, are an approach to understanding intelligent behavior and cognition that grew out of the idea of Unified Theories of Cognition \cite{newell1994unified}. These systems have their roots in AI production systems and some types use rules-based cognition. Many in Computer Science are familiar with inductive themes from a different type, called Connectionism, due to its historic ties with artificial neural networks. Symbolic cognitive architectures, by contrast, are less widely known outside of cognitive science, and are abstracted and explicit like logic programming.

Both ontologies and cognitive architectures deal with symbolic knowledge. Symbolic cognitive architectures typically focus on the plausibility of knowledge and the way in which that knowledge is translated into human behavior within a specific task. Ontologies offer a set of robust mechanisms for reasoning over complex knowledge bases and could help cognitive architectures adapt to tasks in novel environments. One way the two may be integrated is by leveraging the ontology to reduce the specificity of a cognitive agent.

In general, cognitive agents are often specialized, or \emph{differentiated}, to perform a specific task or set of tasks. An \emph{undifferentiated} agent is one that has no specialization. The purpose of such an agent is to be adaptable to new tasks as needed. As part of initial work to develop such an undifferentiated cognitive agent, we have developed a modular ontology that captures instructions for a specific cognitive agent task called ISR-MATB. We discuss this platform in more detail in the next section. 

Currently, the ontology supports the memory of a cognitive agent by adding structure to its knowledge and providing new varieties of query-like recall. And due to design methodology used during the modeling process, the ontology is general enough that it could model other cognitive agent experiments, which could then be evaluated against each other in a structured way. This allows the ontology to act as an invaluable interchange format between researchers developing cognitive agents.

The rest of this paper is organized as follows. Section \ref{sec:ISR-MATB} provides a brief overview of the use-case: ISR-MATB. Section \ref{sec:Ontology Description} provides an in-depth examination of the ontology. Finally, in Section \ref{sec:Conclusion}, we briefly conclude and discuss next steps.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{ISR-MATB}\label{sec:ISR-MATB} % (2 pages)
%%%%%%%%%%%%%%%%%%%
ISR-MATB is a series of cognitive tasks that could be completed by a Cognitive Agent or a human \cite{Frame2019}. A trial starts with one very simple task, the evaluation then branches into two sub-tasks that relate back to the first task. After the two sub-tasks are complete the agent completes one final task requiring integration of remembered information from all previous tasks. The final task is made more difficult by the possibility of incorrect feedback as the agent learns. ISR-MATB is intended to be repeated for a fixed time so that researchers can observe changes in the agent's response time and develop better computational cognitive agents.
\begin{figure}[t]
    \centering
    \includegraphics[width=\textwidth]{figures/isrmatb.png}
    \caption{An example depicting the four ISR-MATB tasks in a single interface.}
    \label{fig:isrmatb}
\end{figure}

\subsection{Psychomotor Vigilance Test}
The Psychomotor Vigilance Test is one of the more basic cognitive tasks \cite{dinges1985microcomputer}. In this task, there is an area of the screen where a letter could appear. The letter will be drawn with a specific color. When the letter does appear an agent must press a button that acknowledges they have seen it. If the agent pushes the button too soon a false start is recorded and the task continues normally. If too much time passes before the agent pushes the button then the task will continue with the letter unacknowledged. The next two tasks reference this letter and color, so the agent is instructed to remember them.

\subsection{Visual Search}
The Visual Search task requires that the agent determine if the letter they remember is among a group of many letters that appear on the screen \cite{treisman1980feature}. The other letters are distractors, and may be the same letter as the target with a different color, or the same color as the target with a different letter, or both color and letter different. The target may or may not appear among the distractors, and never appears more than once. The agent pushes a button to indicate whether the the letter is present or absent.

\subsection{Auditory Search}
The Auditory Search task is very similar to the Visual Search, except of course that the agent must listen instead of look. In this task there are between one and four audio messages that each include a spoken color and letter. If one of the messages is the same as as the letter and color from the first task the agent pushes a button to indicate that it is present, otherwise they indicate that it is absent.

\subsection{Decision Making}
The final task, Decision Making, requires agents to infer a relationship between the outcomes of the Visual and Auditory Search tasks together with a new binary piece of information called ``Intelligence" that appears after choosing whether to hypothetically allocate sensors or not. The rule the agent must guess is not too hard, but it is complex enough that it must be learned by trial-and-error over multiple attempts. Learning the rule is made more difficult by the unlikely but not impossible event that the program responds incorrectly even when a correct answer is given. Responding `yes' or `no' to this sub-task ends one ISR-MATB trial.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Ontology Description}\label{sec:Ontology Description} % (4 pages)
%%%%%%%%%%%%%%%%%%%
In this section we present the Instruction Ontology, a domain ontology built for use with the ISR-MATB experiment platform. This ontology was produced by following the Modular Ontology Modeling (MOM) methodology, outlined in \cite{chess,momtut} MOM is designed to ensure the high quality and reusability of the resulting ontology, both in terms of scope and in terms of granularity, which is a desired outcome.

The ontology consists of six modules: \textsf{ISR-MATB Experiment}, \textsf{Instruction}, \textsf{SituationDescription}, \textsf{ItemRole}, \textsf{Action}, and \textsf{Affordance}. For each module, we describe its purpose, provide a schema diagram,\footnote{A schema diagram is an informal, but intuitive way for conveying information about the structure and contents of an ontology. We use a consistent visual syntax for convenience, detailed in Figure \ref{fig:1}.} and state its axiomatization in both description logic syntax and natural language. The OWL file for this ontology can be found online\footnote{See \url{https://raw.githubusercontent.com/undiffagents/uagent/develop/ontology/uagent.owl}.} as well as the official documentation.\footnote{See \url{https://daselab.cs.ksu.edu/content/domain-ontology-instruction}} Figure \ref{fig:all} shows the schema diagram for the entire ontology. 

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{ISR-MATB Experiment.} 
\label{ssec:isrmatbexp}
%%%%%%%%%%%%%%%%%%%
The \textsf{ISR-MATB Experiment} module is the core module for the ontology. The two main classes are \textsf{ISR-MATB Experiment} and \textsf{ISR-MATB Task}. As noted in Section \ref{sec:ISR-MATB}, an experiment consists of up to four tasks that may require that information be carried between them, where each Task resides in a specific quadrant of the interface. Each Task provides roles to different \textsf{Items}, as well as a set of \textsf{Instructions} for the agent to carry out. We discuss these classes in more detail in their respective Module sections. The schema diagram for this module is shown in Figure \ref{fig:experiment}.

\subsubsection*{Axiomatization:}
\begin{align}
\top &\sqsubseteq \forall\textsf{affords.Affordance} \\
\textsf{ISR-MATBTask} &\sqsubseteq \mathord{\geq}1~\textsf{hasInstruction.Instruction} \\
\textsf{ISR-MATBExperiment} &\sqsubseteq \mathord{\leq}4~\textsf{hasTask.ISR-MATBTask} \\
\top &\sqsubseteq \forall \textsf{hasLocation.Location} \\
\top &\sqsubseteq \forall \textsf{hasName.xsd:string} \\
\textsf{ISR-MATBTask} &\sqsubseteq \mathord{=}1~\textsf{hasName.xsd:string} \\
\textsf{ISR-MATBTask} &\sqsubseteq \forall\textsf{providesRole.ItemRole} \\
\textsf{ISR-MATBTask} &\sqsubseteq \forall\textsf{informs.ISR-MATBTask}
\end{align}

\subsubsection*{Explanation of axioms above:}
\begin{enumerate}
\item Range. The range of \textsf{affords} is \textsf{Affordance}.
\item Minimum Cardinality. An \textsf{ISR-MATBTask} has at least one \textsf{Instruction}.
\item Maximum Cardinality. An \textsf{ISR-MATBExperiment} consists of at most four \textsf{ISR-MATBTasks}.
\item Range. The range of \textsf{hasLocation} is \textsf{Location}.
\item Range. The range of \textsf{hasName} is \textsf{xsd:string}.
\item Scoped Range. The range of \textsf{providesRole} is \textsf{ItemRole} when the domain is \textsf{ISR-MATBTask}.
\item Scoped Range. The range of \textsf{informs} is \textsf{ISR-MATBTask} when the domain is \textsf{ISR-MATBTask}.
\end{enumerate}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Action}
\label{ssec:action}
%%%%%%%%%%%%%%%%%%%
The \textsf{Action} module is an instantiation of the \textsf{Explicit Typing} meta-pattern described in \cite{modl}.\footnote{\cite{modl} is a modular ontology design library; it contains a set of frequently used patterns and respective documentation.}

In this case, we use a class, \textsf{ActionType}, to represent a controlled vocabulary. We believe that using a controlled vocabulary to represent this type information is less invasive to the ontology. This way, adding or removing types of actions from the controlled vocabulary does not actually change the ontology. Some instances of the controlled vocabulary are listed in Figure \ref{fig:all}. 

An \textsf{Action}, in this context, is the physical, actual action that takes place to transition between different states of the experiment, e.g. `the action of clicking a button.' The schema diagram for this module is shown in Figure \ref{fig:action}.

\subsubsection*{Axiomatization:}
\begin{align}
\textsf{Action} &\sqsubseteq \mathord{=}1\textsf{ofType.ActionType}
\end{align}

\subsubsection*{Explanation of axioms above:}
\begin{enumerate}
\item Exact Cardinality. An \textsf{Action} has exactly one \textsf{ActionType}.
\end{enumerate}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Affordance}
\label{ssec:affordance}
%%%%%%%%%%%%%%%%%%%
The \textsf{Affordance} module is also instantiated from the \textsf{Explicit Typing} meta-pattern, explained in more detail in Section \ref{ssec:action} and \cite{modl}. An \textsf{Affordance} is essentially some quality of an \textsf{Item} that indicates that ``something'' may be done with it. Familiar examples might include \emph{clickable} buttons or text highlighted in blue (perhaps indicating that it's a hyperlink). Instances of the \textsf{AffordanceType} can be found in Figure \ref{fig:all}. The schema diagram for this module is shown in Figure \ref{fig:affordance}.

\begin{figure*}[!htb]
    \centering
    \subfloat[The \textsf{Action} module.\label{fig:action}]{{\includegraphics[width=0.45\textwidth]{figures/Action}}}%
    \qquad
    \subfloat[The \textsf{Affordance} module.\label{fig:affordance}]{{\includegraphics[width=0.45\textwidth]{figures/Affordance}}}%
    \subfloat[The \textsf{ISR-MATB Experiment} module.\label{fig:experiment}]{{\includegraphics[width=0.45\textwidth]{figures/ISR-MATBExperiment.pdf}}}%
    \qquad
    \subfloat[The \textsf{ItemRole} module.\label{fig:itemrole}]{{\includegraphics[width=0.45\textwidth]{figures/ItemRole.pdf}}}%
    \caption[Schema Diagrams]{Schema Diagrams\footnotemark}
    \label{fig:1}
\end{figure*}
\footnotetext{Orange boxes are classes and indicate that they are central to the diagram. Blue dashed boxes indicate a reference to another diagram, pattern, or module. Gray frames with a dashed outline contain modules. Arrows depict relations and open arrows represent subclass relations. Yellow ovals indicate data types (and necessarily, arrows pointing to a datatype are data properties). Finally, purple boxes represent controlled vocabularies. That is, they represent a controlled set of IRIs that are of that type.}

\subsubsection*{Axiomatization:}
\begin{align}
\textsf{Affordance} &\sqsubseteq \mathord{=}1\textsf{hasAffordanceType.AffordanceType}
\end{align}

\subsubsection*{Explanation of axioms above:}
\begin{enumerate}
\item Exact cardinality. An \textsf{Affordance} has exactly one \textsf{AffordanceType}.
\end{enumerate}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{ItemRole}
\label{ssec:itemrole}
%%%%%%%%%%%%%%%%%%%
The \textsf{ItemRole} module is an instantiation of the \textsf{AgentRole} pattern, which may also be found in \cite{modl}. We also equip it with an explicit type, in the same manner as \textsf{Action} and \textsf{Affordance}.

Each \textsf{ISR-MATB Task} may provide roles to \textsf{Item}s. That is, certain items may be a target or distractor, but not always. This allows us to assign certain roles to items that may, if they were qualities, be ontologically disjoint. The schema diagram for this module is shown in Figure \ref{fig:itemrole}.

\subsubsection*{Axiomatization:}
\begin{align}
\textsf{ISR-MATBTask} &\sqsubseteq \forall\textsf{providesRole.ItemRole} \\
\top &\sqsubseteq \forall \textsf{hasItemRoleType.ItemRoleType} \\
\textsf{ItemRole} &\sqsubseteq \forall\textsf{assumedBy.Item} \\
\textsf{ItemRole} &\sqsubseteq \exists\textsf{assumedBy.Item}
\end{align}

\subsubsection*{Explanation of axioms above:}
\begin{enumerate}
\item Scoped Range. The range of \textsf{providesRole} is \textsf{ItemRole} when the domain is \textsf{ISR-MATBTask}.
\item Range. The range of \textsf{hasItemRoleType} is \textsf{ItemRoleType}.
\item Scoped Range. \textsf{ItemRoles} are \textsf{assumedBy} \textsf{Items}.
\item Existential. Every \textsf{ItemRole} is \textsf{assumedBy} an \textsf{Item}.
\end{enumerate}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{SituationDescription}
\label{ssec:situation}
%%%%%%%%%%%%%%%%%%%
\begin{figure}%[t]
    \centering
    \includegraphics[width=\textwidth]{figures/SituationDescription.pdf}
    \caption[The schema diagram for the\textsf{SituationDescription} module.]{The schema diagram for the \textsf{SituationDescription} module.\footnotemark}
    \label{fig:situation}
\end{figure}
\footnotetext{Color and shape usage is the same as in previous diagrams.}

For this module, we opted to use the Situation and Description approach. We chose to use this conceptualization due to the non-linear nature of the instructions.\footnote{For a deeper discussion on Descriptions, Situations, and Plans, see \cite{GangemiM03}.} That is, an \textsf{ISR-MATB Task} is not a sequence of instructions, but a collection of directions or descriptions.

An \textsf{Instruction}, is a description of a way to transition between two states. In order to follow out an instruction the state described in the the pre-SituationDescription would need to be met. Following through would result in a new state, the Post-Situtation Description.

Furthermore, the \textsf{SituationDescription} will indicate the presence, or absence, of an item, as well as its description. Descriptions, in this case, are relegated to controlled vocabularies in the same manner as \textsf{Affordance} or \textsf{Action}. We call this an \textsf{ItemDescription} because it is inherent to the \textsf{Instruction} and not the \textsf{Item}, itself. 

The schema diagram for this module is shown in Figure \ref{fig:situation}.


\subsubsection*{Axiomatization:}
\begin{align}
\textsf{SituationDescription} &\sqsubseteq \forall \textsf{hasCurrentCondition.}(\textsf{RelativeCondition}\sqcup\textsf{ItemDescription}) \\
\textsf{SituationDescription} &\sqsubseteq \forall \textsf{hasEarlierCondition.ItemDescription} \\
\top &\sqsubseteq \forall\textsf{hasRelativeConditionType.RelativeConditionType} \\
\textsf{RelativeCondition} &\sqsubseteq \forall \textsf{hasFirstItem.ItemDescription} \\
\textsf{RelativeCondition} &\sqsubseteq \forall \textsf{hasSecondItem.ItemDescription} \\
\textsf{ItemDescription} &\sqsubseteq \forall \textsf{ofItem.Item} \\
\textsf{ItemDescription} &\sqsubseteq \mathord{=}1~\textsf{isPresent.xsd:boolean} \\
\top &\sqsubseteq \forall\textsf{refersToItemLocation.LocationType} \\
\top &\sqsubseteq \forall\textsf{refersToItemColor.ColorType} \\
\top &\sqsubseteq \forall\textsf{refersToShapeType.ShapeType} \\
\top &\sqsubseteq \forall\textsf{refersToItemType.ItemType} \\
\textsf{ItemDescription} &\sqsubseteq \mathord{\geq}0~\textsf{refersToItemLocation.LocationType} \\
\textsf{ItemDescription} &\sqsubseteq \mathord{\geq}0~\textsf{refersToItemColor.ColorType} \\
\textsf{ItemDescription} &\sqsubseteq \mathord{\geq}0~\textsf{refersToItemShape.ShapeType} \\
\textsf{ItemDescription} &\sqsubseteq \mathord{\geq}0~\textsf{refersToItemType.ItemType} \\
\top &\sqsubseteq \forall \textsf{hasItemName.xsd:string} \\
\exists\textsf{hasItemName.}\top &\sqsubseteq \textsf{Item}
\end{align}

\subsubsection*{Explanation of axioms above:}
\begin{enumerate}
\item Scoped Range. The range of \textsf{hasCurrentCondition} is a \textsf{RelativeCondition} or \textsf{ItemDescription} when the domain is \textsf{SituationDescription}.
\item Scoped Range. The range of \textsf{hasEarlierCondition} is \textsf{ItemDescription} when the domain is \textsf{SituationDescription}.
\item Range. The range of \textsf{hasRelativeConditionType} is \textsf{RelativeConditionType}.
\item Scoped Range. The range of \textsf{hasFirstItem} is \textsf{ItemDescription} when the domain is \textsf{RelativeCondition}.
\item Scoped Range. The range of \textsf{hasSecondItem} is \textsf{ItemDescription} when the domain is \textsf{RelativeCondition}.
\item Scoped Range. The range of \textsf{ofItem} is Item when the domain is \textsf{ItemDescription}.
\item Scoped Range. An \textsf{ItemDescription} has exactly one Boolean flag indicating whether or not it is present.
\item Range. The range of \textsf{refersToItemLocation} is \textsf{LocationType}.
\item Range. The range of \textsf{refersToItemColor} is \textsf{ColorType}.
\item Range. The range of \textsf{refersToItemShape} is \textsf{ShapeType}.
\item Range. The range of \textsf{refersToItemType} is \textsf{ItemType}.
\item Structural Tautology. An \textsf{ItemDescriptio}n may refer to a \textsf{LocationType}.
\item Structural Tautology. An \textsf{ItemDescription} may refer to a \textsf{ColorType}.
\item Structural Tautology. An \textsf{ItemDescription} may refer to a \textsf{ShapeType}.
\item Structural Tautology. An \textsf{ItemDescription} may refer to an \textsf{ItemType}.
\item Range. The range of \textsf{hasItemName} is \textsf{xsd:string}.
\item Domain Restriction. The domain of \textsf{hasItemName} is restricted to \textsf{Items}.
\end{enumerate}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Instruction}
\label{ssec:instruction}
%%%%%%%%%%%%%%%%%%%
Instructions are the atomic units of a task. They come in two varieties: descriptions and actions. The former are instructions that are prescriptive or descriptive. They are statements that indicate information about the environment or the task. They may, in natural language, take such form as ``There is a button named `Present'.'' The latter type of instruction instructs when or where to do something. For example, ``Press the button if a high-pitched tone is heard.'' An \textsf{Action-Instruction} prescribes some transition between descriptions of situations, whereas \textsf{Description-Instruction}s directly contribute to said \textsf{SituationDescription}. The module also uses a data property to capture the natural language formulation of the \textsf{Instruction}.  The schema diagram for this module is shown in Figure \ref{fig:instruction}.

\begin{figure}[t]
    \centering
    \includegraphics[width=\textwidth]{figures/Instruction.pdf}
    \caption[The schema diagram for the \textsf{Instruction} module.]{The schema diagram for the \textsf{Instruction} module.\footnotemark}
    \label{fig:instruction}
\end{figure}
\footnotetext{Color and shape usage is the same as in previous diagrams.}

\subsubsection*{Axiomatization:}
\begin{align}
\textsf{ActionInstruction} &\sqsubseteq \textsf{Instruction} \\
\textsf{ActionInstruction} &\sqsubseteq \forall \textsf{prescribes.TransitionDescription} \\
\top &\sqsubseteq \forall \textsf{asString.xsd:string} \\
\textsf{Instruction} &\sqsubseteq \mathord{\geq}0~\textsf{asString.xsd:string} \\
\textsf{DescriptionInstruction} &\sqsubseteq \textsf{Instruction} \\
\textsf{DescriptionInstruction} &\sqsubseteq \forall \textsf{contributesTo.SituationDescription} \\
\top &\sqsubseteq \forall \textsf{hasPreSituationDescription.SituationDescription} \\
\top &\sqsubseteq \forall \textsf{hasPostSituationDescription.SituationDescription} 
\end{align}

\subsubsection*{Explanation of axioms above:}
\begin{enumerate}
\item Subclass. Every \textsf{ActionInstruction} is an \textsf{Instruction}.
\item Scoped Range. The range of prescribes is \textsf{TransitionDescription} when the domain is \textsf{ActionInstruction}.
\item Range. The range of \textsf{asString} is \textsf{xsd:string}.
\item Structural Tautology. An \textsf{Instruction} may have a string representation.
\item Subclass. Every \textsf{DescriptionInstruction} is an \textsf{Instruction}.
\item Scoped Range. The range of \textsf{contributesTo} is \textsf{SituationDescription} when the domain is \textsf{DescriptionInstruction}.
\item Range. The range of \textsf{hasPreSituationDescription} is \textsf{SituationDescription}.
\item Range. The range of \textsf{hasPostSituationDescription} is \textsf{SituationDescription}.
\end{enumerate}

%%%%%%%%%%%%%%%%%%%
\begin{figure}[H]
    \centering
    \includegraphics[angle=90,origin=c,height=.9\textwidth]{figures/all_draft-6.pdf}
    %\includegraphics[width=\textwidth]{figures/all_draft-6.pdf}
    \caption[The schema diagram for the entire ontology.]{The schema diagram for the entire ontology. \footnotemark}
    \label{fig:all}
\end{figure}
\footnotetext{Note that the \textsf{SituationDescription} module is nested in the \textsf{Instruction} Module. Color and shape usage is the same as in previous diagrams.}
%%%%%%%%%%%%%%%%%%%

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Conclusion}\label{sec:Conclusion} % (1 page)
%%%%%%%%%%%%%%%%%%%
In this paper we have presented an ontology for modeling the ISR-MATB cognitive agent task instructions. This ontology can be used, as we have, to directly support the memory of a cognitive agent performing tasks. It also could support experiment design, irrespective of any agent, by providing a structured basis for evaluating similar tasks. The modular structure facilitates adapting the ontology to other use cases and scenarios by replacing or adapting the existing modules. It is also possible to create new modules from the referenced patterns via template-based instantiation \cite{template}.

\subsection{Future Work}
In the future we plan to extend this ontology so that it can support a fully undifferentiated agent. This will include tasks like ISR-MATB, but also many others that could be very different. One such task is supporting materials science research that uses the Autonomous Research System (ARES) framework. An undifferentiated cognitive agent could operate a robotic system that performs research, using software like ARES, saving materials researchers hours of potentially hazardous lab work.
