A Semantics-Based Approach to Machine Perception

Document Type


Publication Date



Machine perception can be formalized using semantic web technologies in order to derive abstractions from sensor data using background knowledge on the Web, and efficiently executed on resource-constrained devices.

Advances in sensing technology hold the promise to revolutionize our ability to observe and understand the world around us. Yet the gap between observation and understanding is vast. As sensors are becoming more advanced and cost-effective, the result is an avalanche of data of high volume, velocity, and of varied type, leading to the problem of too much data and not enough knowledge (i.e., insights leading to actions). Current estimates predict over 50 billion sensors connected to the Web by 2020. While the challenge of data deluge is formidable, a resolution has profound implications. The ability to translate low-level data into high-level abstractions closer to human understanding and decision-making has the potential to disrupt data-driven interdisciplinary sciences, such as environmental science, healthcare, and bioinformatics, as well as enable other emerging technologies, such as the Internet of Things.

The ability to make sense of sensory input is called perception; and while people are able to perceive their environment almost instantaneously, and seemingly without effort, machines continue to struggle with the task. Machine perception is a hard problem in computer science, with many fundamental issues that are yet to be adequately addressed, including: (a) annotation of sensor data, (b) interpretation of sensor data, and (c) efficient implementation and execution. This dissertation presents a semantics-based machine perception framework to address these issues.

The tangible primary contributions created to support the thesis of this dissertation include the development of a Semantic Sensor Observation Service (SemSOS) for accessing and querying sensor data on the Web, an ontology of perception (Intellego) that provides a formal semantics of machine perception and reasoning framework for the interpretation of sensor data, and efficient algorithms for the machine perception inference tasks to enable interpretation of sensor data on resource-constrained devices, such as smart phones. Each of these contributions has been prototyped, evaluated, and applied towards solving real-world problems in multiple domains including weather and healthcare.