Comments have closed. View all of the finalist entries below.

Integrated Human-Machine Intelligence

What is the compelling question or challenge?

Can we seamlessly integrate human and machine intelligence and allow the integrated intelligence to dynamically evolve as new knowledge and data are continuously added?

What do we know now about this Big Idea and what are the key research questions we need to address?

The need to explore elements involved in human knowledge-based cognitive processing and fuse them with machine intelligence, empowered through computational processing of large-scale complex data, has been recognized by a wide spectrum of specialized domains, such as medicine, science, social psychology, security intelligence, and more. Despite the significant advances in machine learning and artificial intelligence (AI), computational/statistical models alone are still insufficient to capture the key insights from highly complicated scenarios, where humans should continue to play an integral role in problem-solving and decision-making. Recent studies demonstrate that prior domain knowledge provides important learning cues that can potentially boost the performance of machine learning algorithms, including deep learning and other advanced statistical learning methods. As a result, domain knowledge from humans is borrowed and injected into a variety of algorithms and used to improve computer-based decision-making. Existing works primarily rely on text-based annotations or hard-coded rules and constraints to incorporate human domain knowledge to enhance machine intelligence in various tasks. However, it is often difficult to use text annotations to fully describe a complex scenario, such as medical images and surveillance videos. Moreover, hard-coded rules/constraints are fundamentally inadequate for knowledge capture and representation as knowledge, acquired through training and experience in complex and specialized domains, usually exists in tacit form that is not easily shared explicitly.

Integrated human-machine intelligence will fully and seamlessly synergize human domain expertise and machine’s computing power, enabling them to collectively tackle highly challenging tasks in specialized domains that neither could individually perform to satisfaction. A fundamental challenge involves extracting, understanding, and ultimately utilizing human expertise. Expert knowledge is predominantly in tacit form that is not easily shared explicitly. Furthermore, human decision-making is an inherently multimodal process involving bottom-up (data-driven) and top-down (knowledge-driven) processing. Novel data collection tools and approaches are required to properly elicit and extract data representing implicit knowledge, in some form, from human experts. Advanced and innovative computational/statistical models need to be developed to understand human domain knowledge once collected and uncover the underlying relationship between different modalities that provide important evidence to trace the use of tacit domain knowledge involved in human’s multimodal cognitive processing. The modeling outcome will guide the systematic fusion of human knowledge data with the task related data, which will ultimately achieve the envisioned human-machine intelligence integration. Instead of being a static process, the fusion should be conducted in an interactive and collaborative fashion, where human and machine take turns to iteratively direct an integrated learning process until a consensus is reached. Such a human-machine collaborative process requires the two parts to communicate in an intuitive way supported by an interactive interface where machine intelligence is directly interpretable to humans and experts can conveniently send their feedback in natural forms. Furthermore, the fusion should automatically adjust to a dynamic environment such that the integrated intelligence can continuously and automatically evolve along with the newly developed human knowledge and additionally collected data. A self-evolving integrated intelligence is essential to handle dynamic scenarios arising in many real-world applications (e.g., self-driving cars and military operations) where both the tasks and data may be fast changing.

Show More


National Science Foundation, 2415 Eisenhower Avenue, Alexandria VA 22314, USA

Tel: (703) 292-5111, FIRS: (800) 877-8339 | TDD: (800) 281-8749