Human-machine collaboration in reshaping visual perception and cognition

Human understanding of the world through visual signals involves two closely intertwined processes: perception and cognition. However, both are inherently constrained by human physiological limitations. This commentary advocates for reshaping visual perception and cognition through human-machine collaboration, with the aim of overcoming these intrinsic human limits (Figure 1). To this end, we explore a feasible human-machine collaborative perception and cognition system, providing concrete operational guidance in two key areas: ensuring perception/cognition consistency and enabling real-time processing. Furthermore, we discuss the future of human-machine deep collaboration systems and propose a new perspective, emphasizing that their core lies in the development of bio-neuron machines. Building on this foundation, we suggest potential strategies for facilitating effective interactions between machine and human brain neurons. However, safety considerations and ethical implications require careful attention.

Challenges in human visual perception and cognition

Perception is the process by which raw visual input from the eyes is processed into clear representations of the external world, enabling us to “see clearly.” Cognition interprets these representations using memory and knowledge to recognize objects, categorize semantics, and understand scenes, allowing us to “understand accurately.”

Seeing clearly at all times is challenging due to two main factors: harsh external conditions and individual physiological impairments. On the one hand, harsh environments (e.g., haze, rain, darkness) degrade visual input by scattering light, creating glare, or obscuring details limiting high-fidelity visual perception. On the other hand, physiological impairments, from myopia to severe disorders (e.g., cataracts, optic nerve damage) can reduce visual clarity. Visual cognition also faces several key challenges that hinder accurate understanding. Specifically, limited attentional restricts the full analysis of complex scenes, causing selective processing and inattentional blindness. In high-risk settings like battlefields, focusing on one target may make soldiers to miss other threats. Besides, human visual cognition is prone to systematic illusions, distorting understanding. For example, certain patterns may mislead judgments of depth, causing a two-dimensional image to be perceived as three-dimensional, showing that even clear visual input can be misinterpreted.

Human-machine collaborative perception and cognition

To tackle challenges in visual perception and cognition, we explore a human-machine collaborative system requiring no implantation. Instead, it primarily extends human perception and enhances cognition, offering a relatively safe collaboration model. First, during visual perception, intelligent machines process degraded signals with advanced restoration algorithms (e.g., exposure correction, dehazing) to produce high-visibility images for observation. This enables humans to perceive scene distributions accurately, even in challenging environments. Second, in the cognition stage, intelligent machines generate semantic-level annotations using AI techniques such as segmentation and object detection. This structured information helps humans quickly understand complex scenes, enhancing cognition and facilitating better decision-making. Remarkably, vision-language models is elevating this capability. They recognize image content and interpret visual information at a knowledge level, providing humans with insightful summaries and suggestions. Moreover, using full receptive field modeling and adversarial training, these AI algorithms can help overcome human cognitive limits, like inattentional blindness and systematic illusions.