【Domestic Papers】Multispectral In-Sensor computing for lmage Recognition Based on the Opposite Photogating Photosynapse

      University of Science and Technology of China, Long Shibing, Zhao Xiaolong et al. published an article titled "Multispectral In-Sensor Computing for Image Recognition Based on the Opposite Photogating Photosynapse” at ACS Nano.

      This paper presents a four-color reservoir computing system based on opposite photogating (OPG) engineering for multispectral optical synapses for image recognition. The system reduces the large amount of data redundancy caused by frequent data conversion and transmission in the traditional Von Neumann architecture by integrating sensing and computing capabilities into a single optical synapse. The optical synapse, based on Ga₂O₃/WSe₂ heterojunction field-effect transistors, exhibits bidirectional responses to different spectral stimuli: deep ultraviolet (DUV) light induces a negative threshold voltage (Vth) shift (excitatory response), and visible light induces a positive Vth shift (inhibitory response). This nonlinear optical response, along with tunable short-term memory characteristics, makes it suitable for reservoir computations in optoelectronics. The system achieved an accuracy of 88.3% in locating corona discharges in high-voltage systems, demonstrating the potential for precise intelligent image recognition in complex multispectral scenarios.

Background

      With the rapid development of artificial intelligence, photoelectric imaging sensors, as key hubs for information interaction, inevitably need to be intelligent. Traditional photoelectric sensors only convert light signals to electrical signals, and the final image recognition process requires frequent conversion and transmission of data between sensors, processors, and storage units, resulting in a large amount of data redundancy and high energy consumption, which is particularly disadvantageous for real-time detection and image recognition scenarios. Inspired by biological systems, neuromorphic architectures offer a promising alternative by integrating sensing and computing capabilities into artificial synapses. The biological visual system constructs optical images through a variety of photoreceptor cells that are sensitive to light of different wavelengths. These photoreceptor cells selectively extract key visual features, such as specific colors or shapes, thereby enhancing neural activity related to specific signals while suppressing irrelevant information. Compared with single-spectrum recognition, multi-spectrum recognition enhances the target recognition ability in complex optical scenes. However, most of the reported optical synapses operate only within a narrow spectral range, and developing multispectral optical synapses is crucial for accurate image recognition.

Abstract

      Through the integration of sensing and computing functions into a single photosynapse, the neuromorphic visual system mitigates the substantial data redundancy caused by frequent data conversion and transmission in Von Neumann architectures. However, most reported photosynapses can produce unidirectional light responses only without electric modulation and are limited to narrow spectral ranges, which limits their effectiveness in target recognition in complex real-world optical scenes. Here, we present a four-color reservoir computing (RC) system based on an opposite photogating (OPG)-engineered multispectral photosynapse. The OPG effect, characterized by light-modulated oppositely shifted threshold voltage (Vth), originates from different carrier dynamics in a Ga₂O₃/WSe₂ heterojunction field-effect transistor. Specifically, hole trapping in Ga₂O₃ under deep ultraviolet (DUV) light induces negative Vth shifts (excitatory responses), while electron trapping in WSe₂ under visible light causes positive Vth shifts (inhibitory responses). The nonlinear photoresponse and tunable short-term memory under external light stimuli make the photosynapse suitable for photoelectric reservoirs. The DUV-specific corona discharge, a critical challenge in high-voltage transmission systems, causes exacerbated equipment aging and significant energy losses. By integration of DUV-specific discharge signals and visible environmental information, the system achieves 88.3% accuracy in localizing the corona discharge among six high-risk components in high-voltage systems. Our multispectral RC system demonstrates a pathway toward precise intelligent image recognition in real-world multispectral scenarios.

Experimental Details

      The researchers first fabricated high-quality Ga₂O₃ and WSe₂ microsheets via mechanical exfoliation and residue-free dry transfer, and subsequently constructed Ga₂O₃/WSe₂ heterojunction field-effect transistors (FETs).

Innovations

      1. A four-color in-memory computing system based on Ga₂O₃/WSe₂ heterojunctions is proposed for multispectral image recognition.

      2. Bidirectional light response under different spectral stimulations was achieved through reverse grating (OPG) engineering, that is, deep ultraviolet (DUV) light induced negative threshold voltage (Vth) shift, and visible light induced forward Vth shift.

      3. Demonstrated the nonlinear optical response and tunable short-term memory properties of the optical synapse, which are crucial for constructing high-dimensional nonlinear feature s

      4. Achieved 88.3% accuracy in corona discharge localization in high-voltage systems and 94.7% accuracy in the four-color handwritten digit recognition task, demonstrating the potential of the system for practical applications.

Conclusion

      This study presents a four-color in-memory computing system based on opposite photogating (OPG) engineering for Ga₂O₃/WSe₂ optical synapses for multispectral image recognition. By achieving bidirectional light response under different spectral stimuli, the system successfully identified corona discharge locations in six high-risk locations in the high-voltage system with an accuracy rate of 88.3%. In addition, the system achieved 94.7% accuracy in the four-color handwritten digit recognition task, further demonstrating its effectiveness in image recognition. The system offers a new approach to achieving efficient image recognition by reducing data redundancy and lowering hardware complexity.

Results and Discussion

Figure 1. Multispectral image recognition system based on OPG-engineered optical synapses with dual excitation/inhibition modes. (a) Schematic of the biological visual system (close-up of synaptic details), illustrating excitatory and inhibitory behaviors of synapses under different spectral stimuli. (b) Schematic of a four-color sensor array for corona discharge identification, fusing deep ultraviolet and visible spectral information; each pixel unit contains four sub-pixels sensitive to the red (R), green (G), blue (B), and deep ultraviolet (U) bands. (c) Schematic of the OPG-engineered multispectral van der Waals optical synapse as a sub-pixel sensor. (d) Wavelength-dependent transfer characteristic curve and (e) bipolar EPSC/IPSC response curve. Negative threshold voltage drift under deep ultraviolet light (NPG effect, purple) corresponds to enhanced photocurrent, while positive drift under visible light (PPG effect, green) corresponds to reduced photocurrent.

 Figure 2. Photoelectric performance and working mechanism of a bipolar multi-band photodetector based on an OPG-engineered Ga₂O₃/WSe₂ van der Waals heterojunction field-effect transistor (vdW-HJFET). (a-i) Schematic of the device structure and (a-ii) optical micrograph of the device composed of Ga₂O₃ and WSe₂ microsheets. (b) Transfer characteristic curves under illumination from deep ultraviolet to visible wavelengths, showing a negative photogating (NPG) effect in the deep ultraviolet band (254–275 nm) and a positive photogating (PPG) effect in the visible band (465–625 nm).(c) Channel current variation (ΔI) under pulsed illumination from 254 to 625 nm, with polarity switching from positive to negative, consistent with the NPG-to-PPG transition observed in panel (b).(d-i) Schematic of in-situ Kelvin probe force microscopy (KPFM) analysis of Ga₂O₃ and WSe₂ materials without applied gate or drain bias, focusing on three photosensitive regions: region A (Ga₂O₃ channel), region B (WSe₂ on Ga₂O₃ channel), and region C (WSe₂ on the SiO₂ substrate).(d-ii) Surface work function variations in region A under dark conditions and after deep ultraviolet (275 and 254 nm) irradiation, revealing hole trapping in Ga₂O₃. (d-iii) Surface work functions of regions B and C in the dark state and under 520 or 254 nm illumination, indicating electron trapping in WSe₂.(e) Mechanism model of the NPG and PPG effects excluding the influence of gate and drain electric fields. Cross-sectional schematics illustrate the trap-induced non-equilibrium electron–hole distribution in (i) the dark state, (ii) under deep ultraviolet illumination, and (iii) under visible illumination.

 Figure 3. Excitatory and inhibitory synaptic simulations in OPG-engineered Ga₂O₃/WSe₂ photodetectors. (a) Schematic analogy between the multi-band photodetector and a biological synapse, where optical signals serve as the presynaptic input and channel current as the postsynaptic signal. (b) Transient ΔPSC responses under pulsed optical stimulation: (i) excitatory postsynaptic current (EPSC) under 254 nm illumination and (ii) inhibitory postsynaptic current (IPSC) under 520 nm illumination. (c) Paired-pulse facilitation (PPF) modulation of (i) EPSC and (ii) IPSC responses (Δt = 300 ms, t_on = 100 ms), where A₁ and A₂ represent the ΔPSC amplitudes of the first and second pulses, respectively. (d) PPF index of excitatory (top) and inhibitory (bottom) modes as a function of pulse interval. (e) Synaptic weight modulation by varying the pulse width and (f) by changing the number of pulses at different light intensities. The upper panels show excitatory behavior (DUV light), and the lower panels show inhibitory behavior (visible light, 520 nm). All experimental data were measured under the same bias conditions (V_d = 15 V, V_g = −0.1 V).

 Figure 4. Four-bit optical input nonlinear mapping based on OPG-engineered HJFET multispectral optical synapses. (a) Time-dependent photoresponse characteristics and dual-feature extraction of four representative 4-bit binary input sequences (“0001”, “0011”, “0111”, and “1111”) under 254 nm (purple) and 520 nm (green) illumination. Each binary input corresponds to an optical pulse sequence (t_on = 200 ms, Δt = 300 ms). (b) Statistical dual-feature analysis over 30 repeated cycles for each 4-bit input sequence under (i, ii) DUV (254 nm) and (iii, iv) visible light (520 nm) stimulation. The output distributions for all inputs (“0000” to “1111”) are presented using box plots and Gaussian fitting.  Figure 5. OPG-engineered multispectral RC system based on vdW-HJFET optical synapses for corona discharge localization. (a) Schematic diagram of the unmanned aerial vehicle (UAV) multispectral imaging system for corona discharge perception and identification, simultaneously achieving environmental background recognition (visible light) and DUV-specific corona discharge localization. (b) Twelve classification categories representing the states of six discharge-prone locations under two conditions: N (no discharge) and Y (discharge). (c) Preprocessing flow of four-color corona discharge images, including spectral separation, binarization, cropping, and recombination steps for subsequent in-sensor computation. (d) Schematic diagram of the multispectral RC system architecture, where the four-channel input reservoir serves as the core processing unit, dynamically converting input optical signals into feature outputs. (e) Training performance of the 12-classification task, showing rapid cross-entropy loss decline and 90% accuracy on the training set. (f) Confusion matrix comparison of classification performance between (i) dual-feature and (ii) single-feature strategies on the test dataset after 100 training epochs. The dual-feature strategy significantly outperforms the single-feature method, with a 16.8% improvement in classification accuracy.

Figure 5. OPG-engineered multispectral RC system based on vdW-HJFET optical synapses for corona discharge localization. (a) Schematic diagram of the unmanned aerial vehicle (UAV) multispectral imaging system for corona discharge perception and identification, simultaneously achieving environmental background recognition (visible light) and DUV-specific corona discharge localization. (b) Twelve classification categories representing the states of six discharge-prone locations under two conditions: N (no discharge) and Y (discharge). (c) Preprocessing flow of four-color corona discharge images, including spectral separation, binarization, cropping, and recombination steps for subsequent in-sensor computation. (d) Schematic diagram of the multispectral RC system architecture, where the four-channel input reservoir serves as the core processing unit, dynamically converting input optical signals into feature outputs. (e) Training performance of the 12-classification task, showing rapid cross-entropy loss decline and 90% accuracy on the training set. (f) Confusion matrix comparison of classification performance between (i) dual-feature and (ii) single-feature strategies on the test dataset after 100 training epochs. The dual-feature strategy significantly outperforms the single-feature method, with a 16.8% improvement in classification accuracy

DOI:

https://doi.org/10.1021/acsnano.5c03453