Vision HgNN | Rajat Sarkar

Abstract

Vision Hypergraph Neural Networks (Vision-HgNN) is a hypergraph-based backbone architecture for electron micrograph classification that models higher-order dependencies between image patches, going beyond pairwise relations. Traditional methods such as Convolutional Neural Networks, Vision Transformers, and Graph Neural Networks fail to capture higher-order relationships among spatial regions in electron micrographs.

Vision-HgNN represents micrographs as visual hypergraphs where image patches serve as hypernodes. The architecture comprises four core modules: Hypergraph Structure Learning (HgSL) for constructing the hypergraph topology, Hypergraph Attention Network (HgAT) for capturing short- and moderate-range dependencies, Hypergraph Transformer (HgT) for modeling long-range pairwise interactions, and Hypergraph Read-out (HgRo) for producing graph-level embeddings for downstream classification.

Key Highlights

                                Higher-Order
                                Learning
                            

                                4 Novel
                                Modules
                            

                                Hypergraph
                                Backbone
                            

Architecture

Vision-HgNN introduces a novel visual hypergraph representation paradigm for scientific imaging. The architecture constructs hypergraphs from electron micrograph patches using Hypergraph Structure Learning (HgSL), then processes them through Hypergraph Attention Networks (HgAT) for local spatial interactions and Hypergraph Transformers (HgT) for global long-range dependencies, before aggregating via Hypergraph Read-out (HgRo) for classification.

Figure: Vision HgNN architecture with HgSL, HgAT, HgT, and HgRo modules.

Key Contributions

Hypergraph Backbone: First hypergraph-based vision architecture for electron micrograph classification.
Higher-Order Modeling: Captures beyond-pairwise relationships using hyperedges connecting multiple patches.
Multi-Scale Attention: Combines HgAT for local and HgT for global spatial interactions.
Novel Representations: Introduces visual hypergraph representation paradigm for scientific imaging.

Results

Vision-HgNN demonstrates the effectiveness of hypergraph-based higher-order modeling for electron micrograph classification, outperforming traditional pairwise approaches by capturing richer spatial relationships among image patches.

Different representation paradigms for electron micrograph analysis.

Learned pattern visualization from the hypergraph model.

Similarity analysis across micrograph representations.