Authors are with the Automation and Robotics Research Group, Interdisciplinary Centre for Security, Reliability, and Trust (SnT), University of Luxembourg, Luxembourg. Holger Voos is also associated with the Faculty of Science, Technology, and Medicine, University of Luxembourg, Luxembourg.

{saad.ejaz, miguel.fernandez, holger.voos, joseluis.sanchezlopez}@uni.lu

Author is with I3A, Universidad de Zaragoza.

jcivera@unizar.es

This research was funded, in whole or in part, by the Luxembourg National Research Fund (FNR) under the DEUS Project (Ref. C22/IS/17387634/DEUS).*For the purpose of open access, and in fulfillment of the obligations arising from the grant agreement, the author has applied a Creative Commons Attribution 4.0 International (CC BY 4.0) license to any Author Accepted Manuscript version arising from this submission.

Abstract

CAD-to-image alignment aims to estimate an object’s 9D pose (rotation, translation, and anisotropic scale) from a single RGB image, enabling applications in robotics and augmented reality. Recent zero-shot methods use visual foundation models to match image regions to CAD models, yet typically their correspondences are appearance-driven and degrade under occlusion or sim-to-real domain shift. To address these limitations, we introduce SUFLECA (Scaling Up Feature LEarning for CAD Alignment), a weakly-supervised framework for zero-shot CAD alignment with two key contributions. First, SUFLECA scales up geometry-grounded feature learning from pretrained visual representations through Normalized Object Coordinates (NOCs) supervision on 674K images spanning 12 real and synthetic datasets, learning compact geometry-aware features that generalize across domains. Second, we propose a geometrically consistent matching algorithm that establishes reliable one-to-one CAD-to-image correspondences. Together, these contributions enable accurate, sub-second alignment per object instance without iterative pose refinement. On ScanNet25k, SUFLECA achieves 33.4%/42.3% category/instance accuracy, outperforming, with a smaller computational footprint, the strongest zero-shot baseline by 10.3/12.2 percentage points and, for the first time on this benchmark, even surpassing fully supervised methods. Code is available at: https://github.com/snt-arg/SUFLECA

I Introduction

Recovering the 3D pose and shape of objects from images is a fundamental challenge in computer vision, with relevant applications in robotic perception [33] and manipulation [9]. A practical pipeline typically first retrieves a CAD model that approximately matches the target object, and then estimates a 9D alignment (rotation, translation, and anisotropic scale) to register it to the image observation [14, 3]. The alignment step is typically solved by establishing correspondences between the CAD model and the observed image, followed by robust pose estimation from the resulting matches.

Early works cast this as a supervised learning problem, using end-to-end [17, 14] or render-and-compare [18, 19] training setups, which lack generalization to out-of-distribution images and require costly pose-annotated data. Advances in foundation models for visual perception have enabled zero-shot CAD alignment through pretrained backbones that produce high-dimensional features for correspondence estimation, improving generalization across categories and distributions [39, 2]. However, features from visual foundation models are primarily appearance-based and lack grounding in the 3D object geometries. This results in a domain gap, leading to suboptimal alignment accuracy.

Weak supervision via synthetic renders of CAD models has been used to align this feature space with the geometry-focused Normalized Object Coordinates (NOCs) [36, 3], yet models trained on synthetic and isolated data struggle to generalize to the clutter, occlusion, and appearance variation present in real scenes. Beyond training data, nearest-neighbor matching in high-dimensional semantic feature spaces, as used in existing works, is costly in both memory and computation and fails to exploit the rigid geometric structure of the problem. This produces noisy matches that often require expensive iterative refinement to recover accuracy.

In this paper we propose SUFLECA (Scaling Up Feature LEarning for CAD-to-image Alignment), which addresses these limitations through two key contributions. First, it scales NOC-supervised feature learning to a large mixture of real and synthetic datasets, substantially improving generalization. The resulting low-dimensional feature space reduces both matching cost and memory footprint, making it well suited for robotic applications. Second, we introduce a correspondence estimation algorithm that replaces conventional semantic-only nearest-neighbor matching with a method that enforces mutuality and geometric consistency. Together, these contributions enable zero-shot alignment accuracy that surpasses supervised methods while maintaining sub-second runtime, as shown in Figure 1.

II Related Work

II-A Normalized Object Coordinates (NOCs)

NOCs [36] represent each object point in a category-level canonical coordinate frame, allowing pose estimation to be formulated as dense coordinate prediction followed by 3D registration. OmniNOCS [15] introduced large-scale NOC supervision for pose estimation, although it draws heavily from single-object datasets such as Objectron [1]. Vision backbones implicitly encode 3D-awareness [31], motivating the use of NOC supervision as an auxiliary signal on top of frozen features to shape a high-dimensional feature space for correspondence estimation [3]. However, this remains limited in coverage of occlusions and real-synthetic pairings, which are particularly important in CAD alignment, where a synthetic CAD model is aligned to a real object observation that may be affected from partial occlusion.

II-B Supervised CAD-to-image alignment

Supervised approaches to single-image CAD alignment span three paradigms. Direct regression methods predict 9D pose in a single feed-forward pass. Total3DUnderstanding [23] jointly regresses layout, boxes, and meshes; and Mask2CAD [17] couples detection with image-shape embeddings. Render-and-compare methods refine an initial CAD pose by minimising the difference between the rendered hypothesis and the input image. 3D-RCNN [16] pioneered this idea with differentiable rendering and CAD priors, while SPARC [18] and MultiObj-SPARC [19] achieve strong results through sparse transformer-based 9D pose updates. In contrast, indirect methods first estimate correspondences and then recover pose through registration. ROCA [14] regresses NOCs from images to establish such correspondences and is trained end-to-end. More recently, CosCAD [38] combines cross-modal CAD retrieval with pose alignment by integrating image, CAD, and text features in a shared representation space, improving retrieval and alignment under occlusion. However, all of these methods rely on direct pose supervision, which limits their scalability.

II-C Zero-shot CAD-to-image alignment

A parallel line of work aims to remove the dependence on pose-annotated training data. DiffCAD [13] trains cascaded diffusion models on synthetic data for scale, pose, and shape estimation, while SDFit [2] fits a morphable SDF through iterative render-and-compare optimization. However, both methods require per-category priors and are computationally expensive. Related category-level approaches [33, 32] aim to estimate optimal object pose and shape by progressively pruning outliers to recover the largest set of compatible measurements. Our correspondence filtering is inspired by this line of work, but these methods still rely on active shape models and keypoint detectors, which are difficult to obtain for most categories. Among other works, Diorama [39] uses an additional off-the-shelf model for scale inference and targets open-world scene-level layout, but its accuracy degrades in real-world scenes with occlusions. Recently, ZeroCAD [3] trains a NOC-supervised adapter over a frozen visual backbone for zero-shot 9D alignment. However, training exclusively on synthetic single-object scenes across nine categories leaves a substantial domain gap, and its simple nearest-neighbor matching admits many-to-one geometrically inconsistent correspondences.

III SUFLECA

媒体内容 · 前往原文查看

RGB frames source	Annotation source	Number of frames	Avg. objects/frame
Pascal3D+ [11]	Included	6, 108	1.00
Objectron [1]	OmniNOCS [15]	7, 357	1.20
ARKitScenes [6]	OmniNOCS [15]	52, 146	3.55
REAL275 [36]	Included	7, 072	5.00
RealEstate10K [42]	CAD-Estate [22]	127, 214	2.16
ScanNet [10]	Scannotate [26]	117, 284	2.58
ScanNet++ [41]	Scannotate++ [26]	88, 198	2.96
Pix3D [35]	Included	11, 622	1.00
ObjectNet3D [40]	Included	20, 520	1.00
3D-Front [12]	Rendering	69, 578	6.43
Hypersim [30]	OmniNOCS [15]	44, 545	13.55
ShapeNet [7]	Rendering	123,207	1.00
Overall		674, 851	3.38

III-A Problem Definition

Given an RGB image, camera intrinsics, and a candidate CAD model retrieved from off-the-shelf methods, the goal is to estimate a 9D transformation from the CAD coordinate frame to the camera coordinate frame , as follows:

(1)

where , is an axis-aligned anisotropic scale matrix, and . We extend this standard formulation and also output a score, , that measures the quality of the estimated alignment, which is useful for ranking alignments or multi-view aggregation.

III-B Overall CAD Fitting Pipeline

SUFLECA follows the standard indirect CAD alignment pipeline used in prior work [14, 39, 3, 2].

Given the input image, the pipeline first extracts an object mask and computes per-pixel descriptors. The same descriptors are also extracted for the candidate CAD model. Image-to-CAD correspondences are then established in feature space, lifted to 3D using monocular depth estimates, and passed to a RANSAC-based Procrustes solver to estimate the alignment in Equation 1. SUFLECA improves two key modules of this pipeline: the feature model, which is learned at scale and unified across real and synthetic data as described in Section III-C, and the correspondence estimation module, which adds geometric consistency as described in Section III-D. Throughout this paper, zero-shot refers specifically to the alignment stage: object detection and segmentation, depth estimation, and CAD retrieval are treated as upstream components inherited from prior work.

III-C Scaling Unified NOC-aligned Feature Learning

Multi-Dataset Setup  We adopt a scaling paradigm similar to [15], focused primarily on the indoor domain and augmented with synthetic data. Since correspondence for CAD alignment occurs between a real image and a synthetic CAD render, we augment, wherever CAD annotations are available, each real image with a synthetic counterpart in which the CAD objects are rendered at approximately the same pose. The synthetic counterparts are rendered with randomized backgrounds and object textures, with some objects occasionally replaced by visually similar ones. This yields a larger and more balanced training set with comparable numbers of real and synthetic samples, sharing similar occlusion patterns and scene context, which encourages the model to learn domain-invariant features. Compared with  [3], which trains on 300K synthetic images of single objects across 9 categories, our dataset comprises 674K images from 12 datasets (see Table I) with an average of 3.38 annotated objects per frame across a broader range of categories, providing denser and more diverse supervision. Furthermore, while most datasets we train on originate from SLAM-focused camera trajectories, the inclusion of casual imagery [35, 11, 40] further broadens the data distribution.

NOC Derivation and Verification  Most datasets do not provide NOC maps directly, but instead supply object poses in 3D from which NOCs must be derived. To this end, we render the ground-truth object onto the image to obtain an approximate bounding box, from which a mask is extracted using SAM2 [27]. The masked region is then backprojected into 3D using available depth maps or a monocular metric depth estimator [20], after which NOCs are computed following [15] (see the middle row in Figure 2). To ensure annotation quality, we verify the computed NOCs against the overlaid CAD NOCs and reject object instances whose mean error exceeds a threshold. Frames for which the total number of valid NOC pixels falls below a threshold are discarded entirely. For datasets derived from high frame-rate cameras [42, 10, 41], we subsample frames at regular intervals and additionally enforce a minimum camera pose displacement to ensure sufficient viewpoint diversity.

Model Architecture  As shown in Figure 3, our model builds upon a pretrained perception encoder, kept frozen during training, that provides rich patch-level features from images. These patch features are upsampled using a Dense Prediction Transformer (DPT) [25], which fuses multi-scale outputs from different layers of the encoder to produce denser feature maps of dimension . The DPT features are decoded by a lightweight NOC head: a binned classifier [36, 15] that partitions each coordinate axis (, , ) of the NOC space into uniform bins, estimating value as the expectation over the bin probabilities. This supervision encourages the DPT features to be spatially smooth and geometrically meaningful. The training loss comprises a categorical cross-entropy term between the predicted bin probabilities and the ground-truth bin, and an regression term on the expected NOC,

(2)

where is the cross-entropy loss, is the predicted NOC value, is the ground-truth NOC value, and is a weighting coefficient. The NOC head is discarded at test time, and the -normalized DPT features are used as descriptors for correspondence estimation in SUFLECA. Optionally, these features can be concatenated with the output layer features of the vision backbone, following [3], to mitigate forgetting caused by weak supervision over a limited set of categories. We refer to this variant as SUFLECA-blend.

III-D Geometrically-consistent Correspondence Estimation

To align a CAD model to a masked image , we start by featurizing the image using the aforementioned feature model , yielding per-patch features interpolated to pixel level, . Unlike prior works that aggregate 2D features across multiple renders of to obtain 3D features over a mesh [3, 2], a process that is both arbitrary and lossy, we keep the matching task entirely within the 2D domain. To this end, we maintain a small set of pre-rendered views of with precomputed features and point maps. Perception encoders have been shown to encode global pose information in an image [39], which we exploit to select the closest render to the query image prior to estimating correspondences. For that, we start by using a monocular metric depth estimator [20] to lift the masked image pixels into a 3D point cloud , and apply Farthest Point Sampling (FPS) to obtain a representative subset of points for correspondence estimation. Then, given the set of pre-rendered views from distinct fixed viewpoints, we select the view that maximises the mean nearest-neighbor cosine similarity over ,

(3)

where is the precomputed feature at pixel of view and is the feature at pixel in .

Using the selected render feature map and image feature map , we need to estimate correspondences between the lifted render and observed object pixels. While nearest-neighbor matching suffices for the aforementioned coarse view selection, it is unreliable for correspondence estimation, as it produces geometrically inconsistent and many-to-one matches. Mutual nearest-neighbor matching reduces these ambiguities but is overly conservative. We therefore adopt mutual -nearest-neighbors matching, retaining a correspondence only when the render and image descriptors lie within each other’s top- neighbors, with duplicates resolved by greedy assignment to the highest-scoring source match. However, this only considers the semantic feature space and does not enforce geometric consistency; and since the transformation includes anisotropic scaling, pairwise distance preservation is not directly applicable. Therefore, we first estimate an approximate scale to normalize the correspondence set before applying pairwise distance preservation. Concretely, let be a set of putative correspondences, where are source points from and are target points from . We seek a boolean inlier mask that retains correspondences geometrically consistent with the unknown 9D transformation. Denoting by the squared anisotropic scale vector, we obtain the following from pairwise differences,

(4)

which yields a linear system in ,

(5)

Solving this system in the presence of outliers requires robust initialisation. To this end, we obtain an isotropic scale estimate by computing the per-pair isotropic log-scale over all pairs and taking the histogram mode, assuming that inlier pairs coarsely concentrate in a common bin. This gives the isotropic log-scale , from which we initialise . From this, we refine the anisotropic log-scales via Iteratively Reweighted Least Squares (IRLS). Since this estimate degrades when correspondences do not cover all axes sufficiently, we shrink towards the more robust isotropic estimate, leading to

(6)

where controls the trust placed in the anisotropic estimate. We then build a consensus matrix , whose entries compare the observed squared distance against the value predicted by ,

(7)

where is the relative inlier threshold. The principal eigenvector of scores each correspondence by its consistency with the inlier group, following which correspondences are retained by,

(8)

where is the relative gating threshold. The -masked set with cardinality is passed to a space-partitioning RANSAC [5] that uses Procrustes registration under anisotropic scaling to recover the final alignment.

Alignment quality  Unlike prior works [14, 3], which use the CAD retrieval score as a proxy for alignment quality, SUFLECA estimates from the registration residuals, providing a more principled measure of alignment quality. The information matrix is computed analytically from the Jacobians of the registration residuals with respect to the parameter vector , where is a left rotation perturbation, is an additive translation, and is an additive perturbation on the diagonal scale. The residual for each correspondence is , with per-correspondence Jacobian,

(9)

where is the rotated and scaled source point and denotes the skew-symmetric cross-product matrix. The information matrix is then

(10)

from which the alignment quality score is .

IV Evaluation

媒体内容 · 前往原文查看

Method	Sup.	Category-specific accuracy (%)									Average (%)
		bathtub	bed	bin	bkshlf	cabinet	chair	display	sofa	table	cat.	inst.
ScanNet25k [10] – Vid2CAD [21] NMS Protocol
ROCA [14]	✓	20.8	8.6	26.3	9.0	13.1	39.9	24.6	10.6	12.7	18.4	25.0
SPARC [18]	✓	25.8	25.7	24.6	14.2	20.8	51.5	17.8	28.3	15.4	24.9	31.8
MultiObj-SPARC [19]	✓	25.8	34.3	44.8	17.0	19.2	64.8	5.8	35.4	25.5	30.3	40.3
CosCAD [38]	✓	24.9	18.9	31.1	20.8	33.1	45.4	27.5	19.9	24.6	27.4	33.2
DINOv3-L [34]	–	7.5	7.1	6.5	4.7	6.5	24.7	11.0	29.2	8.7	11.8	15.1
DUNE-B [31]	–	6.7	2.9	8.6	2.4	6.9	28.6	12.6	23.9	10.1	11.4	16.6
Diorama [39]	–	6.7	1.4	19.8	3.8	4.2	16.7	11.5	9.0	6.0	8.7	11.3
FoundationPose (9D) [37, 3]	*	20.0	22.9	27.6	0.9	3.1	41.8	23.6	15.0	17.5	19.2	25.7
ZeroCAD (w/o refine) [3]	*	16.7	10.0	10.8	15.1	7.3	46.8	16.2	31.0	10.7	18.3	26.0
ZeroCAD [3]	*	16.7	18.6	22.8	12.7	9.2	49.3	24.1	38.1	16.5	23.1	30.1
SUFLECA (Ours)	*	23.3	38.6	27.6	18.9	24.6	64.4	31.4	40.7	30.7	33.4	42.3
SUFLECA-S (Ours)	*	21.7	28.6	30.2	20.3	22.7	60.5	22.0	41.6	28.1	30.6	39.5
SUFLECA-blend (Ours)	*	22.5	24.3	28.5	18.9	22.7	59.4	24.1	44.3	25.5	30.0	38.5
DiffCAD [13] Split
DiffCAD (GT) [13]	*	–	27.1	–	24.4	33.0	65.9	–	46.3	18.3	35.8	41.9
DINOv3-L [34]	–	–	5.2	–	9.2	12.5	25.1	–	23.8	5.9	13.6	16.0
DUNE-B [31]	–	–	3.2	–	9.2	13.1	26.0	–	23.4	8.7	14.0	17.0
Diorama [39]	–	–	0.7	–	5.4	13.1	33.5	–	6.5	12.2	11.9	18.4
DiffCAD [13]	*	–	7.7	–	7.6	10.4	31.1	–	15.3	4.3	12.7	16.7
ZeroCAD [3]	*	–	2.8	–	8.2	12.0	41.3	–	21.4	9.8	15.9	20.9
SUFLECA (Ours)	*	–	23.2	–	21.7	34.3	71.1	–	41.4	24.9	36.1	44.8
SUFLECA-S (Ours)	*	–	20.0	–	23.9	29.6	67.6	–	38.7	21.8	33.6	41.7
SUFLECA-blend (Ours)	*	–	23.9	–	22.8	35.4	71.1	–	42.2	24.1	36.6	44.9

IV-A Implementation Details

Our feature model uses DUNE-B [31] as the frozen perception encoder and produces DPT patch features with . These are fed to the NOC head, which consists of two shallow convolutional layers and classifies each coordinate into bins. We train for epochs using AdamW with and a learning rate of . We also train a smaller variant, SUFLECA-S, which uses DUNE-S [31] as the encoder with and . For alignment, we maintain pre-rendered fixed views per CAD model, and set , , , , , and . All hyperparameters are empirically determined and remain consistent across all experiments, conducted on a workstation with an NVIDIA RTX 5090 GPU.

IV-B Methodology

We evaluate SUFLECA across complementary settings. ScanNet25k [14] serves as the main benchmark for alignment accuracy, while CO3D [28] is adapted to assess generalization to categories not observed during weak supervision (Section IV-D). We further include ablation studies to isolate the contributions of our design choices (Section IV-E), analyze robustness to inexactly retrieved CAD models (Section IV-F), and perform an efficiency analysis (Section IV-G). While our comparisons primarily focus on zero-shot methods, we also include results from supervised methods on ScanNet25k for reference. We also report results with the smaller SUFLECA-S variant (from Section IV-A) and the SUFLECA-blend variant (from Section III-C) of our method.

媒体内容 · 前往原文查看

Method	Seen categories (CAD retrieval accuracy: 77.5%)				Unseen categories (CAD retrieval accuracy: 52.5%)
	3D-IoU ↑	ICP-Rot ↓	ADD-S ↓	ADD-S@0.1↑	3D-IoU ↑	ICP-Rot ↓	ADD-S ↓	ADD-S@0.1↑
DINOv3-L [34]	36.21	27.32	6.54	71.50	8.77	40.44	10.31	46.50
DUNE-B [31]	23.40	35.81	8.48	63.50	4.46	44.44	11.84	34.50
Diorama [39]	39.41	21.06	5.79	82.00	39.12	14.54	6.33	81.50
SUFLECA (Ours)	62.63	10.24	3.80	84.00	48.89	16.64	5.86	78.75
SUFLECA-S (Ours)	62.50	10.31	3.71	85.50	46.42	16.44	6.49	74.00
SUFLECA-blend (Ours)	62.71	10.86	3.77	86.00	49.50	12.93	5.56	81.25

IV-C Datasets

ScanNet25k [14] are a dataset of images derived from the ScanNet [10] dataset with Scan2CAD annotations [4]. An alignment on this dataset is considered correct when translation, rotation, and scale errors are below , , and , respectively. We report both category- and instance-averaged accuracy over the nine categories on the validation split. We follow the NMS-based evaluation protocol [21], ranking detections by the product of the ROCA semantic retrieval score [14] and defined in Equation 10. To ensure zero-shot evaluation, we exclude all ScanNet-derived images during weak supervision for this dataset. Following [3], all methods use ROCA bounding boxes and CAD retrievals [14], SAM2 masks [27], and depth from a monocular metric depth estimator [20] fine-tuned on the ScanNet25k training split.

The DiffCAD split [13] covers six of the ScanNet25k object categories, evaluated without NMS and using non-fine-tuned depth. This split was introduced by DiffCAD [13], whose priors are available only for six categories.

CO3D [28] is an object-centric dataset that we adapt for the inexact CAD fitting setting. We convert its COLMAP point clouds to metric scale by selecting the depth scale with the lowest error against monocular depth estimates. We also simulate occlusions by placing a rectangular occluder within the GroundedSAM [29] bounding box. For CAD retrieval, we use SAM3D [8] to generate category-level CAD pools and retrieve candidates with OSCAR [24]. We restrict the categories to two domains: seen categories (chair, couch), which appear during weak supervision of SUFLECA, and unseen categories (toaster, hairdryer, microwave, suitcase), which are used to evaluate generalization performance. We sample 100 frames per category across 208 objects. Since CO3D lacks ground-truth poses and SAM3D meshes have inconsistent canonical orientations, we report 3D-IoU, ICP-Rot, ADD-S, and ADD-S@0.1 [8] for comparison.

IV-D Results

Table II reports alignment accuracy on ScanNet25k and the respective DiffCAD split. Among zero-shot methods, SUFLECA achieves category-averaged and instance-averaged accuracy using the NMS protocol, surpassing ZeroCAD [3] by and percentage points respectively, and outperforming FoundationPose [37] by a larger margin. This can be attributed to two factors: broader and more diverse NOC-supervised training data, which reduces the domain gap to real images, and our correspondence estimation, which produces geometrically consistent matches. Notably, SUFLECA also surpasses fully supervised methods, achieving the best or second-best accuracy across seven of the nine categories and outperforming MultiObj-SPARC [19] despite the latter being trained with 9D pose supervision on ScanNet. On the DiffCAD split, competing zero-shot methods perform poorly because alignment errors are not suppressed through NMS. In contrast, SUFLECA more than doubles the performance of ZeroCAD [3], achieving gains of and percentage points in category- and instance-averaged accuracy, respectively. Moreover, SUFLECA even outperforms the oracle variant of DiffCAD [13], which uses ground-truth pose to select the best result from eight hypotheses. Notably, the other SUFLECA variants also achieve competitive performance, outperforming competing zero-shot methods by a large margin and remaining comparable to supervised methods. SUFLECA-S attains lower overall accuracy than the base model, but still substantially outperforms other zero-shot methods, highlighting the effectiveness of its compact feature space. SUFLECA-blend performs on par with, or slightly below, SUFLECA, since the added DUNE-B features primarily improve generalization, which is less critical for the common object categories in ScanNet25k.

On CO3D, results are obtained under occluded masks and inexact retrieved CADs, posing significant front–back and left–right geometric ambiguities, as shown in Figure 4. SUFLECA grounds objects in 3D substantially better than competing methods in this challenging setting, with its features more effectively able to disambiguate such cases. In seen categories, all variants of SUFLECA perform comparably and outperform competing methods, highlighting the effectiveness of our weak supervision. For unseen categories, SUFLECA-blend generally outperforms other methods, as it compensates for the limited supervision coverage for uncommon objects at the cost of a larger feature representation.

IV-E Ablation Study

We ablate the main design choices, and report alignment accuracy on ScanNet25k (NMS protocol) in Table IV.

Training data  This ablation evaluates the impact of including in-distribution training data. Training on the complete collected dataset, including ScanNet-derived images, improves performance from to . This modest gain of only percentage points suggests that SUFLECA benefits from additional data, but it already learns strong features without in-distribution images.

Correspondence estimation  We also ablate the correspondence estimation strategy. Using nearest-neighbor matching alone yields the largest drop, reducing accuracy to . Adding mutual consistency improves over nearest-neighbor matching to , but still remains below the full model. Using mutual -nearest-neighbors matching further improves performance to , but still underperforms the full pipeline that applies geometric consistency. These results show that both mutual matching and geometric consensus are important for accurate alignment.

NMS scoring  We further evaluate the score used to rank alignments during NMS. Replacing our combined score based on with the ROCA retrieval score reduces accuracy from to . This drop of percentage points shows that semantic retrieval confidence alone is not a reliable proxy for alignment quality. We also note that the render-selection score from Equation 3 performs even worse, reducing accuracy to , since it is coarse and hence sensitive to the choice of render viewpoints.

媒体内容 · 前往原文查看

Variant	cat.	inst.
SUFLECA (Ours)	33.4	42.3
Training data
ScanNet included during training	34.4↑	42.8↑
Correspondence estimation
Nearest neighbor [14, 3]	26.3↓	34.2↓
Mutual nearest-neighbor	30.7↓	39.4↓
Mutual -nearest-neighbors w/o filtering	32.0↓	41.0↓
NMS scoring
ROCA retrieval score [14]	30.4↓	39.3↓
Render selection score –  Equation 3	29.2↓	38.8↓

IV-F Robustness to Inexact CADs

We further analyze how alignment accuracy depends on the quality of the retrieved CAD model. As shown in  Table V, replacing ground-truth Scan2CAD annotations [4] with zero-shot retrieval using GroundedSAM [29] w/ OSCAR [24] leads to a gap of more than percentage points in both category- and instance-averaged alignment accuracy. This is primarily due to the low instance retrieval accuracy of (most retrieved CADs are inexact), resulting from heavy occlusion, appearance differences, and annotation bias. This indicates that zero-shot CAD retrieval from real-world observations remains an open problem and a key bottleneck for a fully zero-shot CAD fitting pipeline. Nevertheless, as we move from the ground-truth oracle to supervised ROCA [14] and then to fully zero-shot retrieval, the drop in instance retrieval accuracy is much steeper than the corresponding drop in alignment accuracy, suggesting that SUFLECA is moderately robust to CAD inexactness.

IV-G Efficiency Analysis

Setup  We compare the CAD alignment runtime and peak VRAM usage of different zero-shot methods, with the results reported in Table VI. We exclude detection, segmentation, depth estimation, and retrieval, as these stages are consistent across all methods. Moreover, since ZeroCAD [3] is not open-source, we implement a runtime-only proxy, termed as ZeroCAD*, to estimate a lower bound on its true runtime.

媒体内容 · 前往原文查看

Detection and retrieval	Retrieval acc.	Alignment acc.
		cat.	inst.
Scan2CAD annotations (GT) [4]	100.0	43.0	54.2
ROCA [14]	34.3	33.4	42.3
GroundedSAM [29] w/ OSCAR [24]	3.8	22.5	33.1

媒体内容 · 前往原文查看

Method		VRAM	Per-inst.
		(MB) ↓	(s) ↓
DINOv3-L [34]	1024	3394	0.94
DUNE-B [31]	768	2696	0.82
Diorama [39]	1024	3426	2.39
DiffCAD [13]	–	3146	13.1
FoundationPose [37]	–	8544	0.61
ZeroCAD∗ (w/o refine) [3]	2048	4988	1.30
ZeroCAD∗ [3]	2048	5158	3.77
SUFLECA (Ours)	384	2178	0.53
SUFLECA-S (Ours)	256	1556	0.49
SUFLECA-blend (Ours)	1152	3972	0.76
SUFLECA – CAD alignment timing by component (ms)
Featurizer (once per image)			86.6
Target render selection			43.1
Correspondence estimation and filtering			68.7
3D registration w/ RANSAC (300k iterations)			332.0

Discussion  SUFLECA achieves lower runtime and VRAM usage than competing methods owing to its compact feature representation (2.0 to 5.3 smaller than those of competing methods), and the absence of iterative pose refinement stages used in [3, 37]. The smaller SUFLECA-S further improves runtime and VRAM usage without a significant sacrifice in performance, while SUFLECA-blend improves generalization at the cost of increased runtime and VRAM usage. Notably, the majority of the runtime in SUFLECA is spent in RANSAC, enabling a straightforward accuracy-speed trade-off by adjusting the maximum iteration count.

V Conclusion

We introduced SUFLECA, a weakly-supervised zero-shot method for 9D CAD alignment from images. Its two core contributions, scaled NOC-supervised feature learning across diverse real and synthetic data and a mutual -nearest-neighbors matching algorithm with geometric consensus filtering, jointly address the domain gap and spatial inconsistencies that limit prior approaches. SUFLECA uses compact, geometrically discriminative features to enable sub-second CAD alignment without pose-annotated training data or iterative refinement, achieving state-of-the-art performance on ScanNet25K and, for the first time, surpassing supervised methods on this benchmark. Despite these advances, important limitations remain. First, alignment quality is still limited by CAD retrieval accuracy, making robust open-set retrieval from occluded real-world observations an important direction for future work. Second, SUFLECA is currently restricted to indoor scenes and common object categories, motivating the inclusion of outdoor scenes and less common categories to improve scalability and generalization.

References

• [1] A. Ahmadyan, L. Zhang, A. Ablavatski, J. Wei, and M. Grundmann (2021) Objectron: a large scale dataset of object-centric videos in the wild with pose annotations. Proc. IEEE Conference on Computer Vision and Pattern Recognition. Cited by: §II-A, TABLE I.
• [2] D. Antić, G. Paschalidis, S. Tripathi, T. Gevers, S. K. Dwivedi, and D. Tzionas (2025) SDFit: 3D object pose and shape by fitting a morphable SDF to a single image. In Proc. IEEE/CVF International Conference on Computer Vision, Cited by: §I, §II-C, §III-B, §III-D.
• [3] P. Arsomngern, S. Khwanmuang, M. Nießner, and S. Suwajanakorn (2025) Zero-shot inexact cad model alignment from a single image. In Proc. IEEE International Conference on Computer Vision, Cited by: Figure 1, Figure 1, §I, §I, §II-A, §II-C, §III-B, §III-C, §III-C, §III-D, §III-D, §IV-C, §IV-D, §IV-G, §IV-G, TABLE II, TABLE II, TABLE II, TABLE II, TABLE IV, TABLE VI, TABLE VI, TABLE VI, TABLE VI.
• [4] A. Avetisyan, M. Dahnert, A. Dai, M. Savva, A. X. Chang, and M. Nießner (2019) Scan2cad: learning cad model alignment in rgb-d scans. In Proc. Conference on computer vision and pattern recognition, Cited by: §IV-C, §IV-F, TABLE V.
• [5] D. Barath (2025) SupeRANSAC: one ransac to rule them all. arXiv preprint arXiv:2506.04803. Cited by: §III-D.
• [6] G. Baruch, Z. Chen, et al. (2021) ARKitscenes - a diverse real-world dataset for 3d indoor scene understanding using mobile RGB-d data. In Conference on Neural Information Processing Systems, Cited by: TABLE I.
• [7] A. X. Chang, T. Funkhouser, et al. (2015) Shapenet: an information-rich 3d model repository. arXiv preprint arXiv:1512.03012. Cited by: TABLE I.
• [8] X. Chen et al. (2026) Sam 3d: 3dfy anything in images. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition, Cited by: §IV-C.
• [9] A. Collet, D. Berenson, S. S. Srinivasa, and D. Ferguson (2009) Object recognition and full pose registration from a single image for robotic manipulation. In IEEE international conference on robotics and automation, Cited by: §I.
• [10] A. Dai, A. X. Chang, M. Savva, M. Halber, T. Funkhouser, and M. Nießner (2017) ScanNet: richly-annotated 3d reconstructions of indoor scenes. In IEEE Computer Vision and Pattern Recognition, Cited by: §III-C, TABLE I, TABLE I, TABLE I, §IV-C, TABLE II.
• [11] M. Everingham, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman (2010) The pascal visual object classes (voc) challenge. International journal of computer vision. Cited by: §III-C, TABLE I.
• [12] H. Fu et al. (2021) 3d-front: 3d furnished rooms with layouts and semantics. In Proc. International Conference on Computer Vision, Cited by: TABLE I.
• [13] D. Gao, D. Rozenberszki, S. Leutenegger, and A. Dai (2024) DiffCAD: weakly-supervised probabilistic cad model retrieval and alignment from an rgb image. ACM Trans. Graph.. Cited by: §II-C, §IV-C, §IV-D, TABLE II, TABLE II, TABLE II, TABLE VI.
• [14] C. Gümeli, A. Dai, and M. Nießner (2022) Roca: robust cad model retrieval and alignment from a single image. In Proc. IEEE/CVF conference on Computer Vision and Pattern Recognition, Cited by: Figure 1, Figure 1, §I, §I, §II-B, §III-B, §III-D, TABLE I, TABLE I, §IV-B, §IV-C, §IV-F, TABLE II, TABLE IV, TABLE IV, TABLE V.
• [15] A. Krishnan, A. Kundu, K. Maninis, J. Hays, and M. Brown (2024) Omninocs: a unified nocs dataset and model for 3d lifting of 2d objects. In Proc. European Conference on Computer Vision, Cited by: §II-A, §III-C, §III-C, §III-C, TABLE I, TABLE I, TABLE I.
• [16] A. Kundu, Y. Li, and J. M. Rehg (2018) 3d-rcnn: instance-level 3d object reconstruction via render-and-compare. In Proc. IEEE conference on computer vision and pattern recognition, Cited by: §II-B.
• [17] W. Kuo, A. Angelova, T. Lin, and A. Dai (2020) Mask2CAD: 3d shape prediction by learning to segment and retrieve. In Proc. European Conference on Computer Vision, Cited by: §I, §II-B.
• [18] F. Langer, G. Bae, I. Budvytis, and R. Cipolla (2022) SPARC: sparse render-and-compare for cad model alignment in a single rgb image. In Proc. British Machine Vision Conference, Cited by: §I, §II-B, TABLE II.
• [19] F. Langer, R. Cipolla, and I. Budvytis (2023) Sparse multi-object render-and-compare. In Proc. British Machine Vision Conference, Cited by: §I, §II-B, §IV-D, TABLE II.
• [20] B. Ma, J. Yang, D. Di, X. Zhang, J. Cui, H. Li, Y. Xie, and W. Chen (2026) MetricAnything: scaling metric depth pretraining with noisy heterogeneous sources. arXiv preprint arXiv:2601.22054. Cited by: §III-C, §III-D, §IV-C.
• [21] K. Maninis, S. Popov, M. Niesner, and V. Ferrari (2023) Vid2CAD: CAD Model Alignment Using Multi-View Constraints From Videos . IEEE Transactions on Pattern Analysis & Machine Intelligence. Cited by: §IV-C, TABLE II.
• [22] K. Maninis, S. Popov, M. Nießner, and V. Ferrari (2023) Cad-estate: large-scale cad model annotation in rgb videos. In Proc. IEEE/CVF International Conference on Computer Vision, Cited by: TABLE I.
• [23] Y. Nie, X. Han, S. Guo, Y. Zheng, J. Chang, and J. J. Zhang (2020) Total3DUnderstanding: joint layout, object pose and mesh reconstruction for indoor scenes from a single image. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition, Cited by: §II-B.
• [24] T. Pulli, J. Weibel, P. Hönig, M. Hirschmanner, M. Vincze, and A. Holzinger (2026) OSCAR: open-set cad retrieval from a language prompt and a single image. arXiv preprint arXiv:2601.07333. Cited by: §IV-C, §IV-F, TABLE V.
• [25] R. Ranftl, A. Bochkovskiy, and V. Koltun (2021) Vision transformers for dense prediction. In Proc. IEEE/CVF international conference on computer vision, Cited by: §III-C.
• [26] Y. Rao, S. Ainetter, S. Stekovic, V. Lepetit, and F. Fraundorfer (2025) Leveraging automatic cad annotations for supervised learning in 3d scene understanding. arXiv preprint arXiv:2504.13580. Cited by: TABLE I, TABLE I.
• [27] N. Ravi, V. Gabeur, et al. (2024) SAM 2: segment anything in images and videos. arXiv preprint arXiv:2408.00714. Cited by: §III-C, §IV-C.
• [28] J. Reizenstein, R. Shapovalov, P. Henzler, L. Sbordone, P. Labatut, and D. Novotny (2021) Common objects in 3d: large-scale learning and evaluation of real-life 3d category reconstruction. In International Conference on Computer Vision, Cited by: §IV-B, §IV-C, TABLE III, TABLE III.
• [29] T. Ren, S. Liu, A. Zeng, J. Lin, K. Li, H. Cao, J. Chen, X. Huang, Y. Chen, F. Yan, et al. (2024) Grounded sam: assembling open-world models for diverse visual tasks. arXiv preprint arXiv:2401.14159. Cited by: §IV-C, §IV-F, TABLE V.
• [30] M. Roberts, J. Ramapuram, A. Ranjan, A. Kumar, M. A. Bautista, N. Paczan, R. Webb, and J. M. Susskind (2021) Hypersim: A photorealistic synthetic dataset for holistic indoor scene understanding. In International Conference on Computer Vision, Cited by: TABLE I.
• [31] M. B. Sarıyıldız, P. Weinzaepfel, T. Lucas, P. De Jorge, D. Larlus, and Y. Kalantidis (2025) Dune: distilling a universal encoder from heterogeneous 2d and 3d teachers. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition, Cited by: §II-A, §IV-A, TABLE II, TABLE II, TABLE III, TABLE VI.
• [32] L. Shaikewitz, T. Nguyen, and L. Carlone (2026) Category-level object shape and pose estimation in less than a millisecond. IEEE International Conference on Robotics and Automation. Cited by: §II-C.
• [33] J. Shi, H. Yang, and L. Carlone (2023) Optimal and robust category-level perception: object pose and shape estimation from 2-d and 3-d semantic keypoints. IEEE Transactions on Robotics. Cited by: §I, §II-C.
• [34] O. Siméoni, H. V. Vo, et al. (2025) Dinov3. arXiv preprint arXiv:2508.10104. Cited by: TABLE II, TABLE II, TABLE III, TABLE VI.
• [35] X. Sun, J. Wu, X. Zhang, Z. Zhang, C. Zhang, T. Xue, J. B. Tenenbaum, and W. T. Freeman (2018) Pix3D: dataset and methods for single-image 3d shape modeling. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: §III-C, TABLE I.
• [36] H. Wang, S. Sridhar, J. Huang, J. Valentin, S. Song, and L. J. Guibas (2019) Normalized object coordinate space for category-level 6d object pose and size estimation. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: §I, §II-A, §III-C, TABLE I.
• [37] B. Wen, W. Yang, J. Kautz, and S. Birchfield (2024) Foundationpose: unified 6d pose estimation and tracking of novel objects. In Proc. IEEE conference on Computer Vision and Pattern Recognition, Cited by: §IV-D, §IV-G, TABLE II, TABLE VI.
• [38] Z. Wen, H. Chen, Z. Zhu, Z. Wei, L. Nan, and M. Wei (2025) CosCAD: cross-modal cad model retrieval and pose alignment from a single image. In International Conf. on Computational Visual Media, Cited by: §II-B, TABLE II.
• [39] Q. Wu, D. Iliash, D. Ritchie, M. Savva, and A. X. Chang (2025) Diorama: unleashing zero-shot single-view 3d indoor scene modeling. In Proc. IEEE/CVF International Conference on Computer Vision, Cited by: §I, §II-C, §III-B, §III-D, TABLE II, TABLE II, TABLE III, TABLE VI.
• [40] Y. Xiang, W. Kim, W. Chen, J. Ji, C. Choy, H. Su, R. Mottaghi, L. Guibas, and S. Savarese (2016) ObjectNet3D: a large scale database for 3d object recognition. In European Conference Computer Vision, Cited by: §III-C, TABLE I.
• [41] C. Yeshwanth, Y. Liu, M. Nießner, and A. Dai (2023) Scannet++: a high-fidelity dataset of 3d indoor scenes. In Proc. IEEE/CVF International Conference on Computer Vision, Cited by: §III-C, TABLE I.
• [42] T. Zhou, R. Tucker, J. Flynn, G. Fyffe, and N. Snavely (2018) Stereo magnification: learning view synthesis using multiplane images. In SIGGRAPH, Cited by: §III-C, TABLE I.