PromptMoGeMetric Geometry from One Photo and a Phone LiDAR, Computed on the Device

Method

Starting point: MoGe‑3

MoGe‑3 [1] takes one image and predicts, for every pixel, a point in 3‑D. It does not predict metric coordinates directly. Instead it predicts a viewing ray (x/z, y/z) and a log‑depth log z that is only defined up to a global scale and a shift along the viewing direction; this is called an affine-invariant point map, and it is what lets one model train on datasets with wildly different scales. The architecture has three parts. A DINOv2 ViT‑L/14 encoder turns a 420×560 image into a 30×40 grid of 1 200 tokens. A convolutional neck without normalisation layers decodes those tokens over five levels, doubling the resolution each time until it reaches 480×640, and separate heads read out points, surface normals and a validity mask.

The part that distinguishes MoGe‑3 from its predecessors is its refiner. A 2‑D decoder mixes features between pixels that are neighbours in the image, even when one lies on a chair leg and the other on the wall two metres behind it, and this is what blurs depth edges. MoGe‑3 instead lifts every pixel into a voxel at (row, column, round(256·log z)) and runs a sparse 3‑D U‑Net over the occupied voxels. Pixels that are far apart in depth land in different voxels and stop exchanging information, so the refiner can sharpen edges instead of smearing them. Each refinement step outputs a correction to log‑depth and leaves the viewing rays untouched.

Could we skip the prompt entirely and just rescale MoGe‑3's output to match the LiDAR? We tried: with the best possible robust scale and shift, stock MoGe‑3 reaches 3.5 % error on the pixels where the sensor is confident, which is worse than the raw sensor's 1.9 %. A single scale and shift cannot fix the slow, scene-wide distortions in a monocular prediction. The sensor has to go into the network, where it can correct the geometry region by region.

The prompt

The LiDAR arrives as a 256×192 depth map in metres and a confidence map with three levels. We turn it into four channels. The first is log‑depth with the frame's median subtracted, which removes the overall scale of the scene (the network predicts scale-free geometry anyway) while keeping the depth differences that describe its shape. The second marks which pixels have a reading at all, and the third is the sensor's confidence. The fourth is an uncertainty value that combines confidence with distance, rising between 3 and 5 m where phone LiDAR becomes unreliable; because it depends on absolute distance it also stops the prompt from being completely blind to scale. A pixel with no reading is encoded as “no depth, invalid, no confidence, fully uncertain”, so an empty prompt is a well-defined input that the network learns to treat as “use the image only”.

This prompt map feeds two small convolutional encoders (Figure 1). A prompt stem reduces it to the 30×40 token grid, and its features are added to the image tokens before transformer blocks 0, 4 and 8. A prompt pyramid processes the prompt at full output resolution and adds one feature map to each of the five levels of the neck.

Zero-initialised gates, not zero-initialised projections

A new input path should not disturb a pretrained model at the start of training. The usual way to guarantee that, used by ControlNet [9] and by PromptDA [2], is to initialise the projection that adds the new features to zero. With a frozen ViT this did not work for us. Our first run diverged within 160 steps. Lowering the learning rate five-fold stopped the divergence but produced a slow drift instead: the projection weights grew 35‑fold in 160 steps, and the scale of the predicted geometry slid steadily away from the sensor, all while the gradient norm looked perfectly healthy.

The explanation is a property of the Adam optimiser. Adam moves every weight by roughly the learning rate at every step, almost regardless of how informative its gradient is. A zero-initialised 256×1024 projection has 262 144 weights that all start moving at once, so what it injects into the transformer's residual stream is, early on, a growing random walk — and the frozen backbone has no way to adapt to it. We therefore initialise the projection normally and multiply its output by a per-channel gate that starts at zero. The injection is now controlled by 1 024 numbers instead of a quarter of a million, it opens smoothly, and because a closed gate passes no gradient to the layers behind it, the gates get their own, higher learning rate. A unit test checks that the untrained model reproduces MoGe‑3 bit for bit.

Which of the two injection paths matters? We trained three otherwise identical models from stock MoGe‑3. Without the neck pyramid, error rises by 17.8 % and gets worse in all 61 test rooms. Without the token injection, overall error is unchanged (+0.5 %), but thin structures get 2.2 % worse. So the pyramid carries most of the benefit, and the early token injection buys fine detail (ablations).

The metric attachment

The network's output is geometry without units. The metric attachment is the step that converts it into metres using the LiDAR, and it turned out to matter as much as the network itself — so much that any comparison between prompted depth models has to say how each one's output was attached. It has three stages: calibrate the sensor, fit a scale and shift, and optionally correct a smooth residual error across the frame.

Stage 1 — calibrating the sensor. Compared with laser-scan ground truth, ARKit's LiDAR depth reads slightly short, and by an amount that depends on distance: 3.0 % short at 0.35 m, shrinking to 0.4 % at 4.5 m. That pattern is what a fixed range offset in a time-of-flight sensor looks like, and indeed the correction is almost exactly

We store it as a 12-point lookup table of correction factor against log‑depth, fitted on training videos that share no rooms with any test data, and hold it constant outside 0.35–4.5 m. The important detail is where it is applied: once, to the sensor input, before the prompt is built and before anything is fitted to the sensor. A scale-and-shift fit applied afterwards cannot do the same job, because the bias is neither a pure scale nor a pure shift — calibrating the raw sensor with the best global scale and shift only brings its error from 1.93 % to 1.49 %, while the lookup table reaches 1.33 %. For a long time we mistook this for a limit of our architecture; it was a property of the sensor.

Stage 2 — a robust scale and shift. Let z be the network's scale-free depth and d the calibrated LiDAR depth. We look for the two numbers s and t that make s·z + t match d as closely as possible, which for a weighted least-squares criterion has a closed-form answer:

Three details make this reliable on real frames. First, the fit only uses pixels the sensor marks as high-confidence and the network's own mask marks as valid, and it is done at the sensor's resolution: the model's depth is averaged down to the 256×192 grid rather than the sparse sensor being interpolated up. Second, it is trimmed. The sensor is sometimes confidently wrong, and the model is sometimes wrong at depth edges, so after each solve we discard the 20 % of pixels with the largest relative error and solve again, four times in total. Third, if fewer than 32 usable pixels remain, the fit is skipped rather than trusted. The resulting (s, t) is applied to the full-resolution output. This is in the spirit of the alignment used to evaluate relative-depth models such as MiDaS [11], with the difference that we align to the sensor rather than to ground truth, so it is a legitimate part of inference.

Training through the fit. During training the same fit converts the network's output to metres before it is compared with ground truth. Our first implementation treated s and t as constants when back-propagating, and training collapsed: the loss rose from 0.030 to 0.073, the predicted log‑depth drifted upwards by 4.6 (a factor of a hundred in depth), and the fitted scale fell from 0.83 to 0.01 to compensate. The network had discovered that it could reduce its loss a little by changing its overall scale; the fit then absorbed the change at the next step, the gradient pointed the same way again, and the two chased each other indefinitely. We call this the gauge treadmill.

The cure is to let gradients flow through the final least-squares solve, while still choosing which pixels to trim from detached values. With the weights held fixed, rescaling the network's output by any a and b simply changes the fitted values to s/a and t − s·b/a and leaves the metric result identical. The loss therefore becomes exactly insensitive to the network's global scale and shift, the gradient in that direction vanishes (we measured 10⁻⁷), and the network is left to learn shape. Since nothing then pins the absolute scale of its output, a weak extra term ties the mean log‑depth to that of the frozen MoGe‑3 teacher.

Stage 3 — a per-frame quadratic gauge. Two numbers cannot remove an error that varies smoothly across the image, such as a slight tilt or bow. As an optional last step we therefore fit, on the confident pixels, a quadratic surface in image coordinates to the remaining log‑ratio between sensor and prediction,

using three rounds of robustly re-weighted least squares so that outliers do not steer it, and multiply the depth and the lateral coordinates by the result. Scaling all three coordinates together moves each point along its viewing ray, so the geometry stays consistent with the camera. On ARKit data this lowers confident-pixel error by a further 4.5 % and slightly improves fused reconstructions. We only use it at inference: supervising the network “up to a gauge” during training made it 1 % worse. And it should be switched off for sparse sensors, where six free parameters start fitting the sensor's own distortion (error 3.50 % with it, 3.25 % without, on a simulated 112-beam sensor).

How much is the attachment worth? The table below separates the contribution of the attachment from that of the network, on our three-capture development set. Calibration alone takes the raw sensor most of the way to the prompted models. PromptDA‑L improves from 1.37 % to 1.25 % when it is simply given our attachment. And if both networks are instead tied to the uncalibrated LiDAR with a plain scale and shift, they are indistinguishable. This is why every comparison below reports PromptDA both as released and “with our attachment”, meaning its output passed through exactly the three stages above.

A refiner that sees the sensor

At our token budget the stock refiner does almost nothing for metric accuracy: error after three refinement steps equals error after none, to four decimal places. It has no way of knowing where the sensor disagrees with the current estimate. We give it that information as two extra input channels per voxel. Before every step the LiDAR is re-fitted to the current depth (the same robust fit as above, run in the network's scale-free frame), and the refiner receives the log‑difference between sensor and estimate, together with a weight that is 1.0 for high-confidence readings, 0.58 for medium and 0.15 for low. The weight never reaches zero because ARKit's confidence is a heuristic and low-confidence readings are often correct. The projection that introduces these channels starts at zero, like every other new path.

With this change one refinement step lowers error where the sensor is weakest — by about 5 % in holes, 4 % in low-confidence regions and 3 % at depth edges — and also sharpens edges visibly. It has one consistent cost: thin structures get about 4 % worse, in 60 of our 61 test rooms. We ship one refinement step as the default and let the user choose none.

Training

What is trained

The DINOv2 backbone stays frozen in every model we release. We train the prompt stem, the prompt pyramid and their gates, MoGe‑3's neck and heads and, in a separate stage, the refiner — 24 million of the model's 374 million parameters. We did try to adapt the backbone as well, seven times in different ways: LoRA adapters, full fine-tuning at a small learning rate with a penalty for leaving the pretrained weights, and feature distillation. None improved accuracy on real data, and the more aggressive ones made the model forget more of what it knew about images without a prompt. With roughly 16 000 to 160 000 training frames there is simply not enough data to improve a backbone trained on hundreds of millions of images.

Data

Real training frames come from ARKitScenes [3], which pairs iPad video with the device's own LiDAR depth and confidence and with ground truth rendered from stationary Faro laser scans. Most of the development used about 16 000 frames from 800 videos; the final training stage uses a denser sampling of 157 000 frames from 1 570 videos. Rooms are split so that no test room, or any other video of it, is ever used for training.

One training sample in four is synthetic, from Hypersim [4], where depth and normals are exact. Synthetic frames have no LiDAR, so we simulate one — and simple simulations turned out to be too clean to be useful. We measured the real sensor's errors against laser scans and found that they are structured: readings are biased by +3.8 % on the near side of depth edges, the noise is correlated over about 8 pixels, and holes come in large blobs rather than speckles. We therefore trained a small network on 14 000 real LiDAR/laser pairs to reproduce these statistics, including the confidence map, and use it to render a plausible LiDAR frame for every synthetic image. Even so, synthetic data mainly helps synthetic benchmarks; its lasting contribution to real accuracy is exact supervision for normals and edges, which real laser scans cannot provide.

Teaching the network not to copy the sensor

The easiest way for a prompted model to reduce its loss is to pass the prompt straight through, since the sensor is right most of the time. A model that does this is useless exactly where it is needed. We saw the symptoms early: a prompted model's surfaces were rougher than stock MoGe‑3's (normal error 20.3° against 17.7°) because it was reproducing sensor noise, and it followed the sensor into its mistakes.

Several measures counter this. In a quarter of the samples the prompt is removed entirely and the network is trained to reproduce frozen MoGe‑3, which both preserves its image-only ability and teaches it what an empty prompt means. In the remaining samples we punch rectangular holes into the LiDAR, mark regions as low-confidence, and add depth noise that grows with the sensor's stated uncertainty, so the network learns that the prompt can be missing or wrong and that the confidence channel means something. Errors on pixels without a confident reading count four times as much. And a loss on surface normals, computed from the predicted depth, penalises reproducing sensor fuzz directly; with it, normal error falls to 16.9°.

Depth edges need special care, because the ground truth is least reliable there: laser-scan edges sit a median of 4.9 pixels away from the corresponding edges in the image. Training on them teaches the network to blur. We mask the ground truth within 3 pixels of its own depth discontinuities, and supervise edges instead with the gradient of the frozen image-only teacher, whose edges are aligned with the image by construction. This single change is responsible for most of our advantage on thin structures.

Losses and optimisation

The main loss is a log‑depth error against ground truth after the differentiable metric fit, truncated so that gross outliers in the ground truth do not dominate. A second term keeps the output, averaged over a neighbourhood of about 8 sensor pixels, within 0.3 % of the confident LiDAR: the network is free to add detail but not to drift away from measurements that are already good. Scale-invariant, multi-scale gradient and local-patch terms shape the relative geometry, and the frozen teacher supervises viewing rays, normals and the validity mask throughout.

We use AdamW with separate learning rates for the prompt path (5·10⁻⁵), its gates (2·10⁻⁴) and the decoder (2·10⁻⁵), a short warm-up and cosine decay, MoGe‑3's mixed-precision recipe, and a batch of four frames on a single 24–48 GB GPU. The released full-size model is the last of several stages that each start from the previous one; its final stage runs for 12 000 steps and takes about two and a half hours on an RTX 4090.

Running it on an iPad

When we first profiled the full model on an iPad Pro, module by module, it added up to 296 ms per frame, of which 82 ms went into merely building the voxel structure for one refinement step. Reaching 152 ms took two kinds of work: making the model smaller without changing what it computes, and building a runtime in which every part runs on the processor that is fastest at it.

Compression without an accuracy bill (Model A)

A cheaper prompt pyramid. In the original pyramid a single convolution, 256 channels in and out at the full 480×640 resolution, accounted for 84 % of the prompt path's 431 GFLOPs. It feeds the finest level of the neck, which is also where the prompt matters least: there the injected features amount to 0.25 % of the activations they are added to, against 71 % at the second-coarsest level. We redesigned the pyramid to be wide where the resolution is low and narrow where it is high, mirroring the neck. Because every tensor shape changes, no weights carry over; instead we train the new pyramid to reproduce the five feature maps of the old one, which takes two minutes. The result costs 29 GFLOPs and moves no metric by more than 0.5 %.

Exact rescaling for the Neural Engine. The Neural Engine computes in 16-bit floating point, and the neck and heads, having no normalisation layers, produce activations as large as 433 — enough to overflow its accumulators, which on the device turned 99.9 % of the predicted normals into NaNs. Retraining with bounded activations was one option. But these layers use ReLU, and ReLU has a convenient property: scaling its input by a positive constant scales its output by the same constant. So dividing a block's input weights and all its biases by α, and multiplying its output weights by α, leaves the result mathematically unchanged while shrinking every intermediate value. We apply this throughout (the prompt gates must be divided too, since the pyramid adds into the same stream), bringing the peak activation down to 8, and verify for every layer that its worst-case accumulation stays far below the overflow limit.

A smaller, int8 refiner. The refiner's coarsest level used 27 % of its convolution time on 0.8 % of the voxels, so we halve its width. All 31 wide layers are then trained with quantisation in the loop, simulating precisely the arithmetic of our Metal kernels: int8 weights with one scale per output channel, int8 activations with one scale per tensor, and 32-bit integer accumulation. The student is trained to reproduce the uncompressed refiner's per-voxel corrections rather than the ground truth, because the refiner's job is edges and the ground truth is masked at edges. The refiner shrinks from 39.4 to 17.2 million parameters. One trap is worth recording: freezing a quantised layer's weights does not freeze its quantiser, whose activation ranges are statistics updated on every forward pass. A later training run that merely used the frozen refiner let one range drift from 72.9 to 519.2 and silently degraded it, until we froze those statistics explicitly.

Model B removes one doubling stage from the neck, the heads, the prompt pyramid and the refiner, so the same 1 200 tokens produce a 240×320 point map. The levels it shares with the full model are copied across, the final output layers are re-fitted by least squares, and the model is then fine-tuned and quantised like Model A.

Three processors, one frame

The Neural Engine is the most power-efficient processor in the device and handles convolutions and MLPs very well, but self-attention at 1 200 tokens is its weak point: the softmax over a 1 201×1 201 score matrix per head proceeds at a fixed, modest rate, and the ViT alone takes 264 ms there. The GPU's fused attention never materialises that matrix and is several times faster, but the rest of the network runs as fast or faster on the Neural Engine. Core ML will not split a model this way on its own, so we do it explicitly: each of the 24 transformer blocks is exported as two models, the attention half assigned to the GPU and the MLP half to the Neural Engine. Activations pass between them through IOSurface-backed buffers, so nothing is copied, and the prompt pyramid runs on the Neural Engine during the time it would otherwise sit idle waiting for the GPU. The neck and heads stay on the Neural Engine, where they are 1.9 times faster than on the GPU.

The refiner cannot be expressed in Core ML at all, so it runs on Metal kernels we wrote for it. A refinement step first needs the sparse structure: which voxels exist at each level, which voxels pool into which, and each voxel's 27 neighbours. Built with sorting and hashing on the CPU this took 60–80 ms. The key observation is that the required voxel order — sorted by row, column, then depth — is just the image's pixel order with each pixel's voxels sorted by depth. Every level can therefore be stored column by column, pooling becomes a merge of four short lists, and a neighbour lookup is a short scan of nine columns. All of it runs in parallel on the GPU in about 2 ms. Each residual block then takes three GPU dispatches around a tiled int8 matrix multiply, with normalisation, activation, re-quantisation and the skip connection folded into the kernels so that no level of activations is read or written more than necessary. The robust metric fit runs on the GPU too, including its trimming step, which uses a two-level histogram instead of a sort.

One finding outside the model is worth passing on. In a camera app, leaving the capture session running while inference ran made Model A almost three times slower, because the image pipeline and preview compete with the ViT's attention for the GPU. The demo pauses the camera for the 150 ms it needs.

Results

A few conventions apply throughout. Our models use 1 200 tokens and one refinement step. The error measure is absolute relative error (AbsRel): the difference between predicted and true depth divided by the true depth, averaged over pixels, with no alignment to the ground truth of any kind. Besides all pixels, we report subsets that isolate particular difficulties: pixels where the sensor is confident, has low confidence or has no reading (holes); pixels near depth edges; thin structures, meaning the near side of a depth edge; and pixels where the sensor reports high confidence but is wrong by more than 15 %.

PromptMoGe‑L is the full-size model; Model A and Model B are the on-device models. For PromptDA‑L we ran the public checkpoint ourselves at its native 756×1008 resolution (3 888 tokens), once as released and once with our metric attachment applied to its output, which is the like-for-like comparison.

61 held-out rooms

Our main benchmark consists of 61 rooms, 3 340 frames, from the ARKitScenes validation pool. The pool was divided by physical room rather than by video, and the division was fixed before any of the data was downloaded, so that neither training nor design decisions could leak into it. No model in the table was trained on these rooms. The intervals we quote are 95 % bootstrap intervals obtained by resampling rooms, which is the honest unit: frames from one room are far from independent.

Official ARKitScenes upsampling benchmark

ARKitScenes defines an official depth-upsampling benchmark, and PromptDA reports its headline result on it. We evaluated on the 60 % of its validation split that falls inside our development rooms (3 373 frames), following Apple's protocol at ×4 upsampling, and kept the rest untouched as a future test set. On identical frames our models match or slightly beat our PromptDA‑L run.

We want to be clear about what this does not show. The PromptDA paper reports 0.0132 m mean error on the full split; our run of its public checkpoint gives 0.0143 m on our part of it. We checked input resolution and image format and could not close that gap, although our sensor baseline matches the published one to within 1 %. We therefore make no claim against the published number, and label every PromptDA figure on this page as our own run.

Zero-shot on seven other datasets

To test whether the model has learned to use a depth prompt in general, rather than ARKit's in particular, we ran it on seven datasets that neither it nor PromptDA was trained on, from tabletop scenes with transparent objects (HAMMER) to driving (KITTI). These datasets have no phone LiDAR, so the prompt is made from the ground truth: either downsampled 16 times, or downsampled 8 times with about 40 % of it removed in random rectangles. The second setting is the more informative. With a clean, dense ×8 prompt, plain interpolation of the prompt beats every network, so that setting mostly measures how well a model passes its input through. All models receive the same prompt, with gaps filled from the nearest valid pixel, and none is given camera intrinsics. The sensor calibration table is switched off, since these are not ARKit sensors.

3‑D reconstruction

Depth maps are rarely the end product; usually they are fused into a 3‑D model of a room. We fused each model's depth into a volumetric (TSDF) reconstruction along ARKit's recorded camera trajectory, for 163 captures of the 61 rooms, and compared the result with a reconstruction made the same way from laser-scan depth. The F‑score counts how much of the surface is reconstructed to within a threshold; accuracy and completeness are mean distances in the two directions.

Compared room by room, PromptMoGe‑L beats PromptDA‑L as released on F@2cm (+0.009, interval [+0.004, +0.015]) and beats the calibrated sensor on every measure in at least 59 of the 61 rooms. Against PromptDA with our attachment the two are close: we lead narrowly at the standard 4 cm voxel size and tie at 2 cm, while our surface normals are consistently 0.4–0.65° better. The most useful result for deployment is that Model A is indistinguishable from the full model, within 0.0003 F‑score across all 163 captures.

On three longer captures we also refined the camera poses using each model's own depth, without any ground truth. This improves on ARKit's trajectory (mean position error 1.55 → 1.34 cm) and raises F@2cm to 0.856 for both of our models, against 0.846 for PromptDA with our attachment. PromptDA's surfaces are, however, placed slightly more accurately on average (2.69 against 2.83 cm), consistent with its strength on pixels where the sensor is wrong.

Robustness to a degraded prompt

Real sensors misbehave, so we degraded the ARKit prompt in controlled ways and gave every model the same degraded input. Depth noise barely affects our models: with 10 % multiplicative noise their error moves from 1.38 % to 1.40 %, while PromptDA's triples and the sensor's own rises five-fold. The robust fit averages noise out, and the network has been trained not to copy individual readings. A misaligned prompt, as produced by a timestamp or calibration error between camera and LiDAR, hurts all methods about equally. Cutting the prompt off beyond 2 m is handled best by the full-size model and worst by Model A, a case where compression does cost robustness.

Randomly deleting prompt pixels initially looked like a serious weakness: with half of them missing our error almost doubled. It turned out not to be a loss of information — the models were actually more accurate with 5 % of the pixels than with 20 % — but an unfamiliar input pattern, since training only ever removed rectangular blocks. Filling each gap with the nearest valid reading and marking it as lower-confidence restores accuracy to within 3 % of the clean result even when only one pixel in twenty survives. The released code and the demo app do this by default. It matters on a real device, where the raw LiDAR stream has far more gaps than the processed depth in ARKitScenes.

Ablations

The table below collects the controlled comparisons behind claims made earlier on this page. All are paired over the same 61 rooms.

Sparse sensors and confidence

Two further lines of work are not part of this release but shaped it. ARKit's depth map is the product of several frames of temporal integration. A device that exposes only a single pulse of its laser pattern — we simulated 112 beams — provides a far sparser prompt, and a model trained only on ARKit's dense maps handles it badly: error on confident pixels rises from 1.2 % to 3.2 %, and the model copies the prompt's smeared object boundaries so faithfully that its edges become worse than if it had ignored the prompt. Re-drawing a quarter of the training prompts through a learned simulator of that sensor recovers 13 % of the lost accuracy and most of the edge quality, at a cost of 4 % on ARKit data.

We also trained two small heads that predict, per pixel, the error of the sensor and the error of the model's own output. They are trained after the fact on frozen features, so adding them changes no depth value, which we verified bit for bit. They are good at ranking pixels by error, recovering 50–70 % of the gap between a random and a perfect ordering, but they under-estimate its magnitude; a single scale factor per head, fitted on held-out videos, fixes that for the model-error head, whose 90 % intervals then cover 91–92 % of pixels.

Limitations

It is not better than PromptDA‑L everywhere. With identical metric attachment, overall accuracy is a tie. PromptDA is 17 % more accurate where the sensor is confidently wrong and 4–9 % more accurate between 2 and 4 m. We believe the first is a side effect of our fidelity loss, which pulls the output towards every confident sensor reading, including the wrong ones; restricting it to readings that agree with ground truth is an obvious next experiment.

It is an indoor, short-range model. Training data covers rooms up to about 5 m. On outdoor scenes (DIODE outdoor) PromptDA‑L is roughly twice as accurate.

Image-only geometry has eroded. If the prompt path is bypassed entirely, the model's scale-free geometry is 2–22 % worse than stock MoGe‑3 on MoGe's own evaluation suite, worst on KITTI. The gates guarantee an exact start, but the decoder is fine-tuned afterwards and drifts. Predicted scale and field of view are unaffected, and on a device the prompt is always present, but this is a genuine regression that a future training recipe should address, for instance by replaying generic images.

The measurements have a floor. On sensor-confident pixels the laser ground truth disagrees with a fusion of itself by 0.66 %, so differences much below 1 % on that subset mean little. The sparse-sensor results rely on a simulator rather than hardware, and we could not reproduce PromptDA's published ARKitScenes figure.

Refinement trades thin structures for everything else, by about 4 %. A refiner that knows which structures are thin is future work.

Related work

Prompted depth foundation models. PromptDA [2] is the closest system to ours and the one we compare with throughout. It fine-tunes Depth Anything V2, including its backbone, and adds the LiDAR map at four scales of the decoder through zero-initialised convolutions. Its prompt is a single depth channel normalised to the range of each frame, without confidence or validity, and it is trained on ARKitScenes and ScanNet++. Prior Depth Anything [5] first densifies an arbitrary depth prior and then refines it with a conditioned network; LDCM [6] combines a Poisson-based pre-fill with a point-map output; SLIM [7] fuses sparse LiDAR into the neck of MoGe‑2 for long-range driving. To our knowledge none of these reports latency on a phone or tablet; their timings are for datacenter GPUs.

Depth completion. Dedicated completion networks, from propagation-based designs to OMNI‑DC [8], are strong on the sensor pattern they were trained for and tend to break when it changes. Conditioning a frozen foundation model is our way of inheriting robustness that such networks have to learn from scratch.

Conditioning frozen models. Our gates belong to the family of zero-initialised conditioning mechanisms that includes ControlNet [9], and are closest in spirit to the tanh gates with which Flamingo [10] connects new layers to a frozen language model. The metric attachment borrows the closed-form alignment and the residual trimming of MiDaS [11], applies them to sensor readings instead of ground truth, and makes them part of the training graph.

References

1. L. Kong, R. Li, R. Wang, S. Xu, C. Yao, J. Xiang, J. Yang. MoGe‑3: Fine-Detail Monocular Geometry Estimation with Self-Guided Sparse Volumetric Refinement. 2026.
2. H. Lin, S. Peng, J. Chen, S. Peng, J. Sun, M. Liu, H. Bao, J. Feng, X. Zhou, B. Kang. Prompting Depth Anything for 4K Resolution Accurate Metric Depth Estimation. CVPR 2025.
3. G. Baruch et al. ARKitScenes. NeurIPS Datasets and Benchmarks 2021.
4. M. Roberts et al. Hypersim. ICCV 2021.
5. Z. Wang et al. Depth Anything with Any Prior. ICLR 2026.
6. Z. Yu et al. Large Depth Completion Model from Sparse Observations. ICLR 2026.
7. K. Zheng et al. Sparse-LiDAR Prompting of Monocular Geometry Foundations. 2026.
8. Y. Zuo, W. Yang, Z. Ma, J. Deng. OMNI‑DC. ICCV 2025.
9. L. Zhang, A. Rao, M. Agrawala. Adding Conditional Control to Text-to-Image Diffusion Models. ICCV 2023.
10. J.-B. Alayrac et al. Flamingo. NeurIPS 2022.
11. R. Ranftl et al. Towards Robust Monocular Depth Estimation. TPAMI 2020.
12. R. Wang et al. MoGe‑2. NeurIPS 2025. · M. Oquab et al. DINOv2. TMLR 2024. · L. Yang et al. Depth Anything V2. NeurIPS 2024.

BibTeX

@misc{promptmoge2026,
  title  = {PromptMoGe: LiDAR-Prompted Monocular Geometry on the Device},
  author = {Sergii Penner},
  year   = {2026},
  url    = {https://github.com/sergmister/PromptMoGe}
}