Abstract

Audio-based piano transcription performs well on onset, pitch, and velocity, but the sustain pedal lets sound persist long after key release, so audio systems predict pedal-extended offsets rather than physical key release. Yet existing Visual Piano Transcription (VPT) systems focus on onset detection from short video windows, offset accuracy lags onset by a wide margin, and note-level velocity has not been reported. To address these gaps, we present V2N (Video to Notes), the first complete VPT system: a shared temporal backbone feeds task-specific heads for onset, offset, key hold, and velocity, jointly trained with per-frame supervision rather than only at the window center. Ablations show that multi-task supervision enables offset and velocity prediction while improving onset accuracy; longer temporal context yields further improvements. V2N sets new state-of-the-art results on PianoVAM and R3.

1 Introduction

Automatic Music Transcription (AMT) recovers symbolic note attributes such as pitch, onset, offset, and velocity from audio. Audio-based piano transcription has matured in estimating pitch, onset, and velocity [6, 7, 13], but offset remains systematically confounded by the sustain pedal: sound persists after the key returns to rest, so audio systems conventionally extend note offsets to pedal release for both training and evaluation [6, 13, 20, 21], with discrepancies from the MIDI NoteOff event that can reach several seconds.

Visual Piano Transcription (VPT) observes the keyboard directly: a pressed key is visually distinct from a released one regardless of pedal state, making the MIDI-encoded physical key state directly observable. VPT is also robust when audio is ambiguous, degraded, or absent, for example, in multi-instrument recordings, noisy or reverberant environments, and footage with missing or corrupted audio. Yet VPT remains underexplored. Existing methods [12, 18, 22] process at most 0.2 s of video context, leave offset accuracy substantially lower than onset accuracy, or do not report note-level velocity.

We present V2N (Video to Notes),¹¹1Code, trained checkpoints, and predicted MIDI: https://github.com/yonghyunk1m/V2N. the first complete VPT system: four task-specific heads (onset, offset, key hold, and velocity; Figure 1) are trained jointly over 1 s of video and supervised at every frame rather than only at the window center. Our contributions are:

• Video-only complete MIDI transcription. V2N matches Li et al. [14] on PianoVAM Onset and surpasses all prior VPT on R3 Onset, substantially improves physical key-release prediction, and is the first video-only system to report note-level velocity F1, achieving state-of-the-art results on PianoVAM and R3.

• Ablation-validated design choices. (i) Multi-frame loss with a Conformer-style convolutional backbone [5] outperforms the single-frame Sight-to-Sound (S2S) [12] recipe on onset. (ii) Multi-task heads improve onset and offset accuracy; removing these heads (offset, key hold, velocity) reduces onset performance and collapses offset F1. (iii) Offset-guided note decoding, which terminates notes at offset-head peaks (with key hold as a fallback) rather than at a key hold (the video analogue of audio frame-activity) threshold alone, improves over the convention from audio-based piano transcription [6].

We additionally analyze cross-dataset transfer and find that both V2N and prior VPT systems fail similarly, highlighting the limitations of fixed-geometry preprocessing.

2 Related Work

2.1 Audio-based Piano Transcription

Audio-based piano transcription is a mature field [2]. Onsets and Frames [6] established multi-task prediction of onsets, sustained audio activity, and velocity, whose complementarity dramatically improved transcription. Subsequent work refined the paradigm with Transformer architectures [7] and high-resolution onset/offset regression [13] on MAESTRO [8]. Neural semi-CRF event-based decoding [20, 21] defines the current state of the art and, like its predecessors, extends note offsets through sustain-pedal intervals, so reported offset accuracy measures the end of sound rather than key release. We port this multi-task paradigm to vision, where onset cues differ (finger motion vs. spectral transients) and offset prediction becomes a direct observation rather than inferred from sound decay.

2.2 Visual Piano Transcription

Early VPT approaches detected pressed keys via background subtraction [19, 1]. Sight-to-Sound (S2S) [12] introduced end-to-end learning with a ResNet-18 [9] backbone on 5-frame windows. Su et al.’s Video2RollNet [18] (V2R) augmented the same ResNet-18 with multi-scale feature attention as the visual front-end of an audio-generation pipeline, predicting a binary pressed-key roll at the center of a 5-frame window. PPAN [22] applied a Vision Transformer [4] to 6-frame inputs of the R3 rehearsal dataset [3]. These methods process at most 0.2 s of temporal context, supervise only the center frame of each window, and leave offset accuracy well behind onset (PPAN: 45.9% $+$Off on PianoVAM at 50 ms, our re-evaluation). Velocity has been explored only with optical-flow CNNs [10], without note-level evaluation. The closest audio-visual counterpart, Li et al. [14], fuses video with audio via cross-attention but derives offsets from acoustic frame activations and does not report note-level velocity.

3 Methodology

3.1 Task Definition

A piano note is defined by a key press (MIDI NoteOn) and a key release (NoteOff), together with pitch and velocity. Video captures mechanical cues (keys pressing, being held, and returning to rest); audio captures acoustic cues (hammer-strike transient and sound envelope).

With the sustain pedal engaged, sound persists long after key release, so audio-based piano transcription conventionally targets the end of the sound rather than the key release. Audio offsets can diverge from MIDI NoteOff by seconds.

We define four visual prediction targets grounded in key mechanics (Figure 1); each is the head’s continuous output, indexed by video frame $t\in \{1,\dots ,T\}$ and piano key $k\in \{1,\dots ,88\}$:

• Onset $o_{t,k}\in [0,1]$: key press initiation (shared with audio-based piano transcription).

• Offset $r_{t,k}\in [0,1]$: physical key release, aligned with MIDI NoteOff; audio-based piano transcription conventionally predicts the end of the sound (audio offset), which coincides with key release only when the pedal is disengaged and otherwise extends to pedal release.

• Key Hold $f_{t,k}\in [0,1]$: key physically held over $[\text{onset},\text{offset})$; the audio equivalent (active) extends through pedal sustain.

• Velocity $v_{t,k}\in [0,1]$: per-key normalized MIDI velocity, regressed by a linear head with onset-masked loss.

The probability heads $o,r,f$ are sigmoid-bounded and binarized at threshold $\tau =0.5$ during note decoding (Section 4); the velocity head is linear (its raw output may exceed $[0,1]$) and is clipped to $[0,1]$ then rescaled to integers in $\{0,\dots ,127\}$ at inference (Section 3.3).

A NoteOn is then $(t^{\ast },k^{\ast },v_{t^{\ast }})$ at a peak in $o_{t,k}$, and NoteOff is the corresponding peak in $r_{t,k}$; the key hold signal densely labels the $[\text{onset},\text{offset})$ span, and repeated same-key onsets yield independent note pairs while overlapping key hold labels merge via element-wise maximum.²²2We use “frame” exclusively for a video frame (40 ms; all models in this paper operate at 25 fps). This should not be confused with the “frame” terminology in audio-based piano transcription, which denotes per-step activity over a time-frequency representation (e.g., spectrogram or CQT).

3.2 Architecture

Our architecture (Figure 2) consists of three components: (i) a visual feature extractor that converts each video frame into a per-frame feature vector, (ii) a shared temporal backbone that models dependencies across frames, and (iii) task-specific prediction heads for each output.

Visual Feature Extractor. We adopt the S2S feature extractor [12]: a ResNet-18 [9] (grayscale, trained from scratch) augmented with a slope prior, a learned 1D spatial encoding mapping each horizontal position to one of the 88 piano keys. For each frame, a 5-frame window (0.2 s at 25 fps) is processed jointly; global average pooling yields a $T\times 512$ sequence, which a linear projection with dropout ($p=0.5$) maps to the backbone dimension.

Temporal Backbone. The projected features pass through LayerNorm followed by $L=3$ Conformer ConvModule blocks [5] (self-attention omitted): FFN $\to$ depthwise 1D convolution (kernel size 31) $\to$ FFN, with residual connections. Three stacked kernel-31 convolutions provide each output frame with a receptive field spanning all 25 frames of the 1 s input without the quadratic cost of self-attention.

Task-Specific Heads. Four parallel heads each apply a bidirectional LSTM (hidden 256) followed by a linear projection. Onset, offset, and key hold heads output $T\times 88$ logits, sigmoid-mapped to per-key per-frame probabilities; the velocity head regresses normalized MIDI velocity ($T\times 88$) under a per-key mask at ground-truth onset positions. Key hold provides dense per-frame supervision over each $[\text{onset},\text{offset})$ span, complementing the sparse onset/offset kernels and strengthening the backbone during training; at inference, offset peaks define note end; if key hold ends first, the note ends there instead (Table 2).

Computational cost. V2N uses 125 GFLOPs and 28.2 M parameters per 1 s clip (S2S feature extractor accounts for 99%; Conformer and BiLSTM heads add $<$1%). With 0.5 s overlapping stride, the amortized cost is $\approx 250$ GFLOPs per second of input video, or 40 ms of GPU time on an NVIDIA RTX 5080 (bfloat16; RTF $\approx 0.04$). This undercuts Li et al. [14]’s video branch ($\sim 430$ GFLOPs/s, $\sim 1.7\times$) and is comparable in wall-clock to PPAN [22] (9.7 GFLOPs per 6-frame call) despite higher FLOPs, so the 1 s window incurs no practical wall-clock penalty.

3.3 Training

Onset and offset labels are soft triangular kernels centered at each event, linearly decaying from 1.0 at the event frame to 0 at $\pm \ell$ frames ($\ell =2$); this tolerates small labeling jitter and spreads the gradient signal over neighboring frames instead of a single positive surrounded by negatives. Key hold labels are binary 1 throughout $[\text{onset},\text{offset})$. MIDI velocity is normalized to $[0,1]$ for training (dividing by 127) and at inference is rescaled by 127, clipped to $[0,127]$, and rounded to integers in $\{0,\dots ,127\}$ for MIDI export.

$L_{\text{vel}}=\frac{{\sum }_{t,k}{m}_{t,k}{\left(v_{t,k}-v_{t,k}^{\ast }\right)}^2}{{\sum }_{t,k}{m}_{t,k}}$

(1)

where $m_{t,k}\in \{0,1\}$ is a per-key onset mask (1 at ground-truth onset positions, 0 elsewhere). Task weights are onset$=$2, others$=$1, emphasizing the onset head as in Onsets-and-Frames [6]:

$L_{\text{total}}=2L_{\text{onset}}+L_{\text{off}}+L_{\text{khold}}+L_{\text{vel}}$

(2)

At training time, 1 s segments are drawn densely at a 1-frame stride and shuffled across videos, so every frame index appears as the center of some segment and at every other relative offset across epochs. At inference we process the full recording in 1 s segments with 0.5 s stride (50% overlap), retaining only the central 0.5 s of predictions from each segment to avoid boundary artifacts where the receptive field is truncated at segment edges.

媒体内容 · 前往原文查看

	Frame	Onset		$+$Off		$+$Vel		$+$Off$+$Vel
Model	(60 Hz)	50 ms	100 ms	50 ms	100 ms	50 ms	100 ms	50 ms	100 ms
Trained on PianoVAM $\to$ Test: PianoVAM
S2S† [12]	41.7	60.1	94.5	21.8	49.6	(39.4)	(63.2)	(14.3)	(32.7)
V2R† [18]	82.2	86.5	89.8	55.4	81.8	(58.5)	(60.4)	(36.8)	(54.6)
Li et al.† [14]	(25.2)	94.2	97.4	(21.0)	(40.0)	(63.6)	(65.2)	(12.8)	(25.8)
PPAN† [22]	80.4	84.9	94.1	45.9	82.8	(57.0)	(62.5)	(29.7)	(55.3)
V2N (Ours)	90.9∗	94.7	97.0	89.5∗	95.5∗	82.8∗	84.5∗	78.3∗	83.2∗
Trained on R3s+R3x $\to$ Test: R3s
S2S† [12]	60.5	44.0	84.7	17.2	64.7	(24.9)	(46.5)	(9.8)	(35.2)
V2R† [18]	54.7	38.7	76.1	15.8	54.9	(21.6)	(41.4)	(8.8)	(29.6)
PPAN† [22]	57.0	40.7	83.4	14.9	60.3	(23.4)	(46.1)	(8.7)	(33.2)
V2N (Ours)	80.4∗	78.8∗	91.7∗	69.5∗	88.9∗	57.9∗	65.9∗	51.2∗	64.1∗
Trained on R3s+R3x $\to$ Test: R3x
S2S† [12]	58.7	46.6	83.9	14.7	63.6	(33.5)	(57.1)	(10.3)	(44.2)
V2R† [18]	57.0	46.4	80.8	14.9	60.8	(33.7)	(55.7)	(10.3)	(42.6)
PPAN† [22]	56.6	47.7	83.2	13.9	59.3	(33.7)	(57.0)	(9.3)	(41.3)
V2N (Ours)	76.5∗	73.3∗	86.6∗	68.5∗	85.2∗	59.7∗	69.8∗	56.6∗	69.0∗

4 Experimental Setup

4.1 Datasets

PianoVAM [11] contains 107 top-view piano videos with synchronized MIDI from a Yamaha Disklavier, recorded during amateur practice sessions under naturalistic conditions. We follow the proposed splits in PianoVAM v1.1’s metadata.json: train$+$ext-train (81 recordings) for training, valid (9) for validation, and test (9 recordings, 1.63 h, 42,241 notes) for evaluation; we exclude 8 recordings that the release assigns to special(blurry) and special(4hands) splits. Each video is preprocessed with a perspective transform using annotated keyboard corners, producing $800\times 144$ grayscale frames at 25 fps. The frame height includes the keyboard and a bottom margin capturing hand and wrist motion for velocity cues.

R3 [3, 22] consists of 31 h of practice recordings from 62 sessions by two professional pianists, captured with GoPro cameras (primarily 60 fps, downsampled to 25 fps). It is divided into R3s (Rachmaninoff Piano Concerto No. 3; 219 train / 74 test) and R3x (various Western classical repertoire; 495 train / 107 test). Videos are preprocessed with a perspective transform using bounding-box annotations and rotation correction to the same $800\times 144$ format. R3 is more challenging than PianoVAM: it spans multiple pianos, camera positions, lighting, and advanced repertoire.

4.2 Training Configuration

V2N. Trained with AdamW [15] ($\text{lr}=5\times {10}^{-4}$, weight decay $0.01$) and a cosine schedule with 5% linear warmup, in bfloat16 with an effective batch size of 16 (via gradient accumulation). To fit the 1 s window on a single GPU, we apply gradient sampling: backpropagation flows through a random 50% of the 25 input frames while the remainder are forward-passed under torch.no_grad, roughly halving backward-pass memory without reducing frame-level supervision coverage. PianoVAM models use 100k optimizer steps and R3 models use 200k steps (convergence was slower on the more heterogeneous R3 data). All reported numbers come from the final-step checkpoint. For augmentation, we adopt a subset of the PPAN training recipe [22]: brightness jitter ($\pm$10%, $p=0.4$), random rotation ($\pm$0.2^∘, $p=0.4$), random erasing ($p=0.5$), and Gaussian noise ($\sigma =0.1$, $p=0.4$).

Baselines. S2S [12], V2R [18], Li et al. [14], and PPAN [22] are retrained with each author’s default hyperparameters and augmentations. Li et al. and PPAN use official code. S2S and V2R use the CNN and CNN-V2R reproductions from the PPAN codebase (5-frame window, image height$\times$width $160\times 800$ and $138\times 776$, respectively). For Li et al., we train and evaluate only the video branch (no audio input), so all baselines are video-only. On R3, S2S, V2R, and PPAN share the PPAN recipe (AdamW, 10 epochs on R3s+R3x, PPAN-default augmentations).

媒体内容 · 前往原文查看

Training	Inference	Frame	Onset		$+$Off		$+$Vel		$+$Off$+$Vel
		(60 Hz)	50 ms	100 ms	50 ms	100 ms	50 ms	100 ms	50 ms	100 ms
Kh	Kh	90.1	92.8	95.2	81.4	93.1	(62.3)	(63.8)	(55.5)	(62.4)
Kh, V	Kh, V	90.6	92.7	96.1	85.7	94.1	80.6	82.8	74.6	81.0
Kh, V, On	Kh, V	90.2	93.3	96.8	85.9	95.0	82.1	84.9	75.8	83.4
Kh, V, On	Kh, V, On	90.4	94.3	97.0	86.7	95.2	82.8	85.0	76.3	83.4
Kh, V, On, Off	Kh, V, On	90.7	94.7	97.0	87.8	95.4	82.5	84.3	76.7	83.0
Kh, V, On, Off	Kh, V, On, Off	90.9	94.7	97.0	89.5	95.5	82.8	84.5	78.3	83.2

媒体内容 · 前往原文查看

Variant	Window	Onset	$+$Off	$+$Vel
Ours (center-frame loss)	0.2 s	90.5	78.5	77.4
$+$ multi-frame loss	0.2 s	91.8	83.0	78.5
$+$ sequence model	0.2 s	94.3	88.7	80.5
$+$ longer context (V2N)	1.0 s	94.7	89.5	82.8

4.3 Evaluation Protocol

We decode frame-level predictions into MIDI note events using the onset, offset, key hold, and velocity heads. A peak is a rising-edge transition of the binarized probability on a given key (the first above-threshold frame after an off frame). Each note begins at an onset peak above ${\tau }_{\text{on}}=0.5$ and is extended forward until the next offset peak above ${\tau }_{\text{off}}=0.5$ on the same key, or until both onset and key hold probabilities fall below threshold, whichever comes first. Predicted MIDI is evaluated against ground truth with mir_eval [17]. We report one frame-level and four additive note-level F1 metrics (mir_eval function names in parentheses):

• Frame (multipitch): per-frame multipitch F1 after rasterizing predicted and ground-truth notes on a 60 Hz grid, matching the primary original video frame rate. Independent of onset tolerance.

• Onset (note): pitch and onset match within an onset tolerance $\tau$ (50 ms or 100 ms).

• $+$Off (note_with_offsets): offset match within the same $\tau$ as onset rather than mir_eval’s $\max (0.2d,50\mathrm{ms})$, reflecting key release.

• $+$Vel (note_with_velocity): Onset match plus velocity match within 10% (mir_eval rescales velocity to its L2-optimal global scalar). Not cumulative with $+$Off.

• $+$Off$+$Vel (note_with_offsets_and_velocity): the strictest metric; all four MIDI attributes (pitch, onset, offset, velocity) must match.

We report note-level results at both 50 ms (standard in audio-based piano transcription) and 100 ms (matching the protocol of [22]) onset tolerances; under the offset convention above, the offset tolerance tracks the onset tolerance. Predicted note timestamps are quantized to the 25 fps video grid (40 ms resolution), while ground-truth MIDI timestamps are continuous; the 50 ms onset/offset tolerance absorbs up to 40 ms of quantization error, so evaluation is not bottlenecked by the video frame rate. The sustain pedal (MIDI CC64) is not predicted by V2N or any baseline.

5 Results

5.1 Main Results

Table 1 compares V2N against the baselines defined in Section 4. V2N is our default full-configuration model (onset, offset, key hold, velocity heads).

On PianoVAM, V2N matches Li et al. on Onset (within 0.5 %p at both tolerances) and surpasses all baselines on every offset-dependent metric. The largest gains appear on the physical key-release metrics ($+$Off and $+$Off$+$Vel): dedicated offset supervision with 1 s of context nearly doubles $+$Off F1 (PPAN: 45.9 $\to$ V2N: 89.5 at 50 ms) and yields $+$48.6 %p on $+$Off$+$Vel (29.7 $\to$ 78.3) over PPAN, and V2N is the only system achieving high accuracy across all four MIDI attributes (pitch, onset, offset, velocity).

On R3, V2N surpasses prior VPT on Onset at both splits and tolerances, and the gap on offset-dependent metrics grows further: onset-centric baselines fail on $+$Off despite competitive onset accuracy, confirming that physical key release requires dedicated supervision rather than arising as a by-product of key hold (frame-activity) modeling.

Our retrained S2S and V2R baselines reproduce the published R3s/R3x Onset numbers within roughly one percentage point, so the gap V2N opens on offset-dependent metrics reflects model design, not training differences.

Figure 3 illustrates the qualitative consequence of these performance differences on a representative PianoVAM excerpt: Li et al.’s offsets collapse to near-zero duration, PPAN produces plausible note spans but a constant MIDI velocity, whereas V2N reconstructs both the note durations and the dynamic contour of the ground truth.

5.2 Ablation Studies

We organize the ablations to mirror the title: first multi-task heads (Table 2), then multi-frame context (Table 3).

Why multi-task heads? Table 2 shows that additional heads help both as training supervision and as decoding signals. Adding velocity supervision and decoding raises $+$Vel F1 from the constant-velocity proxy to 80.6 at 50 ms. Adding onset as an auxiliary target improves most note metrics before the onset head is used (row 3 vs. row 2), and using it at decoding further improves Onset F1 (93.3 $\to$ 94.3; row 4 vs. row 3). Finally, adding offset supervision improves offset-dependent metrics before the offset head is used (row 5 vs. row 4), and offset-guided decoding yields the best $+$Off and $+$Off$+$Vel scores (row 6). Thus, the heads provide complementary supervision and inference-time cues for complete MIDI prediction.

Why multi-frame loss? Our starting variant (Table 3 row 1) follows the S2S [12] recipe of supervising only the center frame of each 5-frame window. Extending the loss to all five frames (row 2) yields $+$1.3 %p Onset and $+$4.5 %p $+$Off, since the model now sees $\sim 5\times$ more supervised signal per segment at zero inference cost.

Why sequence modeling? Adding a Conformer ConvModule backbone over the 5-frame window (Table 3 row 3) is the largest single gain: $+$2.5 %p Onset and $+$5.7 %p $+$Off. A purely spatial S2S head cannot distinguish pressed from held keys; sequence modeling resolves this ambiguity.

Why a 1 s input window? Extending the window from 0.2 s to 1.0 s (Table 3 row 4) adds only $+$0.4/$+$0.8 %p (Onset/$+$Off) on PianoVAM but is decisive on R3: V2N reaches 91.7% R3s Onset F1 at 100 ms (Table 1) and enables offset prediction (+Off 88.9%) that shorter-window, onset-only baselines cannot provide. R3’s advanced repertoire and heterogeneous conditions demand longer context, so we use 1.0 s as the default on both datasets.

Why parallel heads? Cascaded variants from audio-transcription designs [6, 13] yielded no meaningful improvement, so V2N uses the simpler parallel-head design.

5.3 Cross-Dataset Transfer

媒体内容 · 前往原文查看

Train	Test	S2S	V2R	PPAN	V2N
PianoVAM	R3s	0.6/2.4	0.3/0.5	0.0/0.0	1.5/5.6
PianoVAM	R3x	0.2/1.8	0.0/0.0	0.0/0.0	0.9/2.7
R3s$+$R3x	PianoVAM	0.1/0.1	0.0/0.0	0.3/0.6	1.1/4.9

Table 4 shows that Onset F1 under dataset swaps collapses to near zero for every system. The root cause is geometric: the $800\times 144$ perspective warp standardizes the pixel grid, not the keyboard within it; median keyboard width is $\sim 784$ px on R3 vs. $\sim 606$ px on PianoVAM, so the same column indexes a different key across datasets. Cross-dataset VPT likely requires geometry-invariant representations, as also observed in audio-based AMT [16]. We leave robustness to unseen camera geometries to future work.

6 Discussion

R3 synchronization issues. Cross-correlating MIDI onset trains with video audio energy (10 ms resolution, $\pm$3 s search) reveals systematic video–MIDI offsets exceeding 200 ms in 70 of 895 R3 files, all with MIDI preceding video. Since this affects all models equally, we report results on the original test split. Excluding the 10 sync-affected R3x test files raises V2N F1 at 100 ms by $+7.7$ %p Onset (86.6 $\to$ 94.3), $+7.9$ %p $+$Off (85.2 $\to$ 93.2), $+6.6$ %p $+$Vel (69.8 $\to$ 76.4), and $+6.7$ %p $+$Off$+$Vel (69.0 $\to$ 75.7). Per-file sync offsets are released with the code (Section 1).

Limitations. The current work has two main limitations: (i) cross-dataset transfer fails almost completely: the learned pixel-to-key mapping does not generalize across camera geometries, even after perspective-transform normalization, and (ii) V2N does not estimate sustain-pedal control changes. Future work should address geometry-invariant keyboard localization, joint sync alignment for noisy training data, sustain-pedal estimation, and audio-visual fusion building on V2N’s visual contributions.

7 Conclusion

We presented V2N, a visual piano transcription system that jointly predicts onsets, offsets, key hold states, and velocity through multi-task, multi-frame modeling. V2N substantially improves physical key-release accuracy over prior video baselines, whose offsets either collapse to near-zero duration or are inferred post-hoc from key hold activations, and is the first VPT system to report note-level velocity. Ablations identify three design choices behind complete note-level transcription from video: a dedicated offset head, multi-task training, and offset-guided decoding. Cross-dataset transfer between PianoVAM and R3 collapses in every direction, indicating that current VPT models are tightly coupled to their training preprocessing; geometry-invariant representations and dataset-agnostic keyboard localization remain open problems.

8 Acknowledgements

This work was supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) under Grant RS-2023-NR077289.

References

• [1] M. Akbari and H. Cheng (2015) ClaVision: visual automatic piano music transcription. In International Conference on New Interfaces for Musical Expression (NIME), Cited by: §2.2.
• [2] E. Benetos, S. Dixon, Z. Duan, and S. Ewert (2019) Automatic music transcription: an overview. IEEE Signal Processing Magazine 36 (1), pp. 20–30. Cited by: §2.1.
• [3] C. E. Cancino-Chacón and I. Pilkov (2024) The rach3 dataset: towards data-driven analysis of piano performance rehearsal. In MultiMedia Modeling: 30th International Conference (MMM), Cited by: §2.2, §4.1.
• [4] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al. (2021) An image is worth 16x16 words: transformers for image recognition at scale. In International Conference on Learning Representations (ICLR), Cited by: §2.2.
• [5] A. Gulati, J. Qin, C. Chiu, N. Parmar, Y. Zhang, J. Yu, W. Han, S. Wang, Z. Zhang, Y. Wu, and R. Pang (2020) Conformer: convolution-augmented transformer for speech recognition. In Interspeech, Cited by: 2nd item, §3.2.
• [6] C. Hawthorne, E. Elsen, J. Song, A. Roberts, I. Simon, C. Raffel, J. Engel, S. Oore, and D. Eck (2018) Onsets and frames: dual-objective piano transcription. In International Society for Music Information Retrieval Conference (ISMIR), Cited by: 2nd item, §1, §2.1, §3.3, §5.2.
• [7] C. Hawthorne, I. Simon, R. Swavely, E. Manilow, and J. H. Engel (2021) Sequence-to-sequence piano transcription with transformers. In International Society for Music Information Retrieval Conference (ISMIR), Cited by: §1, §2.1.
• [8] C. Hawthorne, A. Stasyuk, A. Roberts, I. Simon, C. A. Huang, S. Dieleman, E. Elsen, J. Engel, and D. Eck (2019) Enabling factorized piano music modeling and generation with the MAESTRO dataset. In International Conference on Learning Representations (ICLR), Cited by: §2.1.
• [9] K. He, X. Zhang, S. Ren, and J. Sun (2016) Deep residual learning for image recognition. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: §2.2, Figure 2, §3.2.
• [10] S. Kang, J. Kim, and S. Yoon (2019) Virtual piano using computer vision. arXiv preprint arXiv:1910.12539. Cited by: §2.2.
• [11] Y. Kim, J. Park, J. Bae, K. Kim, T. Kwon, A. Lerch, and J. Nam (2025) PianoVAM: a multimodal piano performance dataset. In International Society for Music Information Retrieval Conference (ISMIR), Cited by: §4.1.
• [12] A. S. Koepke, O. Wiles, Y. Moses, and A. Zisserman (2020) Sight to sound: an end-to-end approach for visual piano transcription. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Cited by: 2nd item, §1, §2.2, Figure 2, §3.2, Table 1, Table 1, Table 1, §4.2, §5.2.
• [13] Q. Kong, B. Li, X. Song, Y. Wan, and Y. Wang (2021) High-resolution piano transcription with pedals by regressing onset and offset times. IEEE/ACM Transactions on Audio, Speech, and Language Processing 29, pp. 3707–3717. Cited by: §1, §2.1, §5.2.
• [14] Y. Li, X. Wang, R. Wu, W. Xu, and W. Cheng (2024) A two-stage audio-visual fusion piano transcription model based on the attention mechanism. IEEE/ACM Transactions on Audio, Speech, and Language Processing 32, pp. 3618–3630. Cited by: 1st item, §2.2, §3.2, Table 1, §4.2.
• [15] I. Loshchilov and F. Hutter (2019) Decoupled weight decay regularization. In International Conference on Learning Representations (ICLR), Cited by: §4.2.
• [16] L. S. Marták, P. Hu, and G. Widmer (2024) Quantifying the corpus bias problem in automatic music transcription systems. In International Workshop on Sound Signal Processing Applications (IWSSPA), Cited by: §5.3.
• [17] C. Raffel, B. McFee, E. J. Humphrey, J. Salamon, O. Nieto, D. Liang, and D. P. W. Ellis (2014) Mir_eval: a transparent implementation of common MIR metrics. In International Society for Music Information Retrieval Conference (ISMIR), Cited by: §4.3.
• [18] K. Su, X. Liu, and E. Shlizerman (2020) Audeo: audio generation for a silent performance video. In Advances in Neural Information Processing Systems (NeurIPS), Vol. 33. Cited by: §1, §2.2, Table 1, Table 1, Table 1, §4.2.
• [19] P. Suteparuk (2014) Detection of piano keys pressed in video. Technical Report Stanford University. Cited by: §2.2.
• [20] Y. Yan, F. Cwitkowitz, and Z. Duan (2021) Skipping the frame-level: event-based piano transcription with neural semi-CRFs. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: §1, §2.1.
• [21] Y. Yan and Z. Duan (2024) Scoring time intervals using non-hierarchical transformer for automatic piano transcription. In International Society for Music Information Retrieval Conference (ISMIR), Cited by: §1, §2.1.
• [22] U. Zivanovic, I. Pilkov, and C. Cancino-Chacón (2025) Pay attention to the keys: visual piano transcription using transformers. In International Joint Conference on Artificial Intelligence (IJCAI), Cited by: §1, §2.2, §3.2, Table 1, Table 1, Table 1, Table 1, §4.1, §4.2, §4.2, §4.3.