MedTok — Medical Multimodal Tokenizer & Unified Token Protocol

Overview

MedTok is my undergraduate thesis on designing a medical-specific multimodal tokenizer and a unified data protocol. Medical data is heterogeneous (2D images, physiological signals, surgical video, clinical notes, 3D surfaces), and most models need a separate pipeline for each. MedTok maps all of them into one discrete vocabulary of 13,824 token ids and one sequence format, so one Transformer can consume any combination.

How it works

Each modality has its own codec behind a common Codec interface. Lightweight autoencoders and a vector quantiser produce discrete codes for images and video. Time-series can be encoded as events, as VQ codes, or as continuous projections. Text uses an exact byte-level BPE codec. A MultimodalCodec combines the segments into a TokenSequence with modality, time and position side channels, and every sequence can be decoded back to its source.

On top of that format, the project compares three representation strategies (continuous projection, vector quantisation and event-based tokens). It measures compression ratio, reconstruction error, codebook utilisation, sequence length and downstream accuracy on MedMNIST classification, PhysioNet 2012 mortality prediction and Cataract-101 surgical phase recognition.

Engineering notes

The whole experiment matrix and its ablations run from YAML pipeline plans with a single script. Result tables and figures are generated from experiment outputs and never typed in by hand. The protocol has a formal specification covering id ranges, special tokens, reversibility and bit accounting. Only open datasets are downloaded, and no patient-level data is committed.