ScribbleMind

Overview

ScribbleMind takes a photo of handwriting and answers two questions at once: what does it say, and whose handwriting is it? Both run from a single upload — a phone camera, a webcam, or a file — through a FastAPI backend that serves the two models independently or together.

It is deployed and you can use it right now. The interface ships five preset sample images, so trying it takes one click and no handwriting of your own.

The two halves are genuinely different problems. Transcription is a sequence task: variable-length input, variable-length output, no alignment between them. Style identification is plain image classification. Solving them separately and combining the results at the API layer kept each model simple.

Approach

Transcription uses a CRNN — convolutional layers to read the image, then recurrent layers over the resulting feature sequence, trained with CTC loss so the model never needs character-level alignment labels. Decoding is CTC beam search rather than greedy argmax, followed by a trigram spell-correction pass. That last step matters more than it sounds: the model reliably confuses visually similar characters, and hello wor1d becomes hello world from context alone.

Two separate weight sets handle the two input shapes — 512×32 for full lines, 128×32 for single words — because a model trained on line-length sequences degrades badly on a single word.

Style identification is a ResNet-18 fine-tuned over five writer/font families — Caveat, Courier, Lobster, Merriweather and Roboto. The training set is generated rather than collected: a text corpus is rendered in each font, then augmented with rotation, perspective shift, blur and noise so the classifier learns stroke shape instead of memorising clean glyphs.

Preprocessing turned out to be the difference between working and not. Photos of paper carry shadows and uneven lighting that wreck a model trained on clean scans. Rather than a fixed threshold, the pipeline stretches the white point using the 90th percentile of brightness — that removes background and shadow while leaving faint pencil strokes intact, which a naive threshold erases along with the noise.

What I would do differently

The classifier is trained entirely on rendered fonts, so it has never seen real handwriting during training. It generalises to photographed writing better than I expected, but the honest test is a held-out set of genuine handwriting from writers outside the training set, and that is the number I would want before claiming it works.