LLM pretraining datamixes often emphasize general knowledge, reasoning, and coding. The human “training mix” includes far more samples of visual/spatial/motor tasks, which come about naturally in the embodied human experience.

World models like Sora and Genie are pretrained on video and 3D video game data and excel at predicting the behavior of the real world. But no current model is at the frontier of both reasoning/coding and spatial reasoning/world modeling.

We’d expect (and it seems empirically true) that LLMs trained primarily on text are worse than humans on visual/spatial tasks. Computer-aided design (CAD) tasks require strong 3D reasoning ability as well as common-sense world knowledge, so LLMs might struggle with these tasks.

The experiment

I started with a practical CAD task I wanted done: designing a 3D-printable wall mount for my bike pump. Could some LLM do this task for me?

Current models can’t use graphical CAD programs like FreeCAD, but they’re great at writing code, so I had the models use OpenSCAD. Here’s the prompt:

Design a wall mount for this Lezyne Steel Floor Drive bike pump that I can 3D print. […] It should hold the bike pump by the handle, so that the bike pump hangs with the dial facing outward. It should hold the pump far enough away from the wall that the valve (which sticks out from the bottom of the pump) doesn’t touch the wall. Orient and position the design so that the wall is the YZ plane, and the mount protrudes into the positive X direction and is symmetric about the XZ plane. […]

Implement your design in openscad. […] Keep iterating on your design using the provided tool(s) until your most recent mujoco_mount_sim call returns *ONLY* the status “object_held” and *NO OTHER STATUSES*. If you get any other status, it means your design was not successful. Before each call to mujoco_mount_sim, write 1-3 sentences about how your design will work and/or how you will fix the issue(s) with previous versions of your design. […]

To test the designs, I made a 3D scan of the bike pump using Luma AI and created a simulation using MuJoCo to check whether the mount holds the pump.

I put each model in an agentic loop where it could call the simulator up to 10 times. If the mount held the pump (i.e. the pump was touching the mount after 5 seconds) then the design passed.

I also tried two other objects: a model of a pan from Amazon Berkeley Objects and a mug from Google Scanned Objects. I evaluated 7 LLMs and did 10 trials per (LLM x object) pair.

Code for this project is here.

Results and commentary

See all results here

Claude Opus 4.6 is best at this task. In the table I only evaluate whether the mount held the object, and Claude gets perfect marks. Subjectively, most of its designs are not directly usable but almost all are close. They are sometimes too large, too small, would be too weak if 3D-printed in plastic, or are random shapes that coincidentally work. This capability seems new; I did a smaller run with Claude Opus 4.1 and it failed 100% of the time.

GPT-5.2 has a good pass rate but its designs are subjectively quite bad and almost all would need to be completely reworked. Almost all of its designs have redundant parts, and they often are too weak or have “floating” components that are physically impossible (I could check for this but wanted to avoid scope creep).

Gemini 3.1 Pro and 3 Flash sometimes produce great designs. For example, here is Flash one-shotting a usable design for 2.5 cents. However, these models often end up in loops or fail to make use of all 10 turns. Other times they produce garbled designs similar to GPT-5.2. They often act erratically, producing random words in their commentary. Pro and Flash perform and behave very similarly.

All the open-weight models do poorly. Even in cases where they technically hold the object, their designs are subjectively quite bad and are often overly simplistic, only work by accident, or have floating parts. Kimi K2.5 is noticeably closer to producing usable designs than the other two however.

Under the hood

Creating the simulator and building the agentic harness was the bulk of the work on this project. MuJoCo is complex and powerful and often has surprising behaviors. LLMs often make mistakes when calling tools and I had to carefully validate the simulator input to distinguish tool call failures from legitimate bugs in my code.

One surprising bottleneck was convex decomposition. MuJoCo can only simulate objects composed of convex components, and so concave geometries have to be broken down into multiple convex geoms. The SOTA method for this is CoACD, and it’s quite slow. Generating the above table took 15.9 hours of CoACD processing time on my potato-class Hetzner server (almost as long as the 21.8 hours spent calling the model providers).

Future work

I built a simple custom agent harness for this, but it’s possible I could get better results using an off-the-shelf harness like Codex or Claude Code and turning my MuJoCo simulator into a CLI or MCP tool. These could provide a better system prompt and tools like memory to help keep the agent on-track.

Including more objects would make this into a better, more realistic eval. Amazon Berkeley Objects and Google Scanned Objects have ~8k and ~1k 3D models respectively, and although some are irrelevant (e.g. couches), the set of objects could be expanded without much effort.

The biggest thing that could be improved is the grading of results, by checking many aspects of each design and scoring them on a rubric. Here’s a non-exhaustive list of additional things that could be checked:

• Does the mount have “floating” parts or multiple parts that would have to separately be attached to the wall?
• Does the mount still hold the object if the object is perturbed?
• Can the object be easily lifted off the mount? (Try moving the object along several reasonable trajectories and see if the object hits the mount).
• Can the object be easily grabbed while in the mount? (Define an exclusion zone around the point where one would grab the object and see if it intersects the mount).
• Does the mount have a big enough contact patch with the wall?
• Does the mount intersect the wall?
• How much material does the mount use? (Actually slice it with PrusaSlicer and check the estimated filament usage).
• Does the mount fit in the build volume of a typical 3D printer?
• Are there thin sections of the model which would be weak when printed?
• How much weight does the model hold before deforming, using a finite element analysis?
• Can the screw / nail holes in the mount be accessed by a screwdriver / hammer? (Define exclusion zones around the holes and see if they intersect the mount).

Related stuff

• CAD-Coder: An Open-Source Vision-Language Model for Computer-Aided Design Code Generation
• How good is GPT-5 at 3D?

The law of total variance states that for two random variables \(X\) and \(Y\),

If \(Y\) is a mixture of distributions, this tells you that \(\mathrm{Var}(Y)\) is the mean variance of the components plus the variance of the means of the components (\(X\) in this case is a discrete variable indicating which component you selected).

At first blush this makes sense: jumping from \(\bar{Y}\) to \(\mu_i\) (the mean of component \(i\)) gives you some variance, and then jumping from \(\mu_i\) to your sample gives you some additional variance. But! These are dependent jumps, so you can’t naively add the variances.

Some pretty graphs

Let’s concretely say we have an equal mixture of 4 Gaussians with means -8, -2, 2, and 8. If the 2 outer Gaussians have variance 9 and the inner Gaussians have variance 1, you get this distribution:

Compare this to a version where the inner Gaussians have high variance:

The law of total variance says these two distributions have exactly the same overall variance (which works out to 39). That’s surprising to me! I would intuitively expect the first graph to have higher variance because the wider Gaussians on the edges lead to more extreme values.

A discrete version

Let’s try to understand what’s happening by replacing these Gaussians with Bernoulli distributions ¹ with the same means and variances. For example, a Gaussian with mean 2 and variance 9 becomes a coin flip where heads is 5 and tails is -1.

The variances of the two lists [-11, -5, -3, -1, 1, 3, 5, 11] and [-9, -7, -5, -1, 1, 5, 7, 9] are indeed both 39. Cool I guess? But it’s not really illuminating.

Instead let’s sic ✨algebra✨ on the problem. We’ll zoom in on one of these Bernoulli distributions which is a coin flip between \(\mu_i - \sigma_i\) and \(\mu_i + \sigma_i\). We can add up their contributions to the variance. (Here \(N=8\) is the total number of samples):

So on average, each sample contributes \(\frac{\mu_i^2 + \sigma_i^2}{N}\) to the total variance. That’s what the law of total variance says! It works, at least in this simple case.

OK so why does it actually work

\(\mathrm{Var}(X + Y) = \mathrm{Var}(X) + \mathrm{Var}(Y)\) if \(X\) and \(Y\) are uncorrelated. They don’t necessarily need to be independent (which is a stronger condition). You can see this from the more general formula

which works for all \(X\) and \(Y\). Here, once we pick \(X=\mu_i\), the distribution of \((Y \vert X = \mu_i) - \mu_i\) is zero-mean by definition, and so it’s uncorrelated with which \(\mu_i\) we chose.

money is a social construct, a way to keep track of what society thinks you deserve in terms of goods and services

That is: Bob makes me a sandwich. I give Bob $5. In doing so, I proclaim to the world that Bob’s sandwich has increased my happiness by five units. The whole of society is indebted to Bob for the Good he has done.

Bob then goes to Amazon and buys a pair of socks with the $5. Society repays his debt by giving him a pair of socks that make him five units happier. The money cycle continues: Jeff Bezos has done $5 of Good, and Jeff calls in the favor by buying (part of) a G700, etc.

A challenge to our modern free market system is that people do not believe this. People do not believe that the money you have reflects how much goodwill you deserve. For good reason! Wealth is often earned in ways that have minimal or negative effects on global utility. Wealth is often inherited by heirs who did nothing to earn it. Even a self-made fortune through honest work is highly random and path-dependent.

That money equals social good is a myth of the free market system. A myth in the sense that it is a meme, which, though not entirely true, reflects the ethos of the system.

If you believe that free market capitalism is the least-bad economic system and want it to succeed (and I do ¹²³), it is imperative to continually push for the myth to be more true. That means taxing externalities, raising inheritance taxes, and closing loopholes and regulatory arbitrage. ⁴

• dosing: It’s easy to judge how much you have consumed.
• moderation: It’s physically difficult to consume a lethal dose.
• hydration: Users consume water alongside the drugs.

I’d argue that American society would have a healthier relationship with most recreational drugs if they were legal but could only be sold diluted in drinks. This approach could work for popular but currently scheduled drugs like MDMA, LSD, psilocybin, mescaline, ketamine, cocaine, amphetamine, heroin¹, and morphine¹, and could be an alternative delivery route for legal drugs like nicotine and THC.²

If legalized in this fashion, I’d propose the “beer standard” for intoxicating drugs. Drugs like LSD and psilocybin would be required to be diluted such that one 12 oz drink produces a subjectively similar level of intoxication for the average user as one 12 oz 5% ABV beer. For stimulant drugs like cocaine and amphetamine the standard would be similar to weak coffee.

Dosing is tricky for many recreational drugs because milligram/microgram-scale effective doses can’t be accurately measured without special tools. Illicit drugs generally aren’t sold in standard packaging listing the drug(s) and dosage. This contributes to the over 100,000 US drug overdose deaths per year.

Coca tea, poppy tea, mushroom tea, and cannabis drinks are existing incarnations of this idea.

For instance, I recently accepted an offer for a team at Anthropic that is distributed across San Francisco, New York, and Seattle, and I have the flexibility to live and work in any of those cities. Where should I choose to live? San Francisco is the center of mass of Anthropic and has great nature and weather, New York is where I and most of my friends currently are, and Seattle also has great nature and is lower-tax/lower-cost-of-living.

If I had a “portfolio” of 100 of me and I could allocate them to these different strategies, I’d probably have 45 move to the San Francisco, 35 move to Seattle, 15 stay in New York, and 5 would renege the Anthropic offer and start a startup or pursue a PhD or travel the world or make art. These numbers are from my gut, but what I’m trying to maximize is the “return” in the sense of expected happiness for the whole portfolio.¹

I only have one of me, so it makes most sense to do the modal thing, and I’m in fact planning to move to San Francisco in a few months. (The 45 replicas in San Francisco would in turn make different sub-decisions about which neighborhood to live in, whom to live with, etc.)

This exercise helps you think through decisions, and moreover acknowledges the paths you didn’t take which helps with FOMO (at least for me). Yes, one of the hundred would move abroad and pursue a singing career, but you can take only one route, and generally the mode is the best.

The core functionality comes from 3 lines in my ~/.bashrc:

homegit() {
    git --git-dir=$HOME/.homegit --work-tree=$HOME "$@"
}

This defines a command homegit that works like regular git except that it saves its state in ~/.homegit (as opposed to ~/.git). Having a special command helps avoid accidents, like thinking I’m in a project repo, running git clean -dfx, and deleting everything in my home. I also use git_prompt_info in my Zsh PS1 to show my current Git branch, and by using --git-dir=$HOME/.homegit I avoid seeing branch information all the time.

I also define this command:

homegit-private() {
    git --git-dir=$HOME/.homegit-private --work-tree=$HOME "$@"
}

homegit-private works just like homegit, with the same Git toplevel dir (!) The difference is that I push the contents to a private repo. This allows me to version things like my SSH config and easily keep it in sync across devices. It’s also where I put anything that isn’t the same between my home and work setups.

When I want to pull my dotfiles onto a new device, I run these commands:

git clone git@github.com:kerrickstaley/homedir
mv homedir/.git ~/.homegit
git --git-dir=$HOME/.homegit diff  # check that the next command won't overwrite anything
git --git-dir=$HOME/.homegit reset --hard HEAD

The last piece that helps with all this is the runningon command. This allows me to put conditional blocks in my rc files like this:

# Default to Gnu binaries on macOS.
if runningon macos; then
    export PATH="/opt/homebrew/opt/coreutils/libexec/gnubin:$PATH"  # coreutils
    export PATH="/opt/homebrew/opt/grep/libexec/gnubin:$PATH"       # grep
    export PATH="/opt/homebrew/opt/gnu-sed/libexec/gnubin:$PATH"    # sed
    export PATH="/opt/homebrew/opt/gnu-tar/libexec/gnubin:$PATH"    # tar
fi

That way, I can use the same rc files on both macOS and Linux, at both home and work.

If you want to use this system yourself, you just need to copy these 7 lines into your .bashrc or .zshrc, and copy the runningon script into your ~/bin and modify the list of hostnames.

Update 2023-11-25:

The vcs-home wiki page links to many other approaches to this idea.

To avoid messy git status output, I’ve run

homegit config status.showUntrackedFiles no
homegit-private config status.showUntrackedFiles no

on all my machines as suggested in this HN post.

I enjoy micro-optimizing things, so I have a device that tells me when to leave my apartment to catch the train. At 10:53 AM, it told me to leave, so I did; it was lining me up for a train scheduled to depart at 11:01 AM. The station is 10 minutes away walking but the transit panel thought the train was running 2 minutes late.

At the station I waited about 2 minutes for the train to appear and another 2 for it to leave once I boarded. 4 minutes ¹ of precious time which could have been spent blogging was instead wasted waiting in the near-freezing station. What happened—why did my transit panel mislead me so?

A system’s output is only as good as the input you feed to it, and it turns out the transit panel was itself misled by the API it was using to get realtime PATH departures. That API had an overly-optimistic view of how late the train was running. Let’s look at the data!

Here’s a graph of how many minutes were left before the train departed according to 4 sources, in the 10-odd minutes leading up to departure:

The gray line is how much time was left until departure according to the train schedule. By “departure” I mean when the doors closed. The green line uses data from an API made by Matt Razza.² The blue line uses data from an “official” API.³ The orange line is the actual time left (as if we had a crystal ball which knew the exact departure time). The dashed red line is the departure.

This graph is easier to interpret if you instead look at how late the sources predict the train will be:

The train was ultimately 3m32s late, but intially the two APIs estimated it at 1m48s late. As the actual departure time drew closer, the APIs became more accurate and eventually overshot, predicting it would leave 13 seconds after it did.⁴

To catch the train I needed to leave 10 minutes before departure. Here’s what that looks like superimposed on the first graph:

The intersections of the magenta line with the 4 timelines are the times I should leave my apartment according to these 4 sources:

Following the schedule, I would have left 3m32s too early. Since my transit panel uses the “official” API, I was able to save half of that. If mrazza’s API—which gives better data (see below) but is down—were available, I could have saved another minute. And if the API made perfect predictions, it would have saved the remaining 46 seconds.

Squinting at that second graph, the “official” API looks an awful lot like a lagged version of Matt Razza’s API. Indeed, if we scoot it ahead 80 seconds, we see that it aligns pretty well:

I hypothesize that the same data source feeds both the mrazza and “official” APIs (Matt’s blog post has more information about the data infra that PATH uses on the backend) and something in the “official” API adds a random delay of 70 to 90 seconds. So if you’re looking at the real-time departures on PATH’s website, the info you’re looking at is a little stale compared to what the backend actually knows.

That’s all for now,⁵ but maybe in a future episode I’ll revisit the data that we get from PATH’s API. The Jupyter notebook where I did this analysis is here. The scraper that recorded the data is here. To record the train departure time I used a simple Google Form (Google Forms records the timestamp when you submit a form).

Discussion on Hacker News

To make my life a little easier, I built a web app that runs on a tablet by my front door that tells me when I should leave my apartment to catch the next train. This is what it looks like:

I’ve been using it for 3 months and I’m really happy with it. It feels great to walk at a leisurely pace to the station and show up 2 minutes before the train leaves, every time. Instead of waiting in a cold station I can wait in my warm apartment.

2 minutes before departure, train waiting in station

The rest of this post will explain how I made this thing, which I call a transit panel. It wasn’t hard, and with a bit of HTML/Javascript knowledge and handiness you can make one of your own!

Hardware

I used an Amazon Fire HD 10 tablet. This tablet is cheap ($150 at time of writing, but frequently on sale for $110 or less) and has a large-ish screen, which is important because I need to be able to read the screen from 10 meters away. The web app isn’t demanding and the system is always plugged in, so specs like processor and battery life don’t matter. The app runs full-time so the lockscreen ads aren’t an annoyance.

To mount it on the wall, I used the Dockem Koala Wall Mount 2.0, which worked well. It can be screwed to the wall or adhered using Command Strips; I chose to use these drywall anchors because Command Strips sometimes fall off after several months of use.

mount with tablet removed

I routed the power cable using these Monoprice cable clips.

Software

The transit panel runs a simple web app without any frameworks. The source code is here, with most of the logic in main.js. I’m not a JS wizard; don’t judge my code 😅 (but constructive comments are appreciated! Send me a PR).

currently 3 transit options are supported

Data Source

The app hard-codes schedules for the PATH and ferry (in departure_times.js). I wanted to use real-time data, but NY Waterway ferries don’t have real-time tracking at all, and the PATH has real-time information but they don’t have a Javascript-accessible web API (if you work at PATH, please implement this!).

Luckily, a true internet hero named Matthew Razza runs a web service that exposes the PATH data in a JSON HTTPS API. A second problem however is that the live PATH data doesn’t have a long enough time horizon. I take 10 minutes to walk to the station, and oftentimes the only departures returned by the API are less than 10 minutes in the future. (This affects everyone, including people using the official website and app—if you work at PATH, please fix this!).

I could combine the approaches, using the live API and falling back to the schedule if there is no data, but for now the hardcoded schedule works well enough.

Fonts

I spent a ton of time trying to center the numbers vertically in their rows (e.g. the 4 in “leave in 4 min”). I would tweak the CSS to center it in Firefox, but then I would open it in Chrome on the same computer and it would look different, and the tablet would be different from both.

I thus learned the hard way that if you want your app’s text to have a consistent appearance across platforms, you need to use a font from a font service like Google Fonts instead of relying on the browser’s built-in font library.

It turns out that font geometry is complicated and compensating for different fonts in CSS/JavaScript to vertically center text is hard. The simplest solution is to use a consistent font and tweak the margins in CSS to make it look like you want.

Libraries

I found the Luxon Javascript library really helpful for working with date/time values.

Browser

I used the Fennec browser from the F-Droid app store, which is a reskin of Firefox for Android. This was the only browser I found that behaved correctly when full-screened (hiding both the top and bottom system UI bars).

Reliability

I’ve kept the tablet running continuously for about 4 months. Once or twice in that period, it’s gotten into a bad state and needed a reboot. I’ve noticed no issues with burn-in on the screen. I’m hoping to get several years of use before the hardware needs replacement.

Future Improvements

At some point (maybe when it’s warmer) I’m planning to add a row showing bike availability at the nearest Citi Bike station. Citi Bike has a delightful and easy-to-use REST API. Speaking of weather, I’m also planning to add a row that shows the weather and current time.

This is a little optimistic. A year ago I could barely bumble through a basic conversation in Mandarin, and any sort of “real” Chinese text was totally inaccessible—I could only read things that were designed for language learners. Continuing to cram vocabulary didn’t seem to help. I couldn’t make it more than a sentence into a newspaper article or book without hitting an unfamiliar word and needing to pull out a dictionary.

This article is about some software I wrote, based on the Anki flashcard app, to help me leapfrog from HSK 4 to reading a real Chinese book. If you’re learning Chinese, you can download and use this software too! Fair warning, it’s still a lot of work. Chinese is hard.

The Three Body Problem

The novel The Three Body Problem (三体 [Sān Tǐ] in Chinese) is moderately famous in sci-fi circles. The English translation won the 2015 Hugo Award for Best Novel and to date it is the only novel in translation to have done so. It’s one of Barack Obama’s favorite books. And Amazon was reported in May 2018 to be eyeing film rights to the book for $1 billion USD, in a bid to boost Amazon Prime Video’s original content (no word on whether that did happen).

One of my friends (hi Tommy!) recommended The Three Body Problem to me, and he also mentioned that the book reads a little more fluidly in the original Chinese. And so I decided to try reading the Chinese version. So far, I’m 45 pages in and actually kinda enjoying it. Which is a start, right?

The “Prestudy” Technique

I call the method I’m using to read the book “prestudy”. Here’s how it works:

1. You come up with a list of vocabulary words you want to learn. Using the 3500 most common words works pretty well.
2. You take the first 3 or so pages of the book you’re reading, and you find out which words from (1) are in those pages.
3. You generate Anki flashcards for all the words from (2) that you don’t already have flashcards for.
4. You study the flashcards and learn the words.
5. You read the pages. If you encounter a word you don’t know, you look it up.
6. You repeat steps 2-5. You can pipeline it so that you’re studying words from page N+3 on the same day you’re reading page N.

With this technique you can read a page or so a day with moderate effort. The bottleneck is acquiring vocabulary; each page will generally have about 5-10 new words (starting from a HSK 4 base), and it’s difficult to memorize more than 5-10 new words per day. It gets gradually easier as you build up a vocabulary base and encounter fewer and fewer new words per page.

The technique also works pretty well for newspaper articles and TV shows (for shows, you’ll want to look for a .srt subtitles file).

This is not a totally new idea. It’s similar to many Chinese textbooks where each chapter presents a text passage and some related vocabulary. The difference is that you can make your own study guide, in the form of Anki flashcards, for anything you want to read.

The Tool

The tool that does all this is an Anki add-on which you can get here. You copy/paste in your text (so you’ll need a PDF if it’s a book), enter your target vocabulary size, and select which deck and tags you want to apply to the cards:

It’s only compatible with the beta-channel Anki 2.1, not the stable-channel Anki 2.0. Sorry if this is a dealbreaker for you. I know some people are stuck on 2.0 because certain add-ons only support 2.0. If I have time and Anki 2.1 continues to be stuck in beta, I’ll look at making a 2.0-compatible version.

It also only supports texts with simplified characters. I’ll eventually add support for traditional characters. The silver lining is that when you add a flashcard for a simplified character, you’ll also get a flashcard for the traditional character. It’ll be suspended by default so you’ll have to unsuspend if you want to study it.

The Code

All the code behind this is open-source, and it’s split across several components that can be re-used in other projects:

• Chinese-Prestudy: The Anki plugin itself.
• Chinese-Vocab-List: A giant YAML file of the 4000+ most common Chinese words (in descending order of frequency), with definitions and example sentences. Plus a small Python wrapper for accessing the data.
• chineseflashcards: A Python library, built on genanki, for creating Chinese flashcards in Anki.
• genanki: A Python library for creating Anki flashcards.

With the exception of genanki, none of these projects is in a very contributor-friendly state. Most of their code isn’t very readable or documented and could use more unit tests. Still, I’d encourage interested persons to dive in and make contributions; I’ll try my best to help you out and make the code more hackable as we go.

This project also leans heavily on a lot of great open-source projects:

• CC-CEDICT: A free (Creative Commons) Chinese/English dictionary.
• Tatoeba: Multilingual database of example sentences.
• The Hanyu Shuiping Kaoshi vocab list.
• SUBTLEX-CH: List of Chinese words from most to least common (in spoken usage).
• jieba: Chinese word segmentation library.
• pypinyin: Character-to-pinyin library.

Final Thoughts

Learning this way still demands a lot of perseverence, bordering on masochism, but that’s Chinese. On the bright side, I feel like I’m developing proficiency faster than any point other than when I was in undergrad taking Chinese classes 5 days a week.

Reading better has also lifted my listening and speaking abilities, even though I haven’t spent much time on those recently.

I still have to stop and look up a word every 3 or 4 sentences, which is a pain. I usually use the OCR feature in the Hanping Chinese Camera app, and on an unsteady train (where I normally read) this gets frustrating fast. I’m working on a solution for this too: a “cheatsheet” that lists all the advanced words so you don’t have to study them with flashcards. But it’s not done yet.

I hope that you find this tool useful on your Chinese learning journey! Feel free to leave feedback on GitHub’s issue tracker or by mail to k@kerrickstaley.com. Upvotes on Hacker News are also appreciated!

I wanted to get a transcript of the episode’s dialog so I could study the unfamiliar vocabulary. Unfortunately, the video files I have only have hard subtitles, i.e. the subtitles are images directly composited into the video stream. After an hour spent scouring both the English- and Chinese- language webs, I couldn’t find any soft subs (e.g. SRT format) for the show.

So I thought it’d be interesting to try to convert the hard subs in the video files to text. For example, here’s a frame of the video:

From this frame, we want to extract the text “怎么去这么远的地方“. To approach this, we’re going to use the Tesseract library and the PyOCR binding for it.

We could just try throwing Tesseract at it and see what comes out:

import pyocr
from PIL import Image

LANG = 'chi_sim'

tool = pyocr.get_available_tools()[0]
print(tool.image_to_string(Image.open('car_scene.png'), lang=LANG))

Running it:

$ python snippet_1.py

$ 

Hmm, so that didn’t work. What’s happening?

Tesseract requires that you clean your input image before you do OCR. Our input image is full of irrelevant background features but Tesseract expects clean black text on a white background (or white on black).

To remove the background image and get just the subtitles, we turn to OpenCV. The easiest part is cropping the image. We keep a larger left/right border because some frames have more text:

import cv2

TEXT_TOP = 621
TEXT_BOTTOM = 684
TEXT_LEFT = 250
TEXT_RIGHT = 1030


img = cv2.imread('car_scene.png')

cropped = img[TEXT_TOP:TEXT_BOTTOM, TEXT_LEFT:TEXT_RIGHT]

cv2.imshow('cropped', cropped)
cv2.waitKey(10000)

The result:

Now we want to isolate the text. The text is white, so we can mask out all the areas in the image that aren’t white:

white_region = cv2.inRange(cropped, (200, 200, 200), (255, 255, 255))

This uses the OpenCV inRange function. inRange returns a value of 255 (pure white in an 8-bit grayscale context) for pixels where the red, blue, and green components are all between 200 and 255, and 0 (black) for pixels that are outside this range. This is called thresholding. Here’s what we get:

A lot better! Let’s run Tesseract again:

extracted_text = tool.image_to_string(Image.fromarray(white_region), lang=LANG)
print(extracted_text)

And Tesseract returns (drumroll…):

′…′二′′′'′ 怎么去逯么远的地方 '/′

Now we’re getting somewhere! Several areas in the background are white, so when we pass those through to Tesseract it interprets them as assorted punctuation. Let’s strip out these non-Chinese characters using the built-in Python unicodedata library:

import unicodedata

chinese_text = []
for c in extracted_text:
  if unicodedata.category(c) == 'Lo':
    chinese_text.append(c)
chinese_text = ''.join(chinese_text)

print(chinese_text)

The 'Lo' here is one of the General Categories that Unicode assigns to characters and stands for “Letter, other”. It’s good for extracting East Asian characters. From this code we get:

二怎么去逯么远的地方

There are two mistakes here: a spurious 二 character on the front, and a mismatched character in the middle (that 逯 should be 这). Still, not bad!

That’s all for now, but in Part 2 (and maybe Part 3?) of this post series I’ll discuss how we can use some more advanced techniques to perfect the above example and also handle cases where extracting the text isn’t so straightforward. If you can’t wait until then, the code is on GitHub.

If you have any comments about this post, join the discussion on Hacker News, and if you enjoyed it, please upvote on HN!