Prior art: zoom on a moving ball

After deciding that the vision phase of the project was going to be my focus, I found some prior art there too. The Ishikawa Oku Laboratory at the University of Tokyo has some impressive stuff they've done.

Here's the brief video of their accomplishments in ping pong:

To summarize, they track fast-moving objects -- and stay zoomed on them -- by using some high-speed servos to move mirrors, instead of moving the camera. That allows them to see detail on the ping-pong ball as it flies through the air. They get enough detail that they can identify the spin on the ball, because they can zoom in so much.

I imagine that with that kind of detail, a robot could make some pretty good "thinking" phase decisions, and play a very competitive game.

So what can I learn from them? Thankfully, like good academics, they have released a paper that provides some details: High-speed Gaze Controller for Millisecond-order Pan/tilt Camera.

I learn that they used some expensive equipment -- at least in the scale of the self-funded researcher. They are using two M2 mirrors from GSI, which are high-quality mirrors on high-speed servos, designed for scanning stuff with lasers. I think I saw them costing $700 each but I can't find the link now; maybe I'm fooling myself. They also use a 1000fps camera which, as I've discovered since, is an expensive toy. I think it is described in more detail in this non-free paper. They also use their own custom fast-acting lens to keep everything in focus at those speeds.

Altogether, this is out of my league... by an even greater margin than the rest of the project. But do I really need all that? I don't think so.

Let's think about the needed frame rate on the camera. I'd like to think of it in terms of how far the ball moves between frames, and try to keep that reasonable. That requires knowing how fast the ball is moving. This site concludes 30 m/s is the upper end for professional players. I did some rough calculations on me playing at a very gentle pace against the playback on my table: 4 m/s. I'm going to use 10 m/s as my target speed. So at 1000 fps, the ball moves 10mm. That's pretty small. Not a lot happens in that time. It's not like the ball is actively powered -- it isn't accelerating itself. There should be predictable forces acting on the ball: gravity, air resistance, spin, bounces off a uniform table. I think we could easily get away with the ball moving 100mm between frames, if we had good precision about our measurements in each frame. So that would mean 100 fps. Any slower and I do worry that we wouldn't have enough observations of the ball in flight to accurately determine its future path. This jives with the Zhejiang group from my previous post, who use 120fps cameras.

The second strength Ishikawa Oku has is the ability to zoom in on the ball. That would be nice, as it would allow us to determine its position and velocity more accurately by using the same pixels to cover a smaller area. But the complexity of their tracking system just isn't realistic for my first attempts. Ditto with detecting spin (a product of high frame rate, high resolution via zooming, and high-speed focus). I think it is a level of refinement more than I need to get the basic job done.

So this post reviewed some very cool work, but I've decided that is is overkill for my purposes. I'm looking for something around 100 fps that doesn't need any fancy tracking.


As an aside, the Ishikawa Oku Lab is my most favoritist. They seem to do all sorts of awesome robot stuff. Like this high-speed robot hand that I stumbled across a year ago and that still amazes me.