Wowzer. Now that's a challenging problem! Multiply the difficulty of speech recognition with the difficulty of self-driving cars, and maybe that's about how difficult it would be.
Lip reading is so dependant on context of the conversation. There are so many possible combination of words that could look the same from a lip reading perspective. Classic example is how "elephant shoe" looks like "I love you". Not only that, everyone's lips are different, generally females are easier to lip read, but then what if they have an accent? If english is their second language then that could spell trouble. Some people are mumblers and barely move their lips at all. Sometimes their lips are obscured with facial hair or they keep blocking the view or facing the wrong direction, etc..
Some of the points you mention are not unlike the ones that ASR faces:
Lip reading is so dependant on context of the conversation. -> So is speech recognition. And actually, it's the same context.
There are so many possible combination of words that could look the same from a lip reading perspective. -> That's the same in speech recognition, and it's the reason you don't just model the acoustics but also have a language model.
everyone's lips are different -> Everyone's voice is different.
but then what if they have an accent? -> That is possibly a bigger problem for ASR than for lip reading, but that's just my guess.
However:
facing the wrong direction -> This for me is the real trouble. In the video clips you see in the paper (as stills), you have nice full frontal view of the lips. But when do people actually hold that still outside a lab situation?
In ASR, if people turn their head, your signal quality drops (and hence your recognition rate does too) but when you move the head > 80-90 degrees away from the camera, you have pretty much no signal left to work with.
one example: https://about.sourcegraph.com/strange-loop/strange-loop-2019...
another: I have RSI, and while I use speech recognition, my voice gets fatigued really easily. Also, using speech recognition in public can annoy people. Real time lip reading recognition would be a serious boon for me.
Lip reading is so dependant on context of the conversation. There are so many possible combination of words that could look the same from a lip reading perspective. Classic example is how "elephant shoe" looks like "I love you". Not only that, everyone's lips are different, generally females are easier to lip read, but then what if they have an accent? If english is their second language then that could spell trouble. Some people are mumblers and barely move their lips at all. Sometimes their lips are obscured with facial hair or they keep blocking the view or facing the wrong direction, etc..