This is what I was thinking. I don't know about Japanese, but sentence structure in mandarin is pretty much reversed from what it is in romance languages. And so much of character meaning is based on context that outside of common phrases it might be nearly impossible.
On the other hand, an English to Chinese translator might be more doable, and might also be more commercially profitable.
Even an app that converted characters into the most direct meaning could provide insight in a large number of cases into the intended meaning and would be valuable.
An app that converted from characters to pinyin (a much easier problem) would be gold to someone trying to learn mandarin. I'd easily pay $50 for something with that functionality without it even making an attempt at the english. It's a much smaller market than those trying to understand chinese signs on a vacation, but it's one with a much larger stake and interest in the result.
On the other hand, an English to Chinese translator might be more doable, and might also be more commercially profitable.