All roads lead to porn
Aug. 24th, 2004 01:06 pm![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png){kind=small}
trocheeOur web-crawler corpus of Chinese text (useful in improving speech recognition's language model) has a substantial subsection that includes porn in Chinese. I wouldn't have known -- my Chinese reading skills are for shite -- but my next-desk-neighbor started laughing really hard.
The irony of the whole thing is that this data probably helps our system quite a bit -- although the topic isn't very helpful for certain kinds of dialogue, it's still a pretty good source of statistics about Chinese word frequency.
I'm sure that following links at random in English-language sites would also frequently wind up in pornography too, but now I run dangerously close to a long and probably boring digression on web connectedness graphs.
The irony of the whole thing is that this data probably helps our system quite a bit -- although the topic isn't very helpful for certain kinds of dialogue, it's still a pretty good source of statistics about Chinese word frequency.
I'm sure that following links at random in English-language sites would also frequently wind up in pornography too, but now I run dangerously close to a long and probably boring digression on web connectedness graphs.
