All roads lead to porn

[trochee]

Our web-crawler corpus of Chinese text (useful in improving speech recognition's language model) has a substantial subsection that includes porn in Chinese. I wouldn't have known -- my Chinese reading skills are for shite -- but my next-desk-neighbor started laughing really hard.

The irony of the whole thing is that this data probably helps our system quite a bit -- although the topic isn't very helpful for certain kinds of dialogue, it's still a pretty good source of statistics about Chinese word frequency.

I'm sure that following links at random in English-language sites would also frequently wind up in pornography too, but now I run dangerously close to a long and probably boring digression on web connectedness graphs.

Comments

[evan.livejournal.com]

porn sites often try to spam search engines with keywords, so they're actually bad samples of real text.

[isolt.livejournal.com]

"The internet is for porn!
The internet is for porn!
Why you think the net was born?
Porn, porn, porn!"
(Avenue Q, "The Internet is for Porn")

[trochee.livejournal.com]

but apparently this was dirty stories, so it's probably newsgroups rather than web cruft.

But I didn't get to ask too many details.