This is a project I've been mulling over for several months and what better time to begin than at the start of a new year. I've been an academic and professional in the field of natural language processing (NLP, aka computational linguistics or "data science for language") for about a decade now, and it's time to put those skills toward something fun. On this blog, I'll show how we can use the tools and methods of NLP to solve word puzzles, riddles and the like. I plan to primarily focus on the weekly Sunday Puzzle from Weekend Edition on NPR.
Each Sunday, Will Shortz, editor of the New York Times crossword puzzle hosts the Sunday Puzzle, a roughly five minute segment featuring two puzzles. The first puzzle is the on-air challenge, in which a caller is presented with a series of rapid-fire puzzle questions, usually following some pattern. Then the segment closes with the weekly challenge, where Shortz reads some puzzle and listeners have until Thursday that week to submit the solution. Among the correct responses, a random winner is drawn, and that person is the caller who participates in the next on-air challenge.
This blog will focus on these weekly challenges. I won't be spoiling the answers, however; I plan to post solutions after the Thursday deadline only--assuming I'm able to find the solution, that is! I may also discuss how NLP could augment the on-air puzzles, and sometimes I'll look at older puzzles or puzzles from other sources entirely. I also expect to recruit occasional "guest bloggers" to help when I get stuck or feel that a particular problem plays well to a colleague's strengths or expertise.
In terms of the kinds of NLP tools I'll be discussing here, you can expect to see some Python scripting, NLP and data science packages like NLTK, spaCy, as well as tools like the Stanford CoreNLP and Parser kit and BERT models. We'll no doubt use methods like part-of-speech tagging, dependency parsing, word embeddings, n-gram language modeling, sentence transformers, tf-idf, mutual information, similarity metrics, machine translation, web scraping, code and cypher cracking and so on. My goal is to provide practical examples of how such tools can be used to solve various kinds of language problems. Most of this will involve solving puzzles, but the methods and tools discussed here are applicable to many practical problems as well.
I'd love to see this become a collaborative space where others can share their thoughts on how to solve problems with NLP as well. There's rarely a single "right" approach to this.
No comments:
Post a Comment