Natural Language Processing

Course Description   Readings   Grading & Assignments   Schedule

Mon & Thu 11.45am @ Snell Engineering Center 108

Announcements/Discussions @ Piazza

Course Description

The widespread adoption of digital information systems and the Web led to a deluge of text data from a variety of genres, languages, and domains (e.g., news articles, tweets, clinical notes in Electronic Health Records). How can we use computers to help us sift through and make sense of all this data? What can we learn from analyzing natural language data at scale?

Natural Language Processing is a subfield of Artificial Intelligence that uses methods from Computer Science, Computational Linguistics, Cognitive Science, Statistics, and Machine Learning to give computers the ability to automatically analyze, categorize, understand, and generate natural language. This is challenging because unlike other kinds of language (say, programming languages) natural language is unstructured and often ambiguous, nuanced and subjective. In this course we will learn about:

• the linguistic phenomena that make NLP hard for computers to approach
• the main NLP problems and tasks, and strategies to address them
• the role of data and machine learning in NLP systems
• the ethical considerations and potentials for bias in NLP systems
• how to formulate and evaluate NLP solutions to address real-world problems

The course will be very much hands-on with a great emphasis on methods, meaning that we will spend most of our time discussing and implementing (often from scratch) typical approaches to solve key NLP tasks. Recent advances in neural networks and deep learning models led to remarkable breakthroughs is NLP. However, this is intended to be an introductory course and thus we will not be jumping straight into these models. Instead, we will first become familiarized with classical statistical methods which are both important baselines for NLP tasks and lay the foundation for more sophisticated approaches. Then, in the second half of the semester, we will dive deep into neural networks starting from simple Multilayer Perceptrons and building our way up to state-of-the-art models based on pre-trained Transformers.

Prerequisites

This class has no official prerequisites, however, modern NLP relies heavily on statistical methods, machine learning and deep learning. Therefore, students must be comfortable with basic mathematical concepts from Linear Algebra, Probability and Calculus. We will briefly review some of the main concepts needed for the models and algorithms that we will cover in the class. However, this will be a review and NOT a thorough and rigorous exposition of these subjects. Students are thus encouraged to proactively fill any gaps in their knowledge.

We will make extensive use of python3, scientific computing libraries (e.g., numpy, scipy, matplotlib), and jupyter notebooks. If you have less experience working with python and notebooks, we highly encourage you to make time to come to office hours in the first few weeks of the course.

Readings

Main Text Speech and Language Processing 3rd Edition Dan Jurafsky and James H. Martin Occasionally, we will supplement this text with readings from research papers and other freely available sources.
Dive into Deep Learning Aston Zhang, Zachary C. Lipton, Mu Li and Alexander J. Smola
Mathematics for Machine Learning Marc Peter Deisenroth, A. Aldo Faisal, and Cheng Soon Ong

Grading and Assignments

• Homework (50%)
• Research paper presentation (CS6120 students only; this will count as an additional homework)
  1. Reading and presenting research papers in small groups
  2. Reviewing presentations from your peers
• Quizzes (20%)
• Final project (20%)
  1. proposal
  2. code & write-up
  3. video presentation
  4. presentation review
  5. reflection
• Class participation (10%)
  1. Asking and answering questions in class
  2. Completing lecture notebooks

Homework is due Fridays at 11pm and quizzes are due Sundays at 11pm. Notebooks for a given week can be submitted until Sunday 11pm.

Notes: Homework assignments have a small number of additional questions for CS6120 students (which count as extra-credit for CS4120). You can think of quizzes as mini take-home exams. The questions will typically focus on material covered in class that week, and may have a small number of questions based on readings for the upcoming week. You can make a request to retake one of the quizzes.

Late Policy

All homework should be turned in on time whenever possible. However, you can turn in your work up to 2 days late (48 hours from the deadline) with a penalty: 25% for the first day and 50% for the second day. Once a semester, you may turn in your homework up to 24 hours late without penalty or explanation — no other extensions will be granted. This policy only applies to homework; quizzes and final project may not be completed after the deadline. If you miss a quiz, make sure to attempt the extra-credit on the homework.

Collaboration Policy

We encourage you to collaborate with your classmates, but remember that collaboration is different than working in pairs or as a group. A few key points to remember:

• Strategies: You may talk with your classmates about general strategies but you may not talk about specific solutions.
• Explaining concepts: You may talk with your classmates about how certain techniques work in general but not how to write any part (or sub-part) of the solution needed for the homework.
• Online solutions: You are expected to use the internet as a place for online resources, such as documentation, not as a place to get solutions to your assignments. This includes posting to sources like StackExchange, Reddit, Chegg, etc.
• Plagiarism: assignments and code that you turn in should be written entirely on your own. You should always consult the course instructional staff if you need extra help. A good rule of thumb: do not share your assignments and do not look at your classmates assignments (this makes it very hard to come up with your own solution afterwards); do not write code together unless the assignment explicitly states that you may work in pairs (this includes explaining your solutions).

Collaboration Policy violations will result in a 0 on the assignment in question. The university’s academic integrity policy discusses actions regarded as violations and consequences for students.

Letter Grades

Schedule

This a tentative schedule and subject to change.