CHAPTER 01 · Natural Language Processing · TESTED DEMONSTRATION

Natural Language Processing: your first working example

Natural language processing, often called NLP, works with human language as data. Text must be represented in a form a computer can count, compare or model. Words, tokens and sentences are related units but not always identical.

Why this example matters

Our short sample divides a sentence at spaces and counts the resulting words. Repeated terms appear more often in the count table, making word frequency easy to examine.

What you will be able to do

Recognize the input values and describe what they represent in Natural Language Processing.
Run the Natural Language Processing program and explain why its displayed output follows from the recorded operations.
Identify a limitation of the Natural Language Processing activity before applying similar reasoning to a real system.

Before you start

You do not need prior knowledge of Natural Language Processing. To run the example, install Python 3 or use a trusted Python editor. Type the code into a file, save it and run it with your editor or the command python3 filename.py.

Original worked program

The following Natural Language Processing demonstration uses small invented values so you can trace every operation. The Natural Language Processing activity uses only Python standard-library features; it downloads no model and calls no paid service.

Python 3 · runnable

from collections import Counter
sentence = "robots learn words and robots use words"
tokens = sentence.lower().split()
counts = Counter(tokens)
print("Unique words:", len(counts))
print("robots:", counts["robots"])
print("words:", counts["words"])

Actual output from Python

Unique words: 5
robots: 2
words: 2

Read each instruction

from collections import CounterThis import makes a standard Python tool available in Natural Language Processing. In the Natural Language Processing activity, check the named values on that line rather than memorizing symbols.
sentence = "robots learn words and robots use words"The assignment gives a name to a value or a calculated result in Natural Language Processing. In the Natural Language Processing activity, check the named values on that line rather than memorizing symbols.
tokens = sentence.lower().split()The assignment gives a name to a value or a calculated result in Natural Language Processing. In the Natural Language Processing activity, check the named values on that line rather than memorizing symbols.
counts = Counter(tokens)The assignment gives a name to a value or a calculated result in Natural Language Processing. In the Natural Language Processing activity, check the named values on that line rather than memorizing symbols.

Observe the results

Output line 1Unique words: 5

Output line 2robots: 2

Output line 3words: 2

To verify the Natural Language Processing result, copy the program into a fresh file and run it once without modification. Then change only one input. For the Natural Language Processing trial, record what you predict before pressing Run again. Compare the Natural Language Processing prediction with the observed result.

Second original practice: Cleaning text with care

The Natural Language Processing extension changes the task, not just the numbers in the first example. Study its new data structure or operation, then decide how the result supports the milestone “Cleaning text with care”.

Python 3 · extension verified

words = ["small", "robots", "can", "learn"]
pairs = list(zip(words,words[1:]))
print("Adjacent pairs:", pairs)
print("Pairs found:", len(pairs))

Actual second output

Adjacent pairs: [('small', 'robots'), ('robots', 'can'), ('can', 'learn')]
Pairs found: 3

For the Natural Language Processing extension, read the code from its first assignment or import to its final print statement. Explain the additional Natural Language Processing idea introduced by this extension, then design a check that could reveal a mistake. The Natural Language Processing extension output above was recorded by running that independent program.

What the example does not prove

Splitting only on spaces is insufficient for punctuation, contractions, languages without spaces or meaningful context. Counts cannot tell us whether a statement is true.

A reliable Natural Language Processing project also needs an explicit objective, test cases that were not used to tune the approach, documentation of the assumptions, and review of situations where the result might mislead the user. Do not mistake the small Natural Language Processing program for evidence of accuracy or safety in a deployed application.

How to validate your own result

For Natural Language Processing, a convincing result starts with a written claim about the actual task, not the appearance of a successful print statement. For Natural Language Processing, list the initial values and the one value you will vary; record a prediction that could be disproved. Save both Natural Language Processing outputs as separate records to avoid mixing up the results of the two investigations. A correct Python run only establishes that the particular instructions executed; it does not establish that the assumptions match every situation in which Natural Language Processing might be used.

Connect this trial to “Word frequencies” without claiming that the milestone is already complete. Name a measurement you would collect for Natural Language Processing, a plausible error source and a person able to review the decision. The later topic “Stop words and tradeoffs” deserves its own investigation with different evidence. Keeping these questions separate prevents your Natural Language Processing learning journal from becoming a collection of exercises that all make the same claim.

Finally, keep the Natural Language Processing experiment reproducible: include the Python version, exact script, input values, observed results and any unexpected behavior. If a Natural Language Processing investigation uses external data, document the source and your permission to use those records. A new learner should be able to explain every output line in ordinary words before moving into the next milestone of Natural Language Processing.

Second verified lab: Text in practice

To deepen Chapter 1 of Natural Language Processing, compare the first calculation with a different, executable experiment. The small text lab below illustrates a related technique, not every possible implementation of documents and tokens.

Python 3 · independently executed

# Course: Natural Language Processing | Chapter 1 | Subject: Documents and tokens
# A deliberately simple Unicode-aware token count
message = "A careful learner compares models and records results"
normalized = message.casefold()
words = normalized.split()
counts = {word: words.count(word) for word in sorted(set(words))}
print("Tokens:", len(words))
print("Vocabulary size:", len(counts))
print("Contains learner:", "learner" in counts)

Actual output of the displayed code

Tokens: 8
Vocabulary size: 8
Contains learner: True

While examining your first working example in the natural language processing course (chapter 1), apply this test: This tokenizer splits on whitespace only, demonstrating exactly which units enter the count and why more sophisticated tokenizers are needed. For the farm irrigation schedules exercise, distinguish what the code observes from any stronger predictive or causal claim.

While examining your first working example in the natural language processing course (chapter 1), apply this test: Interpretation limit. Whitespace tokenization does not capture subword tokens or punctuation rules in multilingual systems. Record the changed inputs and compare both outputs before extending the result to a project.Interpretation limit. Whitespace tokenization does not capture subword tokens or punctuation rules in multilingual systems. Record the changed inputs and compare both outputs before extending the result to a project.

20 practice questions and worked answers

Test your understanding of your first working example in the natural language processing course (chapter 1). Compare the claims in both runnable labs, and use the outputs to defend each numerical response.

1. What is the core question in Natural Language Processing — Documents and tokens — chapter 1 (Beginner orientation)?

The chapter examines documents and tokens through beginner orientation. State the specific input, observable outcome, and assumptions before choosing a calculation or AI method.

2. What does the first printed line of the introductory example report? (In your first working example, chapter 1 of natural language processing)

The verified run prints “Unique words: 5”. Check the actual statement in the script and identify the variable(s) that produced this line.

3. What is the first observed result in the follow-up lab? (In your first working example, chapter 1 of natural language processing)

For your first working example in the natural language processing course (chapter 1), evaluate the stated result against the example data: Executing the follow-up code for documents and tokens produces “Tokens: 8”. This is evidence of the example computation only.

4. What does the last line of the follow-up lab say? (In your first working example, chapter 1 of natural language processing)

For your first working example in the natural language processing course (chapter 1), evaluate the stated result against the example data: The recorded program ends with “Contains learner: True”. Identify the arithmetic or conditional expression used to obtain it.

5. How do the two code examples contribute different evidence? (In your first working example, chapter 1 of natural language processing)

For your first working example in the natural language processing course (chapter 1), evaluate the stated result against the example data: The first program provides a basic baseline; the follow-up is a second small experiment about text. Compare their output names and the assumptions required to interpret them.

6. Which input definition must be written before exploring documents and tokens? (In your first working example, chapter 1 of natural language processing)

For your first working example in the natural language processing course (chapter 1), evaluate the stated result against the example data: For farm irrigation schedules, specify a record boundary, its units, when it was observed, and whether records can be missing or duplicated.

7. What would be an invalid claim after running the follow-up code? (In your first working example, chapter 1 of natural language processing)

For your first working example in the natural language processing course (chapter 1), evaluate the stated result against the example data: It would be invalid to declare a production-quality natural language processing system proven merely because the toy text computation executed successfully.

8. Which failure is explicitly excluded from the small lab? (In your first working example, chapter 1 of natural language processing)

For your first working example in the natural language processing course (chapter 1), evaluate the stated result against the example data: Whitespace tokenization does not capture subword tokens or punctuation rules in multilingual systems. Add a specific counterexample before treating the method as dependable.

9. What mathematical or procedural idea does the second program demonstrate? (In your first working example, chapter 1 of natural language processing)

For your first working example in the natural language processing course (chapter 1), evaluate the stated result against the example data: This tokenizer splits on whitespace only, demonstrating exactly which units enter the count and why more sophisticated tokenizers are needed.

10. How can a learner reproduce this documents and tokens experiment? (In your first working example, chapter 1 of natural language processing)

For your first working example in the natural language processing course (chapter 1), evaluate the stated result against the example data: Use Python 3 without third-party packages, run the exact displayed code, record “Tokens: 8”, and compare each subsequent output line.

11. What single change would help probe sensitivity in Natural Language Processing — Documents and tokens — chapter 1 (Beginner orientation)?

For your first working example in the natural language processing course (chapter 1), evaluate the stated result against the example data: Alter one declared numeric input in the text lab, record the old and new value, and explain why all other inputs stay fixed for a fair comparison.

12. Why should a reviewer question the scenario data? (In your first working example, chapter 1 of natural language processing)

For your first working example in the natural language processing course (chapter 1), evaluate the stated result against the example data: The farm irrigation schedules numbers are invented for teaching. Before using actual records, document provenance, permission, missingness, and whether the sample represents the intended context.

13. How would you identify a computational bug in chapter 1? (In your first working example, chapter 1 of natural language processing)

For your first working example in the natural language processing course (chapter 1), evaluate the stated result against the example data: Calculate one output by hand, compare it against a corresponding printed line, and inspect the smallest expression producing any mismatch.

14. How can a method fail even when the Python code is correct? (In your first working example, chapter 1 of natural language processing)

An assumption about documents and tokens may be false; for example the underlying farm irrigation schedules could shift or be measured inconsistently.

15. What distinguishes an appropriate baseline from a more complicated system? (In your first working example, chapter 1 of natural language processing)

For your first working example in the natural language processing course (chapter 1), evaluate the stated result against the example data: The baseline must be understandable and measurable for farm irrigation schedules. A more complicated natural language processing implementation must show a relevant, independently evaluated improvement.

16. What must remain separate when translating this documents and tokens lesson? (In your first working example, chapter 1 of natural language processing)

For your first working example in the natural language processing course (chapter 1), evaluate the stated result against the example data: Keep executable Python code, variable names, mathematical notation and verified output in LTR with unchanged characters; the prose and page layout may use RTL.

17. How can the student avoid data leakage in a farm irrigation schedules evaluation? (In your first working example, chapter 1 of natural language processing)

For your first working example in the natural language processing course (chapter 1), evaluate the stated result against the example data: Form a future or independent holdout set before computing transformations learned from data. Do not let protected answers influence training or threshold selection.

18. What is an appropriate privacy safeguard for the example? (In your first working example, chapter 1 of natural language processing)

For your first working example in the natural language processing course (chapter 1), evaluate the stated result against the example data: Use fabricated data first. For any real farm irrigation schedules, minimize collected identifiers, define access and retention, and document consent or other lawful authority.

19. How should an expert challenge documents and tokens conclusions? (In your first working example, chapter 1 of natural language processing)

For your first working example in the natural language processing course (chapter 1), evaluate the stated result against the example data: Ask for a failure case, a justified metric, a counterexample, a reproducible trace, and an explanation of what beginner orientation did not establish.

20. What practical artifact demonstrates progress on Natural Language Processing — Documents and tokens — chapter 1 (Beginner orientation)?

For your first working example in the natural language processing course (chapter 1), evaluate the stated result against the example data: Submit both runnable scripts, their verified outputs including “Contains learner: True”, a one-page interpretation, one changed-input test, and a documented limitation.

Build one original mini-project

Chapter 1 checkpoint: Explain how documents and tokens affects the farm irrigation schedules example; demonstrate one failing input; then propose a revision with an acceptance test and a recorded output. This checkpoint belongs to the Natural Language Processing course.

Write two sentences about a small robot, count repeated terms and describe which information frequency misses.

A six-step practice plan

Documents and tokens. For the “Documents and tokens” exercise, identify one observable result and record what would demonstrate progress in Natural Language Processing.
Cleaning text with care. For the “Cleaning text with care” exercise, identify one observable result and record what would demonstrate progress in Natural Language Processing.
Word frequencies. For the “Word frequencies” exercise, identify one observable result and record what would demonstrate progress in Natural Language Processing.
Stop words and tradeoffs. For the “Stop words and tradeoffs” exercise, identify one observable result and record what would demonstrate progress in Natural Language Processing.
Stemming versus lemmatization. For the “Stemming versus lemmatization” exercise, identify one observable result and record what would demonstrate progress in Natural Language Processing.
N-gram patterns. For the “N-gram patterns” exercise, identify one observable result and record what would demonstrate progress in Natural Language Processing.

Save your original inputs, your changed inputs, the two actual outputs and a short explanation of what you learned about Natural Language Processing. The Natural Language Processing record forms evidence for the first milestone of your project portfolio.

Where you go next

Your Natural Language Processing roadmap has 18 subject-specific milestones progressing toward evaluation and a capstone. Future Natural Language Processing lessons will require new demonstrations and evidence, beyond the skills taught in this first chapter.

Open course roadmap