Curious Pandas
Explorer0 XP

Small tools, joined by pipes

Each tool does one thing to lines of text. Join them with | and they answer real questions - like which client is hammering your site with failing requests.

Runs in your browserOn your machine
16 min+35 XP for finishing

Every program starts with three streams already open: standard input (where it reads), standard output (where it writes its results) and standard error (where it writes complaints). In a terminal, input is your keyboard and both outputs are your screen.

The pipe, |, rewires that. In a | b, everything a writes to standard output becomes b’s standard input. Neither program knows; each just reads and writes lines. So tools that each do one small job - keep matching lines, cut out a column, sort, count - combine into answers none of them could give alone.

Visual guideWho failed most? One line, five tools
Who failed most? One line, five toolsaccess.log flows into grep, which keeps only lines ending in 500. cut -d ' ' keeps the first space-separated field, the client address. sort puts identical addresses next to each other, uniq -c collapses each run into a count, and sort -rn puts the biggest count first. head -1 keeps the top line, and > saves it to noisy.txt. Error messages go to the terminal, not down the pipe.grep ' 500$'keep failurescut -d ' ' -f 1keep the addresssortline twins upuniq -ccount each runsort -rnbiggest firsthead -1keep the topnoisy.txt> saves ityour terminalstandard error|||||>errorsWho failed most? One line, five toolsaccess.log flows into grep, which keeps only lines ending in 500. cut -d ' ' keeps the first space-separated field, the client address. sort puts identical addresses next to each other, uniq -c collapses each run into a count, and sort -rn puts the biggest count first. head -1 keeps the top line, and > saves it to noisy.txt. Error messages go to the terminal, not down the pipe.grep ' 500$'keep failurescut -d ' ' -f 1keep the addresssortline twins upuniq -ccount each runsort -rnbiggest firsthead -1keep the topnoisy.txt> saves ityour terminalstandard error|||||>errors
  1. grep reads the log and writes out only the lines that end in 500.
  2. Each | hands one tool's output to the next tool's input. cut -d ' ' keeps the first space-separated field: the client's address. Plain cut splits on tabs, and this log has none.
  3. uniq only merges identical lines that sit next to each other - so sort goes first and lines the twins up.
  4. sort -rn ranks the counts biggest first, head -1 keeps the top line, and > saves it to a file instead of the screen.
  5. Error messages travel on standard error, which a pipe does not carry. They still reach your screen, so a failing step cannot hide.

Find the noisy client

Your shop’s access log has one line per request: the client’s address first, the status code last. Something is failing a lot. Find out who.

Twenty-eight requestsa toy Linux in your browser
You are here

    Check yourself+10 XP

    Why does the pipeline sort before uniq -c?

    Redirection: where the output goes

    Pipes connect programs. Redirection connects a program to a file:

    • > file writes output to a file, replacing what was there.
    • >> file appends to the end.
    • < file feeds a file in as input.
    • 2> file sends standard error to a file. 2> /dev/null throws errors away.
    Check yourself+10 XP

    grep pattern missing.txt | wc -l prints 0, plus 'grep: missing.txt: No such file or directory' on screen. Why did the error not get counted?

    Do it for real · your machine+20 XP bonus

    Everything above ran in your browser. Now try it in a real terminal: any Linux machine, a Mac's Terminal app, or Windows with WSL. Nothing to install beyond that.

    Your own shell keeps a history. Ask it which commands you use most:

    history | cut -c 8- | sort | uniq -c | sort -rn | head

    Then the classic “which files are biggest here?”:

    du -a . 2>/dev/null | sort -rn | head -5
    Self-reported. We can't see your machine, and we don't want to. This is a bonus, never required.

    Sources

    Finished reading?

    Missions and quizzes above earn their own XP. This marks the lesson done.