You must process an allowlist file line by line, and some lines have leading spaces or backslashes that have to be preserved exactly. Which loop is correct?

while IFS= read -r line; do ... done < allow.txt. IFS= stops the whitespace trimming and -r keeps backslashes literal, so each line arrives exactly as written.

In plain Bash with no special options set, you run `for f in *.csv; do echo "$f"; done` in a directory that contains no files ending in .csv. What happens?

The body runs once with `f` set to the literal string `*.csv`. Plain Bash leaves an unmatched glob unexpanded, so the loop runs once with the pattern itself, which is why `shopt -s nullglob` exists to make it expand to nothing instead.

A colleague's audit script counts lines with `while IFS= read -r line; do ((n++)); done < export.csv`. It reports one fewer line than the file contains, but only for files from one tool that omits the trailing newline. What is going on, and what fixes it?

`read` returns a failure status when it hits end-of-file on a final chunk with no trailing newline, even though it did store that text in `line`; the loop condition sees the failure and exits before the last line is processed. Add `|| [[ -n "$line" ]]` to the while test. `read` reports end-of-file on an unterminated final line, so `while IFS= read -r line || [[ -n "$line" ]]` is needed to catch that leftover text.

Courses Bash for ops, done safelyLoops: for, while & reading files

Loops: for, while & reading files

Q: In plain Bash with no special options set, you run `for f in *.csv; do echo "$f"; done` in a directory that contains no files ending in .csv. What happens?

The body runs once with `f` set to the literal string `*.csv`. Plain Bash leaves an unmatched glob unexpanded, so the loop runs once with the pattern itself, which is why `shopt -s nullglob` exists to make it expand to nothing instead.

Q: A colleague's audit script counts lines with `while IFS= read -r line; do ((n++)); done < export.csv`. It reports one fewer line than the file contains, but only for files from one tool that omits the trailing newline. What is going on, and what fixes it?

`read` returns a failure status when it hits end-of-file on a final chunk with no trailing newline, even though it did store that text in `line`; the loop condition sees the failure and exits before the last line is processed. Add `|| [[ -n "$line" ]]` to the while test. `read` reports end-of-file on an unterminated final line, so `while IFS= read -r line || [[ -n "$line" ]]` is needed to catch that leftover text.

Repeat work and read input line by line.

Beginner18 min · lesson 6 of 14

Doing the same job on a pile of things

You have forty servers to check, two hundred log files to scan, or a list of user accounts to audit. Typing the same command forty times is slow and easy to fumble. A loop is the fix. It is the assembly-line worker of Bash (the command language you type into a Linux terminal): you describe the steps once, hand it a pile of items, and it runs those steps on each item in turn. This lesson covers the loops you will actually reach for in operations work, plus the one file-reading pattern that quietly breaks security scripts when people get it wrong.

for: one run per item in a list

A for loop is you reading names off a guest list. For each name you do the same thing: check them in. You write the list of items after the word in, and Bash runs the body once with the loop variable set to each item.

~/secopslog — bash

$ for env in dev staging prod; do echo "Deploying to $env" done

Deploying to dev Deploying to staging Deploying to prod

Bash split that list on spaces and ran the body three times. $env held dev, then staging, then prod. The words after in are the whole list, and they can be anything: hostnames, ports, filenames, usernames.

for over a glob: let the shell find the files

Most of the time you do not want to type the filenames by hand. You want to hand the shell a rule instead of a list: "every file whose name ends in .log". That shorthand is a glob (short for global, the old name for filename wildcards). The * matches any run of characters, so *.log means every name ending in .log. The shell expands the glob into the real filenames before the loop starts.

~/secopslog — bash

$ for logfile in *.log; do lines=$(wc -l < "$logfile") echo "$logfile has $lines lines" done

app.log has 3 lines auth.log has 2 lines nginx.log has 1 lines

The shell replaced *.log with the three matching files (in alphabetical order) and the loop ran once per file. Notice the double quotes around "$logfile". A filename can contain spaces, and the quotes keep such a name as one item instead of two. Leave them off and a file called access log.txt would split and break your script. One sharp edge: if no file matches, plain Bash leaves the text *.log unexpanded and the loop runs once with that literal string. Running shopt -s nullglob first makes a no-match glob expand to nothing, so the loop runs zero times, which is usually what you want.

Counting and waiting: C-style for and while

Two loops cover the cases where you are not walking a list. When you want to run something a fixed number of times, use the C-style for, named after the C programming language it copies. It works like a lap counter at a track: it starts at a number, keeps going while a condition holds, and ticks the number up each time around. Inside the double parentheses those three parts sit in order: where the counter starts, how long to keep going, and how the counter changes each pass.

~/secopslog — bash

$ for (( i=0; i<3; i++ )); do echo "attempt $i" done

attempt 0 attempt 1 attempt 2

When you do not know the count ahead of time and instead want to keep going until some condition flips, use while. It works like a bucket under a tap: keep filling while the bucket is not full. Bash checks the condition, runs the body, then checks again, stopping the moment the condition is false.

~/secopslog — bash

$ count=3 while (( count > 0 )); do echo "$count..." (( count-- )) done echo "liftoff"

3... 2... 1... liftoff

(( count-- )) subtracts one from count each pass. Forget that line and the condition never turns false, so you get an infinite loop. Press Ctrl-C to kill one if it happens.

Reading a file line by line, the right way

This is the pattern people get wrong most often, and the wrong version fails in ways that matter for security. Say you have an allowlist of addresses, where the exact content on each line matters:

allowlist.txt

10.0.0.5
  10.0.0.6
C:\logs\path
*

Here is the correct way to read it. Memorize this line as a single unit:

~/secopslog — bash

$ while IFS= read -r line; do echo "[$line]" done < allowlist.txt

[10.0.0.5] [ 10.0.0.6] [C:\logs\path] [*]

Every line arrived exactly as written: the leading spaces on the second address, the backslashes in the path, and the lone *. Read the machinery from right to left. < allowlist.txt feeds the file into the loop as its input. read pulls one line into the variable line and returns success until the file runs out, which is what stops the loop at the end. Two small pieces carry the weight: IFS= and -r.

IFS is the Internal Field Separator, the set of characters Bash treats as gaps between words. Normally read uses it to trim leading and trailing spaces and tabs off each line. Setting IFS= (empty) for this one command turns that trimming off, so 10.0.0.6 keeps its indentation. -r means raw. Without it, read treats a backslash as an escape character and swallows it. The difference is not cosmetic:

~/secopslog — bash

$ # paths.txt contains the literal line: C:\logs\audit while IFS= read line; do echo "[$line]"; done < paths.txt # no -r while IFS= read -r line; do echo "[$line]"; done < paths.txt # with -r

[C:logsaudit] [C:\logs\audit]

Without -r, C:\logs\audit came back as C:logsaudit. The backslashes vanished. On Windows paths, Active Directory names, or regular expression (regex) patterns stored in a rules file, that silent damage changes what your script matches. Keep -r on every read loop.

Now the wrong way, the one you will find in old blog posts and copy-pasted scripts:

~/secopslog — bash

$ for line in $(cat allowlist.txt); do echo "[$line]" done

[10.0.0.5] [10.0.0.6] [C:\logs\path] [allowlist.txt]

for line in $(cat file) is broken and unsafe

Two bugs are visible in that output. First, $(cat file) gets split on every space and newline, not only on newlines, so 10.0.0.6 lost its leading spaces, and any line containing a space would shatter into separate items. Second, the shell runs filename expansion on the result, so the lone * on the last line turned into the filename allowlist.txt. Feed such a loop a file an attacker can write to, drop a * on a line, and you are suddenly looping over your directory contents instead of the data you meant to process. Always read files with while IFS= read -r line; do ... done < file, never with for over cat.

break and continue: skip one, or stop early

Inside any loop, two commands change the flow. continue says "skip the rest of this pass and move to the next item", like passing on a name on your checklist. break says "stop the whole loop now and walk away". A common shape reads a host list, skips comment lines, and halts at a marker:

hosts.txt

web-01
# web-02 is decommissioned
web-03
STOP
web-04

~/secopslog — bash

$ while IFS= read -r host; do [[ "$host" == \#* ]] && continue # skip comment lines [[ "$host" == STOP ]] && break # stop at the marker echo "Scanning $host" done < hosts.txt

Scanning web-01 Scanning web-03

The comment line was skipped by continue, web-03 was scanned, and STOP ended the loop with break before web-04 was ever read. [[ "$host" == \#* ]] is a pattern test: it is true when host starts with a #. The backslash before # keeps the shell from reading it as the start of a comment.

Putting it together: scan a folder of logs

Real work stacks these loops. Here is a script that walks every log file in a folder and, for each one, reads it line by line to count failed logins. The outer loop is a glob over files, and the inner loop is a while read over the lines of each file.

scan.sh

#!/usr/bin/env bash
for logfile in logs/*.log; do
  fails=0
  while IFS= read -r line; do
    [[ "$line" == FAIL* ]] && (( fails++ ))
  done < "$logfile"
  echo "$logfile: $fails failed login(s)"
done

~/secopslog — bash

$ bash scan.sh

logs/auth-web01.log: 2 failed login(s) logs/auth-web02.log: 1 failed login(s)

The fails counter is reset to zero at the top of each file, counts lines starting with FAIL, and gets printed once per file. Because the file is fed in with < "$logfile" rather than piped, the whole loop runs in your current shell, so the counter survives to the echo after it.

A missing final newline can drop your last line

read returns failure when it hits the end of the file without a trailing newline, even though it did place that last chunk into the variable. So a plain while IFS= read -r line loop silently skips the final line of any file that does not end in a newline (truncated logs and some exported data do this). Guard against it: while IFS= read -r line || [[ -n "$line" ]]; do. The extra test is true whenever read pulled in leftover text but reported end-of-file, so that last line still gets processed.

Which loop should you reach for?

Repeating work in Bash

pick by what you are looping over

A list you can name

for x in a b c

envs, ports, users

Files on disk

for f in *.log

glob expands to real names

A fixed count

for (( i=0; i<n; i++ ))

retries, N passes

Until a condition flips

while (( cond ))

wait, drain, count down

Lines of a file

while IFS= read -r line

never for line in $(cat)

Match the loop to the shape of your input.

Try this

Work through “Putting it together: scan a folder of logs” yourself on a sandbox you can throw away, following the commands above in order. Then break one step deliberately and re-run, so you have seen the failure before it finds you.

Takeaway

The trap worth remembering here: for line in $(cat file) is broken and unsafe. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Conditionals: if & caseFunctions & arguments

Project layout and venvsPython for security automation Bash internals: subshells, FDs & process substitutionAdvanced scripting for DevSecOps Bash scripting cheat sheetCheat sheet GitOps interview questionsInterview prep DevSecOps & supply-chain security interview questionsInterview prep

Loops: for, while & reading files

Doing the same job on a pile of things

for: one run per item in a list

for over a glob: let the shell find the files

Counting and waiting: C-style for and while

Reading a file line by line, the right way

break and continue: skip one, or stop early

Putting it together: scan a folder of logs

Try this

Takeaway

Related