Filesystem Operations (os & shutil)

• DevOps scripts often need to create, delete, copy, and move files and directories as part of automation workflows.
• The os module provides low-level filesystem functions, while shutil offers higher-level operations like copying and recursive removal.
• These tools work hand-in-hand with pathlib (for path manipulation) to build robust file management scripts.

Listing Directory Contents

• Use os.listdir(path) to get a list of entry names (files and subdirectories) in a directory.
• Use Path(path).iterdir() to iterate over Path objects, which you can query further with methods like .is_file() or .is_dir().
• os.listdir returns a plain list of strings; iterdir() yields full Path objects, making downstream operations more convenient.

import os
from pathlib import Path
import shutil

"""
Directory structure:

temp_listing_dir/
├── file1.txt
├── file2.log
└── subdir/
    └── subfile.py
"""

tmp_path = Path("temp_listing_dir")
tmp_path.mkdir(exist_ok=True)
(tmp_path / "file1.txt").touch()
(tmp_path / "file2.log").touch()
(tmp_path / "subdir").mkdir(exist_ok=True)
(tmp_path / "subdir" / "subfile.py").touch()

print(f"--- os.listdir(\"{tmp_path}\") ---")
for name in os.listdir(tmp_path):
    print(name)

print(f"--- Path(\"{tmp_path}\").iterdir() ---")
for entry in tmp_path.iterdir():
    print(entry)

shutil.rmtree(tmp_path)

Creating Directories

• os.mkdir(path) creates a single directory and fails if parents don’t exist or if it already exists.
• os.makedirs(path, exist_ok=False) creates all intermediate directories; set exist_ok=True to ignore existing leaf.
• Path(path).mkdir(parents=True, exist_ok=True) is the pathlib equivalent for recursive, idempotent creation.

from pathlib import Path
import shutil

single = Path("my_single_dir")

try:
    single.mkdir(exist_ok=True)
    print(f"Created {single}: {single.exists()}")
finally:
    if single.exists():
        single.rmdir()

nested = Path("parent/child/grandchild")
nested.mkdir(parents=True, exist_ok=True)
print(f"Created nested path {nested}: {nested.exists()}")

shutil.rmtree("parent")

Removing Files and Directories

• os.remove(path) or Path(path).unlink() deletes a single file and raises if missing (unless missing_ok=True).
• os.rmdir(path) or Path(path).rmdir() removes an empty directory only.
• shutil.rmtree(path) recursively deletes a directory tree and all contents; use with extreme caution.

from pathlib import Path
import shutil

"""
Directory structure:

.
├── temp_file.txt
├── empty_dir/
└── tree_root/
    └── child/
        └── inner.txt
"""

temp_file = Path("temp_file.txt")
temp_file.touch()

empty_dir = Path("empty_dir")
empty_dir.mkdir(exist_ok=True)

tree = Path("tree_root/child")
tree.mkdir(parents=True, exist_ok=True)
(tree / "inner.txt").touch()

temp_file.unlink()
print(f"Removed file {temp_file}. Exists? {temp_file.exists()}")
empty_dir.rmdir()
print(f"Removed dir {empty_dir}. Exists? {empty_dir.exists()}")

shutil.rmtree("tree_root")
print(f"Removed \"tree_root\" recursively. Exists? {tree.exists()}")

Copying Files and Directories

• shutil.copy(src, dst) copies a file but does not preserve metadata like timestamps or permissions.
• shutil.copy2(src, dst) copies files and attempts to preserve metadata.
• shutil.copytree(src_dir, dst_dir, dirs_exist_ok=False) recursively copies an entire directory tree.

import shutil
from pathlib import Path

"""
Directory structure:

src_copy/
├── a.txt
└── sub/
    └── b.txt
"""

src = Path("src_copy")
src.mkdir(exist_ok=True)
(src / "a.txt").write_text("A")
(src / "sub").mkdir(exist_ok=True)
(src / "sub" / "b.txt").write_text("B")

dest_file = Path("copied_a.txt")
dest_file_metadata = Path("copied_a_metadata.txt")
shutil.copy(src / "a.txt", dest_file)
shutil.copy2(src / "a.txt", dest_file_metadata)

dest_dir = Path("copied_src")

if dest_dir.exists():
    shutil.rmtree(dest_dir)

shutil.copytree(src, dest_dir)

shutil.rmtree("src_copy")
shutil.rmtree(dest_dir)
dest_file.unlink()
dest_file_metadata.unlink()

Moving Files and Directories

• Use shutil.move(src, dst) to move or rename files and directories in one step.
• If dst is an existing directory, src is moved into it; if dst names a file, src is renamed there.
• Moving across filesystems may involve a copy-and-delete under the hood.

import shutil
from pathlib import Path

"""
Directory structure:

.
├── move_me.txt
├── move_dir/
│   └── inside.txt
└── dest_folder/
"""

file_src = Path("move_me.txt")
file_src.write_text("Moving file.")

dir_src = Path("move_dir")
dir_src.mkdir(exist_ok=True)
(dir_src / "inside.txt").write_text("Inside source dir.")

dest_dir = Path("dest_folder")
dest_dir.mkdir(exist_ok=True)

try:
    shutil.move(file_src, dest_dir)
except Exception as e:
    print(f"Error occurred: {e}")

file_src2 = dest_dir / file_src.name
new_name = Path("renamed.txt")
shutil.move(file_src2, new_name)

try:
    shutil.move(dir_src, dest_dir)
except Exception as e:
    print(f"Error occurred: {e}")

shutil.rmtree(dest_dir)
if new_name.exists():
    new_name.unlink()

Common Pitfalls & How to Avoid Them

• PermissionError: Operations fail if the script lacks rights. Ensure correct ownership or run with appropriate privileges.
• Non-empty Directories: os.rmdir() and Path.rmdir() only remove empty dirs. Use shutil.rmtree() for recursive deletion, but do so carefully.
• Existing Destinations: shutil.copytree() errors if the target exists unless dirs_exist_ok=True. Consider pre-cleanup or that flag.
• Irreversible Deletions: There is no undo for os.remove, os.rmdir, or shutil.rmtree(). Add confirmation or dry-run modes when deleting!

+++