Create K8s cluster in Azure (AKS) with 3-node pool

1. Problem Statement

Organizations running containerized workloads require scalable, reliable, and secure Kubernetes clusters to deploy and manage applications. Manually provisioning and managing Azure Kubernetes Service (AKS) clusters can be complex, time-consuming, and error-prone.
This use c…

Read more

Read more

Provision same VM spec (Ubuntu, 2 vCPU, 4GB RAM, SSD) in AWS, Azure, GCP

1. Problem Statement

In a multi-cloud environment, managing and provisioning identical virtual machines (VMs) across different cloud providers (AWS, Azure, GCP) manually is time-consuming, error-prone, and inconsistent. Each provider has its own interface, configuration se…

Read more

SheLeads Digital

Work-from-Home Empowerment Program for Women

Unlock Your Digital Superpowers. Work, Learn & Earn — From Home.

🎯 Who Is This Program For?

👩‍💼 Women Restarting Their Careers
Took a career break? We’ll help you rebuild confidence, upgrade your skills, and rejoin the digital workforce — on your terms.

🏡 Homemakers Ready to Earn Digitally
You already manage so much. Now, learn how to turn your time, creativity, and passion into real digital income opportunities — from your own home.

🎓 Students & Moms Seeking Flexibility
Study, nurture, and grow — all while working remotely. Find flexible freelance and part-time digital roles that fit your schedule.

💫 Aspiring Personal Brands, Freelancers & Coaches
If you dream of becoming a content creator, freelancer, or online entrepreneur — this program gives you the roadmap to build your brand, grow visibility, and earn from your expertise.

⚙️ Pre-Requisites

✅ Dual-Monitor Setup – for higher productivity
✅ Fast & Reliable Internet Connection
✅ Dedicated Computer or Laptop
✅ Single Gmail ID – use one consistent Gmail ID across all platforms for unified branding

🌐 Create Your Digital Identity

💡 Pro Tip:
Use one consistent handle across all platforms to strengthen your brand.
Example:
📧 CloudByVenkat@gmail.com
🌐 github.com/CloudByVenkat | 💼 linkedin.com/in/CloudByVenkat |

📝 medium.com/@CloudByVenkat | 💻 dev.to/CloudByVenkat |

🧠 hashnode.com/@CloudByVenkat | 🐦 x.com/CloudByVenkat |

🎥 youtube.com/@CloudByVenkat

🎨 Design & Tech Essentials

🧩 Google Workspace Essentials

📄 Google Docs – Create, format & collaborate in real-time
🔗 Sharing Docs – Manage permissions, comments & link access
📊 Google Sheets – Track data, budgets & client records
📝 Google Forms + Sheets Integration – Create forms & automate data collection into linked spreadsheets

✨ Canva Poster Creation

🎨 Learn to design eye-catching visuals for social media, blogs, and clients
📺 YouTube Thumbnails – Design posters in size 1280px x 720px
📰 Dev.to, Medium, Hashnode, Blogger Posters – Create optimized visuals 1600px x 840px
📜 Letterhead Posters – Build personalized templates for your freelance or business use

🎬 Professional Design & Presentation Skills

📊 PPT Mastery – Create visually powerful, story-based presentations
🎤 Presentation Skills – Speak with confidence and impact
📄 Professional Documentation – Learn to draft client-ready documents with structure and polish
🎞️ Basic Video Editing – Edit short-form videos for Instagram, YouTube, and clients

🎙️ Podcasts & Self Videos

🎧 Podcast Creation

• Interview experts, mentors, or industry leaders

• Discuss trending topics: Cloud ☁️ | DevOps ⚙️ | AI 🤖 | Leadership 🌍

• Publish on your YouTube Channel

• Repurpose podcasts into blogs for Medium, Dev.to, and LinkedIn

📹 Self Video Creation

• Record learnings, insights, or project highlights

• Upload to YouTube → Repurpose as blogs

• End every blog with your branding footer (social links & handles)

💼 LinkedIn Mastery

🪶 Profile Optimization

• Craft a professional headline & keyword-rich summary

• Highlight achievements with numbers & impact

• Use the Featured Section to display blogs, presentations, and media

• Add volunteer work, certifications, and skills strategically

📢 Content & Engagement

• Share industry insights, videos, polls, or carousels 3–5 times/week

• Use storytelling to connect with your audience

• Comment meaningfully on others’ posts for visibility

• Celebrate milestones, projects, and achievements

🤝 Networking & Relationship Building

• Send personalized connection requests

• Join active industry-specific groups

• Attend LinkedIn events, lives, and webinars

• Tag mentors & peers authentically for organic visibility

🌟 Thought Leadership

• Write LinkedIn Articles & launch a Newsletter

• Share case studies or lessons learned

• Repurpose posts with your unique perspective

• Post Q&A discussions to boost interaction

📈 Growth & Analytics

• Track profile views, post engagement, and follower growth

• Analyze competitors’ content strategies

• Experiment with hashtags, content types, and post timings

✍️ Blogging & Writing Essentials

📝 Learn to:

• Create accounts on Medium, Dev.to, Hashnode & Blogger

• Write, format & publish SEO-friendly blogs

• Repurpose YouTube or podcast content into written posts

📄 Resume & Cover Letter Writing – Tailored for remote & freelance roles
📰 LinkedIn Articles & Newsletters – Grow credibility through writing
💫 Self-Branding Techniques – Position yourself as a thought leader
🌍 Connect with Industry Leaders – Build genuine professional relationships

💼 Business, Clients & Freelancing

🏢 Understand Your Partner’s Offerings
Learn to confidently explain the course or service offerings to potential clients or learners.

💬 Ready-Made Client Scripts & Training
Use proven templates to respond to inquiries, explain benefits, and close deals effectively.

💡 Lead Generation Mastery

• Learn different question types leads may ask

• Practice answering confidently & empathetically

• Identify real leads vs. time-wasters

💬 Conversion Psychology

• How to convince leads with value-focused communication

• How to overcome objections & guide them to a “yes”

👩‍🏫 Onboarding Students with Partner Approval

• Understand step-by-step onboarding

• Maintain smooth communication & documentation flow

🚀 Create & Convert Leads

• Build your client funnel & master digital selling

• Track, follow-up, and close with clarity

🧠 Respond Professionally

• Craft polite, assertive, and empathetic messages

💻 Freelancing Setup

• Choose the right platforms (Fiverr, Upwork, LinkedIn)

• Build your portfolio & write professional proposals

💬 Sell Use Cases Like a Pro

• Present results, not just features — show real value

🏆 Program Highlights

✅ 100% Online & Beginner Friendly
✅ Real-World Projects & Portfolio Building
✅ Flexible Timings for Women
✅ Weekly Live Classes + Q&A
✅ Peer Community + Personal Mentorship

🔄 Program Flow

Phase 1: Learn & Build Portfolio

Hands-on learning through guided projects and assignments

Phase 2: Evaluation

Submit practical work: designs, documents, videos, and responses

Phase 3: Work & Earn

Qualified participants get paid opportunities & project tasks 💰

🎁 Bonuses Included

🎨 Canva Pro Templates
📄 Resume & Cover Letter Templates
🖼️ LinkedIn Banner + Branding Kit
🎓 Certificate of Completion
👩‍💬 Access to Private Women’s Community

💬 Tagline Options

✨ “Build Your Brand. Work from Anywhere.”
✨ “Digital Skills for Today’s Women.”
✨ “From Kitchen to Keyboard — Rewrite Your Story.”
✨ “Learn. Prove. Work. Earn.”

Red vs Blue Team Scenarios & Simulated Attacks

From Linux Administration to Cyber Defense Mastery

Read more

Course Target Audience:

Experienced Linux administrators who want a hands-on transition into cybersecurity (ethical hacking, web/app security, network security, hardening, incident response). Students will create their own lab environments (Azure VMs and/or local VMs), run safe offensive exercises against intentionally vulnerable systems, and implement …

Read more

🌟 Course Overview

This course is designed to help learners master Generative AI from the ground up, combining theory, real-world applications, and end-to-end projects.

You’ll start by understanding AI, Machine Learning, and Deep Learning fundamentals, then move into LLMs, LangChain, Hugging Face, Vector Databases, and AI-powered app deployment.

By the end…

Read more

📘 Course Overview

This 18-week Linux System Administration Crash Course is designed to transform beginners into confident system administrators capable of managing, automating, securing, and troubleshooting real-world Linux servers.

Students will gain hands-on experience with real Linux environments, mastering everything from basic shell commands to kernel tuning, system hardening, and automation scripting.
By the end of this course, students will be ready to perform system admin tasks in enterprise environments and prepare for Linux certification paths (RHCSA, LFCS, CompTIA Linux+).

🗓️ Week-by-Week Syllabus

Week 1 – Introduction to Linux

• History, architecture, and distributions (RHEL, Ubuntu, CentOS)

• Understanding the Linux kernel, shell, and file system

• The Linux boot process and system targets

• Installing and exploring Linux environments
  🧠 Outcome: Understand how Linux works internally and interact with the shell confidently.

Week 2 – Linux File System & Directory Structure

• Linux file hierarchy and directory roles

• Navigating paths (absolute & relative)

• Mounting/unmounting partitions

• /etc/fstab and persistent mounts

• File operations and management
  🧠 Outcome: Manage files, directories, and storage mounts effectively.

Week 3 – File Permissions and Ownership

• Understanding Linux permissions (rwx)

• Using chmod, chown, and chgrp

• Special permissions (SUID, SGID, Sticky Bit)

• Default permissions (umask)
  🧠 Outcome: Secure files using correct permission models.

Week 4 – Linux Command Line Essentials

• Viewing and filtering content (cat, grep, awk, sed)

• Piping, redirection, and command chaining

• Searching and text manipulation
  🧠 Outcome: Gain command-line mastery for daily admin tasks.

Week 5 – User and Group Administration

• Creating and managing users & groups

• Understanding /etc/passwd, /etc/shadow

• Configuring sudo permissions

• Managing user profiles and shells
  🧠 Outcome: Handle user access and privilege control securely.

Week 6 – Process and Job Management

• Understanding processes and PIDs

• Background/foreground jobs

• Managing processes with ps, top, kill

• Scheduling tasks using cron and at
  🧠 Outcome: Monitor and control system processes efficiently.

Week 7 – Software and Package Management

• Package managers: apt, yum, dnf, rpm

• Installing, updating, and removing software

• Managing repositories

• Compiling software from source
  🧠 Outcome: Install and maintain Linux software reliably.

Week 8 – Disk, Storage & File System Management

• Disk partitioning (lsblk, fdisk)

• Creating and mounting filesystems (mkfs, mount)

• LVM concepts (PV, VG, LV)

• Disk quotas and RAID fundamentals
  🧠 Outcome: Administer Linux storage safely and efficiently.

Week 9 – System Startup and Services

• Understanding systemd and init systems

• Starting/stopping/enabling services

• Boot targets and runlevels

• Troubleshooting failed services
  🧠 Outcome: Manage service lifecycles and system boots confidently.

Week 10 – Networking in Linux

• Configuring IPs and DNS (ip, nmcli)

• Testing connectivity (ping, ss, traceroute)

• SSH configuration and key management

• Basic web & file transfers (HTTP, FTP, SCP)
  🧠 Outcome: Configure and troubleshoot Linux networking.

Week 11 – Logging and System Monitoring

• Linux log architecture and log files (/var/log)

• Viewing logs (journalctl, dmesg)

• Log rotation (logrotate)

• Monitoring tools (top, vmstat, iostat)
  🧠 Outcome: Monitor performance and analyze system events effectively.

Week 12 – Backup and Recovery

• Backup strategies (full, incremental)

• Using tar, rsync, cron for automation

• Data restoration procedures

• System rescue & recovery modes
  🧠 Outcome: Design and execute reliable backup systems.

Week 13 – Linux Security and Hardening

• SSH hardening and key-based access

• Sudo and least privilege principles

• Firewalls (ufw, firewalld, iptables)

• SELinux/AppArmor basics

• Auditing system events (auditd)
  🧠 Outcome: Secure and harden Linux systems using best practices.

Week 14 – Performance Tuning and Troubleshooting

• Analyzing CPU, memory, and I/O bottlenecks

• Using vmstat, iostat, and free

• Tuning kernel parameters (sysctl)

• Swap and GRUB recovery
  🧠 Outcome: Diagnose and optimize system performance.

Week 15 – Bash Scripting & Automation

• Writing shell scripts and using shebang

• Variables, loops, conditionals, and functions

• Debugging scripts (set -x)

• Automating admin tasks (backups, cleanup)
  🧠 Outcome: Automate repetitive operations with Bash scripts.

Week 16 – Git for Linux Administrators

• Introduction to Git & version control

• Managing config files in Git

• Branching, merging, and restoring files

• .gitignore and automated Git backups
  🧠 Outcome: Use Git to safely version and manage system configurations.

Week 17 – Advanced Linux Administration

• Kernel management and upgrades

• GRUB customization (boot control)

• System limits (ulimit, /etc/security/limits.conf)

• Time sync (Chrony/NTP)

• System tuning with sysctl
  🧠 Outcome: Handle kernel-level and system tuning operations confidently.

Week 18 – Real-Time Administration Use Cases

• Server patching automation

• Log cleanup and rotation scripts

• User lifecycle and access management

• Backup validation

• Root Cause Analysis (RCA)
  🧠 Outcome: Apply all skills in real-world administrative workflows.

🧩 Learning Outcomes – What Students Will Learn

After completing this 18-week course, students will be able to:

✅ System Operation & Management

• Understand Linux architecture and command-line fundamentals

• Manage file systems, storage, users, and services

✅ Automation & Optimization

• Write Bash scripts for automation

• Configure cron jobs for scheduled tasks

• Tune performance parameters using sysctl and ulimit

✅ Security & Networking

• Harden systems with SSH keys, firewalls, and SELinux

• Secure user privileges and monitor access logs

• Configure and troubleshoot network connectivity

✅ Monitoring & Recovery

• Analyze system logs and troubleshoot issues

• Perform backup, restore, and recovery operations

• Identify root causes of downtime or performance degradation

✅ Version Control & Advanced Administration

• Track configuration changes using Git

• Manage kernel modules, GRUB, and system boot targets

• Automate patching and routine maintenance

🎯 By the End of This Course, Students Will:

• Be confident to administer Linux servers independently.

• Be able to manage production systems safely.

• Automate daily sysadmin tasks using Bash scripting.

• Troubleshoot, optimize, and secure enterprise-grade systems.

• Be ready for RHCSA, LFCS, or Linux+ certifications.

🧰 Tools and Technologies Covered

• Distributions: Ubuntu, CentOS, RHEL

• Core Utilities: bash, grep, awk, sed, vim

• Service Managers: systemd, cron, logrotate

• Package Managers: apt, yum, dnf, rpm

• Network Tools: ip, ss, nmcli, scp, ping

• Security Tools: ufw, firewalld, iptables, auditd

• Performance Tools: top, vmstat, iostat, free, df

• Version Control: git, .gitignore, Git branches

• Automation: bash, cron, rsync, tar, sysctl

🧾 Certification of Completion

At the end of the 18-week program, participants will receive a
“CareerByteCode Linux System Administrator” certificate
signifying mastery of foundational and real-world Linux administration skills.

Chapter 1 — Introduction to Linux (Week 1)

Objective (simple): Understand what Linux is, why people use different distributions, what the kernel and shell are, how the system boots at a high level, and how to peek around your installed Linux distribution.

I’ll start with Chapter 1 (Week 1 — Introduction to Linux) exactly as you asked:

How to run these examples

• Open a terminal on any Linux machine (Ubuntu/RHEL/CentOS VM is perfect).

• I will use /tmp/linux_course/week1 as a safe working folder so nothing system-critical is changed.

• Copy/paste each code block into the terminal. Commands are safe and non-destructive. Some commands (like dmesg) might show different output on different systems — that’s fine.

1. What is Linux? — Layman explanation

Imagine a car. The kernel is the engine it talks to the metal (hardware) and makes things move. The rest of the car (steering wheel, seats, radio) is the userland programs and tools you interact with. Linux is the engine (kernel) plus a huge group of people who build and share cars (distributions) with different seats and radios (apps, packaging, defaults).

A distribution (distro) is like a specific car model Ubuntu, RHEL, CentOS, Fedora, Debian all share the Linux engine but package different apps and support levels.

2. Linux architecture — simple mapping

• Kernel: engine — manages CPU, memory, devices.

• User space: everything outside kernel — shells, programs, GUI.

• Shell: the steering wheel the driver uses to control the car — bash, zsh.

• Init/system manager: car ignition system that starts everything — systemd or older init.

• File system: how you store things in the trunk and glovebox — /etc, /usr, /var, /home.

3. Boot process overview — plain steps

1. Power on → UEFI/BIOS: the basic electronics turn on and look for boot device.

2. Bootloader (GRUB): like the car key that chooses which engine/configuration to load.

3. Kernel loads: kernel (engine) runs and mounts a temporary root.

4. Init/system manager (systemd): starts services (networking, display, etc.) and user sessions.

5. Login prompt / graphical login: you sit in the driver seat and start using the system.

4. Shell vs terminal

• Terminal: the vehicle’s dashboard screen (a program that shows text).

• Shell: the command interpreter that reads what you type and runs programs. Example shells: bash, zsh, sh.

5. Checking distro and system facts (simple)

Linux distributions keep an /etc/os-release file describing themselves. You can read that along with uname (kernel info).

Quick reference — commands table (copyable explanation)

Make a working folder (run this first)

# create working folder (safe)
mkdir -p /tmp/linux_course/week1
cd /tmp/linux_course/week1
pwd   # should show /tmp/linux_course/week1

Example 1 — Check kernel and distro info (copy-paste runnable)

# Example 1: Check kernel & distro info
echo “=== uname -a ===”
uname -a

echo
echo “=== /etc/os-release (if present) ===”
if [ -f /etc/os-release ]; then
  cat /etc/os-release
else
  echo “/etc/os-release not found on this system”
fi

echo
echo “=== sample output stored to file ===”
uname -a > kernel_info.txt
cat kernel_info.txt

What this does: shows the kernel version & distro. The file kernel_info.txt will contain the kernel line.

Example 2 — Explore basic file system layout (create sample files & table)

We will create a tiny mock of the FHS in /tmp/linux_course/week1/demo_fhs to play with.

# Example 2: create small demo FHS structure (safe place)
BASE=/tmp/linux_course/week1/demo_fhs
rm -rf “$BASE”
mkdir -p “$BASE”/{etc,usr,bin,var,home,tmp}
echo “root config file” > “$BASE/etc/passwd.example”
echo “system binary”  > “$BASE/bin/hello.sh”
chmod +x “$BASE/bin/hello.sh”
cat > “$BASE/bin/hello.sh” <<’EOF’
#!/bin/sh
echo “Hello from demo_fhs /bin/hello.sh”
EOF

# Show tree-like listing (without tree command)
echo “Contents of demo_fhs:”
ls -lR “$BASE” | sed -n ‘1,200p’

# Run the demo binary
“$BASE/bin/hello.sh”

What this does: sets up an example filesystem tree you can inspect. This simulates /etc, /bin, etc., without touching system directories.

Example 3 — Shell basics: variables, simple script, environment

# Example 3: shell basics
cat > /tmp/linux_course/week1/var_demo.sh <<’SH’
#!/bin/bash
# A tiny script to show variables and environment use

MYNAME=”Student”
echo “Hello, $MYNAME. Today is $(date +%A).”

# export a variable
export COURSE=”Linux Basics”
echo “COURSE is set to: $COURSE”

# read input if available
if [ -t 0 ]; then
  echo “No piped input. (Run: echo ‘some text’ | ./var_demo.sh )”
else
  echo “Piped input was:”
  cat -
fi
SH

chmod +x /tmp/linux_course/week1/var_demo.sh
# Run it normally
/tmp/linux_course/week1/var_demo.sh

# Run with piped input to test reading from STDIN:
echo “this is piped text” | /tmp/linux_course/week1/var_demo.sh

What this does: shows variables, exporting, date expansion, and reading piped input.

Example 4 — Look at boot parameters & simple explanation

# Example 4: Boot parameters and dmesg sample (non-root safe)
echo “=== /proc/cmdline (kernel boot parameters) ===”
if [ -f /proc/cmdline ]; then
  cat /proc/cmdline
else
  echo “/proc/cmdline not available”
fi

echo
echo “=== first 20 kernel messages (dmesg may require root to see everything) ===”
dmesg | head -n 20 || true

Note: dmesg sometimes requires root to see full kernel message buffer; normal users will still see many messages. This demonstrates how kernel communicates during boot.

Example 5 — Identify your shell and try simple shell commands

# Example 5: shell identity and common commands
echo “Your current shell is: $SHELL”
echo “Which shells exist on this system (show /etc/shells):”
if [ -f /etc/shells ]; then
  cat /etc/shells
else
  echo “/etc/shells not present”
fi

# Try simple commands: list root & home (non-destructive)
echo
echo “--- ls / ---”
ls -l / | sed -n ‘1,50p’

echo
echo “--- ls ~ (your home) ---”
ls -la ~ | sed -n ‘1,50p’

Example 6 — Check which package manager exists (shows distro family)

# Example 6: check package manager (non-destructive)
echo “Checking package manager presence...”
for cmd in apt dnf yum rpm dpkg zypper pacman; do
  which $cmd >/dev/null 2>&1 && echo “Found: $cmd” || true
done

# Also show /etc/os-release again to correlate
echo
echo “If apt found → Debian/Ubuntu family; if yum/dnf/rpm found → RHEL/CentOS family”
cat /etc/os-release 2>/dev/null || true

Example 7 — Create a small “cheat sheet” CSV table to practice reading files

# Example 7: create a CSV with basic command cheats and show it
cat > /tmp/linux_course/week1/cheats.csv <<’CSV’
Command,Description,Example
uname -a,Show kernel info,uname -a
cat /etc/os-release,Show distro info,cat /etc/os-release
ls -la,List files with details,ls -la /tmp
echo $SHELL,Show current shell,echo $SHELL
dmesg | head,Show kernel messages,dmesg | head -n 5
CSV

echo “Cheat sheet created at /tmp/linux_course/week1/cheats.csv”
column -s, -t /tmp/linux_course/week1/cheats.csv || cat /tmp/linux_course/week1/cheats.csv

column beautifies the CSV into a table; if column isn’t present, the CSV is still readable via cat.

Minimal Troubleshooting tips (layman)

• If a command says “permission denied”, it usually means you tried to touch a system file — use your personal folder like /tmp or your home folder.

• If a command is “command not found”, your distro may not have that tool installed yet — e.g., tree or column may be missing. Use apt install tree on Ubuntu or yum install tree on RHEL if you want them (requires root).

• If dmesg shows nothing as non-root, try sudo dmesg (some distros restrict kernel log access).

10 Hands-On Exercises (step by step — one at a time)

Do these in /tmp/linux_course/week1 (I’ve already created it).

1. Create the course folder — run:

mkdir -p /tmp/linux_course/week1/mywork
cd /tmp/linux_course/week1/mywork
pwd

1. Goal: ensure you’re inside the course folder.

2. Read distro file — show the distro name:

cat /etc/os-release

1. Write down the NAME and VERSION fields.

2. Create a personal README — create a file README.md with your name, distro, and date:

cat > README.md <<’MD’
# Week1 - My Intro
Name: <Your Name>
Distro:
Date:
MD

# then open it to edit using nano/vim or echo commands to fill fields.
sed -n ‘1,200p’ README.md

1. Show kernel info & save:

uname -a > my_kernel.txt
cat my_kernel.txt

1. Make a small script — create hello_name.sh that prompts for your name and says hi:

cat > hello_name.sh <<’SH’
#!/bin/bash
read -p “What’s your name? “ n
echo “Hello, $n — welcome to Linux Week 1!”
SH
chmod +x hello_name.sh
./hello_name.sh

1. Simulate FHS — create directories /tmp/linux_course/week1/demo2/{etc,bin,home} and add sample files; list them.

2. Inspect boot cmdline — run:

cat /proc/cmdline

1. Note what parameters are present.

2. Check available shells — run:

cat /etc/shells
echo “Your shell is: $SHELL”

1. Create a cheat CSV and show as table:

cp /tmp/linux_course/week1/cheats.csv ./my_cheats.csv
column -s, -t my_cheats.csv || cat my_cheats.csv

1. Document everything — zip your work:

cd /tmp/linux_course/week1
tar czvf week1_submission.tar.gz mywork demo_fhs cheats.csv kernel_info.txt README.md
ls -l week1_submission.tar.gz

Each exercise is deliberately non-destructive and runs in /tmp.

10 Assessment Questions (mix of theory & short practical)

Answer these in short sentences or run the commands where applicable.

1. What is the Linux kernel in one or two sentences?

2. Name three popular Linux distributions and one distinguishing feature of each.

3. What is the difference between a shell and a terminal?

4. Explain the high-level boot steps from power-on to login.

5. What file usually contains human-readable distro information (name & version)? Show the command to read it.

6. What command shows you the kernel version and architecture?

7. How would you check which shell you are using right now (give the command)?

8. What is /proc/cmdline and why is it useful?

9. Create a tiny shell script that prints “I love Linux” and explain how to make it executable and run it. (Practical)

10. Why is it safer to do practice in /tmp or in a VM when learning Linux?

Chapter 2 – Linux File System & Directory Structure

Week 2 Objective:
Understand how the Linux filesystem is organized, what each directory means, how to move around, what “mounting” is, and how the system remembers attached drives using /etc/fstab.

🧠 1. Layman’s Explanation — “The Library Analogy”

Think of a Linux system like a huge library:

• The root (/) is the main entrance of the library.

• Every section (like /home, /etc, /usr) is a different room holding different types of materials.

• Each book is a file, and each bookshelf is a directory (folder).

• A path (like /home/student/documents/note.txt) is like saying
  “Go to the library → Home Room → Student shelf → Documents → note.txt book.”

🧩 2. The Linux Filesystem Hierarchy — Common Directories

🧭 3. Absolute vs Relative Paths (simple analogy)

• Absolute path = full address from the root.
  👉 Example: /home/student/project/script.sh

• Relative path = shortcut from where you currently are.
  👉 If you’re already in /home/student, then project/script.sh works fine.

Try this with pwd (print working directory) to know where you are.

🧱 4. Mounting (what it means)

In Windows, each drive gets a letter (C:, D:).
In Linux, everything joins one big tree. When you “mount” a drive, it’s like saying:

“Take this new storage device and hang it inside the big tree at this folder path.”

When you “unmount”, you remove it from that tree — but data stays safe on the drive.

🗂️ 5. /etc/fstab (what it does)

/etc/fstab is a memory book for Linux — it lists which drives or partitions should be mounted automatically every time the system starts.

Example line:

UUID=1234-ABCD  /data  ext4  defaults  0  2

Meaning: “Mount the drive with UUID 1234-ABCD to /data using the ext4 filesystem with default options.”

🧰 Key Commands Table (you can run them safely)

⚙️ Example 1 — Create a Demo File System Tree

# Create a safe workspace
mkdir -p /tmp/linux_course/week2/demo_fs/{etc,usr,bin,home,log,tmp}
echo “User data file” > /tmp/linux_course/week2/demo_fs/home/user.txt
echo “Configuration sample” > /tmp/linux_course/week2/demo_fs/etc/config.conf
echo “Binary placeholder” > /tmp/linux_course/week2/demo_fs/bin/runapp
chmod +x /tmp/linux_course/week2/demo_fs/bin/runapp

# List the created structure
echo “=== Demo File System ===”
ls -lR /tmp/linux_course/week2/demo_fs

Explanation:
This mock tree simulates a Linux system but is safe to explore under /tmp.

⚙️ Example 2 — Practice Navigating Paths

cd /tmp/linux_course/week2/demo_fs
pwd
ls

# Absolute path
cat /tmp/linux_course/week2/demo_fs/etc/config.conf

# Relative path
cd home
cat user.txt
cd ..
pwd

Concept:
Absolute uses / from the start; relative starts from where you are.

⚙️ Example 3 — Symbolic Links (shortcuts)

cd /tmp/linux_course/week2/demo_fs
ln -s home/user.txt user_link
ls -l user_link
cat user_link

Explanation:
ln -s creates a symbolic link — like a shortcut or alias to another file.

⚙️ Example 4 — Create and Mount a Dummy File System (Loopback Mount)

(no USB needed — safe virtual test)

# Create a 20MB dummy file
mkdir -p /tmp/linux_course/week2
cd /tmp/linux_course/week2
dd if=/dev/zero of=mydisk.img bs=1M count=20

# Create an ext4 filesystem inside it
mkfs.ext4 mydisk.img

# Create a mount point and mount it
mkdir -p /tmp/linux_course/week2/mnt_disk
sudo mount -o loop mydisk.img /tmp/linux_course/week2/mnt_disk

# Verify mount
df -h /tmp/linux_course/week2/mnt_disk

# Write a file inside the mounted disk
echo “Data inside mounted disk” | sudo tee /tmp/linux_course/week2/mnt_disk/test.txt

# Unmount
sudo umount /tmp/linux_course/week2/mnt_disk

Explanation:
You created a fake disk file, formatted it, mounted it, wrote data, and unmounted it — all in /tmp.

⚙️ Example 5 — View and Understand /etc/fstab (Read-Only)

# Display only non-comment lines
grep -v ‘^#’ /etc/fstab | column -t

Explanation:
Shows which drives are configured to mount at boot. (Don’t edit this file as a beginner!)

🧩 Mini Data Table for Understanding df -h Output

This is what df -h typically prints — showing drives and their mount points.

🧠 10 Hands-On Exercises (Safe to Try)

1. Create a folder /tmp/linux_course/week2/work and move into it.

2. Create 3 directories: config, data, and scripts.

3. Inside data, create 2 files: file1.txt, file2.txt with any text.

4. Copy both files to the config folder using one cp command.

5. Move file2.txt from config to scripts using mv.

6. Create a symbolic link named latest.txt in work/ pointing to data/file1.txt.

7. Run ls -l to confirm link points correctly.

8. Run df -h and note which filesystem your /tmp belongs to.

9. View the contents of /etc/fstab and identify your / line.

10. Write a small script named showpath.sh that prints:

#!/bin/bash
echo “You are in $(pwd)”
echo “Listing files:”
ls -l

1. Then chmod +x showpath.sh and run it.

🧮 10 Assessment Questions

1. In Linux, what symbol represents the root directory?

2. What is the difference between /home and /etc directories?

3. Explain “absolute path” vs “relative path” with an example.

4. What is mounting in Linux? How is it different from drive letters in Windows?

5. What does /etc/fstab contain?

6. What command shows how much disk space is available on each mounted filesystem?

7. If you run cd .., what happens?

8. What is the difference between ln and ln -s?

9. How can you check which drives are currently mounted?

10. What happens when you unmount a disk?

✅ End of Chapter 2 Summary:
You learned:

• The directory structure of Linux (FHS).

• The difference between absolute and relative paths.

• What mounting means and how /etc/fstab automates it.

• Practical file navigation commands and mount simulations.

Chapter 3 – File Permissions and Ownership

Week 3 Objective:
Understand how Linux controls who can access what, how to use commands like chmod, chown, chgrp, and how special permissions (SUID, SGID, Sticky Bit) work.

🧠 1. Layman’s Explanation — “The House Analogy”

Think of a Linux system as a big apartment building:

• Each file is like a room.

• Every room has:

  • Owner — the person who lives there.

  • Group — friends/family allowed to enter.

  • Others — everyone else in the building.

• The door permissions decide who can read, write, or execute (enter) the room.

So Linux uses 3 types of permissions for 3 types of people:

r = Read
w = Write
x = Execute

u = User (owner)
g = Group
o = Others

🧩 2. Permission Format

When you run ls -l, you’ll see something like this:

-rwxr-xr--

Let’s decode:

🧮 3. Numeric Representation (Octal Mode)

Each permission has a numeric value:

Permission Value Read (r) 4 Write (w) 2 Execute (x) 1

Example:

• rwx = 4+2+1 = 7

• rw- = 4+2 = 6

• r-- = 4 = 4

So chmod 755 file means:

7 (rwx) for user
5 (r-x) for group
5 (r-x) for others

🧰 4. Ownership and Groups

Every file belongs to:

• A user (owner)

• A group

Commands:

🧷 5. Special Permissions (SUID, SGID, Sticky Bit)

You’ll often see:

-rwsr-xr-x   (SUID)
drwxr-sr-x   (SGID)
drwxrwxrwt   (Sticky)

🧪 Example Environment Setup

# Create safe working area
mkdir -p /tmp/linux_course/week3
cd /tmp/linux_course/week3

# Create some demo users and groups (simulation only)
echo “Simulating user and group environment...”
# These won’t actually add real users without root — simulation only
mkdir users
touch users/alice.txt users/bob.txt users/shared.txt

# Make them all readable
chmod 644 users/*.txt
ls -l users

⚙️ Example 1 — Viewing Permissions

cd /tmp/linux_course/week3/users
ls -l

Output looks like:

-rw-r--r-- 1 student student 0 Oct 12 19:30 alice.txt
-rw-r--r-- 1 student student 0 Oct 12 19:30 bob.txt
-rw-r--r-- 1 student student 0 Oct 12 19:30 shared.txt

Meaning:

• User can read/write.

• Group and others can only read.

⚙️ Example 2 — Changing Permissions with chmod

# Add execute permission for the owner only
chmod u+x alice.txt
ls -l alice.txt

# Remove read permission for others
chmod o-r bob.txt
ls -l bob.txt

# Give full permissions to everyone (777)
chmod 777 shared.txt
ls -l shared.txt

✅ Try numeric style:

chmod 644 alice.txt   # rw-r--r--
chmod 755 shared.txt  # rwxr-xr-x

⚙️ Example 3 — Change Ownership with chown and chgrp

# Simulate changing ownership
touch demo1.txt
sudo chown root demo1.txt
ls -l demo1.txt

# Change both user and group
sudo chown root:root demo1.txt
ls -l demo1.txt

💡 If you don’t have sudo, just note the syntax — it needs admin rights.

⚙️ Example 4 — umask (default permission template)

umask   # Show current value

# Example: create file and directory
touch fileA
mkdir dirA
ls -l fileA dirA

Typical default umask = 0022, meaning:

• New files get rw-r--r--

• New dirs get rwxr-xr-x

You can test by changing umask:

umask 0077
touch fileB
ls -l fileB

Now only the owner has access.

⚙️ Example 5 — Special Permissions (SUID, SGID, Sticky)

# Create demo scripts
cd /tmp/linux_course/week3
echo -e ‘#!/bin/bash\necho “Running as $(whoami)”’ > suid_demo.sh
chmod 755 suid_demo.sh

# Set SUID bit (on script - demonstration)
chmod u+s suid_demo.sh
ls -l suid_demo.sh

Notice the “s” instead of “x” on the user permission column.

✅ SGID and Sticky examples:

# SGID
mkdir groupdir
chmod 2775 groupdir
ls -ld groupdir

# Sticky bit
mkdir stickydir
chmod 1777 stickydir
ls -ld stickydir

🧱 Example Permission Table

🧩 10 Hands-On Exercises

Perform these inside /tmp/linux_course/week3.

1. Create 3 files: report.txt, data.txt, script.sh.

2. Use chmod to make:

   • report.txt readable/writable by you only.

   • data.txt readable by everyone.

   • script.sh executable by everyone.

3. Check with ls -l.

4. Use numeric permissions to make report.txt = 600, data.txt = 644, script.sh = 755.

5. Create a folder secure_dir and make it accessible only to you (700).

6. Run umask and note the output.

7. Change umask temporarily to 077 and create a new file; check its default permissions.

8. Create directory group_project, run chmod 2775 group_project, and explain the SGID effect.

9. Create directory shared_area, run chmod 1777 shared_area — what does Sticky Bit do?

10. Create example.sh script that prints whoami, add SUID (chmod u+s example.sh), then ls -l to observe change.

🧠 10 Assessment Questions

1. What do the letters r, w, and x stand for in file permissions?

2. Who are the three categories of users in Linux permissions?

3. Convert the permission string -rw-r--r-- into numeric form.

4. What command is used to change ownership of a file?

5. How do you remove write permission for “others” on a file?

6. What does chmod 755 script.sh mean in words?

7. What does SUID do? Give one real-world example.

8. Why is the Sticky Bit important for directories like /tmp?

9. What does the command umask 077 change?

10. What command lists all files with their permissions?

✅ End of Chapter 3 Summary
You learned:

• What file permissions mean (user, group, others).

• How to use chmod, chown, and chgrp.

• Numeric permission codes (like 644, 755).

• Special permissions: SUID, SGID, Sticky Bit.

• How to view and modify default permissions with umask.

Chapter 4 – Linux Command Line Essentials

Week 4 Objective:
Learn how to navigate, search, filter, manipulate text, and combine commands efficiently using pipes, redirection, and powerful utilities like grep, find, awk, and sed.

🧠 1. Layman’s Explanation — “The Power of Words”

Think of the Linux command line like a kitchen 🍳

• Each command is a tool (knife, pan, blender).

• Each file is an ingredient.

• You can connect tools together with pipes (|) — like sending chopped veggies straight from knife to pan!

• You can redirect output (>) — like saving your finished dish into a container.

The goal: learn to combine simple commands to do complex tasks easily.

🧩 2. The Most Common File Viewing Commands

🧱 3. Searching & Filtering Data

grep (Global Regular Expression Print)

Used to search inside files.

Example:

grep “error” /var/log/syslog

Finds all lines containing the word error.

🧩 4. Redirection and Pipes

🧮 5. Text Manipulation Tools

💻 Let’s Practice (All Examples Runnable)

Setup:

mkdir -p /tmp/linux_course/week4
cd /tmp/linux_course/week4

⚙️ Example 1 – Create Sample Data

cat > employees.txt <<’EOF’
ID,Name,Department,Salary
1,John,IT,5000
2,Jane,HR,5500
3,Raj,Finance,6000
4,Meena,IT,6200
5,Ali,Sales,4800
6,Chen,IT,7000
EOF

✅ Check:

cat employees.txt

⚙️ Example 2 – View Files

echo “=== Using cat ===”
cat employees.txt

echo “=== Using head ===”
head -n 3 employees.txt

echo “=== Using tail ===”
tail -n 2 employees.txt

echo “=== Using less ===”
less employees.txt

⚙️ Example 3 – Searching with grep

# Find who works in IT
grep “IT” employees.txt

# Ignore case
grep -i “it” employees.txt

# Show line numbers
grep -n “Finance” employees.txt

# Exclude HR
grep -v “HR” employees.txt

⚙️ Example 4 – Cut, Sort, and Uniq

# Extract the Department column (3rd field)
cut -d’,’ -f3 employees.txt

# Sort departments alphabetically
cut -d’,’ -f3 employees.txt | sort

# Show unique departments only
cut -d’,’ -f3 employees.txt | sort | uniq

⚙️ Example 5 – Using awk for Simple Reports

# Print name and salary
awk -F’,’ ‘{print $2, $4}’ employees.txt

# Print all IT employees only
awk -F’,’ ‘$3==”IT” {print $2, $3, $4}’ employees.txt

# Calculate total salary
awk -F’,’ ‘NR>1 {sum+=$4} END {print “Total Salary:”, sum}’ employees.txt

# Find average salary
awk -F’,’ ‘NR>1 {sum+=$4; count++} END {print “Average Salary:”, sum/count}’ employees.txt

⚙️ Example 6 – sed (Find and Replace)

# Replace ‘IT’ with ‘Tech’ and save to new file
sed ‘s/IT/Tech/g’ employees.txt > updated.txt
cat updated.txt

# Delete header line
sed ‘1d’ employees.txt

# Add prefix to each line
sed ‘s/^/Employee: /’ employees.txt

⚙️ Example 7 – Pipes & Redirection Together

# List all shell scripts in /bin and count them
ls /bin | grep “\.sh” | wc -l

# Save IT department employees to a separate file
grep “IT” employees.txt > it_team.txt
cat it_team.txt

# Append HR employees at the end
grep “HR” employees.txt >> it_team.txt
cat it_team.txt

⚙️ Example 8 – Combine Tools: Find + Grep

# Create a few text files
echo “Linux Rocks” > note1.txt
echo “DevOps uses Linux” > note2.txt
echo “Python is fun” > note3.txt

# Find all .txt files containing “Linux”
find . -name “*.txt” | xargs grep “Linux”

⚙️ Example 9 – Redirect Errors

# Try accessing non-existent folder
ls /no/such/path 2> error.log

# Show contents of the error log
cat error.log

⚙️ Example 10 – Sort & Filter Combined Example

# Show all salaries sorted in descending order
awk -F’,’ ‘NR>1 {print $4}’ employees.txt | sort -nr

# Show top 3 salaries
awk -F’,’ ‘NR>1 {print $4}’ employees.txt | sort -nr | head -3

🧩 10 Hands-On Exercises

Perform these in /tmp/linux_course/week4:

1. Create a file products.csv with columns: ID,Name,Category,Price.

2. Add at least 5 products manually.

3. Use cat and head to preview it.

4. Find all products belonging to category “Electronics” using grep.

5. Use awk to print only Name and Price columns.

6. Use awk to calculate total price of all items.

7. Use cut and sort to show sorted category list (unique).

8. Use sed to replace all commas with tabs and save as tab_products.txt.

9. Create a pipeline that counts how many products cost more than 1000 (hint: awk ‘$4>1000’).

10. Redirect all commands’ output to summary.log for submission.

🧠 10 Assessment Questions

1. What is the difference between cat and less?

2. How would you show only the last 5 lines of a file?

3. What does the pipe | do between two commands?

4. Write a command to find all lines containing “error” (case-insensitive) in app.log.

5. What is the purpose of cut and sort?

6. Write a command to show unique departments in employees.txt.

7. How can you replace all instances of “Dev” with “Developer” inside a file?

8. What’s the difference between > and >>?

9. Write a command that finds all .txt files and searches for “Linux” inside them.

10. What is the meaning of awk -F’,’ ‘NR>1 {sum+=$4} END {print sum}’ employees.txt?

✅ End of Chapter 4 Summary
You learned:

• How to view, search, and manipulate text in Linux.

• How to redirect, pipe, and filter command outputs.

• How to use tools like grep, awk, and sed to extract insights.

• How to combine multiple commands to automate analysis quickly.

Chapter 5 – User and Group Administration

Week 5 Objective:
Learn how Linux handles accounts, groups, and permissions, and how to manage them using commands and configuration files.

🧠 1. Layman’s Explanation — “The Company Analogy”

Think of a Linux system as a company office:

• Each user is an employee with their own desk (home directory).

• Each group is a team (e.g., Developers, HR).

• The administrator (root) is like the CEO — has all access.

• sudo is like temporary CEO permission given to trusted staff.

Every user has a username, password, user ID (UID), group ID (GID), and home directory.

🧩 2. Key Files Involved

🧰 3. Important Commands

🧮 4. The /etc/passwd Format

Each line looks like this:

username:x:UID:GID:Comment:HomeDirectory:Shell

Example:

bavi:x:1001:1001:Bavithran:/home/bavi:/bin/bash

Explanation:

🧩 5. The /etc/group Format

group_name:x:GID:user1,user2

Example:

devops:x:1002:bavi,janani

💻 Let’s Practice (Safe Demo Environment)

We’ll simulate user/group management in /tmp/linux_course/week5/demo_users
(not real system users — safe for non-root users).

⚙️ Example 1 – Create a Demo Folder for Users

mkdir -p /tmp/linux_course/week5/demo_users/{home,groups}
cd /tmp/linux_course/week5/demo_users

⚙️ Example 2 – Simulate Adding Users (No Root Needed)

We’ll just create files to represent users.

# Create mock user entries
echo “bavi:x:1001:1001:Bavi:/home/bavi:/bin/bash” > users.txt
echo “janani:x:1002:1002:Janani:/home/janani:/bin/bash” >> users.txt
echo “vikram:x:1003:1003:Vikram:/home/vikram:/bin/bash” >> users.txt

cat users.txt

⚙️ Example 3 – View Users and Groups on Real System

# Show first 5 users in /etc/passwd
head -n 5 /etc/passwd

# Show your user info
whoami
id

⚙️ Example 4 – Create Real Groups and Users (requires sudo)

(Run only if you’re on a local VM or have admin rights)

sudo groupadd devops
sudo useradd -m -s /bin/bash bavi
sudo usermod -aG devops bavi
sudo passwd bavi
id bavi

Explanation:

• -m → creates a home directory.

• -s → defines shell.

• -aG → adds user to group.

⚙️ Example 5 – Manage Passwords and Expiry

# Show password aging policy
sudo chage -l bavi

# Set password to expire in 10 days
sudo chage -E $(date -d “+10 days” +%Y-%m-%d) bavi

⚙️ Example 6 – Configuring sudo Permissions

# Add user to sudo group
sudo usermod -aG sudo bavi

# Verify group membership
groups bavi

# Test sudo (if you have privileges)
sudo whoami

/etc/sudoers file controls this. Always use visudo to edit it safely.

⚙️ Example 7 – Locking and Unlocking Accounts

# Lock user account
sudo usermod -L bavi

# Unlock user account
sudo usermod -U bavi

⚙️ Example 8 – Creating and Managing Groups

# Create new group
sudo groupadd testers

# Add user to multiple groups
sudo usermod -aG devops,testers bavi

# Show all groups for user
groups bavi

⚙️ Example 9 – Deleting Users and Groups

# Delete user
sudo userdel -r bavi

# Delete group
sudo groupdel devops

⚙️ Example 10 – View All Users and Groups (Read-only)

cut -d: -f1 /etc/passwd | head -n 10
cut -d: -f1 /etc/group | head -n 10

🧩 10 Hands-On Exercises

Perform these on your VM or using mock files under /tmp/linux_course/week5:

1. Create mock files named users.txt and groups.txt.

2. Add 3 users and 2 groups to these files.

3. Use cut to extract usernames from /etc/passwd.

4. Use grep to find your username in /etc/passwd.

5. Run id to display your UID, GID, and group memberships.

6. Create a new group named analytics (mock using echo).

7. Add user raj to group analytics.

8. Use sort and uniq to list unique groups from /etc/group.

9. Write a script list_users.sh that prints:

#!/bin/bash
echo “Total Users: $(wc -l < /etc/passwd)”
echo “Users in sudo group:”
grep sudo /etc/group

1. Save your script, make it executable (chmod +x list_users.sh), and run it.

🧠 10 Assessment Questions

1. Which file stores basic information about all user accounts?

2. What is the difference between /etc/passwd and /etc/shadow?

3. How can you find your username and UID in Linux?

4. Write the command to create a new user named developer.

5. How do you add a user to the sudo group?

6. What does userdel -r do?

7. What command shows which groups you belong to?

8. What is the safest way to edit /etc/sudoers?

9. What happens when you lock a user account using usermod -L?

10. How do you list all existing users on the system?

✅ End of Chapter 5 Summary
You learned:

• How Linux stores and manages user & group data.

• Commands for user creation, modification, deletion.

• Safe sudo configuration.

• Understanding /etc/passwd, /etc/group, /etc/shadow.

Chapter 6 – Process and Job Management

Week 6 Objective:
Understand what a process is, how to monitor, kill, pause, resume, and schedule jobs in Linux.

🧠 1. Layman’s Explanation — “The Restaurant Analogy”

Imagine your Linux system as a restaurant kitchen 🍽️

• Each process = a chef cooking a dish (a running program).

• Each PID (Process ID) = a badge number identifying a chef.

• The parent process (PPID) = head chef who started others.

• The foreground process = the chef currently at the counter (your terminal).

• The background process = chefs cooking quietly in the back.

• The scheduler = the manager who decides who cooks first (CPU time).

🧩 2. Understanding Process States

🧰 3. Important Commands

🧩 4. Process Hierarchy – Example

Every process starts from PID 1 (systemd or init):

systemd (PID 1)
 ├── sshd
 ├── bash
 │   └── vim
 ├── apache2
 └── cron

Use pstree to see this structure in action.

💻 Let’s Practice (Runnable Examples)

Setup

mkdir -p /tmp/linux_course/week6
cd /tmp/linux_course/week6

⚙️ Example 1 – See All Processes

echo “=== Using ps ===”
ps aux | head -10

echo “=== Using ps -ef ===”
ps -ef | head -10

💡 ps aux = BSD format; ps -ef = UNIX format. Both show who owns the process, PID, CPU%, MEM%, etc.

⚙️ Example 2 – View Process Tree

pstree -p | head -20

This shows the relationship between parent and child processes.

⚙️ Example 3 – Start Background and Foreground Jobs

# Foreground job (runs in front)
sleep 10

# Background job
sleep 60 &

# Check job list
jobs

# Bring job to foreground
fg %1

Explanation:

• sleep 60 & runs in the background.

• jobs lists active jobs.

• fg brings one back to the terminal.

⚙️ Example 4 – Killing a Process

# Run a long process
sleep 1000 &
ps aux | grep sleep

# Note its PID and kill it
PID=$(pgrep -f “sleep 1000”)
echo “Killing PID: $PID”
kill $PID

If the process refuses to die:

kill -9 $PID

⚙️ Example 5 – Process Priorities (nice / renice)

Every process has a nice value between -20 (high priority) and +19 (low priority).

# Run a CPU task with low priority
nice -n 10 bash -c ‘for i in {1..50000}; do :; done’ &

# Find its PID
ps -eo pid,ni,comm | grep bash

# Change its priority
sudo renice -n -5 -p <PID>

⚙️ Example 6 – Monitor Processes (top and htop)

top

Press:

• q → quit

• P → sort by CPU

• M → sort by memory

For a modern UI (if installed):

htop

⚙️ Example 7 – Scheduling Tasks with cron

# Open crontab for current user
crontab -e

Add a line:

* * * * * echo “Hello from cron at $(date)” >> /tmp/linux_course/week6/cron_log.txt

This runs every minute.

To list all scheduled jobs:

crontab -l

⚙️ Example 8 – One-Time Task with at

# Schedule a message in 2 minutes
echo “echo ‘Task ran at $(date)’ >> /tmp/linux_course/week6/at_log.txt” | at now + 2 minutes

⚙️ Example 9 – Monitoring Resource Usage with ps and top

# Show top 5 CPU-consuming processes
ps -eo pid,comm,%cpu --sort=-%cpu | head -6

# Show top 5 memory-consuming processes
ps -eo pid,comm,%mem --sort=-%mem | head -6

⚙️ Example 10 – Automate Process Check Script

cat > process_check.sh <<’EOF’
#!/bin/bash
echo “=== Process Report on $(date) ===”
ps -eo pid,comm,%cpu,%mem --sort=-%cpu | head -10
EOF
chmod +x process_check.sh
./process_check.sh

🧩 10 Hands-On Exercises

1. Run ps aux and note your terminal’s PID and parent PID.

2. Run a sleep 30 & background job and bring it to the foreground.

3. Create two background processes (sleep 100, sleep 200) and list them using jobs.

4. Use kill to stop one of them.

5. Start a CPU-intensive process with nice -n 15, then check its priority using ps -l.

6. Change its priority using renice -n 5 -p <PID>.

7. View all running processes sorted by memory usage (ps -eo %mem,cmd --sort=-%mem).

8. Schedule a cron job that appends System Alive! to a file every 2 minutes.

9. Schedule a one-time at job that prints “Lunch break!” after 3 minutes.

10. Write a script cleanup.sh that kills all sleep processes automatically.

🧠 10 Assessment Questions

1. What is a process in Linux?

2. What is the difference between a foreground and background process?

3. How do you list all running processes on the system?

4. What is a PID and how can you find it?

5. What command displays a process hierarchy?

6. What is the difference between kill and pkill?

7. What does the command nice -n 10 do?

8. How do you schedule a task to run every day at 6 PM?

9. What command lists currently scheduled cron jobs?

10. What does the at command do?

✅ End of Chapter 6 Summary
You learned:

• What processes are and how they’re structured.

• How to manage and monitor processes using ps, top, kill, etc.

• How to schedule tasks with cron and at.

• How to adjust process priorities with nice and renice.

Chapter 7 – Software and Package Management

Week 7 Objective:
Learn how to install, update, and manage packages on both Debian/Ubuntu (using apt) and RHEL/CentOS/Fedora (using yum, dnf, and rpm).
You’ll also understand repositories, package info, and software compilation.

🧠 1. Layman’s Explanation — “The App Store Analogy”

Think of your Linux system as a smartphone 📱

• apt, yum, and dnf are like your App Store managers.

• Packages are the apps.

• Repositories (repos) are the online stores where these apps are kept.

• rpm or dpkg are like manual app installers (.apk or .exe).

• Dependencies are the libraries needed for an app to run (like needing Java before installing a Java app).

🧩 2. Package Types by Distribution

🧰 3. Common Commands

🟦 Debian/Ubuntu (apt / dpkg)

🟥 RHEL/CentOS/Fedora (yum / dnf / rpm)

🧩 4. Repositories Explained

Repositories are like warehouses of software.
They contain thousands of packages and metadata.

Config files for repos are stored in:

• /etc/apt/sources.list and /etc/apt/sources.list.d/ (Debian/Ubuntu)

• /etc/yum.repos.d/*.repo (RHEL/CentOS)

Example (RHEL):

[base]
name=BaseOS
baseurl=http://mirror.centos.org/centos/8/BaseOS/x86_64/os/
enabled=1
gpgcheck=1

💻 Let’s Practice (Runnable Examples)

Create a safe workspace:

mkdir -p /tmp/linux_course/week7
cd /tmp/linux_course/week7

(These commands are read-only or safe to execute on any system.)

⚙️ Example 1 – Check Distro Family

cat /etc/os-release

Output might contain:

ID=ubuntu
ID_LIKE=debian

or

ID=”rhel”
ID_LIKE=”fedora”

⚙️ Example 2 – List Installed Packages

For Ubuntu/Debian:

dpkg -l | head -10

For RHEL/CentOS:

yum list installed | head -10

⚙️ Example 3 – Search and Install Software

(Run only if you have sudo privileges; safe to read otherwise)

# Search for curl
apt search curl    # Ubuntu/Debian
yum search curl    # RHEL/CentOS

# Install curl
sudo apt install curl -y
# OR
sudo yum install curl -y

⚙️ Example 4 – Get Information About a Package

apt show curl 2>/dev/null || yum info curl

You’ll see version, dependencies, and description.

⚙️ Example 5 – Remove and Reinstall a Package

sudo apt remove curl -y 2>/dev/null || sudo yum remove curl -y
sudo apt install curl -y 2>/dev/null || sudo yum install curl -y

⚙️ Example 6 – List Package Files

dpkg -L curl 2>/dev/null || rpm -ql curl

Shows which files were installed by that package.

⚙️ Example 7 – Download a Package Without Installing

apt download curl 2>/dev/null || yumdownloader curl
ls

⚙️ Example 8 – Install Software from Source (safe demo)

We’ll simulate this with a tiny “Hello World” C program.

cat > hello.c <<’EOF’
#include <stdio.h>
int main() {
    printf(”Hello from compiled program!\n”);
    return 0;
}
EOF

# Compile
gcc hello.c -o hello
./hello

If gcc isn’t installed:

sudo apt install build-essential -y 2>/dev/null || sudo yum groupinstall “Development Tools” -y

⚙️ Example 9 – Check Repository Configuration

# Ubuntu/Debian
cat /etc/apt/sources.list | head -n 10

# RHEL/CentOS
ls /etc/yum.repos.d/

⚙️ Example 10 – Verify Package Integrity

# Verify if curl is properly installed
dpkg -V curl 2>/dev/null || rpm -V curl

🧩 10 Hands-On Exercises

1. Check your Linux distro using /etc/os-release.

2. Use dpkg -l or yum list installed to view 10 installed packages.

3. Search for “git” and “wget” using apt search or yum search.

4. Install wget and verify it’s working (wget --version).

5. Remove and reinstall wget.

6. Display full package info using apt show wget or yum info wget.

7. List all files installed by wget (dpkg -L wget or rpm -ql wget).

8. Create and compile a hello.c program using gcc.

9. Run ls /etc/yum.repos.d/ or view /etc/apt/sources.list.

10. Create a small shell script named pkg_report.sh that prints:

#!/bin/bash
echo “=== Package Summary ===”
dpkg -l | wc -l 2>/dev/null || yum list installed | wc -l
echo “Total packages installed”

🧠 10 Assessment Questions

1. What is the difference between a package and a repository?

2. Which file extension is used for packages in Ubuntu vs RHEL?

3. What is the command to install software in Ubuntu?

4. How do you remove software in RHEL?

5. What command lists all installed packages?

6. What does apt update do?

7. What is the difference between yum and dnf?

8. How do you check which files belong to a package?

9. What command can compile source code manually?

10. Where are repository configuration files stored in RHEL/CentOS?

✅ End of Chapter 7 Summary
You learned:

• How Linux installs and manages software.

• Package formats (.deb and .rpm).

• Commands for installing, removing, and verifying packages.

• How repositories work and how to compile software from source.

Chapter 8 – Disk, Storage & File System Management

Week 8 Objective:
Learn how Linux sees storage devices, how to list them, format and mount them, manage virtual disks with LVM, understand /etc/fstab, quotas, and the idea of RAID — all without risking your data.

🧠 1. Layman’s Explanation — “The Warehouse Analogy”

Imagine your computer as a warehouse 🏭:

• The building = your physical disk.

• Rooms (partitions) = different sections inside the warehouse.

• Shelves (file systems) = how things are arranged (ext4, xfs, etc).

• Labels (mount points) = the doors through which you access a room (/mnt, /home).

• LVM = movable walls — you can resize rooms without rebuilding the whole warehouse.

🧩 2. Basic Terminology

🧰 3. Safe Read-Only Inspection Commands

💻 Safe Practice in a Sandbox

Setup

mkdir -p /tmp/linux_course/week8
cd /tmp/linux_course/week8

⚙️ Example 1 – List Disks and Mounts

echo “=== Block devices ===”
lsblk
echo
echo “=== Mounted filesystems ===”
df -hT

lsblk shows hierarchy; df -hT shows space and filesystem type.

⚙️ Example 2 – Create a Fake 20 MB Disk (File-based Loop Device)

(safe; no real disks touched)

dd if=/dev/zero of=fake_disk.img bs=1M count=20
ls -lh fake_disk.img

⚙️ Example 3 – Format and Mount the Fake Disk

sudo mkfs.ext4 fake_disk.img
mkdir -p /tmp/linux_course/week8/mnt_fake
sudo mount -o loop fake_disk.img /tmp/linux_course/week8/mnt_fake
df -h /tmp/linux_course/week8/mnt_fake

💡 You just made an ext4 filesystem inside a file and mounted it!

⚙️ Example 4 – Write and Read from the Mounted Disk

echo “Hello from fake disk $(date)” | sudo tee /tmp/linux_course/week8/mnt_fake/test.txt
cat /tmp/linux_course/week8/mnt_fake/test.txt

⚙️ Example 5 – Unmount and Verify

sudo umount /tmp/linux_course/week8/mnt_fake
file fake_disk.img

Output shows it’s an ext4 filesystem image.

⚙️ Example 6 – View / etc / fstab (Read-Only)

grep -v ‘^#’ /etc/fstab | column -t

Shows which filesystems mount automatically at boot.

⚙️ Example 7 – Display UUID and Mount with UUID

sudo blkid fake_disk.img
# Use UUID to mount again (simulate entry)
UUID=$(sudo blkid -s UUID -o value fake_disk.img)
sudo mount -o loop,uuid=$UUID fake_disk.img /tmp/linux_course/week8/mnt_fake

⚙️ Example 8 – Simulate LVM Workflow (Safe Concept Demo)

We’ll only show the commands; run these inside a disposable VM if you want real practice.

# (Do not run on production systems)
sudo pvcreate /dev/sdb
sudo vgcreate vgdata /dev/sdb
sudo lvcreate -L 5G -n lvbackup vgdata
sudo mkfs.xfs /dev/vgdata/lvbackup
sudo mount /dev/vgdata/lvbackup /mnt

Conceptually:

• PV = Physical Volume (disk)

• VG = Volume Group (pool)

• LV = Logical Volume (mountable “drive”)

⚙️ Example 9 – Disk Usage Analysis

du -sh /var /home /tmp 2>/dev/null | sort -h

Shows folder sizes in ascending order.

⚙️ Example 10 – Quota Concept Demo (Safe Sim)

Quotas limit how much storage a user can use.

Conceptual steps (do inside VM):

sudo apt install quota -y
sudo mount -o usrquota,grpquota /dev/sda1 /
sudo quotacheck -cum /
sudo quotaon /
sudo edquota username

🧩 10 Hands-On Exercises (Safe Mode)

1. Run lsblk and identify your primary disk.

2. Run df -hT and note mount points and filesystem types.

3. Create a 10 MB fake disk (dd if=/dev/zero of=disk10.img bs=1M count=10).

4. Format it with mkfs.ext4 and mount under /tmp/linux_course/week8/mnt10.

5. Copy /etc/hosts into it and verify.

6. Unmount and check its type using file.

7. Run blkid and find its UUID.

8. Open /etc/fstab and locate the line for /.

9. Use du -sh ~ to see how much space your home uses.

10. Write a script disk_report.sh that prints:

#!/bin/bash
echo “=== Disk Summary ===”
df -hT | grep -v tmpfs
echo “Largest directories:”
du -sh /* 2>/dev/null | sort -hr | head -5

🧠 10 Assessment Questions

1. What command lists all block devices?

2. What is the difference between a partition and a filesystem?

3. What file determines which disks mount automatically at boot?

4. What does the mount command do?

5. How do you unmount a filesystem safely?

6. What is a loop device and why is it safe for practice?

7. Name three common Linux filesystems.

8. In LVM, what do PV, VG, and LV stand for?

9. What command shows disk usage per directory?

10. What is the main advantage of using LVM?

✅ End of Chapter 8 Summary
You learned:

• How to inspect and mount disks safely.

• How to create and test filesystems using loop-back images.

• The roles of /etc/fstab, UUIDs, and LVM.

• Concepts of quotas and RAID (theory).

Chapter 9 – System Startup & Services

Week 9 Objective:
Understand the boot process, manage and troubleshoot system services, and control which services start automatically.

🧠 1 – Layman’s Explanation — “The Morning Routine Analogy”

Think of your computer as a person waking up in the morning:

1. BIOS/UEFI → Alarm clock rings (power on).

2. Bootloader (GRUB) → Chooses which outfit (OS kernel) to wear.

3. Kernel → Gets dressed and starts basic body functions (CPU, memory, devices).

4. systemd → Morning planner that starts each task in the right order (network, login, GUI).

5. User login → You’re ready for work!

🧩 2 – Key Startup Components

🧰 3 – Important systemd Commands

💻 Safe Practice Examples

Setup

mkdir -p /tmp/linux_course/week9
cd /tmp/linux_course/week9

⚙️ Example 1 – View Boot Time and Critical Path

systemd-analyze
systemd-analyze blame | head -15

Shows total boot time and which services took longest.

⚙️ Example 2 – List Active Services

systemctl list-units --type=service | head -20

⚙️ Example 3 – Inspect a Specific Service

systemctl status ssh 2>/dev/null || systemctl status systemd-journald

Scroll with arrows / press q to quit.

⚙️ Example 4 – Start, Stop, Restart a Service

(Try this only with a harmless service like cron or ssh if present.)

sudo systemctl restart cron 2>/dev/null || sudo systemctl restart ssh

⚙️ Example 5 – Enable and Disable at Boot

sudo systemctl enable cron
sudo systemctl disable cron

⚙️ Example 6 – Check Default Boot Target

systemctl get-default

Common targets:

• multi-user.target → console mode

• graphical.target → GUI mode

Change temporarily:

sudo systemctl isolate multi-user.target

⚙️ Example 7 – View Failed Services

systemctl --failed

⚙️ Example 8 – Examine Logs with journalctl

# Show messages from this boot
journalctl -b | head -20

# Show logs for a unit
journalctl -u ssh | tail -20

⚙️ Example 9 – Create a Custom Service (Unit File Demo)

cat > hello.service <<’EOF’
[Unit]
Description=Simple Hello Service

[Service]
ExecStart=/bin/echo “Hello from systemd!”
Type=oneshot

[Install]
WantedBy=multi-user.target
EOF

sudo cp hello.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl start hello.service
sudo systemctl status hello.service

💡 This creates a service that prints a message and exits.

⚙️ Example 10 – Simulate Boot Troubleshooting

# View kernel boot messages
dmesg | head -20

# Show last reboot info
last reboot | head -5

🧩 10 Hands-On Exercises

1. Run systemd-analyze and note your boot time.

2. List the 10 slowest boot services (systemd-analyze blame | head -10).

3. View the status of cron or ssh service.

4. Restart that service and verify it is active.

5. Enable the service to start at boot, then disable it.

6. Use systemctl get-default to see current target.

7. Switch to multi-user.target and then back to graphical.target (if applicable).

8. View all failed services.

9. Create and run the custom hello.service demo.

10. Use journalctl -b | grep error to list boot-time errors.

🧠 10 Assessment Questions

1. What is the first process started by the Linux kernel?

2. What does systemctl status sshd show?

3. How do you list all running services?

4. What command shows the system’s default boot target?

5. How can you change the default boot target permanently?

6. Which file defines a custom service (unit file)?

7. What does sudo systemctl enable nginx do?

8. How can you view logs for a specific service?

9. What does systemctl --failed display?

10. Why is systemd-analyze blame useful for admins?

✅ End of Chapter 9 Summary
You learned:

• The entire Linux boot sequence and systemd’s role.

• How to start, stop, enable, and inspect services.

• How to analyze boot performance and troubleshoot failed units.

• How to create a custom systemd service.

Chapter 10 – Networking in Linux

Week 10 Objective:
Understand how Linux handles networking, from interface configuration to connectivity testing, SSH access, and simple web/file transfers.

🧠 1 – Layman’s Explanation — “The Postal System Analogy”

Think of the internet as a global postal network:

• Each computer is a house.

• Its IP address is the street address.

• Packets are the letters you send.

• Routers are the post offices forwarding letters.

• DNS is the phone book translating names into addresses.

🧩 2 – Key Concepts

🧰 3 – Common Networking Commands

💻 Safe Practice Examples

Setup

mkdir -p /tmp/linux_course/week10
cd /tmp/linux_course/week10

⚙️ Example 1 – View Interfaces and Addresses

ip addr show | grep -E ‘inet | ^[0-9]’

⚙️ Example 2 – Display Routing Table

ip route

Shows default gateway and network paths.

⚙️ Example 3 – Check Connectivity

ping -c 4 8.8.8.8
ping -c 4 google.com

-c 4 = stop after 4 pings.

⚙️ Example 4 – Test DNS Resolution

nslookup openai.com || dig openai.com +short

⚙️ Example 5 – Show and Change Hostname

hostname
sudo hostnamectl set-hostname linuxstudent
hostnamectl status

⚙️ Example 6 – Show Listening Ports and Services

sudo ss -tuln | head -n 10

Shows TCP/UDP ports and bound processes.

⚙️ Example 7 – Check Firewall Status (if available)

sudo ufw status 2>/dev/null || sudo firewall-cmd --state 2>/dev/null || echo “No firewall tool detected”

⚙️ Example 8 – Transfer Files Between Systems (via SSH)

(Needs another machine or VM with SSH enabled)

scp hello.txt user@remotehost:/tmp/

⚙️ Example 9 – Run a Simple Web Server (Lab Demo)

echo “<h1>Hello Linux Networking</h1>” > index.html
python3 -m http.server 8080 &
curl http://localhost:8080

Stop it:

pkill -f “http.server”

⚙️ Example 10 – Inspect Network Interfaces with nmcli (if available)

nmcli device status

🧩 10 Hands-On Exercises

1. Show all interfaces with ip addr.

2. Display routing table using ip route.

3. Find your default gateway.

4. Ping 8.8.8.8 and google.com — note any difference.

5. Use nslookup to resolve openai.com.

6. Check your hostname with hostnamectl.

7. View open ports using ss -tuln.

8. Start the temporary Python web server and fetch its page using curl.

9. Stop the web server process.

10. Write a script net_report.sh that prints:

#!/bin/bash
echo “=== Network Report on $(date) ===”
hostname -f
ip addr show | grep inet
echo “Default Gateway:”
ip route | grep default
echo “DNS Check:”
nslookup openai.com | grep Address

🧠 10 Assessment Questions

1. What command shows all network interfaces and IP addresses?

2. What does ip route display?

3. How can you test connectivity to another host?

4. What is the purpose of a default gateway?

5. How do you resolve a hostname to an IP address?

6. What command shows the current system hostname?

7. What command lists listening network ports?

8. What tool starts a temporary HTTP server in Python 3?

9. What is the difference between ping 8.8.8.8 and ping google.com?

10. What is the function of nmcli?

✅ End of Chapter 10 Summary
You learned how to:

• Inspect network interfaces and routes.

• Test connectivity and DNS resolution.

• Manage hostnames and view ports.

• Run simple web/file transfer tests securely.

Chapter 11 – Logging and System Monitoring

Week 11 Objective:
Understand how Linux handles system logs, how to view them using journalctl, analyze /var/log files, set up log rotation, and monitor live system performance.

🧠 1 – Layman’s Explanation — “The Security Camera Analogy”

Think of Linux as a smart building:

• Logs are your CCTV recordings — every door open, alarm, or visitor gets recorded.

• journalctl is your control room monitor, showing live footage.

• logrotate is the security clerk that archives old footage.

• Monitoring tools (top, vmstat) are your building health sensors — checking temperature (CPU), air pressure (RAM), and power (disk I/O).

🧩 2 – Key Log Types

🧰 3 – Important Commands

💻 Safe Practice Examples

Setup

mkdir -p /tmp/linux_course/week11
cd /tmp/linux_course/week11

⚙️ Example 1 – Explore Log Files

ls /var/log | head -20

You’ll see logs like syslog, auth.log, dmesg, kern.log, apt.

⚙️ Example 2 – View System Logs

sudo journalctl -b | head -20

Shows logs from the current boot session.

⚙️ Example 3 – Filter Logs by Service

sudo journalctl -u ssh | tail -20

Shows only SSH-related logs.

⚙️ Example 4 – Follow Logs in Real-Time

sudo journalctl -f

Press Ctrl + C to stop following.

⚙️ Example 5 – Show Only Errors and Warnings

sudo journalctl -p 3 -b | head -20

Priority 3 = errors;
4 = warnings;
5 = notices.

⚙️ Example 6 – View Authentication Logs

sudo cat /var/log/auth.log | grep “Accepted”

Lists successful logins via SSH or terminal.

⚙️ Example 7 – Kernel Logs

dmesg | tail -20

Shows recent hardware or driver messages.

⚙️ Example 8 – Log Rotation (Safe Simulation)

sudo cat /etc/logrotate.conf | head -10
sudo ls /etc/logrotate.d/ | head -5

💡 logrotate automatically compresses and archives old logs (e.g., syslog.1.gz).

⚙️ Example 9 – Monitor System Resources (top & vmstat)

top
# Press q to quit

Or:

vmstat 2 5

Shows CPU, memory, and I/O stats every 2 seconds (5 times).

⚙️ Example 10 – Disk and Memory Monitoring

free -h
df -h
iostat 2 3 2>/dev/null || echo “Install sysstat for iostat”

🧩 10 Hands-On Exercises

1. Run ls /var/log and list 10 log files.

2. Use journalctl -b | head -20 to view current boot logs.

3. View the last 20 SSH-related messages using journalctl -u ssh | tail.

4. Show all error messages from the current boot (journalctl -p 3 -b).

5. Display all authentication logs containing “Failed password”.

6. Use dmesg to view the last 10 hardware messages.

7. Check your logrotate configuration (cat /etc/logrotate.conf).

8. Run top and identify the highest CPU-using process.

9. Run df -h to view free disk space.

10. Write a script sys_report.sh:

#!/bin/bash
echo “=== System Health Report ===”
date
echo
echo “--- Disk Usage ---”
df -h | grep -v tmpfs
echo
echo “--- Memory ---”
free -h
echo
echo “--- Top CPU Consumers ---”
ps -eo pid,comm,%cpu --sort=-%cpu | head -6

1. Then chmod +x sys_report.sh and run it.

🧠 10 Assessment Questions

1. Where are most Linux logs stored?

2. What command shows logs for a specific systemd service?

3. How do you view only the latest logs as they happen?

4. What is logrotate used for?

5. What is the function of /var/log/auth.log?

6. Which command shows kernel messages?

7. How do you check system logs for errors only?

8. What does the command vmstat 2 5 do?

9. What is the difference between df and free?

10. What tool is used for viewing live CPU/memory processes interactively?

✅ End of Chapter 11 Summary
You learned:

• How Linux logging works (journalctl, /var/log, dmesg).

• How to find errors and monitor logs in real time.

• How to rotate logs automatically.

• How to monitor CPU, memory, and disk usage with top, vmstat, and df.

Chapter 12 – Backup and Recovery

Week 12 Objective:
Learn how to create and restore backups using tools like tar, rsync, and cron.
Understand backup strategies, scheduling, and basic recovery principles.

🧠 1 – Layman’s Explanation — “The Library Analogy”

Think of your server as a library 📚:

• The books = your files and data.

• The backup = photocopies stored safely elsewhere.

• The restore = bringing those copies back when something goes wrong.

• The cron job = the librarian who automatically makes those photocopies every night.

Without backups, one accidental delete = disaster.
With backups, you just smile and restore. 😎

🧩 2 – Backup Strategy Types

🧰 3 – Essential Backup Tools

💻 Safe Practice Examples

Setup

mkdir -p /tmp/linux_course/week12/source
mkdir -p /tmp/linux_course/week12/backups
cd /tmp/linux_course/week12

Create sample files:

echo “Config File” > source/config.txt
echo “Data File” > source/data.log
ls source

⚙️ Example 1 – Create a TAR Backup

tar -cvf backups/mybackup.tar source/

• c = create

• v = verbose (show files)

• f = filename

Verify:

tar -tf backups/mybackup.tar

⚙️ Example 2 – Compress Backup

gzip backups/mybackup.tar
ls backups/

File becomes mybackup.tar.gz.

To uncompress:

gunzip backups/mybackup.tar.gz

⚙️ Example 3 – Restore from TAR Backup

rm -rf source
tar -xvf backups/mybackup.tar -C .
ls source/

⚙️ Example 4 – Using rsync for Incremental Backups

mkdir -p /tmp/linux_course/week12/remote_backup
rsync -av source/ remote_backup/

Modify a file and re-run:

echo “Updated line” >> source/config.txt
rsync -av source/ remote_backup/

✅ Only changed files get re-copied.

⚙️ Example 5 – Automate Backup with Cron (Safe Demo)

crontab -l 2>/dev/null
(crontab -l 2>/dev/null; echo “0 2 * * * tar -czf /tmp/linux_course/week12/backups/auto_$(date +\%F).tar.gz /tmp/linux_course/week12/source”) | crontab -

This schedules a backup every day at 2 AM.

⚙️ Example 6 – Verify Backups

tar -tf backups/mybackup.tar | grep config

⚙️ Example 7 – Simulate Remote Copy (scp)

(Won’t actually connect — safe to read example only)

scp backups/mybackup.tar user@192.168.1.100:/backup/

⚙️ Example 8 – Partial Restore

Extract only one file:

tar -xvf backups/mybackup.tar source/config.txt

⚙️ Example 9 – Backup Verification Script

cat > verify_backup.sh <<’EOF’
#!/bin/bash
tar -tf backups/mybackup.tar > /dev/null
if [ $? -eq 0 ]; then
  echo “Backup verified successfully ✅”
else
  echo “Backup corrupted ❌”
fi
EOF
chmod +x verify_backup.sh
./verify_backup.sh

⚙️ Example 10 – Understanding Rescue Mode (Theory)

If the system won’t boot:

1. Boot from a Live Linux ISO.

2. Mount your main partition manually (e.g., /mnt).

3. Chroot into it: chroot /mnt.

4. Repair files, reinstall bootloader, or copy data out.

🧩 10 Hands-On Exercises

1. Create a project/ directory with 3 files and back it up as project_backup.tar.

2. List files inside the TAR without extracting.

3. Extract only one file from the archive.

4. Compress the archive using gzip.

5. Create another folder and use rsync to copy data into it.

6. Modify one file and re-run rsync; observe changed files only.

7. Schedule a cron job to back up /tmp/linux_course/week12/source every minute (for test).

8. Verify the latest TAR file using tar -tf.

9. Write a script daily_backup.sh that creates a timestamped compressed backup.

10. Restore all files from your backup and verify integrity.

🧠 10 Assessment Questions

1. What is the difference between full, incremental, and differential backups?

2. Which command is used to create a compressed archive of a directory?

3. How do you view the contents of a .tar file without extracting it?

4. What is the main advantage of rsync over cp?

5. What is the command to copy files securely to another host?

6. How can backups be automated daily at 2 AM?

7. How do you extract only one file from a TAR archive?

8. What does the cron field 0 2 * * * mean?

9. What is the purpose of rescue mode?

10. Write a one-liner that backs up /etc into /backup/etc_$(date +%F).tar.gz.

✅ End of Chapter 12 Summary
You learned:

• How to create, compress, and restore backups using tar and rsync.

• How to automate backups using cron.

• How to simulate remote backups using scp.

• What to do in system recovery or rescue mode.

Chapter 12 – Backup and Recovery

Week 12 Objective:
Learn how to create and restore backups using tools like tar, rsync, and cron.
Understand backup strategies, scheduling, and basic recovery principles.

🧠 1 – Layman’s Explanation — “The Library Analogy”

Think of your server as a library 📚:

• The books = your files and data.

• The backup = photocopies stored safely elsewhere.

• The restore = bringing those copies back when something goes wrong.

• The cron job = the librarian who automatically makes those photocopies every night.

Without backups, one accidental delete = disaster.
With backups, you just smile and restore. 😎

🧩 2 – Backup Strategy Types

🧰 3 – Essential Backup Tools

💻 Safe Practice Examples

Setup

mkdir -p /tmp/linux_course/week12/source
mkdir -p /tmp/linux_course/week12/backups
cd /tmp/linux_course/week12

Create sample files:

echo “Config File” > source/config.txt
echo “Data File” > source/data.log
ls source

⚙️ Example 1 – Create a TAR Backup

tar -cvf backups/mybackup.tar source/

• c = create

• v = verbose (show files)

• f = filename

Verify:

tar -tf backups/mybackup.tar

⚙️ Example 2 – Compress Backup

gzip backups/mybackup.tar
ls backups/

File becomes mybackup.tar.gz.

To uncompress:

gunzip backups/mybackup.tar.gz

⚙️ Example 3 – Restore from TAR Backup

rm -rf source
tar -xvf backups/mybackup.tar -C .
ls source/

⚙️ Example 4 – Using rsync for Incremental Backups

mkdir -p /tmp/linux_course/week12/remote_backup
rsync -av source/ remote_backup/

Modify a file and re-run:

echo “Updated line” >> source/config.txt
rsync -av source/ remote_backup/

✅ Only changed files get re-copied.

⚙️ Example 5 – Automate Backup with Cron (Safe Demo)

crontab -l 2>/dev/null
(crontab -l 2>/dev/null; echo “0 2 * * * tar -czf /tmp/linux_course/week12/backups/auto_$(date +\%F).tar.gz /tmp/linux_course/week12/source”) | crontab -

This schedules a backup every day at 2 AM.

⚙️ Example 6 – Verify Backups

tar -tf backups/mybackup.tar | grep config

⚙️ Example 7 – Simulate Remote Copy (scp)

(Won’t actually connect — safe to read example only)

scp backups/mybackup.tar user@192.168.1.100:/backup/

⚙️ Example 8 – Partial Restore

Extract only one file:

tar -xvf backups/mybackup.tar source/config.txt

⚙️ Example 9 – Backup Verification Script

cat > verify_backup.sh <<’EOF’
#!/bin/bash
tar -tf backups/mybackup.tar > /dev/null
if [ $? -eq 0 ]; then
  echo “Backup verified successfully ✅”
else
  echo “Backup corrupted ❌”
fi
EOF
chmod +x verify_backup.sh
./verify_backup.sh

⚙️ Example 10 – Understanding Rescue Mode (Theory)

If the system won’t boot:

1. Boot from a Live Linux ISO.

2. Mount your main partition manually (e.g., /mnt).

3. Chroot into it: chroot /mnt.

4. Repair files, reinstall bootloader, or copy data out.

🧩 10 Hands-On Exercises

1. Create a project/ directory with 3 files and back it up as project_backup.tar.

2. List files inside the TAR without extracting.

3. Extract only one file from the archive.

4. Compress the archive using gzip.

5. Create another folder and use rsync to copy data into it.

6. Modify one file and re-run rsync; observe changed files only.

7. Schedule a cron job to back up /tmp/linux_course/week12/source every minute (for test).

8. Verify the latest TAR file using tar -tf.

9. Write a script daily_backup.sh that creates a timestamped compressed backup.

10. Restore all files from your backup and verify integrity.

🧠 10 Assessment Questions

1. What is the difference between full, incremental, and differential backups?

2. Which command is used to create a compressed archive of a directory?

3. How do you view the contents of a .tar file without extracting it?

4. What is the main advantage of rsync over cp?

5. What is the command to copy files securely to another host?

6. How can backups be automated daily at 2 AM?

7. How do you extract only one file from a TAR archive?

8. What does the cron field 0 2 * * * mean?

9. What is the purpose of rescue mode?

10. Write a one-liner that backs up /etc into /backup/etc_$(date +%F).tar.gz.

✅ End of Chapter 12 Summary
You learned:

• How to create, compress, and restore backups using tar and rsync.

• How to automate backups using cron.

• How to simulate remote backups using scp.

• What to do in system recovery or rescue mode.

Chapter 13 – Linux Security and Hardening

Week 13 Objective:
Learn how to secure Linux systems through SSH hardening, sudo control, firewalls, SELinux/AppArmor, file-level protection, and auditing.

🧠 1 – Layman’s Explanation — “The Castle Analogy”

Imagine your Linux system as a castle 🏰:

• Walls = firewalls (block unwanted visitors)

• Keys = SSH keys (secure entry)

• Guards = sudo rules (who can do what)

• Surveillance cameras = audit logs (who did what)

• Locked rooms = SELinux (restrict what processes can access)

🧩 2 – Linux Security Layers

💻 Let’s Begin Practically

Setup

mkdir -p /tmp/linux_course/week13
cd /tmp/linux_course/week13

⚙️ Example 1 – Check Current Users & SSH Access

whoami
who
last | head -5

Shows who is logged in and recent logins.

⚙️ Example 2 – SSH Configuration (Safe Read-Only Check)

View SSH server settings:

sudo grep -E “Port|PermitRootLogin|PasswordAuthentication” /etc/ssh/sshd_config

✅ Recommended values:

Port 22
PermitRootLogin no
PasswordAuthentication no

⚙️ Example 3 – Generate and Use SSH Keys

(Safe to run on your own system)

ssh-keygen -t rsa -b 4096 -C “bavi@careerbytecode”

This creates:

~/.ssh/id_rsa (private key)
~/.ssh/id_rsa.pub (public key)

To copy key to a remote server:

ssh-copy-id user@server_ip

Then login passwordlessly:

ssh user@server_ip

⚙️ Example 4 – Sudo Configuration

List sudo users:

sudo grep ‘sudo’ /etc/group

Give user sudo rights:

sudo usermod -aG sudo bavi

Safely test:

sudo whoami

✅ Output: root

View sudo logs:

sudo cat /var/log/auth.log | grep sudo | tail

⚙️ Example 5 – Using Firewalls (ufw or firewalld)

For Ubuntu/Debian:

sudo ufw status
sudo ufw allow 22/tcp
sudo ufw enable

For RHEL/CentOS:

sudo firewall-cmd --state
sudo firewall-cmd --add-service=ssh --permanent
sudo firewall-cmd --reload

List rules:

sudo iptables -L -n | head

⚙️ Example 6 – SELinux/AppArmor Overview (Read-Only)

Check SELinux status:

getenforce 2>/dev/null || echo “SELinux not installed”

Modes:

• Enforcing: Strict security

• Permissive: Logs violations

• Disabled: Not active

AppArmor alternative:

sudo aa-status 2>/dev/null || echo “AppArmor not installed”

⚙️ Example 7 – Secure File Permissions

Create secure file:

echo “Secret credentials” > secret.txt
chmod 600 secret.txt
ls -l secret.txt

Only the file owner can read/write.

⚙️ Example 8 – File Integrity Check (sha256sum)

sha256sum secret.txt > hash.txt
echo “Secret credentials changed” > secret.txt
sha256sum -c hash.txt

✅ Alerts if file has been altered.

⚙️ Example 9 – Auditing with auditd

Install (if available):

sudo apt install auditd -y 2>/dev/null || sudo yum install audit -y 2>/dev/null
sudo systemctl enable --now auditd

Monitor /etc/passwd changes:

sudo auditctl -w /etc/passwd -p wa -k passwd_changes
sudo ausearch -k passwd_changes

⚙️ Example 10 – Security Check Script

cat > system_check.sh <<’EOF’
#!/bin/bash
echo “=== SECURITY CHECK REPORT ===”
echo “Date: $(date)”
echo
echo “[1] Active Users:”
who
echo
echo “[2] Sudo Users:”
grep sudo /etc/group
echo
echo “[3] SSH Config Summary:”
grep -E “Port|PermitRootLogin|PasswordAuthentication” /etc/ssh/sshd_config
echo
echo “[4] Firewall Status:”
sudo ufw status 2>/dev/null || sudo firewall-cmd --list-all 2>/dev/null
echo
echo “[5] SELinux Status:”
getenforce 2>/dev/null || echo “AppArmor/SELinux not active”
EOF
chmod +x system_check.sh
./system_check.sh

🧩 10 Hands-On Exercises

1. Run who and last to view logged-in users.

2. Open /etc/ssh/sshd_config (read-only) and check if root login is disabled.

3. Generate an SSH key using ssh-keygen.

4. Copy your public key to a remote or local test server (ssh-copy-id).

5. Add your user to the sudo group and verify sudo whoami.

6. Enable your firewall and allow only port 22 (SSH).

7. Check SELinux/AppArmor status using getenforce or aa-status.

8. Create a file secure.txt and set permission chmod 600.

9. Generate SHA checksum for secure.txt and verify it later.

10. Run the system_check.sh script and review the output.

🧠 10 Assessment Questions

1. What is the purpose of disabling PermitRootLogin in SSH?

2. What’s the difference between PasswordAuthentication and KeyAuthentication in SSH?

3. How can you safely give a user administrative privileges?

4. What is the command to allow port 80 in ufw?

5. What is SELinux and what are its modes?

6. How can you verify if a file has been modified using a hash?

7. What does auditd do?

8. What command lists all active firewall rules?

9. What does chmod 600 mean?

10. Why should logs like /var/log/auth.log be regularly monitored?

✅ End of Chapter 13 Summary
You learned how to:

• Secure SSH access using key authentication.

• Configure sudo and least-privilege access.

• Manage firewall rules (ufw/firewalld/iptables).

• Check SELinux/AppArmor enforcement.

• Audit and verify system integrity.

Chapter 14 – Performance Tuning and Troubleshooting

Week 14 Objective:
Learn how to identify, analyze, and optimize performance issues in Linux using built-in tools and safe kernel parameter tuning.

🧠 1 – Layman’s Explanation — “The Car Analogy”

Think of Linux like a car 🚗:

• CPU = engine horsepower.

• Memory (RAM) = fuel tank.

• Disk I/O = tires’ grip on the road.

• Kernel = onboard computer managing everything.

• sysctl = dashboard tuning knobs to adjust engine performance.

If the car slows down — you check the engine, fuel, or road; same logic for servers!

🧩 2 – Common Causes of Poor Performance

🧰 3 – Key Diagnostic Tools

💻 Safe Practice Environment

mkdir -p /tmp/linux_course/week14
cd /tmp/linux_course/week14

⚙️ Example 1 – Checking System Load

uptime

Output:

20:00:01 up 5 days, 3:12, 2 users, load average: 0.35, 0.50, 0.40

💡 Load average = number of running/waiting processes.
Rule of thumb:

• 1.0 = fully used single core.

• On 4-core CPU, up to 4.0 is acceptable.

⚙️ Example 2 – Monitor CPU and Memory

top

• P → sort by CPU

• M → sort by memory

• q → quit

Alternative:

vmstat 2 5

Shows CPU idle, wait time, and swap activity every 2 seconds (5 times).

⚙️ Example 3 – Memory Analysis

free -h

Displays total, used, and free RAM including swap.

To see swap activity:

swapon --show

⚙️ Example 4 – Disk I/O Monitoring

df -hT
iostat -x 2 3 2>/dev/null || echo “Install sysstat package for iostat”

• df → disk space usage

• iostat → read/write activity

⚙️ Example 5 – Find Largest Disk Usage

du -ah /var | sort -rh | head -10

Lists top 10 space-consuming files.

⚙️ Example 6 – Monitor Kernel Messages

dmesg | tail -20

Shows hardware or system warnings (e.g., “Out of memory” or “I/O error”).

⚙️ Example 7 – System Resource Limits

ulimit -a

Shows user limits (max file size, open files, etc.)

To temporarily increase file limit:

ulimit -n 4096

⚙️ Example 8 – Tune Kernel Parameters Safely

Check a value:

sysctl net.ipv4.ip_forward

Enable IP forwarding temporarily:

sudo sysctl -w net.ipv4.ip_forward=1

Make it persistent (demo only):

echo “net.ipv4.ip_forward = 1” | sudo tee -a /etc/sysctl.conf
sudo sysctl -p

⚙️ Example 9 – Check Process Tree and Zombie Processes

pstree -p | head -15
ps aux | grep ‘Z’

Zombie (Z) = process finished but parent didn’t clean up.

⚙️ Example 10 – System Health Script

cat > perf_check.sh <<’EOF’
#!/bin/bash
echo “=== System Performance Report ===”
date
echo “--- CPU Load ---”
uptime
echo
echo “--- Memory ---”
free -h
echo
echo “--- Disk Usage ---”
df -h | grep -v tmpfs
echo
echo “--- Top 5 CPU-consuming Processes ---”
ps -eo pid,comm,%cpu --sort=-%cpu | head -6
EOF
chmod +x perf_check.sh
./perf_check.sh

🧩 10 Hands-On Exercises

1. Run uptime and note the load average.

2. Use top and identify the process using most CPU.

3. Use free -h to check available memory.

4. Run vmstat 2 5 and watch for swap activity.

5. Check disk usage with df -hT.

6. Find top 5 largest files under /var using du.

7. View kernel messages using dmesg | tail.

8. Use ulimit -a to check open file limits.

9. Temporarily enable IP forwarding with sysctl -w.

10. Run your perf_check.sh report script.

🧠 10 Assessment Questions

1. What does “load average” represent?

2. How do you check free and used memory?

3. What is the function of the vmstat command?

4. How do you monitor disk I/O in Linux?

5. What command lists the top 10 largest files?

6. What is a zombie process?

7. What does the command ulimit -a show?

8. How can you temporarily modify a kernel parameter?

9. What command displays kernel log messages?

10. What tool helps you identify the process using most CPU?

✅ End of Chapter 14 Summary
You learned:

• How to measure CPU, memory, and I/O performance.

• How to interpret load averages.

• How to view kernel logs and tune sysctl safely.

• How to automate performance checks with a script.

Chapter 15 – Bash Scripting & Automation 🧠💻

This chapter transforms you from a command runner into a command creator.
You’ll learn how to automate repetitive admin tasks, write reusable scripts, and handle files, loops, conditionals, and functions.

All examples are safe to execute under /tmp/linux_course/week15.

🗓️ Week 15 Objective

Learn how to write, execute, and debug Bash shell scripts to automate Linux tasks like backups, user creation, and cleanup.

🧠 1 – Layman’s Explanation — “The Robot Assistant Analogy”

Imagine Bash as your personal robot assistant 🤖:

• You teach it by writing instructions (scripts).

• It remembers and executes them line by line.

• If you give it logic (if, for, while), it makes decisions on its own.

• If something goes wrong, you can debug and fix its instructions.

🧩 2 – What Is a Bash Script?

A Bash script is just a text file with commands that run automatically.

Basic Example:

#!/bin/bash
echo “Hello, Linux World!”

#!/bin/bash — called a shebang, tells the OS which interpreter to use.

To make it executable:

chmod +x hello.sh
./hello.sh

✅ Output:

Hello, Linux World!

🧰 3 – Script Essentials Cheat Sheet

💻 Practice Setup

mkdir -p /tmp/linux_course/week15
cd /tmp/linux_course/week15

⚙️ Example 1 – Hello Script

cat > hello.sh <<’EOF’
#!/bin/bash
echo “Hello, $(whoami)! Welcome to Bash scripting.”
EOF
chmod +x hello.sh
./hello.sh

⚙️ Example 2 – Variables and User Input

cat > userinfo.sh <<’EOF’
#!/bin/bash
echo “Enter your name:”
read name
echo “Enter your favorite color:”
read color
echo “Hello $name, your favorite color is $color.”
EOF
chmod +x userinfo.sh
./userinfo.sh

⚙️ Example 3 – Conditional Statements

cat > check_disk.sh <<’EOF’
#!/bin/bash
usage=$(df / | tail -1 | awk ‘{print $5}’ | sed ‘s/%//’)
if [ $usage -gt 80 ]; then
  echo “⚠️ Disk usage is high: ${usage}%”
else
  echo “✅ Disk usage is normal: ${usage}%”
fi
EOF
chmod +x check_disk.sh
./check_disk.sh

⚙️ Example 4 – For Loop

cat > loop_demo.sh <<’EOF’
#!/bin/bash
for i in {1..5}
do
  echo “Iteration $i”
  sleep 1
done
EOF
chmod +x loop_demo.sh
./loop_demo.sh

⚙️ Example 5 – While Loop with Counter

cat > countdown.sh <<’EOF’
#!/bin/bash
count=5
while [ $count -gt 0 ]
do
  echo “Counting down: $count”
  ((count--))
  sleep 1
done
echo “🚀 Blast off!”
EOF
chmod +x countdown.sh
./countdown.sh

⚙️ Example 6 – Command-Line Arguments

cat > greet.sh <<’EOF’
#!/bin/bash
echo “Hello $1, welcome to $2!”
EOF
chmod +x greet.sh
./greet.sh Bavi Linux

✅ Output:

Hello Bavi, welcome to Linux!

⚙️ Example 7 – Functions

cat > math.sh <<’EOF’
#!/bin/bash
add() {
  echo “Sum: $(($1 + $2))”
}
multiply() {
  echo “Product: $(($1 * $2))”
}

add 5 7
multiply 3 4
EOF
chmod +x math.sh
./math.sh

⚙️ Example 8 – Backup Script (Real-World Example)

cat > auto_backup.sh <<’EOF’
#!/bin/bash
src=”/etc”
dest=”/tmp/linux_course/week15/backup_$(date +%F_%H-%M-%S).tar.gz”
echo “Backing up $src to $dest”
sudo tar -czf $dest $src 2>/dev/null && echo “✅ Backup done!”
EOF
chmod +x auto_backup.sh
./auto_backup.sh

⚙️ Example 9 – Automation with Cron

(crontab -l 2>/dev/null; echo “0 * * * * /tmp/linux_course/week15/auto_backup.sh”) | crontab -
crontab -l

This runs your backup every hour.

⚙️ Example 10 – Debugging Mode

cat > debug_demo.sh <<’EOF’
#!/bin/bash
set -x
x=10
y=5
z=$((x * y))
echo “Result: $z”
set +x
EOF
chmod +x debug_demo.sh
./debug_demo.sh

💡 Use set -x to trace script execution.

🧩 10 Hands-On Exercises

1. Write a script that prints “System Report” with current date and uptime.

2. Write a script that takes two numbers as arguments and prints their sum.

3. Create a loop that prints numbers 1–10.

4. Create a countdown timer using while.

5. Write a script that checks disk usage and sends a warning if > 70%.

6. Write a function that calculates square of a number.

7. Write a script that asks for your name and greets you with the current time.

8. Automate cleanup of /tmp older than 7 days (demo with dummy files).

9. Use cron to schedule a script every day at midnight.

10. Debug a script using set -x and fix an intentional typo.

🧠 10 Assessment Questions

1. What is the purpose of the #!/bin/bash line?

2. How do you make a script executable?

3. What command reads input from the user?

4. How do you pass parameters to a script?

5. What does $1 represent inside a script?

6. How do you write an if-else condition in Bash?

7. How can you run a script every day automatically?

8. What command enables debugging mode?

9. How do you define and call a function in Bash?

10. What is the benefit of automating routine tasks using Bash?

✅ End of Chapter 15 Summary
You learned:

• How to write and run Bash scripts.

• How to use variables, conditionals, loops, and functions.

• How to schedule and debug scripts.

• How to automate real-world admin tasks like backups.

Chapter 16 – Git for Linux Administrators

Week 16 Objective:
Learn version control fundamentals using Git — initializing repositories, committing changes, branching, pushing to remote, and using Git to manage Linux configuration and automation scripts.

🧠 1 – Layman’s Explanation — “The Time Machine Analogy”

Think of Git as a time machine for your files 🕰️

• Every time you change a file and commit it — you take a snapshot.

• You can travel back in time to see what changed and when.

• If something breaks, you just rollback to an older version.

• For Linux admins, Git protects config files (/etc, scripts, backups) from mistakes.

🧩 2 – Key Git Concepts for Admins

💻 Practice Setup (Local, Safe)

mkdir -p /tmp/linux_course/week16/config_repo
cd /tmp/linux_course/week16/config_repo

⚙️ Example 1 – Initialize a Git Repository

git init

✅ Output:

Initialized empty Git repository in /tmp/linux_course/week16/config_repo/.git/

⚙️ Example 2 – Configure Git User Identity

git config --global user.name “Bavi”
git config --global user.email “bavi@example.com”

View config:

git config --list

⚙️ Example 3 – Create and Commit Files

echo “ServerName careerbytecode.local” > apache.conf
echo “PermitRootLogin no” > sshd_config

git add .
git commit -m “Initial configuration files added”

Check history:

git log --oneline

⚙️ Example 4 – Make and Commit Changes

echo “Port 2222” >> sshd_config
git status
git add sshd_config
git commit -m “Changed SSH port to 2222”

⚙️ Example 5 – View File Differences

git diff HEAD~1 sshd_config

Shows what was added or removed compared to the previous commit.

⚙️ Example 6 – Restore Previous Version

git checkout HEAD~1 sshd_config
cat sshd_config

✅ File reverts to earlier version.

To undo the restore:

git restore sshd_config

⚙️ Example 7 – Using .gitignore

echo “*.log” > .gitignore
echo “temp/” >> .gitignore
mkdir temp
touch debug.log
git status

Files listed in .gitignore will not appear in “untracked files.”

⚙️ Example 8 – Create and Switch Branches

git branch dev
git switch dev
echo “# Dev changes” > notes.txt
git add notes.txt
git commit -m “Added developer notes”

Switch back:

git switch main

Merge changes:

git merge dev

⚙️ Example 9 – Create a Remote Repo Simulation (Optional Demo)

(No internet needed — local simulation)

mkdir -p /tmp/linux_course/week16/remote_repo.git
cd /tmp/linux_course/week16/remote_repo.git
git init --bare

Now connect local repo:

cd /tmp/linux_course/week16/config_repo
git remote add origin /tmp/linux_course/week16/remote_repo.git
git push origin main

✅ You’ve simulated a remote Git server locally!

⚙️ Example 10 – Git for Backup & Config Sync

Automate periodic commits:

cat > git_backup.sh <<’EOF’
#!/bin/bash
repo=”/tmp/linux_course/week16/config_repo”
cd $repo
git add .
git commit -m “Auto backup at $(date +%F_%T)”
EOF

chmod +x git_backup.sh
./git_backup.sh
git log --oneline | head -5

🧩 10 Hands-On Exercises

1. Initialize a new Git repository in /tmp/linux_course/week16/test_repo.

2. Create a notes.txt and commit it.

3. Modify the file and check git status.

4. Commit your changes with a clear message.

5. View your Git commit history with git log.

6. Create and switch to a branch named testing.

7. Add a .gitignore file that ignores .bak files.

8. Simulate a merge between testing and main.

9. Run the git diff command to view changes.

10. Create a small script that automatically commits changes every hour (with cron).

🧠 10 Assessment Questions

1. What is a Git repository?

2. What does the command git init do?

3. How do you view the list of committed changes?

4. What’s the purpose of .gitignore?

5. How do you restore a previous version of a file?

6. What command creates a new branch?

7. What’s the difference between git add and git commit?

8. How do you simulate pushing to a remote repository locally?

9. Why is Git useful for system administrators?

10. What does git diff show?

✅ End of Chapter 16 Summary
You learned:

• How Git tracks configuration changes.

• How to commit, branch, merge, and rollback safely.

• How to use .gitignore to exclude temp/log files.

• How to simulate remote repositories locally.

• How to automate versioned backups.

Chapter 17 – Advanced Linux Administration

Week 17 Objective:
Master kernel management, boot configuration (GRUB), resource limits, time synchronization, and system tuning parameters — safely and practically.

🧠 1 – Layman’s Explanation — “The Brain & Nervous System Analogy” 🧠⚙️

If Linux were a human body:

• The kernel is the brain 🧠 controlling everything — CPU, memory, I/O.

• The bootloader (GRUB) is the alarm clock that wakes the brain.

• sysctl values are the neural settings (reflex speeds, thresholds).

• ulimit defines how much work each organ (process) can handle.

• chrony or NTP keeps the body’s clock in sync with the world.

🧩 2 – Major Concepts

💻 Safe Practice Setup

mkdir -p /tmp/linux_course/week17
cd /tmp/linux_course/week17

⚙️ Example 1 – View Kernel Version & Info

uname -a
cat /proc/version

✅ Output:

Linux hostname 6.8.0-37-generic #1 SMP ...

⚙️ Example 2 – List Loaded Kernel Modules

lsmod | head -10

Shows kernel modules currently loaded.
To describe one:

modinfo ext4 | head -10

⚙️ Example 3 – Bootloader (GRUB) Configuration

cat /etc/default/grub | grep GRUB_

Typical lines:

GRUB_TIMEOUT=5
GRUB_DEFAULT=0
GRUB_CMDLINE_LINUX=”quiet splash”

💡 Never modify this directly on a production system — only in a VM or test.

To apply changes (DEMO ONLY):

# sudo update-grub   ← (commented out for safety)

⚙️ Example 4 – List All sysctl Parameters (Read-only)

sysctl -a | head -20

To view a single parameter:

sysctl net.ipv4.ip_forward

✅ Output:

net.ipv4.ip_forward = 0

⚙️ Example 5 – Temporarily Change a Kernel Parameter (Safe)

sudo sysctl -w net.ipv4.ip_forward=1
sysctl net.ipv4.ip_forward

⚠️ This change resets after reboot.
To make it persistent:

echo “net.ipv4.ip_forward = 1” | sudo tee -a /etc/sysctl.conf
sudo sysctl -p

⚙️ Example 6 – Process & Resource Limits (ulimit)

ulimit -a

To set a new limit temporarily:

ulimit -n 4096

Common limits you can control:

Limit Meaning Example -n Max open files 1024 → 4096 -u Max processes per user 4096 -t Max CPU time (sec) 600 -v Virtual memory 1 GB

Persistent limit location:

/etc/security/limits.conf

⚙️ Example 7 – Time Synchronization with chrony

Check if chrony is active:

sudo systemctl status chronyd 2>/dev/null || sudo systemctl status chrony

List time sources:

chronyc sources 2>/dev/null || timedatectl show-timesync

Manual sync:

sudo chronyc makestep

If chrony isn’t installed:

sudo apt install chrony -y 2>/dev/null || sudo yum install chrony -y 2>/dev/null

⚙️ Example 8 – Analyze System Boot Logs

journalctl -b | grep -E “boot|kernel” | head -20

Check last boot time:

systemd-analyze

⚙️ Example 9 – System Resource Monitoring (Safe Combo Command)

echo “=== Resource Overview ===”
uptime
free -h
df -h
vmstat 2 3

⚙️ Example 10 – Automate Health Check Script

cat > health_check.sh <<’EOF’
#!/bin/bash
echo “=== Advanced System Health Check ===”
date
echo
echo “--- Kernel Info ---”
uname -r
echo “--- Uptime ---”
uptime
echo “--- Memory ---”
free -h
echo “--- Disk Usage ---”
df -h | grep -v tmpfs
echo “--- Sysctl Key (ip_forward) ---”
sysctl net.ipv4.ip_forward
echo “--- Open Files Limit ---”
ulimit -n
echo “--- Time Sync Status ---”
timedatectl | grep “System clock synchronized”
EOF
chmod +x health_check.sh
./health_check.sh

🧩 10 Hands-On Exercises

1. Run uname -a to view your kernel version.

2. Use lsmod and modinfo to inspect kernel modules.

3. View GRUB settings in /etc/default/grub.

4. Run sysctl -a | grep net.ipv4 to list all IPv4 settings.

5. Enable net.ipv4.ip_forward temporarily using sysctl -w.

6. Display all process limits with ulimit -a.

7. Increase the file descriptor limit (ulimit -n 4096) and verify.

8. Check your system time sync with chronyc sources.

9. Run journalctl -b | grep kernel | head -10 to see boot messages.

10. Run your health_check.sh script and review output.

🧠 10 Assessment Questions

1. What is the role of the Linux kernel?

2. What file controls GRUB boot settings?

3. How can you view current kernel parameters?

4. What does sysctl -w net.ipv4.ip_forward=1 do?

5. What is the difference between temporary and persistent sysctl changes?

6. What command shows all user resource limits?

7. Where can you define permanent process limits for a user?

8. What service keeps the Linux clock synchronized?

9. What command shows all loaded kernel modules?

10. Why is it risky to modify GRUB directly on production systems?

✅ End of Chapter 17 Summary
You learned:

• How to inspect and manage kernel parameters safely.

• How GRUB controls system boot behavior.

• How to use sysctl and ulimit for tuning.

• How to maintain accurate time sync with chrony.

• How to automate full system health checks.

Chapter 18 – Real-Time Administration Use Cases

🎯 Week 18 Objective

Practise real-world Linux administration operations such as server patching, log rotation, automated cleanup, user lifecycle management, backup validation, and root-cause analysis — all through hands-on, non-destructive examples under /tmp/linux_course/week18.

🧠 1 – Layman’s Explanation — “The Maintenance Analogy”

Think of your server as a factory 🏭

• Patching = updating the factory’s machines.

• User lifecycle = managing employee badges.

• Log cleanup = removing old security footage.

• Backups = daily safety copies of factory plans.

• Root-cause analysis (RCA) = investigating why a machine broke down.

🧩 2 – Common Real-Time Admin Tasks

Category Task Example System Maintenance Patch OS and packages sudo apt update && sudo apt upgrade -y User Management Create/disable accounts useradd, usermod -L Storage Cleanup Rotate and remove old logs logrotate, find *.log -mtime +30 -delete Backup Validation Verify latest archives tar -tf backup.tar.gz Troubleshooting Analyze downtime and logs journalctl -p err -b

💻 Safe Practice Setup

mkdir -p /tmp/linux_course/week18
cd /tmp/linux_course/week18

⚙️ Example 1 – Server Patching Simulation

echo “=== Check for updates ===”
sudo apt update 2>/dev/null || sudo yum check-update
echo “=== Apply available updates ===”
sudo apt upgrade -y 2>/dev/null || sudo yum update -y

💡 Always snapshot VMs or test servers before patching production.

⚙️ Example 2 – User Lifecycle Management

sudo useradd maria -m -c “QA Engineer”
sudo passwd maria
id maria
sudo usermod -L maria # lock user
sudo userdel -r maria # delete user and home folder

⚙️ Example 3 – Automated Log Cleanup

mkdir logs
touch logs/app_{1..5}.log
sleep 1
find logs -name “*.log” -type f -mtime +30 -delete # (simulate cleanup)

Schedule with cron to run daily.

⚙️ Example 4 – Log Rotation Verification

sudo logrotate -d /etc/logrotate.conf | head -10

💡 Use -d for dry-run mode to avoid actual rotation during tests.

⚙️ Example 5 – Backup Validation Script

cat > validate_backup.sh <<’EOF’
#!/bin/bash
latest=$(ls -t /tmp/linux_course/week12/backups/*.tar* 2>/dev/null | head -n 1)
if [ -z “$latest” ]; then
  echo “No backups found.”
  exit 1
fi
echo “Validating $latest”
tar -tf “$latest” > /dev/null && echo “✅ Backup valid” || echo “❌ Corrupt backup”
EOF
chmod +x validate_backup.sh
./validate_backup.sh

⚙️ Example 6 – System Health and Service Audit

systemctl --failed
sudo journalctl -p err -b | tail -10

⚙️ Example 7 – Resource Cleanup Automation

cat > cleanup_temp.sh <<’EOF’
#!/bin/bash
find /tmp -type f -mtime +7 -delete
echo “Old temporary files cleaned.”
EOF
chmod +x cleanup_temp.sh
./cleanup_temp.sh

Schedule weekly via cron.

⚙️ Example 8 – Patching Automation (Example Script)

cat > auto_patch.sh <<’EOF’
#!/bin/bash
echo “=== Automated Patch Started $(date) ===”
sudo apt update -y 2>/dev/null || sudo yum update -y
sudo apt upgrade -y 2>/dev/null || sudo yum upgrade -y
echo “=== Patch Completed $(date) ===”
EOF
chmod +x auto_patch.sh
./auto_patch.sh

⚙️ Example 9 – Simulated Downtime RCA (Log Analysis)

sudo journalctl -p err --since “1 hour ago” | grep -E “fail|error|critical” | head -10

Helps identify root cause of failures (e.g., service crashes, disk errors).

⚙️ Example 10 – Comprehensive System Maintenance Script

cat > maintenance.sh <<’EOF’
#!/bin/bash
echo “=== System Maintenance Report ===”
date
echo “[1] Disk Usage:”; df -h | grep -v tmpfs
echo “[2] CPU Load:”; uptime
echo “[3] Failed Services:”; systemctl --failed
echo “[4] Old Temp Files Cleanup...”; find /tmp -type f -mtime +7 -delete
echo “[5] Backup Validation:”; bash /tmp/linux_course/week18/validate_backup.sh
echo “=== Maintenance Complete ===”
EOF
chmod +x maintenance.sh
./maintenance.sh

🧩 10 Hands-On Exercises

1. Run sudo apt update && sudo apt upgrade in a test VM.

2. Create two new users and set passwords and sudo access.

3. Lock one account and verify login denied.

4. Create dummy logs and remove older ones using find -delete.

5. List configured logrotate policies under /etc/logrotate.d/.

6. Create and verify a TAR backup using the validation script.

7. Run systemctl --failed and note any non-active units.

8. Schedule cleanup_temp.sh to run daily at 3 AM via cron.

9. Check for error logs in the last hour using journalctl.

10. Execute the full maintenance.sh and analyze its output.

🧠 10 Assessment Questions

1. Why is it important to patch Linux systems regularly?

2. Which commands are used to add, lock, and remove users?

3. How does logrotate help in log management?

4. How do you automate log cleanup using find and cron?

5. What does the backup validation step ensure?

6. Which command shows failed systemd services?

7. What is the purpose of RCA in system administration?

8. What is the difference between manual and automated patching?

9. Why should cleanup scripts be scheduled off-peak hours?

10. How does a comprehensive maintenance script benefit admins?

✅ End of Chapter 18 Summary
You now know how to perform real-world admin duties:

• Apply patches and security updates.

• Manage user lifecycle.

• Rotate and clean logs automatically.

• Validate backups and audit system health.

• Analyze and troubleshoot production issues efficiently.

🎯 Course Description

In the world of Artificial Intelligence and Machine Learning, data is everything and PostgreSQL is one of the most powerful tools to manage, clean, and prepare that data efficiently.

This course is designed for Python learners, data enthusiasts, and aspiring AI/ML professionals who want to build a strong SQL foundation using PostgreSQL, the most advanced open-source database in the world.

We start from absolute scratch understanding databases, tables, and queries and gradually build up to advanced analytical queries, window functions, feature engineering, and performance tuning.

Each chapter includes:

• 💻 Real-world, copy-paste runnable examples

• 🧠 Step-by-step explanations in plain English

• 📊 Mini projects linked to AI and data analytics

• 🧩 Exercises to build hands-on confidence

By the end, you’ll not only master SQL for data extraction and transformation you’ll learn how to bridge SQL with Python for real-time analytics, model-ready datasets, and production-grade pipelines.

🚀 Who This Course Is For

✅ Python learners stepping into AI, Data Science, or ML.
✅ Developers who want to master SQL from zero using practical examples.
✅ Data Analysts who want to transition into AI roles.
✅ AI/ML Engineers who want to design efficient feature engineering pipelines.
✅ Anyone preparing for AI/Data interviews requiring strong SQL query writing.

📚 What You’ll Learn

You’ll go from:

“I know a bit of SELECT…”
to
“I can write complex queries that feed ML models!”

Here’s a glimpse of your journey:

📍 Foundations

• Understanding databases, tables, and data types

• Writing queries with SELECT, WHERE, and ORDER BY

• Using conditions, filters, and aliases

📍 Intermediate SQL

• Aggregations (COUNT, SUM, AVG) and GROUP BY

• All types of Joins and Subqueries

• Common Table Expressions (CTEs)

📍 Advanced SQL

• Window Functions for time-series analytics

• Case statements, ranking, and rolling metrics

• Data cleaning using string and date functions

• JSON and semi-structured data handling

📍 Real-World AI Applications

• SQL for feature engineering

• Handling missing data, anomalies, and outliers

• SQL + Python integration (Pandas, SQLAlchemy)

• Building small ETL pipelines with PostgreSQL

🌟 Key Benefits

1️⃣ Real-World Ready

Every topic connects directly to how data scientists and AI engineers use SQL in real jobs — preparing, cleaning, and transforming data for analysis and model training.

2️⃣ Python + SQL Integration

You’ll learn to connect PostgreSQL with Python, pull datasets using pandas, and preprocess them for your AI projects or dashboards.

3️⃣ Practice Without Frustration

All examples are fully self-contained — if you copy-paste them, they just work. No missing data, no setup errors.

4️⃣ Visual Learning Approach

Each concept includes tables, outputs, and real data. You’ll understand why a query works, not just how.

5️⃣ AI-Focused SQL Mastery

This isn’t generic database training. The entire syllabus is tailored to AI/ML use cases — feature creation, time series, window analytics, and ETL workflows.

6️⃣ Practical Assignments

Each chapter ends with 10 hands-on exercises that can be solved directly in PostgreSQL — designed to reinforce learning.

7️⃣ Career-Oriented Learning Path

You’ll build the SQL confidence needed to answer data engineering, data analyst, and AI interview questions with clarity.

🧠 By the End of This Course, You’ll Be Able To:

✅ Design and query complex PostgreSQL databases.
✅ Clean, join, and transform raw data for AI/ML workflows.
✅ Write advanced SQL queries with subqueries, CTEs, and window functions.
✅ Handle JSON, arrays, and time-series data inside PostgreSQL.
✅ Integrate PostgreSQL queries directly with Python notebooks and pipelines.
✅ Debug and optimize queries for faster performance.
✅ Build complete, model-ready datasets from raw data.

🕒 Duration

6 Weeks (Self-paced, Practice-based)
Each week includes:

• 1 Chapter (detailed theory + 5 runnable examples + 10 exercises)

• 1 Practice project at the end of the module

🧩 Course Structure

💼 Career Impact

After completing this course, you’ll confidently say:

“I can use SQL to build datasets for AI models and analytics — from raw data to insights.”

You’ll be ready for roles like:

• Data Analyst (Python + SQL)

• Data Engineer (PostgreSQL + ETL)

• ML Engineer (Feature Engineering with SQL)

• AI Developer (Data Prep & Integration)

🏆 Certificate of Completion

Upon completing all chapters and final project, you’ll receive a CareerByteCode Certificate showcasing your practical PostgreSQL expertise for AI and Data roles.

🚀 PostgreSQL Roadmap for AI / Data / ML Jobs

🔹Chapter 1. PostgreSQL Basics (Foundations)

Goal: Build a solid foundation in PostgreSQL syntax and structure.

• What is PostgreSQL? (Architecture, Role in AI systems)

• Understanding Databases, Schemas, Tables, Columns, Records

• Data Types: INTEGER, NUMERIC, TEXT, BOOLEAN, DATE, TIMESTAMP

• DDL, DML, DQL, DCL, TCL overview

• Creating and managing databases (CREATE DATABASE, DROP DATABASE)

• Creating tables with constraints (PRIMARY KEY, FOREIGN KEY, UNIQUE, NOT NULL)

• SELECT, FROM, WHERE basics

• Filtering data (=, <, >, <>, BETWEEN, IN, LIKE, ILIKE)

• Sorting with ORDER BY

• Limiting with LIMIT, OFFSET, FETCH

• Aliases with AS

👉 AI Relevance: Core foundation for data extraction and manipulation before model training.

🔹Chapter 2. Aggregations & Grouping

Goal: Summarize and explore datasets effectively.

• COUNT(), SUM(), AVG(), MIN(), MAX()

• GROUP BY with single/multiple columns

• HAVING vs WHERE

• Removing duplicates with DISTINCT

• Statistical queries for EDA (Exploratory Data Analysis)

👉 AI Relevance: Computing aggregates for feature engineering and EDA (mean, count, frequency features).

🔹Chapter 3. Joins (Essential for Data Engineering & AI)

Goal: Combine multiple datasets efficiently.

• INNER JOIN, LEFT JOIN, RIGHT JOIN, FULL OUTER JOIN

• CROSS JOIN and Cartesian products

• SELF JOIN (hierarchical and recursive relationships)

• Multi-table joins

• Natural joins vs explicit joins

👉 AI Relevance: Combine multiple sources for dataset creation, feature enrichment, or entity linking.

🔹Chapter 4. Subqueries & CTEs (Common Table Expressions)

Goal: Create modular, readable, and optimized queries.

• Subqueries in SELECT, FROM, and WHERE

• Correlated vs non-correlated subqueries

• EXISTS vs IN performance

• Using WITH for Common Table Expressions (CTEs)

• Recursive CTEs for hierarchical data

👉 AI Relevance: Break down complex queries for ETL pipelines and data preparation workflows.

🔹Chapter 5. Advanced Filtering & Window Functions

Goal: Learn analytics-level querying (critical for AI feature creation).

• Conditional logic using CASE WHEN

• Handling NULLs: COALESCE, NULLIF, IS NULL

• Ranking functions: ROW_NUMBER(), RANK(), DENSE_RANK()

• Window aggregates: SUM() OVER(), AVG() OVER()

• Frame clauses: ROWS BETWEEN, RANGE BETWEEN

• Lead/Lag functions for time-series (LAG(), LEAD())

• Running totals, moving averages, percentiles

👉 AI Relevance: Perfect for time series modeling, rolling statistics, and trend analysis.

🔹 Chapter 6. Data Manipulation

Goal: Manage data lifecycle inside PostgreSQL.

• INSERT, UPDATE, DELETE

• RETURNING clause (unique to PostgreSQL)

• Upsert operations using INSERT ... ON CONFLICT DO UPDATE

• Bulk inserts with COPY command

• Transactions with BEGIN, COMMIT, ROLLBACK

👉 AI Relevance: Managing data updates efficiently during ETL or experiment iterations.

🔹 Chapter 7. Set Operations

Goal: Combine or compare datasets for complex workflows.

• UNION, UNION ALL, INTERSECT, EXCEPT

• Combining datasets for training/validation splits

• Deduplication with set operations

👉 AI Relevance: Useful for merging multiple datasets, or comparing model prediction results vs actuals.

🔹Chapter 8. Data Cleaning & Transformation (ETL Core)

Goal: Prepare clean, well-structured datasets for ML.

• String operations: TRIM, SUBSTRING, CONCAT, UPPER, LOWER, REPLACE

• Date & time functions: AGE, DATE_PART, EXTRACT, TO_CHAR, NOW()

• Type conversion: CAST, ::type syntax

• Pivoting data using crosstab() (from tablefunc extension)

• Unpivoting / reshaping datasets

• Conditional transformations using CASE

👉 AI Relevance: Critical preprocessing step before model training — clean, normalized, formatted data.

🔹 Chapter 9. JSON & Semi-Structured Data (PostgreSQL Power Feature)

Goal: Handle real-world mixed-format data efficiently.

• JSON & JSONB columns

• Accessing data: ->, ->>, #> operators

• JSON functions: json_each(), jsonb_array_elements(), jsonb_build_object()

• Querying nested structures

• Storing AI/ML model metadata or configurations

👉 AI Relevance: Modern datasets often include API responses or event logs stored as JSON.

🔹Chapter 10. Array & Advanced Data Types

Goal: Work with complex and scientific data structures.

• Array creation and indexing

• unnest() for array flattening

• Range types (int4range, numrange, daterange)

• ENUMs, UUIDs, HSTORE key-value pairs

👉 AI Relevance: Helpful for storing multiple model parameters or embedding vectors.

🔹Chapter 11. Performance & Optimization

Goal: Run large queries efficiently in data pipelines.

• Index types: B-tree, Hash, GIN, GiST

• Creating and analyzing indexes

• Query plans with EXPLAIN and EXPLAIN ANALYZE

• Understanding Sequential vs Index Scan

• Partitioning large tables

• Materialized views

• VACUUM and ANALYZE commands

👉 AI Relevance: Optimized queries mean faster ETL, real-time dashboards, and efficient training loops.

🔹Chapter 12. AI / Data-Specific Use Cases

Goal: Connect SQL knowledge directly to ML & AI workflows.

• Feature engineering queries (aggregates, ratios, time-based deltas)

• Time-series transformations (lags, moving windows)

• Anomaly detection with SQL-based thresholds

• Missing value detection (COUNT(NULL), COALESCE)

• Deduplication and data consistency checks

• Statistical summaries with SQL

• Creating feature stores in PostgreSQL

🔹Chapter 13. PostgreSQL with Python (Integration)

Goal: Combine PostgreSQL and Python seamlessly for end-to-end data work.

• Connecting using psycopg2 or SQLAlchemy

• Reading queries into Pandas DataFrames (pd.read_sql())

• Writing model outputs back to PostgreSQL

• Using COPY for fast bulk inserts from Python

• Building ETL pipelines (Airflow + Postgres + Pandas)

• Query automation via Jupyter or scripts

👉 AI Relevance: The bridge between SQL and ML model development.

🔹Chapter 14. Data Warehousing & Analytical Extensions

Goal: Handle enterprise-scale analytics.

• Understanding extensions like TimescaleDB (for time series)

• CUBE, ROLLUP, and GROUPING SETS

• PostgreSQL partitioning for analytical datasets

• Integration with BI tools (Power BI, Tableau, Metabase)

• Using FDW (Foreign Data Wrapper) for external data access

👉 AI Relevance: Scaling SQL analysis for production AI data pipelines.

✅ Summary: What You Must Master for AI Jobs

🔗 Recommended Tools for Practice

• PostgreSQL (install locally or use Docker)

• pgAdmin / DBeaver (GUI tools)

• Jupyter Notebook + SQLAlchemy (Python integration)

• Kaggle Datasets + PostgreSQL (real-world AI practice)

Chapter 1 — PostgreSQL Basics (Foundations)

What you’ll learn in this chapter (in one sentence)

You’ll learn what PostgreSQL is, the basic building blocks of a database (databases → schemas → tables → rows/columns), how to write simple SELECT queries, filter results, sort them, limit rows, and use column aliases — all explained in plain language.

Layman explanations (super simple)

• What is PostgreSQL?
  PostgreSQL (Postgres) is a program that stores data for you (like a super-organized filing cabinet). You ask it for data using SQL (a language); it gives the data back.

• Database vs Schema vs Table vs Row vs Column

  • Database: A whole filing cabinet.

  • Schema: A drawer inside the cabinet (helps organize).

  • Table: A file folder in the drawer (holds similar records).

  • Row: One sheet of paper in the folder (a single item/record).

  • Column: A labeled line on the sheet (a specific attribute, like “name” or “age”).

• Primary Key
  A column that uniquely identifies each row — like a unique ID number on each sheet.

• Foreign Key
  A column that points to a primary key in another table — like a reference to another file.

• Data types
  Tells Postgres what kind of info a column holds: INTEGER (numbers), TEXT (words), BOOLEAN (true/false), TIMESTAMP (date & time), etc.

• DDL / DML / DQL

  • DDL (Data Definition Language): Commands that create or change the structure (e.g., CREATE TABLE).

  • DML (Data Manipulation Language): Commands that modify data (e.g., INSERT, UPDATE, DELETE).

  • DQL (Data Query Language): Commands that read data (mostly SELECT).

• SELECT, FROM, WHERE

  • SELECT lists which columns you want.

  • FROM says which table.

  • WHERE filters rows (gives conditions).

• Filtering operators
  Use = (equal), <, >, <> (not equal), BETWEEN, IN (in a set), LIKE (pattern matching).

• ORDER BY
  Sort the results (e.g., newest first or lowest price first).

• LIMIT / OFFSET
  Only take the first N rows (LIMIT), or skip some and then take (OFFSET).

• Aliases (AS)
  Rename columns or tables within your query to make output cleaner or queries shorter.

Practical Examples (each is copy-paste runnable)

Note: Each example starts with DROP TABLE IF EXISTS so you can re-run freely.

Example 1 — Basic SELECT and creating a table

Scenario: A tiny people table. Show selected columns.

-- Example 1: basic SELECT
DROP TABLE IF EXISTS ex1_people;

CREATE TABLE ex1_people (
  person_id SERIAL PRIMARY KEY,
  full_name TEXT NOT NULL,
  age INTEGER,
  city TEXT
);

INSERT INTO ex1_people (full_name, age, city) VALUES
(’Anita Sharma’, 28, ‘Brussels’),
(’Mark Johnson’, 35, ‘London’),
(’Sana Ali’, 22, ‘Amsterdam’),
(’Tom Lee’, NULL, ‘Berlin’);  -- Tom’s age unknown

-- 1) select all columns
SELECT * FROM ex1_people;

-- 2) select specific columns
SELECT full_name, city FROM ex1_people;

-- 3) rename a column in output
SELECT full_name AS name, age AS years FROM ex1_people;

What you learned: how to create a table, insert rows, and fetch all or specific columns. NULL means unknown.

Example 2 — Filtering with WHERE and operators

Scenario: Products table — use =, <, >, <>, BETWEEN, IN, LIKE, ILIKE.

-- Example 2: filtering
DROP TABLE IF EXISTS ex2_products;
CREATE TABLE ex2_products (
  product_id SERIAL PRIMARY KEY,
  name TEXT NOT NULL,
  price NUMERIC(8,2),
  category TEXT
);

INSERT INTO ex2_products (name, price, category) VALUES
(’Coffee Beans’, 12.50, ‘Beverage’),
(’Green Tea’, 8.00, ‘Beverage’),
(’Notebook A4’, 3.25, ‘Stationery’),
(’Premium Notebook’, 7.50, ‘Stationery’),
(’Espresso Machine’, 120.00, ‘Appliance’);

-- Equal
SELECT * FROM ex2_products WHERE category = ‘Beverage’;

-- Not equal
SELECT * FROM ex2_products WHERE price <> 12.50;

-- Greater than / Less than
SELECT name, price FROM ex2_products WHERE price > 10.00;

-- BETWEEN
SELECT name, price FROM ex2_products WHERE price BETWEEN 5 AND 50;

-- IN (matches any of a list)
SELECT * FROM ex2_products WHERE category IN (’Beverage’, ‘Appliance’);

-- LIKE (pattern; % means any chars)
SELECT * FROM ex2_products WHERE name LIKE ‘%Notebook%’;

-- ILIKE (case-insensitive)
SELECT * FROM ex2_products WHERE name LIKE ‘coffee%’;

What you learned: common filters and pattern matching (case sensitive LIKE, case-insensitive ILIKE).

Example 3 — ORDER BY, LIMIT, OFFSET

Scenario: Students leaderboard — sort by score and paginate.

-- Example 3: order, limit, offset
DROP TABLE IF EXISTS ex3_students;
CREATE TABLE ex3_students (
  student_id SERIAL PRIMARY KEY,
  name TEXT,
  score INTEGER,
  taken_at TIMESTAMP
);

INSERT INTO ex3_students (name, score, taken_at) VALUES
(’Aisha’, 92, ‘2025-01-05 10:00’),
(’Ben’, 85, ‘2025-01-06 11:30’),
(’Carla’, 78, ‘2025-01-04 09:20’),
(’Derek’, 92, ‘2025-01-07 14:00’),
(’Eve’, 68, ‘2025-01-03 08:00’);

-- Sort by highest score first, newest attempt first
SELECT name, score, taken_at
FROM ex3_students
ORDER BY score DESC, taken_at DESC;

-- Only top 3
SELECT name, score FROM ex3_students ORDER BY score DESC LIMIT 3;

-- Pagination: skip first 2, show next 2
SELECT name, score FROM ex3_students ORDER BY score DESC LIMIT 2 OFFSET 2;

What you learned: ordering, limiting results and simple pagination with LIMIT and OFFSET.

Example 4 — Aliases, expressions, and casting

Scenario: Employee salaries; show monthly pay and format.

-- Example 4: aliases, calculation, type casting
DROP TABLE IF EXISTS ex4_staff;
CREATE TABLE ex4_staff (
  staff_id SERIAL PRIMARY KEY,
  name TEXT,
  annual_salary NUMERIC(12,2),
  join_date DATE
);

INSERT INTO ex4_staff (name, annual_salary, join_date) VALUES
(’Ramesh’, 60000.00, ‘2020-06-15’),
(’Julia’, 48000.50, ‘2021-09-01’),
(’Liam’, 72000.00, ‘2019-03-10’);

-- Calculate monthly salary (simple division), show with alias
SELECT name,
       annual_salary,
       (annual_salary / 12) AS monthly_salary
FROM ex4_staff;

-- Round monthly salary to 2 decimals and show as text label
SELECT name,
       annual_salary,
       ROUND(annual_salary / 12, 2) AS monthly_salary_rounded,
       (ROUND(annual_salary / 12, 2))::TEXT || ‘ EUR/month’ AS pay_label
FROM ex4_staff;

What you learned: you can do math in SELECT, assign a friendly column name with AS, and cast results to text for readable labels.

Example 5 — DISTINCT and basic uniqueness check

Scenario: Orders with cities — find unique cities, and count distinct.

-- Example 5: DISTINCT
DROP TABLE IF EXISTS ex5_orders;
CREATE TABLE ex5_orders (
  order_id SERIAL PRIMARY KEY,
  customer TEXT,
  city TEXT,
  amount NUMERIC(8,2)
);

INSERT INTO ex5_orders (customer, city, amount) VALUES
(’Anna’, ‘Brussels’, 34.50),
(’Ben’, ‘Brussels’, 15.00),
(’Carmen’, ‘Antwerp’, 44.00),
(’David’, ‘Ghent’, 21.25),
(’Ella’, ‘Antwerp’, 99.99);

-- Get unique cities
SELECT DISTINCT city FROM ex5_orders;

-- Count unique cities
SELECT COUNT(DISTINCT city) AS unique_city_count FROM ex5_orders;

-- Distinct combinations (customer+city)
SELECT DISTINCT customer, city FROM ex5_orders;

What you learned: DISTINCT removes duplicates from the result set.

Example 6 — Basic table constraints (NOT NULL, UNIQUE, PRIMARY KEY)

Scenario: Simple contacts table demonstrating constraints and safe inserts.

-- Example 6: constraints
DROP TABLE IF EXISTS ex6_contacts;
CREATE TABLE ex6_contacts (
  contact_id SERIAL PRIMARY KEY,
  email TEXT UNIQUE NOT NULL,
  name TEXT NOT NULL,
  phone TEXT
);

-- valid insert
INSERT INTO ex6_contacts (email, name, phone) VALUES
(’alice@example.com’,’Alice’,’+32-2-555-0100’),
(’bob@example.com’,’Bob’,NULL);

-- Trying to insert NULL into email or duplicate email would fail:
-- (these lines are commented out so script runs; try them manually)
-- INSERT INTO ex6_contacts (email, name) VALUES (NULL, ‘Bad’); -- error due to NOT NULL
-- INSERT INTO ex6_contacts (email, name) VALUES (’alice@example.com’,’Duplicate’); -- UNIQUE violation

SELECT * FROM ex6_contacts;

What you learned: constraints enforce rules — NOT NULL, UNIQUE, PRIMARY KEY keep data valid. I commented the failing inserts so the script runs cleanly; try them yourself to see the errors.

Quick troubleshooting tips

• If you see relation “ex1_people” does not exist, make sure you executed the CREATE TABLE block first.

• If an INSERT fails with duplicate key value violates unique constraint, it means you tried to insert a value that must be unique (like an email or primary key).

• Use \d tablename in psql to inspect table structure.

10 Exercises (practice tasks for Chapter 1)

Try these on your own. For each exercise write SQL to solve it.

1. Make a books table with columns book_id (PK), title, author, price. Insert 6 books, then select all books cheaper than 20.

2. Create a visitors table with visitor_id, name, visit_date. Insert 5 rows. Select rows where visit_date is after ‘2025-01-01’.

3. Filter by multiple conditions: Using a table ex2_products-style, select products where category = ‘Stationery’ and price < 5.

4. Case-insensitive search: Create a users table with usernames in mixed case. Use a query to find username starting with ‘jo’ regardless of case.

5. Top-N: Create a scores table, insert 8 rows and return top 3 scorers.

6. Aliases and calculations: Make a sales table with unit_price and quantity. Show total = unit_price * quantity with alias order_total.

7. Distinct usage: Create an events table with event names and cities. List all distinct cities.

8. Constraint test: Create a members table where email must be unique and name not null. Try inserting duplicates to see the error. (Explain what happened.)

9. Limit + Offset: Create a messages table and return rows 6–10 ordered by timestamp (simulate pagination).

10. Explain why NULL matters: Create a table with a column allowed to be NULL. Insert a row with NULL and another without. Show how WHERE column = ‘something’ ignores the NULL row, but WHERE column IS NULL picks it up.

Chapter 2 — Aggregations & Grouping

(Plain-English explanations + very detailed examples you can copy-paste + 10 hands-on exercises)

Bavi — this chapter teaches how to summarize and roll up data: counts, sums, averages, min/max, and how to group rows into buckets (by product, by day, by city) so you can get the high-level numbers data scientists need for EDA and feature engineering.

What you’ll learn in plain English

• Aggregate functions are small math tools Postgres gives you: COUNT, SUM, AVG, MIN, MAX. They take many rows and return one number (e.g., total sales).

• GROUP BY collects rows into buckets (e.g., one bucket per product) so you can run aggregates per bucket.

• HAVING filters buckets after aggregation (e.g., show only products whose total sales exceed €500).

• COUNT(*) vs COUNT(column) — COUNT(*) counts rows; COUNT(column) only counts non-NULL values.

• COUNT(DISTINCT ...) counts unique values inside each group.

• FILTER (Postgres feature) lets you run conditional aggregates in the same query (e.g., count purchases vs refunds).

• DATE_TRUNC is handy to group timestamps by month/week/day. Very useful for time-series features.

All examples below are fully self-contained (they DROP then CREATE tables and INSERT data). Copy-paste any single example into your psql, pgAdmin, DBeaver, or other Postgres client and it will run.

Example 1 — Simple aggregates: COUNT, SUM, AVG, MIN, MAX

Goal: See the basic aggregate functions in action on a small transactions table.

-- EX1: Basic aggregates
DROP TABLE IF EXISTS ex1_transactions;
CREATE TABLE ex1_transactions (
  txn_id   SERIAL PRIMARY KEY,
  user_id  INTEGER,
  amount   NUMERIC(10,2),
  txn_time TIMESTAMP
);

INSERT INTO ex1_transactions (user_id, amount, txn_time) VALUES
(1, 12.50, ‘2025-01-01 09:10’),
(2, 7.00,  ‘2025-01-01 10:00’),
(1, 5.25,  ‘2025-01-02 11:15’),
(3, NULL,  ‘2025-01-03 12:00’), -- failed / missing amount
(2, 20.00, ‘2025-01-04 13:00’),
(4, 100.00,’2025-01-05 14:30’);

-- a) total number of transactions (rows)
SELECT COUNT(*) AS total_transactions FROM ex1_transactions;

-- b) total amount of all transactions (NULL values ignored by SUM)
SELECT SUM(amount) AS total_amount FROM ex1_transactions;

-- c) average amount per transaction (NULLs ignored)
SELECT AVG(amount) AS avg_amount FROM ex1_transactions;

-- d) smallest and largest transaction amounts
SELECT MIN(amount) AS smallest, MAX(amount) AS largest FROM ex1_transactions;

-- e) combine them in one row for an overview
SELECT
  COUNT(*) AS total_rows,
  COUNT(amount) AS rows_with_amount,
  SUM(amount) AS sum_amount,
  AVG(amount) AS avg_amount,
  MIN(amount) AS min_amount,
  MAX(amount) AS max_amount
FROM ex1_transactions;

Why this matters (layman): COUNT tells you how many rows; SUM adds numbers; AVG finds the mean; MIN and MAX show extremes. Notice SUM and AVG ignore NULL amounts — that prevents broken math when some rows are missing data.

Example 2 — GROUP BY a single column: totals per category

Goal: Group sales by category to see which categories sell the most.

-- EX2: Group by category (sales totals and counts)
DROP TABLE IF EXISTS ex2_category_sales;
CREATE TABLE ex2_category_sales (
  sale_id  SERIAL PRIMARY KEY,
  product  TEXT,
  category TEXT,
  qty      INTEGER,
  unit_price NUMERIC(8,2)
);

INSERT INTO ex2_category_sales (product, category, qty, unit_price) VALUES
(’Espresso Beans’,’Beverage’, 10, 12.50),
(’Green Tea’,’Beverage’, 5, 8.00),
(’Notebook A4’,’Stationery’, 50, 3.25),
(’Premium Notebook’,’Stationery’, 20, 7.50),
(’Espresso Machine’,’Appliance’, 2, 120.00),
(’Mug’,’Merch’, 100, 4.00),
(’Sticker Pack’,’Merch’, 200, 1.50);

-- Calculate total sales per category and number of items sold
SELECT
  category,
  SUM(qty * unit_price) AS total_revenue,
  SUM(qty) AS total_units_sold,
  COUNT(*) AS number_of_line_items
FROM ex2_category_sales
GROUP BY category
ORDER BY total_revenue DESC;

What you see: Each category becomes one row; inside that row we compute SUM(qty * unit_price) to get revenue for that category. GROUP BY is how we “fold” rows into buckets.

Example 3 — GROUP BY multiple columns and time bucketing (DATE_TRUNC)

Goal: For each city and each month, compute total revenue — extremely useful for time-series features.

-- EX3: Group by city and month (time bucketing)
DROP TABLE IF EXISTS ex3_orders;
CREATE TABLE ex3_orders (
  order_id   SERIAL PRIMARY KEY,
  customer   TEXT,
  city       TEXT,
  amount     NUMERIC(10,2),
  order_ts   TIMESTAMP
);

INSERT INTO ex3_orders (customer, city, amount, order_ts) VALUES
(’Anna’,’Brussels’, 34.50, ‘2025-01-05 10:00’),
(’Ben’,’Brussels’, 15.00, ‘2025-01-06 12:00’),
(’Carmen’,’Antwerp’, 44.00, ‘2025-01-20 09:00’),
(’David’,’Ghent’, 21.25, ‘2025-02-02 11:00’),
(’Ella’,’Antwerp’, 99.99, ‘2025-02-10 14:00’),
(’Fahad’,’Brussels’, 12.00, ‘2025-02-12 16:00’),
(’Gina’,’Brussels’, 60.00, ‘2025-03-01 10:30’);

-- Group by city and month
SELECT
  city,
  DATE_TRUNC(’month’, order_ts) AS month,
  COUNT(*) AS orders_count,
  SUM(amount) AS month_revenue,
  AVG(amount) AS avg_order_value
FROM ex3_orders
GROUP BY city, DATE_TRUNC(’month’, order_ts)
ORDER BY month, city;

Why this is handy: DATE_TRUNC(’month’, order_ts) converts a timestamp to the month (e.g., ‘2025-02-01 00:00:00’) so months become grouping keys. Grouping by more than one column (city + month) gives a grid of metrics you can turn into time-series features for ML.

Example 4 — HAVING vs WHERE: filter groups (correct way)

Goal: Show how to keep only buckets that meet a condition (like categories with total revenue > 100).

-- EX4: HAVING to filter groups
DROP TABLE IF EXISTS ex4_sales;
CREATE TABLE ex4_sales (
  id SERIAL PRIMARY KEY,
  category TEXT,
  amount   NUMERIC(8,2)
);

INSERT INTO ex4_sales (category, amount) VALUES
(’Beverage’, 12.50),
(’Beverage’, 8.00),
(’Stationery’, 3.25),
(’Stationery’, 7.50),
(’Appliance’, 120.00),
(’Merch’, 4.00),
(’Merch’, 1.50),
(’Merch’, 150.00);

-- WRONG: this would fail because WHERE cannot use aggregates (commented)
-- SELECT category, SUM(amount) FROM ex4_sales WHERE SUM(amount) > 100 GROUP BY category;

-- RIGHT: use HAVING to filter groups after aggregation
SELECT
  category,
  SUM(amount) AS total_revenue
FROM ex4_sales
GROUP BY category
HAVING SUM(amount) > 100
ORDER BY total_revenue DESC;

-- Alternative: using a subquery with WHERE (also valid)
SELECT category, total_revenue FROM (
  SELECT category, SUM(amount) AS total_revenue
  FROM ex4_sales
  GROUP BY category
) t
WHERE total_revenue > 100;

Layman explanation: WHERE filters individual rows before grouping. HAVING filters the grouped results after you compute SUM/AVG/etc. Use HAVING when your condition uses aggregated numbers.

Example 5 — COUNT(DISTINCT), FILTER clause, and conditional aggregates

Goal: Count unique users and compute purchase-only sums in the same grouped query.

-- EX5: DISTINCT counts and FILTER (Postgres)
DROP TABLE IF EXISTS ex5_user_events;
CREATE TABLE ex5_user_events (
  event_id   SERIAL PRIMARY KEY,
  user_id    INTEGER,
  event_type TEXT,     -- ‘view’, ‘add_to_cart’, ‘purchase’, ‘refund’
  amount     NUMERIC(10,2),
  event_ts   TIMESTAMP
);

INSERT INTO ex5_user_events (user_id, event_type, amount, event_ts) VALUES
(1, ‘view’, NULL,  ‘2025-03-01 09:00’),
(1, ‘add_to_cart’, NULL, ‘2025-03-01 09:05’),
(1, ‘purchase’, 20.00, ‘2025-03-01 09:06’),
(2, ‘view’, NULL, ‘2025-03-01 10:00’),
(2, ‘purchase’, 15.00, ‘2025-03-02 11:00’),
(3, ‘view’, NULL, ‘2025-03-02 12:00’),
(3, ‘purchase’, 7.50, ‘2025-03-02 12:05’),
(3, ‘refund’, -7.50, ‘2025-03-03 13:00’),
(4, ‘purchase’, 100.00, ‘2025-03-04 14:00’),
(4, ‘purchase’, 50.00, ‘2025-03-05 15:00’);

-- Aggregate example: per-day summary
SELECT
  DATE(event_ts) AS day,
  COUNT(*) AS total_events,
  COUNT(DISTINCT user_id) AS unique_users,
  COUNT(*) FILTER (WHERE event_type = ‘purchase’) AS purchase_events,
  SUM(amount) FILTER (WHERE event_type = ‘purchase’) AS purchase_amount,
  SUM(amount) FILTER (WHERE event_type = ‘refund’) AS refund_amount
FROM ex5_user_events
GROUP BY DATE(event_ts)
ORDER BY day;

Plain take-away: COUNT(DISTINCT user_id) gives number of unique users. FILTER (WHERE ...) is a clean Postgres way to say “run this aggregate but only for rows that match this condition” — very useful to compute multiple conditional counts/sums in a single query.

Example 6 — NULLs, COUNT(*) vs COUNT(col), and grouping NULLs with COALESCE

Goal: Understand how NULL affects aggregates and how to group NULL values cleanly.

-- EX6: NULL handling in aggregates
DROP TABLE IF EXISTS ex6_feedback;
CREATE TABLE ex6_feedback (
  fb_id    SERIAL PRIMARY KEY,
  product_id INTEGER,  -- can be NULL (feedback not tied to product)
  rating   INTEGER,    -- 1..5 or NULL if not rated
  comments TEXT
);

INSERT INTO ex6_feedback (product_id, rating, comments) VALUES
(1, 5, ‘Great’),
(1, 4, ‘Good’),
(NULL, 3, ‘General feedback’),
(2, NULL, ‘No rating’),
(2, 2, ‘Poor’),
(NULL, NULL, ‘Anonymous note’);

-- Counts:
SELECT
  COUNT(*) AS total_rows,
  COUNT(product_id) AS rows_with_product_id,
  COUNT(rating) AS rows_with_rating,
  COUNT(DISTINCT product_id) AS distinct_products_referenced
FROM ex6_feedback;

-- Group by product_id will put NULLs in a distinct bucket.
-- Use COALESCE to label NULL product_id as ‘unknown’ when grouping.
SELECT
  COALESCE(product_id::TEXT, ‘unknown’) AS product_label,
  COUNT(*) AS feedback_count,
  AVG(rating) AS avg_rating
FROM ex6_feedback
GROUP BY COALESCE(product_id::TEXT, ‘unknown’);

Key point for beginners: COUNT(*) counts every row. COUNT(product_id) ignores rows where product_id is NULL. When grouping, NULL is a real bucket — use COALESCE to make it human-friendly (e.g., ‘unknown’).

Quick practical tips (layman)

• If you want totals per day/week/month: use DATE_TRUNC(’day’|’week’|’month’, timestamp_col) in GROUP BY.

• Use ORDER BY on aggregated columns (e.g., ORDER BY SUM(amount) DESC) to find top buckets.

• To filter groups by their aggregate value use HAVING.

• Use COUNT(DISTINCT ...) to know how many unique users, products, etc.

• If you need multiple conditional aggregates (e.g., purchases vs refunds), use FILTER — it’s clearer than SUM(CASE WHEN ...). (Both work.)

10 Exercises — practice (all should be solved in PostgreSQL)

Try to write the SQL for each. If you want, I’ll provide the full runnable solution scripts afterwards.

1. Total revenue & average order value: Create an orders table (order_id, customer_id, amount, order_ts). Insert 12 rows across two months. Write a query that returns the total revenue and AVG order amount for the entire dataset.

2. Revenue per product: Create product_sales(product, category, qty, unit_price) (insert 20 rows across 4 categories). Return total revenue and units sold per category, sorted by revenue descending.

3. Monthly active users: Create events(user_id, event_type, event_ts) where event_type can be ‘view’ or ‘purchase’. Write a query that shows month (YYYY-MM) and number of unique users who made a purchase in that month.

4. Top 3 customers: Using an orders table (customer_id, amount), return the top 3 customers by total spending (customer_id and total_spent).

5. Filter groups with HAVING: Create a sales table and find categories whose total sales amount is at least 500. Use HAVING.

6. Conditional aggregates: Create events with types ‘purchase’ and ‘refund’. For each day, compute: total_events, purchase_count, refund_count, purchase_amount, refund_amount. Use FILTER or SUM(CASE WHEN ...).

7. Count vs Count(column): Build a responses table with a nullable rating and show COUNT(*) vs COUNT(rating) and explain the difference in one sentence (as a SQL comment).

8. Group by multiple columns: Using an orders table with city and order_ts, compute city, month, orders_count, total_amount grouped by city and month for the last 90 days.

9. Top-N per group (advanced): For sales(employee_id, region, sale_amount), find the highest sale per region (region + employee_id + sale_amount). (Hint: use ROW_NUMBER() over partition — this crosses into window functions but is very useful. If you prefer not to use window functions yet, use a subquery per region.)

10. Distinct counts in groups: Create a visits(user_id, page, visit_ts) table. For each page, compute how many unique users visited it and the total visits. Sort pages by unique users descending.

Chapter 3 — Joins (Very plain-English + runnable examples + 10 exercises)

Alright Bavi — here’s Chapter 3: Joins. I’ll explain every concept like you’re reading a friendly cheat-sheet, then provide at least 5 fully runnable examples (each example drops/creates its own tables and inserts data so you can copy/paste and run). Every example is followed by a short plain-language explanation. At the end are 10 exercises to practice.

What is a JOIN? (super simple)

A join lets you combine rows from two (or more) tables into a single row in the result — based on some matching rule.
Think of two lists: one is people, another is their phones. A join answers questions like “show me each person and their phone” or “show all people even if they don’t have a phone”.

Key idea: a join picks rows from Table A and rows from Table B and sticks them together when a matching condition is true (usually when an id in A equals an id in B).

Types of joins — in one line each

• INNER JOIN: Only rows that have matches in both tables. (Think: intersection.)

• LEFT (LEFT OUTER) JOIN: All rows from the left table; matching rows from right when available; otherwise NULLs for right columns. (Think: everything on the left, bring matching right info.)

• RIGHT (RIGHT OUTER) JOIN: All rows from the right table; matching left when available. (Less used but symmetric to LEFT.)

• FULL OUTER JOIN: All rows from both tables; where no match exists, fill with NULLs. (Union of left and right with matching combined.)

• CROSS JOIN: Cartesian product — every row from A paired with every row from B (be careful: can explode).

• SELF JOIN: A table joined to itself (useful for hierarchies).

• Multi-table joins: Chain joins across several tables to build richer rows.

Syntax basics

• SELECT ... FROM A INNER JOIN B ON A.key = B.key;

• LEFT JOIN and RIGHT JOIN work the same but preserve rows from one side.

• You can use USING(col) when both tables have the same column name for the join key — this avoids repeating ON A.col = B.col.

Example 1 — INNER JOIN (basic)

-- EX1: INNER JOIN basic
DROP TABLE IF EXISTS ex1_customers;
DROP TABLE IF EXISTS ex1_orders;

CREATE TABLE ex1_customers (
  customer_id SERIAL PRIMARY KEY,
  name TEXT,
  city TEXT
);

CREATE TABLE ex1_orders (
  order_id SERIAL PRIMARY KEY,
  customer_id INTEGER REFERENCES ex1_customers(customer_id),
  amount NUMERIC(8,2),
  order_ts TIMESTAMP
);

INSERT INTO ex1_customers (name, city) VALUES
(’Anita’, ‘Brussels’),
(’Ben’, ‘London’),
(’Carmen’, ‘Antwerp’);

INSERT INTO ex1_orders (customer_id, amount, order_ts) VALUES
(1, 34.50, ‘2025-01-05 10:00’),
(1, 15.00, ‘2025-01-06 12:00’),
(3, 44.00, ‘2025-01-20 09:00’);

-- Inner join: only customers who have orders
SELECT c.customer_id, c.name, o.order_id, o.amount, o.order_ts
FROM ex1_customers c
INNER JOIN ex1_orders o
  ON c.customer_id = o.customer_id
ORDER BY c.customer_id, o.order_id;

Plain take-away: Inner join shows only customers that have orders. Anita and Carmen appear because they have orders; Ben does not because he has none.

Example 2 — LEFT JOIN (include unmatched left rows)

-- EX2: LEFT JOIN shows all customers, even if no orders
DROP TABLE IF EXISTS ex2_customers;
DROP TABLE IF EXISTS ex2_orders;

CREATE TABLE ex2_customers (
  customer_id SERIAL PRIMARY KEY,
  name TEXT
);

CREATE TABLE ex2_orders (
  order_id SERIAL PRIMARY KEY,
  customer_id INTEGER,
  amount NUMERIC(8,2)
);

INSERT INTO ex2_customers (name) VALUES
(’Ali’),
(’Beth’),
(’Cheng’);

INSERT INTO ex2_orders (customer_id, amount) VALUES
(1, 10.00),
(1, 5.00),
(3, 100.00);

-- Left join: all customers; orders when available
SELECT c.customer_id, c.name, o.order_id, o.amount
FROM ex2_customers c
LEFT JOIN ex2_orders o
  ON c.customer_id = o.customer_id
ORDER BY c.customer_id;

Plain take-away: Beth (customer_id 2) will appear with NULLs for order fields — left join preserves all left-table rows even if no match on the right.

Example 3 — RIGHT JOIN and FULL OUTER JOIN

-- EX3: RIGHT JOIN and FULL OUTER JOIN
DROP TABLE IF EXISTS ex3_employees;
DROP TABLE IF EXISTS ex3_departments;

CREATE TABLE ex3_employees (
  emp_id SERIAL PRIMARY KEY,
  name TEXT,
  dept_id INTEGER
);

CREATE TABLE ex3_departments (
  dept_id SERIAL PRIMARY KEY,
  dept_name TEXT
);

INSERT INTO ex3_employees (name, dept_id) VALUES
(’Ravi’, 1),
(’Mia’, 2),
(’Noah’, NULL);  -- Noah not assigned

INSERT INTO ex3_departments (dept_name) VALUES
(’Engineering’),
(’HR’),
(’Legal’);  -- Legal has no employees yet

-- Right join: all departments, employees info when present
SELECT e.emp_id, e.name, d.dept_id, d.dept_name
FROM ex3_employees e
RIGHT JOIN ex3_departments d
  ON e.dept_id = d.dept_id
ORDER BY d.dept_id;

-- Full outer join: everything from both sides
SELECT e.emp_id, e.name, d.dept_id, d.dept_name
FROM ex3_employees e
FULL OUTER JOIN ex3_departments d
  ON e.dept_id = d.dept_id
ORDER BY COALESCE(d.dept_id, 999), e.emp_id;

Plain take-away:

• RIGHT JOIN keeps all departments; Engineering and HR will show matching employees and Legal will show NULL employee columns.

• FULL OUTER JOIN shows Noah (employee with no dept) and Legal (dept with no employees) in the same result.

Example 4 — CROSS JOIN (cartesian product)

-- EX4: CROSS JOIN can blow up rows, use with care
DROP TABLE IF EXISTS ex4_colors;
DROP TABLE IF EXISTS ex4_sizes;

CREATE TABLE ex4_colors (color TEXT);
CREATE TABLE ex4_sizes (size TEXT);

INSERT INTO ex4_colors VALUES (’Red’), (’Blue’);
INSERT INTO ex4_sizes VALUES (’S’), (’M’), (’L’);

-- Cross join: every color combined with every size (2 * 3 = 6 rows)
SELECT color, size FROM ex4_colors CROSS JOIN ex4_sizes ORDER BY color, size;

Plain take-away: CROSS JOIN pairs every row from first table with every row from second. Useful for generating combinations, but be careful with big tables.

Example 5 — SELF JOIN (table joined to itself)

-- EX5: SELF JOIN for manager-employee relationships
DROP TABLE IF EXISTS ex5_people;

CREATE TABLE ex5_people (
  person_id SERIAL PRIMARY KEY,
  name TEXT,
  manager_id INTEGER  -- points to person_id
);

INSERT INTO ex5_people (name, manager_id) VALUES
(’CEO’, NULL),
(’Alice’, 1),
(’Bob’, 1),
(’Charlie’, 2),
(’Diana’, 2);

-- Self join: show employee + their manager name
SELECT e.person_id AS emp_id, e.name AS employee,
       m.person_id AS mgr_id, m.name AS manager
FROM ex5_people e
LEFT JOIN ex5_people m
  ON e.manager_id = m.person_id
ORDER BY e.person_id;

Plain take-away: Self join treats the table as two roles (employee and manager) so we can display relationships stored as IDs in the same table.

Example 6 — Multi-table joins (chain joins) + USING

-- EX6: Multi-table join (orders -> customers -> shipments)
DROP TABLE IF EXISTS ex6_customers;
DROP TABLE IF EXISTS ex6_orders;
DROP TABLE IF EXISTS ex6_shipments;

CREATE TABLE ex6_customers (
  customer_id SERIAL PRIMARY KEY,
  name TEXT
);

CREATE TABLE ex6_orders (
  order_id SERIAL PRIMARY KEY,
  customer_id INTEGER REFERENCES ex6_customers(customer_id),
  amount NUMERIC(8,2)
);

CREATE TABLE ex6_shipments (
  shipment_id SERIAL PRIMARY KEY,
  order_id INTEGER REFERENCES ex6_orders(order_id),
  shipped_at TIMESTAMP
);

INSERT INTO ex6_customers (name) VALUES (’Sam’), (’Tina’);
INSERT INTO ex6_orders (customer_id, amount) VALUES (1, 10),(1,20),(2,50);
INSERT INTO ex6_shipments (order_id, shipped_at) VALUES (1, ‘2025-01-02’), (3, ‘2025-01-05’);

-- Chain join: bring customer, order, and shipment info together
SELECT c.customer_id, c.name, o.order_id, o.amount, s.shipment_id, s.shipped_at
FROM ex6_customers c
JOIN ex6_orders o USING (customer_id)      -- shorthand when same column name
LEFT JOIN ex6_shipments s ON o.order_id = s.order_id
ORDER BY c.customer_id, o.order_id;

Plain take-away: You can chain joins to assemble richer records. USING(column) saves typing when both tables have the same join column name — the output will show that column only once.

Example 7 — JOIN with aggregate (which side to aggregate?)

-- EX7: Join and aggregate — total orders per customer
DROP TABLE IF EXISTS ex7_customers;
DROP TABLE IF EXISTS ex7_orders;

CREATE TABLE ex7_customers (
  customer_id SERIAL PRIMARY KEY,
  name TEXT
);

CREATE TABLE ex7_orders (
  order_id SERIAL PRIMARY KEY,
  customer_id INTEGER,
  amount NUMERIC(8,2)
);

INSERT INTO ex7_customers (name) VALUES (’Umar’), (’Vera’), (’Wes’);
INSERT INTO ex7_orders (customer_id, amount) VALUES (1,10),(1,20),(2,5);

-- Option A: aggregate then join (good when you want customers with totals)
SELECT c.customer_id, c.name, COALESCE(t.total_spent, 0) AS total_spent
FROM ex7_customers c
LEFT JOIN (
  SELECT customer_id, SUM(amount) AS total_spent
  FROM ex7_orders
  GROUP BY customer_id
) t ON c.customer_id = t.customer_id
ORDER BY c.customer_id;

-- Option B: join then group (works too)
SELECT c.customer_id, c.name, SUM(o.amount) AS total_spent
FROM ex7_customers c
LEFT JOIN ex7_orders o ON c.customer_id = o.customer_id
GROUP BY c.customer_id, c.name
ORDER BY c.customer_id;

Plain take-away: You can aggregate before or after joining. Aggregating first reduces row volume (useful for big data). Both produce same results here; choose what reads better and performs well.

Example 8 — JOIN pitfalls: duplicate join columns and ambiguous column names

-- EX8: ambiguous column names - aliasing required
DROP TABLE IF EXISTS ex8_a;
DROP TABLE IF EXISTS ex8_b;

CREATE TABLE ex8_a (id SERIAL PRIMARY KEY, name TEXT);
CREATE TABLE ex8_b (id SERIAL PRIMARY KEY, a_id INTEGER, value TEXT);

INSERT INTO ex8_a (name) VALUES (’X’), (’Y’);
INSERT INTO ex8_b (a_id, value) VALUES (1,’v1’), (1,’v2’), (2,’v3’);

-- If both tables have a column named ‘id’ and you select id alone, SQL errors or is ambiguous.
-- Always qualify or alias columns when names repeat.
SELECT a.id AS a_id, a.name, b.id AS b_id, b.value
FROM ex8_a a
JOIN ex8_b b ON a.id = b.a_id;

Plain take-away: When columns share the same name in joined tables, always prefix with table alias (e.g., a.id) or rename in the SELECT list to avoid confusion.

10 Exercises — Joins (try them out)

For each exercise, create tables and insert sample rows (5–12 rows usually) then write the query.

1. Inner join practice: Create students(student_id, name) and marks(student_id, subject, marks). Show student name + their math mark only (assume subject=’math’).

2. Left join rows with no match: employees(emp_id, name) and asset_assignments(emp_id, asset). Show all employees and their assets; employees without assets should show NULL.

3. Full outer join scenario: local_inventory(item_id, qty) and warehouse_inventory(item_id, qty). Show combined view with COALESCE to show item_id and both quantities; include items only in one location.

4. Self join — reporting chain: Create team(member_id, name, reports_to) and show each person with their manager name. Also show people who have no manager.

5. Multi-table join: users, orders, payments — show user name, order id, payment status. Include orders even if payment not recorded.

6. Top N per group without window functions (subquery approach): For sales(region, rep, amount), find the highest single sale per region. (Hint: group by region, get max(amount), join back.)

7. Using USING shortcut: Create two tables sharing product_id. Demonstrate join with USING(product_id) and show the joined columns (note how product_id appears once).

8. Cross join use-case: colors and sizes — generate SKU combinations with a synthetic SKU code (color || ‘-’ || size). Show how many rows you created.

9. Ambiguous names: Create two tables both with column id. Write a join and show how you alias both ids in the output to avoid ambiguity.

10. Join + filter order: Create courses(course_id, name), enrollments(student_id, course_id, enrolled_at), and students(student_id, name). Show names of students who enrolled in ‘Data Science 101’ after ‘2025-01-01’. Use proper joins and WHERE on the joined data.

Chapter 4 — Subqueries & CTEs (Common Table Expressions)

(Step-by-step explanations + runnable examples + 10 exercises)

Hey Bavi — welcome to Chapter 4 🎯.
This is where we start writing multi-level SQL queries — a foundation for building AI-ready data transformations and feature pipelines.
We’ll learn to use subqueries (queries inside queries) and CTEs (WITH clause) to make your SQL cleaner, modular, and easier to debug.

🧠 What You’ll Learn

By the end of this chapter, you’ll be able to:

• Understand what subqueries are and how they work.

• Use subqueries in SELECT, FROM, and WHERE clauses.

• Use correlated subqueries (where the inner query depends on the outer one).

• Understand EXISTS vs IN.

• Build clean, reusable queries with Common Table Expressions (CTEs) using WITH.

• Chain multiple CTEs to build complex transformations step-by-step.

All examples below are copy-paste runnable — each creates its own data, so nothing conflicts.

🧩 1. What Is a Subquery? (Plain-English Explanation)

Think of a subquery as a small query inside another query — like doing one calculation first, then using its result in a bigger calculation.

It’s like:

“First, find the average salary. Then, find all employees earning more than that.”

Subqueries help break complex problems into smaller steps.

🧩 2. Subquery in WHERE (Find rows based on results of another query)

-- EX1: Subquery in WHERE
DROP TABLE IF EXISTS ex1_employees;
CREATE TABLE ex1_employees (
  emp_id SERIAL PRIMARY KEY,
  name TEXT,
  salary NUMERIC(10,2),
  department TEXT
);

INSERT INTO ex1_employees (name, salary, department) VALUES
(’Alice’, 60000, ‘HR’),
(’Bob’, 45000, ‘HR’),
(’Charlie’, 70000, ‘Engineering’),
(’David’, 80000, ‘Engineering’),
(’Ella’, 50000, ‘Marketing’);

-- Find employees who earn more than the average salary
SELECT name, salary
FROM ex1_employees
WHERE salary > (
  SELECT AVG(salary) FROM ex1_employees
);

Plain Explanation:
The inner query (SELECT AVG(salary) ...) calculates one number (the average salary).
The outer query then picks rows where salary > that number.

🧩 3. Subquery in FROM (Treat a subquery as a temporary table)

-- EX2: Subquery in FROM
DROP TABLE IF EXISTS ex2_sales;
CREATE TABLE ex2_sales (
  sale_id SERIAL PRIMARY KEY,
  category TEXT,
  amount NUMERIC(8,2)
);

INSERT INTO ex2_sales (category, amount) VALUES
(’Stationery’, 120.00),
(’Stationery’, 80.00),
(’Beverage’, 200.00),
(’Beverage’, 300.00),
(’Appliance’, 600.00);

-- Step 1: create a subquery to get total per category
-- Step 2: wrap it in another query to calculate percentage share
SELECT
  category,
  total,
  ROUND(100.0 * total / (SELECT SUM(total) FROM (
    SELECT category, SUM(amount) AS total
    FROM ex2_sales
    GROUP BY category
  ) t), 2) AS percent_share
FROM (
  SELECT category, SUM(amount) AS total
  FROM ex2_sales
  GROUP BY category
) t
ORDER BY percent_share DESC;

Layman’s Explanation:
We first group sales by category inside the inner query.
Then, the outer query uses that as a virtual table (t) to calculate each category’s percentage contribution.
This “query-inside-FROM” pattern is common for analytics reports.

🧩 4. Subquery in SELECT (Compute an extra column)

-- EX3: Subquery in SELECT
DROP TABLE IF EXISTS ex3_orders;
CREATE TABLE ex3_orders (
  order_id SERIAL PRIMARY KEY,
  customer_id INTEGER,
  amount NUMERIC(8,2)
);

DROP TABLE IF EXISTS ex3_customers;
CREATE TABLE ex3_customers (
  customer_id SERIAL PRIMARY KEY,
  name TEXT
);

INSERT INTO ex3_customers (name) VALUES
(’Anita’), (’Ben’), (’Carmen’);

INSERT INTO ex3_orders (customer_id, amount) VALUES
(1, 50.00), (1, 30.00), (2, 100.00);

-- Add a column: total spent by that customer
SELECT
  c.customer_id,
  c.name,
  (SELECT SUM(amount) FROM ex3_orders o WHERE o.customer_id = c.customer_id) AS total_spent
FROM ex3_customers c;

Explanation:
For each customer, we run a mini-query (subquery) that sums their orders.
It’s like a lookup performed row by row.

🧩 5. Correlated Subquery (depends on outer query)

-- EX4: Correlated subquery example
DROP TABLE IF EXISTS ex4_products;
CREATE TABLE ex4_products (
  product_id SERIAL PRIMARY KEY,
  name TEXT,
  category TEXT,
  price NUMERIC(8,2)
);

INSERT INTO ex4_products (name, category, price) VALUES
(’Espresso’, ‘Beverage’, 5.00),
(’Latte’, ‘Beverage’, 7.50),
(’Green Tea’, ‘Beverage’, 4.00),
(’Notebook’, ‘Stationery’, 3.00),
(’Pen’, ‘Stationery’, 2.00),
(’Marker’, ‘Stationery’, 4.50);

-- Find products that cost more than the average in their category
SELECT name, category, price
FROM ex4_products p1
WHERE price > (
  SELECT AVG(price)
  FROM ex4_products p2
  WHERE p2.category = p1.category
);

Plain Explanation:
The inner query (p2) runs once for each row in the outer query (p1).
It compares the product’s price with the average price of products in the same category.

This is the essence of a correlated subquery — the inner query depends on the outer query.

🧩 6. EXISTS vs IN

-- EX5: EXISTS vs IN
DROP TABLE IF EXISTS ex5_users;
DROP TABLE IF EXISTS ex5_orders;

CREATE TABLE ex5_users (
  user_id SERIAL PRIMARY KEY,
  name TEXT
);

CREATE TABLE ex5_orders (
  order_id SERIAL PRIMARY KEY,
  user_id INTEGER,
  total NUMERIC(8,2)
);

INSERT INTO ex5_users (name) VALUES (’Aisha’), (’Ben’), (’Celine’), (’Dinesh’);
INSERT INTO ex5_orders (user_id, total) VALUES (1,50),(1,25),(3,100);

-- Using IN
SELECT name
FROM ex5_users
WHERE user_id IN (SELECT user_id FROM ex5_orders);

-- Using EXISTS
SELECT u.name
FROM ex5_users u
WHERE EXISTS (
  SELECT 1 FROM ex5_orders o WHERE o.user_id = u.user_id
);

Layman’s Explanation:
Both IN and EXISTS check for presence of related rows.

• IN compares a value against a list (from the inner query).

• EXISTS checks “does at least one row exist?” — faster when matching large datasets.

🧩 7. Common Table Expressions (CTEs) — WITH clause

CTEs are like temporary tables you define at the top of a query.
They make your SQL easier to read and debug.

Think of it as “naming a subquery” before using it.

-- EX6: Using CTE to simplify query
DROP TABLE IF EXISTS ex6_sales;
CREATE TABLE ex6_sales (
  id SERIAL PRIMARY KEY,
  category TEXT,
  amount NUMERIC(8,2)
);

INSERT INTO ex6_sales (category, amount) VALUES
(’Beverage’, 100),
(’Beverage’, 50),
(’Stationery’, 200),
(’Stationery’, 100),
(’Appliance’, 400);

-- Step 1: total per category
-- Step 2: calculate share using CTE
WITH category_totals AS (
  SELECT category, SUM(amount) AS total
  FROM ex6_sales
  GROUP BY category
),
grand_total AS (
  SELECT SUM(total) AS total_sum FROM category_totals
)
SELECT
  c.category,
  c.total,
  ROUND(100.0 * c.total / g.total_sum, 2) AS percent_share
FROM category_totals c, grand_total g
ORDER BY percent_share DESC;

Explanation:
We broke the problem into 2 logical pieces:

1. category_totals → group totals

2. grand_total → total of all
   Then we joined them to compute % share.

🧩 8. Multiple CTEs chained together (step-by-step data pipeline)

-- EX7: Multi-step CTEs
DROP TABLE IF EXISTS ex7_logs;
CREATE TABLE ex7_logs (
  user_id INTEGER,
  event TEXT,
  amount NUMERIC(8,2),
  event_ts TIMESTAMP
);

INSERT INTO ex7_logs (user_id, event, amount, event_ts) VALUES
(1,’view’,NULL,’2025-01-01’),
(1,’purchase’,20,’2025-01-02’),
(2,’purchase’,50,’2025-01-03’),
(2,’refund’,-50,’2025-01-04’),
(3,’view’,NULL,’2025-01-05’),
(3,’purchase’,10,’2025-01-06’);

WITH purchases AS (
  SELECT user_id, SUM(amount) AS total_spent
  FROM ex7_logs
  WHERE event = ‘purchase’
  GROUP BY user_id
),
refunds AS (
  SELECT user_id, ABS(SUM(amount)) AS total_refunds
  FROM ex7_logs
  WHERE event = ‘refund’
  GROUP BY user_id
),
combined AS (
  SELECT p.user_id, p.total_spent, COALESCE(r.total_refunds,0) AS total_refunds
  FROM purchases p
  LEFT JOIN refunds r USING(user_id)
)
SELECT *,
       (total_spent - total_refunds) AS net_spent
FROM combined
ORDER BY user_id;

Plain Explanation:
This builds a mini data pipeline:

1. purchases step collects all purchase totals.

2. refunds step collects all refunds.

3. combined step merges them and computes the final metric.

This style is widely used in AI feature engineering pipelines.

🧩 9. Recursive CTE (optional advanced)

-- EX8: Recursive CTE for hierarchy
DROP TABLE IF EXISTS ex8_org;
CREATE TABLE ex8_org (
  emp_id SERIAL PRIMARY KEY,
  name TEXT,
  manager_id INTEGER
);

INSERT INTO ex8_org (name, manager_id) VALUES
(’CEO’, NULL),
(’Alice’, 1),
(’Bob’, 1),
(’Charlie’, 2),
(’Diana’, 2);

-- Recursive CTE to show org hierarchy
WITH RECURSIVE org_cte AS (
  SELECT emp_id, name, manager_id, 1 AS level
  FROM ex8_org
  WHERE manager_id IS NULL
  UNION ALL
  SELECT e.emp_id, e.name, e.manager_id, c.level + 1
  FROM ex8_org e
  JOIN org_cte c ON e.manager_id = c.emp_id
)
SELECT * FROM org_cte ORDER BY level, emp_id;

Explanation:
Recursive CTEs call themselves to walk through hierarchies (e.g., manager → subordinate → next level).
They’re like a SQL version of recursion in Python.

🧩 10. When to Use What

🧩 10 Practice Exercises

Try these on your own (I can give full runnable answers if you want next):

1. Create an employees table (name, dept, salary). Find employees earning above the average salary using a subquery.

2. Create a sales table. Show each category and its % of total revenue using a subquery in FROM.

3. Using a students and marks table, find students who scored above the class average.

4. Build a products table and use a correlated subquery to find products priced above their category’s average.

5. Create customers and orders tables. Use IN and EXISTS to find customers who placed orders.

6. Use a CTE to first compute total sales per region, then compute each region’s share of total sales.

7. Chain multiple CTEs: step 1 = purchases, step 2 = refunds, step 3 = net revenue.

8. Use a CTE to find top 3 customers by total spend.

9. Create a small organization table and write a recursive CTE to display the reporting hierarchy.

10. Create a CTE that filters out customers with zero orders, then use that result to find average spending among active customers.

Chapter 5 — Window Functions

(Plain-English guide + detailed examples with data + 10 exercises)

Welcome back, Bavi 👏 — this is one of the most powerful and useful SQL chapters, especially for AI, analytics, and time-series work.

Window functions let you perform calculations across rows — without grouping them together.
They are the secret behind rankings, running totals, moving averages, time-based comparisons, and feature engineering for machine learning models.

🧠 What You’ll Learn in This Chapter

By the end of this chapter, you’ll be able to:

• Understand what window functions are and how they differ from normal aggregates.

• Use OVER() and PARTITION BY.

• Rank rows using ROW_NUMBER(), RANK(), and DENSE_RANK().

• Compare current vs previous row using LAG() and LEAD().

• Compute running totals and moving averages.

• Use frame clauses like ROWS BETWEEN … for rolling calculations.

Every example below is fully runnable (it drops and creates its own tables).
You can copy each one directly into pgAdmin or DBeaver and it’ll just work ✅

💡 1. What Are Window Functions (In Simple Terms)

A window function looks at a “window” of rows — which can be:

• all rows,

• rows in the same group (partition),

• or a subset around the current row (like “previous 2 and next 2”).

It does not collapse rows like GROUP BY does.
You still see every row, but you can add running totals, ranks, or differences as extra columns.

🔹 2. Basic Window Function Syntax

function_name(expression) OVER (
    PARTITION BY column
    ORDER BY column
    ROWS BETWEEN ... AND ...
)

Parts:

• PARTITION BY: divide rows into groups (like GROUP BY but rows remain visible)

• ORDER BY: defines order inside each group

• ROWS BETWEEN: defines the window range

🧩 Example 1 — Compare Window vs Aggregate

-- EX1: Window vs Group Aggregate
DROP TABLE IF EXISTS ex1_sales;
CREATE TABLE ex1_sales (
  region TEXT,
  rep TEXT,
  sales NUMERIC(8,2)
);

INSERT INTO ex1_sales (region, rep, sales) VALUES
(’East’,’Alice’,100),
(’East’,’Bob’,150),
(’West’,’Cathy’,200),
(’West’,’Dan’,300),
(’West’,’Ella’,100);

-- Normal aggregate (collapses rows)
SELECT region, SUM(sales) AS total_sales
FROM ex1_sales
GROUP BY region;

-- Window aggregate (keeps all rows)
SELECT
  region,
  rep,
  sales,
  SUM(sales) OVER (PARTITION BY region) AS region_total
FROM ex1_sales
ORDER BY region, rep;

Plain Explanation:

• The GROUP BY query loses individual reps — it only shows totals.

• The OVER(PARTITION BY region) query keeps each rep and adds the total for their region beside them.

🧩 Example 2 — ROW_NUMBER(), RANK(), DENSE_RANK()

-- EX2: Ranking examples
DROP TABLE IF EXISTS ex2_scores;
CREATE TABLE ex2_scores (
  student TEXT,
  subject TEXT,
  marks INTEGER
);

INSERT INTO ex2_scores (student, subject, marks) VALUES
(’Anita’,’Math’,90),
(’Ben’,’Math’,85),
(’Carmen’,’Math’,85),
(’David’,’Math’,70),
(’Ella’,’Math’,95);

-- Ranking functions
SELECT
  student,
  subject,
  marks,
  ROW_NUMBER() OVER (PARTITION BY subject ORDER BY marks DESC) AS row_num,
  RANK() OVER (PARTITION BY subject ORDER BY marks DESC) AS rank_value,
  DENSE_RANK() OVER (PARTITION BY subject ORDER BY marks DESC) AS dense_rank_value
FROM ex2_scores
ORDER BY marks DESC;

Layman’s Explanation:

• ROW_NUMBER() gives unique increasing numbers even for ties.

• RANK() leaves gaps after ties (1, 2, 2, 4).

• DENSE_RANK() doesn’t leave gaps (1, 2, 2, 3).
  These are essential for leaderboards, ranking models, or finding top-N per category.

🧩 Example 3 — LAG() and LEAD() for Time-Series Comparison

-- EX3: LAG and LEAD
DROP TABLE IF EXISTS ex3_stock;
CREATE TABLE ex3_stock (
  stock_date DATE,
  symbol TEXT,
  price NUMERIC(8,2)
);

INSERT INTO ex3_stock (stock_date, symbol, price) VALUES
(’2025-01-01’,’AAPL’,100),
(’2025-01-02’,’AAPL’,105),
(’2025-01-03’,’AAPL’,102),
(’2025-01-04’,’AAPL’,110),
(’2025-01-05’,’AAPL’,108);

SELECT
  stock_date,
  symbol,
  price,
  LAG(price) OVER (PARTITION BY symbol ORDER BY stock_date) AS prev_price,
  LEAD(price) OVER (PARTITION BY symbol ORDER BY stock_date) AS next_price,
  price - LAG(price) OVER (PARTITION BY symbol ORDER BY stock_date) AS daily_change
FROM ex3_stock
ORDER BY stock_date;

Plain Explanation:

• LAG() gives you the previous row’s value.

• LEAD() gives the next row’s value.

• You can calculate price - LAG(price) to find daily changes.
  This is perfect for AI time-series features like “change from previous day”.

🧩 Example 4 — Running Total & Cumulative Sum

-- EX4: Running total per customer
DROP TABLE IF EXISTS ex4_orders;
CREATE TABLE ex4_orders (
  order_id SERIAL PRIMARY KEY,
  customer TEXT,
  order_date DATE,
  amount NUMERIC(8,2)
);

INSERT INTO ex4_orders (customer, order_date, amount) VALUES
(’Asha’,’2025-01-01’,100),
(’Asha’,’2025-01-03’,50),
(’Asha’,’2025-01-05’,150),
(’Ben’,’2025-01-02’,200),
(’Ben’,’2025-01-04’,100);

SELECT
  customer,
  order_date,
  amount,
  SUM(amount) OVER (PARTITION BY customer ORDER BY order_date) AS running_total
FROM ex4_orders
ORDER BY customer, order_date;

Explanation:
SUM(...) OVER (PARTITION BY customer ORDER BY order_date) keeps a running total as you move through time.
It doesn’t reset the rows — just adds a new column.

🧩 Example 5 — Moving Average (Rolling Window)

-- EX5: Moving average (3-day window)
DROP TABLE IF EXISTS ex5_temps;
CREATE TABLE ex5_temps (
  reading_date DATE,
  city TEXT,
  temp_c NUMERIC(5,2)
);

INSERT INTO ex5_temps (reading_date, city, temp_c) VALUES
(’2025-01-01’,’Paris’,10),
(’2025-01-02’,’Paris’,12),
(’2025-01-03’,’Paris’,14),
(’2025-01-04’,’Paris’,16),
(’2025-01-05’,’Paris’,18),
(’2025-01-06’,’Paris’,20);

SELECT
  reading_date,
  city,
  temp_c,
  ROUND(AVG(temp_c) OVER (
    PARTITION BY city
    ORDER BY reading_date
    ROWS BETWEEN 2 PRECEDING AND CURRENT ROW
  ),2) AS moving_avg_3days
FROM ex5_temps
ORDER BY reading_date;

Layman’s Explanation:
This keeps a moving average of the last 3 days (2 PRECEDING + current).
It’s essential for trend detection and smoothing in time-series analysis.

🧩 Example 6 — Percent Rank and Cumulative Distribution

-- EX6: Percent rank and cumulative distribution
DROP TABLE IF EXISTS ex6_scores;
CREATE TABLE ex6_scores (
  student TEXT,
  marks INTEGER
);

INSERT INTO ex6_scores (student, marks) VALUES
(’A’,90),(’B’,80),(’C’,70),(’D’,60),(’E’,50);

SELECT
  student,
  marks,
  PERCENT_RANK() OVER (ORDER BY marks) AS percent_rank,
  CUME_DIST() OVER (ORDER BY marks) AS cumulative_distribution
FROM ex6_scores
ORDER BY marks;

Plain Explanation:

• PERCENT_RANK() gives rank as a percentage of total rows.

• CUME_DIST() shows the proportion of rows less than or equal to current row.
  These are great for normalizing scores or creating percentile-based features in ML.

🧩 Example 7 — Combine LAG + Window for Change Detection

-- EX7: Detect change events
DROP TABLE IF EXISTS ex7_device;
CREATE TABLE ex7_device (
  device_id TEXT,
  reading_ts TIMESTAMP,
  status TEXT
);

INSERT INTO ex7_device (device_id, reading_ts, status) VALUES
(’D1’,’2025-01-01 10:00’,’ON’),
(’D1’,’2025-01-01 11:00’,’ON’),
(’D1’,’2025-01-01 12:00’,’OFF’),
(’D1’,’2025-01-01 13:00’,’OFF’),
(’D1’,’2025-01-01 14:00’,’ON’);

SELECT
  device_id,
  reading_ts,
  status,
  LAG(status) OVER (PARTITION BY device_id ORDER BY reading_ts) AS prev_status,
  CASE WHEN status <> LAG(status) OVER (PARTITION BY device_id ORDER BY reading_ts)
       THEN ‘CHANGED’
       ELSE ‘SAME’
  END AS change_flag
FROM ex7_device
ORDER BY reading_ts;

Layman’s Explanation:
We compare the current status with the previous one using LAG().
This helps detect status changes, anomalies, or events — very common in IoT, logs, and monitoring datasets.

🧩 When to Use Window Functions in AI/ML

🧩 10 Practice Exercises

Try writing these queries in PostgreSQL. (I can give full runnable solutions if you want next.)

1. Create a sales(region, rep, sales_amount) table. Show each rep’s sales and total sales of their region using a window function.

2. Create students(name, score) and rank them using ROW_NUMBER(), RANK(), and DENSE_RANK().

3. Create orders(customer, order_date, amount) and compute each customer’s running total of spending.

4. Create a temperature(city, reading_date, temp) table and compute 7-day moving average temperature.

5. Using LAG(), find daily price changes for a stock(symbol, date, price) table.

6. Using LEAD(), predict next day’s value for each row in a time-series table.

7. Create scores(student, subject, marks) and find percentile ranks using PERCENT_RANK().

8. Using LAG(), detect when an IoT device changes its status (ON → OFF or OFF → ON).

9. Combine RANK() + PARTITION BY region to find top 2 sales reps per region.

10. Combine multiple window functions: for each customer, show total spent, running total, and percentage of total revenue.

Chapter 6 — Data Manipulation (INSERT · UPDATE · DELETE · MERGE · Transactions)

(Plain-English explanations + fully runnable SQL + 10 hands-on exercises)

🎯 Why this matters

Up to now you’ve learned how to read and analyze data.
This chapter teaches you how to change it safely: add new rows, fix mistakes, remove bad data, and merge new information — the day-to-day skills of any data engineer or AI-pipeline maintainer.

🧱 Core ideas in very simple words

🧩 Example 1 — INSERT (single & multiple rows)

-- EX1: INSERT rows
DROP TABLE IF EXISTS ex1_products;
CREATE TABLE ex1_products (
  product_id SERIAL PRIMARY KEY,
  name TEXT,
  price NUMERIC(8,2),
  category TEXT
);

-- one row
INSERT INTO ex1_products (name, price, category)
VALUES (’Notebook’, 3.50, ‘Stationery’);

-- multiple rows
INSERT INTO ex1_products (name, price, category) VALUES
(’Pen’, 1.00, ‘Stationery’),
(’Espresso Beans’, 12.00, ‘Beverage’),
(’Green Tea’, 9.00, ‘Beverage’);

SELECT * FROM ex1_products;

Plain talk: INSERT adds new sheets to your filing cabinet. You can insert one or many rows in a single shot.

🧩 Example 2 — UPDATE (change existing data)

-- EX2: UPDATE
UPDATE ex1_products
SET price = price * 1.10
WHERE category = ‘Beverage’;  -- raise Beverage prices by 10%

-- give Pen a better name
UPDATE ex1_products
SET name = ‘Blue Pen’
WHERE name = ‘Pen’;

SELECT * FROM ex1_products;

What’s happening: SET tells PostgreSQL which columns to change. WHERE ensures you only change the right rows.
⚠ Without WHERE, you’d update every row.

🧩 Example 3 — DELETE (remove rows)

-- EX3: DELETE
DELETE FROM ex1_products
WHERE price < 2.00;  -- remove cheap items

SELECT * FROM ex1_products;

Tip: Always run a SELECT … WHERE … first to preview which rows will be deleted.

🧩 Example 4 — RETURNING clause

PostgreSQL can return changed rows immediately.

-- EX4: UPDATE … RETURNING
UPDATE ex1_products
SET price = price + 1.00
WHERE category = ‘Stationery’
RETURNING product_id, name, price;