Duration:

4 Days

Audience:

Employees of federal, state and local governments; and businesses working with the government.

Course Description:

This class meets the needs of advanced Linux Systems Administrators expanding their knowledge of core workings of Linux. Theories and methodologies are covered for tuning, monitoring performance, and analyzing the performance of the Linux operating system.
OBJECTIVES:

  • Learn principles and methods of system tuning
  • Understand Linux architecture
  • Use system and networking benchmark tools for understanding system capabilities
  • Diagnose complex performance and functional system problems
  • Learn to profile system hardware and applications
  • Analyze system performance to identify patterns
  • Learn methods of remote system monitoring

Prerequisites:

An intermediate knowledge of the Linux operating system is required. The equivalent knowledge can be found in the GL120 “Linux Fundamentals” and GL250 “Enterprise Linux Systems Administration” courses.

Supported Distributions:

Red Hat Enterprise Linux 7

Course Outline:

  1. TUNING PRINCIPLES & THE USE METHOD
    1. What is Performance Tuning?
    2. Performance Tuning Methodology
    3. Model System
    4. Establish a Baseline
    5. Establish Performance Targets
    6. Identify Bottleneck
    7. Tune
    8. Measure, Monitor, Evaluate, Repeat?
    9. Final Thoughts
    10. The USE Method
    11. Gathering USE Metrics

    LAB TASKS

    1. Determining the System’s Configuration
    2. USE Method: CPU
    3. USE Method: Memory
    4. USE Method: Network
    5. USE Method: Storage I/O & Capacity
  2. LINUX INTERNALS REVIEW
    1. The Linux Operating System Components
    2. Open Standards
    3. Memory Layout
    4. Important Kernel Threads
    5. Device Files in Linux
    6. Managing Linux Device Files
    7. udev
    8. Random Numbers and /dev/random
    9. Block Devices and the Device Mapper
    10. List Block Devices
    11. SCSI Devices
    12. File Systems with Linux
    13. Hardware Discovery Tools
    14. Cache and Write Barriers
    15. Advanced LVM: Components & Object Tags
    16. Advanced LVM: Automated Storage Tiering
    17. Advanced LVM: Thin Provisioning
    18. Advanced LVM: Striping & Mirroring
    19. Advanced LVM: RAID Volumes
    20. Kernel Modules

    LAB TASKS

    1. Adjusting Kernel Options
    2. Linux Kernel Driver Compilation
  3. LINUX USERSPACE ARCHITECTURE
    1. systemd System and Service Manager
    2. Using systemd
    3. Security Model Overview

    LAB TASKS

    1. Managing Services With Systemd’s systemctl
    2. Setting Limits with the pam_limits Modules
    3. Operation of the setuid() and capset() System Calls
  4. LOG FILE ADMINISTRATION
    1. System Logging
    2. systemd Journal
    3. systemd Journal’s journalctl
    4. Secure Logging with Journal’s Log Sealing
    5. gnome-system-log
    6. Rsyslog
    7. /etc/rsyslog.conf
    8. Log Management
    9. Log Anomaly Detector
    10. Sending logs from the shell

    LAB TASKS

    1. Using the systemd Journal
    2. Setting up a Full Debug Logfile
  5. MONITORING & TROUBLESHOOTING
    1. Viewing Processes
    2. System Status – Memory
    3. System Status – I/O
    4. System Status – CPU
    5. Performance Trending with sar
    6. Determining Service to Process Mapping
    7. Analyzing Process Memory Usage with pmap
    8. Real-time Monitoring of Resources — Cgroups
    9. strace and ltrace
    10. lsof and fuser
    11. ss/netstat and rpcinfo
    12. tcpdump and wireshark

    LAB TASKS

    1. Process Related Tools
    2. System Activity Reporter
    3. Cgroup for Processes
    4. Analyzing Memory Usage
  6. BENCHMARK TOOLS
    1. Benchmark Tools
    2. Disk/Filesystem Benchmarks
    3. Network Benchmarks
    4. Phoronix Test Suite

    LAB TASKS

    1. Benchmarking Disk and Filesystem
    2. Bonnie++ Visualization
  7. SYSTEMTAP
    1. SystemTap Overview
    2. SystemTap Features
    3. SystemTap Requirements
    4. Invoking SystemTap
    5. The 5 passes of stap
    6. SystemTap Probe Point Syntax
    7. SystemTap Probe Modes
    8. SystemTap Probe Points
    9. SystemTap process Probes
    10. Listing and Using Probe Points
    11. Built-in SystemTap Functionality
    12. SystemTap Example Repositories
    13. SystemTap Advanced Example
    14. SystemTap Documentation

    LAB TASKS

    1. System Tap Intro
    2. System Tap
    3. Creating SystemTap Scripts
    4. SystemTap One-liners & Histograms
  8. PERFORMANCE CO-PILOT
    1. Performance Co-Pilot Introduction
    2. PCP Architecture
    3. PCP Performance Metrics
    4. PCP Installation
    5. PCP Performance Monitoring
    6. PCP Centralized Logging
    7. PCP Inference Engine
    8. PCP Deployment Strategies
    9. PCP Advanced Topics

    LAB TASKS

    1. Performance Co-Pilot Introduction
    2. Live Monitoring with pmchart
    3. Install Web Monitoring Tools
  9. REMOTE METRIC GATHERING WITH SNMP
    1. SNMP Overview
    2. SNMP Operation – Query
    3. MIBs and OID Namespace
    4. SNMP Security
    5. SNMP for Linux

    LAB TASKS

    1. SNMP
  10. LINUX PERF
    1. perf Overview
    2. perf Commands Typical Workflow
    3. perf Events
    4. perf stat
    5. perf record
    6. perf report

    LAB TASKS

    1. Perf
  11. LINUX TUNABLES
    1. Operating System Tuning Caveats
    2. Consider OS (or Kernel) Upgrade
    3. Tuning CPU – NUMA
    4. Tuning CPU – Scheduling
    5. Huge Memory Pages
    6. Tuning Virtual Memory Manager
    7. Tuning Virtual Memory Manager (cont.)
    8. Dropping Caches
    9. I/O Scheduler
    10. Tuning Filesystems
    11. Tuning Filesystems (cont.)
    12. Network Tuning
    13. tuned