Automate the Boring Stuff with Python

Automate The Boring Stuff with Python
Practical Programming for Total Beginners
by 
Al Sweigart
April 2015, 504 pp.
ISBN-13: 
978-1-59327-599-0

“The best part of programming is the triumph of seeing the machine do something useful. Automate the Boring Stuff with Python frames all of programming as these small triumphs; it makes the boring fun.”
Hilary Mason, Founder of Fast Forward Labs and Data Scientist in Residence at Accel

Wil Wheaton on reading Automate the Boring Stuff with Python: “I’m having a lot of fun breaking things and then putting them back together, and just remembering the joy of turning a set of instructions into something useful and fun, like I did when I was a kid.”

Automate the Boring Stuff with Python is recommended on the Open Source Summer 2015 Reading List!

If you’ve ever spent hours renaming files or updating hundreds of spreadsheet cells, you know how tedious tasks like these can be. But what if you could have your computer do them for you?
In Automate the Boring Stuff with Python, you’ll learn how to use Python to write programs that do in minutes what would take you hours to do by hand—no prior programming experience required. Once you’ve mastered the basics of programming, you’ll create Python programs that effortlessly perform useful and impressive feats of automation to:

  • Search for text in a file or across multiple files
  • Create, update, move, and rename files and folders
  • Search the Web and download online content
  • Update and format data in Excel spreadsheets of any size
  • Split, merge, watermark, and encrypt PDFs
  • Send reminder emails and text notifications
  • Fill out online forms

Step-by-step instructions walk you through each program, and practice projects at the end of each chapter challenge you to improve those programs and use your newfound skills to automate similar tasks.

Don’t spend your time doing work a well-trained monkey could do. Even if you’ve never written a line of code, you can make your computer do the grunt work. Learn how in Automate the Boring Stuff with Python.

Note: The programs in this book are written to run on Python 3.

Author Bio 

Al Sweigart is a software developer and teaches programming to kids and adults. He has written several Python books for beginners, including Hacking Secret Ciphers with Python, Invent Your Own Computer Games with Python, and Making Games with Python & Pygame.

Table of contents 

Introduction
Part I: Python Programming Basics
Chapter 1: Python Basics
Chapter 2: Flow Control
Chapter 3: Functions
Chapter 4: Lists
Chapter 5: Dictionaries and Structuring Data
Chapter 6: Manipulating Strings
Part II: Automating Tasks
Chapter 7: Pattern Matching with Regular Expressions
Chapter 8: Reading and Writing Files
Chapter 9: Organizing Files
Chapter 10: Debugging
Chapter 11: Web Scraping
Chapter 12: Working with Excel Spreadsheets
Chapter 13: Working with PDF and Word Documents
Chapter 14: Working with CSV Files and JSON Data
Chapter 15: Keeping Time, Scheduling Tasks, and Launching Programs
Chapter 16: Sending Email and Text Messages
Chapter 17: Manipulating Images
Chapter 18: Controlling the Keyboard and Mouse with GUI Animation
A: Installing Third-Party Modules
B: Running Programs
C: Answers to the Practice Questions
View the detailed Table of Contents (PDF)
View the Index (PDF)

Reviews 

“Do you need Automate the Boring Stuff with Python? Yes, if you want to enhance your workflow by using automation, this is an excellent place to start. Highly recommended.”
Network World

Listen to an interview with Al Sweigart about Automate the Boring Stuff with Python on the Talk Python to Me podcast!

“Valuable to have on your shelf...an extremely useful book.”
Kids, Code, and Computer Science Magazine

“​Written in easy to understand language with a gentle approach...Automate the Boring Stuff with Python is the perfect manual to get your computer doing the busywork.”
Games Fiends

Read an interview with Al Sweigart on the Import Python blog!

Al Sweigart was interviewed about practical programming on the Test Talks podcast with Joe Colantonio.

Al Sweigart offers tips on Practical Programming for Non-Engineers on Opensource.com.

Read Al Sweigart's advice on Opensource.com for teaching kids how to code.

Al Sweigart chronicles 6 time-consuming tasks you can automate with code, on Opensource.com.

"Automate the Boring Stuff with Python is perfect for anyone who has menial tasks they don't want to spend hours doing."
GeekMom

"One of the best books for learning Python."
Giles McMullen-Klein, FlickThrough Reviews

Updates 
Page 21: In the second paragraph, "programms" should be "programmers."

Page 38: The indentation is slightly wrong in the code; changes are marked with #Changed below:

if name == 'Mary':
print('Hello Mary')
if password == 'swordfish': #Changed
print('Access granted.') #Changed
else: #Changed
print('Wrong password.') #Changed

Page 57: random.ranint() should actually be random.randint()

Page 70: The end of the following sentences:

“Because eggs is declared global at the top of spam(), when eggs is set to 'spam', this assignment is done to the globally scoped spam. No local spam variable is created.”

...should actually be

“...to the globally scoped eggs. No local eggs variable is created.”

Page 88: The left and right columns of Table 4-1 need to be switched. The headings are fine, but the left column should be:

Augmented assignment statement

spam += 1 spam -= 1 spam *= 1 spam /= 1 spam %= 1

…and the right column should be:

Equivalent assignment statement

spam = spam + 1 Updates, errata, etc. This field will be hidden by default, unless the user clicks "show updates." Input format Extra Stuff: This is just another field that will appear beneath "About the Author." Use it for videos, images, or whatever else doesn't fit in the other categories. Input format XML sitemap Menu settings Revision information Comment settings File attachments Shipping settings Authoring information Publishing options Save Save and continue Preview Delete Want sweet deals? Sign up for our newsletter. You might also like... Write Great Code, Volume 2 Book of CSS3, 2nd Edition Clojure for the Brave and True If Hemingway Wrote JavaScript spam = spam - 1 spam = spam * 1 spam = spam / 1 spam = spam % 1

Page 101: In the first sentence on the page, index 7 should actually be index 1:

“…which is why only the list in cheese is modified when you assign 42 at index 1.”

Page 102: In Question #3, the code should actually be this:

spam[int(int('3' * 2) / 11)]

Page 103: A stray “w” was removed from the second sentence in the second paragraph:

“…and the y-coordinates increase going down.”

Page 108: The spam.keys() example has an error. It's marked with #Changed in the code.

>>> for k in spam.keys():
print(k) #Changed
color
age
>>> for i in spam.items():
print(i)
('color', 'red')
('age', 42)

Page 119 (and 142): The following paragraph should actually appear at the end of the summary section of Chapter 6, on page 142:

“That just about covers all the basic concepts of Python programming!

You’ll continue to learn new concepts throughout the rest of this book [...etc...] So let’s learn how to write real programs to do useful automated tasks.”

Page 120: The helpful hint for the “fantasy game inventory” practice project has a helpful example which is actually the complete solution! I’ve adjusted the code as a result to the following:

def display_inventory(inventory):
print("Inventory:")
item_total = 0
for k, v in inventory.items():
# FILL THIS PART IN #Changed
print("Total number of items: " + str(item_total))

Page 120: The inventory.py program has two errors. They're marked with #changed in the code below.

# inventory.py
stuff = {'rope': 1, 'torch': 6, 'gold coin': 42, 'dagger': 1, 'arrow': 12}
def displayInventory(inventory): #Changed
print("Inventory:")
item_total = 0
for k, v in inventory.items():
print(str(v) + ' ' + k)
item_total += v
print("Total number of items: " + str(item_total))
displayInventory(stuff) #Changed

The example output of the displayinventory() function also has an error. It's marked with #Changed in the code below.

Inventory:
12 arrow
42 gold coin
1 rope
6 torch
1 dagger
Total number of items: 62 #Changed

Page 126: In the first paragraph, the second sentence says that “Escaping single and double quotes is optional in raw strings.” It should read “Escaping single and double quotes is optional in multiline strings.”

Page 158: The findall() example has an error. It's marked with #Changed in the code below.

>>> phoneNumRegex = re.compile(r'(\d\d\d)-(\d\d\d)-(\d\d\d\d)') # has groups
>>> phoneNumRegex.findall('Cell: 415-555-9999 Work: 212-555-0000')
[('415', '555', '9999'), ('212', '555', '0000')] #Changed

The paragraph below should also be updated:

2. When called on a regex that has groups, such as (\d\d\d)-(\d\d\d)-(\d\d\d\d), the method findall() returns a list of tuples of strings (one string for each group), such as [('415', '555', '9999'), ('212', '555', '0000')].

Page 212: The backupToZip.py program has an error. It's marked with #Changed in the code below.

# Add all the files in this folder to the ZIP file.
w for filename in filenames:
newBase = os.path.basename(folder) + '_' #Changed
if filename.startswith(newBase) and filename.endswith('.zip')
continue # don't backup the backup ZIP files
backupZip.write(os.path.join(foldername, filename))
backupZip.close()
print('Done.')
backupToZip('C:\\delicious')

Page 217: In the first paragraph, the last sentence says that “This box shape is printed to the console.” It should say that “This box shape is printed to the screen.”

Page 222: The factorialLog.py program has two errors. They're marked with #Changed in the code below.

import logging
logging.basicConfig(level=logging.DEBUG, format='%(asctime)s - %(levelname)s - %(message)s')
logging.debug('Start of program')
def factorial(n):
logging.debug('Start of factorial(%s%%)' % (n)) # Changed
total = 1
for i in range(1, n + 1):
total *= i
logging.debug('i is ' + str(i) + ', total is ' + str(total))
return total
logging.debug('End of factorial(%s%%)' % (n)) # Changed
print(factorial(5))
logging.debug('End of program')

Page 237: Added a space between “requests.get()” and “function” in the following line:

The requests.get() function takes a string of

Page 237: Replaced the first sentence of the last paragraph to read: “The URL goes to a text web page for the entire play of Romeo and Juliet, provided on this book’s site.”

Pages 237 and 239: Updated the URL for where you can download the entire play of Romeo and Juliet to https://automatetheboringstuff.com/files/rj.txt in the code sections on these pages.

Page 239: Changed the filename from rj.txt to pg1112.txt in the following sentence:

“Note that while the filename on the website was pg1112.txt…”

Page 244: The temperature in the temperature part of the web page should be 59 degrees, to match Figure 11-5:

“...the temperature part of the web page is <p class="myforecast-current-lrg">59°F</p>”

Page 254: Updated the 6th line of code in the following section (marked with #Changed):

# Find the URL of the comic image.
comicElem = soup.select('#comic img')
if comicElem == []:
print('Could not find comic image.')
else:
comicUrl = ‘http:’ + comicElem[0].get('src') #Changed
# Download the image.
print('Downloading image %s...' % (comicUrl))
res = requests.get(comicUrl)
res.raise_for_status()

Page 259-60: The example code in the “Filling Out and Submitting Forms” section should be updated to the following, since Gmail has updated their webpage with the username and password fields on separate forms:

>>> from selenium import webdriver
>>> browser = webdriver.Firefox()
>>> browser.get('https://mail.yahoo.com') #Changed
>>> emailElem = browser.find_element_by_id('login-username') #Changed
>>> emailElem.send_keys('not_my_real_email') #Changed
>>> passwordElem = browser.find_element_by_id('login-passwd') #Changed
>>> passwordElem.send_keys('12345')
>>> passwordElem.submit()

Page 282: This line:

>>> sheet['A'].style/styleObj
should be:
>>> sheet['A1'].style = styleObj

Page 304: Updated key/str.lower to key=str.lower in the following code section:

#! python3
# combinePdfs.py - Combines all the PDFs in the current working directory into
# into a single PDF.

import PyPDF2, os

# Get all the PDF filenames. pdfFiles = [] for filename in os.listdir('.'): if filename.endswith('.pdf'): pdfFiles.append(filename) pdfFiles.sort(key=str.lower) #Changed

pdfWriter = PyPDF2.PdfFileWriter()

# TODO: Loop through all the PDF files. # TODO: Loop through all the pages (except the first) and add them. # TODO: Save the resulting PDF to a file.

Page 304: The last sentence of the paragraph following the first code section should read as follows:

Afterward, this list is sorted in alphabetical order with the key=str.lower keyword argument to sort().

Page 414: The third bullet under "Installing the pyautogui Module" has incomplete instructions for installing the pyautogui module on Linux. It should read as follows:

On Linux, run sudo pip3 install python3-xlib, sudo apt-get install scrot, sudo apt-get install python3-tk, and sudo apt-get install python3-dev. (Scrot is a screenshot program that PyAutoGUI uses.)

Page 453: The answer for question 20 of Chapter 7 has this: (,{3})*$') When it should be this: (,\d{3})*$')