Rather than have users create new login identities and passwords (and worry about securely storing those passwords), I'm going to use social logins. The old LibraryHippo application supported login via a Google account, and I'll do the same here, while leaving the door open to add other vectors in the future.

New Requirements

We'll need new software to perform the authentication and user management. The Flask-Login package adds support for storing user objects and managing their sessions, and Authomatic simplifies authentication via OAuth2 providers such as Google (and many more). Install them both:

pip install Flask-Login
pip install Authomatic
inv freeze

Register with Google

Google requires applications using it as an OAuth provider to register and obtain credentials which must be presented when authenticating users. I visit the [APIs & Services Dashboard]https://console.developers.google.com/apis/dashboard), create a new project, and add a new Web application OAuth Client ID. I added 3 Authorised redirect URIs, which are the URIs that lead to the Flask route that will handle login. Authomatic sends its current URI to Google when authenticating, and Google redirects back to the URI (with an additional token) after authenticating the user. As a security measure, Google will only redirect back to known URIs.

I added

• http://localhost:5000/login/google/ (for testing locally)
• https://libraryhippo.herokuapp.com/login/google/ (for when I deploy to Heroku), and
• http://libraryhippo.com/login/google/ (for when I eventually point the domain name to Heroku)

and was rewarded with a Client ID and a Client secret, which are needed by Authomatic.

I also configured the OAuth consent screen, which controls what users see when they are redirected to Google. I added an application name and Application logo. Since we only need to identify a user, I left the scopes limited to the minimum:

• email
• profile, and
• openid

I don't have an Application Privacy Policy link, so left it blank. This seems to prevent me from "verifying" the application. So far it hasn't been an issue for testing, but I'll keep an eye on it.

New Configuration

Authomatic knows a lot about its underlying providers, but still needs to be told about the Client ID and Client secret obtained above. I'll add those to the secrets file.

# secrets
…
OAUTH_GOOGLE_CLIENT_ID=********.apps.googleusercontent.com
OAUTH_GOOGLE_CLIENT_SECRET=********

Of course these need to be provided to the application through the Config class:

# config.py
from authomatic.providers import oauth2
…

OAUTH = {
    "google": {
        "class_": oauth2.Google,
        "consumer_key": os.environ.get("OAUTH_GOOGLE_CLIENT_ID"),
        "consumer_secret": os.environ.get("OAUTH_GOOGLE_CLIENT_SECRET"),
        "scope": ["profile", "email"],
    }
}

SECRET_KEY = os.environ.get("SECRET_KEY") or "MeF+3?N',Nmsn39v]"

This will tell Authomatic that there's a provider named "google" that is a Google provider, and what the consumer ID and secret are. The "scope" entry in the dictionary controls what Authomatic asks for from Google. If I don't specify this, it won't load the user's email.

The other new entry, SECRET_KEY, is a Flask concept that's used to sign its session cookie. It should be treated as a secret; the default value is just supplied to make local testing easier.

The User Model

Flask-Login will persist to any backing store, but the database makes the most sense for LibraryHippo. The new User model will store the details of the users that have been registered (by logging in):

# __init__.py
from flask_login import LoginManager
…

login_manager = LoginManager(app)

# app/models.py
from app import login_manager
from flask_login import UserMixin
…

class User(UserMixin, db.Model):
    id = db.Column(db.Integer, primary_key=True)
    social_id = db.Column(db.String(64), nullable=False, unique=True)
    nickname = db.Column(db.String(64), nullable=False)
    email = db.Column(db.String(64), nullable=False)


@login_manager.user_loader
def load_user(id):
    return User.query.get(int(id))

The load_user function is registered as the Flask-Login "user_loader". Flask-Login will ensure that there's information in the session that can identify the active user and will use the function to load the rest of the user information from the database.

Of course the new model has to be added via SQLAlchemy:

flask db migrate -m "Add User model"
flask db upgrade

The main event - log a user in

The above has been preparatory work to allow LibraryHippo to authenticate a user via Google, log them into Flask, and save a user record to the database.

First, I add an Authomatic instance to app, initializing it with the OAuth configuration and secret key I set up earlier:

# app/__init__.py
from logging.config import dictConfig
…

authomatic = Authomatic(config=Config.OAUTH, secret=Config.SECRET_KEY)

Now add the login route, which will contain a segment for the Google provider:

# app/routes.py
from flask import flash, make_response, redirect, request, session, url_for
from flask_login import current_user, login_user
from authomatic.adapters import WerkzeugAdapter

from app import authomatic
from app.models User

…

@app.route("/login/<provider>/", methods=["GET", "POST"])
def login(provider):
    if not current_user.is_anonymous:
        return redirect(url_for("index"))

    response = make_response()
    result = authomatic.login(
        WerkzeugAdapter(request, response),
        provider_name=provider,
        session=session,
        session_saver=lambda: app.save_session(session, response),
    )

    if not result:
        return response

    if result.user:
        result.user.update()
        if result.user.id is None:
            flash("Authentication failed.")
            app.logger.error("Authentication failed: %s", result.error)
            return redirect(url_for("index"))

        social_id = provider + ":" + result.user.id
        user = User.query.filter_by(social_id=social_id).first()

    if not user:
        user = User(
            social_id=social_id, nickname=result.user.name, email=result.user.email
        )
        db.session.add(user)
        db.session.commit()

    login_user(user, remember=True)
    return redirect(url_for("index"))

There's a lot going on here. The gist is that if the current user is logged in, the login route just redirects to the main page; there's no need to login if someone's already logged in.

Otherwise, Authomatic attempts to log the user in, via the WerkzeugAdapter, which lets it manipulate the HTTP request and response to direct the flow of the application. It's given the current Flask session as well as a callback it can use to save the session. Once that succeeds, the user is "updated" to fill in extra information such as their name and e-mail address.

Then the method attempts to load the user from the database, looking them up by combining the provider name and the ID assigned by the provider. If no record exists, one is created and saved back to the database for the future. Finally, the user is logged into Flask.

Logging out

Once users are logged in, they might want to log out, maybe so another user can check their library cards. This is much easier than the login process. Again, I add a new route:

# app/routes.py
from flask_login import logout_user

…

app.route("/logout")
    logout():
    if not current_user.is_anonymous:
        logout_user()
    return redirect(url_for("index"))

It just calls Flask's logout_user if the user isn't logged in. Then they're redirected to the main page.

Add login/logout links

Users will need a way to initiate the login process, or to log out if they're already logged in, and these links should be available from every page, so I'll replace my ad hoc page generation with templates that will centralize those functions.

<!-- app/templates/base.jinja -->
<!DOCTYPE html>
<html>
    <head>
        <title>LibraryHippo</title>
    </head>
    <body>
        <nav>
        {% if current_user.is_anonymous %}
        <a href="{{ url_for('login', provider='google') }}">Login</a>
        {% else %}
        <a href="{{ url_for('logout') }}">Logout {{ current_user.nickname }}</a>
        {% endif %}
        {% block body %}{% endblock %}
    </body>
</html>

The base.jinja template sets up common elements to all the pages of the application. Here it checks to see if the user is logged in. If not, it includes a link to the login route, and if so, a link to the logout route. The logout link includes the user's nickname, mostly to make it easier for me to test. The main page has not much to add, so its template is very plain for now:

<!-- app/templates/index.jinja -->
{% extends "base.jinja" %}

{% block body %}
<h1>LibraryHippo 2020</h1>
{% endblock %}

Then the index route is updated to use the template:

# app/routes.py
from flask import render_template

…

@app.route("/")
@app.route("/index")
def index():
    return render_template("index.jinja")

How's it look?

From a not-logged-in account, I visit my local LibraryHippo instance:

The new "Login" navigation link appears. When I click it, I'm taken to a Google-hosted page where I can select the account I want to use. This screen will vary depending on whether I'm already logged into Google, and with how many accounts. Note that it indicates what Google account-specific information will be shared with LibraryHippo:

• name
• email address
• language preference (which LibraryHippo doesn't use), and
• profile picture (which LibraryHippo doesn't use)

And finally once I'm logged in, the "Login" link becomes a "Logout" link, including my name, which was harvested from Google.

Deploying to Heroku

There almost nothing to this. I already added the Heroku-specific URL to the Google configuration, so I just have to generate a SECRET_KEY and set it and the OAUTH_GOOGLE_CLIENT_* values from earlier.

heroku config:set SECRET_KEY=…

Progress

Six of nine requirements have been met!

Move rendering out of library

Last time, the WPL.check_card method scraped a patron's holds and checkouts, and rendered them as HTML for display to the user. It would be better to have the library build a data structure, which can be stored for later use or rendered by the web app.

# app/libraries/wpl.py
def check_card(self, patron, number, pin):
    session = Session()
    summary_page = self.login(session, patron, number, pin)

    holds_url = urllib.parse.urljoin(
        self.login_url(),
        summary_page.find(name="a", href=re.compile("/holds$"))["href"],
    )
    checkouts_url = urllib.parse.urljoin(
        self.login_url(),
        summary_page.find(name="a", href=re.compile("/items$"))["href"],
    )

    holds = self.get_holds(session, holds_url)
    checkouts = self.get_checkouts(session, checkouts_url)

    return {
        "holds": holds,
        "checkouts": checkouts,
    }

"Data structure" is maybe too fancy a term for "dictionary with two values", but it's a start.

# app/routes.py
@app.route("/check")
def check():
    card_check_result = WPL().check_card(
        Config.PATRON_NAME, Config.CARD_NUMBER, Config.PIN
    )

    result = "<h1>Holds</h1>"
    for hold in card_check_result["holds"]:
        result += "<dl>"
        for k, v in hold.items():
            result += f"<dt>{k}</dt><dd>{v}</dd>"
        result += "</dl><hr>"

    result += "<h1>Checkouts</h1>"
    for checkout in card_check_result["checkouts"]:
        result += "<dl>"
        for k, v in checkout.items():
            result += f"<dt>{k}</dt><dd>{v}</dd>"
        result += "</dl><hr>"

    return result

Create a local database

Flask doesn't come with a database of its own, like some web frameworks, but there's an extension, Flask-SQLAlchemy, that helps it work with the SQLAlchemy Object Relational Mapper. These will let LibraryHippo interact with databases both locally and on Heroku. It's good practice to track changes to the database schema using Flask-Migrate, so I'll install that as well.

pip install Flask-SQLAlchemy
pip install Flask-Migrate
inv freeze

Flask needs some configuration settings to access the database. SQLALCHEMY_DATABASE_URI describes how the application can contact the database. In this case, there's a reasonable default, a local SQLite instance. The SQLALCHEMY_TRACK_MODIFICATIONS setting will keep the database from signalling the application whenever the database content changes.

# config.py
class Config(object):
    # …
    # Remove PATRON_NAME, CARD_NUMBER, and PIN, as they'll move to the database

    SQLALCHEMY_DATABASE_URI = os.environ.get(
        "DATABASE_URL"
    ) or "sqlite:///" + os.path.join(basedir, "app.db")
    SQLALCHEMY_TRACK_MODIFICATIONS = False

Then the application needs to be taught about the database and migration facilities:

# app/__init__.py
# …
from flask_migrate import Migrate
from flask_sqlalchemy import SQLAlchemy

# …
app = Flask(__name__)
app.config.from_object(Config)

mail = Mail(app)

db = SQLAlchemy(app)
migrate = Migrate(app, db)

from app import routes, models

Add a Card

The application now has the ability to talk to the database, but there's no schema defined. Let's add a model and insert a record.

# app/__init__.py
class Card(db.Model):
    id = db.Column(db.Integer, primary_key=True)
    patron_name = db.Column(db.String(64))
    number = db.Column(db.String(32))
    pin = db.Column(db.String(16))
    last_state = db.Column(db.Text())

    def __repr__(self):
        return f"<Patron {self.patron_name}>"

Every model gets an id, for convenience, and the next 3 fields will be familiar from the previous article—they identify the card and control access to the library card state. The last field, last_state, will be used to record the last-checked card state. It'll hold a JSON-formatted version of the dictionary that appears above.

Initialize the database

Now initialize a schemaless database, add the first migration script for the Card model, and actually upgrade the database schema:

flask db init
flask db migrate -m "Add Card model"
flask db upgrade

Creating directory D:\Sandbox\LibraryHippo\migrations ...  done
Creating directory D:\Sandbox\LibraryHippo\migrations\versions ...  done
Generating D:\Sandbox\LibraryHippo\migrations\alembic.ini ...  done
Generating D:\Sandbox\LibraryHippo\migrations\env.py ...  done
Generating D:\Sandbox\LibraryHippo\migrations\README ...  done
Generating D:\Sandbox\LibraryHippo\migrations\script.py.mako ...  done
Please edit configuration/connection/logging settings in 'D:\\Sandbox\\LibraryHippo\\migrations\\alembic.ini' before proceeding.

Generating D:\Sandbox\LibraryHippo\migrations\versions\b2fc8df2f32f_add_card_model.py ...  done

Insert a card into the database

Normally cards would be added to the database by the users, via a fancy form. For now, the flask shell will do.

❯ flask shell
Python 3.8.1 (tags/v3.8.1:1b293b6, Dec 18 2019, 22:39:24) [MSC v.1916 32 bit (Intel)] on win32
App: app [production]
Instance: D:\Sandbox\LibraryHippo\instance
>>> from app.models import Card, db
>>> card = Card(patron_name="Blair Conrad", number="123456789", pin="9876")
>>> db.session.add(card)
>>> db.session.commit()
>>> Card.query.all()
[<Patron Blair Conrad>]

Load the card from the database and store the check results

The Config class no longer has the hard-coded patron name, card number, and PIN values added last time, so the check route must load them from the database and save the result back onto the card:

# app/routes.py
# …
import json
from app.models import Card

# …

@app.route("/check")
def check():
    card = Card.query.get(1)  # a hack - we know there's only 1 card for now
    card_check_result = WPL().check_card(card)
    card.last_state = json.dumps(card_check_result)
    db.session.commit()
    # rendering code…

The stored result can be seen by querying the database via flask shell:

❯ flask shell
Python 3.8.1 (tags/v3.8.1:1b293b6, Dec 18 2019, 22:39:24) [MSC v.1916 32 bit (Intel)] on win32
App: app [production]
Instance: D:\Sandbox\LibraryHippo\instance
>>> from app.models import Card
>>> card = Card.query.get(1)
>>> card.last_state

'{"holds": [{"Title": "\\nBlood heir / Am\\u00e9lie Wen Zhao\\n\\n", "Status": " 2 of 2 holds ", "Pickup": "WPL McCormick Branch", "Cancel": "09-17-20", "Freeze": true}, {"Title": "\\nEducated : a memoir / Tara Westover\\n\\n", "Status": " 1 of 2 holds ", "Pickup": "WPL McCormick Branch", "Cancel": "09-28-20", "Freeze": true}, {"Title": "\\nCoders : the making of a new tribe and the remaking of the world / Clive Thompson\\n\\n", "Status": " 1 of 1 holds ", "Pickup": "WPL McCormick Branch", "Cancel": "10-16-20", "Freeze": true}, {"Title": "\\nBecoming Superman : my journey from poverty to Hollywood : with stops along the way at murder, mayhem, movie stars, cults, slums, sociopaths, and war crimes / J. Michael Straczynski ; introduction by Neil Gaiman\\n\\n", "Status": " 1 of 1 holds ", "Pickup": "WPL McCormick Branch", "Cancel": "11-02-20", "Freeze": true}, {"Title": "\\nBatman : Creature of the Night / illustrated by John Paul Leon.\\n\\n", "Status": " 4 of 6 holds ", "Pickup": "WPL McCormick Branch", "Cancel": "11-06-2

Use the stored card check result to send e-mail

Now that the database contains the result of the last card status check, it's relatively straightforward to include that text in the notification e-mails. All that's required is to load the card record, deserialize the saved state using json.loads, and build the HTML:

# app/cli.py
# …
from app import mail, models

# …

def register(app):
    @app.cli.command("notify-all")
    def notify_all():
        card = models.Card.query.get(1)  # a hack - we know there's only 1 card for now
        last_card_state = json.loads(card.last_state)

        html_body = "<h1>Holds</h1>"
        for hold in last_card_state["holds"]:
            html_body += "<dl>"
            for k, v in hold.items():
                html_body += f"<dt>{k}</dt><dd>{v}</dd>"
            html_body += "</dl><hr>"

        html_body += "<h1>Checkouts</h1>"
        for checkout in last_card_state["checkouts"]:
            html_body += "<dl>"
            for k, v in checkout.items():
                html_body += f"<dt>{k}</dt><dd>{v}</dd>"
            html_body += "</dl><hr>"

        now = datetime.now().isoformat()

        msg = Message(
            "LibraryHippo starting notifications", recipients=["blair@blairconrad.com"]
        )
        msg.body = f"starting notifications at {now}"
        msg.html = html_body

        mail.send(msg)

        # …

Deploy to Heroku

There's nothing left to do but try this out on Heroku. It shouldn't be too much work.

Add and configure a database plugin

Heroku has a free hobby-tier PostgreSQL addon that you can add on right from the command line:

heroku addons:add heroku-postgresql:hobby-dev

Creating heroku-postgresql:hobby-dev on ⬢ libraryhippo... free
Database has been created and is available
! This database is empty. If upgrading, you can transfer
! data from another database with pg:copy
Created ·················· as DATABASE_URL
Use heroku addons:docs heroku-postgresql to view documentation

The addon sets the DATABASE_URL environment variable, which is the one that the Config.SQLALCHEMY_TRACK_MODIFICATIONS attribute is populated from.

SQLAlchemy needs a bonus psycopg2 package to connect to the database, and there's no harm in having it installed when I'm testing locally, so I'll just add it to requirements.txt:

pip install psycopg2
inv freeze

Finally, the application startup should perform the database migration, to react to any new model changes. This requires an extra command before starting gunicorn:

web: flask db upgrade; gunicorn libraryhippo:app

And the only thing left to do is deploy.

Store a library card

I'll store the library card to the PostgreSQL database just as with the local sqlite instance. The only difference is that instead of running flask shell directly, I use Heroku's facility to run a one-off command via heroku run:

❯ heroku run flask shell
Running flask shell on ⬢ libraryhippo... up, run.4950 (Free)
Python 3.8.1 (default, Dec 23 2019, 04:19:22)
[GCC 7.4.0] on linux
App: app [production]
Instance: /app/instance
>>> from app.models import Card, db
>>> card = Card(patron_name="Blair Conrad", number="123456789", pin="9876")
>>> db.session.add(card)
>>> db.session.commit()
>>> Card.query.all()
[<Patron Blair Conrad>]

With that done, there was no need to keep the old environment variables that encoded my library credentials, so I removed them:

heroku config:unset PATRON_NAME CARD_NUMBER PIN

Wait for the e-mail

And that's it. I did visit /check on the website to ensure there was a cached card status, and there was nothing else to do but wait until 18:30 local time to see everything work together. Sure enough, the task woke up, read the stored data, and used it in the e-mail:

Progress

Five of nine requirements have been met!

The production application gathers information from libraries using a combination of App Engine's custom URL Fetch service, an older version of the Beautiful Soup HTML parser (which had to be copied into the application source), and some glue that I wrote. Today I'll try to replicate that using modern, commodity components.

Gathering requirements

I've heard that the Requests library is a feature-rich and popular library for sending HTTP requests. Knowing no more than that, I'll try it. Beautiful Soup has worked very well for me in the past, but has had a number of significant releases since the version I've used, so I'll try the latest, 4.8.2 as I write.

pip install requests
pip install BeautifulSoup4
inv freeze

Library Credential Management

Library websites don't allow unfettered access to their patrons' records, thank goodness, so it'll be necessary to use at least one patron's credentials during the "scraping" test. I don't want to embed my credentials in the application source code, so they have to be stored somewhere else. Had I already integrated a permanent datastore and implement user management, the credentials would be tied to the LibraryHippo user's account, but for now I'll read them from environment variables that I'll save in the secrets file established in Sending Email from Heroku.

# secrets
# …
PATRON_NAME=Blair Conrad
CARD_NUMBER=123456789
PIN=9876

with the corresponding change to the Config.

Implementation

When a patron wants to check their library card status at the Waterloo Public Library, they have to visit the login page and provide their credentials, after which they are taken to a summary page that basically just show the number of checkouts and held items, with links to complete lists of their checkouts and holds. That's 4 pages visited, and possibly some hidden redirects, plus one if they manually logout. I planned to have the automated components follow the same path.

Login

My initial thought was that LibraryHippo could login directly, just by posting credentials to the website, but it failed miserably. The reason: hidden fields. The login page looks like it has 4 inputs, counting the "Submit" button as an input:

But there are actually 6. The four we see and two hidden ones, called lt and _eventId:

<input id="name" name="name" type="text" value="" size="25" maxlength="50" tabindex="15">
<input id="code" name="code" type="text" value="" size="25" maxlength="50" tabindex="20">
<input id="pin" name="pin" type="password" value="" size="25" maxlength="64" tabindex="30">
<input type="submit" name="Log In" class="loginSubmit" tabindex="35">
<input type="hidden" name="lt" value="_cB2859561-37E4-542A-1165-9B73858A095A_k57C7CE2E-6774-24D9-E8F6-3A54E706F248" />
<input type="hidden" name="_eventId" value="submit" />

When I submit the form without them, the login fails. The lt field's value is different every time I visit the login page, so I assume the server is using it as a session identifier or some such. The only way to have a successful login is to harvest that value from the login page. So the card-checking flow must be:

1. request the login page
2. find the hidden form fields
3. submit the login form, including the configured credentials as well as the hidden field values
4. read the response and find the links to the checkouts and holds pages
5. request the checkouts page and read the results
6. request the holds page and read the results
7. request the logout page, to logout

The first 3 steps are covered by the login method on my new WPL class:

# app/libraries/wpl.py
from bs4 import BeautifulSoup
from requests import Session

class WPL:
    def login_url(self):
        return (
            "https://books.kpl.org/iii/cas/login?service="
            + "https://books.kpl.org/patroninfo~S3/j_acegi_cas_security_check&lang=eng&scope=3"
        )

    def check_card(self, patron, number, pin):
        session = Session()
        summary_page = self.login(session, patron, number, pin)
        # …

    def login(self, session, patron, number, pin):
        initial_login_page_view = session.get(self.login_url())
        login_page = BeautifulSoup(initial_login_page_view.text, "html.parser")

        form_fields = self.get_form_fields(login_page)
        form_fields.update({"name": patron, "code": number, "pin": pin})

        login_response = session.post(self.login_url(), form_fields)
        return BeautifulSoup(login_response.text, "html.parser")

    def get_form_fields(self, page):
        form_fields = {}
        for input_field in page.find_all("input"):
            if input_field["type"] == "submit":
                form_fields["submit"] = input_field["name"]
            else:
                form_fields[input_field["name"]] = input_field.get("value", "")

        return form_fields

Starting from the top, there are a few things to note:

1. I import key classes from Beautiful Soup (bs4) and the Requests library
2. The login URL is hard-coded here. You have to start somewhere.
3. I'm passing patron, number, and pin into the check_card method, which is the principal entry point into this class
4. check_card immediately creates a requests.Session object to communicate with the outside world. It's possible to call methods like get and post directly on the requests module, but the Session class provides session management by tracking cookies, pools connections, and can persist parameters across requests. Use of the Session class is the key to having the library website grant access on subsequent requests. Under Google App Engine, I had to write request decorators to handle the cookies and session management
5. login first requests the login page as discussed, and parses it using Beautiful Soup before passing to get_form_fields to find all the hidden and special (e.g. "submit") fields to ensure the right values are posted. Not how easy the find_all method makes it to locate all the input fields.
6. login then fills in values specific to this card: patron name, card number ("code") and PIN, before post\ ing the result and returning it to check_card for future processing

Finding the checkouts and holds pages

Once a user logs into the WPL, they see a summary page that contains rather a lot of personal information that they probably don't care to see every day, as well as some links that would allow them to update their account and, most importantly, a link to their holds and one to their checkouts, right at the bottom of the box above the "Log Out" button.

It's those last two that we'll be going after. Unfortunately, the links aren't clearly marked with an ID or even a class:

<div class="patNameAddress">
    <strong>CONRAD, BLAIR</strong><br>
    123 FAKE STREET<br>
    WATERLOO ON<br>
    132-456-7890<br>
    EXP DATE:10-11-2020<br>
    <br>
    <div>
    </div>
    <div>
    <a href="/patroninfo~S3/12345678/holds" target="_self">22 requests (holds).</a>
    </div>
    <div>
    <a href="/patroninfo~S3/12345678/items" target="_self">5 Items currently checked out</a>
    </div>
</div>

The URLs vary from patron to patron, so we can't hard-code them. I'll cheat a little and look for links that end in "/holds" or "/items":

# app/libraries/wpl.py
import re
import urllib.parse

# …

class WPL:
    def check_card(self, patron, number, pin):
        session = Session()
        summary_page = self.login(session, patron, number, pin)

        holds_url = urllib.parse.urljoin(
            self.login_url(),
            summary_page.find(name="a", href=re.compile("/holds$"))["href"],
        )
        items_url = urllib.parse.urljoin(
            self.login_url(),
            summary_page.find(name="a", href=re.compile("/items$"))["href"],
        )

        # …

Beautiful Soup looks on the summary page for <a> tags whose href match the supplied regular expressions ("ends with /holds" or "ends with /items") and returns the results. Indexing by "href" returns that attribute's value. Since the URLs were relative, I join them to the original login URL to getd absolute URLs.

Loading the Holds

The hold page repeats the same personal information from the summary page, and then lists all the patron's holds in a table.

And the HTML behind the table starts like this:

<table lang="en" class="patFunc">
    <tr class="patFuncTitle">
    <th colspan="6" class="patFuncTitle">
        <label id="items_count">22 HOLDS</label>
    </th>
    </tr>

    <tr class="patFuncHeaders">
    <th class="patFuncHeaders"> CANCEL </th>
    <th class="patFuncHeaders"> TITLE </th>
    <th class="patFuncHeaders"> STATUS </th>
    <th class="patFuncHeaders">PICKUP LOCATION</th>
    <th class="patFuncHeaders"> CANCEL IF NOT FILLED BY </th>
    <th class="patFuncHeaders"> FREEZE </th>
    </tr>

    <tr class="patFuncEntry on_ice">
    <td class="patFuncMark" >
        <input type="checkbox" name="cancelb2677337x00" id="cancelb2677337x00" />
    </td>
    <td class="patFuncTitle">
        <label for="cancelb2677337x00">
        <a href="/record=b2677337~S3">
            <span class="patFuncTitleMain">Blood heir / Amélie Wen Zhao</span>
        </a>
        </label>
        <br />
    </td>
    <td class="patFuncStatus"> 2 of 2 holds </td>
    <td class="patFuncPickup">
        <div class="patFuncPickupLabel">
        <label for="locb2677337x00">Pickup Location</label>
        </div>
        <select name=locb2677337x00 id=locb2677337x00 >
        <option value="mn+++" >Central Library-KPL</option>
        <option value="ch+++" >Country Hills Library-KPL</option>
        <option value="fh+++" >Forest Heights Library-KPL</option>
        <option value="gr+++" >Grand River Stanley Pk Lib-KPL</option>
        <option value="pp+++" >Pioneer Park Library-KPL</option>
        <option value="w++++" >WPL Main Library</option>
        <option value="wm   " selected="selected">WPL McCormick Branch</option>
        <option value="ww+++" >WPL John M. Harper Branch</option>
        </select>
    </td>
    <td class="patFuncCancel">09-17-20</td>
    <td class="patFuncFreeze" >
        <div class="patFuncFreezeLabel">
        <label for="freezeb2677337x00" >Freeze</label>
        </div>
        <input type="checkbox" name="freezeb2677337x00" checked />
    </td>
    </tr>

Each significant cell in the table has a class indicator that can be used to interpret the contents. Note that since the table is actually part of a form, where the patron can choose to cancel, freeze, or change the pickup location of an item, some td elements contain input controls, slightly complicating the parsing. Still, it's not that difficult to extract the information:

# app/libraries/wpl.py
def get_holds(self, session, holds_url):
    holds = []
    holds_page = BeautifulSoup(session.get(holds_url).text, "html.parser")

    holds_table = holds_page.find("table", class_="patFunc")

    for hold_row in holds_table.children:
        if hold_row.name != "tr" or "patFuncEntry" not in hold_row["class"]:
            continue

        hold = {}
        for hold_cell in hold_row.children:
            if hold_cell.name != "td":
                continue
            cell_class = hold_cell["class"][0]
            cell_name = cell_class.replace("patFunc", "")
            if cell_name == "Mark":
                continue
            if cell_name == "Pickup":
                hold[cell_name] = hold_cell.find(
                    "option", selected="selected"
                ).string
            elif cell_name == "Freeze":
                hold[cell_name] = "checked" in hold_cell.input.attrs
            else:
                # logger.info("cell " + cell_name)
                hold[cell_name] = "".join(hold_cell.strings)
        holds.append(hold)
    return holds

# …

def check_card(self, patron, number, pin):
    # …
    result = "<h1>Holds</h1>"
    for hold in self.get_holds(session, holds_url):
        result += "<dl>"
        for k, v in hold.items():
            result += f"<dt>{k}</dt><dd>{v}</dd>"
        result += "</dl><hr>"

    return result

I load the page, find the table, and iterate over rows with the patFunEntry class, extracting values to shove in a hold object, which is just a dictionary. the default action is to store the contents of the td, but the "Mark" column is just used to cancel holds, and conveys no information, so I drop it. The "Pickup" column always contains a number of selections, so I'm carful to grab the option element that is "selected". Finally the "Freeze" column is effectively a boolean: if the input has a "checked" attribute, the hold is frozen.

Back in check_card, I just loop over the holds, printing a dl for each one, listing the attributes. It's not pretty, and would be better as a Jinja template, but it's good enough for a proof of concept.

Loading the Checkouts

Parsing the Checkouts was to have been the same as parsing the holds, so I was going to omit it, but that plan fell through when I found that the checkouts page's HTML is malformed in a way that defeated the html.parser library. Some tr tags aren't closed in the table:

<table lang="en" class="patFunc">
<tr class="patFuncTitle">
    <th colspan="5"  class="patFuncTitle">5 ITEMS CHECKED OUT</th>
</tr>

<tr class="patFuncHeaders">
    <th class="patFuncHeaders"> RENEW </th>
    <th class="patFuncHeaders"> TITLE </th>
    <th class="patFuncHeaders"> BARCODE </th>
    <th class="patFuncHeaders"> STATUS </th>
    <th class="patFuncHeaders"> CALL NUMBER </th>
<!-- NO CLOSING TR -->
<tr class="patFuncEntry">
    <td class="patFuncMark"><input type="checkbox" name="renew0" id="renew0" value="i3879884" /></td>
    <td class="patFuncTitle"><label for="renew0"><a href="/record=b2529260~S3"><span class="patFuncTitleMain">Banff, Jasper & Glacier National Parks</span></a></label><br /></td>
    <td class="patFuncBarcode"> 33420013067559 </td>
    <td class="patFuncStatus"> DUE 03-02-20  <span  class="patFuncRenewCount">Renewed 1 time</span></td>
    <td style="text-align:left" class="patFuncCallNo"> 917.1233204 Ban  </td>
</tr>

As a result, Beautiful Soup saw only the "patFuncTitle" and "patFuncHeaders" rows. The workaround is to install the lxml XML and HTML parser and have BeautifulSoup use it:

pip install lxml
inv freeze

# app/libraries/wpl.py
def get_checkouts(self, session, checkouts_url):
    checkouts = []
    checkouts_page = BeautifulSoup(session.get(checkouts_url).text, "lxml")
    # fairly boring parsing hereafter

Deploying to Heroku

Deploying is straightforward. I use my fancy inv deploy command and set the new secret environment variables:

heroku config:set "PATRON_NAME=Blair Conrad"
heroku config:set CARD_NUMBER=123456789
heroku config:set PIN=9876

And voila:

Progress

Four of nine requirements have been met!

Making a Custom Flask command

On Google App Engine, every action had to be run via the web interface, so they had to be secured by special credentials, which could be a little tricky. Being able to write the tasks essentially as scripts under Flask/Heroku removes a lot of complexity. These scripts are what Flask calls custom commands; they can be invoked from outside the web application, but with all the context (such as the e-mail configuration set up last time) of the the full application.

First, I created new app/cli.py file to hold the command:

# app/cli.py
import time

from app import mail
from datetime import datetime
from flask_mail import Message


def register(app):
    @app.cli.command("notify-all")
    def notify_all():
        now = datetime.now().isoformat()

        msg = Message(
            "LibraryHippo starting notifications", recipients=["blair@blairconrad.com"]
        )
        msg.body = f"starting notifications at {now}"
        msg.html = f"<h1>Test mail from LibraryHippo</h1><p>{msg.body}."

        print(msg.body)
        mail.send(msg)

        time.sleep(300)

        now = datetime.now().isoformat()

        msg = Message(
            "LibraryHippo ending notifications", recipients=["blair@blairconrad.com"]
        )
        msg.body = f"ending notifications at {now}"
        msg.html = f"<h1>Test mail from LibraryHippo</h1><p>{msg.body}."

        print(msg.body)
        mail.send(msg)

This was taken from the old sendmail web route, which I removed completely, and then updated to

1. send two e-mails, just to make sure we could, and
2. sleep for 5 minutes between e-mails, to verify that Heroku won't kill a longer-running task

I called the task notify-all, since I'm simulating that action in the existing LibraryHippo: notifying all families of their library card status. The command can be invoked by running

❯ flask notify-all

and it performs exactly how you'd hope.

Once the new version of the application is deployed using inv deploy, it's even possible to run the task on a Heroku dyno via

❯ heroku run flask notify-all

Running flask notify-all on ⬢ libraryhippo... up, run.2562 (Free)
starting notifications at 2020-02-10T11:56:25.840194
ending notifications at 2020-02-10T12:01:26.234036

Scheduling the Task

There are a number of options for scheduling repeated tasks on Heroku, but a very simple (and free!) one is the Heroku Scheduler add-on. It hasn't the flexibility of other schedulers, supporting only daily, hourly, or 10-minutely schedules. Still, LibraryHippo just needs to send e-mails once per day and check users' cards about that often, so it should do.

Adding the scheduler is very easy:

❯ heroku addons:create scheduler:standard

Creating scheduler:standard on ⬢ libraryhippo... free
To manage scheduled jobs run:
heroku addons:open scheduler

Created scheduler-curved-17868
Use heroku addons:docs scheduler to view documentation

A short search didn't reveal a way to affect the schedule from the console, but it was easy enough to open the web-based configuration.

❯ heroku addons:open scheduler

Adding a job is as simple as choosing "Create job", selecting a time to run, and typing the command to execute, which in this case was flask notify-all. I chose to execute daily at 11:30 PM because as I typed, it was 11:26 PM UTC.

Now there's nothing to do but wait. In the meantime I opened up the LibraryHippo application's log view (at https://dashboard.heroku.com/apps/libraryhippo/logs) and watched.

Shortly after 6:30 PM local time, the log started updating, and I received my first e-mail, with further updates and a second e-mail about 5 minutes later. The log looked like this:

Note that there are some earlier entries from the manually-invoked test run I'd done at 2020-02-10T12:01:29, and also from the web worker that had been active from some earlier time and was shut down due to inactivity at 12:26:17.

At 23:30:25, the flask notify-all worker starts up, running achieving an "up" state before logging (via the print statements in the code) the two e-mail messages that it sent, and finally transitioning to a "complete" state and shutting down at 23:35:28.

And the e-mails arrived right on schedule:

A Note on Time Zones

As the documentation states, Heroku Scheduler jobs use a clock in the UTC time zone, but LibrayHippo's customers live in the Eastern Time Zone (of the Americas), which is either 5 or 4 hours behind UTC, depending on whether daylight saving time is in effect. When I ran my test, I wanted the e-mails to be sent near 18:30 in my local time zone, and daylight saving time was not in effect, so I scheduled the job for 23:30 UTC.

Configuring the jobs with an offset is not particularly onerous, but it does mean that once daylight saving time takes effect, users will see their e-mails start arriving an hour later in the day. This is annoying, but can be worked around in a variety of ways. I'll probably just configure the notification job to run at 10:00 UTC, so e-mails arrive near 5:00 local time in the winter and 6:00 in the summer.

Some alternatives to having the e-mail delivery time shift with the seasons are to pay for a more expensive and sophisticated scheduler, or to further workaround by having 2 scheduled jobs. One could run at 10:00 UTC and one at 11:00 UTC. They could each check whether daylight saving time were active in the Eastern Time Zone, ensuring that only the proper job ran. But I'll leave that for later. Or never.

Progress

Three of nine requirements have been met.

Requirements

Flask has a plugin to make sending mail easier, Flask-Mail; I'll install it, but first I'll add a task to freeze the requirements.txt file, since I'm tired of using the Powershell syntax to do that.

# tasks.py
@task
def freeze(c):
    """Freeze pip's requirements.txt. Does not commit the file."""
    import pip

    result = c.run("pip freeze")
    with open("requirements.txt", mode="w") as requirements:
        requirements.write(result.stdout)

Now to install the package:

❯ pip install Flask-Mail
❯ inv freeze

Collecting Flask-Mail
Using cached Flask-Mail-0.9.1.tar.gz (45 kB)
Requirement already satisfied: Flask in d:\sandbox\libraryhippo\venv\lib\site-packages (from Flask-Mail) (1.1.1)
Collecting blinker
Using cached blinker-1.4.tar.gz (111 kB)
Requirement already satisfied: itsdangerous>=0.24 in d:\sandbox\libraryhippo\venv\lib\site-packages (from Flask->Flask-Mail) (1.1.0)
Requirement already satisfied: Werkzeug>=0.15 in d:\sandbox\libraryhippo\venv\lib\site-packages (from Flask->Flask-Mail) (0.16.1)
Requirement already satisfied: Jinja2>=2.10.1 in d:\sandbox\libraryhippo\venv\lib\site-packages (from Flask->Flask-Mail) (2.11.1)
Requirement already satisfied: click>=5.1 in d:\sandbox\libraryhippo\venv\lib\site-packages (from Flask->Flask-Mail) (7.0)
Requirement already satisfied: MarkupSafe>=0.23 in d:\sandbox\libraryhippo\venv\lib\site-packages (from Jinja2>=2.10.1->Flask->Flask-Mail) (1.1.1)
Installing collected packages: blinker, Flask-Mail
    Running setup.py install for blinker ... done
    Running setup.py install for Flask-Mail ... done
Successfully installed Flask-Mail-0.9.1 blinker-1.4

blinker==1.4
Click==7.0
Flask==1.1.1
Flask-Mail==0.9.1
gunicorn==20.0.4
invoke==1.4.1
itsdangerous==1.1.0
Jinja2==2.11.1
MarkupSafe==1.1.1
python-dotenv==0.10.5
Werkzeug==0.16.1

Flask-Mail Configuration

The production LibraryHippo application uses Sendgrid as an email server, and I see no reason to deviate now. Flask-Mail must be configured to use this server. Some of the configuration should remain a secret (the password), and some could be hard-coded right in the app, but I prefer to separate the configuration from the code. I'll put the public settings in a file called configuration, which will be committed, and the sensitive ones in secrets, which I won't commit.

# configuration
MAIL_DEFAULT_SENDER=librarianhippo@gmail.com
MAIL_PORT=587
MAIL_SERVER=smtp.sendgrid.net
MAIL_USE_TLS=True
MAIL_USERNAME=apikey

# secrets
# Do not commit this file. It must not be shared.

MAIL_PASSWORD=AN_API_KEY_THAT_I_WONT_SHARE_WITH_YOU

Code

Now to make Flask aware of the configuration from above and to add Flask-Mail to the application so it can send email.

The Config class is a bridge that gives Flask access to the environment variables. It

1. provides a central location to view all configuration settings
2. supplies sensible defaults for settings that might have some, and
3. converts some settings from strings to their proper types, simplifying usage in the code.

# config.py
import os
from dotenv import load_dotenv

basedir = os.path.abspath(os.path.dirname(__file__))
load_dotenv(os.path.join(basedir, "secrets"))
load_dotenv(os.path.join(basedir, "configuration"))


class Config(object):
    MAIL_DEFAULT_SENDER = os.environ.get("MAIL_DEFAULT_SENDER")
    MAIL_PASSWORD = os.environ.get("MAIL_PASSWORD")
    MAIL_PORT = int(os.environ.get("MAIL_PORT") or 25)
    MAIL_SERVER = os.environ.get("MAIL_SERVER")
    MAIL_USE_TLS = os.environ.get("MAIL_USE_TLS") != "False"
    MAIL_USERNAME = os.environ.get("MAIL_USERNAME")

Then 4 lines are added to the application initialization to hook the configuration class and Flask-Mail into the application:

# app/__init__.py
from config import Config

from flask import Flask
from flask_mail import Mail

app = Flask(__name__)
app.config.from_object(Config)

mail = Mail(app)

from app import routes

Finally, a new route is added to the application to trigger the email. Note that this is completely unprotected and a horrible, horrible idea for a production environment, as someone could just visit the page and spam me. But it makes for an easy test.

#  routes.py
from app import app
from app import mail
from datetime import datetime
from flask_mail import Message

@app.route("/sendmail")
def sendmail():
    now = datetime.now().strftime("%c")
    msg = Message("Mail from LibraryHippo", recipients=["blair@blairconrad.com"])
    msg.body = f"test mail from LibraryHippo at {now}"
    msg.html = f"<h1>Test mail from LibraryHippo</h1><p>It's now {now}."
    mail.send(msg)
    return f"Sent mail at {now}"

And it works! I can trigger the route and get a success message. Nearly instantaneously, I receive the email in my inbox.

❯ inv run

* Serving Flask app "libraryhippo.py"
* Environment: production
WARNING: This is a development server. Do not use it in a production deployment.
Use a production WSGI server instead.
* Debug mode: off
* Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)
127.0.0.1 - - [07/Feb/2020 06:08:38] "GET /sendmail HTTP/1.1" 200 -

Deploying to Heroku

There's very little work to do to deploy to Heroku. All the new configuration settings are in the configuration file except for MAIL_PASSWORD. The Heroku web interface provides a way to set the value, but it's easier to use the Heroku command line interface:

❯ heroku config:set "MAIL_PASSWORD=AN_API_KEY_THAT_I_WONT_SHARE_WITH_YOU"

Setting MAIL_PASSWORD and restarting ⬢ libraryhippo... done, v4
MAIL_PASSWORD: AN_API_KEY_THAT_I_WONT_SHARE_WITH_YOU

And now to deploy and test

❯ inv deploy

remote: Compressing source files... done.
remote: Building source:
remote:
remote: -----> Python app detected
remote: -----> Need to update SQLite3, clearing cache
remote: -----> Installing python-3.8.1
remote: -----> Installing pip
remote: -----> Installing SQLite3
remote: Sqlite3 successfully installed.
remote: -----> Installing requirements with pip
#
# a lot of boring pip stuff
#
remote:        Successfully installed Click-7.0 Flask-1.1.1 Flask-Mail-0.9.1 Jinja2-2.11.1 MarkupSafe-1.1.1 Werkzeug-0.16.1 blinker-1.4 gunicorn-20.0.4 invoke-1.4.1 itsdangerous-1.1.0 python-dotenv-0.10.5
remote:
remote: -----> Discovering process types
remote:        Procfile declares types -> web
remote:
remote: -----> Compressing...
remote:        Done: 47.9M
remote: -----> Launching...
remote:        Released v5
remote:        https://libraryhippo.herokuapp.com/ deployed to Heroku
remote:
remote: Verifying deploy... done.
To https://git.heroku.com/libraryhippo.git
3f0598d..ddf4728  lh2020 -> master

Note the time discrepancy between the time that LibraryHippo reported and the time that GMail said it receive the message. I'm sending from UTC-5, and the Heroku server appears to be in UTC. It's not a problem for now, but may become a factor when scheduling jobs.

Progress

Two of nine requirements have been met.

Lately, the project's not much fun. I've had the urge to make small tweaks or add the odd feature, but I haven't bothered. Whenever I'm tempted to, there've been a few speedbumps:

1. it's written in Python 2.7, which I'm trying to leave behind, and
2. it's running on Google App Engine

I'm grateful to Google for providing a free tier of App Engine for projects such as mine to run on, and I've benefitted a lot from it, but it's not without its downsides:

1. many of the services have custom interfaces, that I only use there, so they fade from my memory, and
2. it seems every time I come back to the project, I have to update my SDK to even work with LibraryHippo, and sometimes learn new commands to deploy or monitor the service

Since Python 2.7 is now unsupported, I'm looking to upgrade to 3.7. App Engine documentation now suggests writing the application in Flask, rather than the older framework. The project seems well-regarded, so I read some documentation and looked at Miguel Grinberg's amazing The Flask Mega-Tutorial, (which I highly recommend: it's eminently readable, obviously well-researched, and espouses good overall development practices), and decided to give that a try.

I figured that since I'm doing a near-complete rewrite anyhow (some of the core code should survive), it might be time to explore different hosting options, making use of commodity databases, authentication, queuing, etc.

After a little searching, it looks as if Heroku might be the platform that meets my needs, which are:

1. web app hosting
2. a small (perhaps a few MB) persistent datastore
3. scheduled jobs
4. authentication via social accounts
5. sending e-mail
6. free, or nearly so; as I said, this is a hobby project, and I'm not willing to dump several tens of dollars into it every month
7. Update 2020-02-11: scraping library websites on users' behalf

I've other requirements, but these aren't likely to be deal-breakers:

1. custom domain name
2. job queues

Over the next few posts, I'm going to pick high-risk requirements from the above list, and attempt to satisfy them via a Flask web application running on Heroku's free tier. If I can satisfy (in the most rudimentary way) these requirements, I'll pursue a full conversion.

I'll assign issues and pull requests that I make for this work to the LibraryHippo 2020 milestone.

Hopefully the series will serve as entertainment or education for you, and a useful reference for me when I wonder how or why I did something.

(The exact cause of the problem won't be of interest to many people, but I thought the way it presented, and how I diagnosed it, might help someone out. Read on!)

I checked the logs, and sure enough, there was an error. Sort of like this:

I 2014-04-28 14:10:43.429 checking [patron name redacted] Waterloo
I 2014-04-28 14:10:47.550 saving checked card for [patron name redacted]
E 2014-04-28 14:10:51.061 Failed to save checked card. Continuing.
Traceback (most recent call last):
  File "/base/data/home/apps/s~libraryhippo27/1.368776574735783966/libraryhippo.py", line 380, in save_checked_card
    checked_card.payload = card_status
  File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/db/__init__.py", line 614, in __set__
    value = self.validate(value)
  File "/base/data/home/apps/s~libraryhippo27/1.368776574735783966/gael/objectproperty.py", line 11, in validate
    result = pickle.dumps(value)
  File "/base/data/home/runtimes/python27/python27_dist/lib/python2.7/pickle.py", line 1374, in dumps
    Pickler(file, protocol).dump(obj)
  File "/base/data/home/runtimes/python27/python27_dist/lib/python2.7/pickle.py", line 224, in dump
    self.save(obj)
  File "/base/data/home/runtimes/python27/python27_dist/lib/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/base/data/home/runtimes/python27/python27_dist/lib/python2.7/pickle.py", line 725, in save_inst
    save(stuff)
  File "/base/data/home/runtimes/python27/python27_dist/lib/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self

[about 80 lines of stack trace elided]

  File "/base/data/home/runtimes/python27/python27_dist/lib/python2.7/pickle.py", line 663, in _batch_setitems
    save(v)
  File "/base/data/home/runtimes/python27/python27_dist/lib/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/base/data/home/runtimes/python27/python27_dist/lib/python2.7/pickle.py", line 725, in save_inst
    save(stuff)
  File "/base/data/home/runtimes/python27/python27_dist/lib/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/base/data/home/runtimes/python27/python27_dist/lib/python2.7/pickle.py", line 649, in save_dict
    self._batch_setitems(obj.iteritems())
  File "/base/data/home/runtimes/python27/python27_dist/...(length 98720)
I 2014-04-28 14:10:51.275 Saved; key: __appstats__:043300, part: 190 bytes, full: 65479 bytes, overhead: 0.004 + 0.005; link: http://libraryhippo27.appspot.com/_ah/stats/details?time=1398708643356

Whenever LibraryHippo checks a patron's library card, it saves the results to the datastore to be used to construct the next day's e-mails, but an inability to save doesn't keep the results from being displayed on the web page. So that part made sense.

The next step was to figure out what was going wrong with the save. The logs indicated that pickle was using more stack frames than were available.

I try to debug it

I added a copy of the offending card to my family's account on the live site. Same problem. Then I fired up the dev environment at home and did the same thing. Everything worked like a charm.

That was unexpected. So, as a last resort, I started thinking.

Why so deep?

The structure that was being pickled (CardStatus) was quite flat. Here are the involved classes' definitions. Unless noted otherwise, everything is a string or datetime:

class CardStatus:
    def __init__(self, card, items=None, holds=None):

        self.library_name = card.library.name
        self.patron_name = card.name
        self.items = items or []  # Items
        self.holds = holds or []  # Holds
        self.info = []            # strings
        self.expires = datetime.date.max

class Thing():
    def __init__(self, library, card):

        self.library_name = library.name
        self.user = card.name
        self.title = ''
        self.author = ''
        self.url = ''
        self.status = ''
        self.status_notes = []    # strings

class Hold(Thing):
    def __init__(self, library, card):
        Thing.__init__(self, library, card)

        self.pickup = ''
        self.holds_url = ''
        self.expires = datetime.date.max

class Item(Thing):
    def __init__(self, library, card):
        Thing.__init__(self, library, card)

        self.items_url = ''

So, one level for the CardStatus, one for Thing, one for Hold (or Item), one for a list, and one for the status_notes strings in the list. That's 5 levels. And probably pickle encodes a dict in a few of the of the complex types. Let's be generous and say 10 levels, each of which take maybe 5 nested pickle functions. That's about 50. Plus however deep we are in the stack before the pickling happens. That shouldn't be more than 100.

How deep is too deep?

Popular wisdom on the web seems to be to increase the recursion limit when pickle runs into these kinds of problems. I was loath to do this, as I'd be constantly worrying about what depth to allow and whether it'd be enough and so on.

While I dithered over that, my wife called to me from the television room, "Just set the limit higher. That should help your user for now and it will give you more time to work on the problem."

So I did. I was worried that the App Engine runtime wouldn't let me change the recursion limit, so I used sys.getrecursionlimit to log the depth, sys.setrecursionlimit, and sys.getrecursionlimit again to verify.

Turns out that:

• The default recursion limit on App Engine's environment is 800.
• You can change the limit. I went up to 20000.
• In the dev environment the limit is 1000.

The higher limit on the production server fixed things. I relaxed a little, and started thinking.

Maybe the 800/1000 difference accounted for things working at home, but not in production. I used sys.setrecursionlimit to change the limit at home, and reproduced the error. Huzzah! Now I could move more quickly.

Inserting diagnostics into pickle

Back to the question of why pickle was recursing so deeply. I don't routinely debug Python, relying instead on the power of logging statements. Thus, I decided to provide a custom pickler that did normal pickling things, but that also, for every object pickled, logged the stack depth, the object type, and its representation:

import pickle
import logging
import traceback

class SpyingPickler(pickle.Pickler, object):
    def save(self, obj):
        logging.info("depth: %d, obj_type: %s, obj: %s",
                     len(traceback.extract_stack()),
                     type(obj), repr(obj))
        super(SpyingPickler, self).save(obj)

I ran this, and got reams of data. Scads and scads and scads, including what looked like to be large HTML documents. So I took out the repr(obj) and repeated. This was more manageable.

…
depth: 176, obj_type: 
depth: 179, obj_type: 
depth: 179, obj_type: 
depth: 182, obj_type: 
depth: 182, obj_type: 
depth: 182, obj_type: 
…

Already we can see that we're quite deep in the stack, but the real surprise was the BeautifulSoup.NavigableString. LibraryHippo uses BeautifulSoup to scrape the libraries' web pages. A NavigableString is basically a Unicode string plus navigation. To support the navigation, the instance points at what looks to be a DOM model of the entire parsed HTML page. That explains the deep deep recursion.

The offending objects were in the pickup field of the Hold class. A quick fix to ensure we're storing a plain string, and the problem was resolved.

What a difference

Aside from the obvious effect of correct e-mails, I was curious to see what other differences this change made.

Before the fix, the serialization descended to a stack depth of 892, and produced a 546221-byte blob. After, the maximum depth is 52, and the blob size is 10779 bytes. So everyone should benefit a little from lower resource usage and quicker card-checking. Fortunately, LibraryHippo isn't popular enough to exceed the free quotas, so I wasn't paying extra for compute or storage. Then again, if I had been, maybe I would've noticed the problem before a user did.

I live in Waterloo, Ontario, a city with weekly residential waste collection. My collection day is Monday. In the warm months, there's an additional biweekly (that's every 2 weeks, not twice a week) collection of what's called yard waste: grass clippings, raked-up leaves, tree branches, and so on.

People in my area (including me) have trouble remembering which weeks are yard waste weeks. This means that every two weeks I get to see collections of plant material left at the curb outside people's homes and then brought in a day or two later, only to reappear the next week.

I use tools to augment my memory. Every year, at the beginning of the warm season, I visit the Region of Waterloo's website where I can learn about their Yard waste residential collection program. Here I find links to two PDFs: a pretty version of the schedule for Waterloo, and an "accessible" version.

I look at one of those schedules, and I create a recurrning Google Calendar event. Then my phone tells me on Sunday night whether to haul the tree parts to the street. It works really well.

However, every year, I've wished that the Region provided an .iCalendar feed or created a Google Calendar for me so I wouldn't have to do this. But I haven't done more than wish.

This year, I'm doing more. For starters, I created a few Google Calendars. One for me (and my fellow Monday-pickup people), and four for everyone else in Waterloo (and Cambridge). The Region's web site gave me start and end weeks for pickups, so it was very easy to make a biweekly repeating event, using these Google Calendar settings:

Lo and behold, it shows up in my calendar:

I created the Tuesday through Friday calendars by making the obvious modification to the schedule above.

I put all the calendars on a separate page that I intend to maintain until the Region provides replacements. Or I move. Go! Get one for your collection day.

Next, I'm going to pester the Region, showing them how easy it was to do this and to see if they'd be willing to carry on with the work next year. If something comes of it, I'll let you know.