LibraryHippo 2020 - A Small Heroku Datastore
Now the Heroku-hosted LibraryHippo can perform periodic tasks, send e-mails, and scrape the Waterloo Public Library's website. All it needs is a datastore to tie these concepts together into a decoupled "push card status to patrons" pipeline.
Move rendering out of library
Last time, the WPL.check_card
method scraped a patron's holds and checkouts,
and rendered them as HTML for display to the user. It would be better to have
the library build a data structure, which can be stored for later use or
rendered by the web app.
"Data structure" is maybe too fancy a term for "dictionary with two values", but it's a start.
Create a local database
Flask doesn't come with a database of its own, like some web frameworks, but there's an extension, Flask-SQLAlchemy, that helps it work with the SQLAlchemy Object Relational Mapper. These will let LibraryHippo interact with databases both locally and on Heroku. It's good practice to track changes to the database schema using Flask-Migrate, so I'll install that as well.
pip install Flask-SQLAlchemy pip install Flask-Migrate inv freeze
Flask needs some configuration settings to access the database.
SQLALCHEMY_DATABASE_URI
describes how the application can contact the
database. In this case, there's a reasonable default, a local SQLite instance.
The SQLALCHEMY_TRACK_MODIFICATIONS
setting will keep the database from
signalling the application whenever the database content changes.
Then the application needs to be taught about the database and migration facilities:
Add a Card
The application now has the ability to talk to the database, but there's no schema defined. Let's add a model and insert a record.
Every model gets an id
, for convenience, and the next 3 fields will be
familiar from the previous article—they identify the card and control access to
the library card state. The last field, last_state
, will be used to record
the last-checked card state. It'll hold a JSON-formatted version of the
dictionary that appears above.
Initialize the database
Now initialize a schemaless database, add the first migration script for the
Card
model, and actually upgrade the database schema:
Insert a card into the database
Normally cards would be added to the database by the users, via a fancy form. For now, the flask shell
will do.
❯ flask shell Python 3.8.1 (tags/v3.8.1:1b293b6, Dec 18 2019, 22:39:24) [MSC v.1916 32 bit (Intel)] on win32 App: app [production] Instance: D:\Sandbox\LibraryHippo\instance >>> from app.models import Card, db >>> card = Card(patron_name="Blair Conrad", number="123456789", pin="9876") >>> db.session.add(card) >>> db.session.commit() >>> Card.query.all() [<Patron Blair Conrad>]
Load the card from the database and store the check results
The Config
class no longer has the hard-coded patron name, card number, and
PIN values added last time, so the check
route must load them from the
database and save the result back onto the card:
The stored result can be seen by querying the database via flask shell
:
Use the stored card check result to send e-mail
Now that the database contains the result of the last card status check, it's
relatively straightforward to include that text in the notification e-mails. All
that's required is to load the card record, deserialize the saved state using
json.loads
, and build the HTML:
Deploy to Heroku
There's nothing left to do but try this out on Heroku. It shouldn't be too much work.
Add and configure a database plugin
Heroku has a free hobby-tier PostgreSQL addon that you can add on right from the command line:
The addon sets the DATABASE_URL
environment variable, which
is the one that the Config.SQLALCHEMY_TRACK_MODIFICATIONS
attribute is
populated from.
SQLAlchemy needs a bonus psycopg2
package to connect to the database, and
there's no harm in having it installed when I'm testing locally, so I'll just
add it to requirements.txt
:
pip install psycopg2 inv freeze
Finally, the application startup should perform the database migration, to react to any new model changes. This requires an extra command before starting gunicorn:
web: flask db upgrade; gunicorn libraryhippo:app
And the only thing left to do is deploy.
Store a library card
I'll store the library card to the PostgreSQL database just as with the local
sqlite instance. The only difference is that instead of running flask shell
directly, I use Heroku's facility to run a one-off command via heroku run
:
❯ heroku run flask shell Running flask shell on ⬢ libraryhippo... up, run.4950 (Free) Python 3.8.1 (default, Dec 23 2019, 04:19:22) [GCC 7.4.0] on linux App: app [production] Instance: /app/instance >>> from app.models import Card, db >>> card = Card(patron_name="Blair Conrad", number="123456789", pin="9876") >>> db.session.add(card) >>> db.session.commit() >>> Card.query.all() [<Patron Blair Conrad>]
With that done, there was no need to keep the old environment variables that encoded my library credentials, so I removed them:
heroku config:unset PATRON_NAME CARD_NUMBER PIN
Wait for the e-mail
And that's it. I did visit /check
on the website to ensure there was a
cached card status, and there was nothing else to do but wait until 18:30 local
time to see everything work together. Sure enough, the task woke up, read the
stored data, and used it in the e-mail:
Progress
Five of nine requirements have been met!
done | web app hosting | |
done | scheduled jobs (run in UTC) | |
done | scraping library websites on users' behalf | |
done | small persistent datastore | |
next | social authentication | |
done | sending e-mail | |
nearly free | ||
job queues | ||
custom domain name |