I started playing with Google App Engine more than a year ago as a way to get my feet wet with Python. However, over time, I’ve noticed that when I want to play around with an idea I have for a web application, I find I spend too much time setting things up before I get to the fun stuff.
Looking over my playground, I’ve noticed a few items that are fairly consistent over all the project carcasses lying around:.
- Nearly all use the Tornado framework
- All use a template system (a couple different ones; I’m covering Jinja2)
- Several had simple comment or forum systems and used reStructuredText (rst) or Markdown as the markup language, which was then converted into HTML for display on the web.
- Several had support for code syntax coloring using Pygments.
When you have an idea, you want to get it out and work with it as fast as possible to see if it’s worth putting more effort into. If you follow this article, you can use it to keep a starter application around in your back pocket.
Prerequisites
Google App Engine
If you want to work through this article, you’ll want to go to the Google App Engine homepage, download the SDK for your platform, and read up on it a bit if you haven’t already. You don’t need to get an account until you are actually ready to deploy a project.
Python 2.5
A requirement of the Google App Engine SDK is Python 2.5. Because Python 2.5 is a little on the old side (2.7.x is the current version), you’ll need to check the installed version if you’re on a Unix/Linux-type system (including Mac OS X) as Python is usually a part of most default installations. Mac OS X Snow Leopard comes with Python 2.6, so you’ll have to install 2.5 yourself if that’s going to be your development system. At the end of the article, I included a Resources section for links and extra stuff. I have also included a subsection titled, Installing Python 2.5 for Mac OS X 10.6.
I myself haven’t had any issues using Python 2.6…yet. (I sometimes forget to change the Python version I’m using). But you shouldn’t count on it not throwing a weird error your way.
Tornado – Request/Response Framework
Tornado is an open-source non-blocking webserver, written in Python. It was created by and is used to power FriendFeed (bought by Facebook in 2009). You can’t use all of its features on App Engine but I’ve grown to like its request/response framework and a couple of the extras it comes with that are nice (XSRF/CSRF attack prevention for one). Tornado is always the first thing I add to my projects. If you’d prefer not to use Tornado, I’ve linked to a page with a list of several other frameworks at the end of this article in the Other Frameworks section.
Adding and using Tornado in your projects
- Download Tornado
- Unarchive
- Copy the “tornado” directory from “tornado-n.n.n” (where n is a number; I downloaded 1.1.1) to the root of your App Engine Project. The root of the project is the directory where app.yaml, index.yaml, and main.py reside after you create a project using Google App Engine Launcher.
The next step is to edit main.py. We’re just going to stick with the “Hello, World!” app that the App Engine SDK initially creates but modify it to use Tornado instead of webapp. So go ahead and open main.py.
The first two lines of code import webapp and webapp.util, which we no longer need.
from google.appengine.ext import webapp
from google.appengine.ext.webapp.util
Go ahead and delete them and then, add the following three lines:
import tornado.web
import tornado.wsgi
import wsgiref.handlers
This gives us access to the Tornado web framework request handler (tornado.web.RequestHandler), which we’ll be using to replace the handlers from webapp (webapp.RequestHandler), the Tornado WSGI (Web Server Gateway Interface) implementation, and the Python WSGI reference handlers to run the Tornado WSGI application you create (in a few minutes).
Since the point of this article is to give you a decent starting point for creating your App Engine projects, let’s define a base handler, which we will use as a foundation for all our projects’ handlers. It will extend tornado.web.RequestHandler so we get all the goodies from there, but also any methods we define under our base handler will also be accessible from any handlers that extend our base handler. This keeps things organized, simple, and keeps us from defining the same functionality over and over and over again. Who wants to keep reinventing the wheel? That doesn’t move you forward.
The code for the base handler at this point is really simple and is as follows:
class BaseHandler(tornado.web.RequestHandler):
"""BaseHandler
[document this class here; what are it's properties? what are it's methods? What are it's hopes and dreams?]
"""
Insert the BaseHandler class above the MainHandler class.
Now modify the MainHandler class to inherit from BaseHandler instead of webapp.RequestHandler. I also will change the output so I know it has truly changed and is not just a cached version. So, instead of “Hello, World!” I have it say, “Hello, Tornado!”.
class MainHandler(BaseHandler):
"""The MainHandler for the application. Just says "Hello, Tornado!""""
self.write('Hello, Tornado!')
Last and definitely not least, is to modify the main() function. Currently, it creates and runs a webapp application:
...
application = webapp.WSGIApplication([('/', MainHandler)], debug=True)
util.run_wsgi_app(application)
However, that’s not what we need. We’re no longer using webapp so we need to use these instead:
settings {
"xsrf_cookies": False, # If True, forces you to include tokens for POSTs
}
application = tornado.wsgi.WSGIApplication([(r"/", MainHandler),],**settings)
wsgiref.handlers.CGIHandler().run(application)
Now, test that puppy out! If you ran the dev_appserver.py from the command line, the URL should be in the last line of the output when you start the server.
dev_appserver.py app-name
...
INFO 2011-02-12 16:32:01,522 dev_appserver_main.py:507] Running application app-name on port 8080: http://localhost:8080
If you’re using the App Engine Launcher, select the appropriate app from the list, click the green Run button and once the icon next to the project is green, click the browse button (the browse button doesn’t always show that it is clickable and sometimes I just have to click it a couple of times).
All should be well and good and you’re now using Tornado instead of webapp! Read up on the framework or read the code (the code is very well documented–another plus).
Jinja2 – Templates
A good template system helps keep things organized. It’s bad form to output HTML directly from your code. It’s cleaner and easier to maintain if you use a template system. It also makes it easier for a designer or front end developer to work but not have to monkey around in code. Jinja2 is based off of Django templates and I find it fits me quite well.
Adding and using Jinja2 in your projects
There are simple instructions for getting Jinja2 to work on App Engine. But wait! Before you go off and do that, you need to install Jinja2! I installed Jinja2 as a Python Egg. Then in step 2, I copied the “jinja2” directory from the Python egg into my project’s root and followed the rest of the instructions.
I also used a couple of suggestion from the comments (comment [5] by “bneijt”) who put the template code in its own class (class TemplateRendering) that your handlers inherit from. I changed it up a bit to suit my needs. I have it return the result instead of writing it to the response (as the name of the method indicated). I did this so I could render a small template like a menu or breadcrumbs and pass the rendered result to another template. I also renamed the method “render_template” from “render_to_response” as it doesn’t write to the response anymore. My code is below but you should check out the original too.
class TemplateRendering:
"""TemplateRendering
A simple class to hold methods for rendering templates.
"""
def render_template(self, template_name, variables):
"""render_template
Returns the result of template.render to be used elsewhere. I think this will be useful to render templates to be passed into other templates.
Gets the template directory from app settings dictionary with a fall back to "templates" as a default.
Probably could use a default output if a template isn't found instead of throwing an exception.
"""
template_dirs = []
if self.settings["templates"] and self.settings["templates"] != '':
template_dirs.append(os.path.join(os.path.dirname(__file__),
self.settings["templates"]))
template_dirs.append(os.path.join(os.path.dirname(__file__), 'templates')) # added a default for fail over.
env = Environment(loader = FileSystemLoader(template_dirs))
try:
template = env.get_template(template_name)
except TemplateNotFound:
raise TemplateNotFound(template_name)
content = template.render(variables)
return content
Of course, if you’re using Tornado you’ll need to adjust accordingly if you haven’t already.
I changed the body of the MainHandler get() method to:
class MainHandler(BaseHandler, TemplateRendering):
...
def get(self, *args):
...
# a bunch of variables to pass into the template
variables = {'message':'Hello, Tornado!', 'page_title': self.settings["page_title"], 'page_description': '', 'page_author': 'The author of the page.', }
content = self.render_template('index.html', variables) # remember, I changed the name of the method from "render_to_response"
self.write(content)
If you’ve noticed “self.settings[“page_title”]” in the code samples above and thought I’d missed telling you something, you have a good eye. This is how you access entries from the settings dictionary. The only problem here is that I haven’t told you to add any entries yet so let’s go ahead and do that.
settings {
"page_title" : "It's a Tornado, Toto!",
"templates" : "views",
"xsrf_cookies": False,
}
Since this project is to set up a starting point for development, let’s use the HTML5 Boilerplate as a base for the templates. Create a “views” or “templates” folder in the root of your project (make sure the name matches the “templates” entry in your settings dictionary) and copy the index.html file there. We’ll deal with styles, JavaScript, and all that jazz in the next section.
If we ran the project now, “Hello, Tornado!” would not be displayed as we haven’t defined where in index.html to output the message variable. So, open index.html and, wherever you want the output to show up, add the following template code:
If you want you can wrap it in an HTML element:
<div class="message">{{ message }}</div>
Now try it out and you should see your message!
CSS, JS, and Other Static Content
Now that we have the templates in place, I think this is a good time to bring up static content such as CSS, Javascript, and Images and how to map or link to them in your templates. If you’re still using the default app.yaml created by App Engine, all requests to the server are routed to main.py. You could set up routes in your application to accommodate these files but I think it’s simpler if we just use use app.yaml and leave the application stuff to main.py. Google has some great documentation on app.yaml in their documentation: Python Application Configuration. To get you started, below is what I usually include:
- url: /static/
static_dir: static
- url: /robots\.txt
static_files: static/robots.txt
upload: static/robots.txt
- url: /favicon\.png
static_files: static/images/favicon.png
upload: static/images/favicon.png
- url: /apple-touch-icon\.png
static_files: static/images/apple-touch-icon.png
upload: static/images/apple-touch-icon.png
# JavaScript
- url: /js/(.*\.js)
static_files: static/js/\1
upload: static/js/(.*\.js)
# StyleSheets
- url: /css/(.*\.css)
static_files: static/css/\1
upload: static/css/(.*\.css)
# Images
- url: /(.*\.(gif|png|jpg|jpeg))
static_files: static/images/\1
upload: static/images/(.*\.(gif|png|jpg|jpeg))
This is all above the entry for main.py. Anything that doesn’t match the special entries above get routed to your application where webapp, Tornado, or whatever framework you’re choosing to use can deal with it appropriately (e.g., a 404 or what have you).
- url: .*
script: main.py
These new entries in app.yaml allow you to organize all your static files in a directory structure like so:
/static/css
/static/images
/static/js
I prefer to keep the root directory as clean as possible so I like organizing all the static files in a “static” directory in the root of the project but I think the URLs are cleaner if they don’t have static in them so the these new entries in app.yaml allow for that.
If you want to link to your css in ~/static/css/, use:
<link rel="stylesheet" href="/css/style.css">
To link to your javascript files in ~/static/js/, use:
<script src="/js/scripts.js"></script>
I’ll let you guess the rest.
The “/” in the front is necessary with how it’s setup. If you used the code from the HTML5 Boilerplate, make sure the leading “/” is present.
Now, you can copy all the CSS and Javascript from the Boilerplate to the static folder in your project and test it out. For the example application, I also used the 1140 grid from cssgrid.net. Pretty cool.
Markup Languages – reStructuredText/Markdown
reStructuredText
Syntax: http://docutils.sourceforge.net/docs/user/rst/quickref.html
The reStructuredText (or rst for short) markup language is used to standardize the documentation strings (docstrings; the triple-quoted strings) to document code in Python but has also been used to write technical books and can be converted to practically any format you’d want (e.g., XML, HTML, LaTeX, ODF, etc.).
Adding and using reStructuredText/Docutils to your project
I got the instructions for how to incorporate docutils into App Engine from Andi Albrecht’s blog: http://andialbrecht.wordpress.com/2008/08/14/using-restructuredtext-on-app-engine/ but I have detailed the instructions below since he just lists the command-line commands he uses and may throw some people off.
Download the docutils-snapshot.tgz and uncompress it resulting in a docutils folder. From inside that folder is another docutils folder. This is the one you want to copy to the root of your project. You will also want roman.py from the extras folder. You also place this in the root of your project.
To use docutils (or at least the part we need) you need to import publish_parts from docutils.core.
from docutils.core import publish_parts
To make things a little cleaner, I put most of the rst stuff in it’s own method in the BaseHandler class. If you remember why we created the BaseHandler, this will allow any of the handlers that extend BaseHandler to call this new method like so: self.render_rst(rst_to_process).
def render_rst(self, rst):
parts = publish_parts(source=rst, writer_name='html4css1',
settings_overrides={'_disable_config': True})
return parts['html_body'] # was 'fragment'
Note: When was was testing this, I noticed that 'fragment' doesn't return any headings unless they are the only thing you input. This is probably what you want for most applications where the title will also be in a textbox input, but for demonstration purposes, I output all the html. If you want to see all the possibilities (including css), just loop through the *parts* dictionary and output the key names.
Pygments – Syntax Highlighting Code Blocks
If you’re hoping to create your own simple blog or plan on having coding samples displayed, a syntax highlighter will make your life much easier and save time and sanity. Enter Pygments.
To get working with Pygments is pretty easy.
Download Pygments from https://bitbucket.org/birkenfeld/pygments-main/src. Then on the right side of the page, there’s a “get source” menu next to a red heart with a little green plus sign. From the “get source” menu you can select what kind of archive you want to download.
The reStructuredText directive comes with the distribution (see http://pygments.org/docs/rstdirective/). It’s in external and called rst-directive.py. You just need to copy it to /docutils/parsers/rst/directives directory in your project. I also renamed rst-directive.py to sourcecode.py since the directive is called “sourcecode”.
Now add the import at the top of main.py.
import docutils.parsers.rst.directives.sourcecode
This registers the new sourcecode directive with docutils so now if you do something like this:
.. sourcecode:: python
def a_function(self):
"""a_function is the simplest function ever! it does nothing!"""
pass
…it’ll get processed with Pygments and spit out properly.
Markdown and Pygments
Syntax: http://daringfireball.net/projects/markdown/syntax
Markdown is another markup language similar to reStructuredText. It is probably simpler to learn as it uses formatting you may already be familiar with as it derives heavily from the formatting of text in text-only email that’s developed naturally over the years. reStructuredText on the otherhand was created to be simple and powerful and has been used to write books. Use the one that best fits the bill.
To integrate Markdown and Pygments into the project, there are very nice instructions in part two of an article by Joey Bratton at http://www.joeyb.org/blog/2009/06/03/django-based-blog-on-google-app-engine-tutorial-part-2 under the heading: “Markdown and Pygments”.
If you have already integrated Pygments into your project (from the previous section) you should only need to read the first paragraph on Markdown from Joey Bratton’s article.
As the article deals mostly with Django and is not necessary for what I’m talking about (though you may want to look into it later) I’ve extracted from the article the few lines of code you do need.
Of course, you’ll need to import Markdown:
And then, below, is the code to use it. I have mirrored what I did with rst and created a function to keep the code in the handlers light. I have also added an HTML wrapper around the output to keep the HTML structure similar to that of the docutils rst output. This is of course to make it easier to style.
def render_md(self, md):
"""render_md(self, md)
self - BaseHandler
md - raw markdown text
"""
markdown_processor = markdown.Markdown(extensions=['codehilite'])
html = markdown_processor.convert(md)
html = '<div class="document">'+html+'</div>'
return html
Putting it all together.
If you’ve been following along, you should now have the code in place for using the Tornado framework, Jinja2 templates, the ability to process both reStructuredText and the Markdown markup languages into HTML, and use Pygments to markup code examples for easy styling using either markup language. Only Pygments relies on other software (either Docutils for reStructuredText or Markdown for Python for…Markdown), so if you don’t like Tornado and would prefer to use a different framework or are already comfortable with say the Cheetah template system, use what you’re comfortable with.
To use all the pieces in this article, I created an application where someone can select a markdown language (rst or Markdown) and in a textarea, type in the markup language they’ve chosen. When they submit, it spits out the HTML result back out on the page. Nothing fancy, but I used it to convert this document from Markdown to HTML.
I hope this helps you create applications more efficiently or at least gives you a taste of Python and App Engine.
Resources
Here is a list of resources used linked throughout this document:
Example Application and Code:
XSRF/CSRF (Cross-site Request Forgeries)
Since XSRF/CSRF attacks can be pretty easy to implement (bad) but also, especially with Tornado, easy to defend against (good) I figured it would be a good idea to link to an article about preventing XSRF/CSRF attacks. Tornadoweb.org has all the documentation for defending against XSRF/CSRF attacks you’ll need (as long as you’re using Tornado). If you’re planning on not using Tornado, look through the documentation or for an implementation you can use for your particular framework.
Profiling, Appstats, and FirePython/FireLogger
Here are several articles on Profiling, Appstats and using FirePython. Profiling is about collecting data about your application as it runs; the function calls, the memory usage, the time it takes for things to execute. It can be very helpful when you are tracking reasons why your application is slow, unresponsive, or crashing.
Other Frameworks
The Tipfy (a framework written specifically for Google App Engine) wiki has a list of frameworks that work (modified or are created specifically for use) on Google’s App Engine: Tipfy wiki – App Engine Frameworks
Installing Python 2.5 for Mac OS X 10.6
Mac OS X Snow Leopard comes with Python 2.6 installed by default. Google App Engine uses 2.5 and to prevent the differences between the two versions from befuddling you and getting in the way of the fun of programming, you should install Python 2.5.x.
Unfortunately, it has been a while so I don’t recall exactly how I did it, but I believe I used MacPorts http://www.macports.org/install.php, which should make it super easy. You do need to have Apple’s Xcode Developer Tools installed, which is a rather large download, so start now. If you still have the disks that came with your Mac, you can get recent enough versions from the extras disk.
Then, after you’re finished installing Python 2.5, you can switch the version you’re using so you’re not stuck using the old version for other software.
If you’re using the GUI Google App Engine Launcher, go to preferences and tell it to use python in /usr/bin/python2.5 (or wherever you’ve installed it). You can also use the other option below if you do it before starting Google App Engine Launcher (or if you restart Google App Engine Launcher afterwords).
If you use the dev_appserver.py command from Terminal, you can switch the version by issuing this command in Terminal:
defaults write com.apple.versioner.python Version 2.5
Got that from StackOverflow: http://stackoverflow.com/questions/1380281/set-snow-leopard-to-use-python-2-5-rather-than-2-6
Notes
Code Styles
I apologize about the code styles not coming out colored correctly. WordPress.com doesn’t let you include inline styles or include custom styles without paying more, which I didn’t anticipate.
Pre Styles
One thing I did notice once I got Pygments working was that in the setup I created (with the html5boilerplate), the styles for the pre element used for code output made the text extend outside of the designated area instead of wrapping as I’d have liked (and as the styles tried to indicate).
Around line 55 of styles.css (for the HTML5 Boilerplate) you’ll see a style like-a-this-a:
pre {
padding: 15px;
white-space: pre;
white-space: pre-wrap;
white-space: pre-line;
word-wrap: break-word;
}
There may be a reason for defining “white-space” three times but it doesn’t seem to work in Chrome, Safari, and Firefox. Or they are in the wrong order? Looking at the comments for the html5boilerplate Github page, https://github.com/paulirish/html5-boilerplate/issues/#issue/294 it looks like there may very well be a problem. So I have changed it below to “fix” it but haven’t had an opportunity to test it thoroughly.
pre {
padding: 15px;
word-wrap: break-word;
white-space: pre-wrap;
}
The Auri Group Tech Blog 