Planet TurboGears

Alessandro Molina: Mastering the TurboGears EasyCrudRestController

Tue, 31 Jan 2012 20:22:06 +0000

One of the key features of TurboGears2 is the great CRUD extension. Mastering the CRUD extension can really make the difference between spending hours or just a few minutes on writing a web app prototype or even a full application.

The CRUD extension provides two main features, the CrudRestController which is meant to help creating totally custom CRUDs and the EasyCrudRestController which provides a quick and easy way to create CRUD interfaces.

I’ll focus on the EasyCrudRestController as it is the easiest and more productive one, moving forward to the CrudRestController is quite straightforward after you feel confident with the Easy one.

The target will be to create, in no more than 40 lines of controller code, a full featured photo gallery application with:

Multiple Albums
Uploads with Thumbnails Generation
Authenticated Access, only users in group “photos” will be able to manage photos
Contextual Management, manage photos of one album at time instead of having all photos mixed together in a generic management section

If you don’t already know how to create a new TurboGears project, start by giving a look at TurboGears Installation for The Impatient guide. Just remember to add tgext.datahelpers to dependencies inside your project setup.py before running the setup.py develop command.

I’ll start by providing a Gallery and Photo model. To store the images I’ll use tgext.datahelpers to avoid having to manage the attachments. Using datahelpers also provides the advantage of having thumbnails support for free.

from tgext.datahelpers.fields import Attachment, AttachedImage
 
class Gallery(DeclarativeBase):
    __tablename__ = 'galleries'
 
   uid = Column(Integer, autoincrement=True, primary_key=True)
   name = Column(Unicode(100), nullable=False)
 
class Photo(DeclarativeBase):
    __tablename__ = 'photos'
 
    uid = Column(Integer, autoincrement=True, primary_key=True)
    name = Column(Unicode(100), nullable=False)
    description = Column(Unicode(2048), nullable=False)
    image = Column(Attachment(AttachedImage))
 
    author_id = Column(Integer, ForeignKey(model.User.user_id)))
    author = relation(app_model.User, backref=backref('photos'))
 
    gallery_id = Column(Integer, ForeignKey(Gallery.uid))
    gallery = relation(Gallery, backref=backref('photos', cascade='all, delete-orphan'))

Now to be able to start using our galleries we will have to provide a place where to view them and a gallery management controller to create and manage them. Viewing them should be quite straightforward, I’ll just retrieve the galleries from the database inside my index method and render them. To access a single gallery I’ll rely on the datahelpers SQLAEntityConverter which will retrieve the gallery for us ensuring it exists and is valid. For the management part I’ll create an EasyCrudRestController mounted as /manage_galleries

from tgext.crud import EasyCrudRestController
 
class GalleriesController(EasyCrudRestController):
    allow_only = predicates.in_group('photos')
    title = "Manage Galleries"
    model = model.Gallery
 
    __form_options__ = {
        '__hide_fields__' : ['uid'],
        '__omit_fields__' : ['photos']
    }
 
class RootController(BaseController):
    manage_galleries = GalleriesController(DBSession)
 
    @expose('photos.templates.index')
    def index(self, *args, **kw):
        galleries = DBSession.query(Gallery).order_by(Gallery.uid.desc()).all()
        return dict(galleries=galleries)
 
    @expose('photos.templates.gallery')
    @validate(dict(gallery=SQLAEntityConverter(Gallery)), error_handler=index)
    def gallery(self, gallery):
        return dict(gallery=gallery)

Logging in with an user inside the photos group and accessing the /manage_galleries url we will be able to create a new gallery and manage the existing ones.

To configure how the crud controller forms should appear and behave the __form_options__ property of the EasyCrudRestController can be used. This property relies on the same options as Sprox FormBase and customizes both the Edit and Add forms.
The next part is probably to be able to upload some photos inside our newly created galleries. To perform this we will create a new EasyCrudRestController for gallery photos management.

from tgext.crud import EasyCrudRestController
from tw.forms import FileField
from tw.forms.validators import FieldStorageUploadConverter
from webhelpers import html
 
class PhotosController(EasyCrudRestController):
    allow_only = predicates.in_group('photos')
    title = "Manage Photos"
    model = model.Photo
    keep_params = ['gallery']
 
    __form_options__ = {
        '__hide_fields__' : ['uid', 'author', 'gallery'],
        '__field_widget_types__' : {'image':FileField},
        '__field_validator_types__' : {'image':FieldStorageUploadConverter},
        '__field_widget_args__' : {'author':{'default':lambda:request.identity['user'].user_id}}
    }
 
    __table_options__ = {
        '__omit_fields__' : ['uid', 'author_id', 'gallery_id', 'gallery'],
        '__xml_fields__' : ['image'],
        'image': lambda filler,row: html.literal('‹img src="%s"/›' % row.image.thumb_url)
    }

Mounting this inside the RootController as manage_photos = PhotosController(DBSession) it will be possible to upload new photos inside any gallery. To manage the photos inside the first gallery for example we will have to access /manage_photos?gallery=1url.

Each parameter passed to the EasyCrudRestController is used to filter the entries to show inside the management table and the keep_params option provides a way to keep the filter around. This makes possible to edit the photos of only one gallery at the time instead of having all the photos mixed together. Also when a new photo is created it will be created in the current gallery.

The PhotosController got more customization than the GalleriesController, through the __field_widget_types__ and __field_validator_types__ options we force the image field to be a file field and using the __field_widget_args__ we ensure that the newly uploaded photos have the current user as the author.

__table_options__ provide a way to customize the management table. The available options are the same as the Sprox TableBase and Sprox TableFiller objects. in this case we hide the indexes of the rows on the database and the gallery itself, as we are managing the photos of a specific gallery we probably don’t need to know which galleries the photos belong to. Using the __xml_fields__ we also specify that the image field provides HTML and so doesn’t have to be escaped. The image entry forces the table to show the image thumbnail for the image column of the table instead of printing the AttachedImage.__repr__ as it would by default.

At first sight it might sound a bit complex, but once you start feeling confident, the CRUD extension makes possible to create entire applications in just a bunch of code lines. With just a few lines of code we created a photo gallery with multiple albums support and we can now focus on the index and gallery templates to make the gallery as pleasant as possible for our visitors.

The complete implementation of the photo gallery is available as a pluggable application on bitbucket, feel free to use it in your TurboGears projects.

Mengu Kagan: 2011 At A Glance

Sat, 31 Dec 2011 19:07:11 +0000

I can call 2011 quite a good year on my behalf however when I look at things I've done, I feel I've done much less than I could. Here is a list of what I have done in 2011.

Things I've Done in 2011

Executed an idea we had with my brother, http://compector.com.
Built the start-up above with Ruby on Rails so I became more familiar with Ruby world.
I have made the switch to PostgreSQL from MySQL.
Got a job in one of Turkey's top web sites where we deal with thousands of concurrent users...

Mengu Kagan: Subqueries With SQLAlchemy

Wed, 21 Dec 2011 20:28:11 +0000

I have been developing the new version of www.osesturkiye.com for the Turkish version of the show called "The Voice". It's already built with TurboGears, mako and SQLAlchemy. In the new version we have a gallery and many photos in it. My SQLAlchemy model is like this:

class GalleryPhoto(DeclarativeBase):
    __tablename__ = 'gallery_photo'

    id = Column(Integer, primary_key=True)
    photo_image = Column(UnicodeText)
    photo_description = Column(UnicodeText)
    dateline = Column(DateTime, default=datetime.now)

The problem I have is, in the photo detail page I will display "Previous" and "Next" links. I can implement this with two ways..

Ralph Bean: threebean

Tue, 13 Dec 2011 05:44:53 +0000

Tonight, VooDooNOFX was asking in IRC in #turbogears how to disable the injection of jquery.js by tw2.jquery into her/his TG2 app. Using the inject_resources=False middleware config value wouldn’t cut it, since she/he wanted tw2 to inject all other resources, they were loading jQuery via google CDN beforehand and tw2′s injection was clobbering their code.

I came up with the following hack to myapp/lib/base.py which will remove tw2.jquery.jquery_js from the list of resources tw2 would inject into each page served by a TG2.1 app.

At the top of myapp/lib/base.py import:

import tw2.core.core
import tw2.jquery

and then replace:

        return TGController.__call__(self, environ, start_response)

with the following:

        stream = TGController.__call__(self, environ, start_response)

        # Disable the injection of tw2.jquery
        offending_link = tw2.jquery.jquery_js.req().link
        local = tw2.core.core.request_local()
        local['resources'] = [
            r for r in local.get('resources', None) if r.link != offending_link
        ]

        return stream

The two tricks to this are

Simply knowing that tw2 resources register themselves with the ‘request_local’ object and that during the return-phase of the WSGI pipeline, the tw2 middleware refers to that list when injecting resources
Figuring out where in a TG2 app’s request flow to place the call to alter that object after all widgets that might register jquery have declared their resources but before the resources list is injected into the output stream.

We came out of it with a bug filed in the tw2 issue tracker so we can take care of it properly in the future.

Michael Pedersen: Announcing TurboGears 2.1.4 Release

noreply@blogger.com (Michael Pedersen) — Mon, 12 Dec 2011 20:18:23 +0000

The TurboGears team is proud to announce the release of TurboGears 2.1.4!

This release has many new features and bugfixes, all of them listed below. The most important one, though, is that this is the final 2.1.x release, and the final release that will support Python 2.4.

Please make sure to update your code to work on Python 2.5, as that will be the next supported Python version.

I would like to take a moment to thank Alessandro Molina especially. His work made this release possible, and as large as it is. He's been a great asset to the team, and I'm glad to have him on board.

Deprecated Python 2.4 support. In 2.2, Python 2.4 support will be removed, and only 2.5-2.7 will be supported
TurboGears extension architecture enhanced. Better support for hooks before and after configuration
Performance enhancements in tg module and in default templates
Enhancements to lazy_url
WebOb version locked in. The change of virtualenv to "distribute" by default has broken dependency_links and allow_hosts in the config files, and this works around that issue.
Jinja2 filters are now automatically loaded
Work arounds for best_match, which was not producing the expected behavior with IE7 and IE8
Add "auto_reload_template" as an .ini option
Performance tuning the default size of the Genshi cache
Added Genshi advanced i18n support
Better compatibility with SQLAlchemy 0.7
Changed default quickstart options to help ensure that some model is always available
Documentation enhancements
Nested RestControllers now work as expected (no longer required to implement "_custom_actions")

Curia: An rsync-like interface for Amazon S3 and Google Storage

Mon, 12 Dec 2011 00:55:06 +0000

Every so often I find myself working on that odd job that requires syncing files with Amazon’s S3. In the beginning, I tried some of the various S3 FUSE interfaces—hoping for something that would play nice with rsync—but FUSE’s stability always left something to be desired and more often than not I’d be left with that one transfer that never would quite finish correctly.

Eventually I discovered boto and settled in to using a crude (yet stable) Python/boto solution for these type of tasks—all the while wondering why nobody took the time to write a “real” rsync-like client for S3.

Well, this last time around I finally decided to stop whining and take matters into my own hands. After a couple of late nights fleshing out my basic boto solutions, I’m happy to announce what I’m calling “boto rsync”—an rsync like wrapper for boto’s cloud storage interfaces (S3 and Google Storage).

Please take a look at the project on github and let me know what you think: http://github.com/seedifferently/boto_rsync

Mengu Kagan: What's Going On With TurboGears?

Sat, 19 Nov 2011 23:55:28 +0000

If you are using Python and looking for a web framework or if you are just looking to do web programming with Python, you will look out your options and among Django, web2py and pyramid you will notice TurboGears. As a Python developer and TurboGears user, I will let you guys know what is going on with TurboGears lately and will let you have a chance with TurboGears.

A Short Introduction First

TurboGears is a full stack MVC web framework that is built on top of Pylons and which includes an ORM, a templating system, database migration tool, web helpers, authentication and authorization system by default just like Django. If you are wondering, there are two main differences between Django and TurboGears.

TurboGears is using class based controllers and object dispatch.
TurboGears is built using external Python libraries such as Pylons, WebOB, paster whereas everything in Django is built for Django.

Alessandro Molina: TurboGears2 DebugBar

Sat, 12 Nov 2011 19:30:47 +0000

Recently some work has been done to extend the hooks support in TurboGears, to play a little with the new hooks I decided to try creating the famous and envied Django Debug Toolbar. I’m quite happy of the result and most of the features are there. In a few days I’ll be able to place it on a public repository and I’ll release it concurrently with the 2.1.4 release of TurboGears.

Debug Toolbar

Timings

Request and Headers

SQLAlchemy Queries

Mounted Controllers

The code has been heavily inspired by the Pyramid Debug Toolbar and have to thank the Pyramid team for the good job they did at making the Toolbar code clean and simple.

Cliff Wells: FreeSWITCH on Scientific Linux 6.1

Sun, 23 Oct 2011 15:27:44 +0000

SL 6.1 (and I assume RHEL and CentOS 6.1 as well) has introduced an issue for building and running FreeSWITCH. Apparently a lot of stuff now relies on dynamically linking to libnss3. libnss3, in turn, depends on libnspr4.so, which depends on libplds4.so. Seemingly, this should not be an issue (stuff depends on chained shared objects all over the place), but somehow it is.

What happens is first you can't compile FreeSWITCH. You get complaints about unresolved symbols in /usr/lib64/libnss3.so. The solution is to run the following commands:

yum install nspr-devel
env LIBS="-lnss3 -lnspr4 -lplds4" ./configure
make && make install

This will get you a compiled version of FreeSWITCH. However, when you actually run it, you'll find that several modules won't load at runtime (including the ODBC driver, should you happen to be using it). The solution for this is similar. Assuming you are using an init script to launch FreeSWITCH, you can add the following line to the top of /etc/init.d/freeswitch:

export LD_PRELOAD="/usr/lib64/libnss3.so /usr/lib64/libnspr4.so /usr/lib64/libplds4.so"

Voila. Everything works. Hopefully the FreeSWITCH devs get on RHEL6 support soon, but meanwhile this should get you by.

Michael Pedersen: Request For Ideas for Hiring Pond

noreply@blogger.com (Michael Pedersen) — Tue, 18 Oct 2011 20:44:25 +0000

So, a favor to ask of people: I'm working on a web application to help people manage their resumes. As I've gotten further in, I've realized I don't have an actual todo list for it. So, since I'm making this to be used by others, I'll ask everybody here:

What would you want to see? Currently, I've added enough code to allow the program to output something very close to my online resume ( http://www.icelus.org/ ). Next up, I have the following features on my todo list already:

add docutils to output strings where appropriate (everywhere?)
add login
add ability to edit own resume
add ability to compile new custom resumes
add ability to output PDF
add ability to output ODT
add ability to output DOC
add generic logo
add generic photo / head (shadow) shot
add ability to login using Google, Yahoo, Facebook, etc
add ability to send resumes directly from Hiring Pond
add ability to track where resumes have gone
add ability to track job hunt progress
add ability to display current employment status (employed, unemployed, contracting)
add ability to display current employment seeking status (not at all, active, passive)
add logo to qrcode
change colors of qrcode
change color scheme for all of hiring pond to sunset and water
add fisherman silhouette as hiring pond logo

What else would you all want to see in order to make you want to use this?

Alessandro Molina: TurboGears2 Performance Improvements

Tue, 04 Oct 2011 15:35:48 +0000

As recently some effort has been involved in improving the performances of TurboGears2, I was curious to see how much things improved. As usually, the test isn’t really reliable in any way and was just for fun.

All the graphs report the request/sec the application has been able to perform on my computer with only 1 concurrent client. So higher is better.

Here is the comparison between TG2.0 and TG2dev (will be 2.1.4)

I also compared various setups with different template engines on TG2dev

The comparison happened on an application similar to the quickstarted one.
Actually as there is no database involved in this application the template engine impacts a lot and so was a good benchmark for the template engines themselves.

Ralph Bean: threebean

Mon, 03 Oct 2011 20:35:34 +0000

It’s done. An over-engineered WSGI middleware component that adds a velociraptor to every page served. Fact: Every WSGI app is better with a raptor.

It’s called raptorizemw (pronounced “awesome”) and the only way to use it is in production.

Michael Pedersen: Announcement: TurboGears2 2.1.3 Released!

noreply@blogger.com (Michael Pedersen) — Wed, 28 Sep 2011 21:22:07 +0000

We are pleased to announce the latest release of TurboGears, 2.1.3!

This release adds support for the Mongo object database and the Ming ORM for it, in addition to bringing in some small bugfixes. The full changelog is below

The number of changes is, admittedly, small, but don't underestimate the importance: NoSQL databases are an important tool for the web, and adding the support for Mongo brings TurboGears into the group of frameworks that not only supports it, but supports it well.

We're already looking forward to expanding support and closing even more issues for 2.1.4, so we'll see you next month for that release!

* Added support for MongoDB into the quickstart template
* Added lurl (lazy_url) support for late evaluation
* Adding tests module back into source distribution. Solves ticket 115
* Template caching is now manageable by TG applications
* Clean up output for flash() in default quickstart
* Performance enhancements
* Fixed missing dependencies in TG quickstart on Python 2.4

Ralph Bean: threebean

Sat, 24 Sep 2011 16:01:21 +0000

I wrote an app that scrapes foreclosure data from my county of residence and plots it six ways from Sunday in a TurboGears2.1 app. You can find it at http://monroe-threebean.rhcloud.com/, hosted on redhat’s openshift cloud.

It’s used by activists with Take Back the Land, Rochester and my local branch of the ISO to find upcoming evictions before they happen and organize the neighborhoods to stop the shuttering of homes. Get a hundred people at the door of the house before the cops come, and no-one is getting evicted (we’ve had some successes).

We’re living in some absurd times where banks got bailed out by the trillions yet still get to collect on our student debt and mortgages. Most of us are being ruined. If you’re not, then your neighbor is.

If you’re in Boston, check out Vida Urbana

or if you’re in Chicago, check out the Chicago Anti-Eviction Campaign. Anywhere you go, check out the ISO.

Fork my code, port it to your home town, and start organizing!

Christpher Arndt: Hurra, hurra, der Herbst ist da!

Thu, 22 Sep 2011 20:38:46 +0000

Warum man zur Zeit beim Spazierengehen immer große Taschen mitnehmen sollte...

Michael Pedersen: TurboGears and Ming/Mongo

noreply@blogger.com (Michael Pedersen) — Sun, 18 Sep 2011 21:29:30 +0000

I know that my followers are few, and my posts are too infrequent. I'm looking for ways to change my posting frequency.

But, even still, I have an important request: TurboGears 2.1.3 is getting ready to be released. We're adding Ming/Mongo support in this release. Anybody who can, please review the changes (core, devtools, and docs), run tests, etc. We need to make sure we've got this as correct as possible for everybody.

Thank you for your time!

Curia: The long overdue “Shootout” update

Mon, 12 Sep 2011 19:08:06 +0000

It’s been several months since I’ve had a chance to update The Great Web Framework Shootout, but this weekend I decided that it was time to dig in and freshen things up a bit.

Not only have most of the frameworks seen new releases since the last revision, but I finally decided to move all of the tests over to Amazon’s “release” version of the Ubuntu LTS AMI.

Below is a quick summary of what’s new in this revision:

All tests were performed on the updated Ubuntu LTS AMI (ami-fbbf7892 ubuntu-images-us/ubuntu-lucid-10.04-amd64-server-20110719.manifest.xml)
The updated AMI was configured with Python 2.6.5, PHP 5.3.2, Ruby 1.9.2p290, Apache 2.2.14 (default config), mod_wsgi 2.8 (embedded mode), and mod_passenger 3.0.9
Rails 2.x and 3.0 were dropped from the “full stack(ish)” tests in favor of Rails 3.1.
CakePHP 1.2 was dropped from the PHP tests in favor of 1.3, but Symfony and Yii were added as they seem to have considerable market share.
CakePHP’s caching engine was incorrectly configured during the last round of tests, and this has been corrected.

Circle me on Google+ to keep track of further updates, and feel free to contact me there with any questions or comments.

Michael Pedersen: Announcement: TurboGears2 2.1.2 Released!

noreply@blogger.com (Michael Pedersen) — Wed, 24 Aug 2011 20:58:33 +0000

We are pleased to announce the latest release of TurboGears, 2.1.2!

This is a maintenance/bugfix release for the 2.1 series which also adds some new features. The full changelog is below.

Highlights for this release include the beginnings of our new book "TurboGears2 Web Applications: Batteries Included", Python 2.7 support, improved Jinja2 support, and quite a few bug fixes.
You can see the book as it stands right now at http://www.turbogears.org/book/ . We're working towards getting more details and more of the older documentation updated to go into the book.

We took longer to reach this release milestone than we had planned, but Real Life(tm) takes its toll on everybody during the summer months, it seems.

Even still, I can speak for the whole team when I say that we look forward to continuing to serve the community with quality code, and we look forward to your continued support!

Full Change Log Here:

For TG2.x Core:

Improved content-type handling ( https://sourceforge.net/p/turbogears2/tickets/73/ )
Fixed dotted filename support for Genshi and Chameleon.Genshi
Removed lxml dependency; now preparation of virtualenv is much easier on 64 bit systems
Upgraded Chameleon.Genshi support to Chameleon1.3rc
Set up render_genshi on demand only, improves Google App Engine support
Support for Python 2.7
Improved Jinja2 and Jinja2 filters support ( https://sourceforge.net/p/turbogears2/tickets/107/ )
Added option to prevent auto-saving sessions ( https://sourceforge.net/p/turbogears2/tickets/83/ )
Fixed crash where multiple templates were registered and user called override_template ( https://sourceforge.net/p/turbogears2/tickets/82/ )
Fixed problem with ":" in error messages ( https://sourceforge.net/p/turbogears2/tickets/86/ )
Improved i18n support
Improved pagination support ( https://sourceforge.net/p/turbogears2/tickets/99/ )
Fixed warning resets when in debug mode ( https://sourceforge.net/p/turbogears2/tickets/84/ )

For TG2.x Devtools:

Improved Genshi support in quickstart template
Improved content-type handling ( https://sourceforge.net/p/turbogears2/tickets/73/ )
Improved bootstrap if database is under version control
Changed default password hashing to SHA256 ( https://sourceforge.net/p/turbogears2/tickets/94/ )
New "--minimal" quickstart template
Separated password hashing mechanism to make it more easily usable ( https://sourceforge.net/p/turbogears2/tickets/67/ )
Fixed Unicode return type in auth.py ( https://sourceforge.net/p/turbogears2/tickets/89/ )
Added missing dependencies for Python 2.4 and the quickstart app ( https://sourceforge.net/p/turbogears2/tickets/100/ )

For TG2.x Docs:

Introduced new book beginnings
Rewrote 20 Minute Wiki, brought it forward to working with TG2.1.2
Rewrote installation instructions, much simplified
Began description of new application for tutorial/book
Improved documentation on how to contribute
Made all standard modules autodocumented in the appendices
"ignore_parameters" documented ( https://sourceforge.net/p/turbogears2/tickets/15/ )
"tg_avoid_touch" (method to avoid saving sessions) documented ( https://sourceforge.net/p/turbogears2/tickets/83/ )
Updated references regarding _default and _lookup ( https://sourceforge.net/p/turbogears2/tickets/5/ )
Documented custom filters for Jinja2
"migrate" command is now documented ( https://sourceforge.net/p/turbogears2/tickets/106/ )
Documented pagination parameters
Documented "--minimal" quickstart option ( https://sourceforge.net/p/turbogears2/tickets/102/ )
Changed references to "aptitude" to "apt-get" ( https://sourceforge.net/p/turbogears2/tickets/104/ )

Ralph Bean: You should be all gravy.

Tue, 19 Jul 2011 16:24:24 +0000

Often, you will need to authenticate against ldap in your webapp. Here’s how to make that happen in a freshly quickstarted TurboGears 2.1 app.

Setting up your environment

mkvirtualenv --no-site-packages repoze-ldap-app
pip install tg.devtools
paster quickstart   # call the app repoze-ldap-app, yes to mako and auth
cd repoze-ldap-app
python setup.py develop
pip install genshi  # This is a workaround.
paster setup-app development.ini
paster serve development.ini  # To test if the basic app works.

Point your browser at http://localhost:8080 just to make sure everything is cool.

Setting up `repoze.who.plugins.ldap`

Add the following line to the install_requires list in setup.py:

    "repoze.who.plugins.ldap",

Run python setup.py develop to install the newly listed repoze plugin.

Add the following four lines to development.ini which reference an as yet unwritten secondary configuration file. Place them just above the sqlalchemy.url=... lines:

# Repoze.who stuff
who.config_file = %(here)s/who.ini
who.log_level = INFO
who.log_stream = stdout

Create a new file who.ini with the following contents:

# This file is adapted from:
# http://threebean.wordpress.com/2011/07/19/using-repoze-who-plugins-ldap-in-a-turbogears-2-1-app/
# which has been adapted from:
# http://static.repoze.org/whodocs/#middleware-configuration-via-config-file
# which has been adapted from:
# http://code.gustavonarea.net/repoze.who.plugins.ldap/Using.html

[plugin:friendlyform]
use = repoze.who.plugins.friendlyform:FriendlyFormPlugin
login_form_url= /login
login_handler_path = /login_handler
logout_handler_path = /logout_handler
rememberer_name = auth_tkt
post_login_url = /post_login
post_logout_url = /post_logout

[plugin:auth_tkt]
use = repoze.who.plugins.auth_tkt:make_plugin
secret = omg_this_is_so_secret_lololololol_2938485#butts

[plugin:ldap_auth]
# Here I use my own ldap_auth, since by default ldap allows affirmative
# authentication with *no password specified*.  That is lame; I override it.
use = repozeldapapp.lib.auth:ReconnectingAuthenticatorPlugin

# This is the URI of wherever you want to connect to.  I work at RIT.
ldap_connection = ldap://ldap.rit.edu

# This is the base of the 'distinguished names' (DNs) of persons in your
# particular LDAP instance.  It will vary from server to server.
base_dn = ou=People,dc=rit,dc=edu

[plugin:ldap_attributes]
# I also do some overriding for more security in how I get attributes for
# users.
use = repozeldapapp.lib.auth:ReconnectingLDAPAttrsPlugin
ldap_connection = ldap://ldap.rit.edu

[general]
request_classifier = repoze.who.classifiers:default_request_classifier
challenge_decider = repoze.who.classifiers:default_challenge_decider

[mdproviders]
plugins =
    ldap_attributes

[identifiers]
plugins =
    friendlyform;browser
    auth_tkt

[authenticators]
plugins =
    ldap_auth

[challengers]
plugins =
    friendlyform;browser

Create another new file repozeldapapp/lib/auth.py with the following contents:

from repoze.who.plugins.ldap import (
    LDAPAttributesPlugin, LDAPAuthenticatorPlugin
)
import ldap


class URISaver(object):
    """ Saves the ldap_connection str given to repoze authn and authz """
    def __init__(self, *args, **kw):
        self.uri = kw['ldap_connection']
        super(URISaver, self).__init__(*args, **kw)


class ReconnectingLDAPAttrsPlugin(LDAPAttributesPlugin, URISaver):
    """ Gets attributes from LDAP.  Refreshes connection if stale. """

    def add_metadata(self, environ, identity):
        """ Add ldap attributes to the `identity` entry. """

        try:
            return super(ReconnectingLDAPAttrsPlugin, self).add_metadata(
                environ, identity)
        except Exception, e:
            print "FAILED TO CONNECT TO LDAP 1 : " + str(e)
            print "Retrying..."
            self.ldap_connection = ldap.initialize(self.uri)
            return super(ReconnectingLDAPAttrsPlugin, self).add_metadata(
                environ, identity)


class ReconnectingAuthenticatorPlugin(LDAPAuthenticatorPlugin, URISaver):
    """ Authenticates against LDAP.

    - Refreshes connection if stale.
    - Denies anonymously-authenticated users

    """

    def authenticate(self, environ, identity):
        """ Extending the repoze.who.plugins.ldap plugin to make it much
        more secure. """

        res = None

        try:
            # This is unbelievable.  Without this, ldap will
            #   let you bind anonymously
            if not identity.get('password', None):
                return None
            try:
                dn = self._get_dn(environ, identity)
            except (KeyError, TypeError, ValueError):
                return None

            res = super(ReconnectingAuthenticatorPlugin, self).authenticate(
                environ, identity)

            # Sanity check here (for the same reason as the above check)
            if "dn:%s" % dn != self.ldap_connection.whoami_s():
                return None

        except ldap.LDAPError, e:
            print "FAILED TO CONNECT TO LDAP 2 : " + str(e)
            print "Retrying..."
            self.ldap_connection = ldap.initialize(self.uri)

        return res

Finally, do two things to repozeldapapp/config/middleware.py.

Edit it and at the top of the file add:

from repoze.who.config import make_middleware_with_config

Add the following inside the make_app(...) function, just below the comment line about Wrap your base TurboGears 2…, like so:

    # Wrap your base TurboGears 2 application with custom middleware here
    app = make_middleware_with_config(
        app, global_conf,
        app_conf['who.config_file'],
        app_conf['who.log_stream'],
        app_conf['who.log_level'])

Give it a test

Restart the paster server and reload http://localhost:8080. Try logging in as a user in your ldap instance and you should be all gravy.

You should be all gravy.

Mengu Kagan: Deploying TurboGears 2.1 Application With Nginx And uWSGI

Fri, 15 Jul 2011 03:03:30 +0000

I have developed a TurboGears 2.1 application and I was asked to deploy it today. I was planning to use Apache and mod_wsgi however on CentOS 5.6 both are painful. I was getting some errors that I really did not want to deal with so I have decided to use nginx and uWSGI. Great combination I believe. The thing is I had to deploy this application to 3 web servers as we are using a load balancer so I have created a deployment script to use on all 3 web servers but surely I won't let you figure out how to deploy from this script.

Let's cut the chase and get to the point.

Christpher Arndt: ELKA PM 13 MIDI Basspedal (Version B)

Tue, 12 Jul 2011 13:23:58 +0000

Beschreibung, Anleitung und Kurzreview Das ELKA EM 13 MIDI Basspedal ist ein reiner MIDI-Controller ohne Klangerzeugung und dafür ausgelegt, mit den Füßen gespielt zu werden, um beide Hände zum Gitarre- oder Keyboardspielen oder Sonstigem freizuhaben. Die Pedale umfassen eine Oktave, von C bis C, ansonsten hat das Gerät keinerlei Klangsteuerungsmöglichkeiten und auch nur einen zusätzlichen [...]

Curia: Google+: It’s a bigger deal than Facebook, but not like you think

Tue, 12 Jul 2011 01:53:41 +0000

I’ve been on Google+ since week 1 and while the initial mood has been overwhelmingly positive, I couldn’t help but notice the content of my stream becoming a bit skewed over the past few days as Google began opening it up more and more to “the laypeople.” Google was wise to restrict G+’s initial membership base to the tech-savvy, because we’re already on to Google about where it’s going with this thing; but now that the rest of the world is jumping onboard there seems to be a bit of confusion about what it’s good for.

In response to this, I thought I’d share a few of my own thoughts on G+, and why I believe it is a valuable and needed addition the online social ecosystem:

1. It’s not Facebook, and it’s not Twitter—it’s a bit of both, and the key is Circles.

What Twitter does best is giving busy professionals, celebrities, and business entities an outlet to interact publicly with their audience on a pseudo-personal level in 140 characters or less. What Facebook does best is giving people an outlet to interact somewhat privately with people they trust on as personal of a level as they like. For many, the separation between Twitter vs. Facebook mirrors their own separation of business vs. personal. For example: A picture of your 2-year old squeezing the cat is more likely to wind up posted to a limited audience on Facebook than publicly shared on Twitter, while a quick blurb updating anyone interested on a recent professional achievement often needs a more public forum such as Twitter to gain the visibility you want it to have. Even in each of their post boxes you can see the difference in what’s expected to be shared: Twitter asks “What’s happening?” expecting you to want the whole world to know, while Facebook asks “What’s on your mind?” which is quite a bit more personal.

What Google+ gives you is both (and much more), through the power of a brilliant feature called Circles. No longer is it a question of public vs. private, but rather the much more honest and straightforward ”Who do you want to share this with?” Through Circles, you now have complete control over who will and who wont see the content you are about to share. You may only want “close friends” to see how crazy that party was last night, while “friends” can be given access to pictures of your recent vacation to Hawaii, and the whole world can have access to that presentation you did last week that received a standing ovation.

(Note: Yes I am aware that Facebook offers something similar to Circles, but its integration is clumsy and inconvenient in comparison.)

2. For now, it may only be relevant to those who manage multiple online identities.

Sure, there are plenty of people who use Twitter and Facebook as practically the same outlet, and if you’re one of those then Google+ is probably going to be little more than a redundant nuisance—one more social network to have to keep track of.

Or perhaps you’re the Facebook-only or Twitter-only type and simply don’t see the point in jumping ship or picking up another service. I applaud you for your loyalty, but in this day and age resistance to “the big G” is futile. Which leads me to my final thought…

3. If you can look to the horizon, it might just be bigger than anything we’ve seen yet!

The real thing to keep in mind in these early days is that G+ is just the beginning of Google’s plan to bring a social element to many of its products and services. In the coming months, you are probably going to see the google.com ecosystem revolve more and more tightly around G+ to a point where involvement in the + will be unavoidable. They have already somewhat merged Picasa and there are whispers of a full-on integration with Gmail. What else could be coming? Well, just think about all of the services that Google currently offers and simply add a “+” to them: Music+, Calendar+, Maps+, Docs+, Checkout+, etc.—and I’m not even going to take the time here to mention Android and the possible mobile implications!

But what about speculative projects? Well, Google recently sank loads of cash into a little company called Zynga and not too long ago partnered with Rovio to bring Angry Birds to Chrome. Can you imagine playing “Words With Friends+” or “Angry Birds+” online with all your G+ friends?

The future of Google is here folks, and it’s as subtle as a +.

Alessandro Molina: TurboGears 2.1.1 released!

Sun, 19 Jun 2011 08:59:05 +0000

After a reflection moment caused by the need to think what will follow after the pylons and repoze.bfg merge in pyramid the TurboGears2 team has decide to gather up all its forces and give to TurboGears2 its own independent life.

The first steps have been oriented to improve the framework reliability and brought to life the TurboGears continuous integrations system and a standard project release process.

After a few months of work 2.1.1 has been released and it brings many fixes and improvements, 2.1.2 is under its way and a 2.2 release with major improvements is already planned!

TG2 Core:
* Fixed dependencies for Python 2.4. Now any packages that are
needed are automatically installed.
* Updated package requirements as high as possible.
* Verified nested RestControllers work as expected
* Added/fixed Kajiki support
* Ignore repoze.who_testutil when running nosetests
* Fixed import order for pylons.middleware
* Fixed crash when PYTHONOPTIMIZE is enabled
* Report a warning about ErrorMiddleware is disabled
* Fixed concurrency issues with use_custom_format
* Fixed 404 errors if a controller uses only custom formats
* Verified that user object is available inside of the error controller/template
* Fixed expansion of arguments on before/after calls
* Fixed wrong header response for 405 error
* Fixed WebOb version requirment. Newer version required
* Added test case to check for replace_header when called from WSGIApp
* Fixed issues with Content-Type header appearing multiples times on 204/205 responses
* Removed redundant hasattr checks on override_template
* Improved support for pylons 1.0 strict_c
* Fixed post traceback, now reports to Pylons correctly
* Added test case to check for spurious content type removal on empty content
* Fixed crash when content type header is missing
* Fixed crash when response Content-Type is set to None
* Fixed support for etags. Pylons 1.0 changes slightly, we support the correct version now
* Added dependency_links and setup.cfg allow_hosts: easy_install TurboGears2 now works
* Fixed DecoratedController. should not call super(), 2.6 revealed a problem
* Fixed Genshi output method. Use XHTML if none specified, instead of XML

TG2 Devtools:
* Fixed Python 2.4 compatibility issues. Dependencies are now automatically specified
* Updated package version requirements as high as possible
* Fixed about.html instructions about where the logo is found
* Set “zip_safe=False” by default in the templates now
* Tests fixed, now pass
* Added support for sqlalchemy-migrate
* Added option to choose config file
* Added archive_tw_resources command for projects
* Fixed deprecated redirect calls
* Set Genshi templating method by default to XHTML
* Adding dependency_links: easy_install tg.devtools now works

Michael Pedersen: Announcement: TurboGears 2.1.1 Released!

noreply@blogger.com (Michael Pedersen) — Wed, 15 Jun 2011 21:21:10 +0000

We are pleased to announce the latest release of TurboGears, 2.1.1!

This is a maintenance/bugfix release for the 2.1 series. The full changelog is below.

This release has been a long time in the making, and I would like to take a few minutes to thank the people who made it happen.

Mark Ramm, Chris Perkins, and Kevin Dangoor, for supporting me in beginning this whole process back in January.

Alessandro Molina, Christoph Zwerschke, and Diez Roggisch, for contributing much of their precious free time to writing code, making suggestions, and keeping me on track.

Florent Aide and Jon Schemoul, for providing us with a server and tgext.pages to make the site work.

And to everybody in the community that I did not name: Without you, I don't think I would have kept myself working. The results are here, and they belong to you as much as any of the people I named above.

Thank you, everybody. Your support and help mean a great deal to me. I only hope I've lived up to some of your expectations.

I can speak for the whole team when I say that we look forward to continuing to serve the community with quality code, and we look forward to your continued support!

And I know I can say it most strongly for myself. We're back on the move. And I'm liking the results.

Full Change Log Here:

TG2 Core:

Fixed dependencies for Python 2.4. Now any packages that are needed are automatically installed.
Updated package requirements as high as possible.
Verified nested RestControllers work as expected
Added/fixed Kajiki support
Ignore repoze.who_testutil when running nosetests
Fixed import order for pylons.middleware
Fixed crash when PYTHONOPTIMIZE is enabled
Report a warning about ErrorMiddleware is disabled
Fixed concurrency issues with use_custom_format
Fixed 404 errors if a controller uses only custom formats
Verified that user object is available inside of the error controller/template
Fixed expansion of arguments on before/after calls
Fixed wrong header response for 405 error
Fixed WebOb version requirment. Newer version required
Added test case to check for replace_header when called from WSGIApp
Fixed issues with Content-Type header appearing multiples times on 204/205 responses
Removed redundant hasattr checks on override_template
Improved support for pylons 1.0 strict_c
Fixed post traceback, now reports to Pylons correctly
Added test case to check for spurious content type removal on empty content
Fixed crash when content type header is missing
Fixed crash when response Content-Type is set to None
Fixed support for etags. Pylons 1.0 changes slightly, we support the correct version now
Added dependency_links and setup.cfg allow_hosts: easy_install TurboGears2 now works
Fixed DecoratedController. should not call super(), 2.6 revealed a problem
Fixed Genshi output method. Use XHTML if none specified, instead of XML

TG2 Devtools:

Fixed Python 2.4 compatibility issues. Dependencies are now automatically specified
Updated package version requirements as high as possible
Fixed about.html instructions about where the logo is found
Set "zip_safe=False" by default in the templates now
Tests fixed, now pass
Added support for sqlalchemy-migrate
Added option to choose config file
Added archive_tw_resources command for projects
Fixed deprecated redirect calls
Set Genshi templating method by default to XHTML
Adding dependency_links: easy_install tg.devtools now works

TG2 Docs:

New book doc project started.
Added note about how/why to set zip_safe to False
Fixed import statement in some examples
Fixed spelling errors
Fixed wiki tutorial errors about bootstrapping, also the missing setup.py
Removed dead references in modwsgi deployment documentation
Fixed warnings. None remain right now
Implemented new theme donated by Christoph Zwerschke
Added documentation on how to extract ToscaWidgets resources
Added tutorials for DataGrid, pagination, sorting, and custom columns

Michael Pedersen: Announcing TurboGears 2.0.4

noreply@blogger.com (Michael Pedersen) — Sat, 04 Jun 2011 21:55:59 +0000

We are pleased to announce the latest release in the 2.0 series, TurboGears 2.0.4.

This is a maintenance/bugfix release for the 2.0 series. The full changelog is below.

Note that, unless security issues are found, this will be the final release of the 2.0.x line. As 2.1 is fully backwards compatible, and release of 2.1.1 is imminent, we strongly encourage everybody to upgrade as soon as possible.

We look forward to continuing to serve the community with quality code, and we look forward to your continued support!

The following changes were added for 2.0.4:

Python 2.4 compatibility was fixed.
Nested RestControllers had a bug preventing the proper
resolution of the "delete" method. This has been fixed.
When all @expose'd methods of a controller had a
CUSTOM_CONTENT_TYPE, a 404 would be returned. This has been fixed.
Problem existed with getting tuples into controllers. Patch from
bug 37 applied to fix this.
Brought the current TG2 private PyPI current. Getting installation
to work is now as easy as "easy_install TurboGears2==2.0.4"
Got automated testing working consistently.
Added tests for mixing positional args, keyword args, and pagination
dependency_links is now in place to make it easier to get the initial
setup done
Several other smaller bugs were fixed.

Cliff Wells: I joined Facebook

Sun, 15 May 2011 23:30:49 +0000

I've been a long-time opponent of Facebook, and social networking sites in general (my opposition started way back when Orkut was the cool thing). My reasons are simple: I value privacy and I feel these sites make too much information available to both civil authorities as well as corporations. My canned summary of this position was to ask: "Are you, or have you ever been, a member of the Communist Party?", quoting our infamous McCarthy, who would have absolutely adored Facebook, with its easy-to-navigate graphs of relationships.

I still feel Facebook should have never existed and is simply too dangerous.

So why did I join (and friend a lot of people)? The reasons are myriad, but at the end of the day it comes down to this: the genie is out of the bottle and can't be put back in. My wife is on Facebook, my friends are on Facebook. Facebook already knows who I am, and who my friends are. Despite the fact that I chose not to engage, I have to finally admit I'm already part of the system. At this point, I can either pretend this isn't the case, or I can choose to engage on my own terms, as best I can.

We are all aware of the abuses of Facebook and its complete and utter disregard for its user's privacy. But I've also seen Facebook used to empower people in ways previously unheard of. The turning point for me came a couple of days ago when I read this article, about a "flash mob" of 40,000 people who came together, mostly via Facebook, to protest against a wealthy neighborhood who wanted to keep them out. This is something previously unheard of, and really, previously impossible. The reality is that Facebook and friends are, like most powerful technologies, a double-edged sword; we can either allow it to be wielded against us, or we can wield it ourselves.

In any case, I've decided that Facebook (and social networks in general), are little more than the culmination of the Internet as a whole. Really, if we wanted to stop Facebook, we'd have needed to stop the Internet (and for many of the same reasons). At the end of the day, for better or worse, the world is now networked. Anonymity is over and has been for some time. While I'm still trying to wrap my head around it, I have a gut feeling that the only hope we have for privacy is to create such a glut of data that it can never be fully digested, that to parade our social networks is possibly the only protection we have left. After all, if they know my wife, my friends, my family, what good is my own anonymity?

Luke Macken: Red Hat OpenShift Express & The Leafy Miracle

Sat, 07 May 2011 20:20:13 +0000

Red Hat made a lot of awesome announcements this week at The Red Hat Summit, one of which being OpenShift.

I had the opportunity to play with the internal beta for a little while now, and I must say that as a developer I am extremely impressed with the service. Just being able to git push my code into to the cloud drastically simplifies large-scale software deployment, and makes it so I don't even have to leave my development environment.

I figured out a way to get TurboGears2 and Pyramid running on OpenShift Express, and documented it here and here. After that, I proceeded to write my very first Pyramid application.

[ The Leafy Miracle ]

In memory of the proposed [and rejected] Fedora 16 codename "Beefy Miracle", this little app is called "Leafy Miracle".

leafy-miracle.rhcloud.com

[ Features & Tech ]

Written in Python using the Pyramid web framework
SQLAlchemy database model of Yum Categories, Groups, Packages, Dependencies, and Dependants
Interactive graph widget, using ToscaWidgets2 and the JavaScript InfoVis Toolkit
Package mouse-over menus linking to downloads (community), code (gitweb), bugs (bugzilla), acls (pkgdb), builds (koji) and updates (bodhi).
Deep linking, which allows you to use the browser back/forward buttons while navigating the tree, as well as share or bookmark links to specific nodes
Search bar with auto-completion of packages, categories, and groups
jQuery under the hood, powering the widgets

[ Running ]

sudo yum -y install python-virtualenv
git clone git://fedorapeople.org/~lmacken/leafymiracle && cd leafymiracle
virtualenv env && source env/bin/activate
python setup.py develop
python leafymiracle/populate.py
paster serve development.ini

[ Code ]

git clone git://fedorapeople.org/~lmacken/leafymiracle

[ Props ]

Mad props go out to RJ Bean, who helped me write this app. He is responsible for writing a ton of amazing Python widgets for various JavaScript visualization libraries. You can see some demos of them here: craftsman.rc.rit.edu.

Cliff Wells: Ubuntu 11.04

Wed, 04 May 2011 03:50:20 +0000

Despite misgivings about Unity, I upgraded to 11.04 on Monday night.

Turns out my misgivings were entirely misplaced, since Unity won't even run (just hangs). Shrugging that off, I proceeded to switch to the "classic" desktop, spent an hour fixing all the settings that are required for Compiz to work properly without Unity (moving and resizing windows, etc). I finally got it into a pretty happy state and was set to move on.

Issue #1: suspend/resume broken. Oops. This worked fine under 10.04 and 10.10. Suspend works, resume doesn't. I figured I could figure out a quirk or wait for an update to fix this, except...

Issue #2: an update on Tuesday (not sure if it was kernel, compiz or xorg Intel drivers) seems to have permanently broken my desktop. Now I can only log in using "classic desktop (no effects)" or "safe mode". Fiddled with it for another hour, before giving up.

At the end of the day, I ended up reinstalling 10.04 and then upgrading to 10.10, where things are more-or-less sane again. 11.04 is an unmitigated disaster as far as I'm concerned. I'll stick with 10.10 until 11.10 or 12.04 is released, but if the next release doesn't improve their QA considerably, I'm thinking I'll be moving to Debian Mint.

Alessandro Molina: Mobile devices detection with TurboGears2

Tue, 19 Apr 2011 23:00:19 +0000

We just released tgext.mobilemiddleware for turbogears2 to make easier to handle templates for mobile devices and detect mobile devices requests.

Indeed it is quite simple to use as it makes possible just to register a different template by using @expoe_mobile decorator which will be used for mobile devices, making possible to create mobile version of web page by using for example jquery mobile

Christpher Arndt: Tag der freien Dokumente

Wed, 30 Mar 2011 15:33:50 +0000

Die Free Software Foundation (FSF) hat für heute, den 30. März 2011, den Document Freedom Day (DFD) ausgerufen. Das nehme ich zum Anlass, den kurzen Informationstext der DFD-Webseite zum Open Document Format vom Englischen ins Deutsche zu übersetzen. Das “Open Document Format” Offene Dokumentformate sind offene Standards für Ihre Bürodokumente: Texte, Tabellen, Präsentationen usw. Wir [...]

Luke Macken: git clone all of your Fedora packages

Thu, 24 Mar 2011 16:10:06 +0000

After doing a fresh Fedora 15 install on my laptop last night, I wanted to quickly clone all of the packages that I maintain. Here is a single command that does the job:



python -c "import pyfedpkg; from fedora.client.pkgdb import PackageDB; [pyfedpkg.clone(pkg['name'], '$USER') for pkg in PackageDB().user_packages('$USER')['pkgs']]"

Luke Macken: Fedora Photobooth @ SXSW

Sun, 13 Mar 2011 02:57:39 +0000

This is the first year that Fedora will have a booth at SXSW! Sadly, I am not going to be attending since it conflicts with PyCon. However, my code will be running at our booth. Usually the Fedora booth at conferences is comprised of a bunch of flyers, media, swag, and some people to help answer questions and tell the Fedora story. However at SXSW, things are going to be a little different.

Aside from the amazing flyers that Máirín created, there will also be a Fedora Photobooth. Someone (probably Spot or Jared) will be dressed in a full Tux costume, and people can come and get their photo taken with them. Spot came to me the other day and asked if I could write some code to streamline the whole process.

An hour or so later, photobooth.py was born. There are definitely lots of improvements that can be made, but here is what it currently does in its initial incarnation:

Uses gphoto2 to automatically detect your camera
When Enter is pressed, it snaps a photo and downloads it locally
A Fedora watermark is applied to the bottom right corner of the image
The photo is uploaded to a server
A QRCode is generated that points to the image URL
A TinyURL is generated for the image
HTML is generated that shows the image, the QRCode, and TinyURL
The page is then displayed in the web browser

In Action
See Mo's blog for photos of this code in action at the Fedora SXSW booth!

* SXSW Expo Day 1 from the show floor
* SXSW Expo Day 2
* A Beefy, Miraculous Day at SXSW (Expo Day 3)

The Code
I threw this in a git repo and tossed it up on GitHub:

github.com/lmacken/photobooth.py

#!/usr/bin/python
# photobooth.py - version 0.3
# Requires: python-imaging, qrencode, gphoto2, surl
# Author: Luke Macken <lmacken@redhat.com>
# License: GPLv3

import os
import surl
import Image
import subprocess

from uuid import uuid4
from os.path import join, basename, expanduser

# Where to spit out our qrcode, watermarked image, and local html
out = expanduser('~/Desktop/sxsw')

# The watermark to apply to all images
watermark_img = expanduser('~/Desktop/fedora.png')

# This assumes ssh-agent is running so we can do password-less scp
ssh_image_repo = 'fedorapeople.org:~/public_html/sxsw/'

# The public HTTP repository for uploaded images
http_image_repo = 'http://lmacken.fedorapeople.org/sxsw/'

# Size of the qrcode pixels
qrcode_size = 10

# Whether or not to delete the photo after uploading it to the remote server
delete_after_upload = True

# The camera configuration
# Use gphoto2 --list-config and --get-config for more information
gphoto_config = {
    '/main/imgsettings/imagesize': 3, # small
    '/main/imgsettings/imagequality': 0, # normal
    '/main/capturesettings/zoom': 70, # zoom factor
}

# The URL shortener to use
shortener = 'tinyurl.com'

class PhotoBooth(object):

    def initialize(self):
        """ Detect the camera and set the various settings """
        cfg = ['--set-config=%s=%s' % (k, v) for k, v in gphoto_config.items()]
        subprocess.call('gphoto2 --auto-detect ' +
                        ' '.join(cfg), shell=True)

    def capture_photo(self):
        """ Capture a photo and download it from the camera """
        filename = join(out, '%s.jpg' % str(uuid4()))
        cfg = ['--set-config=%s=%s' % (k, v) for k, v in gphoto_config.items()]
        subprocess.call('gphoto2 ' +
                        '--capture-image-and-download ' +
                        '--filename="%s" ' % filename,
                        shell=True)
        return filename

    def process_image(self, filename):
        print "Processing %s..." % filename
        print "Applying watermark..."
        image = self.watermark(filename)
        print "Uploading to remote server..."
        url = self.upload(image)
        print "Generating QRCode..."
        qrcode = self.qrencode(url)
        print "Shortening URL..."
        tiny = self.shorten(url)
        print "Generating HTML..."
        html = self.html_output(url, qrcode, tiny)
        subprocess.call('firefox "%s"' % html, shell=True)
        print "Done!"

    def watermark(self, image):
        """ Apply a watermark to an image """
        mark = Image.open(watermark_img)
        im = Image.open(image)
        if im.mode != 'RGBA':
            im = im.convert('RGBA')
        layer = Image.new('RGBA', im.size, (0,0,0,0))
        position = (im.size[0] - mark.size[0], im.size[1] - mark.size[1])
        layer.paste(mark, position)
        outfile = join(out, basename(image))
        Image.composite(layer, im, layer).save(outfile)
        return outfile

    def upload(self, image):
        """ Upload this image to a remote server """
        subprocess.call('scp "%s" %s' % (image, ssh_image_repo), shell=True)
        if delete_after_upload:
            os.unlink(image)
        return http_image_repo + basename(image)

    def qrencode(self, url):
        """ Generate a QRCode for a given URL """
        qrcode = join(out, 'qrcode.png')
        subprocess.call('qrencode -s %d -o "%s" %s' % (
            qrcode_size, qrcode, url), shell=True)
        return qrcode

    def shorten(self, url):
        """ Generate a shortened URL """
        return surl.services.supportedServices()[shortener].get({}, url)

    def html_output(self, image, qrcode, tinyurl):
        """ Output HTML with the image, qrcode, and tinyurl """
        html = """
            <html>
              <center>
                <table>
                  <tr>
                    <td colspan="2">
                        <b><a href="%(tinyurl)s">%(tinyurl)s</a></b>
                    </td>
                  </tr>
                  <tr>
                    <td><img src="%(image)s" border="0"/></td>
                    <td><img src="%(qrcode)s" border="0"/></td>
                  </tr>
                </table>
              </center>
          </html>
        """ % {'image': image, 'qrcode': qrcode, 'tinyurl': tinyurl}
        outfile = join(out, basename(image) + '.html')
        output = file(outfile, 'w')
        output.write(html)
        output.close()
        return outfile

if __name__ == "__main__":
    photobooth = PhotoBooth()
    try:
        photobooth.initialize()
        while True:
            raw_input("Press enter to capture photo.")
            filename = photobooth.capture_photo()
            photobooth.process_image(filename)
    except KeyboardInterrupt:
        print "\nExiting..."

Michael Pedersen: A Git Script for Python

noreply@blogger.com (Michael Pedersen) — Wed, 09 Mar 2011 07:27:37 +0000

Working on TurboGears, I found myself trying to run all the tests we provide. And then I found myself getting a failure even though I didn't change anything. I had simply switched to a different branch.

Turns out the problem was left over .pyc files. Also turns out that the fix is extremely easy. In your git repository, create a file named "hooks/post-checkout". In that file, put the following lines, and make the script executable. All .pyc files will be scrubbed every time you switch, preventing the problem entirely.

#!/bin/bash

find . -type f -print | grep -v ^.git | grep '\.pyc$' | xargs rm

Luke Macken: FUDCon 2011 Tempe

Wed, 16 Feb 2011 21:55:02 +0000

I meant to get this summary out the door weeks ago, however, I didn't want to distract from my post-FUDCon productivity :)

Another FUDCon has come and gone. This time it was in gorgeous Tempe, Arizona. It was my first time in AZ, and I must say that I was thoroughly impressed. It was truly a great location for FUDCon. Thanks to the epic snowstorm of doom, I also was stuck there for a few extra days

Live Widgetry
Before the conference I decided to throw together a little live widget that scrolls all of the new identi.ca posts tagged with #FUDCon. Thanks to the mad design skillz of mizmo, and the feed aggregation and real-time web sockets of Moksha, I was able to throw it together pretty quickly. I plan on taking this code and integrating it in the existing fedoracommunity dashboard and hooking up many different fedora-related feeds to it.

Sessions

AutoQA
Day 1 of the FUDCon sessions were quite interesting. I got a chance to learn a bit more about the exciting AutoQA project, which is coming along nicely. You'll be seeing AutoQA commenting on bodhi updates soon enough.

Security Lab
I caught Joerg Simon's session on the Fedora Security Lab. It was exciting to see how the Security Spin has evolved ever since I created it back in 2007 for a project in my forensics class. It was also interesting to learn more about OSSTMM.

Spins
The next session I attended was about the future of spins. Almost everyone agreed that Spins are useful and a valuable part of Fedora. The problems seem to mostly lie in governance/policy and a lack of communication and coordination between the Spins SIG and QA/Releng/Infrastructure. Once of my ideas below may help with this a little bit.

The Next Big Project

Day 2 was comprised of more sessions, including my team's "Next Big Project" proposals. Of course, the Fedora RPG that Spot and Mo talked about was definitely a hot topic, and got a lot of people excited.

I talked about a handful of project ideas that I would like to work on in the future (actually, I have code written for most of them already). Here is a quick rundown:

Real-time Infrastructure
I want to see us deploy an AMQP message broker inside our production infrastructure. Then, we hook up all of our existing services and have them fire off messages when various events occur (koji builds, bodhi updates, pkgdb additions/removals/changes, git hooks, planet feeds, wiki edits, etc). From here, using some realtime web technology that we created, we could easily expose these message queues via a live dashboard that lets you filter and navigate the stream of activity, along with providing real-time metrics. We can also create desktop notification widgets, so you can get popup bubbles for things that you care about.

This is also key to the whole RPG as well. In order to build a game based on Fedora workflows, we need an underlying expert system that knows what actions can be taken within fedora, how they are accomplished, and who is getting them done.

Meeting app
Currently after a meeting, our Meetbot spits out the logs and an overview in txt/html/rst to a directory on meetbot.fedoraproject.org. Trying to track down who agreed to what when, or even to see what a given team has been up to over the past couple of months, is very tedious.

The data is there, now we just need to make it useful. I would love to see a frontend for this sytem that tracked meetings by team/people/topics/projects, kept track of actions and held people accountable for what they say they are going to do (and make it easy for people to say "I need help with this", or "I don't have time to finish this"), making it simple to go from an #idea in a meeting to implementation. There is so much great data in these logs, and I think we can do some awesome things with it.

Improved upstream monitoring
Most Fedora developers probably don't even know that we already have an Upstream release monitoring service buried in the wiki. I added almost every package I maintain to it, and it will automatically open a bug when a new upstream version is released. Extremely useful.

I wouldn't call this a "big" project, but I would like to see us integrate this service into our existing infrastructure. We could potentially store this per-package upstream data in the pkgdb/bodhi, and when a new release comes out write some code to automatically try doing a simple specfile bump, throw a scratch build at koji, run it through AutoQA, queue up for testing in bodhi, etc. Ideally, this would minimize the massive amounts of effort that our maintainers have to do to keep our packages up to speed with upstream.

Discussions app
Last year, Máirín Duffy and I came up with some interesting ideas for improving our mailing lists. I would like to make this a reality. Since then, I have already written code that can successfully parse all of fedora-devel.mbox (sounds much easier than it really is), populate it into a SQLAlchemy database model, expose a JSON API for quering, and visualize threads and various statistics with a basic widget.

Spin Master
One of the biggest problems with spins that we have right now is that there is no easy way to track how they are evolving. I would like to see us create a system that took the nightly spins and analyzed them, tracking what packages have been added/removed, which packages have grown/shrunk, etc. We could potentially get AutoQA involved here and make sure all spins pass a certain level of sanity checks before they can even be released. There are scripts floating around that can do a lot of this already -- but I want to streamline it and build a frontend.

Source-level package diff viewer
Fedora churns at such a fast pace, yet I can only imagine that a small subset of maintainers actually look at the complete code changes between upstream package releases. The recent sourceforge intrusion should be seen as a reality check to us distributions, and I think we need to step up our game quite a bit to ensure we don't let any malicious code slip into Fedora.

I would like to see a web/cli interface for viewing full source diffs of package updates in Fedora, allowing people to annotate/flag lines of code. Having more eyes view the code changes that go into our distribution is definitely a Good Thing, not just for Fedora, but for Open Source in general.

Hackfests

Now, on to my favorite part of any conference -- The Hackfests. First off, I felt that the hackfests were a bit unorganized this year. There was no opportunity to pitch hackfests, and it was not easy to figure out who was doing what in which rooms. Anyway, I had a fairly productive day of hacking...

bodhi
I integrated our package test cases into bodhi, which will now query the wiki for tests and display them in your updates, like so:

Thanks to James Laska for the code to query the wiki, and to Ian Weller for python-simplemediawiki API.

I also sat down with Máirín Duffy and talked about Bodhi v2.0 interaction design. We discussed what actions we want people to take when they arrive at bodhi's homepage, which essentially boils down to submitting, searching, browsing, and testing updates. Mo quickly threw together an awesome mockup that portrays some of our initial ideas.

Fedora Community Discussions
As mentioned above, I have already started implementing the mailing list interface that mizmo and I designed last year. I worked with Casey Dahlin during the hackfests and helped him get a working fedoracommunity/moksha development environment up and running and become familiar with the existing code.

Even though he wasn't at FUDCon, Jan Hutar has also been working on a couple of great graphs/grids of mailing list statistics for fedoracommunity as well. We'll be hacking away at this stuff over the next few months, so stay tuned.

kernel EFI framebuffer
I spun up a quick kernel patch to enable the EFI framebuffer on a handfull of Macs. I already wrote a patch that got applied upstream that enables this framebuffer on 14 different mac models, but this new patch adds 5 more. A lot of people, especially Sugar on a Stick users, are desperate to get Fedora running on their mactel machines (assuming found in may school labs), so I spun up a fresh livecd with my kernel patch for testing. See Bug #528232 for more information.

liveusb-creator
I did a bunch of work on porting the liveusb-creator from HAL to UDisks. Thankfully, Ubuntu's cleverly-named "usb-creator" already has UDisks support, so I've been happily borrowing ideas from their code :)

I also had some great discussions with Peter Robinson and Bernie Innocenti about solving the persistent overlay problem with our Live USBs. Right now, they are essentially a ticking time bomb, and real world LiveUSB use-cases are getting bit by this all of the time. Over the past few years, the "solution" has been to "wait for unionfs to get merged into the kernel". However, there are a variety of other potential solutions that we are going to look into as well.

Pyramid
The future of Python web development is extremely exciting, innovative, and still evolving at a rapid pace. As a TurboGears developer, I'm still very impressed with TG2, which along with TG1 will be supported for a long time to come -- but I'm also very eager for the next generation framework that has just emerged.

Recently, the Pylons project has merged with repoze.bfg to form Pyramid, which just released version 1.0. I'm quite amazed by the quality of the code, docs, and tests, and benchmarks already show it blowing rails/django/tg/pylons out of the water. I'm looking forward to the PyCon sprints, where all 3 communitites are going to be sitting at the same table working together on this project.

So while I was stuck in Tempe during the snow storm, I wrote 7 RPMs for Pyramid and it's dependencies, which are currently waiting to be reviewed. I plan on writing Bodhi v2.0 using Pyramid, so if this is something that interests you, let me know (and start reading the 600+ pages of docs ;)

Good times
Technical stuff aside, I had a blast at FUDCon, and I feel like it was one of the best. FUDPub was great, as usual. I played a lot of poker [poorly], ate a lot of tasty food, and had some great conversations. I had caught a cold prior to coming to FUDCon, so instead of nursing it with some nyquil and sleep, I decided to try to nurse it with a bottle of Jack, which turned out to be a bad idea.

Also, no FUDCon would be complete without mentioning TheGreatestGameYouWillEverPlay.com. Once the hackfests started winding down, I gave a quick session on nethack, where I tought people various ways to steal from shops, using a bunch of screecasts that I had on my laptop.

Alessandro Molina: Fixing broken inline genshi tags on TG2.1

Fri, 14 Jan 2011 03:49:20 +0000

Turbogears 2.1 got a little issue with genshi templates being rendered as xml instead of xhtml. This causes the most various strange problems with tags nesting being closed in random points as the browser likes.

This is caused by an improvement to the mechanism used to decide with rendering format to use that it is now based on the response content type. If your applications returns Content-Type text/xml or text/plain your templates will be rendered as xml or as plain text.

But what happens with your application returns text/html which is the default case?

Indeed in this case the application asks to your configuration file what to do, and as in your configuration file by default there isn’t any templating.genshi.method entry you get the default genshi behaviour, which is xml. Your browser doesn’t really like receiving some xml when he is expecting html and so handles the tags as not being closed.

In the recent future it will probably be fixed, in the mean time you can work around this problem by declaring in your development.ini

[DEFAULT]
templating.genshi.method = xhtml

This will reassemble the default behaviour that was available in TG2.0 rendering your templates as xhtml and correctly showing them inside the browser. If you have specified in your template the html doctype instead of the default xhtml one, then you might also want to change that value to html.

Luke Macken: liveusb-creator 3.9.3 windows release

Thu, 06 Jan 2011 15:34:12 +0000

I spent the majority of yesterday at a DOS prompt. Thankfully, it wasn't as painful as it sounds, as git, vim and Python make Windows development quite tolerable.

Anyway, I was finally able to track down and fix a couple of major bugs in the liveusb-creator on Windows XP and 7, and I pushed out a new build yesterday with the following changes:

Rebuilt with Python 2.7 and the latest PyQt4/pywin32/py2exe/NSIS
Update to syslinux 4.03 in our Windows package, which works with Fedora 14
Determine if we are running with admin privs, and warn otherwise
Fix how and where we put our error logs
Update our list of Fedora & Sugar releases
Download releases to Downloads or My Documents
Various Windows path-related fixes
Translation updates

Windows users, download it here: http://liveusb-creator.fedorahosted.org

Alessandro Molina: Workaround for empty helpers in Turbogears 2.1

Tue, 04 Jan 2011 11:46:29 +0000

Last release of TG2.1 has a subtle bug due to the changes helpers management to support in pylons 1.0, this makes the “h” object appear as an empty dict instead of your application helpers module.

The bug is already tracked on http://trac.turbogears.org/ticket/2488 and as the fix is quite easy it will be probably fixed soon. In the mean time you can work around it by monkey patching the pylons.templating.pylons_globals (I do it in lib/app_globals.py but any place is fine).

import tg.render, pylons
def patched_pylons_globals():
    x = tg.render.my_pylons_globals()
    if x['h'] == {}:
        conf = pylons.config._current_obj()
        x['h'] = conf['package'].lib.helpers
    return x
pylons.templating.pylons_globals = patched_pylons_globals

This will make your helpers work again in TG2.1.
Please pay attention that this code is specific for TG2.1, TG2.0 didn’t perform pylons_globals monkey patching and so the code won’t work.

Compound Thinking: TurboGears Joins the Pylons Project

Tue, 28 Dec 2010 14:12:40 +0000

After much debate, discussion, and contemplation, we’ve made an important decision, that will best ensure the future of TurboGears, and of the ideas on which it was based. TurboGears is merging into the Pylons Project.

A bit of background

We built TurboGears 2 on top of a the Pylons framework, and I have been working with Ben and the Pylons folks for a couple of years now.

Pylons is a lightweight framework that is neither full stack noropinionated. The Pylons team has recently joined up with the repoze.bfg folks to create a new low-level non-opinionated web application development library called Pyramid which is based on repoze.bfg and now part of the larger Pylons Project.

Pyramid status

You can use Pyramid to build web applications right now. It has more plug points, and is in many ways more flexible and extendable than the original Pylons and will provide an even better foundation for a full stack framework like TurboGears.

Pyramid has a strong, highly documented, well tested, approach, is already starting to show the fruits of merging in ideas from TurboGears, and Pylons. We expect it to be a great choice for those who want a flexible, non-opinionated and very fast framework for their projects.

The future of “Full Stack” frameworks

Fortunately, that’s not where the story ends. The Pylons Project leaders recognize that a “low level” framework is not enough.

Most web developers need to get things done quickly, and want a full stack setup and ready to go, so they can immediately start developing features rather than infrastructure.

That’s where we come in. TurboGears was the pioneer of the “full stack” set of integrated components approach among the modern Python web frameworks, and we have already developed many full stack tools.

Next Steps

So, our first step will be to add the TurboGears2 package to the legacy support in the Pylons Project. So, the Pylons 1.x and Turbogears2 packages will be maintained side by side as part of a single overall project. This change won’t impact the TG2 code, except that we will be officially acknowledging that both codebases are
tightly linked and now are part of a single project.

Maintaining the existing Pylons1+tg code will only be one part of the larger Pylons Project.

Big Harry Audacious Goal

Ultimately, we will also be working with the Pylons Project folks to create a new generation of rapid application development tools on top of Pyramid, using the TurboGears “Full Stack” philosophy. We will help to pick default templating engines, default session support, default data persistence mechanisms, integrate widget libraries and build high level tools like OAuth or OpenID support.

The main benifits of this merger will come when we reach across framework boundaries, work together with with Repoze, Pylons, and other web framework developers to build a set of high level tools that make building complex, modern web applications easier and faster. There’s a lot we can learn from each-other, and even more that we can do if we work through our differences and find new ways to collaborate.

There is no future but what we make

It’s been a great ride. And, it looks like just when we thought things were settling down, there’s another drop, and the roller coaster ride that is TurboGears isn’t done yet. I am forever grateful for the chance
to work with all the TurboGears developers, to face challenges, and to build something that’s been so valuable to so many. And I’m looking forward to what we can do together with the Pylons and Repoze folks.

Compound Thinking: Technical Debt isn’t always Debt

Tue, 05 Oct 2010 03:15:39 +0000

After yesterday’s posts about why you should not focus on reducing technical debt and why that’s not an excuse to ignore it either. Dave brought up a good point in a comment.

Technical debt can be assessed like real debt, to a large degree. How much are you paying monthly because of the debt? An annoying little thing, or wanting to upgrade to a new version just for newness’ sake has a low interest rate.

– Dave Brondsema

But it got me thinking, perhaps “debt” isn’t the right metaphor for this at all. Annoying issues that don’t cost anything, and don’t technically need to be “repaid” aren’t properly debt at all.

Perhaps technical warts would be a better term for those kind of things. It feels good to fix them. But rather than feel righteous for spending time fixing them all and “paying down our debt,” perhaps we ought to feel a little bit self indulgent for spending time and money on what amounts to cosmetic surgery for our code.

I’m not saying cosmetic surgery is never valuable, or that warts are never painful, and certainly not that you should ignore them. After all warts can be cancerous, and could perhaps kill you. But after investigating and discovering that they are benign, perhaps it’s not worth the cost to have them removed.

Michael Pedersen: Announcing gogo, The Bash Project Switcher

noreply@blogger.com (Michael Pedersen) — Tue, 14 Sep 2010 19:32:38 +0000

If you're a developer, then you've faced this problem: When you switch between multiple projects, you have to switch your shell, your editor, and possibly a lot more. If it's time to start a new project, then there's all the busy work of making a new project.

Those problems are now ending.

gogo is a set of bash functions that will enable you to more easily switch things around. Since I wrote it (first) for me, it supports my most common needs up front: Python, Mercurial, Emacs, and TurboGears. The only piece that is not trivially swappable is Mercurial. I'm sorry, that's on my todo list, but I wanted to let people see it and maybe start getting some value from it.

I highly recommend checking out the README file. It includes instructions on getting set up, along with instructions on how to use the tool. For now, a quick summary of usage:

Make a new project: gogo -n myproject
Activate the new project (after having closed your current shell): gogo myproject
Make a new Python project: gogo -n -l python pyproject
Make a new TurboGears project: gogo -n -l python -t tg21 tgproject

Switching between projects is downright trivial. gogo even supports tab completion of project names. It's already helping me, and hopefully it can help you.

You can download the v0.4 release, or clone the latest repository (or even download a .tar.gz file of it).

Alessandro Molina: Lowering Tg2 memory usage by running multiple instances of an app inside same WSGI daemon process

Fri, 06 Aug 2010 15:01:29 +0000

I was recently trying to deploy one app multiple times inside the same
WSGIProcessGroup and WSGIApplicationGroup %{GLOBAL} to reduce memory
usage.
This works quite well except for all the SQLAlchemy sessions which end being
attached to the engine of the last wsgi script started.

The best solution that I have been able to get so far is to create a proxy interface to the application engine. This way each wsgi script gets binded to the same engine, but the engine itself keeps track of all the available real engines and responds to the script requests sending them to the right real engine.

class MultiSiteEngine(object):
   def __init__(self):
       self.engines = {}
 
   def __getattr__(self, name):
       if name == 'engines':
           return object.__getattribute__(self, name)
 
       if not self.engines.has_key(config['sqlalchemy.url']):
           self.engines[config['sqlalchemy.url']] = engine_from_config(config, 'sqlalchemy.')
 
       return getattr(self.engines[config['sqlalchemy.url']], name)

To make this work just allocate a multi_engine = MultiSiteEngine() inside your tg2 app model __init__ and change init_model method to pass multi_engine instead of the engine itself.

Alessandro Molina: Patched TG2 bootstrap script for python 2.6

Wed, 28 Jul 2010 22:41:07 +0000

As actually I end up installing Tg2 on various systems having python2.6 quite often I have patched the bootstrap script to work on it.

If you had problems installing TG2 on python2.6 failing on Extremes or Zope dependencies you can try to use this script, it should work for you:

https://bitbucket.org/_amol_/tg-bootstrap-py2.6/src/tip/tg2-bootstrap.py

Michael Pedersen: Announcing the Availability of tgext.menu v1.0b3

noreply@blogger.com (Michael Pedersen) — Mon, 26 Jul 2010 21:21:26 +0000

tgext.menu v1.0b3 is available for immediate download. It can be found on PyPI in an easy_install'able form. The source may be found at BitBucket. This release now allows specifying the icon for a menu entry in the decorators and functions, fixes an issue where an id could be duplicated, and allows for replacement of the default template.

Michael Pedersen: TurboGears, SQLAlchemy, and Parent/Child Relations on the Same Model

noreply@blogger.com (Michael Pedersen) — Thu, 15 Jul 2010 11:00:29 +0000

Best thing to do is to read this post. I'm just posting it here to try to help it get more exposure to those who might need it.

http://www.5dollarwhitebox.org/drupal/node/110

Michael Pedersen: Announcing the Availability of tgext.menu v1.0b2

noreply@blogger.com (Michael Pedersen) — Mon, 12 Jul 2010 22:59:51 +0000

This is a minor update. It contains new installation instructions due to testing failures right after installation in a new quickstart, along with a fix for a menu display bug with jdMenu, and some new API enhancements from bitbucket user scottawilliams. It is already available on PyPI, and you may upgrade as soon as you are ready.

Michael Pedersen: TurboGears and Amazon EC2 Benchmarking, Part 4: Conclusions

noreply@blogger.com (Michael Pedersen) — Sat, 10 Jul 2010 23:52:53 +0000

Reviewing my methods, and the results I got, I found out quite a bit about TurboGears. I learned about how it will perform in real world scenarios, and I learned how to figure out the level of hardware I will need to accommodate a community. So, what does this mean for you?

For TurboGears, this is about answering the question: "How well does this scale".

The results of that question cover a very limited set of variables: How much hardware do you need to scale? What version of TG provides the best performance? Is Genshi or Mako the better performing template engine?

That's it. No more, no less. By and large, the numbers speak for themselves. The only gotcha in this is that those numbers are for all the dynamic pages. They do not cover the static pages, and that is done deliberately. I configured Apache to handle serving static CSS and JavaScript files. You get the full speed of Apache when the files are static, and then you get the full benefits of TurboGears for all your dynamic pages.

If you need to deploy and scale your TurboGears application, look at those numbers, use the definition of an Amazon EC2 Compute Unit (along with how many each instance provides), and then comes the hard part: You must estimate how many dynamic pages are going to be viewed at a time. I would then leave an additional 50% capacity, just as a buffer.

So, if you expect your site to attract, at peak, 10 dynamic page views per second, and you're on TG2.1b2 with Genshi, then you can get away with the smallest instance. The requirements sound like you need something huge, but you really don't. Remember what I said in yesterday's post: 864,000 dynamic page views per day on something that small. That's a lot of page views, and will support a very active community.

I'm glad to be working with TurboGears. Lots of power, lots of scalability, and it's all built in. I just have to use it.

Now, as I close out this series, I'd just like to say thank you to the developers who work so long and hard with TurboGears. They make things like these posts possible.

Michael Pedersen: TurboGears and Amazon EC2 Benchmarking, Part 3: What I was Testing

noreply@blogger.com (Michael Pedersen) — Fri, 09 Jul 2010 09:43:00 +0000

I actually learned a few things, some of which were surprising, some of which were not. One thing that I don't think I've communicated very well to my readers is that I most emphatically am not comparing TurboGears to any of the other frameworks out there. I'm comparing TurboGears to itself, and finding out what I need to do to scale it up.

I could compare to Ruby on Rails, CakeWalk, CodeIgniter, WebCore, Django, and a host of others. The problem with doing so is that doing that sort of comparison is nearly impossible. Why? Each of these frameworks has their own particular pluses and minuses. PHP versus Python vs Ruby. Pure unadulterated speed vs size of community vs how much scaffolding you need to build vs how locked in you are after you start using it.

For every developer, these are tradeoffs. Some will be okay with scaffolding. Some demand PHP. Others want the framework to put no demands on them. Comparing all of these frameworks in an unbiased fashion just is not feasible. That's why I didn't do it.

All of these, from very very slow to very very fast, will serve the needs of a great many web sites. Even at a "mere" 10 requests per second, that's 864,000 dynamic page views per day. I happen to participate in a forum which transfers around 3G of data every day, and it only gets 380,000 URL requests per day (both dynamic and static).

This is part of why I've stressed that I'm testing the whole application stack so very much in these past few posts. Any of these frameworks are more than capable of serving a rather large and thriving community on very simple hardware.

So, think about those numbers long and hard before you say that a "mere" 14 requests per second is not good enough. If you actually analyze your needs, you're likely to find that you're overly focused on a number that won't matter nearly as much as you think it does.

Michael Pedersen: TurboGears and Amazon EC2 Benchmarking, Part 2: Requests per Second

noreply@blogger.com (Michael Pedersen) — Fri, 09 Jul 2010 09:40:19 +0000

That all important question has been asked, and is now answered: How fast is TurboGears? I started this series yesterday when I described my methodology. I conclude soon with my analysis of what I did, and what I found.

The numbers below are not what you might expect to see. Many people, when doing benchmarks, do the equivalent of writing "hello world", and say "That's how fast my web framework is." No, it's not. It's considerably slower than that. I've seen far too many benchmarks like that. I'll not link to any of them, as I don't wish to encourage more like that. I want to get an idea of how a full featured application will perform, not some dinky "hello world".

When looking at these numbers today, keep one thing in mind: These are not "hello world" requests. These requests include templating, caching, sessions, database requests and transactions, authentication, authorization, and all of the other little pieces that make an actual web application into an application versus something that manages to simply print "hello world" on the screen.

I'll be back tomorrow with some analysis, both on the numbers, and my own biases in the whole process. For now, look these over. I think you'll find them interesting.

In the below table, the columns have the following meanings:

EC2 Instance Name: This is the marketing name of the instance as visible on Amazon's page about EC2.
EC2 Instance Type: This is the instance name that you will see when you create a new instance on EC2.
TG Version: The version of TurboGears that produced these results
Genshi/Mako: When running these tests, I varied the template engine between these two where possible. This column indicates the template engine that produced these results.
Index or Database: The default index page for a quickstarted application does not hit the database. This column indicates whether I was forcing a database hit, or just hitting the index page.
Req/Second: How many requests per second were being processed. I truncated the results (so nothing after a decimal).

TurboGears on EC2 Results
EC2 Instance Name	EC2 Instance Type	TG Version	Genshi/Mako	Index or Database	Req/Second
Small Instance	m1.small	2.0.3	Genshi	Index	24
Small Instance	m1.small	2.0.3	Genshi	Database	13
High-CPU Medium	c1.medium	2.0.3	Genshi	Index	100
High-CPU Medium	c1.medium	2.0.3	Genshi	Database	64
Large	m1.large	2.0.3	Genshi	Index	86
Large	m1.large	2.0.3	Genshi	Database	44
Extra Large	m1.xlarge	2.0.3	Genshi	Index	148
Extra Large	m1.xlarge	2.0.3	Genshi	Database	80
High-Memory Extra Large	m2.xlarge	2.0.3	Genshi	Index	171
High-Memory Extra Large	m2.xlarge	2.0.3	Genshi	Database	94
High-Memory Double Extra Large	m2.2xlarge	2.0.3	Genshi	Index	284
High-Memory Double Extra Large	m2.2xlarge	2.0.3	Genshi	Database	171
High-Memory Quadruple Extra Large	m2.4xlarge	2.0.3	Genshi	Index	388
High-Memory Quadruple Extra Large	m2.4xlarge	2.0.3	Genshi	Database	229
High-CPU Extra Large	x1.xlarge	2.0.3	Genshi	Index	217
High-CPU Extra Large	x1.xlarge	2.0.3	Genshi	Database	123
EC2 Instance Name	EC2 Instance Type	TG Version	Genshi/Mako	Index or Database	Req/Second
Small Instance	m1.small	2.1b2	Genshi	Index	28
Small Instance	m1.small	2.1b2	Genshi	Database	14
High-CPU Medium	c1.medium	2.1b2	Genshi	Index	124
High-CPU Medium	c1.medium	2.1b2	Genshi	Database	58
Large	m1.large	2.1b2	Genshi	Index	96
Large	m1.large	2.1b2	Genshi	Database	41
Extra Large	m1.xlarge	2.1b2	Genshi	Index	168
Extra Large	m1.xlarge	2.1b2	Genshi	Database	70
High-Memory Extra Large	m2.xlarge	2.1b2	Genshi	Index	196
High-Memory Extra Large	m2.xlarge	2.1b2	Genshi	Database	86
High-Memory Double Extra Large	m2.2xlarge	2.1b2	Genshi	Index	349
High-Memory Double Extra Large	m2.2xlarge	2.1b2	Genshi	Database	150
High-Memory Quadruple Extra Large	m2.4xlarge	2.1b2	Genshi	Index	468
High-Memory Quadruple Extra Large	m2.4xlarge	2.1b2	Genshi	Database	194
High-CPU Extra Large	x1.xlarge	2.1b2	Genshi	Index	283
High-CPU Extra Large	x1.xlarge	2.1b2	Genshi	Database	126
EC2 Instance Name	EC2 Instance Type	TG Version	Genshi/Mako	Index or Database	Req/Second
Small Instance	m1.small	2.1b2	Mako	Index	115
Small Instance	m1.small	2.1b2	Mako	Database	22
High-CPU Medium	c1.medium	2.1b2	Mako	Index	480
High-CPU Medium	c1.medium	2.1b2	Mako	Database	93
Large	m1.large	2.1b2	Mako	Index	282
Large	m1.large	2.1b2	Mako	Database	63
Extra Large	m1.xlarge	2.1b2	Mako	Index	415
Extra Large	m1.xlarge	2.1b2	Mako	Database	94
High-Memory Extra Large	m2.xlarge	2.1b2	Mako	Index	630
High-Memory Extra Large	m2.xlarge	2.1b2	Mako	Database	131
High-Memory Double Extra Large	m2.2xlarge	2.1b2	Mako	Index	976
High-Memory Double Extra Large	m2.2xlarge	2.1b2	Mako	Database	220
High-Memory Quadruple Extra Large	m2.4xlarge	2.1b2	Mako	Index	1354
High-Memory Quadruple Extra Large	m2.4xlarge	2.1b2	Mako	Database	274
High-CPU Extra Large	x1.xlarge	2.1b2	Mako	Index	776
High-CPU Extra Large	x1.xlarge	2.1b2	Mako	Database	181

Jorge Vargas: Hoy Dom Sagolla en RD, #sagollaRD y el futuro

Mon, 28 Jun 2010 13:00:43 +0000

El día ha llegado. Hoy se presenta Dom Sagolla (@dom) en Republica Dominicana y le deseo suerte. Esperemos que la audiencia logre sacarle todo el provecho que pueda y que @dom se lleve una muy buena impresión de la representación de nuestra comunidad que fue a escucharlo.

Dicho eso, me parece mas interesante analizar el entorno, ya que al igual que las redes sociales, la huella de esta conferencia (a mi parecer) es mas importante que la conferencia en si. La punta de lanza de este entorno durante las pasadas semanas fue el tag #SagollaRD en twitter, aunque las ramificaciones son bastantes, desde replies públicos con y sin mention, DMs (mensajes directos), google buzz, emails públicos, privados y oficiales (de twitter inc.), llamadas telefónicas, almuerzos, discusiones de pasillo, reuniones para tratar el tema, temas varios después de las reuniones, mesas de tragos, conversaciones de bomba de gasolina, y hasta tema para romper esos silencios incómodos que ocurren de vez en cuando.

En fin la cola es larga e interesante. A continuación una serie de puntos a resaltar sin orden de relevancia:

El “old style” marketing, es decir publicidad masiva, medios de comunicación, rifas, un mensaje unificado, repetir hasta el cansancio, entre otros. Todavia sirve. Irónicamente sirve para mercadear el “new style” marketing, que aboga por como los métodos “tradicionales” son obsoletos. La prueba de esto es simple. Todas (casi?) las boletas del evento se vendieron. El conferencista ganara por su exposición y la compañía organizadora obtendrá el beneficio planificado. Por otro lado el publico pago el precio pactado y saldrá con un conocimiento que no tenia anteriormente.
Los chistes, la ironía y el sarcasmo son una excelente forma de transmitir la información. Así como pueden ser malinterpretados y tomados demasiado en serio.
El expositor responde a las inquietudes, discordancias y malentendidos mientras que los organizadores siguen con el mensaje original. Esto es nuevo e interesante ya que nunca antes se había tenido este nivel de contacto, me presento como ejemplo a mi mismo, ante la confusión, al igual que muchos de ustedes me hice las siguiente preguntas: vale la pena que yo vaya? ganare algo de este evento? el costo de la entrada vale lo que potencialmente ganare? si veo el evento online (o me lo cuentan) valdrá la pena economizar ese dinero? La solución fue simple preguntarle al conferencista. Hace 5 años eso hubiera sido una locura. Hoy en día me tomo unos cuantos minutos para pensar la pregunta http://twitter.com/elpargo/status/16699935795 y a el unos cuantos mas responderme http://twitter.com/elpargo/status/16699935795. El análisis de la pregunta, la respuesta y de si estoy o no sentado en la sala*, es irrelevante aunque queda como un ejercicio para el lector. El punto es contrarrestar toda la inversión en tiempo y dinero del “old style” marketing con la del “new style” y medir su efectividad para cumplir la meta.
El Cambio del bio de @dom, en la tarde del viernes 18 de junio, http://twitter.com/dom cambio de “co-creator of twitter”** a el texto actual “Helped create @Twitter.” este sutil cambio en si mismo es casi irrelevante, pero dado que gran parte de la controversia generada por #SagollaRD se enfocaba en el termino co-creador, lo vuelve muy importante. Lo mas importante a resaltar es la influencia de la red social y sus ramificaciones, los detalles no los tengo (ni realmente los quiero saber). Lo cierto es que aparentemente #sagollaRD fue la gota que desramo el vaso y influencio a las personas indicadas a que tomaran acción para cambiar rompiendo con el status quo.
Otro dato curioso es medir la habilidad para adaptarse al cambio de las entidades involucradas, la fecha anterior es importante porque una semana después del cambio (24 junio), en las dos entrevistas realizadas a @dom todavía el termino era usado. Sera interesante ver durante la charla de @dom cuantas veces el utiliza el termino para referirse a si mismo.

Por ultimo me gustaría notar cuatro cosas negativas que salieron a relucir a lo largo de esta discusión.

Criticas a la persona no a la proyección de la imagen: se critico la habilidad de @dom de dar una excelente charla por ser presentado como un “co-creador de twitter” y la publicidad que sugería un rol mas importante del realmente jugado.
Criticas a la persona no al argumento, se tacho de malo (o bueno) un argumento por venir de algunas personas con conflictos de interés, sin tomar en cuenta la validez del argumento.
Se genero un ambiente de nosotros contra ellos, donde no se podía tener una opinión neutral y tener puntos a favor y en contra.
Muchas personas aprovecharon la oportunidad para “saldar pleitos pasados”

Todas las anteriores simplemente deterioran a la comunidad de personas interesadas en fomentar el avance tecnológico de la República Dominicana, y son practicas que debemos tratar de evitar, podemos discutir y estar radicalmente en desacuerdo sobre algún tema, esto no quiere decir que tengamos que llevar las cosas a titulo personal.

Por ultimo a los que están/fueron al evento de @dom que lo disfruten y que aprendan, a los que no fueron, que también que se lo disfruten y que aprendan.

El futuro? Quien sabe. Estamos aquí para crearlo.

* revisar si he hecho checkin en foursquare es trampa (y la data puede ser alterada con el fin de confundir y ser irónico)

** Si alguien tiene el texto original me encantaría ponerlo de ejemplo, por favor haganmelo llegar.

Luke Macken: Professors' Open Source Summer Experience @ RIT

Fri, 25 Jun 2010 19:59:33 +0000

Last week Red Hat put on a Professors' Open Source Summer Experience (POSSE) at RIT. Being an alumni, I was excited by the opportunity to be able to go back up to The ROC and teach some of the people that taught me. Going into it, I really had no idea what to expect. All I knew is that I was going to help lead the 'deep dive' section of the course, where I would teach professors how to dive in head first and get productively lost in a strange codebase. This is not something that can be accomplished with a set of powerpoint slides. Teaching how to hack on open source requires that you emerse yourself into a codebase, and bring your students with you.

The previous POSSE at Worcester State dove into the Sugar Measure Activity, and we were going to do the same.

Measure is an activity that turns the computer into an oscilloscope. Signals from the microphone (and sensors) can be plotted in time and frequency domains.

I had never used this activity, let alone hacked on it before. I've also never done any sugar activity development, aside from some tweaking of the OpenVideoChat, so I really had no idea what I was getting myself into. The obvious first step was to get it running. All of us were able to start the activity in virtual machines or emulators, except for one install which hit some odd errors upon startup. We were able to quickly track the bug down to a stray return statement in __init__ before some critical initialization code. Right after we fixed the problem we noticed that Walter Bender had already fixed this issue a few hours earlier. After a git pull, we were up and running.

Once we all got the activity running, we took a look at the bug list to see if there was any low-hanging fruit for us to tackle. Since the previous POSSE had already done some work on this activity the week before, there were not any trivial tickets left in the queue. So, in that case, we dove head first into the hardest one, "Measure activity gets stuck after recording". This ticket had very little information, and no log output, so we were on our own to try and track it down. We were able to reliably reproduce the issue on the XO-1.5, but not on the 1.0. In our virtual machines we hit it sporadically. We all agreed that it felt like a race-condition, most likely due to threading. So we started instrumenting the code and adding some debugging statements to try and figure out which line of code was the culprit.

In our efforts to scatter print statements all over the place to try and determine the code path, we noticed that none of our output was hitting the logs. When you have no idea if your code is even being run or not, don't be ashamed to throw in a raise Exception("WTF?!"). We finally realized that since the activity would freeze, it was never able to flush stderr/stdout to the log. A quick find/replace regex later, and we were using the proper logging module and seeing our debugging output hit the logs.

We were then able to track the bug down to a line of code that uses GTK to try and get the coordinates of the parent window. At this point, since none of us were GTK experts, we had to go upstream. So, I dropped into #fedora-devel and asked. Within 10 minutes I had responses from 3 different GTK hackers. One was a typical RTFM response, which humored the professors (who are very used to hearing/saying this), but the others pointed us in the right direction. One mentioned that gtk.gdk calls probably should not be done in a seperate thread. So, a suggested workaround was to add gtk.threads_enter()/gtk.threads_leave() calls before running any gtk.gdk code in the thread. A 2-line patch later, and we had squashed the bug.

We eventually bumped into the inevitable typo in some comments. So, we made the fix locally, and committed it to our git repo. A few minutes later and we found another one. I saw this as a great opportunity to show off some of my git-fu. Instead of sending two "Fix typo" patches upstream, I showed the professors how to use git interactive rebasing to squash multiple commits into a single one. They all followed along closely, and the workflow made sense to them.

While we were looking through the ticket queue, we saw an issue where Measure would apparently leak memory and crash when running it for a long period of time. While keeping this in mind, we kept our eyes peeled while wandering around the codebase to see if we could track the issue down. When looking at the code that takes screenshots of the waveform, I noticed that it created a temporary file with tempfile.mkstemp, saved the pixmap to it, injected it into the Journal, and then deleted the directory. This looked fine at a first glance, until I realized that it never closed the temporary file descriptor. Another 2-line patch later, and this issue was solved.

The next day both of the patches that we sent upstream were applied by Walter Bender.

Overall, POSSE definitely exceeded my expectations, and I'm extremely satisfied with how the 'deep dive' section went. I went into it feeling completely unprepared to teach, but by trusting my "hacker intuition", I feel that it turned out to be a fantastic learning experience for all of us.

Michael Pedersen: Announcing tgext.xmlrpc v0.8

noreply@blogger.com (Michael Pedersen) — Tue, 15 Jun 2010 20:11:38 +0000

This is an important update, and also one that is 100% backwards compatible.

This update allows you to have nice Python method signatures, as opposed to the hacky "*p, **kw" trick that v0.6 required on all methods.

To clarify, an example. v0.6 code required this:

class MyXmlRpc(XmlRpcController):
    @xmlrpc([['int', 'int', 'int']])
    def sumthem(self, *p):
        return p[0] + p[1]

v0.8 finally corrects this ugliness. This now works:

class MyXmlRpc(XmlRpcController):
    @xmlrpc([['int', 'int', 'int']])
    def sumthem(self, i1, i2):
        return i1+i2

Note that any code written for v0.6 will work just fine with v0.8. This is a purely cosmetic change, though a welcome one, I think.

It is available via PyPI and BitBucket (with documentation also on BitBucket).

Note that I did not forget to finish my Amazon EC2 benchmarking series. The next post in the series has been bugging me. I might not be William Shakespeare, but I do have some standards for my writing, and I've not been able to write up anything decent yet. I think I've finally figured out why I hated what I was writing, so will finish that series this week.

Michael Pedersen: TurboGears and Amazon EC2 Benchmarking, Part 1: Methodology

noreply@blogger.com (Michael Pedersen) — Wed, 09 Jun 2010 20:23:49 +0000

This is going to be a multi-part series. As of right now, I've got at least three parts, and I may be adding another one in. I've just gathered a lot of data from Amazon, and the TurboGears EC2 images I made. Enough of it, in fact, that I'm not sure how to present all of it. For now, I'll explain how I gathered the data.

First, I signed up for Amazon's Relational Database Service (RDS). This allowed me to move the database off of my EC2 instances. It did subject me to the limits of their setup, and their network, but this provides two advantages:

Web servers and database servers are typically separated in real world deployments. Using RDS allows me to have a similar configuration.
This allows me to show the speed of TurboGears. It's a fairly safe bet that Amazon has tuned their RDS service very well. Therefore, any speed issues that occur are likely to belong entirely to TurboGears.

From there, I used the EC2 Amazon Machine Images I announced last week. I would start up an instance, create a quickstarted application, and then test against both the index page (which produces no database hits) and against the index page with authentication cookies, which did produce database hits.

That's the summary.

For 2.0.3, I did the following:

Start up the instance of the TG 2.0.3 AMI I created. Use the correct one to get an instance size I've not yet measured.
ssh into the machine
Become the "turbogears" user
Create a quickstart app named "tgtestapp"
Run "python setup.py develop" to register the application
Copy development.ini to tgapp.ini, as expected by the AMI
Edit tgapp.ini: Set debug to False, and set the correct sqlalchemy.url to point to the RDS database.
Use "paster setup-app tgapp.ini" to initialize the database
Use "tglinker tgtestapp 0.1dev" to configure the remaining mod_wsgi files
Restart Apache
Visit the site in a browser to make sure the site is visible
Run the commands I made for benchmarking, and record the requests per second value from them.

For 2.1b2, I did the following (very similar to 2.0.3):

Start up the instance of the TG 2.1b2 AMI I created. Use the correct one to get an instance size I've not yet measured.
ssh into the machine
Become the "turbogears" user
Create a quickstart app named "tgtestapp" using genshi templates
Run "python setup.py develop" to register the application
Copy development.ini to tgapp.ini, as expected by the AMI
Edit tgapp.ini: Set debug to False, and set the correct sqlalchemy.url to point to the RDS database.
Use "paster setup-app tgapp.ini" to initialize the database
Use "tglinker tgtestapp 0.1dev" to configure the remaining mod_wsgi files
Restart Apache
Visit the site in a browser to make sure the site is visible
Run the commands I made for benchmarking, and record the requests per second value from them.
Destroy the current quickstarted app (rm -rf)
Make a new one, using mako templates
Run "python setup.py develop" to register the application
Use "tglinker tgtestapp 0.1dev" to configure the remaining mod_wsgi files
Restart Apache
Visit the site in a browser to make sure the site is visible
Run the commands I made for benchmarking, and record the requests per second value from them.

I repeated all of the steps (for both 2.0.3 and 2.1b2) for all Amazon EC2 instance sizes.

The commands I used for running the actual benchmarks are as follows:

ab -c 10 -n 400 http://localhost/
ab -c 10 -n 1000 http://localhost/
ab -c 10 -n 1000 -C 'tgtestapp=06c4a524f1a6c3196b1c491e95517c61c1770ef7efeaaacdcc29437c70cc8737438fc02d' -C 'authtkt="3a5c5890ea35f1bea327bf5b43123bbe4c0ef565manager!"' -C 'authtkt="3a5c5890ea35f1bea327bf5b43123bbe4c0ef565manager!"' http://localhost/

I have noticed a sometimes significant delay in mod_wsgi getting started serving the site. The first command overcomes that delay, and gets everything flowing as smoothly as possible. The second command focuses on raw rendering speed. The third looks at speed when the database server is in the mix, as it will generate hits.

Many people insist on multiple runs when doing benchmarks. To a degree, they are right. They are looking to know the absolute values, and benchmarking is inherently non-absolute. I did not do multiple runs, because I was looking for approximations of real world performance. Any one run will provide that.

Even those times when I said "Wait, that can't be right", and re-ran the numbers, I found the results were always within a few percentage points of each other. I quickly figured out that, for the needs of this article, anything more than one run would be superfluous.

You will note that the pages being retrieved are not simple "hello world" pages. The entire TurboGears stack gets tried out here. This is a full, real world example of what you can expect to see from TurboGears on Amazon's EC2 service. As such, multiple runs are of questionable value in this case (at best). If enough people are bothered by this, I'll do re-runs, and average out the results (or get the median, whichever), but I really do not feel it is necessary for this particular test.

Preparing the results for viewing on blogspot is actually taking a bit of time. I'll get them up in the next two days, so that everybody can see the numbers that I see.

Next: Requests per Second

Luke Macken: Fedora Updates Report

Tue, 08 Jun 2010 20:41:36 +0000

I recently wrote some code to generate detailed statistics of Fedora & EPEL updates within bodhi. Eventually this will be auto-generated and exposed within bodhi itself, but for now here are the initial metrics.

This report definitely conveys the shortcomings in how we currently utilize bodhi for "testing" updates, however, it does show us improving with each release. For Fedora 13, we implemented the No Frozen Rawhide process with improved Critical Path policies, which were definitely a success. With these enhanced procedures, along with the upcoming implementation of AutoQA and the new Package update acceptance criteria, I think we'll see these numbers drastically improve in the future.

You can find the code that generates these statistics here: metrics.py, log_stats.py. If you have any ideas or suggestions for different types of metrics to generate, or if you find any bugs in my code, please let me know.


Bodhi Statistics Report (Generated on June 8th, 2010)
=====================================================

Out of 17412 total updates, 2958 received feedback (16.99%)
Out of 1045 total unique karma submitters, the top 30 are:
 * notting (424)
 * mclasen (366)
 * jkeating (321)
 * adamwill (283)
 * cwickert (161)
 * rdieter (159)
 * pbrobinson (141)
 * kevin (141)
 * cweyl (122)
 * tomspur (119)
 * mtasaka (110)
 * xake (97)
 * cschwangler (86)
 * kwright (84)
 * peter (83)
 * hadess (80)
 * michich (72)
 * tagoh (69)
 * pfrields (69)
 * bpepple (69)
 * iarnell (68)
 * lkundrak (66)
 * shinobi (65)
 * sundaram (64)
 * spot (62)
 * pravins (62)
 * markmc (62)
 * thomasj (61)
 * smooge (60)
 * fab (59)

================================================================================
     Fedora 13
================================================================================

 * 3562 updates
 * 3065 stable updates
 * 427 testing updates
 * 62 pending updates
 * 8 obsolete updates
 * 2371 bugfix updates (66.56%)
 * 745 enhancement updates (20.92%)
 * 89 security updates (2.50%)
 * 357 newpackage updates (10.02%)
 * 410 critical path updates (11.51%)
 * 333 critical path updates approved
 * 1155 updates received feedback (32.43%)
 * 12120 +0 comments
 * 2477 +1 comments
 * 155 -1 comments
 * 595 unique authenticated karma submitters
 * 133 anonymous users gave feedback (1.57%)
 * 2261 out of 3562 updates went through testing (63.48%)
 * 1317 testing updates were pushed *without* karma (58.25%)
 * 21 critical path updates pushed *without* karma
 * Time spent in testing:
   * mean = 11 days
   * median = 9 days
   * mode = 7 days
 * 4 updates automatically unpushed due to karma (0.11%)
   * 0 of which were critical path updates
 * 231 updates automatically pushed due to karma (6.49%)
   * 2 of which were critical path updates
 * Time spent in testing of updates that were pushed by karma:
   * mean = 11 days
   * median = 7 days
   * mode = 7 days
 * Time spent in testing of updates that were unpushed by karma:
   * mean = 9 days
   * median = 5 days
   * mode = 5 days
 * 2445 packages updated (top 10 shown)
    * selinux-policy: 13
    * jd: 12
    * openoffice.org: 12
    * gdb: 12
    * ibus-pinyin: 11
    * nautilus: 10
    * kernel: 10
    * evolution: 9
    * libfm: 9
    * libmx: 9

================================================================================
     Fedora 12
================================================================================

 * 4844 updates
 * 4291 stable updates
 * 371 testing updates
 * 113 pending updates
 * 69 obsolete updates
 * 2905 bugfix updates (59.97%)
 * 1054 enhancement updates (21.76%)
 * 201 security updates (4.15%)
 * 684 newpackage updates (14.12%)
 * 407 critical path updates (8.40%)
 * 960 updates received feedback (19.82%)
 * 16311 +0 comments
 * 1899 +1 comments
 * 554 -1 comments
 * 758 unique authenticated karma submitters
 * 576 anonymous users gave feedback (5.33%)
 * 2873 out of 4844 updates went through testing (59.31%)
 * 2138 testing updates were pushed *without* karma (74.42%)
 * 188 critical path updates pushed *without* karma
 * Time spent in testing:
   * mean = 14 days
   * median = 13 days
   * mode = 17 days
 * 12 updates automatically unpushed due to karma (0.25%)
   * 4 of which were critical path updates
 * 133 updates automatically pushed due to karma (2.75%)
   * 13 of which were critical path updates
 * Time spent in testing of updates that were pushed by karma:
   * mean = 11 days
   * median = 7 days
   * mode = 7 days
 * Time spent in testing of updates that were unpushed by karma:
   * mean = 9 days
   * median = 5 days
   * mode = 5 days
 * 2902 packages updated (top 10 shown)
    * qbittorrent: 25
    * gdb: 25
    * selinux-policy: 22
    * kernel: 15
    * xorg-x11-server: 14
    * ibus: 13
    * jd: 13
    * abrt: 11
    * gvfs: 11
    * gtk2: 11

================================================================================
     Fedora 11
================================================================================

 * 6987 updates
 * 6381 stable updates
 * 183 testing updates
 * 99 pending updates
 * 324 obsolete updates
 * 3649 bugfix updates (52.23%)
 * 1566 enhancement updates (22.41%)
 * 350 security updates (5.01%)
 * 1422 newpackage updates (20.35%)
 * 383 critical path updates (5.48%)
 * 729 updates received feedback (10.43%)
 * 23427 +0 comments
 * 1197 +1 comments
 * 448 -1 comments
 * 782 unique authenticated karma submitters
 * 481 anonymous users gave feedback (3.58%)
 * 4129 out of 6987 updates went through testing (59.10%)
 * 3620 testing updates were pushed *without* karma (87.67%)
 * 278 critical path updates pushed *without* karma
 * Time spent in testing:
   * mean = 15 days
   * median = 14 days
   * mode = 17 days
 * 7 updates automatically unpushed due to karma (0.10%)
   * 0 of which were critical path updates
 * 64 updates automatically pushed due to karma (0.92%)
   * 11 of which were critical path updates
 * Time spent in testing of updates that were pushed by karma:
   * mean = 11 days
   * median = 7 days
   * mode = 7 days
 * Time spent in testing of updates that were unpushed by karma:
   * mean = 9 days
   * median = 5 days
   * mode = 5 days
 * 3787 packages updated (top 10 shown)
    * libguestfs: 30
    * jd: 24
    * selinux-policy: 23
    * kdebase-workspace: 19
    * kernel: 18
    * gdb: 16
    * dovecot: 16
    * qemu: 16
    * kdebase-runtime: 16
    * kdenetwork: 16

================================================================================
     Fedora EPEL 5
================================================================================

 * 1572 updates
 * 1255 stable updates
 * 198 testing updates
 * 43 pending updates
 * 76 obsolete updates
 * 734 bugfix updates (46.69%)
 * 236 enhancement updates (15.01%)
 * 93 security updates (5.92%)
 * 509 newpackage updates (32.38%)
 * 20 critical path updates (1.27%)
 * 103 updates received feedback (6.55%)
 * 6076 +0 comments
 * 156 +1 comments
 * 19 -1 comments
 * 243 unique authenticated karma submitters
 * 41 anonymous users gave feedback (1.22%)
 * 1176 out of 1572 updates went through testing (74.81%)
 * 1092 testing updates were pushed *without* karma (92.86%)
 * 19 critical path updates pushed *without* karma
 * Time spent in testing:
   * mean = 24 days
   * median = 18 days
   * mode = 16 days
 * 0 updates automatically unpushed due to karma (0.00%)
   * 0 of which were critical path updates
 * 10 updates automatically pushed due to karma (0.64%)
   * 0 of which were critical path updates
 * Time spent in testing of updates that were pushed by karma:
   * mean = 11 days
   * median = 7 days
   * mode = 7 days
 * Time spent in testing of updates that were unpushed by karma:
   * mean = 9 days
   * median = 5 days
   * mode = 5 days
 * 1060 packages updated (top 10 shown)
    * libguestfs: 26
    * znc: 10
    * vrq: 8
    * cherokee: 8
    * 389-ds-base: 8
    * viewvc: 8
    * 389-admin: 7
    * pki-ca: 7
    * wordpress-mu: 7
    * Django: 7

================================================================================
     Fedora EPEL 4
================================================================================

 * 447 updates
 * 359 stable updates
 * 40 testing updates
 * 11 pending updates
 * 37 obsolete updates
 * 222 bugfix updates (49.66%)
 * 68 enhancement updates (15.21%)
 * 40 security updates (8.95%)
 * 117 newpackage updates (26.17%)
 * 5 critical path updates (1.12%)
 * 11 updates received feedback (2.46%)
 * 1592 +0 comments
 * 11 +1 comments
 * 2 -1 comments
 * 85 unique authenticated karma submitters
 * 2 anonymous users gave feedback (0.24%)
 * 320 out of 447 updates went through testing (71.59%)
 * 311 testing updates were pushed *without* karma (97.19%)
 * 5 critical path updates pushed *without* karma
 * Time spent in testing:
   * mean = 18 days
   * median = 16 days
   * mode = 16 days
 * 0 updates automatically unpushed due to karma (0.00%)
   * 0 of which were critical path updates
 * 1 updates automatically pushed due to karma (0.22%)
   * 0 of which were critical path updates
 * Time spent in testing of updates that were pushed by karma:
   * mean = 11 days
   * median = 7 days
   * mode = 7 days
 * Time spent in testing of updates that were unpushed by karma:
   * mean = 9 days
   * median = 5 days
   * mode = 5 days
 * 313 packages updated (top 10 shown)
    * cherokee: 8
    * globus-common: 7
    * R: 6
    * voms: 6
    * globus-gsi-proxy-ssl: 5
    * globus-openssl-module: 5
    * globus-gsi-proxy-core: 5
    * bitlbee: 5
    * flashrom: 5
    * viewvc: 5

Gustavo Narea: Web Site Security With repoze.who and repoze.what

Tue, 01 Jun 2010 14:41:13 +0000

This article first appeared in the May 2009 issue of Python Magazine and has been slightly updated. The contents of the article are only applicable to repoze.who 1.0 and repoze.what 1.0, not repoze.who 2 and repoze.what 1.1 which are under development as of this writing.

Have you ever created a Web application? If so, it’s very likely that you have at one time or another faced “the security problem”; whether to create and maintain a homegrown security sub-system, or to learn to use framework-specific security mechanisms (which may not be as flexible as you wish).

Securing Web applications shouldn’t be a problem. This article explores a highly extensible alternative which you can learn once and use in arbitrary applications, regardless of the Web framework used (if any!).

Application security is a broad field within software development, covering topics ranging from low-level network transmission security and encryption, up to application-level data security and input validation. In this article, we will focus on two of the most basic elements of application security: authentication and authorization.

Authentication vs. Authorization

Even experienced software developers often confuse these two related but not equivalent terms, so we’ll start by explaining what they are.

Authentication, often shortened as “authn”, is what you do when you check the credentials provided by the user to verify that he’s really who he claims to be. The most widely-used credentials are a set made up of a user identifier (like user name, or email address) and a password.

Authorization, often shortened as “authz”, is what you do when you check whether the user has permission to make the request or perform the transaction he’s currently making. Most of the time this depends on who the subject is, but it may also depend on what is requested, when it is requested, and/or how it is requested.

A common shortening for “authentication and authorization” is “auth”.

And there’s one more related term that comes into play: Identification, which is to check whether he was successfully authenticated previously to avoid challenging him again unnecessarily.

Authentication, identification and authorization in Python Web applications

Web applications powered by Python may take advantage of repoze.who to handle authentication and identification, and repoze.what to handle authorization. Both are security frameworks that can be used on top of your application, or integrated with your application’s Web framework if you are using one.

You can use them as long as your application is WSGI-compliant. WSGI is a Python standard which defines an interface between HTTP servers and Python applications, similar to CGI, which eases the writing of cross Web server libraries (which can be framework-independent) and applications.

The WSGI standard mandates the availability of a Python dictionary, referred to as the “WSGI environment”. This dictionary stores CGI environment variables, as well as others specific to the WSGI standard; other components used by your application may also use the WSGI environment to store data.

Any framework or raw application that exposes the WSGI environment dictionary can use both repoze.who and repoze.what. The popular WSGI-compliant frameworks CherryPy, Django, Pylons, TurboGears and Werkzeug expose this variable.

How repoze.who and repoze.what work

Figure 1 illustrates how a typical request is processed in WSGI and when repoze.who comes into play.

Figure 1

repoze.who is a WSGI middleware. A WSGI middleware is a layer that wraps your application, processing each request before your application does. WSGI middleware can also post-process your application’s response. WSGI applications can have zero or more of these middleware layers.

As shown in Figure 2, when a request is made and before it reaches your application, repoze.who will try to check if the user is already logged in or if he’s currently trying to log in. If the user is trying to log in and has supplied the right credentials, authentication will succeed and the user will be “remembered” in future requests by default. The request is then passed to the next WSGI middleware, if any, and eventually on to your WSGI application.

Figure 2

The following fully customizable components can be used at this stage:

The request classifier: It matters if the agent is a browser, a Subversion client or a library which access a Web Service; you wouldn’t want to display a login form if the agent is a Subversion client, for example. Thus, this component classifies the current request so that only the appropriate plugins (out of your pre-selected ones) are used.
Identifier plugins: These are the components that “identify” the user. That is, they are able to tell if the user is already logged from in a previous request, or if he is trying to log in in the current request. When authentication succeeds, these plugins are in charge of “remembering” the user for future requests (e.g., by defining a session cookie). Likewise, when a challenge is required, they “forget” the user (discarding a session cookie).
Authenticator plugins: When the user is trying to log in in the current request, these components verify the user-supplied credentials (such as username and password) against a database, LDAP server, .htaccess file, etc.
Metadata provider plugins: If the user was authenticated, metadata providers can load data about the current user (email address, full name, etc.) so that such data is ready to be used by your application.

After your application issues a response and before it reaches the HTTP server, as shown in Figure 3, repoze.who will check if a challenge is necessary; that is, ask the user in some way to identify himself. If so and the user was previously authenticated, the previous authentication will be forgotten, and the related session cookie will be discarded. After the challenge is complete, your application’s response will be replaced with a new response which will allow the user to log in (for example, sending a “WWW-Authenticate” header or displaying a login form).

repoze.who’s response-handling functionality is also driven by customizable components:

The challenge decider: This is the component which will determine whether a challenge is required. The default challenge decider will order a challenge if and only if the response’s HTTP status code is 401.
Challenger plugins: When a challenge is required, those challenger plugins which support the current request type will be run until the first of them returns a valid response.

repoze.who may seem hard to use at first sight, with all its components and terminology; so much flexibility comes at the cost of being a little hard to understand. But one of the goals of this article is to help people who had never heard of repoze.who to deal with basic and advanced authentication settings.

And how does repoze.what work?

repoze.what is mostly used inside of your application, where you define the access rules that must be met for a given routine to be performed.

Access rules in repoze.what are made of atomic units called “predicate checkers”, re-usable objects that check whether a given condition is met. Predicate checkers can be single or compound to form complex access rules. For example, a single condition (or predicate) might be “The user is not anonymous”, while a compound predicate could be “The user is not anonymous and their IP address is X.X.X.X”. You can write your own predicate checkers, usually in just a few lines of code.

repoze.what has built-in support for the widely-used authorization pattern whereby you assign groups to your users and then grant permissions to such groups; it ships with a comprehensive set of checkers for the relevant predicates (e.g., “The current user belongs to the ‘directors’ group”, “The current user is allowed to edit user accounts”). If you use this authorization pattern, the groups and permissions for the authenticated user will be loaded by a repoze.who metadata provider; in repoze.what 1.1, they’ll be loaded on demand by repoze.what itself, so you wouldn’t need repoze.who. Nevertheless, it is entirely optional and you can use any authorization pattern that best suits your needs.

Predicate checkers allow you to control access based not only on who makes the request, but also on how the request is made and what exactly is requested. For example, if you have a blog application, you might use the simple predicate “The user is allowed to remove posts”, focusing on who the user is to control access to the post edition routine. Or you can have the more specific compound predicate “The user is allowed to remove posts, as long as the user is the post’s author”. This predicate focuses on both who makes the request and what is requested. You can have an even more complex access rule for the article edition routine, such as “The user is allowed to remove posts, as long as the user is the post’s author — except post #1 which nobody can remove”.

repoze.what 1.0.X supports only blacklist-based authorization, that is, authorization is granted unless explicitly denied. Whitelist-based authorization, in which authorization is denied unless explicitly granted, will be supported as of version 1.1 (under development as of this writing).

Creating a Web application protected with repoze.who and repoze.what

It’s time to go practical! We’re going to create a WSGI application powered by repoze.who and repoze.what.

I’ll use the TurboGears 2 Web Application framework to make this simple application, but after reading the article you should be able to put what you just learned into practice in other frameworks, or standalone applications.

(For the sake of demonstration, we will skip typical overhead features such as input validation, so we can focus the examples on the repoze.who and repose.what’s integration of authentication and authorization.)

Let’s get started!

We’re going to develop this application using an isolated Python environment using a rather famous utility called “virtualenv”. This is very handy because everything you install or remove won’t affect your system-wide Python environment. To install it, run:

easy_install virtualenv

You may need administration rights to install it, depending on where you install it.

Next, create an environment for our application and activate it; on Unix systems, the commands for this are:

virtualenv --no-site-packages appenv source appenv/bin/activate

On Windows systems, enter the following in a directory whose path doesn’t contain spaces:

C:\Python25\python.exe "C:\Path-to-VE\virtualenv.py" appenv C:\Path-to-newly-created-environment\Scripts\activate.bat

When you’re done and want to deactivate it, you should run the command “deactivate” on Unix systems. For Windows, use “C:\Path-to-newly-created-environment\Scripts\deactivate.bat”.

I’ve called my virtual environment “appenv”, but you can use any name you like.

For help installing virtualenv, you can check its documentation at http://pypi.python.org/pypi/virtualenv.

Now to install TurboGears 2:

easy_install -i http://www.turbogears.org/2.0/downloads/current/index tg.devtools

Generating an application

TurboGears allows you to start coding using a minimal application, so that you don’t have to start from scratch (unless you really want to). We’ll use just that minimal application so that we can get off to a quick start, which you can optionally download.

The Web application we’re going to create is a classifieds service, like craiglist.org or gumtree.com; for lack of imagination, I’ll call it “classifieds”. To start coding it from a minimal application, we’ll ask TurboGears to generate it with the following command:

paster quickstart classifieds

Then you’ll be asked a couple of questions. The first will ask you for the package name to be generated in this project; hit ENTER to accept “classifieds” as the default package name. The second question will ask if you need authentication and authorization features for this project, hit ENTER to accept the default answer of “yes”. By answering “yes” to the second question, repoze.who and repoze.what will be used in the application by default. Now switch to the application’s directory, install it in development mode and set it up:

cd classifieds python setup.py develop paster setup-app development.ini

We’re not going to start coding yet — I’d like you to try the generated application, so that you can see repoze.who and repoze.what in action. So, start the application (the “reload” switch will restart the application whenever one of its files is modified):

paster serve --reload development.ini

and then open the following URL in your browser: http://localhost:8080/.

You should keep this simple procedure in mind because we’ll use it very often. Then, when you need to stop the development server, you have to hit Ctrl+C.

Now try, at least, the following (in order):

Log in: Visit http://localhost:8080/login or click on the “Login” link in the upper-right side of any Web page of our application. Enter “manager” and “managepass” in the login and password fields. You’ll get redirected to the main page and a welcome message will be displayed.
Visit a page you shouldn’t see: Like http://localhost:8080/editor_user_only; you should get a 403 page and a message which reads “Only for the editor”.
Log out: Visit http://localhost:8080/logout_handler or click on the “Logout” link in the upper-right side of any Web page of our application. You’ll get redirected to the main page and a goodbye message will be displayed.
Visit a private page as anonymous: If you’re currently logged in, log out first. Then visit http://localhost:8080/manage_permission_only; you should be redirected to the login form, where also the message “Only for managers” is displayed. Log in with the previous credentials (“manager”/”managepass”) and you’ll be redirected to the page you requested initially.

What you’ve seen is repoze.who, repoze.what and some of their official plugins in action; those notification messages are TurboGears-specific, though, but it shouldn’t be hard to port them to other frameworks or raw applications.

Before moving forward, I’ll describe some of the files and directories that the “paster quickstart” command above generated:

devdata.db: The sqlite database file used for development.
classifieds/: Your application’s package itself.
classifieds/config/middleware.py: The file where the extra WSGI middleware for your application is added.
classifieds/controllers/root.py: Your application’s root controller. Sub-controllers should be attached to the one defined in this file; below I’ll explain how.
classifieds/model/: The application’s model definitions powered by SQLAlchemy.
classifieds/model/auth.py: The model definitions specific to authentication, identification and authorization.
classifieds/templates/: The application’s XHTML templates powered by Genshi.
classifieds/websetup.py: Contains the function which will get run when the application is set up (used by the paster setup-app command). By default it just adds rows to the database.

On the other hand, this is how authentication, identification and authorization is configured right now:

We have a table for our application’s users (tg_user), which contains the self-explanatory fields user_name and password, among others. repoze.who is configured to authenticate by using these two fields.
repoze.what is configured to use its groups/permission-based pattern. The groups and permissions are stored in the database, in the tg_group and tg_permission tables.
One user can belong to zero or more groups; one group can be granted zero or more permissions.
Right now we have two users, “manager” and “editor” (with passwords “managepass” and “editpass”). “manager” belongs to the only group defined so far, “managers”. The group “managers” is granted the only permission defined so far, “manage”.

Let’s start coding

“At last!”, I heard you say.

While implementing repoze.who and repoze.what in an application which doesn’t come with them enabled out-of-the-box, the first thing you have to do is add their middleware to your application.

However, we’ll skip that part and keep the default settings for now, so that we can go to fun part right away: Learning how to protect areas in your Web application. The setup will be addressed later on, where you’ll learn how to configure repoze.who and repoze.what the TurboGears-independent way.

This is what we’re going to implement in our classifieds application:

A simple user registration system.
A classified visualization mechanism, for any user (anonymous or authenticated).
A classified addition mechanism, for registered users.
A classified edition mechanism, for registered users to edit their own classifieds.

So it’s finally time to fire up your Python editor, a terminal and a window of your browser!

First of all, let’s add groups and permissions for authorization in our application, instead of sticking to the ones created by default. We’re going to add one more group called “posters” and the “add-classifieds” permission to be granted to posters. (Before continuing, you should stop the server running in the terminal — With Ctrl+C).

To add the posters group, add this code to the setup_app() function, defined in classifieds/websetup.py:

posters = model.Group(group_name=u'posters', display_name=u'Classified posters') model.DBSession.add(posters)

The addclassifieds permission is created with this code:

addclassifieds = model.Permission(permission_name=u'add-classifieds', description=u'Allowed to add classifieds') addclassifieds.groups.append(posters) model.DBSession.add(addclassifieds)

Add these sections right before the following lines:

model.DBSession.flush() transaction.commit()

Now, since we have a classifieds application, we have to define the SQLAlchemy model for the classifieds. To keep things simple, we’ll define just four columns:

classified_id: The classified’s identifier.
poster_id: The poster’s user id.
classified_title: The classified title.
classified_contents: The classified contents.

Listing 1 implements this model definition. You have to create classifieds/model/posts.py and store that definition in there. Then you have to import that model at the end of classifieds/model/__init__.py:

from classifieds.model.posts import Classified

To apply the changes, let’s remove the development database, re-create it with our new model and rows, and finally start the server again:

rm devdata.db paster setup-app development.ini paster serve --reload development.ini

That’s it, now we have the model definition for the classifieds. Let the fun part begin!

Creating a user registration system

We’re going to create a simple registration system made up of two controller actions: One to display the registration form and another to process the submitted form.

We’re going to implement them in the root controller for our application, the class RootController found in the source file classifieds/controllers/root.py.

To define the action for the registration form, first import the repoze.what predicate checker that verifies that the current user is anonymous:

from repoze.what.predicates import is_anonymous

Next, add the following method in RootController:

@expose('classifieds.templates.register') @require(is_anonymous(msg='Only one account per user is allowed')) def register(self): """Display the user registration form""" return {'page': 'user registration'}

What the two decorators above do is specify that the template for the “register” action is classifieds/templates/register.html and that access is granted to users who don’t have an account, respectively.

@require is a TurboGears-specific decorator that evaluates a repoze.what predicate before calling the action in question. When such a predicate is not met, the action is not called and a 401 response is returned (403 if the user is already logged in). Then repoze.who’s default challenge decider will find the 401 response and will replace it with the challenge.

Internally, the predicate is evaluated by calling its check_authorization() method (which raises the repoze.what.predicates.NotAuthorizedError exception if the predicate isn’t met, whose message is the user-friendly explanation) or is_met() (which returns a boolean). Listing 2 illustrates a fictitious implementation, using the decorator package; you’d find it useful if you want to use repoze.what in another framework or raw application (Pylons users may want to check repoze.what-pylons).

Going back to the creation of the action, you’ll have to create the template that will display the form: Create classifieds/templates/register.html with the contents of Listing 3.

Now it’s time to create the action which will process the contents of the submitted form by adding the user to the database, including them in the “posters” group and finally redirecting them to the login form so that they can use the newly created account. For that, you have to define the add_user method shown in Listing 4 (in RootController).

That’s it! Our registration system is done. Now you can try it by visiting http://localhost:8080/register.

The classifieds visualization mechanism

We’re going to write the part of the interface which will allow us to see the classifieds hosted by our service. This will be accomplished by means of two controller actions: One to see all the classifieds available, in the main page of the application, and another to see individual classifieds.

For the first, we have to re-write the “index” action of the root controller, to make it look like this:

@expose('classifieds.templates.index') def index(self): query = DBSession.query(model.Classified) all_classifieds = query.all() return dict(page='index', classifieds=all_classifieds)

Then its template, classifieds/templates/index.html, should be replaced with the contents of Listing 5, which lists the available classifieds with a link to their individual pages.

The second action, the one to show the classifieds individually, will be implemented as “view” and will be defined in the root controller too:

@expose('classifieds.templates.view') def view(self, classified): query = DBSession.query(model.Classified) classified_obj = query.get(classified) # Is the user allowed to edit the classified? checker = user_is_poster() can_edit = checker.is_met(request.environ) return {'page': 'Classified page', 'classified': classified_obj, 'classified_id': classified, 'can_edit': can_edit}

Note that in the “view” action we do something new: Handle a predicate checker directly. Sometimes it is necessary to evaluate them directly and get a boolean result to know whether it’s met or not thanks to the is_met() method of the predicate checker. For example, right now we use it to display a link to edit that predicate, if and only if the current user is allowed to edit it.

Then the template for “view”, classifieds/templates/index.html, is defined as shown in Listing 6.

Implementing the classified submission mechanism

To allow users publish classifieds, we’ll write two controller actions (once again, one to display the form and the other to process it).

The action that will display the form should first check that the user is allowed to add a classified; this is, we should use repoze.what’s has_permission predicate so that it checks whether they are granted the “add-classified” permission. You have to import it at the top of classifieds/controllers/root.py:

from repoze.what.predicates import has_permission

And then write the action as shown below:

@expose('classifieds.templates.add') @require(has_permission('add-classifieds')) def add(self): return {'page': 'Classified submission'}

This action uses the template classifieds/templates/add.html, which we have to create with the contents of Listing 7.

Finally, Listing 8 shows how the action that processes the submitted form is implemented. There we use something we hadn’t used before: The repoze.who identity dictionary. Do you remember that I said that repoze.who optionally uses so-called “metadata providers”, which are plugins that load data about the current user into the request? Well, that data is kept in the WSGI environment dictionary, under the repoze.who.identity key.

Then we use one of the items of that dictionary, “user”, which contains the database object for the current user. It is loaded by the metadata provider defined in the repoze.who SQLAlchemy plugin (repoze.who.plugins.sa), a component that is enabled by default in TurboGears.

Implementing the classified edition mechanism with custom predicate checkers

So far we’ve used a few repoze.what predicate checkers, which are all very basic. Very often you have to write your own checkers. For example, if the URL where classifieds are edited looks like http://localhost:8080/edit/X (where “X” represents the classified identifier), and we want that classifieds can only be edited by their posters, we’ll need a predicate checker that finds the poster of the “X” classified and checks if it is the current user.

This predicate checker is implemented in Listing 9, which should be stored in classifieds/lib/authz.py (a file you should create). That’s a good sample checker, which helps us to understand how a predicate checker is defined:

It must extend the repoze.what.predicates.Predicate class.
It must define a user-friendly explanation in the message attribute, which may be shown to the user when the predicate is not met.
Its logic is defined in the evaluate() instance method, which must call the unmet() method when the predicate is not met; keyword arguments passed to this method will replace the placeholders defined in message (if any), although that message can be replaced on-the-fly with a string passed as the first positional argument. evaluate() receives the WSGI environment and the repoze.what credentials dictionaries as arguments.
If the checker relies on arguments such as GET or POST variables, or other arguments available in the URL, the parse_variables() method should be used to retrieve them. It will return a dictionary whose items are: “get” for variables in the query string and “post” for POST variables, plus “named_args” and “positional_args” for named and positional arguments in the URL (which must be set by a routing software like Selector or Routes).

Because this predicate relies on a named argument in the URL (“classified”), we have to use a URL router software compliant with the wsgiorg.routing_args standard; we’ll use Routes. To configure Routes in TurboGears 2, you have to insert the following contents in classifieds/config/app_cfg.py (right after the imports) and then replace the line base_config = AppConfig() with base_config = ClassifiedsConfig():

class ClassifiedsConfig(AppConfig): def setup_routes(self): """Customize routing""" from tg import config from routes import Mapper dir = config['pylons.paths']['controllers'] map = Mapper(directory=dir, always_scan=config['debug']) # Defining our custom routes: map.connect('/{action}/{classified:\d+}', controller='root') # Required by TurboGears: map.connect('*url', controller='root', action='routes_placeholder') config['routes.map'] = map

At this point we’re ready to use the user_is_poster checker.

Now import the custom checker into classifieds/controllers/root.py:

from classifieds.lib.authz import user_is_poster

Next, use the following contents to define the “edit” action and the contents of Listing 10 for its template (classifieds/templates/edit.html):

@expose('classifieds.templates.edit') @require(user_is_poster()) def edit(self, classified): query = DBSession.query(model.Classified) classified_obj = query.get(classified) return {'page': 'Classified edition page', 'classified': classified_obj, 'classified_id': classified}

Finally, for the action that processes the submitted form, we can use:

@expose() @require(user_is_poster()) def edit_classified(self, classified, title, contents): # Fetching and updating the classified object: query = DBSession.query(model.Classified) classified_obj = query.get(classified) classified_obj.classified_title = title classified_obj.classified_contents = contents DBSession.update(classified_obj) # Notifying the user: flash('Classified "%s" updated!' % title) redirect(url('/view/%s' % classified))

You can now play with the classifieds edition mechanism, to see by yourself how authorization is denied when somebody tries to edit somebody else’s classified, thanks to our “user_is_poster” checker.

Configuring it all by ourselves

At this point you’re able to deal with identification (using the repoze.who identity dictionary, which contains user data) and control authorization in your application with repoze.what predicate checkers (even how to write your own!), and you should also be able to put this knowledge to the test in frameworks other than TurboGears.

What we’re missing now is to know how to configure repoze.who and repoze.what by ourselves, since we skipped that part initially because TurboGears configures them for us. But you have to know this to use both packages with other frameworks or even to continue with TurboGears in a more advanced setup.

Therefore we’re going to stop TurboGears from configuring repoze.who and repoze.what, so that we have full control on how they are configured and learn how to do it in other frameworks.

To disable the automatic setup of repoze.who and repoze.what, go to classifieds/config/app_cfg.py and set the variable base_config.auth_backend to None. Then all those variables that start by base_config.sa_auth will be ignored, so you can remove them if you want.

Now let’s handle the configuration by adding the middleware to our WSGI application. I’ll use a function called add_auth() (defined in classifieds/config/auth.py) which receives the WSGI application as the only argument and returns it with the middleware added, as shown in Listing 11.

Because add_auth() configures repoze.who and repoze.what the same way we’ve been using them, but with the hidden details revealed, we’ll be able to identify their components.

In this function we see that repoze.who is configured with the following plugins:

AuthTktCookiePlugin, an identifier which remembers and forgets authenticated users using cookies (using the string “secret” as the encryption key and “authtkt” as the cookie name).
FriendlyFormPlugin, the component that handles our login form and logouts. As an identifier, when the user is logging in it extracts the “login” and “password” from the request so that the authenticator(s) can use such data, and when the user tries to log out, it asks AuthTktCookiePlugin to forget the user. As a challenger, it redirects the user to the login form (at “/login”).
SQLAlchemyAuthenticatorPlugin, as the only authenticator used. It connects to the “tg_user” table to check if there’s a match for the supplied “login” and “password”.
SQLAlchemyUserMDPlugin, the metadata provider that loads the current user’s SQLAlchemy object into the repoze.who identity item “user”.
Because we didn’t specified request classifiers or challenge deciders, the default ones will be used.

Meanwhile, repoze.what is configured using the groups/permissions-based authorization pattern, where the groups and permissions are retrieved thanks to the following adapters:

SqlGroupsAdapter, which loads the groups from the “tg_group” table.
SqlPermissionsAdapter, which loads the groups from the “tg_permission” table.

If we don’t use this authorization pattern, our app_with_mw variable would have been defined without passing groups/permission adapters:

app_with_mw = setup_auth(app, **who_args)

And what about using repoze.who but not repoze.what? Listing 12 shows how to configure repoze.who just like we did above, but without repoze.what. Note that this time we had to pass the request classifier and challenge decider explicitly.

The opposite, using repoze.what without repoze.who, is not yet possible as of this writing because repoze.what 1.0′s credentials are loaded through a repoze.who metadata provider. repoze.what 1.1 will be completely repoze.who-independent, optionally.

Finally, it’s time to use add_auth(). It can be used like any other WSGI middleware, so in the case of this TurboGears 2 application, it is in classifieds/config/middleware.py; if using another framework, consult the relevant documentation to find the equivalent. First, you should import the function:

from classifieds.config.auth import add_auth

And then use it inside the make_app() function, under the specified line:

# Wrap your base TurboGears 2 application with custom (...) app = add_auth(app)

And voila! Now our classifieds service behaves the same way as before, with its authentication, identification and authorization settings now being controlled by our own code instead of the generated defaults.

Going beyond with repoze.who and repoze.what

The topics covered in this article are just the tip of the iceberg. Both Repoze packages are created with extensibility in mind; their core is minimalist but they already have many ready-to-use plugins, not only the repoze.who and repoze.what SQLAlchemy plugins we used here.

There are repoze.who plugins for OpenId, LDAP, .htaccess and RADIUS authentication, as well as a built-in challenger plugin for HTTP authentication — just to name some of the available plugins. And you can easily create your own plugins, following the patterns described in this article.

Although repoze.what is a relatively new piece of software as of this writing, it has several ready-to-use plugins as well. It has plugins to store the groups and permissions in XML files or .ini files, not only in databases, as well as a plugin called repoze.what-quickstart which allows us to have repoze.who and repoze.what working quickly (that’s what TurboGears uses to set them up).

Although they were not used in our classifieds service, just mentioned in the beginning, you can have so-called compound predicates. Access rules aren’t always as simple as “The user must be logged in” or “The user belongs to the ‘posters’ group”. For example, in our classified edition mechanism we way want to allow administrators to edit classifieds, even those not posted by themselves; to do so, instead of using our user_is_poster checker alone, we could use it along with the Any and in_group checkers:

from repoze.what.predicates import Any, in_group p = Any(in_group('manager'), user_is_poster())

In this article we didn’t even use half of the built-in repoze.what predicate checkers. A full list is available in the repoze.what manual.

Settings are very flexible. It is even possible to configure repoze.who and repoze.what through .ini files, so that those settings can be adjusted while deploying the application (which would be a replacement for our add_auth() function defined above).

It is also worth noting that both projects are actively developed, well documented, well tested and supported by the Repoze community. As a result, it is most likely that you’ll have a good experience using them.

repoze.who and repoze.what aim to solve “the security problem” we Web developers face so often, and they have proved to be the right choice in many scenarios. The likelihood of them being the right choice for your next Web application is strong, specially now that you’re familiar with them!

To ask questions about repoze.who and/or repoze.what, please use the repoze-dev mailing list. For everything else that is related to this article, please leave a comment below.

Michael Pedersen: Announcing TurboGears EC2 Images

noreply@blogger.com (Michael Pedersen) — Mon, 31 May 2010 20:00:07 +0000

Amazon has long offered a service whereby the community can purchase time on their clusters. Today, TurboGears is able to take advantage of this in a new way. It is now easier than ever before to get a TurboGears instance up, running, and viewable by you, allowing you to find out how that TurboGears based application really looks and functions.

The whole process is actually fairly simple. You will need to know the AMI ID of the version of TurboGears you wish to test. Here are the AMI IDs we have available now. This post will be updated when new versions of the AMIs are made available.

TurboGears 2.0.3, 32 Bit - AMI ID: ami-c5648dac
TurboGears 2.0.3, 64 Bit - AMI ID: ami-a17e97c8
TurboGears 2.1b2, 32 Bit - AMI ID: ami-67678e0e
TurboGears 2.1b2, 64 Bit - AMI ID: ami-7b719812

Installing your application and getting it live is very simple. If you don't know how to work with Amazon's EC2 service, please read their documentation for it. Here's the process:

Launch an instance of the AMI you have selected.
Log in to the instance:
ssh ubuntu@public.dns.name
sudo su - turbogears
Install the application following whatever directions are provided by the application, but do not run "paster setup-app" yet.
Instead of making "production.ini" or "deployment.ini", you will make "/home/turbogears/app.ini". This is absolutely identical to the other files, it's just the name that is special. Yes, you can modify configuration to use one of the other names, but you do not have to.

When you do make this file, don't forget to set "debug=false" in it. This is meant to be a production environment, and will break if "debug=true" is set.
Run the following two commands:
cd $HOME
paster setup-app tgapp.ini
If your configuration uses a SQLite database, don't forget to fix the permissions. Either make www-data the owner, or set the permissions to 0777.
A couple of files need to be made, but will be made automatically for you. When you installed the application in step 3, a Python egg was installed. You will need the name of that egg (but can find it fairly easily). A command has been written to make this whole part of the process painless. Do this:

tglinker eggname eggversion

If you are unsure of the eggname, simply type "tglinker", followed by a space, and then press the tab key twice. You will be given a list of all installed eggs, and can start typing the name in from that list. Once you get enough to be unique, press tab again, and the name will be completed.

Once you have that, the eggversion will be even easier: Pressing tab twice will likely fill in the full version number. If not, simply type it in again usaing the tab completion as above.

Press enter, and the required files will be generated.
Finally, logout of the turbogears user account (either by using the "exit" command, or the "logout" command), and restart apache using this command:

sudo /etc/init.d/apache2 restart

You're done, and by visiting "http://public.dns.name" in your browser, you will now see your web site function.

A word of caution: If you are a new user of the EC2 service, you might well terminate any instance that you create. All data on the local drive will be destroyed. Their documentation covers how to preserve data. I recommend you read it.

Luke Macken: liveusb-creator trojan in the wild

Tue, 25 May 2010 21:11:25 +0000

I've been noticing many different copies of my Windows liveusb-creator popping up on various sketchy-looking download sites. The majority of these copies contain a variant of the Vundo Trojan.

"Vundo, or the Vundo Trojan (also known as Virtumonde or Virtumondo and sometimes referred to as MS Juan) is a Trojan horse that is known to cause popups and advertising for rogue antispyware programs, and sporadically other misbehavior including performance degradation and denial of service with some websites including Google and Facebook."

So, if you downloaded a copy of the Windows liveusb-creator from anywhere other than https://fedorahosted.org/liveusb-creator -- you could be infected. Apparently the latest variation of this trojan is undetectable by most antivirus (although, clamav was able to recognize the one that I found), so you may need to look around for some of the common symptoms. There is apparently a tool that will remove this trojan which can be found here, however I have not tested it and cannot vouch for its validity.

If anyone was actually hit by this, I'd be interested to hear about it.

Also, to state the blatantly obvious: only download the liveusb-creator from the homepage!

Michael Pedersen: Announcing tgext.xmlrpc

noreply@blogger.com (Michael Pedersen) — Mon, 24 May 2010 10:43:57 +0000

One of the questions that we in the TurboGears community get asked is "How can I have XMLRPC in my TurboGears application?" Until now, we didn't have a useful answer beyond "Write it up". Now we have tgext.xmlrpc.

tgext.xmlrpc allows you to have an XMLRPC hierarchy. Your code becomes a case of simply setting up your controller and working with it like any other controller. So, if you need to have XMLRPC in your TurboGears application, check it out (and don't forget to read the docs!). I think you'll find it quite helpful.

Jorge Vargas: Twittdo en 3923 caracteres (primera parte)

Mon, 19 Apr 2010 17:03:40 +0000

El dia comenzo triste, lloviendo y nublado. Sin embargo la gente llego desde temprano y todo el mundo madrugo (menos yo). Dato curioso cuando me levante y vi que eran las 9 y pico, lo primero que hice fue abrir @tweetdeck y el stream de @onlineteamDO cuando me di cuenta que me estaban esperando para comenzar me aliste lo mas rapido que pude y llegue cuando ya habia pasado “el incidente de la fila” asi que entre de una vez y le dieron permiso a @ggroovin de que comenzara. Por lo tanto les pido disculpas a todos por el retraso en el comienzo del evento, fue 100% culpa mia.

Dejando el chiste malo de lado la charla de Ricardo Guerrero fue interesante, me gusto el hecho de lo rapido que se dieron cuenta de la excelente herramienta que encontraron generalmente las empresas grandes tienen muchos problemas con eso, creo que les ayudo mucho el funcionar independiente. Creo que la mejor leccion de esa charla es que una gran empresa puede funcionar como una pequeña si se estructura bien.

Luego el coffee break, la parte mas interesante de este fue el ponerse al lado de alguien aleatoreo y decir hola.

El panel de las empresas me lo encontre 50/50 la tipa de @tricom sinceramente me durmio no recuerdo una sola palabra de lo que dijo. Por otro lado la presentacion de @yokomosushi fue excelente como le mande a decir lastima que a mi el sushi no me gusta, voy a tener que cambiar eso, simplemente una ejemplo de lo que el servicio al cliente deberia de ser. La gente de @aeropaq y @diariolibre se manejaron bien pero no vi nada resaltable.

La charla de @tpago fue interesante es un projecto excelente ya que cubre el tercer modelo de revenue que ha funcionado en Internet (siendo los otros dos ads y subscriptions) y si logran implementar los micropayments estoy seguro de que todos lo vamos a querer usar. Yo soy uno que cada vez que firmo la tarjeta uso algo diferente. Y por otro lado me encantaria poder implementar micropayments en cualquiera de mis apps orientados a RD. Aunque tengo dos dudas que me tienen preocupado. #1 Quien paga por el sistema? La respuesta de @yaqui a la pregunta me dio a entender 3 cosas. a) Pagan los usuarios con una mensualidad (ya que hay que ser miembro) b) Pagan los bancos y empresas crediticias (por tener el pribilegio? de cobrar por ese sistema) c- el sistema es mantenido por GCSystems y se mantiene magicamente. El miedo es simplemente porque si nos apuntamos en este sistema queremos que funcione por los proximos 5 años y para eso se necesita un modelo de negocio valido y solido #2 Como yo cobro por aca? si yo tengo un app, un negocio, una fruteria (ejemplo que le molesto mucho a @cor3d) como cobro?? Lamentablemente en materia de $$$ hay que tener un plan solido y no estoy diciendo que no exista sino que no lo he visto, hay que publicar eso. Update: mientras escribia esto me llego un tweet de @tpago http://twitter.com/tPago/status/12464325608 diciendo que sea a, la pregunta entonces cambia es a+b o solo a?

El panel de tasa cero me encanto. Lo unico que necesitamos es llevar eso a la realidad. QUe dicha que hay mas gente pensando “Hay que declarar el internet como bien público que de cáracter universal” Lo segundo que nos tenemos que llevar de esto es la burbuja en la que vivimos solo el 30% de los dominicanos tienen acceso a internet y tener internet en tu casa cuesta el 35% del sueldo minimo.

Luego la foto de grupo (link ?) y el almuerzo

La charla de Norberto Gobbi (@ngobbi) fue la que mas me gusto mucha informacion interesante y muchos pequeños detalles ese es un video que hay que verlo para tratar de captar todas los detalles que se perdieron.

Luego la seccion de preguntas y respuestas no se de donde salio, ironicamente fue una de las que mas me aburrio aunque hubo una gran participacion. Eso estaba planeado?

El #barcamp fue la parte mas interesante del evento entero, pero ya me harte de escribir :) asi que se quedara para un segundo post.

Alessandro Molina: ACR got support for user permissions

Wed, 07 Apr 2010 21:15:28 +0000

ACR opensource Turbogears2 CMS got support for user permissions to allow users to edit only some pages and create children only in some sections. This should permit to separate work between multiple people in ACR based sites.

ACR also got support for blog/news slice template. This permits to create blogs in ACR with just two clicks instead of having to declare the ACR slicegroup youself.

As usual you can download ACR from http://repo.axant.it/hg/acr by using mercurial

Luke Macken: Fedora Community Statistics

Sat, 27 Mar 2010 03:48:25 +0000

I'm pleased to announce that version 0.4.0 of the Fedora Community dashboard has just hit production. Along with the usual batch of bugfixes, this release contains a new 'Statistics' section that contains metrics from a variety of different pieces of Fedora Infrastructure.

Thanks goes to Ian Weller for the wiki stats, Seth Vidal for the torrent stats, and Matt Domsch & Jef Spaleta for the map generation code. I ended up writing the updates metrics, package stats, and users/mirrors widgets. Enjoy!

Planet TurboGears

Alessandro Molina: Mastering the TurboGears EasyCrudRestController

Mengu Kagan: 2011 At A Glance

Things I've Done in 2011

Mengu Kagan: Subqueries With SQLAlchemy

Ralph Bean: threebean

Michael Pedersen: Announcing TurboGears 2.1.4 Release

Curia: An rsync-like interface for Amazon S3 and Google Storage

Mengu Kagan: What's Going On With TurboGears?

A Short Introduction First

Alessandro Molina: TurboGears2 DebugBar

Cliff Wells: FreeSWITCH on Scientific Linux 6.1

Michael Pedersen: Request For Ideas for Hiring Pond

Alessandro Molina: TurboGears2 Performance Improvements

Ralph Bean: threebean

Michael Pedersen: Announcement: TurboGears2 2.1.3 Released!

Ralph Bean: threebean

Christpher Arndt: Hurra, hurra, der Herbst ist da!

Michael Pedersen: TurboGears and Ming/Mongo

Curia: The long overdue “Shootout” update

Michael Pedersen: Announcement: TurboGears2 2.1.2 Released!

Ralph Bean: You should be all gravy.

Setting up your environment

Setting up repoze.who.plugins.ldap

Give it a test

Mengu Kagan: Deploying TurboGears 2.1 Application With Nginx And uWSGI

Christpher Arndt: ELKA PM 13 MIDI Basspedal (Version B)

Curia: Google+: It’s a bigger deal than Facebook, but not like you think

Alessandro Molina: TurboGears 2.1.1 released!

Michael Pedersen: Announcement: TurboGears 2.1.1 Released!

Michael Pedersen: Announcing TurboGears 2.0.4

Cliff Wells: I joined Facebook

Luke Macken: Red Hat OpenShift Express & The Leafy Miracle

Cliff Wells: Ubuntu 11.04

Alessandro Molina: Mobile devices detection with TurboGears2

Christpher Arndt: Tag der freien Dokumente

Luke Macken: git clone all of your Fedora packages

Luke Macken: Fedora Photobooth @ SXSW

Michael Pedersen: A Git Script for Python

Luke Macken: FUDCon 2011 Tempe

Sessions

The Next Big Project

Hackfests

Alessandro Molina: Fixing broken inline genshi tags on TG2.1

Luke Macken: liveusb-creator 3.9.3 windows release

Alessandro Molina: Workaround for empty helpers in Turbogears 2.1

Compound Thinking: TurboGears Joins the Pylons Project

Compound Thinking: Technical Debt isn’t always Debt

Michael Pedersen: Announcing gogo, The Bash Project Switcher

Alessandro Molina: Lowering Tg2 memory usage by running multiple instances of an app inside same WSGI daemon process

Alessandro Molina: Patched TG2 bootstrap script for python 2.6

Michael Pedersen: Announcing the Availability of tgext.menu v1.0b3

Michael Pedersen: TurboGears, SQLAlchemy, and Parent/Child Relations on the Same Model

Michael Pedersen: Announcing the Availability of tgext.menu v1.0b2

Michael Pedersen: TurboGears and Amazon EC2 Benchmarking, Part 4: Conclusions

Michael Pedersen: TurboGears and Amazon EC2 Benchmarking, Part 3: What I was Testing

Michael Pedersen: TurboGears and Amazon EC2 Benchmarking, Part 2: Requests per Second

Jorge Vargas: Hoy Dom Sagolla en RD, #sagollaRD y el futuro

Luke Macken: Professors' Open Source Summer Experience @ RIT

Michael Pedersen: Announcing tgext.xmlrpc v0.8

Michael Pedersen: TurboGears and Amazon EC2 Benchmarking, Part 1: Methodology

Luke Macken: Fedora Updates Report

Gustavo Narea: Web Site Security With repoze.who and repoze.what

Authentication vs. Authorization

Authentication, identification and authorization in Python Web applications

How repoze.who and repoze.what work

And how does repoze.what work?

Creating a Web application protected with repoze.who and repoze.what

Generating an application

Let’s start coding

Creating a user registration system

The classifieds visualization mechanism

Implementing the classified submission mechanism

Implementing the classified edition mechanism with custom predicate checkers

Configuring it all by ourselves

Going beyond with repoze.who and repoze.what

Michael Pedersen: Announcing TurboGears EC2 Images

Luke Macken: liveusb-creator trojan in the wild

Michael Pedersen: Announcing tgext.xmlrpc

Jorge Vargas: Twittdo en 3923 caracteres (primera parte)

Setting up `repoze.who.plugins.ldap`