Planet Turbogears

Compound Thinking: TurboGears 2.0.1 now available on pypi

Mon, 22 Jun 2009 20:21:32 +0000

Since the beginning (nearly) of TurboGears 2.x development we use a private index to store all our dependencies and point people to it for installation. This was nice because it helped us control our dependency tree and make sure installation of our unstable software was as easy as possible.

Since we had released a stable 2.0 version, followed by a 2.0.1 bugfix, we really were lacking a real pypi install mechanism. This is now a thing of the past! You can install a full blown TurboGears2 environment in one simple command:

easy_install tg.devtools

and this will fetch everything using normal lookups from the pypi. I hope this will help people out there.

At the same time I’d like to ask for help in that domain: If you are using TurboGears2 and find out that some dependency that needs compilation is not found in a compiled state (either on pypi or on our index), please help us:

By sending a compiled version of the egg to me (florent aide, you’ll find me on the mailing list easily)
By pointing us the missing dependency (send a mail in the turbogears or turbogears-trunk mailing list)
Or even better: try to become a maintainer for the specific dependency, that you know will always be missing for your architecture, and let us know about that great news. The whole open source community will become better, your karma (the real one, not the one you can admire on ohloh) will bump-up and you’ll be forever remembered on endor.

Now this leaves us with one more thing to do: finish the web site for turbogears that is currently in a poor state. If you would like to help us on the engine, please let us know in the mailing lists or via private message. If you want to help-out with content, please let us know also and we’ll be more than glad to give you an editor account and some directions to get some content online in this new website.

Cheers all,

Florent.

Compound Thinking: TurboGears 2.0 presentation video

Mon, 15 Jun 2009 08:29:50 +0000

In the last post I said we presented TurboGears 2.0 at the Pycon FR event. We have received the video feeds from the organization. This is a French presentation, sorry for the non-french speakers.

As a reminder the code of application we presented is available on bitbucket.

Compound Thinking: TurboGears 1.x to Turbogears 2.0

Fri, 12 Jun 2009 10:50:56 +0000

Two weeks ago we (Jonathan Schemoul, Thomas Zaffran and me) animated a TurboGears 2.0 presentation (in french) and workshop at PyconFR 2009. PyconFR is an annual French Python event organized by the AFPY.

The presention was separated in two phases. The first part covered an history of the project and a technical overview. The second part was a live coding session during which we created a small todolist application to show-off the new TurboGears 2.0 features.

At the end of this presentation, one of the attendees came to us and gave us a nice remark that I would like to share with you here: “I did not remark any changes in TG2 since my TG1 days, did you change something?”

This was a pleasant surprise to me. This means one thing: we changed the whole underlying layers, rewrote authentication and authorization, switched tgwidgets with ToscaWidgets, based our Framework on Pylons instead of CherryPy — and yet this person, looking at the controller’s code of our todolist application felt “at home like if it was a TurboGears 1.0 application”.

As we already stated, we want to ease-out the tg1–>tg2 transitions as much as possible. We know there is some room for improvement in that quarter! But since we (core devs) have live and running tg1 applications out there the itch is already there and we’ll make sure to scratch it.

As an example, Paris Envies is a TurboGears 1.1 application, coming from an 1.0 background, using Genshi, SQLObject and TGWidgets. And I know for sure that Jon has a real incentive to move to TG2 and get SO support for it :)

Stay tuned people, TurboGears is live and well!

Christopher Perkins: Sprox 0.6.1 released

Fri, 05 Jun 2009 19:28:21 +0000

Well, it’s been longer than I wanted since the last release. (almost 4 months) I was holding out because I wanted to get a few more features working, but I realized this week that there was a lot of new functionality, and so it was time to cut a new release. It turns out, the codebase grew by almost 20% with this release. I see this as a very stable release because I use almost every new feature at my day job. Here’s a rundown on what is new.

* TableFiller can provide field_methods for customization of output.
* You can now customize the query TableFiller uses.
* TableBase now has __xml_field__ modifier that un-escapes the data provided by the given list of fields.
* There is a new tutorial for TableFiller: http://www.sprox.org/tutorials/table.html .
* Sqashed #8: Synonym Properties causing Tracebacks.
* Added Mako Templates for every sprox widget. (SPEED!)
* Fixed the way relation form fields render.
* Brandon Rhodes provided a fix for TableBase that allows fields to render with any Widget type.
* Teajay Bernard provided *experimental* support for Editable Dojo Grid.
* Test framework fixed to be more forgiving of XML.
* Declarative Validation overrides now work properly.
* Field Class added so that you may override both the Widget _and_ a Validator for a field (documentation pending).

Special thanks to Brandon Rhodes and Teajay Bernard for their contributions.

You can get sprox from pypi at: http://pypi.python.org/packages/source/s/sprox/sprox-0.6.1.tar.gz

Jorge Vargas: “Everything” and Wave

Mon, 01 Jun 2009 02:58:02 +0000

Outline: I talk a little about what is wave, then what I think of wave and last about “everything” which was my idea/prototype of what it should be.

So What is wave?
Apparently most of the people on the “blogosphere” have no clue. And currently the best source (other than the video, go watch that first) is at the 2 google groups. I’ll start with a brief explanation. Wave is actually 3 totally separate things.

A communication protocol
A very efficient client
A set of extensions to that client.

What most people miss is that wave is a protocol (apparently a set of extensions to XMPP), now be careful! this protocol is ONLY for server-server communications, which means this is how waves travel across the interwebs and has nothing to do with posting to the wave or reading them. The real advantage of this is that synchronization happens at a very fast level which means thin clients, With something they call “operational transformation” as this guy explains in this video. And the other really interesting thing is that it enables private servers with private waves, without creating walls.

Then we have the client which is what everyone is drooling about, which is simply a realllly cool editor and collector. It is build with GWT and apparently comet. This seems to implement the realtime typing, search, the playback and drag & drop goodness we saw. It is a remarkable webapp and I can’t wait for them to release it.

The third component which is the most interesting part for me to play with. Is actually 3 different things Robots, Embed, and Gadgets.

The gadgets are nothing new, we all know how you can add a youtube video in a site that is not youtube and do stuff with it, same with a google map, etc. The only innovation here is that the gadget can access the data from the wave.
The Embed api is pretty much a “viral tool”, the goal is to take waves to non-wave environments. Which gets you several bridges to pretty much everything out there. I call this viral because it is what is going to make this a huge thing. After someone sees that my “blog’s comments” are in this super cool system, they will track that system and say ohhhh wave is cool!
The part I’m most exited about is the robots, simply because they are first class citizens. And that means that anyone with a little knowledge of how the system works is going to be able to create automatization, and this is so powerful that it is scary. In the good uses of this I can see (and will write) 100s of really useful tools. Just think about it for a second, the robot will know everything the people will know about the wave. Now the scary part is when someone takes this freedom and applies it to evil.

That pretty much covers it, how is this different from anything else out there. It’s goal is to unify not subdivide the fact that they are releasing the protocol means you will be able to run your own wave server, which means private really private waves. This is totally useless for “where are we going to eat?” but it’s very important for “We are broke and only have 3 million dollars left on the bank account, what shall we do?”. The other really cool thing is that they are going to open source everything (let’s hope that’s true) which means the client will solve the “how to build a real time system” problem, and developers can focus on their tools.

But this is not a me-too post. In fact someone recommended me not to post this but I have to. I have been thinking of a system like this for a long time, seriously I have chatlogs, trac tickets, svn repositories (yes that long!), a prototype on appengine, another in mercurial and another as a TG2 app with first had a zodb backend and recently a mongodb. I also have some work on the frontend but I wasn’t thinking real time. So all in all it has to be at least 5 years of working around this idea. This is also on the planned roadmap for “codemill” and “DVDev” (a totally different post). And my last prototype of this idea is named “everything”, hence the title of this post. I though it was a little cocky at first but I think I was right, wave is indeed about everything.

Of course as the famous quote “There is someone smarter than you outside of your organization”, I have not been able to lift off my idea as I’m pretty much a one man team with no resources, and I’m not smarter than the wave team. Which means I’m going to be an early adopter here and build “everything” on top of wave.

Analysis of Wave

What they got really right.

Robots as first class citizens
- This means they realize the potential behind having computer helpers on your day to day tasks. With the correct set of bots you should be able to enhance your writing to scary levels of enlightenment.
One instance of the wave
- I think this the most important part of the system (but I put robots first because I’m so exited about them!), the web today is too fragmented, just as an example try to search for the first public non-official announcement of wave. This is practically impossible because the web is a huge copy&paste fiasco. The wave format is something to unify all the systems under the same data structure but it keeps them all independent.
Server-Server replication
- The beauty of this is that all the wave servers take care of the heavy lifting which means the clients are very simple (in coding). This is probably going to be a key feature for the adoption of the system. Most existing clients (mail,IM,GUIs,webpages) and forms of presenting the data (websites,phones,native clients,etc) simply need to learn how to read and write from the wave format.
Plugable as hell
- you can write your own client, your own bits (gadgets), your own functions (robots) and your own server
one data structure to rule them all
- IF wave-protocol gets to be the success I hope it will be, this is probably as big as XML. Before XML each system/company/app/team used their own little format for moving data across applications (most binary, but a lot text). So back then you will have to code something to talk to each one of the applications you wanted to talk to. Why XML was so big? It lifted the barriers on the data. How wave-protocol is like XML, it’s going to lift barriers on the data.
It’s a messaging system
- Which coincidentally is how people communicate. All the old messaging system share one fundamental flaw they are synchronous you can measure when a message starts and when it ends. In wave the message is one character long which is as close to speaking as we can get, in fact it’s better because it’s not noisy
A solution to the “where”
- today you need to answer where shall we discuss this, should I call you, email, write to the mailing list, IM, twitter, bah! wave’s answers: “everywhere”

What they got almost right

XML, to be honest they have a very good and clean xml file. But why not json or even yaml? Although this post is enlightening. I honestly think they didn’t choose XML, XMPP did that for them. Maybe json is a better storage and transfer system but they will probably make them have to sit and re-write XMPP and I’ll be writting this post in 2012 or later.
One true editor, I’m not sold to the idea of having one editor. I really think we’ll need several specialized editors for the different content we’ll like to create on waves.
Appengine only bots. I assume this is either because of
- a) to be able to kill an offensive bot
- b) to protect wave with the sandbox (you can’t compile a worn on appengine)
- c) to give appengine popularity.
In any case I hope this is lifted at some point, or I’ll just use a comunity server

The biggest challenges

spam: I think this is the most important aspect of the system. will it die like usenet? could the protocol be spam-proof? how can you prevent spam without coercing the users?
robots: leaving spammy out. Where is the line between helper robots and annoying robots, how can you trust a robot? how will you identify the evil robots?
uncooperative behavior: this is similar to the robots problem, and in fact it’s tied to it. How do you deal with people that where added to the wave and are against it’s usefulness.
authorization: Who should be allowed to edit the wave? some waves need to have ACL, should this be at the protocol level?
some readonly parts: how and who should declare what is “set in stone”?
massive forking/endless hours finding the important waves: if we give too much power to the owner of the wave then we risk fragmenting the conversation and having 100s of “missed update” where you could be useful. If we give too little power caos will take over.
ownership: who owns the content? who owns the wave? up to which point my original wave is really mine?

The future of “Everything”

So as I said above I am/was writing “everything”. And I think it is still a viable project. I’m just going to retarget it to be wave-compliant.

how is “everything” different from wave?

specialized editors: I am really thinking of having editors for each type of data. For example a ticket editor, code review editor or the more traditional rest,html,plain editors. Since this is pure javascript I’ll probably retrofit them to be gadgets ones the client is release, or they could evolved into their own client.
Per server robots: To be honest I’m a bit scared about the robots in the wave, I think the wave team went too far with robots and I’m really interested in knowing how they plan to keep them at bay. In my idea the robots where local and I was thinking of them as helpers rather than contributors, in wave terms my robots where limited to the blip, and I think this is a good idea for your own set of minions you don’t want other people to use. I’m going to push this idea forward and I’ll code them in wave terms so they are compatible and most likely useable as google’s wave-client bots. From the source it seems you can build a standalone robot server with what they released.
publishing system. I originally though of “everything” as a publishing system. But in the form of editorial reviews rather than wiki-style. My idea was to have one true editor to a website with the means for a proper feedback. And I’ll push this to happen on the wave. I really think we need a two tier system for some waves, where you have a set of authors and a massive set of feedback. In the simplest way this will be used to write a blog. In a more complex case it will be a mailing list to discuss a project/product/idea. In a even more complex case it will serve to write a paper or a book. And in the most complex case it will serve to record your live and thoughts.

So that’s about it. Sorry for the huge ass wave but I think this is a topic that deserves a lot of thought I’ll leave you with what I think is the most inspiring message currently at any of the google groups regarding wave.

Compound Thinking

Wed, 27 May 2009 18:21:29 +0000

The long wait is finally over!

I am happy to announce the release of TurboGears 2.0 final. This release is the product of a lot of work by the whole TurboGears team, and we’re very happy to have a final stable release. TurboGears 2.0 final includes all kinds of goodies for those making web applications, from one of the most powerful and flexible Object Relational Mappers available in any language, to a powerful and flexible template system. But just as important as the quality of the parts, is the out-of-the-box integration to help get you started quickly:

We have quickstart template that helps get you going quickly with everything you need: from sample templates, to sample controllers and tests.
We have an extensible user/groups/permission system that you can easily configure into your app when quickstarting a project.
We have zero config needed support for development database backed by SQLite.
We have a working admin system for editing your database while your app is in development.
Our admin system is extensible and reusable as a component of your application.

There’s lots more. But equally important, we don’t think that out of the box defaults should be constraints on our users. So, a trivial configuration change lets you use DB2, or Oracle, or SQLServer, and everything we’ve wired up for you is easy enough to customize or replace. For example, we support configs for three major python template engines out of the box, and you can easily make your own render function to handle anything else you want.

One of the goals of TurboGears 2 is to use standard python components, that are valuable in all kinds of other contexts, so you are not tied into one monolythic system. Learning SQLAlchemy can help you write command line tools, GUI apps, web-services that don’t use a framework; Genshi is valuable when generating all kinds of xml data for interchange between systems; the beaker is a great caching system that’s valuable in all kind of web contexts, etc.

TurboGears 2 final is just now comming out, but it’s already in production use at places like ShootQ, RedHat? (for a large set of Fedora infrastructure projects) and many other places. And we’re already looking forward to a few more high profile TG2 deployments in the next few weeks.

Jorge Vargas: TurboGears and the Google Summer of Code 09

Mon, 25 May 2009 03:02:26 +0000

The GSOC started officially yesterday. For those of you that don’t know it it’s a program from google to get university student involved in open source, google it for more info.

TurboGears got accepted for 3 projects this year! You can read the proposals here.

The first one (in points) is by a second time student with us. His name is Sanjiv and he is from India, in his last GSOC he created a bunch of ToscaWidgets for a big set of libraries, and since he was so good at it he started to work on tgext.geo which is an extension to TurboGears to provide geospatial libraries support. We liked that so much that he now has the full summer to get that going, and I’m really looking forward for his work.
Our second project is going to be developed by Jesse Howarth and it is something I have been expecting from quite some time. And it’s actually two small projects and I love both of them. The first one (in order of appearance) is to finish up our last year GSOC project to provide a webUI for Builbot, for those that don’t know it buildbot is a “integration testing tool” it will checkout, build, run tests, run your project and yell at you if you break the tip, together with a webUI to manage it, that rocks! The second is a very interesting tool for building online documentation based on sphinx, there is little to know about how it’s going to work but it will be awesome if we could get something that will allow people to manage documentation just as if it where code with a web based review interface.
And the third it’s a problem I have been wanting to work on for quite some time. I won’t lie to you. I wrote the idea and Alexandru Marinescu is going to step up and code it, and in the process we are going to leave a ton of very useful tgext to be used by the community. The project is currently split into 3 areas,
1. User management, we are building tgext.registration, tgext.profile and maybe tgext.oath and tgext.openid, those last two should probably be repoze.who plugins instead. The idea is that tgext.registration will provide everything that is needed so no one will ever need to write a registration module again Profile on the other hand is a very nice tool to manage users it will provide a custom admin for ‘administrators’ to manage their users, as well as password reset forms and more
2. A complete web based frontend for mercurial, to view a repository. This is in spirit a rewrite of hgweb with the goals of making it more flexible and to provide some additional features
3. And this is where the cool stuff starts, a web based repository management tool for mercurial. This part of the project is a bit fusy but it it’s initial feature set is support for ‘create repo’, ‘fork’ and ‘pull request’, after that it’s open to adding as many features as people need.

I have great hopes for us students and I’m looking forward for their work. If you want to follow you should join us on the mailing list turbogears-gsoc@googlegroups.com and check out the site we have put up for this http://gsoc.turbogears.org/

Matthew Wilson: Need help with data files and setup.py

Wed, 22 Apr 2009 03:00:28 +0000

I’m working on a package that includes some files that are meant to be copied and edited by people using the package.

My project is named “pitz” and it is a bugtracker. Instead of using a config file to set the options for a project, I want to use python files.

When somebody installs pitz, I want to save some .py files somewhere so that when they run my pitz-setup script, I can go find those .py files and copy them into their working directory.

I have two questions:

Do I need to write my setup.py file to specify that the .py files in particular directory need to be treated like data, not code? For example, I don’t want the installer to hide those files inside an egg.
How can I find those .py files later and copy them?

Here’s my setup.py so far:


from setuptools import setup, find_packages
version = '0.1'
setup(name='pitz',
    version=version,
    description="Python to-do tracker inspired by ditz (ditz.rubyforge.org)",

    long_description="""
ditz (http://ditz.rubyforge.org) is the best distributed ticketing
system that I know of.  There's a few things I want to change, so I
started pitz.""",

    classifiers=[],
    keywords='ditz',
    author='Matt Wilson',
    author_email='matt@tplus1.com',
    url='http://tplus1.com',
    license='',
    packages=find_packages(exclude=['ez_setup', 'examples', 'tests']),

    include_package_data=True,
    package_dir={'pitz':'pitz'},

    data_files=[('share/pitz',
        [   
            'pitz/pitztypes/agilepitz.py.sample',
            'pitz/pitztypes/tracpitz.py.sample',
        ])],

    zip_safe=False,
    install_requires=[
        # 'PyYAML',
        # 'sphinx',
        # 'nose',
        # 'jinja2',
          # -*- Extra requirements: -*-
    ],

    # I know about the much fancier entry points, but I prefer this
    # solution.  Why does everything have to be zany?
    scripts = ['scripts/pitz-shell'],

    test_suite = 'nose.collector',
)

When I run python setup.py install, I do get those .sample files copied, but they get copied into a folder way inside of my pitz install:


$ cd ~/virtualenvs/scratch/lib/
$ find -type f -name '*.sample'
./python2.6/site-packages/pitz-0.1dev-py2.6.egg/share/pitz/tracpitz.py.sample
./python2.6/site-packages/pitz-0.1dev-py2.6.egg/share/pitz/agilepitz.py.sample

I don’t know how I can write a script to copy those tracpitz.py.sample files out. Maybe I can ask pitz what its version is, and then build a tring and use os.path.join, but that doesn’t look like any fun at all.

So, what should I do instead?

Compound Thinking: Working at SourceForge

Thu, 09 Apr 2009 14:20:53 +0000

I’ve been at SourceForge for a couple of months now, it’s been great, the work is surprisingly fun and rewarding. There’s a local office, and so I actually get to g and hang out with smart people whenever I want. I can still work from home, but having someplace to go in to has been a refreshing change.

I haven’t gotten to know many people outside the engineering team in Dexter, but they are great guys.

There’s lots of good stuff happening here, support for bazar, mercurial, git, trac, and other options on SourceForge itself, improved feeds, and other API’s for getting at SF data, etc. But I’m only peripherally aware of all that at the moment because I was hired to work on “totally new stuff” which is written in Python.

What I’m working on

Our first new project is a site called FossFor.Us, and it was the vision for this site, and the team that is working on this and other new stuff, that sold me on the coming to work for Sourceforge. It’s written in Django, and it’s been my first really large Django project, and while the experience has been pretty positive, there have been a number of things that have renewed my commitment to TurboGears development — but that’s a blog post for another day.

The backstory to the FossFor.Us site is that open source project hosting providers (Sourceforge and it’s recent competitors) have traditionally been pulled in two very different directions by two very different sets of users:

developers of open source software
and people who just want to.

And that tension has held us back in the past, we have to serve everybody with the same portal, and it ends up not serving either community as well as it should. But since developers are the most vocal users, it’s been the second class of user that’s been most neglected.

These people are just looking to get things done, and don’t care about the “project” part of open source software, they are, at least at first, only interested in the “product.” In many ways the Free and Open Source Software community has not served these people well.

Fossfor.us is in it’s first incarnation an attempt to create a window on the free software world, that’s just about finding and using software. But in a larger sense it’s an attempt to help us as a community to connect with potential users better.

I think connecting FOSS geeks and users is actually important

It’s important because people aren’t aware that there are free options, and are paying for software they can’t afford. There’s a prototypical user (based on a real person) that we talk about a lot, who’s a single mom, has an old laptop, and struggles week to week to pay her bills, but who bought Photoshop, because “that’s how you edit photos.” Her family could have used that money to more productive ends, but because she needed to edit photos, and didn’t know about the free alternatives all those opportunities are just lost.

Of course the same thing is true of small business owners, who could use free software to reduce their “overhead” costs, and actually spend money on creating things people love. Free software has the potential to lubricate the wheels of the economy, encourage entrepreneurial activity, and enrich people’s lives.

All of this is to say I think fossfor.us is a way to serve the world by making the product of all the open source developer’s labor more easily available and more accessible to real people. And when my mom actually used it to find some software a couple weeks ago, I knew we’d done something right.

Christopher Perkins: Pycon 2009 Recap

Sat, 04 Apr 2009 00:08:31 +0000

It felt like this year Pycon was executed to near perfection. Many struggles I had with last years Pycon were addressed both by the organizers, and some creative thinking.
In this post I will recap everything that happened from my perspective.

WSGI House

I gathered a few close friends from the TG team and a couple of wildcards for perspective to share a house for the continuum of the conference. Having a house gave us a place to go home to at night and meet with friends, often staying up late talking about issues surrounding our favorite software. Having a focused group I feel is important because you spend less time off on wild tangents. The first (and pretty much only) rule of the house was that you pay the same amount whether you stay one night or nine. At least one of our members was encouraged by this rule to stay for the sprints which he hadn’t done before. Success!

Tutorials

For me, tutorials got off to a shaky start, but we seemed to recover nicely. TurboGears has a lot of momentum right now, and it makes it hard to come up with a succinct tutorial when there is so much functionality to cover. I think we were able to recover and that our students managed to soak in enough knowledge from our proverbial fire hose to create some useful applications. I think we have a good start on a new book.

I was extremely impressed with the quality of students who were attending my ToscaWidgets tutorial. Every single student finished every example. I chose Pylons to give the tutorial, and although it is a little harder to integrate TW in the stream than does TurboGears2, it installed quickly and flawlessly. Overall, I think the tutorial was a success.

Talks

This year I did not focus on attending the talks, but instead chose wisely based on speaker and topic and allowed my feet to do the walking if the talk became uninteresting. I definitely missed some talks, but the AV team has done an incredible job putting the talks up on blip.tv so that I can review them later.

This year I did not miss Raymond Hettinger’s talk on AI in python and was enthralled by a speaker who could successfully put a page of code on the screen and keep my interest. I showed up to support Philip Jenvey in his talk on Pylons on Jython but was impressed by his ability to provide a succinct example on where Jython really shines. I am hoping that more people take a second look at this really well done presentation.

Now, I am a SQLAlchemy supporter through and through, but find the domain of database mapping an interesting echosystem. While the ORM panel was littered by advertising chatter from one of the panelists who did not even write an ORM, an obvious dis-inclusion was Robert Brewer who wrote Dejavu, a very nice way to map persistent resources of different types for use in an “objecty” way. Bob’s talk was especially interesting and makes me wonder if SQLAlchemy could leverage some of the work with AST that Bob beautifully displayed with some of the most amazing one-handed keyboarding I have ever seen.

Open Space

Well, I said I was going to give a talk at the Open Space, and ended up not doing so. Part of the problem was the utter lack of projectors in the OS rooms, and part of it was a reluctance to break up the collaborative/discussive vibe that was going on in these sessions. WSGIers hammered out a 2.0 spec, which involved a discussion I only monitored in passing. I was disappointed by the lack of people who showed up for the GSoC BOF, but I think the economy held back a lot of students from attending Pycon. It was also nice to allow my feet to walk around and see what was up in different projects. I met one guy who took REST way to far and got to express some of my dissatisfaction with one of the available tools. On a more positive note, the TG BOF was well-attended and it was nice to see so many users wondering what was up in TG land.

Sprints

This year I refused to let the noobs get me down and actually wrote some code. I am sorry if I did not act as a good host of the TG project, but we have some important milestones coming up and I just wanted to get work done on that. Sprinting remains a cornerstone of our development process and I will see if we can’t get our monthly
sprints happening again in 2009. I was however able to completely re-engineer our dispatch system, and while it is not currently 100% complete, it should be finished in a matter of days. RestController now supports variable arguments for get_one, delete, and put, as well as supporting lookup and default. Anyone can actually now create their own dispatch mechanism, since this functionality has been generalized. Simply subclass Dispatcher, override _dispatch() and go to town. I look forward
to seeing what kind of crazy code this brings to TG land. A lot of discussion has been had on how to make “plugins” or “extensions” for TG, and you can rest assured that we will have this functionality soon.

Thanks

Thanks to all of my house mates who put up with my “mothering”. Thanks to all of you who tolerated my “um”s at my talk on Sphinx, and especially to Georg Brandl who answered some questions. Thanks to the organizers, volunteers and staff that came together to create what has been my best Pycon to date.

Compound Thinking: Turbogears 2.0 RC1 Release

Sun, 29 Mar 2009 18:46:43 +0000

I missed the boat by a few days here, but I posted up the first Release Candidate for TG2 earlier this week before pycon. And given the interest in the sprints, it’s very likely that we’ll have a 2.0 final release avalable by the end of next week.

Jorge Vargas: TG2: after 3 and a half months and 6 beta releases

Mon, 02 Mar 2009 05:09:47 +0000

Last time I wrote about TG here I said I was committed to get TGb1 out of the door and that was December 15th, 2008, wow time runs by when you are having fun. Lets see, today is March 1 that’s 3.5 months exactly or 76 days according to python.
>>> datetime.date.today()-datetime.date(2008,12,15) datetime.timedelta(76)

Since then we have had TG2b1, TG2b2, TG2b3, TG2b4, TG2b5 and TG2b6 which averages us to one release every 12 days! that’s a lot of work!, to be fair it’s more like 15 days each as b4 was short lived due to …. that’s in the pass who cares

TG2b1 had 47 tickets It was basically a cleanup release and we applied a ton of patches from trac, the most notable thing is the upgrades for repoze.what, with Gustavo making a clear goal to get this package in shape for real world usage.

TG2b2 had 12 tickets Was mainly Marks work, He fixed our WebError to look nice and fixed the error controller (after I failed at understanding it) then he fixed tg.url to work with the newest pylons.url (pylons 0.9.7b4 I think) which was 15% faster (or some nice number) due to some optimizations from benbangert.

Tg2b3 had 16 tickets and was probably the most important of the betas, even though it had such a small ticket count, as it gave us a ton of features mostly from the hand of Christopher Perkins,

He gave us the very cool content type request dispatch, which despite the name it’s something not that scary and very powerful. You simply add .json to a url that’s behind a controller with @expose(’json’) and it will give you ‘application/json’, this works with minetypes so it will allow you to build any type of file on the fly. Or simply serve different requests from the same controller, for example .xml, .json and.html could serve as legacy API, ajax and browser, all from the same controller!

The other very nice feature by Chris is the RestController (RC) and it’s extension CrudRestController (CRC) (in the package tgext.crud) which is the base for tgext.admin and Catwalk2. Wow that’s a lot of names. I saved this from IRC which I think explains it nicely.

1 percious_: Sprox = SA + TW
2 percious_: RC = Restful Dispatch
3 jon1012: hey guys
4 percious_: CRC = Forms/Fillers/Tables + RC
5 percious_: Admin = CRC + config + index page
6 elpargo: hi jon1012
7 percious_: Catwalk = Admin + TG-quickstart-specific Config
8 percious_: hi jon1012
9 elpargo: awesome I’ll turn that into a doc soon TM
10 percious_: catwalk is a dumb name we should probably eliminate…
11 pjenvey_: holy acronyms, batman
12 percious_: sounds awesome
13 percious_: AC = Admin Controller
14 percious_: CRC = CrudRestController
15 percious_: RC = RestController
16 percious_: SA = SqlAlchemy
17 percious_: TW = ToscaWidgets

still too cryptic? ok, here are some words. You start with the very basic SQLAlchemy definitions and ToscaWidgets then you have sprox which links one to the other with some very nice basic Widgets, then you take a rest controller and make available over the web as a resource, then you take a group of resources and transform it into a CRUD with CrudRestController, then you take a group of CRCs and transform them into an AdminController, or you could just skip all that and let Catwalk2 give you a very nice set of defaults. From all this I find CRC to be the best for most applications it’s a very nice sweet spot between fully automatic and fully by hand. And I strongly think this is the best tool TG2 has to offer for database driven applications (sorry Rum I know you are very good but I just can’t customize you)

The other important change in TGb3 was the new flash based on a cookie, instead of the session and this was done by Alberto Valverde (don’t be fool by his blog, he is very active coding). This is a rather interesting feature mainly because it will allow you to cache your application. And in the best of WSGI spirit this was released as a separate package.

The last important change here was a small project started by Alberto and your trully, it’s a base class inside TurboGears named WSGIAppController and a package named tgext.wsgiapps (yet unreleased, although it’s hg is public). The idea behind WSGIAppController is to prepare TG to serve a downstream application. The purpose of wsgiapps is to make controllers for the most popular apps out there (we have Trac, Mercurial, Zine (no link as a vote of protest as it has no setup.py file), and TileCache) and fix all the small little details you will encounter. This is all heavily under-documented but I’ll get around to it ones I have a proper alpha for codemill (but codemill is another post).

The next feature (I believe this was here correct me if I’m wrong). jon1012 and florent (hi french people) sat down and gave the TG quickstart a totally fresh look and not conformed with that they made a very nice tutorial for new people. The new CSS is very nice and clean and will serve as a nice base for any project.

TG2b4 was another bugfix release and it had a very small ticket count (just 2) this was because most of the work happen outside of TG, and it was a bugfix release, with so many feature in b3 we had to find some bugs didn’t we?

TG2b5 was another biggiee (21 tickets) and it saw many more enhancements to the auth & auth stack, with the last set of features from Gustavo to make this super configurable, we also had the __before__ added to TG which is a very nice feature from people comming from pylons and the error controller now has a nice styling (again thanks Mark!)

And last but not least TG2b6 which was a little overdue (mainly because mramm has a life and couldn’t cut us a release). This was another rather big release as everyone was rushing to get their fixes in before feature freeze and made a total ticket count of 27 I’m a little bias here so I’ll say what I did first. We now have a new (and awesome) default model unit test class which was originally given to me (for private code) by percious and I finally manage to sneak into TG2, it makes testing your model as simple as creating an instance of each class with such a nice api that it doesn’t even needs docs (yes I know I’ll write them), on top of that I commited a cleanup of the dependencies which makes the installer a lot smaller and made TG2 pip compatible, you know because everyone loves pip. On the auth front Gustavo’s work was so good that people from pylons wanted to use his repoze.what package so he split TG’s functionality into repoze.what-pylons and also enhanced the whole thing to look a little more like TG1 identity which allows a far better use with less surprises.

Also here (I believe) one of our ‘08 google summer of code students (now active contributor) all the way from India Mr. Sanjiv Singh (ok that was too much of an intro) , updated the tgext.geo package to be a flag in quickstart. So now you can enjoy all his geo goodness with 5 keystrokes (–geo)

So there you have it that’s the super small summary of what We (TurboGears people) have been up to in the last… wow I still can’t believe it’s been 3.5 months.

What’s next? of course TG2.1! We are already building up on tickets for that. But before we get our hands on the coding we need to get the rc and final out and write some documentation and some more documentation yes that is confusing we are moving from a “component documentation” to a “type documentation” but most tickets are still in the wrong slot. Anyway it’s all work to be done and needs reviews as well as comments from newbies and pros alike. TG2 is here and ready to rock now lets test it in the wild so we can write docs, posts and patches, to get a TG2.0!

Oh and see you at pycon!

Compound Thinking: TurboGears 2 Beta 6 released

Fri, 27 Feb 2009 02:07:03 +0000

TurboGears 2.0 beta 6 comes with lots of fixes, and a total feature freeze in preparation for the first Release Candidate. From here on out to 2.0, we’ll only be doing bugfixes and improving our Docs. Speaking of Docs, Beta 6 comes on the heals of this last weekend’s doc sprint, so it already comes with new and improved documentation.

One thing which you should be aware of in Beta 6 is that after you install tg.devtools and quickstart a new application you have to do python setup.py develop on your quickstarted app because some of the dependencies for the app are not installed until that time. This allows you to remove optional dependencies from your application if you’re not using them.

Jorge Vargas has completely rewritten the Install doc. Anita has done great work on the Wiki 20 doc, which is providing an ever improving introduction to TG2.

Christoph Zwerschke has update the ToscaWidgets forms doc, Chris Perkins has documented the new RestController stuff which simplifies common Create, Read, Update, and Delete operations. And I created an overview of the TurboGears 2 stack. There are also new docs on the alternative template engines.

And of course there were also many, many smaller updates spread across the TG2 docs.

We are nowhere near where we want to be with the TG2 docs, but they are shaping up quite nicely. And we’re getting very close to the final release.

It’s been a great ride, and I’m already looking forward to some of the great things in store for 2.1.

Compound Thinking: TG2 doc sprint this weekend

Fri, 20 Feb 2009 03:03:30 +0000

We’ve declared a final feature-freeze on TurboGears in the runnup to the 2.0 release.

But that doesn’t mean that there’s not still lots of work to do. We’re working on a updates to the TG website, radically improved documentation, and better marketing materials. So, we’re having another TG2 sprint this weekend, but this sprint will be focused on infrastructure documentation, and will be the most newbie friendly sprint ever since one of the primary tasks is to make sure that the docs make good sense to new users.

Documentation is hard because the people who write it generally are experts, and have forgotten what’s complicated/confusing to new users. So we really, really need fresh eyes and fresh perspectives.

And one of the most useful things you can do is read and comment on our docs:

http://www.turbogears.org/2.0/docs/index.html

If you have a few hours, please join us in #turbogears on IRC on freenode (see this tutorial if you’re new to IRC) and sign up for the sprint here:

Compound Thinking: NGINX+SSI+TG2/Pylons

Thu, 12 Feb 2009 01:41:07 +0000

Interesting stuff here. This came up again at my new job, and I remembered that I’d forgotten to blog about it when I first saw this blog-post.

NGINX is an amazingly fast lightweight web server, and Server Side Includes (SSI) seem to be making a comeback as “composed” pages become more popular and traffic levels go up.

This is interesting because it lets NGINX provide super-fast and easy to use access to memcachd cached pages, and with SSI you can restrict the dynamic portions that have to be handled by python to the smallest possible portion of the page. If the pattern fits your application really well, you can see very significant increases in the number of requests-per-second you can achieve this way.

I’ve been fooling around with SSI+nginx too for another project, but and have been meaning to blog about it but haven’t had time to get anything ready to publish. I do intend to write a detailed tutorial on this setup someday, but for know this is a great if super-high velocity introduction to the how of SSI+Memcached+Pylons. And of course you can use TG2 with this in exactly the same way.

http://www.reshetseret.com/app/blog/?p=3

Compound Thinking: TurboGears 2 tutorials at PyCon

Wed, 11 Feb 2009 01:40:36 +0000

This year at PyCon we’ll be using turbogears 2.0 final in the beginning and intermediate TG tutorials, and there’s lots of great stuff to cover:

* A new “admin” system
* New REST support in controllers
* Improved automatic form generation via sprox
* Built in user/groups/permissions system

This year’s tutorial will be much more interactive than last, and will be focused on TurboGears more narrowly than last year when we did TG2+Pylons. We’ll be covering lots of stuff that can be used in Pylons, but since TG2 configures all this for us, we’ll be doing it the TG2 way.

In addition to the beginning and intermediate TG tutorials, there will also be a full day of SQLAlchemy tutorials, and a ToscaWidgets tutorial.

If you really want to maximize your learning experience, I’d recommend this list of tutorials:

If you haven’t used SQLAlchemy before and want more introduction than the TG tutorials provide, you might want to switch out the ToscaWidgets tutorial for the Introduction to SQLAlchemy.

And if you really want to maximize your TG experience, there will be a couple of TG experts that will work with you on some open source TurboGears applications at the sprints after the conference. I think working with the core TG team on real application code is a great way to round out the Tutorials and maximizing your learning.

This is an incredibly great two day learning experience at an incredible price. I’m really excited about how much PyCon in general and the tutorials specifically have to offer TurboGears users this year. I think the two full days TG related tutorial, opens up the possibility go go into much greater depth, and when coupled with the sprints, provides a great way to learn more about turbogears.

Compound Thinking: Database Migrations

Mon, 09 Feb 2009 02:53:51 +0000

SQLAlchemy Migrate 0.5 was released last week, and it got me thinking about how important it is.

Evolutionary Database design and data schema refactoring critical to most projects these days, and pretty much everywhere I’ve worked over the past 10 years, we’ve handled database upgrades via a set of sql upgrade and downgrade scripts, a db version number in the db, and a upgrader/downgrader python thingamabober.

A few years ago I did a couple projects in Rails, and Rails Migrations made this easier, by providing a standard API. There were limitations, and worries about lost data, and sometimes it wasn’t clear what SQL would be generated — but it was good enough, and worked well enough, that everybody used it.

But what I don’t like about ActiveRecord, and other ORMs like it is that every object-property change ultimately requires an underlying database change. On large projects I really want the flexiblity of a data-mapper to insulate some of my OO code and OO design decisions from the need for database level changes. I think this is incredibly important on very complex projects where the “data model” needs to change quite frequently, but the underlying database schema needs to change less frequently. SQLAlchemy on the other hand provides a clear, simple, elegant solution to that problem by decoupling the Objects from the Relations by providing an explicit Mapper.

But even with the flexibility of a Data Mapper based ORM, you still need to update your database schema sometimes. And that’s where SQLAlchemy-Migrate comes in. I think it or something like it is an indispensable tool in the SA user’s toolkit.

TG2 ships with both SQLALchemy and SQLALchemy-Migrate. And provides a little bit of help to make migration development even easier.

There’s more we can do to integrate Migrations into TG2, mainly by documenting it better, and setting it up a bit more in quickstart. I know there’s a trade-off there too, because for a lot of smaller applications migrations are too much overhead at the beginning, you can more easily wipe and recreate your database as you make changes, and we should support that way of working — so nothing we do should make that harder.

With that said, I’ll know we have arrived at the right place when we’ve made evolutionary database development or database schema refactoring easy enough that it feels natural and easy, and not doing it feels uncomfortable.

Christopher Perkins: Pycon 2009

Wed, 04 Feb 2009 23:33:42 +0000

So, Pycon registration has been up for a few days, I will be speaking both on and off-podium (read: open space) and providing assistance to and presenting tutorials. Here is a run down of what I am planning in case you wanted a little bit more in-depth information.

Tutorials:

Turbogears2 Beginner and Intermediate:

I will be assisting Mark Ramm by giving individuals help installing and using the new TurboGears2 framework. Mark is an experienced tutorial presenter, an expert in the technology, and in general a fun character to spend a few hours with. When you leave his tutorials you should expect to have a working version of TG2 on your machine, along with an understanding of Model, View, and Controller paradigms. Middleware, Forms, and REST will also be covered. One note, if you are getting started with TG2, it’s best to have it installed and running if you plan to attend only the Intermediate Section. We will not be going over installation in the second-half.

Toscawidgets: Test Driven Modular Ajax:

I am presenting this tutorial which will describe how to use the valuable Toscawidgets package to create web content. If you are currently use WSGI technology, and are interested in creating reusable, modular web content, this is a perfect way to get started. I will show you how to configure TW middleware to work with pylons (which is applicable to other frameworks like repoze.bfg, paste, or even plone/Grok). I will then describe how you might use this middleware to generate web forms. The last few hours of class will be devoted to using the JavaScript utilities of TW to create an Ajaxified website, and test it using YUITest.

The Big F’ing Tutorial: Development Using the repoze.bfg Web Framework

I will assist/present with Chris McDonough about this up-and-coming framework who’s goals are to utilize bits of the zope 3 framework, wsgi, and new technologies to make a lighting-fast web server. Those of you who are familiar with Zope technologies may be interested to find how nicely some of the familiar bits of zope are integrated with wsgi with repoze.bfg.

Presentations:

Using Sphinx and Doctests to provide Robust Documentation

This is a 1/2 hour slot which describes how you can integrate tested documentation with your source code… with sanity! I go over a quick install of Sphinx, and use some screencasts to demenstrate how to add, run, and display doctests using it.

Open Space:Agile Development with SQLAlchemy and Python Testing Tools

I really enjoy giving this talk, and even though it was not accepted as a formal talk, I will find a venue by way of Open Space to express my knowledge of Testing, SA, and Nose. I have given this talk a few times now, and it’s fairly polished. My presentation, while on some dry topics, won’t put you to sleep. Carefully prepared screencasts and photograph-punctuated slides makes the 45 minutes breeze by. Questioneers/Hecklers welcome!

Sprint Topics

I want to spend some time with the Dispatch of TG2, and probably push Sprox further a bit. If you are just starting with TG, please feel free to contribute. Sprinting is a great way to learn a lot from the experts in the domain. We usually do a meet-greet-install the night before the sprints. Oh, and I’ve been known to provide refreshments to all of our sprinting hordes (read: FREE BEER).

So, I hope to see all of you there! If you see me in the hall, feel free to introduce yourself and tell me what you are using Python for!

Compound Thinking: WSGI and cooperation

Wed, 04 Feb 2009 04:49:47 +0000

Wow, lots of interesting discussion based on my last post about a Django developer discovering how rich the WSGI middleware ecosystem has become. And I actually think that the discussion there is very enlightening.

But I think those who criticize WSGI often miss the real point. I think WSGI is interesting not because of how amazing it’s API is, but because it’s provided a clear way for the many python web development communities to work together, and to swap and trade technologies.

Many things that used to be part of the Zope application framework, have been turned into WSGI middleware by the repoze folks (repoze.profile, repoze.tm, repoze.retry, etc) while things that used to be TurboGears framework specific have moved out to become libraries with middleware components (ToscaWidgets, repoze.what, etc).

Pylons was built from the ground up using WSGI Components, and has contributed a bunch of great components.

And then there’s Ian Bicking who has contributed a huge array of tools for working with WSGI stacks paste contains all kinds of stuff like the paste cascade app, paste script, and most importantly webob. WebOb is very lightweight, just an OO proxy over the environ, and a few helper methods, but it makes writing WSGI middleware and WSGI applications much easier.

The point of all this is that the wider python web world is seeing a huge increase in cross-polination of ideas, of working together on projects, and sharing code across major frameworks from TurboGears to Zope, to Pylons. And at the core of all this is the WSGI specification. WSGI may not be pefect, but it’s still great, because it provides a defined way for web framework developers to work together. And, that coupled with the fact that we’re all pretty friendly people, and we all like each other and want to work together on things, has created a renasance of sorts in python web tools world.

It’s now easy enough to pull tools together and make your own webframework that I know a couple people folks who’ve created 2-3 specialized frameworks for various projects in the last year. This is great because they can take only what they need, and also because they have contributed back inovative bits and pieces, and have also helped improve some of existing pieces at the same time.

Sure, it’s a little bit messy and chaotic when compared to more centrally controlled framework development, but it’s also a whole heck of a lot more interesting, flexible, and fun. And there’s a whole lot of interesting new ideas getting tried out.

TG2’s goal is to provide an opinionated, coherent, facade across this complexity. We want to be coherent, without being monolythic, to be full-stack, while allowing the user control over the stack.

Django developers sometimes criticize the wsgi world for not providing a complete and integrated toolset. And that’s exactly the problem that TG2 is determined to solve. If we’re doing our jobs right TG2 will provide what we think are best of breed components, put together in a coherent, and easy-to-use way. We want users to be able to depend on the stack and to be able to build reusable applications on top of it. And that requires being opinionated. But we also want to make it possible for users with different needs to make different choices when and where the need arises.

One TG2 developer said it this way “We want TG2 to have strong opinions, loosely held,” and I think that’s about right. We want to provide a clear path that works and works well, with a solid foundation, but we also want to have the humility to know that the opinions we bake into our framework are not always right for every web-app. And we want to trust our development community to know when to stay one the easy path, and when to try out new ways of doing things, or use WSGI components that work better for their apps.

Jonathan LaCour: Buy My Expertise and Save Lives!

Thu, 29 Jan 2009 16:00:17 +0000

I’m proud to report that my employer, ShootQ is sponsoring the ThirstRelief Mentor Auction to benefit ThirstRelief International. Bidding starts tonight, Thursday, January 29 at 10 p.m. EST. If you have any interest in photography, I’d encourage you to participate in the auction, as there are some great opportunities available to interact with some of the best in the world.

One of the auctions available to bid on is a 90-minute mentor session with the ShootQ development team, including yours truly. While we may not be as glitzy as some of the other folks up for auction, I’d like to think that we could still fetch a nice sum. So join in the cause, and start bidding tonight!

Its a joy to be a part of a business that puts such an emphasis on making a difference in the world, and I am excited to be able to chip in a bit myself.

Christopher Perkins: Coding Binge

Tue, 27 Jan 2009 23:20:40 +0000

I haven’t written to the blog in a while. Quite frankly, I’ve been busy. In the last 30 days, I have released 3 software new packages, updated 1, deprecated 1, participated in a sprint that lasted a virtual 2 weeks, closed countless tickets, and pushed forward TG2 functionality.

TG2b4 was released last Saturday. This was mostly a bug-fix release, but b3 is where the new functionality really came into the scene. TG2b3 is the first build to include Sprox, a new library for schema-generated widget generation. Sprox is the offspring of DBSprockets. I decided I liked the declarative part of DBSprockets so much I wanted to spin it off as it’s own entity. Sprox looses DBSprocket’s table-based dependency, utilizing the mapping provided by SQLAlchemy. I realized that much of DBSprocket’s code was doing precisely what SQLSoup was designed to do, and decided to focus my efforts on making and extremely configurable widget base. The result was a considerable removal of the cruft that was associated with DBSprockets. Sprox releases with an excellent documentation base provided by Sphinx.

There has been a bit of resistance to Sprox, people were/are confused/upset about my providing yet more options for schema based widget generation. The fact is I have yet to find anything that performs as well as Sprox from a developer/speed standpoint, and I needed to provide our TurboGears user base with a better way to administrate their site, and also allow them to use that tool component-wise in their system. I think this method for developing widgets is well done in other frameworks, and we need a solid answer to this problem. Sprox is just that.

The next step was to re-work Catwalk to use Sprox. This took a little effort, and I put in RESTful URLs while I was at it, but struggled with making the URLs work within TG2’s dispatch system. The result was as close to REST as you can get without conforming to a set standard.

The result of hacking REST into Catwalk got me thinking, and I decided to implement content-type dispatch as well as RESTful dipatch in TG2. I went back for another round on Catwalk, and converted it over to the standard.

I’ve also been toying around with Dojo at NREL. I’m pretty much done with ExtJS due to licensing issues, a not-so-hot codebase, and weak support from IRC. It’s bad when you go on to ask a question on the channel as a 6 month-user of a software project and end up spending all your time answering everyone else’s questions (as the most experienced person in the room). Something must be said for an organization that does not push paid consulting as a primary focus on their website… #dojo has been an exceptional resource for getting my work done. Those guys know their software, and lend a great hand to help you with it.

Back to the topic at hand… I was able to shoe-horn Dojo into Sprox with little effort, and implemented DojoCatwalk, which worked, but was ultimately not what I wanted. What I really wanted was configurability. I started work on tgext.admin, which was supposed to provide enough functionality to replace tgcrud, a command to auto-create crud in your own TG application. To support tgext.admin, I created a new package called tgext.crud, which provided a CrudRestController, which is a simple way of providing crud for any object in your model. AdminController combines this functionality with that of Mark’s lookup code to provide a fast, configurable set of tables/forms/etc for all objects in your model. AdminController takes a declarative AdminConfig as input which provides a consistent way to create your administrative toolset. Did I mention it does Dojo tables with ajax loading? Yeah.

I’m not done with this binge yet. Catwalk is going to mutate one more time before I’m through with it. It is going to become a default-configured AdminController specifically designed to work within the context of a quickstarted TG2 application. I had one blocker ticket which was solved last weekend, so it’s time to get Catwalk good and finished.

Jonathan LaCour: Metaclasses Demystified [5]

Sat, 10 Jan 2009 19:00:12 +0000

This article originally appeared in the July 2008 issue of Python Magazine and was adapted from Jonathan LaCour’s CLEVERtrain professional Python training services.

The word “metaclass” often strikes fear into the hearts of Python programmers. This article explores metaclasses in an approachable way, in order to take the mystery out of metaclasses, and enable you to leverage metaprogramming techniques to solve real problems.

There are few things in the Python world that are so misunderstood and feared as metaclasses. In fact, when I recently was training a group of seasoned programmers about metaclasses, I asked the room what sprung to mind when they heard the word “metaclass.” The students shouted out their feelings in chorus – “confusing,” “magical,” and worst of all, “difficult.”

Well, as I told my students that day, when it comes to metaclasses, I’d recommend that we called upon the wise words of Douglas Adams in his famed book “The Hitchhiker’s Guide to the Galaxy.”

“Don’t panic!”

Once you take a closer look at how object-oriented programming works in Python, you’ll quickly discover that not only are metaclasses easy, they’re also extremely useful. Here, we’ll do our best to take all of the mystery out of metaclasses, and show you that there’s nothing to be afraid of!

Revisiting Classes

Most Python programmers are familiar with creating classes, and might even create or modify their own classes several times a week in the course of their work. Many of us were even taught about classes and object-oriented programming in the course of our education. But, when was the last time any of us really thought about classes in depth since we were first introduced to the concept? I’d wager that most of us take classes for granted, and don’t really think about what they are providing for us, and more importantly, how they provide it to us.

Understanding metaclasses can be greatly simplified by taking another look at Python classes, from a slightly different perspective. In order to do this, lets pose a simple question. What exactly does a class do?

My first instinct when attempting to answer this question is to rely on my education, which unearths big, fancy, computer-sciency words like “encapsulation” and “abstraction.” Its tempting to allow this to cloud our thinking, so lets do our best to think at a much simpler level when considering this question. So, I ask again – what does a class do?

Fundamentally, a class is used to construct objects called instances. This is essentially all a class does – creates instances. The process of creating instances using a class is called instantiation. For example, given a class Person, we can instantiate it to create an “instance” of that Person class.

    class Person(object):
        pass

    jonathan = Person()

This instance is related back to its class in that the class is what constructs the instance. What do we mean by “construct”? Well, when a class is instantiated, it constructs an instance by providing it with its “namespace.” We all know from Tim Peters’ “The Zen of Python” that namespaces are a “honking great idea,” and classes are a great example of why! The attributes in the namespace of a class are used to define the namespace of its instances, thus providing those instances with behavior and state. Lets enhance our Person class, to see how this works.

    >>> class Person(object):
    ...    greeting = 'Hello'
    ...    
    ...    def greet(self, who):
    ...        print self.greeting, who

    >>> jonathan = Person()
    >>> jonathan.greet('Readers')
    'Hello Readers'

The Person class now has two attributes in its namespace – greeting and greet. When we instantiate our Person class, our instance is given both state, in the form of the greeting attribute, and behavior, in the form of the greet method. In this way, a class is of critical and primary importance in defining how its instances will behave.

So, what does a class do? Lets summarize. In a nutshell, a class is an object with constructs instances through a process called instantiation, therefore defining the namespace, behavior, and state of that instance.

Defining Metaclasses

Now that we’ve established clearly what a class is, we can confidently pose the question – what exactly is a metaclass? When sifting through documentation, the most common answer you’ll find to that question is short, but often difficult to unpack:

A metaclass is the class of a class.

While this answer is certainly correct and concise, and tells us what a metaclass is, it still doesn’t tell us what a metaclass does. Lets look at an alternate answer to the question:

A metaclass is a class whose instances are classes.

I will concede that this answer is just as much of a mouthful as its counterpart, but it has the benefit of giving us a clue as to what a metaclass does! In short, a metaclass constructs classes. We also know that a metaclass is just another class, and we’ve just spent some time outlining what a class does. As a result, we should be able to build upon what we’ve learned about classes earlier to determine a bit more about metaclasses.

Recall that a class helps define the behavior and state of its instances. Metaclasses provide the same basic capability to classes, giving you the ability to change the way a class will behave at runtime. This technique is commonly referred to as “metaprogramming.”

Having a definition of metaclasses and metaprogramming is useful, but lets take a deeper look by investigating an existing metaclass.

The `type` Metaclass

In Python, classes which inherit from object are called “new-style classes.” All such new-style classes have a default metaclass called type. We can prove that type is the metaclass of object through simple introspection in the Python interpreter. Recall, a metaclass is the class of a class, so what is the class of object:

    >>> print object.__class__
    <type 'type'>

Just as we expected! The metaclass of object is type, and thus all classes which inherit from object will be provided with this metaclass by default. Classes that do not inherit from object are called “old-style classes” and will disappear in Python 3.0. While old-style classes also support metaclasses and metaprogramming, we’ll focus on new-style classes in this article for the sake of simplicity.

Typically, classes are defined using the class statement in Python, as we saw in our earlier Person example. However, we have just learned that metaclasses create classes when they are instantiated. This means that we should be able to define a class by instantiating the type metaclass manually.

Lets define our original Person class again, but this time, lets do it without using the class statement by instantiating the type metaclass:

    >>> def greet(self, who):
    ...    print self.greeting, who

    >>> Person = type(
    ...    'Person', 
    ...    (object,),
    ...    {'greet': greet, 'greeting': 'Hello'}
    ... )
    >>>
    >>> jonathan = Person()
    >>> jonathan.greet('Readers')
    'Hello Readers'

This method of creating classes is equivalent to using the class statement, and reveals quite a bit about how metaclasses work. The constructor for a metaclass expects very specific arguments:

The first argument is the name of the class.
The second argument is a tuple of the base classes for the class.
The last argument is a dictionary representing the namespace of the class. This dictionary contains all of the attributes that would typically appear within the body of a class statement.

Now, we’ve seen a metaclass in action, and we know how to instantiate them to create classes. Armed with this knowledge of the default type metaclass, we can now tackle the much more interesting problem of creating our own metaclasses.

Defining Metaclasses

Defining metaclasses in Python is as simple as creating a class that inherits from the built-in type metaclass. The constructor for our metaclass will take the same arguments as the constructor for the type metaclass:

The class name.
A tuple of the class bases.
A dictionary representing the namespace of the class.

Constructors typically perform some action on their instances, so lets make our constructor set a flag on the instance that we can inspect to make sure that our metaclass is being used.

    class MyFirstMeta(type):
        def __init__(cls, name, bases, ns):
            cls.uses_my_metaclass = True

One important thing to note here is that the first argument to the constructor of a class is typically called self, as it refers to the instance being constructed. It is conventional to name the first argument of a metaclass constructor cls as the metaclass instance is actually a class.

Now that we’ve defined our metaclass, we know that we can construct a new class called MyClass simply by instantiating the metaclass:

    >>> MyClass = MyFirstMeta('MyClass', (object,), {})
    >>> print MyClass.uses_my_metaclass
    True

This very simple metaclass has given us our first glimpse into the power of metaclasses. Within our metaclass constructor, we gave our class some state in the form of a boolean attribute uses_my_metaclass. Metaclasses have the power to add, remove, or modify any attribute of the class being constructed. Metaclasses will frequently add or replace methods on their instances, based upon the data in the namespace of the class. Many Python object-relational mappers use metaclasses to transform the attributes of a class into database table definitions, for example.

While you can certainly construct classes by manually instantiating custom metaclasses, it is much more convenient to use the class statment to create your classes. Python allows you to define the metaclass for a class by using the special __metaclass__ attribute in your class statement:

    class MyClass(object):
        __metaclass__ = MyFirstMeta

This is the preferred method of attaching metaclasses to your classes. An important thing to note about this syntax is that while you are not manually instantiating the metaclass when defining your classes this way, the Python interpeter will instantiate the metaclass. The metaclass instantiation will occur immediately after the class statement has been fully executed. As a result, bugs in metaclasses often are triggered during imports. In a way, the class statement is simply syntactic sugar for instantiating metaclasses!

Mystery Metaclass Methods

Before we move onto some practical examples of metaclasses, lets investigate metaclass definition a bit further. Our Person class defines a method called greet within its namespace. Instances of Person will thus have an “instance method” called greet. We know what happens when we define methods on a class, but what happens if we define a method on a metaclass?

Methods defined on a class become instance methods. Since instances of metaclasses are classes, methods defined on metaclasses become _class methods_. Lets take a look at this in practice:

    >>> class MysteryMeta(type):
    >>> ... def mystery_method(cls):
    >>> ...     return 'I am a class method!'
    >>> 
    >>> class Mystery(object):
            __metaclass__ = MysteryMeta
    >>> 
    >>> print Mystery.mystery_method()
    I am a class method!

This revelation often surprises people, but is a logical outcome of the fact that metaclasses are simply classes which produce classes. Many metaclasses utilize this capability of defining class methods, but it is often preferable to define such class methods on a base class, which is easier to document and understand.

We’ve now established what metaclasses are, how they work, and how we can define our own by inheriting from type. But, what about practical use cases for metaclasses? Lets take a look at several examples of how we might come to use metaprogramming in practice.

Example: The Enforcer

When I am teaching Python to programmers with strong backgrounds in “bondage and discipline” languages like Java, I frequently hear the same complaints about Python. One such complaint is that class definitions are “loose” and you cannot enforce the type of variables by declaring them up-front. In order to illustrate how to use metaclasses, lets define a little library that will address this particular complaint.

Our goal is to create a base class called Enforcer that will enforce the types of attributes on its subclasses. Lets say we wanted to enforce that the name attribte of our Person class was a string, and that the age attribute was an integer. Attempting to set attribtues with the wrong type should trigger a TypeError to be raised, just like the Java compiler would catch such errors.

    class Person(Enforcer):
        name = Field(str)
        age = Field(int)

The first thing we need to do is define the Field class, which we will use to hold onto the type of the variable we’re attempting to restrict. Lets also give it the ability to validate whether or not a value is of the right type for that particular attribute:

    class Field(object):
        def __init__(self, ftype):
            self.ftype = ftype

        def is_valid(self, value):
            return isinstance(value, self.ftype)

Now that we have our Field class, we need to create a metaclass that will look at the namespace of our class, searching for Field definitions, and then storing them in a dictionary for later use. Recall that the last argument to the constructor of a metaclass is a dictionary containing the namespace of the class. We can loop through this namespace to find Field definitions:

    class EnforcerMeta(type):
        def __init__(cls, name, bases, ns):
            # store the field definitions on the class as a dictionary 
            # mapping the field name to the Field instance.
            cls._fields = {}

            # loop through the namespace looking for Field instances
            for key, value in ns.items():
                if isinstance(value, Field):
                    cls._fields[key] = value

Our metaclass first attaches a _fields dictionary to the class itself. This data structure is where we will store Field definitions for later use. We then loop through the items in the namespace looking for Field instances, and finally we store them in our _fields dictionary.

Next up is the Enforcer base class itself. The Enforcer base class first needs to attach the EnforcerMeta metaclass we’ve just defined. This is a very common way for libraries to distribute their metaclasses, by defining a base class to inherit from, rather than requiring users to even know that a metaclass is being used, or how to attach the metaclass to their classes.

The second thing the Enforcer base class needs to do is to override the __setattr__ method. This is a special method on Python classes that allows you to override the default attribute setting behavior on your Python objects. The __setattr__ method takes in the name of the attribute being set, and the value being set.

    class Enforcer(object):
        # attach the metaclass
        __metaclass__ = EnforcerMeta

        def __setattr__(self, key, value):
            if key in self._fields:
                if not self._fields[key].is_valid(value):
                    raise TypeError('Invalid type for field!')
            super(Enforcer, self).__setattr__(key, value)

Our Enforcer class first attaches the metaclass. Then, it overrides the __setattr__ method so that it can watch for field assignments. First, we check to see if the attribute being set is one of our defined fields. Then, we ask the field definition if the value that is being passed is valid for the field definition. If it is not a valid type for the field, we raise a TypeError.

The last line of the Enforcer class is extremely important. This line is instructing the Python interpreter to call the __setattr__ implementation on the appropriate superclass definition, in this case object. Without this line, attribute setting would fail on all Enforcer subclasses.

Lets try out our new creation from the Python interactive prompt:

    >>> class Person(Enforcer):
    ...    name = Field(str)
    ...    age = Field(int)
    ... 
    >>> jonathan = Person()
    >>> jonathan.name = 3
    TypeError: Invalid type
    >>> jonathan.age = 'Old'
    TypeError: Invalid type
    >>> jonathan.name = 'Jonathan'
    >>> jonathan.age = 28

Our metaclass has completely changed the way that our class behaves, transforming it into a more rigid class definition that keeps our Java-loving friends happy. While such restrictive enforcement is atypical in Python, it certainly shows how powerful metaclasses can be.

This example also illustrates a common pattern for metaclasses, in which a class describes how it wants to behave in a “declarative” way without actually writing any code to implement that behavior. The metaclass subsequently takes that metadata, and uses it to “reprogram” the class. This is the essence of what metaprogramming brings to the table, and is the most popular use-case for metaclasses.

Example: Auto Decorator

Since being given a special syntax in Python 2.4, decorators have become a very commonly used feature. A decorator is essentially a wrapper around a function or method. In versions of Python prior to 2.4, decorators were applied by manually replacing the method definition with a “decorated” version of the method. Starting in Python 2.4, decorators could be applied using the now popular @decorator syntax.

Sometimes, I find myself in the situation where I need to decorate all of the methods of a class with the same decorator. Lets create a metaclass that will simplify this process by automatically applying a decorator that is declared in the namespace of the class:

    import inspect

    class AutoDecorateMeta(type):
        def __init__(cls, name, bases, ns):
            deco = ns.get('decorator', lambda f: f)
            for key, value in ns.items():
                # skip the decorator and constructor
                if key in ('decorator', '__init__'): continue

                # skip objects in the namespace that aren't methods
                if not inspect.isfunction(value): continue

                # apply the decorator
                setattr(cls, key, deco(value))

Our metaclass first looks up an attribute named decorator in the namespace of the class. This is the decorator that we will apply to every method of the class. Note that if such a decorator cannot be found in the namespace of the class, we define a fake decorator that will simply return the existing method definition.

Next, we loop through the namespace of the class looking for methods. If we encounter the decorator or the constructor, we skip them, as we don’t want to decorate our constructor or the decorator itself. We then make use of the inspect module to determine if the value is a method, and if it is, we replace the method definition with a decorated version of that method.

Lets put our metaclass to the test by creating a class that defines several properties. Typically, we’d have to decorate every method with the @property decorator, which could add a lot of verbosity to our class, depending on the number of methods we’d have to decorate.

    class Person(object):
        __metaclass__ = AutoDecorateMeta
        decorator = property

        def __init__(self, first, middle, last):
            self.first = first
            self.middle = middle
            self.last = last

        def name(self):
            return '%s %s' % (self.first, self.last)

        def full_name(self):
            return '%s %s %s' % (self.first, self.middle, self.last)

        def initials(self):
            return '%s%s%s' % (self.first[0], self.middle[0], self.last[0])

Now that we’ve created our class, we should be able to try it out from our Python interactive prompt to see if all of our methods were transformed into properties:

    
    >>> mlk = Person('Martin', 'Luther', 'King')
    >>> print mlk.name
    Martin King
    >>> print mlk.full_name
    Martin Luther King
    >>> print mlk.initials
    MLK

Not only is this example useful in practice, it also illustrates another use case for metaclasses and metaprogramming – automating repetitive tasks that occur in class definition. This metaclass could be used to wrap every method of a class in a transaction, to add debugging hooks, or even to synchronize all of the methods using a thread lock.

Cautionary Notes

Metaclasses are fantastically cool, and as we’ve seen, they can be pretty useful. The rise of object-relational mappers and web frameworks have put metaclasses into use by an increasing number of Python users, and has increased the visibility of metaclasses substantially. That being said, metaclasses are a feature of Python that must be used carefully. Because metaclasses do their work at class definition time, bugs in your metaclasses can result in errors that are triggered at import time. In addition, metaclasses are often hidden from the programmer behind a base class, which can cause confusion.

Applied judiciously, metaclasses can be a great tool for solving problems, and give Python programmers the ability to take advantage of metaprogramming techniques in their own code. I hope that this article has shed some light on what metaclasses are, and has taken some of the mystery out of metaclasses.

Andrew Grover: Moving to TG2, Part 1

Mon, 05 Jan 2009 14:12:39 +0000

This is the start of a series on the work we're doing upgrading BandRadar's components. BandRadar's development started in 2006, and used the default components of TurboGears at the time: SQLObject for ORM, kid for templating, tgwidgets for widgeting, mochikit for js library, and CherryPy as appserver.

Basically all of these components are now obsolete, or at least no longer recommended for new TG/TG2 projects. But they still work fine, so why change?

Good question. I'm hoping that re-examining the choices we made in implementing BandRadar version 1 will bear dividends in terms of simplifying the code or making it easier to add new functionality, but honestly it's not clear at this point whether time spent on this work will have more value than spending it on extending the current code. It also depends on how hard moving to the new components is. Thankfully, TurboGears will let us transition to new components in stages, instead of all at once.

The plan:

SQLObject to SQLAlchemy (Elixir)
kid to Genshi
tg.widgets to ToscaWidgets
CherryPy to Pylons/mod_wsgi

Currently working on #1. Converting classes derived from class SQLObject to class Entity and defining the relationships has been relatively painless. All our table names are not what Elixir is expecting, but it's easy to tell it what they should be. The first little bit of trouble is that SQLObject did not create db constraints for required fields, it keeps track of these itself and throws an error if a required field is omitted when creating an object. SA appears to rely on the database for this (which makes sense) but our existing db schema (originally created by SO) will need to be updated by hand for the new not-NULL constraints. We've also added a fair number of additional methods to the objects on model.py, and these will need to be reimplemented in SA-ese. Finally, our SO model includes a feature where all modifications to some tables are logged to a separate table (so we could recover from vandalism if necessary)... this will need to be reimplemented in SA, or we need to jump to versioned objects. I'd prefer the former, frankly, but I don't know if SA has a feature equivalent to sqlobject.events, or if I could just override __setattr__() in my model objects.

Jorge Vargas: TurboGears2 and ExtJS tutorial app

Wed, 17 Dec 2008 09:16:29 +0000

For some time someone has been asking me, actually saying near me “We should write a tutorial on how to use ExtJS with TurboGears”, this morning someone on IRC was having big troubles getting it to work and I figured out that it will take me less time to write it than to sit down and explain, I was wrong but I did it anyway.

The example is very simple. It takes one basic ExtJS tutorial which originally was a locally loaded grid and transforms it into a remote(”Ajaxy”) grid with the same data, so even though it looks the same this is loaded from the Database. I wrote it with TurboGears trunk mainly because that is what I had around but also I wanted to test the new quickstart template in it’s non-auth version, in theory it should run on TG1.9.7b2 but I haven’t tested it. You can get a copy from the temporal mercurial repositiory here, I say temporal becuase I’m going to finally enable hg.maetico.com soonTM.

Back to the tutorial, you can follow the changesets as they explain each step in detail, (together with mistakes!) the steps are simple

quickstart a TG2 project
Add extjs to the public/javascript folder
Create the model and copy the ‘myData” variable, notice how much JS looks like python, all I had to do was delete the var and the ;
Then we take advantage of one of the great features of turbogears, automagical json from the ORM. Don’t believe me? take a look at this changeset. 3 lines will take all objects from the dband output json. And the exact same code works for any SQLAlchemy object.
copy over the rest of the extjs sample to a JS file, and modify it to have a JsonStore (http+json reader) instead of the SimpleStore(local array reader)
Add all your JS to the template and your done! well except one detail.
EXTJS likes to send a _dc parameter I really don’t know why I believe it’s to prevent proxy’s cache or something, therefore we need to tell TG about it and then simple don’t do anything with it.

So what are you waiting for? go get the trunk we need lots of testing before our 2.0b1 release! since you already clicked on that link, why don’t you take a look at the remaining tickets, they are mostly documentation and tests, easy things

Peace out.

Jorge Vargas: TurboGears 2 and I

Mon, 15 Dec 2008 21:50:03 +0000

This is no news to the people close to the project but it will be for everyone else.

I have decided to exercise my committer rights, and actually give back to the project, in terms of code. I have been giving a lot back in terms of help, if you don’t believe me check this and this, on the “top posters part”. And yes I have had SVN checking privileges for a long time, several years actually, I just never made it public until now.

What I plan (actually been doing for at least a week) to do in the short term.

Get TG2 b1 out of the door, we did a lot of work this weekend and cut the pending tickets more than half, we started with around 40 and at this moment at 11, which by the way are almost all documentation tickets
Finish my TG1.1 to TG2 migration guide, the first draft is here
Work on my Widgets 3.0 idea, search the trunk list of it, I need a name by the way, so suggestions are welcome

In the mid term

Work on a TWsite code, to make it usable for www.turbogears.org
Work on finally getting TurboGears on mercurial
writing and rewriting several apps I have and launching them on top of TG2.

Anyway enough talk back to the tickets.

Long term

TurboGears 2.1

Planet Turbogears

Compound Thinking: TurboGears 2.0.1 now available on pypi

Compound Thinking: TurboGears 2.0 presentation video

Compound Thinking: TurboGears 1.x to Turbogears 2.0

Christopher Perkins: Sprox 0.6.1 released

Jorge Vargas: “Everything” and Wave

Compound Thinking

Jorge Vargas: TurboGears and the Google Summer of Code 09

Matthew Wilson: Need help with data files and setup.py

Compound Thinking: Working at SourceForge

Christopher Perkins: Pycon 2009 Recap

Compound Thinking: Turbogears 2.0 RC1 Release

Jorge Vargas: TG2: after 3 and a half months and 6 beta releases

Compound Thinking: TurboGears 2 Beta 6 released

Compound Thinking: TG2 doc sprint this weekend

Compound Thinking: NGINX+SSI+TG2/Pylons

Compound Thinking: TurboGears 2 tutorials at PyCon

Compound Thinking: Database Migrations

Christopher Perkins: Pycon 2009

Compound Thinking: WSGI and cooperation

Jonathan LaCour: Buy My Expertise and Save Lives!

Christopher Perkins: Coding Binge

Jonathan LaCour: Metaclasses Demystified [5]

Revisiting Classes

Defining Metaclasses

The type Metaclass

Defining Metaclasses

Mystery Metaclass Methods

Example: The Enforcer

Example: Auto Decorator

Cautionary Notes

Andrew Grover: Moving to TG2, Part 1

Jorge Vargas: TurboGears2 and ExtJS tutorial app

Jorge Vargas: TurboGears 2 and I

The `type` Metaclass