Tim Bray is blogging about “inflection points” in the uptake of various technologies.

Python get’s a very positive review:

Today you’d be nuts not to look seriously at PHP, Python, and Ruby.

So, the rise of the so-called scripting languages is one of the inflection points, but it’s not the only one.

He singles out web-framework development as one place where there’s a lot of stuff happening, and a lot of new “rails-like” frameworks are cropping up all the time. TurboGears will live or die in the context of a much larger web-development revolution, and we need to be prepared to make our way forward in the midst of that.

What comes after rails will not be a rails clone. It will learn the right lessons from rails, avoid the pitfalls of rails, but it will also need to carve out something new and better than rails. For RDBMS users, I think the key difference between TG and Rails is the power and flexibility of SQLAlchemy. We need to “sell” this better.

There are a lot of other revolutions coming according to Tim. And I do think we’re looking at big changes in terms of everything from programming language choice, to web-development tools, to end-user desktops, and data persistence mechanisms. We’re also just beginning to see what the world of high-end javascript and other “rich” internet applications is going to do to our view of end-user software.

He doesn’t even mention the rise of EC2 and the Google App Engine as sea-changes in the way we buy computational resources, and I think that’s going to have a huge impact.

In the end my prediction is that the way we develop applications will change more in the next 5 years than it did in the last 5, and it’s time to start getting our heads wrapped around these issues, or we’ll be left behind.

April 30, 2008 04:28 PM

Chris McDonough just posted a pretty extensive writeup of something he and Florent hacked up recently. Basically he helped us to put together some helpers that turn the repoze.who project into very similar authentication/authorization system to what we had in Turbogears 1.0.

The cool thing about this is that we get to share the stuff that makes sense to share, and yet maintain a backwards compatable API. We started a project (authority) which did the same thing, all on our own, but it’s much better to be able to share. ;) The Authority project turned out to provide a lot of useful bits and pieces which have since found homes, and that process will likely continue.

I’ve said before that I’m very impressed with the work that the Repoze folks have been doing, but I think this is a particularly good example of how it’s possible to collaborate in interesting new ways on top of a shared set of interfaces.

There’s a lot more to be done to make python web development work better, and I’m excited because we continue to move towards a world where innovating in one area doesn’t require building a whole new stack.

I think there are big benefits for users of having a healthy and diverse ecosystem.

April 30, 2008 01:28 AM

I don’t think it’s possible to over emphasize the importance of developing teams in software companies. Software production is a group activity, and Brooks, Lister, Demarco and Weinberg all announced this same thing in various ways. And they have been saying it for a long time. The Mythical Man Month, The Peopleware Papers, and The Psychology of Computer were all written over a quarter century ago.

But, in spite of 30 years, and thousands of pages written, people still don’t get it.

Up until Monday I was working on a suite of fantastic applications for patient data and hospital management. I was part of a small, but very talented team of developers doing amazing things. Sure we had great tools like Python, Ext.js, TurboGears, etc. But the fundamental reason we were so successful is that we had a great team that worked together really well.

How do you know you’ve got a great team?

Chad Fowler’s very good (but very poorly titled) book My Job Went to India: and all I got was this book has a chapter explaining how before he became a programmer he was a musician, and someone gave him this powerful advice:

“Always try to be the worst player in the band.”

If you seek out people who are better than you, people who will push you — people who will make you grow just to keep up — you won’t have the option of stagnating. You either get better, or you get out. That’s how I felt on this team, and I know it’s how we all felt.

We knew each-other, knew our strengths and weaknesses, and we knew that together we could make things happen. We knew how to challenge one another, and if something went wrong, we would take action to fix it as a team.

Great teams are a huge asset

I can honestly say that it was the best team of people I’ve ever worked with, and we were producing products with real revenue attached — in other words, we were making money, and lots of it.

But as you can probably guess, after the acquisition came layoffs. And the team was chopped in half. Even though the team consisted of the best developers in the company, even though we were the most profitable part of the company, even though things were going amazingly well in our little corner of the world — we were still torn asunder by layoffs.

There may be some higher corporate logic behind this. And perhaps at some level it was the right thing to do, though I doubt it. I am convinced that Lister and Demarco are right is arguing (via peopleware) that a “jelled team” is an incredibly valuable resource for a company. They get stuff done, and get it done quickly, because they already know how to work together well. We cared about one another, cared about our work, and cared about our company, we were motivated, skilled, and we got things done. Loosing that is loosing a lot.

I got lucky

I was fortunate enough to be the newest member of the team, and was one of the folks that was let go. Why fortunate? Because I don’t have to sit around every day trying to get stuff done without the rest of the team, because I don’t have to be reminded every single day of what once was, but is no more.

And because I can take this opportunity to take a breath, look around and try to find a job which fits my long term goals. I don’t know exactly what I want to do yet, but I do know that I want to do something good for python, something good for the world, and something where I get to work with great people.

So if you’re interested, feel free to drop me an e-mail (mark.mchristensen@gmail.com).

Please hire these guys — they are amazing!

But more importantly there are some other great developers who are also looking for jobs. I’ve seen them perform miracles in C#, Java, Python, and any number of other languages. And I know them well enough to know that they would each be a huge asset to any company who’s looking to develop a great team.

One of the things you need the most in team development is someone who has a picture of how it could be better, of what’s possible, and the will to make it so. In addition to their technical skills, I know that’s something they all have. So, if you’re looking for one or more highly skilled software developers in the Atlanta area (or you’re willing to work with remote people) I can’t recommend them highly enough. Please send me e-mail if you’re interested and I’ll pass the info on to the right people.

April 24, 2008 03:35 AM

The TurboGears community is proud to welcome the participation of six Google Summer of Code participants. These students have proposed very interesting projects, done a lot of research, and promise to make TurboGears and related projects better.

The goal of the Google Summer of Code project is to get students more involved in the Open Source community, and I’m very excited about getting to know this group of students better, and seeing them show their stuff. And of course I’m looking forward to all the things they promise to do, from building automated build and test systems, to creating a super-easy CRUD interface builder, and improving the performance of Genshi templates.

We also had a large number of student proposals which weren’t accepted for the GSoC project, some of which were very, very good. The competition was very tough this year, and I am hopeful that some of those students will get involved in the TurboGears community anyway.

• DBSprockets User Interface by Alberto Valverde González
• Documentation Production System by Bruno José de Moreaes Melo
• Genshi optimization & XPath rewrite by Marcin Kurczych
• ToscaWidgets Library for OpenLayers and jQuery by Sanjiv Singh
• Improving TurboGears2 support on Jython by Ariane Paola Gomes
• TurboGears Quality Assurance Initiative by Steven Mohr

As you can see the majority of these projects are about improving stuff that’s used in TurboGears, but are generally useful tools on their own. We’re committed to principle of re-usable components, and the GSoC proposals we received have shown that there are a lot of other people commited to that same approach.

I’m particularly interested in Stephen Mohr’s project, because I think having a reliable build and test system available so that we can easily determine if a change in one of our dependencies breaks things is critical in for a world of interconnected but independent components to be viable.

But most of all I’m very excited that Google chose us to participate and that we had such a great set of proposals to choose from. Thanks Google! and thanks everybody who wrote a proposal.

April 22, 2008 02:32 PM

April 21, 2008 12:44 PM

The tg-admin script that is bundled with turbogears is really helpful, but I had a hard time learning how to use it.

Before you read any more, you should know that this only works when you use SQLObject, not SQLAlchemy, for your ORM.

These are my notes on how I use tg-admin to upgrade an existing database.

• I have a production database that uses prod.cfg;
• I have a development database that uses dev.cfg;
• Neither databases have a sqlobject_db_version table initially, because I never payed attention to it yet.

The development database has a bunch of new columns, tables, and indexes that I want to add to the production database. For this example, I’ll pretend that all I want to do is add an index to a table.

First, I made sure that the dev database matches sqlobject classes:

tg-admin -c dev.cfg sql status


If those are out of sync, then do whatever you need to do to make sure your actual dev database matches your classes. Of course, tg-admin sql status is not perfect. For example, it overlooks missing indexes and constraints, at least with postgres.

Next, I recorded the state of the development database:

tg-admin -c dev.cfg sql record --force-db-version=2008-03-21

This will make a new table in the dev database called sqlobject_db_version. I am forcing it to have a value of today’s date (March 21st, 2008).

Now I connect to the production database and set a version on it with yesterday’s date:

tg-admin -c prod.cfg sql record --force-db-version=2008-03-20


Now I run this to try to upgrade the production database to match the development database:

tg-admin -c prod.cfg sql upgrade


Of course, that should fail, and I see some error message sort of like this:

$ tg-admin -c prod.cfg sql upgrade
Using database URI postgres://staffknex:staffknex@localhost/staffknex320
No way to upgrade from 2008-03-20 to 2008-03-21
(you need a 2008-03-20/upgrade_postgres_2008-03-21.sql script)


This is an example of a helpful error message. I need to write a script that will explain how to upgrade from yesterday’s version to today’s version.

That script will be really simple:

BEGIN;
CREATE UNIQUE INDEX majestic12 ON ufo_theorists (first_name, last_name);
END;


I suggest using BEGIN and END so that in case something goes wrong in the middle, your transaction will be rolled back automatically.

Now I can run this:

tg-admin -c prod.cfg sql upgrade

And my production database will be upgraded with the new index.

Now for some complaints:

• Why isn’t this advertised better? This is a really nice feature.
• You’re supposed to be able to specify the URI on the command-line with the –connection option, but I could never get it to work.
• I really wish that tg-admin sql status detected stuff like missing indexes and constraints. I use these things heavily.
• It would be nice to be able to mix python into the upgrade script, rather than just SQL. For example, I recently dropped a column that had both an employee’s first and last name, and separated this into two columns. I used SQL to make the new columns, then I used python to read data out of the old single column and write it to the two new columns. Then I used SQL again to drop the old column.

Like I said at the beginning, this is a really helpful script and I’m very grateful to whoever wrote it.

April 16, 2008 05:02 PM

April 16, 2008 12:53 AM

I thought the top 10 command line meme was interesting, and it seems to be going around.

For me nosetests seems to be by far the most run command on my system — I guess that means I’ve been infected the testing meme as well.


ramoth:mramm markramm$ history|awk '{a[$2]++} END{for(i in a){printf “%5d\t%s\n”,a[i],i}}’|sort -rn|head

169 nosetests
52 ls
48 cd
31 Python
27 sphinx-build
27 bzr
20 svn
17 hg
15 ping
11 easy_install

Also I’ve been working on TurboGears 2 docs, and I’m loving sphinx. I also note that 6 of the items on the list are python, and 3 of them are version control systems that I use to manage python code. I expected svn to come in a lot higher, but I reallized that this list is has been influenced by the fact that i use QuickSilver to launch my text editor, and I have been using the svn plug-in a lot.

April 11, 2008 01:29 AM

$ history|awk ‘{a[$2]++ } END{for(i in a){print a[i] ” ” i}}’ |sort -rn|head
241 nosetests
97 ls
85 cd
20 svn
15 python
7 vi
7 rm
5 easy_install
4 pylint
3 history|awk

April 11, 2008 12:39 AM

Are you a software engineer? Programmer? QA specialist? How long have you been at your current position? How long do you expect to be? I bet it’s less than 5 years.

Why would that be an important issue for open source? Besides, most of the time you spend most of your time working OSS projects at home, right? But I bet someday you’d like to get paid for that work, unless like some OSS developers you base your happiness on the proliferation of your software, of which I have met few.

So one day you are going to pack up and leave your company, and all of it’s proprietary software behind. And if you are developing Python, that probably means you are leaving a gaping, expensive hole for your former employer. But what if you could keep working for that employer, after you left? What if what you were doing at the new job was so transferable that you could be productive from day 1?

This is the promise that is OSS. And while it is probably idealist, it has at least one tentacle sticking out from reality. The fact is, workers are migratory these days. Personally I think this is because short-sighted investors value fiscal gains over a longer payoff, but we live in the day of day-trading. The fact is, many people don’t see programmers for what they really are, a mixture of engineer, mathematician and artist. I can always seem to find someone who will pay me more money to do the same job I already have. When something I just can’t turn down for my family’s sake comes up, I have to take it. For this reason programmers, and even the greater workforce migrates.

Employers have already realized that employees are migratory. They have all but done-away with pension plans in exchange for portability: the 401k. Who cares what kind of health insurance we offer? 80/20 is now the norm, because employers don’t expect to have to pay in the long run, and employees accept it too.

But lets get back to the task at hand. What does a migratory worker meant to open source. Portability of work. I can take my Django/Turbogears/Zope knowledge to any other company using these technologies and pick up right where I left off. If I am one of the few developing software in the OSS world (as apposed to just using it) I can actually contribute more to my OSS project on my companies dime. The question is, are the companies really losing out by providing their software to the world?

Definitely not. In fact, they gain much more then they lost by choosing something “free” and open. They retain their worker after said person has left the company (which they undoubtedly will). Bug patches, new features, etc. all pulled from the same pot they were putting their goodies in before. Since the software is open and free, more people have access to it in the age where replication of information is almost cost-less. That means they have a larger work force to choose from, and more experts and shared expertise to gain from.

If you are a company who is battling with the decision to contribute to, or even use open source, I implore you to look seriously at your options, your [migratory] workforce, and your ability to develop a product for the long haul. I bet you find more value in free software than the sticker price. Software, more than ever, is about the people, not the product.

April 04, 2008 09:15 PM

In June 2006, I shipped the TurboGears Ultimate DVD, featuring several hours of useful screencast material for TurboGears programmers. I produced the DVD with TurboGears 0.9a6 (or so) code, but much of what is talked about there applies to TurboGears 1.0x users. A big thanks to Ian Oszvald and Kyran Dale for putting the effort into getting 2GB of material transcoded and online: TurboGears Ultimate DVD (TG v1.0) - video tutorials to learn turbogears, web_development, web_framework, python, javascript, web_application, cheeseshop, generic_function, JSON, metaclass, widget, API, sqlalchemy, cherrypy, sqlobject, WSGI

ShareThis

April 04, 2008 04:06 PM

I have recorded a screencast of my PyCon 2008 talk and put the code for the demo app online. Check it out on the SitePen Blog » Rich UI Webapps with TurboGears 2 and Dojo Screencast.

ShareThis

March 31, 2008 05:23 PM

Zed Shaw and I spent quite a bit of time talking about Documentation tools, since we are both working on books for Prentice Hall and we both want to have an open-source, reusable tool-chain that helps us get tested code into our documentation easily and effectively.

Zed did a bunch of hacking and came up with this:

http://www.zedshaw.com/projects/zapps/idiopidae.html

Which has a very simple syntax for importing code into your plain-text documentation file. It’s a simple parser that we use to parse the document source and the export files, create parse trees, and do a bit of simple processing. One of the main advantages of this for book writers is that you can have the source code imported into your plain text documentation automatically, and repeatedly. That means you can look at the code while you’re writing. The final parsed output will allow you to convert the plain text document into your final output format (HTML, LaTeX, or whatever) and it will use pygments to code formating/colorization, and interlieve the code into the rendered document.

I also learned that the tool being used to do the new Python 2.6/3.0 documentation has been cleaned up and released as a general purpose documentation tool. It’s ReST (ReStructured Text) all the way, which seems to be a small price to pay for such attractive docs, and such a wide feature set. It also relies on imports from specific line numbers — which is a lot more fragile than marking sections for import, but does not require you to be able to include delimiters in your source code files.

I haven’t had a chance to use it yet, but I’m very excited about it as a tool for the TurboGears online Documentation.

Hooray for better doc tools!

March 21, 2008 08:50 PM

First I want to congratulate Chris Arent and Chris Perkens who put a lot of work into the Google Summer of Code application for TurboGears. The GSoC has been very good for python, and good for TurboGears in the past, and it’s really nice to take this to the next step and become a GSoC mentoring organization this year.

With that said, there is one project being sponsored by the TG team which I want to highlight because I really want to see get picked up by talented and motivated student. The first would be the Genshi speedup stuff, this is going to require some python profiling and optimization skills (obviously with lots of mentor help) and will likely require some C coding as well. But it will be a huge benift not just to TurboGears, but also to Trac, and all kinds of other Python projects which want to produce XML output for whatever reason. Recent benchmarks have shown that TG2 is very fast when you don’t use genshi match templates — so making this stuff faster will have a big performance impact.

So if you’re a student, and interested in doing something challenging, interesting, and useful, please take a look at this, and the rest of our GSoC ideas on the wiki — and feel free to suggest new things on the mailing list.

March 21, 2008 08:26 PM

TurboGears has been accepted as a Google Summer of Code organization. Chris Arndt has put together a page at docs.turbogears.org to pull together all of the TG Summer of Code information. Congrats to Chris and the others for pulling together this year’s Summer of Code effort for TG!

Next week is signup time for students (March 24-31), so if you’re interested in getting paid to work on TurboGears this summer, check out the TG SoC info page.

ShareThis

March 20, 2008 11:19 AM

March 18, 2008 03:03 AM

I just returned from PyCon 2008 in Chicago, where I connected up again with a few people I met last year and met a few new people. Rather than write a travelogue, I'll just highlight a few of the main things that I learned/did.

It looks like some of the niceties of Zope2 are going to leave the silo under the Repose project. This means that the zope transaction manager (repoze.tm), retry engine (repoze.retry) and other zope features can be used by your WSGI application. Also it means that Zope can be hosted as a WSGI app.

I learned there are ways to extend _import_ that don't involve rewriting _import_. I don't know if I'll ever need to know this, but it might be useful in a plugin loader of some sort. There's a flow chart in Brett's SVN.

I should probably use more weak references in my Python code. Dr. Tim Couper gave a lucid overview of references weak and strong, and practical ways of dealing with circular references, including how to detect them using unittest. I hope he posts his slides soon.

Kevin Dangoor gave a slick demo of TG 2 and Dojo, and gave a quick introduction to Comet servers. It looks like dojo does pretty much everything that Mochikit does, but adds support for Javascript UI widgets. Comet looks really cool. More about that later.

Tahoe is a nifty looking distributed remote filesystem concept. It needs fuse drivers though before it's really a filesystem rather than a storage mechanism (like S3). It's all RESTy though, which makes some people happy.

There are metrics for measuring code complexity, and python has tools to do that. Basically every branch in a method increases complexity by one. Keep complexity down to 10 or less to make unit testing feasible. More than 10 and the number of unittests required to cover every branch starts to get unmanageable. Apparently there is a PyMetrics module for measuring code complexity. Here are the slides from a very interesting talk.

I held a BoF on i18n'izing web applications, and a few people helped me to brainstorm this problem. The general consensus was that you should do translation as close to the user as prudent; view certainly, controller is ok, but never in the model, if you have messages that are generated by some other process, or that gets cached to the database, the "data" should be stored separately from the operational message. Generally you should avoid sending this kind of data to the end user, and instead abstract it with messages of your own. Barry Warsaw came in to the BoF with a hard problem; usually when you're handling web templates, you filter out the non-element text and build a string table out of that. What about the cases however, where you have <em> or <b> tags? I think that if you had an XML parser that would extract the text elements, and if sub-elements are present lump them with the text. For example, if you had "This text <b>needs</b> to be translated", it would be collected as one string, but if you had "<div><span>Phrase 1</span><span>Phrase 2</span></div>" it would be two phrases.

There are a couple of cool tools that have stemmed from the PyPy project. The first offers a framework for distributed testing. Py.test does stack introspection so that simple "assert" statements can be used instead of the pseudo self.assertEquals().

I got together with some people to talk about Python Packaging. Here are some notes from the meeting, and there's been some additional discussion on the distutils-SIG mailing list. Jeff Rush did a tutorial on Thursday, and his slides and exercises are available. Hopefully something coherent comes out of this, but it looks like more of the same resistance to making application installation reasonable.

I talked briefly with Ivan Krstić and Noah Kantrowitz about python plugin frameworks. Hopefully we'll be able to collaborate some on a generic system for application extensions.

Probably the coolest thing I learned about was a couple of protocols, both implemented by orbited. Orbited's client libraries allow for a user to connect to a orbited server to provide a push mechanism for sending messages to a web browser. The Orbited team was shooting for a 0.4 release after the PyCon Sprints this week. Check out their IRC client demo.

PyCryptoPP was suggested to me as a decent Python based Crypto library, since it's simply python wrappers for Crypto++. PyOpenssl has a new maintainer, and should be getting some much needed attention.

I also enjoyed Raymond Hettinger's talk on "Core Python Containers". It was very helpful in understanding what list, dict, deque and set do behind the scenes, and how to use them most efficiently.

Overall, I had a great experience at PyCon 2008. The venue was big enough for all of us (there were more than 1000 registrants), and there was more than enough room for Open Space/BoF talks.

Chicago style pizza is definitely different from anything I have eaten before. The jury is still out as to whether it's worth the hype.

March 17, 2008 02:57 PM

These are a few of the rules I try to follow in my design. So far, they’ve helped me out.

I aim to finish all interaction with the database before I get to the template layer.

This is non-trivial because it is so easy to forget that a method or an attribute will evaluate into a query. I use this rule because it lets me be certain about the number of interactions each page will have with the database.

I avoid branching (if-else clause) in my templates as much as possible.

I have a really hard time detangling code when I find a bunch of nested if statements. For all but the most trivial instances, I prefer to have a bunch of similar templates and then choose the best one. For example, instead of handling both a successful login and a failed login in a single template, I’ll make two different files and then choose the right one in my controller.

In practice, I have some really similar templates. But then I go back and strip out as much of the common code as possible and put those into widgets.

Any time I find a select() call in my controller, I consider making a new method in my model.

When I write something like this in a controller:

bluebirds = model.Bird.select(Bird.q.color == 'blue')


I usually come back later and put in something like this into the Bird class:

class Bird(SQLObject):
    color = UnicodeCol()
 
    @classmethod
    def by_color(cls, color)
       return cls.select(cls.q.color == color)


Now I have something that I can reuse. If I’m feeling whimsical I’ll use functools.partial to do something like this:

class Bird(SQLObject):
    color = UnicodeCol()
 
    def by_color(self, color):
        return self.select(self.q.color == color)
 
    redbirds = classmethod(partial(by_color, color='red'))
    bluebirds = classmethod(partial(by_color, color='blue'))


Sidenote: I couldn’t figure out how to use the @classmethod decorator in the second version of by_color because partial complained. Appararently, callable(some_class_method) returns False, and partial requires the first argument to be a callable.

Maybe a reader can explain to me what’s going on there…

March 16, 2008 04:18 PM

Ben Bangert and I did a tutorial at PyCon this year, and it was hard. We’d just cut eggs for TG2 and pylons pre-release versions that morning after months of trying to have releases done before PyCon. So, I personally was a bit frazzled before the tutorial even got started, and then we had a huge number of technical issues with the pycon ireless, which lead to various egg-dependency problems.

Our back up strategy for sharing the eggs was to set up an ad-hock wireless network, and sharing them out directly. But the PyCon folks specifically asked that nobody setup ad-hock networks. So we had to fall back to burning CD’s, putting eggs on flash drives, and walking around helping people get installed.

Ben Bangert and Chris Perkins did a great job of helping people get stuff installed, while I walked the people in the room through a basic WSGI tutorial, so they would have the conceptual framework needed to understand how some of the TurboGears2 and pylons code works.

We then went into the TG2 tutorial proper, but by this time we were running pretty late, and the need to provide both TG2 and Pylons examples became somewhat problematic in the tight timeline we had. I worked hard to make sure that there were materials that the tutorial participants could take away with them, and I think that paid off because we didn’t get to finish the TG2 tutorial. But hopefully people will be able to go through the tutorial on their own.

And I created a Google Group for tutorial members so that they can ask questions, get help, and otherwise continue to learn stuff even after the talk itself is done. If you were at the tutorial, and you didn’t get a chance to join the group — but still want to, send me an e-mail and I’ll get you added.

I learned a lot about how to handle crazy problems in the context of a large tutorial group, and I’ll be doing a number of things differently next time.

March 15, 2008 03:28 PM

I recently ordered a TiVo HD from Woot, in order to replace the absolutely horrifically bad Motorola HD DVR that was provided to me by Comcast, the world’s leader in being terrible at everything. I have been extremely excited about getting the new TiVo HD set up, and after placing the order, my first task was to call Comcast to order my CableCards.

On my first attempt calling Comcast, I politely explained that I would like to get a single Multi-Stream CableCard to install in my new TiVo HD. The representative on the other end of the line said “we don’t support TiVo.” Understanding that the person on the other end of the line was not actually a human being, but a tech support monkey, I politely asked for her supervisor. The supervisor was nice enough, and told me that I could pick up a CableCard at my local office and that I didn’t need to do anything until the TiVo HD arrived. Comcastic!

The next day, on a sneaking suspicion that the supervisor had their head up their posterior, I initiated a chat with Comcast’s technical support team through their website. This time, the technical support engineer not only knew what a CableCard was, they explained that I would need to place an order for one, which they walked me through. “You’ll be able to pick it up at your local Comcast office when your TiVo arrives.” I felt reassured, and later that day, I received shipping confirmation for my TiVo HD from FedEx. I was on my way! Comcastic!

The next morning, I received a call from my local Comcast office. They explained to me that I could not, in fact, pick up the CableCard. They would have to send out an engineer. Okay, fine, I said. Anything to get rid of this terrible Motorola DVR. The appointment was made, and I was told to be home on Friday from 2:00 PM until 5:00 PM, when a technician would arrive and install a single Multi-Stream CableCard. Thankfully, I work from home, so this would not be a problem for me.

Thursday night, I hooked my TiVo HD up to my TV, disconnecting the old Motorola DVR, and went through the guided setup, which was straightforward and pleasant. In the process, I spent 3 minutes on the phone with TiVo, who activated my service quickly and efficiently. All I needed now were the CableCards from Comcast.

Friday morning came. Then Friday afternoon came. Around 3:00 PM, square in the middle of my “window” for service, the phone rings. Its Comcast. They wanted to confirm with me that I’d be getting two new cable outlets installed in my home today. Huh? What?! That can’t be right…. I explain to the representative that is not what I have asked for, and she assures me that she’s put a note on the account, and that I should expect the technician to arrive within the scheduled time.

Its 4:59 PM. Comcast still isn’t here. I pick up my phone and call Comcast. The representative says that she’ll send a note “to dispatch” and see what’s up. “It will probably take about an hour or two for them to respond.” But, we had a window. A three hour window! Knowing that there was nothing else that could be done, I waited.

5:34 PM. A Comcast truck pulls up in front of my house, and a technician gets out of the car with two Single-Stream CableCards in his hand. Okay, so its not exactly what I asked for, but it’ll have to do.

6:28 PM. The CableCards are in my TiVo HD, and after no less than 14 phone calls to Comcast, most of which were spent on hold, or being transferred to someone who knew what a CableCard was, my TiVo HD seems to be receiving its channels through the CableCard.

6:29 PM. We realize that in the process of installing the CableCards, the technician has managed to remove my cable modem from my account, and I am now without internet. The technician explains that this is not his problem, and that he wants to leave. I firmly ensure that he stays with a glare that would put fear in the heart of the devil himself.

6:54 PM. Internet has been restored. The TiVo HD seems to work. I tell the technician to leave.

Comcastic!

As a short aside, anyone who thinks that Apple is going to get into the DVR game is insane. If they were uncomfortable with the cell phone industry, what do you think they’ll think of the mess that is cable television?

March 15, 2008 01:00 AM

Last year, I made the difficult decision to change my career path and start working with some very talented family members on a new endeavor – ShootQ, a studio management application for photographers. Its been a pleasure to work on a product for people who are always searching for ways that they can help change the world through their art form.

Today, I am immensely proud to be a part of ShootQ, as we announce the ShootQ Grant, which will help a project that raises public awareness about an important social, environmental or economic issue through a $10,000 award. From the website:

ShootQ’s goal is to provide tools for photographers which free them from the tedious work of running a business. With ShootQ Grant, our mission is to empower photographers to invest time working on projects that raise awareness about important issues. More than a monetary award, the ShootQ Grant will provide photographers the opportunity to work on a meaningful project, unencumbered by daily work, with the creative freedom and financial support necessary to produce photographs that raise public awareness about important social, environmental or economic issues.

In my previous work experience, the measure of success has been sales, stock price, and accomplishments. I am proud to be a part of a business that isn’t solely about helping ourselves succeed, but is also about helping others to succeed. Not only that, we’re all committed to helping spread important messages, and have a positive impact not only on our industry, but on our world.

And for those of you waiting to see what I’ve been working on for the past few months, I promise you that your patience will be rewarded. I am extremely happy with the progress that we’re making, and can’t wait to share more. Stay tuned!

March 13, 2008 06:00 PM

TurboGears is undergoing a monumental effort to participate in GSoC.  Ok, maybe not monumental, but at least 5 of the developers have been hard at working putting together an application even Google wouldn’t turn down. Even if we are not accepted, we are planning on participating by way of PSF, as we have done in previous Google Highly Open Participation contests.  We have developed a number of ideas which students can choose from, or students are welcome to come up with their own TurboGears ideas, and I am sure that one of our mentors will be able to match up with you.      

My own ideas revolve around DBSprockets and TwTools, which is not surprising since I am the owner of said projects.  The largest project and the one which I have the most desire to see put into action is that of a TurboGears CMS.  There had been some work done last year by the guys at Pagoda which produced a brilliant screencast, but the project seems stalled out and it would be nice to see it revived.  Furthermore, the solution I proposed is intended to be much more modular, so you could pick apart portions of the CMS and put them in your existing applications.  I think this would be the most flexible solution, and also one which would employ much that DBSprockets, TwTools, and Toscawidgets have to offer.  If you are a student who is interested in working on this project, please don’t hesitate to drop me a line.  You can also track me down at Pycon for the remainder of the week, and at the TurboGears sprint next week.  If you are a student who is eager to get started feel free to participate in our sprint, remotely or in person.   

One of the great things I see coming from this mini-project is that we now have a very nice set of concrete ideas about how to make TurboGears better.  Whether or not students participate in the development we still get a huge benefit from the creation of ideas, and it gives the development team and possibly new developers a target to make TurboGears the best it can be.  My hope is that these ideas will not only bring a students to our project, but also bring some developers out of the woodwork who may have started similar projects and would like to contribute.  The bottom line is, if you aren’t a contributor, and want to be, here is a great place to start.

March 12, 2008 09:46 PM

I love SQLObject, but this is a rant about the tiny frustrations I face with it.

First, this is a minor point. I don’t really care about database independence that much. Postgres has a lot of wonderful features: I never have to worry about choosing the table engine that will enforce foreign key constraints, I like creating indexes with function inside:

create unique index nodup_parent on category (org_id, parent_cat, lower(name));

and I really like how easy it is to write stored procedures. Anyway, since I know I’m going to use postgresql, I don’t want to be restricted to only the features that exist or can be emulated in every platform. I know all about sqlmeta and createSQL and use it plenty. But I don’t like how when I set a default value, sometimes it is set in the database table, and other times, it isn’t.

Anyway, in practice, the most dangerous part of using SQLObject is that it hypnotizes you into forgetting about the queries behind everything. Imagine you have employees, departments, and a join table between them. You can set this up in SQLObject like this:

class Employee(SQLobject):
    name = UnicodeCol(alternateID=True)
    departments = RelatedJoin('Department')
 
class Department(SQLObject):
    name = UnicodeCol(alternateID=True)
    employees = RelatedJoin('Employee')

You want to draw a grid that indicates whether each user is a member in every group, so you might dash off some code like this:

for emp in Employee.select():
    for d in Department.select():
        if d in emp.departments:
            print "yes!"
        else:
            print "no!"


In an ideal scenario, you can do this with three simple queries:

• You need a list of employees
• You need a list of departments
• You need the list of employee-department of associations.

People that talk about how you can use outer joins to cram all that into one query will be dropped into a bottomless pit. Besides, I profiled it, and three separate queries is often much cheaper.

Anyway, back to the point. SQLObject will only run a single query to get the employees and a separate single query to get all the departments. So that’s good.

However, the place where all hell breaks loose is that if clause in the middle. If we have three employees and four departments, this statement

if d in emp.departments:

executes a dozen times. That’s unavoidable. The problem is that each time it executes, SQLObject runs a query like:

select department_id from department_employee where employee_id = (whatever);

Every time you say “is this particular department in this employee’s list of departments?” SQLObject grabs the full list of departments for that employee. So, if you ask about 10 different departments, you will run the exact same query ten times. Sure, the database is likely to cache the results of the query for you, but it is still very wasteful.

With just a few employees and a few departments, that’s not so bad. Eventually, though, as the number of employees and departments grow, the cost of that code grows at N², which is just geek slang for sucky.

So, in conclusion, this may sound like a rant, but it really isnt. SQLObject is great. But it isn’t magic. It’s a great scaffolding system. But now I find that I’m rewriting a fair portion of code in order to reduce the database costs.

Aside: when I started paying attention to the queries generated by SQLObject, I found it really useful to edit postgresql.conf and enable log_min_duration_statement. Then every query and its cost will be logged for you. This is really useful stuff. It’s helped me to relax about doing a lot of things that I used to think were really bad.

March 11, 2008 07:16 PM

A lot of people tell me that they want to write a technical book for one reason or another. And I think that’s a great goal that can really stretch you as a communicator, as a programmer, and as a human being — so go for it. But if you’re thinking about it, I’d suggest that you learn from a couple of my mistakes. ;)

People might tell you that writing technical books sucks because you don’t make much money. (Which is true, as far as it goes). Or they may tell you that writing books sucks because it’s hard work. Or they might tell you how much time you spend away from those you love. And those things are true. But I don’t regret any of those things about writing the TurboGears book.

I do however, have a couple of process related regrets, and I’ve felt for a long time that I needed to write an article to codify some of the things I’ve learned about writing, so that prospective book authors and open source framework/library documenters have a shot at avoiding some of my rookie mistakes.

The two most important things that I learned from writing the TurboGears book were:

• Every single line of code needs to be tested, not just before it goes in the book, but every time you make changes. If you don’t do this code will get broken in the process of last minute reorganization, rewrites, and crazy insanity.
• It’s better to take time to do it right, than to rush something out the door that’s not what people need.

The testing issue is the most critical thing about book writing and it comes in two parts — both of which are far too easy to ignore. First code needs to be tested to make sure that: it runs, it does the right thing, and it makes sense. The first two tests are automatable, and really need to be automated. Refactoring, and rewriting are fundamental to making good code and good books, and you can’t confidently refactor without tests. And since I think book authors should be testing the code to make sure it makes sense, but getting target audience readers to read-and-understand it and making it shockingly easy for them to provide feedback, it’s likely that lots of refactoring opportunities will come up.

Unfortunately, though the Pragmatic Press people have one, as do many, many authors, I’m not aware of a single openly available tool which is designed to testing book-code easy. And I think this is a shame because even if you’re not writing books, every open source library needs documentation, and most of them need tutorial style documentation which requires the same basic tools. So, I’m hoping that some of us can join forces to get a tool like this started at the PyCon Sprints next week.

There have been two approaches to the problem:

1. Suck code from external source-code into the document itself.
2. Take code from the document, and mark it up with a list of external resources needed to test it.

Based on my unscientific results it looks like the first approach is more popular than the second. But the second approach has one very significant advantage — all of the code is visible while you’re writing the text and therefore you are less likely to have “refactoring” bugs that cross the text/code boundary (a method name is changed in the code, but not the text that describes it).

With that said, there are a number of compelling advantages of the suck-in code method. First, it’s relatively language independent. You just need to define what comments you’ll use to mark off code in the project (to add formatting, and mark the beginning and ending boundaries) and create a simple structure that runs the native language tests, and then builds the document. You may need to adjust things slightly for languages with different commenting conventions. And it certianly seems like multi-language support would be a lot harder to achieve when pulling code out of the document.

Also, I’m very much a believer in the idea that both the source code and the document-text source should be in an plain text format that’s easy to keep in version control, easy to track and easy to diff. I also want to be able to use the same editor for both my document source and my source code.

But in order to mitigate the possibility of the kind of “refactoring” problems I mentioned a minute ago, we ought top make it really easy to create rendered documents. I suppose you could work in two windows with the source-document in one and the rendered version in a second, but it would be even better if you could leave the “processing directives” that grab the code above the rendered-source in a plain text document, and then mark the end of the code samples in the rendered document, so that a document could be safely edited (while looking at the source) and then re-rendered at will.

If you’ve got an internal toolchain you think might be valuable as a reference for us, please let me know. And if you’ve got a couple of days and want to contribute to making Open Source documentation better, while making it easier to write good technical books, feel free to drop in (in person, or virtually) to the TurboGears sprint at PyCon next week and we’ll see what we can do.

http://docs.turbogears.org/SprintOrganization

March 11, 2008 01:53 AM

Well, in the week since my post on Jython, some really good news has come to my attention.

Ted, and Frank are now working on Jython and Python stuff for Sun Microsystems. I’ve heard from Frank, Jim, and Ted that the work on the JVM is actually already a much more hospitable environment for dynamic languages than I thought. And all of them have pointed to the Da Vinci Machine project as being an active project with a community that’s very concerned with making the JVM into a really great platform for dynamic languages.

Ian Bicking suggested that the runtime environment of the JVM isn’t all that friendly to Open Source sensibilities (the VM takes a while to load, it’s got WAY to many options, and is too complex to tweek properly for memory issues.) And perhaps that’s true, but I think the least friendly bit of the JVM is the what feels like open hostility towards C libraries. In my opinion this is the biggest single issue in terms of hostility to the Open Source way. So much of the Open Source community is steeped in a Unix+C ethos that’s very hard to shake. And for significant number of problems, there are indisputable performance gains that you can get when you manage the way that your data structures are arranged in memory yourself. I’ve been told that this is getting better too, and that the horrible days of the Java Native Interface (JNI) are numbered.

On another front, I had hoped that the Jython and Jruby people could reach across their respective language community cultures, and work together. And I’ve been told a good chunk of the JRuby team is coming to the PyCon sprints to help with the Jython sprint — which is awesome!

In other news:

Somebody suggested that perhaps the Tamarin virtual machine might be a viable alternative to the JVM, with the IronMonkey project porting IronPython to that vm. I honestly don’t think that will happen, both because the IronMonkey project seems stalled, and because the IronMonkey project was just a IL to Tamarin VM “bytecode” cross compiler, so it would likely always be a step behind the .NET guys in terms of performance.

March 06, 2008 05:11 PM

At CodeMash Neil Ford gave a keynote about polyglot programming, a subject about which I’ve been thinking a lot — and a subject which I have a slightly different (and somewhat more pragmatic) set of opinions than Neil expresses.

But, at the moment I’m hung up on a throw away section of his talk which I think indicates a misunderstanding about the way language and metaphor work. A misunderstanding which is probably particularly common among programmers, but which goes to the heart of why I think software development is so difficult.

Niel takes exception with the notion that software development is software engineering saying something along these lines:

It’s not true. It’s a metaphor. In fact, it’s a ‘tortured’ metaphor. Software and engineering really are different. What we need is to get away from these metaphors, since they cloud the issue, and describe what it is we actually do.

He goes on to say a lot of interesting stuff about how engineering is different than software development, which is of course true. But I think that misses the point a bit. Methaphors aren’t supposed to be flat descriptions of fact. If I say “I’m as tired as a wet dishrag” nobody expects that they could measure my metabolic state against that of the dishrag and determine the “truth” of my statement. It’s true in a very different way. And I’m not just talking about creative language, or artistic merrit. Truth in this case is all about explanatory power.

Natural language is imprecise, analogical, and metaphorical — and after 4 years of philosophy, linguistics, literature, in school and 10 years of software development I’m convinced that these aren’t just surface traits of most language, they are the fundamental constructs from which all human language interactions are built. And, more than that I’m certain that this is actually a good thing, because the world is filled with things that we can only talk about that way. But I’ll get back to that in a second.

First, let’s dive into why I think Neil is wrong, and why I think his error is the kind of error that programmers are particularly prone to fall into.

Programming is both more abstract and more concrete than natural language. As Ocean tells us:

That there are no metaphors in programming, that there is zero ambiguity and every single line of code always has a definite, unambiguous, unarguable, and precise meaning has a very important consequence: integration is extremely difficult. Unlike poets who can take practically anything and throw it at the page and make it stick with a bit of jiggling and handwaving, a programmer must speak in a language that a computer can understand — ultimately, in 1’s and 0’s! And so it’s almost impossible to seamlessly combine programs to create new programs.

Every line of code has a specific well defined meaning. And unlike language that meaning is not the internal subjective representation of the code in the reader’s mind, nor a shared but nebulous social construct. It’s totally external and absolutely quantifiable. Code is defined by the way in which it will be parsed and executed by a pile of silicon.

Natural language is totally different.

One of the most influential philosophers of the 20th century, and very smart dude — Ludwig Wittgenstein — reminds us, it’s very hard to define many words even though everybody knows what they mean. The famous Wittgenstinian example is the word game which applies to things like ring-around-the-rosy, professional football, chess, World of Warcraft, hopscotch, and solitaire. The important thing to notice about this example is some games are played competitively, and others are not, some have well defined rules, others do not, some are frustrating and complex, others are simple and fun, some are professional, and others are done for pure pleasure.

You might suggest that there’s some core idea of amusement involved in all games, but that begs the question what is amusement? How could it be the same for children playing duck-duck-goose, football players at work, and chess masters practicing.

But the most important thing to remember is that even with all these “complications”, it’s obvious to everybody what we mean when we say game — even if they can’t provide an adequate working definition of “game” that actually covers all the things we commonly call games.

As I understand him, Wittgenstein, goes further to suggest that it is the lack of logical rigor, this tendency towards analogy and metaphor, this embrace of ambiguity that makes language useful. I think that’s very much true. Without ambiguity and metaphor we couldn’t communicate complex new ideas, and we couldn’t extend our base of knowledge from one area to another, and we would be entirely unable to communicate anything about huge swaths of human experience at all.

Back to programming. Program languages can’t express love, or disdain, there’s no such thing as irony. There’s no way to say anything meaningful about art or love or spirituality. It’s all concrete instructions that operate in a well known context. So, I see what Ocean is saying when he argues that programming is more concrete than natural languages.

But at the same time, I’m convinced that software development is all about creating systems that encapsulate various kinds of complexity. Another way to say that same thing is that software is all about building up layer upon layer of abstractions, and software debugging is all about the complex process of learning to traveling up and down the “abstraction stack.” And the terminology in software development is all divorced from any easy-real-world referent, “objects” aren’t physical, “stacks” and “heaps” don’t really “take up space” and memory “locations” are only metaphorical. The layers of metaphor requires to say the simplest thing about software development can be enormously daunting. In contrast with natural language it’s so easy to see the relationship between the word “pencil” and the real world object it represents.

So, software development is inherently much more abstract, and is often almost completely divorced from the concrete referents that make up everyday language. Which means it makes lots of sense to say that any kind of reasonable discussion of the the act of programming is ultimately less concrete than most natural language, as is any discussion of the code. And, really what else would you expect from a discipline build around building abstractions and encapsulation of ideas in executable forms.

Ultimately good programmers build up metaphors which provide layers of abstraction which are effective presisely because they offer a non-literal way of talking about the complex things that happen “undernieth” that layer. Take away the metaphors, and you won’t be able to talk about what you do. You won’t be able to explain what you do to other programmers, and you won’t be able to think lucidly about the problem domain and the code at the same time.

All this is the very long way round to saying that programming requires all kinds of mental tricks, and that ultimately talking about writing code is going to require metaphors. With all respect to Niel Ford, the problem is not that we use metaphors, it’s that we don’t understand how to use metaphors well. The more metaphorical our metaphors are, the better, because then they aren’t easily taken for flat descriptions of fact. As far as I am concerned, Software Development is like Engineering, but it’s also like washing windows on a skyscraper, playng chess, or trying to understand what language your teenage daughter is speaking.

Fundamentally I’m convinced that learning to do software development really well is about learning how to communicate really well. And communicating really well means working at the level of metaphor and analogy because the problems, requirements, and users of your system all operate at this level. It also means learning to translate that into code which expresses the metaphors that help people solve their problems in a medium (code) that has zero ambiguity, but to arrange that code into systems built around sensible — and easily communicated — metaphors.

March 05, 2008 10:04 PM

February 29, 2008 11:00 PM

There are a number of very powerful template languages available in Python. Some template languages, such as Genshi, allow complex Python statements and even full Python code within the template file. Others, such as Django templates, prefer to restrict templates to presentation only and do not allow general Python expressions ...

February 28, 2008 08:00 PM

For years, I have been able to keep my personal email address virtually spam-free by never putting it in any public place, entering it into any forms, or sharing it with anyone I didn't actually want to talk to. For me, this has been blissful. Generally, when I receive email, its actual communication with a person I genuinely want to talk to, and I enjoy reading email.

A few years ago, my email bliss ended abruptly.

Spam? You May Rely on It

At my previous employer, I was forced to use Microsoft Outlook for my work-related email. From day one, I received hundreds of spam messages a day at my work email account. The Exchange server died routinely, leaving me without access to important communication. When I sent email, sometimes it just wouldn't be delivered, silently failing.

One day, in a moment of weakness, wanting to send a link home to myself so I could remember to read it later, I opened Outlook, pasted the link into a new message, and emailed it to my personal email address. The moment I pressed "Send", I knew I had made a mistake. But there was no going back. It was too late. My email address now lived in the Exchange Server.

The next day, spam started trickling in. A few days later, the trickling turned into gushing. As of last week, I was receiving 50 or 60 emails in my inbox a day that Spamassassin wasn't catching. My bliss had ended, and the era of spam had begun.

That is, until yesterday.

Outlook Not So Good

In a bout of frustration, I started looking at the headers of all the spam that was getting through my server-side filter to see what the spam scores were. Many of them were "close" to being marked as spam, but even more of them passed completely through. One thing I immediately noticed, though, was that all of the emails he had a specific header, with the same value, every time. Last night, I added the following rule to my procmail configuration, out of curiousity:

:0:
* ^X-Mailer:.*Microsoft.*
$DEFAULT/.Spam.Outlook/

This morning, my inbox had three new emails from friends, family, or acquaintances. My "Outlook Spam" folder had 87 new messages, all of which were spam. For the first time in over a year and a half, I had no spam slip through the cracks.

I am sure some of you will point out, now any email that I receive from people who use Outlook will end up being flagged as spam. Fair enough, but do I actually want to talk to those people? My magic 8 ball says: "Very doubtful."

Saved by Outlook. Sweet, sweet irony. Good riddance, spam.

February 20, 2008 05:00 PM

One kind of 500 seeing I've been seeing is this:

GET /
(traceback)
TypeError: index() got an unexpected keyword argument 'gclid'

I shrugged these off (stupid bots) but no these are VERY BAD.

After a little Googling, I discovered that Google AdWords adds this parameter to requests to track when people click through your ads. (We do a small amount of ads on Google.) Well every one of these that we've been paying for (!) have been getting 500 errors instead of the main page because of this added parameter! I have tg.strict_parameters set.

If you run a site and are using AdWords, make sure you aren't getting these errors. I would never have known if I hadn't turned on error reporting (see here).

February 18, 2008 10:37 AM