Planet Turbogears

Plumbling Life's Depths: Best pattern for "captive" servers in testing?

Tue, 09 Feb 2010 07:21:13 +0000

So I'm wondering what other people do in this scenario:

You have an executable that provides a (networked) service
You want to test against the executable's service
You want to be able to start/stop the executable during the tests
You want to be able to run multiple executables between test suites (i.e. with different configs for each suite of tests)

I'm mostly been using subprocess, and this works; but I always seem to wind up with situations where the child process is hung, or winds up running past the execution of the tearDown() for the suite. Any given failure can almost always be fixed, but it always feels messy and low-level, and it just feels like there should be something better.

So, how do you run captive servers?

Plumbling Life's Depths: ab is a cruel mistress

Sun, 07 Feb 2010 16:18:44 +0000

Got started on testing performance tonight for ChatTrack. Wow. Nowhere near where it needs to be, not even on basic page rendering... seeing something like 14.5rps sustained on 1000 requests with 100 concurrency, and there's significant numbers of requests taking more than 4s to reply (eep!). And that's on the laptop, the poor little virtual server is just going to keel over.

Good news is, I haven't optimized much yet, just tested, so likely there's just something "oopsie"'d in there. I can see at least 2 queries for the user record on each request... that's not right. There also appears to be some unnecessary queries for the survey options. After that, it's off to the wonderful world of caching! And at some point I need to test the streaming-scaling...

I did find some time to work on the actual presentation this evening, but the next two days I'm off on paying customer's work, so no work on the PyCon stuff for a bit. Still, presentation is shaping up well.

Plumbling Life's Depths: Specs can really suck the life out of you...

Wed, 03 Feb 2010 14:15:25 +0000

Just spent hours tracking down the "DOCSIS TLV" config-file format and implementing it. This is one of those things where the project spec says "do what that spec says to do", pointing you at a deprecated version of the big spec, then the updated version of that spec says "do what this other spec says to do", but that spec is then defined in terms of 2 other specs... an enormous amount of reading to specify something that can be described in a tiny amount of code.

The key thing that gets left out AFAICS is that the spec is recursive; so for option 43.1, 43.2, or 43.3.4.5.2.3.4.5.6, for example, you do all of the encoding steps as though you were doing a top-level encoding for the various named (numbered) sub-options at each level, then you break the resulting (potentially very long) value into 255-byte chunks of top-level encodings. For instance, our last example there with a value of 'hello' looks like:

[43,21,3,19,4,17,5,15,2,13,3,11,4,9,5,7,6,5,104,101,108,108,111]

Once you actually get all that, it's one of those relatively simple specs that likely should just be wrapped up in a standard library... but that would be hundreds of times more work than putting together something that can do what you need for this tiny project.

There are likely stupid little corner cases, such as whether to do splitting of long option-values only at the top-level, or whether you are intended to also split them in the recursive elements if an individual sub-element is > 255 chars. There's also some checksums and the like, basically just some sha1 hashes.

The fact that you need lookup tables for all non-trivial interpretation makes the whole thing look way too messy to bother with beyond the needs of your clients of the day. In the example above, unless you know that 43.3 has internal structure you can't parse the result meaningfully. It might just as easily be a single long string. The T (type) in TLV is just a single-byte-integer reference into the current table space, not a direct data-type value. Those tables are vendor, and potentially even project, specific, so you then need to create a mechanism for users to define the tables... ick. Makes one appreciate context-free formats all the more.

Plumbling Life's Depths: Quicky testing TFTP Client (and server) with tftpy

Tue, 02 Feb 2010 22:59:32 +0000

To run a server (easy_install tftpy first):

from tftpy.TftpServer import TftpServer
TftpServer( 'tftp' ).listen( 'localhost', 8069 )

Which serves up the files in the 'tftp' directory on port 8069 (69 is the standard tftp port). To download an existing file from the server:

from tftpy.TftpClient import TftpClient
TftpClient( 'localhost',8069).download( 'exists', 'downloaded.txt' )

which should download the file "exists" from the server to the file 'downloaded.txt' (as long as the file exists).

Plumbling Life's Depths: Two Weeks to PyCon Dress Rehearsal @ PyGTA

Tue, 02 Feb 2010 18:32:22 +0000

Mark your calendars. Mark your icals. Mark your minds. 3 PyCon presenters will present at the regular Greater Toronto Area Python User's Group (PyGTA). Real-time feedback tools, a chance to chat with the presenters and give feedback after the talks, and a rollicking good time for all. 7pm -> ~11pm Tuesday the 16th. Be there, revel in being square!

Plumbling Life's Depths: Linux is kinda fun...

Mon, 01 Feb 2010 11:58:40 +0000

You're on a Linux box. You want to set up an unprivileged user (yourself, maybe) to be able to run on port 80... how do you do it? There's actually quite a few ways, some that use permissions in security-focused versions of linux, some that have root run the process then shed privileges (the traditional way). There's also a neat little hack where you just tell the firewall to forward port 80 to port 8080 and trust that your users will only ever run good and happy programs on 8080 .

I've mucked about with FreeBSD firewalls before, but this was the first time I'd tried playing with Linux iptables... it's kinda neat, there's a live command "iptables" that lets you modify the current configuration, test, and make sure you haven't screwed up. When you have what you want, you can save out the results with iptables-save and install the resulting saved configuration into your init script (which uses iptables-restore on the result).

Have the server mostly set up for the little feedback system now. Still need to get it a domain name, probably just set it up as a vrplumber.com sub-domain for now and put buying a real one and a certificate on the todo list. For now, however, should get back to trying to make money.

Plumbling Life's Depths: Didn't even think to enable epoll in Twisted+TG

Sun, 31 Jan 2010 14:19:03 +0000

Enabling the epoll reactor in Twisted is a two-line change:

from twisted.internet import epollreactor
epollreactor.install()
from twisted.internet import reactor

From there on the app is unchanged. I didn't have time to really test that today (paying work and all), but I don't really expect to see much of a change. EPoll is about scaling up, and my current speed annoyances are all about single-query latencies (I see most queries taking 300-430ms).

Anyway, I've got more paying work to get done this week, so the ChatTrack stuff is going to have to wait a bit. It needs to be done for PyGTA's PyCon Dress Rehearsal on the 19th at the latest, but I can likely get another 2-3 days between then and now, maybe as much as a week if I finish these projects with few speed-bumps.

Plumbling Life's Depths: Testing and Compressing and Timing, Oh My!

Sat, 30 Jan 2010 18:48:01 +0000

Finally got a URL set up for the little chatting application. And actually sat down and played with it for a while with the 3 browsers I have handy (Firefox 3.5, Chromium and IE . Good news is that (after having to fix a known jquery-on-ie bug), it all seems to pretty much work. Bad news is that there's lots of little fit-and-finish bugs, UI elements that just don't quite work. Most are little things that can each be easily coded around, but there's likely a solid day's work in there with all of them.

One that's very annoying is that Chromium seems to get "tired" after a while if there are too many incoming messages. I gather something in JQuery gets borked, as the symptom is basically that "K" is not defined (ah, the joys of compressed js). Speaking of which, I ran the JS and CSS through YUI compressor. I've done that before, but tonight I realized, why not just gzip compress the minimized results? Saves rather a lot of transfer; almost 500KB down to 71KB for the js and from 34KB down to 5KB for the CSS. Seems to work perfectly fine in the 3 big browsers. Of course, once I finished and actually tested, it doesn't really make much of a difference to the user, as everything is in the cache after the first page-hit.

I'm really not satisfied with the main query-speed so far. Seeing ~400ms for the page load. Will have to track that down if I'm going to have even a dozen simultaneous users, let alone hundreds. The streaming seems perfectly fine, it's just the initial page-load that's taking far longer than I'd like.

I also spent quite a while trying to set up Ajax login... then finally decided that I'll have to just go with non-Ajax login for now; the repoze.who.friendlyform stuff just isn't set up to do async login and fixing that isn't something I want to spend time on right now.

Plumbling Life's Depths: PyCon 2010 Schedule is Up

Thu, 28 Jan 2010 15:26:51 +0000

How did I miss the announcement? Looks like I'm the last talk on Saturday afternoon. Greg's talking at the same time on Basie, Ned on Testing. Hrm, maybe don't need to worry about my server melting. Anyway, the schedule is up, now we can all start planning which sessions we have to give up to see the ones we want to see.

Enough with the procrastinating... I will keep writing.

Plumbling Life's Depths: Ported back to Twisted (but with TurboGears)

Thu, 28 Jan 2010 07:18:45 +0000

For those who are wondering about the Twisted + TurboGears posts. Yes, I did move ChatTrack back to Twisted, and I did go with an embedded WSGI-hosted TurboGears for the "webish" stuff.

As some of you may recall, I started off in December writing my little tool for real-time feedback using Athena... and wound up running smack into the hiccup that is the Nevow/Web2/Web issues in Twisted. Since it was a personal project, I decided to play around and see what Tornado (and then CouchDB) would have to offer.

I got the whole project pretty-much finished in Tornado + CouchDB. It worked perfectly fine, it was as fast as I could ask... but I'd written far too much of the stack myself, and there really didn't seem to be any interest from other people in working on e.g. the CouchDB wrapper, or the generic channel-subscription mechanism, with me. Tossing "hey, here's something you could use" posts out and finding that no-one is interested is very clarifying... it tells you "hey, this isn't going to be fun to work on in the future". If no-one else is going to contribute (or even use it), that's a lot of code you have to maintain going forward.

What's more, I ran smack into the query mechanisms of CouchDB... they're fine if you're doing "document" stuff, but they get painful when you want to do heavily-joined queries and the like. I got to a point where I was looking at re-implementing query languages or having to create an async postgresql client (when what I really wanted was SQLAlchemy). Those are not things I want to be maintaining, particularly as I don't see it as likely I'll be using them in another project.

Which is where running TurboGears under Twisted came in. The Twisted code is really good at low-level multi-protocol "networking done right". TurboGears is good for writing large, complex database (SQLAlchemy) driven web-sites rapidly. What's more, I'm comfortable enough in TurboGears (I use it for many of my clients) to write solid, tested, code without even having to pause to think about it.

With this architecture, what's more, the Twisted and TurboGears "resources" are largely separable. The Twisted channel protocol is in a separate project that doesn't have anything to do with the TurboGears site other than that it receives "send" commands from it. I could readily put that into another process and have the TG site run under Apache and use RPC to do the sending. That is, the web-site-development environment isn't tied to the low-level networking environment.

The Twisted implementation of the channel server is a trivial set of changes from the Tornado one (just a different sub-class of the same basic code that handles the different request-argument-parsing and callback-registration mechanism). The real difference is that now I can hook up that implementation to X other protocols, that is, it is now sitting in an environment where people have written code to interact with dozens of other protocols... and I can use that code, instead of having to write my own. Ironically, I likely won't have time to do that hooking up before PyCon, but whatever.

Tornado is a perfectly fine web framework. But compared to TurboGears for writing web applications it's early in its game, and I can get patches into TurboGears reasonably easily (I'm mostly working on the docs, but I work with TG enough to hack on the internals). I likely could have used WSGI to host TurboGears within Tornado... but honestly the callback-based networking in Tornado is nowhere near as elegant as the Twisted Deferred pattern (just my opinion), so if I'm going to use a WSGI abstraction anyway, I'd rather go with Twisted to write (and maintain) the trivial protocol.

Plumbling Life's Depths: Getting close on ChatTrack

Thu, 28 Jan 2010 06:56:16 +0000

I'm almost to the point where I want to get a few friends playing with the tool. I've still got Search, User "Blogs", RSS Feeds, and Highest-rated views to do, as well as lots of clean-ups. Also need to get the domain-name set up for the poor little server.

I expect that the VM will just melt under even 100 simultaneous users (which is why I'm not going to try running on Vex's machines, as I get that access for free). I may wind up having to pay for a real server for a few weeks over PyCon. Want to get an SSL cert as well, though running SSL on an overloaded box probably isn't a great idea. That needs to wait on the domain name anyway, I suppose. I'm tempted to just go with "http://apps.vrplumber.com/ct" since I'd like to keep the server around for other projects where I don't want to impact Vex's other users.

Also need to get tested on at least an iPhone (didn't get that done at PyGTA) and maybe an Android... oh, and I should test on IE 7/8 too . Hrm, probably need to start setting priorities so I can get it properly stress-tested and deployed before PyGTA this month.

For tonight, however, working on the presentation.

Plumbling Life's Depths: Making your Twisted resource(s) a url sub-tree of your WSGI resource...

Wed, 27 Jan 2010 21:09:48 +0000

For those paddling about in the Twisted + WSGI-hosted app world (here I'm playing with TurboGears), one thing you may wind up wanting to do is to have your WSGI application be the "default" URL-tree, with only certain sub-trees handled by Twisted. To be more concrete, say you want your URLs to look like this:

/* -> WSGI URL handling for all urls except
/channels -> Twisted long-polling protocol (or Soap, or XMLRPC, or whatever

Normally (i.e. with normal twisted.web resources), you'd do either a putChild or write a getChild handler. putChild won't work on a WSGI resource, however, nor does it provide a getChild that you could delegate to... so you'll have to do some hacking here.

You create a root resource with a getChild, but you warp the request's paths so that your root controller doesn't "consume" any path fragments. You can use putChild() on the root resource to put "channels" in (or any other root name, such as "RPC"), and only undefined children will be delegated to the WSGI.

class Root( Resource ):
    """Root resource that combines the two sites/entry points"""
    WSGI = None
    def getChild( self, child, request ):
        request.prepath.pop()
        request.postpath.insert(0,child)
        return self.WSGI
    def render( self, request ):
        """Delegate to the WSGI resource"""
        return self.WSGI.render( request )

The WSGI child is "promoted" to the root by being delegated to for both the root render and the lookup of all non-statically-defined root url fragments. Enjoy.

Plumbling Life's Depths: Form Handling, is there a better path?

Tue, 26 Jan 2010 16:16:19 +0000

I've now written, worked with and generally explored form handling in web development a great deal more than I've enjoyed. I haven't seen anything that satisfies me.

You can code basic form handling in minutes, but then sharing with anyone else is lost. The intricate messy details of formatting/layout of the GUI widgets, the placement of error text, the integration of internationalization, handling complex widgets, and getting all of the data to the right spot when the time comes to render... they're all simple problems, but together they're the kind of thing that no-one wants to spend their lives doing.

The shared form-handling libraries, however, seem to grow so complex as to make them ridiculously complex to debug when they fall down. Your data doesn't show up in the widget, nor your error messages... well, you'd darn well better feel like spelunking down 20 levels deep into the meta-programming to figure out what went wrong.

Sometimes it will be easy, like "oh, it's trying to take the len of a None because there's no 'name' field on the widget and an error was encountered for that field value". Sometimes it just won't work no matter what you do. And sometimes you'll find yourself thinking it would take 1/10th the time to just write your own bloody validation routine for these 4 fields on two forms... but then you're back into maintaining the tarmac forever.

My big projects have all wound up paving new tarmac. I tend to want complete control over rendering, so I create template functions that render each given control, they take the value, higher-level functions render table-rows with labels, error messages, with a couple of layouts, and then the template decides what rows to render, dispatching the values passed into the widget-renderers.

I normally do custom FormEncode validators, but I often find that using schemas/compound validators seems to make the code very obtuse feeling, so I wind up doing per-field validation with FormEncode/TG and then separate validation for multi-field consistency tests... but now I need to hack TG to re-hook that non-standard validation into the mainline error-reporting/handling machinery, and we still hit issues with TG's validation/error_handler stuff where we'll wind up with un-validated results showing up. Blech.

It feels like there should be something better available, something simple, obvious, easily debugged, readily composable and decomposable, loosely coupled... or maybe it's just that at 4:30am I'm tired of debugging what I was intending to have finished by midnight. It feels like it should be some sort of tool-set of functions that are all explicitly invoked in the order you want them in a function that you write for each controller (you couldn't believe how much time I spent one day tracking down a bug due to an "All" constraint being applied in reversed order)...

def validator( req ):

    login = req.set( 'login', req.re_match(

        req.longer_than(

            req.as_text( req.get( login) ),

            5,

        ),

        re.compile( r'^\w+$', re.I|re.U),

        message = _('Only non-white-space characters allowed' ),

    ))

    password = req.set( 'password',req.equals(

        req.strong_password(

            req.longer_than(

                req.as_text( req.get( 'password' ) ),

                6,

            )

        ),

        req.as_text( req.get( 'confirm' )),

        message = _('Password and confirmation do not match'),

    ))

    if not req.errors:

        # no errors so far, so do more checking...

        # here we see custom code, rather than something

        # composed into a reusable piece...

        other = req.db_lookup( User, User.id, login )

        if other:

            req.set( 'login', req.error(  _('That login is in use')))

The idea being that each little helper function/method of the request is doing some trivial operation. It returns either the converted/validated result or an Invalid object. The method wrapping for the helper functions then short-circuits if there's already an Invalid as the value and returns the original Invalid. You set the converted value for the parameter with the .set() method. If it's an Invalid, it registers as an error, otherwise it's a converted value. Helpers where it makes sense to allow message customization can just let you pass in the messages as arguments. Extra arguments (e.g. limits, or other parameters) can similarly be passed in.

You'd keep the helper functions as simple as you can, something like:

def longer_than( 

    req, value, 

    minimum=1, 

    message=_('Require value greater than %(minimum)s in length'), 

    type_message=_('Unable to determine length of %(typ)s value'),

):

    """Validation function, takes value, checks it, returns checked/converted value or Invalid instance"""

    try:

        length = len( value )

    except (ValueError,TypeError), err:

        return req.error( type_message%dict(

            typ = type(value),

        ))

    else:

        if len < minimum:

            return req.error(message%dict(

                value = value,

                minimum = minimum,

            ))

        return value

The request would look something like:

class Request( object ):

    def get( self, key, default=None, required=False ):

        """Retrieve argument for key"""

    def set( self, key, value=None ):

        """Set a (partially) validated value"""

        if isinstance( value, Invalid ):

            self.errors[key] = value

        self._values[key] = value

    def error( self, message=_('Error')):

        """Create an error message for the given key



        These are sentinel values.  for sanity-checking, would likely

        make them report an error if they are deallocated without being

        set() on the request?

        """

    def rawvalue( self, key, default=None ):

        """Retrieve original, untouched value for key (for user to correct)"""

    def register_helper( cls, name, helper ):

        """Register a validation helper function (and wrap)"""

Of course now that I've written all that, it's not really any different than the dozens of other validation/conversion mechanisms out there. So now it's 1.25 hours later and I'm just going to stop writing and accept that web development is a messy mess of mess and that tomorrow whatever it is that's making FormEncode/TG fail will have just magically disappeared.

Christopher Perkins: Contribution can free you

Mon, 25 Jan 2010 23:34:52 +0000

In the past few months I’ve talked a bit about contribution, sprints, and my own form of freedom of expression: Open Source Software. I think it’s great to find something that works for you, something that you can latch onto and become a part of. I’m not talking about becoming a parasite or a sink on a piece of software, but an actually joining forces with like minds and producing something new, interesting, unique. Sharing your knowledge benefits your own community, friends, family, and yourself. Doing so freely is extremely liberating.

One of my goals for 2010 is to liberate TurboGears from the bonds of our technical-biased development strategy. In 2009 we needed to focus on the technology and build a stable platform with which to build upon. We are there. TG 2.1b1 is released tonight, and it has the required speed, ease of use, and a heck of a lot more docs than last year. We do realize that more documentation is needed and have asked numerous times for help.

The problem with asking for help is that first of all people want to be part of something successful, something well known in their own community. They also want to give back to something they use, or someone who has helped them. This is where I come in. I can offer the descriptions of technical supremacy of the components TurboGears brings together, like SQAlchemy, Mako, Abstract Dispatch, until I’m blue in the face, but if no one tries it on their own, or there is an insufficient killer app to use something, it will not garner the attention, nor the following to be a successful product. I’ve been explaining the technical bits of TG for a while, and have seen modest results.

Deciding to take action is one of the hardest things in life to do. The thing is, once you have made a decision to take action, taking actual action becomes easier. I choose to bring TG to you, the developers, hackers, engineers. I am putting myself out there to help you make web applications more quickly, easily, and without hitting the proverbial wall that so many developers using other frameworks hit.

On Saturday I will visit Dallas Texas, where my good friend Jason Galyon will be hosting my workshop. The goal of the workshop is to work with people interested in TurboGears, to work at their level to bring forth a web application that is applicable to them. Bring a database, we’ll see how it works with the new TG admin. Bring nothing and walk home with a wiki. Bring an existing app and spruce it up with a widget or two. Providing a self-driven workshop is the goal. Company Dallas is the location, we are starting around 8:30am, cost: $0. 8 people are already signed up.

On Sunday you have the opportunity to give-back by sprinting. We’ll start at 8:30 again, same place as the workshop. The goal of the sprint is to breath some life into c5t a cms I started with a few others to bridge mongodb, mako, and TurboGears into a blazing-fast, useful way of storing documents. We need: Someone to finish up the calendaring, a search, someone artistic to create an eye-pleasing theme, and lots and lots of testing. I plan to work entirely on testing, you guys can do the fun stuff. We could also use someone to help start some documentation. I’m hoping to release a beta of c5t before Pycon.

I plan on doing the first half of my workshop at Pycon. The difference is that here it will cost between $100 and $150 to take the tutorial depending on the total number of tutorials you are taking, but it is well worth the money if you need to get up and going with a TurboGears app. The tutorial cost is free for me, as I have waived my honoraria. My tutorial is actually in danger of being axed because I need 10 members to sign up and currently have 6. Please register today if you have not already done so!

Also at pycon, I will be participating in a discussion panel for form generation hosted by Brandon Rhodes. I will be expressing my knowledge of Sprox, my own personal foray into the arena. Other experts such as Jacob Kaplan Moss and Chris McDonnough will sit in on the panel as well.

Pycon is such an awesome conference because they build a 4-day sprint into every event. What’s even more awesome is the number of people that show up and contribute. You can bet i’ll be there all 4 days.

A week after I return to Denver, I will leave for Montreal for my first visit outside of the country. It’s amazing I have made it this long without leaving my beloved States. Whatfor would I leave you ask? A developer conference called Confoo.ca. Here I will be giving my workshop again, this time it will be a full-day workshop with lunch provided. The cost is $300, which includes lunch. I waived my honoraria to bring the cost down by $300.

At Confoo.ca I will also give a general talk on TurboGears, to spread the good word about how we have evolved over our lifetime, and where we fit into the future of WSGI and the greater web ecosphere.

Again, I am working to organize a 1 day sprint in Montreal following the conference. Stay tuned for details.

It’s going to be a busy few months. I’m looking forward to it professionally, but it’s going to definitely be a challenge to be away from my family for all that time. What helps me get through that is knowing that people out there benefit from my work and I help make their development lives better. So, if you see me in the hall say hello and tell me what you are up to, or drop me an email if something I contributed has helped you out. Better yet, show up to a sprint and give back and see how your contributions can make a difference while also liberating you.

Plumbling Life's Depths: TurboGears 2.1b1

Mon, 25 Jan 2010 22:49:53 +0000

Percious just released the first 2.1 beta. I've been using tip for my little project that embeds TG in Twisted, so I'm already there. If you're planning on moving your codebases to 2.1 from 2.0, now is the time to test your apps with it so you can report any bugs before the final release.

Plumbling Life's Depths: Haven't done much poetry lately...

Fri, 22 Jan 2010 20:54:34 +0000

Colleague (Mike Lin) has (re)launched his almost-rhyme-finding play site (that is, it finds almost rhymes, it's not trying to almost do rhyme finding). Don't know that I'll have enough time to get back to poetry soon (Planet Python sighs a sigh of relief), but fun to play with.

Plumbling Life's Depths: 64-bit Ubuntu Web Devs, don't let Firebug update to 1.5

Thu, 21 Jan 2010 03:21:16 +0000

There's a bug that makes Firebug crash Firefox 3.5.x on 64-bit Ubuntu. There's a fix to be released from Mozilla, but for right now Firebug 1.5 is not a good idea... of course, you're likely reading this in Firefox, so likely too late. You can revert to Firebug 1.4.x by going here. Details on the Ubuntu/Launchpad bug report.

Plumbling Life's Depths: Things I'd like to play with (given enough time)

Wed, 20 Jan 2010 07:17:19 +0000

I'm finding myself spending too much time (again) on this little "chat track" project... which leads me to think about other things I could be playing with:

3D renderers and/or game engines
Haskel
Mobile games
WebGL
XMPP (and/or Wave)
PyPy
Educational tools

I still haven't finished the remaining core piece of the project, but I did get the user's contribution page working and began work on being able to page through previous discussions. I'm really beginning to regret the choice of CouchDB for the back-end. It's working perfectly well, but I have literally dozens of pieces of code hanging around for doing SQL-based paging... none of which are applicable to CouchDB.

Maybe I can code up the paging in Haskel... or make the chatting interface 3D... or send XMPP messages . Or maybe I should concentrating on getting the tool working first .

Michael Pedersen: Announcing Availability of tgext.menu, v0.2

Tue, 19 Jan 2010 07:00:49 +0000

Long story cut short: I've not been moving on Bit Vizier as fast as I wanted for a number of reasons. One of them is the pain that is managing the navigation bar from a development perspective. If I want to move things around, I have to update my controllers and my navigation bar template. If I need to rename something, I have to do the same. If I want to add a feature, I have to do the same. The navigation bar, the side bar, and any other menus are all easily determined from the code. Why do I have to maintain a separate template?

With tgext.menu, I don't have to do so anymore. Instead of having a separate template, I simply have to provide a list of the menu entries, and the rest just happens. It makes dealing with things much easier.

tgext.menu is available on the Python Package Index (for use with easy_install), and also on bitbucket.org (to get the source code).

I'll apologize now for the shortness of this post. I'm still recovering from ear surgery, so mildly unavailable, but didn't want to hold this back any longer.

Luke Macken: nose 0.11

Mon, 18 Jan 2010 22:58:00 +0000

I know nose 0.11 is old news, but I've only recently discovered it's new multiprocess module.


lmacken@tomservo ~/bodhi $ nosetests
................................................................................................
----------------------------------------------------------------------
Ran 96 tests in 725.111s

OK

lmacken@tomservo ~/bodhi $ nosetests --processes=50
................................................................................................
----------------------------------------------------------------------
Ran 96 tests in 10.915s

OK

Nose 0.11 is already in rawhide, and will soon be in updates-testing.

Note to self (and others): Buy the nose developers beer at PyCon next month

Christopher Perkins: Circuit Resume

Fri, 15 Jan 2010 16:25:01 +0000

Hey guys,

So, I thought I’d continue this meme a little further. I got some great comments the last time. This resume idea is inspired by my friend who is staying with me while he finds a job here in Colorado. Ben’s an electrical engineer with 6 years experience and someone I trust with my life on a regular basis. Anyway, here is my circuit resume, “feedback” appreciated.

The OmniGraffle source. Oh, and here’s my “real” resume.

Plumbling Life's Depths: Tornado @ PyGTA this month, PyCon Dress Rehearsal next

Thu, 14 Jan 2010 03:19:52 +0000

This upcoming Tuesday we're going to look at, and hopefully play with, Tornado at PyGTA. Next month we've got the big PyCon Dress Rehearsal with 3 speakers.

Michael Pedersen: Choose The Right Data Structure

Wed, 13 Jan 2010 10:45:20 +0000

This one is actually almost embarrassing to post. I've been working on an extension for TurboGears called tgext.menu. I'll not get into long explanations of why, or what I hope to accomplish with it (I'll leave that for when the first release happens, hopefully early next week).

I will say this, though: If you don't choose the right data structure, right off the bat, you are condemning yourself to a hellish experience coding.

In my case, I had a list that looked like this:

ExitApp
Foo Spot
Foo Spot || Bar
Foo Spot || Baz
Foo Spot || Foo
Foo Spot || Sub || Bar
Foo Spot || Sub || Baz
Foo Spot || Sub || Foo
Sub || Sub 1
Sub || Sub 1 || Nested 1
TestHome

I wanted that to become a nested unordered list in HTML, looking like this:

ExitApp
Foo Spot
- Bar
- Baz
- Foo
- Sub
  - Bar
  - Baz
  - Foo
Sub
- Sub 1
  - Nested 1
TestHome

Looks really simple, doesn't it? All you have to do is loop over the list, split each item on the || symbols, generate openings, put your entries, and generate closings. Except you have to compare the current base of the tree with the next base of the tree and the previous base of the tree.

I can't count the number of methods I tried, and every one of them had a small, subtle, error that threw off all the output. Sometimes elements would not be closed properly. Sometimes they would not be opened properly. I tried iterating, I tried recursion. All of them failed with errors that I could not find a generic way to correct.

Saying this was frustrating is a major understatement. Last night, I finally changed my data structure that represented the output: Instead of a list, I went with a tree. Using a tree, I was able to write up (in less than a dozen lines) a simple recursive loop that wrote out the entire tree, with appropriate whitespace, CSS class tags, the work.

In one hour, I solved problems that had been plaguing me for two weeks.

Lesson learned: If I'm spending a lot of time trying to solve a specific problem, and my algorithms are always producing errors, I need to look at the underlying data structures. I'll probably save myself a lot of time that way.

Plumbling Life's Depths: IE6 Can Bite My Shiny

Mon, 11 Jan 2010 07:09:47 +0000

Figured I should test on IE7 and 8, and as usual when I go to test on IE, I found myself thinking "really?" about the bugs that showed up... missing "push" and "forEach" on arrays? Crashing because of a check for whether an attribute is defined? Requiring a different event-binding to capture <return>? Sigh. Whatever. There's still a dumb bug where IE8 loses focus if you chat too fast (looks like a timing issue somewhere, bah).

I considered downloading the MS IE6 compatibility VM image... but then I realized it's 700+MB for something that's time-bombed to stop working in April... there's something really cathartic in saying to yourself "I refuse to support IE6 in this project." I could just fire up my old Win2K image and use that to test, but honestly, a big fat "Get Real, Update your Browser, You're at a Tech Conference" message feels better .

Also tested on the BlackBerry browser, I'm going to put that on the "don't bother" list for now. I don't have a debugger there, and while the errors look pretty shallow, I just don't have that much motivation. iPhone and Safari will likely work (being pretty close to Chromium). I'll see about getting a friend to test those, maybe at PyGTA this month.

I did a bit of refactoring of the torchannels stuff, making the entry points a little more meaningfully named, adding documentation, using a JSON parser/linearizer instead of JQuery's eval-based approach, and generally trying to wrap up the "plumbing" part so I can get to the tool itself... hopefully I really am there now and can start coding the fun bits.

Michael Pedersen: Are You So Arrogant ...

Wed, 06 Jan 2010 09:08:14 +0000

I was insulted last night. Now, there's nothing new about that, nor even anything particularly blog worthy. After all, we all get insulted in some fashion or another on a regular basis. We shrug it off, and we move on. This time, though, I've heard this particular insult too often. I want to address this, in particular: I was called arrogant.

Why? Because I answered a question.

The person who called me arrogant had asked about a particular piece of a program I've written, wondering what would happen if the user did a specific action. I replied that them doing so would be irrelevant, since it could do no harm. The reply came back, and amounted to "Are you sure?"

Suddenly, I wasn't sure. I went back to the code, and re-read it. I checked the actual behavior while running. I verified the contents of the database. Now, I was sure again: It simply didn't matter if the user did this action. It produced an off-by-one sum error for something which already has a much larger margin of error (counting cards that got sent and receieved from around the world). Being off by one in that situation causes no concern whatsoever.

Despite this, I then checked the database to see who was doing it, and found a whopping four people out of fifty three had done it. I spelled all of this out for the original questioner. Since the original questioner is not a technical person, I made sure to use terms that would make sense to her.

And then she asked the question that started with the title of this post: "Are you so arrogant ... " (yes, I've left the rest out, as I'm trying very hard not to make this an attack against her, and quoting her excessively would do just that).

I'm arrogant because I answered her question in terms that I knew she would understand.

After that, she mentioned that, in her experience, computer people (programmers, techs, administrators, etc) come across as if dealing with idiot children when dealing with non-computer people. So, instead of just being arrogant, I'm arrogant and condescending.

This is where the great disconnect occurs, and is the main point of this post: As a developer, I have to know things about the computer that many people do not. Doing web development, I have to know HTML, CSS, and JavaScript. If I'm doing anything with dynamic data, I also have to know whatever programming language I'm using on the server, and some database dialect so I can store, retrieve, and manipulate that data. All of that is the bare minimum required to do that work.

In order to do it well, I need to know some bits about the underlying operating system, the web server software being used, basics about HTTP (and possibly https), and the quirks of the various browsers in use around the world (IE6/7/8, Firefox, Opera, Chrome). Possibly more importantly, I have to have the knowledge that I can make no assumptions about anything. I can't count the number of times I've said "That's not possible to happen!" only to find out later that not only was it possible, it happened due to a bug in my code, and that bug came from an assumption.

In other words, I have to have a huge amount of specific, detailed, technical knowledge rolling around in my head just to do the basics of my job well. When I get somebody who is a self-proclaimed non-technical person asking me for answers that require at least some of that knowledge be handed out, I have to mentally change gears and use terms that I know will work to explain that knowledge. Of course I sound condescending, simply because I try very hard to make no assumptions.

The part that I find most interesting to me, in all of this, is one thing that my experience has taught me: The more understanding people have of their own systems, the less likely they are to call me arrogant when I explain an answer to their question.

I care a great deal about making sure I'm understood. There are times when my doing so comes across as rude, arrogant, or condescending. I do not set out to do so, but there are times when I cannot think of a way to avoid it. I actually don't like doing so. To have the insult lobbed in my direction on top of that is more than I can accept. I had to say something.

My final thought on all of this comes from her final remark: "Not everybody is a computer genius." That's correct, not everybody is. Not even me. If you want to find out about a few of them, allow me to give you some names to look up: Alan Turing, Steve Wozniak, Linus Torvalds, Alan Cox, Theo de Raadt, Brian Kernighan, Dennis Ritchie, Larry Wall, and Guido von Rossum. I am not, and never will be, in their league.

Plumbling Life's Depths: Simple Channels for Tornado (rev 0.0.1)

Tue, 05 Jan 2010 07:49:34 +0000

I pushed some of the refactored code from my little test project out today. Can't say I'm in love with the code, nor with how it was written (I'm afraid I hacked it from spike test right up to final implementation). Anyway, the result is a poll/long-poll/streaming channel-server for Tornado + JQuery. You use it something like this:

class ChannelHandler( longpoll.LongPollMixin, basehandler.BaseHandler ):

    def can_access( self, sub ):

        '''Override to provide authenication for channels'''

        if not(super(ChannelHandler,self).can_access(sub)):

            return False

        if my_logic_says_this_user_cannot_see_this_channel( sub ):

            return False

        return True



class Application(tornado.web.Application):

    def _init_(self):

        handlers = [

            (r"^/channels$", ChannelHandler),

            (r"^/([-a-zA-Z]+)$",YourInitialUploadHandler),

        ]

        settings = dict(

            # Note: you must copy longpoll.js to the static file directory!

            static_path=os.path.join(os.path.dirname(_file_), "static"),

        )

        tornado.web.Application._init_(self, handlers, **settings)

With the Javascript looking like this:

    var subscriptions = $.longPollSubscriptions( {

        url: '/channels',

        stream: true

    } );

    subscriptions.subscribe( 'some-channel', function( msg ) { } );

You write standard Handlers to handle initial state transfer, posting messages to the channel, etceteras, so you use your application's normal authorization scheme for that. To write to a channel:

torchannels.channels.Channels.get_channel( channel_name ).send( { } )

Anyway, the code is extremely alpha, but it's up on LaunchPad if you want to play with it.

bzr branch lp:torchannels

Have fun.

Plumbling Life's Depths: Are we having fun yet?

Mon, 04 Jan 2010 11:56:54 +0000

Okay, there's fun, and then there's fun. I think I've passed the point of diminishing entertainment playing with the little tool/site for my PyCon talk... somewhere in there I implemented a publish-subscribe system that looks a lot like a simpler version of Bayeux (with fewer transports, mind you, just poll and long-poll). While learning Tornado and CouchDB was fun, I really need to just get the actual UI written (instead of the plumbing) and then get some paying work in the door.

I don't know that I'd use this technology stack in a real-world application; it's fine, but pulling an off-the-shelf comet server and client would likely be more practical for the comet stuff... even if the implementation isn't particularly accessible having someone professionally maintaining it is way more important. That said, the implementation is bloody simple, so maybe for some simple task some day...

As for CouchDB... I'm not really in love. Maybe it's just the years of PostgreSQL experience and the familiarity of SQLAlchemy's ORM but I find myself seeing Couch as more of a low-level data-store than a database... joins are just so convenient. I'm afraid I'm relegating it to the same position as ZODB in my toolbox, that is, something that might show up some day as being useful for a particular project, but not something I intend to work with enough to mold it to my hands. Maybe as I finish the project I'll find the zen there, but not yet.

Maybe just take a few days off of the project to get my joy on again.

Plumbling Life's Depths: More CouchDB Fun...

Mon, 04 Jan 2010 03:03:21 +0000

Continuing to work on my little project. Mostly messing about with UI and getting CouchDB to do what I want it to do. The query mechanism is a little weird when you're used to SQL(Alchemy) and complex referential schemas. For instance, say you want to be able to total votes across M objects with N users able to vote up/down (once) for any given object (and you want to be able to restrict the search to a single query).

{ 'type': 'vote', 'survey':'surveyname', 'target': targetid, 'vote': -1 }

The map function is pretty straightforward, you extract the keys you want to query against with the vote-value like so:

function (doc) {

    if (doc.type == 'vote') {

        emit( [doc.survey,doc.target], doc.vote );

    }

}

The reduce function is where it gets interesting. CouchDB will potentially rereduce your partially reduced values, so you need to be able to handle both direct summing/counting and summing partial values in your reduce function:

function (keys,values,rereduce) {

    var sum = 0;

    var count = 0;

    values.forEach( function( v ) {

        if (rereduce) {

            sum = sum + v.sum;

            count = count + v.count;

        } else {

            sum = sum + v;

            count = count + 1;

        }

    } );

    return {sum:sum,count:count};

}

Works reasonably well, but I'm still finding myself wanting to do "joins" so that I can retrieve the vote counts when I retrieve the "target" objects. Haven't found the equivalent of that for CouchDB yet.

Christopher Perkins: New Resume

Sun, 03 Jan 2010 04:38:51 +0000

Hey guys, I’ve been busy for a bit, but I made up this new resume for the TG tour, looking for feedback if you’d like to comment.

cheers.

-chris

Plumbling Life's Depths: PyCon Dress Rehearsal Confirmed for February PyGTA

Tue, 29 Dec 2009 20:37:32 +0000

All three local PyCon presenters are booked in for the PyGTA on February 16th. Should be pretty darn cool.