fidzu : Turbogears

My last post I said that I had to remove my internet query tools due to some bugs that were a concern. Some of the code was hard to maintain and probably had holes and I had noticed that it looped at times.

I'm happy to say that I have restored some of those tools now, still located at http://enc.com.au/itools

This code is completely re-written in Python using the TurboGears toolkit which means it is a lot cleaner in how it works and how it looks. Some of the lookup tables use a database rather than an array for ease of updating and querying. The downside is the backends will take time. It currently only does nslookup queries and whois only works for IPv4 addresses. The domain name queries will be a while off as these are the most complicated to handle. To give you an idea, all IPv4 and IPv6 address information comes from 5 sources with two formats while domain names come from over 200 sources with about 40 formats. This means the information from Regional Internet Registries will be done first.

15 May 2013 1:14pm GMT

You might of looked at Ralph Bean's tutorial on graphs and thought, that's nice but I'd like something different. The great thing about ToscaWidgets using the jqPlot library is that pretty much anything you can do in jqPlot, you can do in ToscaWidgets and by extension in TurboGears. You want different graph types? jqPlot has heaps!

My little application needed a Pie Chart to display the overall status of attributes. There are 6 states an attribute can be in: Up, Alert, Down, Testing, Admin Down and Unknown. The Pie Chart would show them all with some sort of colour representing the status. For this example I've used hard-coded data but you would normally extract it from the database using a TurboGears model and possibly some bucket sorting code.

I've divided my code into widget specific, which is found in myapp/widgets/attribute.py and the controller code found unsurprisingly at myapp/controllers/attribute.py Also note that some versions of ToscaWidgets have a bug which means any jqPlot widget won't work, version 2.0.4 has the fix for issue 80 that explains briefly the bug.

The widget code looks like this:

from tw2.jqplugins.jqplot import JQPlotWidget
from tw2.jqplugins.jqplot.base import pieRenderer_js
import tw2.core as twc
 
class AttributeStatusPie(JQPlotWidget):
    """
    Pie Chart of the Attributes' Status """
    id = 'attribute-status-pie'
    resources = JQPlotWidget.resources + [
            pieRenderer_js,
            ]
 
    options = {
            'seriesColors': [ "#468847", "#F89406", "#B94A48", "#999999", "#3887AD", "#222222"], 
            'seriesDefaults' : {
                'renderer': twc.js_symbol('$.jqplot.PieRenderer'),
                },
            'legend': {
                'show': True, 
                'location': 'e',
                },
            }

Some important things to note are:

• resources are the way of pulling in the javascript includes that actually do the work, generally if you have something like a renderer using js_symbol further on, it needs to be listed in the resources too.
• seriesColors is how you make a specific data item a specific colour, or perhaps change the range of colours. It's not required if you use the default set, which is defined in the jqPlot options.
• The renderer tells jqPlot what sort of graph we want, the line above says we want pie

Next the controller needs to be defined:

from myapp.widgets.attribute import AttributeStatusPie
 
@expose('myapp.templates.widget')
    def statuspie(self):
        data = [[
                ['Up', 20], ['Alert', 7], ['Down', 12], ['Admin Down', 3], ['Testing', 1], ['Unknown', 4],
                ]]
        pie = AttributeStatusPie(data=data)
        return dict(w=pie)

And that is about it, we now have a controller path attributes/statuspie which shows us the pie chart.
My template is basically a bare template with a ${w.display | n} in it to just show the widget for testing.

Pie Chart in Turbogears

12 Apr 2013 11:40pm GMT

Filmed these last week weekend. You can hear my year-old daughter talking to them.

13 Mar 2013 12:08pm GMT

For those who haven't been keeping up with all of the awesome code Ralph Bean has been churning out lately, be sure to checkout fedmsg.com. Hop on #fedora-fedmsg on Freenode or load up busmon to see it in action. Not all of the Fedora Infrastructure services currently fire off fedmsgs, but we're getting very close.

This technology is built on top of Moksha, which I created many years ago while writing the first version of the fedoracommunity app. It's come a long way since then, and now can speak ØMQ over WebSockets, as well as AMQP and STOMP over Orbited. Now the time has finally come to bring Moksha to the desktop!

Introducing fedmsg-notify

fedmsg-notify lets you get realtime desktop notifications of activity within the Fedora community. It allows you to tap into the firehose of contributions as they happen and filter them to your liking. It works with any window manager that supports the notification-spec, however I've only seen the gravatars show up using GNOME.

For GNOME Shell users, you can [optionally] install gnome-shell-extension-fedmsg, and then enable it with the gnome-tweak-tool or by running `gnome-shell-extension-tool -e fedmsg@lmacken-redhat.com` (and then hit alt+f2 and type 'r' to reload the shell). You will then be graced with the presence of The Bus:

For those who aren't running GNOME shell, you can simply yum install fedmsg-notify, and then run fedmsg-notify-config, or launch it from your Settings menu. Due to a dependency on Twisted's gtk3reactor, fedmsg-notify is currently only available on Fedora 18 and newer.

The first tab shows you all services that are currently hooked into fedmsg. As we add new ones, the gui will automatically display them. These services are defined in the fedmsg_meta_fedora_infrastructure package.

The Advanced tab lets you further customize what messages you want to see. The "Bugs that you have encountered" option will display all messages that reference any Bugzilla numbers for crashes that you have hit locally with ABRT. The other filters involve querying your local yum database or the PackageDB.

Under the hood

The fedmsg-notify-daemon itself is fairly minimal (see daemon.py). At it's core, it's just a Twisted reactor that consumes ØMQ messages. Moksha does all of the heavy lifting behind the scenes, so all we really have to do is specify a topic to subscribe to and define a consume method that gets called with each message. This is essentially just a basic Moksha Consumer with some fedmsg + DBus glue.

class FedmsgNotifyService(dbus.service.Object, fedmsg.consumers.FedmsgConsumer):
    topic = 'org.fedoraproject.*'

    def consume(self, msg): …

The daemon will automatically startup upon login, or will get activated by DBus when enabled via the GUI. When a message arrives, it filters it accordingly, downloads & caches the icons, [optionally] relays the message over DBus, and then displays the notification on your desktop.

The API for writing custom filters is dead simple (see filters.py). Here is an example of one:

class MyPackageFilter(Filter):
    """ Matches messages regarding packages that a given user has ACLs on """
    __description__ = 'Packages that these users maintain'
    __user_entry__ = 'Usernames'

    def __init__(self, settings):
        self.usernames = settings.replace(',', ' ').split()
        self.packages = set()
        reactor.callInThread(self._query_pkgdb)

    def _query_pkgdb(self):
        for username in self.usernames:
            log.info("Querying the PackageDB for %s's packages" % username)
            for pkg in PackageDB().user_packages(username)['pkgs']:
                self.packages.add(pkg['name'])

    def match(self, msg, processor):
        packages = processor.packages(msg)
        for package in self.packages:
            if package in packages:
                return True

The fedmsg-notify-config interface (see gui.py), automatically introspects the filters and populates the Advanced tab with the appropriate labels, switches, and text entries.

Consuming fedmsg over DBus

Let's say you want to write an application that listens to fedmsg, but you don't want to deal with spinning up your own connection, or you're not using Python, etc. For these cases, fedmsg-notify supports relaying messages over DBus. This functionality can be enabled by running `gsettings set org.fedoraproject.fedmsg.notify emit-dbus-signals true`. You can then easily listen for the MessageReceived DBus signal, like so:

import json, dbus

from gi.repository import GObject
from dbus.mainloop.glib import DBusGMainLoop

def consume(topic, body):
    print(topic)
    print(json.loads(body))

DBusGMainLoop(set_as_default=True)
bus = dbus.SessionBus()
bus.add_signal_receiver(consume, signal_name='MessageReceived',
                        dbus_interface='org.fedoraproject.fedmsg.notify',
                        path='/org/fedoraproject/fedmsg/notify')
loop = GObject.MainLoop()
loop.run()

Contributing

If you're interested in helping out with any layer of the fedmsg stack, hop in #fedora-apps, and fork it on GitHub:

• lmacken/gnome-shell-extension-fedmsg
  
• fedora-infra/fedmsg-notify
  
• fedora-infra/fedmsg_meta_fedora_infrastructure
  
• fedora-infra/fedmsg
  
• mokshaproject/moksha
  

Hop on the bus!

07 Mar 2013 5:30pm GMT

When I say "more accessible" I mean any of these:

• useful for people with hearing / reading / seeing / cognitive / anything differences
• approachable for people at many different skill levels
• not intimidating. not insulting or offensive or exclusive.
• useful for people that are studying the material afterward
• Help me out here - what are some more examples?

I'm not an expert on how to do this, but I know this is a problem. I want help figuring out solutions, so please get in touch with me if you can help with that. I will ignore defenders of the status quo.

More detail on the problem

We pump a lot of energy into making really cool presentations for conferences, and then, when the conference is over, a lot of times, that great content usually just disappears.

Or if it doesn't disappear, we don't do a good job of getting it out where more people can benefit from our work. Maybe there's a zipfile with our slides on a page for our talk on the conference website afterwards.

Maybe the slides (without most of the commentary) will show up online in one of those flash widgets.

Or if you're really lucky, a video recording of the presentation will show up. And that's great. For example, Next Day Video records and edits the PyOhio presentations, and does fantastic work, but just a video is not sufficient for all audiences.

A video recording is great for some things, but not for others. It isn't easy how to copy a URL mentioned in a video, for example, or copy-paste a block of code. Or bookmark something 25-minutes in.

Consider that for every person in your audience, over the next few years, there's probably at least 10 or a hundred or maybe even a thousand people that will be doing searches online for the facts you're covering right now.

A lot of those people might be brand new to the language or library. A lot of those people might not be native English speakers. And maybe they're on slow internet connections too.

A few ideas to make this better

I have a few ideas for what to do, listed below, but I'm more interested in getting feedback from readers. So please, let me know how what you think we should do.

Anyhow, my ideas:

• Require presenters to submit something like a paper, not just a stack of LOLCATS slides for their proposal.
• Bundle the materials from the presenters as soon as possible and get those out on the web. The SAS Global Forum does this, and it is great. I have read nearly all the papers presented at their conferences, because they make it so easy to get their material, and because everybody is writing actual papers, not doing stream of consciousness performance art.
• Use open formats for text. Avoid PDFs, power-point slide desks, and similar stuff like the plague. Take SEO to heart. Not because we want to sell advertising, but because we want to share our knowledge.
• Encourage attendees to critically react to the presentations. Maybe even consider the presentation material as open-source. A presentation contains some code sample that confuses people in the audience, then people should rewrite that with something more intuitive and more obvious.

22 Feb 2013 8:06pm GMT

My little flock of four birds got some attention in this article.

12 Feb 2013 4:24pm GMT

Here it is

If stuff goes according to plan, 2013 is going to be the year when this business really takes off.

28 Jan 2013 5:01pm GMT

I've been trying to tidy up my Mako templates within my Turbogears 2 project. As part of that I was looking at some of them that are quickstarted including one which is the About page.

What was curious was there was all this CSS work all done already for you, including icons. Digging further I found out that one of the many projects Turbogears takes in, is Bootstrap. It's website describes Bootstrap as "Sleek, intuitive, and powerful front-end framework for faster and easier web development." but what it means to me is a bunch of guys who understand HTML and CSS way better than me have made it easy for me to build a decent web application.

While the framework won't suit everyone's tastes, it makes a lot of the formatting decisions so much easier. The thing is, in all the Turbogears documentation I have read I've not heard it mentioned. Not sure why, its pretty awesome (not SQLAlchemy awesome but not many things are).

Related articles

• TurboGears2 2.2.1 (pypi.python.org)
• twitter bootstrap CSS mobile (stackoverflow.com)
• 11 Reasons to Use Twitter Bootstrap (sitepoint.com)
• Book Review : Twitter BootStrap Web Development How-To (anujgakhar.com)
• Bootstrap in 2013 (epiphanysearch.co.uk)

24 Jan 2013 12:05pm GMT

That's right, it's out right now, and you can begin using it in your own projects! Follow the directions on http://www.turbogears.org/ you'll get 2.2.0.

23 Jan 2013 9:24pm GMT

My Turbogears project has recently reached an important milestone; one of the back-end processes now runs pretty much continuously and the plugins it use (or at least the ones I can see) are also working. That means I can turn to the front-end which displays the data the back-end collected.

For some of the data I am using a ToscaWidgets (or TW2) widget called a jqGridWidget which is a very nice jquery device that separates the presentation and data using a json query. I've mentioned previously about how to get a jqGridWidget working but left the pre-filtering out until now. This meant that my grid showed all the items in the database, not just the ones I wanted to see, but it was a start.

Now this widget displays things called Attributes which are basically children of another model called Hosts. Basically, Attributes are things you want to check or track about a Host. My widget used to show all Attributes but often on a Host screen, I want to see its hosts only. So, this is how I got my widget to behave; I'm not sure this is THE CORRECT way of doing it, but it does work.

First, in the Hosts controller you need to create the widget and pass along the host_id that the controller has obtained. I was not able to use the sub-classing trick you see in some TW2 documentation but actually make a widget instance.

class HostsController(BaseController):
    # other stuff here
    class por2(portlets.Portlet):
        title = 'Host Attributes'
        widget = AttributeGrid()
        widget.host_id = host_id

Next, the prepare() method in the Widget needs to get hold of the host_id and put it into the postData list. I needed to do it before calling super() because the options become one long string in the sub-class.

class AttributeGrid(jqgrid.jqGridWidget):
    def prepare(self, **kw):
        if self.host_id is not None:
            self.options['postData'] = {'hostid': self.host_id}
        super(AttributeGrid, self).prepare()

This means the jqgrid when it asks for its JSON data will include the hostid parameter as well. We need that method to "see" the host ID so we can filter the database access.

Finally in the JSON method for the Attribute we pick up and filter on the hostid.

    @expose('json')
    @validate(validators={'hostid':validators.Int()})
    def jqsumdata(self, hostid=0, page=1, rows=1, *args, **kw):
        conditions = []
        if hostid > 0:
            conditions.append(model.Attribute.host_id == hostid)
        attributes =model.DBSession.query(model.Attribute).filter(and_(*conditions))

From there you run through the attributes variable and build your JSON reply.

Related articles

• tw2.core 2.1.3 (pypi.python.org)

17 Jan 2013 9:57pm GMT

The days between Christmas and New Year's Eve have always been a time for me where I find the tranquility to dig into personal projects without much distraction. The past few days I have been writing and recording a new song, which is a first for me, in that I wrote a two-hand Piano part [...]

28 Dec 2012 4:41pm GMT

Kleine Spielerei mit dem Step-Sequenzer des Yamaha AN1x: Share & Enjoy!

01 Dec 2012 2:00am GMT

Carp is for code templating.

You know, for stuff like starting HTML files with a bunch of predefined javascript libraries, or for starting a new python project without having to write a setup.py file from scratch.

You can use carp for a single file or for a tree of files and folders.

It makes it easy to take a folder of code files and subfolders, replace parts of text you want to be parameterized, and then store it. Later, you render that template by passing in the required parameters.

Get the code at github here.

08 Nov 2012 11:12pm GMT

Here's the contents of my ~/.psqlrc file:

-- Report time used for each query.
timing

-- Set a cute border around query results.
pset border 2

-- Set pager to less, rather than more.
setenv PAGER /usr/bin/less

-- Monkey up the prompt.

-- This one shows user@host:database
-- set PROMPT1 '%n@%m:%/%# '

-- Same thing, but with pretty colors, across several lines.
set PROMPT1 'n%[%033[0;36;29m%]%n@%m:%/%[%033[0m%]n%# '

Now I get a psql prompt that looks like this:

user@localhost:dbname
# select * from current_timestamp;
+-------------------------------+
|              now              |
+-------------------------------+
| 2012-10-23 12:22:38.263557-04 |
+-------------------------------+
(1 row)

Time: 1.907 ms

23 Oct 2012 4:24pm GMT

One of the new additions in TG2.1.4 has been the support for the so called pluggable applications, this is a really powerful and convenient feature that probably not enough TurboGears users started embracing.

For people that never used them, pluggable applications provide a python package that can be installed and "plugged" inside any existing TurboGears application to add new features. Django has been probably the first framework to bring this feature to Python world and TurboGears implementation tries to be as convenient by making pluggable applications identical to plain TurboGears applications and providing a "quickstart-pluggable" command that creates the application skeleton for you. Pluggable applications can be installed using easy_install or pip and they can off course depend on any other pluggable application they need.

This year, at EuroPython 2012, I have been pleased to present a talk about using TurboGears for rapid prototyping (both in Italian and English, you should be able to find the videos on EuroPython youtube channel), so I decided to dedicate a part of it to pluggable applications as they are actually the fastest way to rapidly prototype a project. With my surprise most the questions I received were about the EasyCrudRestController and not about pluggable applications.

While the EasyCrudRestController is definitively a powerful tool, it's far from being the answer to all the web developments needs. In most of the applications you are going to develop, users will probably prefer consulting content from something more engaging than an administration table of database entries.

This month, to create a set of utilities that can help people with their everyday needs, I decided to ask guys that work with me to make every part of the web sites that they were writing as pluggable applications. The result of this experiment has been that most of the pluggable apps that I did in my spare time (tgapp-smallpress, tgapp-photos, tgapp-tgcomments, tgext.tagging and so on) ended being used in real world projects and started to improve exposing hooks and ways to customize their behavior for the project they were going to be used.

After a few weeks, new pluggables like tgapp-fbauth, tgapp-userprofile, tgapp-calendarevents, tgapp-fbcontest, tgapp-youtubevideo has seen light and developing the target application started becoming blazing fast: Just plug what you need and customize it.

Embracing this philosophy the last project I'm working on has an app_cfg.py file that looks like:

plug(base_config, 'tgext.debugbar', inventing=True)
plug(base_config, 'tgext.scss')
plug(base_config, 'tgext.browserlimit')
plug(base_config, 'registration')
plug(base_config, 'photos')
plug(base_config, 'smallpress', 'press', form='XXX.lib.forms.ArticleForm')
plug(base_config, 'tgcomments', allow_anonymous=False)
from XXX.lib.matches import MyKindOfEvent
plug(base_config, 'calendarevents', 'eventi', event_types=[MyKindOfEvent()])
replace_template(base_config, 'smallpress.templates.article', 
                              'XXX.templates.press.article')
replace_template(base_config, 'smallpress.templates.excerpt', 
                              'XXX.templates.press.excerpt')

Thanks to this our development process has really improved: whenever a developer finds a bug he just has to propose a patch for the target pluggable, whenever someone notices a missing index on a query he has just to add it to the given pluggable. All the websites under development improved like people were working on the same project.

While existing pluggables might be limited, buggy or slow I'm getting confident that they will continue to improve, and some day they will surpass whatever custom implementation I can think of. I think I'm going to heavily rely on pluggable applications for any future project sticking to only one rule: "make it opensource". This way, apart from probably helping other people, I'm also improving my own projects through other people feedbacks, bug reports and patches to the pluggables I used.

So, next time you have to start a new project give a look at the TurboGears CogBin and check if there is a pluggable application that looks like what you need. If you find any issue or find space for improvements just fork it and send a pull request, or send an email on the TurboGears Mailing List I'll do my best to address any reported issue thanking you for your feedbacks as I'm aware that you are actually improving any past and future project that relies on that pluggable.

08 Sep 2012 9:30pm GMT

That's right, we finally did it! It's up right now, and if you follow the directions on http://www.turbogears.org/ you'll get 2.2.0.

I'd like to take a minute to thank Alessandro Molina in particular. I did what support I could, but this release really belongs to him. Please pass along congratulations and thanks to him. And if there's bugs, I'll take the blame myself for not doing more to help.

The full changelog is below.



Important Commit Messages:

• Fix template engine extension collision in tw2.core >= 2.1
• Permit TGController subclasses to keep the decorations made on the parent class methods around
• Support repoze.who v2
• New auth configuration layer that doesn't require repoze.what anymore while still being compatible
• Support validation dict using TW2 validators and make validation less dependant from FormEncode
• Always prefer default renderer when multiple engines are registered for the same content_type
• Fix empty documentation for some decorators
• Remove url kwargs which were marked for removal in 2.2
• Remove legacy renderers support
• Disable AuthMetadataProvider when is None
• Improved kajiki loader to solve some issues with py:extends
• use .jinja extension for jinja templates by default
• On Mako when using file loader templates where reloaded using the auto_reload_templates option, while when using dotted loader the mako.reloadfromdisk option was used. Merge the two options in auto_reload_templates as all the other templates engines use this one
• __before__ and __after__ were renamed to _before and _after
• Expose render call as a public api with render_template name
• Create sqla module for sqlalchemy related things and add BalancedSession for master/slave load balancing
• Change prefix for tw2 to make resources archiving work correctly
• prefer_toscawidgets2 option in app_cfg enables tw2 and disables tw2. This should be the default in 2.2 newly quickstarted projects
• Add option for setting the template rendering parameters.
• Permit to avoid doctype injection when manually rendering templates with tg.render.render
• Object dispatch has been delegated to the crank package, improving performance and test coverage
• Quickstart new applications using repoze.who v2
• Add dynamic response.charset to meta in mako templates
• New quickstart template
• Quickstart projects without repoze.what requirement using new auth layer
• Fix issue with kajiki quickstart
• Adapt the bootstrap based template to jinja changes, always provide master.html for extensions and remove the minimal template as the new one is minimal enough
• Make genshi always available to have a lingua franca for pluggable apps and extensions
• Storing sessions in encrypted cookies is easier to manage then the file based solution for newly quickstarted projects
• tgext.admin now works with ToscaWidgets2
• Paste quickstart command will now ask if you want to use an alternative templating system instead of asking for each one
• New Upgrading TG2 page
• Document new inherit option of @expose with examples
• Fix missing kajiki setup_renderer reference and remove reference to default_renderer
• Update Authentication documentation
• Document flash and caching
• Document lurl usage
• Document WebFlash
• Add more docs and example about app_globals and tmpl_context
• Document ugettext and lazy_ugettext
• Document DBsession().using_engine switch to a context manager
• Minor fixes in MultipleDtabases documentation
• Added documentation about master slave load balancing
• Removing some old and now invalid parts of the documentation and add ToscaWidgets2 documentation
• Remove some references to tw1 in core parts of the doc

Tickets Closed:
1 Improve documentation on authentication, identification and authorization
2 Include lazy translations in the documentation
11 Document SecureFormMixin from TW
19 DataGrid and pagination tutorial/​reference missing for TG2
21 sqlite db creation error in wiki20 tutorial
26 Documentation for app_globals unclear
27 Document webflash aka tg.flash
51 Turbogears 2.1b1 tgext.admin Doesn't Handle Boolean Options
57 add a --clean option to quickstart
68 Simplify database migration in TG2
70 Migrating turbogears to pip
71 Create some kind of "component" architecture for TG2
75 quickstart graphics aren't easy to modify
137 Proper use of url() for widgets in the documentation
138 Backport repoze.what-quickstart and repoze.what-pylonshq removal
139 Move dispatch to Crank
140 Verify quickstart and devtools dependency
149 Expose the tg.render.render function as render_template
150 Full ToscaWidgets2 support and make it default when quickstarting app
151 Minimal Search and Sorting functions in admin
152 Jinja Support in quickstart
153 Master /​ Slave database support
154 Merge autoreload options for template engines
156 Provide a way to quickstart with Kajiki
157 repoze.who-1.0 is bugged when using auth_tkt
158 Permit to inherit decorations on TGController subclasses
160 allow_only doesn't get checked in dynamically dispatched RestController

23 Aug 2012 11:15pm GMT