15 May 2013

feedPlanet TurboGears

Craig Small: itools is back

My last post I said that I had to remove my internet query tools due to some bugs that were a concern. Some of the code was hard to maintain and probably had holes and I had noticed that it looped at times.

I'm happy to say that I have restored some of those tools now, still located at http://enc.com.au/itools

This code is completely re-written in Python using the TurboGears toolkit which means it is a lot cleaner in how it works and how it looks. Some of the lookup tables use a database rather than an array for ease of updating and querying. The downside is the backends will take time. It currently only does nslookup queries and whois only works for IPv4 addresses. The domain name queries will be a while off as these are the most complicated to handle. To give you an idea, all IPv4 and IPv6 address information comes from 5 sources with two formats while domain names come from over 200 sources with about 40 formats. This means the information from Regional Internet Registries will be done first.

15 May 2013 1:14pm GMT

12 Apr 2013

feedPlanet TurboGears

Craig Small: Pie Charts in TurboGears

You might of looked at Ralph Bean's tutorial on graphs and thought, that's nice but I'd like something different. The great thing about ToscaWidgets using the jqPlot library is that pretty much anything you can do in jqPlot, you can do in ToscaWidgets and by extension in TurboGears. You want different graph types? jqPlot has heaps!

My little application needed a Pie Chart to display the overall status of attributes. There are 6 states an attribute can be in: Up, Alert, Down, Testing, Admin Down and Unknown. The Pie Chart would show them all with some sort of colour representing the status. For this example I've used hard-coded data but you would normally extract it from the database using a TurboGears model and possibly some bucket sorting code.

I've divided my code into widget specific, which is found in myapp/widgets/attribute.py and the controller code found unsurprisingly at myapp/controllers/attribute.py Also note that some versions of ToscaWidgets have a bug which means any jqPlot widget won't work, version 2.0.4 has the fix for issue 80 that explains briefly the bug.

The widget code looks like this:

from tw2.jqplugins.jqplot import JQPlotWidget
from tw2.jqplugins.jqplot.base import pieRenderer_js
import tw2.core as twc
 
class AttributeStatusPie(JQPlotWidget):
    """
    Pie Chart of the Attributes' Status """
    id = 'attribute-status-pie'
    resources = JQPlotWidget.resources + [
            pieRenderer_js,
            ]
 
    options = {
            'seriesColors': [ "#468847", "#F89406", "#B94A48", "#999999", "#3887AD", "#222222"], 
            'seriesDefaults' : {
                'renderer': twc.js_symbol('$.jqplot.PieRenderer'),
                },
            'legend': {
                'show': True, 
                'location': 'e',
                },
            }

Some important things to note are:

Next the controller needs to be defined:

from myapp.widgets.attribute import AttributeStatusPie
 
@expose('myapp.templates.widget')
    def statuspie(self):
        data = [[
                ['Up', 20], ['Alert', 7], ['Down', 12], ['Admin Down', 3], ['Testing', 1], ['Unknown', 4],
                ]]
        pie = AttributeStatusPie(data=data)
        return dict(w=pie)

And that is about it, we now have a controller path attributes/statuspie which shows us the pie chart.
My template is basically a bare template with a ${w.display | n} in it to just show the widget for testing.

Pie Chart in Turbogears

Pie Chart in Turbogears

12 Apr 2013 11:40pm GMT

13 Mar 2013

feedPlanet TurboGears

Matthew Wilson: Barred rocks and a frosty morning

Filmed these last week weekend. You can hear my year-old daughter talking to them.

13 Mar 2013 12:08pm GMT

07 Mar 2013

feedPlanet TurboGears

Luke Macken: Keeping your finger on the pulse of the Fedora community

For those who haven't been keeping up with all of the awesome code Ralph Bean has been churning out lately, be sure to checkout fedmsg.com. Hop on #fedora-fedmsg on Freenode or load up busmon to see it in action. Not all of the Fedora Infrastructure services currently fire off fedmsgs, but we're getting very close.

This technology is built on top of Moksha, which I created many years ago while writing the first version of the fedoracommunity app. It's come a long way since then, and now can speak ØMQ over WebSockets, as well as AMQP and STOMP over Orbited. Now the time has finally come to bring Moksha to the desktop!

Introducing fedmsg-notify

fedmsg-notify lets you get realtime desktop notifications of activity within the Fedora community. It allows you to tap into the firehose of contributions as they happen and filter them to your liking. It works with any window manager that supports the notification-spec, however I've only seen the gravatars show up using GNOME.


For GNOME Shell users, you can [optionally] install gnome-shell-extension-fedmsg, and then enable it with the gnome-tweak-tool or by running `gnome-shell-extension-tool -e fedmsg@lmacken-redhat.com` (and then hit alt+f2 and type 'r' to reload the shell). You will then be graced with the presence of The Bus:


For those who aren't running GNOME shell, you can simply yum install fedmsg-notify, and then run fedmsg-notify-config, or launch it from your Settings menu. Due to a dependency on Twisted's gtk3reactor, fedmsg-notify is currently only available on Fedora 18 and newer.


The first tab shows you all services that are currently hooked into fedmsg. As we add new ones, the gui will automatically display them. These services are defined in the fedmsg_meta_fedora_infrastructure package.


The Advanced tab lets you further customize what messages you want to see. The "Bugs that you have encountered" option will display all messages that reference any Bugzilla numbers for crashes that you have hit locally with ABRT. The other filters involve querying your local yum database or the PackageDB.

Under the hood

The fedmsg-notify-daemon itself is fairly minimal (see daemon.py). At it's core, it's just a Twisted reactor that consumes ØMQ messages. Moksha does all of the heavy lifting behind the scenes, so all we really have to do is specify a topic to subscribe to and define a consume method that gets called with each message. This is essentially just a basic Moksha Consumer with some fedmsg + DBus glue.

class FedmsgNotifyService(dbus.service.Object, fedmsg.consumers.FedmsgConsumer):
    topic = 'org.fedoraproject.*'

    def consume(self, msg): 

The daemon will automatically startup upon login, or will get activated by DBus when enabled via the GUI. When a message arrives, it filters it accordingly, downloads & caches the icons, [optionally] relays the message over DBus, and then displays the notification on your desktop.

The API for writing custom filters is dead simple (see filters.py). Here is an example of one:

class MyPackageFilter(Filter):
    """ Matches messages regarding packages that a given user has ACLs on """
    __description__ = 'Packages that these users maintain'
    __user_entry__ = 'Usernames'

    def __init__(self, settings):
        self.usernames = settings.replace(',', ' ').split()
        self.packages = set()
        reactor.callInThread(self._query_pkgdb)

    def _query_pkgdb(self):
        for username in self.usernames:
            log.info("Querying the PackageDB for %s's packages" % username)
            for pkg in PackageDB().user_packages(username)['pkgs']:
                self.packages.add(pkg['name'])

    def match(self, msg, processor):
        packages = processor.packages(msg)
        for package in self.packages:
            if package in packages:
                return True

The fedmsg-notify-config interface (see gui.py), automatically introspects the filters and populates the Advanced tab with the appropriate labels, switches, and text entries.

Consuming fedmsg over DBus

Let's say you want to write an application that listens to fedmsg, but you don't want to deal with spinning up your own connection, or you're not using Python, etc. For these cases, fedmsg-notify supports relaying messages over DBus. This functionality can be enabled by running `gsettings set org.fedoraproject.fedmsg.notify emit-dbus-signals true`. You can then easily listen for the MessageReceived DBus signal, like so:

import json, dbus

from gi.repository import GObject
from dbus.mainloop.glib import DBusGMainLoop

def consume(topic, body):
    print(topic)
    print(json.loads(body))

DBusGMainLoop(set_as_default=True)
bus = dbus.SessionBus()
bus.add_signal_receiver(consume, signal_name='MessageReceived',
                        dbus_interface='org.fedoraproject.fedmsg.notify',
                        path='/org/fedoraproject/fedmsg/notify')
loop = GObject.MainLoop()
loop.run()

Contributing

If you're interested in helping out with any layer of the fedmsg stack, hop in #fedora-apps, and fork it on GitHub:

Hop on the bus!

07 Mar 2013 5:30pm GMT

22 Feb 2013

feedPlanet TurboGears

Matthew Wilson: We need to make our conference presentations more accessible

When I say "more accessible" I mean any of these:

I'm not an expert on how to do this, but I know this is a problem. I want help figuring out solutions, so please get in touch with me if you can help with that. I will ignore defenders of the status quo.

More detail on the problem

We pump a lot of energy into making really cool presentations for conferences, and then, when the conference is over, a lot of times, that great content usually just disappears.

Or if it doesn't disappear, we don't do a good job of getting it out where more people can benefit from our work. Maybe there's a zipfile with our slides on a page for our talk on the conference website afterwards.

Maybe the slides (without most of the commentary) will show up online in one of those flash widgets.

Or if you're really lucky, a video recording of the presentation will show up. And that's great. For example, Next Day Video records and edits the PyOhio presentations, and does fantastic work, but just a video is not sufficient for all audiences.

A video recording is great for some things, but not for others. It isn't easy how to copy a URL mentioned in a video, for example, or copy-paste a block of code. Or bookmark something 25-minutes in.

Consider that for every person in your audience, over the next few years, there's probably at least 10 or a hundred or maybe even a thousand people that will be doing searches online for the facts you're covering right now.

A lot of those people might be brand new to the language or library. A lot of those people might not be native English speakers. And maybe they're on slow internet connections too.

A few ideas to make this better

I have a few ideas for what to do, listed below, but I'm more interested in getting feedback from readers. So please, let me know how what you think we should do.

Anyhow, my ideas:

22 Feb 2013 8:06pm GMT

12 Feb 2013

feedPlanet TurboGears

Matthew Wilson: Backyard chickens in Ohio

My little flock of four birds got some attention in this article.

me holding a chicken

12 Feb 2013 4:24pm GMT

28 Jan 2013

feedPlanet TurboGears

Matthew Wilson: Check out the awesome logo for 216 Software, LLC!

Here it is

Logo

If stuff goes according to plan, 2013 is going to be the year when this business really takes off.

28 Jan 2013 5:01pm GMT

24 Jan 2013

feedPlanet TurboGears

Craig Small: Bootstrap – The hidden gem in Turbogears

I've been trying to tidy up my Mako templates within my Turbogears 2 project. As part of that I was looking at some of them that are quickstarted including one which is the About page.

What was curious was there was all this CSS work all done already for you, including icons. Digging further I found out that one of the many projects Turbogears takes in, is Bootstrap. It's website describes Bootstrap as "Sleek, intuitive, and powerful front-end framework for faster and easier web development." but what it means to me is a bunch of guys who understand HTML and CSS way better than me have made it easy for me to build a decent web application.

While the framework won't suit everyone's tastes, it makes a lot of the formatting decisions so much easier. The thing is, in all the Turbogears documentation I have read I've not heard it mentioned. Not sure why, its pretty awesome (not SQLAlchemy awesome but not many things are).

Related articles

24 Jan 2013 12:05pm GMT

23 Jan 2013

feedPlanet TurboGears

Michael Pedersen: Announcing TurboGears 2.2.1!


That's right, it's out right now, and you can begin using it in your own projects! Follow the directions on http://www.turbogears.org/ you'll get 2.2.0.


I'd like to thank +Alessandro Molina, +Christoph Zwerschke, and +Carlos Daniel Ruvalcaba Valenzuela. Between the three of them, TurboGears is becoming better every day, and they deserve much praise and thanks.

The full changelog is below.
Ming / Mongo
  • Support connection options for ming/mongodb
  • Support new ming configuration API
  • Fix datastore options for ming
Templating
  • Jinja autoload filters
  • Added jinja filter autoloading namespace protection test cases.
  • When autoloading jinja filters use the special __all__ module variable to import the filters, if it is not defined then just import any defined callables.
  • Jinja2 didn't perform escaping while mako, genshi and kajiki did
  • Closes #17, mako bytecaching only works when use_dotted_templatenames = False.
Authentication
  • Better configuration of authenticators.
  • More flexible configuration of authenticators. By adding ('default', None) to the list of authenticators, you can now include the default authenticator in the list of custom authenticators.
Documentation
  • Fix more references to pylons.c
  • Update doc to deprecated pylons ns
  • Update Wiki20 tutorial to TG2.2 and try to make it easier to follow and faster to read by hightlighting code and reducing parts that are more advanced or pure web developemnt/python related
TurboGears Project
  • Fix for travis due to repoze.who 2.1 now bringing in WebOb 1.2
  • Travis-ci for development branch
  • Add contributing file.
Everything Else
  • urljoin behavior on schemas seems to change depending on python version
  • Refactor request_type detection usage
  • Fix issue with request extensions being wrongly handled two times
  • Use the new built-in translations provided by ToscaWidgets 2.
  • @beaker_cache was not available inside the deacorators module while it should have been
  • @require now provides smart behavior, and can be used to just abort, rather than do redirect. Closes #27
  • Make possible to import controllers (and so expose templates) before AppConfig.init_config has been called
  • Most packages used by TG currently don't work with SQLAlchemy 0.8
  • Permission._groups is currently defined as an array, so it expects to receive a list

23 Jan 2013 9:24pm GMT

17 Jan 2013

feedPlanet TurboGears

Craig Small: Pre-selecting ToscaWidgets jqgrid values in TurboGears

My Turbogears project has recently reached an important milestone; one of the back-end processes now runs pretty much continuously and the plugins it use (or at least the ones I can see) are also working. That means I can turn to the front-end which displays the data the back-end collected.

For some of the data I am using a ToscaWidgets (or TW2) widget called a jqGridWidget which is a very nice jquery device that separates the presentation and data using a json query. I've mentioned previously about how to get a jqGridWidget working but left the pre-filtering out until now. This meant that my grid showed all the items in the database, not just the ones I wanted to see, but it was a start.

Now this widget displays things called Attributes which are basically children of another model called Hosts. Basically, Attributes are things you want to check or track about a Host. My widget used to show all Attributes but often on a Host screen, I want to see its hosts only. So, this is how I got my widget to behave; I'm not sure this is THE CORRECT way of doing it, but it does work.

First, in the Hosts controller you need to create the widget and pass along the host_id that the controller has obtained. I was not able to use the sub-classing trick you see in some TW2 documentation but actually make a widget instance.

class HostsController(BaseController):
    # other stuff here
    class por2(portlets.Portlet):
        title = 'Host Attributes'
        widget = AttributeGrid()
        widget.host_id = host_id

Next, the prepare() method in the Widget needs to get hold of the host_id and put it into the postData list. I needed to do it before calling super() because the options become one long string in the sub-class.

class AttributeGrid(jqgrid.jqGridWidget):
    def prepare(self, **kw):
        if self.host_id is not None:
            self.options['postData'] = {'hostid': self.host_id}
        super(AttributeGrid, self).prepare()

This means the jqgrid when it asks for its JSON data will include the hostid parameter as well. We need that method to "see" the host ID so we can filter the database access.

Finally in the JSON method for the Attribute we pick up and filter on the hostid.

    @expose('json')
    @validate(validators={'hostid':validators.Int()})
    def jqsumdata(self, hostid=0, page=1, rows=1, *args, **kw):
        conditions = []
        if hostid > 0:
            conditions.append(model.Attribute.host_id == hostid)
        attributes =model.DBSession.query(model.Attribute).filter(and_(*conditions))

From there you run through the attributes variable and build your JSON reply.

Related articles

17 Jan 2013 9:57pm GMT

28 Dec 2012

feedPlanet TurboGears

Christpher Arndt: A Song For The New Year

The days between Christmas and New Year's Eve have always been a time for me where I find the tranquility to dig into personal projects without much distraction. The past few days I have been writing and recording a new song, which is a first for me, in that I wrote a two-hand Piano part [...]

28 Dec 2012 4:41pm GMT

01 Dec 2012

feedPlanet TurboGears

Christpher Arndt: On The AN1x Machine

Kleine Spielerei mit dem Step-Sequenzer des Yamaha AN1x: Share & Enjoy!

01 Dec 2012 2:00am GMT

08 Nov 2012

feedPlanet TurboGears

Matthew Wilson: Announcing carp!

Carp is for code templating.

You know, for stuff like starting HTML files with a bunch of predefined javascript libraries, or for starting a new python project without having to write a setup.py file from scratch.

You can use carp for a single file or for a tree of files and folders.

It makes it easy to take a folder of code files and subfolders, replace parts of text you want to be parameterized, and then store it. Later, you render that template by passing in the required parameters.

Get the code at github here.

08 Nov 2012 11:12pm GMT

23 Oct 2012

feedPlanet TurboGears

Matthew Wilson: Customize your psql terminal

Here's the contents of my ~/.psqlrc file:



-- Report time used for each query.
timing

-- Set a cute border around query results.
pset border 2

-- Set pager to less, rather than more.
setenv PAGER /usr/bin/less

-- Monkey up the prompt.

-- This one shows user@host:database
-- set PROMPT1 '%n@%m:%/%# '

-- Same thing, but with pretty colors, across several lines.
set PROMPT1 'n%[%033[0;36;29m%]%n@%m:%/%[%033[0m%]n%# '

Now I get a psql prompt that looks like this:


user@localhost:dbname
# select * from current_timestamp;
+-------------------------------+
|              now              |
+-------------------------------+
| 2012-10-23 12:22:38.263557-04 |
+-------------------------------+
(1 row)

Time: 1.907 ms

23 Oct 2012 4:24pm GMT

08 Sep 2012

feedPlanet TurboGears

Alessandro Molina: It’s a Pluggable World

One of the new additions in TG2.1.4 has been the support for the so called pluggable applications, this is a really powerful and convenient feature that probably not enough TurboGears users started embracing.

For people that never used them, pluggable applications provide a python package that can be installed and "plugged" inside any existing TurboGears application to add new features. Django has been probably the first framework to bring this feature to Python world and TurboGears implementation tries to be as convenient by making pluggable applications identical to plain TurboGears applications and providing a "quickstart-pluggable" command that creates the application skeleton for you. Pluggable applications can be installed using easy_install or pip and they can off course depend on any other pluggable application they need.

This year, at EuroPython 2012, I have been pleased to present a talk about using TurboGears for rapid prototyping (both in Italian and English, you should be able to find the videos on EuroPython youtube channel), so I decided to dedicate a part of it to pluggable applications as they are actually the fastest way to rapidly prototype a project. With my surprise most the questions I received were about the EasyCrudRestController and not about pluggable applications.

While the EasyCrudRestController is definitively a powerful tool, it's far from being the answer to all the web developments needs. In most of the applications you are going to develop, users will probably prefer consulting content from something more engaging than an administration table of database entries.

This month, to create a set of utilities that can help people with their everyday needs, I decided to ask guys that work with me to make every part of the web sites that they were writing as pluggable applications. The result of this experiment has been that most of the pluggable apps that I did in my spare time (tgapp-smallpress, tgapp-photos, tgapp-tgcomments, tgext.tagging and so on) ended being used in real world projects and started to improve exposing hooks and ways to customize their behavior for the project they were going to be used.

After a few weeks, new pluggables like tgapp-fbauth, tgapp-userprofile, tgapp-calendarevents, tgapp-fbcontest, tgapp-youtubevideo has seen light and developing the target application started becoming blazing fast: Just plug what you need and customize it.

Embracing this philosophy the last project I'm working on has an app_cfg.py file that looks like:

plug(base_config, 'tgext.debugbar', inventing=True)
plug(base_config, 'tgext.scss')
plug(base_config, 'tgext.browserlimit')
plug(base_config, 'registration')
plug(base_config, 'photos')
plug(base_config, 'smallpress', 'press', form='XXX.lib.forms.ArticleForm')
plug(base_config, 'tgcomments', allow_anonymous=False)
from XXX.lib.matches import MyKindOfEvent
plug(base_config, 'calendarevents', 'eventi', event_types=[MyKindOfEvent()])
replace_template(base_config, 'smallpress.templates.article', 
                              'XXX.templates.press.article')
replace_template(base_config, 'smallpress.templates.excerpt', 
                              'XXX.templates.press.excerpt')

Thanks to this our development process has really improved: whenever a developer finds a bug he just has to propose a patch for the target pluggable, whenever someone notices a missing index on a query he has just to add it to the given pluggable. All the websites under development improved like people were working on the same project.

While existing pluggables might be limited, buggy or slow I'm getting confident that they will continue to improve, and some day they will surpass whatever custom implementation I can think of. I think I'm going to heavily rely on pluggable applications for any future project sticking to only one rule: "make it opensource". This way, apart from probably helping other people, I'm also improving my own projects through other people feedbacks, bug reports and patches to the pluggables I used.

So, next time you have to start a new project give a look at the TurboGears CogBin and check if there is a pluggable application that looks like what you need. If you find any issue or find space for improvements just fork it and send a pull request, or send an email on the TurboGears Mailing List I'll do my best to address any reported issue thanking you for your feedbacks as I'm aware that you are actually improving any past and future project that relies on that pluggable.

08 Sep 2012 9:30pm GMT

23 Aug 2012

feedPlanet TurboGears

Michael Pedersen: Announcing TurboGears 2.2.0!

That's right, we finally did it! It's up right now, and if you follow the directions on http://www.turbogears.org/ you'll get 2.2.0.

I'd like to take a minute to thank Alessandro Molina in particular. I did what support I could, but this release really belongs to him. Please pass along congratulations and thanks to him. And if there's bugs, I'll take the blame myself for not doing more to help.

The full changelog is below.



Important Commit Messages:




Tickets Closed:
1 Improve documentation on authentication, identification and authorization
2 Include lazy translations in the documentation
11 Document SecureFormMixin from TW
19 DataGrid and pagination tutorial/​reference missing for TG2
21 sqlite db creation error in wiki20 tutorial
26 Documentation for app_globals unclear
27 Document webflash aka tg.flash
51 Turbogears 2.1b1 tgext.admin Doesn't Handle Boolean Options
57 add a --clean option to quickstart
68 Simplify database migration in TG2
70 Migrating turbogears to pip
71 Create some kind of "component" architecture for TG2
75 quickstart graphics aren't easy to modify
137 Proper use of url() for widgets in the documentation
138 Backport repoze.what-quickstart and repoze.what-pylonshq removal
139 Move dispatch to Crank
140 Verify quickstart and devtools dependency
149 Expose the tg.render.render function as render_template
150 Full ToscaWidgets2 support and make it default when quickstarting app
151 Minimal Search and Sorting functions in admin
152 Jinja Support in quickstart
153 Master /​ Slave database support
154 Merge autoreload options for template engines
156 Provide a way to quickstart with Kajiki
157 repoze.who-1.0 is bugged when using auth_tkt
158 Permit to inherit decorations on TGController subclasses
160 allow_only doesn't get checked in dynamically dispatched RestController


23 Aug 2012 11:15pm GMT