10 Mar 2014

feedPlanet TurboGears

Matthew Wilson: RIP hen #1

For two years now, I've been letting my hens free range around in a fenced-in section in my back yard during the day. They love it.

Went out to check for eggs today after work and found one of my girls had been killed in the coop.

Head bitten off. No other signs of predation. Internet forums say this was likely a possum or a raccoon.

Going to set up a live trap tonight. And some wire snares.

I raised these girls since they were a day old. Can't really remember the last time I was this angry and sad.

10 Mar 2014 11:01pm GMT

02 Jan 2014

feedPlanet TurboGears

Matthew Wilson: What’s good and bad about github issues

Ticketing / workflow / bugtracker systems are always nasty. Github's is pretty good. Maybe the best of what's out there. But it ain't perfect.

Here's what I like:

And what I dislike:

02 Jan 2014 4:09pm GMT

08 Dec 2013

feedPlanet TurboGears

Matthew Wilson: I really like my gardening boots

You don't need many tools to start gardening. You can dig holes with a stick or a sharp rock. You can start seeds in tin cans. You can use all sorts of stuff to carry water. You really only need dirt, sun, and seeds. So don't run out and buy a bunch of stuff!

But when you realize you've got the gardening bug bad, there's a few tools that really help. First off, you need some rubber boots. Otherwise, you're going to track mud everywhere. That's going to make your significant other very annoyed!

I bought these boots in 2006. They've held up very well over the last seven years. They're waterproof, thick enough to block thorns, easy to hose off, and the sole is thick enough that I can push on a shovel with them.

They're made here in Illinois, USA, by Boss Manufacturing Company. They've been around since 1893!

You can order them on Amazon. You can't order them direct from the company.

Note: I will get some commission if you order the boots from the link below, so if you hate me, you should not click on that link.

08 Dec 2013 4:50pm GMT

16 Nov 2013

feedPlanet TurboGears

Matthew Wilson: Upload files directly to Rackspace from the browser with AJAX PUT

I just wrote this up on my biz site.

Upload files directly to Rackspace Cloudfiles from the browser with AJAX PUT

I hope it helps somebody out!

16 Nov 2013 9:43pm GMT

10 Nov 2013

feedPlanet TurboGears

Alessandro Molina: TurboGears 2.3 Hidden Gems #2 - Application Wrappers

One of the less known features introduced in TurboGears 2.3 are application wrappers.
Application wrappers are much like controller wrappers (available since 2.2), but instead of wrapping controllers they actually wrap the whole application providing an easier way to implement what in plain WSGI is done through middlewares.

The advantage of application wrappers over middlewares is that they have full access to TurboGears stack, they can access the current request, the database, session and caches as the application itself would do.

The great part is that, as they run between TGApp and TGController, they can also replace the TurboGears Context and the TurboGears Response providing a great way to hijack requests, responses or even replace entire features of the framework like the cache layer. A very similar concept is available in other frameworks like Pyramid Tweens.

A very simple application wrapper that intercepts exceptions and logs them without messing with the standard TurboGears error handling might look like:

class ErrorLoggingWrapper(object):
    def __init__(self, handler, config):
        self.handler = handler
        self.config = config
 
    def __call__(self, controller, environ, context):
        path = context.request.path
        try:
            return self.handler(controller, environ, context)
        except:
            log.exception('Error while handling %s', path)
            raise

The wrapper can then be enabled calling

base_config.register_wrapper(ErrorLoggingWrapper)

inside config/app_cfg.py

Now that we have an application wrapper able to log exceptions we can decide for example to add another one that suppresses exceptions and prints "Something went wrong!", as it is possible to specify the order of execution for application wrappers we can register a SuppressErrorsWrapper that should execute after the ErrorLoggingWrapper:

from webob import Response
 
class SuppressErrorsWrapper(object):
    def __init__(self, handler, config):
        self.handler = handler
        self.config = config
 
    def __call__(self, controller, environ, context):
        try:
            return self.handler(controller, environ, context)
        except:
            return Response('Oh! Oh! Something went wrong!', status=500, content_type='text/plain')

Then it can be registered after the ErrorLoggingWrapper using:

base_config.register_wrapper(SuppressErrorsWrapper, after=ErrorLoggingWrapper)

While applications wrappers are a powerful feature, most of their power comes from the new response management refactoring that makes possible to access the current context and replace the outgoing response while working with high level objects instead of having to manually cope with WSGI.

10 Nov 2013 1:16pm GMT

02 Nov 2013

feedPlanet TurboGears

Alessandro Molina: TurboGears 2.3 Hidden Gems #1 - New Response Management

TurboGears2.3 has been a major improvement for the framework, most of its code got rewritten to achieve less dependencies, cleaner codebase a cleaner API and a faster framework. This resulted in reduction to only 7 dependencies in minimal mode and a 3x faster codebase.

While those are the core changes for the release, there are a lot of side effects that users can exploit at their benefit. This is the reason why I decided to start this set of posts to describe some of those hidden gems and explain users how to achieve the best from the new release.

The first change I'm going to talk about is how the response management got refactored and simplified. While this has some direct benefits it also provided some interesting side effects it makes sense to explore.

How TurboGears on Pylons did it

TurboGears tried to abstract a lot of response complexity through tg.response object and as there were not many reasons to override TGController.__call__ it was common that the response object body was always set by TurboGears itself.

Due to the fact that Pylons controllers were somehow compliant to WSGI itself the TGController was then in charge of calling the start_response function by actually providing all the headers user set into tg.response

response = self._dispatch_call()
 
# Here the response body got set, removed for brevity
 
if hasattr(response, 'wsgi_response'):
  # Copy the response object into the testing vars if we're testing
  if 'paste.testing_variables' in environ:
      environ['paste.testing_variables']['response'] = response
  if log_debug:
      log.debug("Calling Response object to return WSGI data")
 
  return response(environ, self.start_response)

While this made sense for Pylons, where you are expected to subclass the controller to perform advanced customizations, it was actually something unexposed to TurboGears users.

TurboGears made possible to change application behaviour using hooks and controller_wrappers. So the use for subclassing the TGController was actually strictly related to custom dispatching methods, which was usually better solved by specializing the TGController._dispatch method (tgext.routes is a simple enough example of this).

Cleaning Up Things

This lead to a curious situation where the TGController needed to speak with TGApp through WSGI to make Pylons happy, so it needed to call start_response and return the response iterator itself. TGApp was supposed to be the WSGI application, but in fact most of the real work was happening into TGController, in the end we had two WSGI applications: both TGController and TGApp were callable that spoke WSGI.

The 2.3 rewrite has been a great occasion to solve this ambiguity by providing a clear communication channel between TGController and TGApp by assigning each one a specific responsibility.

Communication Channel

In TG2.3 only the TGApp is now in charge of exposing the WSGI application interface. The TGController is expected to get a TurboGears Request Context object and provide back a TurboGears Response object. The TGApp will then use the provided response object to submit headers and response body.

The TGController code got much more straightforward and the whole testing and call response part was moved to the TGApp itself:

try:
    response = self._perform_call(context)
except HTTPException as httpe:
    response = httpe
 
# Here the response body got set, removed for brevity
 
return response

This has been possible without breaking backward compatibility thanks to the fact that the only subclassing of TGController common in TurboGears world was the BaseController class implemented by most applications.

The BaseController usually acts just as a pass-through between TGApp and TGController to setup some shortcuts to authentication data and other helpers for each request. So the fact that the parameters received by BaseController.__call__ changed didn't cause an huge issue as they were just forwarded to TGController.__call__

A little side effect

One of the interesting effects of this change is that your controllers are now enabled to return any instance of webob.Response.

In previous versions it was possible to return practically only webob. WSGIHTTPException subclasses (as they exposed a wsgi_response property which was consumed by Pylons), so it was possible to return an HTTPFound instance to force a redirect, but it was not possible to return a plain response.

A consequence of the new change is enabling your controller to call third party WSGI applications by using tg.request.get_reponse with a given application. The returned response can be directly provided as the return value of your controller.

This behaviour also makes easier to write reusable components that don't need to rely on tg.response and change it. Your application can forward the request to them and proxy back the response they return.

Part #2 will cover Application Wrappers, which greatly benefit from the new response management.

02 Nov 2013 10:51pm GMT

17 Sep 2013

feedPlanet TurboGears

Craig Small: jqGrid in TurboGears2 Admin Screens

I wanted to use the jqGrid for my admin pages as I liked that look and it matches some of the other screens outside the admin controller. The admin controller, or rather the CrudRestController out of tgext.crud, has a way of exporting results in json format. So surely its a matter of changing the TableBase to use jqGrid in the admin controller and we're done?

Well, no.

First you need to adjust the jsonReader options so that it lines up to the field names that the controller sends and this was one of the first (or last snags). The json output looks like:

{
  "value_list": {
    "total": 20,
    "items_per_page": 7,
    "page": 1,
    "entries": [(lots of entries)...]
  }
}

Now this is a little odd because of the top-level dictionary that is being used here. Most of the examples have everything that is inside the value_list being returned. In fact adjusting the controller to return only those items in the value_list values works.

To look inside this structure we need to adjust the jsonReader options. jqGrid documentation uses the format "toptier>subtier" for the XML reader so that was the intial attempt. It was also an intial fail, it doesn't work and you get your very familiar empty grid.

The trick is close to this format, but slightly different for json. You change the options to "toptier.subtier". In other words change the greater than symbol to a full stop for json access.

The jqGridWidget now has the following options (amongst others):

options = {
  'datatype': 'json',
  'jsonReader': {
    'repeatitems': False,
    'root': 'value_list.entries',
    'total': 'value_list.total',
    'page': 'value_list.page',
  }
}

There might be a way of saying all entries sit under value_list inside jqGrid, but I couldn't find it. Those options given above do give a working jqGrid on the admin screens.

17 Sep 2013 12:29pm GMT

23 Jul 2013

feedPlanet TurboGears

Matthew Wilson: old-school code checklist

You can't call it old-school code unless a majority of these are true:

23 Jul 2013 5:06pm GMT

18 Jul 2013

feedPlanet TurboGears

Craig Small: Step or Sloping Graphs

Even though the backend of Rosenberg NMS uses rrdtool RRD files, the front end graphs are created using jqPlot. The idea is to have a set of templates for the different types of graphs and just apply them to the various data sets. It makes things a lot simpler for new graphs because you just select which one you want; unless you want something a lot different which would involve a new graph template.

In any case, anyone that looks enough at the standard rrdtool graphs will know they are a series of steps. While it depends on the RRA, usually they are 5 minute steps, so a graph showing an increasing rate might show 5 minutes of 2 Mbps and then the next 5 minutes of 11 Mbps. jqPlot graphs as I've currently got them draws a line between two data points, so there would be a sloping line starting at 2.5 minutes (half of the first 5 minute interval) and 2Mbps sloping up to 11 Mbps at 7.5 minutes.

At first I thought this was wrong and spent some time attempting to "fix" the graph by making it look like a rrdtool graph more. Someway through that process I stopped and wondered, what IS the right way? The answer like a lot of other things, is "it depends".

For a graph showing a rate, such as the output bits per second on an interface, the way this is done is at regular intervals a counter is measured. So if at time 0 the counter is 140 and time 60 the counter is 200 and finally at time 120 the counter is at 800 there has been an average rate of 1 [(200-140)/60] and 100 [(800-200)/60)]. rrdtool would show a horizontal bar at 1 bps and then another horizontal bar at 100 at the next time interval. jqPlot would show a sloping line going from 30,1 up to 90,100.

Two graphs looking very different from the same data, what gives? Each graph is right, or rather is showing an estimation of different things, hence their differences.

rrdtool is showing the average rate for that time period. It is in some ways accurate because leaving aside missed wrap-arounds and resets that many bits did pass through that interface for the time specified. However often the graphs are interpreted incorrectly as the real rate and not an average. We can be reasonably sure for a rate that it would not be 1 for exactly a minute and then immediately jump to 100 for another minute. This isn't rrdtool's fault, it is just how the graphs can be interpreted.

jqplot will show more "realistic" graphs, with a curve sloping up. However this too makes assumptions that the rate increase is linear which often it is not. It just gives the illusion that the graph knows more about the data than it really does.

In the end, both graphs are at the same time accurate and misleading. It's important when looking at any graph in general (not just these two types) that you understand its limits and assumptions. To give one example of the problems that can be missed, traffic interfaces may have microbursts (large amount of traffic in short amount of time) which, due to the averaging that goes on in graphing are invisible to graphs and give an incorrect account of what is going on.

18 Jul 2013 1:05pm GMT

28 Jun 2013

feedPlanet TurboGears

Matthew Wilson: Obscure python syntax error

Been writing python for a long time. When I wrote this code, I could not figure out why I was getting a syntax error.


d1 = dict(
    user_id=99,
    display_name='Matt Wilson',)

d2 = dict(
    email_address='matt@tplus1.com',
    **d1,)

It is the trailing comma after **d1. It is not OK. Which is really weird, because the trailing comma after display_name='Matt Wilson' is just fine.

28 Jun 2013 6:53pm GMT

23 Jan 2013

feedPlanet TurboGears

Michael Pedersen: Announcing TurboGears 2.2.1!


That's right, it's out right now, and you can begin using it in your own projects! Follow the directions on http://www.turbogears.org/ you'll get 2.2.0.


I'd like to thank +Alessandro Molina, +Christoph Zwerschke, and +Carlos Daniel Ruvalcaba Valenzuela. Between the three of them, TurboGears is becoming better every day, and they deserve much praise and thanks.

The full changelog is below.
Ming / Mongo
  • Support connection options for ming/mongodb
  • Support new ming configuration API
  • Fix datastore options for ming
Templating
  • Jinja autoload filters
  • Added jinja filter autoloading namespace protection test cases.
  • When autoloading jinja filters use the special __all__ module variable to import the filters, if it is not defined then just import any defined callables.
  • Jinja2 didn't perform escaping while mako, genshi and kajiki did
  • Closes #17, mako bytecaching only works when use_dotted_templatenames = False.
Authentication
  • Better configuration of authenticators.
  • More flexible configuration of authenticators. By adding ('default', None) to the list of authenticators, you can now include the default authenticator in the list of custom authenticators.
Documentation
  • Fix more references to pylons.c
  • Update doc to deprecated pylons ns
  • Update Wiki20 tutorial to TG2.2 and try to make it easier to follow and faster to read by hightlighting code and reducing parts that are more advanced or pure web developemnt/python related
TurboGears Project
  • Fix for travis due to repoze.who 2.1 now bringing in WebOb 1.2
  • Travis-ci for development branch
  • Add contributing file.
Everything Else
  • urljoin behavior on schemas seems to change depending on python version
  • Refactor request_type detection usage
  • Fix issue with request extensions being wrongly handled two times
  • Use the new built-in translations provided by ToscaWidgets 2.
  • @beaker_cache was not available inside the deacorators module while it should have been
  • @require now provides smart behavior, and can be used to just abort, rather than do redirect. Closes #27
  • Make possible to import controllers (and so expose templates) before AppConfig.init_config has been called
  • Most packages used by TG currently don't work with SQLAlchemy 0.8
  • Permission._groups is currently defined as an array, so it expects to receive a list

23 Jan 2013 9:24pm GMT

28 Dec 2012

feedPlanet TurboGears

Christpher Arndt: A Song For The New Year

The days between Christmas and New Year's Eve have always been a time for me where I find the tranquility to dig into personal projects without much distraction. The past few days I have been writing and recording a new song, which is a first for me, in that I wrote a two-hand Piano part […]

28 Dec 2012 4:41pm GMT

01 Dec 2012

feedPlanet TurboGears

Christpher Arndt: On The AN1x Machine

Kleine Spielerei mit dem Step-Sequenzer des Yamaha AN1x: Share & Enjoy!

01 Dec 2012 2:00am GMT

08 Sep 2012

feedPlanet TurboGears

Alessandro Molina: It’s a Pluggable World

One of the new additions in TG2.1.4 has been the support for the so called pluggable applications, this is a really powerful and convenient feature that probably not enough TurboGears users started embracing.

For people that never used them, pluggable applications provide a python package that can be installed and "plugged" inside any existing TurboGears application to add new features. Django has been probably the first framework to bring this feature to Python world and TurboGears implementation tries to be as convenient by making pluggable applications identical to plain TurboGears applications and providing a "quickstart-pluggable" command that creates the application skeleton for you. Pluggable applications can be installed using easy_install or pip and they can off course depend on any other pluggable application they need.

This year, at EuroPython 2012, I have been pleased to present a talk about using TurboGears for rapid prototyping (both in Italian and English, you should be able to find the videos on EuroPython youtube channel), so I decided to dedicate a part of it to pluggable applications as they are actually the fastest way to rapidly prototype a project. With my surprise most the questions I received were about the EasyCrudRestController and not about pluggable applications.

While the EasyCrudRestController is definitively a powerful tool, it's far from being the answer to all the web developments needs. In most of the applications you are going to develop, users will probably prefer consulting content from something more engaging than an administration table of database entries.

This month, to create a set of utilities that can help people with their everyday needs, I decided to ask guys that work with me to make every part of the web sites that they were writing as pluggable applications. The result of this experiment has been that most of the pluggable apps that I did in my spare time (tgapp-smallpress, tgapp-photos, tgapp-tgcomments, tgext.tagging and so on) ended being used in real world projects and started to improve exposing hooks and ways to customize their behavior for the project they were going to be used.

After a few weeks, new pluggables like tgapp-fbauth, tgapp-userprofile, tgapp-calendarevents, tgapp-fbcontest, tgapp-youtubevideo has seen light and developing the target application started becoming blazing fast: Just plug what you need and customize it.

Embracing this philosophy the last project I'm working on has an app_cfg.py file that looks like:

plug(base_config, 'tgext.debugbar', inventing=True)
plug(base_config, 'tgext.scss')
plug(base_config, 'tgext.browserlimit')
plug(base_config, 'registration')
plug(base_config, 'photos')
plug(base_config, 'smallpress', 'press', form='XXX.lib.forms.ArticleForm')
plug(base_config, 'tgcomments', allow_anonymous=False)
from XXX.lib.matches import MyKindOfEvent
plug(base_config, 'calendarevents', 'eventi', event_types=[MyKindOfEvent()])
replace_template(base_config, 'smallpress.templates.article', 
                              'XXX.templates.press.article')
replace_template(base_config, 'smallpress.templates.excerpt', 
                              'XXX.templates.press.excerpt')

Thanks to this our development process has really improved: whenever a developer finds a bug he just has to propose a patch for the target pluggable, whenever someone notices a missing index on a query he has just to add it to the given pluggable. All the websites under development improved like people were working on the same project.

While existing pluggables might be limited, buggy or slow I'm getting confident that they will continue to improve, and some day they will surpass whatever custom implementation I can think of. I think I'm going to heavily rely on pluggable applications for any future project sticking to only one rule: "make it opensource". This way, apart from probably helping other people, I'm also improving my own projects through other people feedbacks, bug reports and patches to the pluggables I used.

So, next time you have to start a new project give a look at the TurboGears CogBin and check if there is a pluggable application that looks like what you need. If you find any issue or find space for improvements just fork it and send a pull request, or send an email on the TurboGears Mailing List I'll do my best to address any reported issue thanking you for your feedbacks as I'm aware that you are actually improving any past and future project that relies on that pluggable.

08 Sep 2012 9:30pm GMT

23 Aug 2012

feedPlanet TurboGears

Michael Pedersen: Announcing TurboGears 2.2.0!

That's right, we finally did it! It's up right now, and if you follow the directions on http://www.turbogears.org/ you'll get 2.2.0.

I'd like to take a minute to thank Alessandro Molina in particular. I did what support I could, but this release really belongs to him. Please pass along congratulations and thanks to him. And if there's bugs, I'll take the blame myself for not doing more to help.

The full changelog is below.



Important Commit Messages:




Tickets Closed:
1 Improve documentation on authentication, identification and authorization
2 Include lazy translations in the documentation
11 Document SecureFormMixin from TW
19 DataGrid and pagination tutorial/​reference missing for TG2
21 sqlite db creation error in wiki20 tutorial
26 Documentation for app_globals unclear
27 Document webflash aka tg.flash
51 Turbogears 2.1b1 tgext.admin Doesn't Handle Boolean Options
57 add a --clean option to quickstart
68 Simplify database migration in TG2
70 Migrating turbogears to pip
71 Create some kind of "component" architecture for TG2
75 quickstart graphics aren't easy to modify
137 Proper use of url() for widgets in the documentation
138 Backport repoze.what-quickstart and repoze.what-pylonshq removal
139 Move dispatch to Crank
140 Verify quickstart and devtools dependency
149 Expose the tg.render.render function as render_template
150 Full ToscaWidgets2 support and make it default when quickstarting app
151 Minimal Search and Sorting functions in admin
152 Jinja Support in quickstart
153 Master /​ Slave database support
154 Merge autoreload options for template engines
156 Provide a way to quickstart with Kajiki
157 repoze.who-1.0 is bugged when using auth_tkt
158 Permit to inherit decorations on TGController subclasses
160 allow_only doesn't get checked in dynamically dispatched RestController


23 Aug 2012 11:15pm GMT

16 Aug 2012

feedPlanet TurboGears

Alessandro Molina: What’s new about Sprox 0.8

Today Sprox 0.8 got released, it is the first release to add ToscaWidgets2 support. Depending on which version of ToscaWidgets is available inside your environment Sprox will either use TW1 or TW2 to generate its forms.

Being mostly a TW2 oriented release it might seem that not a lot changed since the previous version, but a little gem is hidden between all the TW2 changes as Sprox now supports setting default behavior for models themselves using the __sprox__ attribute inside model declaration.

class Parent(DeclarativeBase):
    __tablename__ = 'parents'
 
    uid = Column(Integer, primary_key=True)
    data = Column(String(100))
 
class Child(DeclarativeBase):
    __tablename__ = 'children'
 
    class __sprox__(object):
        dropdown_field_names = {'owner': ['data']}
 
    uid = Column(Integer, primary_key=True)
    data = Column(String(100))
 
    owner_id = Column(Integer, ForeignKey(Parent.uid))
    owner = relation('Parent')

The previous code example makes Sprox use the Parent data field for selection fields when choosing the parent of Child entities.

Apart from making easier to share options between your AddRecordForm and EditableForm __sprox__ attribute opens a great way to customize the TurboGears admin.

By adding a __sprox__ attribute inside your models you will be able to change the TurboGears admin behavior without having to create a custom admin configuration. Setting __sprox__ attribute makes possible to change most sprox properties changing CrudRestController behavior, the same properties that are documented on sprox.org can be specified inside the __sprox__ attribute by simply removing the underscores.

16 Aug 2012 7:53pm GMT