05 Aug 2015

feedPlanet Python

Mauveweb: Get a Kanban! (or Scrum board)

I continue to be staggered at the effectiveness, as a process management technique, of simply sticking cards representing tasks onto a whiteboard. Whatever your industry or project management methodology, the ability it offers to visualise the flow of work is immensely powerful. It lets us plan the current work and the near future to maximise our productivity.

/2015/kanban-example.png

It's valuable whether you're working on your own or working as a team. When working as a team, it can be used to schedule work among team members. When on your own, it merely helps with clarity of thought (we'll look at why a little later).

Yet this is largely unknown outside of software development. All sorts of industries would benefit from this approach, from farming to law.

Terminology

There's lots of variation in the terminology around kanbans, so let me lay out the terms as I use them.

The idea of a kanban originates in manufacturing in Japan. The word itself means sign board and refers to the board itself. Specific processes built around a kanban are called kanban methodologies. Scrum calls the kanban a "Scrum Board" and naturally there are all manner of other terms and practices for using a similar approach in other methodologies too.

Onto the kanban we need to stick cards representing tasks - small pieces of work that are easy to pick up and get done. Sometimes tasks will relate to bigger projects. Some call these bigger projects epics, and may use additional cards to represent the relationship of tasks to epics.

A backlog is the totality of the work yet to do (and again, terms differ; some practices may exclude work that is already scheduled).

How to run a kanban

First of all, get yourself a real, physical whiteboard. If you can get a magnetic whiteboard, you can stick task cards to it with magnets, which is nice and clean. But otherwise your tasks can be cards stuck to the board with blu-tak, or post-it notes. I favour index cards of a weighty paper density, about the size of your hand when flat. This lets you write large, clear letters on them, which are easier to see from a distance, and they are somewhat resistant to being scuffed as you stack them into a deck and riffle through it.

Next, you need to come up with your backlog. If you're in the middle of a piece of work, you can start by braindumping the current state. Otherwise, get into a quiet room, with the appropriate people if necessary, and a stack of index cards, and write out cards, or break them down, or tear them up, until you have a set of concrete tasks that will get you to your goal. Make sure everyone agrees the cards are correct.

The cards can include all kinds of extra information that will help you plan the work. For example, you might include deadlines or an estimate (in hours, days or your own unit - I like "ideal hours").

/2015/kanban-task-metadata.png

Sometimes tasks are easy to describe on a card but if you were to pick up the card as something to work on, it wouldn't be immediately obvious where to start. These should be broken down into smaller pieces of work during this planning phase. This allows you to see with better granularity how much of the large piece of work is done. I like tasks that are of an appropriate size for each person to do several of them in a week. However, it's OK to break down the card into smaller tasks later if the task is probably going to be something to tackle further in the future.

Now, divide the whiteboard into columns. You will need at least two: something like backlog, and in progress. But you could have many more. Kanban is about flow. Tasks flow through the columns. The flow represents the phases of working on a task. You might start by quoting for work and finish by billing for it. Or you might start by making sure you have all the raw materials required and finish by taking inventory of materials used.

/2015/kanban-flow.png

None of these practices are set in stone - you can select them and reselect them as your practices evolve. For example, you could focus on longer-range planning:

/2015/kanban-schedule.png

So with your whiteboard drawn, you can put your tasks on the board. Naturally many of your cards may not fit, so you can keep your backlog stack somewhere else. Choosing what to put on the board becomes important.

Now, just move the cards to reflect the current state. When a task is done, you update the board and choose the next most valuable task to move forward. You might put initials by a card to indicate who is working on it.

Visit the kanban regularly, as a team. Stop and replan frequently - anything from a couple of times a week up to a couple of times a day - especially when new information becomes available. This might involve pulling cards from the backlog onto the board, writing new cards, tearing up cards that have become redundant, and rearranging the board to reprioritise. Make sure the right people are present every time if possible.

Less frequently you might make a bigger planning effort: pick up all the cards from your backlog pile or column, and sit down again with the relevant people to replan these and reassess all their priorities. Some of the cards may be redundant and some new work may have been identified.

The value of the kanban will then naturally begin to flow:

  • Higher productivity as you see how what you're working on fits into a whole
  • A greater ability to reschedule - for example, to park work in progress to tackle something urgent
  • Team collaboration around tasks that seem to be problematic
  • Estimates of when something might get done or which deadines are at risk

Tips

A physical whiteboard seems to be very important. A lot of the practices don't seem to evolve properly if you use some sort of digital version of a kanban. There are lots of reasons for this. One obvious one is that physical whiteboards offer the ability to annotate the kanban with little hints, initials, or whatever. Another one is that an online whiteboard doesn't beg to be looked at; a physical whiteboard up in your workplace is something to notice frequently, as well as offer a designated place to get away from a screen and plan work.

Naturally, having a physical whiteboard is only possible if your team is not geographically distributed. Geographically distributed teams are challenging for a whole host of reasons, and this is just one. A digital version of a kanban may be a good approach in those cases. Or perhaps frequent photos of a physical whiteboard elsewhere in the world can help to keep things in sync.

Readability from a distance helps get value from your kanban. Write in capital letters because these are more readable from a distance. Use a broad felt pen. Use differently coloured index cards or magnets to convey additional information.

It's somewhat important to ensure that the kanban captures all streams of work. There's a tendency to think "This isn't part of the project we're planning; let's not get distracted by it". But that reduces the value of the kanban in tracking what is actually happening in your workflow. Obviously, different streams of work can be put in a different place on the kanban, or use differently coloured cards.

You can also track obstacles to delivering work on the board. I like to reserve red cards to indicate obstacles. Removing those obstacles may require work!

Why Kanbans work

Kanbans are certainly a form of process visualisation. Enabling you to visualise how tasks are flowing will let you spot problems in the process, such as too much work building up that only a certain team member can do. You can design workarounds to a problem like this also right there on the kanban.

Stepping back from this, the reason I've found having a kanban useful even for solo work may be related to the psychological idea of transactive memory, where we use our memory not as a primary store of information, but as an index over other stores of information, such as those in other people's heads, or on paper. The model of thought is then very much like a database transaction - we might "read" a number of facts from different sources into working memory, generate some new insight, and "write" that insight back to an external source.

By committing our understanding of our backlog of work to index cards, we can free our memories to focus on the task at hand. And when that task is done, we waste no time in switching back to a view of our workflow that can tell us immediately "what's next". Or say we encounter new information that we suspect affects something in the backlog - being able to go straight back to that card and recover exactly how we defined the task turns out to be useful: it allows us to quickly assess the impact of new information to our existing ideas and plans.

The final reason I believe kanbans work so well is that both the kanban and the stack of cards that represent your backlog are artifacts that are constructed collaboratively in a group. Taking some concrete artifact out of a meeting as a record of what was said cuts down a lot on misremembered conclusions afterwards. Some people try to take "action points" out of meetings for the same reason, and then quote them back to everyone by e-mail afterwards. This doesn't seem to work as well - I often find myself thinking "I don't recall agreeing that!" One reason for this is that the record of the action points is not written down for all to see and approve/veto, but a personal list written by the person taking the minutes.

Writing tasks out on index cards in front of people, and reading them out repeatedly or handing them around (or laying them out on the table for people to move around and reorganise - related in principle to CRC Cards), means that everyone gets a chance to internalise or reject the wording on the card.

Similarly, the organisation of kanban is not only a concrete artifact that is modified with other people standing around: it is ever-present to consult and correct. Nobody can have an excuse to leave the kanban in an incorrect state. Thus the kanban is a reliable source of truth.

So whatever your industry, whatever your process methodology, set yourself up a kanban and give it a try. Happy kanbanning!

05 Aug 2015 12:22am GMT

Mauveweb: Get a Kanban! (or Scrum board)

I continue to be staggered at the effectiveness, as a process management technique, of simply sticking cards representing tasks onto a whiteboard. Whatever your industry or project management methodology, the ability it offers to visualise the flow of work is immensely powerful. It lets us plan the current work and the near future to maximise our productivity.

/2015/kanban-example.png

It's valuable whether you're working on your own or working as a team. When working as a team, it can be used to schedule work among team members. When on your own, it merely helps with clarity of thought (we'll look at why a little later).

Yet this is largely unknown outside of software development. All sorts of industries would benefit from this approach, from farming to law.

Terminology

There's lots of variation in the terminology around kanbans, so let me lay out the terms as I use them.

The idea of a kanban originates in manufacturing in Japan. The word itself means sign board and refers to the board itself. Specific processes built around a kanban are called kanban methodologies. Scrum calls the kanban a "Scrum Board" and naturally there are all manner of other terms and practices for using a similar approach in other methodologies too.

Onto the kanban we need to stick cards representing tasks - small pieces of work that are easy to pick up and get done. Sometimes tasks will relate to bigger projects. Some call these bigger projects epics, and may use additional cards to represent the relationship of tasks to epics.

A backlog is the totality of the work yet to do (and again, terms differ; some practices may exclude work that is already scheduled).

How to run a kanban

First of all, get yourself a real, physical whiteboard. If you can get a magnetic whiteboard, you can stick task cards to it with magnets, which is nice and clean. But otherwise your tasks can be cards stuck to the board with blu-tak, or post-it notes. I favour index cards of a weighty paper density, about the size of your hand when flat. This lets you write large, clear letters on them, which are easier to see from a distance, and they are somewhat resistant to being scuffed as you stack them into a deck and riffle through it.

Next, you need to come up with your backlog. If you're in the middle of a piece of work, you can start by braindumping the current state. Otherwise, get into a quiet room, with the appropriate people if necessary, and a stack of index cards, and write out cards, or break them down, or tear them up, until you have a set of concrete tasks that will get you to your goal. Make sure everyone agrees the cards are correct.

The cards can include all kinds of extra information that will help you plan the work. For example, you might include deadlines or an estimate (in hours, days or your own unit - I like "ideal hours").

/2015/kanban-task-metadata.png

Sometimes tasks are easy to describe on a card but if you were to pick up the card as something to work on, it wouldn't be immediately obvious where to start. These should be broken down into smaller pieces of work during this planning phase. This allows you to see with better granularity how much of the large piece of work is done. I like tasks that are of an appropriate size for each person to do several of them in a week. However, it's OK to break down the card into smaller tasks later if the task is probably going to be something to tackle further in the future.

Now, divide the whiteboard into columns. You will need at least two: something like backlog, and in progress. But you could have many more. Kanban is about flow. Tasks flow through the columns. The flow represents the phases of working on a task. You might start by quoting for work and finish by billing for it. Or you might start by making sure you have all the raw materials required and finish by taking inventory of materials used.

/2015/kanban-flow.png

None of these practices are set in stone - you can select them and reselect them as your practices evolve. For example, you could focus on longer-range planning:

/2015/kanban-schedule.png

So with your whiteboard drawn, you can put your tasks on the board. Naturally many of your cards may not fit, so you can keep your backlog stack somewhere else. Choosing what to put on the board becomes important.

Now, just move the cards to reflect the current state. When a task is done, you update the board and choose the next most valuable task to move forward. You might put initials by a card to indicate who is working on it.

Visit the kanban regularly, as a team. Stop and replan frequently - anything from a couple of times a week up to a couple of times a day - especially when new information becomes available. This might involve pulling cards from the backlog onto the board, writing new cards, tearing up cards that have become redundant, and rearranging the board to reprioritise. Make sure the right people are present every time if possible.

Less frequently you might make a bigger planning effort: pick up all the cards from your backlog pile or column, and sit down again with the relevant people to replan these and reassess all their priorities. Some of the cards may be redundant and some new work may have been identified.

The value of the kanban will then naturally begin to flow:

  • Higher productivity as you see how what you're working on fits into a whole
  • A greater ability to reschedule - for example, to park work in progress to tackle something urgent
  • Team collaboration around tasks that seem to be problematic
  • Estimates of when something might get done or which deadines are at risk

Tips

A physical whiteboard seems to be very important. A lot of the practices don't seem to evolve properly if you use some sort of digital version of a kanban. There are lots of reasons for this. One obvious one is that physical whiteboards offer the ability to annotate the kanban with little hints, initials, or whatever. Another one is that an online whiteboard doesn't beg to be looked at; a physical whiteboard up in your workplace is something to notice frequently, as well as offer a designated place to get away from a screen and plan work.

Naturally, having a physical whiteboard is only possible if your team is not geographically distributed. Geographically distributed teams are challenging for a whole host of reasons, and this is just one. A digital version of a kanban may be a good approach in those cases. Or perhaps frequent photos of a physical whiteboard elsewhere in the world can help to keep things in sync.

Readability from a distance helps get value from your kanban. Write in capital letters because these are more readable from a distance. Use a broad felt pen. Use differently coloured index cards or magnets to convey additional information.

It's somewhat important to ensure that the kanban captures all streams of work. There's a tendency to think "This isn't part of the project we're planning; let's not get distracted by it". But that reduces the value of the kanban in tracking what is actually happening in your workflow. Obviously, different streams of work can be put in a different place on the kanban, or use differently coloured cards.

You can also track obstacles to delivering work on the board. I like to reserve red cards to indicate obstacles. Removing those obstacles may require work!

Why Kanbans work

Kanbans are certainly a form of process visualisation. Enabling you to visualise how tasks are flowing will let you spot problems in the process, such as too much work building up that only a certain team member can do. You can design workarounds to a problem like this also right there on the kanban.

Stepping back from this, the reason I've found having a kanban useful even for solo work may be related to the psychological idea of transactive memory, where we use our memory not as a primary store of information, but as an index over other stores of information, such as those in other people's heads, or on paper. The model of thought is then very much like a database transaction - we might "read" a number of facts from different sources into working memory, generate some new insight, and "write" that insight back to an external source.

By committing our understanding of our backlog of work to index cards, we can free our memories to focus on the task at hand. And when that task is done, we waste no time in switching back to a view of our workflow that can tell us immediately "what's next". Or say we encounter new information that we suspect affects something in the backlog - being able to go straight back to that card and recover exactly how we defined the task turns out to be useful: it allows us to quickly assess the impact of new information to our existing ideas and plans.

The final reason I believe kanbans work so well is that both the kanban and the stack of cards that represent your backlog are artifacts that are constructed collaboratively in a group. Taking some concrete artifact out of a meeting as a record of what was said cuts down a lot on misremembered conclusions afterwards. Some people try to take "action points" out of meetings for the same reason, and then quote them back to everyone by e-mail afterwards. This doesn't seem to work as well - I often find myself thinking "I don't recall agreeing that!" One reason for this is that the record of the action points is not written down for all to see and approve/veto, but a personal list written by the person taking the minutes.

Writing tasks out on index cards in front of people, and reading them out repeatedly or handing them around (or laying them out on the table for people to move around and reorganise - related in principle to CRC Cards), means that everyone gets a chance to internalise or reject the wording on the card.

Similarly, the organisation of kanban is not only a concrete artifact that is modified with other people standing around: it is ever-present to consult and correct. Nobody can have an excuse to leave the kanban in an incorrect state. Thus the kanban is a reliable source of truth.

So whatever your industry, whatever your process methodology, set yourself up a kanban and give it a try. Happy kanbanning!

05 Aug 2015 12:22am GMT

04 Aug 2015

feedPlanet Python

Mike Driscoll: Python 101 Screencast Available for Pre-Order

The Python 101 Screencast is now available for Pre-Order. If you pre-order the screencast series, then you will receive what I currently have finished (12 videos + the eBook) and then receive updates as I add new ones. There will be a minimum of 44 videos. Upon purchase, you will be able to stream or download the videos at any time.

The screencasts are based off my book, Python 101. Each screencast is based on a chapter from the book. The first 11 videos are available free of charge so you can try-before-you-buy! You can check them out on Youtube here.

04 Aug 2015 4:44pm GMT

Mike Driscoll: Python 101 Screencast Available for Pre-Order

The Python 101 Screencast is now available for Pre-Order. If you pre-order the screencast series, then you will receive what I currently have finished (12 videos + the eBook) and then receive updates as I add new ones. There will be a minimum of 44 videos. Upon purchase, you will be able to stream or download the videos at any time.

The screencasts are based off my book, Python 101. Each screencast is based on a chapter from the book. The first 11 videos are available free of charge so you can try-before-you-buy! You can check them out on Youtube here.

04 Aug 2015 4:44pm GMT

Daily Tech Video (Python): [Video 254] Raymond Hettinger: Keynote — What makes Python awesome?

Python is popular, and is rapidly growing in popularity - not what you might expect from a 25-year-old language. What is it that makes Python so special? Why are so many people, in so many places adopting Python? In this talk, Raymond Hettinger describes the features of the Python language, ecosystem, and community that combine to make it such a compelling choice for universities, companies, and individuals.

The post [Video 254] Raymond Hettinger: Keynote - What makes Python awesome? appeared first on Daily Tech Video.

04 Aug 2015 1:32pm GMT

Daily Tech Video (Python): [Video 254] Raymond Hettinger: Keynote — What makes Python awesome?

Python is popular, and is rapidly growing in popularity - not what you might expect from a 25-year-old language. What is it that makes Python so special? Why are so many people, in so many places adopting Python? In this talk, Raymond Hettinger describes the features of the Python language, ecosystem, and community that combine to make it such a compelling choice for universities, companies, and individuals.

The post [Video 254] Raymond Hettinger: Keynote - What makes Python awesome? appeared first on Daily Tech Video.

04 Aug 2015 1:32pm GMT

eGenix.com: eGenix Talks & Videos: Python Idioms Talk

EuroPython 2015 in Bilbao, Basque Country, Spain

Marc-André Lemburg, Python Core Developer, one of the EuroPython 2015 organizers and Senior Software Architect, held a talk at EuroPython focusing on programmers just starting with Python.

We have now turned the talk into a video presentations for easy viewing and also released the presentation slides.

Python Idioms to help you write good code

Talk given at the EuroPython 2015 conference in Bilbao, Basque Country, Spain, presenting Python idioms which are especially useful for programmers new to Python.

Click to proceed to the talk video and slides ...

Python focuses a lot on writing readable code and also tries to make solutions obvious, but this doesn't necessarily mean that you cannot write unreadable code or design your code in ways which makes it hard to extend or maintain.

The talk shows some useful idioms to apply when writing Python code, how to structure your modules and also goes into details on which techniques to use and which to think about twice, based on 20 years of experience writing Python.

-- Marc-André Lemburg

More interesting eGenix presentations are available in the presentations and talks community section of our website.

Related Python Coaching and Consulting

If you are interested in learning more about these advanced techniques, eGenix now offers Python project coaching and consulting services to give your project teams advice on how to design Python applications, successfully run projects, or find excellent Python programmers. Please contact our eGenix Sales Team for information.

Enjoy !

Charlie Clark, eGenix.com Sales & Marketing

04 Aug 2015 8:00am GMT

eGenix.com: eGenix Talks & Videos: Python Idioms Talk

EuroPython 2015 in Bilbao, Basque Country, Spain

Marc-André Lemburg, Python Core Developer, one of the EuroPython 2015 organizers and Senior Software Architect, held a talk at EuroPython focusing on programmers just starting with Python.

We have now turned the talk into a video presentations for easy viewing and also released the presentation slides.

Python Idioms to help you write good code

Talk given at the EuroPython 2015 conference in Bilbao, Basque Country, Spain, presenting Python idioms which are especially useful for programmers new to Python.

Click to proceed to the talk video and slides ...

Python focuses a lot on writing readable code and also tries to make solutions obvious, but this doesn't necessarily mean that you cannot write unreadable code or design your code in ways which makes it hard to extend or maintain.

The talk shows some useful idioms to apply when writing Python code, how to structure your modules and also goes into details on which techniques to use and which to think about twice, based on 20 years of experience writing Python.

-- Marc-André Lemburg

More interesting eGenix presentations are available in the presentations and talks community section of our website.

Related Python Coaching and Consulting

If you are interested in learning more about these advanced techniques, eGenix now offers Python project coaching and consulting services to give your project teams advice on how to design Python applications, successfully run projects, or find excellent Python programmers. Please contact our eGenix Sales Team for information.

Enjoy !

Charlie Clark, eGenix.com Sales & Marketing

04 Aug 2015 8:00am GMT

Talk Python to Me: #19 Automate the Boring Stuff with Python

Some of the things we do in life are tedious and boring. It's the kind of thing that machines or robots could do. So let's build those machines! <more></more> This week we talk Al Sweigart, the author of Automating the Boring Stuff. You'll learn how he hopes to engage and teach Python to a unique and broad segment of the population. We'll discuss why, at first, it might make more sense to keep things simple rather than insisting on the "right" patterns and best practices. Links from the show: <div style="font-size: .85em;"> <b>Book: Automate the boring stuff</b>: <a href='https://automatetheboringstuff.com/' target='_blank'>automatetheboringstuff.com</a> <b>Invent with Python</b>: <a href='http://inventwithpython.com/' target='_blank'>inventwithpython.com</a> <b>Al at Github</b>: <a href='https://github.com/asweigart' target='_blank'>github.com/asweigart</a> <b>@alsweigart</b>: <a href='https://twitter.com/alsweigart' target='_blank'>twitter.com/alsweigart</a> </div>

04 Aug 2015 8:00am GMT

Talk Python to Me: #19 Automate the Boring Stuff with Python

Some of the things we do in life are tedious and boring. It's the kind of thing that machines or robots could do. So let's build those machines! <more></more> This week we talk Al Sweigart, the author of Automating the Boring Stuff. You'll learn how he hopes to engage and teach Python to a unique and broad segment of the population. We'll discuss why, at first, it might make more sense to keep things simple rather than insisting on the "right" patterns and best practices. Links from the show: <div style="font-size: .85em;"> <b>Book: Automate the boring stuff</b>: <a href='https://automatetheboringstuff.com/' target='_blank'>automatetheboringstuff.com</a> <b>Invent with Python</b>: <a href='http://inventwithpython.com/' target='_blank'>inventwithpython.com</a> <b>Al at Github</b>: <a href='https://github.com/asweigart' target='_blank'>github.com/asweigart</a> <b>@alsweigart</b>: <a href='https://twitter.com/alsweigart' target='_blank'>twitter.com/alsweigart</a> </div>

04 Aug 2015 8:00am GMT

Mauveweb: Pygame Zero 1.1 is out!

Pygame Zero 1.1 is released today! Pygame Zero is a zero-boilerplate games programming framework for education.

This release brings a number of bug fixes and usability enhancements, as well as one or two new features. The full changelog is available in the documentation, but here are a couple of the highlights for me:

  • A new spell checker will point out hook or parameter names that have been misspelled when the program starts. This goes towards helping beginner programmers understand what they have done wrong in cases where normally no feedback would be given.
  • We fixed a really bad bug on Mac OS X where Pygame Zero's window can't be focused when it is installed in a virtualenv.
  • Various contributors have contributed open-source implementations of classic retro games using Pygame Zero. This is an ongoing project, but there are now implementations of Snake, Pong, Lunar Lander and Minesweeper included in the examples/ directory. These can be used as a reference or turned into course material for teaching with Pygame Zero.

Pygame Zero was well-received at Europython. Carrie-Anne Philbin covered Pygame Zero in her keynote; I gave a lightning talk introducing the library and its new spellchecker features; and the sprints introduced several new collaborators to the project, who worked to deliver several of the features and bugfixes that are being released today.

A big thank you to everyone who helped make this release happen!

04 Aug 2015 7:33am GMT

Mauveweb: Pygame Zero 1.1 is out!

Pygame Zero 1.1 is released today! Pygame Zero is a zero-boilerplate games programming framework for education.

This release brings a number of bug fixes and usability enhancements, as well as one or two new features. The full changelog is available in the documentation, but here are a couple of the highlights for me:

  • A new spell checker will point out hook or parameter names that have been misspelled when the program starts. This goes towards helping beginner programmers understand what they have done wrong in cases where normally no feedback would be given.
  • We fixed a really bad bug on Mac OS X where Pygame Zero's window can't be focused when it is installed in a virtualenv.
  • Various contributors have contributed open-source implementations of classic retro games using Pygame Zero. This is an ongoing project, but there are now implementations of Snake, Pong, Lunar Lander and Minesweeper included in the examples/ directory. These can be used as a reference or turned into course material for teaching with Pygame Zero.

Pygame Zero was well-received at Europython. Carrie-Anne Philbin covered Pygame Zero in her keynote; I gave a lightning talk introducing the library and its new spellchecker features; and the sprints introduced several new collaborators to the project, who worked to deliver several of the features and bugfixes that are being released today.

A big thank you to everyone who helped make this release happen!

04 Aug 2015 7:33am GMT

David MacIver: Hypothesis 1.10.0 is out

The general consensus when I asked about this, which was conveniently what I was already planning to do, was to save Hypothesis 2.0 for a big release where I wanted to make some significant changes and was happy to cut the deprecated APIs and such at that point. So for now we're continuing on with the 1.x series.

This release actually doesn't have any new features externally. I've introduced a new 'cleanup' function, which lets you register cleanup handlers that execute at the end of your tests (this is more useful inside strategies than it is inside tests, but you can use it in either), but its semantics as it interacts with find are a bit weird and so I'm not currently declaring it part of the public API.

Other than that, this is mostly a bug fix and performance release. It was supposed to be a small release which just contained the performance improvements to one_of that I blogged about previously, but it got held up by a pypy bug (I believe this is the first time that's happened - previous bugs have been easy to diagnose and work around) and then snowballed into a bit more significant of a release.

Also, reminder that I have a Patreon now, and that if you like this release schedule some support for it would be appreciated.

This is just a bugfix and performance release, but it changes some semi-public APIs, hence the minor version bump.

  • Significant performance improvements for strategies which are one_of() many branches. In particular this included recursive() strategies. This should take the case where you use one recursive() strategy as the base strategy of another from unusably slow (tens of seconds per generated example) to reasonably fast.
  • Better handling of just() and sampled_from() for values which have an incorrect __repr__ implementation that returns non-ASCII unicode on Python 2.
  • Better performance for flatmap from changing the internal morpher API to be significantly less general purpose.
  • Introduce a new semi-public BuildContext/cleanup API. This allows strategies to register cleanup activities that should run once the example is complete. Note that this will interact somewhat weirdly with find.
  • Better simplification behaviour for streaming strategies.
  • Don't error on lambdas which use destructuring arguments in Python 2.
  • Add some better reprs for a few strategies that were missing good ones.
  • The Random instances provided by randoms() are now copyable.
  • Slightly more debugging information about simplify when using a debug verbosity level.
  • Support using given for functions with varargs, but not passing arguments to it as positional.

04 Aug 2015 6:52am GMT

David MacIver: Hypothesis 1.10.0 is out

The general consensus when I asked about this, which was conveniently what I was already planning to do, was to save Hypothesis 2.0 for a big release where I wanted to make some significant changes and was happy to cut the deprecated APIs and such at that point. So for now we're continuing on with the 1.x series.

This release actually doesn't have any new features externally. I've introduced a new 'cleanup' function, which lets you register cleanup handlers that execute at the end of your tests (this is more useful inside strategies than it is inside tests, but you can use it in either), but its semantics as it interacts with find are a bit weird and so I'm not currently declaring it part of the public API.

Other than that, this is mostly a bug fix and performance release. It was supposed to be a small release which just contained the performance improvements to one_of that I blogged about previously, but it got held up by a pypy bug (I believe this is the first time that's happened - previous bugs have been easy to diagnose and work around) and then snowballed into a bit more significant of a release.

Also, reminder that I have a Patreon now, and that if you like this release schedule some support for it would be appreciated.

This is just a bugfix and performance release, but it changes some semi-public APIs, hence the minor version bump.

  • Significant performance improvements for strategies which are one_of() many branches. In particular this included recursive() strategies. This should take the case where you use one recursive() strategy as the base strategy of another from unusably slow (tens of seconds per generated example) to reasonably fast.
  • Better handling of just() and sampled_from() for values which have an incorrect __repr__ implementation that returns non-ASCII unicode on Python 2.
  • Better performance for flatmap from changing the internal morpher API to be significantly less general purpose.
  • Introduce a new semi-public BuildContext/cleanup API. This allows strategies to register cleanup activities that should run once the example is complete. Note that this will interact somewhat weirdly with find.
  • Better simplification behaviour for streaming strategies.
  • Don't error on lambdas which use destructuring arguments in Python 2.
  • Add some better reprs for a few strategies that were missing good ones.
  • The Random instances provided by randoms() are now copyable.
  • Slightly more debugging information about simplify when using a debug verbosity level.
  • Support using given for functions with varargs, but not passing arguments to it as positional.

04 Aug 2015 6:52am GMT

Vasudev Ram: Converting numeric strings to integers with handrolled code


By Vasudev Ram -


A while ago, I was explaining to someone, how different data types such as numbers and strings are represented in a computer. That made me think of writing a simple program, called str_to_int.py, similar to Python's built-in int() function, to show them roughly what steps are involved in the process of converting a numeric string, such as '123', to an integer.

There are many different solutions possible for this task, and some may be more efficient or simpler than others, of course. This was just my first quick attempt at it (in Python, because I had no need to do it earlier, though I've done it before in assembly language and in C; in C it would be like the atoi() function in the C standard library). Note that this program does not handle all the cases that Python's int() does. It's just meant to show the basics of the process.

Here is the source code for str_to_int.py:

def str_to_int(s):
ctr = i = 0
for c in reversed(s):
i += (ord(c) - 48) * (10 ** ctr)
ctr += 1
return i

print
for s in ('0', '1', '2', '3', '12', '123', '234', '456', '567'):
i = str_to_int(s)
print "s = {}, i = {} |".format(s, i),

print
print

for i in range(50):
s = str(i)
j = str_to_int(s)
print "s = {}, j = {} |".format(s, j),

And here is its output:

$ py str_to_int.py

s = 0, i = 0 | s = 1, i = 1 | s = 2, i = 2 | s = 3, i = 3 | s = 12, i = 12 | s =
123, i = 123 | s = 234, i = 234 | s = 456, i = 456 | s = 567, i = 567 |

s = 0, j = 0 | s = 1, j = 1 | s = 2, j = 2 | s = 3, j = 3 | s = 4, j = 4 | s = 5
, j = 5 | s = 6, j = 6 | s = 7, j = 7 | s = 8, j = 8 | s = 9, j = 9 | s = 10, j
= 10 | s = 11, j = 11 | s = 12, j = 12 | s = 13, j = 13 | s = 14, j = 14 | s = 1
5, j = 15 | s = 16, j = 16 | s = 17, j = 17 | s = 18, j = 18 | s = 19, j = 19 |
s = 20, j = 20 | s = 21, j = 21 | s = 22, j = 22 | s = 23, j = 23 | s = 24, j =
24 | s = 25, j = 25 | s = 26, j = 26 | s = 27, j = 27 | s = 28, j = 28 | s = 29,
j = 29 | s = 30, j = 30 | s = 31, j = 31 | s = 32, j = 32 | s = 33, j = 33 | s
= 34, j = 34 | s = 35, j = 35 | s = 36, j = 36 | s = 37, j = 37 | s = 38, j = 38
| s = 39, j = 39 | s = 40, j = 40 | s = 41, j = 41 | s = 42, j = 42 | s = 43, j
= 43 | s = 44, j = 44 | s = 45, j = 45 | s = 46, j = 46 | s = 47, j = 47 | s =
48, j = 48 | s = 49, j = 49 |


To get the documentation for int(), you can do this:

>>> print int.__doc__

which gives this as the output:

int(x=0) -> int or long
int(x, base=10) -> int or long

Convert a number or string to an integer, or return 0 if no arguments
are given. If x is floating point, the conversion truncates towards zero.
If x is outside the integer range, the function returns a long instead.

If x is not a number or if base is given, then x must be a string or
Unicode object representing an integer literal in the given base. The
literal can be preceded by '+' or '-' and be surrounded by whitespace.
The base defaults to 10. Valid bases are 0 and 2-36. Base 0 means to
interpret the base from the string as an integer literal.
>>> int('0b100', base=0)
4

Learn more about this overall topic at the Wikipedia article on numeral systems.
Also check out my earlier post about Bhaskaracharya and the man who found zero.

Kthxbye :)

Vasudev Ram - Online Python training and programming Dancing Bison EnterprisesSignup to hear about new Python products or services that I create. Posts about Python Posts about xtopdf Contact Page

Share |
Vasudev Ram

04 Aug 2015 4:05am GMT

Vasudev Ram: Converting numeric strings to integers with handrolled code


By Vasudev Ram -


A while ago, I was explaining to someone, how different data types such as numbers and strings are represented in a computer. That made me think of writing a simple program, called str_to_int.py, similar to Python's built-in int() function, to show them roughly what steps are involved in the process of converting a numeric string, such as '123', to an integer.

There are many different solutions possible for this task, and some may be more efficient or simpler than others, of course. This was just my first quick attempt at it (in Python, because I had no need to do it earlier, though I've done it before in assembly language and in C; in C it would be like the atoi() function in the C standard library). Note that this program does not handle all the cases that Python's int() does. It's just meant to show the basics of the process.

Here is the source code for str_to_int.py:

def str_to_int(s):
ctr = i = 0
for c in reversed(s):
i += (ord(c) - 48) * (10 ** ctr)
ctr += 1
return i

print
for s in ('0', '1', '2', '3', '12', '123', '234', '456', '567'):
i = str_to_int(s)
print "s = {}, i = {} |".format(s, i),

print
print

for i in range(50):
s = str(i)
j = str_to_int(s)
print "s = {}, j = {} |".format(s, j),

And here is its output:

$ py str_to_int.py

s = 0, i = 0 | s = 1, i = 1 | s = 2, i = 2 | s = 3, i = 3 | s = 12, i = 12 | s =
123, i = 123 | s = 234, i = 234 | s = 456, i = 456 | s = 567, i = 567 |

s = 0, j = 0 | s = 1, j = 1 | s = 2, j = 2 | s = 3, j = 3 | s = 4, j = 4 | s = 5
, j = 5 | s = 6, j = 6 | s = 7, j = 7 | s = 8, j = 8 | s = 9, j = 9 | s = 10, j
= 10 | s = 11, j = 11 | s = 12, j = 12 | s = 13, j = 13 | s = 14, j = 14 | s = 1
5, j = 15 | s = 16, j = 16 | s = 17, j = 17 | s = 18, j = 18 | s = 19, j = 19 |
s = 20, j = 20 | s = 21, j = 21 | s = 22, j = 22 | s = 23, j = 23 | s = 24, j =
24 | s = 25, j = 25 | s = 26, j = 26 | s = 27, j = 27 | s = 28, j = 28 | s = 29,
j = 29 | s = 30, j = 30 | s = 31, j = 31 | s = 32, j = 32 | s = 33, j = 33 | s
= 34, j = 34 | s = 35, j = 35 | s = 36, j = 36 | s = 37, j = 37 | s = 38, j = 38
| s = 39, j = 39 | s = 40, j = 40 | s = 41, j = 41 | s = 42, j = 42 | s = 43, j
= 43 | s = 44, j = 44 | s = 45, j = 45 | s = 46, j = 46 | s = 47, j = 47 | s =
48, j = 48 | s = 49, j = 49 |


To get the documentation for int(), you can do this:

>>> print int.__doc__

which gives this as the output:

int(x=0) -> int or long
int(x, base=10) -> int or long

Convert a number or string to an integer, or return 0 if no arguments
are given. If x is floating point, the conversion truncates towards zero.
If x is outside the integer range, the function returns a long instead.

If x is not a number or if base is given, then x must be a string or
Unicode object representing an integer literal in the given base. The
literal can be preceded by '+' or '-' and be surrounded by whitespace.
The base defaults to 10. Valid bases are 0 and 2-36. Base 0 means to
interpret the base from the string as an integer literal.
>>> int('0b100', base=0)
4

Learn more about this overall topic at the Wikipedia article on numeral systems.
Also check out my earlier post about Bhaskaracharya and the man who found zero.

Kthxbye :)

Vasudev Ram - Online Python training and programming Dancing Bison EnterprisesSignup to hear about new Python products or services that I create. Posts about Python Posts about xtopdf Contact Page

Share |
Vasudev Ram

04 Aug 2015 4:05am GMT

Twisted Matrix Labs: Twisted 15.3 Released

On behalf of Twisted Matrix Labs, I am honoured to announce the release of Twisted 15.3.

We're marching confidently into the future, and this release demonstrates this:

You can find the downloads at https://pypi.python.org/pypi/Twisted (or alternatively http://twistedmatrix.com/trac/wiki/Downloads). The NEWS file can be found at https://github.com/twisted/twisted/blob/trunk/NEWS.

Many thanks to everyone who had a part in this release - the supporters of the Twisted Software Foundation, the developers who contributed code as well as documentation, and all the people building great things with Twisted!

Twisted Regards,

Amber "Hawkie" Brown

04 Aug 2015 12:07am GMT

Twisted Matrix Labs: Twisted 15.3 Released

On behalf of Twisted Matrix Labs, I am honoured to announce the release of Twisted 15.3.

We're marching confidently into the future, and this release demonstrates this:

You can find the downloads at https://pypi.python.org/pypi/Twisted (or alternatively http://twistedmatrix.com/trac/wiki/Downloads). The NEWS file can be found at https://github.com/twisted/twisted/blob/trunk/NEWS.

Many thanks to everyone who had a part in this release - the supporters of the Twisted Software Foundation, the developers who contributed code as well as documentation, and all the people building great things with Twisted!

Twisted Regards,

Amber "Hawkie" Brown

04 Aug 2015 12:07am GMT

03 Aug 2015

feedPlanet Python

Gal Varoquaux: Nilearn sprint: hacking neuroimaging machine learning

A couple of weeks ago, we had in Paris the second international nilearn sprint, dedicated to making machine learning in neuroimaging easier and more powerful.

It was such a fantastic experience, as nilearn is really shaping up as a simple yet powerful tool, and there is a lot of enthusiasm. For me, this sprint is a turning point, as I could see people other than the original core team (that spanned out of our research team) excited about the project's future. Thank you to all who came:

The sprint was a joint sprint with the MNE-Python team, that makes MEG processing awesome. We also need to thank Alex Gramfort, who did most of the work to set up the sprint, as well as NeuroSaclay for funding, and La paillasse, Telecom, and INRIA for hosting.

Highlights of the sprints results

Plotting of multiple maps

A function to visualize overlays of various maps, eg for a probabilistic atlas, with defaults that try to adapt to the number of maps (see the example). It's very useful for example for easy visualizing of ICA components.

Sign of activation in glass brain

Our glass brain plotting was greatly improved adding amongst other things the option to capture the sign of the activation in the color (see this example).

Spatially-regularized decoder

Decoders based on GraphNet and total variation have finally landed in nilearn. This has required a lot of work to get fast convergence and robust parameter selection. At the end of the day, it is much slower than an SVM, but the maps look splendid (see this example).

Sparse dictionary learning

We have almost merged sparse dictionnary learning as a alternative to ICA. Experience shows that on resting-state data, it gives more contrasted segmentation of networks (see this example).

New installation docs

New webpage layout using tabs to display only the installation instruction relevant to the OS of the user (see here). The results are more compact and more clear instructions, that I hope will make our users' life easier.

CircleCI integration

We now use CircleCI to run the examples and build the docs. This is challenging because our examples are real cases of neuroimaging data analysis, and thus require heavy datasets and computing horse power.

Neurodebian packaging

There are now neurodebian packages for nilearn.

And much more!

Warning

Features listed above are not in the released version of nilearn. You need to wait a month or so.

03 Aug 2015 10:00pm GMT

Gal Varoquaux: Nilearn sprint: hacking neuroimaging machine learning

A couple of weeks ago, we had in Paris the second international nilearn sprint, dedicated to making machine learning in neuroimaging easier and more powerful.

It was such a fantastic experience, as nilearn is really shaping up as a simple yet powerful tool, and there is a lot of enthusiasm. For me, this sprint is a turning point, as I could see people other than the original core team (that spanned out of our research team) excited about the project's future. Thank you to all who came:

The sprint was a joint sprint with the MNE-Python team, that makes MEG processing awesome. We also need to thank Alex Gramfort, who did most of the work to set up the sprint, as well as NeuroSaclay for funding, and La paillasse, Telecom, and INRIA for hosting.

Highlights of the sprints results

Plotting of multiple maps

A function to visualize overlays of various maps, eg for a probabilistic atlas, with defaults that try to adapt to the number of maps (see the example). It's very useful for example for easy visualizing of ICA components.

Sign of activation in glass brain

Our glass brain plotting was greatly improved adding amongst other things the option to capture the sign of the activation in the color (see this example).

Spatially-regularized decoder

Decoders based on GraphNet and total variation have finally landed in nilearn. This has required a lot of work to get fast convergence and robust parameter selection. At the end of the day, it is much slower than an SVM, but the maps look splendid (see this example).

Sparse dictionary learning

We have almost merged sparse dictionnary learning as a alternative to ICA. Experience shows that on resting-state data, it gives more contrasted segmentation of networks (see this example).

New installation docs

New webpage layout using tabs to display only the installation instruction relevant to the OS of the user (see here). The results are more compact and more clear instructions, that I hope will make our users' life easier.

CircleCI integration

We now use CircleCI to run the examples and build the docs. This is challenging because our examples are real cases of neuroimaging data analysis, and thus require heavy datasets and computing horse power.

Neurodebian packaging

There are now neurodebian packages for nilearn.

And much more!

Warning

Features listed above are not in the released version of nilearn. You need to wait a month or so.

03 Aug 2015 10:00pm GMT

Robert Collins: The merits of (careful) impatience

The Python packaging ecosystem has long desired a overhaul and implementation of designed features, but it often stalls on adoption.

I think its time to propose a guiding principle for incremental change.

be carefully impatient

The cautious approach for delivering a new feature in the infrastructure looks like this:

  1. Design the change.
  2. Implement the change(s) needed, in a new major version (e.g Metadata-2.0).
  3. Wait for the new version to be the default everywhere.
  4. Tell users they can use it.

This is frankly terrible. Firstly, we cannot really identify 'default everywhere'. We can identify 'default in known distributions', but behind the firewall setups may lag arbitrarily far behind. Secondly, it makes the cycle time for getting user feedback extraordinarily long: decade plus time windows. Thirdly, as a consequence, we run a large risk of running ahead of our users and delivering less good fixes and improvements than we might do if they were using our latest things and giving us feedback.

So here is how I think we should deliver things instead:

  1. Design the change with specific care that it fails closed and is opt-in.
  2. Implement the change(s) needed, in a new minor version of the tools.
  3. Tell users they can use it.

So, why do I think we can skip waiting for it to be a default?

pip, wheel and setuptools are just as able to be updated as any other Python component. If someone is installing (say) numpy via pip (or easy-install), then by definition they are willing to use things from PyPI, and pip and setuptools are in that category.

And if they are not installing via pip, then the Python packaging ecosystem does not affect them.

If we have opt-in as a design principle, then the adoption process will be bottom up: projects that are willing to say to their users 'you need new versions of pip and setuptools' can do so, and use the feature immediately. Projects that want to support users installing their packages with pip but aren't willing to ask that they also upgrade their pip can hold off.

If we have fails-closed as a design principle, then when a project has opted in, and the user installing the package hasn't upgraded their pip, things will at least fail rather than silently doing the wrong thing.

I had experience of this in Mock recently: the 1.1.0 and up releases depended on setuptools 17.1. The minimum setuptools we could have depended on (while publishing wheels) was still newer than that in Ubuntu Precise (not to mention RHEL!), so we were forcing an upgrade regardless.

This worked ok but we had two significant issues. Firstly, folk with incorrect Python paths can end up shadowing system installed packages, and for some reason 'six' triggered this for multiple users. Secondly, we had a number of different attempts to clearly signal the dependency, as the new features we were using did not fail closed: they were silently ignored by sufficiently old setuptools.

We ended up with a setup_requires="setuptools>17.1" clause in setup.py, which we're hopeful will fail, or Just Work, consistently.


03 Aug 2015 9:50pm GMT

Robert Collins: The merits of (careful) impatience

The Python packaging ecosystem has long desired a overhaul and implementation of designed features, but it often stalls on adoption.

I think its time to propose a guiding principle for incremental change.

be carefully impatient

The cautious approach for delivering a new feature in the infrastructure looks like this:

  1. Design the change.
  2. Implement the change(s) needed, in a new major version (e.g Metadata-2.0).
  3. Wait for the new version to be the default everywhere.
  4. Tell users they can use it.

This is frankly terrible. Firstly, we cannot really identify 'default everywhere'. We can identify 'default in known distributions', but behind the firewall setups may lag arbitrarily far behind. Secondly, it makes the cycle time for getting user feedback extraordinarily long: decade plus time windows. Thirdly, as a consequence, we run a large risk of running ahead of our users and delivering less good fixes and improvements than we might do if they were using our latest things and giving us feedback.

So here is how I think we should deliver things instead:

  1. Design the change with specific care that it fails closed and is opt-in.
  2. Implement the change(s) needed, in a new minor version of the tools.
  3. Tell users they can use it.

So, why do I think we can skip waiting for it to be a default?

pip, wheel and setuptools are just as able to be updated as any other Python component. If someone is installing (say) numpy via pip (or easy-install), then by definition they are willing to use things from PyPI, and pip and setuptools are in that category.

And if they are not installing via pip, then the Python packaging ecosystem does not affect them.

If we have opt-in as a design principle, then the adoption process will be bottom up: projects that are willing to say to their users 'you need new versions of pip and setuptools' can do so, and use the feature immediately. Projects that want to support users installing their packages with pip but aren't willing to ask that they also upgrade their pip can hold off.

If we have fails-closed as a design principle, then when a project has opted in, and the user installing the package hasn't upgraded their pip, things will at least fail rather than silently doing the wrong thing.

I had experience of this in Mock recently: the 1.1.0 and up releases depended on setuptools 17.1. The minimum setuptools we could have depended on (while publishing wheels) was still newer than that in Ubuntu Precise (not to mention RHEL!), so we were forcing an upgrade regardless.

This worked ok but we had two significant issues. Firstly, folk with incorrect Python paths can end up shadowing system installed packages, and for some reason 'six' triggered this for multiple users. Secondly, we had a number of different attempts to clearly signal the dependency, as the new features we were using did not fail closed: they were silently ignored by sufficiently old setuptools.

We ended up with a setup_requires="setuptools>17.1" clause in setup.py, which we're hopeful will fail, or Just Work, consistently.


03 Aug 2015 9:50pm GMT

End Point: img.bi, a secret encrypted image sharing service tool

After a fairly good experience with dnote installed on our own servers as an encrypted notes sharing service, my team decided that it would have been nice to have a similar service for images.

We found a nice project called img.bi that is based on NodeJS, Python, Redis and a lot of client-side JavaScript.

The system is divided into two components: the HTML/JS frontend and a Python FastCGI API.

Unfortunately the documentation is a still in its very early stage and it's lacking a meaningful structure and a lot of needed information.

Here's an overview of the steps we followed to setup img.bi on our own server behind nginx.

First of all we chose that we wanted to have as much as possible running and confined to a regular user, which is always a good idea with such young and potentially vulnerable tools. We chose to use the imgbi user.

Then since we wanted to keep as clean as possible the root user environment (and system status), we also decided to use pyenv. To be conservative we chose the latest Python 2.7 stable release, 2.7.10.

git clone https://github.com/yyuu/pyenv.git ~/.pyenv
echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.bash_profile
echo 'export PATH="$PYENV_ROOT/bin:$PATH"' >> ~/.bash_profile
echo 'eval "$(pyenv init -)"' >> ~/.bash_profile
echo $SHELL -l
pyenv install -l  | grep 2\\.7
pyenv install 2.7.10
pyenv global 2.7.10
pyenv version
which python
python --version

In order to use img.bi, we also needed NodeJS and following the same approach we chose to use nvm and install the latest NodeJS stable version:

curl -o- https://raw.githubusercontent.com/creationix/nvm/v0.25.4/install.sh | bash
nvm install stable
nvm list
nvm use stable
nvm alias default stable
node --version

As a short note to the usage of the bad practice of blindly using:

curl -o- https://some_obscure_link_or_not | bash

We want to add that we do not endorse this practice as it's dangerous and exposes your system to many security risks. On the other hand, though, it's true that cloning the source via Git and compile/installing it blindly is not much safer, so it's always up to how much you trust the peer review on the project you're about to use. And at least with an https URL you should be talking to the destination you want, whereas an http URL is much more dangerous.

Furthermore going through the entire Python and NodeJS installation as a regular user, was far beyond the scope of this post and the steps proposed here assumes that you're doing everything as the regular user, except where specifically stated differently.

Anyway after that we updated pip and then installed all the needed Python modules:

pip install --upgrade pip
pip install redis m2crypto web.py bcrypt pysha3 zbase62 pyutil flup

Then it's time to clone the actual img.bi code from the GitHub repo, install a few missing dependencies and then use the bower and npm .json files to add the desired packages:

git clone https://github.com/imgbi/img.bi.git
cd img.bi/
npm install -g bower grunt grunt-cli grunt-multiresize
npm install -g grunt-webfont --save-dev
npm install
bower install

We also faced an issue which made Grunt fail to start correctly. Grunt was complaining about an "undefined property" called "prototype". If you happen to have the same problem just type

cd node_modules/grunt-connect-proxy/node_modules/http-proxy
npm install eventemitter3@0.1.6
cd -

That'll basically install the eventemitter3 NodeJS package module locally to the grunt-connect-proxy module so to overcome the compatibility issues which in turn causes the error mentioned above.

You should use your favourite editor to change the file config.json, which basically contains all your local needed configuration. In particular our host is not exposed on the I2P or Tor network, so we "visually" disabled those options.

# lines with "+" needs to be replace the ones starting with a "-"
-  "name": "img.bi",
+  "name": "img.bi - End Point image sharing service",

-  "maxSize": "3145728",
+  "maxSize": "32145728",

-  "clearnet": "https://img.bi",
+  "clearnet": "https://imgbi.example",

-  "i2p": "http://imgbi.i2p",
+  "i2p": "http://NOTAVAILABLE.i2p",

-  "tor": "http://imgbifwwqoixh7te.onion",
+  "tor": "http://NOTAVAILABLE.onion",

Save and close the file. At this point you should be able to run "grunt" to build the project but if it fails on the multiresize task, just run

grunt --force

to ignore the warnings.

That's about everything you need for the frontend part, so it's now time to take care of the API.

cd
git clone https://github.com/imgbi/img.bi-api.git
cd /home/imgbi/img.bi-api/

You now need to edit the two Python files which are the core of the API.

# edit code.py expired.py
-upload_dir = '/home/img.bi/img.bi-files'
+upload_dir = '/home/imgbi/img.bi-files'

Verify that you're not having any Python import related error, due to missing modules or else, by running the Python code.py file directly.

./code.py

If that's working okay, just create a symlink in the build directory in order to have the API created files available to the frontend

ln -s /home/imgbi/img.bi-files /home/imgbi/img.bi/build/download

And then it's time to spawn the actual Python daemon:

spawn-fcgi -f /home/imgbi/img.bi-api/code.py -a 127.0.0.1 -p 1234

The expired.py file is used by a cronjob which periodically checks if there's any image/content that should be removed because its time has expired. First of all let's call the script directly and if there's no error, let's create the crontab:

python /home/imgbi/img.bi-api/expired.py

crontab -e

@reboot spawn-fcgi -f /home/imgbi/img.bi-api/code.py -a 127.0.0.1 -p 1234
30 4 * * * python /home/imgbi/img.bi-api/expired.py

It's now time to install nginx and Redis (if you still haven't done so), and then configure them. For Redis you can just follow the usual simple, basic installation and that'll be just okay. Same is true for nginx but we'll add our configuration/vhost file content here as an example /etc/nginx/sites-enabled/imgbi.example.conf for everyone who may need it:

upstream imgbi-fastcgi {
  server 127.0.0.1:1234;
}

server {
  listen 80;
  listen [::]:80;
  server_name imgbi.example;
  access_log /var/log/nginx/sites/imgbi.example/access.log;
  error_log /var/log/nginx/sites/imgbi.example/error.log;
  rewrite ^ https://imgbi.example/ permanent;
}

server {
  listen 443 ssl spdy;
  listen [::]:443 ssl spdy;
  server_name  imgbi.example;
  server_name  imgbi.example;
  access_log /var/log/nginx/sites/imgbi.example/access.log;
  error_log /var/log/nginx/sites/imgbi.example/error.log;

  client_max_body_size 4G;

  include include/ssl-wildcard-example.inc;

  add_header Strict-Transport-Security max-age=31536000;
  add_header X-Frame-Options SAMEORIGIN;
  add_header X-Content-Type-Options nosniff;
  add_header X-XSS-Protection "1; mode=block";

  location / {
    root /home/imgbi/img.bi/build;
  }

  location /api {
    fastcgi_param QUERY_STRING $query_string;
    fastcgi_param REQUEST_METHOD $request_method;
    fastcgi_param CONTENT_TYPE $content_type;
    fastcgi_param CONTENT_LENGTH $content_length;

    fastcgi_param SCRIPT_NAME "";
    fastcgi_param PATH_INFO $uri;
    fastcgi_param REQUEST_URI $request_uri;
    fastcgi_param DOCUMENT_URI $document_uri;
    fastcgi_param DOCUMENT_ROOT $document_root;
    fastcgi_param SERVER_PROTOCOL $server_protocol;

    fastcgi_param GATEWAY_INTERFACE CGI/1.1;
    fastcgi_param SERVER_SOFTWARE nginx/$nginx_version;

    fastcgi_param REMOTE_ADDR $remote_addr;
    fastcgi_param REMOTE_PORT $remote_port;
    fastcgi_param SERVER_ADDR $server_addr;
    fastcgi_param SERVER_PORT $server_port;
    fastcgi_param SERVER_NAME $server_name;
    fastcgi_param HTTPS on;

    fastcgi_pass imgbi-fastcgi;
    fastcgi_keep_conn on;
  }
}

Well, that should be enough to get you started and at least have all the components in place. Enjoy your secure image sharing now.

03 Aug 2015 9:25pm GMT

End Point: img.bi, a secret encrypted image sharing service tool

After a fairly good experience with dnote installed on our own servers as an encrypted notes sharing service, my team decided that it would have been nice to have a similar service for images.

We found a nice project called img.bi that is based on NodeJS, Python, Redis and a lot of client-side JavaScript.

The system is divided into two components: the HTML/JS frontend and a Python FastCGI API.

Unfortunately the documentation is a still in its very early stage and it's lacking a meaningful structure and a lot of needed information.

Here's an overview of the steps we followed to setup img.bi on our own server behind nginx.

First of all we chose that we wanted to have as much as possible running and confined to a regular user, which is always a good idea with such young and potentially vulnerable tools. We chose to use the imgbi user.

Then since we wanted to keep as clean as possible the root user environment (and system status), we also decided to use pyenv. To be conservative we chose the latest Python 2.7 stable release, 2.7.10.

git clone https://github.com/yyuu/pyenv.git ~/.pyenv
echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.bash_profile
echo 'export PATH="$PYENV_ROOT/bin:$PATH"' >> ~/.bash_profile
echo 'eval "$(pyenv init -)"' >> ~/.bash_profile
echo $SHELL -l
pyenv install -l  | grep 2\\.7
pyenv install 2.7.10
pyenv global 2.7.10
pyenv version
which python
python --version

In order to use img.bi, we also needed NodeJS and following the same approach we chose to use nvm and install the latest NodeJS stable version:

curl -o- https://raw.githubusercontent.com/creationix/nvm/v0.25.4/install.sh | bash
nvm install stable
nvm list
nvm use stable
nvm alias default stable
node --version

As a short note to the usage of the bad practice of blindly using:

curl -o- https://some_obscure_link_or_not | bash

We want to add that we do not endorse this practice as it's dangerous and exposes your system to many security risks. On the other hand, though, it's true that cloning the source via Git and compile/installing it blindly is not much safer, so it's always up to how much you trust the peer review on the project you're about to use. And at least with an https URL you should be talking to the destination you want, whereas an http URL is much more dangerous.

Furthermore going through the entire Python and NodeJS installation as a regular user, was far beyond the scope of this post and the steps proposed here assumes that you're doing everything as the regular user, except where specifically stated differently.

Anyway after that we updated pip and then installed all the needed Python modules:

pip install --upgrade pip
pip install redis m2crypto web.py bcrypt pysha3 zbase62 pyutil flup

Then it's time to clone the actual img.bi code from the GitHub repo, install a few missing dependencies and then use the bower and npm .json files to add the desired packages:

git clone https://github.com/imgbi/img.bi.git
cd img.bi/
npm install -g bower grunt grunt-cli grunt-multiresize
npm install -g grunt-webfont --save-dev
npm install
bower install

We also faced an issue which made Grunt fail to start correctly. Grunt was complaining about an "undefined property" called "prototype". If you happen to have the same problem just type

cd node_modules/grunt-connect-proxy/node_modules/http-proxy
npm install eventemitter3@0.1.6
cd -

That'll basically install the eventemitter3 NodeJS package module locally to the grunt-connect-proxy module so to overcome the compatibility issues which in turn causes the error mentioned above.

You should use your favourite editor to change the file config.json, which basically contains all your local needed configuration. In particular our host is not exposed on the I2P or Tor network, so we "visually" disabled those options.

# lines with "+" needs to be replace the ones starting with a "-"
-  "name": "img.bi",
+  "name": "img.bi - End Point image sharing service",

-  "maxSize": "3145728",
+  "maxSize": "32145728",

-  "clearnet": "https://img.bi",
+  "clearnet": "https://imgbi.example",

-  "i2p": "http://imgbi.i2p",
+  "i2p": "http://NOTAVAILABLE.i2p",

-  "tor": "http://imgbifwwqoixh7te.onion",
+  "tor": "http://NOTAVAILABLE.onion",

Save and close the file. At this point you should be able to run "grunt" to build the project but if it fails on the multiresize task, just run

grunt --force

to ignore the warnings.

That's about everything you need for the frontend part, so it's now time to take care of the API.

cd
git clone https://github.com/imgbi/img.bi-api.git
cd /home/imgbi/img.bi-api/

You now need to edit the two Python files which are the core of the API.

# edit code.py expired.py
-upload_dir = '/home/img.bi/img.bi-files'
+upload_dir = '/home/imgbi/img.bi-files'

Verify that you're not having any Python import related error, due to missing modules or else, by running the Python code.py file directly.

./code.py

If that's working okay, just create a symlink in the build directory in order to have the API created files available to the frontend

ln -s /home/imgbi/img.bi-files /home/imgbi/img.bi/build/download

And then it's time to spawn the actual Python daemon:

spawn-fcgi -f /home/imgbi/img.bi-api/code.py -a 127.0.0.1 -p 1234

The expired.py file is used by a cronjob which periodically checks if there's any image/content that should be removed because its time has expired. First of all let's call the script directly and if there's no error, let's create the crontab:

python /home/imgbi/img.bi-api/expired.py

crontab -e

@reboot spawn-fcgi -f /home/imgbi/img.bi-api/code.py -a 127.0.0.1 -p 1234
30 4 * * * python /home/imgbi/img.bi-api/expired.py

It's now time to install nginx and Redis (if you still haven't done so), and then configure them. For Redis you can just follow the usual simple, basic installation and that'll be just okay. Same is true for nginx but we'll add our configuration/vhost file content here as an example /etc/nginx/sites-enabled/imgbi.example.conf for everyone who may need it:

upstream imgbi-fastcgi {
  server 127.0.0.1:1234;
}

server {
  listen 80;
  listen [::]:80;
  server_name imgbi.example;
  access_log /var/log/nginx/sites/imgbi.example/access.log;
  error_log /var/log/nginx/sites/imgbi.example/error.log;
  rewrite ^ https://imgbi.example/ permanent;
}

server {
  listen 443 ssl spdy;
  listen [::]:443 ssl spdy;
  server_name  imgbi.example;
  server_name  imgbi.example;
  access_log /var/log/nginx/sites/imgbi.example/access.log;
  error_log /var/log/nginx/sites/imgbi.example/error.log;

  client_max_body_size 4G;

  include include/ssl-wildcard-example.inc;

  add_header Strict-Transport-Security max-age=31536000;
  add_header X-Frame-Options SAMEORIGIN;
  add_header X-Content-Type-Options nosniff;
  add_header X-XSS-Protection "1; mode=block";

  location / {
    root /home/imgbi/img.bi/build;
  }

  location /api {
    fastcgi_param QUERY_STRING $query_string;
    fastcgi_param REQUEST_METHOD $request_method;
    fastcgi_param CONTENT_TYPE $content_type;
    fastcgi_param CONTENT_LENGTH $content_length;

    fastcgi_param SCRIPT_NAME "";
    fastcgi_param PATH_INFO $uri;
    fastcgi_param REQUEST_URI $request_uri;
    fastcgi_param DOCUMENT_URI $document_uri;
    fastcgi_param DOCUMENT_ROOT $document_root;
    fastcgi_param SERVER_PROTOCOL $server_protocol;

    fastcgi_param GATEWAY_INTERFACE CGI/1.1;
    fastcgi_param SERVER_SOFTWARE nginx/$nginx_version;

    fastcgi_param REMOTE_ADDR $remote_addr;
    fastcgi_param REMOTE_PORT $remote_port;
    fastcgi_param SERVER_ADDR $server_addr;
    fastcgi_param SERVER_PORT $server_port;
    fastcgi_param SERVER_NAME $server_name;
    fastcgi_param HTTPS on;

    fastcgi_pass imgbi-fastcgi;
    fastcgi_keep_conn on;
  }
}

Well, that should be enough to get you started and at least have all the components in place. Enjoy your secure image sharing now.

03 Aug 2015 9:25pm GMT

Nathan Lemoine: Multivariate Techniques in Python: EcoPy Alpha Launch!

I'm announcing the alpha launch of EcoPy: Ecological Data Analysis in Python. EcoPy is a Python module that contains a number of techniques (PCA, CA, CCorA, nMDS, MDS, RDA, etc.) for exploring complex multivariate data. For those of you familiar … Continue reading

03 Aug 2015 5:09pm GMT

Nathan Lemoine: Multivariate Techniques in Python: EcoPy Alpha Launch!

I'm announcing the alpha launch of EcoPy: Ecological Data Analysis in Python. EcoPy is a Python module that contains a number of techniques (PCA, CA, CCorA, nMDS, MDS, RDA, etc.) for exploring complex multivariate data. For those of you familiar … Continue reading

03 Aug 2015 5:09pm GMT

David MacIver: New Patreon page for work on Hypothesis

After a bunch of people have suggested it, I've created a Patreon to fund ongoing work on Hypothesis. Rewards include receiving in depth weekly updates about Hypothesis - news, advice on how to use it, etc. That probably means there will be a bit less of such things here.

I don't think this will be enough to keep me in food and drink, let alone rent, and I still have ongoing plans for commercial development of things built on top of Hypothesis. This is purely to fund the development of the open source core library, and to give me enough money that I can feasibly keep going until I've got a more solid plan on that front.

So, if you like my work on Hypothesis (or even on here)! I'd really appreciate some funding. I've put a lot of work into this, and it sure would be nice to see something back for that.

03 Aug 2015 2:28pm GMT

David MacIver: New Patreon page for work on Hypothesis

After a bunch of people have suggested it, I've created a Patreon to fund ongoing work on Hypothesis. Rewards include receiving in depth weekly updates about Hypothesis - news, advice on how to use it, etc. That probably means there will be a bit less of such things here.

I don't think this will be enough to keep me in food and drink, let alone rent, and I still have ongoing plans for commercial development of things built on top of Hypothesis. This is purely to fund the development of the open source core library, and to give me enough money that I can feasibly keep going until I've got a more solid plan on that front.

So, if you like my work on Hypothesis (or even on here)! I'd really appreciate some funding. I've put a lot of work into this, and it sure would be nice to see something back for that.

03 Aug 2015 2:28pm GMT

Mike Driscoll: PyDev of the Week: Patrick Maupin

This week we welcome Patrick Maupin as our PyDev of the Week. Let's spend some time getting to know Patrick a bit better.

Can you tell us a little about yourself (hobbies, education, etc):

I was born in San Antonio, Texas and raised in Austin. I spent several months working in England, and met and married a wonderful English lady and brought her home. We've lived in Austin for the last 30 years, except for one year we spent in Toronto. We have two great girls - one is in medical school, and the other is a budding composer/author.

I rotate through a few hobbies - bicycling, sailing, playing tuba in a community band. The tuba is actually something I re-started just a few years ago. I wound up dropping out of school when my lung collapsed when I was playing tuba with the UT Longhorn Band. Fortunately, my lung was repaired later…

I had been pursuing a degree in electrical engineering, because playing with electrons was my passion ever since I can remember. When I dropped out of school, I couldn't find a job doing electrical stuff, but I did find one programming, and then worked myself into positions where I was doing both hardware and software. I have mostly worked near the hardware - drivers, embedded stuff, modem firmware, or on the hardware - board design, Verilog blocks for chips, etc., or on tools that help to use and test the hardware - schematic checkers, JTAG programmers, etc.

For the last 20 years, I've only worked for chip companies, and helping to design chips suits me, because successful chip companies are committed to practices that lead to working products - a failed tapeout can cost hundreds of thousands, or even millions of dollars, plus the lost opportunity costs of waiting one to three months for your first chips to come back so you can even test them. A product that fails in the field on a customer board can be unimaginably costly, as well, so chip manufacturers usually don't just give lip-service to words like "testing" and "quality."

I've worked at the same place for the last nine years, but we've been acquired a couple of times. The name on the door now says Microsemi, and it's a great place to work.

I've sort of worked myself into a position where the hardware guys think I'm a software guy, and the software guys think I'm a hardware guy. That probably means I don't do either one very well.

Why did you start using Python?

I was trying to teach myself a bit about encryption and hashing, back in 1996, and I found Python's long number support, combined with its automatic memory management, quite compelling.

A quick google search will show you that opinions vary widely on what computer language syntax and core language features should be, but I have to say that Python matches the way I think and work much better than most high-level language. I'm always amused and bemused by assertions from aficionados of other language about Python's supposed shortcomings - some of those I consider features, and some really are shortcomings, but usually nowhere near bad enough to seriously consider using whichever language is being touted at the time.

The first time I remember using Python for work was in 1999. My boss had been working on a hardware block (as his main full-time task) for 6 weeks, and just couldn't get it to work. He finally asked for my help, and I wrote a Python program in about five hours that wrote some perfectly working Verilog code for him. He wasn't very happy about that.

Using Python to write code has been a recurring theme in my career, and in general, I like the syntax so much that I even sometimes write some simple translators, which is why I contribute to projects like astor.

I'd like to do a hardware language with Python. There are a few efforts in that direction, but I don't find any of them compelling. Some of the building blocks that people are making that could help out with this, like mython, seem interesting.

What other programming languages do you know and which is your favorite?

I have made money programming in FORTRAN, Z-80 assembly, X86 assembly, 68000 assembly, TI DSP assembly, Pascal, Modula-2, C, and Verilog. Interestingly, I have never "made money" programming in Python, in the sense that it has never been in my job description like those other languages have, but in general, people at work don't bother me when I'm using it rather than a "real" language, and often seek me out for basic scripts that they can tweak.

I always liked assembly language, and I really liked Modula-2 at the time, but when C acquired all the ANSI extensions and the compilers got really good at warning you about your stupid mistakes, C became "good enough" and took over the world.

I have dabbled in several other languages, but with Python and C in my toolbelt (and these days, Cython does a great job of merging them if ctypes isn't fast enough), I just haven't happened to have a project where any of the others are compelling yet.

In fact, my C may be getting a bit rusty - computers are fast enough that I probably only break it out once every couple of years. These days, if I'm typing stuff on an keyboard, it's probably either Python or Verilog, and that works out well - Verilog satisfies the urge to write something tiny and polished at a really low level, and Python satisfies the urge to write something tiny and polished at a really high level (as well as the necessity to bang out some ugly throw-away code for a task that somebody needs immediately). What else is there?

I don't know where I first heard it, but I often echo the sentiment that "C is the new assembly language and Python is the new C." But I'm not sure how true that really is any more - for in-browser stuff, it appears that javascript is rapidly becoming the new assembly language.

Anyway, for most general computing tasks, Python has to be my overall favorite by a large margin.

What projects are you working on now?

At work, I'm working on emulation for a new test chip. I use Python for many of the tests, with a setup where the device is represented by a class instance on the computer, and when you do something like "dut.timer1.ctl.start = True" it will actually go out over USB and provoke the dut (aka "device under test") and do a read/modify/write to timer 1's control register, setting the start bit high. Put py.test and jenkins on top of that, and you have really simple
reporting that even a manger will love.

The fact that "start" is a bit field in the "ctl" register of the "timer1″ block is maintained in a file that is also parsed by a Python script (via rsonlite) to produce Verilog, include files, and documentation (via rst2pdf).

I also use Python for several other tasks, ranging from checking schematic netlists to writing scripts to help out with CAD and production tests.

In my spare time, I'm busy trying to clean up a lot of my old stuff and port it to Python 3 (if it didn't already support it) and move it off googlecode to github. I started with pdfrw, and released a 0.2 with support for Python 3 as well as 2, and a few bug fixes and enhancements. I am actually fairly happy with how github works, and am working on a few more enhancements (and documentation - need to do a better job of that!)

Maybe 0.3 will come out soon - pdfrw is actually my most popular project. Although not as popular or full-featured as PyPDF2, it has some nice niche features. The core started off as some personal scripts to do page imposition for a neighborhood newsletter my wife used to edit, and then I released it as open-source when I started using and contributing to rst2pdf and realized that none of the available tools for reusing vector content with that package were very good -either they rasterized the source, or had bugs and some things didn't display properly. It turns out that one PDF feature - form XObjects - lets you reuse pieces of existing PDFs as if they were images, so you can use something like inkscape to convert your SVG file into a PDF, and then use pdfrw with reportlab (either directly or via rst2pdf) to add any sort of preexisting PDF content (including those converted SVG images) in a new PDF.

After I'm happy with pdfrw, I'll probably clean up and move rson to github next. rson is another package that was born to work with rst2pdf. Actually, I had some other use-cases for it as well, but its initial usage was with rst2pdf stylesheets. Those were originally in JSON, and when I converted them to YAML, the rst2pdf test time went up by something stupid. I don't remember exactly, but it was in the hundreds of percent. To parse stylesheets. Anyway, I read the YAML spec, downloaded and tested several implementations in different languages, realized that none of the implementations agreed on the multitude of corner cases, and was appalled, so I wrote something that was simpler than YAML and more readable than JSON.

Obviously, the world doesn't agree with me (or maybe YAML has been cleaned up?) because YAML seems to be the in-thing these days.

Which Python libraries are your favorite (core or 3rd party)?

When it comes to the core, that's a very difficult question. The core is so balanced that I use a lot of libraries - subprocess, hashlib, shutil, time, bisect, textwrap, array, glob, atexit, wave, gc, ast, elementtree - the list goes on and on.

I think one of your previous interviewees, Brian Curtin, said something like "anything by Raymond Hettinger" and that's a great answer - personally, I use itertools a lot, and collections would also be a great module even if the only thing in it was the defaultdict.

For interfacing to the world, ctypes and re are both indispensible. Finally, binascii is a great sleeper module - for some tasks it's much faster than using ctypes or struct.

When it comes to external modules, obviously I have a fondness for things I've written or worked on, but honestly, the overall winner is not in either of those categories - xlrd is the hands-down winner, because it helps to keep me sane. I don't use Windows, much less Excel, but I have come to accept that, for many users, Excel is a productive data entry tool. xlrd lets me accept data from those guys, get it into a text form that can be version-controlled, and then have my way with it.

Where do you see Python going as a programming language?

Psychologically, I'm still recovering from the 2-to-3 transition, which created a huge chicken-and-egg situation in the community. I know, I'm a Luddite, but for a few years there, I wasn't really interested in doing anything with Python 3, and didn't feel that other people would be all that interested in my Python 2 stuff.

Obviously the core developers are committed to 3, and with some of the later versions of 3 and some of the backports to 2.7, it's not nearly as painful to transition as it seemed it would be for awhile.

I don't know where it goes after this - I'm just hoping for a bit of breathing room, to where I can reliably get back to looking forward to the next release of Python with unbridled anticipation. I'm starting to get there - for example, I think 3.5 will have some killer features, and now I'm engaged enough with Python 3 to care - but that hasn't been the case for a long time.

Overall, I think there are a lot of positive things going on in the community with new packaging, etc. I think the next few years have the potential to be quite exciting. On the one hand, it sometimes seems that, despite things like pyjs and kivy, Python isn't really going to participate in the web/mobile transition. But on the other hand, iPython/jupyter has some fantastic technology, and if they keep improving their security model, who knows where that could lead?

Maybe I can have my own iPython notebook at my bank that knows all about my accounts and presents the data the way I want to see it, and a notebook at my brokerage that helps to handle my trades for me, and a notebook at amazon that helps to buy stuff for me.

Is there anything else you'd like to say?

One thing I've noticed since I started re-engaging with the Python community a few months ago is how huge it's gotten in Austin. There are apparently a zillion companies in downtown Austin using Python heavily, and sometimes over 100 people show up at the monthly meetup.

We just started a couple of "hack nights" to try to recapture the spirit of the early days of the meetup, and even there, the response has been phenomenal - the downtown one probably had 40 people show up, and the one I set up in the coffee shop around the corner from my house had over 20. It's energizing to be able to walk around the corner and engage in relevant technical discussions, so if you don't have any meetup like that in your neighborhood, you should go talk to a coffee shop owner and maybe slip them a few bucks to reserve a room, and then go to meetup.com and let everybody know about it.

That's something I wish I figured out many years earlier.

Another thing in that category is this: if you have a project, it doesn't have to be perfect. Before I understood this, I shelved a few things that I actually had working code for, simply because I didn't think they were good enough (and frankly, some of that was due to negative reactions from people I admired). If you have something that works well for you, even in just one use case, put it out there, and if you get some feedback and make a few mods, pretty soon you might have a core group of users.

On a related note, if you're working for a company that won't ever let you contribute anything you do at work back to the wider community, whether that's because they think that every single thing they do is a sooper-sekret competitive advantage, or because they don't want anybody finding out about you, you're
probably working for the wrong people - in the first case, they are completely deluded, and in the second case, they're probably not paying you enough.

Finally, I think your blog in general, and this interview series in particular, are both awesome, and I'd like to thank you for doing this!

Thanks!

The Last 10 PyDevs of the Week

03 Aug 2015 12:30pm GMT

Mike Driscoll: PyDev of the Week: Patrick Maupin

This week we welcome Patrick Maupin as our PyDev of the Week. Let's spend some time getting to know Patrick a bit better.

Can you tell us a little about yourself (hobbies, education, etc):

I was born in San Antonio, Texas and raised in Austin. I spent several months working in England, and met and married a wonderful English lady and brought her home. We've lived in Austin for the last 30 years, except for one year we spent in Toronto. We have two great girls - one is in medical school, and the other is a budding composer/author.

I rotate through a few hobbies - bicycling, sailing, playing tuba in a community band. The tuba is actually something I re-started just a few years ago. I wound up dropping out of school when my lung collapsed when I was playing tuba with the UT Longhorn Band. Fortunately, my lung was repaired later…

I had been pursuing a degree in electrical engineering, because playing with electrons was my passion ever since I can remember. When I dropped out of school, I couldn't find a job doing electrical stuff, but I did find one programming, and then worked myself into positions where I was doing both hardware and software. I have mostly worked near the hardware - drivers, embedded stuff, modem firmware, or on the hardware - board design, Verilog blocks for chips, etc., or on tools that help to use and test the hardware - schematic checkers, JTAG programmers, etc.

For the last 20 years, I've only worked for chip companies, and helping to design chips suits me, because successful chip companies are committed to practices that lead to working products - a failed tapeout can cost hundreds of thousands, or even millions of dollars, plus the lost opportunity costs of waiting one to three months for your first chips to come back so you can even test them. A product that fails in the field on a customer board can be unimaginably costly, as well, so chip manufacturers usually don't just give lip-service to words like "testing" and "quality."

I've worked at the same place for the last nine years, but we've been acquired a couple of times. The name on the door now says Microsemi, and it's a great place to work.

I've sort of worked myself into a position where the hardware guys think I'm a software guy, and the software guys think I'm a hardware guy. That probably means I don't do either one very well.

Why did you start using Python?

I was trying to teach myself a bit about encryption and hashing, back in 1996, and I found Python's long number support, combined with its automatic memory management, quite compelling.

A quick google search will show you that opinions vary widely on what computer language syntax and core language features should be, but I have to say that Python matches the way I think and work much better than most high-level language. I'm always amused and bemused by assertions from aficionados of other language about Python's supposed shortcomings - some of those I consider features, and some really are shortcomings, but usually nowhere near bad enough to seriously consider using whichever language is being touted at the time.

The first time I remember using Python for work was in 1999. My boss had been working on a hardware block (as his main full-time task) for 6 weeks, and just couldn't get it to work. He finally asked for my help, and I wrote a Python program in about five hours that wrote some perfectly working Verilog code for him. He wasn't very happy about that.

Using Python to write code has been a recurring theme in my career, and in general, I like the syntax so much that I even sometimes write some simple translators, which is why I contribute to projects like astor.

I'd like to do a hardware language with Python. There are a few efforts in that direction, but I don't find any of them compelling. Some of the building blocks that people are making that could help out with this, like mython, seem interesting.

What other programming languages do you know and which is your favorite?

I have made money programming in FORTRAN, Z-80 assembly, X86 assembly, 68000 assembly, TI DSP assembly, Pascal, Modula-2, C, and Verilog. Interestingly, I have never "made money" programming in Python, in the sense that it has never been in my job description like those other languages have, but in general, people at work don't bother me when I'm using it rather than a "real" language, and often seek me out for basic scripts that they can tweak.

I always liked assembly language, and I really liked Modula-2 at the time, but when C acquired all the ANSI extensions and the compilers got really good at warning you about your stupid mistakes, C became "good enough" and took over the world.

I have dabbled in several other languages, but with Python and C in my toolbelt (and these days, Cython does a great job of merging them if ctypes isn't fast enough), I just haven't happened to have a project where any of the others are compelling yet.

In fact, my C may be getting a bit rusty - computers are fast enough that I probably only break it out once every couple of years. These days, if I'm typing stuff on an keyboard, it's probably either Python or Verilog, and that works out well - Verilog satisfies the urge to write something tiny and polished at a really low level, and Python satisfies the urge to write something tiny and polished at a really high level (as well as the necessity to bang out some ugly throw-away code for a task that somebody needs immediately). What else is there?

I don't know where I first heard it, but I often echo the sentiment that "C is the new assembly language and Python is the new C." But I'm not sure how true that really is any more - for in-browser stuff, it appears that javascript is rapidly becoming the new assembly language.

Anyway, for most general computing tasks, Python has to be my overall favorite by a large margin.

What projects are you working on now?

At work, I'm working on emulation for a new test chip. I use Python for many of the tests, with a setup where the device is represented by a class instance on the computer, and when you do something like "dut.timer1.ctl.start = True" it will actually go out over USB and provoke the dut (aka "device under test") and do a read/modify/write to timer 1's control register, setting the start bit high. Put py.test and jenkins on top of that, and you have really simple
reporting that even a manger will love.

The fact that "start" is a bit field in the "ctl" register of the "timer1″ block is maintained in a file that is also parsed by a Python script (via rsonlite) to produce Verilog, include files, and documentation (via rst2pdf).

I also use Python for several other tasks, ranging from checking schematic netlists to writing scripts to help out with CAD and production tests.

In my spare time, I'm busy trying to clean up a lot of my old stuff and port it to Python 3 (if it didn't already support it) and move it off googlecode to github. I started with pdfrw, and released a 0.2 with support for Python 3 as well as 2, and a few bug fixes and enhancements. I am actually fairly happy with how github works, and am working on a few more enhancements (and documentation - need to do a better job of that!)

Maybe 0.3 will come out soon - pdfrw is actually my most popular project. Although not as popular or full-featured as PyPDF2, it has some nice niche features. The core started off as some personal scripts to do page imposition for a neighborhood newsletter my wife used to edit, and then I released it as open-source when I started using and contributing to rst2pdf and realized that none of the available tools for reusing vector content with that package were very good -either they rasterized the source, or had bugs and some things didn't display properly. It turns out that one PDF feature - form XObjects - lets you reuse pieces of existing PDFs as if they were images, so you can use something like inkscape to convert your SVG file into a PDF, and then use pdfrw with reportlab (either directly or via rst2pdf) to add any sort of preexisting PDF content (including those converted SVG images) in a new PDF.

After I'm happy with pdfrw, I'll probably clean up and move rson to github next. rson is another package that was born to work with rst2pdf. Actually, I had some other use-cases for it as well, but its initial usage was with rst2pdf stylesheets. Those were originally in JSON, and when I converted them to YAML, the rst2pdf test time went up by something stupid. I don't remember exactly, but it was in the hundreds of percent. To parse stylesheets. Anyway, I read the YAML spec, downloaded and tested several implementations in different languages, realized that none of the implementations agreed on the multitude of corner cases, and was appalled, so I wrote something that was simpler than YAML and more readable than JSON.

Obviously, the world doesn't agree with me (or maybe YAML has been cleaned up?) because YAML seems to be the in-thing these days.

Which Python libraries are your favorite (core or 3rd party)?

When it comes to the core, that's a very difficult question. The core is so balanced that I use a lot of libraries - subprocess, hashlib, shutil, time, bisect, textwrap, array, glob, atexit, wave, gc, ast, elementtree - the list goes on and on.

I think one of your previous interviewees, Brian Curtin, said something like "anything by Raymond Hettinger" and that's a great answer - personally, I use itertools a lot, and collections would also be a great module even if the only thing in it was the defaultdict.

For interfacing to the world, ctypes and re are both indispensible. Finally, binascii is a great sleeper module - for some tasks it's much faster than using ctypes or struct.

When it comes to external modules, obviously I have a fondness for things I've written or worked on, but honestly, the overall winner is not in either of those categories - xlrd is the hands-down winner, because it helps to keep me sane. I don't use Windows, much less Excel, but I have come to accept that, for many users, Excel is a productive data entry tool. xlrd lets me accept data from those guys, get it into a text form that can be version-controlled, and then have my way with it.

Where do you see Python going as a programming language?

Psychologically, I'm still recovering from the 2-to-3 transition, which created a huge chicken-and-egg situation in the community. I know, I'm a Luddite, but for a few years there, I wasn't really interested in doing anything with Python 3, and didn't feel that other people would be all that interested in my Python 2 stuff.

Obviously the core developers are committed to 3, and with some of the later versions of 3 and some of the backports to 2.7, it's not nearly as painful to transition as it seemed it would be for awhile.

I don't know where it goes after this - I'm just hoping for a bit of breathing room, to where I can reliably get back to looking forward to the next release of Python with unbridled anticipation. I'm starting to get there - for example, I think 3.5 will have some killer features, and now I'm engaged enough with Python 3 to care - but that hasn't been the case for a long time.

Overall, I think there are a lot of positive things going on in the community with new packaging, etc. I think the next few years have the potential to be quite exciting. On the one hand, it sometimes seems that, despite things like pyjs and kivy, Python isn't really going to participate in the web/mobile transition. But on the other hand, iPython/jupyter has some fantastic technology, and if they keep improving their security model, who knows where that could lead?

Maybe I can have my own iPython notebook at my bank that knows all about my accounts and presents the data the way I want to see it, and a notebook at my brokerage that helps to handle my trades for me, and a notebook at amazon that helps to buy stuff for me.

Is there anything else you'd like to say?

One thing I've noticed since I started re-engaging with the Python community a few months ago is how huge it's gotten in Austin. There are apparently a zillion companies in downtown Austin using Python heavily, and sometimes over 100 people show up at the monthly meetup.

We just started a couple of "hack nights" to try to recapture the spirit of the early days of the meetup, and even there, the response has been phenomenal - the downtown one probably had 40 people show up, and the one I set up in the coffee shop around the corner from my house had over 20. It's energizing to be able to walk around the corner and engage in relevant technical discussions, so if you don't have any meetup like that in your neighborhood, you should go talk to a coffee shop owner and maybe slip them a few bucks to reserve a room, and then go to meetup.com and let everybody know about it.

That's something I wish I figured out many years earlier.

Another thing in that category is this: if you have a project, it doesn't have to be perfect. Before I understood this, I shelved a few things that I actually had working code for, simply because I didn't think they were good enough (and frankly, some of that was due to negative reactions from people I admired). If you have something that works well for you, even in just one use case, put it out there, and if you get some feedback and make a few mods, pretty soon you might have a core group of users.

On a related note, if you're working for a company that won't ever let you contribute anything you do at work back to the wider community, whether that's because they think that every single thing they do is a sooper-sekret competitive advantage, or because they don't want anybody finding out about you, you're
probably working for the wrong people - in the first case, they are completely deluded, and in the second case, they're probably not paying you enough.

Finally, I think your blog in general, and this interview series in particular, are both awesome, and I'd like to thank you for doing this!

Thanks!

The Last 10 PyDevs of the Week

03 Aug 2015 12:30pm GMT

pgcli: Release v0.19.0

Pgcli is a command line interface for Postgres database that does auto-completion and syntax highlighting. You can install this version by:

$ pip install -U pgcli

Check detailed instructions if you're having difficulty.

This release comes with metadata information in the completion menus. Now you can see whether a completion suggestion is a table or a schema or a view. New special command (\i), latest prompt_toolkit with wider completion menus.

Features:

  • Wider completion menus can be enabled via the config file. (Thanks: Jonathan Slenders)

    Open the config file (~/.pgclirc) and check if you have wider_completion_menu option available. If not add it in and set it to True.

  • Completion menu now has metadata information such as schema, table, column, view, etc., next to the suggestions. (Thanks: Darik Gamble)

  • Customizable history file location via config file. (Thanks: Çağatay Yüksel)

    Add this line to your config file (~/.pgclirc) to customize where to store the history file.

history_file = /path/to/history/file
  • Add support for running queries from a file using \i special command. (Thanks: Michael Kaminsky)

Bug Fixes:

  • Always use utf-8 for database encoding regardless of the default encoding used by the database.
  • Fix for None dereference on \d schemaname. with sequence. (Thanks: Nathan Jhaveri)
  • Fix a crashing bug in the autocompletion engine for some JOIN queries.
  • Handle KeyboardInterrupt in pager and not quit pgcli as a consequence.

Internal Changes:

03 Aug 2015 7:00am GMT

pgcli: Release v0.19.0

Pgcli is a command line interface for Postgres database that does auto-completion and syntax highlighting. You can install this version by:

$ pip install -U pgcli

Check detailed instructions if you're having difficulty.

This release comes with metadata information in the completion menus. Now you can see whether a completion suggestion is a table or a schema or a view. New special command (\i), latest prompt_toolkit with wider completion menus.

Features:

  • Wider completion menus can be enabled via the config file. (Thanks: Jonathan Slenders)

    Open the config file (~/.pgclirc) and check if you have wider_completion_menu option available. If not add it in and set it to True.

  • Completion menu now has metadata information such as schema, table, column, view, etc., next to the suggestions. (Thanks: Darik Gamble)

  • Customizable history file location via config file. (Thanks: Çağatay Yüksel)

    Add this line to your config file (~/.pgclirc) to customize where to store the history file.

history_file = /path/to/history/file
  • Add support for running queries from a file using \i special command. (Thanks: Michael Kaminsky)

Bug Fixes:

  • Always use utf-8 for database encoding regardless of the default encoding used by the database.
  • Fix for None dereference on \d schemaname. with sequence. (Thanks: Nathan Jhaveri)
  • Fix a crashing bug in the autocompletion engine for some JOIN queries.
  • Handle KeyboardInterrupt in pager and not quit pgcli as a consequence.

Internal Changes:

03 Aug 2015 7:00am GMT

10 Nov 2011

feedPython Software Foundation | GSoC'11 Students

Benedict Stein: King Willams Town Bahnhof

Gestern musste ich morgens zur Station nach KWT um unsere Rerservierten Bustickets für die Weihnachtsferien in Capetown abzuholen. Der Bahnhof selber ist seit Dezember aus kostengründen ohne Zugverbindung - aber Translux und co - die langdistanzbusse haben dort ihre Büros.


Größere Kartenansicht




© benste CC NC SA

10 Nov 2011 10:57am GMT

09 Nov 2011

feedPython Software Foundation | GSoC'11 Students

Benedict Stein

Niemand ist besorgt um so was - mit dem Auto fährt man einfach durch, und in der City - nahe Gnobie- "ne das ist erst gefährlich wenn die Feuerwehr da ist" - 30min später auf dem Rückweg war die Feuerwehr da.




© benste CC NC SA

09 Nov 2011 8:25pm GMT

08 Nov 2011

feedPython Software Foundation | GSoC'11 Students

Benedict Stein: Brai Party

Brai = Grillabend o.ä.

Die möchte gern Techniker beim Flicken ihrer SpeakOn / Klinke Stecker Verzweigungen...

Die Damen "Mamas" der Siedlung bei der offiziellen Eröffnungsrede

Auch wenn weniger Leute da waren als erwartet, Laute Musik und viele Leute ...

Und natürlich ein Feuer mit echtem Holz zum Grillen.

© benste CC NC SA

08 Nov 2011 2:30pm GMT

07 Nov 2011

feedPython Software Foundation | GSoC'11 Students

Benedict Stein: Lumanyano Primary

One of our missions was bringing Katja's Linux Server back to her room. While doing that we saw her new decoration.

Björn, Simphiwe carried the PC to Katja's school


© benste CC NC SA

07 Nov 2011 2:00pm GMT

06 Nov 2011

feedPython Software Foundation | GSoC'11 Students

Benedict Stein: Nelisa Haircut

Today I went with Björn to Needs Camp to Visit Katja's guest family for a special Party. First of all we visited some friends of Nelisa - yeah the one I'm working with in Quigney - Katja's guest fathers sister - who did her a haircut.

African Women usually get their hair done by arranging extensions and not like Europeans just cutting some hair.

In between she looked like this...

And then she was done - looks amazing considering the amount of hair she had last week - doesn't it ?

© benste CC NC SA

06 Nov 2011 7:45pm GMT

05 Nov 2011

feedPython Software Foundation | GSoC'11 Students

Benedict Stein: Mein Samstag

Irgendwie viel mir heute auf das ich meine Blogposts mal ein bischen umstrukturieren muss - wenn ich immer nur von neuen Plätzen berichte, dann müsste ich ja eine Rundreise machen. Hier also mal ein paar Sachen aus meinem heutigen Alltag.

Erst einmal vorweg, Samstag zählt zumindest für uns Voluntäre zu den freien Tagen.

Dieses Wochenende sind nur Rommel und ich auf der Farm - Katja und Björn sind ja mittlerweile in ihren Einsatzstellen, und meine Mitbewohner Kyle und Jonathan sind zu Hause in Grahamstown - sowie auch Sipho der in Dimbaza wohnt.
Robin, die Frau von Rommel ist in Woodie Cape - schon seit Donnerstag um da ein paar Sachen zur erledigen.
Naja wie dem auch sei heute morgen haben wir uns erstmal ein gemeinsames Weetbix/Müsli Frühstück gegönnt und haben uns dann auf den Weg nach East London gemacht. 2 Sachen waren auf der Checkliste Vodacom, Ethienne (Imobilienmakler) außerdem auf dem Rückweg die fehlenden Dinge nach NeedsCamp bringen.

Nachdem wir gerade auf der Dirtroad losgefahren sind mussten wir feststellen das wir die Sachen für Needscamp und Ethienne nicht eingepackt hatten aber die Pumpe für die Wasserversorgung im Auto hatten.

Also sind wir in EastLondon ersteinmal nach Farmerama - nein nicht das onlinespiel farmville - sondern einen Laden mit ganz vielen Sachen für eine Farm - in Berea einem nördlichen Stadteil gefahren.

In Farmerama haben wir uns dann beraten lassen für einen Schnellverschluss der uns das leben mit der Pumpe leichter machen soll und außerdem eine leichtere Pumpe zur Reperatur gebracht, damit es nicht immer so ein großer Aufwand ist, wenn mal wieder das Wasser ausgegangen ist.

Fego Caffé ist in der Hemmingways Mall, dort mussten wir und PIN und PUK einer unserer Datensimcards geben lassen, da bei der PIN Abfrage leider ein zahlendreher unterlaufen ist. Naja auf jeden Fall speichern die Shops in Südafrika so sensible Daten wie eine PUK - die im Prinzip zugang zu einem gesperrten Phone verschafft.

Im Cafe hat Rommel dann ein paar online Transaktionen mit dem 3G Modem durchgeführt, welches ja jetzt wieder funktionierte - und übrigens mittlerweile in Ubuntu meinem Linuxsystem perfekt klappt.

Nebenbei bin ich nach 8ta gegangen um dort etwas über deren neue Deals zu erfahren, da wir in einigen von Hilltops Centern Internet anbieten wollen. Das Bild zeigt die Abdeckung UMTS in NeedsCamp Katjas Ort. 8ta ist ein neuer Telefonanbieter von Telkom, nachdem Vodafone sich Telkoms anteile an Vodacom gekauft hat müssen die komplett neu aufbauen.
Wir haben uns dazu entschieden mal eine kostenlose Prepaidkarte zu testen zu organisieren, denn wer weis wie genau die Karte oben ist ... Bevor man einen noch so billigen Deal für 24 Monate signed sollte man wissen obs geht.

Danach gings nach Checkers in Vincent, gesucht wurden zwei Hotplates für WoodyCape - R 129.00 eine - also ca. 12€ für eine zweigeteilte Kochplatte.
Wie man sieht im Hintergrund gibts schon Weihnachtsdeko - Anfang November und das in Südafrika bei sonnig warmen min- 25°C

Mittagessen haben wir uns bei einem Pakistanischen Curry Imbiss gegönnt - sehr empfehlenswert !
Naja und nachdem wir dann vor ner Stunde oder so zurück gekommen sind habe ich noch den Kühlschrank geputzt den ich heute morgen zum defrosten einfach nach draußen gestellt hatte. Jetzt ist der auch mal wieder sauber und ohne 3m dicke Eisschicht...

Morgen ... ja darüber werde ich gesondert berichten ... aber vermutlich erst am Montag, denn dann bin ich nochmal wieder in Quigney(East London) und habe kostenloses Internet.

© benste CC NC SA

05 Nov 2011 4:33pm GMT

31 Oct 2011

feedPython Software Foundation | GSoC'11 Students

Benedict Stein: Sterkspruit Computer Center

Sterkspruit is one of Hilltops Computer Centres in the far north of Eastern Cape. On the trip to J'burg we've used the opportunity to take a look at the centre.

Pupils in the big classroom


The Trainer


School in Countryside


Adult Class in the Afternoon


"Town"


© benste CC NC SA

31 Oct 2011 4:58pm GMT

Benedict Stein: Technical Issues

What are you doing in an internet cafe if your ADSL and Faxline has been discontinued before months end. Well my idea was sitting outside and eating some ice cream.
At least it's sunny and not as rainy as on the weekend.


© benste CC NC SA

31 Oct 2011 3:11pm GMT

30 Oct 2011

feedPython Software Foundation | GSoC'11 Students

Benedict Stein: Nellis Restaurant

For those who are traveling through Zastron - there is a very nice Restaurant which is serving delicious food at reasanable prices.
In addition they're selling home made juices jams and honey.




interior


home made specialities - the shop in the shop


the Bar


© benste CC NC SA

30 Oct 2011 4:47pm GMT

29 Oct 2011

feedPython Software Foundation | GSoC'11 Students

Benedict Stein: The way back from J'burg

Having the 10 - 12h trip from J'burg back to ELS I was able to take a lot of pcitures including these different roadsides

Plain Street


Orange River in its beginngings (near Lesotho)


Zastron Anglican Church


The Bridge in Between "Free State" and Eastern Cape next to Zastron


my new Background ;)


If you listen to GoogleMaps you'll end up traveling 50km of gravel road - as it was just renewed we didn't have that many problems and saved 1h compared to going the official way with all it's constructions sites




Freeway


getting dark


© benste CC NC SA

29 Oct 2011 4:23pm GMT

28 Oct 2011

feedPython Software Foundation | GSoC'11 Students

Benedict Stein: Wie funktioniert eigentlich eine Baustelle ?

Klar einiges mag anders sein, vieles aber gleich - aber ein in Deutschland täglich übliches Bild einer Straßenbaustelle - wie läuft das eigentlich in Südafrika ?

Ersteinmal vorweg - NEIN keine Ureinwohner die mit den Händen graben - auch wenn hier mehr Manpower genutzt wird - sind sie fleißig mit Technologie am arbeiten.

Eine ganz normale "Bundesstraße"


und wie sie erweitert wird


gaaaanz viele LKWs


denn hier wird eine Seite über einen langen Abschnitt komplett gesperrt, so das eine Ampelschaltung mit hier 45 Minuten Wartezeit entsteht


Aber wenigstens scheinen die ihren Spaß zu haben ;) - Wie auch wir denn gücklicher Weise mussten wir nie länger als 10 min. warten.

© benste CC NC SA

28 Oct 2011 4:20pm GMT