19 Sep 2014

feedPlanet Plone

Andreas Jung: Copying Dexterity fields from one instance to another instance

19 Sep 2014 2:58pm GMT

18 Sep 2014

feedPlanet Plone

Connexions Developers Blog: Rewrite Technologies

Our team has been hard at work rewriting the OpenStax CNX site. A frequently asked question is "What tech are you using?". This post covers a high level overview of the new site.

Architecture


Rewrite is a Single-Page App which means most of the logic lives in a Javascript client. The Client accesses data via REST APIs that are written in Python. All of our data is stored in a PostgresSQL database. The details of the separate components follow.

Webview

Webview is the Javascript Client. The basis of the Client is Backbone.js and Bootstrap. We use several other Javascript packages as well.

Most of the Javascript is written in Coffeescript and compiled to Javascript. CSS is developed using Less.

Webview requests json from the Archive via REST APIs. The json contains the HTML for the content and any metadata. The Client parses the json and displays it.

Editing is a separate view in Webview. When the user selects to edit, the views are swapped out. The new editor is based on the open-source HTML5 editor Aloha. We have added several plugins to Aloha that are textbook editing specific. The development on the editor was done by our team and the OERPub team with OERPub doing the bulk of the work.

We are using Nginx as our web server.

Archive

Archive stores published content and handles search. Content retrieving and search are handled via APIs written in Python. When content is requested, the json is built via stored procedures in Postgres. The json is built using the json functions in Postgres.

Search is done with optimized queries. We are caching subject and one word searches long term and all searches short term to improve performance. When a user pages through search results, the cached result is used to populate the next page.

Archive will run on an WSGI compatible server. We are currently using Waitress as our server.

Publishing

The publishing application integrates with the Archive database. It allows users and third-party applications to publish content to the Archive, where it can be read and distributed to the public. Publishing is built similar to Archive, but differs in many ways. Archive is a read-only content API. Publishing provides an additional set of APIs that handle the publishing workflow, which includes user interactions like license and role (e.g. author or translator) acceptance, as well as the triggering of export files. Users of the OpenStax CNX system will typically never directly interact with this application. Almost all of the backend business logic is handled within the publishing application.

Authoring

Unpublished content is stored in Authoring in Postgres. Authoring also has APIs used by the Editor and the Workspace. When content is published, an EPUB is generated and passed from Authoring to Publishing. The EPUB format was selected to pass information between components because it encapsulates all of the info needed.

The Workspace is a listing of all content a user has access to edit. Books and Pages can be created in the Workspace and they can be deleted as well.

OpenStax Accounts

Users are now stored in a shared accounts component. This was developed so users could have the same account on all OpenStax sites. All CNX users have been migrated to the new Accounts. Accounts uses OAuth so users will also be able to log in using Google, Twitter and Facebook. CNX will no longer create CNX user accounts. New users will need to use one of their existing OAuth accounts to log in.

Logging

We are currently logging information, errors and user interactions to Syslog. Our goal is to load the logs into Graphite so we can visually see how our site is being used.

Transformation Services

Transformation Services generates Export files(PDF, EPUB, Zip, etc.) and imports content to the editor. The initial design uses the same import and export code from Legacy CNX inside of a messaging system wrapper.

The messaging using RabbitMQ and several messaging queues. The queues will give us persistence of the file generation requests. The requests will be sent by Publishing after content has been added to Archive. Publishing will pass an EPUB to Transformation Services that contains all of the data needed to generate the files.


OpenStax CNX is a deceptively complex site that has many moving parts. Our goal with this architecture was to design a component based system that can be easily updated and tested without impacting all of the site. All of our code is on Github.

Many thanks to CNX team members Michael Mulich and Derek Kent for reviewing and contributing to this post.

18 Sep 2014 5:51pm GMT

15 Sep 2014

feedPlanet Plone

Cosent: Social intranet sprint Berlin

The Plone social intranet sprint in Berlin at the Humboldt University was attended to by about 16 participants across two tracks, plus a social program in the evenings. Apart from all the hard work we also had a lot of fun. Berlin is a vibrant city and the Ploners who are also Berliners took us to some great restaurants.

strategy track

The strategy track saw enthousiastic responses to a plan for joint investment by ~10 Plone companies into a new Plone intranet software suite. We'll use the coming weeks to solidify the momentum for this initiative and try and convert positive intentions into hard investment commitments.

A key part of the proposed plan is a design-first process to create a compelling user experience, leveraging Plone5-compatible frontend technologies. Netsight, Cosent and Syslab have already made an exploratory first iteration with this design process, focussed on re-designing the microblogging and social activity stream interactions in Plonesocial.

coding track

In the coding track, we've taken these new frontend designs and implemented them on top of the existing Plonesocial code base. The result is, that we now have a Patternslib based Plonesocial implementation that can be installed and run in a Plone 4 installation. Because Plone5 Mockup is also Patternslib-based, the work we're doing is forward compatible and will be easily portable to Plone 5.

In addition to implementing the existing Plonesocial features, the sprint also resulted in the integration of plonesocial.messaging (private one-on-one messages) and new "reply" functionality (conversation threading).

The new frontend introduces a host of new features that are not yet provided by the backend and need to be architected and coded: file uploads, URL and file previews, "@mentions", "liking", "favoriting" etc. Also, we're already working on extending the design with a number of subtle but powerful micro-interactions in the form of shortcodes to provide a pluggable linking system.

The current development version of Plonesocial is more advanced than the last released version, but it needs more work before we can make a production-quality release - so come and join us in the next sprint at Plone Conference 2014 in Bristol.

15 Sep 2014 8:58am GMT

14 Sep 2014

feedPlanet Plone

Tim Knapp: Surprise features that you didn't ask for - Mike Haworth

Server Side Request Forgery

Vulnerable systems:

Urllib: accepts file://

Pycurl accepts gopher://

Memcached: gopher://127.0.0.1:11211/2get

Preventing: blacklist is error prone.

XML External Entity (XXE):

Safer: http://pypi.python.org/pypi/diffusedxml

Signed cookies - session data is visible to user.

Also unless it's got discount applied, just reapply discount. Can have a unique number that is issued each time we give discount.

Don't use pickle as could execute shellcode but instead use JSON.

Pentesting challenges: http://tinyurl.com/h4ckit

14 Sep 2014 5:18am GMT

Tim Knapp: Automated Deployment with Ansible - Juergen Brendel

Configuration - system packages, settings, users, apps

CM Tools - describe the desired state. E.g. ensure all system packages are updated.

Puppet, chef - feel bit like working with Java.

Salt, Ansible - simple.

Ansible - orchestration engine. Written in Python. Uses YAML to write "playbooks". Can use it to configure Windows.

No central configuration server/client (on target).

Just needs SSH access and Python on target machine.

Modules - 100s of them. They know how to do stuff.

Deletes module code from target after running it on target.

Can define groups to operate against.

group_vars directory/ all, groupname1

Ansible 'cloud' modules: AWS, Docker.

Modules: all output to stdout + JSON.

https://github.com/jbrendel/ansible_cluster_setup

14 Sep 2014 4:29am GMT

Tim Knapp: Lightning Talks

LCAuckland

LCA: 12-16 January 2015

Program online in next year or 2.

Singapore also put in proposal for future LCA.

6-700 attendees hoped.

Heritage Preserve #WeWantJam

Grabbed 30TBs of .co.nz internet.

2 days of catering, hacking.

If you'd like to do it again tweet #WeWantJam

Less Boilerplate

import argparse

Command line

http://github.com/aliles/begins

Command line programs for busy developers.

Show us the world

rbenv - isolate different

pyenv shell 3.4.1

pyenv virtualenv name

pyenv local dirname

MySQL performance schema

Debugging schema for the database.

Can configure at runtime.

5.5 or 6 dbs.

Julia

Very pythonic. Stolen lots of good ideas from Python.

Compiled just in time. Timing a file that does nothing it takes 2.25s.

The Python Promotion Pamphlet

International Python Promotion Pamphlet. Glossy & stylised.

Creating NZ-version. Will have high-quality printable version and web-version.

Make your company more visible to Python programmers rather than pay recruitment agents.

Blackbox

http://sheltered-forest-9460.herokuapp.com

http://github.com/tuxbert/blackbox

NZPUG

When you subscribe to the mailing list (607 members on mailing list) you aren't a member of the incorporated society.

Jessica McKellar's Kiwi PyCon73,000 views. Far more than Guido McV's keynote at PyCon US.

Thomi is commandeering LT and announcing my leaving committee and asking if someone can join NZPUG committee. See committee members if would like to join.

Docker for Python

Will do to IT industry what container box did to shipping industry.

LXC (Linux Lightweight Containers) + UnionFS.

Docker file is essentially like a bash script file of commands.

You can inherit from other docker files.

Fig - Vagrant for Docker.

Ship deltas not images.

Shipyard - 3rd party ecosystem.

Gotchas

Object Factories

Sole object is to create objects for testing purposes.

Don't care about what object is. Just assign random attributes to objects.

Code Obfuscation

Weird bugs: object() > object()

If doing tests against time it will fail at midnight depending on your timezone.

False = True

Arduino + Thermal Printer + Python

Arduino sketch, Python script, Git

http://github.com/nick-NZ/Arduino-pull-request-printer

Dynamically creating Python tutorials and presentations

Massey Computer Science moved to Python in 2011.

Emacs: an operating system disguised as an editor.

http://www.orgmode.org

Org-babel - generate code.

Org-slidy

Bad-ass Postgres Tricks

http://bit.ly/pgtrix1

  1. Template Databases - clone dbs
  2. Estimated Counts
  3. Hstore - values have to be strings
  4. Smart Indexes
  5. Schema Change Locks
  6. PL/Python

"One button" test and deploy on AWS

Use Vagrant+Ansible

Scripted creation of AMI and deployment.

Python Antipatterns

Better: list comprehension

Better: raise AssertionError

Better: if pattern in input_str

Don't pass empty list as default are.

if 5 <= i < 10: print "something"

Better: return NotImplementedError

Map reduction using Python scripts

Script to create ArcMap maps.

Making weird maps with Python

Pulled data from census + meshblock boundaries (Koordinates.com) + map box (tile mill) + GDAL.

Should have used Fiona + Shapely.

14 Sep 2014 4:00am GMT

Tim Knapp: Multimedia programming using Gstreamer (and, of course, Python) - Douglas Bagnall

Pipelines consist of linked elements.

Elements:

$ gst-launch-1.0 filesrc location=video.ogv ! oggdemux ! theoradec ! autovideosink

playbin - just play file = mplayer

Probably about 100 media players in world that are based on gstreamer

Muxer: combining 2 streams into 1 file.

Really good example - video wall.

Decode bin -> split into 4 streams (tee) -> queues (often freezes as all running in 1 thread and bump into eachother) -> crop (I.e. Remove 3/4s of video). And also play audio.

Can just use gstreamer cli syntax when calling gi calls in Python.

github: douglasbagnall

http://gstreamer.freedesktop.org

14 Sep 2014 1:58am GMT

Tim Knapp: Understanding human language with Python - Alyona Medelyan

Android Auto: hands-free operation via voice commands.

WordLense: "augmented reality translation"

"Her" movie in reality: "Siri will you marry me? Siri doesn't seem to be available."

Segmentation complexities.

Disambiguation complexities. E.g. Flying planes can be dangerous.

Categories - taxonomy terms, entities - biological/chemical, sentiment.

NKTK - Python toolkit for NLP.

How to get to the core words? Remove stopwords.

from nltk.corpus import stopwords from nltk.tokenizie import word_tokenize

Keyword scoring: TFxIDF

Gensim:

from gensim import corpora, models

When ranking words can use score to discard them.

Text Categorisation with NLTK

Sentiment Analysis:

from textblob import TextBlob

http://deeplearning.net/software/theano

http://textblob.readthedocs.org

http://scikit-learn.org/stable

http://nltk.org

http://radimrehurek.com/gensim

http://github.com/zelandiya/KiwiPyCon-NLP-tutorial

14 Sep 2014 1:30am GMT

13 Sep 2014

feedPlanet Plone

Tim Knapp: PyPy.js: What? How? Why? - Ryan Kelly

@rfkelly

Mozilla - protecting & promoting the web.

The Web? Technology + Culture (open - don't need to submit to 'web pool', ubiquitous, secure, trustworthy)

"For Mozilla, anything that the Web can't do, or anything that the Web is not faster and better at than native technologies, is a bug. We should file it in our Bugzilla system, so we can start writing a patch to fix it" - Andreas Gal (Mozilla CEO)

var vm = new PyPyJS() import js js.eval()

An experiment in building a fast, compliant, in-browser Python environment.

PyPy.js is … Not so Fast. Incompatibilities between the JIT compilers.

Is … Humongous.

How? PyPy + Emscripten.

Emscripten: an LLVM backend that generates JavaScript.

"It's awful all the way down"

Alts: http://GitHub.com/PythonJS

"Please, file bugs against the Web"

http://pypyjs.org

http://github.com/rfk/talk-pypyjs-what-how-why

13 Sep 2014 11:51pm GMT

Tim Knapp: Deploying a Django application using Juju - Tim Penhey

Or "How I used Juju to deploy my Django app up to AWS"

Used backbone.js

Juju is written in Go. Service orchestration in the cloud.

Charms:

Existing Charms

Juju can help with scaling later.

Simple repeatable deployments.

Writing a subordinate charm.

Hooks are just scripts. Used Install hooks and Everything else hooks.

Promoted charms are only ones that are officially available.

"As a Juju core developer it was interesting that if I didn't know the value that Juju would give me later, I would've thrown it out ages ago."

Insufficient documentation around using the Python-Django charm, had to read the code.

Insufficient documentation around how to write a payload charm for Python-Django, very much work it out as you go.

Python-Django and gunicorn still needs nginx to be really useful.

Should have a full stack Django charm.

juju bootstrap|deploy|add-relation|deploy|expose

13 Sep 2014 11:11pm GMT

Tim Knapp: 2nd Keynote: A Snake in Space - The rise of scientific Python in Astrodynamics and Astronomy - Francesco Biscani

Max Planck Institute for Astronomy, European Space Agency.

PaGMO and Astropy - will mainly talk about PaGMO as wrote this and more familiar with this.

PaGMO - Parallel Global Multiobjective Optimiser. Optimisation tool. Initially a trajectory optimisation tool, evolved as a general-purpose optimiser.

Focused on parallel and distributed computing. Can use via C++ or Python.

Astropy - community effort to develop a single core package for Astronomy.

Optimisation - a large area of applied mathematics. "The selection of a best element (with regard to some criteria) from some set of available alternatives." E.g. Travelling salesman (TSP).

E.g. algorithms: gradient-based methods, evolutionary algorithms, stochastic algorithms.

Interplanetary trajectories - space mission trajectories are defined by sets of parameters: launch date, initial velocity vector, sequence of flybys, sequence of deep-space manoeuvres (DSM).

Usually we want to minimise fuel consumption.

The resulting optimisation problem is hard: multimodal objective function, highly nonlinear, highly dimensional.

Traditionally tackled by teams of human experts.

Bio-inspired algorithms:

Island model - name inspired by Darwin's trip to the Galápagos Islands.

History of PaGMO - pattern of scientific programming code: created as part of some research and then abandoned for many years; not useable by anyone else and then picked up later and made more consumable by others.

Initially created Python bindings to initially created C/C++ 'research' code (2008-2009) and followed 'eat own dog food' approach by using it a lot for internal research. Been through 2 GSoCs and now 'fully-fledged' open source project.

Pros of Scientific Computing

Cons of Scientific Computing

PaGMO framework:

Implemented via Boost.Python.

Challenging, non-trivial issues:

Leveraging Python's strengths:

PaGMO uses: evolution of neural networks for autonomous Martian rovers, selection of Near Earth Asteroids for future human missions.

Astropy - "a community package for Astronomers":

13 Sep 2014 10:37pm GMT

Daniel Nouri: Identifying birds, butterflies, and wildflowers with a snap

It's been a while since I started my little adventure of building an image recognition pipeline that would reliably identify animal and plant species from photos. I'm pretty happy with what the results are so far, and these days I'm even successfully scratching my own itch with it. Which is how I came up with the idea in the first place: I suck at identifying birds, butterflies, and plants, but I want to learn about them. And knowing an animal or plant's name is really the gateway to learning more about it.

So here's a painless way of identifying these species. Say you want to identify one bird that you see (and you're lucky to have a camera with a big zoom with you). You take a photo of it and then send it to @id_birds on Twitter using Twitter's own image upload. After a minute or two, you'll get the automatic reply, with hopefully the right species name. @id_birds can with a remarkably high accuracy identify yours between the 256 different birds that it knows.

There's two other bots that are also available to use for free: @id_butterflies knows around 50 butterfly species, and @id_wildflowers around 200 wildflowers. All three bots know species occurring natively in Europe only, at least for now.

"But does it actually work?" I hear you say. And the answer is: it works pretty well, especially if you send good quality photos where the animal or plant is clearly visible.

Seeing is believing though, so to prove that it works well, let's try and identify the ten birds that are in The Wildlife Trust's river bird spotter sheet.

http://danielnouri.org/media/river-bird-spotter.jpg

Image credits: Grey heron (c) Amy Lewis / Moorhen and mallard (c) Steve Waterhouse / Kingfisher (c) Margaret Holland / Mute swan (c) Neil Aldridge / Dipper and grey wagtail (c) Tom Marshall / Coot (c) David Longshaw / Great crested grebe (c) Don Sutherland / Tufted duck (c) Bob Coyle

Let's start with the first bird, a grey heron. To the left we have me addressing the Twitter bot and sending the image. To the right is @id_birds' reply. And it's correct! Ardea cinerea is indeed the scientific name for the grey heron:

(If you're not seeing the tweets below, I suggest you view this on my blog.)

@id_birds hey can you ID me this bird from @wildlifetrust's river birds spotter sheet? pic.twitter.com/z5JnE3WIb3

- Daniel Nouri (@dnouri) September 13, 2014

@dnouri 1. Ardea cinerea http://t.co/gGuu8ombMs 2. Anthropoides virgo http://t.co/exqiChamSg

- id_birds (@id_birds) September 13, 2014

Notice how @id_birds will always respond with the two most likely guesses. The second guess here was the demoiselle crane (Anthropoides virgo), which looks sort of similar.

The next bird on the river bird spotter sheet is the common moorhen (Gallinula chloropus). Let's try to ID it:

@id_birds how about this one? pic.twitter.com/9csoNaAaJw

- Daniel Nouri (@dnouri) September 13, 2014

@dnouri 1. Anas platyrhynchos http://t.co/9xoDyXJGMJ 2. Gallinula chloropus http://t.co/ISQ0N7m2Lu

- id_birds (@id_birds) September 13, 2014

And already we have @id_birds' first mis-classification. It thinks that this moorhen is a mallard (Anas platyrhynchos). Only the second guess is correct. Thus so far we're counting one correct, one error.

The next bird we want to identify is incidentally a mallard:

@id_birds pic.twitter.com/x9h89dKkkb

- id_mushrooms (@id_mushrooms) September 13, 2014

@id_mushrooms 1. Anas platyrhynchos http://t.co/9xoDyXJGMJ 2. Coracias garrulus http://t.co/MKJTOTfeOv

- id_birds (@id_birds) September 13, 2014

And the response is correct. Notice how the second guess (Coracias garrulus) seems a little weird. I guess it's the similarities in color here. We don't count the second guesses here though, so it's two correct, one error.

What about the kingfisher? That's a bird with a pretty unique appearance among birds native to Europe. So it's probably one of the easier birds to identify:

@id_birds pic.twitter.com/NNy1t6izLQ

- id_butterflies (@id_butterflies) September 13, 2014

@id_butterflies 1. Alcedo atthis http://t.co/Q1fUKAlYSt 2. Hirundo rustica http://t.co/mtVqrMC8oG

- id_birds (@id_birds) September 13, 2014

And sure enough, @id_birds knows what it is. The latin name for the kingfisher is Alcedo atthis.

The fifth bird on the spotter sheet is the mute swan (Cygnus olor). Everyone probably knows this one, but does @id_birds?

@id_birds and this? pic.twitter.com/RiFk3pP38B

- id_butterflies (@id_butterflies) September 13, 2014

@id_butterflies 1. Cygnus olor http://t.co/jz7DLdqVW2 2. Cygnus cygnus http://t.co/wx8MSFv3sy

- id_birds (@id_birds) September 13, 2014

Yes, the answer is correct. Note that the second guess is also a swan, namely the whooper swan, quite similar in appearance. A good job!

So out of the first five birds we got four correct, not bad. But let's see about the rest of the birds from the spotter sheet.

I'll post the responses to the remaining five birds without further comment. But do note that of the remaining ones, only the coot (Fulica atra) is mis-classified, that's the third one below:

@id_birds I found this one in @wildlifetrust's river birds spotter sheet. Do you know what it is? pic.twitter.com/R9Qf5s2sSf

- id_wildflowers (@id_wildflowers) September 13, 2014

@id_wildflowers 1. Cinclus cinclus http://t.co/u2W2R8NYZX 2. Corvus corone http://t.co/1JjU7HtcSj

- id_birds (@id_birds) September 13, 2014

@id_birds Here's another cute bird from that same spotter sheet. Can you ID it? pic.twitter.com/onxOeALtaL

- Daniel Nouri (@dnouri) September 13, 2014

@dnouri 1. Motacilla cinerea http://t.co/RhZyl5ltwy 2. Motacilla flava http://t.co/TWG5YgTxBQ

- id_birds (@id_birds) September 13, 2014

@id_birds pic.twitter.com/qYVqQGKa4I

- id_wildflowers (@id_wildflowers) September 13, 2014

@id_wildflowers 1. Gallinula chloropus http://t.co/ISQ0N7m2Lu 2. Podiceps nigricollis http://t.co/WgflyFJyCU

- id_birds (@id_birds) September 13, 2014

@id_birds Can you ID this as Podiceps cristatus? pic.twitter.com/jQwS5cjuEi

- id_wildflowers (@id_wildflowers) September 13, 2014

@id_wildflowers 1. Podiceps cristatus http://t.co/JzrmePWuf9 2. Branta ruficollis http://t.co/X6fCvqF0xo

- id_birds (@id_birds) September 13, 2014

@id_birds here's the last one: a tufted duck (Aythya fuligula) pic.twitter.com/S8VwHd6rry

- id_butterflies (@id_butterflies) September 13, 2014

@id_butterflies 1. Aythya fuligula http://t.co/n4O4tTIwtv 2. Aythya marila http://t.co/H9g0mXy6Eu

- id_birds (@id_birds) September 13, 2014

So all in all we have two mis-classifications out of ten. That's 8 out of 10 accuracy, which is pretty remarkable when you consider that the differences between these 256 different birds species can sometimes be minuscule. As an example, compare the grey wagtail (Motacilla cinerea) from above with the yellow wagtail. According to Wikipedia, the grey wagtail "looks similar to the yellow wagtail but has the yellow on its underside restricted to the throat and vent."

Of course ten picture is a tiny test set, but the score roughly matches what I see with the much larger test set that I use for development, so it's cool. :-)

I'll write about how the computer vision works under the hood another time (but here's a hint). We call the stack that we've developed and that's running behind @id_birds and the other bots PhotoID. So far PhotoID has proven to have very competitive performance and be remarkably flexible to use for different image classification problems.

Let's conclude this post with two other IDs, one from @id_butterflies, the other from @id_wildflowers:

#WansteadFlats It's an upside down world sometimes. Maybe a Meadow Brown? @id_butterflies @wildlife_uk pic.twitter.com/yDnkTUTMO8

- TheNatureOfTheFlats (@RoseStephensArt) August 22, 2014

@id_butterflies 1. Maniola jurtina http://t.co/4YGYG6WQHc 2. Cupido minimus http://t.co/2hbLKCYzpF

- id_butterflies (@id_butterflies) August 22, 2014

Can @id_wildflowers identify this wildflower photographed by @LornaCrowcroft? Wondering what it is? See the reply. pic.twitter.com/WI31e7aN8x

- id_wildflowers (@id_wildflowers) September 4, 2014

@id_wildflowers 1. Solanum nigrum http://t.co/kvMbrC7KB4 2. Cyclamen hederifolium http://t.co/mcZHVmRecq

- id_wildflowers (@id_wildflowers) September 4, 2014

Happy ID'ing!

(Plug: Feel free to contact me through my company if you have a challenging image recognition problem that you want us to help solve.)

13 Sep 2014 6:45pm GMT

Daniel Nouri: Idenitfying birds, butterflies, and wildflowers with a snap

It's been a while since I started my little adventure of building an image recognition pipeline that would reliably identify animal and plant species from photos. I'm pretty happy with what the results so far, and these days I'm even successfully scratching my own itch with it. Which is how I came up with the idea in the first place: I suck at identifying birds, butterflies, and plants, but I want to learn about them. And knowing a species' name is really the gateway to learning more about it.

So here's a painless way of identifying these species. Say you want to identify one bird that you see. You take a photo of it and send it to @id_birds on Twitter using Twitter's own image upload. After a minute or two, you'll get the automatic reply, with hopefully the right species name. @id_birds can with a remarkably high accuracy identify yours between the 256 different birds that it knows.

There's two other bots that are also available to use for free: @id_butterflies knows around 50 butterfly species, and @id_wildflowers around 200 wildflowers. All three bots know species occurring natively in Europe only, at least for now.

"But does it actually work?" I hear you say. And the answer is: it works pretty well, especially if you're sending it good quality photos where the animal or plant is clearly visible, is not too blurry, has good white balance etc.

Seeing is believing though, so to prove that it works well, let's try and identify the ten birds that are in The Wildlife Trust's river bird spotter sheet.

http://danielnouri.org/media/river-bird-spotter.jpg

Image credits: Grey heron (c) Amy Lewis / Moorhen and mallard (c) Steve Waterhouse / Kingfisher (c) Margaret Holland / Mute swan (c) Neil Aldridge / Dipper and grey wagtail (c) Tom Marshall / Coot (c) David Longshaw / Great crested grebe (c) Don Sutherland / Tufted duck (c) Bob Coyle

Let's start with the first bird, a grey heron. To the left we have me addressing the Twitter bot and sending the image. To the right is @id_birds' reply. And it's correct! Ardea cinerea is indeed the scientific name for the grey heron:

@id_birds hey can you ID me this bird from @wildlifetrust's river birds spotter sheet? pic.twitter.com/z5JnE3WIb3

- Daniel Nouri (@dnouri) September 13, 2014

@dnouri 1. Ardea cinerea http://t.co/gGuu8ombMs 2. Anthropoides virgo http://t.co/exqiChamSg

- Bird auto-ID (@id_birds) September 13, 2014

Notice how @id_birds will always respond with the two most likely guesses. The second guess here was the demoiselle crane (Anthropoides virgo), which looks sort of similar.

The next bird on the river bird spotter sheet is the common moorhen (Gallinula chloropus). Let's try to ID it:

@id_birds how about this one? pic.twitter.com/9csoNaAaJw

- Daniel Nouri (@dnouri) September 13, 2014

@dnouri 1. Anas platyrhynchos http://t.co/9xoDyXJGMJ 2. Gallinula chloropus http://t.co/ISQ0N7m2Lu

- Bird auto-ID (@id_birds) September 13, 2014

And already we have @id_birds' first mis-classification. It thinks that this is a mallard (Anas platyrhynchos). Only the second guess was correct. Thus so far we're counting one correct, one error.

The next bird we want to identify is incidentally a mallard:

@id_birds pic.twitter.com/x9h89dKkkb

- Mushroom auto-id (@id_mushrooms) September 13, 2014

@id_mushrooms 1. Anas platyrhynchos http://t.co/9xoDyXJGMJ 2. Coracias garrulus http://t.co/MKJTOTfeOv

- Bird auto-ID (@id_birds) September 13, 2014

And the response is correct. Notice how the second guess (Coracias garrulus) seems a little weird. I guess it's the similarities in color here. We don't count the second guesses here though, so it's two correct, one error.

What about the kingfisher? That's a bird with some pretty unique appearance among birds native to Europe. So it's probably one of the easier birds to identify:

@id_birds pic.twitter.com/NNy1t6izLQ

- Butterfly auto-ID (@id_butterflies) September 13, 2014

@id_butterflies 1. Alcedo atthis http://t.co/Q1fUKAlYSt 2. Hirundo rustica http://t.co/mtVqrMC8oG

- Bird auto-ID (@id_birds) September 13, 2014

And sure enough, @id_birds knows what it is. The latin name for the kingfisher is Alcedo atthis.

The fifth bird on the spotter sheet is the mute swan (Cygnus olor). Everyone probably knows this one, but does @id_birds?

@id_birds and this? pic.twitter.com/RiFk3pP38B

- Butterfly auto-ID (@id_butterflies) September 13, 2014

@id_butterflies 1. Cygnus olor http://t.co/jz7DLdqVW2 2. Cygnus cygnus http://t.co/wx8MSFv3sy

- Bird auto-ID (@id_birds) September 13, 2014

Yes, the answer is correct. Note that the second guess is also a swan, namely the whooper swan, quite similar in appearance. A good job!

So out of the first five birds we got four correct, not bad. But let's see about the rest of the birds from the spotter sheet.

I'll post the responses to the remaining five birds without further comment. But do note that of the remaining ones, only the coot (Fulica atra) is mis-classified, that's the third one below:

@id_birds I found this one in @wildlifetrust's river birds spotter sheet. Do you know what it is? pic.twitter.com/R9Qf5s2sSf

- Wildflower auto-ID (@id_wildflowers) September 13, 2014

@id_wildflowers 1. Cinclus cinclus http://t.co/u2W2R8NYZX 2. Corvus corone http://t.co/1JjU7HtcSj

- Bird auto-ID (@id_birds) September 13, 2014

@id_birds Here's another cute bird from that same spotter sheet. Can you ID it? pic.twitter.com/onxOeALtaL

- Daniel Nouri (@dnouri) September 13, 2014

@dnouri 1. Motacilla cinerea http://t.co/RhZyl5ltwy 2. Motacilla flava http://t.co/TWG5YgTxBQ

- Bird auto-ID (@id_birds) September 13, 2014

@id_birds pic.twitter.com/qYVqQGKa4I

- Wildflower auto-ID (@id_wildflowers) September 13, 2014

@id_wildflowers 1. Gallinula chloropus http://t.co/ISQ0N7m2Lu 2. Podiceps nigricollis http://t.co/WgflyFJyCU

- Bird auto-ID (@id_birds) September 13, 2014

@id_birds Can you ID this as Podiceps cristatus? pic.twitter.com/jQwS5cjuEi

- Wildflower auto-ID (@id_wildflowers) September 13, 2014

@id_wildflowers 1. Podiceps cristatus http://t.co/JzrmePWuf9 2. Branta ruficollis http://t.co/X6fCvqF0xo

- Bird auto-ID (@id_birds) September 13, 2014

@id_birds here's the last one: a tufted duck (Aythya fuligula) pic.twitter.com/S8VwHd6rry

- Butterfly auto-ID (@id_butterflies) September 13, 2014

@id_butterflies 1. Aythya fuligula http://t.co/n4O4tTIwtv 2. Aythya marila http://t.co/H9g0mXy6Eu

- Bird auto-ID (@id_birds) September 13, 2014

So all in all we have two mis-classifications out of ten. That's 80% accuracy, which is pretty remarkable when you consider that differences between these birds can sometimes be minuscule (grey wagtail versus yellow wagtail, anyone?). As mentioned, @id_birds knows 256 bird species. Which means that if you made guesses at random, then you'd have 0.4% accuracy.

I'll write about how the computer vision algorithm works under the hood another time. I call the stack that I've developed and that's running behind @id_birds and the other bots PhotoID. So far PhotoID has proven to have very competitive performance and be remarkably flexible, that is, I can use it as a basis for different kinds of classification problems without too much adaptation work.

Let's conclude this post with two other IDs, one from @id_butterflies, the other from @id_wildflowers:

#WansteadFlats It's an upside down world sometimes. Maybe a Meadow Brown? @id_butterflies @wildlife_uk pic.twitter.com/yDnkTUTMO8

- TheNatureOfTheFlats (@RoseStephensArt) August 22, 2014

@id_butterflies 1. Maniola jurtina http://t.co/4YGYG6WQHc 2. Cupido minimus http://t.co/2hbLKCYzpF

- Butterfly auto-ID (@id_butterflies) August 22, 2014

Can @id_wildflowers identify this wildflower photographed by @LornaCrowcroft? Wondering what it is? See the reply. pic.twitter.com/WI31e7aN8x

- Wildflower auto-ID (@id_wildflowers) September 4, 2014

@id_wildflowers 1. Solanum nigrum http://t.co/kvMbrC7KB4 2. Cyclamen hederifolium http://t.co/mcZHVmRecq

- Wildflower auto-ID (@id_wildflowers) September 4, 2014

Happy ID'ing!

(Plug: Feel free to contact me through my company if you have a challenging image recognition problem that you want me to help solve.)

13 Sep 2014 4:34pm GMT

Tim Knapp: Java for Python Developers - Chris Neugebauer

One company who likes Python but in reality likes Java a lot more = …. Google!

.java -> [compiler] -> JVM ByteCode -> JVM

Jython - setuptools - make .jars

from java.Lang.system

What can do in Python 2.7 can also do in Jython.

Android devices outsold full-size computers 2x.

Dalvik Virtual Machine = Android Runtime (ART)

.java -> [compiler] -> .dex (Dalvik exe) -> DVM

No dynamic 3rd-party libraries. All statically compiled into Dalvik exe.

Java Native Interface (JNI).

Kivy

Python-For-Android: Cpython API for Java.

from jnius import autoclass

PyJnius runs on Android!

JNI method signatures are a nightmare.

If Java's more important, use Jython! Support for C extensions is not complete.

If Python's more important, use PyJnius!

Jython is corporate sponsored by Sun.

2.7 support on Jython is only recent.

Kivy doesn't have a very native look-and-feel. They say they're working on this but "they don't know what they're talking about".

Toga aims to use the native APIs for theming but currently has no Android support.

13 Sep 2014 5:01am GMT

Tim Knapp: Intro to flask-security - Beau Butler

Why Flask? Steep Django learning curve. Like to start simple and then become more complex.

App example code: http://github.com/oddy/bangingminimal

Don't get Async mail sending, Pretty forms.

You get: sess auth + RBAC, base users db, all std workflows, passwd crypto, token/JSON auth.

pip install flask flask-security flask-SQLAlchemy pyBcrypt paste

Google: Prove Domain ownership - use button option.

Black box testing: Know lots of hackers.

Need motivate hackers - apply drink.

How hacked? ARP Redir (had PC on same subnet), SSLSplit, click forgot password, intercept reset URL, reset URL.

Actual hack - other PC pretending to be Google SMTP server (on same subnet) intercepted program call and passed call on to Google SMTP server and hacker's app.

Py smtplib doesn't check Google's cert! 2.x smtplib can't do it at all.

Admin Ronacher FTW.

Flask-Sec RBAC itself still not broken.

Have a go: pwnoddy.com.

13 Sep 2014 4:25am GMT