26 Jul 2014

feedPlanet Python

End Point: Python Subprocess Wrapping with sh

When working with shell scripts written in bash/csh/etc. one of the primary tools you have to rely on is a simple method of piping output and input from subprocesses called by the script to create complex logic to accomplish the goal of the script. When working with python, this same method of calling subprocesses to redirect the input/output is available, but the overhead of using this method in python would be so cumbersome as to make python a less desirable scripting language. In effect you were implementing large parts of the I/O facilities, and potentially even writing replacements for the existing shell utilities that would perform the same work. Recently, python developers attempted to solve this problem, by updating an existing python subprocess wrapper library called pbs, into an easier to use library called sh.

Sh can be installed using pip, and the author has posted some documentation for the library here: http://amoffat.github.io/sh/

Using the sh library

After installing the library into your version of python, there will be two ways to call any existing shell command available to the system, firstly you can import the command as though it was itself a python library:

from sh import hostname
print(hostname())


In addition, you can also call the command directly by just referencing the sh namespace prior to the command name:

import sh
print(sh.hostname())


When running this command on my linux workstation (hostname atlas) it will return the expected results:

Python 2.7.6 (default, Mar 22 2014, 22:59:56)
[GCC 4.8.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sh
>>> print(sh.hostname())
atlas


However at this point, we are merely replacing a single shell command which prints output to the screen, the real benefit of the shell scripts was that you could chain together commands in order to create complex logic to help you do work.

Advanced Gymnastics

A common use of shell scripts is to provide administrators the ability to quickly filter log file output and to potentially search for specific conditions within those logs, to alert in the event that an application starts throwing errors. With python piping in sh we can create a simple log watcher, which would be capable of calling anything we desire in python when the log file contains any of the conditions we are looking for.

To pipe together commands using the sh library, you would encapsulate each command in series to create a similar syntax to bash piping:

>>> print(sh.wc(sh.ls("-l", "/etc"), "-l"))
199


This command would have been equivalent to the bash pipe of "ls -l /etc | wc -l" indicating that the long listing of /etc on my workstation contained 199 lines of output. Each piped command is encapsulated inside the parenthesis of the command the precedes it.

For our log listener we will use the tail command along with a python iterator to watch for a potential error condition, which I will represent with the string "ERROR":

>>> for line in sh.tail("-f", "/tmp/test_log", _iter=True):
...     if "ERROR" in line:
...         print line


In this example, once executed, python will call the tail command to follow a particular log file. It will iterate over each line of output produced by tail and if any of the lines contain the string we are watching for python will print that line to standard output. At this point, this would be similar to using the tail command and piping the output to a string search command, like grep. However, you could replace the third line of the python with a more complex action, emailing the error condition out to a developer or administrator for review, or perhaps initiating a procedure to recover from the error automatically.

Conclusions


In this manner with just a few lines of python, much like with bash, one could create a relatively complex process without recreating all the shell commands which already perform this work, or create a convoluted wrapping process of passing output from command to command. This combination of the existing shell commands and the power of python; you get all the functions available to any python environment, with the ease of using the shell commands to do some of the work. In the future I will definitely be using this python library for my own shell scripting needs, as I have generally preferred the syntax and ease of use of python over that of bash, but now I will be able to enjoy both at the same time.

26 Jul 2014 12:38am GMT

End Point: Python Subprocess Wrapping with sh

When working with shell scripts written in bash/csh/etc. one of the primary tools you have to rely on is a simple method of piping output and input from subprocesses called by the script to create complex logic to accomplish the goal of the script. When working with python, this same method of calling subprocesses to redirect the input/output is available, but the overhead of using this method in python would be so cumbersome as to make python a less desirable scripting language. In effect you were implementing large parts of the I/O facilities, and potentially even writing replacements for the existing shell utilities that would perform the same work. Recently, python developers attempted to solve this problem, by updating an existing python subprocess wrapper library called pbs, into an easier to use library called sh.

Sh can be installed using pip, and the author has posted some documentation for the library here: http://amoffat.github.io/sh/

Using the sh library

After installing the library into your version of python, there will be two ways to call any existing shell command available to the system, firstly you can import the command as though it was itself a python library:

from sh import hostname
print(hostname())


In addition, you can also call the command directly by just referencing the sh namespace prior to the command name:

import sh
print(sh.hostname())


When running this command on my linux workstation (hostname atlas) it will return the expected results:

Python 2.7.6 (default, Mar 22 2014, 22:59:56)
[GCC 4.8.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sh
>>> print(sh.hostname())
atlas


However at this point, we are merely replacing a single shell command which prints output to the screen, the real benefit of the shell scripts was that you could chain together commands in order to create complex logic to help you do work.

Advanced Gymnastics

A common use of shell scripts is to provide administrators the ability to quickly filter log file output and to potentially search for specific conditions within those logs, to alert in the event that an application starts throwing errors. With python piping in sh we can create a simple log watcher, which would be capable of calling anything we desire in python when the log file contains any of the conditions we are looking for.

To pipe together commands using the sh library, you would encapsulate each command in series to create a similar syntax to bash piping:

>>> print(sh.wc(sh.ls("-l", "/etc"), "-l"))
199


This command would have been equivalent to the bash pipe of "ls -l /etc | wc -l" indicating that the long listing of /etc on my workstation contained 199 lines of output. Each piped command is encapsulated inside the parenthesis of the command the precedes it.

For our log listener we will use the tail command along with a python iterator to watch for a potential error condition, which I will represent with the string "ERROR":

>>> for line in sh.tail("-f", "/tmp/test_log", _iter=True):
...     if "ERROR" in line:
...         print line


In this example, once executed, python will call the tail command to follow a particular log file. It will iterate over each line of output produced by tail and if any of the lines contain the string we are watching for python will print that line to standard output. At this point, this would be similar to using the tail command and piping the output to a string search command, like grep. However, you could replace the third line of the python with a more complex action, emailing the error condition out to a developer or administrator for review, or perhaps initiating a procedure to recover from the error automatically.

Conclusions


In this manner with just a few lines of python, much like with bash, one could create a relatively complex process without recreating all the shell commands which already perform this work, or create a convoluted wrapping process of passing output from command to command. This combination of the existing shell commands and the power of python; you get all the functions available to any python environment, with the ease of using the shell commands to do some of the work. In the future I will definitely be using this python library for my own shell scripting needs, as I have generally preferred the syntax and ease of use of python over that of bash, but now I will be able to enjoy both at the same time.

26 Jul 2014 12:38am GMT

End Point: Python Imports

For a Python project I'm working on, I wrote a parent class with multiple child classes, each of which made use of various modules that were imported in the parent class. A quick solution to making these modules available in the child classes would be to use wildcard imports in the child classes:

from package.parent import *

however, PEP8 warns against this stating "they make it unclear which names are present in the namespace, confusing both readers and many automated tools."

For example, suppose we have three files:

# a.py
import module1
class A(object):
    def __init__():
        pass

# b.py
import module2
class B(A):
    def __init__():
        super(B, self).__init__()

# c.py
class C(B):
    def __init__():
        super(C, self).__init__()

To someone reading just b.py or c.py, it is unknown that module1 is present in the namespace of B and that both module1 and module2 are present in the namespace of C. So, following PEP8, I just explicitly imported any module needed in each child class. Because in my case there were many imports and because it seemed repetitive to have all those imports duplicated in each of the many child classes, I wanted to find out if there was a better solution. While I still don't know if there is, I did go down the road of how imports work in Python, at least for 3.4.1, and will share my notes with you.

Python allows you to import modules using the import statement, the built-in function __import__(), and the function importlib.import_module(). The differences between these are:

The import statement first "searches for the named module, then it binds the results of that search to a name in the local scope" (Python Documentation). Example:

Python 3.4.1 (default, Jul 15 2014, 13:05:56) 
[GCC 4.8.2] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import re
>>> re
<module 're' from '/home/miguel/.pythonbrew/pythons/Python-3.4.1/lib/python3.4/re.py'>
>>> re.sub('s', '', 'bananas')
'banana'

Here the import statement searches for a module named re then binds the result to the variable named re. You can then call re module functions with re.function_name().

A call to function __import__() performs the module search but not the binding; that is left to you. Example:

>>> muh_regex = __import__('re')
>>> muh_regex
<module 're' from '/home/miguel/.pythonbrew/pythons/Python-3.4.1/lib/python3.4/re.py'>
>>> muh_regex.sub('s', '', 'bananas')
'banana'

Your third option is to use importlib.import_module() which, like __import__(), only performs the search:

>>> import importlib
>>> muh_regex = importlib.import_module('re')
>>> muh_regex
<module 're' from '/home/miguel/.pythonbrew/pythons/Python-3.4.1/lib/python3.4/re.py'>
>>> muh_regex.sub('s', '', 'bananas')
'banana'

Let's now talk about how Python searches for modules. The first place it looks is in sys.modules, which is a dictionary that caches previously imported modules:

>>> import sys
>>> 're' in sys.modules
False
>>> import re
>>> 're' in sys.modules
True
>>> sys.modules['re']
&ltmodule 're' from '/home/miguel/.pythonbrew/pythons/Python-3.4.1/lib/python3.4/re.py'>

If the module is not found in sys.modules Python searches sys.meta_path, which is a list that contains finder objects. Finders, along with loaders, are objects in Python's import protocol. The job of a finder is to return a module spec, using method find_spec(), containing the module's import-related information which the loader then uses to load the actual module. Let's see what I have in my sys.meta_path:

>>> sys.meta_path
[<class '_frozen_importlib.BuiltinImporter'>, <class '_frozen_importlib.FrozenImporter'>, <class '_frozen_importlib.PathFinder'>]

Python will use each finder object in sys.meta_path until the module is found and will raise an ImportError if it is not found. Let's call find_spec() with parameter 're' on each of these finder objects:

>>> sys.meta_path[0].find_spec('re')
>>> sys.meta_path[1].find_spec('re')
>>> sys.meta_path[2].find_spec('re')
ModuleSpec(name='re', loader=_frozen_importlib.SourceFileLoader object at 0x7ff7eb314438>, origin='/home/miguel/.pythonbrew/pythons/Python-3.4.1/lib/python3.4/re.py')

The first finder knows how to find built-in modules and since re is not a built-in module, it returns None.

>>> 're' in sys.builtin_module_names
False

The second finder knows how to find frozen modules, which re is not. The third knows how to find modules from a list of path entries called an import path. For re the import path is sys.path but for subpackages the import path can be the parent's __path__ attribute.

>>>sys.path
['', '/home/miguel/.pythonbrew/pythons/Python-3.4.1/lib/python3.4/site-packages/distribute-0.6.49-py3.4.egg', '/home/miguel/.pythonbrew/pythons/Python-3.4.1/lib', '/home/miguel/.pythonbrew/pythons/Python-3.4.1/lib/python34.zip', '/home/miguel/.pythonbrew/pythons/Python-3.4.1/lib/python3.4', '/home/miguel/.pythonbrew/pythons/Python-3.4.1/lib/python3.4/plat-linux', '/home/miguel/.pythonbrew/pythons/Python-3.4.1/lib/python3.4/lib-dynload', '/home/miguel/.pythonbrew/pythons/Python-3.4.1/lib/python3.4/site-packages', '/home/miguel/.pythonbrew/pythons/Python-3.4.1/lib/python3.4/site-packages/setuptools-0.6c11-py3.4.egg-info']

Once the module spec is found, the loading machinery takes over. That's as far as I dug but you can read more about the loading process by reading the documentation.

26 Jul 2014 12:33am GMT

End Point: Python Imports

For a Python project I'm working on, I wrote a parent class with multiple child classes, each of which made use of various modules that were imported in the parent class. A quick solution to making these modules available in the child classes would be to use wildcard imports in the child classes:

from package.parent import *

however, PEP8 warns against this stating "they make it unclear which names are present in the namespace, confusing both readers and many automated tools."

For example, suppose we have three files:

# a.py
import module1
class A(object):
    def __init__():
        pass

# b.py
import module2
class B(A):
    def __init__():
        super(B, self).__init__()

# c.py
class C(B):
    def __init__():
        super(C, self).__init__()

To someone reading just b.py or c.py, it is unknown that module1 is present in the namespace of B and that both module1 and module2 are present in the namespace of C. So, following PEP8, I just explicitly imported any module needed in each child class. Because in my case there were many imports and because it seemed repetitive to have all those imports duplicated in each of the many child classes, I wanted to find out if there was a better solution. While I still don't know if there is, I did go down the road of how imports work in Python, at least for 3.4.1, and will share my notes with you.

Python allows you to import modules using the import statement, the built-in function __import__(), and the function importlib.import_module(). The differences between these are:

The import statement first "searches for the named module, then it binds the results of that search to a name in the local scope" (Python Documentation). Example:

Python 3.4.1 (default, Jul 15 2014, 13:05:56) 
[GCC 4.8.2] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import re
>>> re
<module 're' from '/home/miguel/.pythonbrew/pythons/Python-3.4.1/lib/python3.4/re.py'>
>>> re.sub('s', '', 'bananas')
'banana'

Here the import statement searches for a module named re then binds the result to the variable named re. You can then call re module functions with re.function_name().

A call to function __import__() performs the module search but not the binding; that is left to you. Example:

>>> muh_regex = __import__('re')
>>> muh_regex
<module 're' from '/home/miguel/.pythonbrew/pythons/Python-3.4.1/lib/python3.4/re.py'>
>>> muh_regex.sub('s', '', 'bananas')
'banana'

Your third option is to use importlib.import_module() which, like __import__(), only performs the search:

>>> import importlib
>>> muh_regex = importlib.import_module('re')
>>> muh_regex
<module 're' from '/home/miguel/.pythonbrew/pythons/Python-3.4.1/lib/python3.4/re.py'>
>>> muh_regex.sub('s', '', 'bananas')
'banana'

Let's now talk about how Python searches for modules. The first place it looks is in sys.modules, which is a dictionary that caches previously imported modules:

>>> import sys
>>> 're' in sys.modules
False
>>> import re
>>> 're' in sys.modules
True
>>> sys.modules['re']
&ltmodule 're' from '/home/miguel/.pythonbrew/pythons/Python-3.4.1/lib/python3.4/re.py'>

If the module is not found in sys.modules Python searches sys.meta_path, which is a list that contains finder objects. Finders, along with loaders, are objects in Python's import protocol. The job of a finder is to return a module spec, using method find_spec(), containing the module's import-related information which the loader then uses to load the actual module. Let's see what I have in my sys.meta_path:

>>> sys.meta_path
[<class '_frozen_importlib.BuiltinImporter'>, <class '_frozen_importlib.FrozenImporter'>, <class '_frozen_importlib.PathFinder'>]

Python will use each finder object in sys.meta_path until the module is found and will raise an ImportError if it is not found. Let's call find_spec() with parameter 're' on each of these finder objects:

>>> sys.meta_path[0].find_spec('re')
>>> sys.meta_path[1].find_spec('re')
>>> sys.meta_path[2].find_spec('re')
ModuleSpec(name='re', loader=_frozen_importlib.SourceFileLoader object at 0x7ff7eb314438>, origin='/home/miguel/.pythonbrew/pythons/Python-3.4.1/lib/python3.4/re.py')

The first finder knows how to find built-in modules and since re is not a built-in module, it returns None.

>>> 're' in sys.builtin_module_names
False

The second finder knows how to find frozen modules, which re is not. The third knows how to find modules from a list of path entries called an import path. For re the import path is sys.path but for subpackages the import path can be the parent's __path__ attribute.

>>>sys.path
['', '/home/miguel/.pythonbrew/pythons/Python-3.4.1/lib/python3.4/site-packages/distribute-0.6.49-py3.4.egg', '/home/miguel/.pythonbrew/pythons/Python-3.4.1/lib', '/home/miguel/.pythonbrew/pythons/Python-3.4.1/lib/python34.zip', '/home/miguel/.pythonbrew/pythons/Python-3.4.1/lib/python3.4', '/home/miguel/.pythonbrew/pythons/Python-3.4.1/lib/python3.4/plat-linux', '/home/miguel/.pythonbrew/pythons/Python-3.4.1/lib/python3.4/lib-dynload', '/home/miguel/.pythonbrew/pythons/Python-3.4.1/lib/python3.4/site-packages', '/home/miguel/.pythonbrew/pythons/Python-3.4.1/lib/python3.4/site-packages/setuptools-0.6c11-py3.4.egg-info']

Once the module spec is found, the loading machinery takes over. That's as far as I dug but you can read more about the loading process by reading the documentation.

26 Jul 2014 12:33am GMT

25 Jul 2014

feedPlanet Python

Geert Vanderkelen: MySQL Connector/Python v2.0.0 alpha

A new major version of Connector/Python is available: v2.0.0 alpha has been been released and is available for download! As with any alpha-software, it's probably not good to throw it in production just yet.

Our manual has the full change log but here's an overview of most important changes for this relase.

Some incompatibilities

The world evolves, at least the software does, and Python is not different. I'm not as bold as the guys at Django who dropped support of Python v2.6 with the Django v1.7 release. I'm leaving it in because I'm nice.

Supported Python: 2.6 and 2.7 and 3.3 and 3.4

We do not support any longer Python 3.1 and 3.2. One of the reasons is that we consolidated the code bases for both major Python versions, and the unicode syntax brought back in 3.3 was a blessing (for example, u'パイソン').

Raw Cursors Return bytearray Objects

Since we consolidated the code bases for Python 2 and 3, we needed make the behaviour as much as possible the same between the two. It's not easy with Unicode strings, but with bytes we have the bytearray type. Raw cursors will return them instead of strings in Python 2 and bytes in Python 3.

If you want to have previous behaviour back, you can inherit from MySQLCursorRaw and change some methods. Please leave comments if you'd like an example for this.

LOAD LOCAL DATA INFILE On by Default

In Connector/Python v1.x you needed to set the client flags to enable the LOAD LOCAL DATA INFILE on the client side. Here an example:

# Connector/Python v1.2
import mysql.connector
from mysql.connector import ClientFlag
cnx = mysql.connector.connect(.. , client_flags=[-ClientFlag.[LOCAL_FILES])

Now in Connector/Python v2.0 it is on. However, some people might not like it so there is a switch to disable it:

# Connector/Python v2.0
import mysql.connector
cnx = mysql.connector.connect(.. , allow_local_infile=False)

Note that you still need to make sure that the MySQL Server is configured to allow this statement.

New Cursors: dict and namedtuple

At last, we have cursors which return rows as dictionaries or named tuples. PEP-249 does not define these since not all database systems might return the columns in a case insensitive or sensitive way.

But this is MySQL.

Here is an example how to use cursor returning dictionaries:

query = (
    "SELECT TABLE_NAME, TABLE_ROWS "
    "FROM INFORMATION_SCHEMA.TABLES "
    " WHERE TABLE_SCHEMA='mysql' ORDER BY TABLE_NAME"
)
cur = cnx.cursor(dictionary=True)
cur.execute(query)
for row in cur:
    print("{TABLE_NAME:>30s} {TABLE_ROWS}".format(**row))

That's far less code for something simple. Each row would look this:

    {u'TABLE_NAME': u'user', u'TABLE_ROWS': 11}

If you like named tuples better, you can do the same, simply giving the named_tuple argument.

cur = cnx.cursor(named_tuple=True)
cur.execute(query)
for row in cur:
    if row.TABLE_ROWS > 0:
        print("{name:>30s} {rows}".format(
            name=row.TABLE_NAME,
            rows=row.TABLE_ROWS))

You can also combine it with the raw=True argument to have raw cursors.

Options Files Support Added

Option files can now be read so you don't have to have all these connection arguments repeated everywhere in your source code. There are lots of ways to do it this, but we needed to be able to read and support the MySQL options files read by client tools and server.

import mysql.connector
cnx = mysql.connector.connect(options_files='/etc/mysql/connectors.cnf')

By default we do not read any file. You have to explicitly specify which files and in which order have to be read. The options groups that are read are client and connector_python. You can also override this and specify which particular group(s) using the argument option_groups.

And more..

Useful links

25 Jul 2014 6:56pm GMT

Geert Vanderkelen: MySQL Connector/Python v2.0.0 alpha

A new major version of Connector/Python is available: v2.0.0 alpha has been been released and is available for download! As with any alpha-software, it's probably not good to throw it in production just yet.

Our manual has the full change log but here's an overview of most important changes for this relase.

Some incompatibilities

The world evolves, at least the software does, and Python is not different. I'm not as bold as the guys at Django who dropped support of Python v2.6 with the Django v1.7 release. I'm leaving it in because I'm nice.

Supported Python: 2.6 and 2.7 and 3.3 and 3.4

We do not support any longer Python 3.1 and 3.2. One of the reasons is that we consolidated the code bases for both major Python versions, and the unicode syntax brought back in 3.3 was a blessing (for example, u'パイソン').

Raw Cursors Return bytearray Objects

Since we consolidated the code bases for Python 2 and 3, we needed make the behaviour as much as possible the same between the two. It's not easy with Unicode strings, but with bytes we have the bytearray type. Raw cursors will return them instead of strings in Python 2 and bytes in Python 3.

If you want to have previous behaviour back, you can inherit from MySQLCursorRaw and change some methods. Please leave comments if you'd like an example for this.

LOAD LOCAL DATA INFILE On by Default

In Connector/Python v1.x you needed to set the client flags to enable the LOAD LOCAL DATA INFILE on the client side. Here an example:

# Connector/Python v1.2
import mysql.connector
from mysql.connector import ClientFlag
cnx = mysql.connector.connect(.. , client_flags=[-ClientFlag.[LOCAL_FILES])

Now in Connector/Python v2.0 it is on. However, some people might not like it so there is a switch to disable it:

# Connector/Python v2.0
import mysql.connector
cnx = mysql.connector.connect(.. , allow_local_infile=False)

Note that you still need to make sure that the MySQL Server is configured to allow this statement.

New Cursors: dict and namedtuple

At last, we have cursors which return rows as dictionaries or named tuples. PEP-249 does not define these since not all database systems might return the columns in a case insensitive or sensitive way.

But this is MySQL.

Here is an example how to use cursor returning dictionaries:

query = (
    "SELECT TABLE_NAME, TABLE_ROWS "
    "FROM INFORMATION_SCHEMA.TABLES "
    " WHERE TABLE_SCHEMA='mysql' ORDER BY TABLE_NAME"
)
cur = cnx.cursor(dictionary=True)
cur.execute(query)
for row in cur:
    print("{TABLE_NAME:>30s} {TABLE_ROWS}".format(**row))

That's far less code for something simple. Each row would look this:

    {u'TABLE_NAME': u'user', u'TABLE_ROWS': 11}

If you like named tuples better, you can do the same, simply giving the named_tuple argument.

cur = cnx.cursor(named_tuple=True)
cur.execute(query)
for row in cur:
    if row.TABLE_ROWS > 0:
        print("{name:>30s} {rows}".format(
            name=row.TABLE_NAME,
            rows=row.TABLE_ROWS))

You can also combine it with the raw=True argument to have raw cursors.

Options Files Support Added

Option files can now be read so you don't have to have all these connection arguments repeated everywhere in your source code. There are lots of ways to do it this, but we needed to be able to read and support the MySQL options files read by client tools and server.

import mysql.connector
cnx = mysql.connector.connect(options_files='/etc/mysql/connectors.cnf')

By default we do not read any file. You have to explicitly specify which files and in which order have to be read. The options groups that are read are client and connector_python. You can also override this and specify which particular group(s) using the argument option_groups.

And more..

Useful links

25 Jul 2014 6:56pm GMT

Europython: EuroPython 2014 Feedback

Now that EuroPython 2014 is almost over, we would like to say a

BIG THANK YOU

to the local organizers in Berlin! You did a wonderful job with the conference organization.

Please provide feedback

Going forward, we would like to ask all EuroPython attendees to send us your feedback for EuroPython 2014, so we can use this information to plan for EuroPython 2015.

Please use our

EuroPython 2014 Feedback Form

for sending us your feedback.

Helping with EuroPython 2015

If you would like to help with EuroPython 2015, we invite you to join the EuroPython Society. Membership is free. Just go to our application page and enter your details.

In the coming months, we will start the discussions about the new work group model we've announced here at the conference.

Thanks to all EuroPython attendees

Thank you very much for attending and have a safe trip home.

We're all looking forward to seeing you again for EuroPython 2015.

Enjoy,
-
EuroPython Society

25 Jul 2014 3:50pm GMT

Europython: EuroPython 2014 Feedback

Now that EuroPython 2014 is almost over, we would like to say a

BIG THANK YOU

to the local organizers in Berlin! You did a wonderful job with the conference organization.

Please provide feedback

Going forward, we would like to ask all EuroPython attendees to send us your feedback for EuroPython 2014, so we can use this information to plan for EuroPython 2015.

Please use our

EuroPython 2014 Feedback Form

for sending us your feedback.

Helping with EuroPython 2015

If you would like to help with EuroPython 2015, we invite you to join the EuroPython Society. Membership is free. Just go to our application page and enter your details.

In the coming months, we will start the discussions about the new work group model we've announced here at the conference.

Thanks to all EuroPython attendees

Thank you very much for attending and have a safe trip home.

We're all looking forward to seeing you again for EuroPython 2015.

Enjoy,
-
EuroPython Society

25 Jul 2014 3:50pm GMT

EuroPython Society: EuroPython 2014 Feedback

Now that EuroPython 2014 is almost over, we would like to say a

BIG THANK YOU

to the local organizers in Berlin! You did a wonderful job with the conference organization.

Please provide feedback

Going forward, we would like to ask all EuroPython attendees to send us your feedback for EuroPython 2014, so we can use this information to plan for EuroPython 2015.

Please use our

EuroPython 2014 Feedback Form

for sending us your feedback.

Helping with EuroPython 2015

If you would like to help with EuroPython 2015, we invite you to join the EuroPython Society. Membership is free. Just go to our application page and enter your details.

In the coming months, we will start the discussions about the new work group model we've announced here at the conference.

Thanks to all EuroPython attendees

Thank you very much for attending and have a safe trip home.

We're all looking forward to seeing you again for EuroPython 2015.

Enjoy,
-
EuroPython Society

25 Jul 2014 3:46pm GMT

EuroPython Society: EuroPython 2014 Feedback

Now that EuroPython 2014 is almost over, we would like to say a

BIG THANK YOU

to the local organizers in Berlin! You did a wonderful job with the conference organization.

Please provide feedback

Going forward, we would like to ask all EuroPython attendees to send us your feedback for EuroPython 2014, so we can use this information to plan for EuroPython 2015.

Please use our

EuroPython 2014 Feedback Form

for sending us your feedback.

Helping with EuroPython 2015

If you would like to help with EuroPython 2015, we invite you to join the EuroPython Society. Membership is free. Just go to our application page and enter your details.

In the coming months, we will start the discussions about the new work group model we've announced here at the conference.

Thanks to all EuroPython attendees

Thank you very much for attending and have a safe trip home.

We're all looking forward to seeing you again for EuroPython 2015.

Enjoy,
-
EuroPython Society

25 Jul 2014 3:46pm GMT

Rob Galanakis: goless 0.7 released, with Python3 support and bug fixes

goless version 0.7.0 is out on PyPI. goless facilitates writing Go language style concurrent programs in Python, including functionality for channels, select, and goroutines.

I forgot to blog about 0.6 at the start of July, which brought Python 3.3 and 3.4 support to goless (#17). I will support pypy3 as soon as Travis supports it.

Version 0.7 includes:
- A "fix" for a gevent problem on Windows (socket must be imported!). #28
- Errors in the case of a deadlock will be more informative. For example, if the last greenlet/tasklet tries to do a blocking send or recv, a DeadlockError will be raised, instead of the underlying error being raised. #25
- goless now has a small exception hierarchy instead of exposing the underlying errors.
- Better PyPy stackless support. #29
- goless.select can be called with (case1, case2, case3), etc., in addition to a list of cases (ie, ([case1, case2, case3])). #22

Thanks to Michael Az for several contributions to this release.

Happy concurrent programming!

25 Jul 2014 2:10pm GMT

Rob Galanakis: goless 0.7 released, with Python3 support and bug fixes

goless version 0.7.0 is out on PyPI. goless facilitates writing Go language style concurrent programs in Python, including functionality for channels, select, and goroutines.

I forgot to blog about 0.6 at the start of July, which brought Python 3.3 and 3.4 support to goless (#17). I will support pypy3 as soon as Travis supports it.

Version 0.7 includes:
- A "fix" for a gevent problem on Windows (socket must be imported!). #28
- Errors in the case of a deadlock will be more informative. For example, if the last greenlet/tasklet tries to do a blocking send or recv, a DeadlockError will be raised, instead of the underlying error being raised. #25
- goless now has a small exception hierarchy instead of exposing the underlying errors.
- Better PyPy stackless support. #29
- goless.select can be called with (case1, case2, case3), etc., in addition to a list of cases (ie, ([case1, case2, case3])). #22

Thanks to Michael Az for several contributions to this release.

Happy concurrent programming!

25 Jul 2014 2:10pm GMT

PyCon Australia: Final weekend for Dinner tickets

We're passing along numbers to our venue for catering on Monday, so this is your last weekend to grab tickets for our conference dinner! As well as fine food and company, we have an important speaker lined up, Paul Gampe.

Get along to the registration page and be sure to select dinner as one of the options.

We'd also like to know if you're coming along to the sprints so we can cater for them appropriately, please head along to our handy-dandy form and let us know!

25 Jul 2014 12:14pm GMT

PyCon Australia: Final weekend for Dinner tickets

We're passing along numbers to our venue for catering on Monday, so this is your last weekend to grab tickets for our conference dinner! As well as fine food and company, we have an important speaker lined up, Paul Gampe.

Get along to the registration page and be sure to select dinner as one of the options.

We'd also like to know if you're coming along to the sprints so we can cater for them appropriately, please head along to our handy-dandy form and let us know!

25 Jul 2014 12:14pm GMT

AppNeta Blog: Faking the Funk: Mocking External Services in Python Tests

In this day and age, it's difficult to build an application that does not rely on some type of external service. Whether the service is handling user identity, analyzing interesting data, or hurling RESTful insults, you have to accept the fact that you now have a dependency on something you do not control. One place […]

Related Posts:

25 Jul 2014 8:00am GMT

AppNeta Blog: Faking the Funk: Mocking External Services in Python Tests

In this day and age, it's difficult to build an application that does not rely on some type of external service. Whether the service is handling user identity, analyzing interesting data, or hurling RESTful insults, you have to accept the fact that you now have a dependency on something you do not control. One place […]

Related Posts:

25 Jul 2014 8:00am GMT

Europython: Upcoming EuroPython sprints and barcamp

Over the weekend there will be a Python BarCamp and of course sprints.

All EuroPython attendees can attend sprints and BarCamp but please register yourself for logistic reasons (registration links on the pages above).

Both events are also free for non-attendees. So if you have friends, co-workers or other people that are interested in Python, please bring them for with you - for free. We only require that they should register at the EuroPython info desk inside the foyer of the bcc.

The sprint orientation and introduction is today, Friday at 2pm in C01.

Join and get some more detailed information about the different sprints and topics.

25 Jul 2014 6:10am GMT

Europython: Upcoming EuroPython sprints and barcamp

Over the weekend there will be a Python BarCamp and of course sprints.

All EuroPython attendees can attend sprints and BarCamp but please register yourself for logistic reasons (registration links on the pages above).

Both events are also free for non-attendees. So if you have friends, co-workers or other people that are interested in Python, please bring them for with you - for free. We only require that they should register at the EuroPython info desk inside the foyer of the bcc.

The sprint orientation and introduction is today, Friday at 2pm in C01.

Join and get some more detailed information about the different sprints and topics.

25 Jul 2014 6:10am GMT

24 Jul 2014

feedPlanet Python

Gael Varoquaux: The 2014 international scikit-learn sprint

A week ago, the 2014 edition of the scikit-learn sprint was held in Paris. This was the third time that we held an internation sprint and it was hugely productive, and great fun, as always.

Great people and great venues

We had a mix of core contributors and newcomers, which is a great combination, as it enables us to be productive, but also to foster the new generation of core developers. Were present:

As the sprint extended through a French bank holiday and the week end, we were hosted in a variety of venues:

I must say that we were treated like kings during the whole stay; each host welcoming us as well they could. Thank you to all of our hosts!

Sponsored by the Paris-Saclay Center for Data Science

Beyond our hosts, we need to thank the Paris-Saclay Center for Data Science. The CDS gave us funding that covered some of the lunches, acomodations, and travel expenses to bring in our
contributors from abroad.

Achievements during the sprint

The first day of the sprint was dedicated to polishing the 0.15 release, which was finally released on the morning of the second day, after 10 months of development.

A large part of the efforts of the sprint were dedicated to improving the coding base, rather than directly adding new features. Some files were reorganized. The input validation code was cleaned up (opening the way for better support of pandas structures in scikit-learn). We hunted dead code, deprecation warnings, numerical instabilities and tests randomly failing. We made the test suite faster, and refactored our common tests that scan all the model.

Some work of our GSOC student, Manoj Kumar, was merged, making some linear models faster.

Our online documentation was improved with the API documentation pointing to examples and source code.

Still work in progress:

24 Jul 2014 10:59pm GMT

Gael Varoquaux: The 2014 international scikit-learn sprint

A week ago, the 2014 edition of the scikit-learn sprint was held in Paris. This was the third time that we held an internation sprint and it was hugely productive, and great fun, as always.

Great people and great venues

We had a mix of core contributors and newcomers, which is a great combination, as it enables us to be productive, but also to foster the new generation of core developers. Were present:

As the sprint extended through a French bank holiday and the week end, we were hosted in a variety of venues:

I must say that we were treated like kings during the whole stay; each host welcoming us as well they could. Thank you to all of our hosts!

Sponsored by the Paris-Saclay Center for Data Science

Beyond our hosts, we need to thank the Paris-Saclay Center for Data Science. The CDS gave us funding that covered some of the lunches, acomodations, and travel expenses to bring in our
contributors from abroad.

Achievements during the sprint

The first day of the sprint was dedicated to polishing the 0.15 release, which was finally released on the morning of the second day, after 10 months of development.

A large part of the efforts of the sprint were dedicated to improving the coding base, rather than directly adding new features. Some files were reorganized. The input validation code was cleaned up (opening the way for better support of pandas structures in scikit-learn). We hunted dead code, deprecation warnings, numerical instabilities and tests randomly failing. We made the test suite faster, and refactored our common tests that scan all the model.

Some work of our GSOC student, Manoj Kumar, was merged, making some linear models faster.

Our online documentation was improved with the API documentation pointing to examples and source code.

Still work in progress:

24 Jul 2014 10:59pm GMT

Data Community DC: Building Data Apps with Python on August 23rd

Python_Building_Data_Apps

Data Community DC and District Data Labs are excited to be hosting another Building Data Apps with Python workshop on August 23rd. For more info and to sign up, go to http://bit.ly/V4used. There's even an early bird discount if you register before the end of this month!

Overview

Data products are usually software applications that derive their value from data by leveraging the data science pipeline and generate data through their operation. They aren't apps with data, nor are they one time analyses that produce insights - they are operational and interactive. The rise of these types of applications has directly contributed to the rise of the data scientist and the idea that data scientists are professionals "who are better at statistics than any software engineer and better at software engineering than any statistician."

These applications have been largely built with Python. Python is flexible enough to develop extremely quickly on many different types of servers and has a rich tradition in web applications. Python contributes to every stage of the data science pipeline including real time ingestion and the production of APIs, and it is powerful enough to perform machine learning computations. In this class we'll produce a data product with Python, leveraging every stage of the data science pipeline to produce a book recommender.

What You Will Learn

Python is one of the most popular programming languages for data analysis. Therefore, it is important to have a basic working knowledge of the language in order to access more complex topics in data science and natural language processing. The purpose of this one-day course is to introduce the development process in Python using a project-based, hands-on approach. In particular you will learn how to structure a data product using every stage of the data science pipeline including ingesting data from the web, wrangling data into a structured database, computing a non-negative matrix factorization with Python, then producing a web based report.

Course Outline

The workshop will cover the following topics:

After this course you should understand how to build a data product using Python and will have built a recommender system that implements the entire data science pipeline.

Instructor: Benjamin Bengfort

Benjamin is an experienced Data Scientist and Python developer who has worked in military, industry, and academia for the past eight years. He is currently pursuing his PhD in Computer Science at The University of Maryland, College Park, doing research in Metacognition and Active Logic. He is also a Data Scientist at Cobrain Company in Bethesda, MD where he builds data products including recommender systems and classifier models. He holds a Masters degree from North Dakota State University where he taught undergraduate Computer Science courses. He is also adjunct faculty at Georgetown University where he teaches Data Science and Analytics.

The post Building Data Apps with Python on August 23rd appeared first on Data Community DC.

24 Jul 2014 6:16pm GMT

Data Community DC: Building Data Apps with Python on August 23rd

Python_Building_Data_Apps

Data Community DC and District Data Labs are excited to be hosting another Building Data Apps with Python workshop on August 23rd. For more info and to sign up, go to http://bit.ly/V4used. There's even an early bird discount if you register before the end of this month!

Overview

Data products are usually software applications that derive their value from data by leveraging the data science pipeline and generate data through their operation. They aren't apps with data, nor are they one time analyses that produce insights - they are operational and interactive. The rise of these types of applications has directly contributed to the rise of the data scientist and the idea that data scientists are professionals "who are better at statistics than any software engineer and better at software engineering than any statistician."

These applications have been largely built with Python. Python is flexible enough to develop extremely quickly on many different types of servers and has a rich tradition in web applications. Python contributes to every stage of the data science pipeline including real time ingestion and the production of APIs, and it is powerful enough to perform machine learning computations. In this class we'll produce a data product with Python, leveraging every stage of the data science pipeline to produce a book recommender.

What You Will Learn

Python is one of the most popular programming languages for data analysis. Therefore, it is important to have a basic working knowledge of the language in order to access more complex topics in data science and natural language processing. The purpose of this one-day course is to introduce the development process in Python using a project-based, hands-on approach. In particular you will learn how to structure a data product using every stage of the data science pipeline including ingesting data from the web, wrangling data into a structured database, computing a non-negative matrix factorization with Python, then producing a web based report.

Course Outline

The workshop will cover the following topics:

After this course you should understand how to build a data product using Python and will have built a recommender system that implements the entire data science pipeline.

Instructor: Benjamin Bengfort

Benjamin is an experienced Data Scientist and Python developer who has worked in military, industry, and academia for the past eight years. He is currently pursuing his PhD in Computer Science at The University of Maryland, College Park, doing research in Metacognition and Active Logic. He is also a Data Scientist at Cobrain Company in Bethesda, MD where he builds data products including recommender systems and classifier models. He holds a Masters degree from North Dakota State University where he taught undergraduate Computer Science courses. He is also adjunct faculty at Georgetown University where he teaches Data Science and Analytics.

The post Building Data Apps with Python on August 23rd appeared first on Data Community DC.

24 Jul 2014 6:16pm GMT

Logilab: EP14 Pylint sprint Day 1 report

https://ep2014.europython.eu/static_media/assets/images/logo.png

We've had a fairly enjoyable and productive first day in our little hidden room at EuroPython in Berlin ! Below are some noticeable things we've worked on and discussed about.

First, we discussed and agreed that while we should at some point cut the cord to the logilab.common package, it will take some time notably because of the usage logilab.common.configuration which would be somewhat costly to replace (and is working pretty well). There are some small steps we should do but basically we should mostly get back some pylint/astroid specific things from logilab.common to astroid or pylint. This should be partly done during the sprint, and remaining work will go to tickets in the tracker.

We also discussed about release management. The point is that we should release more often, so every pylint maintainers should be able to do that easily. Sylvain will write some document about the release procedure and ensure access are granted to the pylint and astroid projects on pypi. We shall release pylint 1.3 / astroid 1.2 soon, and those releases branches will be the last one supporting python < 2.7.

During this first day, we also had the opportunity to meet Carl Crowder, the guy behind http://landscape.io, as well as David Halter which is building the Jedi completion library (https://github.com/davidhalter/jedi). Landscape.io runs pylint on thousands of projects, and it would be nice if we could test beta release on some part of this panel. On the other hand, there are probably many code to share with the Jedi library like the parser and ast generation, as well as a static inference engine. That deserves a sprint on his own though, so we agreed that a nice first step would be to build a common library for import resolution without relying on the python interpreter for that, while handling most of the python dark import features like zip/egg import, .pth files and so one. Indeed that may be two nice future collaborations!

Last but not least, we got some actual work done:

24 Jul 2014 2:39pm GMT

Logilab: EP14 Pylint sprint Day 1 report

https://ep2014.europython.eu/static_media/assets/images/logo.png

We've had a fairly enjoyable and productive first day in our little hidden room at EuroPython in Berlin ! Below are some noticeable things we've worked on and discussed about.

First, we discussed and agreed that while we should at some point cut the cord to the logilab.common package, it will take some time notably because of the usage logilab.common.configuration which would be somewhat costly to replace (and is working pretty well). There are some small steps we should do but basically we should mostly get back some pylint/astroid specific things from logilab.common to astroid or pylint. This should be partly done during the sprint, and remaining work will go to tickets in the tracker.

We also discussed about release management. The point is that we should release more often, so every pylint maintainers should be able to do that easily. Sylvain will write some document about the release procedure and ensure access are granted to the pylint and astroid projects on pypi. We shall release pylint 1.3 / astroid 1.2 soon, and those releases branches will be the last one supporting python < 2.7.

During this first day, we also had the opportunity to meet Carl Crowder, the guy behind http://landscape.io, as well as David Halter which is building the Jedi completion library (https://github.com/davidhalter/jedi). Landscape.io runs pylint on thousands of projects, and it would be nice if we could test beta release on some part of this panel. On the other hand, there are probably many code to share with the Jedi library like the parser and ast generation, as well as a static inference engine. That deserves a sprint on his own though, so we agreed that a nice first step would be to build a common library for import resolution without relying on the python interpreter for that, while handling most of the python dark import features like zip/egg import, .pth files and so one. Indeed that may be two nice future collaborations!

Last but not least, we got some actual work done:

24 Jul 2014 2:39pm GMT

Andrew Dalke: Lausanne Cheminformatics workshop and contest

Dietrich Rordorf from MDPI sent an announcement to the CHMINF mailing list about the upcoming 9th Workshop in Chemical Information. It will be on 12 September 2014 in Lausanne, Switzerland. It seems like it will be a nice meeting, so I thought to forward information about it here. They also have a software contest, with a 2,000 CHF prize, which I think will interest some of my readers.

The workshop has been around for 10 years, so I was a bit suprised that I hadn't heard of it before. Typically between 30 and 50 people attend, which I think is a nice size. The preliminary program is structured around 20 minute presentations, including:

If you know the authors, you might recognize that one is from Strasbourg and another London, and the rest from Switzerland. I can understand. From where I live in Sweden it will cost over US $300 in order to get there, and Lausanne doesn't have its own commercial airport so I would need to fly into Geneva or Bern, while my local air hub doesn't fly there directly.

But I live in a corner of Europe, and my constraints aren't yours.

Source code contest

I had an email conversation with Luc Patiny about an interesting aspect of the workshop. They are running a contest to identify the best open source cheminformatics tool of the year, with a prize of 2000 CHF. That's 1650 EUR, 2200 USD, 1300 GBP, or 15000 SEK, which is plenty enough for someone in Europe or even the US to be able to travel there! They have a time slot set aside for the winner of the contest to present the work. The main requirement is that contest participants are selected from submissions (1-2 pages long) to the open access journal Challenges. (And no, there are no journal page fees for this contest, so it doesn't seem like a sneaky revenue generating trick.)

The other requirement is that the submission be "open source". I put that in quotes because much of my conversation with Luc was to understand what they mean. They want people to be able to download the (unobsfucated) software source code for no cost and be able to read and review it to gain insight.

I think this is very much in line with classical peer review thought, even though it can include software which are neither open source nor free software. For example, software submissions for this contest could be "for research purposes only" or "not for use in nuclear reactors", or "redistributions of modified versions are not permitted."

Instead, I think their definition is more in line with Microsoft terms shared source.

In my case, my latest version of chemfp is 'commercial open source', meaning that those who pay me money for it get a copy of it under the MIT open source license. It's both "free software" and "open source", but it's not eligible for this workshop because it costs money to download it.

But I live in a corner of open source, and my constraints aren't yours. ;) If you have a free software project, open source software project, or shared source software project, then you might be interested in submitting it to this workshop and journal. If you win, think of it as an all-expenses paid trip to Switzerland. If you don't win, think of it as a free publication.

24 Jul 2014 12:00pm GMT

Andrew Dalke: Lausanne Cheminformatics workshop and contest

Dietrich Rordorf from MDPI sent an announcement to the CHMINF mailing list about the upcoming 9th Workshop in Chemical Information. It will be on 12 September 2014 in Lausanne, Switzerland. It seems like it will be a nice meeting, so I thought to forward information about it here. They also have a software contest, with a 2,000 CHF prize, which I think will interest some of my readers.

The workshop has been around for 10 years, so I was a bit suprised that I hadn't heard of it before. Typically between 30 and 50 people attend, which I think is a nice size. The preliminary program is structured around 20 minute presentations, including:

If you know the authors, you might recognize that one is from Strasbourg and another London, and the rest from Switzerland. I can understand. From where I live in Sweden it will cost over US $300 in order to get there, and Lausanne doesn't have its own commercial airport so I would need to fly into Geneva or Bern, while my local air hub doesn't fly there directly.

But I live in a corner of Europe, and my constraints aren't yours.

Source code contest

I had an email conversation with Luc Patiny about an interesting aspect of the workshop. They are running a contest to identify the best open source cheminformatics tool of the year, with a prize of 2000 CHF. That's 1650 EUR, 2200 USD, 1300 GBP, or 15000 SEK, which is plenty enough for someone in Europe or even the US to be able to travel there! They have a time slot set aside for the winner of the contest to present the work. The main requirement is that contest participants are selected from submissions (1-2 pages long) to the open access journal Challenges. (And no, there are no journal page fees for this contest, so it doesn't seem like a sneaky revenue generating trick.)

The other requirement is that the submission be "open source". I put that in quotes because much of my conversation with Luc was to understand what they mean. They want people to be able to download the (unobsfucated) software source code for no cost and be able to read and review it to gain insight.

I think this is very much in line with classical peer review thought, even though it can include software which are neither open source nor free software. For example, software submissions for this contest could be "for research purposes only" or "not for use in nuclear reactors", or "redistributions of modified versions are not permitted."

Instead, I think their definition is more in line with Microsoft terms shared source.

In my case, my latest version of chemfp is 'commercial open source', meaning that those who pay me money for it get a copy of it under the MIT open source license. It's both "free software" and "open source", but it's not eligible for this workshop because it costs money to download it.

But I live in a corner of open source, and my constraints aren't yours. ;) If you have a free software project, open source software project, or shared source software project, then you might be interested in submitting it to this workshop and journal. If you win, think of it as an all-expenses paid trip to Switzerland. If you don't win, think of it as a free publication.

24 Jul 2014 12:00pm GMT

Europython: EuroPython Society Sessions at EuroPython 2014

We are having three EuroPython Society (EPS) sessions today at EuroPython 2014. They are all held in room B09.

All EuroPython attendees are invited to join in to these sessions and to become EuroPython Society members.

If you would like to support the EuroPython Society and want to sign up for membership, please visit our membership application form.

Membership is free and we'd like to get as many EuroPython attendees signed up as members as possible, because the EuroPython conference series is all about its attendees.

Enjoy,
-
EuroPython Society

24 Jul 2014 9:02am GMT

Europython: EuroPython Society Sessions at EuroPython 2014

We are having three EuroPython Society (EPS) sessions today at EuroPython 2014. They are all held in room B09.

All EuroPython attendees are invited to join in to these sessions and to become EuroPython Society members.

If you would like to support the EuroPython Society and want to sign up for membership, please visit our membership application form.

Membership is free and we'd like to get as many EuroPython attendees signed up as members as possible, because the EuroPython conference series is all about its attendees.

Enjoy,
-
EuroPython Society

24 Jul 2014 9:02am GMT

EuroPython Society: EuroPython Society Sessions at EuroPython 2014

We are having three EuroPython Society (EPS) sessions today at EuroPython 2014. They are all held in room B09.

All EuroPython attendees are invited to join in to these sessions and to become EuroPython Society members.

If you would like to support the EuroPython Society and want to sign up for membership, please visit our membership application form.

Membership is free and we'd like to get as many EuroPython attendees signed up as members as possible, because the EuroPython conference series is all about its attendees.

Enjoy,
-
EuroPython Society

24 Jul 2014 9:01am GMT

EuroPython Society: EuroPython Society Sessions at EuroPython 2014

We are having three EuroPython Society (EPS) sessions today at EuroPython 2014. They are all held in room B09.

All EuroPython attendees are invited to join in to these sessions and to become EuroPython Society members.

If you would like to support the EuroPython Society and want to sign up for membership, please visit our membership application form.

Membership is free and we'd like to get as many EuroPython attendees signed up as members as possible, because the EuroPython conference series is all about its attendees.

Enjoy,
-
EuroPython Society

24 Jul 2014 9:01am GMT

S. Lott: Building Probabilistic Graphical Models with Python

A deep dive into probability and scipy: https://www.packtpub.com/building-probabilistic-graphical-models-with-python/book

I have to admit up front that this book is out of my league.

The Python is sensible to me. The subject matter -- graph models, learning and inference -- is above my pay grade.

Asking About a Book

Let me summarize before diving into details.

Asking someone else if a book is useful is really not going to reveal much. Their background is not my background. They found it helpful/confusing/incomplete/boring isn't really going to indicate anything about how I'll find it.

Asking someone else for a vague, unmeasurable judgement like "useful" or "appropriate" or "helpful" is silly. Someone else's opinions won't apply to you.

Asking if a book is technically correct is more measurable. However. Any competent publisher has a thorough pipeline of editing. It involves at least three steps: Acceptance, Technical Review, and a Final Review. At least three. A good publisher will have multiple technical reviewers. All of this is detailed in the front matter of the book.

Asking someone else if the book was technically correct is like asking if it was reviewed: a silly question. The details of the review process are part of the book. Just check the front matter online before you buy.

It doesn't make sense to ask judgement questions. It doesn't make sense to ask questions answered in the front matter. What can you ask that might be helpful?

I think you might be able to ask completeness questions. "What's omitted from the tutorial?" "What advanced math is assumed?" These are things that can be featured in online reviews.

Sadly, these are not questions I get asked.

Irrational Questions

A colleague had some questions about the book named above. Some of which were irrational. I'll try to tackle the rational questions since emphasis my point on ways not to ask questions about books.


2. Is the Python code good at solidifying the mathematical concepts?

This is a definite maybe situation. The concept of "solidifying" as expressed here bothers me a lot.

Solid mathematics -- to me -- means solid mathematics. Outside any code considerations. I failed a math course in college because I tried to convert everything to algorithms and did not get the math part. A kindly professor explained that "F" very, very clearly. A life lesson. The math exists outside any implementation.

I don't think code can ever "solidify" the mathematics. It goes the other way: the code must properly implement the mathematical concepts. The book depends on scipy, and scipy is a really good implementation of a great deal of advanced math. The implementation of the math sits squarely on the rock-solid foundation of scipy. For me, that's a ringing endorsement of the approach.

If the book reinvented the algorithms available in scipy, that would be reason for concern. The book doesn't reinvent that wheel: it uses scipy to solve problems.

4. Can the code be used to build prototypes?

Um. What? What does the word prototype mean in that question? If we use the usual sense of software prototype, the answer is a trivial "Yes." The examples are prototypes in that sense. That can't be what the question means.

In this context the word might mean "model". Or it might mean "prototype of a model". If we reexamine the question with those other senses of prototype, we might have an answer that's not trivially "yes." Might.

When they ask about prototype, could they mean "model?" The code in the book is a series of models of different kinds of learning. The models are complete, consistent, and work. That can't be what they're asking.

Could they mean "prototype of a model?" It's possible that we're talking about using the book to build a prototype of a model. For example, we might have a large and complex problem with several more degrees of freedom than the text book examples. In this case, perhaps we might want to simplify the complex problem to make it more like one of the text book problems. Then we could use Python to solve that simplified problem as a prototype for building a final model which is appropriate for the larger problem.

In this sense of prototype, the answer remains "What?" Clearly, the book solves a number of simplified problems and provides code samples that can be expanded and modified to solve larger and more complex problems.

To get past the trivial "yes" for this question, we can try to examine this in a negative sense. What kind of thing is the book unsuitable for? It's unsuitable as a final implementation of anything but the six problems it tackles. It can't be that "prototype" means "final implementation." The book is unsuitable as a tutorial on Python. It's not possible this is what "prototype" means.

Almost any semantics we assign to "prototype" lead to an answer of "yes". The book is suitable for helping someone build a lot of things.

Summary

Those two were the rational questions. The irrational questions made even less sense.

Including the other irrational questions, it appears that the real question might have been this.

Q: "Can I learn Python from this book?"

A: No.

It's possible that the real question was this:

Q: "Can I learn advanced probabilistic modeling with this book?"

A: Above my pay grade. I'm not sure I could learn probabilistic modeling from this book. Maybe I could. But I don't think that I have the depth required.

It's possible that the real questions was this:

Q: Can I learn both Python and advanced probabilistic modeling with this book?"

A: Still No.

Gaps In The Book

Here's what I could say about the book.

You won't learn much Python from this book. It assumes Python; it doesn't tutor Python. Indeed, it assumes some working scipy knowledge and a scipy installation. It doesn't include a quick-start tutorial on scipy or any of that other hand-holding.

This is not even a quibble with the presentation. It's just an observation: the examples are all written in Python 2. Small changes are required for Python 3. Scipy will work with Python 3. http://www.scipy.org/scipylib/faq.html#do-numpy-and-scipy-support-python-3-x. Reworking the examples seems to involve only small changes to replace print statements. In that respect, the presentation is excellent.




24 Jul 2014 9:00am GMT

S. Lott: Building Probabilistic Graphical Models with Python

A deep dive into probability and scipy: https://www.packtpub.com/building-probabilistic-graphical-models-with-python/book

I have to admit up front that this book is out of my league.

The Python is sensible to me. The subject matter -- graph models, learning and inference -- is above my pay grade.

Asking About a Book

Let me summarize before diving into details.

Asking someone else if a book is useful is really not going to reveal much. Their background is not my background. They found it helpful/confusing/incomplete/boring isn't really going to indicate anything about how I'll find it.

Asking someone else for a vague, unmeasurable judgement like "useful" or "appropriate" or "helpful" is silly. Someone else's opinions won't apply to you.

Asking if a book is technically correct is more measurable. However. Any competent publisher has a thorough pipeline of editing. It involves at least three steps: Acceptance, Technical Review, and a Final Review. At least three. A good publisher will have multiple technical reviewers. All of this is detailed in the front matter of the book.

Asking someone else if the book was technically correct is like asking if it was reviewed: a silly question. The details of the review process are part of the book. Just check the front matter online before you buy.

It doesn't make sense to ask judgement questions. It doesn't make sense to ask questions answered in the front matter. What can you ask that might be helpful?

I think you might be able to ask completeness questions. "What's omitted from the tutorial?" "What advanced math is assumed?" These are things that can be featured in online reviews.

Sadly, these are not questions I get asked.

Irrational Questions

A colleague had some questions about the book named above. Some of which were irrational. I'll try to tackle the rational questions since emphasis my point on ways not to ask questions about books.


2. Is the Python code good at solidifying the mathematical concepts?

This is a definite maybe situation. The concept of "solidifying" as expressed here bothers me a lot.

Solid mathematics -- to me -- means solid mathematics. Outside any code considerations. I failed a math course in college because I tried to convert everything to algorithms and did not get the math part. A kindly professor explained that "F" very, very clearly. A life lesson. The math exists outside any implementation.

I don't think code can ever "solidify" the mathematics. It goes the other way: the code must properly implement the mathematical concepts. The book depends on scipy, and scipy is a really good implementation of a great deal of advanced math. The implementation of the math sits squarely on the rock-solid foundation of scipy. For me, that's a ringing endorsement of the approach.

If the book reinvented the algorithms available in scipy, that would be reason for concern. The book doesn't reinvent that wheel: it uses scipy to solve problems.

4. Can the code be used to build prototypes?

Um. What? What does the word prototype mean in that question? If we use the usual sense of software prototype, the answer is a trivial "Yes." The examples are prototypes in that sense. That can't be what the question means.

In this context the word might mean "model". Or it might mean "prototype of a model". If we reexamine the question with those other senses of prototype, we might have an answer that's not trivially "yes." Might.

When they ask about prototype, could they mean "model?" The code in the book is a series of models of different kinds of learning. The models are complete, consistent, and work. That can't be what they're asking.

Could they mean "prototype of a model?" It's possible that we're talking about using the book to build a prototype of a model. For example, we might have a large and complex problem with several more degrees of freedom than the text book examples. In this case, perhaps we might want to simplify the complex problem to make it more like one of the text book problems. Then we could use Python to solve that simplified problem as a prototype for building a final model which is appropriate for the larger problem.

In this sense of prototype, the answer remains "What?" Clearly, the book solves a number of simplified problems and provides code samples that can be expanded and modified to solve larger and more complex problems.

To get past the trivial "yes" for this question, we can try to examine this in a negative sense. What kind of thing is the book unsuitable for? It's unsuitable as a final implementation of anything but the six problems it tackles. It can't be that "prototype" means "final implementation." The book is unsuitable as a tutorial on Python. It's not possible this is what "prototype" means.

Almost any semantics we assign to "prototype" lead to an answer of "yes". The book is suitable for helping someone build a lot of things.

Summary

Those two were the rational questions. The irrational questions made even less sense.

Including the other irrational questions, it appears that the real question might have been this.

Q: "Can I learn Python from this book?"

A: No.

It's possible that the real question was this:

Q: "Can I learn advanced probabilistic modeling with this book?"

A: Above my pay grade. I'm not sure I could learn probabilistic modeling from this book. Maybe I could. But I don't think that I have the depth required.

It's possible that the real questions was this:

Q: Can I learn both Python and advanced probabilistic modeling with this book?"

A: Still No.

Gaps In The Book

Here's what I could say about the book.

You won't learn much Python from this book. It assumes Python; it doesn't tutor Python. Indeed, it assumes some working scipy knowledge and a scipy installation. It doesn't include a quick-start tutorial on scipy or any of that other hand-holding.

This is not even a quibble with the presentation. It's just an observation: the examples are all written in Python 2. Small changes are required for Python 3. Scipy will work with Python 3. http://www.scipy.org/scipylib/faq.html#do-numpy-and-scipy-support-python-3-x. Reworking the examples seems to involve only small changes to replace print statements. In that respect, the presentation is excellent.




24 Jul 2014 9:00am GMT

11 Oct 2013

feedPython Software Foundation | GSoC'11 Students

Yeswanth Swami: How I kicked off GSoC

Zero to hero

What Prompted me??

I started my third year thinking I should do something that would put me different from the rest and one of my professors suggested me as to why don't I apply for GSoC. I don't know why but I took the suggestion rather seriously, thanks to the bet I had with one of my friend(who is about to complete his MBBS) that whoever earns first will buy the other a "RayBan shades". Well, that's it. I was determined. I started my research early, probably during the start of February(I knew I want to buy my friend, his shades and also buy mine too, in the process).

What experiences I had before??

I started looking at previous years' GSoC projects(having had little experience with Open Source) and started learning how to contribute. I was also very fascinated to the amount of knowledge one could gain just by googling and browsing web pages . I discovered very soon, as to what an immensely great tool , email, through which I could chat with anyone in the open source world and ask seemingly stupid questions and always expect to get a gentle reply back with an answer. Well, that held me spell bound and I knew I want to contribute to Open Source.

How did I begin??

About the middle of March, I discovered that my passion for Python as a programming language increased , after understanding how easy it is as a language. Added to that, my popularity among my fellow classmates increased when I started evangelizing Python(thanks to my seniors for introducing it, I guess I did a decent job popularizing the language). And I started contributing to PSF(Python Software Foundation) , started with a simple bug to fix documentation and slowly my interactivity in IRC increased and I started liking one of the project one of the community member proposed.

A twist in the story??

There I was, still a noob and not knowing how to convince my probable mentor that I could complete the project, given direction. About this juncture, a fellow student(from some university in France) mailed this particular mentor that he was interested in the project . Do, remember, I was part of the mailing list and follow the happenings of it. So, I was furious knowing that I had a competition(having put so much effort) and I was not willing to compromise my project (knowing that this is the one project I actually understood and started researching a little bit too). The other projects require me to have some domain knowledge. I went back to my teachers, seniors, friends and Google and started asking the question , "how would i solve the problem the mentor posted?" . I framed a couple of answers, though very noobish , but at least I could reply the email thread posting my understanding of the problem and how I would solve it and also ask various questions I had in my mind. Well, the mentor replied, immediately to my surprise, and responded back with comments as well as answers to the questions I posed. Again, my nemesis/competitor replied back(he having good knowledge about the problem domain). I knew it was not going to be easy. Hence, I went back again, through all my sources, made further understanding of the problem and posted back again. I guess, about 20 mails in the thread , till we(all three of us) decided we should catch up in IRC and discuss more.

The conclusion:

Well, at IRC , most of senior members from the community were present, and they suggested that they should probably extend the scope of the project(since two students were interested in one project and showed immense passion). Unsurprisingly, over multiple meetings, the project scope was expanded, both the students were given equal important but independent tasks and both the students got opportunity to say they are Google Summer of Code students. Thank goodness, we decided to built the project from scratch giving us more than enough work on our plate.

Movie titles:

1) In the open source world, there is no competition , it is only "COLLABORATION".

2) Why give up, when you can win??

3) From Zero to Hero!!

4) A prodigy in making

p.s. I still owe my friend his shades . *sshole, I am still waiting for him to visit me so that I can buy him his shades and buy mine too. Also, I know its been two years since the story happened, but it is never too late to share, don't you agree??


11 Oct 2013 5:39am GMT