Hello everyone!

It's time to kick off the lead up to a great PyTN 2017! I'm very excited about this year event, and I hope you are as well. I'm pleased to welcome a few new organizers to this year's event: Bill Israel (co-chair) and Jessica Wynn (Volunteer Chair). I know you'll find their extra efforts will help the conference feel more coordinated and organized, so please thank them if you see them. Let's start with the keynotes, which we'll be posting full profiles of in the coming weeks. However, here are the reasons they are special to me:

- Sophie Rapoport: A software engineer from Nashville who has made a distinct impression on me with her sharp intellect, dedication to learning, and teaching nature. Simply an amazing person.

- Courey Elliott: A software developer who I first meet about 2 years ago with an amazing capacity for reasoning, a wide array of interests, a constant desire to learn, and a wonderful soul.

- Jurnell Cockhren: An amazingly wide and deeply learned individual inside and outside of technology. Working right at the line between software engineering, operations, and teaching, Jurnell's sharp mind, excellent sense of humor, and beautiful humanity endeared him to me from the beginning.

- Sarah Guido: One of the people I greatly admire in the data science world, Sarah can take complex concepts break them down into understandable pieces with ease. Sarah's impressive engineering skills lead her to solve complex problems in data science and development alike. With a great compassion to empower others, powerful logic, and writing, Sarah is truly an asset to our community.

CFP Opens Saturday Oct 1st!

PyTennessee 2017 is accepting all types of proposals starting this Saturday Oct 1st through Nov 15th, 2017. Due to the competitive nature of the selection process, we encourage prospective speakers to submit their proposals as early as possible. We're looking forward to receiving your best proposals for tutorials and talks. Lightning talk sign-ups will be available at the registration desk the day of the event.

Note: To submit a proposal, sign up or log in to your account and proceed to your account dashboard!

Special Sponsorship Profile

Intellovations has been the first sponsor of PyTN for all 4 years we have had the conference, and in addition to their financial support, they have been providing words of encouragement behind the scenes for all those years as well. They are truly part of the PyTN family, and for that, we are greatly appreciative and lucky to call them friends.

Young Coders

We will be having young coders again this year. Tickets will be available December 15th at noon and they normally go fast. The program will be lead by Brad Montgomery, a masterful python teacher, again this year. This free program is open to ages 12-17, and all attendees will receive take home books and a Chromebook.

After Party

Last year, we had a games night put on by Eventbrite, and behind the scenes of that party was a group called Meeple Mountain. We heard from many MANY of you how much you enjoyed the game night, and it will be returning this year! The Meeple Mountain crew will be on hand and you can learn more about this and this event at their website.

A Closing Note

Four years ago, a collection of us started PyTN to fill what we felt was a gap in our community. We wanted a place to come together to learn about python, share our experiences, and add a human touch to what is sometimes a tough profession in which to do that. This year as we started planning for PyTN, I realized that we had grown from 110 to over 360 people during these years. On behalf of myself, my wife and all the other organizers, we personally want to thank you for sharing in our vision of this event, you all have made it far more than we every imagined. I'll greatly cherish all the experiences we shared, Katie Cunningham painting my nails, Lars telling us the story of two braids, Eric eating the Publix sushi, and so many other countless fun things. On that note, PyTN 2017 will be my last one as Chair of the conference, and I want to throw a party! So I hope you'll join me and all the other organizers in a celebration of the community you all built.

Thank you,

Jason Myers

PyTN 2017 Chair

29 Sep 2016 1:27pm GMT

Hello everyone!

It's time to kick off the lead up to a great PyTN 2017! I'm very excited about this year event, and I hope you are as well. I'm pleased to welcome a few new organizers to this year's event: Bill Israel (co-chair) and Jessica Wynn (Volunteer Chair). I know you'll find their extra efforts will help the conference feel more coordinated and organized, so please thank them if you see them. Let's start with the keynotes, which we'll be posting full profiles of in the coming weeks. However, here are the reasons they are special to me:

- Sophie Rapoport: A software engineer from Nashville who has made a distinct impression on me with her sharp intellect, dedication to learning, and teaching nature. Simply an amazing person.

- Courey Elliott: A software developer who I first meet about 2 years ago with an amazing capacity for reasoning, a wide array of interests, a constant desire to learn, and a wonderful soul.

- Jurnell Cockhren: An amazingly wide and deeply learned individual inside and outside of technology. Working right at the line between software engineering, operations, and teaching, Jurnell's sharp mind, excellent sense of humor, and beautiful humanity endeared him to me from the beginning.

- Sarah Guido: One of the people I greatly admire in the data science world, Sarah can take complex concepts break them down into understandable pieces with ease. Sarah's impressive engineering skills lead her to solve complex problems in data science and development alike. With a great compassion to empower others, powerful logic, and writing, Sarah is truly an asset to our community.

CFP Opens Saturday Oct 1st!

PyTennessee 2017 is accepting all types of proposals starting this Saturday Oct 1st through Nov 15th, 2017. Due to the competitive nature of the selection process, we encourage prospective speakers to submit their proposals as early as possible. We're looking forward to receiving your best proposals for tutorials and talks. Lightning talk sign-ups will be available at the registration desk the day of the event.

Note: To submit a proposal, sign up or log in to your account and proceed to your account dashboard!

Special Sponsorship Profile

Intellovations has been the first sponsor of PyTN for all 4 years we have had the conference, and in addition to their financial support, they have been providing words of encouragement behind the scenes for all those years as well. They are truly part of the PyTN family, and for that, we are greatly appreciative and lucky to call them friends.

Young Coders

We will be having young coders again this year. Tickets will be available December 15th at noon and they normally go fast. The program will be lead by Brad Montgomery, a masterful python teacher, again this year. This free program is open to ages 12-17, and all attendees will receive take home books and a Chromebook.

After Party

Last year, we had a games night put on by Eventbrite, and behind the scenes of that party was a group called Meeple Mountain. We heard from many MANY of you how much you enjoyed the game night, and it will be returning this year! The Meeple Mountain crew will be on hand and you can learn more about this and this event at their website.

A Closing Note

Four years ago, a collection of us started PyTN to fill what we felt was a gap in our community. We wanted a place to come together to learn about python, share our experiences, and add a human touch to what is sometimes a tough profession in which to do that. This year as we started planning for PyTN, I realized that we had grown from 110 to over 360 people during these years. On behalf of myself, my wife and all the other organizers, we personally want to thank you for sharing in our vision of this event, you all have made it far more than we every imagined. I'll greatly cherish all the experiences we shared, Katie Cunningham painting my nails, Lars telling us the story of two braids, Eric eating the Publix sushi, and so many other countless fun things. On that note, PyTN 2017 will be my last one as Chair of the conference, and I want to throw a party! So I hope you'll join me and all the other organizers in a celebration of the community you all built.

Thank you,

Jason Myers

PyTN 2017 Chair

29 Sep 2016 1:27pm GMT

Worthy Read Python alias commands that play nice with virtualenv Over the years, I've come up with my own Python aliases that play nice with virtual environments. For this post, I tried to stay as generic as possible such that any alias here can be used by every Pythonista. Keep detailed records of the performance of your Django code. django , performance "Keep detailed records of the performance of your Django code.". django-perf-rec is like Django's assertNumQueries on steroids. It lets you track the individual queries and cache operations that occur in your code. This blog post explains the workings of this project https://tech.yplanapp.com/2016/09/26/introducing-django-perf-rec/ . Practical ML for Engineers talk at #pyconuk last weekend machine learning Last weekend I had the pleasure of introducing Machine Learning for Engineers (a practical walk-through, no maths) at PyConUK 2016 ( Video link on page ). My talk covered a practical guide to a 2 class classification challenge (Kaggle's Titanic) with scikit-learn, backed by a longer Jupyter Notebook (github) and further backed by Ezzeri's 2 hour tutorial from PyConUK 2014. Mocks and Monkeypatching in Python testing This tutorial will help you understand why mocking is important, and show you how to mock in Python with Mock and Pytest monkeypatch. Abu Ashraf Masnun: Introduction to Django Channels Yet another introduction to Django Channels. This one is a lot more clear and step by step tutorial. If you still don't know what Django channels is / how to get started, read this. Python has come a long way. So has job hunting. Try Hired and get in front of 4,000+ companies with one application. No more pushy recruiters, no more dead end applications and mismatched companies, Hired puts the power in your hands. Sponsor Python Mocks: a gentle introduction - Part 1 and 2 testing , mock In this series of posts I am going to review the Python mock library and exemplify its use. I will not cover everything you may do with mock, obviously, but hopefully I'll give you the information you need to start using this powerful library. Note it's a two part series as of now, here is the second part's url http://blog.thedigitalcatonline.com/blog/2016/09/27/python-mocks-a-gentle-introduction-part-2/#.V-ysf9HhXQo Decorators: The Function's Function - Weekly Python Chat with Trey Hunner webcast , video Decorators are one of those features in Python that people like to talk about. Why? Because they're different. Because they're a little weird. Because they're a little mind-bending. Let's talk about decorators: how do you make them and when should you use them? Simple REST APIs for charts and datasets charts The Plotly V2 API suite is a simple alternative to the Google Charts API. Make a request to a Plotly URL and get a link to a dataset or D3.js chart. Python code snippet are included on the page. Python Code Review: Unplugged - Episode 2 - By Daniel Bader code review Daniel is doing a series of code review sessions with Python developers. Have a look at the accompanied video where he gives his opinion on a open source project by Milton. Python by the C side c binding CPython, the primary implementation of Python used by millions, is written in C. Python core developers embraced and exposed Python's strong C roots, taking a traditional tack on portability, contrasting with the "write once, debug everywhere" approach popularized elsewhere. The community followed suit with the core developers, developing several methods for linking to C. This has given us a lot of choices for interfacing with c, let us look at them. Django Tips #15 Using Mixins With Class-Based Views django General rules to use mixins to compose your own view classes with code examples. How to set up tab completion for django-admin.py and manage.py ? django In this short article Mike shows us how to set auto complete for django-admin.py / manage.py arguments. Specially helpful if you have tons of management commands. Python Engineering at Microsoft core python That's the opening paragraph from the Python Insider blog post discussing the 2016 Python core sprint that recently took place. In the case of Microsoft's participation in the sprint, both Steve Dower and I (Brett Cannon) were invited to participate (which meant Microsoft had one of the largest company representations at the sprint). Between the two of us we spent the week completing work on four of our own PEPs for Python 3.6: Adding a file system path protocol (PEP 519), Adding a frame evaluation API to CPython (PEP 523), Change Windows console encoding to UTF-8 (PEP 528), Change Windows filesystem encoding to UTF-8 (PEP 529). GitHub - beanbaginc/django: Unofficial security backports for Django: The Web framework for perfectionists with deadlines. security This is an unofficial fork of Django, which focuses entirely on backporting official, publicly-announced security fixes to Django 1.6.11. It does not contain any other bug fixes or features, and any branches other than security-backports/1.6.x are unlikely to be up-to-date. Upcoming Conference / User Group Meet Reunión Python Valencia Reunión Python Valencia Sydney Python User Group PyConES - Almería Edmonton Python User Group IndyPy Monthly Meetup Python Brasil [12] Santa Cruz Python Meetup Projects fmap - 6 Stars, 0 Fork fmap.py - a single dispatch version of fmap for Python3. While there are multiple Haskellesque 'lets put monads in Python!' style libraries out there, most don't seem to focus on taking the nice bits of Haskell's functional approach and giving them a nice Pythonic interface. fmap.py is a very simple take on fmap that lets you remove some unnecesary boiler plate when you are applying a function to each element of a collection. I hope you like it! fbtftp - 5 Stars, 0 Fork fbtftp is Facebook's implementation of a dynamic TFTP server framework. It lets you create custom TFTP servers and wrap your own logic into it in a very simple manner. Facebook currently uses it in production, and it's deployed at global scale across all of our data centers. unfurl - 4 Stars, 0 Fork Python utility to move items in a directory tree to the topmost level possible chalk - 2 Stars, 1 Fork Simple, easy to learn interpreted programming language. human-to-geojson - 2 Stars, 1 Fork Convert raw Human exports to geoJSON

29 Sep 2016 10:40am GMT

Worthy Read Python alias commands that play nice with virtualenv Over the years, I've come up with my own Python aliases that play nice with virtual environments. For this post, I tried to stay as generic as possible such that any alias here can be used by every Pythonista. Keep detailed records of the performance of your Django code. django , performance "Keep detailed records of the performance of your Django code.". django-perf-rec is like Django's assertNumQueries on steroids. It lets you track the individual queries and cache operations that occur in your code. This blog post explains the workings of this project https://tech.yplanapp.com/2016/09/26/introducing-django-perf-rec/ . Practical ML for Engineers talk at #pyconuk last weekend machine learning Last weekend I had the pleasure of introducing Machine Learning for Engineers (a practical walk-through, no maths) at PyConUK 2016 ( Video link on page ). My talk covered a practical guide to a 2 class classification challenge (Kaggle's Titanic) with scikit-learn, backed by a longer Jupyter Notebook (github) and further backed by Ezzeri's 2 hour tutorial from PyConUK 2014. Mocks and Monkeypatching in Python testing This tutorial will help you understand why mocking is important, and show you how to mock in Python with Mock and Pytest monkeypatch. Abu Ashraf Masnun: Introduction to Django Channels Yet another introduction to Django Channels. This one is a lot more clear and step by step tutorial. If you still don't know what Django channels is / how to get started, read this. Python has come a long way. So has job hunting. Try Hired and get in front of 4,000+ companies with one application. No more pushy recruiters, no more dead end applications and mismatched companies, Hired puts the power in your hands. Sponsor Python Mocks: a gentle introduction - Part 1 and 2 testing , mock In this series of posts I am going to review the Python mock library and exemplify its use. I will not cover everything you may do with mock, obviously, but hopefully I'll give you the information you need to start using this powerful library. Note it's a two part series as of now, here is the second part's url http://blog.thedigitalcatonline.com/blog/2016/09/27/python-mocks-a-gentle-introduction-part-2/#.V-ysf9HhXQo Decorators: The Function's Function - Weekly Python Chat with Trey Hunner webcast , video Decorators are one of those features in Python that people like to talk about. Why? Because they're different. Because they're a little weird. Because they're a little mind-bending. Let's talk about decorators: how do you make them and when should you use them? Simple REST APIs for charts and datasets charts The Plotly V2 API suite is a simple alternative to the Google Charts API. Make a request to a Plotly URL and get a link to a dataset or D3.js chart. Python code snippet are included on the page. Python Code Review: Unplugged - Episode 2 - By Daniel Bader code review Daniel is doing a series of code review sessions with Python developers. Have a look at the accompanied video where he gives his opinion on a open source project by Milton. Python by the C side c binding CPython, the primary implementation of Python used by millions, is written in C. Python core developers embraced and exposed Python's strong C roots, taking a traditional tack on portability, contrasting with the "write once, debug everywhere" approach popularized elsewhere. The community followed suit with the core developers, developing several methods for linking to C. This has given us a lot of choices for interfacing with c, let us look at them. Django Tips #15 Using Mixins With Class-Based Views django General rules to use mixins to compose your own view classes with code examples. How to set up tab completion for django-admin.py and manage.py ? django In this short article Mike shows us how to set auto complete for django-admin.py / manage.py arguments. Specially helpful if you have tons of management commands. Python Engineering at Microsoft core python That's the opening paragraph from the Python Insider blog post discussing the 2016 Python core sprint that recently took place. In the case of Microsoft's participation in the sprint, both Steve Dower and I (Brett Cannon) were invited to participate (which meant Microsoft had one of the largest company representations at the sprint). Between the two of us we spent the week completing work on four of our own PEPs for Python 3.6: Adding a file system path protocol (PEP 519), Adding a frame evaluation API to CPython (PEP 523), Change Windows console encoding to UTF-8 (PEP 528), Change Windows filesystem encoding to UTF-8 (PEP 529). GitHub - beanbaginc/django: Unofficial security backports for Django: The Web framework for perfectionists with deadlines. security This is an unofficial fork of Django, which focuses entirely on backporting official, publicly-announced security fixes to Django 1.6.11. It does not contain any other bug fixes or features, and any branches other than security-backports/1.6.x are unlikely to be up-to-date. Upcoming Conference / User Group Meet Reunión Python Valencia Reunión Python Valencia Sydney Python User Group PyConES - Almería Edmonton Python User Group IndyPy Monthly Meetup Python Brasil [12] Santa Cruz Python Meetup Projects fmap - 6 Stars, 0 Fork fmap.py - a single dispatch version of fmap for Python3. While there are multiple Haskellesque 'lets put monads in Python!' style libraries out there, most don't seem to focus on taking the nice bits of Haskell's functional approach and giving them a nice Pythonic interface. fmap.py is a very simple take on fmap that lets you remove some unnecesary boiler plate when you are applying a function to each element of a collection. I hope you like it! fbtftp - 5 Stars, 0 Fork fbtftp is Facebook's implementation of a dynamic TFTP server framework. It lets you create custom TFTP servers and wrap your own logic into it in a very simple manner. Facebook currently uses it in production, and it's deployed at global scale across all of our data centers. unfurl - 4 Stars, 0 Fork Python utility to move items in a directory tree to the topmost level possible chalk - 2 Stars, 1 Fork Simple, easy to learn interpreted programming language. human-to-geojson - 2 Stars, 1 Fork Convert raw Human exports to geoJSON

29 Sep 2016 10:40am GMT

By Vasudev Ram

Peewee => PDF

Peewee is a small, expressive ORM for Python, created by Charles Leifer.

After trying out Peewee a bit, I thought of writing another application of xtopdf (my Python toolkit for PDF creation), to publish Peewee data to PDF. I used an SQLite database underlying the Peewee ORM, but it also supports MySQL and PostgreSQL, per the docs. Here is the program, in file PeeweeToPDF.py:

# PeeweeToPDF.py
# Purpose: To show basics of publishing Peewee ORM data to PDF.
# Requires: Peewee ORM and xtopdf.
# Author: Vasudev Ram
# Copyright 2016 Vasudev Ram
# Web site: https://vasudevram.github.io
# Blog: http://jugad2.blogspot.com
# Product store: https://gumroad.com/vasudevram

from peewee import *
from PDFWriter import PDFWriter

def print_and_write(pw, s):
    print s
    pw.writeLine(s)

# Define the database.
db = SqliteDatabase('contacts.db')

# Define the model for contacts.
class Contact(Model):
    name = CharField()
    age = IntegerField()
    skills = CharField()
    title = CharField()

    class Meta:
        database = db

# Connect to the database.
db.connect() 

# Drop the Contact table if it exists.
db.drop_tables([Contact])

# Create the Contact table.
db.create_tables([Contact])

# Define some contact rows.
contacts = (
    ('Albert Einstein', 22, 'Science', 'Physicist'),
    ('Benjamin Franklin', 32, 'Many', 'Polymath'),
    ('Samuel Johnson', 42, 'Writing', 'Writer')
)

# Save the contact rows to the contacts table.
for contact in contacts:
    c = Contact(name=contact[0], age=contact[1], \
    skills=contact[2], title=contact[3])
    c.save()

sep = '-' * (20 + 5 + 10 + 15)

# Publish the contact rows to PDF.
with PDFWriter('contacts.pdf') as pw:
    pw.setFont('Courier', 12)
    pw.setHeader('Demo of publishing Peewee ORM data to PDF')
    pw.setFooter('Generated by xtopdf: slides.com/vasudevram/xtopdf')
    print_and_write(pw, sep)
    print_and_write(pw, 
        "Name".ljust(20) + "Age".center(5) + 
        "Skills".ljust(10) + "Title".ljust(15))
    print_and_write(pw, sep)

    # Loop over all rows queried from the contacts table.
    for contact in Contact.select():
        print_and_write(pw, 
            contact.name.ljust(20) + 
            str(contact.age).center(5) + 
            contact.skills.ljust(10) + 
            contact.title.ljust(15))
    print_and_write(pw, sep)

# Close the database connection.
db.close()

I could have used Python's namedtuple feature instead of tuples, but did not do it for this small program.

I ran the program with:

python PeeweeToPDF.py

Here is a screenshot of the output as seen in Foxit PDF Reader (click image to enlarge):

- Enjoy.

- Vasudev Ram - Online Python training and consulting Get updates on my software products / ebooks / courses. Jump to posts: Python DLang xtopdf Subscribe to my blog by email My ActiveState recipes FlyWheel - Managed WordPress Hosting

29 Sep 2016 3:28am GMT

By Vasudev Ram

Peewee => PDF

Peewee is a small, expressive ORM for Python, created by Charles Leifer.

After trying out Peewee a bit, I thought of writing another application of xtopdf (my Python toolkit for PDF creation), to publish Peewee data to PDF. I used an SQLite database underlying the Peewee ORM, but it also supports MySQL and PostgreSQL, per the docs. Here is the program, in file PeeweeToPDF.py:

# PeeweeToPDF.py
# Purpose: To show basics of publishing Peewee ORM data to PDF.
# Requires: Peewee ORM and xtopdf.
# Author: Vasudev Ram
# Copyright 2016 Vasudev Ram
# Web site: https://vasudevram.github.io
# Blog: http://jugad2.blogspot.com
# Product store: https://gumroad.com/vasudevram

from peewee import *
from PDFWriter import PDFWriter

def print_and_write(pw, s):
    print s
    pw.writeLine(s)

# Define the database.
db = SqliteDatabase('contacts.db')

# Define the model for contacts.
class Contact(Model):
    name = CharField()
    age = IntegerField()
    skills = CharField()
    title = CharField()

    class Meta:
        database = db

# Connect to the database.
db.connect() 

# Drop the Contact table if it exists.
db.drop_tables([Contact])

# Create the Contact table.
db.create_tables([Contact])

# Define some contact rows.
contacts = (
    ('Albert Einstein', 22, 'Science', 'Physicist'),
    ('Benjamin Franklin', 32, 'Many', 'Polymath'),
    ('Samuel Johnson', 42, 'Writing', 'Writer')
)

# Save the contact rows to the contacts table.
for contact in contacts:
    c = Contact(name=contact[0], age=contact[1], \
    skills=contact[2], title=contact[3])
    c.save()

sep = '-' * (20 + 5 + 10 + 15)

# Publish the contact rows to PDF.
with PDFWriter('contacts.pdf') as pw:
    pw.setFont('Courier', 12)
    pw.setHeader('Demo of publishing Peewee ORM data to PDF')
    pw.setFooter('Generated by xtopdf: slides.com/vasudevram/xtopdf')
    print_and_write(pw, sep)
    print_and_write(pw, 
        "Name".ljust(20) + "Age".center(5) + 
        "Skills".ljust(10) + "Title".ljust(15))
    print_and_write(pw, sep)

    # Loop over all rows queried from the contacts table.
    for contact in Contact.select():
        print_and_write(pw, 
            contact.name.ljust(20) + 
            str(contact.age).center(5) + 
            contact.skills.ljust(10) + 
            contact.title.ljust(15))
    print_and_write(pw, sep)

# Close the database connection.
db.close()

I could have used Python's namedtuple feature instead of tuples, but did not do it for this small program.

I ran the program with:

python PeeweeToPDF.py

Here is a screenshot of the output as seen in Foxit PDF Reader (click image to enlarge):

- Enjoy.

- Vasudev Ram - Online Python training and consulting Get updates on my software products / ebooks / courses. Jump to posts: Python DLang xtopdf Subscribe to my blog by email My ActiveState recipes FlyWheel - Managed WordPress Hosting

29 Sep 2016 3:28am GMT

Introduction

One of the critical things that developers have not been able to do in previous versions of the Google Sheets API is to format cells... that's a big deal! Anyway, the past is the past, and I choose to look ahead. In my earlier post on the Google Sheets API, I introduced Sheets API v4 with a tutorial on how to transfer data from a SQL database to a Sheet. You'd do that primarily to make database data more presentable rather than deliberately switching to a storage mechanism with weaker querying capabilities. At the very end of that post, I challenged readers to try formatting. If you got stuck, confused, or haven't had a chance yet, today's your lucky day. One caveat is that there's more JavaScript in this post than Python... you've been warned!

Using the Google Sheets API

We need to write (formatting) into a Google Sheet, so we need the same scope as last time, read-write:

• 'https://www.googleapis.com/auth/spreadsheets' - Read-write access to Sheets and Sheet properties

Since we've fully covered the authorization boilerplate fully in earlier posts and videos, including how to connect to the Sheets API, we're going to skip that here and jump right to the action.

Formatting cells in Google Sheets

The way the API works, in general, is to take one or more commands, and execute them on the Sheet. This comes in the form of individual requests, either to cells, a Sheet, or the entire spreadsheet. A group if requests is organized as a JavaScript array. Each request in the array is represented by a JSON object. Yes, this part of the post may seem like a long exercise in JavaScript, but stay with me here. Continuing... once your array is complete, you send all the requests to the Sheet using the SHEETS.spreadsheets().batchUpdate() command. Here's pseudocode sending 5 commands to the API:

SHEET_ID =  . . .

reqs = {'requests': [
    {'updateSheetProperties':
        . . .
    {'repeatCell':
        . . .
    {'setDataValidation':
        . . .
    {'sortRange':
        . . .
    {'addChart':
        . . .
]}
SHEETS.spreadsheets().batchUpdate(
        spreadsheetId=SHEET_ID, body=reqs).execute()

What we're executing will be similar. The target spreadsheet will be the one you get when you run the code from the previous post, only without the timestamp in its title as it's unnecessary:


Once you've run the earlier script and created a Sheet of your own, be sure to assign it to the SHEET_ID variable. The goal is to send enough formatting commands to arrive at the same spreadsheet but with improved visuals:


Four (4) requests are needed to bring the original Sheet to this state:

1. Set the top row as "frozen", meaning it doesn't scroll even when the data does
2. Also bold the first row, as these are the column headers
3. Format column E as US currency with dollar sign & 2 decimal places
4. Set data validation for column F, requiring values from a fixed set

Creating Sheets API requests

As mentioned before, each request is represented by a JSON object, cleverly disguised as Python dictionaries in this post, and the entire request array is implemented as a Python list. Let's take a look at what it takes to together the individual requests:

Frozen rows

Frozen rows is a Sheet property, so in order to change it, users must employ the updateSheetProperties command. Specifically, frozenRowCount is a grid property, meaning the field that must be updated is gridProperties.frozenRowCount, set to 1. Here's the Python dict (that gets converted to a JSON object) representing this request:

{'updateSheetProperties': {
    'properties': {'gridProperties': {'frozenRowCount': 1}},
    'fields': 'gridProperties.frozenRowCount',
}},

The properties attribute specifies what is changing and what the new value is. The fields property serves as an attribute "mask." It's how you specify what to alter and what to leave alone when applying a change. In this case, both the properties and fields attributes refer to the same thing: the frozen row count grid property. If you leave out the fields attribute here, sure the frozen row count would be set but all other grid properties would be undone, not such a good side effect. It's okay if it doesn't make a lot of sense yet... there are more examples coming.

Bold formatting

Text formatting, such as bold or italics, is a cell operation. Since we want to apply this formatting to multiple cells, the correct command is, repeatCell. Specifically, what needs to be changed about a cell? A cell's userEnteredFormat.textFormat.bold attribute. This is a simple Boolean value, so we set it to True. The fields masks are as described above... we need to tell the API to explicitly change the just userEnteredFormat.textFormat.bold attribute. Everything else should stay as-is.

The last thing we need is to tell the API which cells in the Sheet should be formatted. For this we have range. This attribute tells the API what Sheet (by ID) and which cells (column and row ranges) in that Sheet to format. Above, you see that we want to bold just one row, row #1. Similarly, there are currently six columns to bold, A-F.

However, like most modern computer systems, the API supports start and end index values beginning with zero... not alphabetic column names nor rows consisting of whole numbers, and the range is exclusive of the end index, meaning it goes up to but does not include the ending row or column. For row 1, this means a starting index of 0 and an ending index of 1. Similarly, columns A-F have start & end index value of 0 and 6, respectively. Visually, here's how you compare traditional row & column values to 0-based index counting:


Here's the dict representing this request:

{'repeatCell': {
    'range': {
        'sheetId': 0,
        'startColumnIndex': 0,
        'endColumnIndex': 6,
        'startRowIndex': 0,
        'endRowIndex': 1
    },
    'cell': {'userEnteredFormat': {'textFormat': {'bold': True}}},
    'fields': 'userEnteredFormat.textFormat.bold',
}},

Before we move on, let's talk about some shortcuts we can make. The ID of the first Sheet created for you is 0. If that's the Sheet you're using, then you can omit passing the Sheet ID. Similarly, the starting row and column indexes default to 0, so you can leave those out too if those are the values to be used. Finally, while an ending column index of 6 works, it won't if more columns are added later. It's best if you just omit the ending index altogether, meaning you want that entire row formatted. All this means that the only thing in the range you need is the ending row index. Instead of the above, your request can be shortened to:

{'repeatCell': {
    'range': {'endRowIndex': 1},
    'cell': {'userEnteredFormat': {'textFormat': {'bold': True}}},
    'fields': 'userEnteredFormat.textFormat.bold',
}},

Range quiz

Okay, now that you know about ranges, take this quiz: assumming the Sheet ID is 0, what are the starting and ending column and row indexes for the four cells highlighted in blue in this Sheet?


If you went with starting and ending column indexes of 3 and 5 and row indexes of 2 and 4, then you're right on the money and ready for more!

Currency formatting

Currency formatting is similar to text formatting, only with numbers, meaning that instead of userEnteredFormat.textFormat, you'd be setting a cell's userEnteredFormat.numberFormat attribute. The command is also repeatCell. Clearly the starting and ending column indexes should be 4 and 5 with a starting row index of 1. But just like the cell bolding we did above, there's no need to restrict ourselves to just the 5 rows of data as more may be coming. Yes, it's best to leave off the ending row index so that the rest of the column is formatted. The only thing you need to learn is how to format cells using US currency, but that's pretty easy to do after a quick look at the docs on formatting numbers:

{'repeatCell': {
    'range': {
        'startRowIndex': 1,
        'startColumnIndex': 4,
        'endColumnIndex': 5,
    },
    'cell': {
        'userEnteredFormat': {
            'numberFormat': {
                'type': 'CURRENCY',
                'pattern': '"$"#,##0.00',
            },
        },
    },
    'fields': 'userEnteredFormat.numberFormat',
}}

More on fields

One caveat to our sample app here is that all of the fields mask only have a single value, the one we want to change, but that's not always the case. There may be situations where you want to effect a variety of changes to cells. To see more examples of fields, check out this page in the docs featuring more formatting examples. To learn more about how masking works, check out this page and this one too.

Cell validation

The final formatting request implements cell validation on column F as well as restricting their possible values. The command used here is setDataValidation. The range is similar to that of currency formatting, only for column F, meaning a starting row index of 1, and starting and ending column indexes of 5 and 6, respectively. The rule implements the restriction. Similar to other spreadsheet software, you can restrict cell values in any number of ways, as outlined by the ConditionType documentation page. Ours is to allow for one of three possible values, so the ConditionType is ONE_OF_LIST.

When you restrict cell values, you can choose to allow but flag it (weak enforcement) or disallow any value outside of what you specify (strict enforcement). If you wish to employ strict enforcement, you need to pass in a strict attribute with a True value. The default is weak enforcement, or False. In either case, users entering invalid values will get a default warning message that the input is not allowed. If you prefer a custom message over the default option, you can pass that to the API as the inputMessage attribute. I prefer the system default and elect not to use it here. Here are the 4 combinations of what shows up when you use or don't use inputMessage with strict and weak enforcement:


No inputMessage (default) + weak enforcement


With inputMessage (my custom msg) + weak enforcement


No inputMessage (default) + strict enforcement


With inputMessage (my custom msg) + weak enforcement

The last attribute you can send is showCustomUi. If the showCustomUi flag is set to True, the Sheets user interface will display a small pulldown menu listing the values accepted by the cell. It's a pretty poor user experience without it (because users won't know what the available choices are), so I recommend you always use it too. With that, this request looks like this:

{'setDataValidation': {
    'range': {
        'startRowIndex': 1,
        'startColumnIndex': 5,
        'endColumnIndex': 6,
    },
    'rule': {
        'condition': {
            'type': 'ONE_OF_LIST',
            'values': [
                {'userEnteredValue': 'PENDING'},
                {'userEnteredValue': 'SHIPPED'},
                {'userEnteredValue': 'DELIVERED'},
            ]
        },
        #'inputMessage': 'Select PENDING, SHIPPED, or DELIVERED',
        #'strict': True,
        'showCustomUi': True,
    },
}}

Since we're not modifying cell attributes, but instead focusing on validation, you'll notice there's no fields mask in these types of requests.

Running our script

Believe it or not, that's the bulk of this application. With the reqs list of these four requests, the last line of code calls the Sheets API exactly like the pseudocode above. Now you can simply run it:

$ python sheets_cell_format.py # or python3
$

There's no output from this script, so you should only expect that your Sheet will be formatted once it has completed. If you bring up the Sheet in the user interface, you should see the changes happening in near real-time:

Conclusion

Below is the entire script for your convenience which runs on both Python 2 and Python 3 (unmodified!):

from apiclient import discovery
from httplib2 import Http
from oauth2client import file, client, tools

SCOPES = 'https://www.googleapis.com/auth/spreadsheets'
store = file.Storage('storage.json')
creds = store.get()
if not creds or creds.invalid:
    flow = client.flow_from_clientsecrets('client_id.json', SCOPES)
    creds = tools.run_flow(flow, store)
SHEETS = discovery.build('sheets', 'v4', http=creds.authorize(Http()))

SHEET_ID = ... # add your Sheet ID here
reqs = {'requests': [
    # frozen row 1
    {'updateSheetProperties': {
        'properties': {'gridProperties': {'frozenRowCount': 1}},
        'fields': 'gridProperties.frozenRowCount',
    }},
    # embolden row 1
    {'repeatCell': {
        'range': {'endRowIndex': 1},
        'cell': {'userEnteredFormat': {'textFormat': {'bold': True}}},
        'fields': 'userEnteredFormat.textFormat.bold',
    }},
    # currency format for column E (E2:E7)
    {'repeatCell': {
        'range': {
            'startRowIndex': 1,
            'endRowIndex': 6,
            'startColumnIndex': 4,
            'endColumnIndex': 5,
        },
        'cell': {
            'userEnteredFormat': {
                'numberFormat': {
                    'type': 'CURRENCY',
                    'pattern': '"$"#,##0.00',
                },
            },
        },
        'fields': 'userEnteredFormat.numberFormat',
    }},
    # validation for column F (F2:F7)
    {'setDataValidation': {
        'range': {
            'startRowIndex': 1,
            'endRowIndex': 6,
            'startColumnIndex': 5,
            'endColumnIndex': 6,
        },
        'rule': {
            'condition': {
                'type': 'ONE_OF_LIST',
                'values': [
                    {'userEnteredValue': 'PENDING'},
                    {'userEnteredValue': 'SHIPPED'},
                    {'userEnteredValue': 'DELIVERED'},
                ]
            },
            #'inputMessage': 'Select PENDING, SHIPPED, or DELIVERED',
            #'strict': True,
            'showCustomUi': True,
        },
    }},
]}

res = SHEETS.spreadsheets().batchUpdate(
        spreadsheetId=SHEET_ID, body=reqs).execute()

As with our other code samples, you can now customize for your own needs, for a mobile frontend, sysadmin script, or a server-side backend, perhaps accessing other Google APIs.

Code challenge

Once you fully grasp this sample and are ready for a challenge: Use the API to create a column "G" with a "Total Cost" header in cell G1, set cell G2 with the formula to calculate the cost based on toys ordered & cost in columns D & E then and create an autoFill request to replicate that formula down column G. When you're done, the right-hand side of your Sheet now looks like this:

Here are some steps you can take to achieve this improvement:

1. Create column G with a "Total Cost" header in cell G1; make sure it's bold too (or do you have to?)
2. Set cell G2 with formula =MULTIPLY(D2,E2)
3. Use autoFill command to copy formula from G2 down the entire column (HINT: you only need the range attribute)

You're now well under way to being able to writing useful applications with the Sheets API!

28 Sep 2016 7:32pm GMT

Introduction

One of the critical things that developers have not been able to do in previous versions of the Google Sheets API is to format cells... that's a big deal! Anyway, the past is the past, and I choose to look ahead. In my earlier post on the Google Sheets API, I introduced Sheets API v4 with a tutorial on how to transfer data from a SQL database to a Sheet. You'd do that primarily to make database data more presentable rather than deliberately switching to a storage mechanism with weaker querying capabilities. At the very end of that post, I challenged readers to try formatting. If you got stuck, confused, or haven't had a chance yet, today's your lucky day. One caveat is that there's more JavaScript in this post than Python... you've been warned!

Using the Google Sheets API

We need to write (formatting) into a Google Sheet, so we need the same scope as last time, read-write:

• 'https://www.googleapis.com/auth/spreadsheets' - Read-write access to Sheets and Sheet properties

Since we've fully covered the authorization boilerplate fully in earlier posts and videos, including how to connect to the Sheets API, we're going to skip that here and jump right to the action.

Formatting cells in Google Sheets

The way the API works, in general, is to take one or more commands, and execute them on the Sheet. This comes in the form of individual requests, either to cells, a Sheet, or the entire spreadsheet. A group if requests is organized as a JavaScript array. Each request in the array is represented by a JSON object. Yes, this part of the post may seem like a long exercise in JavaScript, but stay with me here. Continuing... once your array is complete, you send all the requests to the Sheet using the SHEETS.spreadsheets().batchUpdate() command. Here's pseudocode sending 5 commands to the API:

SHEET_ID =  . . .

reqs = {'requests': [
    {'updateSheetProperties':
        . . .
    {'repeatCell':
        . . .
    {'setDataValidation':
        . . .
    {'sortRange':
        . . .
    {'addChart':
        . . .
]}
SHEETS.spreadsheets().batchUpdate(
        spreadsheetId=SHEET_ID, body=reqs).execute()

What we're executing will be similar. The target spreadsheet will be the one you get when you run the code from the previous post, only without the timestamp in its title as it's unnecessary:


Once you've run the earlier script and created a Sheet of your own, be sure to assign it to the SHEET_ID variable. The goal is to send enough formatting commands to arrive at the same spreadsheet but with improved visuals:


Four (4) requests are needed to bring the original Sheet to this state:

1. Set the top row as "frozen", meaning it doesn't scroll even when the data does
2. Also bold the first row, as these are the column headers
3. Format column E as US currency with dollar sign & 2 decimal places
4. Set data validation for column F, requiring values from a fixed set

Creating Sheets API requests

As mentioned before, each request is represented by a JSON object, cleverly disguised as Python dictionaries in this post, and the entire request array is implemented as a Python list. Let's take a look at what it takes to together the individual requests:

Frozen rows

Frozen rows is a Sheet property, so in order to change it, users must employ the updateSheetProperties command. Specifically, frozenRowCount is a grid property, meaning the field that must be updated is gridProperties.frozenRowCount, set to 1. Here's the Python dict (that gets converted to a JSON object) representing this request:

{'updateSheetProperties': {
    'properties': {'gridProperties': {'frozenRowCount': 1}},
    'fields': 'gridProperties.frozenRowCount',
}},

The properties attribute specifies what is changing and what the new value is. The fields property serves as an attribute "mask." It's how you specify what to alter and what to leave alone when applying a change. In this case, both the properties and fields attributes refer to the same thing: the frozen row count grid property. If you leave out the fields attribute here, sure the frozen row count would be set but all other grid properties would be undone, not such a good side effect. It's okay if it doesn't make a lot of sense yet... there are more examples coming.

Bold formatting

Text formatting, such as bold or italics, is a cell operation. Since we want to apply this formatting to multiple cells, the correct command is, repeatCell. Specifically, what needs to be changed about a cell? A cell's userEnteredFormat.textFormat.bold attribute. This is a simple Boolean value, so we set it to True. The fields masks are as described above... we need to tell the API to explicitly change the just userEnteredFormat.textFormat.bold attribute. Everything else should stay as-is.

The last thing we need is to tell the API which cells in the Sheet should be formatted. For this we have range. This attribute tells the API what Sheet (by ID) and which cells (column and row ranges) in that Sheet to format. Above, you see that we want to bold just one row, row #1. Similarly, there are currently six columns to bold, A-F.

However, like most modern computer systems, the API supports start and end index values beginning with zero... not alphabetic column names nor rows consisting of whole numbers, and the range is exclusive of the end index, meaning it goes up to but does not include the ending row or column. For row 1, this means a starting index of 0 and an ending index of 1. Similarly, columns A-F have start & end index value of 0 and 6, respectively. Visually, here's how you compare traditional row & column values to 0-based index counting:


Here's the dict representing this request:

{'repeatCell': {
    'range': {
        'sheetId': 0,
        'startColumnIndex': 0,
        'endColumnIndex': 6,
        'startRowIndex': 0,
        'endRowIndex': 1
    },
    'cell': {'userEnteredFormat': {'textFormat': {'bold': True}}},
    'fields': 'userEnteredFormat.textFormat.bold',
}},

Before we move on, let's talk about some shortcuts we can make. The ID of the first Sheet created for you is 0. If that's the Sheet you're using, then you can omit passing the Sheet ID. Similarly, the starting row and column indexes default to 0, so you can leave those out too if those are the values to be used. Finally, while an ending column index of 6 works, it won't if more columns are added later. It's best if you just omit the ending index altogether, meaning you want that entire row formatted. All this means that the only thing in the range you need is the ending row index. Instead of the above, your request can be shortened to:

{'repeatCell': {
    'range': {'endRowIndex': 1},
    'cell': {'userEnteredFormat': {'textFormat': {'bold': True}}},
    'fields': 'userEnteredFormat.textFormat.bold',
}},

Range quiz

Okay, now that you know about ranges, take this quiz: assumming the Sheet ID is 0, what are the starting and ending column and row indexes for the four cells highlighted in blue in this Sheet?


If you went with starting and ending column indexes of 3 and 5 and row indexes of 2 and 4, then you're right on the money and ready for more!

Currency formatting

Currency formatting is similar to text formatting, only with numbers, meaning that instead of userEnteredFormat.textFormat, you'd be setting a cell's userEnteredFormat.numberFormat attribute. The command is also repeatCell. Clearly the starting and ending column indexes should be 4 and 5 with a starting row index of 1. But just like the cell bolding we did above, there's no need to restrict ourselves to just the 5 rows of data as more may be coming. Yes, it's best to leave off the ending row index so that the rest of the column is formatted. The only thing you need to learn is how to format cells using US currency, but that's pretty easy to do after a quick look at the docs on formatting numbers:

{'repeatCell': {
    'range': {
        'startRowIndex': 1,
        'startColumnIndex': 4,
        'endColumnIndex': 5,
    },
    'cell': {
        'userEnteredFormat': {
            'numberFormat': {
                'type': 'CURRENCY',
                'pattern': '"$"#,##0.00',
            },
        },
    },
    'fields': 'userEnteredFormat.numberFormat',
}}

More on fields

One caveat to our sample app here is that all of the fields mask only have a single value, the one we want to change, but that's not always the case. There may be situations where you want to effect a variety of changes to cells. To see more examples of fields, check out this page in the docs featuring more formatting examples. To learn more about how masking works, check out this page and this one too.

Cell validation

The final formatting request implements cell validation on column F as well as restricting their possible values. The command used here is setDataValidation. The range is similar to that of currency formatting, only for column F, meaning a starting row index of 1, and starting and ending column indexes of 5 and 6, respectively. The rule implements the restriction. Similar to other spreadsheet software, you can restrict cell values in any number of ways, as outlined by the ConditionType documentation page. Ours is to allow for one of three possible values, so the ConditionType is ONE_OF_LIST.

When you restrict cell values, you can choose to allow but flag it (weak enforcement) or disallow any value outside of what you specify (strict enforcement). If you wish to employ strict enforcement, you need to pass in a strict attribute with a True value. The default is weak enforcement, or False. In either case, users entering invalid values will get a default warning message that the input is not allowed. If you prefer a custom message over the default option, you can pass that to the API as the inputMessage attribute. I prefer the system default and elect not to use it here. Here are the 4 combinations of what shows up when you use or don't use inputMessage with strict and weak enforcement:


No inputMessage (default) + weak enforcement


With inputMessage (my custom msg) + weak enforcement


No inputMessage (default) + strict enforcement


With inputMessage (my custom msg) + weak enforcement

The last attribute you can send is showCustomUi. If the showCustomUi flag is set to True, the Sheets user interface will display a small pulldown menu listing the values accepted by the cell. It's a pretty poor user experience without it (because users won't know what the available choices are), so I recommend you always use it too. With that, this request looks like this:

{'setDataValidation': {
    'range': {
        'startRowIndex': 1,
        'startColumnIndex': 5,
        'endColumnIndex': 6,
    },
    'rule': {
        'condition': {
            'type': 'ONE_OF_LIST',
            'values': [
                {'userEnteredValue': 'PENDING'},
                {'userEnteredValue': 'SHIPPED'},
                {'userEnteredValue': 'DELIVERED'},
            ]
        },
        #'inputMessage': 'Select PENDING, SHIPPED, or DELIVERED',
        #'strict': True,
        'showCustomUi': True,
    },
}}

Since we're not modifying cell attributes, but instead focusing on validation, you'll notice there's no fields mask in these types of requests.

Running our script

Believe it or not, that's the bulk of this application. With the reqs list of these four requests, the last line of code calls the Sheets API exactly like the pseudocode above. Now you can simply run it:

$ python sheets_cell_format.py # or python3
$

There's no output from this script, so you should only expect that your Sheet will be formatted once it has completed. If you bring up the Sheet in the user interface, you should see the changes happening in near real-time:

Conclusion

Below is the entire script for your convenience which runs on both Python 2 and Python 3 (unmodified!):

from apiclient import discovery
from httplib2 import Http
from oauth2client import file, client, tools

SCOPES = 'https://www.googleapis.com/auth/spreadsheets'
store = file.Storage('storage.json')
creds = store.get()
if not creds or creds.invalid:
    flow = client.flow_from_clientsecrets('client_id.json', SCOPES)
    creds = tools.run_flow(flow, store)
SHEETS = discovery.build('sheets', 'v4', http=creds.authorize(Http()))

SHEET_ID = ... # add your Sheet ID here
reqs = {'requests': [
    # frozen row 1
    {'updateSheetProperties': {
        'properties': {'gridProperties': {'frozenRowCount': 1}},
        'fields': 'gridProperties.frozenRowCount',
    }},
    # embolden row 1
    {'repeatCell': {
        'range': {'endRowIndex': 1},
        'cell': {'userEnteredFormat': {'textFormat': {'bold': True}}},
        'fields': 'userEnteredFormat.textFormat.bold',
    }},
    # currency format for column E (E2:E7)
    {'repeatCell': {
        'range': {
            'startRowIndex': 1,
            'endRowIndex': 6,
            'startColumnIndex': 4,
            'endColumnIndex': 5,
        },
        'cell': {
            'userEnteredFormat': {
                'numberFormat': {
                    'type': 'CURRENCY',
                    'pattern': '"$"#,##0.00',
                },
            },
        },
        'fields': 'userEnteredFormat.numberFormat',
    }},
    # validation for column F (F2:F7)
    {'setDataValidation': {
        'range': {
            'startRowIndex': 1,
            'endRowIndex': 6,
            'startColumnIndex': 5,
            'endColumnIndex': 6,
        },
        'rule': {
            'condition': {
                'type': 'ONE_OF_LIST',
                'values': [
                    {'userEnteredValue': 'PENDING'},
                    {'userEnteredValue': 'SHIPPED'},
                    {'userEnteredValue': 'DELIVERED'},
                ]
            },
            #'inputMessage': 'Select PENDING, SHIPPED, or DELIVERED',
            #'strict': True,
            'showCustomUi': True,
        },
    }},
]}

res = SHEETS.spreadsheets().batchUpdate(
        spreadsheetId=SHEET_ID, body=reqs).execute()

As with our other code samples, you can now customize for your own needs, for a mobile frontend, sysadmin script, or a server-side backend, perhaps accessing other Google APIs.

Code challenge

Once you fully grasp this sample and are ready for a challenge: Use the API to create a column "G" with a "Total Cost" header in cell G1, set cell G2 with the formula to calculate the cost based on toys ordered & cost in columns D & E then and create an autoFill request to replicate that formula down column G. When you're done, the right-hand side of your Sheet now looks like this:

Here are some steps you can take to achieve this improvement:

1. Create column G with a "Total Cost" header in cell G1; make sure it's bold too (or do you have to?)
2. Set cell G2 with formula =MULTIPLY(D2,E2)
3. Use autoFill command to copy formula from G2 down the entire column (HINT: you only need the range attribute)

You're now well under way to being able to writing useful applications with the Sheets API!

28 Sep 2016 7:32pm GMT

I just upgraded to Python 2.7.12 and my code is now erroring with rather unhelpful Mixing iteration and read methods would lose data errors. Googling for this yields a number of StackOverflow posts where people are iterating over a file, and then within the loop, doing calls on the file. This sounds like a good error for their case, but in my case the fixes are somewhat more inane.

This errors:

    persistence.write_dict_uint32_to_list_of_uint32s(output_file, { k: [] for k in read_set_of_uint32s(input_file) })

This fixes that error:

    set_value = persistence.read_set_of_uint32s(input_file)
    persistence.write_dict_uint32_to_list_of_uint32s(output_file, { k: [] for k in set_value })

Another error at another location:

    persistence.write_set_of_uint32s(output_file, persistence.read_set_of_uint32s(input_file))

Some simple rearranging makes this error go away:

    set_value = persistence.read_set_of_uint32s(input_file)
    persistence.write_set_of_uint32s(output_file, set_value)

And finally, the next error at another location:

def read_string(f):
    s = ""
    while 1:
        v = f.read(1) # errors here
        if v == '\0':
            break
        s += v
    return s

And the fix for this:

def read_bytes(f, num_bytes):
    return f.read(num_bytes)

def read_string(f):
    s = ""
    while 1:
        v = read_bytes(f, 1)
        if v == '\0':
            break
        s += v
    return s

This error seems like a horribly broken idea, that is popping up in places it wasn't intended.

28 Sep 2016 5:58pm GMT

I just upgraded to Python 2.7.12 and my code is now erroring with rather unhelpful Mixing iteration and read methods would lose data errors. Googling for this yields a number of StackOverflow posts where people are iterating over a file, and then within the loop, doing calls on the file. This sounds like a good error for their case, but in my case the fixes are somewhat more inane.

This errors:

    persistence.write_dict_uint32_to_list_of_uint32s(output_file, { k: [] for k in read_set_of_uint32s(input_file) })

This fixes that error:

    set_value = persistence.read_set_of_uint32s(input_file)
    persistence.write_dict_uint32_to_list_of_uint32s(output_file, { k: [] for k in set_value })

Another error at another location:

    persistence.write_set_of_uint32s(output_file, persistence.read_set_of_uint32s(input_file))

Some simple rearranging makes this error go away:

    set_value = persistence.read_set_of_uint32s(input_file)
    persistence.write_set_of_uint32s(output_file, set_value)

And finally, the next error at another location:

def read_string(f):
    s = ""
    while 1:
        v = f.read(1) # errors here
        if v == '\0':
            break
        s += v
    return s

And the fix for this:

def read_bytes(f, num_bytes):
    return f.read(num_bytes)

def read_string(f):
    s = ""
    while 1:
        v = read_bytes(f, 1)
        if v == '\0':
            break
        s += v
    return s

This error seems like a horribly broken idea, that is popping up in places it wasn't intended.

28 Sep 2016 5:58pm GMT

The title for the blog post is not something I wrote but copy pasted a tweet by @kushaldas. Kushal is being attending PyCon India since 2010, a year after it started. This one tweet of his says if not the whole but lot of it. So, lets explore our journey of PyLadies in PyCon India

August

In the first week of August I had mailed the PyCon India Organizers requesting a table for PyLadies during PyCon India. But mid August got a reply that they will be unable to give one since we were not the sponsors. Then Rupali got into the matter and she said that Red Hat (the Yoda for PyLadies Pune) would love to give us a space in their booth to promote PyLadies. With the that support we started our planning for PyCon. We had a lot of things to do. The prime of them was that to break to earlier notion about PyLadies in India. A Glibc developer, Siddhesh Poyarekar actually sponsored Python tee shirts for us. This is an example the true diversity and the beauty of the Open Source Community.

Day 0

Went to the venue to check things and the Red Hat booth. Was a little disappointed to see that it is on the last corner, and not in front of the main auditorium. Asked Janki and Dhriti to prepare their lightning talks about their work in Python.

Day 1

The day 1 started with a shrill alarm set at 5:30 in the morning. I had to practice my talk before reaching the venue. In the venue met Nisha (what a relief to see her). Me, Nisha, Pooja, Trishna, Rupali, Janki did all the booth set up ready. Rupali managed to give us a separate table inside the Red Hat booth so that we could keep our stuffs. Nisha had done a great job in designing the posters. Actually the new PyLadies Pune logo fetched attention of many people. Again Red Hat sponsored PyLadies posters too. As we never had the money to print stickers, so we have made print outs of the PyLadies Logo. Trishna made 50 print outs of them. We just cut that out and pined it up on our tee shirts and PyCon India badges. These badges made us noticed. Actually people came and asked but unfortunately we could not give it to everyone (we had only 50 of them and never thought it would be so popular :)). We had the photo prop, the PyLadies Umbrella, I had suggested Nisha to bring it over. Thank you Nisha for making it so nicely.

Then came the tough part going to people and telling them about PyLadies. As we are a group who are starting off, So, when initially I asked if they want to know and come to PyLadies here are some replies I got:

So I was going to people and asking them "Do you want to know about PyLadies?" or "Are you interested in PyLadies?" or "Do you want to join PyLadies?" Here are some answers which I got:

1. "Do you have PyLadies in India! Never heard of that."
2. " What is PyLadies?"
3. " Coming in PyCon India since its inception. Never saw anything happening regarding PyLadies. Does PyLadies exist in India?"
4. " Do you actually code in PyLadies?"
5. " I am interested in Python but not in PyLadies."

That was disappointing (but I was not). We actually explained them about PyLadies. Most importantly that what binds PyLadies is the love to code in Python. Then I explained that what we do in our regular Pyladies Pune meet up. In the first half of the meetups we actually learn some Python syntax. In the second half we write code using them, so when we go back home we actually have something which we have created. We also have sessions on things important to contribute in Open Source. We listen to talks. We have a mailing list in which we keep posting problems regularly. After all these explanations the questions changed to:

1. "How can I join PyLadies?"
2. "I don't live in Pune, so how can I be apart of it?"
3. "Can we open our PyLadies Chapter?"
4. "Can you add me in your mailing list?"
5. " Is it free?"

I directed them towards the Red Hat booth. Meanwhile Nisha, Puja, Trishna, Rupali answered them. After grabbing a quick lunch we all came back. But I should mention here that Rupali never had her lunch only so that she could be there at the booth. I had my talk on PyPI licensing trends. As soon as my talk ended a guy said "you PyLadies is such vibrant group with people doing such different work."

Day 2

We decided that we should have a PyLadies meet up at the Open Space. Since we are building the PyLadies community in India, we wanted some tips some directions regarding that. We approached Paul Everitt, Dmitry Filippov, Van Lindberg and Jeff Rush for that. The best persons we can have to talk about "how to build a community". We wanted an informal and personal discussion so that the audience could actually connect to it, put up questions and ideas. Rupali started the it with an appropriate note, actually stating what is the scenario of Pyladies in India. Then Paul, Dmitry, Van and Jeff took over the session. Then pointed the following tips about building community : a community

1. must have a forum for discussion, i.e it should be open to any discussions,
2. it must have new faces i.e people who can grow the community.
3. visibility and consistency are the other two primary factors of building a community.
4. There should be actual contribution and everyone should share their contribution with each other.
5. The more you read code the more you will learn.
6. Code reviews should be done in unison.
7. Strong leadership is another point on which they stressed on.
8. The leaders should prepare their replacements who will grow the community further.

Such thoughtful insights. They shared their personal experiences that how they started PyCon with very few people and never thought that will grow this big. I had always this question in mind that hat do people mean by technical work? Is legal stuff which I do primarily can be called technical? I asked Van about it and he answered it in affirmative. It was a true personal boost for me. It was really inspiring when they said that PyLadies in India is in safe hands and we are actually going to the correct directions ( I was actually jumping around after that :))

After lunch Trishna gave her Lightning Talk on " Fedora Atomic", which received nice response.

Outcome : What has changed after PyCon India for PyLadies?

In the feedback session I asked that this time we were denied a table, so how can we get it next year? The Organizers replied that as we were late this time they could not give us one. So I have requested for a PyLadies table next year in the conference, a year in advance (we are quite early this time right) So, hopefully we will have one in the next year as they said.

During lunch of day 1 a senior guy (one of the organizers) came to me and said that, "We never knew that you, PyLadies are doing so much actual work. We had some different ideas based on our previous not that good experiences. But we really want to help you guys. Please let us know how can we do that." After the lunch of day 1 we really did not need to go to people anymore, they were actually coming to us, PyLadies. People were thrilled to know that they can actually join and learn coding with us. Many people came to us to know how they can start their own PyLadies Chapter (at least 4). The biggest compliment came at the end of the conference when a group of girls came and said " You, PyLadies rocked PyCon India. Next time we will come for you guys only."

"Came for the language, stayed for the community"

This quote actually echos my feeling after PyCon. I met people like Paul Everitt, Dmitry Filippov, Van Lindberg and Jeff Rush who has made me realize that the more successful you are the more humble you be. For Jeff, I just requested him just before the panel discussion was starting, and he readily agreed, Thank you Jeff for that and also for the valuable advice you have given me. When I approached rest of the panelists for the discussion, they said just to remind them the time. In fact they came in the open space 5 minutes before the due time. So much to learn from you guys not only professionally, but personally too.

Now its a list of PyLadies whom I need to thank:

• Nisha, I know how tough this was for you to leave your daughter and come here. (I know because I was going through the same thing). You did a great job in everything from making PyLadies Umbrella, designing Posters, Logo to managing the booth, guiding people with how to create their own chapter and every other thing which I am forgetting. I know you were also ready with you Lightning Talk but could not give one because of the unavailability of the slots. Kudos to you lady.

• Rupali, for giving us a space in the Red Hat booth, sponsoring Posters without which we would not have any presence there in PyCon. Giving the awesome introduction in the PyLadies Open Space and managing the booth so well. You are my inspiration in my ways girl.

• Trishna, for printing the badges, managing booth, talking to people, giving the Lightning Talk. You don't know girl how many people you inspire.

• Pooja, this is one member of PyLadies Pune who does all her work silently. She was there everywhere be it the booth duty, talking to people, running around in the whole conference, setting up the booth or writing the mail ids (at 30 of them) for joining PyLadies mailing list and distributing it to everyone in the panel discussion, its a huge job lady. And of course not to forget fetching her husband at the correct time to help us :).

• Janki, though she was very busy with her voluntary duties she manged to talk to people about PyLadies. Most importantly thank you for getting us the space in Open Space for panel discussion.

• Devi for supporting us and for saying that she will send some chapters, problems in the mailing list for us to solve.

Also not forget about Dhriti, Nirupama(who came on the spot and helped us), Pusplata, Twinkle and many other. Thank you all for making PyCon India a success for PyLadies. With all our hard work we actually broke the earlier notion about PyLadies in India.

See you Ladies in the next PyCon India 2017.

28 Sep 2016 11:06am GMT

The title for the blog post is not something I wrote but copy pasted a tweet by @kushaldas. Kushal is being attending PyCon India since 2010, a year after it started. This one tweet of his says if not the whole but lot of it. So, lets explore our journey of PyLadies in PyCon India

August

In the first week of August I had mailed the PyCon India Organizers requesting a table for PyLadies during PyCon India. But mid August got a reply that they will be unable to give one since we were not the sponsors. Then Rupali got into the matter and she said that Red Hat (the Yoda for PyLadies Pune) would love to give us a space in their booth to promote PyLadies. With the that support we started our planning for PyCon. We had a lot of things to do. The prime of them was that to break to earlier notion about PyLadies in India. A Glibc developer, Siddhesh Poyarekar actually sponsored Python tee shirts for us. This is an example the true diversity and the beauty of the Open Source Community.

Day 0

Went to the venue to check things and the Red Hat booth. Was a little disappointed to see that it is on the last corner, and not in front of the main auditorium. Asked Janki and Dhriti to prepare their lightning talks about their work in Python.

Day 1

The day 1 started with a shrill alarm set at 5:30 in the morning. I had to practice my talk before reaching the venue. In the venue met Nisha (what a relief to see her). Me, Nisha, Pooja, Trishna, Rupali, Janki did all the booth set up ready. Rupali managed to give us a separate table inside the Red Hat booth so that we could keep our stuffs. Nisha had done a great job in designing the posters. Actually the new PyLadies Pune logo fetched attention of many people. Again Red Hat sponsored PyLadies posters too. As we never had the money to print stickers, so we have made print outs of the PyLadies Logo. Trishna made 50 print outs of them. We just cut that out and pined it up on our tee shirts and PyCon India badges. These badges made us noticed. Actually people came and asked but unfortunately we could not give it to everyone (we had only 50 of them and never thought it would be so popular :)). We had the photo prop, the PyLadies Umbrella, I had suggested Nisha to bring it over. Thank you Nisha for making it so nicely.

Then came the tough part going to people and telling them about PyLadies. As we are a group who are starting off, So, when initially I asked if they want to know and come to PyLadies here are some replies I got:

So I was going to people and asking them "Do you want to know about PyLadies?" or "Are you interested in PyLadies?" or "Do you want to join PyLadies?" Here are some answers which I got:

1. "Do you have PyLadies in India! Never heard of that."
2. " What is PyLadies?"
3. " Coming in PyCon India since its inception. Never saw anything happening regarding PyLadies. Does PyLadies exist in India?"
4. " Do you actually code in PyLadies?"
5. " I am interested in Python but not in PyLadies."

That was disappointing (but I was not). We actually explained them about PyLadies. Most importantly that what binds PyLadies is the love to code in Python. Then I explained that what we do in our regular Pyladies Pune meet up. In the first half of the meetups we actually learn some Python syntax. In the second half we write code using them, so when we go back home we actually have something which we have created. We also have sessions on things important to contribute in Open Source. We listen to talks. We have a mailing list in which we keep posting problems regularly. After all these explanations the questions changed to:

1. "How can I join PyLadies?"
2. "I don't live in Pune, so how can I be apart of it?"
3. "Can we open our PyLadies Chapter?"
4. "Can you add me in your mailing list?"
5. " Is it free?"

I directed them towards the Red Hat booth. Meanwhile Nisha, Puja, Trishna, Rupali answered them. After grabbing a quick lunch we all came back. But I should mention here that Rupali never had her lunch only so that she could be there at the booth. I had my talk on PyPI licensing trends. As soon as my talk ended a guy said "you PyLadies is such vibrant group with people doing such different work."

Day 2

We decided that we should have a PyLadies meet up at the Open Space. Since we are building the PyLadies community in India, we wanted some tips some directions regarding that. We approached Paul Everitt, Dmitry Filippov, Van Lindberg and Jeff Rush for that. The best persons we can have to talk about "how to build a community". We wanted an informal and personal discussion so that the audience could actually connect to it, put up questions and ideas. Rupali started the it with an appropriate note, actually stating what is the scenario of Pyladies in India. Then Paul, Dmitry, Van and Jeff took over the session. Then pointed the following tips about building community : a community

1. must have a forum for discussion, i.e it should be open to any discussions,
2. it must have new faces i.e people who can grow the community.
3. visibility and consistency are the other two primary factors of building a community.
4. There should be actual contribution and everyone should share their contribution with each other.
5. The more you read code the more you will learn.
6. Code reviews should be done in unison.
7. Strong leadership is another point on which they stressed on.
8. The leaders should prepare their replacements who will grow the community further.

Such thoughtful insights. They shared their personal experiences that how they started PyCon with very few people and never thought that will grow this big. I had always this question in mind that hat do people mean by technical work? Is legal stuff which I do primarily can be called technical? I asked Van about it and he answered it in affirmative. It was a true personal boost for me. It was really inspiring when they said that PyLadies in India is in safe hands and we are actually going to the correct directions ( I was actually jumping around after that :))

After lunch Trishna gave her Lightning Talk on " Fedora Atomic", which received nice response.

Outcome : What has changed after PyCon India for PyLadies?

In the feedback session I asked that this time we were denied a table, so how can we get it next year? The Organizers replied that as we were late this time they could not give us one. So I have requested for a PyLadies table next year in the conference, a year in advance (we are quite early this time right) So, hopefully we will have one in the next year as they said.

During lunch of day 1 a senior guy (one of the organizers) came to me and said that, "We never knew that you, PyLadies are doing so much actual work. We had some different ideas based on our previous not that good experiences. But we really want to help you guys. Please let us know how can we do that." After the lunch of day 1 we really did not need to go to people anymore, they were actually coming to us, PyLadies. People were thrilled to know that they can actually join and learn coding with us. Many people came to us to know how they can start their own PyLadies Chapter (at least 4). The biggest compliment came at the end of the conference when a group of girls came and said " You, PyLadies rocked PyCon India. Next time we will come for you guys only."

"Came for the language, stayed for the community"

This quote actually echos my feeling after PyCon. I met people like Paul Everitt, Dmitry Filippov, Van Lindberg and Jeff Rush who has made me realize that the more successful you are the more humble you be. For Jeff, I just requested him just before the panel discussion was starting, and he readily agreed, Thank you Jeff for that and also for the valuable advice you have given me. When I approached rest of the panelists for the discussion, they said just to remind them the time. In fact they came in the open space 5 minutes before the due time. So much to learn from you guys not only professionally, but personally too.

Now its a list of PyLadies whom I need to thank:

• Nisha, I know how tough this was for you to leave your daughter and come here. (I know because I was going through the same thing). You did a great job in everything from making PyLadies Umbrella, designing Posters, Logo to managing the booth, guiding people with how to create their own chapter and every other thing which I am forgetting. I know you were also ready with you Lightning Talk but could not give one because of the unavailability of the slots. Kudos to you lady.

• Rupali, for giving us a space in the Red Hat booth, sponsoring Posters without which we would not have any presence there in PyCon. Giving the awesome introduction in the PyLadies Open Space and managing the booth so well. You are my inspiration in my ways girl.

• Trishna, for printing the badges, managing booth, talking to people, giving the Lightning Talk. You don't know girl how many people you inspire.

• Pooja, this is one member of PyLadies Pune who does all her work silently. She was there everywhere be it the booth duty, talking to people, running around in the whole conference, setting up the booth or writing the mail ids (at 30 of them) for joining PyLadies mailing list and distributing it to everyone in the panel discussion, its a huge job lady. And of course not to forget fetching her husband at the correct time to help us :).

• Janki, though she was very busy with her voluntary duties she manged to talk to people about PyLadies. Most importantly thank you for getting us the space in Open Space for panel discussion.

• Devi for supporting us and for saying that she will send some chapters, problems in the mailing list for us to solve.

Also not forget about Dhriti, Nirupama(who came on the spot and helped us), Pusplata, Twinkle and many other. Thank you all for making PyCon India a success for PyLadies. With all our hard work we actually broke the earlier notion about PyLadies in India.

See you Ladies in the next PyCon India 2017.

28 Sep 2016 11:06am GMT

What kind of applications can you build with python? You hear me featuring many people on this show that build websites, web services, or some data science driven application. Of course, all of those are wonderful but I know many of you have dreamed of building a game. <br/> <br/> This episode I'm interviewing Joseph Cherlin. He created the game Epikos and the entire tool chain entirely in Python. He has a great story about how he came across Python, why he decided to use it in his game, and advice he has for anyone out there taking on a large project like this. <br/> <br/> Links from the show: <br/> <div style="font-size: .85em;"> <br/> <b>Joseph on twitter</b>: <a href='https://twitter.com/JJC_Coder' target='_blank'>@JJC_Coder</a> <br/> <b>Epikos website</b>: <a href='http://www.epikosrpg.com' target='_blank'>epikosrpg.com</a> <br/> <b>Screenshots</b>: <a href='http://www.epikosrpg.com/screenshots/' target='_blank'>epikosrpg.com/screenshots</a> <br/> <b>Famous Reddit post</b>: <br/> <a href='https://www.reddit.com/r/Python/comments/4hazs4/how_i_built_an_entire_game_and_toolchain_100_in/?st=ithiamy1&sh=eed4703d' target='_blank'>reddit.com/r/Python/comments/4hazs4/how_i_built_an_entire_game_and_toolchain_100_in</a> <br/> <b>PyGame</b>: <a href=' http://www.pygame.org/hifi.html' target='_blank'>pygame.org</a> <br/> <b>Indie Overlook review</b>: <a href='http://indieoverlook.com/2016/04/15/epikos/' target='_blank'>indieoverlook.com/2016/04/15/epikos</a> <br/> <b>Indie Game: The Movie</b>: <a href='http://buy.indiegamethemovie.com/ ' target='_blank'>indiegamethemovie.com</a> <br/> </div>

28 Sep 2016 8:00am GMT

What kind of applications can you build with python? You hear me featuring many people on this show that build websites, web services, or some data science driven application. Of course, all of those are wonderful but I know many of you have dreamed of building a game. <br/> <br/> This episode I'm interviewing Joseph Cherlin. He created the game Epikos and the entire tool chain entirely in Python. He has a great story about how he came across Python, why he decided to use it in his game, and advice he has for anyone out there taking on a large project like this. <br/> <br/> Links from the show: <br/> <div style="font-size: .85em;"> <br/> <b>Joseph on twitter</b>: <a href='https://twitter.com/JJC_Coder' target='_blank'>@JJC_Coder</a> <br/> <b>Epikos website</b>: <a href='http://www.epikosrpg.com' target='_blank'>epikosrpg.com</a> <br/> <b>Screenshots</b>: <a href='http://www.epikosrpg.com/screenshots/' target='_blank'>epikosrpg.com/screenshots</a> <br/> <b>Famous Reddit post</b>: <br/> <a href='https://www.reddit.com/r/Python/comments/4hazs4/how_i_built_an_entire_game_and_toolchain_100_in/?st=ithiamy1&sh=eed4703d' target='_blank'>reddit.com/r/Python/comments/4hazs4/how_i_built_an_entire_game_and_toolchain_100_in</a> <br/> <b>PyGame</b>: <a href=' http://www.pygame.org/hifi.html' target='_blank'>pygame.org</a> <br/> <b>Indie Overlook review</b>: <a href='http://indieoverlook.com/2016/04/15/epikos/' target='_blank'>indieoverlook.com/2016/04/15/epikos</a> <br/> <b>Indie Game: The Movie</b>: <a href='http://buy.indiegamethemovie.com/ ' target='_blank'>indiegamethemovie.com</a> <br/> </div>

28 Sep 2016 8:00am GMT

Python is getting very popular as the first programming language in both home and aborad. I know many of the Bangladeshi universities have started using Python to introduce beginners to the wonderful world of programming. This also seems to be the case in the US. I have talked to a few friends from other countries and they agree to the fact that Python is quickly becoming the language people learn first. A quick google search could explain why Python is getting so popular among the learners.

Python in Programming Contests

Recently Python has been been included in ICPC, before that Python has usually had less visibility / presence in programming contests. And of course there are valid reasons behind that. The defacto implementation of Python - "CPython" is quite slow. It's a dynmaic language and that costs in terms of execution speed. C / C++ / Java is way faster than Python and programming contests are all about speed / performance. Python would allow you to solve problems in less lines of code but you may often hit the time limit. Despite the limitation, people have continiously chosen Python to learn programming and solve problems on numerous programming related websites. This might have convnced the authority to include Python in ICPC. But we do not yet know which flavor (read implementation) and version of Python will be available to the ICPC contestants. From different sources I gather that Python will be supported but the time limit issue remains - it is not guranteed that a problem can be solved within the time limit using Python. That makes me wonder, can Cython help in such cases?

Introduction to Cython

From the official website:

Cython is an optimising static compiler for both the Python programming language and the extended Cython programming language (based on Pyrex). It makes writing C extensions for Python as easy as Python itself.

With Cython, we can add type hints to our existing Python programs and compile them to make them run faster. But what is more awesome is the Cython language - it is a superset of Python and allows us to write Python like code which performs like C.

Don't trust my words, see for yourself in the Tutorial and Cython Language Basics.

Cython is Fast

When I say fast, I really mean - very very fast.

Image Source: http://ibm.co/20XSZ4F

The above image is taken from an article from IBM Developer Works which shows how Cython compares to C in terms of speed.

You can also check out these links for random benchmarks from different people:

• Cython beating C++
• Cython being 30% faster than the C++
• Another Benchmark

And finally, do try yourself and benchmark Cython against C++ and see how it performs!

Bonus article - Blazing fast Python networking :-)

Cython is easy to Setup

OK, so is it easy to make Cython available in the contest environments? Yes, it is! The only requirements of Cython is that you must have a C Compiler installed on your system along with Python. Any computer used for contest programming is supposed to have a C compiler installed anyway.

We just need one command to install Cython:

pip install Cython

PS: Many Scientific distributions of Python (ie. Anaconda) already ships Cython.

Cython in Programming Contests

Since we saw that Cython is super fast and easy to setup, programming contests can make Cython available along with CPython to allow the contestants make their programs faster and get along with Java / C++. It will make Python an attractive choice for serious problem solving.

I know the Cython language is not exactly Python. It is a superset of the Python language. So beginners might not be familiar with the language and that's alright. Beginners can start with Python and start solving the easier problems with Python. When they start competitive programming and start hitting the time limits, then Cython is one of the options they can choose to make their code run faster. Of course Cython needs some understanding of how C works - that's fine too because Cython still feels more productive than writing plain old C or C++.

Final words

PyPy is already quite popular in the Python community. Dropbox and Microsoft are also working on their Python JITs. I believe that someday Python JITs would be as fast as Java / C++. Today, Python is making programming fun for many beginners. I hope with Cython, we can worry less about the time limits and accept Python as a fitting tool in our competitive programming contests!

28 Sep 2016 2:00am GMT

Python is getting very popular as the first programming language in both home and aborad. I know many of the Bangladeshi universities have started using Python to introduce beginners to the wonderful world of programming. This also seems to be the case in the US. I have talked to a few friends from other countries and they agree to the fact that Python is quickly becoming the language people learn first. A quick google search could explain why Python is getting so popular among the learners.

Python in Programming Contests

Recently Python has been been included in ICPC, before that Python has usually had less visibility / presence in programming contests. And of course there are valid reasons behind that. The defacto implementation of Python - "CPython" is quite slow. It's a dynmaic language and that costs in terms of execution speed. C / C++ / Java is way faster than Python and programming contests are all about speed / performance. Python would allow you to solve problems in less lines of code but you may often hit the time limit. Despite the limitation, people have continiously chosen Python to learn programming and solve problems on numerous programming related websites. This might have convnced the authority to include Python in ICPC. But we do not yet know which flavor (read implementation) and version of Python will be available to the ICPC contestants. From different sources I gather that Python will be supported but the time limit issue remains - it is not guranteed that a problem can be solved within the time limit using Python. That makes me wonder, can Cython help in such cases?

Introduction to Cython

From the official website:

Cython is an optimising static compiler for both the Python programming language and the extended Cython programming language (based on Pyrex). It makes writing C extensions for Python as easy as Python itself.

With Cython, we can add type hints to our existing Python programs and compile them to make them run faster. But what is more awesome is the Cython language - it is a superset of Python and allows us to write Python like code which performs like C.

Don't trust my words, see for yourself in the Tutorial and Cython Language Basics.

Cython is Fast

When I say fast, I really mean - very very fast.

Image Source: http://ibm.co/20XSZ4F

The above image is taken from an article from IBM Developer Works which shows how Cython compares to C in terms of speed.

You can also check out these links for random benchmarks from different people:

• Cython beating C++
• Cython being 30% faster than the C++
• Another Benchmark

And finally, do try yourself and benchmark Cython against C++ and see how it performs!

Bonus article - Blazing fast Python networking :-)

Cython is easy to Setup

OK, so is it easy to make Cython available in the contest environments? Yes, it is! The only requirements of Cython is that you must have a C Compiler installed on your system along with Python. Any computer used for contest programming is supposed to have a C compiler installed anyway.

We just need one command to install Cython:

pip install Cython

PS: Many Scientific distributions of Python (ie. Anaconda) already ships Cython.

Cython in Programming Contests

Since we saw that Cython is super fast and easy to setup, programming contests can make Cython available along with CPython to allow the contestants make their programs faster and get along with Java / C++. It will make Python an attractive choice for serious problem solving.

I know the Cython language is not exactly Python. It is a superset of the Python language. So beginners might not be familiar with the language and that's alright. Beginners can start with Python and start solving the easier problems with Python. When they start competitive programming and start hitting the time limits, then Cython is one of the options they can choose to make their code run faster. Of course Cython needs some understanding of how C works - that's fine too because Cython still feels more productive than writing plain old C or C++.

Final words

PyPy is already quite popular in the Python community. Dropbox and Microsoft are also working on their Python JITs. I believe that someday Python JITs would be as fast as Java / C++. Today, Python is making programming fun for many beginners. I hope with Cython, we can worry less about the time limits and accept Python as a fitting tool in our competitive programming contests!

28 Sep 2016 2:00am GMT

From September 5th to the 9th a group of Python core developers gathered for a sprint hosted at Instagram and sponsored by Instagram, Microsoft, and the Python Software Foundation. The goal was to spend a week working towards the Python 3.6.0b1 release, just in time for the Python 3.6 feature freeze on Monday, September 12, 2016. The inspiration for this sprint was the Need for Speed sprint held in Iceland a decade ago, where many performance improvements were made to Python 2.5. How time flies!

That's the opening paragraph from the Python Insider blog post discussing the 2016 Python core sprint that recently took place. In the case of Microsoft's participation in the sprint, both Steve Dower and I (Brett Cannon) were invited to participate (which meant Microsoft had one of the largest company representations at the sprint). Between the two of us we spent the week completing work on four of our own PEPs for Python 3.6:

1. Adding a file system path protocol (PEP 519)
2. Adding a frame evaluation API to CPython (PEP 523)
3. Change Windows console encoding to UTF-8 (PEP 528)
4. Change Windows filesystem encoding to UTF-8 (PEP 529)

I also helped review the patches implementing PEP 515 and PEP 526 ("Underscores in Numeric Literals" and "Syntax for Variable Annotations", respectively). Both Steve and I also participated in many technical discussions on various topics and we cleared out our backlog of bug reports.

If you're curious as to what else has made it into Python 3.6 (so far), the rough draft of the "What's New" document for Python 3.6 is a good place to start (feature freeze has been reached, so no new features will be added to Python 3.6 but bugs are constantly being fixed). We also strongly encourage everyone to download Python 3.6.0b1 and try it with their code. If you find bugs, please file a bug report. There are also various features which will stay or be removed based on community feedback, so please do give this beta release a try!

Overall the week was very productive not only for the two of us but for everyone at the sprints and Python as a project. We hope that the success of this sprint will help lead to it becoming an annual event so that Python can benefit from such a huge burst of productivity every year. And if you or your company want to help with these sorts of sprints in the future or Python's development in general, then please consider helping with Python's development and/or sponsoring the Python Software Foundation.

27 Sep 2016 7:11pm GMT

From September 5th to the 9th a group of Python core developers gathered for a sprint hosted at Instagram and sponsored by Instagram, Microsoft, and the Python Software Foundation. The goal was to spend a week working towards the Python 3.6.0b1 release, just in time for the Python 3.6 feature freeze on Monday, September 12, 2016. The inspiration for this sprint was the Need for Speed sprint held in Iceland a decade ago, where many performance improvements were made to Python 2.5. How time flies!

That's the opening paragraph from the Python Insider blog post discussing the 2016 Python core sprint that recently took place. In the case of Microsoft's participation in the sprint, both Steve Dower and I (Brett Cannon) were invited to participate (which meant Microsoft had one of the largest company representations at the sprint). Between the two of us we spent the week completing work on four of our own PEPs for Python 3.6:

1. Adding a file system path protocol (PEP 519)
2. Adding a frame evaluation API to CPython (PEP 523)
3. Change Windows console encoding to UTF-8 (PEP 528)
4. Change Windows filesystem encoding to UTF-8 (PEP 529)

I also helped review the patches implementing PEP 515 and PEP 526 ("Underscores in Numeric Literals" and "Syntax for Variable Annotations", respectively). Both Steve and I also participated in many technical discussions on various topics and we cleared out our backlog of bug reports.

If you're curious as to what else has made it into Python 3.6 (so far), the rough draft of the "What's New" document for Python 3.6 is a good place to start (feature freeze has been reached, so no new features will be added to Python 3.6 but bugs are constantly being fixed). We also strongly encourage everyone to download Python 3.6.0b1 and try it with their code. If you find bugs, please file a bug report. There are also various features which will stay or be removed based on community feedback, so please do give this beta release a try!

Overall the week was very productive not only for the two of us but for everyone at the sprints and Python as a project. We hope that the success of this sprint will help lead to it becoming an annual event so that Python can benefit from such a huge burst of productivity every year. And if you or your company want to help with these sorts of sprints in the future or Python's development in general, then please consider helping with Python's development and/or sponsoring the Python Software Foundation.

27 Sep 2016 7:11pm GMT

Decorators are one of those features in Python that people like to talk about.

Why? Because they're different. Because they're a little weird. Because they're a little mind-bending.

Let's talk about decorators: how do you make them and when should you use them?

27 Sep 2016 6:30pm GMT

Decorators are one of those features in Python that people like to talk about.

Why? Because they're different. Because they're a little weird. Because they're a little mind-bending.

Let's talk about decorators: how do you make them and when should you use them?

27 Sep 2016 6:30pm GMT

27 Sep 2016 5:15pm GMT

27 Sep 2016 5:15pm GMT

27 Sep 2016 12:18pm GMT

27 Sep 2016 12:18pm GMT

From September 5th to the 9th a group of Python core developers gathered for a sprint hosted at Instagram and sponsored by Instagram, Microsoft, and the Python Software Foundation. The goal was to spend a week working towards the Python 3.6b1 release, just in time for the Python 3.6 feature freeze on Monday, September 12, 2016. The inspiration for this sprint was the Need for Speed sprint held in Iceland a decade ago, where many performance improvements were made to Python 2.5. How time flies!

By any measurement, the sprint was extremely successful. All participants left feeling accomplished both in the work they did and in the discussions they held. Being in the same room encouraged many discussions related to various PEPs, and many design decisions were made. There was also the camaraderie of working in the same room together - typically most of us only see each other at the annual PyCon US, where there are other distractions that prevent getting to simply spend time together. (This includes the Python development sprints at PyCon, where the focus is more on helping newcomers contribute to Python - that's why this sprint was not public.)

From a quantitative perspective, the sprint was the most productive week for Python ever! According to the graphs from the GitHub mirror of CPython, the week of September 4th saw more commits than the preceding 7 weeks combined! And in terms of issues, the number of open issues dropped by 62 with a total of 166 issues closed.

A large portion of the work performed during the sprint week revolved around various PEPs that had either not yet been accepted or were not yet fully implemented. In the end, 12 PEPs were either implemented from scratch or had their work completed during the week, out of Python 3.6's total of 16 PEPs:

1. Preserving the order of **kwargs in a function (PEP 468)
2. Add a private version to dict (PEP 509)
3. Underscores in Numeric Literals (PEP 515)
4. Adding a file system path protocol (PEP 519)
5. Preserving Class Attribute Definition Order (PEP 520)
6. Adding a frame evaluation API to CPython (PEP 523)
7. Make os.urandom() blocking on Linux, add os.getrandom() (PEP 524)
8. Asynchronous Generators (PEP 525)
9. Syntax for Variable Annotations (PEP 526)
10. Change Windows console encoding to UTF-8 (PEP 528)
11. Change Windows filesystem encoding to UTF-8 (PEP 529)
12. Asynchronous Comprehensions (PEP 530)

Some large projects were also worked on that are not represented as PEPs. For instance, Python 3.6 now contains support for DTrace and SystemTap. This will give people more tools to introspect and monitor Python. See the HOWTO for usage instructions and examples showing some of the new possibilities.

CPython also gained a more memory efficient dictionary implementation at the sprint. The new implementation shrinks memory usage of dictionaries by about 25% and also preserves insertion order, without speed penalties. Based on a proposal by Raymond Hettinger, the patch was written by INADA Naoki prior to the sprint but it was reviewed and heavily discussed at the sprint, as changing the underlying implementation of dictionaries can have profound implications on Python itself. In the end, the patch was accepted, directly allowing for PEP 468 to be accepted and simplifying PEP 520.

Work was also done on the Gilectomy (see the presentation on the topic from PyCon US for more background info on the project). Progress was made such that Python would run without any reference counting turned on (i.e. Python turned into a huge memory leak). Work then was started on trying the latest design on how to turn reference counting back on in a way that would allow Python to scale with the number of threads and CPU cores used. There's still a long road ahead before the Gilectomy will be ready to merge though, and we even jokingly considered branding the result as Python 4.

Much of the work done during the sprint led not only to improvements in the language and library, but to better performance as well. A quick performance comparison between Python 3.5.2+ and 3.6b1+ under OS X shows that 3.6 is generally faster, with double-digit speed improvements not uncommon. Similar benchmarking under Windows 10 has been reported to show similar performance gains.

A huge thanks goes out to the participants of the sprints! They are listed below in alphabetical order, along with thanks to the organizations that helped finance their attendance. Many of them traveled to attend and gave up the US Labor Day holiday with their families to participate. In the end, we had participants from 3 countries on 2 continents (We actually invited more people from other countries and continents, but not everybody invited could attend.)

• Brett Cannon (Microsoft)
• Ned Deily
• Steve Dower (Microsoft)
• Larry Hastings
• Raymond Hettinger
• Senthil Kumaran
• Łukasz Langa (Instagram)
• R. David Murray
• Benjamin Peterson
• Davin Potts
• Lisa Roach (Cisco)
• Guido van Rossum
• Yury Selivanov (magic.io)
• Gregory P. Smith (Google)
• Eric Snow (Canonical)
• Victor Stinner (Red Hat)
• Zachary Ware

Special thanks to Łukasz for making the event happen and to Larry for designing the logo.

27 Sep 2016 11:03am GMT

From September 5th to the 9th a group of Python core developers gathered for a sprint hosted at Instagram and sponsored by Instagram, Microsoft, and the Python Software Foundation. The goal was to spend a week working towards the Python 3.6b1 release, just in time for the Python 3.6 feature freeze on Monday, September 12, 2016. The inspiration for this sprint was the Need for Speed sprint held in Iceland a decade ago, where many performance improvements were made to Python 2.5. How time flies!

By any measurement, the sprint was extremely successful. All participants left feeling accomplished both in the work they did and in the discussions they held. Being in the same room encouraged many discussions related to various PEPs, and many design decisions were made. There was also the camaraderie of working in the same room together - typically most of us only see each other at the annual PyCon US, where there are other distractions that prevent getting to simply spend time together. (This includes the Python development sprints at PyCon, where the focus is more on helping newcomers contribute to Python - that's why this sprint was not public.)

From a quantitative perspective, the sprint was the most productive week for Python ever! According to the graphs from the GitHub mirror of CPython, the week of September 4th saw more commits than the preceding 7 weeks combined! And in terms of issues, the number of open issues dropped by 62 with a total of 166 issues closed.

A large portion of the work performed during the sprint week revolved around various PEPs that had either not yet been accepted or were not yet fully implemented. In the end, 12 PEPs were either implemented from scratch or had their work completed during the week, out of Python 3.6's total of 16 PEPs:

1. Preserving the order of **kwargs in a function (PEP 468)
2. Add a private version to dict (PEP 509)
3. Underscores in Numeric Literals (PEP 515)
4. Adding a file system path protocol (PEP 519)
5. Preserving Class Attribute Definition Order (PEP 520)
6. Adding a frame evaluation API to CPython (PEP 523)
7. Make os.urandom() blocking on Linux, add os.getrandom() (PEP 524)
8. Asynchronous Generators (PEP 525)
9. Syntax for Variable Annotations (PEP 526)
10. Change Windows console encoding to UTF-8 (PEP 528)
11. Change Windows filesystem encoding to UTF-8 (PEP 529)
12. Asynchronous Comprehensions (PEP 530)

Some large projects were also worked on that are not represented as PEPs. For instance, Python 3.6 now contains support for DTrace and SystemTap. This will give people more tools to introspect and monitor Python. See the HOWTO for usage instructions and examples showing some of the new possibilities.

CPython also gained a more memory efficient dictionary implementation at the sprint. The new implementation shrinks memory usage of dictionaries by about 25% and also preserves insertion order, without speed penalties. Based on a proposal by Raymond Hettinger, the patch was written by INADA Naoki prior to the sprint but it was reviewed and heavily discussed at the sprint, as changing the underlying implementation of dictionaries can have profound implications on Python itself. In the end, the patch was accepted, directly allowing for PEP 468 to be accepted and simplifying PEP 520.

Work was also done on the Gilectomy (see the presentation on the topic from PyCon US for more background info on the project). Progress was made such that Python would run without any reference counting turned on (i.e. Python turned into a huge memory leak). Work then was started on trying the latest design on how to turn reference counting back on in a way that would allow Python to scale with the number of threads and CPU cores used. There's still a long road ahead before the Gilectomy will be ready to merge though, and we even jokingly considered branding the result as Python 4.

Much of the work done during the sprint led not only to improvements in the language and library, but to better performance as well. A quick performance comparison between Python 3.5.2+ and 3.6b1+ under OS X shows that 3.6 is generally faster, with double-digit speed improvements not uncommon. Similar benchmarking under Windows 10 has been reported to show similar performance gains.

A huge thanks goes out to the participants of the sprints! They are listed below in alphabetical order, along with thanks to the organizations that helped finance their attendance. Many of them traveled to attend and gave up the US Labor Day holiday with their families to participate. In the end, we had participants from 3 countries on 2 continents (We actually invited more people from other countries and continents, but not everybody invited could attend.)

• Brett Cannon (Microsoft)
• Ned Deily
• Steve Dower (Microsoft)
• Larry Hastings
• Raymond Hettinger
• Senthil Kumaran
• Łukasz Langa (Instagram)
• R. David Murray
• Benjamin Peterson
• Davin Potts
• Lisa Roach (Cisco)
• Guido van Rossum
• Yury Selivanov (magic.io)
• Gregory P. Smith (Google)
• Eric Snow (Canonical)
• Victor Stinner (Red Hat)
• Zachary Ware

Special thanks to Łukasz for making the event happen and to Larry for designing the logo.

27 Sep 2016 11:03am GMT

In the first post I introduced you to Python mocks, objects that can imitate other objects and work as placeholders, replacing external systems during unit testing. I described the basic behaviour of mock objects, the return_value and side_effect attributes, and the assert_called_with() method.

In this post I will briefly review the remaining assert_* methods and some interesting attributes that allow to check the calls received by the mock object. Then I will introduce and exemplify patching, which is a very important topic in testing.

Other assertions and attributes

The official documentation of the mock library lists many other assertion, namely assert_called_once_with(), assert_any_call(), assert_has_calls(), assert_not_called(). If you grasped how assert_called_with() works, you will have no troubles in understanding how those other behave. Be sure to check the documentation to get a full description of what mock object can assert about their history after being used by your code.

Together with those methods, mock objects also provide some useful attributes, two of which have been already reviewed in the first post. The remaining attributes are as expected mostly related to calls, and are called, call_count, call_args, call_args_list, method_calls, mock_calls. While these also are very well descripted in the official documentation, I want to point out the two method_calls and mock_calls attributes, that store the detailed list of methods which are called on the mock, and the call_args_list attribute that lists the parameters of every call.

Do not forget that methods called on a mock object are mocks themselves, so you may first access the main mock object to get information about the called methods, and then access those methods to get the arguments they received.

Patching

Mocks are very simple to introduce in your tests whenever your objects accept classes or instances from outside. In that case, as described, you just have to instantiate the Mock class and pass the resulting object to your system. However, when the external classes instantiated by your library are hardcoded this simple trick does not work. In this case you have no chance to pass a mock object instead of the real one.

This is exactly the case addressed by patching. Patching, in a testing framework, means to replace a globally reachable object with a mock, thus achieving the target of having the code run unmodified, while part of it has been hot swapped, that is, replaced at run time.

A warm-up example

Let us start with a very simple example. Patching can be complex to grasp at the beginning so it is better to learn it with trivial code. If you do not have it yet, create the testing environment mockplayground with the instruction given in the previous post.

I want to develop a simple class that returns information about a given file. The class shall be instantiated with the filename, which can be a relative path.

For the sake of brevity I will not show you every step of the TDD development of the class. Remember that TDD requires you to write a test and then implement the code, but sometimes this could be too fine grained, so do not use the TDD rules without thinking.

The tests for the initialization of the class are

from fileinfo import FileInfo

def test_init():
    filename = 'somefile.ext'
    fi = FileInfo(filename)
    assert fi.filename == filename

def test_init():
    filename = 'somefile.ext'
    relative_path = '../{}'.format(filename)
    fi = FileInfo(relative_path)
    assert fi.filename == filename

You can put them into the tests/test_fileinfo.py file. The code that makes the tests pass could be something like

import os


class FileInfo:
    def __init__(self, path):
        self.original_path = path
        self.filename = os.path.basename(path)

Up to now I didn't introduce any new feature. Now I want the get_info() function to return a tuple with the file name, the original path the class was instantiated with, and the absolute path of the file.

You immediately realise that you have an issue in writing the test. There is no way to easily test something as "the absolute path", since the outcome of the function called in the test is supposed to vary with the path of the test itself. Let us write part of the test

def test_get_info():
    filename = 'somefile.ext'
    original_path = '../{}'.format(filename)
    fi = FileInfo(original_path)
    assert fi.get_info() == (filename, original_path, '???')

where the '???' string highlights that I cannot put something sensible to test the absolute path of the file.

Patching is the way to solve this problem. You know that the function will use some code to get the absolute path of the file. So in the scope of the test only you can replace that code with different code and perform the test. Since the replacement code has a known outcome writing the test is now possible.

Patching, thus, means to inform Python that in some scope you want a globally accessible module/object replaced by a mock. Let's see how we can use it in our example

from unittest.mock import patch

[...]

def test_get_info():
    filename = 'somefile.ext'
    original_path = '../{}'.format(filename)

    with patch('os.path.abspath') as abspath_mock:
        test_abspath = 'some/abs/path'
        abspath_mock.return_value = test_abspath
        fi = FileInfo(original_path)
        assert fi.get_info() == (filename, original_path, test_abspath)

Remember that if you are using Python 2 you installed the mock module with pip, so your import statement becomes form mock import patch.

You clearly see the context in which the patching happens, as it is enclosed in a with statement. Inside this statement the module os.path.abspath will be replaced by a mock created by the function patch and called abspath_mock. We can now give the function a return_value as we did with standard mocks in the first post and run the test.

The code that make the test pass is

class FileInfo:
    [...]

    def get_info(self):
        return self.filename, self.original_path, os.path.abspath(self.filename)

Obviously to write the test you have to know that you are going to use the os.path.abspath function, so patching is somehow a "less pure" practice in TDD. In pure OOP/TDD you are only concerned with the external behaviour of the object, and not with its internal structure. This example, however, shows that you have to cope with some real world issues, and patching is a clean way to do it.

The patching decorator

The patch function we imported from the unittest.mock module is very powerful, and can be used as a function decorator as well. When used in this fashion you need to change the decorated function to accept a mock as last argument.

@patch('os.path.abspath')
def test_get_info(abspath_mock):
    filename = 'somefile.ext'
    original_path = '../{}'.format(filename)

    test_abspath = 'some/abs/path'
    abspath_mock.return_value = test_abspath
    fi = FileInfo(original_path)
    assert fi.get_info() == (filename, original_path, test_abspath)

As you can see the patch decorator works like a big with statement for the whole function. Obviously in this way you replace the target function os.path.abspath in the scope of the whole function. It is then up to you to decide if you need to use patch as a decorator or in a with block.

Multiple patches

We can also patch more that one object. Say for example that we want to change the above test to check that the outcome of the FileInfo.get_info() method also contains the size of the file. To get the size of a file in Python we may use the os.path.getsize() function, which returns the size of the file in bytes.

So now we have to patch os.path.getsize as well, and this can be done with another patch decorator.

@patch('os.path.getsize')
@patch('os.path.abspath')
def test_get_info(abspath_mock, getsize_mock):
    filename = 'somefile.ext'
    original_path = '../{}'.format(filename)

    test_abspath = 'some/abs/path'
    abspath_mock.return_value = test_abspath

    test_size = 1234
    getsize_mock.return_value = test_size

    fi = FileInfo(original_path)
    assert fi.get_info() == (filename, original_path, test_abspath, test_size)

Please notice that the decorator which is nearest to the function is applied first. Always remember that the decorator syntax with @ is a shortcut to replace the function with the output of the decorator, so two decorators result in

@decorator1
@decorator2
def myfunction():
    pass

which is a shorcut for

def myfunction():
    pass
myfunction = decorator1(decorator2(myfunction))

This explains why, in the test code, the function receives first abspath_mock and then getsize_mock. The first decorator applied to the function is the patch of os.path.abspath, which appends the mock that we call abspath_mock. Then the patch of os.path.getsize is applied and this appends its own mock.

The code that makes the test pass is

class FileInfo:
    [...]

    def get_info(self):
        return self.filename, self.original_path, os.path.abspath(self.filename), os.path.getsize(self.filename)

We can write the above test using two with statements as well

def test_get_info():
    filename = 'somefile.ext'
    original_path = '../{}'.format(filename)

    with patch('os.path.abspath') as abspath_mock:
        test_abspath = 'some/abs/path'
        abspath_mock.return_value = test_abspath

        with patch('os.path.getsize') as getsize_mock:
            test_size = 1234
            getsize_mock.return_value = test_size

            fi = FileInfo(original_path)
            assert fi.get_info() == (filename, original_path, test_abspath, test_size)

Using more than one with statement, however, makes the code difficult to read, in my opinion, so in general I prefer to avoid complex with trees if I do not need a limited scope of the patching.

Patching immutable objects

The most widespread version of Python is CPython, which is written, as the name suggests, in C. Part of the standard library is also written in C, while the rest is written in Python itself.

The objects (classes, modules, functions, etc) that are implemented in C are shared between interpreters, which is something that you can do embedding the Python interpreter in a C program, for example. This requires those objects to be immutable, so that you cannot alter them at runtime from a single interpreter.

For an example of this immutability just check the following code

>>> a = 1
>>> a.conjugate = 5
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'int' object attribute 'conjugate' is read-only

Here I'm trying to replace a method with an integer, which is pointless, but nevertheless shows the issue we are facing.

What has this immutability to do with patching? What patch does is actually to temporarily replace an attibute of an object (method of a class, class of a module, etc), so if that object is immutable the patching action fails.

A typical example of this problem is the datetime module, which is also one of the best candidates for patching, since the output of time functions is by definition time-varying.

Let me show the problem with a simple class that logs operations. The class is the following (you can put it into a file called logger.py)

import datetime

class Logger:
    def __init__(self):
        self.messages = []

    def log(self, message):
        self.messages.append((datetime.datetime.now(), message))

This is pretty simple, but testing this code is problematic, because the log() method produces results that depend on the actual execution time.

If we try to write a test patching datetime.datetime.now we have a bitter surprise. This is the test code, that you can put in tests/test_logger.py

from unittest.mock import patch

from logger import Logger

def test_init():
    l = Logger()
    assert l.messages == []

@patch('datetime.datetime.now')
def test_log(mock_now):
    test_now = 123
    test_message = "A test message"
    mock_now.return_value = test_now

    l = Logger()
    l.log(test_message)
    assert l.messages == [(test_now, test_message)]

and the execution of pytest returns a TypeError: can't set attributes of built-in/extension type 'datetime.datetime', which is exactly a problem of immutability.

There are several ways to address this problem, but all of them leverage the fact that, when you import of subclass an immutable object what you get is a "copy" of that is now mutable.

The easiest example in this case is the module datetime itself. In the test_log function we try to patch directly the datetime.datetime.now object, affecting the builtin module datetime. The file logger.py, however, does import datetime, so that this latter becomes a local symbol in the logger module. This is exactly the key for our patching. Let us change the code to

@patch('logger.datetime.datetime')
def test_log(mock_datetime):
    test_now = 123
    test_message = "A test message"
    mock_datetime.now.return_value = test_now

    l = Logger()
    l.log(test_message)
    assert l.messages == [(test_now, test_message)]

As you see running the test now the patching works. What we did was to patch logger.datetime.datetime instead of datetime.datetime.now. Two things changed, thus, in our test. First, we are patching the module imported in the logger.py file and not the module provided globally by the Python interpreter. Second, we have to patch the whole module because this is what is imported by the logger.py file. If you try to patch logger.datetime.datetime.now you will find that it is still immutable.

Another possible solution to this problem is to create a function that invokes the immutable object and returns its value. This last function can be easily patched, because it just uses the builtin objects and thus is not immutable. This solution, however, requires to change the source code to allow testing, which is far from being desirable. Obviously it is better to introduce a small change in the code and have it tested than to leave it untested, but whenever is possible I avoid solutions that introduce code which wouldn't be required without tests.

Final words

In this second part of this small series on Python testing we reviewd the patching mechanism and run through some of its subleties. Patching is a really effective technique, and patch-based tests can be found in many different packages. Take your time to become confident with mocks and patching, since they will be one of your main tools while working with Python and any other object-oriented language.

As always, I strongly recommend finding some time to read the official documentation of the mock library.

Feedback

Feel free to use the blog Google+ page to comment the post. The GitHub issues page is the best place to submit corrections

27 Sep 2016 9:00am GMT

In the first post I introduced you to Python mocks, objects that can imitate other objects and work as placeholders, replacing external systems during unit testing. I described the basic behaviour of mock objects, the return_value and side_effect attributes, and the assert_called_with() method.

In this post I will briefly review the remaining assert_* methods and some interesting attributes that allow to check the calls received by the mock object. Then I will introduce and exemplify patching, which is a very important topic in testing.

Other assertions and attributes

The official documentation of the mock library lists many other assertion, namely assert_called_once_with(), assert_any_call(), assert_has_calls(), assert_not_called(). If you grasped how assert_called_with() works, you will have no troubles in understanding how those other behave. Be sure to check the documentation to get a full description of what mock object can assert about their history after being used by your code.

Together with those methods, mock objects also provide some useful attributes, two of which have been already reviewed in the first post. The remaining attributes are as expected mostly related to calls, and are called, call_count, call_args, call_args_list, method_calls, mock_calls. While these also are very well descripted in the official documentation, I want to point out the two method_calls and mock_calls attributes, that store the detailed list of methods which are called on the mock, and the call_args_list attribute that lists the parameters of every call.

Do not forget that methods called on a mock object are mocks themselves, so you may first access the main mock object to get information about the called methods, and then access those methods to get the arguments they received.

Patching

Mocks are very simple to introduce in your tests whenever your objects accept classes or instances from outside. In that case, as described, you just have to instantiate the Mock class and pass the resulting object to your system. However, when the external classes instantiated by your library are hardcoded this simple trick does not work. In this case you have no chance to pass a mock object instead of the real one.

This is exactly the case addressed by patching. Patching, in a testing framework, means to replace a globally reachable object with a mock, thus achieving the target of having the code run unmodified, while part of it has been hot swapped, that is, replaced at run time.

A warm-up example

Let us start with a very simple example. Patching can be complex to grasp at the beginning so it is better to learn it with trivial code. If you do not have it yet, create the testing environment mockplayground with the instruction given in the previous post.

I want to develop a simple class that returns information about a given file. The class shall be instantiated with the filename, which can be a relative path.

For the sake of brevity I will not show you every step of the TDD development of the class. Remember that TDD requires you to write a test and then implement the code, but sometimes this could be too fine grained, so do not use the TDD rules without thinking.

The tests for the initialization of the class are

from fileinfo import FileInfo

def test_init():
    filename = 'somefile.ext'
    fi = FileInfo(filename)
    assert fi.filename == filename

def test_init():
    filename = 'somefile.ext'
    relative_path = '../{}'.format(filename)
    fi = FileInfo(relative_path)
    assert fi.filename == filename

You can put them into the tests/test_fileinfo.py file. The code that makes the tests pass could be something like

import os


class FileInfo:
    def __init__(self, path):
        self.original_path = path
        self.filename = os.path.basename(path)

Up to now I didn't introduce any new feature. Now I want the get_info() function to return a tuple with the file name, the original path the class was instantiated with, and the absolute path of the file.

You immediately realise that you have an issue in writing the test. There is no way to easily test something as "the absolute path", since the outcome of the function called in the test is supposed to vary with the path of the test itself. Let us write part of the test

def test_get_info():
    filename = 'somefile.ext'
    original_path = '../{}'.format(filename)
    fi = FileInfo(original_path)
    assert fi.get_info() == (filename, original_path, '???')

where the '???' string highlights that I cannot put something sensible to test the absolute path of the file.

Patching is the way to solve this problem. You know that the function will use some code to get the absolute path of the file. So in the scope of the test only you can replace that code with different code and perform the test. Since the replacement code has a known outcome writing the test is now possible.

Patching, thus, means to inform Python that in some scope you want a globally accessible module/object replaced by a mock. Let's see how we can use it in our example

from unittest.mock import patch

[...]

def test_get_info():
    filename = 'somefile.ext'
    original_path = '../{}'.format(filename)

    with patch('os.path.abspath') as abspath_mock:
        test_abspath = 'some/abs/path'
        abspath_mock.return_value = test_abspath
        fi = FileInfo(original_path)
        assert fi.get_info() == (filename, original_path, test_abspath)

Remember that if you are using Python 2 you installed the mock module with pip, so your import statement becomes form mock import patch.

You clearly see the context in which the patching happens, as it is enclosed in a with statement. Inside this statement the module os.path.abspath will be replaced by a mock created by the function patch and called abspath_mock. We can now give the function a return_value as we did with standard mocks in the first post and run the test.

The code that make the test pass is

class FileInfo:
    [...]

    def get_info(self):
        return self.filename, self.original_path, os.path.abspath(self.filename)

Obviously to write the test you have to know that you are going to use the os.path.abspath function, so patching is somehow a "less pure" practice in TDD. In pure OOP/TDD you are only concerned with the external behaviour of the object, and not with its internal structure. This example, however, shows that you have to cope with some real world issues, and patching is a clean way to do it.

The patching decorator

The patch function we imported from the unittest.mock module is very powerful, and can be used as a function decorator as well. When used in this fashion you need to change the decorated function to accept a mock as last argument.

@patch('os.path.abspath')
def test_get_info(abspath_mock):
    filename = 'somefile.ext'
    original_path = '../{}'.format(filename)

    test_abspath = 'some/abs/path'
    abspath_mock.return_value = test_abspath
    fi = FileInfo(original_path)
    assert fi.get_info() == (filename, original_path, test_abspath)

As you can see the patch decorator works like a big with statement for the whole function. Obviously in this way you replace the target function os.path.abspath in the scope of the whole function. It is then up to you to decide if you need to use patch as a decorator or in a with block.

Multiple patches

We can also patch more that one object. Say for example that we want to change the above test to check that the outcome of the FileInfo.get_info() method also contains the size of the file. To get the size of a file in Python we may use the os.path.getsize() function, which returns the size of the file in bytes.

So now we have to patch os.path.getsize as well, and this can be done with another patch decorator.

@patch('os.path.getsize')
@patch('os.path.abspath')
def test_get_info(abspath_mock, getsize_mock):
    filename = 'somefile.ext'
    original_path = '../{}'.format(filename)

    test_abspath = 'some/abs/path'
    abspath_mock.return_value = test_abspath

    test_size = 1234
    getsize_mock.return_value = test_size

    fi = FileInfo(original_path)
    assert fi.get_info() == (filename, original_path, test_abspath, test_size)

Please notice that the decorator which is nearest to the function is applied first. Always remember that the decorator syntax with @ is a shortcut to replace the function with the output of the decorator, so two decorators result in

@decorator1
@decorator2
def myfunction():
    pass

which is a shorcut for

def myfunction():
    pass
myfunction = decorator1(decorator2(myfunction))

This explains why, in the test code, the function receives first abspath_mock and then getsize_mock. The first decorator applied to the function is the patch of os.path.abspath, which appends the mock that we call abspath_mock. Then the patch of os.path.getsize is applied and this appends its own mock.

The code that makes the test pass is

class FileInfo:
    [...]

    def get_info(self):
        return self.filename, self.original_path, os.path.abspath(self.filename), os.path.getsize(self.filename)

We can write the above test using two with statements as well

def test_get_info():
    filename = 'somefile.ext'
    original_path = '../{}'.format(filename)

    with patch('os.path.abspath') as abspath_mock:
        test_abspath = 'some/abs/path'
        abspath_mock.return_value = test_abspath

        with patch('os.path.getsize') as getsize_mock:
            test_size = 1234
            getsize_mock.return_value = test_size

            fi = FileInfo(original_path)
            assert fi.get_info() == (filename, original_path, test_abspath, test_size)

Using more than one with statement, however, makes the code difficult to read, in my opinion, so in general I prefer to avoid complex with trees if I do not need a limited scope of the patching.

Patching immutable objects

The most widespread version of Python is CPython, which is written, as the name suggests, in C. Part of the standard library is also written in C, while the rest is written in Python itself.

The objects (classes, modules, functions, etc) that are implemented in C are shared between interpreters, which is something that you can do embedding the Python interpreter in a C program, for example. This requires those objects to be immutable, so that you cannot alter them at runtime from a single interpreter.

For an example of this immutability just check the following code

>>> a = 1
>>> a.conjugate = 5
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'int' object attribute 'conjugate' is read-only

Here I'm trying to replace a method with an integer, which is pointless, but nevertheless shows the issue we are facing.

What has this immutability to do with patching? What patch does is actually to temporarily replace an attibute of an object (method of a class, class of a module, etc), so if that object is immutable the patching action fails.

A typical example of this problem is the datetime module, which is also one of the best candidates for patching, since the output of time functions is by definition time-varying.

Let me show the problem with a simple class that logs operations. The class is the following (you can put it into a file called logger.py)

import datetime

class Logger:
    def __init__(self):
        self.messages = []

    def log(self, message):
        self.messages.append((datetime.datetime.now(), message))

This is pretty simple, but testing this code is problematic, because the log() method produces results that depend on the actual execution time.

If we try to write a test patching datetime.datetime.now we have a bitter surprise. This is the test code, that you can put in tests/test_logger.py

from unittest.mock import patch

from logger import Logger

def test_init():
    l = Logger()
    assert l.messages == []

@patch('datetime.datetime.now')
def test_log(mock_now):
    test_now = 123
    test_message = "A test message"
    mock_now.return_value = test_now

    l = Logger()
    l.log(test_message)
    assert l.messages == [(test_now, test_message)]

and the execution of pytest returns a TypeError: can't set attributes of built-in/extension type 'datetime.datetime', which is exactly a problem of immutability.

There are several ways to address this problem, but all of them leverage the fact that, when you import of subclass an immutable object what you get is a "copy" of that is now mutable.

The easiest example in this case is the module datetime itself. In the test_log function we try to patch directly the datetime.datetime.now object, affecting the builtin module datetime. The file logger.py, however, does import datetime, so that this latter becomes a local symbol in the logger module. This is exactly the key for our patching. Let us change the code to

@patch('logger.datetime.datetime')
def test_log(mock_datetime):
    test_now = 123
    test_message = "A test message"
    mock_datetime.now.return_value = test_now

    l = Logger()
    l.log(test_message)
    assert l.messages == [(test_now, test_message)]

As you see running the test now the patching works. What we did was to patch logger.datetime.datetime instead of datetime.datetime.now. Two things changed, thus, in our test. First, we are patching the module imported in the logger.py file and not the module provided globally by the Python interpreter. Second, we have to patch the whole module because this is what is imported by the logger.py file. If you try to patch logger.datetime.datetime.now you will find that it is still immutable.

Another possible solution to this problem is to create a function that invokes the immutable object and returns its value. This last function can be easily patched, because it just uses the builtin objects and thus is not immutable. This solution, however, requires to change the source code to allow testing, which is far from being desirable. Obviously it is better to introduce a small change in the code and have it tested than to leave it untested, but whenever is possible I avoid solutions that introduce code which wouldn't be required without tests.

Final words

In this second part of this small series on Python testing we reviewd the patching mechanism and run through some of its subleties. Patching is a really effective technique, and patch-based tests can be found in many different packages. Take your time to become confident with mocks and patching, since they will be one of your main tools while working with Python and any other object-oriented language.

As always, I strongly recommend finding some time to read the official documentation of the mock library.

Feedback

Feel free to use the blog Google+ page to comment the post. The GitHub issues page is the best place to submit corrections

27 Sep 2016 9:00am GMT

This article is brought with ❤ to you by Semaphore.

Post originally published on http://krzysztofzuraw.com/. Republished with author's permission.

Introduction

In this post I will look into the essential part of testing - mocks.

First of all, what I want to accomplish here is to give you basic examples of how to mock data using two tools - mock and pytest monkeypatch.

Why bother mocking?

Some of the parts of our application may have dependencies for other libraries or objects. To isolate the behaviour of our parts, we need to substitute external dependencies. Here comes the mocking. We mock an external API to check certain behaviours, such as proper return values, that we previously defined.

Mocking function

Let's say we have a module called function.py:

def square(value):
    return value ** 2

def cube(value): 
    return value ** 3

def main(value): 
    return square(value) + cube(value)

Then let's see how these functions are mocked using the mock library:

    try:
        import mock
    except ImportError:
        from unittest import mock

    import unittest

    from function import square, main


    class TestNotMockedFunction(unittest.TestCase):

        @mock.patch('__main__.square', return_value=1)
        def test_function(self, mocked_square):
            # because you need to patch in exact place where function that has to be mocked is called
            self.assertEquals(square(5), 1)

        @mock.patch('function.square')
        @mock.patch('function.cube')
        def test_main_function(self, mocked_square, mocked_cube):
            # underling function are mocks so calling main(5) will return mock
            mocked_square.return_value = 1
            mocked_cube.return_value = 0
            self.assertEquals(main(5), 1)
            mocked_square.assert_called_once_with(5)
            mocked_cube.assert_called_once_with(5)


    if __name__ == '__main__':
        unittest.main()

What is happening here? Lines 1-4 are for making this code compatible between Python 2 and 3. In Python 3, mock is part of the standard library, whereas in Python 2 you need to install it by pip install mock.

In line 13, I patched the square function. You have to remember to patch it in the same place you use it. For instance, I'm calling square(5) in the test itself so I need to patch it in __main__. This is the case if I'm running this by using python tests/test_function.py. If I'm using pytest for that, I need to patch it as test_function.square.

In lines 18-19, I patch the square and cube functions in their module because they are used in the main function. The last two asserts come from the mock library, and are there to make sure that mock was called with proper values.

The same can be accomplished using mokeypatching for py.test:

from function import square, main

def test_function(monkeypatch):
    monkeypatch.setattr("test_function_pytest.square", lambda x: 1)
    assert square(5) == 1

def test_main_function(monkeypatch): 
    monkeypatch.setattr('function.square', lambda x: 1) 
    monkeypatch.setattr('function.cube', lambda x: 0) 
    assert main(5) == 1

As you can see, I'm using monkeypatch.setattr for setting up a return value for given functions. I still need to monkeypatch it in proper places - test_function_pytest and function.

Mocking classes

I have a module called square:

import math

class Square(object): 
    def __init__(radius): 
        self.radius = radius

        def calculate_area(self):
            return math.sqrt(self.radius) * math.pi 

and mocks using standard lib:

try: 
    import mock 
except ImportError: 
    from unittest import mock

import unittest

from square import Square

class TestClass(unittest.TestCase):

       @mock.patch('__main__.Square') # depends in witch from is run
       def test_mocking_instance(self, mocked_instance):
           mocked_instance = mocked_instance.return_value
           mocked_instance.calculate_area.return_value = 1
           sq = Square(100)
           self.assertEquals(sq.calculate_area(), 1)


       def test_mocking_classes(self):
           sq = Square
           sq.calculate_area = mock.MagicMock(return_value=1)
           self.assertEquals(sq.calculate_area(), 1)

       @mock.patch.object(Square, 'calculate_area')
       def test_mocking_class_methods(self, mocked_method):
           mocked_method.return_value = 20
           self.assertEquals(Square.calculate_area(), 20)

if __name__ == '__main__':
    unittest.main()

At line 13, I patch the class Square. Lines 15 and 16 present a mocking instance. mocked_instance is a mock object which returns another mock by default, and to these mock.calculate_area I add return_value 1. In line 23, I'm using MagicMock, which is a normal mock class, except in that it also retrieves magic methods from the given object. Lastly, I use patch.object to mock the method in the Square class.

The same using pytest:

try: 
    from mock import MagicMock 
except ImportError: 
    from unittest.mock import MagicMock

from square import Square

def test_mocking_class_methods(monkeypatch):
    monkeypatch.setattr('test_class_pytest.Square.calculate_area', lambda: 1)
    assert Square.calculate_area() ==  1


def test_mocking_classes(monkeypatch):
    monkeypatch.setattr('test_class_pytest.Square', MagicMock(Square))
    sq = Square
    sq.calculate_area.return_value = 1
    assert sq.calculate_area() ==  1

The issue here is with test_mocking_class_methods, which works well in Python 3, but not in Python 2.

All examples can be found in this repo.

If you have any questions and comments, feel free to leave them in the section below.

References:

1. What is Mocking?
2. Mocking With Kung Fu Panda

This article is brought with ❤ to you by Semaphore.

27 Sep 2016 7:31am GMT

This article is brought with ❤ to you by Semaphore.

Post originally published on http://krzysztofzuraw.com/. Republished with author's permission.

Introduction

In this post I will look into the essential part of testing - mocks.

First of all, what I want to accomplish here is to give you basic examples of how to mock data using two tools - mock and pytest monkeypatch.

Why bother mocking?

Some of the parts of our application may have dependencies for other libraries or objects. To isolate the behaviour of our parts, we need to substitute external dependencies. Here comes the mocking. We mock an external API to check certain behaviours, such as proper return values, that we previously defined.

Mocking function

Let's say we have a module called function.py:

def square(value):
    return value ** 2

def cube(value): 
    return value ** 3

def main(value): 
    return square(value) + cube(value)

Then let's see how these functions are mocked using the mock library:

    try:
        import mock
    except ImportError:
        from unittest import mock

    import unittest

    from function import square, main


    class TestNotMockedFunction(unittest.TestCase):

        @mock.patch('__main__.square', return_value=1)
        def test_function(self, mocked_square):
            # because you need to patch in exact place where function that has to be mocked is called
            self.assertEquals(square(5), 1)

        @mock.patch('function.square')
        @mock.patch('function.cube')
        def test_main_function(self, mocked_square, mocked_cube):
            # underling function are mocks so calling main(5) will return mock
            mocked_square.return_value = 1
            mocked_cube.return_value = 0
            self.assertEquals(main(5), 1)
            mocked_square.assert_called_once_with(5)
            mocked_cube.assert_called_once_with(5)


    if __name__ == '__main__':
        unittest.main()

What is happening here? Lines 1-4 are for making this code compatible between Python 2 and 3. In Python 3, mock is part of the standard library, whereas in Python 2 you need to install it by pip install mock.

In line 13, I patched the square function. You have to remember to patch it in the same place you use it. For instance, I'm calling square(5) in the test itself so I need to patch it in __main__. This is the case if I'm running this by using python tests/test_function.py. If I'm using pytest for that, I need to patch it as test_function.square.

In lines 18-19, I patch the square and cube functions in their module because they are used in the main function. The last two asserts come from the mock library, and are there to make sure that mock was called with proper values.

The same can be accomplished using mokeypatching for py.test:

from function import square, main

def test_function(monkeypatch):
    monkeypatch.setattr("test_function_pytest.square", lambda x: 1)
    assert square(5) == 1

def test_main_function(monkeypatch): 
    monkeypatch.setattr('function.square', lambda x: 1) 
    monkeypatch.setattr('function.cube', lambda x: 0) 
    assert main(5) == 1

As you can see, I'm using monkeypatch.setattr for setting up a return value for given functions. I still need to monkeypatch it in proper places - test_function_pytest and function.

Mocking classes

I have a module called square:

import math

class Square(object): 
    def __init__(radius): 
        self.radius = radius

        def calculate_area(self):
            return math.sqrt(self.radius) * math.pi 

and mocks using standard lib:

try: 
    import mock 
except ImportError: 
    from unittest import mock

import unittest

from square import Square

class TestClass(unittest.TestCase):

       @mock.patch('__main__.Square') # depends in witch from is run
       def test_mocking_instance(self, mocked_instance):
           mocked_instance = mocked_instance.return_value
           mocked_instance.calculate_area.return_value = 1
           sq = Square(100)
           self.assertEquals(sq.calculate_area(), 1)


       def test_mocking_classes(self):
           sq = Square
           sq.calculate_area = mock.MagicMock(return_value=1)
           self.assertEquals(sq.calculate_area(), 1)

       @mock.patch.object(Square, 'calculate_area')
       def test_mocking_class_methods(self, mocked_method):
           mocked_method.return_value = 20
           self.assertEquals(Square.calculate_area(), 20)

if __name__ == '__main__':
    unittest.main()

At line 13, I patch the class Square. Lines 15 and 16 present a mocking instance. mocked_instance is a mock object which returns another mock by default, and to these mock.calculate_area I add return_value 1. In line 23, I'm using MagicMock, which is a normal mock class, except in that it also retrieves magic methods from the given object. Lastly, I use patch.object to mock the method in the Square class.

The same using pytest:

try: 
    from mock import MagicMock 
except ImportError: 
    from unittest.mock import MagicMock

from square import Square

def test_mocking_class_methods(monkeypatch):
    monkeypatch.setattr('test_class_pytest.Square.calculate_area', lambda: 1)
    assert Square.calculate_area() ==  1


def test_mocking_classes(monkeypatch):
    monkeypatch.setattr('test_class_pytest.Square', MagicMock(Square))
    sq = Square
    sq.calculate_area.return_value = 1
    assert sq.calculate_area() ==  1

The issue here is with test_mocking_class_methods, which works well in Python 3, but not in Python 2.

All examples can be found in this repo.

If you have any questions and comments, feel free to leave them in the section below.

References:

1. What is Mocking?
2. Mocking With Kung Fu Panda

This article is brought with ❤ to you by Semaphore.

27 Sep 2016 7:31am GMT

A Closer Look at the Django ORM and Many-To-Many Relationships

In the last post I worked some on the data model for the KidsTasks app and discovered that a many-to-many relationship would not allow multiple copies of the same task to exist in a given schedule. Further reading showed me, without much explanation, that using a "through" parameter on the relationship definition fixed that. In this post I want to take a closer look at what's going on in that django model magic.

Django Shell

As part of my research for this topic, I was lead to a quick description of the Django shell which is great for testing out ideas and playing with the models you're developing. I found a good description here. (which also gives a look at filters and QuerySets).

Additionally, I'll note for anyone wanting to play along at home, that the following sequence of commands was quite helpful to have handy when testing different models.

 $ rm tasks/migrations db.sqlite3 -rf
 $ ./manage.py makemigrations tasks
 $ ./manage.py migrate
 $ ./manage.py shell
 Python 3.4.3 (default, Oct 14 2015, 20:33:09)
 [GCC 4.8.4] on linux
 Type "help", "copyright", "credits" or "license" for more information.
 (InteractiveConsole)

Many To Many without an Intermediate Class

I'll start by examining what happened with my original model design where a DayOfWeekSchedule had a ManyToMany relationship with Task.

Simple Solution Code

The simplified model I'll use here looks like this.

class Task(models.Model):
 name = models.CharField(max_length=256)
 required = models.BooleanField()

def __str__(self):
 return self.name

class DayOfWeekSchedule(models.Model):
 tasks = models.ManyToManyField(Task)
 name = models.CharField(max_length=20)

def __str__(self):
 return self.name

Note that the ManyToMany field directly accesses the Task class. (Also note that I retained the __str__ methods to make the shell output more meaningful.)

Experiment

In the shell experiment show in the listing below, I set up a few Tasks and
a couple of DayOfWeekSchedules and then add "first task" and "second
task" to one of the schedules. Once this is done, I attempt to add "first
task" to the schedule again and we see that it does not have the desired
effect.

>>> # import our models
>>> from tasks.models import Task, DayOfWeekSchedule
>>>
>>> # populate our database with some simple tasks and schedules
>>> Task.objects.create(name="first task", required=False)
<Task: first task>
>>> Task.objects.create(name="second task", required=True)
<Task: second task>
>>> Task.objects.create(name="third task", required=False)
<Task: third task>
>>> DayOfWeekSchedule.objects.create(name="sched1")
<DayOfWeekSchedule: sched1>
>>> DayOfWeekSchedule.objects.create(name="sched2")
<DayOfWeekSchedule: sched2>
>>> Task.objects.all()
<QuerySet [<Task: first task>, <Task: second task>, <Task: third task>]>
>>> DayOfWeekSchedule.objects.all()
<QuerySet [<DayOfWeekSchedule: sched1>, <DayOfWeekSchedule: sched2>]>
>>>
>>> # add a task to a schedule
>>> s = DayOfWeekSchedule.objects.get(name='sched2')
>>> t = Task.objects.get(name='first task')
>>> s.tasks.add(t)
>>> s.tasks.all()
<QuerySet [<Task: first task>]>
>>>
>>> # add other task to that schedule
>>> t = Task.objects.get(name='second task')
>>> s.tasks.add(t)
>>> s.tasks.all()
<QuerySet [<Task: first task>, <Task: second task>]>
>>>
>>> # attempt to add the first task to the schedule again
>>> s = DayOfWeekSchedule.objects.get(name='sched2')
>>> t = Task.objects.get(name='first task')
>>> s.tasks.add(t)
>>> s.tasks.all()
<QuerySet [<Task: first task>, <Task: second task>]>

Note that at the end, we still only have a single copy of "first task" in the schedule.

Many To Many with an Intermediate Class

Now we'll retry the experiment with the "through=" intermediate class specified in the ManyToMany relationship.

Not-Quite-As-Simple Solution Code

The model code for this is quite similar. Note the addition of the "through=" option and of the DayTask class.

from django.db import models

class Task(models.Model):
 name = models.CharField(max_length=256)
 required = models.BooleanField()

def __str__(self):
 return self.name

class DayOfWeekSchedule(models.Model):
 tasks = models.ManyToManyField(Task, through='DayTask')
 name = models.CharField(max_length=20)

def __str__(self):
 return self.name

class DayTask(models.Model):
 task = models.ForeignKey(Task)
 schedule = models.ForeignKey(DayOfWeekSchedule)

Experiment #2

This script is as close as possible to the first set. The only difference being the extra steps we need to take to add the ManyToMany relationship. We need to manually create the object of DayTask, initializing it with the Task and Schedule objects and then saving it. While this is slightly more cumbersome in the code, it does produce the desired results; two copies of "first task" are present in the schedule at the end.

>>> # import our models
>>> from tasks.models import Task, DayOfWeekSchedule, DayTask
>>>
>>> # populate our database with some simple tasks and schedules
>>> Task.objects.create(name="first task", required=False)
<Task: first task>
>>> Task.objects.create(name="second task", required=True)
<Task: second task>
>>> Task.objects.create(name="third task", required=False)
<Task: third task>
>>> DayOfWeekSchedule.objects.create(name="sched1")
<DayOfWeekSchedule: sched1>
>>> DayOfWeekSchedule.objects.create(name="sched2")
<DayOfWeekSchedule: sched2>
>>> Task.objects.all()
<QuerySet [<Task: first task>, <Task: second task>, <Task: third task>]>
>>> DayOfWeekSchedule.objects.all()
<QuerySet [<DayOfWeekSchedule: sched1>, <DayOfWeekSchedule: sched2>]>
>>>
>>> # add a task to a schedule
>>> s = DayOfWeekSchedule.objects.get(name='sched2')
>>> t = Task.objects.get(name='first task')
>>> # cannot simply add directly, must create intermediate object see
>>> # https://docs.djangoproject.com/en/1.9/topics/db/models/#extra-fields-on-many-to-many-relationships
>>> # s.tasks.add(t)
>>> d1 = DayTask(task=t, schedule=s)
>>> d1.save()
>>> s.tasks.all()
<QuerySet [<Task: first task>]>
>>>
>>> # add other task to that schedule
>>> t = Task.objects.get(name='second task')
>>> dt2 = DayTask(task=t, schedule=s)
>>> dt2.save()
>>> # s.tasks.add(t)
>>> s.tasks.all()
<QuerySet [<Task: first task>, <Task: second task>]>
>>>
>>> # attempt to add the first task to the schedule again
>>> s = DayOfWeekSchedule.objects.get(name='sched2')
>>> t = Task.objects.get(name='first task')
>>> dt3 = DayTask(task=t, schedule=s)
>>> dt3.save()
>>> s.tasks.all()
<QuerySet [<Task: first task>, <Task: second task>, <Task: first task>]>

But…Why?

The short answer is that I'm not entirely sure why the intermediate class is needed to allow multiple instances. It's fairly clear that it is tied to how the Django code manages those relationships. Evidence confirming that can be seen in the migration script generated for each of the models.

The first model generates these operations:

operations = [
 migrations.CreateModel(
 name='DayOfWeekSchedule',
 fields=[
 ('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
 ('name', models.CharField(max_length=20)),
 ],
 ),
 migrations.CreateModel(
 name='Task',
 fields=[
 ('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
 ('name', models.CharField(max_length=256)),
 ('required', models.BooleanField()),
 ],
 ),
 migrations.AddField(
 model_name='dayofweekschedule',
 name='tasks',
 field=models.ManyToManyField(to='tasks.Task'),
 ),
 ]

Notice the final AddField call which adds "tasks" to the "dayofweekschedule" model directly.

The second model (shown above) generates a slightly different set of migration operations:

operations = [
 migrations.CreateModel(
 name='DayOfWeekSchedule',
 fields=[
 ('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
 ('name', models.CharField(max_length=20)),
 ],
 ),
 migrations.CreateModel(
 name='DayTask',
 fields=[
 ('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
 ('schedule', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, to='tasks.DayOfWeekSchedule')),
 ],
 ),
 migrations.CreateModel(
 name='Task',
 fields=[
 ('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
 ('name', models.CharField(max_length=256)),
 ('required', models.BooleanField()),
 ],
 ),
 migrations.AddField(
 model_name='daytask',
 name='task',
 field=models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, to='tasks.Task'),
 ),
 migrations.AddField(
 model_name='dayofweekschedule',
 name='tasks',
 field=models.ManyToManyField(through='tasks.DayTask', to='tasks.Task'),
 ),
 ]

This time it adds task to the daytask and dayofweekschedule classes. I have to admit here that I really wanted this to show the DayTask object being used in the DayOfWeekSchedule class as a proxy, but that's not the case.

Examining the databases generated by these two models showed no significant differences there, either.

A Quick Look at the Source

One of the beauties of working with open source software is the ability to dive in and see for yourself what's going on. Looking at the Django source, you can find the code that adds a relationship in django/db/models/fields/related_descriptors.py (at line 918 in the version I checked out).

        def add(self, *objs):
            ... stuff deleted ...
            self._add_items(self.source_field_name, 
                            self.target_field_name, *objs)

(actually _add_items can be called twice, once for a forward and once for a reverse relationship). Looking at _add_items (line 1041 in my copy), we see after building the list of new_ids to insert, this chunk of code:

                db = router.db_for_write(self.through, 
                                         instance=self.instance)
                vals = (self.through._default_manager.using(db)
                        .values_list(target_field_name, flat=True)
                        .filter(**{
                            source_field_name: self.related_val[0],
                            '%s__in' % target_field_name: new_ids,
                        }))
                new_ids = new_ids - set(vals)

which I suspect of providing the difference. This code gets the list of current values in the relation table and removes that set from the set of new_ids. I believe that the filter here will respond differently if we have a intermediate class defined. NOTE: I did not run this code live to test this theory, so if I'm wrong, feel free to point out how and where in the comments.

Even if this is not quite correct, after walking through some code, I'm satisfied that the intermediate class definitely causes some different behavior internally in Django.

Next time I'll jump back into the KidsTasks code.

Thank for reading!

27 Sep 2016 1:57am GMT

A Closer Look at the Django ORM and Many-To-Many Relationships

In the last post I worked some on the data model for the KidsTasks app and discovered that a many-to-many relationship would not allow multiple copies of the same task to exist in a given schedule. Further reading showed me, without much explanation, that using a "through" parameter on the relationship definition fixed that. In this post I want to take a closer look at what's going on in that django model magic.

Django Shell

As part of my research for this topic, I was lead to a quick description of the Django shell which is great for testing out ideas and playing with the models you're developing. I found a good description here. (which also gives a look at filters and QuerySets).

Additionally, I'll note for anyone wanting to play along at home, that the following sequence of commands was quite helpful to have handy when testing different models.

 $ rm tasks/migrations db.sqlite3 -rf
 $ ./manage.py makemigrations tasks
 $ ./manage.py migrate
 $ ./manage.py shell
 Python 3.4.3 (default, Oct 14 2015, 20:33:09)
 [GCC 4.8.4] on linux
 Type "help", "copyright", "credits" or "license" for more information.
 (InteractiveConsole)

Many To Many without an Intermediate Class

I'll start by examining what happened with my original model design where a DayOfWeekSchedule had a ManyToMany relationship with Task.

Simple Solution Code

The simplified model I'll use here looks like this.

class Task(models.Model):
 name = models.CharField(max_length=256)
 required = models.BooleanField()

def __str__(self):
 return self.name

class DayOfWeekSchedule(models.Model):
 tasks = models.ManyToManyField(Task)
 name = models.CharField(max_length=20)

def __str__(self):
 return self.name

Note that the ManyToMany field directly accesses the Task class. (Also note that I retained the __str__ methods to make the shell output more meaningful.)

Experiment

In the shell experiment show in the listing below, I set up a few Tasks and
a couple of DayOfWeekSchedules and then add "first task" and "second
task" to one of the schedules. Once this is done, I attempt to add "first
task" to the schedule again and we see that it does not have the desired
effect.

>>> # import our models
>>> from tasks.models import Task, DayOfWeekSchedule
>>>
>>> # populate our database with some simple tasks and schedules
>>> Task.objects.create(name="first task", required=False)
<Task: first task>
>>> Task.objects.create(name="second task", required=True)
<Task: second task>
>>> Task.objects.create(name="third task", required=False)
<Task: third task>
>>> DayOfWeekSchedule.objects.create(name="sched1")
<DayOfWeekSchedule: sched1>
>>> DayOfWeekSchedule.objects.create(name="sched2")
<DayOfWeekSchedule: sched2>
>>> Task.objects.all()
<QuerySet [<Task: first task>, <Task: second task>, <Task: third task>]>
>>> DayOfWeekSchedule.objects.all()
<QuerySet [<DayOfWeekSchedule: sched1>, <DayOfWeekSchedule: sched2>]>
>>>
>>> # add a task to a schedule
>>> s = DayOfWeekSchedule.objects.get(name='sched2')
>>> t = Task.objects.get(name='first task')
>>> s.tasks.add(t)
>>> s.tasks.all()
<QuerySet [<Task: first task>]>
>>>
>>> # add other task to that schedule
>>> t = Task.objects.get(name='second task')
>>> s.tasks.add(t)
>>> s.tasks.all()
<QuerySet [<Task: first task>, <Task: second task>]>
>>>
>>> # attempt to add the first task to the schedule again
>>> s = DayOfWeekSchedule.objects.get(name='sched2')
>>> t = Task.objects.get(name='first task')
>>> s.tasks.add(t)
>>> s.tasks.all()
<QuerySet [<Task: first task>, <Task: second task>]>

Note that at the end, we still only have a single copy of "first task" in the schedule.

Many To Many with an Intermediate Class

Now we'll retry the experiment with the "through=" intermediate class specified in the ManyToMany relationship.

Not-Quite-As-Simple Solution Code

The model code for this is quite similar. Note the addition of the "through=" option and of the DayTask class.

from django.db import models

class Task(models.Model):
 name = models.CharField(max_length=256)
 required = models.BooleanField()

def __str__(self):
 return self.name

class DayOfWeekSchedule(models.Model):
 tasks = models.ManyToManyField(Task, through='DayTask')
 name = models.CharField(max_length=20)

def __str__(self):
 return self.name

class DayTask(models.Model):
 task = models.ForeignKey(Task)
 schedule = models.ForeignKey(DayOfWeekSchedule)

Experiment #2

This script is as close as possible to the first set. The only difference being the extra steps we need to take to add the ManyToMany relationship. We need to manually create the object of DayTask, initializing it with the Task and Schedule objects and then saving it. While this is slightly more cumbersome in the code, it does produce the desired results; two copies of "first task" are present in the schedule at the end.

>>> # import our models
>>> from tasks.models import Task, DayOfWeekSchedule, DayTask
>>>
>>> # populate our database with some simple tasks and schedules
>>> Task.objects.create(name="first task", required=False)
<Task: first task>
>>> Task.objects.create(name="second task", required=True)
<Task: second task>
>>> Task.objects.create(name="third task", required=False)
<Task: third task>
>>> DayOfWeekSchedule.objects.create(name="sched1")
<DayOfWeekSchedule: sched1>
>>> DayOfWeekSchedule.objects.create(name="sched2")
<DayOfWeekSchedule: sched2>
>>> Task.objects.all()
<QuerySet [<Task: first task>, <Task: second task>, <Task: third task>]>
>>> DayOfWeekSchedule.objects.all()
<QuerySet [<DayOfWeekSchedule: sched1>, <DayOfWeekSchedule: sched2>]>
>>>
>>> # add a task to a schedule
>>> s = DayOfWeekSchedule.objects.get(name='sched2')
>>> t = Task.objects.get(name='first task')
>>> # cannot simply add directly, must create intermediate object see
>>> # https://docs.djangoproject.com/en/1.9/topics/db/models/#extra-fields-on-many-to-many-relationships
>>> # s.tasks.add(t)
>>> d1 = DayTask(task=t, schedule=s)
>>> d1.save()
>>> s.tasks.all()
<QuerySet [<Task: first task>]>
>>>
>>> # add other task to that schedule
>>> t = Task.objects.get(name='second task')
>>> dt2 = DayTask(task=t, schedule=s)
>>> dt2.save()
>>> # s.tasks.add(t)
>>> s.tasks.all()
<QuerySet [<Task: first task>, <Task: second task>]>
>>>
>>> # attempt to add the first task to the schedule again
>>> s = DayOfWeekSchedule.objects.get(name='sched2')
>>> t = Task.objects.get(name='first task')
>>> dt3 = DayTask(task=t, schedule=s)
>>> dt3.save()
>>> s.tasks.all()
<QuerySet [<Task: first task>, <Task: second task>, <Task: first task>]>

But…Why?

The short answer is that I'm not entirely sure why the intermediate class is needed to allow multiple instances. It's fairly clear that it is tied to how the Django code manages those relationships. Evidence confirming that can be seen in the migration script generated for each of the models.

The first model generates these operations:

operations = [
 migrations.CreateModel(
 name='DayOfWeekSchedule',
 fields=[
 ('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
 ('name', models.CharField(max_length=20)),
 ],
 ),
 migrations.CreateModel(
 name='Task',
 fields=[
 ('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
 ('name', models.CharField(max_length=256)),
 ('required', models.BooleanField()),
 ],
 ),
 migrations.AddField(
 model_name='dayofweekschedule',
 name='tasks',
 field=models.ManyToManyField(to='tasks.Task'),
 ),
 ]

Notice the final AddField call which adds "tasks" to the "dayofweekschedule" model directly.

The second model (shown above) generates a slightly different set of migration operations:

operations = [
 migrations.CreateModel(
 name='DayOfWeekSchedule',
 fields=[
 ('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
 ('name', models.CharField(max_length=20)),
 ],
 ),
 migrations.CreateModel(
 name='DayTask',
 fields=[
 ('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
 ('schedule', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, to='tasks.DayOfWeekSchedule')),
 ],
 ),
 migrations.CreateModel(
 name='Task',
 fields=[
 ('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
 ('name', models.CharField(max_length=256)),
 ('required', models.BooleanField()),
 ],
 ),
 migrations.AddField(
 model_name='daytask',
 name='task',
 field=models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, to='tasks.Task'),
 ),
 migrations.AddField(
 model_name='dayofweekschedule',
 name='tasks',
 field=models.ManyToManyField(through='tasks.DayTask', to='tasks.Task'),
 ),
 ]

This time it adds task to the daytask and dayofweekschedule classes. I have to admit here that I really wanted this to show the DayTask object being used in the DayOfWeekSchedule class as a proxy, but that's not the case.

Examining the databases generated by these two models showed no significant differences there, either.

A Quick Look at the Source

One of the beauties of working with open source software is the ability to dive in and see for yourself what's going on. Looking at the Django source, you can find the code that adds a relationship in django/db/models/fields/related_descriptors.py (at line 918 in the version I checked out).

        def add(self, *objs):
            ... stuff deleted ...
            self._add_items(self.source_field_name, 
                            self.target_field_name, *objs)

(actually _add_items can be called twice, once for a forward and once for a reverse relationship). Looking at _add_items (line 1041 in my copy), we see after building the list of new_ids to insert, this chunk of code:

                db = router.db_for_write(self.through, 
                                         instance=self.instance)
                vals = (self.through._default_manager.using(db)
                        .values_list(target_field_name, flat=True)
                        .filter(**{
                            source_field_name: self.related_val[0],
                            '%s__in' % target_field_name: new_ids,
                        }))
                new_ids = new_ids - set(vals)

which I suspect of providing the difference. This code gets the list of current values in the relation table and removes that set from the set of new_ids. I believe that the filter here will respond differently if we have a intermediate class defined. NOTE: I did not run this code live to test this theory, so if I'm wrong, feel free to point out how and where in the comments.

Even if this is not quite correct, after walking through some code, I'm satisfied that the intermediate class definitely causes some different behavior internally in Django.

Next time I'll jump back into the KidsTasks code.

Thank for reading!

27 Sep 2016 1:57am GMT

Gestern musste ich morgens zur Station nach KWT um unsere Rerservierten Bustickets für die Weihnachtsferien in Capetown abzuholen. Der Bahnhof selber ist seit Dezember aus kostengründen ohne Zugverbindung - aber Translux und co - die langdistanzbusse haben dort ihre Büros.


Größere Kartenansicht

© benste CC NC SA

10 Nov 2011 10:57am GMT

Niemand ist besorgt um so was - mit dem Auto fährt man einfach durch, und in der City - nahe Gnobie- "ne das ist erst gefährlich wenn die Feuerwehr da ist" - 30min später auf dem Rückweg war die Feuerwehr da.

© benste CC NC SA

09 Nov 2011 8:25pm GMT

Brai = Grillabend o.ä.

Die möchte gern Techniker beim Flicken ihrer SpeakOn / Klinke Stecker Verzweigungen...

Die Damen "Mamas" der Siedlung bei der offiziellen Eröffnungsrede

Auch wenn weniger Leute da waren als erwartet, Laute Musik und viele Leute ...

Und natürlich ein Feuer mit echtem Holz zum Grillen.

© benste CC NC SA

08 Nov 2011 2:30pm GMT

One of our missions was bringing Katja's Linux Server back to her room. While doing that we saw her new decoration.

Björn, Simphiwe carried the PC to Katja's school

© benste CC NC SA

07 Nov 2011 2:00pm GMT

Today I went with Björn to Needs Camp to Visit Katja's guest family for a special Party. First of all we visited some friends of Nelisa - yeah the one I'm working with in Quigney - Katja's guest fathers sister - who did her a haircut.

African Women usually get their hair done by arranging extensions and not like Europeans just cutting some hair.

In between she looked like this...

And then she was done - looks amazing considering the amount of hair she had last week - doesn't it ?

© benste CC NC SA

06 Nov 2011 7:45pm GMT

Irgendwie viel mir heute auf das ich meine Blogposts mal ein bischen umstrukturieren muss - wenn ich immer nur von neuen Plätzen berichte, dann müsste ich ja eine Rundreise machen. Hier also mal ein paar Sachen aus meinem heutigen Alltag.

Erst einmal vorweg, Samstag zählt zumindest für uns Voluntäre zu den freien Tagen.

Dieses Wochenende sind nur Rommel und ich auf der Farm - Katja und Björn sind ja mittlerweile in ihren Einsatzstellen, und meine Mitbewohner Kyle und Jonathan sind zu Hause in Grahamstown - sowie auch Sipho der in Dimbaza wohnt.
Robin, die Frau von Rommel ist in Woodie Cape - schon seit Donnerstag um da ein paar Sachen zur erledigen.
Naja wie dem auch sei heute morgen haben wir uns erstmal ein gemeinsames Weetbix/Müsli Frühstück gegönnt und haben uns dann auf den Weg nach East London gemacht. 2 Sachen waren auf der Checkliste Vodacom, Ethienne (Imobilienmakler) außerdem auf dem Rückweg die fehlenden Dinge nach NeedsCamp bringen.

Nachdem wir gerade auf der Dirtroad losgefahren sind mussten wir feststellen das wir die Sachen für Needscamp und Ethienne nicht eingepackt hatten aber die Pumpe für die Wasserversorgung im Auto hatten.

Also sind wir in EastLondon ersteinmal nach Farmerama - nein nicht das onlinespiel farmville - sondern einen Laden mit ganz vielen Sachen für eine Farm - in Berea einem nördlichen Stadteil gefahren.

In Farmerama haben wir uns dann beraten lassen für einen Schnellverschluss der uns das leben mit der Pumpe leichter machen soll und außerdem eine leichtere Pumpe zur Reperatur gebracht, damit es nicht immer so ein großer Aufwand ist, wenn mal wieder das Wasser ausgegangen ist.

Fego Caffé ist in der Hemmingways Mall, dort mussten wir und PIN und PUK einer unserer Datensimcards geben lassen, da bei der PIN Abfrage leider ein zahlendreher unterlaufen ist. Naja auf jeden Fall speichern die Shops in Südafrika so sensible Daten wie eine PUK - die im Prinzip zugang zu einem gesperrten Phone verschafft.

Im Cafe hat Rommel dann ein paar online Transaktionen mit dem 3G Modem durchgeführt, welches ja jetzt wieder funktionierte - und übrigens mittlerweile in Ubuntu meinem Linuxsystem perfekt klappt.

Nebenbei bin ich nach 8ta gegangen um dort etwas über deren neue Deals zu erfahren, da wir in einigen von Hilltops Centern Internet anbieten wollen. Das Bild zeigt die Abdeckung UMTS in NeedsCamp Katjas Ort. 8ta ist ein neuer Telefonanbieter von Telkom, nachdem Vodafone sich Telkoms anteile an Vodacom gekauft hat müssen die komplett neu aufbauen.
Wir haben uns dazu entschieden mal eine kostenlose Prepaidkarte zu testen zu organisieren, denn wer weis wie genau die Karte oben ist ... Bevor man einen noch so billigen Deal für 24 Monate signed sollte man wissen obs geht.

Danach gings nach Checkers in Vincent, gesucht wurden zwei Hotplates für WoodyCape - R 129.00 eine - also ca. 12€ für eine zweigeteilte Kochplatte.
Wie man sieht im Hintergrund gibts schon Weihnachtsdeko - Anfang November und das in Südafrika bei sonnig warmen min- 25°C

Mittagessen haben wir uns bei einem Pakistanischen Curry Imbiss gegönnt - sehr empfehlenswert !
Naja und nachdem wir dann vor ner Stunde oder so zurück gekommen sind habe ich noch den Kühlschrank geputzt den ich heute morgen zum defrosten einfach nach draußen gestellt hatte. Jetzt ist der auch mal wieder sauber und ohne 3m dicke Eisschicht...

Morgen ... ja darüber werde ich gesondert berichten ... aber vermutlich erst am Montag, denn dann bin ich nochmal wieder in Quigney(East London) und habe kostenloses Internet.

© benste CC NC SA

05 Nov 2011 4:33pm GMT

Sterkspruit is one of Hilltops Computer Centres in the far north of Eastern Cape. On the trip to J'burg we've used the opportunity to take a look at the centre.

Pupils in the big classroom

The Trainer

School in Countryside

Adult Class in the Afternoon

"Town"

© benste CC NC SA

31 Oct 2011 4:58pm GMT

What are you doing in an internet cafe if your ADSL and Faxline has been discontinued before months end. Well my idea was sitting outside and eating some ice cream.
At least it's sunny and not as rainy as on the weekend.

© benste CC NC SA

31 Oct 2011 3:11pm GMT

For those who are traveling through Zastron - there is a very nice Restaurant which is serving delicious food at reasanable prices.
In addition they're selling home made juices jams and honey.

interior

home made specialities - the shop in the shop

the Bar

© benste CC NC SA

30 Oct 2011 4:47pm GMT

Having the 10 - 12h trip from J'burg back to ELS I was able to take a lot of pcitures including these different roadsides

Plain Street

Orange River in its beginngings (near Lesotho)

Zastron Anglican Church

The Bridge in Between "Free State" and Eastern Cape next to Zastron

my new Background ;)

If you listen to GoogleMaps you'll end up traveling 50km of gravel road - as it was just renewed we didn't have that many problems and saved 1h compared to going the official way with all it's constructions sites

Freeway

getting dark

© benste CC NC SA

29 Oct 2011 4:23pm GMT

Klar einiges mag anders sein, vieles aber gleich - aber ein in Deutschland täglich übliches Bild einer Straßenbaustelle - wie läuft das eigentlich in Südafrika ?

Ersteinmal vorweg - NEIN keine Ureinwohner die mit den Händen graben - auch wenn hier mehr Manpower genutzt wird - sind sie fleißig mit Technologie am arbeiten.

Eine ganz normale "Bundesstraße"

und wie sie erweitert wird

gaaaanz viele LKWs

denn hier wird eine Seite über einen langen Abschnitt komplett gesperrt, so das eine Ampelschaltung mit hier 45 Minuten Wartezeit entsteht

Aber wenigstens scheinen die ihren Spaß zu haben ;) - Wie auch wir denn gücklicher Weise mussten wir nie länger als 10 min. warten.

© benste CC NC SA

28 Oct 2011 4:20pm GMT