12 Feb 2016

feedPlanet Python

Ned Batchelder: The value of unit tests

Seems like testing and podcasts are in the air... First, I was interviewed on Brian Okken's Python Test podcast. I wasn't sure what to expect. The conversation went in a few different directions, and it was really nice to just chat with Brian for 45 minutes. We talked about coverage.py, testing, doing presentations, edX, and a few other things.

Then I see that Brian was himself a guest on Talk Python to Me, Michael Kennedy's podcast about all things Python.

On that episode, Brian does a good job arguing against some of the prevailing beliefs about testing. For example, he explains why unit tests are bad, and integration tests are good. His argument boils down to, you should test the promises you've made. Unit tests mostly deal with internal details that are not promises you've made to the outside world, so why focus on testing them? The important thing is whether your product behaves right from the outside.

I liked this argument, it made sense. But I don't think I agree with it. Or, I completely agree with it, and come to a different conclusion.

When I build a complex system, I can't deal with the whole thing at once. I need to think of it as a collection of smaller pieces. And the boundaries between those pieces need to remain somewhat stable. So they are promises, not to the outside world, but to myself. And since I have made those promises to myself, I want unit tests to be sure I'm keeping those promises.

Another value of unit tests is that they are a way to chop up combinatorial explosions. If my system has three main components, and each of them can be in ten different states, I'll need 1000 integration tests to cover all the possibilities. If I can test each component in isolation, then I only need 30 unit tests to cover the possibilities, plus a small number of integration tests to consider everything mashed together. Not to mention, the unit tests will be faster than the integration tests. Which would you rather have? 1000 slow tests, or 30 fast tests plus 20 slow tests?

Sure, it's possible to overdo unit testing. And it's really easy to have all your unit tests pass and still have a broken system. You need integration tests to be sure everything fits together properly. Finding the right balance is an art. I really like hearing Brian's take on it. Give it a listen.

12 Feb 2016 1:31am GMT

Ned Batchelder: The value of unit tests

Seems like testing and podcasts are in the air... First, I was interviewed on Brian Okken's Python Test podcast. I wasn't sure what to expect. The conversation went in a few different directions, and it was really nice to just chat with Brian for 45 minutes. We talked about coverage.py, testing, doing presentations, edX, and a few other things.

Then I see that Brian was himself a guest on Talk Python to Me, Michael Kennedy's podcast about all things Python.

On that episode, Brian does a good job arguing against some of the prevailing beliefs about testing. For example, he explains why unit tests are bad, and integration tests are good. His argument boils down to, you should test the promises you've made. Unit tests mostly deal with internal details that are not promises you've made to the outside world, so why focus on testing them? The important thing is whether your product behaves right from the outside.

I liked this argument, it made sense. But I don't think I agree with it. Or, I completely agree with it, and come to a different conclusion.

When I build a complex system, I can't deal with the whole thing at once. I need to think of it as a collection of smaller pieces. And the boundaries between those pieces need to remain somewhat stable. So they are promises, not to the outside world, but to myself. And since I have made those promises to myself, I want unit tests to be sure I'm keeping those promises.

Another value of unit tests is that they are a way to chop up combinatorial explosions. If my system has three main components, and each of them can be in ten different states, I'll need 1000 integration tests to cover all the possibilities. If I can test each component in isolation, then I only need 30 unit tests to cover the possibilities, plus a small number of integration tests to consider everything mashed together. Not to mention, the unit tests will be faster than the integration tests. Which would you rather have? 1000 slow tests, or 30 fast tests plus 20 slow tests?

Sure, it's possible to overdo unit testing. And it's really easy to have all your unit tests pass and still have a broken system. You need integration tests to be sure everything fits together properly. Finding the right balance is an art. I really like hearing Brian's take on it. Give it a listen.

12 Feb 2016 1:31am GMT

11 Feb 2016

feedPlanet Python

Brett Cannon: How the heck does async/await work in Python 3.5?

Being a core developer of Python has made me want to understand how the language generally works. I realize there will always be obscure corners where I don't know every intricate detail, but to be able to help with issues and the general design of Python I feel like I should try and understand its core semantics and how things work under the hood.

But until recently I didn't understand how async/await worked in Python 3.5. I knew that yield from in Python 3.3 combined with asyncio in Python 3.4 had led to this new syntax. But having not done a lot of networking stuff -- which asyncio is not limited to but does focus on -- had led to me not really paying much attention to all of this async/await stuff. I mean I knew that:

yield from iterator

was (essentially) equivalent to:

for x in iterator:
    yield x

And I knew that asyncio was an event loop framework which allowed for asynchronous programming, and I knew what those words (basically) meant on their own. But having never dived into the async/await syntax to understand how all of this came together, I felt I didn't understand asynchronous programming in Python which bothered me. So I decided to take the time and try and figure out how the heck all of it worked. And since I have heard from various people that they too didn't understand how this new world of asynchronous programming worked, I decided to write this essay (yes, this post has taken so long in time and is so long in words that my wife has labeled it an essay).

Now because I wanted a properly understanding of how the syntax worked, this essay has some low-level technical detail about how CPython does things. It's totally okay if it's more detail than you want or that you don't fully understand it as I don't explain every nuance of CPython internals in order to keep this from turning into a book (e.g., if you don't know that code objects have flags, let alone what a code object is, it's okay and you don't need to care to get something from this essay). I have tried to provide a more accessible summary at the end of every section so that you can skim the details if they turn out to be more than you want to deal with.

A history lesson about coroutines in Python

According to Wikipedia, "Coroutines are computer program components that generalize subroutines for nonpreemptive multitasking, by allowing multiple entry points for suspending and resuming execution at certain locations". That's a rather technical way of saying, "coroutines are functions whose execution you can pause". And if you are saying to yourself, "that sounds like generators", you would be right.

Back in Python 2.2, generators were first introduced by PEP 255 (they are also called generator iterators since generators implement the iterator protocol). Primarily inspired by the Icon programming language, generators allowed for a way to create an iterator that didn't waste memory when calculating the next value in the iteration. For instance, if you wanted to create your own version of range(), you could do it in an eager fashion by creating a list of integers:

def eager_range(up_to):
    """Create a list of integers, from 0 to up_to, exclusive."""
    sequence = []
    index = 0
    while index < up_to:
        sequence.append(index)
        index += 1
    return sequence

The problem with this, though, is that if you want a large sequence like the integers from 0 to 1,000,000, you have to create a list long enough to hold 1,000,000 integers. But when generators were added to the language, you could suddenly create an iterator that didn't need to create the whole sequence upfront. Instead, all you had to do is have enough memory for one integer at a time.

def lazy_range(up_to):
    """Generator to return the sequence of integers from 0 to up_to, exclusive."""
    index = 0
    while index < up_to:
        yield index
        index += 1

Having a function pause what it is doing whenever it hit a yield expression -- although it was a statement until Python 2.5 -- and then be able to resume later is very useful in terms of using less memory, allowing for the idea of infinite sequences, etc.

But as you may have noticed, generators are all about iterators. Now having a better way to create iterators is obviously great (and this is shown when you define an __iter__() method on an object as a generator), but people knew that if we took the "pausing" part of generators and added in a "send stuff back in" aspect to them, Python would suddenly have the concept of coroutines in Python (but until I say otherwise, consider this all just a concept in Python; concrete coroutines in Python are discussed later on). And that exact feature of sending stuff into a paused generator was added in Python 2.5 thanks to PEP 342. Among other things, PEP 342 introduced the send() method on generators. This allowed one to not only pause generators, but to send a value back into a generator where it paused. Taking our range() example further, you could make it so the sequence jumped forward or backward by some amount:

def jumping_range(up_to):
    """Generator for the sequence of integers from 0 to up_to, exclusive.

    Sending a value into the generator will shift the sequence by that amount.
    """
    index = 0
    while index < up_to:
        jump = yield index
        if jump is None:
            jump = 1
        index += jump


if __name__ == '__main__':
    iterator = jumping_range(5)
    print(next(iterator))  # 0
    print(iterator.send(2))  # 2
    print(next(iterator))  # 3
    print(iterator.send(-1))  # 2
    for x in iterator:
        print(x)  # 3, 4

Generators were not mucked with again until Python 3.3 when PEP 380 added yield from. Strictly speaking, the feature empowers you to refactor generators in a clean way by making it easy to yield every value from an iterator (which a generator conveniently happens to be).

def lazy_range(up_to):
    """Generator to return the sequence of integers from 0 to up_to, exclusive."""
    index = 0
    def gratuitous_refactor():
        while index < up_to:
            yield index
            index += 1
    yield from gratuitous_refactor()

By virtue of making refactoring easier, yield from also lets you chain generators together so that values bubble up and down the call stack without code having to do anything special.

def bottom():
    # Returning the yield lets the value that goes up the call stack to come right back
    # down.
    return (yield 42)

def middle():
    return (yield from bottom())

def top():
    return (yield from middle())

# Get the generator.
gen = top()
value = next(gen)
print(value)  # Prints '42'.
try:
    value = gen.send(value * 2)
except StopIteration as exc:
    value = exc.value
print(value)  # Prints '84'.

Summary

Generators in Python 2.2 let the execution of code be paused. Once the ability to send values back into the paused generators were introduced in Python 2.5, the concept of coroutines in Python became possible. And the addition of yield from in Python 3.3 made it easier to refactor generators as well as chain them together.

What is an event loop?

It's important to understand what an event loop is and how they make asynchronous programming possible if you're going to care about async/await. If you have done GUI programming before -- including web front-end work -- then you have worked with an event loop. But since having the concept of asynchronous programming as a language construct is new in Python, it's okay if you don't happen to know what an event loop is.

Going back to Wikipedia, an event loop "is a programming construct that waits for and dispatches events or messages in a program". Basically an event loop lets you go, "when A happens, do B". Probably the easiest example to explain this is that of the JavaScript event loop that's in every browser. Whenever you click something ("when A happens"), the click is given to the JavaScript event loop which checks if any onclick callback was registered to handle that click ("do B"). If any callbacks were registered then the callback is called with the details of the click. The event loop is considered a loop because it is constantly collecting events and loops over them to find what to with the event.

In Python's case, asyncio was added to the standard library to provide an event loop. There's a focus on networking in asyncio which in the case of the event loop is to make the "when A happens" to be when I/O from a socket is ready for reading and/or writing (via the selectors module). Other than GUIs and I/O, event loops are also often used for executing code in another thread or subprocess and have the event loop act as the scheduler (i.e., cooperative multitasking). If you happen to understand Python's GIL, event loops are useful in cases where releasing the GIL is possible and useful.

Summary

Event loops provide a loop which lets you say, "when A happens then do B". Basically an event loop watches out for when something occurs, and when something that the event loop cares about happens it then calls any code that cares about what happened. Python gained an event loop in the standard library in the form of asyncio in Python 3.4.

How async and await work

The way it was in Python 3.4

Between the generators found in Python 3.3 and an event loop in the form of asyncio, Python 3.4 had enough to support asynchronous programming in the form of concurrent programming. Asynchronous programming is basically programming where execution order is not known ahead of time (hence asynchronous instead of synchronous). Concurrent programming is writing code to execute independently of other parts, even if it all executes in a single thread (concurrency is not parallelism). For example, the following is Python 3.4 code to count down every second in two asynchronous, concurrent function calls.

import asyncio

# Borrowed from http://curio.readthedocs.org/en/latest/tutorial.html.
@asyncio.coroutine
def countdown(number, n):
    while n > 0:
        print('T-minus', n, '({})'.format(number))
        yield from asyncio.sleep(1)
        n -= 1

loop = asyncio.get_event_loop()
tasks = [
    asyncio.ensure_future(countdown("A", 2)),
    asyncio.ensure_future(countdown("B", 3))]
loop.run_until_complete(asyncio.wait(tasks))
loop.close()

In Python 3.4, the asyncio.coroutine decorator was used to label a function as acting as a coroutine that was meant for use with asyncio and its event loop. This gave Python its first concrete definition of a coroutine: an object who implemented the methods added to generators in PEP 342 and represented by the collections.abc.Coroutine abstract base class. This meant that suddenly all generators implemented the coroutine interface even if they weren't meant to be used in that fashion. To fix this, asyncio required that all generators meant to be used as a coroutine had to be decorated with asyncio.coroutine.

With this concrete definition of a coroutine (which matched an API that generators provided), you then used yield from on any asyncio.Future object to pass it down to the event loop, pausing execution of the coroutine while you waited for something to happen (being a future object is an implementation detail of asyncio and not important). Once the future object reached the event loop it was monitored there until the future object was done doing whatever it needed to do. Once the future was done doing its thing, the event loop noticed and the coroutine that was paused waiting for the future's result started again with its result sent back into the coroutine using its send() method.

Take our example above. The event loop starts each of the countdown() coroutine calls, executing until it hits yield from and the asyncio.sleep() function in one of them. That returns an asyncio.Future object which gets passed down to the event loop and pauses execution of the coroutine. There the event loop watches the future object until the one second is over (as well as checking on other stuff it's watching, like the other coroutine). Once the one second is up, the event loop takes the paused countdown() coroutine that gave the event loop the future object, sends the result of the future object back into the coroutine that gave it the future object in the first place, and the coroutine starts running again. This keeps going until all of the countdown() coroutines are finished running and the event loop has nothing to watch. I'll actually show you a complete example of how exactly all of this coroutine/event loop stuff works later, but first I want to explain how async and await work.

Going from yield from to await in Python 3.5

In Python 3.4, a function that was flagged as a coroutine for the purposes of asynchronous programming looked like:

# This also works in Python 3.5.
@asyncio.coroutine
def py34_coro():
    yield from stuff()

In Python 3.5, the types.coroutine decorator has been added to also flag a generator as a coroutine like asyncio.coroutine does. You can also use async def to syntactically define a function as being a coroutine, although it cannot contain any form of yield expression; only return and await are allowed for returning a value from the coroutine.

async def py35_coro():
    await stuff()

A key thing async and types.coroutine do, though, istighten the definition of what a coroutine is. It takescoroutines from simply being an interface to an actual type, making the distinction between any generator and a generator that is meant to be a coroutine much more stringent (and the inspect.iscoroutine() function is even stricter by saying async has to be used).

You will also notice that beyond just async, the Python 3.5 example introduces await expressions (which are only valid within an async def). While await operates much like yield from, the objects that are acceptable to an await expression are different. Coroutines are definitely allowed in an await expression since the concept of coroutines are fundamental in all of this. But when you call await on an object , it technically needs to be an awaitable object: an object that defines an __await__() method which returns an iterator which is not a coroutine itself . Coroutines themselves are also considered awaitable objects (hence why collections.abc.Coroutine inherits from collections.abc.Awaitable). This definition follows a Python tradition of making most syntax constructs translate into a method call underneath the hood, much like a + b is a.__add__(b) or b.__radd__(a) underneath it all.

How does the difference between yield from and await play out at a low level (i.e., a generator with types.coroutine vs. one with async def)? Let's look at the bytecode of the two examples above in Python 3.5 to get at the nitty-gritty details. The bytecode for py34_coro() is:

>>> dis.dis(py34_coro)
  2           0 LOAD_GLOBAL              0 (stuff)
              3 CALL_FUNCTION            0 (0 positional, 0 keyword pair)
              6 GET_YIELD_FROM_ITER
              7 LOAD_CONST               0 (None)
             10 YIELD_FROM
             11 POP_TOP
             12 LOAD_CONST               0 (None)
             15 RETURN_VALUE

The bytecode for py35_coro() is :

>>> dis.dis(py35_coro)
  1           0 LOAD_GLOBAL              0 (stuff)
              3 CALL_FUNCTION            0 (0 positional, 0 keyword pair)
              6 GET_AWAITABLE
              7 LOAD_CONST               0 (None)
             10 YIELD_FROM
             11 POP_TOP
             12 LOAD_CONST               0 (None)
             15 RETURN_VALUE

Ignoring the difference in line number due to py34_coro() having the asyncio.coroutine decorator, the only visible difference between them is the GET_YIELD_FROM_ITER opcode versus the GET_AWAITABLE opcode. Both functions are properly flagged as being coroutines, so there's no difference there. In the case of GET_YIELD_FROM_ITER, it simply checks if its argument is a generator or coroutine, otherwise it calls iter() on its argument (the acceptance of a coroutine object by the opcode for yield from is only allowed when the opcode is used from within a coroutine itself, which is true in this case thanks to the types.coroutine decorator flagging the generator as such at the C level with the CO_ITERABLE_COROUTINE flag on the code object).

But GET_AWAITABLE does something different. While the bytecode will accept a coroutine just like GET_YIELD_FROM_ITER, it will not accept a generator if has not been flagged as a coroutine. Beyond just coroutines, though, the bytecode will accepted an awaitable object as discussed earlier. This makes yield from expressions and await expressions both accept coroutines while differing on whether they accept plain generators or awaitable objects, respectively.

You may be wondering why the difference between what an async-based coroutine and a generator-based coroutine will accept in their respective pausing expressions? The key reason for this is to make sure you don't mess up and accidentally mix and match objects that just happen to have the same API to the best of Python's abilities. Since generators inherently implement the API for coroutines then it would be easy to accidentally use a generator when you actually expected to be using a coroutine. And since not all generators are written to be used in a coroutine-based control flow, you need to avoid accidentally using a generator incorrectly. But since Python is not statically compiled, the best the language can offer is runtime checks when using a generator-defined coroutine. This means that when types.coroutine is used, Python's compiler can't tell if a generator is going to be used as a coroutine or just a plain generator (remember, just because the syntax says types.coroutine that doesn't mean someone hasn't earlier done types = spam earlier), and thus different opcodes that have different restrictions are emitted by the compiler based on the knowledge it has at the time.

One very key point I want to make about the difference between a generator-based coroutine and an async one is that only generator-based coroutines can actually pause execution and force something to be sent down to the event loop. You typically don't see this very important detail because you usually call event loop-specific functions like the asyncio.sleep() function since event loops implement their own APIs and these are the kind of functions that have to worry about this little detail. For the vast majority of us, we will work with event loops rather than be writing them and thus only be writing async coroutines and never need to really care about this. But if you're like me and were wondering why you couldn't write something like asyncio.sleep() using only async coroutines, this can be quite the "aha!" moment.

Summary

Let's summarize all of this into simpler terms. Defining a method with async def makes it a coroutine. The other way to make a coroutine is to flag a generator with types.coroutine -- technically the flag is the CO_ITERABLE_COROUTINE flag on a code object -- or a subclass of collections.abc.Coroutine. You can only make a coroutine call chain pause with a generator-based coroutine.

An awaitable object is either a coroutine or an object that defines __await__() -- technically collections.abc.Awaitable -- which returns an iterator that is not a coroutine. An await expression is basically yield from but with restrictions of only working with awaitable objects (plain generators will not work with an await expression). An async function is a coroutine that either has return statements -- including the implicit return None at the end of every function in Python -- and/or await expressions (yield expressions are not allowed). The restrictions for async functions is to make sure you don't accidentally mix and match generator-based coroutines with other generators since the expected use of the two types of generators are rather different.

Think of async/await as an API for asynchronous programming

A key thing that I want to point out is actually something I didn't really think deeply about until I watched David Beazley's Python Brasil 2015 keynote. In that talk, David pointed out that async/await is really an API for asynchronous programming (which he reiterated to me on Twitter). What David means by this is that people shouldn't think that async/await as synonymous with asyncio, but instead think that asyncio is a framework that can utilize the async/await API for asynchronous programming.

David actually believes this idea of async/await being an asynchronous programming API that he has created the curio project to implement his own event loop. This has helped make it clear to me that async/await allows Python to provide the building blocks for asynchronous programming, but without tying you to a specific event loop or other low-level details (unlike other programming languages which integrate the event loop into the language directly). This allows for projects like curio to not only operate differently at a lower level (e.g., asyncio uses future objects as the API for talking to its event loop while curio uses tuples), but to also have different focuses and performance characteristics (e.g., asyncio has an entire framework for implementing transport and protocol layers which makes it extensible while curio is simpler and expects the user to worry about that kind of thing but also allows it to run faster).

Based on the (short) history of asynchronous programming in Python, it's understandable that people might think that async/await == asyncio. I mean asyncio was what helped make asynchronous programming possible in Python 3.4 and was a motivating factor for adding async/await in Python 3.5. But the design of async/await is purposefully flexible enough to not require asyncio or contort any critical design decision just for that framework. In other words, async/await continues Python's tradition of designing things to be as flexible as possible while still being pragmatic to use (and implement).

An example

At this point your head might be awash with new terms and concepts, making it a little hard to fully grasp how all of this is supposed to work to provide you asynchronous programming. To help make it all much more concrete, here is a complete (if contrived) asynchronous programming example, end-to-end from event loop and associated functions to user code. The example has coroutines which represents individual rocket launch countdowns but that appear to be counting down simultaneously . This is asynchronous programming through concurrency; three separate coroutines will be running independently, and yet it will all be done in a single thread.

import datetime
import heapq
import types
import time


class Task:

    """Represent how long a coroutine should before starting again.

    Comparison operators are implemented for use by heapq. Two-item
    tuples unfortunately don't work because when the datetime.datetime
    instances are equal, comparison falls to the coroutine and they don't
    implement comparison methods, triggering an exception.

    Think of this as being like asyncio.Task/curio.Task.
    """

    def __init__(self, wait_until, coro):
        self.coro = coro
        self.waiting_until = wait_until

    def __eq__(self, other):
        return self.waiting_until == other.waiting_until

    def __lt__(self, other):
        return self.waiting_until < other.waiting_until


class SleepingLoop:

    """An event loop focused on delaying execution of coroutines.

    Think of this as being like asyncio.BaseEventLoop/curio.Kernel.
    """

    def __init__(self, *coros):
        self._new = coros
        self._waiting = []

    def run_until_complete(self):
        # Start all the coroutines.
        for coro in self._new:
            wait_for = coro.send(None)
            heapq.heappush(self._waiting, Task(wait_for, coro))
        # Keep running until there is no more work to do.
        while self._waiting:
            now = datetime.datetime.now()
            # Get the coroutine with the soonest resumption time.
            task = heapq.heappop(self._waiting)
            if now < task.waiting_until:
                # We're ahead of schedule; wait until it's time to resume.
                delta = task.waiting_until - now
                time.sleep(delta.total_seconds())
                now = datetime.datetime.now()
            try:
                # It's time to resume the coroutine.
                wait_until = task.coro.send(now)
                heapq.heappush(self._waiting, Task(wait_until, task.coro))
            except StopIteration:
                # The coroutine is done.
                pass


@types.coroutine
def sleep(seconds):
    """Pause a coroutine for the specified number of seconds.

    Think of this as being like asyncio.sleep()/curio.sleep().
    """
    now = datetime.datetime.now()
    wait_until = now + datetime.timedelta(seconds=seconds)
    # Make all coroutines on the call stack pause; the need to use `yield`
    # necessitates this be generator-based and not an async-based coroutine.
    actual = yield wait_until
    # Resume the execution stack, sending back how long we actually waited.
    return actual - now


async def countdown(label, length, *, delay=0):
    """Countdown a launch for `length` seconds, waiting `delay` seconds.

    This is what a user would typically write.
    """
    print(label, 'waiting', delay, 'seconds before starting countdown')
    delta = await sleep(delay)
    print(label, 'starting after waiting', delta)
    while length:
        print(label, 'T-minus', length)
        waited = await sleep(1)
        length -= 1
    print(label, 'lift-off!')


def main():
    """Start the event loop, counting down 3 separate launches.

    This is what a user would typically write.
    """
    loop = SleepingLoop(countdown('A', 5), countdown('B', 3, delay=2),
                        countdown('C', 4, delay=1))
    start = datetime.datetime.now()
    loop.run_until_complete()
    print('Total elapsed time is', datetime.datetime.now() - start)


if __name__ == '__main__':
    main()

As I said, it's contrived, but if you run this in Python 3.5 you will notice that all three coroutines run independently in a single thread and yet the total amount of time taken to run is about 5 seconds. You can consider Task, SleepingLoop, and sleep() as what an event loop provider like asyncio and curio would give you. For a normal user, only the code in countdown() and main() are of importance. As you can see, there is no magic to async, await, or this whole asynchronous programming deal; it's just an API that Python provides you to help make this sort of thing easier.

My hopes and dreams for the future

Now that I understand how this asynchronous programming works in Python, I want to use it all the time! It's such an awesome concept that's so much better than something you would have used threads for previously. The problem is that Python 3.5 is so new that async/await is also very new. That means there are not a lot of libraries out there supporting asynchronous programming like this. For instance, to do HTTP requests you either have to construct the HTTP request yourself by hand (yuck), use a project like the aiohttp framework which adds HTTP on top of another event loop (in this case, asyncio), or hope more projects like the hyper library continue to spring up to provide an abstraction for things like HTTP which allow you to use whatever I/O library you want (although unfortunately hyper only supports HTTP/2 at the moment).

Personally, I hope projects like hyper take off so that we have a clear separation between getting binary data from I/O and how we interpret that binary data. This kind of abstraction is important because most I/O libraries in Python are rather tightly coupled to how they do I/O and how they handle data coming from I/O. This is a problem with the http package in Python's standard library as it doesn't have an HTTP parser but a connection object which does all the I/O for you. And if you were hoping requests would support asynchronous programming, your hopes have already been dashed because the synchronous I/O that requests uses is baked into its design. This shift in ability to do asynchronous programming gives the Python community a chance to fix a problem it has with not having abstractions at the various layers of the network stack. And we have the perk of it not being hard to make asynchronous code run as if its synchronous, so tools filling the void for asynchronous programming can work in both worlds.

I also hope that Python gains some form of support in async coroutines for yield. Maybe this will require yet another keyword (maybe something like anticipate?), but the fact that you actually can't implement an event loop system with just async coroutines bothers me. Luckily, it turns out I'm not the only one who thinks this, and since the author of PEP 492 agrees with me, I think there's a chance of getting this quirk removed.

Conclusion

Basically async and await are fancy generators that we call coroutines and there is some extra support for things called awaitable objects and turning plain generators in to coroutines. All of this comes together to support concurrency so that we have better support for asynchronous programming in Python. It's awesome and much easier to use than comparable approaches like threads -- I wrote an end-to-end example of asynchronous programming in under 100 lines of commented Python code -- while still being quite flexible and fast (the curio FAQ says that it runs faster than twisted by 30-40% but slower than gevent by 10-15%, and all while being implemented in pure Python; remember that Python 2 + Twisted can use less memory and is easier to debug than Go, so just imagine what you could do with this!). I'm very happy that this landed in Python 3 and I look forward to the community embracing it and helping to flesh out its support in libraries and frameworks so we can all benefit from asynchronous programming in Python.

11 Feb 2016 11:33pm GMT

Brett Cannon: How the heck does async/await work in Python 3.5?

Being a core developer of Python has made me want to understand how the language generally works. I realize there will always be obscure corners where I don't know every intricate detail, but to be able to help with issues and the general design of Python I feel like I should try and understand its core semantics and how things work under the hood.

But until recently I didn't understand how async/await worked in Python 3.5. I knew that yield from in Python 3.3 combined with asyncio in Python 3.4 had led to this new syntax. But having not done a lot of networking stuff -- which asyncio is not limited to but does focus on -- had led to me not really paying much attention to all of this async/await stuff. I mean I knew that:

yield from iterator

was (essentially) equivalent to:

for x in iterator:
    yield x

And I knew that asyncio was an event loop framework which allowed for asynchronous programming, and I knew what those words (basically) meant on their own. But having never dived into the async/await syntax to understand how all of this came together, I felt I didn't understand asynchronous programming in Python which bothered me. So I decided to take the time and try and figure out how the heck all of it worked. And since I have heard from various people that they too didn't understand how this new world of asynchronous programming worked, I decided to write this essay (yes, this post has taken so long in time and is so long in words that my wife has labeled it an essay).

Now because I wanted a properly understanding of how the syntax worked, this essay has some low-level technical detail about how CPython does things. It's totally okay if it's more detail than you want or that you don't fully understand it as I don't explain every nuance of CPython internals in order to keep this from turning into a book (e.g., if you don't know that code objects have flags, let alone what a code object is, it's okay and you don't need to care to get something from this essay). I have tried to provide a more accessible summary at the end of every section so that you can skim the details if they turn out to be more than you want to deal with.

A history lesson about coroutines in Python

According to Wikipedia, "Coroutines are computer program components that generalize subroutines for nonpreemptive multitasking, by allowing multiple entry points for suspending and resuming execution at certain locations". That's a rather technical way of saying, "coroutines are functions whose execution you can pause". And if you are saying to yourself, "that sounds like generators", you would be right.

Back in Python 2.2, generators were first introduced by PEP 255 (they are also called generator iterators since generators implement the iterator protocol). Primarily inspired by the Icon programming language, generators allowed for a way to create an iterator that didn't waste memory when calculating the next value in the iteration. For instance, if you wanted to create your own version of range(), you could do it in an eager fashion by creating a list of integers:

def eager_range(up_to):
    """Create a list of integers, from 0 to up_to, exclusive."""
    sequence = []
    index = 0
    while index < up_to:
        sequence.append(index)
        index += 1
    return sequence

The problem with this, though, is that if you want a large sequence like the integers from 0 to 1,000,000, you have to create a list long enough to hold 1,000,000 integers. But when generators were added to the language, you could suddenly create an iterator that didn't need to create the whole sequence upfront. Instead, all you had to do is have enough memory for one integer at a time.

def lazy_range(up_to):
    """Generator to return the sequence of integers from 0 to up_to, exclusive."""
    index = 0
    while index < up_to:
        yield index
        index += 1

Having a function pause what it is doing whenever it hit a yield expression -- although it was a statement until Python 2.5 -- and then be able to resume later is very useful in terms of using less memory, allowing for the idea of infinite sequences, etc.

But as you may have noticed, generators are all about iterators. Now having a better way to create iterators is obviously great (and this is shown when you define an __iter__() method on an object as a generator), but people knew that if we took the "pausing" part of generators and added in a "send stuff back in" aspect to them, Python would suddenly have the concept of coroutines in Python (but until I say otherwise, consider this all just a concept in Python; concrete coroutines in Python are discussed later on). And that exact feature of sending stuff into a paused generator was added in Python 2.5 thanks to PEP 342. Among other things, PEP 342 introduced the send() method on generators. This allowed one to not only pause generators, but to send a value back into a generator where it paused. Taking our range() example further, you could make it so the sequence jumped forward or backward by some amount:

def jumping_range(up_to):
    """Generator for the sequence of integers from 0 to up_to, exclusive.

    Sending a value into the generator will shift the sequence by that amount.
    """
    index = 0
    while index < up_to:
        jump = yield index
        if jump is None:
            jump = 1
        index += jump


if __name__ == '__main__':
    iterator = jumping_range(5)
    print(next(iterator))  # 0
    print(iterator.send(2))  # 2
    print(next(iterator))  # 3
    print(iterator.send(-1))  # 2
    for x in iterator:
        print(x)  # 3, 4

Generators were not mucked with again until Python 3.3 when PEP 380 added yield from. Strictly speaking, the feature empowers you to refactor generators in a clean way by making it easy to yield every value from an iterator (which a generator conveniently happens to be).

def lazy_range(up_to):
    """Generator to return the sequence of integers from 0 to up_to, exclusive."""
    index = 0
    def gratuitous_refactor():
        while index < up_to:
            yield index
            index += 1
    yield from gratuitous_refactor()

By virtue of making refactoring easier, yield from also lets you chain generators together so that values bubble up and down the call stack without code having to do anything special.

def bottom():
    # Returning the yield lets the value that goes up the call stack to come right back
    # down.
    return (yield 42)

def middle():
    return (yield from bottom())

def top():
    return (yield from middle())

# Get the generator.
gen = top()
value = next(gen)
print(value)  # Prints '42'.
try:
    value = gen.send(value * 2)
except StopIteration as exc:
    value = exc.value
print(value)  # Prints '84'.

Summary

Generators in Python 2.2 let the execution of code be paused. Once the ability to send values back into the paused generators were introduced in Python 2.5, the concept of coroutines in Python became possible. And the addition of yield from in Python 3.3 made it easier to refactor generators as well as chain them together.

What is an event loop?

It's important to understand what an event loop is and how they make asynchronous programming possible if you're going to care about async/await. If you have done GUI programming before -- including web front-end work -- then you have worked with an event loop. But since having the concept of asynchronous programming as a language construct is new in Python, it's okay if you don't happen to know what an event loop is.

Going back to Wikipedia, an event loop "is a programming construct that waits for and dispatches events or messages in a program". Basically an event loop lets you go, "when A happens, do B". Probably the easiest example to explain this is that of the JavaScript event loop that's in every browser. Whenever you click something ("when A happens"), the click is given to the JavaScript event loop which checks if any onclick callback was registered to handle that click ("do B"). If any callbacks were registered then the callback is called with the details of the click. The event loop is considered a loop because it is constantly collecting events and loops over them to find what to with the event.

In Python's case, asyncio was added to the standard library to provide an event loop. There's a focus on networking in asyncio which in the case of the event loop is to make the "when A happens" to be when I/O from a socket is ready for reading and/or writing (via the selectors module). Other than GUIs and I/O, event loops are also often used for executing code in another thread or subprocess and have the event loop act as the scheduler (i.e., cooperative multitasking). If you happen to understand Python's GIL, event loops are useful in cases where releasing the GIL is possible and useful.

Summary

Event loops provide a loop which lets you say, "when A happens then do B". Basically an event loop watches out for when something occurs, and when something that the event loop cares about happens it then calls any code that cares about what happened. Python gained an event loop in the standard library in the form of asyncio in Python 3.4.

How async and await work

The way it was in Python 3.4

Between the generators found in Python 3.3 and an event loop in the form of asyncio, Python 3.4 had enough to support asynchronous programming in the form of concurrent programming. Asynchronous programming is basically programming where execution order is not known ahead of time (hence asynchronous instead of synchronous). Concurrent programming is writing code to execute independently of other parts, even if it all executes in a single thread (concurrency is not parallelism). For example, the following is Python 3.4 code to count down every second in two asynchronous, concurrent function calls.

import asyncio

# Borrowed from http://curio.readthedocs.org/en/latest/tutorial.html.
@asyncio.coroutine
def countdown(number, n):
    while n > 0:
        print('T-minus', n, '({})'.format(number))
        yield from asyncio.sleep(1)
        n -= 1

loop = asyncio.get_event_loop()
tasks = [
    asyncio.ensure_future(countdown("A", 2)),
    asyncio.ensure_future(countdown("B", 3))]
loop.run_until_complete(asyncio.wait(tasks))
loop.close()

In Python 3.4, the asyncio.coroutine decorator was used to label a function as acting as a coroutine that was meant for use with asyncio and its event loop. This gave Python its first concrete definition of a coroutine: an object who implemented the methods added to generators in PEP 342 and represented by the collections.abc.Coroutine abstract base class. This meant that suddenly all generators implemented the coroutine interface even if they weren't meant to be used in that fashion. To fix this, asyncio required that all generators meant to be used as a coroutine had to be decorated with asyncio.coroutine.

With this concrete definition of a coroutine (which matched an API that generators provided), you then used yield from on any asyncio.Future object to pass it down to the event loop, pausing execution of the coroutine while you waited for something to happen (being a future object is an implementation detail of asyncio and not important). Once the future object reached the event loop it was monitored there until the future object was done doing whatever it needed to do. Once the future was done doing its thing, the event loop noticed and the coroutine that was paused waiting for the future's result started again with its result sent back into the coroutine using its send() method.

Take our example above. The event loop starts each of the countdown() coroutine calls, executing until it hits yield from and the asyncio.sleep() function in one of them. That returns an asyncio.Future object which gets passed down to the event loop and pauses execution of the coroutine. There the event loop watches the future object until the one second is over (as well as checking on other stuff it's watching, like the other coroutine). Once the one second is up, the event loop takes the paused countdown() coroutine that gave the event loop the future object, sends the result of the future object back into the coroutine that gave it the future object in the first place, and the coroutine starts running again. This keeps going until all of the countdown() coroutines are finished running and the event loop has nothing to watch. I'll actually show you a complete example of how exactly all of this coroutine/event loop stuff works later, but first I want to explain how async and await work.

Going from yield from to await in Python 3.5

In Python 3.4, a function that was flagged as a coroutine for the purposes of asynchronous programming looked like:

# This also works in Python 3.5.
@asyncio.coroutine
def py34_coro():
    yield from stuff()

In Python 3.5, the types.coroutine decorator has been added to also flag a generator as a coroutine like asyncio.coroutine does. You can also use async def to syntactically define a function as being a coroutine, although it cannot contain any form of yield expression; only return and await are allowed for returning a value from the coroutine.

async def py35_coro():
    await stuff()

A key thing async and types.coroutine do, though, istighten the definition of what a coroutine is. It takescoroutines from simply being an interface to an actual type, making the distinction between any generator and a generator that is meant to be a coroutine much more stringent (and the inspect.iscoroutine() function is even stricter by saying async has to be used).

You will also notice that beyond just async, the Python 3.5 example introduces await expressions (which are only valid within an async def). While await operates much like yield from, the objects that are acceptable to an await expression are different. Coroutines are definitely allowed in an await expression since the concept of coroutines are fundamental in all of this. But when you call await on an object , it technically needs to be an awaitable object: an object that defines an __await__() method which returns an iterator which is not a coroutine itself . Coroutines themselves are also considered awaitable objects (hence why collections.abc.Coroutine inherits from collections.abc.Awaitable). This definition follows a Python tradition of making most syntax constructs translate into a method call underneath the hood, much like a + b is a.__add__(b) or b.__radd__(a) underneath it all.

How does the difference between yield from and await play out at a low level (i.e., a generator with types.coroutine vs. one with async def)? Let's look at the bytecode of the two examples above in Python 3.5 to get at the nitty-gritty details. The bytecode for py34_coro() is:

>>> dis.dis(py34_coro)
  2           0 LOAD_GLOBAL              0 (stuff)
              3 CALL_FUNCTION            0 (0 positional, 0 keyword pair)
              6 GET_YIELD_FROM_ITER
              7 LOAD_CONST               0 (None)
             10 YIELD_FROM
             11 POP_TOP
             12 LOAD_CONST               0 (None)
             15 RETURN_VALUE

The bytecode for py35_coro() is :

>>> dis.dis(py35_coro)
  1           0 LOAD_GLOBAL              0 (stuff)
              3 CALL_FUNCTION            0 (0 positional, 0 keyword pair)
              6 GET_AWAITABLE
              7 LOAD_CONST               0 (None)
             10 YIELD_FROM
             11 POP_TOP
             12 LOAD_CONST               0 (None)
             15 RETURN_VALUE

Ignoring the difference in line number due to py34_coro() having the asyncio.coroutine decorator, the only visible difference between them is the GET_YIELD_FROM_ITER opcode versus the GET_AWAITABLE opcode. Both functions are properly flagged as being coroutines, so there's no difference there. In the case of GET_YIELD_FROM_ITER, it simply checks if its argument is a generator or coroutine, otherwise it calls iter() on its argument (the acceptance of a coroutine object by the opcode for yield from is only allowed when the opcode is used from within a coroutine itself, which is true in this case thanks to the types.coroutine decorator flagging the generator as such at the C level with the CO_ITERABLE_COROUTINE flag on the code object).

But GET_AWAITABLE does something different. While the bytecode will accept a coroutine just like GET_YIELD_FROM_ITER, it will not accept a generator if has not been flagged as a coroutine. Beyond just coroutines, though, the bytecode will accepted an awaitable object as discussed earlier. This makes yield from expressions and await expressions both accept coroutines while differing on whether they accept plain generators or awaitable objects, respectively.

You may be wondering why the difference between what an async-based coroutine and a generator-based coroutine will accept in their respective pausing expressions? The key reason for this is to make sure you don't mess up and accidentally mix and match objects that just happen to have the same API to the best of Python's abilities. Since generators inherently implement the API for coroutines then it would be easy to accidentally use a generator when you actually expected to be using a coroutine. And since not all generators are written to be used in a coroutine-based control flow, you need to avoid accidentally using a generator incorrectly. But since Python is not statically compiled, the best the language can offer is runtime checks when using a generator-defined coroutine. This means that when types.coroutine is used, Python's compiler can't tell if a generator is going to be used as a coroutine or just a plain generator (remember, just because the syntax says types.coroutine that doesn't mean someone hasn't earlier done types = spam earlier), and thus different opcodes that have different restrictions are emitted by the compiler based on the knowledge it has at the time.

One very key point I want to make about the difference between a generator-based coroutine and an async one is that only generator-based coroutines can actually pause execution and force something to be sent down to the event loop. You typically don't see this very important detail because you usually call event loop-specific functions like the asyncio.sleep() function since event loops implement their own APIs and these are the kind of functions that have to worry about this little detail. For the vast majority of us, we will work with event loops rather than be writing them and thus only be writing async coroutines and never need to really care about this. But if you're like me and were wondering why you couldn't write something like asyncio.sleep() using only async coroutines, this can be quite the "aha!" moment.

Summary

Let's summarize all of this into simpler terms. Defining a method with async def makes it a coroutine. The other way to make a coroutine is to flag a generator with types.coroutine -- technically the flag is the CO_ITERABLE_COROUTINE flag on a code object -- or a subclass of collections.abc.Coroutine. You can only make a coroutine call chain pause with a generator-based coroutine.

An awaitable object is either a coroutine or an object that defines __await__() -- technically collections.abc.Awaitable -- which returns an iterator that is not a coroutine. An await expression is basically yield from but with restrictions of only working with awaitable objects (plain generators will not work with an await expression). An async function is a coroutine that either has return statements -- including the implicit return None at the end of every function in Python -- and/or await expressions (yield expressions are not allowed). The restrictions for async functions is to make sure you don't accidentally mix and match generator-based coroutines with other generators since the expected use of the two types of generators are rather different.

Think of async/await as an API for asynchronous programming

A key thing that I want to point out is actually something I didn't really think deeply about until I watched David Beazley's Python Brasil 2015 keynote. In that talk, David pointed out that async/await is really an API for asynchronous programming (which he reiterated to me on Twitter). What David means by this is that people shouldn't think that async/await as synonymous with asyncio, but instead think that asyncio is a framework that can utilize the async/await API for asynchronous programming.

David actually believes this idea of async/await being an asynchronous programming API that he has created the curio project to implement his own event loop. This has helped make it clear to me that async/await allows Python to provide the building blocks for asynchronous programming, but without tying you to a specific event loop or other low-level details (unlike other programming languages which integrate the event loop into the language directly). This allows for projects like curio to not only operate differently at a lower level (e.g., asyncio uses future objects as the API for talking to its event loop while curio uses tuples), but to also have different focuses and performance characteristics (e.g., asyncio has an entire framework for implementing transport and protocol layers which makes it extensible while curio is simpler and expects the user to worry about that kind of thing but also allows it to run faster).

Based on the (short) history of asynchronous programming in Python, it's understandable that people might think that async/await == asyncio. I mean asyncio was what helped make asynchronous programming possible in Python 3.4 and was a motivating factor for adding async/await in Python 3.5. But the design of async/await is purposefully flexible enough to not require asyncio or contort any critical design decision just for that framework. In other words, async/await continues Python's tradition of designing things to be as flexible as possible while still being pragmatic to use (and implement).

An example

At this point your head might be awash with new terms and concepts, making it a little hard to fully grasp how all of this is supposed to work to provide you asynchronous programming. To help make it all much more concrete, here is a complete (if contrived) asynchronous programming example, end-to-end from event loop and associated functions to user code. The example has coroutines which represents individual rocket launch countdowns but that appear to be counting down simultaneously . This is asynchronous programming through concurrency; three separate coroutines will be running independently, and yet it will all be done in a single thread.

import datetime
import heapq
import types
import time


class Task:

    """Represent how long a coroutine should before starting again.

    Comparison operators are implemented for use by heapq. Two-item
    tuples unfortunately don't work because when the datetime.datetime
    instances are equal, comparison falls to the coroutine and they don't
    implement comparison methods, triggering an exception.

    Think of this as being like asyncio.Task/curio.Task.
    """

    def __init__(self, wait_until, coro):
        self.coro = coro
        self.waiting_until = wait_until

    def __eq__(self, other):
        return self.waiting_until == other.waiting_until

    def __lt__(self, other):
        return self.waiting_until < other.waiting_until


class SleepingLoop:

    """An event loop focused on delaying execution of coroutines.

    Think of this as being like asyncio.BaseEventLoop/curio.Kernel.
    """

    def __init__(self, *coros):
        self._new = coros
        self._waiting = []

    def run_until_complete(self):
        # Start all the coroutines.
        for coro in self._new:
            wait_for = coro.send(None)
            heapq.heappush(self._waiting, Task(wait_for, coro))
        # Keep running until there is no more work to do.
        while self._waiting:
            now = datetime.datetime.now()
            # Get the coroutine with the soonest resumption time.
            task = heapq.heappop(self._waiting)
            if now < task.waiting_until:
                # We're ahead of schedule; wait until it's time to resume.
                delta = task.waiting_until - now
                time.sleep(delta.total_seconds())
                now = datetime.datetime.now()
            try:
                # It's time to resume the coroutine.
                wait_until = task.coro.send(now)
                heapq.heappush(self._waiting, Task(wait_until, task.coro))
            except StopIteration:
                # The coroutine is done.
                pass


@types.coroutine
def sleep(seconds):
    """Pause a coroutine for the specified number of seconds.

    Think of this as being like asyncio.sleep()/curio.sleep().
    """
    now = datetime.datetime.now()
    wait_until = now + datetime.timedelta(seconds=seconds)
    # Make all coroutines on the call stack pause; the need to use `yield`
    # necessitates this be generator-based and not an async-based coroutine.
    actual = yield wait_until
    # Resume the execution stack, sending back how long we actually waited.
    return actual - now


async def countdown(label, length, *, delay=0):
    """Countdown a launch for `length` seconds, waiting `delay` seconds.

    This is what a user would typically write.
    """
    print(label, 'waiting', delay, 'seconds before starting countdown')
    delta = await sleep(delay)
    print(label, 'starting after waiting', delta)
    while length:
        print(label, 'T-minus', length)
        waited = await sleep(1)
        length -= 1
    print(label, 'lift-off!')


def main():
    """Start the event loop, counting down 3 separate launches.

    This is what a user would typically write.
    """
    loop = SleepingLoop(countdown('A', 5), countdown('B', 3, delay=2),
                        countdown('C', 4, delay=1))
    start = datetime.datetime.now()
    loop.run_until_complete()
    print('Total elapsed time is', datetime.datetime.now() - start)


if __name__ == '__main__':
    main()

As I said, it's contrived, but if you run this in Python 3.5 you will notice that all three coroutines run independently in a single thread and yet the total amount of time taken to run is about 5 seconds. You can consider Task, SleepingLoop, and sleep() as what an event loop provider like asyncio and curio would give you. For a normal user, only the code in countdown() and main() are of importance. As you can see, there is no magic to async, await, or this whole asynchronous programming deal; it's just an API that Python provides you to help make this sort of thing easier.

My hopes and dreams for the future

Now that I understand how this asynchronous programming works in Python, I want to use it all the time! It's such an awesome concept that's so much better than something you would have used threads for previously. The problem is that Python 3.5 is so new that async/await is also very new. That means there are not a lot of libraries out there supporting asynchronous programming like this. For instance, to do HTTP requests you either have to construct the HTTP request yourself by hand (yuck), use a project like the aiohttp framework which adds HTTP on top of another event loop (in this case, asyncio), or hope more projects like the hyper library continue to spring up to provide an abstraction for things like HTTP which allow you to use whatever I/O library you want (although unfortunately hyper only supports HTTP/2 at the moment).

Personally, I hope projects like hyper take off so that we have a clear separation between getting binary data from I/O and how we interpret that binary data. This kind of abstraction is important because most I/O libraries in Python are rather tightly coupled to how they do I/O and how they handle data coming from I/O. This is a problem with the http package in Python's standard library as it doesn't have an HTTP parser but a connection object which does all the I/O for you. And if you were hoping requests would support asynchronous programming, your hopes have already been dashed because the synchronous I/O that requests uses is baked into its design. This shift in ability to do asynchronous programming gives the Python community a chance to fix a problem it has with not having abstractions at the various layers of the network stack. And we have the perk of it not being hard to make asynchronous code run as if its synchronous, so tools filling the void for asynchronous programming can work in both worlds.

I also hope that Python gains some form of support in async coroutines for yield. Maybe this will require yet another keyword (maybe something like anticipate?), but the fact that you actually can't implement an event loop system with just async coroutines bothers me. Luckily, it turns out I'm not the only one who thinks this, and since the author of PEP 492 agrees with me, I think there's a chance of getting this quirk removed.

Conclusion

Basically async and await are fancy generators that we call coroutines and there is some extra support for things called awaitable objects and turning plain generators in to coroutines. All of this comes together to support concurrency so that we have better support for asynchronous programming in Python. It's awesome and much easier to use than comparable approaches like threads -- I wrote an end-to-end example of asynchronous programming in under 100 lines of commented Python code -- while still being quite flexible and fast (the curio FAQ says that it runs faster than twisted by 30-40% but slower than gevent by 10-15%, and all while being implemented in pure Python; remember that Python 2 + Twisted can use less memory and is easier to debug than Go, so just imagine what you could do with this!). I'm very happy that this landed in Python 3 and I look forward to the community embracing it and helping to flesh out its support in libraries and frameworks so we can all benefit from asynchronous programming in Python.

11 Feb 2016 11:33pm GMT

Import Python: ImportPython Issue 61


Word From Our Sponsor


Python Programmers let companies apply to you, not the other way around. Receive interview offers which include salary and equity information. Companies see each other's offers, and compete for your attention. Engage only with the companies you like. REGISTER

Worthy Read

podcast
In this episode I interview Ned Batchelder. I know that coverage.py is very important to a lot of people to understand how much of their code is being covered by their test suites. Since I'm far from an expert on coverage, I asked Ned to discuss it on the show.

Today, I'm pleased to announce the first major release of Zappa - a system for running "serverless" Python web applications using AWS Lambda and AWS API Gateway. Zappa handles all of the configuration and deployment automatically - now, you can deploy an infinitely scalable application to the cloud with a single command - all for a minute fraction of the cost of a traditional web server.

pycon
The Talks committee has been hard at work since the Call For Proposals closed 5 weeks ago, and today we are thrilled to present the result - here are the talks that the committee has chosen for PyCon 2016 in Portland, Oregon!

django
I think we can all agree that React and Django Rest Framework are both awesome. But hooking React into your Django app can really be a nightmare, especially if you're unfamiliar with webpack, npm, and babel. I'm going to walk you through how to get Django Rest Framework (DRF) to work with React.

The Python 101 Screencast has been finished for a little over a month now and I am now releasing it for general consumption. The Python 101 Screencast is based on my book, Python 101. I went through all 44 chapters of the book and turned each of them into a standalone screencast.

pycon
PyCon accepted my talk "Write an Excellent Programming Blog". If you got in, too, congratulations! Now we have to write our talks. Steps Plan your time, Inspire, Outline, Rehearse immediately, Make room for new insights, Put off making slides, Rehearse with friends, Get a coach, Excel.

webcast
It's time for another free hour-long Webinar! This time, I'll be talking about the increasingly popular tools for data science in Python, namely Pandas and Matplotlib. How can you read data into Pandas, manipulate it, and then plot it? I'll show you a large number of examples and use cases, and we'll also have lots of time for Q&A.

docker
Docker is an open source infrastructure management platform for running and deploying software. The Docker platform is constantly evolving so an exact definition is currently a moving target. Docker can package up applications along with their necessary operating system dependencies for easier deployment across environments. In the long run it has the potential to be the abstraction layer that easily manages containers running on top of any type of server, regardless of whether that server is on Amazon Web Services, Google Compute Engine, Linode, Rackspace or elsewhere.

testing
The pytest core group is heading towards the biggest sprint in its history, to take place in the black forest town Freiburg in Germany. As of February 2016 we have started a funding campaign on Indiegogo to cover expenses The page also mentions some preliminary topics. Here is the campaign URL https://www.indiegogo.com/projects/python-testing-sprint-mid-2016#/

interview
Aisha Bello is a current student at Cardiff Metropolitan University, where she's finishing up a MSc in Information Technology. Her final project is centered on open source data mining technologies for small and medium-sized hospitality organizations. Aisha co-organized and coached at Django Girls Windhoek in January 2016, and is also organizing a Django Girls workshop in Lagos, Nigeria in February 2016.

PEP
This PEP proposes the creation of a new platform tag for Python package built distributions, such as wheels, called manylinux1_{x86_64,i686} with external dependencies limited to a standardized, restricted subset of the Linux kernel and core userspace ABI. It proposes that PyPI support uploading and distributing wheels with this platform tag, and that pip support downloading and installing these packages on compatible platforms.

podcast
Looking for an open source alternative to Mathematica or MatLab for solving algebraic equations? Look no further than the excellent SymPy project. It is a well built and easy to use Computer Algebra System (CAS) and in this episode we spoke with the current project maintainer Aaron Meurer about its capabilities and when you might want to use it.

core python
Why are you recommending Python? That's the question a colleague of mine asked when I was pitching Python for data science work. It is a fair question, and I tried to answer with facts and not opinions.

django
I've been using this pattern for many years in my applications, and I always found strange nobody ever mentioned it. Since it has prove useful to me in many different projects, I think it's a perfect occasion to put some life again in my blog.

Luthor utilizes all efficient tricks from pholcidae to make XML parsing as simple as never before. Luthor uses lxml's iterable parsing mechanism to parse files of any size.


Jobs

Paris, France
At Gorgias (Techstars NYC '15), we're making customer support software to easily handle big volumes of customer requests. We're using machine-learning to automatically group requests, suggest the right responses and show only the relevant information for the customer support agent to take swift action on repetitive tasks.

Bangalore, Karnataka, India
Springboard is an online education startup with a big vision for transforming education using open online learning. We believe we are in the early days of a revolution that will not only increase access to great education, but also transform the way people learn.

Cornwall, United Kingdom
Headforwards are looking for a Python developer to join their team on an existing project. Some front end experience with web technologies such as HTML / CSS / JQuery / AJAX / JavaScript / HTML5 would be useful additional knowledge.



Projects

BinaryNet - 76 Stars, 5 Fork
Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1

pip-check - 54 Stars, 1 Fork
Gives you a quick overview of all installed packages and their update status. Very much like pip list -o but with colors. Inspired by npm-check though lacks most of its features (yet).

bigchaindb - 47 Stars, 6 Fork
A scalable blockchain database

pyq - 45 Stars, 1 Fork
A tool to search for Python code using jQuery-like selectors

Implementation of Dynamic memory networks by Kumar et al. http://arxiv.org/abs/1506.07285

grabbySub2 - 15 Stars, 1 Fork
Grab movie/tv subtitles, hand crafted for you!

models - 9 Stars, 3 Fork
Models built with TensorFlow

django_simpleCrud - 8 Stars, 1 Fork
This is django simple crud featuring Bootstrap CSS + jQuery

kaggle-homesite - 7 Stars, 1 Fork
Top15 Model for Kaggle-Competition "Homesite Quote Conversion"

AWSInventoryLambda - 6 Stars, 3 Fork
Save AWS inventory as CSV on S3 and trigger emails

modpack-helper - 6 Stars, 0 Fork
Ease the deployement of modded Minecraft servers

pkgquiz - 4 Stars, 3 Fork
Match debian package names to their descriptions

pygraph - 4 Stars, 0 Fork
CLI interface to python graphviz

erandom - 4 Stars, 0 Fork
Like /dev/random but with emojis!

bin_analyzer - 4 Stars, 0 Fork
Toy project for static analysis of ELF binaries

animated-win-background - 4 Stars, 1 Fork
Animated backgrounds on Windows (GIF frames)

django-auth-adfs - 3 Stars, 0 Fork
A Django authentication backend for ADFS

11 Feb 2016 7:40pm GMT

Import Python: ImportPython Issue 61


Word From Our Sponsor


Python Programmers let companies apply to you, not the other way around. Receive interview offers which include salary and equity information. Companies see each other's offers, and compete for your attention. Engage only with the companies you like. REGISTER

Worthy Read

podcast
In this episode I interview Ned Batchelder. I know that coverage.py is very important to a lot of people to understand how much of their code is being covered by their test suites. Since I'm far from an expert on coverage, I asked Ned to discuss it on the show.

Today, I'm pleased to announce the first major release of Zappa - a system for running "serverless" Python web applications using AWS Lambda and AWS API Gateway. Zappa handles all of the configuration and deployment automatically - now, you can deploy an infinitely scalable application to the cloud with a single command - all for a minute fraction of the cost of a traditional web server.

pycon
The Talks committee has been hard at work since the Call For Proposals closed 5 weeks ago, and today we are thrilled to present the result - here are the talks that the committee has chosen for PyCon 2016 in Portland, Oregon!

django
I think we can all agree that React and Django Rest Framework are both awesome. But hooking React into your Django app can really be a nightmare, especially if you're unfamiliar with webpack, npm, and babel. I'm going to walk you through how to get Django Rest Framework (DRF) to work with React.

The Python 101 Screencast has been finished for a little over a month now and I am now releasing it for general consumption. The Python 101 Screencast is based on my book, Python 101. I went through all 44 chapters of the book and turned each of them into a standalone screencast.

pycon
PyCon accepted my talk "Write an Excellent Programming Blog". If you got in, too, congratulations! Now we have to write our talks. Steps Plan your time, Inspire, Outline, Rehearse immediately, Make room for new insights, Put off making slides, Rehearse with friends, Get a coach, Excel.

webcast
It's time for another free hour-long Webinar! This time, I'll be talking about the increasingly popular tools for data science in Python, namely Pandas and Matplotlib. How can you read data into Pandas, manipulate it, and then plot it? I'll show you a large number of examples and use cases, and we'll also have lots of time for Q&A.

docker
Docker is an open source infrastructure management platform for running and deploying software. The Docker platform is constantly evolving so an exact definition is currently a moving target. Docker can package up applications along with their necessary operating system dependencies for easier deployment across environments. In the long run it has the potential to be the abstraction layer that easily manages containers running on top of any type of server, regardless of whether that server is on Amazon Web Services, Google Compute Engine, Linode, Rackspace or elsewhere.

testing
The pytest core group is heading towards the biggest sprint in its history, to take place in the black forest town Freiburg in Germany. As of February 2016 we have started a funding campaign on Indiegogo to cover expenses The page also mentions some preliminary topics. Here is the campaign URL https://www.indiegogo.com/projects/python-testing-sprint-mid-2016#/

interview
Aisha Bello is a current student at Cardiff Metropolitan University, where she's finishing up a MSc in Information Technology. Her final project is centered on open source data mining technologies for small and medium-sized hospitality organizations. Aisha co-organized and coached at Django Girls Windhoek in January 2016, and is also organizing a Django Girls workshop in Lagos, Nigeria in February 2016.

PEP
This PEP proposes the creation of a new platform tag for Python package built distributions, such as wheels, called manylinux1_{x86_64,i686} with external dependencies limited to a standardized, restricted subset of the Linux kernel and core userspace ABI. It proposes that PyPI support uploading and distributing wheels with this platform tag, and that pip support downloading and installing these packages on compatible platforms.

podcast
Looking for an open source alternative to Mathematica or MatLab for solving algebraic equations? Look no further than the excellent SymPy project. It is a well built and easy to use Computer Algebra System (CAS) and in this episode we spoke with the current project maintainer Aaron Meurer about its capabilities and when you might want to use it.

core python
Why are you recommending Python? That's the question a colleague of mine asked when I was pitching Python for data science work. It is a fair question, and I tried to answer with facts and not opinions.

django
I've been using this pattern for many years in my applications, and I always found strange nobody ever mentioned it. Since it has prove useful to me in many different projects, I think it's a perfect occasion to put some life again in my blog.

Luthor utilizes all efficient tricks from pholcidae to make XML parsing as simple as never before. Luthor uses lxml's iterable parsing mechanism to parse files of any size.


Jobs

Paris, France
At Gorgias (Techstars NYC '15), we're making customer support software to easily handle big volumes of customer requests. We're using machine-learning to automatically group requests, suggest the right responses and show only the relevant information for the customer support agent to take swift action on repetitive tasks.

Bangalore, Karnataka, India
Springboard is an online education startup with a big vision for transforming education using open online learning. We believe we are in the early days of a revolution that will not only increase access to great education, but also transform the way people learn.

Cornwall, United Kingdom
Headforwards are looking for a Python developer to join their team on an existing project. Some front end experience with web technologies such as HTML / CSS / JQuery / AJAX / JavaScript / HTML5 would be useful additional knowledge.



Projects

BinaryNet - 76 Stars, 5 Fork
Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1

pip-check - 54 Stars, 1 Fork
Gives you a quick overview of all installed packages and their update status. Very much like pip list -o but with colors. Inspired by npm-check though lacks most of its features (yet).

bigchaindb - 47 Stars, 6 Fork
A scalable blockchain database

pyq - 45 Stars, 1 Fork
A tool to search for Python code using jQuery-like selectors

Implementation of Dynamic memory networks by Kumar et al. http://arxiv.org/abs/1506.07285

grabbySub2 - 15 Stars, 1 Fork
Grab movie/tv subtitles, hand crafted for you!

models - 9 Stars, 3 Fork
Models built with TensorFlow

django_simpleCrud - 8 Stars, 1 Fork
This is django simple crud featuring Bootstrap CSS + jQuery

kaggle-homesite - 7 Stars, 1 Fork
Top15 Model for Kaggle-Competition "Homesite Quote Conversion"

AWSInventoryLambda - 6 Stars, 3 Fork
Save AWS inventory as CSV on S3 and trigger emails

modpack-helper - 6 Stars, 0 Fork
Ease the deployement of modded Minecraft servers

pkgquiz - 4 Stars, 3 Fork
Match debian package names to their descriptions

pygraph - 4 Stars, 0 Fork
CLI interface to python graphviz

erandom - 4 Stars, 0 Fork
Like /dev/random but with emojis!

bin_analyzer - 4 Stars, 0 Fork
Toy project for static analysis of ELF binaries

animated-win-background - 4 Stars, 1 Fork
Animated backgrounds on Windows (GIF frames)

django-auth-adfs - 3 Stars, 0 Fork
A Django authentication backend for ADFS

11 Feb 2016 7:40pm GMT

Mike Driscoll: Python Partials

Python comes with a fun module called functools. One of its classes is the partial class. You can use it create a new function with partial application of the arguments and keywords that you pass to it. You can use partial to "freeze" a portion of your function's arguments and/or keywords which results in a new object. Another way to put it is that partial creates a new function with some defaults. Let's look at an example!

>>> from functools import partial
>>> def add(x, y):
...     return x + y
... 
>>> p_add = partial(add, 2)
>>> p_add(4)
6

Here we create a simple adding function that returns the result of adding its arguments, x and y. Next we create a new callable by creating an instance of partial and passing it our function and an argument for that function. In other words, we are basically defaulting the x parameter of our add function to the number 2. Finally we call our new callable, p_add, with the argument of the number 4 which results in 6 because 2 + 4 = 6.

One handy use case for partials is to pass arguments to callbacks. Let's take a look using wxPython:

import wx
 
from functools import partial 
 
 
########################################################################
class MainFrame(wx.Frame):
    """
    This app shows a group of buttons
    """
 
    #----------------------------------------------------------------------
    def __init__(self, *args, **kwargs):
        """Constructor"""
        super(MainFrame, self).__init__(parent=None, title='Partial')
        panel = wx.Panel(self)
 
        sizer = wx.BoxSizer(wx.VERTICAL)
        btn_labels = ['one', 'two', 'three']
        for label in btn_labels:
            btn = wx.Button(panel, label=label)
            btn.Bind(wx.EVT_BUTTON, partial(self.onButton, label=label))
            sizer.Add(btn, 0, wx.ALL, 5)
 
        panel.SetSizer(sizer)
        self.Show()
 
    #----------------------------------------------------------------------
    def onButton(self, event, label):
        """
        Event handler called when a button is pressed
        """
        print 'You pressed: ', label
 
 
if __name__ == '__main__':
    app = wx.App(False)
    frame = MainFrame()
    app.MainLoop()

Here we use partial to call the onButton event handler with an extra argument, which happens to be the button's label. This might not seem all that useful to you, but if you do much GUI programming, you'll see a lot of people asking how to do this sort of thing. Of course, you could also use a lambda instead for passing arguments to callbacks.

One use case that we've used at work was for our automated test framework. We test a UI with Python and we wanted to be able to pass a function along to dismiss certain dialogs. Basically you would pass a function along with the name of the dialog to dismiss, but it would need to be called at a certain point in the process to work correctly. Since I can't show that code, here's a really basic example of passing a partial function around:

from functools import partial
 
#----------------------------------------------------------------------
def add(x, y):
    """"""
    return x + y
 
#----------------------------------------------------------------------
def multiply(x, y):
    """"""
    return x * y
 
#----------------------------------------------------------------------
def run(func):
    """"""
    print func()
 
#----------------------------------------------------------------------
def main():
    """"""
    a1 = partial(add, 1, 2)
    m1 = partial(multiply, 5, 8)
    run(a1)
    run(m1)
 
if __name__ == "__main__":
    main()

Here we create a couple of partial functions in our main function. Next we pass those partials to our run function, call it and then print out the result of the function that was called.


Wrapping Up

At this point, you should know how to use functools partial to create your own "frozen" callables. Partials have many uses, but they're not always obvious. I recommend that you just start experimenting with them and you might start seeing uses for your own code. Have fun!


Related Reading

11 Feb 2016 6:15pm GMT

Mike Driscoll: Python Partials

Python comes with a fun module called functools. One of its classes is the partial class. You can use it create a new function with partial application of the arguments and keywords that you pass to it. You can use partial to "freeze" a portion of your function's arguments and/or keywords which results in a new object. Another way to put it is that partial creates a new function with some defaults. Let's look at an example!

>>> from functools import partial
>>> def add(x, y):
...     return x + y
... 
>>> p_add = partial(add, 2)
>>> p_add(4)
6

Here we create a simple adding function that returns the result of adding its arguments, x and y. Next we create a new callable by creating an instance of partial and passing it our function and an argument for that function. In other words, we are basically defaulting the x parameter of our add function to the number 2. Finally we call our new callable, p_add, with the argument of the number 4 which results in 6 because 2 + 4 = 6.

One handy use case for partials is to pass arguments to callbacks. Let's take a look using wxPython:

import wx
 
from functools import partial 
 
 
########################################################################
class MainFrame(wx.Frame):
    """
    This app shows a group of buttons
    """
 
    #----------------------------------------------------------------------
    def __init__(self, *args, **kwargs):
        """Constructor"""
        super(MainFrame, self).__init__(parent=None, title='Partial')
        panel = wx.Panel(self)
 
        sizer = wx.BoxSizer(wx.VERTICAL)
        btn_labels = ['one', 'two', 'three']
        for label in btn_labels:
            btn = wx.Button(panel, label=label)
            btn.Bind(wx.EVT_BUTTON, partial(self.onButton, label=label))
            sizer.Add(btn, 0, wx.ALL, 5)
 
        panel.SetSizer(sizer)
        self.Show()
 
    #----------------------------------------------------------------------
    def onButton(self, event, label):
        """
        Event handler called when a button is pressed
        """
        print 'You pressed: ', label
 
 
if __name__ == '__main__':
    app = wx.App(False)
    frame = MainFrame()
    app.MainLoop()

Here we use partial to call the onButton event handler with an extra argument, which happens to be the button's label. This might not seem all that useful to you, but if you do much GUI programming, you'll see a lot of people asking how to do this sort of thing. Of course, you could also use a lambda instead for passing arguments to callbacks.

One use case that we've used at work was for our automated test framework. We test a UI with Python and we wanted to be able to pass a function along to dismiss certain dialogs. Basically you would pass a function along with the name of the dialog to dismiss, but it would need to be called at a certain point in the process to work correctly. Since I can't show that code, here's a really basic example of passing a partial function around:

from functools import partial
 
#----------------------------------------------------------------------
def add(x, y):
    """"""
    return x + y
 
#----------------------------------------------------------------------
def multiply(x, y):
    """"""
    return x * y
 
#----------------------------------------------------------------------
def run(func):
    """"""
    print func()
 
#----------------------------------------------------------------------
def main():
    """"""
    a1 = partial(add, 1, 2)
    m1 = partial(multiply, 5, 8)
    run(a1)
    run(m1)
 
if __name__ == "__main__":
    main()

Here we create a couple of partial functions in our main function. Next we pass those partials to our run function, call it and then print out the result of the function that was called.


Wrapping Up

At this point, you should know how to use functools partial to create your own "frozen" callables. Partials have many uses, but they're not always obvious. I recommend that you just start experimenting with them and you might start seeing uses for your own code. Have fun!


Related Reading

11 Feb 2016 6:15pm GMT

Nikola: Nikola v7.7.5 is out!

On behalf of the Nikola team, I am pleased to announce the immediate availability of Nikola v7.7.5. It fixes some bugs and adds new features.

What is Nikola?

Nikola is a static site and blog generator, written in Python. It can use Mako and Jinja2 templates, and input in many popular markup formats, such as reStructuredText and Markdown - and can even turn Jupyter (IPython) Notebooks into blog posts! It also supports image galleries, and is multilingual. Nikola is flexible, and page builds are extremely fast, courtesy of doit (which is rebuilding only what has been changed).

Find out more at the website: https://getnikola.com/

Downloads

Install using pip install Nikola or download tarballs on GitHub and PyPI.

Changes

Features

  • Add nikola theme --new command for creating new themes (Issue #2231)
  • Add nikola theme --copy-template command for copying templates to customize them (Issue #2231)
  • Add nikola theme --uninstall command for deleting themes (Issue #2231)
  • Replace nikola install_theme with more capable nikola theme command (Issue #2231)
  • Allow for customizing github_deploy commit messages with -m (Issue #2198)
  • Commit to source branch automatically in github_deploy if GITHUB_COMMIT_SOURCE is set to True (Issue #2186)
  • Hugo-like shortcodes (Issue #1707)
  • New Galician translation
  • New facilities for data persistence and data caching (Issues #2209 and #2009)
  • (internal) allow scripts/jinjify.py usage with scripts (Issue #2240)

Bugfixes

  • Fix some rebuilds with indexes and galleries
  • Make state files work on Python 3
  • Don't attempt to create redirects for URLs with query strings in WordPress imports if the site is in a subdirectory (Issue #2224)
  • Avoid some random file rebuilds (Issue #2220)
  • Honor MATHJAX_CONFIG setting
  • Display tags and archives in a unified format, with the date on the left, instead of a misplaced dash in tags (Issue #2212)
  • Decide is_mathjax based on current language tags (Issue #2205)
  • Don't duplicate images in flowr when resizing page (Issue #2202)

11 Feb 2016 5:55pm GMT

Nikola: Nikola v7.7.5 is out!

On behalf of the Nikola team, I am pleased to announce the immediate availability of Nikola v7.7.5. It fixes some bugs and adds new features.

What is Nikola?

Nikola is a static site and blog generator, written in Python. It can use Mako and Jinja2 templates, and input in many popular markup formats, such as reStructuredText and Markdown - and can even turn Jupyter (IPython) Notebooks into blog posts! It also supports image galleries, and is multilingual. Nikola is flexible, and page builds are extremely fast, courtesy of doit (which is rebuilding only what has been changed).

Find out more at the website: https://getnikola.com/

Downloads

Install using pip install Nikola or download tarballs on GitHub and PyPI.

Changes

Features

  • Add nikola theme --new command for creating new themes (Issue #2231)
  • Add nikola theme --copy-template command for copying templates to customize them (Issue #2231)
  • Add nikola theme --uninstall command for deleting themes (Issue #2231)
  • Replace nikola install_theme with more capable nikola theme command (Issue #2231)
  • Allow for customizing github_deploy commit messages with -m (Issue #2198)
  • Commit to source branch automatically in github_deploy if GITHUB_COMMIT_SOURCE is set to True (Issue #2186)
  • Hugo-like shortcodes (Issue #1707)
  • New Galician translation
  • New facilities for data persistence and data caching (Issues #2209 and #2009)
  • (internal) allow scripts/jinjify.py usage with scripts (Issue #2240)

Bugfixes

  • Fix some rebuilds with indexes and galleries
  • Make state files work on Python 3
  • Don't attempt to create redirects for URLs with query strings in WordPress imports if the site is in a subdirectory (Issue #2224)
  • Avoid some random file rebuilds (Issue #2220)
  • Honor MATHJAX_CONFIG setting
  • Display tags and archives in a unified format, with the date on the left, instead of a misplaced dash in tags (Issue #2212)
  • Decide is_mathjax based on current language tags (Issue #2205)
  • Don't duplicate images in flowr when resizing page (Issue #2202)

11 Feb 2016 5:55pm GMT

Reuven Lerner: All 50 “Practice Makes Python” screencasts are complete!

I'm delighted to announce that I've completed a screencast for every single one of the 50 exercises in my ebook, "Practice Makes Python." This is more than 300 minutes (5 hours!) of Python instruction, helping you to become a more expert Python programmer.

Each screencast consists of me solving one of the exercises in real-time, describing what I'm doing and what I'm doing it. They range in length from 4 to 10 minutes. The idea is that you'll do the exercise, and then watch my video to compare your answer (and approach) with mine.

If you enjoy my Webinars or in-person courses, then I think you'll also enjoy these videos.

The screencasts, available with the two higher-tier "Practice Makes Python" packages, can be streamed in HD video quality, or can be downloaded (DRM-free) to your computer for more convenient viewing.

To celebrate finally finishing these videos, I'm offering the two higher-end packages at 20% off for the coming week, until February 18th. Just use the offer code "videodone" with either the "consultant" or "developer" package, and enjoy a huge amount of Python video.

You can explore these packages at the "Practice Makes Python" Web site.

Not interested in my book, but still want to improve your Python skills? You can always take one of my two free e-mail courses, on Python variable scoping and working with files. Those are and will remain free forever. And of course, there's my free Webinar on Python and data science next week.

The post All 50 "Practice Makes Python" screencasts are complete! appeared first on Lerner Consulting Blog.

11 Feb 2016 2:50pm GMT

Reuven Lerner: All 50 “Practice Makes Python” screencasts are complete!

I'm delighted to announce that I've completed a screencast for every single one of the 50 exercises in my ebook, "Practice Makes Python." This is more than 300 minutes (5 hours!) of Python instruction, helping you to become a more expert Python programmer.

Each screencast consists of me solving one of the exercises in real-time, describing what I'm doing and what I'm doing it. They range in length from 4 to 10 minutes. The idea is that you'll do the exercise, and then watch my video to compare your answer (and approach) with mine.

If you enjoy my Webinars or in-person courses, then I think you'll also enjoy these videos.

The screencasts, available with the two higher-tier "Practice Makes Python" packages, can be streamed in HD video quality, or can be downloaded (DRM-free) to your computer for more convenient viewing.

To celebrate finally finishing these videos, I'm offering the two higher-end packages at 20% off for the coming week, until February 18th. Just use the offer code "videodone" with either the "consultant" or "developer" package, and enjoy a huge amount of Python video.

You can explore these packages at the "Practice Makes Python" Web site.

Not interested in my book, but still want to improve your Python skills? You can always take one of my two free e-mail courses, on Python variable scoping and working with files. Those are and will remain free forever. And of course, there's my free Webinar on Python and data science next week.

The post All 50 "Practice Makes Python" screencasts are complete! appeared first on Lerner Consulting Blog.

11 Feb 2016 2:50pm GMT

PyCharm: PyCharm 5.1 EAP is Open

Today we're extremely glad to announce that the first PyCharm 5.1 Early Access Preview (EAP) build 144.3891.17 is finally available for download! Head on over to the PyCharm Early Access Preview (EAP) page and have a look at the download section where you can get it for your platform right now.

What is the Early Access Program (EAP)?
We at JetBrains believe that making tools for developers should greatly involve listening to developers. Early Access Programs allow you to try pre-release versions of our software to evaluate features that will be added to the next release, closely participate in discussions about PyCharm, and influence the release from early stages onwards.

Once you've downloaded the latest EAP build (from its official page), you'll start getting updates with new features and improvements every week (to receive notifications about updates, make sure to set "Early Access Program" in your update settings). Your feedback is welcome and very much appreciated in our issue tracker or in the comments section of this blog post.

PyCharm 5.1 EAP build # 144.3891.17 introduces a lot of new features, bug-fixes and improvements. There are a number of recently added features that are unique to PyCharm, as well as additional features and improvements from the Web and IntelliJ Platform sides.

Python Related Improvements

Tox support

We're pleased to announce that PyCharm 5.1 EAP now supports Tox, the virtualenv management and testing tool. Simply right-click the tox.ini file in the project view and choose Run "Tox" or press Ctrl (Cmd) + Shift + F10. See the tests running and results nicely represented in the graphical test runner:

Tox

Devpi and Optional PyPI repositories support

Take full control over package repositories and PyPI mirroring solutions with support for optional repositories in PyCharm. To specify a custom repository (including devpi repos):

  1. Go to Settings (Preferences) | Project | Project Interpreter and click the "+" icon:

devpi1

2. In the Available Packages dialog box, click "Manage Repositories".
3. In the Manage Repositories dialog box, add, delete or edit the URLs of any repositories:

devpi2

Folding for Python code blocks, collection literals, and multi-line comments

In addition to the existing code folding functionality in PyCharm's code editor, we've implemented code folding for other code structure elements such as multi-line dictionaries, lists and comments, and if-else, for/while and try-except-finally code blocks in case a code block contains more than one statement:

folding

You can find additional options in Settings (Preferences) | Editor | General | Code folding ("Long string literals", "Long collection literals", "Sequential comments"). Read more about code folding options in PyCharm help.

Stepping over yield from and await

A small but neat improvement for developers who use Python 3, PyCharm's debugger is now able to step over yield from and await expressions. Put a breakpoint before a yield from or an await expression and press the Step over button (F8). The debugger will step to the next line after the expression:

yieldfrom

Debugger performance improvements

In addition to stepping over yield from and await expressions as described above, PyCharm's debugger receives a number of performance improvements that we hope you'll enjoy. One year before we released PyCharm 4, we joined efforts with Fabio Zadrozny, a PyDev maintainer, and merged our debugger code with that from the original PyDev repository. Now the work on the debugger continues from both sides (PyCharm and PyDev) in one place. As a result of the unification and the work done since then, the debugger in PyCharm has grown more powerful.

The very recent improvement that you can now see in PyCharm 5.1 EAP is that the debugger became much faster and now has optional Cython modules for even additional speedups. The debugger is reported to be now overall 40% faster without Cython and 138% faster with Cython in the benchmarks created. In real world cases, even better gains are expected as the benchmarks were done for the cases where the debugger has more overhead than usual. You can read more about the performance improvements and see some detailed benchmark statistics in the original blog post written by Fabio.

Web Development Improvements

As usual, PyCharm 5.1 will have all features of the upcoming WebStorm 12 release. WebStorm 12 preview builds are already available (read more in the WebStorm blog), so here is just a short summary of what's new and available in the PyCharm 5.1 preview build 144.3891.17 from the Web side:

Some of these features are already bundled with PyCharm 5.1 EAP, while others are available for installation from the JetBrains plugin repository.

Platform Improvements

In addition to the new features and improvements from the Web side, PyCharm 5.1 receives a lot of changes from the underlying IntelliJ Platform. The most noteworthy changes include:

PyCharm 5.1 runs on Java 8

Another important announcement is that the whole IntelliJ Platform has migrated to Java 8. That means that now you can not launch PyCharm under a JDK older than Java 8. This change affects all the EAP builds (144.*) and further major releases this upcoming spring.

Download PyCharm 5.1 preview build 144.3891.17 for your platform right from the project EAP page, and please report any bugs and feature request to our Issue Tracker.

Note: The PyCharm 5.1 preview build is available in two editions - free/open source PyCharm Community Edition and full-fledged PyCharm Professional Edition. While both of them are available for download and use at no charge, PyCharm Professional Edition has a 30-day time-limited evaluation period.

PyCharm Team
The Drive to Develop

11 Feb 2016 2:04pm GMT

PyCharm: PyCharm 5.1 EAP is Open

Today we're extremely glad to announce that the first PyCharm 5.1 Early Access Preview (EAP) build 144.3891.17 is finally available for download! Head on over to the PyCharm Early Access Preview (EAP) page and have a look at the download section where you can get it for your platform right now.

What is the Early Access Program (EAP)?
We at JetBrains believe that making tools for developers should greatly involve listening to developers. Early Access Programs allow you to try pre-release versions of our software to evaluate features that will be added to the next release, closely participate in discussions about PyCharm, and influence the release from early stages onwards.

Once you've downloaded the latest EAP build (from its official page), you'll start getting updates with new features and improvements every week (to receive notifications about updates, make sure to set "Early Access Program" in your update settings). Your feedback is welcome and very much appreciated in our issue tracker or in the comments section of this blog post.

PyCharm 5.1 EAP build # 144.3891.17 introduces a lot of new features, bug-fixes and improvements. There are a number of recently added features that are unique to PyCharm, as well as additional features and improvements from the Web and IntelliJ Platform sides.

Python Related Improvements

Tox support

We're pleased to announce that PyCharm 5.1 EAP now supports Tox, the virtualenv management and testing tool. Simply right-click the tox.ini file in the project view and choose Run "Tox" or press Ctrl (Cmd) + Shift + F10. See the tests running and results nicely represented in the graphical test runner:

Tox

Devpi and Optional PyPI repositories support

Take full control over package repositories and PyPI mirroring solutions with support for optional repositories in PyCharm. To specify a custom repository (including devpi repos):

  1. Go to Settings (Preferences) | Project | Project Interpreter and click the "+" icon:

devpi1

2. In the Available Packages dialog box, click "Manage Repositories".
3. In the Manage Repositories dialog box, add, delete or edit the URLs of any repositories:

devpi2

Folding for Python code blocks, collection literals, and multi-line comments

In addition to the existing code folding functionality in PyCharm's code editor, we've implemented code folding for other code structure elements such as multi-line dictionaries, lists and comments, and if-else, for/while and try-except-finally code blocks in case a code block contains more than one statement:

folding

You can find additional options in Settings (Preferences) | Editor | General | Code folding ("Long string literals", "Long collection literals", "Sequential comments"). Read more about code folding options in PyCharm help.

Stepping over yield from and await

A small but neat improvement for developers who use Python 3, PyCharm's debugger is now able to step over yield from and await expressions. Put a breakpoint before a yield from or an await expression and press the Step over button (F8). The debugger will step to the next line after the expression:

yieldfrom

Debugger performance improvements

In addition to stepping over yield from and await expressions as described above, PyCharm's debugger receives a number of performance improvements that we hope you'll enjoy. One year before we released PyCharm 4, we joined efforts with Fabio Zadrozny, a PyDev maintainer, and merged our debugger code with that from the original PyDev repository. Now the work on the debugger continues from both sides (PyCharm and PyDev) in one place. As a result of the unification and the work done since then, the debugger in PyCharm has grown more powerful.

The very recent improvement that you can now see in PyCharm 5.1 EAP is that the debugger became much faster and now has optional Cython modules for even additional speedups. The debugger is reported to be now overall 40% faster without Cython and 138% faster with Cython in the benchmarks created. In real world cases, even better gains are expected as the benchmarks were done for the cases where the debugger has more overhead than usual. You can read more about the performance improvements and see some detailed benchmark statistics in the original blog post written by Fabio.

Web Development Improvements

As usual, PyCharm 5.1 will have all features of the upcoming WebStorm 12 release. WebStorm 12 preview builds are already available (read more in the WebStorm blog), so here is just a short summary of what's new and available in the PyCharm 5.1 preview build 144.3891.17 from the Web side:

Some of these features are already bundled with PyCharm 5.1 EAP, while others are available for installation from the JetBrains plugin repository.

Platform Improvements

In addition to the new features and improvements from the Web side, PyCharm 5.1 receives a lot of changes from the underlying IntelliJ Platform. The most noteworthy changes include:

PyCharm 5.1 runs on Java 8

Another important announcement is that the whole IntelliJ Platform has migrated to Java 8. That means that now you can not launch PyCharm under a JDK older than Java 8. This change affects all the EAP builds (144.*) and further major releases this upcoming spring.

Download PyCharm 5.1 preview build 144.3891.17 for your platform right from the project EAP page, and please report any bugs and feature request to our Issue Tracker.

Note: The PyCharm 5.1 preview build is available in two editions - free/open source PyCharm Community Edition and full-fledged PyCharm Professional Edition. While both of them are available for download and use at no charge, PyCharm Professional Edition has a 30-day time-limited evaluation period.

PyCharm Team
The Drive to Develop

11 Feb 2016 2:04pm GMT

PythonClub - A Brazilian collaborative blog about Python: Salvando gráfico de contribuições do Github com Python e Selenium

Como alguns sabem, sou apaixonado por Python. Atualmente, minha linguagem favorita por conta de sua simplicade e poder (além de ficar LINDJA toda indentada, hahahaha).

Uma das coisas mais legais da linguagem é a enorme quantidade de bibliotecas disponíveis. Cada dia que abro um grupo de discussão acabo conhecendo alguma funcionalidade interessante. Se você faz parte de algum desses grupos, provavelmente já viu o post do Alex Recker "Using Selenium to Buy a Bus Pass", em que ele mostra como automatizou a compra de passagens de ônibus com Selenium e Python.

Eu já havia ouvido falar do Selenium, mas nunca tinha experimentado na prática e o post do Alex foi o empurrão que faltava.

Obviamente, meu projetinho é bem mais simples, mas foi algo divertido de se fazer como forma de aprendizado. Batizei-o de GHSS(Github Screenshot). Como o próprio nome sugere, ele entra no seu perfil do Github e tira um screenshot do gráfico de contribuições, salvando com a data atual.

Abaixo, irei mostrar como fazer. Visto que há muita gente que usa Python sem ser programador por profissão, tentarei explicar de forma mais simples possível. O código completo pode ser encontrado no meu Github.


Neste código, utilizaremos o Python2.

Primeiramente, temos que importar todas as bibliotecas necessárias.

Na linha 1, importamos o "OS", que será utilizado para "acharmos" o arquivo secrets.yml. Explicarei daqui a pouco.

Na linha 2, importamos do Selenium o Webdriver, responsável pela automatização (abertura das páginas e preenchimento dos campos).

Nas próximas duas linhas, importamos as bibliotecas restantes que são responsáveis pelo nosso arquivo secrets.yml, no qual o username e password serão guardados, e pela data que será salva no nome do arquivo final.

Na última linha, importamos o responsável por tirar o screenshot.

import os
from selenium import webdriver
import yaml
from datetime import date
import pyscreenshot as ImageGrab

Neste bloco de código, mostramos ao nosso programa onde está nosso arquivo secrets.yml e o carregamos.

cur_dir = os.path.dirname(os.path.realpath(__file__))
secret_path = os.path.join(cur_dir, 'secrets.yml')

with open(secret_path, 'r') as stream:
    data = yaml.load(stream)
    USERNAME = data.get('user','')
    PASSWORD = data.get('password')

O arquivo secrets.yml é composto por apenas dois campos, "password" e "user", que, PASMEM, são para inserir sua senha e seu usuário.

password: senha_do_zezinho_hacker
user: zezinhohacker123

Nestas três linhas abrimos o Firefox, passamos para ele qual o endereço desejamos acessar e maximizamos a janela, respectivamente.

driver = webdriver.Firefox()
driver.get("https://github.com/login")
driver.maximize_window()

Aqui é onde a "mágica" acontece.

Na primeira linha, a propriedade "find_element_by_id" busca o campo "login_field", onde devemos inserir o nome de usuário. Na linha posterior, enviamos aquele username informado lá no secrets, lembra?

Nas próximas duas linhas, é feito o mesmo procedimento, mas, desta vez, com a senha.

Na última, clicamos o botão para logarmos.

email = driver.find_element_by_id("login_field")
email.send_keys(USERNAME)
senha = driver.find_element_by_id("password")
senha.send_keys(PASSWORD)
driver.find_element_by_name('commit').click()

Nesta linha, nós entramos no nosso perfil do Github.

Quando utilizamos {0}, "guardamos" o espaço, para o que informarmos adiante. Ou seja, no espaço reservado, será inserido o username.

driver.get("https://github.com/{0}" .format(USERNAME))

Por exemplo, se fizermos o seguinte código:

print("Meus esportes preferidos são: {0}, {1} e {2}" .format("futebol", "basquete", "corrida"))

O resultado será:

 Meus esportes preferidos são: futebol, basquete e corrida.

Deu para entender?


Na última linha do programa, salvamos a imagem.

No campo bbox, informamos qual área da tela queremos dar o screenshot, na ordem: X1, Y1, X2, Y2. Você pode alterá-lo de acordo com seu navegador.

No save, utilizamos o que ensinei acima para gerar o arquivo da seguinte maneira: "dataatual_gitshot_nomedousuario".

img = ImageGrab.grab(bbox=(460,540,770,208)).save("{0}_gitshot_{1}.png" .format(date.today(), USERNAME))

Este será o resultado. O nome do arquivo, no meu caso, ficou "2016-01-24_gitshot_othonalberto.png".

Resultado


Código completo:

#!/usr/bin/env python
# -*- coding: utf-8 -*-
import os

from selenium import webdriver
import yaml
from datetime import date
import pyscreenshot as ImageGrab

cur_dir = os.path.dirname(os.path.realpath(__file__))
secret_path = os.path.join(cur_dir, 'secrets.yml')

with open(secret_path, 'r') as stream:
    data = yaml.load(stream)
    USERNAME = data.get('user','')
    PASSWORD = data.get('password')

driver = webdriver.Firefox()
driver.get("https://github.com/login")
driver.maximize_window()

email = driver.find_element_by_id("login_field")
email.send_keys(USERNAME)
senha = driver.find_element_by_id("password")
senha.send_keys(PASSWORD)
driver.find_element_by_name('commit').click()

driver.get("https://github.com/{0}" .format(USERNAME))

img = ImageGrab.grab(bbox=(460,540,770,208)).save("{0}_gitshot_{1}.png" .format(date.today(), USERNAME))
# bbox=(X1, Y1, X2, Y2)

É isso! Espero ter contribuído com o conhecimento de vocês com este post e gerado curiosidade para que experimentem o Selenium.

Quem quiser contribuir, seja com código ou sugestões, sinta-se à vontade.

Abraços!

11 Feb 2016 1:47pm GMT

PythonClub - A Brazilian collaborative blog about Python: Salvando gráfico de contribuições do Github com Python e Selenium

Como alguns sabem, sou apaixonado por Python. Atualmente, minha linguagem favorita por conta de sua simplicade e poder (além de ficar LINDJA toda indentada, hahahaha).

Uma das coisas mais legais da linguagem é a enorme quantidade de bibliotecas disponíveis. Cada dia que abro um grupo de discussão acabo conhecendo alguma funcionalidade interessante. Se você faz parte de algum desses grupos, provavelmente já viu o post do Alex Recker "Using Selenium to Buy a Bus Pass", em que ele mostra como automatizou a compra de passagens de ônibus com Selenium e Python.

Eu já havia ouvido falar do Selenium, mas nunca tinha experimentado na prática e o post do Alex foi o empurrão que faltava.

Obviamente, meu projetinho é bem mais simples, mas foi algo divertido de se fazer como forma de aprendizado. Batizei-o de GHSS(Github Screenshot). Como o próprio nome sugere, ele entra no seu perfil do Github e tira um screenshot do gráfico de contribuições, salvando com a data atual.

Abaixo, irei mostrar como fazer. Visto que há muita gente que usa Python sem ser programador por profissão, tentarei explicar de forma mais simples possível. O código completo pode ser encontrado no meu Github.


Neste código, utilizaremos o Python2.

Primeiramente, temos que importar todas as bibliotecas necessárias.

Na linha 1, importamos o "OS", que será utilizado para "acharmos" o arquivo secrets.yml. Explicarei daqui a pouco.

Na linha 2, importamos do Selenium o Webdriver, responsável pela automatização (abertura das páginas e preenchimento dos campos).

Nas próximas duas linhas, importamos as bibliotecas restantes que são responsáveis pelo nosso arquivo secrets.yml, no qual o username e password serão guardados, e pela data que será salva no nome do arquivo final.

Na última linha, importamos o responsável por tirar o screenshot.

import os
from selenium import webdriver
import yaml
from datetime import date
import pyscreenshot as ImageGrab

Neste bloco de código, mostramos ao nosso programa onde está nosso arquivo secrets.yml e o carregamos.

cur_dir = os.path.dirname(os.path.realpath(__file__))
secret_path = os.path.join(cur_dir, 'secrets.yml')

with open(secret_path, 'r') as stream:
    data = yaml.load(stream)
    USERNAME = data.get('user','')
    PASSWORD = data.get('password')

O arquivo secrets.yml é composto por apenas dois campos, "password" e "user", que, PASMEM, são para inserir sua senha e seu usuário.

password: senha_do_zezinho_hacker
user: zezinhohacker123

Nestas três linhas abrimos o Firefox, passamos para ele qual o endereço desejamos acessar e maximizamos a janela, respectivamente.

driver = webdriver.Firefox()
driver.get("https://github.com/login")
driver.maximize_window()

Aqui é onde a "mágica" acontece.

Na primeira linha, a propriedade "find_element_by_id" busca o campo "login_field", onde devemos inserir o nome de usuário. Na linha posterior, enviamos aquele username informado lá no secrets, lembra?

Nas próximas duas linhas, é feito o mesmo procedimento, mas, desta vez, com a senha.

Na última, clicamos o botão para logarmos.

email = driver.find_element_by_id("login_field")
email.send_keys(USERNAME)
senha = driver.find_element_by_id("password")
senha.send_keys(PASSWORD)
driver.find_element_by_name('commit').click()

Nesta linha, nós entramos no nosso perfil do Github.

Quando utilizamos {0}, "guardamos" o espaço, para o que informarmos adiante. Ou seja, no espaço reservado, será inserido o username.

driver.get("https://github.com/{0}" .format(USERNAME))

Por exemplo, se fizermos o seguinte código:

print("Meus esportes preferidos são: {0}, {1} e {2}" .format("futebol", "basquete", "corrida"))

O resultado será:

 Meus esportes preferidos são: futebol, basquete e corrida.

Deu para entender?


Na última linha do programa, salvamos a imagem.

No campo bbox, informamos qual área da tela queremos dar o screenshot, na ordem: X1, Y1, X2, Y2. Você pode alterá-lo de acordo com seu navegador.

No save, utilizamos o que ensinei acima para gerar o arquivo da seguinte maneira: "dataatual_gitshot_nomedousuario".

img = ImageGrab.grab(bbox=(460,540,770,208)).save("{0}_gitshot_{1}.png" .format(date.today(), USERNAME))

Este será o resultado. O nome do arquivo, no meu caso, ficou "2016-01-24_gitshot_othonalberto.png".

Resultado


Código completo:

#!/usr/bin/env python
# -*- coding: utf-8 -*-
import os

from selenium import webdriver
import yaml
from datetime import date
import pyscreenshot as ImageGrab

cur_dir = os.path.dirname(os.path.realpath(__file__))
secret_path = os.path.join(cur_dir, 'secrets.yml')

with open(secret_path, 'r') as stream:
    data = yaml.load(stream)
    USERNAME = data.get('user','')
    PASSWORD = data.get('password')

driver = webdriver.Firefox()
driver.get("https://github.com/login")
driver.maximize_window()

email = driver.find_element_by_id("login_field")
email.send_keys(USERNAME)
senha = driver.find_element_by_id("password")
senha.send_keys(PASSWORD)
driver.find_element_by_name('commit').click()

driver.get("https://github.com/{0}" .format(USERNAME))

img = ImageGrab.grab(bbox=(460,540,770,208)).save("{0}_gitshot_{1}.png" .format(date.today(), USERNAME))
# bbox=(X1, Y1, X2, Y2)

É isso! Espero ter contribuído com o conhecimento de vocês com este post e gerado curiosidade para que experimentem o Selenium.

Quem quiser contribuir, seja com código ou sugestões, sinta-se à vontade.

Abraços!

11 Feb 2016 1:47pm GMT

Robin Wilson: Hacking the Worcester Wave thermostat in Python – Part 2

In the previous part we had established that the Worcester Wave thermostat app communicates with a remote server (run by Worcester Bosch) using the XMPP protocol, with TLS encryption. However, because of the encryption we haven't yet managed to see the actual content of any of these messages!

To decrypt the messages we need to do a man in the middle attack. In this sort of attack we put insert ourselves between the two ends of the conversation and intercept messages, decrypt them and then forward them on to the original destination. Although it is called an 'attack', here I am basically attacking myself - because I'm basically monitoring communications going from a phone I own through a network I own. Beware that doing this to networks that you do not own or have permission to use in this way is almost certainly illegal.

There is a good guide to setting all of this up using a tool called sslsplit, although I had to do things slightly differently as I couldn't get sslsplit to work with the STARTTLS method used by the Worcester Wave (as you may remember from the previous part, STARTTLS is a way of starting the communication in an unencrypted manner, and then 'turning on' the encryption part-way through the communication).

The summary of the approach that I used is:

  1. I configured a Linux server on my network to use Network Address Translation (NAT) - so it works almost like a router. This means that I can then set up another device on the network to use that server as a 'gateway', which means it will send all traffic to that server, which will then forward it on to the correct place.
  2. I created a self-signed root certificate on the server. A root certificate is a 'fully trusted' certificate that can be used to trust any other certificates or keys derived from it (that explanation is probably technically wrong, but it's conceptually right).
  3. I installed this root certificate on a spare Android phone, connected the phone to my home wifi and configured the Linux server as the gateway. I then tested, and could access the internet fine from the phone, with all of the communications going through the server.

Now, if I use the Worcester Wave app on the phone, all of the communications will go through the server - and the phone will believe the server when it says that it is the Bosch server at the other end, because of the root certificate we installed.

Now we've got all of the certificates and networking stuff configured, we just need to actually decrypt the messages. As I said above, I tried using SSLSplit, but it couldn't seem to cope with STARTTLS. I found the same with Wireshark itself, so looked for another option.

Luckily, I found a great tool called starttls-mitm which does work with STARTTLS. Even more impressively it's barely 80 lines of Python, and so it's very easy to understand the code. This does mean that it is less configurable than a more complex tool like SSLSplit - but that's not a problem for me, as the tool does exactly what I want . It is even configured for XMPP by default too! (Of course, as it is written in Python I could always modify the code myself if I needed to).

So, running starttls-mitm with the appropriate command-line parameters (basically your keys, certificates etc) will print out all communications: both the unencrypted ones before the STARTTLS call, and the decrypted version of the encrypted ones after the STARTTLS call. If we then start doing something with the app while this is running, what do we get?

Well, first we get the opening logging information starttls-mitm telling us what it is doing:

LISTENER ready on port 8443
CLIENT CONNECT from: ('192.168.0.42', 57913)
RELAYING

We then start getting the beginnings of the communication:

C->S 129 '<stream:stream to="wa2-mz36-qrmzh6.bosch.de" xmlns="jabber:client" xmlns:stream="http://etherx.jabber.org/streams" version="1.0">'
S->C 442 '<?xml version=\'1.0\' encoding=\'UTF-8\'?><stream:stream xmlns:stream="http://etherx.jabber.org/streams" xmlns="jabber:client" from="wa2-mz36-qrmzh6.bosch.de" id="260d2859" xml:lang="en" version="1.0"><stream:features><starttls xmlns="urn:ietf:params:xml:ns:xmpp-tls"></starttls><mechanisms xmlns="urn:ietf:params:xml:ns:xmpp-sasl"><mechanism>DIGEST-MD5</mechanism></mechanisms><auth xmlns="http://jabber.org/features/iq-auth"/></stream:features>'

Fairly obviously, C->S are messages from the client to the server, and S->C are messages back from the server (the numbers directly afterwards are just the length of the message). These are just the initial handshaking communications for the start of XMPP communication, and aren't particularly exciting as we saw this in Wireshark too.

However, now we get to the interesting bit:

C->S 51 '<starttls xmlns="urn:ietf:params:xml:ns:xmpp-tls"/>'
S->C 50 '<proceed xmlns="urn:ietf:params:xml:ns:xmpp-tls"/>'
Wrapping sockets.

The STARTTLS message is sent, and the server says PROCEED, and - crucially - starttls-mitm notices this and announces that it is 'wrapping sockets' (basically enabling the decryption of the communication from this point onwards).

I'll skip the boring TLS handshaking messages now, and skip to the initialisation of the XMPP protocol itself. I'm no huge XMPP expert, but basically the iq messages are 'info/query' messages which are part of the handshaking process with each side saying who they are, what they support etc. This part of the communication finishes with each side announcing its 'presence' (remember, XMPP is originally a chat protocol, so this is the equivalent of saying you are 'Online' or 'Active' on Skype, Facebook Messenger or whatever).

C->S 110 '<iq id="lj8Vq-1" type="set"><bind xmlns="urn:ietf:params:xml:ns:xmpp-bind"><resource>70</resource></bind></iq>'
S->C 188 '<iq type="result" id="lj8Vq-1" to="wa2-mz36-qrmzh6.bosch.de/260d2859"><bind xmlns="urn:ietf:params:xml:ns:xmpp-bind"><jid>rrccontact_458921440@wa2-mz36-qrmzh6.bosch.de/70</jid></bind></iq>'
C->S 87 '<iq id="lj8Vq-2" type="set"><session xmlns="urn:ietf:params:xml:ns:xmpp-session"/></iq>'
S->C 86 '<iq type="result" id="lj8Vq-2" to="rrccontact_458921440@wa2-mz36-qrmzh6.bosch.de/70"/>'
C->S 74 '<iq id="lj8Vq-3" type="get"><query xmlns="jabber:iq:roster" ></query></iq>'
S->C 123 '<iq type="result" id="lj8Vq-3" to="rrccontact_458921440@wa2-mz36-qrmzh6.bosch.de/70"><query xmlns="jabber:iq:roster"/></iq>'
C->S 34 '<presence id="lj8Vq-4"></presence>'
C->S 34 '<presence id="lj8Vq-5"></presence>'

Now all of the preliminary messages are dealt with, we get to the good bit. The message below is sent from the client (the phone app) to the server:

C->S 162 '<message id="lj8Vq-6" to="rrcgateway_458921440@wa2-mz36-qrmzh6.bosch.de" type="chat"><body>GET /ecus/rrc/uiStatus HTTP /1.0\nUser-Agent: NefitEasy</body></message>'

It basically seems to be a HTTP GET request embedded within an XMPP message. That seems a bit strange to me - why not just use HTTP directly? - but at least it is easy to understand. The URL that is being requested also makes sense - I was on the 'home screen' of the app at that point, so it was grabbing the status for displaying in the user-interface (things like current temperature, set-point temperature, whether the boiler is on or not, etc).

Now we can see the response from the server:

S->C 904 '<message to="rrccontact_458921440@wa2-mz36-qrmzh6.bosch.de/70" type="chat" xml:lang="en" from="rrcgateway_458921440@wa2-mz36-qrmzh6.bosch.de/RRC-RestApi"><body>HTTP/1.0 200 OK\nContent-Length: 640\nContent-Type: application/json\nconnection: close\n\n5EBW5RuFo7QojD4F1Uv0kOde1MbeVA46P3RDX6ZEYKaKkbLxanqVR2I8ceuQNbxkgkfzeLgg6D5ypF9jo7yGVRbR/ydf4L4MMTHxvdxBubG5HhiVqJgSc2+7iPvhcWvRZrRKBEMiz8vAsd5JleS4CoTmbN0vV7kHgO2uVeuxtN5ZDsk3/cZpxiTvvaXWlCQGOavCLe55yQqmm3zpGoNFolGPTNC1MVuk00wpf6nbS7sFaRXSmpGQeGAfGNxSxfVPhWZtWRP3/ETi1Z+ozspBO8JZRAzeP8j0fJrBe9u+kDQJNXiMkgzyWb6Il6roSBWWgwYuepGYf/dSR9YygF6lrV+iQdZdyF08ZIgcNY5g5XWtm4LdH8SO+TZpP9aocLUVR1pmFM6m19MKP+spMg8gwPm6L9YuWSvd62KA8ASIQMtWbzFB6XjanGBQpVeMLI1Uzx4wWRaRaAG5qLTda9PpGk8K6LWOxHwtsuW/CDST/hE5jXvWqfVmrceUVqHz5Qcb0sjKRU5TOYA+JNigSf0Z4CIh7xD1t7bjJf9m6Wcyys/NkwZYryoQm99J2yH2khWXyd2DRETbsynr1AWrSRlStZ5H9ghPoYTqvKvgWsyMVTxbMOht86CzoufceI2W+Rr9</body></message>'

Oh. This looks a bit more complicated, and not very easily interpretable. Lets format the main body of the message formatted a bit more nicely:

HTTP/1.0 200 OK
Content-Length: 640
Content-Type: application/json
connection: close

5EBW5RuFo7QojD4F1Uv0kOde1MbeVA46P3RDX6ZEYKaKkbLxanqVR2I8ceuQNbxkgkfzeLgg6D5ypF9jo7yGVRbR/ydf4L4MMTHxvdxBubG5HhiVqJgSc2+7iPvhcWvRZrRKBEMiz8vAsd5JleS4CoTmbN0vV7kHgO2uVeuxtN5ZDsk3/cZpxiTvvaXWlCQGOavCLe55yQqmm3zpGoNFolGPTNC1MVuk00wpf6nbS7sFaRXSmpGQeGAfGNxSxfVPhWZtWRP3/ETi1Z+ozspBO8JZRAzeP8j0fJrBe9u+kDQJNXiMkgzyWb6Il6roSBWWgwYuepGYf/dSR9YygF6lrV+iQdZdyF08ZIgcNY5g5XWtm4LdH8SO+TZpP9aocLUVR1pmFM6m19MKP+spMg8gwPm6L9YuWSvd62KA8ASIQMtWbzFB6XjanGBQpVeMLI1Uzx4wWRaRaAG5qLTda9PpGk8K6LWOxHwtsuW/CDST/hE5jXvWqfVmrceUVqHz5Qcb0sjKRU5TOYA+JNigSf0Z4CIh7xD1t7bjJf9m6Wcyys/NkwZYryoQm99J2yH2khWXyd2DRETbsynr1AWrSRlStZ5H9ghPoYTqvKvgWsyMVTxbMOht86CzoufceI2W+Rr9

So, it seems to be a standard HTTP response (200 OK), but the body looks like it is encoded somehow. I assume that the decoded body would be something like JSON or XML or something containing the various status values - but how do we decode it to get that?

I tried all sorts of things like Base64, MD5 and so on but nothing seemed to work. I gave up on this for a few days, while gently pondering it in the back of my mind. When I came back to it, I realised that the data here was probably actually encrypted, using the Access Code that comes with the Wave and the password that you set up when you first connect the Wave. Of course, to decrypt it we need to know how it was encrypted…so time to break out the next tool: a decompiler.

Yes, that's right: to fully understand exactly what the Wave app is doing, I needed to decompile the Android app's APK file and look at the code. I did this using the aptly named Android APK Decompiler, and got surprisingly readable Java code out of it! (I mean, it had a lot of goto statements, but at least the variables had sensible names!)

It's difficult to explain the full details of the encryption/decryption algorithm in prose - so I've included the Python code I implemented to do this below. However, a brief summary is that: the main encryption is AES using ECB, with keys generated from the MD5 sums of combinations of the Access Code, the password and a 'secret' (a value hard-coded into the app).

def encode(s):
    abyte1 = get_md5(access + secret)
    abyte2 = get_md5(secret + password)

    key = abyte1 + abyte2

    a = AES.new(key)
    a = AES.new(key, AES.MODE_ECB)
    res = a.encrypt(s)

    encoded = base64.b64encode(res)

    return encoded

def decode(data):
    decoded = base64.b64decode(data)

    abyte1 = get_md5(access + secret)
    abyte2 = get_md5(secret + password)

    key = abyte1 + abyte2

    a = AES.new(key)
    a = AES.new(key, AES.MODE_ECB)
    res = a.decrypt(decoded)

    return res

Using these functions we can decrypt the response to the GET /ecus/rrc/uiStatus message that we saw earlier, and we get this:

{'id': '/ecus/rrc/uiStatus',
 'recordable': 0,
 'type': 'uiUpdate',
 'value': {'ARS': 'init',
  'BAI': 'CH',
  'BBE': 'false',
  'BLE': 'false',
  'BMR': 'false',
  'CPM': 'auto',
  'CSP': '31',
  'CTD': '2014-12-26T12:34:27+00:00 Fr',
  'CTR': 'room',
  'DAS': 'off',
  'DHW': 'on',
  'ESI': 'off',
  'FPA': 'off',
  'HED_DB': '',
  'HED_DEV': 'false',
  'HED_EN': 'false',
  'HMD': 'off',
  'IHS': 'ok',
  'IHT': '16.70',
  'MMT': '15.5',
  'PMR': 'false',
  'RS': 'off',
  'TAS': 'off',
  'TOD': '0',
  'TOR': 'on',
  'TOT': '17.0',
  'TSP': '17.0',
  'UMD': 'clock'},
 'writeable': 0}

This makes far more sense!

It may not be immediately apparent what each field is (three character variable names - great!), but some of them are fairly obvious (CTD presumably stands for something like Current Time/Date), or can be established by decoding a number of messages with the boiler in different states (showing that DHW stands for Domestic Hot Water and BAI for Burner Active Indicator).

We've made a lot of progress in the second part of this guide: we've now decrypted the communications, and worked out how to get all of the status information that is shown on the app home screen. At this point I set up a simple temperature monitoring system to produce nice graphs of temperature over time - but I'll leave the description of that to later in the series. In the next part we're going to look at sending messages to actually change the state of the thermostat (such as setting a new temperature, or switching to manual mode), and then have a look at the Python library I've written to control the thermostat.

11 Feb 2016 12:02pm GMT

Robin Wilson: Hacking the Worcester Wave thermostat in Python – Part 2

In the previous part we had established that the Worcester Wave thermostat app communicates with a remote server (run by Worcester Bosch) using the XMPP protocol, with TLS encryption. However, because of the encryption we haven't yet managed to see the actual content of any of these messages!

To decrypt the messages we need to do a man in the middle attack. In this sort of attack we put insert ourselves between the two ends of the conversation and intercept messages, decrypt them and then forward them on to the original destination. Although it is called an 'attack', here I am basically attacking myself - because I'm basically monitoring communications going from a phone I own through a network I own. Beware that doing this to networks that you do not own or have permission to use in this way is almost certainly illegal.

There is a good guide to setting all of this up using a tool called sslsplit, although I had to do things slightly differently as I couldn't get sslsplit to work with the STARTTLS method used by the Worcester Wave (as you may remember from the previous part, STARTTLS is a way of starting the communication in an unencrypted manner, and then 'turning on' the encryption part-way through the communication).

The summary of the approach that I used is:

  1. I configured a Linux server on my network to use Network Address Translation (NAT) - so it works almost like a router. This means that I can then set up another device on the network to use that server as a 'gateway', which means it will send all traffic to that server, which will then forward it on to the correct place.
  2. I created a self-signed root certificate on the server. A root certificate is a 'fully trusted' certificate that can be used to trust any other certificates or keys derived from it (that explanation is probably technically wrong, but it's conceptually right).
  3. I installed this root certificate on a spare Android phone, connected the phone to my home wifi and configured the Linux server as the gateway. I then tested, and could access the internet fine from the phone, with all of the communications going through the server.

Now, if I use the Worcester Wave app on the phone, all of the communications will go through the server - and the phone will believe the server when it says that it is the Bosch server at the other end, because of the root certificate we installed.

Now we've got all of the certificates and networking stuff configured, we just need to actually decrypt the messages. As I said above, I tried using SSLSplit, but it couldn't seem to cope with STARTTLS. I found the same with Wireshark itself, so looked for another option.

Luckily, I found a great tool called starttls-mitm which does work with STARTTLS. Even more impressively it's barely 80 lines of Python, and so it's very easy to understand the code. This does mean that it is less configurable than a more complex tool like SSLSplit - but that's not a problem for me, as the tool does exactly what I want . It is even configured for XMPP by default too! (Of course, as it is written in Python I could always modify the code myself if I needed to).

So, running starttls-mitm with the appropriate command-line parameters (basically your keys, certificates etc) will print out all communications: both the unencrypted ones before the STARTTLS call, and the decrypted version of the encrypted ones after the STARTTLS call. If we then start doing something with the app while this is running, what do we get?

Well, first we get the opening logging information starttls-mitm telling us what it is doing:

LISTENER ready on port 8443
CLIENT CONNECT from: ('192.168.0.42', 57913)
RELAYING

We then start getting the beginnings of the communication:

C->S 129 '<stream:stream to="wa2-mz36-qrmzh6.bosch.de" xmlns="jabber:client" xmlns:stream="http://etherx.jabber.org/streams" version="1.0">'
S->C 442 '<?xml version=\'1.0\' encoding=\'UTF-8\'?><stream:stream xmlns:stream="http://etherx.jabber.org/streams" xmlns="jabber:client" from="wa2-mz36-qrmzh6.bosch.de" id="260d2859" xml:lang="en" version="1.0"><stream:features><starttls xmlns="urn:ietf:params:xml:ns:xmpp-tls"></starttls><mechanisms xmlns="urn:ietf:params:xml:ns:xmpp-sasl"><mechanism>DIGEST-MD5</mechanism></mechanisms><auth xmlns="http://jabber.org/features/iq-auth"/></stream:features>'

Fairly obviously, C->S are messages from the client to the server, and S->C are messages back from the server (the numbers directly afterwards are just the length of the message). These are just the initial handshaking communications for the start of XMPP communication, and aren't particularly exciting as we saw this in Wireshark too.

However, now we get to the interesting bit:

C->S 51 '<starttls xmlns="urn:ietf:params:xml:ns:xmpp-tls"/>'
S->C 50 '<proceed xmlns="urn:ietf:params:xml:ns:xmpp-tls"/>'
Wrapping sockets.

The STARTTLS message is sent, and the server says PROCEED, and - crucially - starttls-mitm notices this and announces that it is 'wrapping sockets' (basically enabling the decryption of the communication from this point onwards).

I'll skip the boring TLS handshaking messages now, and skip to the initialisation of the XMPP protocol itself. I'm no huge XMPP expert, but basically the iq messages are 'info/query' messages which are part of the handshaking process with each side saying who they are, what they support etc. This part of the communication finishes with each side announcing its 'presence' (remember, XMPP is originally a chat protocol, so this is the equivalent of saying you are 'Online' or 'Active' on Skype, Facebook Messenger or whatever).

C->S 110 '<iq id="lj8Vq-1" type="set"><bind xmlns="urn:ietf:params:xml:ns:xmpp-bind"><resource>70</resource></bind></iq>'
S->C 188 '<iq type="result" id="lj8Vq-1" to="wa2-mz36-qrmzh6.bosch.de/260d2859"><bind xmlns="urn:ietf:params:xml:ns:xmpp-bind"><jid>rrccontact_458921440@wa2-mz36-qrmzh6.bosch.de/70</jid></bind></iq>'
C->S 87 '<iq id="lj8Vq-2" type="set"><session xmlns="urn:ietf:params:xml:ns:xmpp-session"/></iq>'
S->C 86 '<iq type="result" id="lj8Vq-2" to="rrccontact_458921440@wa2-mz36-qrmzh6.bosch.de/70"/>'
C->S 74 '<iq id="lj8Vq-3" type="get"><query xmlns="jabber:iq:roster" ></query></iq>'
S->C 123 '<iq type="result" id="lj8Vq-3" to="rrccontact_458921440@wa2-mz36-qrmzh6.bosch.de/70"><query xmlns="jabber:iq:roster"/></iq>'
C->S 34 '<presence id="lj8Vq-4"></presence>'
C->S 34 '<presence id="lj8Vq-5"></presence>'

Now all of the preliminary messages are dealt with, we get to the good bit. The message below is sent from the client (the phone app) to the server:

C->S 162 '<message id="lj8Vq-6" to="rrcgateway_458921440@wa2-mz36-qrmzh6.bosch.de" type="chat"><body>GET /ecus/rrc/uiStatus HTTP /1.0\nUser-Agent: NefitEasy</body></message>'

It basically seems to be a HTTP GET request embedded within an XMPP message. That seems a bit strange to me - why not just use HTTP directly? - but at least it is easy to understand. The URL that is being requested also makes sense - I was on the 'home screen' of the app at that point, so it was grabbing the status for displaying in the user-interface (things like current temperature, set-point temperature, whether the boiler is on or not, etc).

Now we can see the response from the server:

S->C 904 '<message to="rrccontact_458921440@wa2-mz36-qrmzh6.bosch.de/70" type="chat" xml:lang="en" from="rrcgateway_458921440@wa2-mz36-qrmzh6.bosch.de/RRC-RestApi"><body>HTTP/1.0 200 OK\nContent-Length: 640\nContent-Type: application/json\nconnection: close\n\n5EBW5RuFo7QojD4F1Uv0kOde1MbeVA46P3RDX6ZEYKaKkbLxanqVR2I8ceuQNbxkgkfzeLgg6D5ypF9jo7yGVRbR/ydf4L4MMTHxvdxBubG5HhiVqJgSc2+7iPvhcWvRZrRKBEMiz8vAsd5JleS4CoTmbN0vV7kHgO2uVeuxtN5ZDsk3/cZpxiTvvaXWlCQGOavCLe55yQqmm3zpGoNFolGPTNC1MVuk00wpf6nbS7sFaRXSmpGQeGAfGNxSxfVPhWZtWRP3/ETi1Z+ozspBO8JZRAzeP8j0fJrBe9u+kDQJNXiMkgzyWb6Il6roSBWWgwYuepGYf/dSR9YygF6lrV+iQdZdyF08ZIgcNY5g5XWtm4LdH8SO+TZpP9aocLUVR1pmFM6m19MKP+spMg8gwPm6L9YuWSvd62KA8ASIQMtWbzFB6XjanGBQpVeMLI1Uzx4wWRaRaAG5qLTda9PpGk8K6LWOxHwtsuW/CDST/hE5jXvWqfVmrceUVqHz5Qcb0sjKRU5TOYA+JNigSf0Z4CIh7xD1t7bjJf9m6Wcyys/NkwZYryoQm99J2yH2khWXyd2DRETbsynr1AWrSRlStZ5H9ghPoYTqvKvgWsyMVTxbMOht86CzoufceI2W+Rr9</body></message>'

Oh. This looks a bit more complicated, and not very easily interpretable. Lets format the main body of the message formatted a bit more nicely:

HTTP/1.0 200 OK
Content-Length: 640
Content-Type: application/json
connection: close

5EBW5RuFo7QojD4F1Uv0kOde1MbeVA46P3RDX6ZEYKaKkbLxanqVR2I8ceuQNbxkgkfzeLgg6D5ypF9jo7yGVRbR/ydf4L4MMTHxvdxBubG5HhiVqJgSc2+7iPvhcWvRZrRKBEMiz8vAsd5JleS4CoTmbN0vV7kHgO2uVeuxtN5ZDsk3/cZpxiTvvaXWlCQGOavCLe55yQqmm3zpGoNFolGPTNC1MVuk00wpf6nbS7sFaRXSmpGQeGAfGNxSxfVPhWZtWRP3/ETi1Z+ozspBO8JZRAzeP8j0fJrBe9u+kDQJNXiMkgzyWb6Il6roSBWWgwYuepGYf/dSR9YygF6lrV+iQdZdyF08ZIgcNY5g5XWtm4LdH8SO+TZpP9aocLUVR1pmFM6m19MKP+spMg8gwPm6L9YuWSvd62KA8ASIQMtWbzFB6XjanGBQpVeMLI1Uzx4wWRaRaAG5qLTda9PpGk8K6LWOxHwtsuW/CDST/hE5jXvWqfVmrceUVqHz5Qcb0sjKRU5TOYA+JNigSf0Z4CIh7xD1t7bjJf9m6Wcyys/NkwZYryoQm99J2yH2khWXyd2DRETbsynr1AWrSRlStZ5H9ghPoYTqvKvgWsyMVTxbMOht86CzoufceI2W+Rr9

So, it seems to be a standard HTTP response (200 OK), but the body looks like it is encoded somehow. I assume that the decoded body would be something like JSON or XML or something containing the various status values - but how do we decode it to get that?

I tried all sorts of things like Base64, MD5 and so on but nothing seemed to work. I gave up on this for a few days, while gently pondering it in the back of my mind. When I came back to it, I realised that the data here was probably actually encrypted, using the Access Code that comes with the Wave and the password that you set up when you first connect the Wave. Of course, to decrypt it we need to know how it was encrypted…so time to break out the next tool: a decompiler.

Yes, that's right: to fully understand exactly what the Wave app is doing, I needed to decompile the Android app's APK file and look at the code. I did this using the aptly named Android APK Decompiler, and got surprisingly readable Java code out of it! (I mean, it had a lot of goto statements, but at least the variables had sensible names!)

It's difficult to explain the full details of the encryption/decryption algorithm in prose - so I've included the Python code I implemented to do this below. However, a brief summary is that: the main encryption is AES using ECB, with keys generated from the MD5 sums of combinations of the Access Code, the password and a 'secret' (a value hard-coded into the app).

def encode(s):
    abyte1 = get_md5(access + secret)
    abyte2 = get_md5(secret + password)

    key = abyte1 + abyte2

    a = AES.new(key)
    a = AES.new(key, AES.MODE_ECB)
    res = a.encrypt(s)

    encoded = base64.b64encode(res)

    return encoded

def decode(data):
    decoded = base64.b64decode(data)

    abyte1 = get_md5(access + secret)
    abyte2 = get_md5(secret + password)

    key = abyte1 + abyte2

    a = AES.new(key)
    a = AES.new(key, AES.MODE_ECB)
    res = a.decrypt(decoded)

    return res

Using these functions we can decrypt the response to the GET /ecus/rrc/uiStatus message that we saw earlier, and we get this:

{'id': '/ecus/rrc/uiStatus',
 'recordable': 0,
 'type': 'uiUpdate',
 'value': {'ARS': 'init',
  'BAI': 'CH',
  'BBE': 'false',
  'BLE': 'false',
  'BMR': 'false',
  'CPM': 'auto',
  'CSP': '31',
  'CTD': '2014-12-26T12:34:27+00:00 Fr',
  'CTR': 'room',
  'DAS': 'off',
  'DHW': 'on',
  'ESI': 'off',
  'FPA': 'off',
  'HED_DB': '',
  'HED_DEV': 'false',
  'HED_EN': 'false',
  'HMD': 'off',
  'IHS': 'ok',
  'IHT': '16.70',
  'MMT': '15.5',
  'PMR': 'false',
  'RS': 'off',
  'TAS': 'off',
  'TOD': '0',
  'TOR': 'on',
  'TOT': '17.0',
  'TSP': '17.0',
  'UMD': 'clock'},
 'writeable': 0}

This makes far more sense!

It may not be immediately apparent what each field is (three character variable names - great!), but some of them are fairly obvious (CTD presumably stands for something like Current Time/Date), or can be established by decoding a number of messages with the boiler in different states (showing that DHW stands for Domestic Hot Water and BAI for Burner Active Indicator).

We've made a lot of progress in the second part of this guide: we've now decrypted the communications, and worked out how to get all of the status information that is shown on the app home screen. At this point I set up a simple temperature monitoring system to produce nice graphs of temperature over time - but I'll leave the description of that to later in the series. In the next part we're going to look at sending messages to actually change the state of the thermostat (such as setting a new temperature, or switching to manual mode), and then have a look at the Python library I've written to control the thermostat.

11 Feb 2016 12:02pm GMT

eGenix.com: Python Meeting Düsseldorf - 22 New Videos Online

The following text is in German, since we're announcing videos available from a regional user group meeting in Düsseldorf, Germany.

Was ist das Python Meeting Düsseldorf ?

Das Python Meeting Düsseldorf ist eine Veranstaltung, die alle drei Monate in Düsseldorf stattfindet und sich an Python Begeisterte aus der Region wendet.

Bei jedem Treffen werden Vorträge gehalten und anschließend in Diskussionen vertieft. Die Meetings dauern üblicherweise ca. 2 Stunden und münden anschließend in eine Restaurant-Session.

Teilnehmer kommen aus ganz Nordrhein-Westfalen, hauptsächlich allerdings aus der näheren Umgebung.

Neue Videos

Um die Vorträge auch für andere Python Enthusiasten zugänglich zu machen, nehmen wir die Vorträge auf, produzieren daraus Videos und laden diese auf unseren PyDDF YouTube Channel hoch.

In den letzten Wochen haben wir die Videos der letzten Treffen aufgearbeitet. Insgesamt sind 22 neue Videos dazugekommen. Viel Spaß damit:

Python Meeting Düsseldorf 2016-01-19

Python Meeting Düsseldorf 2015-10-21

Python Meeting Düsseldorf 2015-07-29

Python Meeting Düsseldorf 2015-04-29


Die vollständige Liste aller mehr als 90 Python Meeting Videos ist über unsere Video Liste verfügbar.

Weitere Informationen

Weitere Informationen und Termine rund um das Python Meeting Düsseldorf stehen auf unserer Webseite:

http://pyddf.de/

Viel Spaß !

Marc-Andre Lemburg, eGenix.com

11 Feb 2016 9:00am GMT

eGenix.com: Python Meeting Düsseldorf - 22 New Videos Online

The following text is in German, since we're announcing videos available from a regional user group meeting in Düsseldorf, Germany.

Was ist das Python Meeting Düsseldorf ?

Das Python Meeting Düsseldorf ist eine Veranstaltung, die alle drei Monate in Düsseldorf stattfindet und sich an Python Begeisterte aus der Region wendet.

Bei jedem Treffen werden Vorträge gehalten und anschließend in Diskussionen vertieft. Die Meetings dauern üblicherweise ca. 2 Stunden und münden anschließend in eine Restaurant-Session.

Teilnehmer kommen aus ganz Nordrhein-Westfalen, hauptsächlich allerdings aus der näheren Umgebung.

Neue Videos

Um die Vorträge auch für andere Python Enthusiasten zugänglich zu machen, nehmen wir die Vorträge auf, produzieren daraus Videos und laden diese auf unseren PyDDF YouTube Channel hoch.

In den letzten Wochen haben wir die Videos der letzten Treffen aufgearbeitet. Insgesamt sind 22 neue Videos dazugekommen. Viel Spaß damit:

Python Meeting Düsseldorf 2016-01-19

Python Meeting Düsseldorf 2015-10-21

Python Meeting Düsseldorf 2015-07-29

Python Meeting Düsseldorf 2015-04-29


Die vollständige Liste aller mehr als 90 Python Meeting Videos ist über unsere Video Liste verfügbar.

Weitere Informationen

Weitere Informationen und Termine rund um das Python Meeting Düsseldorf stehen auf unserer Webseite:

http://pyddf.de/

Viel Spaß !

Marc-Andre Lemburg, eGenix.com

11 Feb 2016 9:00am GMT

10 Feb 2016

feedPlanet Python

Vladimir Iakolev: Writing breakout clone with micropython

I have pyboard, OLED display (SSD1306) and joystick (Keyes_SJoys), so I decided to try to make breakout clone. First of all I decided to create something like a little framework, that will be a bit similar to React, and all game can be formalized just in two functions:

For example, code that draws chess cells and invert it on click, will be like:

from lib.ssd1306 import Display
from lib.keyes import Joystick
from lib.engine import Game, rectangle, text


def is_filled(x, y, inverted):
    fill = (x + y) % 40 == 0
    if inverted:
        return not fill
    else:
        return fill


def view(state):
    # Display data available in state['display']
    for x in range(0, state['display']['width'], 20):
        for y in range(0, state['display']['height'], 20):
            # rectangle is bundled view that yields points
            yield from rectangle(x=x, y=y, w=20, h=20,
                                 fill=is_filled(x, y, state['inverted']))


def controller(state):
    # Joystick data available in state['display']
    if state['joystick']['clicked']:
        return dict(state, inverted=not state['inverted'])
    else:
        return state

initial_state = {'inverted': False}
chess_deck = Game(display=Display(pinout={'sda': 'Y10',
                                          'scl': 'Y9'},
                                  height=64,
                                  external_vcc=False),
                  joystick=Joystick('X1', 'X2', 'X3'),
                  initial_state=initial_state,
                  view=view,
                  controller=controller)

if __name__ == '__main__':
    chess_deck.run()

In action:

Chess photo

From the code you can see, that views can be easily nested with yield from. So if we want to move cell to separate view:

def cell(x, y, inverted):
    yield from rectangle(x=x, y=y, w=20, h=20,
                         fill=is_filled(x, y, inverted))


def view(state):
    for x in range(0, state['display']['width'], 20):
        for y in range(0, state['display']['height'], 20):
            yield from cell(x, y, state['inverted'])

And another nice thing about this approach, is that because of generators we consume not a lot of memory, if we'll make it eager, we'll fail with MemoryError: memory allocation failed soon.

Back to breakout, let's start with views, first of all implement splash screen:

def splash(w, h):
    for n in range(0, w, 20):
        yield from rectangle(x=n, y=0, w=10, h=h, fill=True)

    yield from rectangle(x=0, y=17, w=w, h=30, fill=False)
    # text is bundled view
    yield from text(x=0, y=20, string='BREAKOUT', size=3)


def view(state):
    yield from splash(state['display']['width'],
                      state['display']['height'])

It will draw a nice splash screen:

Splash screen

On splash screen game should be started when user press joystick, so we should update code a bit:

GAME_NOT_STARTED = 0
GAME_ACTIVE = 1
GAME_OVER = 2

def controller(state):
    if state['status'] == GAME_NOT_STARTED and state['joystick']['clicked']:
        state['status'] = GAME_ACTIVE
    return state


initial_state = {'status': GAME_NOT_STARTED}

Now when joystick is pressed, game changes status to GAME_ACTIVE. And now it's time to create view for game screen:

BRICK_W = 8
BRICK_H = 4
BRICK_BORDER = 1
BRICK_ROWS = 4

PADDLE_W = 16
PADDLE_H = 4

BALL_W = 3
BALL_H = 3


def brick(data):
    yield from rectangle(x=data['x'] + BRICK_BORDER,
                         y=data['y'] + BRICK_BORDER,
                         w=BRICK_W - BRICK_BORDER,
                         h=BRICK_H - BRICK_BORDER,
                         fill=True)


def paddle(data):
    yield from rectangle(x=data['x'], y=data['y'],
                         w=PADDLE_W, h=PADDLE_H,
                         fill=True)


def ball(data):
    yield from rectangle(x=data['x'], y=data['y'],
                         w=BALL_W, h=BALL_H, fill=True)


def deck(state):
    for brick_data in state['bricks']:
        yield from brick(brick_data)

    yield from paddle(state['paddle'])
    yield from ball(state['ball'])


def view(state):
    if state['status'] == GAME_NOT_STARTED:
        yield from splash(state['display']['width'],
                          state['display']['height'])
    else:
        yield from deck(state)


def get_initial_game_state(state):
    state['status'] = GAME_ACTIVE
    state['bricks'] = [{'x': x, 'y': yn * BRICK_H}
                       for x in range(0, state['display']['width'], BRICK_W)
                       for yn in range(BRICK_ROWS)]
    state['paddle'] = {'x': (state['display']['width'] - PADDLE_W) / 2,
                       'y': state['display']['height'] - PADDLE_H}
    state['ball'] = {'x': (state['display']['width'] - BALL_W) / 2,
                     'y': state['display']['height'] - PADDLE_H * 2 - BALL_W}
    return state


def controller(state):
    if state['status'] == GAME_NOT_STARTED and state['joystick']['clicked']:
        state = get_initial_game_state(state)
    return state

Game screen

And last part of views - game over screen:

def game_over():
    yield from text(x=0, y=20, string='GAMEOVER', size=3)


def view(state):
    if state['status'] == GAME_NOT_STARTED:
        yield from splash(state['display']['width'],
                          state['display']['height'])
    else:
        yield from deck(state)
        if state['status'] == GAME_OVER:
            yield from game_over()

Game screen

So we ended up with views, now we should add ability to move paddle with joystick:

def update_paddle(paddle, joystick, w):
    paddle['x'] += int(joystick['x'] / 10)
    if paddle['x'] < 0:
        paddle['x'] = 0
    elif paddle['x'] > (w - PADDLE_W):
        paddle['x'] = w - PADDLE_W
    return paddle


def controller(state):
    if state['status'] == GAME_NOT_STARTED and state['joystick']['clicked']:
        state = get_initial_game_state(state)
    elif state['status'] == GAME_ACTIVE:
        state['paddle'] = update_paddle(state['paddle'], state['joystick'],
                                        state['display']['width'])
    return state

Never mind performance, it'll be fixed in the end of the article:

Now it's time for teh hardest thing - moving and bouncing ball, so there's no real physics, for simplification ball movements will be represented as vx and vy, so when ball:

And I implemented something like this with a few hacks:

BALL_SPEED = 6
BALL_SPEED_BORDER = 0.5


def get_initial_game_state(state):
    state['status'] = GAME_ACTIVE
    state['bricks'] = [{'x': x, 'y': yn * BRICK_H}
                       for x in range(0, state['display']['width'], BRICK_W)
                       for yn in range(BRICK_ROWS)]
    state['paddle'] = {'x': (state['display']['width'] - PADDLE_W) / 2,
                       'y': state['display']['height'] - PADDLE_H}

    # Initial velocity for ball:
    ball_vx = BALL_SPEED_BORDER + pyb.rng() % (BALL_SPEED - BALL_SPEED_BORDER)
    ball_vy = -math.sqrt(BALL_SPEED ** 2 - ball_vx ** 2)
    state['ball'] = {'x': (state['display']['width'] - BALL_W) / 2,
                     'y': state['display']['height'] - PADDLE_H * 2 - BALL_W,
                     'vx': ball_vx,
                     'vy': ball_vy}
    return state


def calculate_velocity(ball, item_x, item_w):
    """Calculates velocity for collision."""
    intersection = (item_x + item_w - ball['x']) / item_w
    vx = ball['vx'] + BALL_SPEED * (0.5 - intersection)
    if vx > BALL_SPEED - BALL_SPEED_BORDER:
        vx = BALL_SPEED - BALL_SPEED_BORDER
    elif vx < BALL_SPEED_BORDER - BALL_SPEED:
        vx = BALL_SPEED_BORDER - BALL_SPEED

    vy = math.sqrt(BALL_SPEED ** 2 - vx ** 2)
    if ball['vy'] > 0:
        vy = - vy
    return vx, vy


def collide(ball, item, item_w, item_h):
    return item['x'] - BALL_W < ball['x'] < item['x'] + item_w \
           and item['y'] - BALL_H < ball['y'] < item['y'] + item_h


def update_ball(state):
    state['ball']['x'] += state['ball']['vx']
    state['ball']['y'] += state['ball']['vy']

    # Collide with left/right wall
    if state['ball']['x'] <= 0 or state['ball']['x'] >= state['display']['width']:
        state['ball']['vx'] = - state['ball']['vx']

    # Collide with top wall
    if state['ball']['y'] <= 0:
        state['ball']['vy'] = -state['ball']['vy']

    # Collide with paddle
    if collide(state['ball'], state['paddle'], PADDLE_W, PADDLE_H):
        vx, vy = calculate_velocity(state['ball'], state['paddle']['x'], PADDLE_W)
        state['ball'].update(vx=vx, vy=vy)

    # Collide with brick
    for n, brick in enumerate(state['bricks']):
        if collide(state['ball'], brick, BRICK_W, BRICK_H):
            vx, vy = calculate_velocity(state['ball'], brick['x'], BRICK_W)
            state['ball'].update(vx=vx, vy=vy)
            state['bricks'].pop(n)

    return state


def controller(state):
    if state['status'] == GAME_NOT_STARTED and state['joystick']['clicked']:
        state = get_initial_game_state(state)
    elif state['status'] == GAME_ACTIVE:
        state['paddle'] = update_paddle(state['paddle'], state['joystick'],
                                        state['display']['width'])
        state = update_ball(state)
    return state

And it seems to be wroking:

So now the last part, we should show "Game Over" when ball hits the bottom wall or when all bricks destroyed, and then start game again if user clicks joystick:

def is_game_over(state):
    return not state['bricks'] or state['ball']['y'] > state['display']['height']


def controller(state):
    if state['status'] in (GAME_NOT_STARTED, GAME_OVER)\
            and state['joystick']['clicked']:
        state = get_initial_game_state(state)
    elif state['status'] == GAME_ACTIVE:
        state['paddle'] = update_paddle(state['paddle'], state['joystick'],
                                        state['display']['width'])
        state = update_ball(state)
        if is_game_over(state):
            state['status'] = GAME_OVER
    return state

And it works too:

Performance so bad because drawing pixel on the screen is relatively time consuming operation, and we can easily fix performance by just decreasing count of bricks:

BRICK_W = 12
BRICK_H = 6
BRICK_BORDER = 4
BRICK_ROWS = 3

And now it's smooth:

Source code.

10 Feb 2016 5:05pm GMT

Vladimir Iakolev: Writing breakout clone with micropython

I have pyboard, OLED display (SSD1306) and joystick (Keyes_SJoys), so I decided to try to make breakout clone. First of all I decided to create something like a little framework, that will be a bit similar to React, and all game can be formalized just in two functions:

For example, code that draws chess cells and invert it on click, will be like:

from lib.ssd1306 import Display
from lib.keyes import Joystick
from lib.engine import Game, rectangle, text


def is_filled(x, y, inverted):
    fill = (x + y) % 40 == 0
    if inverted:
        return not fill
    else:
        return fill


def view(state):
    # Display data available in state['display']
    for x in range(0, state['display']['width'], 20):
        for y in range(0, state['display']['height'], 20):
            # rectangle is bundled view that yields points
            yield from rectangle(x=x, y=y, w=20, h=20,
                                 fill=is_filled(x, y, state['inverted']))


def controller(state):
    # Joystick data available in state['display']
    if state['joystick']['clicked']:
        return dict(state, inverted=not state['inverted'])
    else:
        return state

initial_state = {'inverted': False}
chess_deck = Game(display=Display(pinout={'sda': 'Y10',
                                          'scl': 'Y9'},
                                  height=64,
                                  external_vcc=False),
                  joystick=Joystick('X1', 'X2', 'X3'),
                  initial_state=initial_state,
                  view=view,
                  controller=controller)

if __name__ == '__main__':
    chess_deck.run()

In action:

Chess photo

From the code you can see, that views can be easily nested with yield from. So if we want to move cell to separate view:

def cell(x, y, inverted):
    yield from rectangle(x=x, y=y, w=20, h=20,
                         fill=is_filled(x, y, inverted))


def view(state):
    for x in range(0, state['display']['width'], 20):
        for y in range(0, state['display']['height'], 20):
            yield from cell(x, y, state['inverted'])

And another nice thing about this approach, is that because of generators we consume not a lot of memory, if we'll make it eager, we'll fail with MemoryError: memory allocation failed soon.

Back to breakout, let's start with views, first of all implement splash screen:

def splash(w, h):
    for n in range(0, w, 20):
        yield from rectangle(x=n, y=0, w=10, h=h, fill=True)

    yield from rectangle(x=0, y=17, w=w, h=30, fill=False)
    # text is bundled view
    yield from text(x=0, y=20, string='BREAKOUT', size=3)


def view(state):
    yield from splash(state['display']['width'],
                      state['display']['height'])

It will draw a nice splash screen:

Splash screen

On splash screen game should be started when user press joystick, so we should update code a bit:

GAME_NOT_STARTED = 0
GAME_ACTIVE = 1
GAME_OVER = 2

def controller(state):
    if state['status'] == GAME_NOT_STARTED and state['joystick']['clicked']:
        state['status'] = GAME_ACTIVE
    return state


initial_state = {'status': GAME_NOT_STARTED}

Now when joystick is pressed, game changes status to GAME_ACTIVE. And now it's time to create view for game screen:

BRICK_W = 8
BRICK_H = 4
BRICK_BORDER = 1
BRICK_ROWS = 4

PADDLE_W = 16
PADDLE_H = 4

BALL_W = 3
BALL_H = 3


def brick(data):
    yield from rectangle(x=data['x'] + BRICK_BORDER,
                         y=data['y'] + BRICK_BORDER,
                         w=BRICK_W - BRICK_BORDER,
                         h=BRICK_H - BRICK_BORDER,
                         fill=True)


def paddle(data):
    yield from rectangle(x=data['x'], y=data['y'],
                         w=PADDLE_W, h=PADDLE_H,
                         fill=True)


def ball(data):
    yield from rectangle(x=data['x'], y=data['y'],
                         w=BALL_W, h=BALL_H, fill=True)


def deck(state):
    for brick_data in state['bricks']:
        yield from brick(brick_data)

    yield from paddle(state['paddle'])
    yield from ball(state['ball'])


def view(state):
    if state['status'] == GAME_NOT_STARTED:
        yield from splash(state['display']['width'],
                          state['display']['height'])
    else:
        yield from deck(state)


def get_initial_game_state(state):
    state['status'] = GAME_ACTIVE
    state['bricks'] = [{'x': x, 'y': yn * BRICK_H}
                       for x in range(0, state['display']['width'], BRICK_W)
                       for yn in range(BRICK_ROWS)]
    state['paddle'] = {'x': (state['display']['width'] - PADDLE_W) / 2,
                       'y': state['display']['height'] - PADDLE_H}
    state['ball'] = {'x': (state['display']['width'] - BALL_W) / 2,
                     'y': state['display']['height'] - PADDLE_H * 2 - BALL_W}
    return state


def controller(state):
    if state['status'] == GAME_NOT_STARTED and state['joystick']['clicked']:
        state = get_initial_game_state(state)
    return state

Game screen

And last part of views - game over screen:

def game_over():
    yield from text(x=0, y=20, string='GAMEOVER', size=3)


def view(state):
    if state['status'] == GAME_NOT_STARTED:
        yield from splash(state['display']['width'],
                          state['display']['height'])
    else:
        yield from deck(state)
        if state['status'] == GAME_OVER:
            yield from game_over()

Game screen

So we ended up with views, now we should add ability to move paddle with joystick:

def update_paddle(paddle, joystick, w):
    paddle['x'] += int(joystick['x'] / 10)
    if paddle['x'] < 0:
        paddle['x'] = 0
    elif paddle['x'] > (w - PADDLE_W):
        paddle['x'] = w - PADDLE_W
    return paddle


def controller(state):
    if state['status'] == GAME_NOT_STARTED and state['joystick']['clicked']:
        state = get_initial_game_state(state)
    elif state['status'] == GAME_ACTIVE:
        state['paddle'] = update_paddle(state['paddle'], state['joystick'],
                                        state['display']['width'])
    return state

Never mind performance, it'll be fixed in the end of the article:

Now it's time for teh hardest thing - moving and bouncing ball, so there's no real physics, for simplification ball movements will be represented as vx and vy, so when ball:

And I implemented something like this with a few hacks:

BALL_SPEED = 6
BALL_SPEED_BORDER = 0.5


def get_initial_game_state(state):
    state['status'] = GAME_ACTIVE
    state['bricks'] = [{'x': x, 'y': yn * BRICK_H}
                       for x in range(0, state['display']['width'], BRICK_W)
                       for yn in range(BRICK_ROWS)]
    state['paddle'] = {'x': (state['display']['width'] - PADDLE_W) / 2,
                       'y': state['display']['height'] - PADDLE_H}

    # Initial velocity for ball:
    ball_vx = BALL_SPEED_BORDER + pyb.rng() % (BALL_SPEED - BALL_SPEED_BORDER)
    ball_vy = -math.sqrt(BALL_SPEED ** 2 - ball_vx ** 2)
    state['ball'] = {'x': (state['display']['width'] - BALL_W) / 2,
                     'y': state['display']['height'] - PADDLE_H * 2 - BALL_W,
                     'vx': ball_vx,
                     'vy': ball_vy}
    return state


def calculate_velocity(ball, item_x, item_w):
    """Calculates velocity for collision."""
    intersection = (item_x + item_w - ball['x']) / item_w
    vx = ball['vx'] + BALL_SPEED * (0.5 - intersection)
    if vx > BALL_SPEED - BALL_SPEED_BORDER:
        vx = BALL_SPEED - BALL_SPEED_BORDER
    elif vx < BALL_SPEED_BORDER - BALL_SPEED:
        vx = BALL_SPEED_BORDER - BALL_SPEED

    vy = math.sqrt(BALL_SPEED ** 2 - vx ** 2)
    if ball['vy'] > 0:
        vy = - vy
    return vx, vy


def collide(ball, item, item_w, item_h):
    return item['x'] - BALL_W < ball['x'] < item['x'] + item_w \
           and item['y'] - BALL_H < ball['y'] < item['y'] + item_h


def update_ball(state):
    state['ball']['x'] += state['ball']['vx']
    state['ball']['y'] += state['ball']['vy']

    # Collide with left/right wall
    if state['ball']['x'] <= 0 or state['ball']['x'] >= state['display']['width']:
        state['ball']['vx'] = - state['ball']['vx']

    # Collide with top wall
    if state['ball']['y'] <= 0:
        state['ball']['vy'] = -state['ball']['vy']

    # Collide with paddle
    if collide(state['ball'], state['paddle'], PADDLE_W, PADDLE_H):
        vx, vy = calculate_velocity(state['ball'], state['paddle']['x'], PADDLE_W)
        state['ball'].update(vx=vx, vy=vy)

    # Collide with brick
    for n, brick in enumerate(state['bricks']):
        if collide(state['ball'], brick, BRICK_W, BRICK_H):
            vx, vy = calculate_velocity(state['ball'], brick['x'], BRICK_W)
            state['ball'].update(vx=vx, vy=vy)
            state['bricks'].pop(n)

    return state


def controller(state):
    if state['status'] == GAME_NOT_STARTED and state['joystick']['clicked']:
        state = get_initial_game_state(state)
    elif state['status'] == GAME_ACTIVE:
        state['paddle'] = update_paddle(state['paddle'], state['joystick'],
                                        state['display']['width'])
        state = update_ball(state)
    return state

And it seems to be wroking:

So now the last part, we should show "Game Over" when ball hits the bottom wall or when all bricks destroyed, and then start game again if user clicks joystick:

def is_game_over(state):
    return not state['bricks'] or state['ball']['y'] > state['display']['height']


def controller(state):
    if state['status'] in (GAME_NOT_STARTED, GAME_OVER)\
            and state['joystick']['clicked']:
        state = get_initial_game_state(state)
    elif state['status'] == GAME_ACTIVE:
        state['paddle'] = update_paddle(state['paddle'], state['joystick'],
                                        state['display']['width'])
        state = update_ball(state)
        if is_game_over(state):
            state['status'] = GAME_OVER
    return state

And it works too:

Performance so bad because drawing pixel on the screen is relatively time consuming operation, and we can easily fix performance by just decreasing count of bricks:

BRICK_W = 12
BRICK_H = 6
BRICK_BORDER = 4
BRICK_ROWS = 3

And now it's smooth:

Source code.

10 Feb 2016 5:05pm GMT

John Cook: Maximum principle and approximating boundary value problems

Solutions to differential equations often satisfy some sort of maximum principle, which can in turn be used to construct upper and lower bounds on solutions.

We illustrate this in one dimension, using a boundary value problem for an ordinary differential equation (ODE).

Maximum principles

If the second derivative of a function is positive over an open interval (a, b), the function cannot have a maximum in that interval. If the function has a maximum over the closed interval [a, b] then it must occur at one of the ends, at a or b.

This can be generalized, for example, to the following maximum principle. Let L be the differential operator

L[u] = u" + g(x)u' + h(x)

where g and h are bounded functions on some interval [a, b] and h is non-positive. Suppose L[u] ≥ 0 on (a, b). If u has an interior maximum, then u must be constant.

Boundary value problems

Now suppose that we're interested in the boundary value problem L[u] = f where we specify the values of u at the endpoints a and b, i.e. u(a) = ua and u(b) = ub. We can construct an upper bound on u as follows.

Suppose we find a function z such that L[z] ≤ f and z(a) ≥ ua and z(b) ≥ ub. Then by applying the maximum principle to u - z, we see that u - z must be ≤ 0, and so z is an upper bound for u.

Similarly, suppose we find a function w such that L[w] ≥ f and w(a) ≤ ua and w(b) ≤ ub. Then by applying the maximum principle to w - u, we see that w - u must be ≤ 0, and so w is an lower bound for u.

Note that any functions z and w that satisfy the above requirements give upper and lower bounds, though the bounds may not be very useful. By being clever in our choice of z and w we may be able to get tighter bounds. We might start by choosing polynomials, exponentials, etc. Any functions that are easy to work with and see how good the resulting bounds are.

Tomorrow's post is similar to this one but looks at bounds for an initial value problem rather than a boundary value problem.

Airy equation example

The following is an elaboration on an example from [1]. Suppose we want to bound solutions to

u"(x) - x u(x) = 0

where u(0) = 0 and u(1) = 1. (This is a well-known equation, but for purposes of illustration we'll pretend at first that we know nothing about its solutions.)

For our upper bound, we can simply use z(x) = x. We have L[z] ≤ 0 and z satisfies the boundary conditions exactly.

For our lower bound, we use w(x) = x - βx(1 - x). Why? The function z already satisfies the boundary condition. If we add some multiple of x(1 - x) we'll maintain the boundary condition since x(1 - x) is zero at 0 and 1. The coefficient β gives us some room to maneuver. Turns out L[w] ≥ 0 if β ≥ 1/2. If we choose β = 1/2 we have

(x + x2)/2 ≤ u(x) ≤ x

In general, you don't know the function you're trying to bound. That's when bounds are most useful. But this is a sort of toy example because we do know the solution. The equation in this example is well known and is called Airy's equation. The Airy functions Ai and Bi are independent solutions. Here's a plot of the solution with its upper and lower bounds.

Here's the Python code I used to solve for the coefficients of Ai and Bi and make the plot.

import numpy as np
from scipy.linalg import solve
from scipy.special import airy
import matplotlib.pyplot as plt

# airy(x) returns (Ai(x), Ai'(x), Bi(x), Bi'(x))
def Ai(x):
    return airy(x)[0]

def Bi(x):
    return airy(x)[2]

M = np.matrix([[Ai(0), Bi(0)], [Ai(1), Bi(1)]])
c = solve(M, [0, 1])

t = np.linspace(0, 1, 100)
plt.plot(t, (t + t**2)/2, 'r-', t, c[0]*Ai(t) + c[1]*Bi(t), 'k--', t, t, 'b-',)
plt.legend(["lower bound $(x + x^2)/2$", 
    "exact solution $c_0Ai + c_1Bi$", 
    "upper bound $x$"], loc="upper left")
plt.show()

SciPy's function airy has an optimization that we waste here. The function computes Ai and Bi and their first derivatives all at the same time. We could take advantage of that to remove some redundant computations, but that would make the code harder to read. We chose instead to wait an extra nanosecond for the plot.

Help with differential equations

* * *

[1] Murray Protter and Hans Weinberger. Maximum Principles in Differential Equations.

10 Feb 2016 3:46pm GMT

John Cook: Maximum principle and approximating boundary value problems

Solutions to differential equations often satisfy some sort of maximum principle, which can in turn be used to construct upper and lower bounds on solutions.

We illustrate this in one dimension, using a boundary value problem for an ordinary differential equation (ODE).

Maximum principles

If the second derivative of a function is positive over an open interval (a, b), the function cannot have a maximum in that interval. If the function has a maximum over the closed interval [a, b] then it must occur at one of the ends, at a or b.

This can be generalized, for example, to the following maximum principle. Let L be the differential operator

L[u] = u" + g(x)u' + h(x)

where g and h are bounded functions on some interval [a, b] and h is non-positive. Suppose L[u] ≥ 0 on (a, b). If u has an interior maximum, then u must be constant.

Boundary value problems

Now suppose that we're interested in the boundary value problem L[u] = f where we specify the values of u at the endpoints a and b, i.e. u(a) = ua and u(b) = ub. We can construct an upper bound on u as follows.

Suppose we find a function z such that L[z] ≤ f and z(a) ≥ ua and z(b) ≥ ub. Then by applying the maximum principle to u - z, we see that u - z must be ≤ 0, and so z is an upper bound for u.

Similarly, suppose we find a function w such that L[w] ≥ f and w(a) ≤ ua and w(b) ≤ ub. Then by applying the maximum principle to w - u, we see that w - u must be ≤ 0, and so w is an lower bound for u.

Note that any functions z and w that satisfy the above requirements give upper and lower bounds, though the bounds may not be very useful. By being clever in our choice of z and w we may be able to get tighter bounds. We might start by choosing polynomials, exponentials, etc. Any functions that are easy to work with and see how good the resulting bounds are.

Tomorrow's post is similar to this one but looks at bounds for an initial value problem rather than a boundary value problem.

Airy equation example

The following is an elaboration on an example from [1]. Suppose we want to bound solutions to

u"(x) - x u(x) = 0

where u(0) = 0 and u(1) = 1. (This is a well-known equation, but for purposes of illustration we'll pretend at first that we know nothing about its solutions.)

For our upper bound, we can simply use z(x) = x. We have L[z] ≤ 0 and z satisfies the boundary conditions exactly.

For our lower bound, we use w(x) = x - βx(1 - x). Why? The function z already satisfies the boundary condition. If we add some multiple of x(1 - x) we'll maintain the boundary condition since x(1 - x) is zero at 0 and 1. The coefficient β gives us some room to maneuver. Turns out L[w] ≥ 0 if β ≥ 1/2. If we choose β = 1/2 we have

(x + x2)/2 ≤ u(x) ≤ x

In general, you don't know the function you're trying to bound. That's when bounds are most useful. But this is a sort of toy example because we do know the solution. The equation in this example is well known and is called Airy's equation. The Airy functions Ai and Bi are independent solutions. Here's a plot of the solution with its upper and lower bounds.

Here's the Python code I used to solve for the coefficients of Ai and Bi and make the plot.

import numpy as np
from scipy.linalg import solve
from scipy.special import airy
import matplotlib.pyplot as plt

# airy(x) returns (Ai(x), Ai'(x), Bi(x), Bi'(x))
def Ai(x):
    return airy(x)[0]

def Bi(x):
    return airy(x)[2]

M = np.matrix([[Ai(0), Bi(0)], [Ai(1), Bi(1)]])
c = solve(M, [0, 1])

t = np.linspace(0, 1, 100)
plt.plot(t, (t + t**2)/2, 'r-', t, c[0]*Ai(t) + c[1]*Bi(t), 'k--', t, t, 'b-',)
plt.legend(["lower bound $(x + x^2)/2$", 
    "exact solution $c_0Ai + c_1Bi$", 
    "upper bound $x$"], loc="upper left")
plt.show()

SciPy's function airy has an optimization that we waste here. The function computes Ai and Bi and their first derivatives all at the same time. We could take advantage of that to remove some redundant computations, but that would make the code harder to read. We chose instead to wait an extra nanosecond for the plot.

Help with differential equations

* * *

[1] Murray Protter and Hans Weinberger. Maximum Principles in Differential Equations.

10 Feb 2016 3:46pm GMT

Chris Warrick: Deploying Python Web Applications with nginx and uWSGI Emperor

You just wrote a great Python web application. Now, you want to share it with the world. In order to do that, you need a server, and some software to do that for you.

The following is a comprehensive guide on how to accomplish that, on multiple Linux-based operating systems, using nginx and uWSGI Emperor. It doesn't force you to use any specific web framework - Flask, Django, Pyramid, Bottle will all work. Written for Ubuntu, Fedora and Arch Linux (should be helpful for other systems, too)

Getting Started

In order to deploy your web application, you need a server that gives you root and ssh access - in other words, a VPS (or a dedicated server, or a datacenter lease…). If you're looking for a great VPS service for a low price, I recommend DigitalOcean (reflink [1]), which offers a $5/mo service [2]. If you want to play along at home, without buying a VPS, you can create a virtual machine on your own, or use a Vagrant with a Vagrant box for Fedora 23 (I recommend disabling SELinux, more on that later).

Your server should also run a modern Linux-based operating system. I tested and wrote this guide for Ubuntu 15.10 [3], Fedora 23 and Arch Linux, but other Linux distributions (and perhaps *BSD) will work (in places where the instructions are split three-way, try coming up with your own, reading documentation and config files). Unfortunately, all Linux distributions have their own ideas when it comes to running and managing nginx and UWSGI.

Note

All the commands in this tutorial are meant to be run as root - run su or sudo su first to get an administrative shell.

Start by installing virtualenv, nginx and uWSGI. I recommend using your operating system packages. For uWSGI, we need the logfile and python3 plugins. (Arch Linux names the python3 plugin python; the logfile plugin may be built-in - check with your system repositories!).

Ubuntu:

aptitude install virtualenv python3 uwsgi uwsgi-emperor uwsgi-plugin-python3 nginx-full

Fedora:

dnf install python3-virtualenv uwsgi uwsgi-plugin-python3 uwsgi-logger-file nginx

Arch Linux:

pacman -S python-virtualenv uwsgi uwsgi-plugin-python nginx

Preparing your application

This tutorial will work for any web framework. I will, use a really basic Flask app that has just one route (/), a static hello.png file and a favicon.ico for demonstration purposes. Note that the app does not use app.run(). While you could add it, it would be used for local development and debugging only, and would be prepended by if __name__ == '__main__': - uWSGI doesn't work alongside it.

The app will be installed somewhere under the /srv directory, which is a great place to store things like this. I'll choose /srv/myapp for this tutorial, but for real deployments, you should use sometihing more distinguishable - the domain name is a great idea.

We'll start by creating a virtualenv:

Ubuntu:

cd srv
virtualenv -p /usr/bin/python3 myapp

Fedora:

cd /srv
virtualenv-3.4 myapp

Arch Linux:

cd /srv
virtualenv3 myapp

(Make sure you create a Python 3 environment!)

Now, we need to get our app there and install requirements. An example for the tutorial demo app (adjust for your clone/download path):

cd myapp
cp -r ~/git/flask-demo-app appdata
bin/pip install -r appdata/requirements.txt

I'm storing my application data in the appdata subdirectory so that it doesn't clutter the virtualenv (or vice versa). You may also install the uwsgi package in the virtualenv, but it's optional.

What this directory should be depends on your web framework. For example, for a Django app, you should have an appdata/manage.py file (in other words, appdata is where your app structure starts). I also assumed that the appdata folder should have a static subdirectory with all static files, including favicon.ico if you have one (we will add support for both in nginx).

At this point, you should chown this directory to the user and group your server is going to run as. This is especially important if uwsgi and nginx run as different users (as they do on Fedora). Run one of the following commands:

Ubuntu:

chown -R www-data:www-data /srv/myapp

Fedora:

chown -R uwsgi:nginx /srv/myapp

Arch Linux:

chown -R http:http /srv/myapp

Configuring uWSGI and nginx

Note

Parts of the configuration depend on your operating system. I tried to provide advice for Ubuntu, Fedora and Arch Linux. If you experience any issues, in particular with plugins, please consult the documentation.

We need to write a configuration file for uWSGI and nginx.

uWSGI configuration

Start with this, but read the notes below and change the values accordingly:

[uwsgi]
socket = /srv/myapp/uwsgi.sock
chmod-socket = 775
chdir = /srv/myapp/appdata
master = true
binary-path = /srv/myapp/bin/uwsgi
virtualenv = /srv/myapp
module = flaskapp:app
uid = www-data
gid = www-data
processes = 1
threads = 1
plugins = python3,logfile
logger = file:/srv/myapp/uwsgi.log

Save this file as:

  • Ubuntu: /etc/uwsgi-emperor/vassals/myapp.ini
  • Fedora: /etc/uwsgi.d/myapp.ini
  • Arch Linux: /etc/uwsgi/vassals/myapp.ini (create the directory first and chown it to http: mkdir -p /etc/uwsgi/vassals; chown -R http:http /etc/uwsgi/vassals)

The options are:

  • socket - the socket file that will be used by your application. It's usually a file path (Unix domain socket). You could use a local TCP socket, but it's not recommended.
  • chdir - the app directory.
  • binary-path - the uWSGI executable to use. Remove if you didn't install the (optional) uwsgi package in your virtualenv.
  • virtualenv - the virtualenv for your application.
  • module - the name of the module that houses your application, and the object that speaks the WSGI interface, separated by colons. This depends on your web framework:
    • For Flask: module = filename:app, where filename is the name of your Python file (without the .py part) and app is the Flask object
    • For Django: module = project.wsgi:application, where project is the name of your project (directory with settings.py). You should also add an environment variable: env = DJANGO_SETTINGS_MODULE=project.settings
    • For Bottle: module = filename:app, where app = bottle.default_app()
    • For Pyramid: module = filename:app, where app = config.make_wsgi_app() (make sure it's not in a if __name__ == '__main__': block - the demo app does that!)
  • uid and gid - the names of the user account to use for your server. Use the same values as in the chown command above.
  • processes and threads - control the resources devoted to this application. Because this is a simple hello app, I used one process with one thread, but for a real app, you will probably need more (you need to see what works the best; there is no algorithm to decide). Also, remember that if you use multiple processes, they don't share data, so you need an out-of-process database if you want that.
  • plugins - the list of uWSGI plugins to use. For Arch Linux, use plugins = python (the logfile plugin is always active).
  • logger - the path to your app-specific logfile. (Other logging facilities are available, but this one is the easiest, especially for multiple applications on the same server)

You can test your configuration by running uwsgi --ini /path/to/myapp.ini (disable the logger for stderr output or run tail -f /srv/myapp/uwsgi.log in another window).

If you're using Fedora, there are two configuration changes you need to make globally: in /etc/uwsgi.ini, disable the emperor-tyrant option (which seems to be buggy) and set gid = nginx. We'll need this so that nginx can talk to your socket.

nginx configuration

We need to configure our web server. Here's a basic configuration that will get us started:

Save this file as:

  • Ubuntu: /etc/nginx/sites-enabled/myapp.conf
  • Fedora: /etc/nginx/conf.d/myapp.conf
  • Arch Linux: add include /etc/nginx/conf.d/*.conf; to your http directive in /etc/nginx/nginx.conf and use /etc/nginx/conf.d/myapp.conf
server {
    listen 8080;
    server_name localhost myapp.local;

    location / {
        include uwsgi_params;
        uwsgi_pass unix:/srv/myapp/uwsgi.sock;
    }

    location /static {
        alias /srv/myapp/appdata/static;
    }

    location /favicon.ico {
        alias /srv/myapp/appdata/static/favicon.ico;
    }
}

Note that this file is a very basic and rudimentary configuration. This configuration is fine for local testing, but for a real deployment, you will need to adjust it:

  • set listen to 443 ssl and create a http→https redirect on port 80 (you can get a free SSL certificate from Let's Encrypt; make sure to configure SSL properly).
  • set server_name to your real domain name
  • you might also want to add custom error pages, or change anything else that relates to your web server - consult other nginx guides for details
  • nginx might have some server already enabled by default - edit /etc/nginx/nginx.conf to disable it

Service setup

After you've configured uWSGI and nginx, you need to enable and start the system services.

I'm going to use systemd here. If your system does not support systemd, please consult your OS documentation for instructions.

For Arch Linux

All you need is:

systemctl enable nginx emperor.uwsgi
systemctl start nginx emperor.uwsgi

Verify the service is running with systemctl status emperor.uwsgi

For Fedora

Make sure you followed the extra note about editing /etc/uwsgi.ini and run:

systemctl enable nginx uwsgi
systemctl start nginx uwsgi

Verify the service is running with systemctl status uwsgi

This is enough to get an app working, if you disabled SELinux (if you want to do it, edit /etc/selinux/config and reboot), but if you want to keep SELinux happy, you need to do the following:

setenforce 0
chcon -R system_u:system_r:httpd_t:s0 /srv/myapp/appdata/static
setenforce 1

We now need to install a SELinux policy (that I created for this project). If it doesn't work, look into audit2allow.

semodule -i nginx-uwsgi.pp

Hopefully, this is enough. In case it isn't, please read SELinux documentation, and check audit logs.

Also if you're on Fedora, to make your website accessible from the outside Internet, you need to configure the built-in firewall accordingly - for ports 80/443, use:

firewall-cmd --add-service http
firewall-cmd --add-service https

For Ubuntu

Ubuntu does not ship the uWSGI Emperor service by default. However, you can easily create it. Copy the .service file from the uWSGI systemd documentation to /etc/systemd/system/emperor.uwsgi.service. Change the ExecStart line to:

ExecStart=/usr/bin/uwsgi --ini /etc/uwsgi-emperor/emperor.ini

You can now reload systemd daemons and enable the services:

systemctl daemon-reload
systemctl enable nginx emperor.uwsgi
systemctl start nginx emperor.uwsgi

Verify the service is running with systemctl status emperor.uwsgi

Testing - end result

Your web service should now be running at http://localhost:8080/.

If you used the demo application, you should see something like this (complete with the favicon and image greeting):

/images/nginx-uwsgi-demo.png

Hopefully, everything works. If it doesn't, check nginx and uwsgi logs for details, and make sure you followed all instructions.


For easy linking, I set up some aliases: https://go.chriswarrick.com/pyweb and https://go.chriswarrick.com/uwsgi-tut (powered by a Django web application, deployed with nginx and uwsgi!)

Update 2016-02-10 17:00 UTC: This guide uses nginx and uWSGI, because they are considered best practices by most people. nginx is a fast, modern web server, with uWSGI support built in (without resorting to reverse proxying). uWSGI is similarly aimed at speed. The Emperor mode of uWSGI is recommended for init system integration by the uWSGI team, and it's especially useful for multi-app deployments. (This guide is opinionated.)

[1] This reflink gives you $10 in credit, which is enough to run a server for up to two months without paying a thing. I earn $15.
[2] If you're in the EU (and thus have to pay VAT), or want DO to handle your backups, it will cost you a little more.
[3] Ubuntu 14.04 LTS does not use systemd - you're on your own (upstart services exist, figure out how to use them yourself). Note that other software might be outdated as well - proceed with care, or just use something more modern.

10 Feb 2016 2:00pm GMT

Chris Warrick: Deploying Python Web Applications with nginx and uWSGI Emperor

You just wrote a great Python web application. Now, you want to share it with the world. In order to do that, you need a server, and some software to do that for you.

The following is a comprehensive guide on how to accomplish that, on multiple Linux-based operating systems, using nginx and uWSGI Emperor. It doesn't force you to use any specific web framework - Flask, Django, Pyramid, Bottle will all work. Written for Ubuntu, Fedora and Arch Linux (should be helpful for other systems, too)

Getting Started

In order to deploy your web application, you need a server that gives you root and ssh access - in other words, a VPS (or a dedicated server, or a datacenter lease…). If you're looking for a great VPS service for a low price, I recommend DigitalOcean (reflink [1]), which offers a $5/mo service [2]. If you want to play along at home, without buying a VPS, you can create a virtual machine on your own, or use a Vagrant with a Vagrant box for Fedora 23 (I recommend disabling SELinux, more on that later).

Your server should also run a modern Linux-based operating system. I tested and wrote this guide for Ubuntu 15.10 [3], Fedora 23 and Arch Linux, but other Linux distributions (and perhaps *BSD) will work (in places where the instructions are split three-way, try coming up with your own, reading documentation and config files). Unfortunately, all Linux distributions have their own ideas when it comes to running and managing nginx and UWSGI.

Note

All the commands in this tutorial are meant to be run as root - run su or sudo su first to get an administrative shell.

Start by installing virtualenv, nginx and uWSGI. I recommend using your operating system packages. For uWSGI, we need the logfile and python3 plugins. (Arch Linux names the python3 plugin python; the logfile plugin may be built-in - check with your system repositories!).

Ubuntu:

aptitude install virtualenv python3 uwsgi uwsgi-emperor uwsgi-plugin-python3 nginx-full

Fedora:

dnf install python3-virtualenv uwsgi uwsgi-plugin-python3 uwsgi-logger-file nginx

Arch Linux:

pacman -S python-virtualenv uwsgi uwsgi-plugin-python nginx

Preparing your application

This tutorial will work for any web framework. I will, use a really basic Flask app that has just one route (/), a static hello.png file and a favicon.ico for demonstration purposes. Note that the app does not use app.run(). While you could add it, it would be used for local development and debugging only, and would be prepended by if __name__ == '__main__': - uWSGI doesn't work alongside it.

The app will be installed somewhere under the /srv directory, which is a great place to store things like this. I'll choose /srv/myapp for this tutorial, but for real deployments, you should use sometihing more distinguishable - the domain name is a great idea.

We'll start by creating a virtualenv:

Ubuntu:

cd srv
virtualenv -p /usr/bin/python3 myapp

Fedora:

cd /srv
virtualenv-3.4 myapp

Arch Linux:

cd /srv
virtualenv3 myapp

(Make sure you create a Python 3 environment!)

Now, we need to get our app there and install requirements. An example for the tutorial demo app (adjust for your clone/download path):

cd myapp
cp -r ~/git/flask-demo-app appdata
bin/pip install -r appdata/requirements.txt

I'm storing my application data in the appdata subdirectory so that it doesn't clutter the virtualenv (or vice versa). You may also install the uwsgi package in the virtualenv, but it's optional.

What this directory should be depends on your web framework. For example, for a Django app, you should have an appdata/manage.py file (in other words, appdata is where your app structure starts). I also assumed that the appdata folder should have a static subdirectory with all static files, including favicon.ico if you have one (we will add support for both in nginx).

At this point, you should chown this directory to the user and group your server is going to run as. This is especially important if uwsgi and nginx run as different users (as they do on Fedora). Run one of the following commands:

Ubuntu:

chown -R www-data:www-data /srv/myapp

Fedora:

chown -R uwsgi:nginx /srv/myapp

Arch Linux:

chown -R http:http /srv/myapp

Configuring uWSGI and nginx

Note

Parts of the configuration depend on your operating system. I tried to provide advice for Ubuntu, Fedora and Arch Linux. If you experience any issues, in particular with plugins, please consult the documentation.

We need to write a configuration file for uWSGI and nginx.

uWSGI configuration

Start with this, but read the notes below and change the values accordingly:

[uwsgi]
socket = /srv/myapp/uwsgi.sock
chmod-socket = 775
chdir = /srv/myapp/appdata
master = true
binary-path = /srv/myapp/bin/uwsgi
virtualenv = /srv/myapp
module = flaskapp:app
uid = www-data
gid = www-data
processes = 1
threads = 1
plugins = python3,logfile
logger = file:/srv/myapp/uwsgi.log

Save this file as:

  • Ubuntu: /etc/uwsgi-emperor/vassals/myapp.ini
  • Fedora: /etc/uwsgi.d/myapp.ini
  • Arch Linux: /etc/uwsgi/vassals/myapp.ini (create the directory first and chown it to http: mkdir -p /etc/uwsgi/vassals; chown -R http:http /etc/uwsgi/vassals)

The options are:

  • socket - the socket file that will be used by your application. It's usually a file path (Unix domain socket). You could use a local TCP socket, but it's not recommended.
  • chdir - the app directory.
  • binary-path - the uWSGI executable to use. Remove if you didn't install the (optional) uwsgi package in your virtualenv.
  • virtualenv - the virtualenv for your application.
  • module - the name of the module that houses your application, and the object that speaks the WSGI interface, separated by colons. This depends on your web framework:
    • For Flask: module = filename:app, where filename is the name of your Python file (without the .py part) and app is the Flask object
    • For Django: module = project.wsgi:application, where project is the name of your project (directory with settings.py). You should also add an environment variable: env = DJANGO_SETTINGS_MODULE=project.settings
    • For Bottle: module = filename:app, where app = bottle.default_app()
    • For Pyramid: module = filename:app, where app = config.make_wsgi_app() (make sure it's not in a if __name__ == '__main__': block - the demo app does that!)
  • uid and gid - the names of the user account to use for your server. Use the same values as in the chown command above.
  • processes and threads - control the resources devoted to this application. Because this is a simple hello app, I used one process with one thread, but for a real app, you will probably need more (you need to see what works the best; there is no algorithm to decide). Also, remember that if you use multiple processes, they don't share data, so you need an out-of-process database if you want that.
  • plugins - the list of uWSGI plugins to use. For Arch Linux, use plugins = python (the logfile plugin is always active).
  • logger - the path to your app-specific logfile. (Other logging facilities are available, but this one is the easiest, especially for multiple applications on the same server)

You can test your configuration by running uwsgi --ini /path/to/myapp.ini (disable the logger for stderr output or run tail -f /srv/myapp/uwsgi.log in another window).

If you're using Fedora, there are two configuration changes you need to make globally: in /etc/uwsgi.ini, disable the emperor-tyrant option (which seems to be buggy) and set gid = nginx. We'll need this so that nginx can talk to your socket.

nginx configuration

We need to configure our web server. Here's a basic configuration that will get us started:

Save this file as:

  • Ubuntu: /etc/nginx/sites-enabled/myapp.conf
  • Fedora: /etc/nginx/conf.d/myapp.conf
  • Arch Linux: add include /etc/nginx/conf.d/*.conf; to your http directive in /etc/nginx/nginx.conf and use /etc/nginx/conf.d/myapp.conf
server {
    listen 8080;
    server_name localhost myapp.local;

    location / {
        include uwsgi_params;
        uwsgi_pass unix:/srv/myapp/uwsgi.sock;
    }

    location /static {
        alias /srv/myapp/appdata/static;
    }

    location /favicon.ico {
        alias /srv/myapp/appdata/static/favicon.ico;
    }
}

Note that this file is a very basic and rudimentary configuration. This configuration is fine for local testing, but for a real deployment, you will need to adjust it:

  • set listen to 443 ssl and create a http→https redirect on port 80 (you can get a free SSL certificate from Let's Encrypt; make sure to configure SSL properly).
  • set server_name to your real domain name
  • you might also want to add custom error pages, or change anything else that relates to your web server - consult other nginx guides for details
  • nginx might have some server already enabled by default - edit /etc/nginx/nginx.conf to disable it

Service setup

After you've configured uWSGI and nginx, you need to enable and start the system services.

I'm going to use systemd here. If your system does not support systemd, please consult your OS documentation for instructions.

For Arch Linux

All you need is:

systemctl enable nginx emperor.uwsgi
systemctl start nginx emperor.uwsgi

Verify the service is running with systemctl status emperor.uwsgi

For Fedora

Make sure you followed the extra note about editing /etc/uwsgi.ini and run:

systemctl enable nginx uwsgi
systemctl start nginx uwsgi

Verify the service is running with systemctl status uwsgi

This is enough to get an app working, if you disabled SELinux (if you want to do it, edit /etc/selinux/config and reboot), but if you want to keep SELinux happy, you need to do the following:

setenforce 0
chcon -R system_u:system_r:httpd_t:s0 /srv/myapp/appdata/static
setenforce 1

We now need to install a SELinux policy (that I created for this project). If it doesn't work, look into audit2allow.

semodule -i nginx-uwsgi.pp

Hopefully, this is enough. In case it isn't, please read SELinux documentation, and check audit logs.

Also if you're on Fedora, to make your website accessible from the outside Internet, you need to configure the built-in firewall accordingly - for ports 80/443, use:

firewall-cmd --add-service http
firewall-cmd --add-service https

For Ubuntu

Ubuntu does not ship the uWSGI Emperor service by default. However, you can easily create it. Copy the .service file from the uWSGI systemd documentation to /etc/systemd/system/emperor.uwsgi.service. Change the ExecStart line to:

ExecStart=/usr/bin/uwsgi --ini /etc/uwsgi-emperor/emperor.ini

You can now reload systemd daemons and enable the services:

systemctl daemon-reload
systemctl enable nginx emperor.uwsgi
systemctl start nginx emperor.uwsgi

Verify the service is running with systemctl status emperor.uwsgi

Testing - end result

Your web service should now be running at http://localhost:8080/.

If you used the demo application, you should see something like this (complete with the favicon and image greeting):

/images/nginx-uwsgi-demo.png

Hopefully, everything works. If it doesn't, check nginx and uwsgi logs for details, and make sure you followed all instructions.


For easy linking, I set up some aliases: https://go.chriswarrick.com/pyweb and https://go.chriswarrick.com/uwsgi-tut (powered by a Django web application, deployed with nginx and uwsgi!)

Update 2016-02-10 17:00 UTC: This guide uses nginx and uWSGI, because they are considered best practices by most people. nginx is a fast, modern web server, with uWSGI support built in (without resorting to reverse proxying). uWSGI is similarly aimed at speed. The Emperor mode of uWSGI is recommended for init system integration by the uWSGI team, and it's especially useful for multi-app deployments. (This guide is opinionated.)

[1] This reflink gives you $10 in credit, which is enough to run a server for up to two months without paying a thing. I earn $15.
[2] If you're in the EU (and thus have to pay VAT), or want DO to handle your backups, it will cost you a little more.
[3] Ubuntu 14.04 LTS does not use systemd - you're on your own (upstart services exist, figure out how to use them yourself). Note that other software might be outdated as well - proceed with care, or just use something more modern.

10 Feb 2016 2:00pm GMT

John Cook: Musical pitch notation

How can you convert the frequency of a sound to musical notation? I wrote in an earlier post how to calculate how many half steps a frequency is above or below middle C, but it would be useful go further have code to output musical pitch notation.

In scientific pitch notation, the C near the threshold of hearing, around 16 Hz, is called C0. The C an octave higher is C1, the next C2, etc. Octaves begin with C; other notes use the octave number of the closest C below.

C4, middle C

The lowest note on a piano is A0, a major sixth up from C0. Middle C is C4 because it's 4 octaves above C0. The highest note on a piano is C8.

Math

A4, the A above middle C, has a frequency of 440 Hz. This is nine half steps above C4, so the pitch of C4 is 440*2-9/12. C0 is four octaves lower, so it's 2-4 = 1/16 of the pitch of C4. (Details for this calculation and the one below are given in here.)

For a pitch P, the number of half steps from C0 to P is

h = 12 log2(P / C0).

Software

The Python code below calculates the number of half steps h from C0 up to a pitch, then computes the corresponding pitch notation.

from math import log2, pow

A4 = 440
C0 = A4*pow(2, -4.75)
name = ["C", "C#", "D", "D#", "E", "F", "F#", "G", "G#", "A", "A#", "B"]
    
def pitch(freq):
    h = round(12*log2(freq/C0))
    octave = h // 12
    n = h % 12
    return name[n] + str(octave)

The pitch for A4 is its own variable in case you'd like to modify the code for a different tuning. While 440 is common, it used to be lower in the past, and you'll sometimes see higher values like 444 today.

If you'd like to port this code to a language that doesn't have a log2 function, you can use log(x)/log(2) for log2(x).

Powers of 2

When scientific pitch notation was first introduced, C0 was defined to be exactly 16 Hz, whereas now it works out to around 16.35. The advantage of the original system is that all C's have frequency a power of 2, i.e. Cn has frequency 2n+4 Hz. The formula above for the number of half steps a pitch is above C0 simplifies to

h = 12 log2P - 48.

If C0 has frequency 16 Hz, the A above middle C has frequency 28.75 = 430.54, a little flat compared to A 440. But using the A 440 standard, C0 = 16 Hz is a convenient and fairly accurate approximation.

Related posts

10 Feb 2016 1:02pm GMT

John Cook: Musical pitch notation

How can you convert the frequency of a sound to musical notation? I wrote in an earlier post how to calculate how many half steps a frequency is above or below middle C, but it would be useful go further have code to output musical pitch notation.

In scientific pitch notation, the C near the threshold of hearing, around 16 Hz, is called C0. The C an octave higher is C1, the next C2, etc. Octaves begin with C; other notes use the octave number of the closest C below.

C4, middle C

The lowest note on a piano is A0, a major sixth up from C0. Middle C is C4 because it's 4 octaves above C0. The highest note on a piano is C8.

Math

A4, the A above middle C, has a frequency of 440 Hz. This is nine half steps above C4, so the pitch of C4 is 440*2-9/12. C0 is four octaves lower, so it's 2-4 = 1/16 of the pitch of C4. (Details for this calculation and the one below are given in here.)

For a pitch P, the number of half steps from C0 to P is

h = 12 log2(P / C0).

Software

The Python code below calculates the number of half steps h from C0 up to a pitch, then computes the corresponding pitch notation.

from math import log2, pow

A4 = 440
C0 = A4*pow(2, -4.75)
name = ["C", "C#", "D", "D#", "E", "F", "F#", "G", "G#", "A", "A#", "B"]
    
def pitch(freq):
    h = round(12*log2(freq/C0))
    octave = h // 12
    n = h % 12
    return name[n] + str(octave)

The pitch for A4 is its own variable in case you'd like to modify the code for a different tuning. While 440 is common, it used to be lower in the past, and you'll sometimes see higher values like 444 today.

If you'd like to port this code to a language that doesn't have a log2 function, you can use log(x)/log(2) for log2(x).

Powers of 2

When scientific pitch notation was first introduced, C0 was defined to be exactly 16 Hz, whereas now it works out to around 16.35. The advantage of the original system is that all C's have frequency a power of 2, i.e. Cn has frequency 2n+4 Hz. The formula above for the number of half steps a pitch is above C0 simplifies to

h = 12 log2P - 48.

If C0 has frequency 16 Hz, the A above middle C has frequency 28.75 = 430.54, a little flat compared to A 440. But using the A 440 standard, C0 = 16 Hz is a convenient and fairly accurate approximation.

Related posts

10 Feb 2016 1:02pm GMT

A. Jesse Jiryu Davis: Do Your Slides At The Last Minute: 8 Steps To Writing Your PyCon Talk

Jennie Baines addressing a rally of Suffragettes at Trafalgar Square, 1908

Jennie Baines didn't take questions at the end.

PyCon accepted my talk "Write an Excellent Programming Blog". If you got in, too, congratulations! Now we have to write our talks.

Plan your time.

Plan to begin your talk late, and not to take questions.

This is not the consensus approach to conference talks, so I'll argue for it in detail when we get to your actual delivery of the talk, in a future article. Briefly: The beginning and end are the most important parts of your talk. You ruin the beginning by starting before people are ready to listen, and you ruin the end, as often as not, by wincing through a couple bad questions and finishing on a low note.

Better to start a few minutes late, when the audience is ready and waiting for you. At the end, go out with a bang.

Now that you've decided when you'll begin and end, you know your real slot. I have a putative half hour slot at PyCon, so I'll write my talk as if I have 25 minutes.

Inspire.

Every great talk contains an element of inspiration, I believe, and a call to action. Even though most PyCon talks are technical-they explain how something works, or how to do something-you must still inspire us somehow. "Go try this yourself" is a good call to action for almost any technical talk, but "stop doing this wrong and start doing it right", or "do more of this" can also work. If it's a non-technical talk, you might call on us to "be generous" or "nurture our community" or "advance in your career".

Organized her talk around a call to action.

So decide on a call to action. (I put mine in the title: "Write an Excellent Programming Blog".) Then, determine how to motivate us to respond to your call to action. What is the outcome if we do? Will we have more fun, be better programmers, be better humans? I may spend more than half my talk on motivation, rather than on how-to. After all, a conference talk is a poor vehicle for teaching technical details, but it's pretty good for inspiring people.

Once you have a call to action, build a talk whose purpose is to explain why your call to action matters, and inspires your audience to respond to it.

Outline.

If your proposal was thorough enough to be accepted by PyCon, great: you have already outlined your talk! It's a good first draft. Now revise it.

But some conferences' Calls For Proposals are barely two text boxes. (I think these CFPs are a disservice to everyone.) If this is all you've done, go back to my tips for writing a proposal and plan your talk to a PyCon level of rigor. Outline your talk the same as you outline an essay: decide if you are making an argument, or a telling story, or explaining a how-to. Brainstorm as many ideas as you can, then choose the best and arrange them to make an impact.

Finished his talk with a great kicker.

How do you arrange the talk? Same as an article, I think:

Notice what's missing? Introducing yourself. Those opening moments are key, and "who am I" just wastes them. I might say only "I'm a Python and C programmer at MongoDB" and leave it at that, or not introduce myself at all. The audience doesn't want to know who I am, they want to hear my idea!

No one cared how he earned his medals.

Rehearse immediately.

Before your outline is close to finished, start rehearsing it. Stand up and say stuff to the walls. Discover what you really know and care about, by listening to the parts of your talk that flow freely. (This is not my original thought: I know that I have read this on someone else's blog but I can't find it.) If you spend too much time outlining before you begin talking to the walls, your outline becomes a prison for your real passion.

Obsessed over his outline.

I know you've sat through boring talks. The speaker wanted to be there, originally-where did that passion go? It might have got imprisoned during the outlining phase.

Make room for new insights.

So don't let this happen to you! Extemporize to the walls, rehearse in your head while you ride the subway, do research, and think more about your topic. What is it that drives you to risk your dignity on stage at a conference? It's doubtless evolved since you proposed your talk; after all, you wrote the proposal months ago.

What is the thought have now that's so exciting you can't contain it? Once you're certain of the answer, then you can finish your outline.

Put off making slides.

Besides outlines, slides are the other enemy of your actual talk. For me, slides and outlines are dangerous in different ways: outlining curdles my thinking before it's ready, whereas slides are a way to procrastinate the task of thinking at all.

Painter Felix Ziem sitting at his easel

Worked on his slides so long he never wrote his talk.

It's not your fine slide art that will move the audience, it's the clarity and passion of your thinking. Try to rely only on a rough outline as long as possible while rehearsing your talk. It leaves you free to rearrange your talk, or overhaul it to reflect a new insight. If you have a week left before the conference and you haven't finished your slides, don't worry-it might be better that way, it forces you to keep them simple.

Just make sure that you've finished thinking by then!

Rehearse with friends.

You'll know if you really care about your talk, and if you really mean it, when you deliver it face-to-face with friends. Your friends don't have to be tough critics: you'll just get that icky dead feeling if you're giving a rote speech to someone close. Use that feeling. Go back and find where you left your passion, and pick it up again.

I'll trade rehearsals with my friends like Anna Herlihy and Amy Hanlon before PyCon this spring. They've spoken at conferences and they know my subjects, my audience, and me-they'll point out how I can improve.

Spent the soirée rehearsing their PyCon talks for each other.

Get a coach.

Friends help, but a professional speaking coach takes you to a different level. My friend and coach Melissa Collom helps me craft my argument and delivery (and she can help you with yours-shameless plug). She's an opera singer, so she can teach me to command a stage and use my voice, but she's also got a knack for structuring a story or an opinion. In my sessions with her I don't just refine the talk I'm prepping now; I learn techniques for future talks, and for any kind of persuasive speaking. Besides, working with her is a lot more fun than rehearsing to the walls.

Excel.

Your talk doesn't have to be great. I think that most software conferences, unfortunately, set a low bar. PyCon's standards are higher, but still-you won't be sent away for being dull.

But what if you want to give a great talk? It won't happen automatically. You have to think hard, discover the idea that you really care about, and write an inspiring talk that motivates us to respond to your call to action. And then, prepare as you would for any performance, like singing or ballet: get a coach. Plan, train, and practice.

I'm looking forward to your talk!


Images:

10 Feb 2016 12:23pm GMT

A. Jesse Jiryu Davis: Do Your Slides At The Last Minute: 8 Steps To Writing Your PyCon Talk

Jennie Baines addressing a rally of Suffragettes at Trafalgar Square, 1908

Jennie Baines didn't take questions at the end.

PyCon accepted my talk "Write an Excellent Programming Blog". If you got in, too, congratulations! Now we have to write our talks.

Plan your time.

Plan to begin your talk late, and not to take questions.

This is not the consensus approach to conference talks, so I'll argue for it in detail when we get to your actual delivery of the talk, in a future article. Briefly: The beginning and end are the most important parts of your talk. You ruin the beginning by starting before people are ready to listen, and you ruin the end, as often as not, by wincing through a couple bad questions and finishing on a low note.

Better to start a few minutes late, when the audience is ready and waiting for you. At the end, go out with a bang.

Now that you've decided when you'll begin and end, you know your real slot. I have a putative half hour slot at PyCon, so I'll write my talk as if I have 25 minutes.

Inspire.

Every great talk contains an element of inspiration, I believe, and a call to action. Even though most PyCon talks are technical-they explain how something works, or how to do something-you must still inspire us somehow. "Go try this yourself" is a good call to action for almost any technical talk, but "stop doing this wrong and start doing it right", or "do more of this" can also work. If it's a non-technical talk, you might call on us to "be generous" or "nurture our community" or "advance in your career".

Organized her talk around a call to action.

So decide on a call to action. (I put mine in the title: "Write an Excellent Programming Blog".) Then, determine how to motivate us to respond to your call to action. What is the outcome if we do? Will we have more fun, be better programmers, be better humans? I may spend more than half my talk on motivation, rather than on how-to. After all, a conference talk is a poor vehicle for teaching technical details, but it's pretty good for inspiring people.

Once you have a call to action, build a talk whose purpose is to explain why your call to action matters, and inspires your audience to respond to it.

Outline.

If your proposal was thorough enough to be accepted by PyCon, great: you have already outlined your talk! It's a good first draft. Now revise it.

But some conferences' Calls For Proposals are barely two text boxes. (I think these CFPs are a disservice to everyone.) If this is all you've done, go back to my tips for writing a proposal and plan your talk to a PyCon level of rigor. Outline your talk the same as you outline an essay: decide if you are making an argument, or a telling story, or explaining a how-to. Brainstorm as many ideas as you can, then choose the best and arrange them to make an impact.

Finished his talk with a great kicker.

How do you arrange the talk? Same as an article, I think:

Notice what's missing? Introducing yourself. Those opening moments are key, and "who am I" just wastes them. I might say only "I'm a Python and C programmer at MongoDB" and leave it at that, or not introduce myself at all. The audience doesn't want to know who I am, they want to hear my idea!

No one cared how he earned his medals.

Rehearse immediately.

Before your outline is close to finished, start rehearsing it. Stand up and say stuff to the walls. Discover what you really know and care about, by listening to the parts of your talk that flow freely. (This is not my original thought: I know that I have read this on someone else's blog but I can't find it.) If you spend too much time outlining before you begin talking to the walls, your outline becomes a prison for your real passion.

Obsessed over his outline.

I know you've sat through boring talks. The speaker wanted to be there, originally-where did that passion go? It might have got imprisoned during the outlining phase.

Make room for new insights.

So don't let this happen to you! Extemporize to the walls, rehearse in your head while you ride the subway, do research, and think more about your topic. What is it that drives you to risk your dignity on stage at a conference? It's doubtless evolved since you proposed your talk; after all, you wrote the proposal months ago.

What is the thought have now that's so exciting you can't contain it? Once you're certain of the answer, then you can finish your outline.

Put off making slides.

Besides outlines, slides are the other enemy of your actual talk. For me, slides and outlines are dangerous in different ways: outlining curdles my thinking before it's ready, whereas slides are a way to procrastinate the task of thinking at all.

Painter Felix Ziem sitting at his easel

Worked on his slides so long he never wrote his talk.

It's not your fine slide art that will move the audience, it's the clarity and passion of your thinking. Try to rely only on a rough outline as long as possible while rehearsing your talk. It leaves you free to rearrange your talk, or overhaul it to reflect a new insight. If you have a week left before the conference and you haven't finished your slides, don't worry-it might be better that way, it forces you to keep them simple.

Just make sure that you've finished thinking by then!

Rehearse with friends.

You'll know if you really care about your talk, and if you really mean it, when you deliver it face-to-face with friends. Your friends don't have to be tough critics: you'll just get that icky dead feeling if you're giving a rote speech to someone close. Use that feeling. Go back and find where you left your passion, and pick it up again.

I'll trade rehearsals with my friends like Anna Herlihy and Amy Hanlon before PyCon this spring. They've spoken at conferences and they know my subjects, my audience, and me-they'll point out how I can improve.

Spent the soirée rehearsing their PyCon talks for each other.

Get a coach.

Friends help, but a professional speaking coach takes you to a different level. My friend and coach Melissa Collom helps me craft my argument and delivery (and she can help you with yours-shameless plug). She's an opera singer, so she can teach me to command a stage and use my voice, but she's also got a knack for structuring a story or an opinion. In my sessions with her I don't just refine the talk I'm prepping now; I learn techniques for future talks, and for any kind of persuasive speaking. Besides, working with her is a lot more fun than rehearsing to the walls.

Excel.

Your talk doesn't have to be great. I think that most software conferences, unfortunately, set a low bar. PyCon's standards are higher, but still-you won't be sent away for being dull.

But what if you want to give a great talk? It won't happen automatically. You have to think hard, discover the idea that you really care about, and write an inspiring talk that motivates us to respond to your call to action. And then, prepare as you would for any performance, like singing or ballet: get a coach. Plan, train, and practice.

I'm looking forward to your talk!


Images:

10 Feb 2016 12:23pm GMT

Reinout van Rees: Just one comma

In Django, if you say managed = False in a model's Meta, you tell Django not to touch the database table. So: no automatic database migrations, for instance.

Now, what is the problem if you have managed = False, and Django does do migrations?

Some of you will have seen the error already.

The comma after False is the problem:

>>> a = False,
>>> a
(False,)
>>> bool(a)
True

The comma turns it into a tuple. And a non-empty tuple evaluates to True!

I found it quite funny. My colleague was torn between "extremely relieved" and "extremely annoyed" :-)

10 Feb 2016 11:28am GMT

Reinout van Rees: Just one comma

In Django, if you say managed = False in a model's Meta, you tell Django not to touch the database table. So: no automatic database migrations, for instance.

Now, what is the problem if you have managed = False, and Django does do migrations?

Some of you will have seen the error already.

The comma after False is the problem:

>>> a = False,
>>> a
(False,)
>>> bool(a)
True

The comma turns it into a tuple. And a non-empty tuple evaluates to True!

I found it quite funny. My colleague was torn between "extremely relieved" and "extremely annoyed" :-)

10 Feb 2016 11:28am GMT

10 Nov 2011

feedPython Software Foundation | GSoC'11 Students

Benedict Stein: King Willams Town Bahnhof

Gestern musste ich morgens zur Station nach KWT um unsere Rerservierten Bustickets für die Weihnachtsferien in Capetown abzuholen. Der Bahnhof selber ist seit Dezember aus kostengründen ohne Zugverbindung - aber Translux und co - die langdistanzbusse haben dort ihre Büros.


Größere Kartenansicht




© benste CC NC SA

10 Nov 2011 10:57am GMT

09 Nov 2011

feedPython Software Foundation | GSoC'11 Students

Benedict Stein

Niemand ist besorgt um so was - mit dem Auto fährt man einfach durch, und in der City - nahe Gnobie- "ne das ist erst gefährlich wenn die Feuerwehr da ist" - 30min später auf dem Rückweg war die Feuerwehr da.




© benste CC NC SA

09 Nov 2011 8:25pm GMT

08 Nov 2011

feedPython Software Foundation | GSoC'11 Students

Benedict Stein: Brai Party

Brai = Grillabend o.ä.

Die möchte gern Techniker beim Flicken ihrer SpeakOn / Klinke Stecker Verzweigungen...

Die Damen "Mamas" der Siedlung bei der offiziellen Eröffnungsrede

Auch wenn weniger Leute da waren als erwartet, Laute Musik und viele Leute ...

Und natürlich ein Feuer mit echtem Holz zum Grillen.

© benste CC NC SA

08 Nov 2011 2:30pm GMT

07 Nov 2011

feedPython Software Foundation | GSoC'11 Students

Benedict Stein: Lumanyano Primary

One of our missions was bringing Katja's Linux Server back to her room. While doing that we saw her new decoration.

Björn, Simphiwe carried the PC to Katja's school


© benste CC NC SA

07 Nov 2011 2:00pm GMT

06 Nov 2011

feedPython Software Foundation | GSoC'11 Students

Benedict Stein: Nelisa Haircut

Today I went with Björn to Needs Camp to Visit Katja's guest family for a special Party. First of all we visited some friends of Nelisa - yeah the one I'm working with in Quigney - Katja's guest fathers sister - who did her a haircut.

African Women usually get their hair done by arranging extensions and not like Europeans just cutting some hair.

In between she looked like this...

And then she was done - looks amazing considering the amount of hair she had last week - doesn't it ?

© benste CC NC SA

06 Nov 2011 7:45pm GMT

05 Nov 2011

feedPython Software Foundation | GSoC'11 Students

Benedict Stein: Mein Samstag

Irgendwie viel mir heute auf das ich meine Blogposts mal ein bischen umstrukturieren muss - wenn ich immer nur von neuen Plätzen berichte, dann müsste ich ja eine Rundreise machen. Hier also mal ein paar Sachen aus meinem heutigen Alltag.

Erst einmal vorweg, Samstag zählt zumindest für uns Voluntäre zu den freien Tagen.

Dieses Wochenende sind nur Rommel und ich auf der Farm - Katja und Björn sind ja mittlerweile in ihren Einsatzstellen, und meine Mitbewohner Kyle und Jonathan sind zu Hause in Grahamstown - sowie auch Sipho der in Dimbaza wohnt.
Robin, die Frau von Rommel ist in Woodie Cape - schon seit Donnerstag um da ein paar Sachen zur erledigen.
Naja wie dem auch sei heute morgen haben wir uns erstmal ein gemeinsames Weetbix/Müsli Frühstück gegönnt und haben uns dann auf den Weg nach East London gemacht. 2 Sachen waren auf der Checkliste Vodacom, Ethienne (Imobilienmakler) außerdem auf dem Rückweg die fehlenden Dinge nach NeedsCamp bringen.

Nachdem wir gerade auf der Dirtroad losgefahren sind mussten wir feststellen das wir die Sachen für Needscamp und Ethienne nicht eingepackt hatten aber die Pumpe für die Wasserversorgung im Auto hatten.

Also sind wir in EastLondon ersteinmal nach Farmerama - nein nicht das onlinespiel farmville - sondern einen Laden mit ganz vielen Sachen für eine Farm - in Berea einem nördlichen Stadteil gefahren.

In Farmerama haben wir uns dann beraten lassen für einen Schnellverschluss der uns das leben mit der Pumpe leichter machen soll und außerdem eine leichtere Pumpe zur Reperatur gebracht, damit es nicht immer so ein großer Aufwand ist, wenn mal wieder das Wasser ausgegangen ist.

Fego Caffé ist in der Hemmingways Mall, dort mussten wir und PIN und PUK einer unserer Datensimcards geben lassen, da bei der PIN Abfrage leider ein zahlendreher unterlaufen ist. Naja auf jeden Fall speichern die Shops in Südafrika so sensible Daten wie eine PUK - die im Prinzip zugang zu einem gesperrten Phone verschafft.

Im Cafe hat Rommel dann ein paar online Transaktionen mit dem 3G Modem durchgeführt, welches ja jetzt wieder funktionierte - und übrigens mittlerweile in Ubuntu meinem Linuxsystem perfekt klappt.

Nebenbei bin ich nach 8ta gegangen um dort etwas über deren neue Deals zu erfahren, da wir in einigen von Hilltops Centern Internet anbieten wollen. Das Bild zeigt die Abdeckung UMTS in NeedsCamp Katjas Ort. 8ta ist ein neuer Telefonanbieter von Telkom, nachdem Vodafone sich Telkoms anteile an Vodacom gekauft hat müssen die komplett neu aufbauen.
Wir haben uns dazu entschieden mal eine kostenlose Prepaidkarte zu testen zu organisieren, denn wer weis wie genau die Karte oben ist ... Bevor man einen noch so billigen Deal für 24 Monate signed sollte man wissen obs geht.

Danach gings nach Checkers in Vincent, gesucht wurden zwei Hotplates für WoodyCape - R 129.00 eine - also ca. 12€ für eine zweigeteilte Kochplatte.
Wie man sieht im Hintergrund gibts schon Weihnachtsdeko - Anfang November und das in Südafrika bei sonnig warmen min- 25°C

Mittagessen haben wir uns bei einem Pakistanischen Curry Imbiss gegönnt - sehr empfehlenswert !
Naja und nachdem wir dann vor ner Stunde oder so zurück gekommen sind habe ich noch den Kühlschrank geputzt den ich heute morgen zum defrosten einfach nach draußen gestellt hatte. Jetzt ist der auch mal wieder sauber und ohne 3m dicke Eisschicht...

Morgen ... ja darüber werde ich gesondert berichten ... aber vermutlich erst am Montag, denn dann bin ich nochmal wieder in Quigney(East London) und habe kostenloses Internet.

© benste CC NC SA

05 Nov 2011 4:33pm GMT

31 Oct 2011

feedPython Software Foundation | GSoC'11 Students

Benedict Stein: Sterkspruit Computer Center

Sterkspruit is one of Hilltops Computer Centres in the far north of Eastern Cape. On the trip to J'burg we've used the opportunity to take a look at the centre.

Pupils in the big classroom


The Trainer


School in Countryside


Adult Class in the Afternoon


"Town"


© benste CC NC SA

31 Oct 2011 4:58pm GMT

Benedict Stein: Technical Issues

What are you doing in an internet cafe if your ADSL and Faxline has been discontinued before months end. Well my idea was sitting outside and eating some ice cream.
At least it's sunny and not as rainy as on the weekend.


© benste CC NC SA

31 Oct 2011 3:11pm GMT

30 Oct 2011

feedPython Software Foundation | GSoC'11 Students

Benedict Stein: Nellis Restaurant

For those who are traveling through Zastron - there is a very nice Restaurant which is serving delicious food at reasanable prices.
In addition they're selling home made juices jams and honey.




interior


home made specialities - the shop in the shop


the Bar


© benste CC NC SA

30 Oct 2011 4:47pm GMT

29 Oct 2011

feedPython Software Foundation | GSoC'11 Students

Benedict Stein: The way back from J'burg

Having the 10 - 12h trip from J'burg back to ELS I was able to take a lot of pcitures including these different roadsides

Plain Street


Orange River in its beginngings (near Lesotho)


Zastron Anglican Church


The Bridge in Between "Free State" and Eastern Cape next to Zastron


my new Background ;)


If you listen to GoogleMaps you'll end up traveling 50km of gravel road - as it was just renewed we didn't have that many problems and saved 1h compared to going the official way with all it's constructions sites




Freeway


getting dark


© benste CC NC SA

29 Oct 2011 4:23pm GMT

28 Oct 2011

feedPython Software Foundation | GSoC'11 Students

Benedict Stein: Wie funktioniert eigentlich eine Baustelle ?

Klar einiges mag anders sein, vieles aber gleich - aber ein in Deutschland täglich übliches Bild einer Straßenbaustelle - wie läuft das eigentlich in Südafrika ?

Ersteinmal vorweg - NEIN keine Ureinwohner die mit den Händen graben - auch wenn hier mehr Manpower genutzt wird - sind sie fleißig mit Technologie am arbeiten.

Eine ganz normale "Bundesstraße"


und wie sie erweitert wird


gaaaanz viele LKWs


denn hier wird eine Seite über einen langen Abschnitt komplett gesperrt, so das eine Ampelschaltung mit hier 45 Minuten Wartezeit entsteht


Aber wenigstens scheinen die ihren Spaß zu haben ;) - Wie auch wir denn gücklicher Weise mussten wir nie länger als 10 min. warten.

© benste CC NC SA

28 Oct 2011 4:20pm GMT