22 Jun 2018 8:00am GMT

22 Jun 2018 8:00am GMT

I've just released qutebrowser v1.3.3, which fixes an XSS vulnerability on the qute://history page (:history).

qutebrowser is a keyboard driven browser with a vim-like, minimalistic interface. It's written using PyQt and cross-platform.

The vulnerability allowed websites to inject HTML into the page via a crafted title tag …

22 Jun 2018 12:04am GMT

I've just released qutebrowser v1.3.3, which fixes an XSS vulnerability on the qute://history page (:history).

qutebrowser is a keyboard driven browser with a vim-like, minimalistic interface. It's written using PyQt and cross-platform.

The vulnerability allowed websites to inject HTML into the page via a crafted title tag …

22 Jun 2018 12:04am GMT

I wrote an article sometime ago on the iterator protocol that powers Python's for loops. One thing I left out of that article was how to make your own iterators.

In this article I'm going to discuss why you'd want to make your own iterators and then show you how to do so.

What is an iterator?

First let's quickly address what an iterator is. For a much more detailed explanation, consider watching my Loop Better talk or reading the article based on the talk.

An iterable is anything you're able to loop over.

An iterator is the object that does the actual iterating.

You can get an iterator from any iterable by calling the built-in iter function on the iterable.

1
2
3

>>> favorite_numbers = [6, 57, 4, 7, 68, 95]
>>> iter(favorite_numbers)
<list_iterator object at 0x7fe8e5623160>

You can use the built-in next function on an iterator to get the next item from it (you'll get a StopIteration exception if there are no more items).

1
2
3
4
5
6

>>> favorite_numbers = [6, 57, 4, 7, 68, 95]
>>> my_iterator = iter(favorite_numbers)
>>> next(my_iterator)
6
>>> next(my_iterator)
57

There's one more rule about iterators that makes everything interesting: iterators are also iterables and their iterator is themselves. I explain the consequences of that more fully in that Loop Better talk I mentioned above.

Why make an iterator?

Iterators allow you to make an iterable that computes its items as it goes. Which means that you can make iterables that are lazy, in that they don't determine what their next item is until you ask them for it.

Using an iterator instead of a list, set, or another iterable data structure can sometimes allow us to save memory. For example, we can use itertools.repeat to create an iterable that provides 100 million 4's to us:

1
2

>>> from itertools import repeat
>>> lots_of_fours = repeat(4, times=100_000_000)

This iterator takes up 56 bytes of memory on my machine:

1
2
3

>>> import sys
>>> sys.getsizeof(lots_of_fours)
56

An equivalent list of 100 million 4's takes up many megabytes of memory:

1
2
3
4

>>> lots_of_fours = [4] * 100_000_000
>>> import sys
>>> sys.getsizeof(lots_of_fours)
800000064

While iterators can save memory, they can also save time. For example if you wanted to print out just the first line of a 10 gigabyte log file, you could do this:

1
2

>>> print(next(open('giant_log_file.txt')))
This is the first line in a giant file

File objects in Python are implemented iterators. As you loop over a file, data is read into memory one line at a time. If we instead used the readlines method to store all lines in memory, we might run out of system memory.

So iterators can save us memory, but iterators can sometimes save us time also.

Additionally, iterators have abilities that other iterables don't. For example, the laziness of iterables can be used to make iterables that have an unknown length. In fact, you can even make infinitely long iterators.

For example, the itertools.count utility will give us an iterator that will provide every number from 0 upward as we loop over it:

1
2
3
4
5
6
7
8

>>> from itertools import count
>>> for n in count():
...     print(n)
...
0
1
2
(this goes on forever)

That itertools.count object is essentially an infinitely long iterable. And it's implemented as an iterator.

Making an iterator: the object-oriented way

So we've seen that iterators can save us memory, save us CPU time, and unlock new abilities to us.

Let's make our own iterators. We'll start be re-inventing the itertools.count iterator object.

Here's an iterator implemented using a class:

1
2
3
4
5
6
7
8
9
10
11
12
13
14

class Count:

    """Iterator that counts upward forever."""

    def __init__(self, start=0):
        self.num = start

    def __iter__(self):
        return self

    def __next__(self):
        num = self.num
        self.num += 1
        return num

This class has an initializer that initializes our current number to 0 (or whatever is passed in as the start). The things that make this class usable as an iterator are the __iter__ and __next__ methods.

When an object is passed to the str built-in function, its __str__ method is called. When an object is passed to the len built-in function, its __len__ method is called.

1
2
3
4
5

>>> numbers = [1, 2, 3]
>>> str(numbers), numbers.__str__()
('[1, 2, 3]', '[1, 2, 3]')
>>> len(numbers), numbers.__len__()
(3, 3)

Calling the built-in iter function on an object will attempt to call its __iter__ method. Calling the built-in next function on an object will attempt to call its __next__ method.

The iter function is supposed to return an iterator. So our __iter__ function must return an iterator. But our object is an iterator, so should return ourself. Therefore our Count object returns self from its __iter__ method because it is its own iterator.

The next function is supposed to return the next item in our iterator or raise a StopIteration exception when there are no more items. We're returning the current number and incrementing the number so it'll be larger during the next __next__ call.

We can manually loop over our Count iterator class like this:

1
2
3
4
5

>>> c = Count()
>>> next(c)
0
>>> next(c)
1

We could also loop over our Count object like using a for loop, as with any other iterable:

1
2
3
4
5
6
7

>>> for n in Count():
...     print(n)
...
0
1
2
(this goes on forever)

This object-oriented approach to making an iterator is cool, but it's not the usual way that Python programmers make iterators. Usually when we want an iterator, we make a generator.

Generators: the easy way to make an iterator

The easiest ways to make our own iterators in Python is to create a generator.

There are two ways to make generators in Python.

Given this list of numbers:

1

>>> favorite_numbers = [6, 57, 4, 7, 68, 95]

We can make a generator that will lazily provide us with all the squares of these numbers like this:

1
2
3
4
5

>>> def square_all(numbers):
...     for n in numbers:
...         yield n**2
...
>>> squares = square_all(favorite_numbers)

Or we can make the same generator like this:

1

>>> squares = (n**2 for n in favorite_numbers)

The first one is called a generator function and the second one is called a generator expression.

Both of these generator objects work the same way. They both have a type of generator and they're both iterators that provide squares of the numbers in our numbers list.

1
2
3
4
5
6

>>> type(squares)
<class 'generator'>
>>> next(squares)
36
>>> next(squares)
3249

We're going to talk about both of these approaches to making a generator, but first let's talk about terminology.

The word "generator" is used in quite a few ways in Python:

• A generator, also called a generator object, is an iterator whose type is generator
• A generator function is a special syntax that allows us to make a function which returns a generator object when we call it
• A generator expression is a comprehension-like syntax that allows you to create a generator object inline

With that terminology out of the way, let's take a look at each one of these things individually. We'll look at generator functions first.

Generator functions

Generator functions are distinguished from plain old functions by the fact that they have one or more yield statements.

Normally when you call a function, its code is executed:

1
2
3
4
5
6
7
8

>>> def gimme4_please():
...     print("Let me go get that number for you.")
...     return 4
...
>>> num = gimme4_please()
Let me go get that number for you.
>>> num
4

But if the function has a yield statement in it, it isn't a typical function anymore. It's now a generator function, meaning it will return a generator object when called. That generator object can be looped over to execute it until a yield statement is hit:

1
2
3
4
5
6
7
8
9
10
11

>>> def gimme4_later_please():
...     print("Let me go get that number for you.")
...     yield 4
...
>>> get4 = gimme4_later_please()
>>> get4
<generator object gimme4_later_please at 0x7f78b2e7e2b0>
>>> num = next(get4)
Let me go get that number for you.
>>> num
4

The mere presence of a yield statement turns a function into a generator function. If you see a function and there's a yield, you're working with a different animal. It's a bit odd, but that's the way generator functions work.

Okay let's look at a real example of a generator function. We'll make a generator function that does the same thing as our Count iterator class we made earlier.

1
2
3
4
5

def count(start=0):
    num = start
    while True:
        yield num
        num += 1

Just like our Counter iterator class, we can manually loop over the generator we get back from calling count:

1
2
3
4
5

>>> c = count()
>>> next(c)
0
>>> next(c)
1

And we can loop over this generator object using a for loop, just like before:

1
2
3
4
5
6
7

>>> for n in count():
...     print(n)
...
0
1
2
(this goes on forever)

But this function is considerably shorter than our Count class we created before.

Generator expressions

Generator expressions are a list comprehension-like syntax that allow us to make a generator object.

Let's say we have a list comprehension that filters empty lines from a file and strips newlines from the end:

1
2
3
4
5

lines = [
    line.rstrip('\n')
    for line in poem_file
    if line != '\n'
]

We could create a generator instead of a list, by turning the square brackets of that comprehension into parenthesis:

1
2
3
4
5

lines = (
    line.rstrip('\n')
    for line in poem_file
    if line != '\n'
)

Just as our list comprehension gave us a list back, our generator expression gives us a generator object back:

1
2
3
4
5
6

>>> type(lines)
<class 'generator'>
>>> next(lines)
' This little bag I hope will prove'
>>> next(lines)
'To be not vainly made--'

Generator expressions use a shorter inline syntax compared to generator functions. They're not as powerful though.

If you can write your generator function in this form:

1
2
3
4

def get_a_generator(some_iterable):
    for item in some_iterable:
        if some_condition(item):
            yield item

Then you can replace it with a generator expression:

1
2
3
4
5
6

def get_a_generator(some_iterable):
    return (
        item
        for item in some_iterable
        if some_condition(item)
    )

If you can't write your generator function in that form, then you can't create a generator expression to replace it.

Generator expressions vs generator functions

You can think of generator expressions as the list comprehensions of the generator world.

If you're not familiar with list comprehensions, I recommend reading my article on list comprehensions in Python. I note in that article that you can copy-paste your way from a for loop to a list comprehension.

You can also copy-paste your way from a generator function to a function that returns a generator expression:

Generator expressions are to generator functions as list comprehensions are to a simple for loop with an append and a condition.

Generator expressions are so similar to comprehensions, that you might even be tempted to say generator comprehension instead of generator expression. That's not technically the correct name, but if you say it everyone will know what you're talking about. Ned Batchelder actually proposed that we should all start calling generator expressions generator comprehensions and I tend to agree that this would be a clearer name.

So what's the best way to make an iterator?

To make an iterator you could create an iterator class, a generator function, or a generator expression. Which way is the best way though?

Generator expressions are very succinct, but they're not nearly as flexible as generator functions. Generator functions are flexible, but if you need to attach extra methods or attributes to your iterator object, you'll probably need to switch to using an iterator class.

I'd recommend reaching for generator expressions the same way you reach for list comprehensions. If you're doing a simple mapping or filtering operation, a generator expression is a great solution. If you're doing something a bit more sophisticated, you'll likely need a generator function.

I'd recommend using generator functions the same way you'd use for loops that append to a list. Everywhere you'd see an append method, you'd often see a yield statement instead.

And I'd say that you should almost never create an iterator class. If you find you need an iterator class, try to write a generator function that does what you need and see how it compares to your iterator class.

Generators can help when making iterables too

You'll see iterator classes in the wild, but there's rarely a good opportunity to write your own.

While it's rare to create your own iterator class, it's not unusual to make your own iterable class. And iterable classes require a __iter__ method which returns an iterator. Since generators are the easy way to make an iterator, we can use a generator function or a generator expression to create our __iter__ methods.

For example here's an iterable that provides x-y coordinates:

1
2
3
4
5
6

class Point:
    def __init__(self, x, y):
        self.x, self.y = x, y
    def __iter__(self):
        yield self.x
        yield self.y

Note that our Point class here creates an iterable when called (not an iterator). That means our __iter__ method must return an iterator. The easiest way to create an iterator is by making a generator function, so that's just what we did.

We stuck yield in our __iter__ to make it into a generator function and now our Point class con be looped over, just like any other iterable.

1
2
3
4
5
6

>>> p = Point(1, 2)
>>> x, y = p
>>> print(x, y)
1 2
>>> list(p)
[1, 2]

Generator functions are a natural fit for creating __iter__ methods on your iterable classes.

Generators are the way to make iterators

Dictionaries are the typical way to make a mapping in Python. Functions are the typical way to make a callable object in Python. Likewise, generators are the typical way to make an iterator in Python.

So when you're thinking "it sure would be nice to implement an iterable that lazily computes things as it's looped over," think of iterators.

And when you're considering how to create your own iterator, think of generator functions and generator expressions.

21 Jun 2018 11:00pm GMT

I wrote an article sometime ago on the iterator protocol that powers Python's for loops. One thing I left out of that article was how to make your own iterators.

In this article I'm going to discuss why you'd want to make your own iterators and then show you how to do so.

What is an iterator?

First let's quickly address what an iterator is. For a much more detailed explanation, consider watching my Loop Better talk or reading the article based on the talk.

An iterable is anything you're able to loop over.

An iterator is the object that does the actual iterating.

You can get an iterator from any iterable by calling the built-in iter function on the iterable.

1
2
3

>>> favorite_numbers = [6, 57, 4, 7, 68, 95]
>>> iter(favorite_numbers)
<list_iterator object at 0x7fe8e5623160>

You can use the built-in next function on an iterator to get the next item from it (you'll get a StopIteration exception if there are no more items).

1
2
3
4
5
6

>>> favorite_numbers = [6, 57, 4, 7, 68, 95]
>>> my_iterator = iter(favorite_numbers)
>>> next(my_iterator)
6
>>> next(my_iterator)
57

There's one more rule about iterators that makes everything interesting: iterators are also iterables and their iterator is themselves. I explain the consequences of that more fully in that Loop Better talk I mentioned above.

Why make an iterator?

Iterators allow you to make an iterable that computes its items as it goes. Which means that you can make iterables that are lazy, in that they don't determine what their next item is until you ask them for it.

Using an iterator instead of a list, set, or another iterable data structure can sometimes allow us to save memory. For example, we can use itertools.repeat to create an iterable that provides 100 million 4's to us:

1
2

>>> from itertools import repeat
>>> lots_of_fours = repeat(4, times=100_000_000)

This iterator takes up 56 bytes of memory on my machine:

1
2
3

>>> import sys
>>> sys.getsizeof(lots_of_fours)
56

An equivalent list of 100 million 4's takes up many megabytes of memory:

1
2
3
4

>>> lots_of_fours = [4] * 100_000_000
>>> import sys
>>> sys.getsizeof(lots_of_fours)
800000064

While iterators can save memory, they can also save time. For example if you wanted to print out just the first line of a 10 gigabyte log file, you could do this:

1
2

>>> print(next(open('giant_log_file.txt')))
This is the first line in a giant file

File objects in Python are implemented iterators. As you loop over a file, data is read into memory one line at a time. If we instead used the readlines method to store all lines in memory, we might run out of system memory.

So iterators can save us memory, but iterators can sometimes save us time also.

Additionally, iterators have abilities that other iterables don't. For example, the laziness of iterables can be used to make iterables that have an unknown length. In fact, you can even make infinitely long iterators.

For example, the itertools.count utility will give us an iterator that will provide every number from 0 upward as we loop over it:

1
2
3
4
5
6
7
8

>>> from itertools import count
>>> for n in count():
...     print(n)
...
0
1
2
(this goes on forever)

That itertools.count object is essentially an infinitely long iterable. And it's implemented as an iterator.

Making an iterator: the object-oriented way

So we've seen that iterators can save us memory, save us CPU time, and unlock new abilities to us.

Let's make our own iterators. We'll start be re-inventing the itertools.count iterator object.

Here's an iterator implemented using a class:

1
2
3
4
5
6
7
8
9
10
11
12
13
14

class Count:

    """Iterator that counts upward forever."""

    def __init__(self, start=0):
        self.num = start

    def __iter__(self):
        return self

    def __next__(self):
        num = self.num
        self.num += 1
        return num

This class has an initializer that initializes our current number to 0 (or whatever is passed in as the start). The things that make this class usable as an iterator are the __iter__ and __next__ methods.

When an object is passed to the str built-in function, its __str__ method is called. When an object is passed to the len built-in function, its __len__ method is called.

1
2
3
4
5

>>> numbers = [1, 2, 3]
>>> str(numbers), numbers.__str__()
('[1, 2, 3]', '[1, 2, 3]')
>>> len(numbers), numbers.__len__()
(3, 3)

Calling the built-in iter function on an object will attempt to call its __iter__ method. Calling the built-in next function on an object will attempt to call its __next__ method.

The iter function is supposed to return an iterator. So our __iter__ function must return an iterator. But our object is an iterator, so should return ourself. Therefore our Count object returns self from its __iter__ method because it is its own iterator.

The next function is supposed to return the next item in our iterator or raise a StopIteration exception when there are no more items. We're returning the current number and incrementing the number so it'll be larger during the next __next__ call.

We can manually loop over our Count iterator class like this:

1
2
3
4
5

>>> c = Count()
>>> next(c)
0
>>> next(c)
1

We could also loop over our Count object like using a for loop, as with any other iterable:

1
2
3
4
5
6
7

>>> for n in Count():
...     print(n)
...
0
1
2
(this goes on forever)

This object-oriented approach to making an iterator is cool, but it's not the usual way that Python programmers make iterators. Usually when we want an iterator, we make a generator.

Generators: the easy way to make an iterator

The easiest ways to make our own iterators in Python is to create a generator.

There are two ways to make generators in Python.

Given this list of numbers:

1

>>> favorite_numbers = [6, 57, 4, 7, 68, 95]

We can make a generator that will lazily provide us with all the squares of these numbers like this:

1
2
3
4
5

>>> def square_all(numbers):
...     for n in numbers:
...         yield n**2
...
>>> squares = square_all(favorite_numbers)

Or we can make the same generator like this:

1

>>> squares = (n**2 for n in favorite_numbers)

The first one is called a generator function and the second one is called a generator expression.

Both of these generator objects work the same way. They both have a type of generator and they're both iterators that provide squares of the numbers in our numbers list.

1
2
3
4
5
6

>>> type(squares)
<class 'generator'>
>>> next(squares)
36
>>> next(squares)
3249

We're going to talk about both of these approaches to making a generator, but first let's talk about terminology.

The word "generator" is used in quite a few ways in Python:

• A generator, also called a generator object, is an iterator whose type is generator
• A generator function is a special syntax that allows us to make a function which returns a generator object when we call it
• A generator expression is a comprehension-like syntax that allows you to create a generator object inline

With that terminology out of the way, let's take a look at each one of these things individually. We'll look at generator functions first.

Generator functions

Generator functions are distinguished from plain old functions by the fact that they have one or more yield statements.

Normally when you call a function, its code is executed:

1
2
3
4
5
6
7
8

>>> def gimme4_please():
...     print("Let me go get that number for you.")
...     return 4
...
>>> num = gimme4_please()
Let me go get that number for you.
>>> num
4

But if the function has a yield statement in it, it isn't a typical function anymore. It's now a generator function, meaning it will return a generator object when called. That generator object can be looped over to execute it until a yield statement is hit:

1
2
3
4
5
6
7
8
9
10
11

>>> def gimme4_later_please():
...     print("Let me go get that number for you.")
...     yield 4
...
>>> get4 = gimme4_later_please()
>>> get4
<generator object gimme4_later_please at 0x7f78b2e7e2b0>
>>> num = next(get4)
Let me go get that number for you.
>>> num
4

The mere presence of a yield statement turns a function into a generator function. If you see a function and there's a yield, you're working with a different animal. It's a bit odd, but that's the way generator functions work.

Okay let's look at a real example of a generator function. We'll make a generator function that does the same thing as our Count iterator class we made earlier.

1
2
3
4
5

def count(start=0):
    num = start
    while True:
        yield num
        num += 1

Just like our Counter iterator class, we can manually loop over the generator we get back from calling count:

1
2
3
4
5

>>> c = count()
>>> next(c)
0
>>> next(c)
1

And we can loop over this generator object using a for loop, just like before:

1
2
3
4
5
6
7

>>> for n in count():
...     print(n)
...
0
1
2
(this goes on forever)

But this function is considerably shorter than our Count class we created before.

Generator expressions

Generator expressions are a list comprehension-like syntax that allow us to make a generator object.

Let's say we have a list comprehension that filters empty lines from a file and strips newlines from the end:

1
2
3
4
5

lines = [
    line.rstrip('\n')
    for line in poem_file
    if line != '\n'
]

We could create a generator instead of a list, by turning the square brackets of that comprehension into parenthesis:

1
2
3
4
5

lines = (
    line.rstrip('\n')
    for line in poem_file
    if line != '\n'
)

Just as our list comprehension gave us a list back, our generator expression gives us a generator object back:

1
2
3
4
5
6

>>> type(lines)
<class 'generator'>
>>> next(lines)
' This little bag I hope will prove'
>>> next(lines)
'To be not vainly made--'

Generator expressions use a shorter inline syntax compared to generator functions. They're not as powerful though.

If you can write your generator function in this form:

1
2
3
4

def get_a_generator(some_iterable):
    for item in some_iterable:
        if some_condition(item):
            yield item

Then you can replace it with a generator expression:

1
2
3
4
5
6

def get_a_generator(some_iterable):
    return (
        item
        for item in some_iterable
        if some_condition(item)
    )

If you can't write your generator function in that form, then you can't create a generator expression to replace it.

Generator expressions vs generator functions

You can think of generator expressions as the list comprehensions of the generator world.

If you're not familiar with list comprehensions, I recommend reading my article on list comprehensions in Python. I note in that article that you can copy-paste your way from a for loop to a list comprehension.

You can also copy-paste your way from a generator function to a function that returns a generator expression:

Generator expressions are to generator functions as list comprehensions are to a simple for loop with an append and a condition.

Generator expressions are so similar to comprehensions, that you might even be tempted to say generator comprehension instead of generator expression. That's not technically the correct name, but if you say it everyone will know what you're talking about. Ned Batchelder actually proposed that we should all start calling generator expressions generator comprehensions and I tend to agree that this would be a clearer name.

So what's the best way to make an iterator?

To make an iterator you could create an iterator class, a generator function, or a generator expression. Which way is the best way though?

Generator expressions are very succinct, but they're not nearly as flexible as generator functions. Generator functions are flexible, but if you need to attach extra methods or attributes to your iterator object, you'll probably need to switch to using an iterator class.

I'd recommend reaching for generator expressions the same way you reach for list comprehensions. If you're doing a simple mapping or filtering operation, a generator expression is a great solution. If you're doing something a bit more sophisticated, you'll likely need a generator function.

I'd recommend using generator functions the same way you'd use for loops that append to a list. Everywhere you'd see an append method, you'd often see a yield statement instead.

And I'd say that you should almost never create an iterator class. If you find you need an iterator class, try to write a generator function that does what you need and see how it compares to your iterator class.

Generators can help when making iterables too

You'll see iterator classes in the wild, but there's rarely a good opportunity to write your own.

While it's rare to create your own iterator class, it's not unusual to make your own iterable class. And iterable classes require a __iter__ method which returns an iterator. Since generators are the easy way to make an iterator, we can use a generator function or a generator expression to create our __iter__ methods.

For example here's an iterable that provides x-y coordinates:

1
2
3
4
5
6

class Point:
    def __init__(self, x, y):
        self.x, self.y = x, y
    def __iter__(self):
        yield self.x
        yield self.y

Note that our Point class here creates an iterable when called (not an iterator). That means our __iter__ method must return an iterator. The easiest way to create an iterator is by making a generator function, so that's just what we did.

We stuck yield in our __iter__ to make it into a generator function and now our Point class con be looped over, just like any other iterable.

1
2
3
4
5
6

>>> p = Point(1, 2)
>>> x, y = p
>>> print(x, y)
1 2
>>> list(p)
[1, 2]

Generator functions are a natural fit for creating __iter__ methods on your iterable classes.

Generators are the way to make iterators

Dictionaries are the typical way to make a mapping in Python. Functions are the typical way to make a callable object in Python. Likewise, generators are the typical way to make an iterator in Python.

So when you're thinking "it sure would be nice to implement an iterable that lazily computes things as it's looped over," think of iterators.

And when you're considering how to create your own iterator, think of generator functions and generator expressions.

21 Jun 2018 11:00pm GMT

Hi there! Welcome to the second and final installment of my trending twitter hashtags example series. In part 1, we covered the basic dataflow and logic of the application. In part 2, we are going to take a look at how windowing for the "trending" aspect of our application is implemented. When implementing any sort of "trending" application, what we are really doing is implementing some kind of windowing. That is, for some duration of time, we want to know what was popular, what was "trending" during that period of time.

21 Jun 2018 4:49pm GMT

Hi there! Welcome to the second and final installment of my trending twitter hashtags example series. In part 1, we covered the basic dataflow and logic of the application. In part 2, we are going to take a look at how windowing for the "trending" aspect of our application is implemented. When implementing any sort of "trending" application, what we are really doing is implementing some kind of windowing. That is, for some duration of time, we want to know what was popular, what was "trending" during that period of time.

21 Jun 2018 4:49pm GMT

I am just a hobbyist, Python enthusiast who has been, over the course of many years, writing what is now a relatively big Javascript program (close to 20,000 lines of code so far). If you like Python and the Pythonic way of programming, and find yourself writing more JavaScript code than you'd like for a fun side-project meant as a hobby, you may find some merit in the approach I use. I wish I could have read something like this blog post when I started my project or even just a few years ago, when I did a major rewrite, and started using some of the tools described in this post.

If you are a professional programmer, you can just stop reading as you know much more than I do, and you surely have a better, more efficient and cutting edge way of doing things right now - and you will likely use yet a different way next year, if not next month. So, you would likely find my advice to look the same as my site: dated and not using the latest and coolest techniques - in short, for you, not worth looking at. ;-)

Summary:

• If you must, choose a well-supported Javascript library and stick with it.
• Use npm for installing Javascript tools
• • Avoid depending on third-party packages whenever possible
• Use npm to manage your workflow
• • Supplement with your own Python or shell/batch script when needed
• Use browserify to concatenate all your Javascript files
• Use tape for unit testing
• • Use faucet for formatting of unit test results
• Use QUnit for integration testing
• Optional: use Madge for identifying circular dependencies
• Optional: use Dependo to identify any overlooked module
• Optional: use JSDoc for creating an API
• Use jshint instead of jslint

Warning

The context

21 Jun 2018 1:19pm GMT

I am just a hobbyist, Python enthusiast who has been, over the course of many years, writing what is now a relatively big Javascript program (close to 20,000 lines of code so far). If you like Python and the Pythonic way of programming, and find yourself writing more JavaScript code than you'd like for a fun side-project meant as a hobby, you may find some merit in the approach I use. I wish I could have read something like this blog post when I started my project or even just a few years ago, when I did a major rewrite, and started using some of the tools described in this post.

If you are a professional programmer, you can just stop reading as you know much more than I do, and you surely have a better, more efficient and cutting edge way of doing things right now - and you will likely use yet a different way next year, if not next month. So, you would likely find my advice to look the same as my site: dated and not using the latest and coolest techniques - in short, for you, not worth looking at. ;-)

Summary:

• If you must, choose a well-supported Javascript library and stick with it.
• Use npm for installing Javascript tools
• • Avoid depending on third-party packages whenever possible
• Use npm to manage your workflow
• • Supplement with your own Python or shell/batch script when needed
• Use browserify to concatenate all your Javascript files
• Use tape for unit testing
• • Use faucet for formatting of unit test results
• Use QUnit for integration testing
• Optional: use Madge for identifying circular dependencies
• Optional: use Dependo to identify any overlooked module
• Optional: use JSDoc for creating an API
• Use jshint instead of jslint

Warning

The context

21 Jun 2018 1:19pm GMT

homesteading.com

Long ago, when I worked at MongoDB I created Flask-PyMongo to make it easy for programmers using Flask to use the database. Fast forward almost 8 years, during which time I wasn't a consistent user of either Flask or MongoDB, and Flask-PyMongo has fallen into disrepair.

MongoDB, PyMongo, and Flask have moved on, and Flask-PyMongo hasn't been kept up to date. There are more than twice as many forks as pull requests, a GitHub ratio I'm not proud of. Fotunately, the future for Flask-PyMongo is bright.

False Starts and New Beginnings

At PyCon US 2017, I first had the idea to restore Flask-PyMongo. PyCon always has this effect on me, but sadly the effect is often short lived. In 2017, I got as far as mentioning plans for a 2.0 release, but did not go into any detail, nor begin to make any progress toward that goal.

This year at PyCon 2018, I once again had the urge to work on Flask-PyMongo. I had the same conversation with Jesse, decided, again, to jump from 0.x to 2.0, and even came up with the same technical plan as the previous year (about which more below). All without realizing that I had been down this road once before.

I can now confidently say that Flask-PyMongo 2.0 is (soon to be) a real thing, and it will set the stage for easier maintainability into the future and a better experience for users and contributors. Flask-PyMongo 2.0 will be released in early July, and pre-release versions are available today.

What's Changing

Flask-PyMongo 2.0 is not backwards compatible!

A lot of the historical problems with Flask-PyMongo have cenetered on the confusing and difficult configuration system. Originally, I envisioned that users would want configuration abstracted from PyMongo itself, and created a system where you could set Flask configurations for MONGO_HOST, MONGO_PORT, and MONGO_DBNAME, and be off to the races. For a while this worked, and many users seemed to like it. Unfortunately, there are quite a lot of configuration options for PyMongo, so the list of configurations grew. Worse, PyMongo and MongoDB are under active development, and grow and lose features over time. Attempts to make Flask-PyMongo version-agnostic added tremendous complexity to the configuration system, and evidently frustrated many users over the years.

In any event, it turns out that there's a better way to configure PyMongo -- with MongoDB URIs. Most hosted PyMongo services already provide configuration information in exactly this format. Going forward in 2.0, MongoDB URIs are the preferred configuration method for Flask-PyMongo. Flask-PyMongo will only look for or respect a single Flask configuration variable, MONGO_URI.

If you prefer, you may also pass positional and keyword arguments directly to Flask-PyMongo, which will be passed through to the underlying PyMongo MongoClient object.

Flask-PyMongo no longer supports configurating multiple instances via Flask configuration. If you wish to use multiple Flask-PyMongo instances, you must configure at least some of them using a URI or direct argument passing.

Flask-PyMongo 2.0 also clarifies the support policy for versions of Flask, PyMongo, MongoDB, and Python that are supported. For Flask and PyMongo, it supports "recent" versions -- those versions with releases in the preceding 3 years (give or take). For MongoDB, we follow the MongoDB server support policy, and support versions that are not end-of-lifed. For Python, we support 2.7 for as long as it is supported by the CPython core maintainers; and the most recent 3 versions of the 3.x series. For an exact list of supported versions and combinations, see the build matrix.

What You Should Do

If you are a Flask-PyMongo user and you are using the 0.x series, you should immediately pin a particular version. Flask-PyMongo 2.0 is not backwards compatible, so you should take steps to ensure that you don't accidentally break your application.

If you are already using a URI for Flask-PyMongo configuration, or if that is an easy change for you, I would appreciate if you could upgrade, test compatibility, and report any issues on GitHub. You can install Flask-PyMongo 2.0 pre-releases with pip install --pre flask-pymongo. You may also want to follow the general discussion and release notices in issue #110.

I also hope for Flask-PyMongo to be a place that supports Flask and MongoDB with more than just connection assistance. Please suggest ideas and propose contributions!

21 Jun 2018 1:00pm GMT

homesteading.com

Long ago, when I worked at MongoDB I created Flask-PyMongo to make it easy for programmers using Flask to use the database. Fast forward almost 8 years, during which time I wasn't a consistent user of either Flask or MongoDB, and Flask-PyMongo has fallen into disrepair.

MongoDB, PyMongo, and Flask have moved on, and Flask-PyMongo hasn't been kept up to date. There are more than twice as many forks as pull requests, a GitHub ratio I'm not proud of. Fotunately, the future for Flask-PyMongo is bright.

False Starts and New Beginnings

At PyCon US 2017, I first had the idea to restore Flask-PyMongo. PyCon always has this effect on me, but sadly the effect is often short lived. In 2017, I got as far as mentioning plans for a 2.0 release, but did not go into any detail, nor begin to make any progress toward that goal.

This year at PyCon 2018, I once again had the urge to work on Flask-PyMongo. I had the same conversation with Jesse, decided, again, to jump from 0.x to 2.0, and even came up with the same technical plan as the previous year (about which more below). All without realizing that I had been down this road once before.

I can now confidently say that Flask-PyMongo 2.0 is (soon to be) a real thing, and it will set the stage for easier maintainability into the future and a better experience for users and contributors. Flask-PyMongo 2.0 will be released in early July, and pre-release versions are available today.

What's Changing

Flask-PyMongo 2.0 is not backwards compatible!

A lot of the historical problems with Flask-PyMongo have cenetered on the confusing and difficult configuration system. Originally, I envisioned that users would want configuration abstracted from PyMongo itself, and created a system where you could set Flask configurations for MONGO_HOST, MONGO_PORT, and MONGO_DBNAME, and be off to the races. For a while this worked, and many users seemed to like it. Unfortunately, there are quite a lot of configuration options for PyMongo, so the list of configurations grew. Worse, PyMongo and MongoDB are under active development, and grow and lose features over time. Attempts to make Flask-PyMongo version-agnostic added tremendous complexity to the configuration system, and evidently frustrated many users over the years.

In any event, it turns out that there's a better way to configure PyMongo -- with MongoDB URIs. Most hosted PyMongo services already provide configuration information in exactly this format. Going forward in 2.0, MongoDB URIs are the preferred configuration method for Flask-PyMongo. Flask-PyMongo will only look for or respect a single Flask configuration variable, MONGO_URI.

If you prefer, you may also pass positional and keyword arguments directly to Flask-PyMongo, which will be passed through to the underlying PyMongo MongoClient object.

Flask-PyMongo no longer supports configurating multiple instances via Flask configuration. If you wish to use multiple Flask-PyMongo instances, you must configure at least some of them using a URI or direct argument passing.

Flask-PyMongo 2.0 also clarifies the support policy for versions of Flask, PyMongo, MongoDB, and Python that are supported. For Flask and PyMongo, it supports "recent" versions -- those versions with releases in the preceding 3 years (give or take). For MongoDB, we follow the MongoDB server support policy, and support versions that are not end-of-lifed. For Python, we support 2.7 for as long as it is supported by the CPython core maintainers; and the most recent 3 versions of the 3.x series. For an exact list of supported versions and combinations, see the build matrix.

What You Should Do

If you are a Flask-PyMongo user and you are using the 0.x series, you should immediately pin a particular version. Flask-PyMongo 2.0 is not backwards compatible, so you should take steps to ensure that you don't accidentally break your application.

If you are already using a URI for Flask-PyMongo configuration, or if that is an easy change for you, I would appreciate if you could upgrade, test compatibility, and report any issues on GitHub. You can install Flask-PyMongo 2.0 pre-releases with pip install --pre flask-pymongo. You may also want to follow the general discussion and release notices in issue #110.

I also hope for Flask-PyMongo to be a place that supports Flask and MongoDB with more than just connection assistance. Please suggest ideas and propose contributions!

21 Jun 2018 1:00pm GMT

Well-structured code with the thought out architecture is your goal, but the ordinary books and articles seem too confusing? Then this article is for you! We've used the simplest analogies to describe two classic design patterns: Abstract Factory and Strategy. The article also includes Python code examples of patterns implementation and links to the coding challenges, so you can practice right away and understand how the patterns work once and for all.

21 Jun 2018 9:43am GMT

Well-structured code with the thought out architecture is your goal, but the ordinary books and articles seem too confusing? Then this article is for you! We've used the simplest analogies to describe two classic design patterns: Abstract Factory and Strategy. The article also includes Python code examples of patterns implementation and links to the coding challenges, so you can practice right away and understand how the patterns work once and for all.

21 Jun 2018 9:43am GMT

The post Ethical Algorithms - Notes from the DISC Unconference appeared first on NumFOCUS.

20 Jun 2018 8:02pm GMT

The post Ethical Algorithms - Notes from the DISC Unconference appeared first on NumFOCUS.

20 Jun 2018 8:02pm GMT

In this course I cover statistics and machine learning topics. The course assumes little knowledge about what statistics or machine learning involves. I touch lightly on the theory of statistics and machine learning to motivate the tasks performed in the videos.

20 Jun 2018 5:10pm GMT

In this course I cover statistics and machine learning topics. The course assumes little knowledge about what statistics or machine learning involves. I touch lightly on the theory of statistics and machine learning to motivate the tasks performed in the videos.

20 Jun 2018 5:10pm GMT

We're now in our fourth installment of a pretty big 2018.2 Early Access Program cycle. Lots to take a look at by downloading EAP 4 from our website.

New in PyCharm 2018.2 EAP 4

Pipenv support

We know many of you have been waiting for this for a long time, so here you go: Pipenv is supported in PyCharm 2018.2. There is still a lot of work before we finally release stable PyCharm 2018.2 so your input with bug reports or suggestions is very welcome in our issue tracker.

Currently supported Pipenv-related features in PyCharm:

• The Pipenv environment type in the Python interpreter dialog (for new or existing projects)
• A quick fix for setting up the Pipenv for a project with Pipfile
• Automatically create/set up Pipenv for a directory with Pipfile opened for the first time
• Auto-suggestion to download the Toml plugin for Pipfile
• Selection of the base Python executable while configuring a new Pipenv
• Use Pipenv for managing packages and updating Pipfile
• Display the list of installed packages via Pipenv
• Use the indices defined in Pipfile.lock to browse available packages
• A configurable location for Pipenv via Python Integrated Tools
• Using Pipenv in the quick fix to install an unresolved import
• Checking missing package requirements based on Pipfile.lock
• Quick-fix to install missing packages via Pipenv
• Notification about changed and not locked pipfile and a link to lock it

pytest-bdd Support

In this EAP we introduce an initial support for pytest-bdd. To enable the pytest-bdd support open the BDD settings dialog (File | Settings/Preferences | Languages & Frameworks | BDD ) and from the Preferred BDD framework list select pytest-bdd. We're continuing to work on py-bdd support, so your input is much appreciated.

More details on pytest-bdd support in PyCharm

Type hints validation

Any time you're applying type hints, PyCharm checks if the type is used correctly. If there is a usage error, the corresponding warning is shown and the recommended action is suggested.

Learn more about type hints validation in PyCharm

New Front-End Development Functionality

As you might already know, PyCharm bundles all features available in WebStorm, a front-end development IDE by JetBrains. PyCharm EAP 4 adds several WebStorm EAP features:

• Code coverage tool with visual indicators on folders, files, and code about the percentage of coverage
• Test files are automatically added to the Tests scope to help adjust IDE settings such as inspection severity
• New UI for inspection tooltips with key binding to apply the first fix
• Support for Node.js on Windows Subsystem for Linux (WSL)
• Remote mappings in Attach to Node.js configuration
• New intentions in JavaScript, TypeScript, and JSON: implement an interface, create derived class, generate cases for the switch, add properties missing from a JSON schema, and more
• That's right: an earlier EAP had improved validation from a JSON Schema

PyCharm 2018.2 EAP 4 Release Notes

Interested?

Download this EAP from our website. Alternatively, you can use the JetBrains Toolbox App to stay up to date throughout the entire EAP.

If you're on Ubuntu 16.04 or later, you can use snap to get PyCharm EAP, and stay up to date. You can find the installation instructions on our website.

PyCharm 2018.2 is in development during the EAP phase, therefore not all new features are already available. More features will be added in the coming weeks. As PyCharm 2018.2 is pre-release software, it is not as stable as the release versions. Furthermore, we may decide to change and/or drop certain features as the EAP progresses.

All EAP versions will ship with a built-in EAP license, which means that these versions are free to use for 30 days after the day that they are built. As EAPs are released weekly, you'll be able to use PyCharm Professional Edition EAP for free for the duration of the EAP program, as long as you upgrade at least once every 30 days.

20 Jun 2018 4:12pm GMT

We're now in our fourth installment of a pretty big 2018.2 Early Access Program cycle. Lots to take a look at by downloading EAP 4 from our website.

New in PyCharm 2018.2 EAP 4

Pipenv support

We know many of you have been waiting for this for a long time, so here you go: Pipenv is supported in PyCharm 2018.2. There is still a lot of work before we finally release stable PyCharm 2018.2 so your input with bug reports or suggestions is very welcome in our issue tracker.

Currently supported Pipenv-related features in PyCharm:

• The Pipenv environment type in the Python interpreter dialog (for new or existing projects)
• A quick fix for setting up the Pipenv for a project with Pipfile
• Automatically create/set up Pipenv for a directory with Pipfile opened for the first time
• Auto-suggestion to download the Toml plugin for Pipfile
• Selection of the base Python executable while configuring a new Pipenv
• Use Pipenv for managing packages and updating Pipfile
• Display the list of installed packages via Pipenv
• Use the indices defined in Pipfile.lock to browse available packages
• A configurable location for Pipenv via Python Integrated Tools
• Using Pipenv in the quick fix to install an unresolved import
• Checking missing package requirements based on Pipfile.lock
• Quick-fix to install missing packages via Pipenv
• Notification about changed and not locked pipfile and a link to lock it

pytest-bdd Support

In this EAP we introduce an initial support for pytest-bdd. To enable the pytest-bdd support open the BDD settings dialog (File | Settings/Preferences | Languages & Frameworks | BDD ) and from the Preferred BDD framework list select pytest-bdd. We're continuing to work on py-bdd support, so your input is much appreciated.

More details on pytest-bdd support in PyCharm

Type hints validation

Any time you're applying type hints, PyCharm checks if the type is used correctly. If there is a usage error, the corresponding warning is shown and the recommended action is suggested.

Learn more about type hints validation in PyCharm

New Front-End Development Functionality

As you might already know, PyCharm bundles all features available in WebStorm, a front-end development IDE by JetBrains. PyCharm EAP 4 adds several WebStorm EAP features:

• Code coverage tool with visual indicators on folders, files, and code about the percentage of coverage
• Test files are automatically added to the Tests scope to help adjust IDE settings such as inspection severity
• New UI for inspection tooltips with key binding to apply the first fix
• Support for Node.js on Windows Subsystem for Linux (WSL)
• Remote mappings in Attach to Node.js configuration
• New intentions in JavaScript, TypeScript, and JSON: implement an interface, create derived class, generate cases for the switch, add properties missing from a JSON schema, and more
• That's right: an earlier EAP had improved validation from a JSON Schema

PyCharm 2018.2 EAP 4 Release Notes

Interested?

Download this EAP from our website. Alternatively, you can use the JetBrains Toolbox App to stay up to date throughout the entire EAP.

If you're on Ubuntu 16.04 or later, you can use snap to get PyCharm EAP, and stay up to date. You can find the installation instructions on our website.

PyCharm 2018.2 is in development during the EAP phase, therefore not all new features are already available. More features will be added in the coming weeks. As PyCharm 2018.2 is pre-release software, it is not as stable as the release versions. Furthermore, we may decide to change and/or drop certain features as the EAP progresses.

All EAP versions will ship with a built-in EAP license, which means that these versions are free to use for 30 days after the day that they are built. As EAPs are released weekly, you'll be able to use PyCharm Professional Edition EAP for free for the duration of the EAP program, as long as you upgrade at least once every 30 days.

20 Jun 2018 4:12pm GMT

After finishing our previous tutorial on Python variables in this series, you should now have a good grasp of creating and naming Python objects of different types. Let's do some work with them!

Here's what you'll learn in this tutorial: You'll see how calculations can be performed on objects in Python. By the end of this tutorial, you will be able to create complex expressions by combining objects and operators.

In Python, operators are special symbols that designate that some sort of computation should be performed. The values that an operator acts on are called operands.

Here is an example:

>>> a = 10
>>> b = 20
>>> a + b
30

In this case, the + operator adds the operands a and b together. An operand can be either a literal value or a variable that references an object:

>>> a = 10
>>> b = 20
>>> a + b - 5
25

A sequence of operands and operators, like a + b - 5, is called an expression. Python supports many operators for combining data objects into expressions. These are explored below.

Arithmetic Operators

The following table lists the arithmetic operators supported by Python:

Here are some examples of these operators in use:

>>> a = 4
>>> b = 3
>>> +a
4
>>> -b
-3
>>> a + b
7
>>> a - b
1
>>> a * b
12
>>> a / b
1.3333333333333333
>>> a % b
1
>>> a ** b
64

The result of standard division (/) is always a float, even if the dividend is evenly divisible by the divisor:

>>> 10 / 5
2.0
>>> type(10 / 5)
<class 'float'>

When the result of floor division (//) is positive, it is as though the fractional portion is truncated off, leaving only the integer portion. When the result is negative, the result is rounded down to the next smallest (greater negative) integer:

>>> 10 / 4
2.5
>>> 10 // 4
2
>>> 10 // -4
-3
>>> -10 // 4
-3
>>> -10 // -4
2

Note, by the way, that in a REPL session, you can display the value of an expression by just typing it in at the >>> prompt without print(), the same as you can with a literal value or variable:

>>> 25
25
>>> x = 4
>>> y = 6
>>> x
4
>>> y
6
>>> x * 25 + y
106

Comparison Operators

Here are examples of the comparison operators in use:

>>> a = 10
>>> b = 20
>>> a == b
False
>>> a != b
True
>>> a <= b
True
>>> a >= b
False

>>> a = 30
>>> b = 30
>>> a == b
True
>>> a <= b
True
>>> a >= b
True

Comparison operators are typically used in Boolean contexts like conditional and loop statements to direct program flow, as you will see later.

Equality Comparison on Floating-Point Values

Recall from the earlier discussion of floating-point numbers that the value stored internally for a float object may not be precisely what you'd think it would be. For that reason, it is poor practice to compare floating-point values for exact equality. Consider this example:

>>> x = 1.1 + 2.2
>>> x == 3.3
False

Yikes! The internal representations of the addition operands are not exactly equal to 1.1 and 2.2, so you cannot rely on x to compare exactly to 3.3.

The preferred way to determine whether two floating-point values are "equal" is to compute whether they are close to one another, given some tolerance. Take a look at this example:

>>> tolerance = 0.00001
>>> x = 1.1 + 2.2
>>> abs(x - 3.3) < tolerance
True

abs() returns absolute value. If the absolute value of the difference between the two numbers is less than the specified tolerance, they are close enough to one another to be considered equal.

Logical Operators

The logical operators not, or, and and modify and join together expressions evaluated in Boolean context to create more complex conditions.

Logical Expressions Involving Boolean Operands

As you have seen, some objects and expressions in Python actually are of Boolean type. That is, they are equal to one of the Python objects True or False. Consider these examples:

>>> x = 5
>>> x < 10
True
>>> type(x < 10)
<class 'bool'>

>>> t = x > 10
>>> t
False
>>> type(t)
<class 'bool'>

>>> callable(x)
False
>>> type(callable(x))
<class 'bool'>

>>> t = callable(len)
>>> t
True
>>> type(t)
<class 'bool'>

In the examples above, x < 10, callable(x), and t are all Boolean objects or expressions.

Interpretation of logical expressions involving not, or, and and is straightforward when the operands are Boolean:

Take a look at how they work in practice below.

"not" and Boolean Operands

x = 5
not x < 10
False
not callable(x)
True

"or" and Boolean Operands

x = 5
x < 10 or callable(x)
True
x < 0 or callable(x)
False

"and" and Boolean Operands

x = 5
x < 10 and callable(x)
False
x < 10 and callable(len)
True

Evaluation of Non-Boolean Values in Boolean Context

Many objects and expressions are not equal to True or False. Nonetheless, they may still be evaluated in Boolean context and determined to be "truthy" or "falsy."

So what is true and what isn't? As a philosophical question, that is outside the scope of this tutorial!

But in Python, it is well-defined. All the following are considered false when evaluated in Boolean context:

• The Boolean value False
• Any value that is numerically zero (0, 0.0, 0.0+0.0j)
• An empty string
• An object of a built-in composite data type which is empty (see below)
• The special value denoted by the Python keyword None

Virtually any other object built into Python is regarded as true.

You can determine the "truthiness" of an object or expression with the built-in bool() function. bool() returns True if its argument is truthy and False if it is falsy.

Numeric Value

A zero value is false.
A non-zero value is true.

>>> print(bool(0), bool(0.0), bool(0.0+0j))
False False False

>>> print(bool(-3), bool(3.14159), bool(1.0+1j))
True True True

String

An empty string is false.
A non-empty string is true.

>>> print(bool(''), bool(""), bool(""""""))
False False False

>>> print(bool('foo'), bool(" "), bool(''' '''))
True True True

Built-In Composite Data Object

Python provides built-in composite data types called list, tuple, dict, and set. These are "container" types that contain other objects. An object of one of these types is considered false if it is empty and true if it is non-empty.

The examples below demonstrate this for the list type. (Lists are defined in Python with square brackets.)

For more information on the list, tuple, dict, and set types, see the upcoming tutorials.

>>> type([])
<class 'list'>
>>> bool([])
False

>>> type([1, 2, 3])
<class 'list'>
>>> bool([1, 2, 3])
True

The "None" Keyword

None is always false:

>>> bool(None)
False

Logical Expressions Involving Non-Boolean Operands

Non-Boolean values can also be modified and joined by not, or and, and. The result depends on the "truthiness" of the operands.

"not" and Non-Boolean Operands

Here is what happens for a non-Boolean value x:

Here are some concrete examples:

>>> x = 3
>>> bool(x)
True
>>> not x
False

>>> x = 0.0
>>> bool(x)
False
>>> not x
True

"or" and Non-Boolean Operands

This is what happens for two non-Boolean values x and y:

Note that in this case, the expression x or y does not evaluate to either True or False, but instead to one of either x or y:

>>> x = 3
>>> y = 4
>>> x or y
3

>>> x = 0.0
>>> y = 4.4
>>> x or y
4.4

Even so, it is still the case that the expression x or y will be truthy if either x or y is truthy, and falsy if both x and y are falsy.

"and" and Non-Boolean Operands

Here's what you'll get for two non-Boolean values x and y:

>>> x = 3
>>> y = 4
>>> x and y
4

>>> x = 0.0
>>> y = 4.4
>>> x and y
0.0

As with or, the expression x and y does not evaluate to either True or False, but instead to one of either x or y. x and y will be truthy if both x and y are truthy, and falsy otherwise.

Compound Logical Expressions and Short-Circuit Evaluation

So far, you have seen expressions with only a single or or and operator and two operands:

x or y
x and y

Multiple logical operators and operands can be strung together to form compound logical expressions.

Compound "or" Expressions

Consider the following expression:

x₁ or x₂ or x₃ or … x_n

This expression is true if any of the x_i are true.

In an expression like this, Python uses a methodology called short-circuit evaluation, also called McCarthy evaluation in honor of computer scientist John McCarthy. The x_i operands are evaluated in order from left to right. As soon as one is found to be true, the entire expression is known to be true. At that point, Python stops and no more terms are evaluated. The value of the entire expression is that of the x_i that terminated evaluation.

To help demonstrate short-circuit evaluation, suppose that you have a simple "identity" function f() that behaves as follows:

• f() takes a single argument.
• It displays the argument to the console.
• It returns the argument passed to it as its return value.

(You will see how to define such a function in the upcoming tutorial on Functions.)

Several example calls to f() are shown below:

>>> f(0)
-> f(0) = 0
0

>>> f(False)
-> f(False) = False
False

>>> f(1.5)
-> f(1.5) = 1.5
1.5

Because f() simply returns the argument passed to it, we can make the expression f(arg) be truthy or falsy as needed by specifying a value for arg that is appropriately truthy or falsy. Additionally, f() displays its argument to the console, which visually confirms whether or not it was called.

Now, consider the following compound logical expression:

>>> f(0) or f(False) or f(1) or f(2) or f(3)
-> f(0) = 0
-> f(False) = False
-> f(1) = 1
1

The interpreter first evaluates f(0), which is 0. A numeric value of 0 is false. The expression is not true yet, so evaluation proceeds left to right. The next operand, f(False), returns False. That is also false, so evaluation continues.

Next up is f(1). That evaluates to 1, which is true. At that point, the interpreter stops because it now knows the entire expression to be true. 1 is returned as the value of the expression, and the remaining operands, f(2) and f(3), are never evaluated. You can see from the display that the f(2) and f(3) calls do not occur.

Compound "and" Expressions

A similar situation exists in an expression with multiple and operators:

x₁ and x₂ and x₃ and … x_n

This expression is true if all the x_i are true.

In this case, short-circuit evaluation dictates that the interpreter stop evaluating as soon as any operand is found to be false, because at that point the entire expression is known to be false. Once that is the case, no more operands are evaluated, and the falsy operand that terminated evaluation is returned as the value of the expression:

>>> f(1) and f(False) and f(2) and f(3)
-> f(1) = 1
-> f(False) = False
False

>>> f(1) and f(0.0) and f(2) and f(3)
-> f(1) = 1
-> f(0.0) = 0.0
0.0

In both examples above, evaluation stops at the first term that is false-f(False) in the first case, f(0.0) in the second case-and neither the f(2) nor f(3) call occurs. False and 0.0, respectively, are returned as the value of the expression.

If all the operands are truthy, they all get evaluated and the last (rightmost) one is returned as the value of the expression:

>>> f(1) and f(2.2) and f('bar')
-> f(1) = 1
-> f(2.2) = 2.2
-> f(bar) = bar
'bar'

Idioms That Exploit Short-Circuit Evaluation

There are some common idiomatic patterns that exploit short-circuit evaluation for conciseness of expression.

Avoiding an Exception

Suppose you have defined two variables a and b, and you want to know whether (b / a) > 0:

>>> a = 3
>>> b = 1
>>> (b / a) > 0
True

But you need to account for the possibility that a might be 0, in which case the interpreter will raise an exception:

>>> a = 0
>>> b = 1
>>> (b / a) > 0
Traceback (most recent call last):
  File "<pyshell#2>", line 1, in <module>
    (b / a) > 0
ZeroDivisionError: division by zero

You can avoid an error with an expression like this:

>>> a = 0
>>> b = 1
>>> a != 0 and (b / a) > 0
False

When a is 0, a != 0 is false. Short-circuit evaluation ensures that evaluation stops at that point. (b / a) is not evaluated, and no error is raised.

If fact, you can be even more concise than that. When a is 0, the expression a by itself is falsy. There is no need for the explicit comparison a != 0:

>>> a = 0
>>> b = 1
>>> a and (b / a) > 0
0

Selecting a Default Value

Another idiom involves selecting a default value when a specified value is zero or empty. For example, suppose you want to assign a variable s to the value contained in another variable called string. But if string is empty, you want to supply a default value.

Here is a concise way of expressing this using short-circuit evaluation:

s = string or '<default_value>'

If string is non-empty, it is truthy, and the expression string or '<default_value>' will be true at that point. Evaluation stops, and the value of string is returned and assigned to s:

>>> string = 'foo bar'
>>> s = string or '<default_value>'
>>> s
'foo bar'

On the other hand, if string is an empty string, it is falsy. Evaluation of string or '<default_value>' continues to the next operand, '<default_value>', which is returned and assigned to s:

>>> string = ''
>>> s = string or '<default_value>'
>>> s
'<default_value>'

Chained Comparisons

Comparison operators can be chained together to arbitrary length. For example, the following expressions are nearly equivalent:

x < y <= z
x < y and y <= z

They will both evaluate to the same Boolean value. The subtle difference between the two is that in the chained comparison x < y <= z, y is evaluated only once. The longer expression x < y and y <= z will cause y to be evaluated twice.

x < f() <= z
x < f() and f() <= z

More generally, if op₁, op₂, …, op_n are comparison operators, then the following have the same Boolean value:

x₁ op₁ x₂ op₂ x₃ … x_n-1 op_n x_n

x₁ op₁ x₂ and x₂ op₂ x₃ and … x_n-1 op_n x_n

In the former case, each x_i is only evaluated once. In the latter case, each will be evaluated twice except the first and last, unless short-circuit evaluation causes premature termination.

Bitwise Operators

Bitwise operators treat operands as sequences of binary digits and operate on them bit by bit. The following operators are supported:

Here are some examples:

>>> '0b{:04b}'.format(0b1100 & 0b1010)
'0b1000'
>>> '0b{:04b}'.format(0b1100 | 0b1010)
'0b1110'
>>> '0b{:04b}'.format(0b1100 ^ 0b1010)
'0b0110'
>>> '0b{:04b}'.format(0b1100 >> 2)
'0b0011'
>>> '0b{:04b}'.format(0b0011 << 2)
'0b1100'

Identity Operators

Python provides two operators, is and is not, that determine whether the given operands have the same identity-that is, refer to the same object. This is not the same thing as equality, which means the two operands refer to objects that contain the same data but are not necessarily the same object.

Here is an example of two object that are equal but not identical:

>>> x = 1001
>>> y = 1000 + 1
>>> print(x, y)
1001 1001

>>> x == y
True
>>> x is y
False

Here, x and y both refer to objects whose value is 1001. They are equal. But they do not reference the same object, as you can verify:

>>> id(x)
60307920
>>> id(y)
60307936

x and y do not have the same identity, and x is y returns False.

You saw previously that when you make an assignment like x = y, Python merely creates a second reference to the same object, and that you could confirm that fact with the id() function. You can also confirm it using the is operator:

>>> a = 'I am a string'
>>> b = a
>>> id(a)
55993992
>>> id(b)
55993992

>>> a is b
True
>>> a == b
True

In this case, since a and b reference the same object, it stands to reason that a and b would be equal as well.

Unsurprisingly, the opposite of is is is not:

>>> x = 10
>>> y = 20
>>> x is not y
True

Operator Precedence

Consider this expression:

>>> 20 + 4 * 10
60

There is ambiguity here. Should Python perform the addition 20 + 4 first and then multiply the sum by 10? Or should the multiplication 4 * 10 be performed first, and the addition of 20 second?

Clearly, since the result is 60, Python has chosen the latter; if it had chosen the former, the result would be 240. This is standard algebraic procedure, found universally in virtually all programming languages.

All operators that the language supports are assigned a precedence. In an expression, all operators of highest precedence are performed first. Once those results are obtained, operators of the next highest precedence are performed. So it continues, until the expression is fully evaluated. Any operators of equal precedence are performed in left-to-right order.

Here is the order of precedence of the Python operators you have seen so far, from lowest to highest:

Operators at the top of the table have the lowest precedence, and those at the bottom of the table have the highest. Any operators in the same row of the table have equal precedence.

It is clear why multiplication is performed first in the example above: multiplication has a higher precedence than addition.

Similarly, in the example below, 3 is raised to the power of 4 first, which equals 81, and then the multiplications are carried out in order from left to right (2 * 81 * 5 = 810):

>>> 2 * 3 ** 4 * 5
810

Operator precedence can be overridden using parentheses. Expressions in parentheses are always performed first, before expressions that are not parenthesized. Thus, the following happens:

>>> 20 + 4 * 10
60
>>> (20 + 4) * 10
240

>>> 2 * 3 ** 4 * 5
810
>>> 2 * 3 ** (4 * 5)
6973568802

In the first example, 20 + 4 is computed first, then the result is multiplied by 10. In the second example, 4 * 5 is calculated first, then 3 is raised to that power, then the result is multiplied by 2.

There is nothing wrong with making liberal use of parentheses, even when they aren't necessary to change the order of evaluation. In fact, it is considered good practice, because it can make the code more readable, and it relieves the reader of having to recall operator precedence from memory. Consider the following:

(a < 10) and (b > 30)

Here the parentheses are fully unnecessary, as the comparison operators have higher precedence than and does and would have been performed first anyhow. But some might consider the intent of the parenthesized version more immediately obvious than this version without parentheses:

a < 10 and b > 30

On the other hand, there are probably those who would prefer the latter; it's a matter of personal preference. The point is, you can always use parentheses if you feel it makes the code more readable, even if they aren't necessary to change the order of evaluation.

Augmented Assignment Operators

You have seen that a single equal sign (=) is used to assign a value to a variable. It is, of course, perfectly viable for the value to the right of the assignment to be an expression containing other variables:

>>> a = 10
>>> b = 20
>>> c = a * 5 + b
>>> c
70

In fact, the expression to the right of the assignment can include references to the variable that is being assigned to:

>>> a = 10
>>> a = a + 5
>>> a
15

>>> b = 20
>>> b = b * 3
>>> b
60

The first example is interpreted as "a is assigned the current value of a plus 5," effectively increasing the value of a by 5. The second reads "b is assigned the current value of b times 3," effectively increasing the value of b threefold.

Of course, this sort of assignment only makes sense if the variable in question has already previously been assigned a value:

>>> z = z / 12
Traceback (most recent call last):
  File "<pyshell#11>", line 1, in <module>
    z = z / 12
NameError: name 'z' is not defined

Python supports a shorthand augmented assignment notation for these arithmetic and bitwise operators:

For these operators, the following are equivalent:

x <op>= y
x = x <op> y

Take a look at these examples:

Conclusion

In this tutorial, you learned about the diverse operators Python supports to combine objects into expressions.

Most of the examples you have seen so far have involved only simple atomic data, but you saw a brief introduction to the string data type. The next tutorial will explore string objects in much more detail.

[ Improve Your Python With 🐍 Python Tricks 💌 - Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

20 Jun 2018 2:00pm GMT

After finishing our previous tutorial on Python variables in this series, you should now have a good grasp of creating and naming Python objects of different types. Let's do some work with them!

Here's what you'll learn in this tutorial: You'll see how calculations can be performed on objects in Python. By the end of this tutorial, you will be able to create complex expressions by combining objects and operators.

In Python, operators are special symbols that designate that some sort of computation should be performed. The values that an operator acts on are called operands.

Here is an example:

>>> a = 10
>>> b = 20
>>> a + b
30

In this case, the + operator adds the operands a and b together. An operand can be either a literal value or a variable that references an object:

>>> a = 10
>>> b = 20
>>> a + b - 5
25

A sequence of operands and operators, like a + b - 5, is called an expression. Python supports many operators for combining data objects into expressions. These are explored below.

Arithmetic Operators

The following table lists the arithmetic operators supported by Python:

Here are some examples of these operators in use:

>>> a = 4
>>> b = 3
>>> +a
4
>>> -b
-3
>>> a + b
7
>>> a - b
1
>>> a * b
12
>>> a / b
1.3333333333333333
>>> a % b
1
>>> a ** b
64

The result of standard division (/) is always a float, even if the dividend is evenly divisible by the divisor:

>>> 10 / 5
2.0
>>> type(10 / 5)
<class 'float'>

When the result of floor division (//) is positive, it is as though the fractional portion is truncated off, leaving only the integer portion. When the result is negative, the result is rounded down to the next smallest (greater negative) integer:

>>> 10 / 4
2.5
>>> 10 // 4
2
>>> 10 // -4
-3
>>> -10 // 4
-3
>>> -10 // -4
2

Note, by the way, that in a REPL session, you can display the value of an expression by just typing it in at the >>> prompt without print(), the same as you can with a literal value or variable:

>>> 25
25
>>> x = 4
>>> y = 6
>>> x
4
>>> y
6
>>> x * 25 + y
106

Comparison Operators

Here are examples of the comparison operators in use:

>>> a = 10
>>> b = 20
>>> a == b
False
>>> a != b
True
>>> a <= b
True
>>> a >= b
False

>>> a = 30
>>> b = 30
>>> a == b
True
>>> a <= b
True
>>> a >= b
True

Comparison operators are typically used in Boolean contexts like conditional and loop statements to direct program flow, as you will see later.

Equality Comparison on Floating-Point Values

Recall from the earlier discussion of floating-point numbers that the value stored internally for a float object may not be precisely what you'd think it would be. For that reason, it is poor practice to compare floating-point values for exact equality. Consider this example:

>>> x = 1.1 + 2.2
>>> x == 3.3
False

Yikes! The internal representations of the addition operands are not exactly equal to 1.1 and 2.2, so you cannot rely on x to compare exactly to 3.3.

The preferred way to determine whether two floating-point values are "equal" is to compute whether they are close to one another, given some tolerance. Take a look at this example:

>>> tolerance = 0.00001
>>> x = 1.1 + 2.2
>>> abs(x - 3.3) < tolerance
True

abs() returns absolute value. If the absolute value of the difference between the two numbers is less than the specified tolerance, they are close enough to one another to be considered equal.

Logical Operators

The logical operators not, or, and and modify and join together expressions evaluated in Boolean context to create more complex conditions.

Logical Expressions Involving Boolean Operands

As you have seen, some objects and expressions in Python actually are of Boolean type. That is, they are equal to one of the Python objects True or False. Consider these examples:

>>> x = 5
>>> x < 10
True
>>> type(x < 10)
<class 'bool'>

>>> t = x > 10
>>> t
False
>>> type(t)
<class 'bool'>

>>> callable(x)
False
>>> type(callable(x))
<class 'bool'>

>>> t = callable(len)
>>> t
True
>>> type(t)
<class 'bool'>

In the examples above, x < 10, callable(x), and t are all Boolean objects or expressions.

Interpretation of logical expressions involving not, or, and and is straightforward when the operands are Boolean:

Take a look at how they work in practice below.

"not" and Boolean Operands

x = 5
not x < 10
False
not callable(x)
True

"or" and Boolean Operands

x = 5
x < 10 or callable(x)
True
x < 0 or callable(x)
False

"and" and Boolean Operands

x = 5
x < 10 and callable(x)
False
x < 10 and callable(len)
True

Evaluation of Non-Boolean Values in Boolean Context

Many objects and expressions are not equal to True or False. Nonetheless, they may still be evaluated in Boolean context and determined to be "truthy" or "falsy."

So what is true and what isn't? As a philosophical question, that is outside the scope of this tutorial!

But in Python, it is well-defined. All the following are considered false when evaluated in Boolean context:

• The Boolean value False
• Any value that is numerically zero (0, 0.0, 0.0+0.0j)
• An empty string
• An object of a built-in composite data type which is empty (see below)
• The special value denoted by the Python keyword None

Virtually any other object built into Python is regarded as true.

You can determine the "truthiness" of an object or expression with the built-in bool() function. bool() returns True if its argument is truthy and False if it is falsy.

Numeric Value

A zero value is false.
A non-zero value is true.

>>> print(bool(0), bool(0.0), bool(0.0+0j))
False False False

>>> print(bool(-3), bool(3.14159), bool(1.0+1j))
True True True

String

An empty string is false.
A non-empty string is true.

>>> print(bool(''), bool(""), bool(""""""))
False False False

>>> print(bool('foo'), bool(" "), bool(''' '''))
True True True

Built-In Composite Data Object

Python provides built-in composite data types called list, tuple, dict, and set. These are "container" types that contain other objects. An object of one of these types is considered false if it is empty and true if it is non-empty.

The examples below demonstrate this for the list type. (Lists are defined in Python with square brackets.)

For more information on the list, tuple, dict, and set types, see the upcoming tutorials.

>>> type([])
<class 'list'>
>>> bool([])
False

>>> type([1, 2, 3])
<class 'list'>
>>> bool([1, 2, 3])
True

The "None" Keyword

None is always false:

>>> bool(None)
False

Logical Expressions Involving Non-Boolean Operands

Non-Boolean values can also be modified and joined by not, or and, and. The result depends on the "truthiness" of the operands.

"not" and Non-Boolean Operands

Here is what happens for a non-Boolean value x:

Here are some concrete examples:

>>> x = 3
>>> bool(x)
True
>>> not x
False

>>> x = 0.0
>>> bool(x)
False
>>> not x
True

"or" and Non-Boolean Operands

This is what happens for two non-Boolean values x and y:

Note that in this case, the expression x or y does not evaluate to either True or False, but instead to one of either x or y:

>>> x = 3
>>> y = 4
>>> x or y
3

>>> x = 0.0
>>> y = 4.4
>>> x or y
4.4

Even so, it is still the case that the expression x or y will be truthy if either x or y is truthy, and falsy if both x and y are falsy.

"and" and Non-Boolean Operands

Here's what you'll get for two non-Boolean values x and y:

>>> x = 3
>>> y = 4
>>> x and y
4

>>> x = 0.0
>>> y = 4.4
>>> x and y
0.0

As with or, the expression x and y does not evaluate to either True or False, but instead to one of either x or y. x and y will be truthy if both x and y are truthy, and falsy otherwise.

Compound Logical Expressions and Short-Circuit Evaluation

So far, you have seen expressions with only a single or or and operator and two operands:

x or y
x and y

Multiple logical operators and operands can be strung together to form compound logical expressions.

Compound "or" Expressions

Consider the following expression:

x₁ or x₂ or x₃ or … x_n

This expression is true if any of the x_i are true.

In an expression like this, Python uses a methodology called short-circuit evaluation, also called McCarthy evaluation in honor of computer scientist John McCarthy. The x_i operands are evaluated in order from left to right. As soon as one is found to be true, the entire expression is known to be true. At that point, Python stops and no more terms are evaluated. The value of the entire expression is that of the x_i that terminated evaluation.

To help demonstrate short-circuit evaluation, suppose that you have a simple "identity" function f() that behaves as follows:

• f() takes a single argument.
• It displays the argument to the console.
• It returns the argument passed to it as its return value.

(You will see how to define such a function in the upcoming tutorial on Functions.)

Several example calls to f() are shown below:

>>> f(0)
-> f(0) = 0
0

>>> f(False)
-> f(False) = False
False

>>> f(1.5)
-> f(1.5) = 1.5
1.5

Because f() simply returns the argument passed to it, we can make the expression f(arg) be truthy or falsy as needed by specifying a value for arg that is appropriately truthy or falsy. Additionally, f() displays its argument to the console, which visually confirms whether or not it was called.

Now, consider the following compound logical expression:

>>> f(0) or f(False) or f(1) or f(2) or f(3)
-> f(0) = 0
-> f(False) = False
-> f(1) = 1
1

The interpreter first evaluates f(0), which is 0. A numeric value of 0 is false. The expression is not true yet, so evaluation proceeds left to right. The next operand, f(False), returns False. That is also false, so evaluation continues.

Next up is f(1). That evaluates to 1, which is true. At that point, the interpreter stops because it now knows the entire expression to be true. 1 is returned as the value of the expression, and the remaining operands, f(2) and f(3), are never evaluated. You can see from the display that the f(2) and f(3) calls do not occur.

Compound "and" Expressions

A similar situation exists in an expression with multiple and operators:

x₁ and x₂ and x₃ and … x_n

This expression is true if all the x_i are true.

In this case, short-circuit evaluation dictates that the interpreter stop evaluating as soon as any operand is found to be false, because at that point the entire expression is known to be false. Once that is the case, no more operands are evaluated, and the falsy operand that terminated evaluation is returned as the value of the expression:

>>> f(1) and f(False) and f(2) and f(3)
-> f(1) = 1
-> f(False) = False
False

>>> f(1) and f(0.0) and f(2) and f(3)
-> f(1) = 1
-> f(0.0) = 0.0
0.0

In both examples above, evaluation stops at the first term that is false-f(False) in the first case, f(0.0) in the second case-and neither the f(2) nor f(3) call occurs. False and 0.0, respectively, are returned as the value of the expression.

If all the operands are truthy, they all get evaluated and the last (rightmost) one is returned as the value of the expression:

>>> f(1) and f(2.2) and f('bar')
-> f(1) = 1
-> f(2.2) = 2.2
-> f(bar) = bar
'bar'

Idioms That Exploit Short-Circuit Evaluation

There are some common idiomatic patterns that exploit short-circuit evaluation for conciseness of expression.

Avoiding an Exception

Suppose you have defined two variables a and b, and you want to know whether (b / a) > 0:

>>> a = 3
>>> b = 1
>>> (b / a) > 0
True

But you need to account for the possibility that a might be 0, in which case the interpreter will raise an exception:

>>> a = 0
>>> b = 1
>>> (b / a) > 0
Traceback (most recent call last):
  File "<pyshell#2>", line 1, in <module>
    (b / a) > 0
ZeroDivisionError: division by zero

You can avoid an error with an expression like this:

>>> a = 0
>>> b = 1
>>> a != 0 and (b / a) > 0
False

When a is 0, a != 0 is false. Short-circuit evaluation ensures that evaluation stops at that point. (b / a) is not evaluated, and no error is raised.

If fact, you can be even more concise than that. When a is 0, the expression a by itself is falsy. There is no need for the explicit comparison a != 0:

>>> a = 0
>>> b = 1
>>> a and (b / a) > 0
0

Selecting a Default Value

Another idiom involves selecting a default value when a specified value is zero or empty. For example, suppose you want to assign a variable s to the value contained in another variable called string. But if string is empty, you want to supply a default value.

Here is a concise way of expressing this using short-circuit evaluation:

s = string or '<default_value>'

If string is non-empty, it is truthy, and the expression string or '<default_value>' will be true at that point. Evaluation stops, and the value of string is returned and assigned to s:

>>> string = 'foo bar'
>>> s = string or '<default_value>'
>>> s
'foo bar'

On the other hand, if string is an empty string, it is falsy. Evaluation of string or '<default_value>' continues to the next operand, '<default_value>', which is returned and assigned to s:

>>> string = ''
>>> s = string or '<default_value>'
>>> s
'<default_value>'

Chained Comparisons

Comparison operators can be chained together to arbitrary length. For example, the following expressions are nearly equivalent:

x < y <= z
x < y and y <= z

They will both evaluate to the same Boolean value. The subtle difference between the two is that in the chained comparison x < y <= z, y is evaluated only once. The longer expression x < y and y <= z will cause y to be evaluated twice.

x < f() <= z
x < f() and f() <= z

More generally, if op₁, op₂, …, op_n are comparison operators, then the following have the same Boolean value:

x₁ op₁ x₂ op₂ x₃ … x_n-1 op_n x_n

x₁ op₁ x₂ and x₂ op₂ x₃ and … x_n-1 op_n x_n

In the former case, each x_i is only evaluated once. In the latter case, each will be evaluated twice except the first and last, unless short-circuit evaluation causes premature termination.

Bitwise Operators

Bitwise operators treat operands as sequences of binary digits and operate on them bit by bit. The following operators are supported:

Here are some examples:

>>> '0b{:04b}'.format(0b1100 & 0b1010)
'0b1000'
>>> '0b{:04b}'.format(0b1100 | 0b1010)
'0b1110'
>>> '0b{:04b}'.format(0b1100 ^ 0b1010)
'0b0110'
>>> '0b{:04b}'.format(0b1100 >> 2)
'0b0011'
>>> '0b{:04b}'.format(0b0011 << 2)
'0b1100'

Identity Operators

Python provides two operators, is and is not, that determine whether the given operands have the same identity-that is, refer to the same object. This is not the same thing as equality, which means the two operands refer to objects that contain the same data but are not necessarily the same object.

Here is an example of two object that are equal but not identical:

>>> x = 1001
>>> y = 1000 + 1
>>> print(x, y)
1001 1001

>>> x == y
True
>>> x is y
False

Here, x and y both refer to objects whose value is 1001. They are equal. But they do not reference the same object, as you can verify:

>>> id(x)
60307920
>>> id(y)
60307936

x and y do not have the same identity, and x is y returns False.

You saw previously that when you make an assignment like x = y, Python merely creates a second reference to the same object, and that you could confirm that fact with the id() function. You can also confirm it using the is operator:

>>> a = 'I am a string'
>>> b = a
>>> id(a)
55993992
>>> id(b)
55993992

>>> a is b
True
>>> a == b
True

In this case, since a and b reference the same object, it stands to reason that a and b would be equal as well.

Unsurprisingly, the opposite of is is is not:

>>> x = 10
>>> y = 20
>>> x is not y
True

Operator Precedence

Consider this expression:

>>> 20 + 4 * 10
60

There is ambiguity here. Should Python perform the addition 20 + 4 first and then multiply the sum by 10? Or should the multiplication 4 * 10 be performed first, and the addition of 20 second?

Clearly, since the result is 60, Python has chosen the latter; if it had chosen the former, the result would be 240. This is standard algebraic procedure, found universally in virtually all programming languages.

All operators that the language supports are assigned a precedence. In an expression, all operators of highest precedence are performed first. Once those results are obtained, operators of the next highest precedence are performed. So it continues, until the expression is fully evaluated. Any operators of equal precedence are performed in left-to-right order.

Here is the order of precedence of the Python operators you have seen so far, from lowest to highest:

Operators at the top of the table have the lowest precedence, and those at the bottom of the table have the highest. Any operators in the same row of the table have equal precedence.

It is clear why multiplication is performed first in the example above: multiplication has a higher precedence than addition.

Similarly, in the example below, 3 is raised to the power of 4 first, which equals 81, and then the multiplications are carried out in order from left to right (2 * 81 * 5 = 810):

>>> 2 * 3 ** 4 * 5
810

Operator precedence can be overridden using parentheses. Expressions in parentheses are always performed first, before expressions that are not parenthesized. Thus, the following happens:

>>> 20 + 4 * 10
60
>>> (20 + 4) * 10
240

>>> 2 * 3 ** 4 * 5
810
>>> 2 * 3 ** (4 * 5)
6973568802

In the first example, 20 + 4 is computed first, then the result is multiplied by 10. In the second example, 4 * 5 is calculated first, then 3 is raised to that power, then the result is multiplied by 2.

There is nothing wrong with making liberal use of parentheses, even when they aren't necessary to change the order of evaluation. In fact, it is considered good practice, because it can make the code more readable, and it relieves the reader of having to recall operator precedence from memory. Consider the following:

(a < 10) and (b > 30)

Here the parentheses are fully unnecessary, as the comparison operators have higher precedence than and does and would have been performed first anyhow. But some might consider the intent of the parenthesized version more immediately obvious than this version without parentheses:

a < 10 and b > 30

On the other hand, there are probably those who would prefer the latter; it's a matter of personal preference. The point is, you can always use parentheses if you feel it makes the code more readable, even if they aren't necessary to change the order of evaluation.

Augmented Assignment Operators

You have seen that a single equal sign (=) is used to assign a value to a variable. It is, of course, perfectly viable for the value to the right of the assignment to be an expression containing other variables:

>>> a = 10
>>> b = 20
>>> c = a * 5 + b
>>> c
70

In fact, the expression to the right of the assignment can include references to the variable that is being assigned to:

>>> a = 10
>>> a = a + 5
>>> a
15

>>> b = 20
>>> b = b * 3
>>> b
60

The first example is interpreted as "a is assigned the current value of a plus 5," effectively increasing the value of a by 5. The second reads "b is assigned the current value of b times 3," effectively increasing the value of b threefold.

Of course, this sort of assignment only makes sense if the variable in question has already previously been assigned a value:

>>> z = z / 12
Traceback (most recent call last):
  File "<pyshell#11>", line 1, in <module>
    z = z / 12
NameError: name 'z' is not defined

Python supports a shorthand augmented assignment notation for these arithmetic and bitwise operators: