On the Internet, it's important to secure all of your communications.

There are a number of applications which purport to give you "secure chat", "secure email", or "secure phone calls".

The problem with these applications is that they advertise their presence. Since "insecure chat", "insecure email" and "insecure phone calls" all have a particular, detectable signature, an interested observer may easily detect your supposedly "secure" communication. Not only that, but the places that you go to obtain them are suspicious in their own right. In order to visit Whisper Systems, you have to be looking for "secure" communications.

This allows the adversary to use "security" technologies such as encryption as a sort of stencil, to outline and highlight the communication that they really want to be capturing. In the case of the NSA, this dumps anyone who would like to have a serious private conversation with a friend into the same bucket, from the perspective of the authorities, as a conspiracy of psychopaths trying to commit mass murder.

The Snowden documents already demonstrate that the NSA does exactly this; if you send a normal email, they will probably lose interest and ignore it after a little while, whereas if you send a "secure" email, they will store it forever and keep trying to crack it to see what you're hiding.

If you're running a supposedly innocuous online service or writing a supposedly harmless application, the hassle associated with setting up TLS certificates and encryption keys may seem like a pointless distraction. It isn't.

For one thing, if you have anywhere that user-created content enters your service, you don't know what they are going to be using it to communicate. Maybe you're just writing an online game but users will use your game for something as personal as courtship. Can we agree that the state security services shouldn't be involved in that?. Even if you were specifically writing an app for dating, you might not anticipate that the police will use it to show up and arrest your users so that they will be savagely beaten in jail.

The technology problems that "secure" services are working on are all important. But we can't simply develop a good "secure" technology, consider it a niche product, and leave it at that. Those of us who are software development professionals need to build security into every product, because users expect it. Users expect it because we are, in a million implicit ways, telling them that they have it. If we put a "share with your friend!" button into a user interface, that's a claim: we're claiming that the data the user indicates is being shared only with their friend. Would we want to put in a button that says "share with your friend, and with us, and with the state security apparatus, and with any criminal who can break in and steal our database!"? Obviously not. So let's stop making the "share with your friend!" button actually do that.

Those of us who understand the importance of security and are in the business of creating secure software must, therefore, take on the Sisyphean task of not only creating good security, but of competing with the insecure software on its own turf, so that people actually use it. "Slightly worse to use than your regular email program, but secure" is not good enough. (Not to mention the fact that existing security solutions are more than "slightly" worse to use). Secure stuff has to be as good as or better than its insecure competitors.

I know that this is a monumental undertaking. I have personally tried and failed to do something like this more than once. As the Rabbi Tarfon put it, though:

It is not incumbent upon you to complete the work, but neither are you at liberty to desist from it.

25 Jan 2015 12:16am GMT

On the Internet, it's important to secure all of your communications.

There are a number of applications which purport to give you "secure chat", "secure email", or "secure phone calls".

The problem with these applications is that they advertise their presence. Since "insecure chat", "insecure email" and "insecure phone calls" all have a particular, detectable signature, an interested observer may easily detect your supposedly "secure" communication. Not only that, but the places that you go to obtain them are suspicious in their own right. In order to visit Whisper Systems, you have to be looking for "secure" communications.

This allows the adversary to use "security" technologies such as encryption as a sort of stencil, to outline and highlight the communication that they really want to be capturing. In the case of the NSA, this dumps anyone who would like to have a serious private conversation with a friend into the same bucket, from the perspective of the authorities, as a conspiracy of psychopaths trying to commit mass murder.

The Snowden documents already demonstrate that the NSA does exactly this; if you send a normal email, they will probably lose interest and ignore it after a little while, whereas if you send a "secure" email, they will store it forever and keep trying to crack it to see what you're hiding.

If you're running a supposedly innocuous online service or writing a supposedly harmless application, the hassle associated with setting up TLS certificates and encryption keys may seem like a pointless distraction. It isn't.

For one thing, if you have anywhere that user-created content enters your service, you don't know what they are going to be using it to communicate. Maybe you're just writing an online game but users will use your game for something as personal as courtship. Can we agree that the state security services shouldn't be involved in that?. Even if you were specifically writing an app for dating, you might not anticipate that the police will use it to show up and arrest your users so that they will be savagely beaten in jail.

The technology problems that "secure" services are working on are all important. But we can't simply develop a good "secure" technology, consider it a niche product, and leave it at that. Those of us who are software development professionals need to build security into every product, because users expect it. Users expect it because we are, in a million implicit ways, telling them that they have it. If we put a "share with your friend!" button into a user interface, that's a claim: we're claiming that the data the user indicates is being shared only with their friend. Would we want to put in a button that says "share with your friend, and with us, and with the state security apparatus, and with any criminal who can break in and steal our database!"? Obviously not. So let's stop making the "share with your friend!" button actually do that.

Those of us who understand the importance of security and are in the business of creating secure software must, therefore, take on the Sisyphean task of not only creating good security, but of competing with the insecure software on its own turf, so that people actually use it. "Slightly worse to use than your regular email program, but secure" is not good enough. (Not to mention the fact that existing security solutions are more than "slightly" worse to use). Secure stuff has to be as good as or better than its insecure competitors.

I know that this is a monumental undertaking. I have personally tried and failed to do something like this more than once. As the Rabbi Tarfon put it, though:

It is not incumbent upon you to complete the work, but neither are you at liberty to desist from it.

25 Jan 2015 12:16am GMT

Many Western governors are pushing for laws mandating all private communication can be secretly read and analyzed for them.The latest attack targets the one technology that still enables some privacy on a massively surveilled internet: end-to-end encryption. As hackers or IT people we can not afford to lament that the public doesn't understand the significance of end-to-end crypto or privacy if we don't appreciate its value for societies at home and abroad ourselves.

Responding to the renewed surveillance attacks with quick technical or narrow economic counter arguments is not going to work. An appropriate response needs to consider the political history and context of the current crypto and surveillance debates. Moreover, to stem the never-ending waves of new secret agency laws a re-framing of the common security debates is crucial to avoid the never-ending succession of new powers for government.

Let me start by rejecting the idea that governmental surveillance attacks have anything to do with fighting ruthless killers ("terrorists") however often this claim is repeated in broadcast media. This is not to disregard the power of repetition, see the endlessly repeated claims of the existence of "Weapons of mass destruction" as a pretext for the Iraq war, or the fact that advertisements work. But despite endless repetition, governmental surveillance attacks don't have anything to do with fighting terrorists. To turn it around, and i think the burden ought to be on the framers, where is the hard evidence that mass surveillance of civilians has significant effect, if any, on preventing terrorist attacks against civilians? And even if surveillance would prevent a few attacks how would it compare to the dangers of more government power?

The "fight terrorists with surveillance" discussion framing is seriously flawed also for another reason. Within it you are always going to lose the argument against more surveillance. If not now then after the next terror event. Because proponents can always argue they were right: if no attack happens it proves surveillance works and we need more of it. If an attack happens it also proves we need more surveillance. In this framed logic there can never be any rolling back of government powers.

The way out is to unframe the discussion and discuss the political and historical contexts of "terror attacks" and "expanding surveillance" separately. Let's start with surveillance. If fighting terrorism is a red herring what are the motivations and politics of expanding government surveillance?

Governors worry about their power base

Governors of all kinds worry that people decide to change things in ways which endanger the power their associated networks hold. And they are particularly afraid today because they know there are many reasons why people want to change things in more fundamental ways. As much as people have lost trust in governors, governors have lost trust into people to keep them and their ilk in power.

The fear of governors seems justified if you look at the example of Spain in 2015: big parts of Spain's social movements associate with a very new party on the block: Podemos. It aims to win the election in December and currently is leading the polls against the two parties which have governed Spain since 1975. It could actually happen despite the German chancellor Merkel supporting the Spanish president Rajoy who just introduced draconian laws against protesters and is generally sending his troops everywhere to avert the decline of his power network. Having to resort to direct repression is a sign of lost political power and in the case of Spain, panic. If you remember that Spain is a major EU country it's understandable that many other governors in the West are worried something similar might happen to them soon.

Governors are always afraid they could lose their sight and grip over what people in their constituency are up to. Today it is not enough to have buddies in broadcast media which frame the discussion and interpretation of events to the governor's liking. You also need to understand and contain, if possible, wider internet discussions before they can effect change you don't want. Governors learned from Hannah Arendt that private discussions form the basis for public opinions which in turn structure and determine governmental power. If that weren't the case how could feminist and really any social struggle have succeeded? It certainly never was the broadcast media or governors who first talked about and demanded rights for women or other oppressed groups.

How to contain decentralized communication?

New realities are co-created in a more decentralized manner and quicker than ever. Communication platforms grew in the last decade because of the interests of people to communicate and connect with one another. Maybe that's due to a lost sense of community in disintegrating city neighborhoods which make people use "social media". But in any case, Youtube, Twitter, Gmail, Facebook and IOS/Android app platforms became big because they facilitated decentralized communication and sharing between people. This presents a problem to governors because web communications are harder to contain in acceptable ways.

For a typical broadcast media discussion format you can send allied experts and construct "position" and "opposition" and thus frame the discussion. For example, it's acceptable to discuss the virtues and dangers of "press freedom", how to deal with "islamist militants" or how to "defend our values and rights". Western Governors find it much less acceptable to link the Hebdo killing of or the rise of the "Islamic State" to the recent Western wars in Iraq, Libya and Syria, or to the everyday killing of civilians through Western drones and torture. Governors can't yet directly contain such unacceptable linking activities and they are worried about it. For the time being, they try to frame it as irrelevant and repeat the "we are being attacked by ruthless killers" on broadcast media some more. It still kind of works but it's unclear for how long.

What helps to contain discussions is to implant "You are being watched!" into the minds of people discussing the future of their governance. Putting up some public examples of punishment for unacceptable dissent refines the message into "Watch your words (and internet links!)" … also known as internalized or self-censorship. That's not just effective for governors in Saudi Arabia but for their Western allies as well. The recent US sentencing of journalist Barret Brown to 48 months of prison for posting a link to some leaked data on an IRC channel can be seen as an example of a public punishment with chilling effects.

Arguments and national tactics against crypto attacks

Governors have long realized they can exploit central communication platform ownership to tap into most private communications. But to their apparent shock, many IT companies in the Post-Snowden era are implementing decentralized encryption because they in turn want to assure users that they can't surveil their private messages. As a reaction, governors are conspiring to prevent decentralized encryption reaching the masses which would see them losing their current in-depth access to private communication. Psychologically speaking, losing power is always harder to accept than not having it in the first place.

A response to the crypto attacks which I consider optimistic, if not shallow, is "it's not technically feasible to regulate or ban end-to-end crypto". It underestimates the ability of governors to write laws which will drastically change the playing field even if in an incremental manner. To begin with, why shouldn't it be possible to prevent companies from distributing apps which incorporate decentralized encryption? Google and Apple already employ their own regulation on what kind of apps are distributed through their stores. Another regulation on decentralized-crypto apps can probably be added by the governors in the US. And that would prevent decentralized encryption reaching the masses at least in the short term.

As to government access to end-to-end encryption, it's true that backdooring crypto would make people more vulnerable against all kind of exploiting attacks, not just governmental ones. Governors might frame this dillema by claiming that security against physical attacks is more important than security against someone reading your messages. Such an argument already incorporates the flawed "it's all about anti-terror" framing. The increased vulnerability of everyone's devices is a bit of a tricky issue for governors given they couldn't protect their own data against Snowden. If neccessary, governors will try to make concessions. Some applications such as online banking could be allowed to use non-backdoored crypto. They have all the banking data already, anyway. They probably will want to exempt governmental communication itself as well. With that we'd end up with a complete reversal of the democratic principle: public governments to act in secret and private communication to be constantly surveilled.

Western Governors have learned from the last Cryptowars battles. They know full well that they can only break private communication encryption if they outlaw it in a synchronized international manner. Otherwise they would have a harder time to overcome national arguments like "companies are going to leave the country if you ban decentral encryption". Therefore, we need to fend off attacks on decentralized crypto in at least some Western countries to make such commercial arguments useful. Concretely, US companies like Google and Apple will more strongly resist if the EU does not also illegalize decentralized crypto.

It is as crucial to prevent EU crypto regulations as it was two decades ago. During the crypto battles in the 1990ties I studied with the deeply inspiring Prof. Andreas Pfitzmann who consulted the German government on crypto regulation. Along with other colleagues and groups he tirelessly worked and finally turned the tides and prevented Germany and thus the EU from introducing government backdoors to crypto algorithms. This in turn lead France and then the US to drop their plans and eventually relax crypto export regulations to keep their companies competitive. Today, we are back to square zero and must again convince some EU governments or parliaments to refrain from crypto banning laws. It's a fight we better not lose.

Lastly, I'd like to be clear if maybe controversial on the dreadful Anti-Terror topic: If the Western governments want to stop killers from targetting western individuals they first need to stop ruthlessly killing and terrorizing individuals from abroad. Nothing else will bring more physical security against terrorist attacks. It reminds me of the 2500 year old question from the chinese politician and philosopher Confucius: "The way out is via the door. Why is it that no one will use this method?"

24 Jan 2015 11:40pm GMT

Many Western governors are pushing for laws mandating all private communication can be secretly read and analyzed for them.The latest attack targets the one technology that still enables some privacy on a massively surveilled internet: end-to-end encryption. As hackers or IT people we can not afford to lament that the public doesn't understand the significance of end-to-end crypto or privacy if we don't appreciate its value for societies at home and abroad ourselves.

Responding to the renewed surveillance attacks with quick technical or narrow economic counter arguments is not going to work. An appropriate response needs to consider the political history and context of the current crypto and surveillance debates. Moreover, to stem the never-ending waves of new secret agency laws a re-framing of the common security debates is crucial to avoid the never-ending succession of new powers for government.

Let me start by rejecting the idea that governmental surveillance attacks have anything to do with fighting ruthless killers ("terrorists") however often this claim is repeated in broadcast media. This is not to disregard the power of repetition, see the endlessly repeated claims of the existence of "Weapons of mass destruction" as a pretext for the Iraq war, or the fact that advertisements work. But despite endless repetition, governmental surveillance attacks don't have anything to do with fighting terrorists. To turn it around, and i think the burden ought to be on the framers, where is the hard evidence that mass surveillance of civilians has significant effect, if any, on preventing terrorist attacks against civilians? And even if surveillance would prevent a few attacks how would it compare to the dangers of more government power?

The "fight terrorists with surveillance" discussion framing is seriously flawed also for another reason. Within it you are always going to lose the argument against more surveillance. If not now then after the next terror event. Because proponents can always argue they were right: if no attack happens it proves surveillance works and we need more of it. If an attack happens it also proves we need more surveillance. In this framed logic there can never be any rolling back of government powers.

The way out is to unframe the discussion and discuss the political and historical contexts of "terror attacks" and "expanding surveillance" separately. Let's start with surveillance. If fighting terrorism is a red herring what are the motivations and politics of expanding government surveillance?

Governors worry about their power base

Governors of all kinds worry that people decide to change things in ways which endanger the power their associated networks hold. And they are particularly afraid today because they know there are many reasons why people want to change things in more fundamental ways. As much as people have lost trust in governors, governors have lost trust into people to keep them and their ilk in power.

The fear of governors seems justified if you look at the example of Spain in 2015: big parts of Spain's social movements associate with a very new party on the block: Podemos. It aims to win the election in December and currently is leading the polls against the two parties which have governed Spain since 1975. It could actually happen despite the German chancellor Merkel supporting the Spanish president Rajoy who just introduced draconian laws against protesters and is generally sending his troops everywhere to avert the decline of his power network. Having to resort to direct repression is a sign of lost political power and in the case of Spain, panic. If you remember that Spain is a major EU country it's understandable that many other governors in the West are worried something similar might happen to them soon.

Governors are always afraid they could lose their sight and grip over what people in their constituency are up to. Today it is not enough to have buddies in broadcast media which frame the discussion and interpretation of events to the governor's liking. You also need to understand and contain, if possible, wider internet discussions before they can effect change you don't want. Governors learned from Hannah Arendt that private discussions form the basis for public opinions which in turn structure and determine governmental power. If that weren't the case how could feminist and really any social struggle have succeeded? It certainly never was the broadcast media or governors who first talked about and demanded rights for women or other oppressed groups.

How to contain decentralized communication?

New realities are co-created in a more decentralized manner and quicker than ever. Communication platforms grew in the last decade because of the interests of people to communicate and connect with one another. Maybe that's due to a lost sense of community in disintegrating city neighborhoods which make people use "social media". But in any case, Youtube, Twitter, Gmail, Facebook and IOS/Android app platforms became big because they facilitated decentralized communication and sharing between people. This presents a problem to governors because web communications are harder to contain in acceptable ways.

For a typical broadcast media discussion format you can send allied experts and construct "position" and "opposition" and thus frame the discussion. For example, it's acceptable to discuss the virtues and dangers of "press freedom", how to deal with "islamist militants" or how to "defend our values and rights". Western Governors find it much less acceptable to link the Hebdo killing of or the rise of the "Islamic State" to the recent Western wars in Iraq, Libya and Syria, or to the everyday killing of civilians through Western drones and torture. Governors can't yet directly contain such unacceptable linking activities and they are worried about it. For the time being, they try to frame it as irrelevant and repeat the "we are being attacked by ruthless killers" on broadcast media some more. It still kind of works but it's unclear for how long.

What helps to contain discussions is to implant "You are being watched!" into the minds of people discussing the future of their governance. Putting up some public examples of punishment for unacceptable dissent refines the message into "Watch your words (and internet links!)" … also known as internalized or self-censorship. That's not just effective for governors in Saudi Arabia but for their Western allies as well. The recent US sentencing of journalist Barret Brown to 48 months of prison for posting a link to some leaked data on an IRC channel can be seen as an example of a public punishment with chilling effects.

Arguments and national tactics against crypto attacks

Governors have long realized they can exploit central communication platform ownership to tap into most private communications. But to their apparent shock, many IT companies in the Post-Snowden era are implementing decentralized encryption because they in turn want to assure users that they can't surveil their private messages. As a reaction, governors are conspiring to prevent decentralized encryption reaching the masses which would see them losing their current in-depth access to private communication. Psychologically speaking, losing power is always harder to accept than not having it in the first place.

A response to the crypto attacks which I consider optimistic, if not shallow, is "it's not technically feasible to regulate or ban end-to-end crypto". It underestimates the ability of governors to write laws which will drastically change the playing field even if in an incremental manner. To begin with, why shouldn't it be possible to prevent companies from distributing apps which incorporate decentralized encryption? Google and Apple already employ their own regulation on what kind of apps are distributed through their stores. Another regulation on decentralized-crypto apps can probably be added by the governors in the US. And that would prevent decentralized encryption reaching the masses at least in the short term.

As to government access to end-to-end encryption, it's true that backdooring crypto would make people more vulnerable against all kind of exploiting attacks, not just governmental ones. Governors might frame this dillema by claiming that security against physical attacks is more important than security against someone reading your messages. Such an argument already incorporates the flawed "it's all about anti-terror" framing. The increased vulnerability of everyone's devices is a bit of a tricky issue for governors given they couldn't protect their own data against Snowden. If neccessary, governors will try to make concessions. Some applications such as online banking could be allowed to use non-backdoored crypto. They have all the banking data already, anyway. They probably will want to exempt governmental communication itself as well. With that we'd end up with a complete reversal of the democratic principle: public governments to act in secret and private communication to be constantly surveilled.

Western Governors have learned from the last Cryptowars battles. They know full well that they can only break private communication encryption if they outlaw it in a synchronized international manner. Otherwise they would have a harder time to overcome national arguments like "companies are going to leave the country if you ban decentral encryption". Therefore, we need to fend off attacks on decentralized crypto in at least some Western countries to make such commercial arguments useful. Concretely, US companies like Google and Apple will more strongly resist if the EU does not also illegalize decentralized crypto.

It is as crucial to prevent EU crypto regulations as it was two decades ago. During the crypto battles in the 1990ties I studied with the deeply inspiring Prof. Andreas Pfitzmann who consulted the German government on crypto regulation. Along with other colleagues and groups he tirelessly worked and finally turned the tides and prevented Germany and thus the EU from introducing government backdoors to crypto algorithms. This in turn lead France and then the US to drop their plans and eventually relax crypto export regulations to keep their companies competitive. Today, we are back to square zero and must again convince some EU governments or parliaments to refrain from crypto banning laws. It's a fight we better not lose.

Lastly, I'd like to be clear if maybe controversial on the dreadful Anti-Terror topic: If the Western governments want to stop killers from targetting western individuals they first need to stop ruthlessly killing and terrorizing individuals from abroad. Nothing else will bring more physical security against terrorist attacks. It reminds me of the 2500 year old question from the chinese politician and philosopher Confucius: "The way out is via the door. Why is it that no one will use this method?"

24 Jan 2015 11:40pm GMT

23 Jan 2015 7:44pm GMT

23 Jan 2015 7:44pm GMT

Ever since WhatsApp announced their web app, I have seen various people complain about having to keep your phone on to send messages. But in all of these posts people seem to be overlooking two key points about the design of WhatsApp which either necessitate this design or at least facilitate WhatsApp in keeping their service lean and fast.

Encryption

In case you didn't hear, WhatsApp (supposedly) does end-to-end encryption. That means WhatsApp can't read what people say in their messages. What this also means, though, is that if someone sends you a message and you have your phone and web app, there is no way to read that message independently of both devices without sharing the decryption key, which is not good security practice. You could have separate keys for your phone and web browser, but then you would then need to make sure to share keys between your phone, your browser, and all of your contacts so they could receive messages from any of your devices.

But if you treat your phone as your personal WhatsApp server, then your phone can continue to have the master keys for your account and then you only need to manage keys between your phone and your other devices, making it a hub-and-spoke system where you phone is the hub and your other devices are the spokes. This keeps WhatsApp out of the key management business and allows you to easily revoke keys for your other devices without WhatsApp having to store anything for you. So tying everything through your phone keeps everything encrypted and simplifies key sharing between users to just their phones and keeps security simple which is how you want it to be.

Storage

Someone pointed out to me that WhatsApp is not in the storage business, and when you stop and look at how the service was structured before the web app came along you will realize that WhatsApp tried to minimize what data it had to store from its inception. For instance, if you get a new phone you actually have to migrate your messages over yourself. Since WhatsApp doesn't keep messages on their servers, your phone ends up being the keeper of truth when it comes to what messages you sent and received.

But the key thing about WhatsApp not storing messages on their servers is how much it simplifies their service. Consider their last publicly stated user count of 500,000,000. Since WhatsApp doesn't store messages for you, they really only need to store messages that have yet to be delivered to a user's phone and your account's configuration data. So let's assume every user suddenly sent a bunch of photos that came to a total of 1 MB of pending messages (remember that WhatsApp is only going to show you a small version of a photo and so they can compress them such that they don't take up much space; 1 MB should go a long way). That's 1 MB * 500,000,000 = 500 TB of storage. OK, not a puny number for most services.

But let's look at this from a cloud perspective. Let's say you wanted an extremely fast service, so you would want to use local SSD which Google Compute Engine offers. As I write this, a local SSD on GCE is 375 GB and you can have up to 4 per instance. At 375 GB that gets means you would need 1,334 SSDs to store 500 TB of data (it would also require at minimum 334 instances and probably enough for the service to run, but I'm not pricing out computation or bandwidth costs to keep this simple). Now according to GCE's pricing, a local SSD costs USD 81.75/month. That means it would cost you USD 81.75 * 1,334 SSDs = USD 109,054.50/month or USD 1,308,654/year. Now let's be totally extravagant and say you want the data replicated near the sender, near the receiver, and on one other continent for safekeeping until delivery (when the sender and receiver are on the same continent the data could go to two separate clusters). So we are talking about 3N data replication. That works out to USD 3,925,962/year in local SSD storage costs if you used Google Compute Engine at full retail and always maintained this level of storage constantly. In the current startup climate, USD 4 million/year is pittance (and unnecessary as I bet WhatsApp could get away with 2N data replication if that since this is only for buffering purposes). And for a company like Facebook who have their own clusters? Storing 1.5 PB of data would not be difficult at all, so WhatsApp could go as far as backing up the data on every populated continent if they wanted to for 3 PB or less than USD 8 million/year.

In other words by relying on your phone as the storage mechanism for messages and not having to keep anything on their own servers, WhatsApp can run very cheaply and efficiently (and only cost you USD 1/year as a user). This makes WhatsApp basically nothing more than a fast routing service with some buffering between phones, much like the telcos (which might be what inspired WhatsApp to use Erlang). It's rather ingenious and probably why the service seems so fast and has such great uptime.

And I bet users don't care about the potential loss of messages either; when was the last time you scrolled back into the history on your phone to a point that preceded you getting that phone or having catastrophic data loss on the device? Plus you have to consider how many users of WhatsApp are really going to use the web app; I bet a large portion of their users don't even have their own computers beyond their phones so this web app service probably isn't critical to WhatsApp's success.

23 Jan 2015 5:27pm GMT

Ever since WhatsApp announced their web app, I have seen various people complain about having to keep your phone on to send messages. But in all of these posts people seem to be overlooking two key points about the design of WhatsApp which either necessitate this design or at least facilitate WhatsApp in keeping their service lean and fast.

Encryption

In case you didn't hear, WhatsApp (supposedly) does end-to-end encryption. That means WhatsApp can't read what people say in their messages. What this also means, though, is that if someone sends you a message and you have your phone and web app, there is no way to read that message independently of both devices without sharing the decryption key, which is not good security practice. You could have separate keys for your phone and web browser, but then you would then need to make sure to share keys between your phone, your browser, and all of your contacts so they could receive messages from any of your devices.

But if you treat your phone as your personal WhatsApp server, then your phone can continue to have the master keys for your account and then you only need to manage keys between your phone and your other devices, making it a hub-and-spoke system where you phone is the hub and your other devices are the spokes. This keeps WhatsApp out of the key management business and allows you to easily revoke keys for your other devices without WhatsApp having to store anything for you. So tying everything through your phone keeps everything encrypted and simplifies key sharing between users to just their phones and keeps security simple which is how you want it to be.

Storage

Someone pointed out to me that WhatsApp is not in the storage business, and when you stop and look at how the service was structured before the web app came along you will realize that WhatsApp tried to minimize what data it had to store from its inception. For instance, if you get a new phone you actually have to migrate your messages over yourself. Since WhatsApp doesn't keep messages on their servers, your phone ends up being the keeper of truth when it comes to what messages you sent and received.

But the key thing about WhatsApp not storing messages on their servers is how much it simplifies their service. Consider their last publicly stated user count of 500,000,000. Since WhatsApp doesn't store messages for you, they really only need to store messages that have yet to be delivered to a user's phone and your account's configuration data. So let's assume every user suddenly sent a bunch of photos that came to a total of 1 MB of pending messages (remember that WhatsApp is only going to show you a small version of a photo and so they can compress them such that they don't take up much space; 1 MB should go a long way). That's 1 MB * 500,000,000 = 500 TB of storage. OK, not a puny number for most services.

But let's look at this from a cloud perspective. Let's say you wanted an extremely fast service, so you would want to use local SSD which Google Compute Engine offers. As I write this, a local SSD on GCE is 375 GB and you can have up to 4 per instance. At 375 GB that gets means you would need 1,334 SSDs to store 500 TB of data (it would also require at minimum 334 instances and probably enough for the service to run, but I'm not pricing out computation or bandwidth costs to keep this simple). Now according to GCE's pricing, a local SSD costs USD 81.75/month. That means it would cost you USD 81.75 * 1,334 SSDs = USD 109,054.50/month or USD 1,308,654/year. Now let's be totally extravagant and say you want the data replicated near the sender, near the receiver, and on one other continent for safekeeping until delivery (when the sender and receiver are on the same continent the data could go to two separate clusters). So we are talking about 3N data replication. That works out to USD 3,925,962/year in local SSD storage costs if you used Google Compute Engine at full retail and always maintained this level of storage constantly. In the current startup climate, USD 4 million/year is pittance (and unnecessary as I bet WhatsApp could get away with 2N data replication if that since this is only for buffering purposes). And for a company like Facebook who have their own clusters? Storing 1.5 PB of data would not be difficult at all, so WhatsApp could go as far as backing up the data on every populated continent if they wanted to for 3 PB or less than USD 8 million/year.

In other words by relying on your phone as the storage mechanism for messages and not having to keep anything on their own servers, WhatsApp can run very cheaply and efficiently (and only cost you USD 1/year as a user). This makes WhatsApp basically nothing more than a fast routing service with some buffering between phones, much like the telcos (which might be what inspired WhatsApp to use Erlang). It's rather ingenious and probably why the service seems so fast and has such great uptime.

And I bet users don't care about the potential loss of messages either; when was the last time you scrolled back into the history on your phone to a point that preceded you getting that phone or having catastrophic data loss on the device? Plus you have to consider how many users of WhatsApp are really going to use the web app; I bet a large portion of their users don't even have their own computers beyond their phones so this web app service probably isn't critical to WhatsApp's success.

23 Jan 2015 5:27pm GMT

I love writing blog posts, and today I'm setting out to do something I've never tried before: write a blog post about writing blog posts. A big part of our mission at Caktus is to foster and help grow the Python and Django development communities, both locally and nationally. Part of how we've tried to accomplish this in the past is through hosting development sprints, sponsoring and attending conferences such as PyCon and DjangoCon, and building a knowledge base of common problems in Python and Django development in our blog. Many in the Django community first get to know Caktus through our blog, and it's both gratifying and humbling when I meet someone at a conference and the person thanks me for a post Caktus wrote that helped him or her solve a technical problem at some point in the past.

While I personally don't do as much software development as I used to and hence no longer write as many technical posts, the Caktus blog and many others in the community continue as a constant source of inspiration and education to me. As software developers we are constantly trying to work ourselves out of a job, building tools that organize information and help people communicate. Sharing a brief, highly specific technical blog post serves in a similar capacity; after I've spent 1-2 hours or more researching something that ultimately took 5-10 minutes to fix, I'd hate for someone else to have to go through the same experience. Writing up a quick, 1-2 paragraph technical post about the issue not only helps me think through the problem, but also hopefully saves a few minutes of someone else's life at some point in the future.

To help me better understand what I like so much about blogging, I went back and reviewed the history of Caktus blogging efforts over the past 5 years and separated our posts into categories. While I'm sure there are innumerable ways to do this, in case it serves as a source of inspiration to others, what follows are the categories I came up with:

• Technical Tidbits. These types of posts are small, usually a paragraph or two, along with a code or configuration snippet. They might cover upgrading a specific open source package or reusable app in your project, or augment existing Django release notes when you find the built-in Django documentation lacking for a specific use case. Posts in this category that we've written in the past at Caktus include upgrading django-photologue and changing or SHMMAX setting (for PostgreSQL) on a Mac. These are great posts to write after you've just done something for the first time. You'll have a fresher perspective than someone who's done the task many times before. Because of this, you can easily anticipate many of the common problems someone coming to the task for the first time might face.

• Debugging Sugar. Posts handy for debugging purposes often rely on a specific error message or stack trace. Another good candidate for this type of post is documenting an existing Django bug that requires a specific workaround. Posts we've written in this category include using strace to debug stuck celery tasks and the (thankfully now obsolete) parsing microseconds in the Django admin. A good sign you need to write a post like this is that you had to spend more than 5-10 minutes Googling for an answer to something or asking your co-workers. If you're looking for an answer and having trouble finding it, there's a good chance someone else out there is doing the same and would benefit from your blog post.

• Open Source Showcase. Open Source Showcase posts are a great way to spread the word about a project you have or a teammate has written, or to validate a 3rd party app or Django feature you've found particularly helpful. These are typically longer, more in-depth analyses of a project or feature rather than an answer to a specific technical problems (though the two are not always mutually exclusive). At Caktus we've written about our django-scribbler app as well as several new features in Django, including bulk inserts, class-based views, and support for custom user models. While these posts can require a significant time investment to get right, their value as augmentation to or 3rd-party validation of Python and Django development patterns cannot be underrated. Patterns are set through a community rallying around an open source package or approach. Proposing and sharing these ideas openly is what drives the open source community forward.

• Mini How-tos. Mini How-tos are generally a combination of other types of posts. They start with a specific goal in mind -- setting up a server, installing a reusable app -- and walk the reader through all the necessary steps, services, and packages required to get there. If you feel passionately that something should be done in a certain way, this is a great way to set a standard for the community to be aware of and potentially follow. This could cover anything from configuring a Jenkins slave to using Amazon S3 to store your static and uploaded media. Similar to an Open Source Showcase, Mini How-tos are an asset to the community insofar as they help advance and disseminate common approaches to software development problems. At the same time, they're open to review and critique by the wider open source community.

A big thank you to everyone in the Python and Django community for being open and willing to share your experiences and problem solving efforts. Without this, Caktus would not be where it is today and for that I am deeply grateful. If this post happens to inspire at least one short technical post from someone who hasn't written one before, I'll consider it a success.

23 Jan 2015 4:06pm GMT

I love writing blog posts, and today I'm setting out to do something I've never tried before: write a blog post about writing blog posts. A big part of our mission at Caktus is to foster and help grow the Python and Django development communities, both locally and nationally. Part of how we've tried to accomplish this in the past is through hosting development sprints, sponsoring and attending conferences such as PyCon and DjangoCon, and building a knowledge base of common problems in Python and Django development in our blog. Many in the Django community first get to know Caktus through our blog, and it's both gratifying and humbling when I meet someone at a conference and the person thanks me for a post Caktus wrote that helped him or her solve a technical problem at some point in the past.

While I personally don't do as much software development as I used to and hence no longer write as many technical posts, the Caktus blog and many others in the community continue as a constant source of inspiration and education to me. As software developers we are constantly trying to work ourselves out of a job, building tools that organize information and help people communicate. Sharing a brief, highly specific technical blog post serves in a similar capacity; after I've spent 1-2 hours or more researching something that ultimately took 5-10 minutes to fix, I'd hate for someone else to have to go through the same experience. Writing up a quick, 1-2 paragraph technical post about the issue not only helps me think through the problem, but also hopefully saves a few minutes of someone else's life at some point in the future.

To help me better understand what I like so much about blogging, I went back and reviewed the history of Caktus blogging efforts over the past 5 years and separated our posts into categories. While I'm sure there are innumerable ways to do this, in case it serves as a source of inspiration to others, what follows are the categories I came up with:

• Technical Tidbits. These types of posts are small, usually a paragraph or two, along with a code or configuration snippet. They might cover upgrading a specific open source package or reusable app in your project, or augment existing Django release notes when you find the built-in Django documentation lacking for a specific use case. Posts in this category that we've written in the past at Caktus include upgrading django-photologue and changing or SHMMAX setting (for PostgreSQL) on a Mac. These are great posts to write after you've just done something for the first time. You'll have a fresher perspective than someone who's done the task many times before. Because of this, you can easily anticipate many of the common problems someone coming to the task for the first time might face.

• Debugging Sugar. Posts handy for debugging purposes often rely on a specific error message or stack trace. Another good candidate for this type of post is documenting an existing Django bug that requires a specific workaround. Posts we've written in this category include using strace to debug stuck celery tasks and the (thankfully now obsolete) parsing microseconds in the Django admin. A good sign you need to write a post like this is that you had to spend more than 5-10 minutes Googling for an answer to something or asking your co-workers. If you're looking for an answer and having trouble finding it, there's a good chance someone else out there is doing the same and would benefit from your blog post.

• Open Source Showcase. Open Source Showcase posts are a great way to spread the word about a project you have or a teammate has written, or to validate a 3rd party app or Django feature you've found particularly helpful. These are typically longer, more in-depth analyses of a project or feature rather than an answer to a specific technical problems (though the two are not always mutually exclusive). At Caktus we've written about our django-scribbler app as well as several new features in Django, including bulk inserts, class-based views, and support for custom user models. While these posts can require a significant time investment to get right, their value as augmentation to or 3rd-party validation of Python and Django development patterns cannot be underrated. Patterns are set through a community rallying around an open source package or approach. Proposing and sharing these ideas openly is what drives the open source community forward.

• Mini How-tos. Mini How-tos are generally a combination of other types of posts. They start with a specific goal in mind -- setting up a server, installing a reusable app -- and walk the reader through all the necessary steps, services, and packages required to get there. If you feel passionately that something should be done in a certain way, this is a great way to set a standard for the community to be aware of and potentially follow. This could cover anything from configuring a Jenkins slave to using Amazon S3 to store your static and uploaded media. Similar to an Open Source Showcase, Mini How-tos are an asset to the community insofar as they help advance and disseminate common approaches to software development problems. At the same time, they're open to review and critique by the wider open source community.

A big thank you to everyone in the Python and Django community for being open and willing to share your experiences and problem solving efforts. Without this, Caktus would not be where it is today and for that I am deeply grateful. If this post happens to inspire at least one short technical post from someone who hasn't written one before, I'll consider it a success.

23 Jan 2015 4:06pm GMT

By Vasudev Ram

"PrettyTable to PDF is pretty easy with xtopdf."

How's that for some alliteration? :)

PrettyTable is a Python library to help you generate nice tables with ASCII characters as the borders, plus alignment of text within columns, headings, padding, etc.

Excerpt from the site:

[ PrettyTable is a simple Python library designed to make it quick and easy to represent tabular data in visually appealing ASCII tables.
...
PrettyTable lets you control many aspects of the table, like the width of the column padding, the alignment of text within columns, which characters are used to draw the table border, whether you even want a border, and much more. You can control which subsets of the columns and rows are printed, and you can sort the rows by the value of a particular column.

PrettyTable can also generate HTML code with the data in a <table> structure. ]

I came across PrettyTable via this blog post:

11 Python Libraries You Might Not Know.

Then I thought of using it with my PDF creation toolkit, to generate such ASCII tables, but as PDF. Here's a program, PrettyTableToPDF.py, that shows how to do that:

"""
PrettyTableToPDF.py
A demo program to show how to convert the output generated 
by the PrettyTable library, to PDF, using the xtopdf toolkit 
for PDF creation from other formats.
Author: Vasudev Ram - http://www.dancingbison.com
xtopdf is at: http://slides.com/vasudevram/xtopdf

Copyright 2015 Vasudev Ram
"""

from prettytable import PrettyTable
from PDFWriter import PDFWriter

pt = PrettyTable(["City name", "Area", "Population", "Annual Rainfall"])
pt.align["City name"] = "l" # Left align city names
pt.padding_width = 1 # One space between column edges and contents (default)
pt.add_row(["Adelaide",1295, 1158259, 600.5])
pt.add_row(["Brisbane",5905, 1857594, 1146.4])
pt.add_row(["Darwin", 112, 120900, 1714.7])
pt.add_row(["Hobart", 1357, 205556, 619.5])
pt.add_row(["Sydney", 2058, 4336374, 1214.8])
pt.add_row(["Melbourne", 1566, 3806092, 646.9])
pt.add_row(["Perth", 5386, 1554769, 869.4])
lines = pt.get_string()

pw = PDFWriter('Australia-Rainfall.pdf')
pw.setFont('Courier', 12)
pw.setHeader('Demo of PrettyTable to PDF')
pw.setFooter('Demo of PrettyTable to PDF')
for line in lines.split('\n'):
    pw.writeLine(line)
pw.close()

You can run the program with:

$ python PrettyTableToPDF.py

And here is a screenshot of the output PDF in Foxit PDF Reader:

- Enjoy.

--- Posts about Python --- Posts about xtopdf ---

- Vasudev Ram - Dancing Bison Enterprises - Python programming and trainingSignup to be informed about my new products or services.Contact Page

23 Jan 2015 8:52am GMT

By Vasudev Ram

"PrettyTable to PDF is pretty easy with xtopdf."

How's that for some alliteration? :)

PrettyTable is a Python library to help you generate nice tables with ASCII characters as the borders, plus alignment of text within columns, headings, padding, etc.

Excerpt from the site:

[ PrettyTable is a simple Python library designed to make it quick and easy to represent tabular data in visually appealing ASCII tables.
...
PrettyTable lets you control many aspects of the table, like the width of the column padding, the alignment of text within columns, which characters are used to draw the table border, whether you even want a border, and much more. You can control which subsets of the columns and rows are printed, and you can sort the rows by the value of a particular column.

PrettyTable can also generate HTML code with the data in a <table> structure. ]

I came across PrettyTable via this blog post:

11 Python Libraries You Might Not Know.

Then I thought of using it with my PDF creation toolkit, to generate such ASCII tables, but as PDF. Here's a program, PrettyTableToPDF.py, that shows how to do that:

"""
PrettyTableToPDF.py
A demo program to show how to convert the output generated 
by the PrettyTable library, to PDF, using the xtopdf toolkit 
for PDF creation from other formats.
Author: Vasudev Ram - http://www.dancingbison.com
xtopdf is at: http://slides.com/vasudevram/xtopdf

Copyright 2015 Vasudev Ram
"""

from prettytable import PrettyTable
from PDFWriter import PDFWriter

pt = PrettyTable(["City name", "Area", "Population", "Annual Rainfall"])
pt.align["City name"] = "l" # Left align city names
pt.padding_width = 1 # One space between column edges and contents (default)
pt.add_row(["Adelaide",1295, 1158259, 600.5])
pt.add_row(["Brisbane",5905, 1857594, 1146.4])
pt.add_row(["Darwin", 112, 120900, 1714.7])
pt.add_row(["Hobart", 1357, 205556, 619.5])
pt.add_row(["Sydney", 2058, 4336374, 1214.8])
pt.add_row(["Melbourne", 1566, 3806092, 646.9])
pt.add_row(["Perth", 5386, 1554769, 869.4])
lines = pt.get_string()

pw = PDFWriter('Australia-Rainfall.pdf')
pw.setFont('Courier', 12)
pw.setHeader('Demo of PrettyTable to PDF')
pw.setFooter('Demo of PrettyTable to PDF')
for line in lines.split('\n'):
    pw.writeLine(line)
pw.close()

You can run the program with:

$ python PrettyTableToPDF.py

And here is a screenshot of the output PDF in Foxit PDF Reader:

- Enjoy.

--- Posts about Python --- Posts about xtopdf ---

- Vasudev Ram - Dancing Bison Enterprises - Python programming and trainingSignup to be informed about my new products or services.Contact Page

23 Jan 2015 8:52am GMT

Disclaimer: I do not represent the Django Software Foundation in any way, nor has anything below been endorsed by the DSF. The following opinions are my own, unsolicited rambling.

If you hadn't been looking for it specifically, you may have missed it. The Django Softare Foundation is running a fundraising effort for the new Django Fellowship program. It sounds like they're still trying to figure out how to get the word out, so I wanted to do what I could to tell you why you should chip in.

This particular blog post is going to focus on encouraging (peer-pressuring) commercial Django users in particular, though enthusiasts are welcome to read along!

23 Jan 2015 7:21am GMT

Disclaimer: I do not represent the Django Software Foundation in any way, nor has anything below been endorsed by the DSF. The following opinions are my own, unsolicited rambling.

If you hadn't been looking for it specifically, you may have missed it. The Django Softare Foundation is running a fundraising effort for the new Django Fellowship program. It sounds like they're still trying to figure out how to get the word out, so I wanted to do what I could to tell you why you should chip in.

This particular blog post is going to focus on encouraging (peer-pressuring) commercial Django users in particular, though enthusiasts are welcome to read along!

23 Jan 2015 7:21am GMT

A recent HackerNews post announced Gotalk, a simple bidirectional protocol. I can imagine your collective eyeballs rolling. "Oh great, yet another half-baked way for… things to talk to one other". But keep following along, maybe you'll see something you like. Here are some highlights:

• By Rasmus Andersson - You may know him from his work at Facebook, Spotify, and Dropbox.
• Bidirectional - There's no discrimination on capabilities depending on who connected or who accepted. Both "servers" and "clients" can expose operations as well as send requests to the other side.
• Concurrent - Not too earthshattering, but this is a non-blocking, multiplexed protocol.
• Debuggable - ASCII-based wire format. You can, of course, encode your payloads any number of way, but you'll have an easy time with your packet sniffers and debug tools.
• Pretty damned simple - There's not a whole lot to Gotalk. A handful of message types. A pretty simple, easy-to-parse wire format. Nothing crazy for terminology or concepts. If you were curious about network protocols and wanted a gentle intro, this would be a great one to look at.

The official Gotalk repo has a bunch of examples and even some helper libraries in Go and JS. The official implementation is in Go, with the JS libraries being built on top of WebSockets.

23 Jan 2015 5:55am GMT

A recent HackerNews post announced Gotalk, a simple bidirectional protocol. I can imagine your collective eyeballs rolling. "Oh great, yet another half-baked way for… things to talk to one other". But keep following along, maybe you'll see something you like. Here are some highlights:

• By Rasmus Andersson - You may know him from his work at Facebook, Spotify, and Dropbox.
• Bidirectional - There's no discrimination on capabilities depending on who connected or who accepted. Both "servers" and "clients" can expose operations as well as send requests to the other side.
• Concurrent - Not too earthshattering, but this is a non-blocking, multiplexed protocol.
• Debuggable - ASCII-based wire format. You can, of course, encode your payloads any number of way, but you'll have an easy time with your packet sniffers and debug tools.
• Pretty damned simple - There's not a whole lot to Gotalk. A handful of message types. A pretty simple, easy-to-parse wire format. Nothing crazy for terminology or concepts. If you were curious about network protocols and wanted a gentle intro, this would be a great one to look at.

The official Gotalk repo has a bunch of examples and even some helper libraries in Go and JS. The official implementation is in Go, with the JS libraries being built on top of WebSockets.

23 Jan 2015 5:55am GMT

So there's a big flamey thread over on python-list about the Python 3.x type-annotation. Thing is, I can't say the annotations seem all *that* important to the behaviour (static type checking): Consider this Python 2.7 decorator-set:

def _func_annotation_for(func):
    """Retrieve the function annotation for a given function or create it"""
    current = getattr(func,'func_annotation',None)
    if current is None:
        current = func.func_annotation = {}
    return current 
def types( *types, **named ):
    def annotate(function):
        base = _func_annotation_for(function)
        base.update(named)
        if hasattr( function, 'func_code' ): # python 2.x
            names = function.func_code.co_varnames
        else:
            names = function.__code__.co_varnames
        
        for name, typ in zip(names, types):
            base[name] = typ
        return function 
    return annotate
def returns( typ ):
    def annotate(function):
        _func_annotation_for(function)['return'] = typ 
        return function 
    return annotate

That's a few lines of code and would seem to do the bulk of what type annotations do, namely populate a func_annotation dictionary and provide a textual clue as to where the type annotations are for AST or similar tools. It's true that it doesn't look as integrated as the syntax-based annotations, but to my eyes it actually looks cleaner when reading:

@types((int, float, complex), d=int, b=str)
@returns(float)
def x(a, b, c, d=3):
    return float(d)

The extra fluff (type annotations) is separated out and doesn't cloud the parameter list, you can use long type expressions to specify unions-of-types and the like without making it impossible to see your parameters. You can just annotate the functions/parameters you care about... etc. Compare the same example with inline syntactic annotations:

def x(a: (int,float,complex), b: str, c, d=3:int) -> float:
    return float(d)

Read that second example and ask yourself "what are the parameter names"... hmm, that took a bit to parse, now what happens when those type expressions grow long and convoluted? You get the nice locality of reference, but now you have type-itis all through your parameter lists.

Bonus points that you can do the decorator-based annotations in Python 2.7 and they should work fine in 3.x as well, while the inline static stuff is 3.x only AFAIK. The inline static stuff, however, does allow you to annotate nested parameters and wildcards (named/positional args). I don't know how often those are really going to get type annotations, but I wouldn't consider it a blocker until I saw people really stumbling on it... and then I'd likely tell people to write a couple of more lines of code to solve it...

Just for reference, PyOpenGL has a decorator that does type-annotations in order to get functions pulled out of DLLs via ctypes, that looks like this:

@_f
@_p.types(None,_cs.GLsync,_cs.GLenum,_cs.GLsizei,arrays.GLsizeiArray,arrays.GLintArray)
def glGetSynciv(sync,pname,bufSize,length,values):pass

Where @_f is the "turn it into a ctypes function" and @_p.types( returntype, argtypes ) is the type-annotation code that just adds the type-annotations to the function's dictionary. That works across Python versions, allows IDEs to provide argument names, is fairly concise, etc.

23 Jan 2015 4:09am GMT

So there's a big flamey thread over on python-list about the Python 3.x type-annotation. Thing is, I can't say the annotations seem all *that* important to the behaviour (static type checking): Consider this Python 2.7 decorator-set:

def _func_annotation_for(func):
    """Retrieve the function annotation for a given function or create it"""
    current = getattr(func,'func_annotation',None)
    if current is None:
        current = func.func_annotation = {}
    return current 
def types( *types, **named ):
    def annotate(function):
        base = _func_annotation_for(function)
        base.update(named)
        if hasattr( function, 'func_code' ): # python 2.x
            names = function.func_code.co_varnames
        else:
            names = function.__code__.co_varnames
        
        for name, typ in zip(names, types):
            base[name] = typ
        return function 
    return annotate
def returns( typ ):
    def annotate(function):
        _func_annotation_for(function)['return'] = typ 
        return function 
    return annotate

That's a few lines of code and would seem to do the bulk of what type annotations do, namely populate a func_annotation dictionary and provide a textual clue as to where the type annotations are for AST or similar tools. It's true that it doesn't look as integrated as the syntax-based annotations, but to my eyes it actually looks cleaner when reading:

@types((int, float, complex), d=int, b=str)
@returns(float)
def x(a, b, c, d=3):
    return float(d)

The extra fluff (type annotations) is separated out and doesn't cloud the parameter list, you can use long type expressions to specify unions-of-types and the like without making it impossible to see your parameters. You can just annotate the functions/parameters you care about... etc. Compare the same example with inline syntactic annotations:

def x(a: (int,float,complex), b: str, c, d=3:int) -> float:
    return float(d)

Read that second example and ask yourself "what are the parameter names"... hmm, that took a bit to parse, now what happens when those type expressions grow long and convoluted? You get the nice locality of reference, but now you have type-itis all through your parameter lists.

Bonus points that you can do the decorator-based annotations in Python 2.7 and they should work fine in 3.x as well, while the inline static stuff is 3.x only AFAIK. The inline static stuff, however, does allow you to annotate nested parameters and wildcards (named/positional args). I don't know how often those are really going to get type annotations, but I wouldn't consider it a blocker until I saw people really stumbling on it... and then I'd likely tell people to write a couple of more lines of code to solve it...

Just for reference, PyOpenGL has a decorator that does type-annotations in order to get functions pulled out of DLLs via ctypes, that looks like this:

@_f
@_p.types(None,_cs.GLsync,_cs.GLenum,_cs.GLsizei,arrays.GLsizeiArray,arrays.GLintArray)
def glGetSynciv(sync,pname,bufSize,length,values):pass

Where @_f is the "turn it into a ctypes function" and @_p.types( returntype, argtypes ) is the type-annotation code that just adds the type-annotations to the function's dictionary. That works across Python versions, allows IDEs to provide argument names, is fairly concise, etc.

23 Jan 2015 4:09am GMT

Over the last two years Caktus' design portfolio has rapidly been growing. We've taken on new projects primarily focused on design and have received community recognition for those efforts. We are happy to have grown our design capabilities to match the level of quality we demand from our Django developers. We have found it's important to have strength on both sides of the table as each side challenges the other and forces the final product of our process to be as high quality as possible.

In an effort to continue to push ourselves and expand our web design skill sets, Caktus is looking to hire a new Web Design Director. We're searching for someone who can do a bit of wireframing and user experience and then has the tools necessary to design and code pages. We're looking for someone who is attune to both form and function and knows where to focus depending on clients' needs. Caktus is committed to doing good in our development communities as well as through the projects that we choose to work on, so we are also interested in finding someone who is engaged in the design community.

If you or someone you know would be a good fit, please apply to the position! If you have any questions get in touch.

22 Jan 2015 1:00pm GMT

Over the last two years Caktus' design portfolio has rapidly been growing. We've taken on new projects primarily focused on design and have received community recognition for those efforts. We are happy to have grown our design capabilities to match the level of quality we demand from our Django developers. We have found it's important to have strength on both sides of the table as each side challenges the other and forces the final product of our process to be as high quality as possible.

In an effort to continue to push ourselves and expand our web design skill sets, Caktus is looking to hire a new Web Design Director. We're searching for someone who can do a bit of wireframing and user experience and then has the tools necessary to design and code pages. We're looking for someone who is attune to both form and function and knows where to focus depending on clients' needs. Caktus is committed to doing good in our development communities as well as through the projects that we choose to work on, so we are also interested in finding someone who is engaged in the design community.

If you or someone you know would be a good fit, please apply to the position! If you have any questions get in touch.

22 Jan 2015 1:00pm GMT

struct dict {
   long num_items;
   dict_entry* items;   /* pointer to array */
}

struct dict_entry {
   long hash;
   PyObject* key;
   PyObject* value;
}

struct dict {
    long num_items;
    variable_int *sparse_array;
    dict_entry* compact_array;
}

struct dict_entry {
    long hash;
    PyObject *key;
    PyObject *value;
}

22 Jan 2015 12:31pm GMT

struct dict {
   long num_items;
   dict_entry* items;   /* pointer to array */
}

struct dict_entry {
   long hash;
   PyObject* key;
   PyObject* value;
}

struct dict {
    long num_items;
    variable_int *sparse_array;
    dict_entry* compact_array;
}

struct dict_entry {
    long hash;
    PyObject *key;
    PyObject *value;
}

22 Jan 2015 12:31pm GMT

Unexpectedly and amazingly even the third €1200 goal was reached in my funding campaign! The Python community is fantastic! Thanks to everyone!

This is the last day of funding (it says 0 days left), so there is just hours to go (I'm not sure at exactly what time it closes). It's unlikely that the last goal of €2500 will be reached, but I can find ways to put any money over €1200 to good use. For example, I could print copies to give away at PyCon in Montreal. But what to do with that money depends a lot on how much it is.

https://www.fundedbyme.com/en/campaign/5065/supporting-python-3/

Filed under: book, python, python3

22 Jan 2015 8:15am GMT

Unexpectedly and amazingly even the third €1200 goal was reached in my funding campaign! The Python community is fantastic! Thanks to everyone!

This is the last day of funding (it says 0 days left), so there is just hours to go (I'm not sure at exactly what time it closes). It's unlikely that the last goal of €2500 will be reached, but I can find ways to put any money over €1200 to good use. For example, I could print copies to give away at PyCon in Montreal. But what to do with that money depends a lot on how much it is.

https://www.fundedbyme.com/en/campaign/5065/supporting-python-3/

Filed under: book, python, python3

22 Jan 2015 8:15am GMT

tl;dr Using Blaze in conjunction with Numba helps speed up code by reducing temporary variables from intermediate computations while allowing you to express computations at a high level.

22 Jan 2015 8:04am GMT

tl;dr Using Blaze in conjunction with Numba helps speed up code by reducing temporary variables from intermediate computations while allowing you to express computations at a high level.

22 Jan 2015 8:04am GMT

22 Jan 2015 5:57am GMT

22 Jan 2015 5:57am GMT

Wing IDE 5.1 adds multi-process and child process debugging, syntax highlighting in the shells, persistent time-stamped unit test results, auto-conversion of indents on paste, an XCode keyboard personality, support for Flask and Django 1.7, and many other minor features and improvements.

22 Jan 2015 1:00am GMT

Wing IDE 5.1 adds multi-process and child process debugging, syntax highlighting in the shells, persistent time-stamped unit test results, auto-conversion of indents on paste, an XCode keyboard personality, support for Flask and Django 1.7, and many other minor features and improvements.

22 Jan 2015 1:00am GMT

Location - Gordmans Office Building in Aksarben thanks to Aaron Keck.

Meeting starts at 6pm, Wednesday, 1/21/15

Parking and entry details:

The building is the northwest corner of 67th and Frances and the Gordmans entrance is next to the "g" sign, about midway along the East side of the building. There's parking directly out front, but it sometimes fills up in the evenings. The garage around back is open to the public after 5:30 or 6 as well.

The building doors lock at 5, so Aaron will be standing by to badge people in starting around 5:45. If you're running late, or early, just shoot him or the list an email.

21 Jan 2015 9:57pm GMT

Location - Gordmans Office Building in Aksarben thanks to Aaron Keck.

Meeting starts at 6pm, Wednesday, 1/21/15

Parking and entry details:

The building is the northwest corner of 67th and Frances and the Gordmans entrance is next to the "g" sign, about midway along the East side of the building. There's parking directly out front, but it sometimes fills up in the evenings. The garage around back is open to the public after 5:30 or 6 as well.

The building doors lock at 5, so Aaron will be standing by to badge people in starting around 5:45. If you're running late, or early, just shoot him or the list an email.

21 Jan 2015 9:57pm GMT

Gestern musste ich morgens zur Station nach KWT um unsere Rerservierten Bustickets für die Weihnachtsferien in Capetown abzuholen. Der Bahnhof selber ist seit Dezember aus kostengründen ohne Zugverbindung - aber Translux und co - die langdistanzbusse haben dort ihre Büros.


Größere Kartenansicht

© benste CC NC SA

10 Nov 2011 10:57am GMT

Niemand ist besorgt um so was - mit dem Auto fährt man einfach durch, und in der City - nahe Gnobie- "ne das ist erst gefährlich wenn die Feuerwehr da ist" - 30min später auf dem Rückweg war die Feuerwehr da.

© benste CC NC SA

09 Nov 2011 8:25pm GMT

Brai = Grillabend o.ä.

Die möchte gern Techniker beim Flicken ihrer SpeakOn / Klinke Stecker Verzweigungen...

Die Damen "Mamas" der Siedlung bei der offiziellen Eröffnungsrede

Auch wenn weniger Leute da waren als erwartet, Laute Musik und viele Leute ...

Und natürlich ein Feuer mit echtem Holz zum Grillen.

© benste CC NC SA

08 Nov 2011 2:30pm GMT

One of our missions was bringing Katja's Linux Server back to her room. While doing that we saw her new decoration.

Björn, Simphiwe carried the PC to Katja's school

© benste CC NC SA

07 Nov 2011 2:00pm GMT

Today I went with Björn to Needs Camp to Visit Katja's guest family for a special Party. First of all we visited some friends of Nelisa - yeah the one I'm working with in Quigney - Katja's guest fathers sister - who did her a haircut.

African Women usually get their hair done by arranging extensions and not like Europeans just cutting some hair.

In between she looked like this...

And then she was done - looks amazing considering the amount of hair she had last week - doesn't it ?

© benste CC NC SA

06 Nov 2011 7:45pm GMT

Irgendwie viel mir heute auf das ich meine Blogposts mal ein bischen umstrukturieren muss - wenn ich immer nur von neuen Plätzen berichte, dann müsste ich ja eine Rundreise machen. Hier also mal ein paar Sachen aus meinem heutigen Alltag.

Erst einmal vorweg, Samstag zählt zumindest für uns Voluntäre zu den freien Tagen.

Dieses Wochenende sind nur Rommel und ich auf der Farm - Katja und Björn sind ja mittlerweile in ihren Einsatzstellen, und meine Mitbewohner Kyle und Jonathan sind zu Hause in Grahamstown - sowie auch Sipho der in Dimbaza wohnt.
Robin, die Frau von Rommel ist in Woodie Cape - schon seit Donnerstag um da ein paar Sachen zur erledigen.
Naja wie dem auch sei heute morgen haben wir uns erstmal ein gemeinsames Weetbix/Müsli Frühstück gegönnt und haben uns dann auf den Weg nach East London gemacht. 2 Sachen waren auf der Checkliste Vodacom, Ethienne (Imobilienmakler) außerdem auf dem Rückweg die fehlenden Dinge nach NeedsCamp bringen.

Nachdem wir gerade auf der Dirtroad losgefahren sind mussten wir feststellen das wir die Sachen für Needscamp und Ethienne nicht eingepackt hatten aber die Pumpe für die Wasserversorgung im Auto hatten.

Also sind wir in EastLondon ersteinmal nach Farmerama - nein nicht das onlinespiel farmville - sondern einen Laden mit ganz vielen Sachen für eine Farm - in Berea einem nördlichen Stadteil gefahren.

In Farmerama haben wir uns dann beraten lassen für einen Schnellverschluss der uns das leben mit der Pumpe leichter machen soll und außerdem eine leichtere Pumpe zur Reperatur gebracht, damit es nicht immer so ein großer Aufwand ist, wenn mal wieder das Wasser ausgegangen ist.

Fego Caffé ist in der Hemmingways Mall, dort mussten wir und PIN und PUK einer unserer Datensimcards geben lassen, da bei der PIN Abfrage leider ein zahlendreher unterlaufen ist. Naja auf jeden Fall speichern die Shops in Südafrika so sensible Daten wie eine PUK - die im Prinzip zugang zu einem gesperrten Phone verschafft.

Im Cafe hat Rommel dann ein paar online Transaktionen mit dem 3G Modem durchgeführt, welches ja jetzt wieder funktionierte - und übrigens mittlerweile in Ubuntu meinem Linuxsystem perfekt klappt.

Nebenbei bin ich nach 8ta gegangen um dort etwas über deren neue Deals zu erfahren, da wir in einigen von Hilltops Centern Internet anbieten wollen. Das Bild zeigt die Abdeckung UMTS in NeedsCamp Katjas Ort. 8ta ist ein neuer Telefonanbieter von Telkom, nachdem Vodafone sich Telkoms anteile an Vodacom gekauft hat müssen die komplett neu aufbauen.
Wir haben uns dazu entschieden mal eine kostenlose Prepaidkarte zu testen zu organisieren, denn wer weis wie genau die Karte oben ist ... Bevor man einen noch so billigen Deal für 24 Monate signed sollte man wissen obs geht.

Danach gings nach Checkers in Vincent, gesucht wurden zwei Hotplates für WoodyCape - R 129.00 eine - also ca. 12€ für eine zweigeteilte Kochplatte.
Wie man sieht im Hintergrund gibts schon Weihnachtsdeko - Anfang November und das in Südafrika bei sonnig warmen min- 25°C

Mittagessen haben wir uns bei einem Pakistanischen Curry Imbiss gegönnt - sehr empfehlenswert !
Naja und nachdem wir dann vor ner Stunde oder so zurück gekommen sind habe ich noch den Kühlschrank geputzt den ich heute morgen zum defrosten einfach nach draußen gestellt hatte. Jetzt ist der auch mal wieder sauber und ohne 3m dicke Eisschicht...

Morgen ... ja darüber werde ich gesondert berichten ... aber vermutlich erst am Montag, denn dann bin ich nochmal wieder in Quigney(East London) und habe kostenloses Internet.

© benste CC NC SA

05 Nov 2011 4:33pm GMT

Sterkspruit is one of Hilltops Computer Centres in the far north of Eastern Cape. On the trip to J'burg we've used the opportunity to take a look at the centre.

Pupils in the big classroom

The Trainer

School in Countryside

Adult Class in the Afternoon

"Town"

© benste CC NC SA

31 Oct 2011 4:58pm GMT

What are you doing in an internet cafe if your ADSL and Faxline has been discontinued before months end. Well my idea was sitting outside and eating some ice cream.
At least it's sunny and not as rainy as on the weekend.

© benste CC NC SA

31 Oct 2011 3:11pm GMT

For those who are traveling through Zastron - there is a very nice Restaurant which is serving delicious food at reasanable prices.
In addition they're selling home made juices jams and honey.

interior

home made specialities - the shop in the shop

the Bar

© benste CC NC SA

30 Oct 2011 4:47pm GMT

Having the 10 - 12h trip from J'burg back to ELS I was able to take a lot of pcitures including these different roadsides

Plain Street

Orange River in its beginngings (near Lesotho)

Zastron Anglican Church

The Bridge in Between "Free State" and Eastern Cape next to Zastron

my new Background ;)

If you listen to GoogleMaps you'll end up traveling 50km of gravel road - as it was just renewed we didn't have that many problems and saved 1h compared to going the official way with all it's constructions sites

Freeway

getting dark

© benste CC NC SA

29 Oct 2011 4:23pm GMT

Klar einiges mag anders sein, vieles aber gleich - aber ein in Deutschland täglich übliches Bild einer Straßenbaustelle - wie läuft das eigentlich in Südafrika ?

Ersteinmal vorweg - NEIN keine Ureinwohner die mit den Händen graben - auch wenn hier mehr Manpower genutzt wird - sind sie fleißig mit Technologie am arbeiten.

Eine ganz normale "Bundesstraße"

und wie sie erweitert wird

gaaaanz viele LKWs

denn hier wird eine Seite über einen langen Abschnitt komplett gesperrt, so das eine Ampelschaltung mit hier 45 Minuten Wartezeit entsteht

Aber wenigstens scheinen die ihren Spaß zu haben ;) - Wie auch wir denn gücklicher Weise mussten wir nie länger als 10 min. warten.

© benste CC NC SA

28 Oct 2011 4:20pm GMT