27 Apr 2015

feedPlanet Python

Ludovic Gasc: TechEmpower FrameworkBenchmarks round 10: The Python results

TechEmpower FrameworkBenchmarks round 10 results are available.

What is TechEmpower FrameworkBenchmarks ?

As explained on their website:

This is a performance comparison of many web application frameworks executing fundamental tasks such as JSON serialization, database access, and server-side template composition. Each framework is operating in a realistic production configuration. Results are captured on Amazon EC2 and on physical hardware. The test implementations are largely community-contributed and all source is available at the GitHub repository.

You can read the technical description of each test on their website.

Goal of this blogpost

I'm concentrated on the results for Python ecosystem, and especially for AsyncIO+aiohttp.web+API-Hour, because it's the first time that an independent organization benchmarks publicly AsyncIO on several use cases with a macro-benchmark that implies a complete system stack (Network+Linux+Databases), closer to a production environment.

/!\ /!\ /!\ Warning - Warning - Warning /!\ /!\ /!\

As explained several times, do not follow any benchmarks results to immediately decide on the architecture of your applications, especially micro-benchmarks:

  1. You must understand a minimum what is tested to be sure you are in the same use case as the benchmark.
  2. Benchmark by yourself with your needs: It will help you to better understand the interactions between layers and maybe to discover that your bias about efficient architecture are false.
  3. Benchmark your complete stack: Sometimes, you can have bigger bottlenecks effects in your complete stack that in your Python code: you can optimize a big loop with Cython, if your database schema contains a lot of data and sucks, you'll wait more your database than in your loop.

The good results

Note: Click on the images to see directly the results on the FrameworkBenchmarks website.

IMPORTANT: FrameworkBenchmarks isn't only for Python, you can compare Web frameworks in other programming languages. Check the main website: https://www.techempower.com/benchmarks/

With no surprises, databases benchmarks are good for AsyncIO+aiohttp.web+API-Hour:

Data updates


Fortunes


Multiple queries



The average result

With less interactions with database, API-Hour results are in the average of others Web frameworks.

Single query


The bad results

The benchmarks without databases interactions are bad.

JSON serialization


Plaintext

Why theses benchmarks are less good that them provided on my blog ?

The difference of results between my DB benchmarks and DB benchmarks in FrameworkBenchmarks can be explained because, in my benchmarks, I use a bigger payload and a SQL query more complex that in FrameworkBenchmarks.
I did this because, as explained above, it is important to me to be as close as a production scenario when benchmarking. A realistic SQL query takes more time than a trivial SELECT, and the framework therefore has a serious advantage if it can manage several other things while waiting for the query result. Hence AsyncIO can better show its power there.
For round 11, TechEmpower should add a more realistic test, with a bigger payload and more DB interactions.

Conclusion

Nevertheless, for a first time participation, I'm really happy of the results. We have now one month to improve results for round 11, if you want to help, send me an e-mail.

My goal in theses benchmarks isn't only to find bottlenecks in the Python stack I'm using everyday, but also to make sure that this Python stack continues to be efficient over the time.
Most stacks become bloatwares over the time.

Future improvements

To increase a bit performance in the slowest test, the HTTP pipelining in aiohttp should be improved, but as it stands today, this might have positive effects in some cases (such as multiplexed HTTP requests), while bringing negative side effect in some other cases (such as old-fashioned HTTP requests or keep-alive requests).

But even with a better HTTP pipelining support, this test won't probably give very good results, as pure JSON provider.

I'll reorganize tests to include more changes like several event loops and more databases.

For the next round, several improvements are on the road:

  1. aiohttp new release has several speed improvements.
  2. An alternative event loop like aiouv might help, even if it isn't really the case for now, some improvements are necessary.
  3. A potential AsyncIO booster written in C is discussed on Python mailing-list
  4. AsyncIO support in the next Cypthon release
  5. During Python language submit 2015, several presentations had some parts to improve efficiency of Python:
    1. Making Python 3 more attractive
    2. Python 3 adoption
    3. Atomicity of operations
    4. PyParallel
  6. PyPy3: with a patched version of AsyncIO and Py3.3 branch of PyPy3, it's possible to execute some FrameworkBenchmarks tests. For now, PyPy3 is slower than CPython, nevertheless, you must understand that the goal of PyPy developers is to have a working version of PyPy3 that support Python 3.3, not to improve efficiency. You can donate to the PyPy3 project.

However, don't expect a magic change in the results for the next round, the efficiency problematic in Python ecosystem isn't new.
Nevertheless, to my knowledge, I've never seen as many solutions to boost code in other interpreted languages, except in Python, I hope that more solutions will emerge in the future.


Special thanks

Even if I was pleased to get some attention for these benchmarks, and even if sometimes I had some "viril exchanges"/ping-pong "fights" (another blogpost should be published) on mailing-lists or during PyCONs, I won't forget that theses results aren't possible without Python community.

Thanks a lot for everybody for your help, especially:

  1. Ukrainian (+Russian ?) Dream Team: Sorry to reduce you to your nationality/language, nevertheless, you're over-represented in AsyncIO community, especially in aiohttp ;-): aiohttp/aio-libs, they helped a lot to boost performances and I received numerous excellent advices from Andrew Svetlov, Nikolay Kim, Alexander Shorin, Alexey Popravka, Yury Selivanov and all others.
  2. Victor Stinner for AsyncIO improvements, help, and benchmarks tests.
  3. Antoine Pitrou for CPython optimizations like --with-computed-gotos and to have accepted AsyncIO PEP.
  4. Inada Naoki for all tips and challenged me in my benchmarks.
  5. Benoit Chesneau for his help to integrate Gunicorn in API-Hour.
  6. Stéphane Wirtel and Fabien Benetou to help me to promote Python in Belgium and for their support.
  7. All people I've forgotten in my list, especially AsyncIO libraries maintainers like Saúl Ibarra Corretgé (aiodns, aiouv), Jonathan Slenders (asyncio_redis), Gael Pasgrimaud (Panoramisk, irc3) Ron Frederick (asyncssh) and Tin Tvrtković (aiofiles, pytest-asyncio).
  8. Guido Van Rossum, because to maintain a community having the size of Python's ecosystem, political skills are as important as technical skills. And also to have implemented and pushed AsyncIO on the Python scene.
  9. My Tech team at ALLOcloud/Eyepea, especially Nicolas Stein and Sylvain Francis, to challenge me everyday and to be the right place for technical innovations in Europe: Almost no management, no "reserved area" in our Tech team (Everybody must have some skills of others: Dev team must have the hands in sysadmin infrastructure, Telco team must be capable to edit source code from Dev team) and the Tech team's trust in me: I've changed their toolbox for AsyncIO, and even if we are a small team, in retrospect, they were very open to change their habits and improve the toolbox, instead of blocking innovations just to stay in their comfort zone. A working environment like that is precious and should be more mainstream.
  10. Last by not least, my family for their unconditional support.


I'm proud to be an atom inside Python community ;-)

And don't forget

IT world has as bias that Python is slow because you (as member of the Python community) believe that it is slow: With absolute values on microbenchmarks, it might be true. But, with some tips & tricks, it can easily be more than fast enough for most production use cases in companies AND with Python, you keep the simplicity of coding.
Be honest with yourself: does it make sense to try to compare yourself with the very large player such as Google or Facebook ? Most IT companies are far from having the same constraints as them, and the same human resources to address them. The main thing everyone has in common with them is "time to market". Therefore, the simplicity of coding your business logic has much more weight in your case.
Developer speed is really more important than programs performance, but unfortunately, there are no benchmarks to measure this ;-).

27 Apr 2015 9:38pm GMT

Ludovic Gasc: TechEmpower FrameworkBenchmarks round 10: The Python results

TechEmpower FrameworkBenchmarks round 10 results are available.

What is TechEmpower FrameworkBenchmarks ?

As explained on their website:

This is a performance comparison of many web application frameworks executing fundamental tasks such as JSON serialization, database access, and server-side template composition. Each framework is operating in a realistic production configuration. Results are captured on Amazon EC2 and on physical hardware. The test implementations are largely community-contributed and all source is available at the GitHub repository.

You can read the technical description of each test on their website.

Goal of this blogpost

I'm concentrated on the results for Python ecosystem, and especially for AsyncIO+aiohttp.web+API-Hour, because it's the first time that an independent organization benchmarks publicly AsyncIO on several use cases with a macro-benchmark that implies a complete system stack (Network+Linux+Databases), closer to a production environment.

/!\ /!\ /!\ Warning - Warning - Warning /!\ /!\ /!\

As explained several times, do not follow any benchmarks results to immediately decide on the architecture of your applications, especially micro-benchmarks:

  1. You must understand a minimum what is tested to be sure you are in the same use case as the benchmark.
  2. Benchmark by yourself with your needs: It will help you to better understand the interactions between layers and maybe to discover that your bias about efficient architecture are false.
  3. Benchmark your complete stack: Sometimes, you can have bigger bottlenecks effects in your complete stack that in your Python code: you can optimize a big loop with Cython, if your database schema contains a lot of data and sucks, you'll wait more your database than in your loop.

The good results

Note: Click on the images to see directly the results on the FrameworkBenchmarks website.

IMPORTANT: FrameworkBenchmarks isn't only for Python, you can compare Web frameworks in other programming languages. Check the main website: https://www.techempower.com/benchmarks/

With no surprises, databases benchmarks are good for AsyncIO+aiohttp.web+API-Hour:

Data updates


Fortunes


Multiple queries



The average result

With less interactions with database, API-Hour results are in the average of others Web frameworks.

Single query


The bad results

The benchmarks without databases interactions are bad.

JSON serialization


Plaintext

Why theses benchmarks are less good that them provided on my blog ?

The difference of results between my DB benchmarks and DB benchmarks in FrameworkBenchmarks can be explained because, in my benchmarks, I use a bigger payload and a SQL query more complex that in FrameworkBenchmarks.
I did this because, as explained above, it is important to me to be as close as a production scenario when benchmarking. A realistic SQL query takes more time than a trivial SELECT, and the framework therefore has a serious advantage if it can manage several other things while waiting for the query result. Hence AsyncIO can better show its power there.
For round 11, TechEmpower should add a more realistic test, with a bigger payload and more DB interactions.

Conclusion

Nevertheless, for a first time participation, I'm really happy of the results. We have now one month to improve results for round 11, if you want to help, send me an e-mail.

My goal in theses benchmarks isn't only to find bottlenecks in the Python stack I'm using everyday, but also to make sure that this Python stack continues to be efficient over the time.
Most stacks become bloatwares over the time.

Future improvements

To increase a bit performance in the slowest test, the HTTP pipelining in aiohttp should be improved, but as it stands today, this might have positive effects in some cases (such as multiplexed HTTP requests), while bringing negative side effect in some other cases (such as old-fashioned HTTP requests or keep-alive requests).

But even with a better HTTP pipelining support, this test won't probably give very good results, as pure JSON provider.

I'll reorganize tests to include more changes like several event loops and more databases.

For the next round, several improvements are on the road:

  1. aiohttp new release has several speed improvements.
  2. An alternative event loop like aiouv might help, even if it isn't really the case for now, some improvements are necessary.
  3. A potential AsyncIO booster written in C is discussed on Python mailing-list
  4. AsyncIO support in the next Cypthon release
  5. During Python language submit 2015, several presentations had some parts to improve efficiency of Python:
    1. Making Python 3 more attractive
    2. Python 3 adoption
    3. Atomicity of operations
    4. PyParallel
  6. PyPy3: with a patched version of AsyncIO and Py3.3 branch of PyPy3, it's possible to execute some FrameworkBenchmarks tests. For now, PyPy3 is slower than CPython, nevertheless, you must understand that the goal of PyPy developers is to have a working version of PyPy3 that support Python 3.3, not to improve efficiency. You can donate to the PyPy3 project.

However, don't expect a magic change in the results for the next round, the efficiency problematic in Python ecosystem isn't new.
Nevertheless, to my knowledge, I've never seen as many solutions to boost code in other interpreted languages, except in Python, I hope that more solutions will emerge in the future.


Special thanks

Even if I was pleased to get some attention for these benchmarks, and even if sometimes I had some "viril exchanges"/ping-pong "fights" (another blogpost should be published) on mailing-lists or during PyCONs, I won't forget that theses results aren't possible without Python community.

Thanks a lot for everybody for your help, especially:

  1. Ukrainian (+Russian ?) Dream Team: Sorry to reduce you to your nationality/language, nevertheless, you're over-represented in AsyncIO community, especially in aiohttp ;-): aiohttp/aio-libs, they helped a lot to boost performances and I received numerous excellent advices from Andrew Svetlov, Nikolay Kim, Alexander Shorin, Alexey Popravka, Yury Selivanov and all others.
  2. Victor Stinner for AsyncIO improvements, help, and benchmarks tests.
  3. Antoine Pitrou for CPython optimizations like --with-computed-gotos and to have accepted AsyncIO PEP.
  4. Inada Naoki for all tips and challenged me in my benchmarks.
  5. Benoit Chesneau for his help to integrate Gunicorn in API-Hour.
  6. Stéphane Wirtel and Fabien Benetou to help me to promote Python in Belgium and for their support.
  7. All people I've forgotten in my list, especially AsyncIO libraries maintainers like Saúl Ibarra Corretgé (aiodns, aiouv), Jonathan Slenders (asyncio_redis), Gael Pasgrimaud (Panoramisk, irc3) Ron Frederick (asyncssh) and Tin Tvrtković (aiofiles, pytest-asyncio).
  8. Guido Van Rossum, because to maintain a community having the size of Python's ecosystem, political skills are as important as technical skills. And also to have implemented and pushed AsyncIO on the Python scene.
  9. My Tech team at ALLOcloud/Eyepea, especially Nicolas Stein and Sylvain Francis, to challenge me everyday and to be the right place for technical innovations in Europe: Almost no management, no "reserved area" in our Tech team (Everybody must have some skills of others: Dev team must have the hands in sysadmin infrastructure, Telco team must be capable to edit source code from Dev team) and the Tech team's trust in me: I've changed their toolbox for AsyncIO, and even if we are a small team, in retrospect, they were very open to change their habits and improve the toolbox, instead of blocking innovations just to stay in their comfort zone. A working environment like that is precious and should be more mainstream.
  10. Last by not least, my family for their unconditional support.


I'm proud to be an atom inside Python community ;-)

And don't forget

IT world has as bias that Python is slow because you (as member of the Python community) believe that it is slow: With absolute values on microbenchmarks, it might be true. But, with some tips & tricks, it can easily be more than fast enough for most production use cases in companies AND with Python, you keep the simplicity of coding.
Be honest with yourself: does it make sense to try to compare yourself with the very large player such as Google or Facebook ? Most IT companies are far from having the same constraints as them, and the same human resources to address them. The main thing everyone has in common with them is "time to market". Therefore, the simplicity of coding your business logic has much more weight in your case.
Developer speed is really more important than programs performance, but unfortunately, there are no benchmarks to measure this ;-).

27 Apr 2015 9:38pm GMT

Robert Collins: Dealing with deps in OpenStack

We've got a problem in OpenStack.. dependency management.

In this post I explore it as input to the design summit session on this in Vancouver.

Goals

We have some goals that are broadly agreed:

  1. Guarantee co-installability of a single release of OpenStack
  2. Be able to deliver known-good installs of OpenStack at any point in time - e.g. 'this is known to work'
  3. Deliver good, clear dependency metadata to redistributors
  4. Support CD deployments of OpenStack from git. Both production and devstack for developers to hack on/with
  5. Avoid firedrills in CI - both internal situations where we run incompatible things we produced, and external situations where some dependency releases a broken version, like the pycparsing one last week
  6. Deployments using the Python dependencies should be up to date and secure
  7. Support doing upgrades in the same Python environment

Assumptions

And we have some baseline assumptions:

  1. We cooperate with the Python ecosystem - publishing our libraries to PyPI for instance
  2. Every commit of server projects is a 'release' from the perspective of e.g. schema management
  3. Other things release when they release, not per-commit

The current approach uses a single global list of acceptable install-requires for all our projects, and then merges that into the git trees being tested during the test. Note in particular that this doesn't take place for things not being tested, which we install from PyPI. We create a branch of that global list for each stable release, and we also create branches of nearly everything when we do the stable release, a system that has evolved in part due to the issues in CI when new releases would break stable releases. These new branches have tightly defined constraints - e.g. "DEP >= version-at-this-release < next-point-release"'. The idea behind this is that if the transitive closure of deps is constrained, we can install from PyPI such a version, and it won't bring in a different version. One of the reasons we needed that was PIP bug 988, where pip takes the first occurrence of a dependency, and so servers would depend on oslo.utils which would depend on an unversioned cliff or some such, and if cliff wasn't already installed we'd get the next releases cliff. Now - semver says we're keeping those things compatible, but mistakes happen, and for stable branches there's really little reason to upgrade.

Issues

We have some practical issues with the current system:

  1. Just one dependency uncapped anywhere in the wider ecosystem (including packages outside of OpenStack) that depends on a dependency that we wanted to stay unchanged, and if that dep is encountered first by the pip scanner… game over. Worse, there are components out there that introspect the installed dependencies and fail hard if one is not listed as compatible, which takes a 'testing with unexpected version' situation and makes it a hard error
  2. We have to run stable branches for everything, even things like OpenStackClient which are intended for end users, and are aimed at a semver rather than branched release model
  3. Due to PIP bug 2687 each time we call pip may introduce the skew that breaks the gate
  4. We don't deliver goal 1:- because we override the requirements at test time, the actual co-installability may be different, and we don't know
  5. We deliver goal 2 but its hard to use:- you have to dig through a specific CI log, and if the CI system has pruned it, you're toast
  6. We don't avoid external firedrills:- because most of our external dependencies are broad, external releases break us trivially and frequently
  7. Lastly, our requirements are too tight to support upgrades: if bug 2687 was fixed, installing the first upgraded server component would error because its requirements are declared as being incompatible with the last release.

We do deliver goals 3,4 and 6 though, which is good.

So what can we do differently? In an ideal world, can we get all 6 goals?

Proposal

I think we can. Here's one way it could work:

  1. We fix the two pip bugs above (I'm working on that now)
  2. We teach pip about constraints *if* something is requested without actually requesting it
  3. We change our project overrides in CI to use a single constraints file rather than merging into each projects requirements
  4. The single constraints file would be exactly specified: "DEP == VERSION", not semver or compatible matched.
  5. We make changes to the single constraints file by running a proposed set of constraints
  6. We find out that we should change the constraints file by having a periodic task which compares the constraints file to the published versions on PyPI and proposes changes to the constraints repository automatically
  7. We loosen up the constraints in all our release branches to permit upgrade co-installability

And some optional bits…

  1. We could start testing new-library old-servers again
  2. We could potentially change our branching strategy for non-server components, but I don't think it harms things - it may just be unnecessary
  3. We could add periodic jobs for testing with unreleased versions of dependencies

Working through each point. Bug 988 causes compatible requirements to be ignored - if we have one constraint of "X > 1.4″ and another of "X > 1.3, !=1.5.1″ but the "> 1.4″ constraint is encountered first, we can end up with 1.5.1 installed, violating a known-bad constraint. Fixing this means that rather than having to have global knowledge of deps at the point where pip is being entered, we can have local knowledge about compatible versions in each package, and as long as the union of requirements is satisfiable, we'll be ok. Bug 2687 causes the constraints that thing A had when it was installed by pip be ignored by the requirements checking for thing B. For instance, pip install python-openstackclient after pip install nova, will meet python-openstackclient's requirements even if that means breaking nova's requirements.

The reason we can't just use a requirements file today, is that a requirements file specifies what needs to be installed as well as what versions are acceptable. We don't want devstack, when configured for nova-network, to install neutron dependencies. But it would today unless we put in place a bunch of complex processing logic. Whereas pip could do this very easily internally.

Merging each requirement into things we're installing from git fails when we install releases - e.g. of client libraries, in particular because of the interactions with bug 988 above. A single constraints file could include all known good versions of everything we might use, and would apply globally in concert with local project requirements. Best of both worlds, in theory :)

The use of inexact versions is a hard limitation today - we can't upgrade multiple project trees local version needs atomically, and because we're supplying all the version constraints in one place - the project's merged install_requirements - they have to be broad enough to co-exist during changes to the requirements, and to remain co-installed during upgrades from release to release of OpenStack. But inexact versions leads to variation in CI - every single run becomes a gamble. The primary goal of CI is to tell us whether a new commit X meets all of our quality criteria - change one thing at a time. Running with every new version of every dependency doesn't tell us more about X, it tells us about ecosystem things. Using exact constraints will solve this: we'll decouple 'update dependencies' or 'pycparsing Y is broken' from testing X - e.g. 'improve nova cells'.

We need to be able to update those dependencies though, and the existing global requirements mechanisms are pretty much right, they just need to work with a constraints file instead of patching each repo at test time. We will still want to check that the local requirements are compatible with the global constraints file.

One of the big holes such approaches have is that we may miss out on important improvements - security, performance or just plain old features - if we don't update our constraints. So we need to be on top of that. A small amount of automation can give us a lot of assistance on that. Just try the new versions and if they work - great. If they don't, show a failing proposal where we can assess what to do.

As I mentioned earlier today we can't actually upgrade: kilo's version locks exclude liberty versions of our libraries, meaning that trying to upgrade nova/kilo to nova/liberty will bring in library versions that conflict with the version deps neutron expresses. We need to open up the project local requirements to avoid this - and we also need to make some guarantees about compatibility with our prior release in our library development (otherwise rebooting a server with only one component upgraded will be a gamble).

Making those guarantees will either require testing every commit against the prior server, or if we can find some way of doing it, testing proposed releases against the prior servers - which would allow more latitude during development of our libraries. The use of constraints files will give us hermetic insulation against bad releases though - we'll be able to stay productive while we fix the issue and issue a new better release. The crucial thing is to have a tight feedback loop though - so I'm in favour of us either testing each commit against last-stable, or figuring out the 'tests before releases' logic (perhaps by removing direct tag access and instead having a thing we propose the intent to as a review).

All this might be enough that we choose to make less stable branches of libraries and go back to plain semver - but its not a requirement: thats something we can discuss in detail if people care, or just wait and see what the overheads and benefits of keeping those branches are.

Lastly, this new structure will make it possible, if we want to, to test that unreleased versions of external dependencies work with a given component, by using a periodic job. Why periodic? There are two sides to each dependencies, and neither side would want their gate to wedge if an accident breaks the other side. E.g. using two of our own components - oslo.messaging and nova. oslo.messaging releases must not break nova, but an individual oslo.messaging commit isn't necessarily constrained (if we have the before-release testing described above). External dependencies are exactly the same, except even less closely aligned than intra-OpenStack components. So running tests with a git version of e.g. libvirt in a periodic job might give us (and libvirt) valuable prior warning about issues.


27 Apr 2015 9:35pm GMT

Robert Collins: Dealing with deps in OpenStack

We've got a problem in OpenStack.. dependency management.

In this post I explore it as input to the design summit session on this in Vancouver.

Goals

We have some goals that are broadly agreed:

  1. Guarantee co-installability of a single release of OpenStack
  2. Be able to deliver known-good installs of OpenStack at any point in time - e.g. 'this is known to work'
  3. Deliver good, clear dependency metadata to redistributors
  4. Support CD deployments of OpenStack from git. Both production and devstack for developers to hack on/with
  5. Avoid firedrills in CI - both internal situations where we run incompatible things we produced, and external situations where some dependency releases a broken version, like the pycparsing one last week
  6. Deployments using the Python dependencies should be up to date and secure
  7. Support doing upgrades in the same Python environment

Assumptions

And we have some baseline assumptions:

  1. We cooperate with the Python ecosystem - publishing our libraries to PyPI for instance
  2. Every commit of server projects is a 'release' from the perspective of e.g. schema management
  3. Other things release when they release, not per-commit

The current approach uses a single global list of acceptable install-requires for all our projects, and then merges that into the git trees being tested during the test. Note in particular that this doesn't take place for things not being tested, which we install from PyPI. We create a branch of that global list for each stable release, and we also create branches of nearly everything when we do the stable release, a system that has evolved in part due to the issues in CI when new releases would break stable releases. These new branches have tightly defined constraints - e.g. "DEP >= version-at-this-release < next-point-release"'. The idea behind this is that if the transitive closure of deps is constrained, we can install from PyPI such a version, and it won't bring in a different version. One of the reasons we needed that was PIP bug 988, where pip takes the first occurrence of a dependency, and so servers would depend on oslo.utils which would depend on an unversioned cliff or some such, and if cliff wasn't already installed we'd get the next releases cliff. Now - semver says we're keeping those things compatible, but mistakes happen, and for stable branches there's really little reason to upgrade.

Issues

We have some practical issues with the current system:

  1. Just one dependency uncapped anywhere in the wider ecosystem (including packages outside of OpenStack) that depends on a dependency that we wanted to stay unchanged, and if that dep is encountered first by the pip scanner… game over. Worse, there are components out there that introspect the installed dependencies and fail hard if one is not listed as compatible, which takes a 'testing with unexpected version' situation and makes it a hard error
  2. We have to run stable branches for everything, even things like OpenStackClient which are intended for end users, and are aimed at a semver rather than branched release model
  3. Due to PIP bug 2687 each time we call pip may introduce the skew that breaks the gate
  4. We don't deliver goal 1:- because we override the requirements at test time, the actual co-installability may be different, and we don't know
  5. We deliver goal 2 but its hard to use:- you have to dig through a specific CI log, and if the CI system has pruned it, you're toast
  6. We don't avoid external firedrills:- because most of our external dependencies are broad, external releases break us trivially and frequently
  7. Lastly, our requirements are too tight to support upgrades: if bug 2687 was fixed, installing the first upgraded server component would error because its requirements are declared as being incompatible with the last release.

We do deliver goals 3,4 and 6 though, which is good.

So what can we do differently? In an ideal world, can we get all 6 goals?

Proposal

I think we can. Here's one way it could work:

  1. We fix the two pip bugs above (I'm working on that now)
  2. We teach pip about constraints *if* something is requested without actually requesting it
  3. We change our project overrides in CI to use a single constraints file rather than merging into each projects requirements
  4. The single constraints file would be exactly specified: "DEP == VERSION", not semver or compatible matched.
  5. We make changes to the single constraints file by running a proposed set of constraints
  6. We find out that we should change the constraints file by having a periodic task which compares the constraints file to the published versions on PyPI and proposes changes to the constraints repository automatically
  7. We loosen up the constraints in all our release branches to permit upgrade co-installability

And some optional bits…

  1. We could start testing new-library old-servers again
  2. We could potentially change our branching strategy for non-server components, but I don't think it harms things - it may just be unnecessary
  3. We could add periodic jobs for testing with unreleased versions of dependencies

Working through each point. Bug 988 causes compatible requirements to be ignored - if we have one constraint of "X > 1.4″ and another of "X > 1.3, !=1.5.1″ but the "> 1.4″ constraint is encountered first, we can end up with 1.5.1 installed, violating a known-bad constraint. Fixing this means that rather than having to have global knowledge of deps at the point where pip is being entered, we can have local knowledge about compatible versions in each package, and as long as the union of requirements is satisfiable, we'll be ok. Bug 2687 causes the constraints that thing A had when it was installed by pip be ignored by the requirements checking for thing B. For instance, pip install python-openstackclient after pip install nova, will meet python-openstackclient's requirements even if that means breaking nova's requirements.

The reason we can't just use a requirements file today, is that a requirements file specifies what needs to be installed as well as what versions are acceptable. We don't want devstack, when configured for nova-network, to install neutron dependencies. But it would today unless we put in place a bunch of complex processing logic. Whereas pip could do this very easily internally.

Merging each requirement into things we're installing from git fails when we install releases - e.g. of client libraries, in particular because of the interactions with bug 988 above. A single constraints file could include all known good versions of everything we might use, and would apply globally in concert with local project requirements. Best of both worlds, in theory :)

The use of inexact versions is a hard limitation today - we can't upgrade multiple project trees local version needs atomically, and because we're supplying all the version constraints in one place - the project's merged install_requirements - they have to be broad enough to co-exist during changes to the requirements, and to remain co-installed during upgrades from release to release of OpenStack. But inexact versions leads to variation in CI - every single run becomes a gamble. The primary goal of CI is to tell us whether a new commit X meets all of our quality criteria - change one thing at a time. Running with every new version of every dependency doesn't tell us more about X, it tells us about ecosystem things. Using exact constraints will solve this: we'll decouple 'update dependencies' or 'pycparsing Y is broken' from testing X - e.g. 'improve nova cells'.

We need to be able to update those dependencies though, and the existing global requirements mechanisms are pretty much right, they just need to work with a constraints file instead of patching each repo at test time. We will still want to check that the local requirements are compatible with the global constraints file.

One of the big holes such approaches have is that we may miss out on important improvements - security, performance or just plain old features - if we don't update our constraints. So we need to be on top of that. A small amount of automation can give us a lot of assistance on that. Just try the new versions and if they work - great. If they don't, show a failing proposal where we can assess what to do.

As I mentioned earlier today we can't actually upgrade: kilo's version locks exclude liberty versions of our libraries, meaning that trying to upgrade nova/kilo to nova/liberty will bring in library versions that conflict with the version deps neutron expresses. We need to open up the project local requirements to avoid this - and we also need to make some guarantees about compatibility with our prior release in our library development (otherwise rebooting a server with only one component upgraded will be a gamble).

Making those guarantees will either require testing every commit against the prior server, or if we can find some way of doing it, testing proposed releases against the prior servers - which would allow more latitude during development of our libraries. The use of constraints files will give us hermetic insulation against bad releases though - we'll be able to stay productive while we fix the issue and issue a new better release. The crucial thing is to have a tight feedback loop though - so I'm in favour of us either testing each commit against last-stable, or figuring out the 'tests before releases' logic (perhaps by removing direct tag access and instead having a thing we propose the intent to as a review).

All this might be enough that we choose to make less stable branches of libraries and go back to plain semver - but its not a requirement: thats something we can discuss in detail if people care, or just wait and see what the overheads and benefits of keeping those branches are.

Lastly, this new structure will make it possible, if we want to, to test that unreleased versions of external dependencies work with a given component, by using a periodic job. Why periodic? There are two sides to each dependencies, and neither side would want their gate to wedge if an accident breaks the other side. E.g. using two of our own components - oslo.messaging and nova. oslo.messaging releases must not break nova, but an individual oslo.messaging commit isn't necessarily constrained (if we have the before-release testing described above). External dependencies are exactly the same, except even less closely aligned than intra-OpenStack components. So running tests with a git version of e.g. libvirt in a periodic job might give us (and libvirt) valuable prior warning about issues.


27 Apr 2015 9:35pm GMT

Al-Ahmadgaid Asaad: Parametric Inference: Likelihood Ratio Test by Example

Hypothesis testing have been extensively used on different discipline of science. And in this post, I will attempt on discussing the basic theory behind this, the Likelihood Ratio Test (LRT) defined below from Casella and Berger (2001), see reference 1.

Definition. The likelihood ratio test statistic for testing $H_0:\theta\in\Theta_0$ versus $H_1:\theta\in\Theta_0^c$ is \begin{equation} \label{eq:lrt} \lambda(\mathbf{x})=\frac{\displaystyle\sup_{\theta\in\Theta_0}L(\theta|\mathbf{x})}{\displaystyle\sup_{\theta\in\Theta}L(\theta|\mathbf{x})}. \end{equation} A likelihood ratio test (LRT) is any test that has a rejection region of the form $\{\mathbf{x}:\lambda(\mathbf{x})\leq c\}$, where $c$ is any number satisfying $0\leq c \leq 1$.

The numerator of equation (\ref{eq:lrt}) gives us the supremum probability of the parameter, $\theta$, over the restricted domain (null hypothesis, $\Theta_0$) of the parameter space $\Theta$, that maximizes the joint probability of the sample, $\mathbf{x}$. While the denominator of the LRT gives us the supremum probability of the parameter, $\theta$, over the unrestricted domain, $\Theta$, that maximizes the joint probability of the sample, $\mathbf{x}$. Therefore, if the value of $\lambda(\mathbf{x})$ is small such that $\lambda(\mathbf{x})\leq c$, for some $c\in [0, 1]$, then the true value of the parameter that is plausible in explaining the sample is likely to be in the alternative hypothesis, $\Theta_0^c$.

Example 1. Let $X_1,X_2,\cdots,X_n\overset{r.s.}{\sim}f(x|\theta)=\frac{1}{\theta}\exp\left[-\frac{x}{\theta}\right],x>0,\theta>0$. From this sample, consider testing $H_0:\theta = \theta_0$ vs $H_1:\theta\theta_0$.

Solution:
The parameter space $\Theta$ is the set $(0,\Theta_0]$, where $\Theta_0=\{\theta_0\}$. Hence, using the likelihood ratio test, we have $$ \lambda(\mathbf{x})=\frac{\displaystyle\sup_{\theta=\theta_0}L(\theta|\mathbf{x})}{\displaystyle\sup_{\theta\leq\theta_0}L(\theta|\mathbf{x})}, $$ where, $$ \begin{aligned} \sup_{\theta=\theta_0}L(\theta|\mathbf{x})&=\sup_{\theta=\theta_0}\prod_{i=1}^{n}\frac{1}{\theta}\exp\left[-\frac{x_i}{\theta}\right]\\ &=\sup_{\theta=\theta_0}\left(\frac{1}{\theta}\right)^n\exp\left[-\displaystyle\frac{\sum_{i=1}^{n}x_i}{\theta}\right]\\ &=\left(\frac{1}{\theta_0}\right)^n\exp\left[-\displaystyle\frac{\sum_{i=1}^{n}x_i}{\theta_0}\right], \end{aligned} $$ and $$ \begin{aligned} \sup_{\theta\leq\theta_0}L(\theta|\mathbf{x})&=\sup_{\theta\leq\theta_0}\prod_{i=1}^{n}\frac{1}{\theta}\exp\left[-\frac{x_i}{\theta}\right]\\ &=\sup_{\theta\leq\theta_0}\left(\frac{1}{\theta}\right)^n\exp\left[-\displaystyle\frac{\sum_{i=1}^{n}x_i}{\theta}\right]=\sup_{\theta\leq\theta_0}f(\mathbf{x}|\theta). \end{aligned} $$ Now the supremum of $f(\mathbf{x}|\theta)$ over all values of $\theta\leq\theta_0$ is the MLE (maximum likelihood estimator) of $f(x|\theta)$, which is $\bar{x}$, provided that $\bar{x}\leq \theta_0$.

So that, $$ \begin{aligned} \lambda(\mathbf{x})&=\frac{\left(\frac{1}{\theta_0}\right)^n\exp\left[-\displaystyle\frac{\sum_{i=1}^{n}x_i}{\theta_0}\right]} {\left(\frac{1}{\bar{x}}\right)^n\exp\left[-\displaystyle\frac{\sum_{i=1}^{n}x_i}{\bar{x}}\right]},\quad\text{provided that}\;\bar{x}\leq \theta_0\\ &=\left(\frac{\bar{x}}{\theta_0}\right)^n\exp\left[-\displaystyle\frac{\sum_{i=1}^{n}x_i}{\theta_0}\right]\exp[n]. \end{aligned} $$ And we say that, if $\lambda(\mathbf{x})\leq c$, $H_0$ is rejected. That is, $$ \begin{aligned} \left(\frac{\bar{x}}{\theta_0}\right)^n\exp\left[-\displaystyle\frac{\sum_{i=1}^{n}x_i}{\theta_0}\right]\exp[n]&\leq c\\ \left(\frac{\bar{x}}{\theta_0}\right)^n\exp\left[-\displaystyle\frac{\sum_{i=1}^{n}x_i}{\theta_0}\right]&\leq c',\quad\text{where}\;c'=\frac{c}{\exp[n]}\\ n\log\left(\frac{\bar{x}}{\theta_0}\right)-\frac{n}{\theta_0}\bar{x}&\leq \log c'\\ \log\left(\frac{\bar{x}}{\theta_0}\right)-\frac{\bar{x}}{\theta_0}&\leq \frac{1}{n}\log c'\\ \log\left(\frac{\bar{x}}{\theta_0}\right)-\frac{\bar{x}}{\theta_0}&\leq \frac{1}{n}\log c-1. \end{aligned} $$ Now let $h(x)=\log x - x$, then $h'(x)=\frac{1}{x}-1$. So the critical point of $h'(x)$ is $x=1$. And to test if this is maximum or minimum, we apply second derivative test. That is, $$ h''(x)=-\frac{1}{x^2}0,\forall x. $$ Thus, $x=1$ is a maximum. Hence, $ \log\left(\frac{\bar{x}}{\theta_0}\right)-\frac{\bar{x}}{\theta_0} $ is maximized if $\frac{\bar{x}}{\theta_0}=1\Rightarrow\bar{x}=\theta_0$. To see this consider the following plot,

LRT and its Critical Value

Above figure is the plot of $h(\bar{x})$ function with $\theta_0=1$. Given the assumption that $\bar{x}\leq \theta_0$ then assuming $R=\frac{1}{n}\log c-1$ designates the orange line above, then we reject $H_0$ if $h(\bar{x})\leq R$, if and only if $\bar{x}\leq k$. In practice, $k$ is specified to satisfy, $$ \mathrm{P}(\bar{x}\leq k|\theta=\theta_0)\leq \alpha, $$ where $\alpha$ is called the level of the test.

It follows that $X_i|\theta = \theta_0\overset{r.s.}{\sim}\exp[\theta_0]$, then $\mathrm{E}X_i=\theta_0$ and $\mathrm{Var}X_i=\theta_0^2$. If $\bar{x}=\frac{1}{n}\sum_{i=1}^{n}X_i$ and if $G_n$ is the distribution of $\frac{(\bar{x}_n-\theta_0)}{\sqrt{\frac{\theta_0^2}{n}}}$. By CLT (central limit theorem) $\lim_{n\to\infty}G_n$ converges to standard normal distribution. That is, $\bar{x}|\theta = \theta_0\overset{r.s.}{\sim}AN\left(\theta_0,\frac{\theta_0^2}{n}\right)$. $AN$ - assymptotically normal.

Thus, $$ \mathrm{P}(\bar{x}\leq k|\theta=\theta_0)=\Phi\left(\frac{k-\theta_0}{\theta_0/\sqrt{n}}\right),\quad\text{for large }n. $$ So that, $$ \mathrm{P}(\bar{x}\leq k|\theta=\theta_0)=\Phi\left(\frac{k-\theta_0}{\theta_0/\sqrt{n}}\right)\leq \alpha. $$ Plotting this gives us,

CDF

with corresponding PDF given by,

Density Function

Implying, $$ \frac{k-\theta_0}{\theta_0/\sqrt{n}}=z_{\alpha}\Rightarrow k=\theta_0+z_{\alpha}\frac{\theta_0}{\sqrt{n}}. $$ Therefore, a level-$\alpha$ test of $H_0:\theta=\theta_0$ vs $H_1:\theta\theta_0$ is the test that rejects $H_0$ when $\bar{x}\leq\theta_0+z_{\alpha}\frac{\theta_0}{\sqrt{n}}$.

Plot's Python Codes

In case you might ask how above plots were generated:

Reference

  1. Casella, G. and Berger, R.L. (2001). Statistical Inference. Thomson Learning, Inc.
  2. Plotly Python Library Documentation

27 Apr 2015 8:44pm GMT

Al-Ahmadgaid Asaad: Parametric Inference: Likelihood Ratio Test by Example

Hypothesis testing have been extensively used on different discipline of science. And in this post, I will attempt on discussing the basic theory behind this, the Likelihood Ratio Test (LRT) defined below from Casella and Berger (2001), see reference 1.

Definition. The likelihood ratio test statistic for testing $H_0:\theta\in\Theta_0$ versus $H_1:\theta\in\Theta_0^c$ is \begin{equation} \label{eq:lrt} \lambda(\mathbf{x})=\frac{\displaystyle\sup_{\theta\in\Theta_0}L(\theta|\mathbf{x})}{\displaystyle\sup_{\theta\in\Theta}L(\theta|\mathbf{x})}. \end{equation} A likelihood ratio test (LRT) is any test that has a rejection region of the form $\{\mathbf{x}:\lambda(\mathbf{x})\leq c\}$, where $c$ is any number satisfying $0\leq c \leq 1$.

The numerator of equation (\ref{eq:lrt}) gives us the supremum probability of the parameter, $\theta$, over the restricted domain (null hypothesis, $\Theta_0$) of the parameter space $\Theta$, that maximizes the joint probability of the sample, $\mathbf{x}$. While the denominator of the LRT gives us the supremum probability of the parameter, $\theta$, over the unrestricted domain, $\Theta$, that maximizes the joint probability of the sample, $\mathbf{x}$. Therefore, if the value of $\lambda(\mathbf{x})$ is small such that $\lambda(\mathbf{x})\leq c$, for some $c\in [0, 1]$, then the true value of the parameter that is plausible in explaining the sample is likely to be in the alternative hypothesis, $\Theta_0^c$.

Example 1. Let $X_1,X_2,\cdots,X_n\overset{r.s.}{\sim}f(x|\theta)=\frac{1}{\theta}\exp\left[-\frac{x}{\theta}\right],x>0,\theta>0$. From this sample, consider testing $H_0:\theta = \theta_0$ vs $H_1:\theta\theta_0$.

Solution:
The parameter space $\Theta$ is the set $(0,\Theta_0]$, where $\Theta_0=\{\theta_0\}$. Hence, using the likelihood ratio test, we have $$ \lambda(\mathbf{x})=\frac{\displaystyle\sup_{\theta=\theta_0}L(\theta|\mathbf{x})}{\displaystyle\sup_{\theta\leq\theta_0}L(\theta|\mathbf{x})}, $$ where, $$ \begin{aligned} \sup_{\theta=\theta_0}L(\theta|\mathbf{x})&=\sup_{\theta=\theta_0}\prod_{i=1}^{n}\frac{1}{\theta}\exp\left[-\frac{x_i}{\theta}\right]\\ &=\sup_{\theta=\theta_0}\left(\frac{1}{\theta}\right)^n\exp\left[-\displaystyle\frac{\sum_{i=1}^{n}x_i}{\theta}\right]\\ &=\left(\frac{1}{\theta_0}\right)^n\exp\left[-\displaystyle\frac{\sum_{i=1}^{n}x_i}{\theta_0}\right], \end{aligned} $$ and $$ \begin{aligned} \sup_{\theta\leq\theta_0}L(\theta|\mathbf{x})&=\sup_{\theta\leq\theta_0}\prod_{i=1}^{n}\frac{1}{\theta}\exp\left[-\frac{x_i}{\theta}\right]\\ &=\sup_{\theta\leq\theta_0}\left(\frac{1}{\theta}\right)^n\exp\left[-\displaystyle\frac{\sum_{i=1}^{n}x_i}{\theta}\right]=\sup_{\theta\leq\theta_0}f(\mathbf{x}|\theta). \end{aligned} $$ Now the supremum of $f(\mathbf{x}|\theta)$ over all values of $\theta\leq\theta_0$ is the MLE (maximum likelihood estimator) of $f(x|\theta)$, which is $\bar{x}$, provided that $\bar{x}\leq \theta_0$.

So that, $$ \begin{aligned} \lambda(\mathbf{x})&=\frac{\left(\frac{1}{\theta_0}\right)^n\exp\left[-\displaystyle\frac{\sum_{i=1}^{n}x_i}{\theta_0}\right]} {\left(\frac{1}{\bar{x}}\right)^n\exp\left[-\displaystyle\frac{\sum_{i=1}^{n}x_i}{\bar{x}}\right]},\quad\text{provided that}\;\bar{x}\leq \theta_0\\ &=\left(\frac{\bar{x}}{\theta_0}\right)^n\exp\left[-\displaystyle\frac{\sum_{i=1}^{n}x_i}{\theta_0}\right]\exp[n]. \end{aligned} $$ And we say that, if $\lambda(\mathbf{x})\leq c$, $H_0$ is rejected. That is, $$ \begin{aligned} \left(\frac{\bar{x}}{\theta_0}\right)^n\exp\left[-\displaystyle\frac{\sum_{i=1}^{n}x_i}{\theta_0}\right]\exp[n]&\leq c\\ \left(\frac{\bar{x}}{\theta_0}\right)^n\exp\left[-\displaystyle\frac{\sum_{i=1}^{n}x_i}{\theta_0}\right]&\leq c',\quad\text{where}\;c'=\frac{c}{\exp[n]}\\ n\log\left(\frac{\bar{x}}{\theta_0}\right)-\frac{n}{\theta_0}\bar{x}&\leq \log c'\\ \log\left(\frac{\bar{x}}{\theta_0}\right)-\frac{\bar{x}}{\theta_0}&\leq \frac{1}{n}\log c'\\ \log\left(\frac{\bar{x}}{\theta_0}\right)-\frac{\bar{x}}{\theta_0}&\leq \frac{1}{n}\log c-1. \end{aligned} $$ Now let $h(x)=\log x - x$, then $h'(x)=\frac{1}{x}-1$. So the critical point of $h'(x)$ is $x=1$. And to test if this is maximum or minimum, we apply second derivative test. That is, $$ h''(x)=-\frac{1}{x^2}0,\forall x. $$ Thus, $x=1$ is a maximum. Hence, $ \log\left(\frac{\bar{x}}{\theta_0}\right)-\frac{\bar{x}}{\theta_0} $ is maximized if $\frac{\bar{x}}{\theta_0}=1\Rightarrow\bar{x}=\theta_0$. To see this consider the following plot,

LRT and its Critical Value

Above figure is the plot of $h(\bar{x})$ function with $\theta_0=1$. Given the assumption that $\bar{x}\leq \theta_0$ then assuming $R=\frac{1}{n}\log c-1$ designates the orange line above, then we reject $H_0$ if $h(\bar{x})\leq R$, if and only if $\bar{x}\leq k$. In practice, $k$ is specified to satisfy, $$ \mathrm{P}(\bar{x}\leq k|\theta=\theta_0)\leq \alpha, $$ where $\alpha$ is called the level of the test.

It follows that $X_i|\theta = \theta_0\overset{r.s.}{\sim}\exp[\theta_0]$, then $\mathrm{E}X_i=\theta_0$ and $\mathrm{Var}X_i=\theta_0^2$. If $\bar{x}=\frac{1}{n}\sum_{i=1}^{n}X_i$ and if $G_n$ is the distribution of $\frac{(\bar{x}_n-\theta_0)}{\sqrt{\frac{\theta_0^2}{n}}}$. By CLT (central limit theorem) $\lim_{n\to\infty}G_n$ converges to standard normal distribution. That is, $\bar{x}|\theta = \theta_0\overset{r.s.}{\sim}AN\left(\theta_0,\frac{\theta_0^2}{n}\right)$. $AN$ - assymptotically normal.

Thus, $$ \mathrm{P}(\bar{x}\leq k|\theta=\theta_0)=\Phi\left(\frac{k-\theta_0}{\theta_0/\sqrt{n}}\right),\quad\text{for large }n. $$ So that, $$ \mathrm{P}(\bar{x}\leq k|\theta=\theta_0)=\Phi\left(\frac{k-\theta_0}{\theta_0/\sqrt{n}}\right)\leq \alpha. $$ Plotting this gives us,

CDF

with corresponding PDF given by,

Density Function

Implying, $$ \frac{k-\theta_0}{\theta_0/\sqrt{n}}=z_{\alpha}\Rightarrow k=\theta_0+z_{\alpha}\frac{\theta_0}{\sqrt{n}}. $$ Therefore, a level-$\alpha$ test of $H_0:\theta=\theta_0$ vs $H_1:\theta\theta_0$ is the test that rejects $H_0$ when $\bar{x}\leq\theta_0+z_{\alpha}\frac{\theta_0}{\sqrt{n}}$.

Plot's Python Codes

In case you might ask how above plots were generated:

Reference

  1. Casella, G. and Berger, R.L. (2001). Statistical Inference. Thomson Learning, Inc.
  2. Plotly Python Library Documentation

27 Apr 2015 8:44pm GMT

PyCon PL Conference: PyCon PL 2015 conference website will launch soon!

This year's conference website will appear a bit earlier than usual - in 2nd half of May. The conference itself will happen on 15th-18th of October. More information coming soon.

27 Apr 2015 8:31pm GMT

PyCon PL Conference: PyCon PL 2015 conference website will launch soon!

This year's conference website will appear a bit earlier than usual - in 2nd half of May. The conference itself will happen on 15th-18th of October. More information coming soon.

27 Apr 2015 8:31pm GMT

Caktus Consulting Group: Why did Caktus Group start Astro Code School?

Our Astro Code School is now officially accepting applications to its twelve-week Python & Django Web Development class for intermediate programmers! To kick off Astro's opening, we asked Caktus' CTO and co-founder Colin Copeland, who recently won a 2015 Triangle Business Journal 40 Under 40 Leadership Award, and Astro's Director Brian Russell to reflect on the development of Astro as well as the role they see the school playing in the Django community.


Why open the East Coast's first Django and Python-focused code school?

Colin: Technology is an important part of economic growth in the Triangle area and we wanted to make sure those opportunities reached as many residents as possible. We saw that there were no East Coast formal adult training programs for Django or Python, our specialities. We have experience in this area, having hosted successful Django boot camps and private corporate trainings. Opening a code school was a way to consolidate the training side of Caktus' business while also giving back to the Triangle-area community by creating a training center to help those looking to learn new skills.

Brian: Ultimately, Caktus noticed a need for developers and the lack of a central place to train them. The web framework Django is written in Python and Python is a great language for beginning coders. Python is the top learning language for the nation's best universities.Those are the skills prominent here at Caktus. It was an opportunity to train more people and prepare them for the growing technology industry at firms like Caktus.

How has demand for Django-based web applications changed since Caktus first began?

Colin: It has increased significantly. We only do Django development now, we weren't specialized in that way when we first started. The sheer number of inbound sales requests is much higher than before. More people are aware of Django, conferences are bigger. Most significantly, it has an ever-growing reputation as a more professional, stable, and maintainable framework than other languages.

How does Astro, then, fit into this growth timeline?

Colin: It's a pretty simple supply and demand ratio. Astro comes out of a desire to add more developers to the field and meet a growing demand for Django coders. The Bureau of Labor Statistics projects a 20% growth in demand for web developers by 2020. It is not practical to wait for today's college, high school, or even middle-school students to become developers. Many great software developers are adults coming from second or third careers. Our staff certainly reflects this truth. Astro provides one means for talented adults to move into the growing technology industry.

Where do you see Astro fitting in to the local Python and Django community? For instance, how do you envision Astro's relationship to some of the groups Caktus maintains a strong relationship with, such as Girl Develop It or TriPython?

Colin: Astro's goals clearly align with those of Girl Develop It in terms of training and support. And the space will be a great place to host events for local groups and classes.

Brian: Yeah, I see it as a very natural fit. We hope to help those organizations by sponsoring meetups, hosting events, and providing free community programs and workshops. And there is the obvious hope that folks from those groups will enroll as students at Astro. I think it's also important to note that Chris Calloway, one of the lead organizers for TriPython, is a member of the Astro advisory committee. There is a natural friendship with that community.

How do you hope Astro will change and add to Durham's existing technical community?

Brian: In general there are a lot students with training from Astro who will be able to bring their skills to local businesses, schools, non-profits-all sorts of organizations. For me, computer programming is like reading, writing, and arithmetic: it should be a part of core curriculum for students these days. It helps people improve their own net worth and contribute to the local economy. Astro is all about workforce development and improving technical literacy: two things that help entrepreneurs and entrepreneurial enterprises.

What are some of the main goals for Astro in its first year?

Brian: I want to help people find better, higher paying jobs by obtaining skills that are usable in the current economy through our 12-week classes. I'm personally interested in social economic justice and one way to achieve that is by being highly skilled. Training helps people better themselves no matter what kind of education it is. In the 21st century, computer programming education is one of the most powerful tools for job preparedness and improvement.

Colin: I would love to follow alumni who make it through the classes and see how their skills help them in their careers.

A huge amount of work has gone into getting Astro licensed with the North Carolina Community College Board. A lot of code schools are not licensed. Why was this an important step for Astro?

Brian: Mainly because we wanted to demonstrate to potential students and the public at large that we've done our due diligence, that other groups and professionals have vetted us and qualified us as prepared to serve. Ultimately we are licensed in order to protect consumers. Not just licensed-we're bonded, licensed, and insured. And this is an ongoing guarantee to our students. We will be audited annually for six years. I see it as a promise for continuous and ongoing protection, betterment, and improvement.

So, who would you describe as the ideal student for an Astro course?

Brian: A lot of students. Any. All different kinds. But, more specifically? I would recommend it to folks changing their career. Or people who graduated from high school, but for one reason or another are not able to go onto higher education. Astro classes will be excellent for job preparedness and training so anyone looking to market themselves in the current economy.

Additionally, anyone fine tuning their career after college or even after grad school. Coding and learning to code is an excellent way to earn money to pay for school without getting into debt. Astro is in no way a replacement for higher ed, but coding classes can augment a well-rounded education. Successful people have a diverse education. And learning to code enables people to align their toolkits for the modern job market.


To learn more about Astro, meet Colin and Brian in person, and celebrate the opening of Astro Code School, be sure to stop by the school's Launch Party on Friday, May 1st from 6:00 to 9:00 pm. Registration is required.

27 Apr 2015 7:55pm GMT

Caktus Consulting Group: Why did Caktus Group start Astro Code School?

Our Astro Code School is now officially accepting applications to its twelve-week Python & Django Web Development class for intermediate programmers! To kick off Astro's opening, we asked Caktus' CTO and co-founder Colin Copeland, who recently won a 2015 Triangle Business Journal 40 Under 40 Leadership Award, and Astro's Director Brian Russell to reflect on the development of Astro as well as the role they see the school playing in the Django community.


Why open the East Coast's first Django and Python-focused code school?

Colin: Technology is an important part of economic growth in the Triangle area and we wanted to make sure those opportunities reached as many residents as possible. We saw that there were no East Coast formal adult training programs for Django or Python, our specialities. We have experience in this area, having hosted successful Django boot camps and private corporate trainings. Opening a code school was a way to consolidate the training side of Caktus' business while also giving back to the Triangle-area community by creating a training center to help those looking to learn new skills.

Brian: Ultimately, Caktus noticed a need for developers and the lack of a central place to train them. The web framework Django is written in Python and Python is a great language for beginning coders. Python is the top learning language for the nation's best universities.Those are the skills prominent here at Caktus. It was an opportunity to train more people and prepare them for the growing technology industry at firms like Caktus.

How has demand for Django-based web applications changed since Caktus first began?

Colin: It has increased significantly. We only do Django development now, we weren't specialized in that way when we first started. The sheer number of inbound sales requests is much higher than before. More people are aware of Django, conferences are bigger. Most significantly, it has an ever-growing reputation as a more professional, stable, and maintainable framework than other languages.

How does Astro, then, fit into this growth timeline?

Colin: It's a pretty simple supply and demand ratio. Astro comes out of a desire to add more developers to the field and meet a growing demand for Django coders. The Bureau of Labor Statistics projects a 20% growth in demand for web developers by 2020. It is not practical to wait for today's college, high school, or even middle-school students to become developers. Many great software developers are adults coming from second or third careers. Our staff certainly reflects this truth. Astro provides one means for talented adults to move into the growing technology industry.

Where do you see Astro fitting in to the local Python and Django community? For instance, how do you envision Astro's relationship to some of the groups Caktus maintains a strong relationship with, such as Girl Develop It or TriPython?

Colin: Astro's goals clearly align with those of Girl Develop It in terms of training and support. And the space will be a great place to host events for local groups and classes.

Brian: Yeah, I see it as a very natural fit. We hope to help those organizations by sponsoring meetups, hosting events, and providing free community programs and workshops. And there is the obvious hope that folks from those groups will enroll as students at Astro. I think it's also important to note that Chris Calloway, one of the lead organizers for TriPython, is a member of the Astro advisory committee. There is a natural friendship with that community.

How do you hope Astro will change and add to Durham's existing technical community?

Brian: In general there are a lot students with training from Astro who will be able to bring their skills to local businesses, schools, non-profits-all sorts of organizations. For me, computer programming is like reading, writing, and arithmetic: it should be a part of core curriculum for students these days. It helps people improve their own net worth and contribute to the local economy. Astro is all about workforce development and improving technical literacy: two things that help entrepreneurs and entrepreneurial enterprises.

What are some of the main goals for Astro in its first year?

Brian: I want to help people find better, higher paying jobs by obtaining skills that are usable in the current economy through our 12-week classes. I'm personally interested in social economic justice and one way to achieve that is by being highly skilled. Training helps people better themselves no matter what kind of education it is. In the 21st century, computer programming education is one of the most powerful tools for job preparedness and improvement.

Colin: I would love to follow alumni who make it through the classes and see how their skills help them in their careers.

A huge amount of work has gone into getting Astro licensed with the North Carolina Community College Board. A lot of code schools are not licensed. Why was this an important step for Astro?

Brian: Mainly because we wanted to demonstrate to potential students and the public at large that we've done our due diligence, that other groups and professionals have vetted us and qualified us as prepared to serve. Ultimately we are licensed in order to protect consumers. Not just licensed-we're bonded, licensed, and insured. And this is an ongoing guarantee to our students. We will be audited annually for six years. I see it as a promise for continuous and ongoing protection, betterment, and improvement.

So, who would you describe as the ideal student for an Astro course?

Brian: A lot of students. Any. All different kinds. But, more specifically? I would recommend it to folks changing their career. Or people who graduated from high school, but for one reason or another are not able to go onto higher education. Astro classes will be excellent for job preparedness and training so anyone looking to market themselves in the current economy.

Additionally, anyone fine tuning their career after college or even after grad school. Coding and learning to code is an excellent way to earn money to pay for school without getting into debt. Astro is in no way a replacement for higher ed, but coding classes can augment a well-rounded education. Successful people have a diverse education. And learning to code enables people to align their toolkits for the modern job market.


To learn more about Astro, meet Colin and Brian in person, and celebrate the opening of Astro Code School, be sure to stop by the school's Launch Party on Friday, May 1st from 6:00 to 9:00 pm. Registration is required.

27 Apr 2015 7:55pm GMT

Reuven Lerner: Free one-hour Webinar about Python’s magic methods on May 6th

I'll be giving another free one-hour Webinar about Python - this time, about the magic methods that Python offers developers who want to change the ways in which their objects work.

We'll start off with some of the simplest magic methods, but will quickly move onto some of the more interesting and advanced ones - affecting the way that our objects are compared, formatted, hashed, pickled, and sauteed. (OK, maybe not sauteed.) Some familiarity with Python objects is expected, but not too much advanced knowledge is necessary.

Register now at EventBrite; if you have any questions, please contact me at reuven@lerner.co.il, or as @reuvenmlerner on Twitter. It should be a lot of fun; I hope to see you there!

The post Free one-hour Webinar about Python's magic methods on May 6th appeared first on Lerner Consulting Blog.

27 Apr 2015 1:26pm GMT

Reuven Lerner: Free one-hour Webinar about Python’s magic methods on May 6th

I'll be giving another free one-hour Webinar about Python - this time, about the magic methods that Python offers developers who want to change the ways in which their objects work.

We'll start off with some of the simplest magic methods, but will quickly move onto some of the more interesting and advanced ones - affecting the way that our objects are compared, formatted, hashed, pickled, and sauteed. (OK, maybe not sauteed.) Some familiarity with Python objects is expected, but not too much advanced knowledge is necessary.

Register now at EventBrite; if you have any questions, please contact me at reuven@lerner.co.il, or as @reuvenmlerner on Twitter. It should be a lot of fun; I hope to see you there!

The post Free one-hour Webinar about Python's magic methods on May 6th appeared first on Lerner Consulting Blog.

27 Apr 2015 1:26pm GMT

Mike Driscoll: PyDev of the Week: Adina Howe

This week we welcome Adina Howe as our PyDev of the week. I first heard about Mrs. Howe from an article on Nature.com about how Python is doing well in scientific circles and is worth learning. Currently she is a Biosystem Engineering professor at Iowa State University. Let's spend some time getting to know more about her!

howe_swc_carpentry

Can you tell us a little about yourself (hobbies, education, etc):

I am a new professor at Iowa State University in the Department of Agricultural and Biosystems Engineering. There, my team and I research the impacts of microorganisms on various environments ranging from soils, waters, to even our guts. Besides research, I really enjoy ultimate frisbee and am trying out for a roller derby team.

Why did you start using Python?

I started using Python because the datasets I was working with were too big to open in Excel and Word. In my research, I had also become tired of doing the same thing over and over again. I became very interested in reproducible and automated approaches to perform my analyses. When I started writing code in Python, I found that I'd have small enough "wins" every day that I was convinced to keep using it - and that continues even today.

What other programming languages do you know and which is your favorite?

My transcript says I should know C++, but I don't remember any of it (that was more than 15 years ago). I regularly use R and shell scripts.

What projects are you working on now?

Mainly, I use programming for bioinformatic analysis, mainly data wrangling and munging. I also integrate different types of datasets together, for example, to study water quality I integrate biological, chemical, and land use datasets.

Which Python libraries are your favorite (core or 3rd party)?

My favorite would be IPython (Notebook). I also use NumPy and SciPy frequently, and also matplotlib for visualization. I've been using Requests lately as I am working with more automated queries to large databases. Someone introduced me to BeautifulSoup the other day - it seemed quite nifty to parse html.

Is there anything else you'd like to say?

I think Python is fantastic. The community of users and developers is terrific - in my experience, insanely productive, helpful, and unselfish. So I'd like to thank you and those who participate for being so welcoming and let you know that I am using your tools to try to make positive impact with my work.

Thank you for your time!

27 Apr 2015 12:30pm GMT

Mike Driscoll: PyDev of the Week: Adina Howe

This week we welcome Adina Howe as our PyDev of the week. I first heard about Mrs. Howe from an article on Nature.com about how Python is doing well in scientific circles and is worth learning. Currently she is a Biosystem Engineering professor at Iowa State University. Let's spend some time getting to know more about her!

howe_swc_carpentry

Can you tell us a little about yourself (hobbies, education, etc):

I am a new professor at Iowa State University in the Department of Agricultural and Biosystems Engineering. There, my team and I research the impacts of microorganisms on various environments ranging from soils, waters, to even our guts. Besides research, I really enjoy ultimate frisbee and am trying out for a roller derby team.

Why did you start using Python?

I started using Python because the datasets I was working with were too big to open in Excel and Word. In my research, I had also become tired of doing the same thing over and over again. I became very interested in reproducible and automated approaches to perform my analyses. When I started writing code in Python, I found that I'd have small enough "wins" every day that I was convinced to keep using it - and that continues even today.

What other programming languages do you know and which is your favorite?

My transcript says I should know C++, but I don't remember any of it (that was more than 15 years ago). I regularly use R and shell scripts.

What projects are you working on now?

Mainly, I use programming for bioinformatic analysis, mainly data wrangling and munging. I also integrate different types of datasets together, for example, to study water quality I integrate biological, chemical, and land use datasets.

Which Python libraries are your favorite (core or 3rd party)?

My favorite would be IPython (Notebook). I also use NumPy and SciPy frequently, and also matplotlib for visualization. I've been using Requests lately as I am working with more automated queries to large databases. Someone introduced me to BeautifulSoup the other day - it seemed quite nifty to parse html.

Is there anything else you'd like to say?

I think Python is fantastic. The community of users and developers is terrific - in my experience, insanely productive, helpful, and unselfish. So I'd like to thank you and those who participate for being so welcoming and let you know that I am using your tools to try to make positive impact with my work.

Thank you for your time!

27 Apr 2015 12:30pm GMT

Tennessee Leeuwenburg: Interlude: Kaggle Scripting

I don't like to support specific companies, in general I'm more interested in how people can be productive as independently and autonomously as possible.

However, Kaggle just did something pretty amazing.

They've launched a free online scripting environment for machine learning, with access to their competition data sets. That means that everything I've been writing about so far -- loading data, setting up your environment, using notebooks, can be easily sidestepped for instant results. The best thing -- it supports Python!

I haven't yet determined where the limitations are, I've simply forked one of the sample scripts that I found, but it worked wonderfully. I found that the execution time of the script was similar to running it locally on my 2011 Macbook Air, but that's not bad for something totally free that runs on someone else's infrastructure. I was able to sidestep data downloading entirely, and "leap into the middle" my forking an existing script.

Here's a direct link to one:

http://www.kaggle.com/users/320654/cs822-am/otto-group-product-classification-challenge/benchmark

Note, I'm logged into Kaggle, so I'm not sure how this will go for people who aren't signed in.

My expectation is that for solutions which demand a lot of computational resources (or rely on the GPU for reasonable performance) or use a unusual libraries, a self-service approach to computing will still win out, but this is an amazing way to get started.

27 Apr 2015 11:03am GMT

Tennessee Leeuwenburg: Interlude: Kaggle Scripting

I don't like to support specific companies, in general I'm more interested in how people can be productive as independently and autonomously as possible.

However, Kaggle just did something pretty amazing.

They've launched a free online scripting environment for machine learning, with access to their competition data sets. That means that everything I've been writing about so far -- loading data, setting up your environment, using notebooks, can be easily sidestepped for instant results. The best thing -- it supports Python!

I haven't yet determined where the limitations are, I've simply forked one of the sample scripts that I found, but it worked wonderfully. I found that the execution time of the script was similar to running it locally on my 2011 Macbook Air, but that's not bad for something totally free that runs on someone else's infrastructure. I was able to sidestep data downloading entirely, and "leap into the middle" my forking an existing script.

Here's a direct link to one:

http://www.kaggle.com/users/320654/cs822-am/otto-group-product-classification-challenge/benchmark

Note, I'm logged into Kaggle, so I'm not sure how this will go for people who aren't signed in.

My expectation is that for solutions which demand a lot of computational resources (or rely on the GPU for reasonable performance) or use a unusual libraries, a self-service approach to computing will still win out, but this is an amazing way to get started.

27 Apr 2015 11:03am GMT

PyCon Sweden: Keynote presentation: Ian Ozsvald

Hello, my name is Ian and I'm giving the opening keynote on Data Science Deployed at PyConSE this May. I'll be talking about the challenges in developing and deploying valuable data science solutions based on 15 years hard-won experience.

In the UK I run a boutique data science agency called ModelInsight and I co-run the PyDataLondon meetup and conference series. I run these events to help bind the data science community in London together. As a result teams with the right skills can come together to help companies and groups turn their data into valuable results.

My goal in this keynote is to show you how you can get started on this journey. I encourage you to ask me questions! I'm particularly intrigued to learn about the sort of data that Swedish companies have and how they currently exploit it.

27 Apr 2015 9:47am GMT

PyCon Sweden: Keynote presentation: Ian Ozsvald

Hello, my name is Ian and I'm giving the opening keynote on Data Science Deployed at PyConSE this May. I'll be talking about the challenges in developing and deploying valuable data science solutions based on 15 years hard-won experience.

In the UK I run a boutique data science agency called ModelInsight and I co-run the PyDataLondon meetup and conference series. I run these events to help bind the data science community in London together. As a result teams with the right skills can come together to help companies and groups turn their data into valuable results.

My goal in this keynote is to show you how you can get started on this journey. I encourage you to ask me questions! I'm particularly intrigued to learn about the sort of data that Swedish companies have and how they currently exploit it.

27 Apr 2015 9:47am GMT

Turnkey Linux: pyproject-pub: A simple Python project template

I hate repeating myself. It's boring. Life is too short. Like any self respecting hacker I will go out of my way to avoid it, even when I suspect it would cost me more to automate something away than to just do it by hand.

On the other hand, doing stuff I've done before by hand is no fun, while writing scripts is fun. Even when it does take longer, time is relative, or so Einstein said.

Plus, everyone knows you're much likely to make mistakes doing mind numbingly stuff, so factor that into the cost comparison. Also, doing stupid stuff over and over again makes you stupid. I'm pretty sure I read that somewhere but I might just be making it up.

So anyhow, I like writing scripts to automate stuff I'd rather not do over and over again by hand. But the zinger is that writing a new script can itself involve repetitive tasks. I'll need to create an installation routine, Debian packaging, Git repository, etc.

To avoid the catch-22, you'll want to automate that as well, which is exactly what I created pyproject-pub for. The -pub stands for public. pyproject-pub is actually a simpler, stripped down implementation of pyproject, a former Python program template I used privately which is about 4 years older and has a ton of additional complexity and a tricky set of dependencies, including one that's circular. Thankfully, most of my public projects don't need all my usual bells and whistles, hence the reboot.

You can find the latest branch of pyproject-pub on GitHub. Here's the documentation:

Why does this exist?

Minimize friction in creating a new Python program.

Features

  1. Auto-creates Git repository.
  2. Local installation via "make install" target
  3. Debian packaging support.
  4. Supports programs with a single executable command (I.e., foo) or with multiple executable commands (example-foo, example-bar).

Usage

  1. Initializing a new project:

    cp -a pyproject-pub myproject
    cd myproject
    make init
    

Installation paths

$(prefix)/lib: where Python files go. $(prefix)/bin: where symbolic links to executable commands in lib/ go. $(prefix)/share/contrib: where shared contrib stuff goes.

$(prefix) defaults to...

/usr/local: when installed locally with make install 
/usr: when installed from the built Debian package

A program with a executable command

make init creates an Python file under $(yourprojectname).py containing cli stub code. Just implement your command there. When installed you will be able to execute $(yourprojectname):

cp -a pyproject-pub mytest
cd mytest
make init

# edit your program
vim mytest.py

make install
mytest

A program with multiple executable commands

Create multiple executable Python files:

cp -a pyproject-pub mytest
cd mytest
make init
cp mytest.py foo.py
mv mytest.py bar.py

make install

# symlinks in /usr/local/bin
mytest-foo
mytest-bar

27 Apr 2015 5:55am GMT

Turnkey Linux: pyproject-pub: A simple Python project template

I hate repeating myself. It's boring. Life is too short. Like any self respecting hacker I will go out of my way to avoid it, even when I suspect it would cost me more to automate something away than to just do it by hand.

On the other hand, doing stuff I've done before by hand is no fun, while writing scripts is fun. Even when it does take longer, time is relative, or so Einstein said.

Plus, everyone knows you're much likely to make mistakes doing mind numbingly stuff, so factor that into the cost comparison. Also, doing stupid stuff over and over again makes you stupid. I'm pretty sure I read that somewhere but I might just be making it up.

So anyhow, I like writing scripts to automate stuff I'd rather not do over and over again by hand. But the zinger is that writing a new script can itself involve repetitive tasks. I'll need to create an installation routine, Debian packaging, Git repository, etc.

To avoid the catch-22, you'll want to automate that as well, which is exactly what I created pyproject-pub for. The -pub stands for public. pyproject-pub is actually a simpler, stripped down implementation of pyproject, a former Python program template I used privately which is about 4 years older and has a ton of additional complexity and a tricky set of dependencies, including one that's circular. Thankfully, most of my public projects don't need all my usual bells and whistles, hence the reboot.

You can find the latest branch of pyproject-pub on GitHub. Here's the documentation:

Why does this exist?

Minimize friction in creating a new Python program.

Features

  1. Auto-creates Git repository.
  2. Local installation via "make install" target
  3. Debian packaging support.
  4. Supports programs with a single executable command (I.e., foo) or with multiple executable commands (example-foo, example-bar).

Usage

  1. Initializing a new project:

    cp -a pyproject-pub myproject
    cd myproject
    make init
    

Installation paths

$(prefix)/lib: where Python files go. $(prefix)/bin: where symbolic links to executable commands in lib/ go. $(prefix)/share/contrib: where shared contrib stuff goes.

$(prefix) defaults to...

/usr/local: when installed locally with make install 
/usr: when installed from the built Debian package

A program with a executable command

make init creates an Python file under $(yourprojectname).py containing cli stub code. Just implement your command there. When installed you will be able to execute $(yourprojectname):

cp -a pyproject-pub mytest
cd mytest
make init

# edit your program
vim mytest.py

make install
mytest

A program with multiple executable commands

Create multiple executable Python files:

cp -a pyproject-pub mytest
cd mytest
make init
cp mytest.py foo.py
mv mytest.py bar.py

make install

# symlinks in /usr/local/bin
mytest-foo
mytest-bar

27 Apr 2015 5:55am GMT

Vasudev Ram: pyvim, an experimental pure Python vim clone

By Vasudev Ram


Saw this via Twitter:

pyvim, a pure Python vim clone, by Jonathan Slenders from Belgium.

It's new and experimental, so will not have a lot of features, and may have some bugs.

I tried it out briefly after installing it with:

pip install pyvim

(That's all there is to it. It's location even got added to my PATH.)

Here is a screenshot of me editing a file in pyvim:


And here is a screenshot of running pyvim without specifying a filename, i.e. editing a new file (same image as the one at the top of this post, except that the one at the top links to the pyvim site on Github):


The really interesting thing about the editor being written in Python, is that, as they say, it may later be possible to script the editor in Python:

"Further, when the project develops, it should also become possible to write extensions in Python, and use Python as a scripting language. (Instead of vimscript, for instance.)"

Speaking of vim, if you are new to vim or vi (vim's predecessor), you may like to check out my vi quickstart tutorial, first published in Linux For You magazine. I first wrote the tutorial at the request of some Windows sysadmin friends who were transitioning to Linux, and they told me that it helped them to come up to speed with vi quickly, for basic text editing tasks. I then sent it to Linux For You as an article proposal and they published it.

- Vasudev Ram - Online Python training and programming Dancing Bison EnterprisesSignup to hear about new products or services that I create. Posts about Python Posts about xtopdf Contact Page

Share |
Vasudev Ram

27 Apr 2015 12:10am GMT

Vasudev Ram: pyvim, an experimental pure Python vim clone

By Vasudev Ram


Saw this via Twitter:

pyvim, a pure Python vim clone, by Jonathan Slenders from Belgium.

It's new and experimental, so will not have a lot of features, and may have some bugs.

I tried it out briefly after installing it with:

pip install pyvim

(That's all there is to it. It's location even got added to my PATH.)

Here is a screenshot of me editing a file in pyvim:


And here is a screenshot of running pyvim without specifying a filename, i.e. editing a new file (same image as the one at the top of this post, except that the one at the top links to the pyvim site on Github):


The really interesting thing about the editor being written in Python, is that, as they say, it may later be possible to script the editor in Python:

"Further, when the project develops, it should also become possible to write extensions in Python, and use Python as a scripting language. (Instead of vimscript, for instance.)"

Speaking of vim, if you are new to vim or vi (vim's predecessor), you may like to check out my vi quickstart tutorial, first published in Linux For You magazine. I first wrote the tutorial at the request of some Windows sysadmin friends who were transitioning to Linux, and they told me that it helped them to come up to speed with vi quickly, for basic text editing tasks. I then sent it to Linux For You as an article proposal and they published it.

- Vasudev Ram - Online Python training and programming Dancing Bison EnterprisesSignup to hear about new products or services that I create. Posts about Python Posts about xtopdf Contact Page

Share |
Vasudev Ram

27 Apr 2015 12:10am GMT

26 Apr 2015

feedPlanet Python

Alex Clark: Plock Rocks

Plock Meme

Plock is a Plone installer for the pip-loving crowd. Plone is the ultimate open source enterprise CMS.

Understanding Plock

To understand Plock you must understand a few things:

  • The complexity of the Plone stack [1].
  • My desire to simplify, clarify and reduce-to-bare-elements everything I touch.
  • My willingness to mask complexity when eliminating it is not possible, despite the risk of contributing to it.

Pyramid author Chris McDonough [2] once made a comment a long time ago to the effect: "Let's stop piling more crap on top of Plone" and that sentiment still resonates today. That's why even though I love small and useful tools like Plock, it pains me to know what Plock is doing "under the hood" [7]. Nevertheless, I felt compelled to write it and make it work well because not having it is even more painful.

Before I tell you what Plock is [8], let me briefly describe what Plone is.

What is Plone, really?

What is the complexity I mention above? Briefly, with as few loaded statements as possible:

  • Zope2 "application server". This is something you can pip install but the results will not be usable [3].

  • Zope2 add-ons AKA "products", most notably the Zope2 Content Management Framework (CMF). This is something you install on top of Zope2 (conceptually but not literally, pip install Products.CMFCore) that provides typical content management features e.g. personalization, workflow, cataloging, etc.

  • Zope3 technologies e.g. the Zope Component Architecture (ZCA). These are things that are included-or-integrated with Zope2 and Plone. [4]

  • Buildout technologies e.g. setuptools, console scripts, recipes, extensions, etc. You can't easily build Plone without them, so we may as well declare them as dependencies.

  • Plone technologies. Plone was originally known as a "skin for CMF" but has become much more than that.

    • Archetypes Legacy content type framework.
    • Dexterity Modern content type framework based on modern Zope concepts e.g. "Reuse over reinvention".
    • Diazo Modern theming engine based on XSLT that "maps Plone content to generic website themes."

In total, if you pip install Plone over 200 Python packages are installed [5].

What is Plock, really?

OK now it's time to explain Plock. Plock is something:

  • you install from PyPI via pip install plock. "Pip installs packages. Plock installs Plone."
  • you use to install Plone without having to know about tarballs or Buildout.
  • you use to install Plone add-ons without having to know about Buildout.

In one sentence: Plock runs Buildout so you don't have to, at least initially.

First steps with Plock

Step #1

The first step with Plock [9] is that light bulb moment when you say to yourself: "I've heard that Plone is the ultimate open source enterprise CMS and I'd love to try it!" But you aren't willing to download a compressed archive and run the installer nor are you willing to pip install zc.buildout and figure the rest out for yourself. Enter Plock.

Step #2

The second step with Plock is knowing that it exists you can install it with: pip install plock.

Step #3

The third step with Plock is using it to install Plone:

$ plock plone
Creating virtualenv... (plone)
Installing buildout...
Downloading installer (https://launchpad.net/plone/4.3/4.3.4/+download/Plone-4.3.4-r1-UnifiedInstaller.tgz)
Unpacking installer...
Unpacking cache...
Installing eggs...
Installing cmmi & dist...
Configuring cache...
Running buildout...
Done, now run:
  plone/bin/plone fg

Now Plock's work is done; visit the following URL: http:://localhost:8080 and you should see:

Plock Screen 1

Create a Plone site:

Plock Screen 2

Start using Plone:

Plock Screen 3

Next steps with Plock

Plock is more than just a way to install the latest stable version of Plone quickly and easily. It's also a way to find and install Plone add-ons quickly and easily, and a way to install almost any version of Plone including the upcoming Plone 5 release.

Installing Add-ons

Step #1

List all Plone-related packages on PyPI:

$ plock -l
1) 73.unlockItems                           - A small tool for unlocking web_dav locked item in a plone portal.
2) actionbar.panel                          - Provides a (old) facebook style action panel at the bottom of your  Plone site
3) adi.init                                 - Deletes Plone's default contents
4) adi.samplecontent                        - Deletes Plone's default content and adds some sample content
5) adi.slickstyle                           - A slick style for Plone portals, easily extendable for your own styles.
6) affinitic.simplecookiecuttr              - Basic integration of jquery.cookiecuttr.js for Plone 3
7) anthill.querytool                        - GUI for AdvancedQuery with some extensions - searching the easy way for Plone
8) anthill.skinner                          - Skinning for plone made easy
9) anz.dashboard                            - Plone netvibes like dashboard implementation
10) anz.ijabbar                              - Integrate iJab(an open source XMPP web chat client recommended by xmpp.org) to your plone site.
…
1,352) zopeskel.diazotheme                      - Paster templates for Plone Diazo theme package
1,353) zopeskel.niteoweb                        - Paster templates for standard NiteoWeb Plone projects
1,354) zopyx.ecardsng                           - An ECard implementation for Plone
1,355) zopyx.existdb                            - Plone-ExistDB integration
1,356) zopyx.ipsumplone                         - Lorem ipsum text and image demo content for Plone
1,357) zopyx.multieventcalendar                 - A multi-event calendar for Plone 3.X
1,358) zopyx.plone.cassandra                    - Show all assigned local roles within a subtree for any Plone 4 site
1,359) zopyx.plone.migration                    - Export/import scripts for migration Plone 2+3 to Plone 4
1,360) zopyx.smartprintng.plone                 - Produce & Publisher server integration with Plone
1,361) zopyx.together                           - Plone integration with together.js

Step #2

Note

Plock currently only supports the initial creation of buildout.cfg, so if you have already run plock once and you want to install add-ons you'll have to use -f to overwrite buildout.cfg.

Pick a few interesting things and install them:

$ plock plone -i "Products.PloneFormGen collective.plonetruegallery eea.facetednavigation"
Creating virtualenv... (plone)
Installing buildout...
Downloading installer (https://launchpad.net/plone/4.3/4.3.4/+download/Plone-4.3.4-r1-UnifiedInstaller.tgz)
Unpacking installer...
Unpacking cache...
Installing eggs...
Installing cmmi & dist...
Configuring cache...
Installing addons...
- https://pypi.python.org/pypi/Products.PloneFormGen
- https://pypi.python.org/pypi/collective.plonetruegallery
- https://pypi.python.org/pypi/eea.facetednavigation
Running buildout...
Done, now run:
  plone/bin/plone fg

Now you should see your add-ons available in Plone:

Plock Screen 6

Upgrading Plone

Step #1

Realize Plock has created a buildout.cfg file you can edit with a text editor.

Step #2

Also realize Plock hosts Buildout configuration files called Pins you can extend from your local buildout.cfg file [10].

Step #3

Edit your buildout.cfg file. Change the first extends URL from:

[buildout]
extends =
    https://raw.github.com/plock/pins/master/plone-4-3
#    https://raw.github.com/plock/pins/master/dev

To:

[buildout]
extends =
    https://raw.github.com/plock/pins/master/plone-5-0
#    https://raw.github.com/plock/pins/master/dev

Run Buildout and start Plone:

$ bin/buildout
$ bin/plone fg

Enjoy the Plone 5 running man:

Plock Screen 5

TL;DR

Cut and paste this into a terminal:

pip install plock; plock plone; plone/bin/plone fg

Now open http://localhost:8080 and happy Ploning.

Plock 0.3.0 is out! Install with pip install plock and report issues here: https://github.com/plock/plock/issues.

Footnotes

[1] Whether or not dealing with that complexity is "worth it" I will not address here. Suffice it to say people still use and care about Plone and with Plone 5 coming "real soon now" there is some excitement building.
[2] He probably made it many times, and rightfully so.
[3] You can create an "instance" after pip install zope2 with bin/mkzopeinstance but $INSTANCE/bin/runzope fails with ImportError: cannot import name _error_start probably due to mismanaged package versions. Maybe we can fix this with version specs included in a dummy package's setup.py?
[4] The integration is not seemless, an undisputed fact as far as I know.
[5] 235
[7] Creating and executing a buildout.cfg file for the end user. Buildout configuration files are written in INI-style text. Ideally the end user sees this file and says "Ah, now I understand how this works."
[8] I've also covered Plock before here.
[9] As someone familiar with Python and a UNIX shell already, because that is the market I like to serve.
[10] Yes, there is a security and/or reliability issue with doing this; you are clearly trading security and reliability for convenience.

26 Apr 2015 10:10pm GMT

Alex Clark: Plock Rocks

Plock Meme

Plock is a Plone installer for the pip-loving crowd. Plone is the ultimate open source enterprise CMS.

Understanding Plock

To understand Plock you must understand a few things:

  • The complexity of the Plone stack [1].
  • My desire to simplify, clarify and reduce-to-bare-elements everything I touch.
  • My willingness to mask complexity when eliminating it is not possible, despite the risk of contributing to it.

Pyramid author Chris McDonough [2] once made a comment a long time ago to the effect: "Let's stop piling more crap on top of Plone" and that sentiment still resonates today. That's why even though I love small and useful tools like Plock, it pains me to know what Plock is doing "under the hood" [7]. Nevertheless, I felt compelled to write it and make it work well because not having it is even more painful.

Before I tell you what Plock is [8], let me briefly describe what Plone is.

What is Plone, really?

What is the complexity I mention above? Briefly, with as few loaded statements as possible:

  • Zope2 "application server". This is something you can pip install but the results will not be usable [3].

  • Zope2 add-ons AKA "products", most notably the Zope2 Content Management Framework (CMF). This is something you install on top of Zope2 (conceptually but not literally, pip install Products.CMFCore) that provides typical content management features e.g. personalization, workflow, cataloging, etc.

  • Zope3 technologies e.g. the Zope Component Architecture (ZCA). These are things that are included-or-integrated with Zope2 and Plone. [4]

  • Buildout technologies e.g. setuptools, console scripts, recipes, extensions, etc. You can't easily build Plone without them, so we may as well declare them as dependencies.

  • Plone technologies. Plone was originally known as a "skin for CMF" but has become much more than that.

    • Archetypes Legacy content type framework.
    • Dexterity Modern content type framework based on modern Zope concepts e.g. "Reuse over reinvention".
    • Diazo Modern theming engine based on XSLT that "maps Plone content to generic website themes."

In total, if you pip install Plone over 200 Python packages are installed [5].

What is Plock, really?

OK now it's time to explain Plock. Plock is something:

  • you install from PyPI via pip install plock. "Pip installs packages. Plock installs Plone."
  • you use to install Plone without having to know about tarballs or Buildout.
  • you use to install Plone add-ons without having to know about Buildout.

In one sentence: Plock runs Buildout so you don't have to, at least initially.

First steps with Plock

Step #1

The first step with Plock [9] is that light bulb moment when you say to yourself: "I've heard that Plone is the ultimate open source enterprise CMS and I'd love to try it!" But you aren't willing to download a compressed archive and run the installer nor are you willing to pip install zc.buildout and figure the rest out for yourself. Enter Plock.

Step #2

The second step with Plock is knowing that it exists you can install it with: pip install plock.

Step #3

The third step with Plock is using it to install Plone:

$ plock plone
Creating virtualenv... (plone)
Installing buildout...
Downloading installer (https://launchpad.net/plone/4.3/4.3.4/+download/Plone-4.3.4-r1-UnifiedInstaller.tgz)
Unpacking installer...
Unpacking cache...
Installing eggs...
Installing cmmi & dist...
Configuring cache...
Running buildout...
Done, now run:
  plone/bin/plone fg

Now Plock's work is done; visit the following URL: http:://localhost:8080 and you should see:

Plock Screen 1

Create a Plone site:

Plock Screen 2

Start using Plone:

Plock Screen 3

Next steps with Plock

Plock is more than just a way to install the latest stable version of Plone quickly and easily. It's also a way to find and install Plone add-ons quickly and easily, and a way to install almost any version of Plone including the upcoming Plone 5 release.

Installing Add-ons

Step #1

List all Plone-related packages on PyPI:

$ plock -l
1) 73.unlockItems                           - A small tool for unlocking web_dav locked item in a plone portal.
2) actionbar.panel                          - Provides a (old) facebook style action panel at the bottom of your  Plone site
3) adi.init                                 - Deletes Plone's default contents
4) adi.samplecontent                        - Deletes Plone's default content and adds some sample content
5) adi.slickstyle                           - A slick style for Plone portals, easily extendable for your own styles.
6) affinitic.simplecookiecuttr              - Basic integration of jquery.cookiecuttr.js for Plone 3
7) anthill.querytool                        - GUI for AdvancedQuery with some extensions - searching the easy way for Plone
8) anthill.skinner                          - Skinning for plone made easy
9) anz.dashboard                            - Plone netvibes like dashboard implementation
10) anz.ijabbar                              - Integrate iJab(an open source XMPP web chat client recommended by xmpp.org) to your plone site.
…
1,352) zopeskel.diazotheme                      - Paster templates for Plone Diazo theme package
1,353) zopeskel.niteoweb                        - Paster templates for standard NiteoWeb Plone projects
1,354) zopyx.ecardsng                           - An ECard implementation for Plone
1,355) zopyx.existdb                            - Plone-ExistDB integration
1,356) zopyx.ipsumplone                         - Lorem ipsum text and image demo content for Plone
1,357) zopyx.multieventcalendar                 - A multi-event calendar for Plone 3.X
1,358) zopyx.plone.cassandra                    - Show all assigned local roles within a subtree for any Plone 4 site
1,359) zopyx.plone.migration                    - Export/import scripts for migration Plone 2+3 to Plone 4
1,360) zopyx.smartprintng.plone                 - Produce & Publisher server integration with Plone
1,361) zopyx.together                           - Plone integration with together.js

Step #2

Note

Plock currently only supports the initial creation of buildout.cfg, so if you have already run plock once and you want to install add-ons you'll have to use -f to overwrite buildout.cfg.

Pick a few interesting things and install them:

$ plock plone -i "Products.PloneFormGen collective.plonetruegallery eea.facetednavigation"
Creating virtualenv... (plone)
Installing buildout...
Downloading installer (https://launchpad.net/plone/4.3/4.3.4/+download/Plone-4.3.4-r1-UnifiedInstaller.tgz)
Unpacking installer...
Unpacking cache...
Installing eggs...
Installing cmmi & dist...
Configuring cache...
Installing addons...
- https://pypi.python.org/pypi/Products.PloneFormGen
- https://pypi.python.org/pypi/collective.plonetruegallery
- https://pypi.python.org/pypi/eea.facetednavigation
Running buildout...
Done, now run:
  plone/bin/plone fg

Now you should see your add-ons available in Plone:

Plock Screen 6

Upgrading Plone

Step #1

Realize Plock has created a buildout.cfg file you can edit with a text editor.

Step #2

Also realize Plock hosts Buildout configuration files called Pins you can extend from your local buildout.cfg file [10].

Step #3

Edit your buildout.cfg file. Change the first extends URL from:

[buildout]
extends =
    https://raw.github.com/plock/pins/master/plone-4-3
#    https://raw.github.com/plock/pins/master/dev

To:

[buildout]
extends =
    https://raw.github.com/plock/pins/master/plone-5-0
#    https://raw.github.com/plock/pins/master/dev

Run Buildout and start Plone:

$ bin/buildout
$ bin/plone fg

Enjoy the Plone 5 running man:

Plock Screen 5

TL;DR

Cut and paste this into a terminal:

pip install plock; plock plone; plone/bin/plone fg

Now open http://localhost:8080 and happy Ploning.

Plock 0.3.0 is out! Install with pip install plock and report issues here: https://github.com/plock/plock/issues.

Footnotes

[1] Whether or not dealing with that complexity is "worth it" I will not address here. Suffice it to say people still use and care about Plone and with Plone 5 coming "real soon now" there is some excitement building.
[2] He probably made it many times, and rightfully so.
[3] You can create an "instance" after pip install zope2 with bin/mkzopeinstance but $INSTANCE/bin/runzope fails with ImportError: cannot import name _error_start probably due to mismanaged package versions. Maybe we can fix this with version specs included in a dummy package's setup.py?
[4] The integration is not seemless, an undisputed fact as far as I know.
[5] 235
[7] Creating and executing a buildout.cfg file for the end user. Buildout configuration files are written in INI-style text. Ideally the end user sees this file and says "Ah, now I understand how this works."
[8] I've also covered Plock before here.
[9] As someone familiar with Python and a UNIX shell already, because that is the market I like to serve.
[10] Yes, there is a security and/or reliability issue with doing this; you are clearly trading security and reliability for convenience.

26 Apr 2015 10:10pm GMT

Daily Tech Video (Python): Ian Oszvald: Cleaning Confused Collections of Characters

The world is a messy place, and trying to make sense of it can be quite demanding for a program - or the programmer writing that program. If you're trying to make sense of text files, such as Word documents or PDF, then it's particularly difficult to extract useful meaning. Adjacent words in the final output might not really be adjacent in the file, character encodings might not be set correctly, and poorly standardized things such as measurements and dates can also cause trouble. For this reason, any sort of serious data analysis starts with the cleaning up of the data source, turning it into something that can be handled reasonably. In this talk, Ian Oszvald describes some of the Python programming techniques he employs in his job to clean data, so that he can then manipulate and work with it.

The post Ian Oszvald: Cleaning Confused Collections of Characters appeared first on Daily Tech Video.

26 Apr 2015 3:00pm GMT

Daily Tech Video (Python): Ian Oszvald: Cleaning Confused Collections of Characters

The world is a messy place, and trying to make sense of it can be quite demanding for a program - or the programmer writing that program. If you're trying to make sense of text files, such as Word documents or PDF, then it's particularly difficult to extract useful meaning. Adjacent words in the final output might not really be adjacent in the file, character encodings might not be set correctly, and poorly standardized things such as measurements and dates can also cause trouble. For this reason, any sort of serious data analysis starts with the cleaning up of the data source, turning it into something that can be handled reasonably. In this talk, Ian Oszvald describes some of the Python programming techniques he employs in his job to clean data, so that he can then manipulate and work with it.

The post Ian Oszvald: Cleaning Confused Collections of Characters appeared first on Daily Tech Video.

26 Apr 2015 3:00pm GMT

بايثون العربي: بايثون عصر جديد من البرمجة


السوق يطلب أكثر : مع طرح العديد من التكنولوجيا والأجهزة الحديثة والجديدة فان شروط ومتطلبات المستخدم ستتغير ايضا مع الوقت ، والتكنولوجيا تنخفض قيمتها بوتيرة سريعة جدا أكثر مما نتوقع ففي هذا العالم دائما هناك الجديد وكما نعلم ان الجديد يقتل القديم ويحطمه وبغض النظر عن التكنولوجيا التي تقوم ببرمجة التطبيقات والبرامج الخاصة بها عليك ان تقوم بتعديل وتطوير ذلك البرنامج بما يتناسب مع التطويرات السريعة التي تطرا على التكنولوجيا وطلبات السوق التي لا تنتهي .
و مع دخول أجهزة الكمبيوتر عهد جديد من التطور مثل الهواتف الذكية والأجهزة اللوحية أصبح التحدي أكبر بالنسبة لمطوري البرمجيات لمواكبة طموحات العميل والحفاظ على السوق، ومع وجود العديد من المنصات و الأجهزة والتكنولوجيا المختلفة يصبح الوضع صعب جدا للمبرمجين للتعامل مع هذا الوضع .
في الوقت الحالي هناك تقنيات قليلة والتي يكثر عليها الطلب في السوق وهي : JSP , ASP , PHP ,PYTHON إلخ.
بايثون لغة قديمة تم إعادة تطويرها من جديد في القرن الواحد والعشرين من طرف العملاق قوقل وقد مرت على عدة مراحل حتى وصلت الى ماهي عليه الأن.

يمكن لبايثون التعامل مع طلبات العصر : خلال العامين الماضيين رأينا نمو هائل الحاصل في صناعة الاجهزة المحمولة مثل الاجهزة اللوحية أو الهواتف الذكية حيث تم تمكين هذه الأجهزة من تكنولوجيا الجيل الثالث وبالتالي يمكن إستخدامها لتصفح الانترنيت كما يمكن أيضا إستخدام هذه الأجهزة للتحكم في الألعاب ثلاثية الأبعاد والتي تتطلب شاشات عالية الدقة ، هل هناك المزيد ؟ طبعا هناك المزيد حيث يمكن أيضا لهذه الأجهزة أت تأتي محملة بالعديد من التطبيقات الأصلية والتي تساعد المستخدم من تنظيم أيامه الشخصية والعملية منها وكما يمكن استخدامها كساعات ومنبهات، الاستماع الى الراديو الخ ... وكل هذا يدل على إمكانيات هذه الأجهزة في السوق والتي من المفترض أن يزيد حجم نموها في السوق بوتيرة متسارعة ، ومن اجل كل هذا يحتاج المستخدم الى مزيد من التطبيقات التي تعمل على تحسين وتعزيز تجربتهم في الميدان، ولتحقيق جميع مطالبهم التي لا تنتهي هناك العديد من التقنيات التي تسمع بذلك وبطرق كثيرة ومتعددة، وحظ بايثون في هذا المجال حظ وفير وفريد من نوعه بحيث يمكن لمبرمجي بايثون التعامل مع جميع متطلبات ومطالب المستخدم في جوانب مختلفة مثل تطوير تطبيقات الويب للمؤسسات ، تطبيقات الويب ، العاب الفبدبو الخ .. وبالتالي يمكننا أن نقول أن بايثون تمتلك جميع المميزات التي يمكنها دخول غمار المنافسة .

بايثون ممتازة وسريعة وموثوقة : بالإضافة الى إمكانيات بايثون في التحكم في مختلف مجالات تطوير التطبيقات والبرامج تعتبر لغة بايثون الأمثل في تطوير البرمجيات التي تعمل بسرعة عالية حيث تتوفر على جميع الدوال والوحدات والمكتبات المناسبة والتي تجعل من عملية التطوير عملية فعالة وذات كفاءة عالية .

بايثون يمتلك دعم ومستقبل رائع : هناك العديد من المجتمعات الضخمة الخاصة بلغة بايثون الموجهة لجميع أصناف المطورين من المبتدئين الى غاية المحترفين وهناك الكثير من المتطوعين ذوي المهارات العالية مستعدين لمساعدة أي شخص يحتاج الى مساعدة أو توجيه مهما كان نوعه وبالتالي ليست هناك مشكلة دعم عملية التطوير ببايثون ويتوقع مزدهر باهر في المستقبل .

بايثون والعرب: قد لا يختلف إثنان عن مستقبل بياثون عند الغرب ولكن عندنا نحن العرب مازلنا نستهلك في جميع الميادين حتى لا أتكلم عن بايثون فقط ، وحتى نبقى في بايثون فقط وما نلاحظه من نقص الدعم والمحتوى وحتى تطبيقات عربية خالصة الا أننا نرى بعض الاهتمام المتزايد باللغة من خلال بعض المجموعات على الفايسبوك والمنتديات .



26 Apr 2015 2:35pm GMT

بايثون العربي: بايثون عصر جديد من البرمجة


السوق يطلب أكثر : مع طرح العديد من التكنولوجيا والأجهزة الحديثة والجديدة فان شروط ومتطلبات المستخدم ستتغير ايضا مع الوقت ، والتكنولوجيا تنخفض قيمتها بوتيرة سريعة جدا أكثر مما نتوقع ففي هذا العالم دائما هناك الجديد وكما نعلم ان الجديد يقتل القديم ويحطمه وبغض النظر عن التكنولوجيا التي تقوم ببرمجة التطبيقات والبرامج الخاصة بها عليك ان تقوم بتعديل وتطوير ذلك البرنامج بما يتناسب مع التطويرات السريعة التي تطرا على التكنولوجيا وطلبات السوق التي لا تنتهي .
و مع دخول أجهزة الكمبيوتر عهد جديد من التطور مثل الهواتف الذكية والأجهزة اللوحية أصبح التحدي أكبر بالنسبة لمطوري البرمجيات لمواكبة طموحات العميل والحفاظ على السوق، ومع وجود العديد من المنصات و الأجهزة والتكنولوجيا المختلفة يصبح الوضع صعب جدا للمبرمجين للتعامل مع هذا الوضع .
في الوقت الحالي هناك تقنيات قليلة والتي يكثر عليها الطلب في السوق وهي : JSP , ASP , PHP ,PYTHON إلخ.
بايثون لغة قديمة تم إعادة تطويرها من جديد في القرن الواحد والعشرين من طرف العملاق قوقل وقد مرت على عدة مراحل حتى وصلت الى ماهي عليه الأن.

يمكن لبايثون التعامل مع طلبات العصر : خلال العامين الماضيين رأينا نمو هائل الحاصل في صناعة الاجهزة المحمولة مثل الاجهزة اللوحية أو الهواتف الذكية حيث تم تمكين هذه الأجهزة من تكنولوجيا الجيل الثالث وبالتالي يمكن إستخدامها لتصفح الانترنيت كما يمكن أيضا إستخدام هذه الأجهزة للتحكم في الألعاب ثلاثية الأبعاد والتي تتطلب شاشات عالية الدقة ، هل هناك المزيد ؟ طبعا هناك المزيد حيث يمكن أيضا لهذه الأجهزة أت تأتي محملة بالعديد من التطبيقات الأصلية والتي تساعد المستخدم من تنظيم أيامه الشخصية والعملية منها وكما يمكن استخدامها كساعات ومنبهات، الاستماع الى الراديو الخ ... وكل هذا يدل على إمكانيات هذه الأجهزة في السوق والتي من المفترض أن يزيد حجم نموها في السوق بوتيرة متسارعة ، ومن اجل كل هذا يحتاج المستخدم الى مزيد من التطبيقات التي تعمل على تحسين وتعزيز تجربتهم في الميدان، ولتحقيق جميع مطالبهم التي لا تنتهي هناك العديد من التقنيات التي تسمع بذلك وبطرق كثيرة ومتعددة، وحظ بايثون في هذا المجال حظ وفير وفريد من نوعه بحيث يمكن لمبرمجي بايثون التعامل مع جميع متطلبات ومطالب المستخدم في جوانب مختلفة مثل تطوير تطبيقات الويب للمؤسسات ، تطبيقات الويب ، العاب الفبدبو الخ .. وبالتالي يمكننا أن نقول أن بايثون تمتلك جميع المميزات التي يمكنها دخول غمار المنافسة .

بايثون ممتازة وسريعة وموثوقة : بالإضافة الى إمكانيات بايثون في التحكم في مختلف مجالات تطوير التطبيقات والبرامج تعتبر لغة بايثون الأمثل في تطوير البرمجيات التي تعمل بسرعة عالية حيث تتوفر على جميع الدوال والوحدات والمكتبات المناسبة والتي تجعل من عملية التطوير عملية فعالة وذات كفاءة عالية .

بايثون يمتلك دعم ومستقبل رائع : هناك العديد من المجتمعات الضخمة الخاصة بلغة بايثون الموجهة لجميع أصناف المطورين من المبتدئين الى غاية المحترفين وهناك الكثير من المتطوعين ذوي المهارات العالية مستعدين لمساعدة أي شخص يحتاج الى مساعدة أو توجيه مهما كان نوعه وبالتالي ليست هناك مشكلة دعم عملية التطوير ببايثون ويتوقع مزدهر باهر في المستقبل .

بايثون والعرب: قد لا يختلف إثنان عن مستقبل بياثون عند الغرب ولكن عندنا نحن العرب مازلنا نستهلك في جميع الميادين حتى لا أتكلم عن بايثون فقط ، وحتى نبقى في بايثون فقط وما نلاحظه من نقص الدعم والمحتوى وحتى تطبيقات عربية خالصة الا أننا نرى بعض الاهتمام المتزايد باللغة من خلال بعض المجموعات على الفايسبوك والمنتديات .



26 Apr 2015 2:35pm GMT

Filipe Saraiva: Cantor in KDE Applications 15.04

KDE Applications 15.04 release brings a new version of the scientific programming software Cantor, with a lot of news. I am specially happy with this release because I worked in several parts of these new features. =)

Come with me™ and let's see what is new in Cantor.

Cantor ported to Qt5/KF5

cantor-kf5

Cantor Qt5/KF5 + Breeze theme. In the image it is possible to see the terminal/worksheet, variable management panel, syntax highlighting, code completion, and the standard interface

I started the Cantor port to Qt5/KF5 during previous LaKademy and I continued the development along the year. Maybe I had pushed code from 5 different countries since the beginning of this work.

The change for this new technology was successfully completed, and for the moment we don't notice any feature missed or new critical bug. All the backends and plugins were ported, and some new bugs created during this work were fixed.

We would like to ask for Cantor users to report any problem or bug in bugzilla. Anyway, the software is really very stable.

When you run Cantor Qt5/KF5 version on the first time, the software will look for Cantor Qt4 configurations and, if it exists, the configurations will be automagically migrated to Cantor Qt5/KF5.

Backend for Python 3

In Season of KDE 2014 I was the mentor of Minh Ngo in the project to create a backend for Python 3, increasing the number of backends in Cantor to 10!

cantor-backends

Backend selection screen: Python 3 and their 9 brothers

The backend developed by Minh uses D-Bus protocol to allow communication between Cantor and Python 3. This architecture is different of Python 2, but it is present in others backends, as in the backend for R.

The cool thing is Cantor can be interesting for pythonistas using Python 2 and/or Python 3 now. We would like to get feedback from you, guys!

Icon!

Cantor first release was originally in 2009, with KDE SC 4.4. Since that date the software did not have an icon.

The Cantor Qt5/KF5 release marks a substantial change in the development of the application, then it is also a good time to release an icon to the software.

Ícone do Cantor

Cantor icon

The art is excellent! It presents the idea of Cantor: a blackboard to you write and develop your equations and formulas while scratches his head and think "and now, what I need to do to solve it?". =)

Thank you Andreas Kainz and Uri Herrera, members of VDG team and authors of Cantor icon!

Other changes and bug fixes

Most bugs added in the Qt5/KF5 port were fixed before the release.

There are some small changes to be cited: in KNewStuff categories world, "Python2″ category was changed to "Python 2″ and "Python 3″ category was added; the automatic loading of pylab module in Python backends was dropped; now it is possible to run Python commands mixed with comments in the worksheet; and more.

You can see a complete log of commits, bugfixes, and new features added in this release in this page.

Future works

As future work maybe the high-priority for this moment is to drop KDELibs4Support from Cantor. Lucas developed part of this work and we would like to finish it for the next release.

I intend to test if D-Bus communication can be a good solution for Scilab backend. Another task is to redesign the graphical generation assistants of Python backends. A long-term work is to follow the creation of Jupyter project, the future of IPython notebooks. If Cantor can to be compatible with Jupyter, it will be really nice for users and to encourage the collaboration between different communities interested in scientific programming and open science.

I will take advantage of the Cantor Qt5/KF5 release to write about how to use Cantor in two different ways: the Matlab way and the IPython notebooks way. Keep your eyes in the updates from this blog! =)

If you would like to help in Cantor development, please contact me or mail kde-edu maillist and let's talk about bug fixes, development of new features, and more.

Donations to KDE Brasil - LaKademy 2015!

If you would like to support my work, please make a donation to KDE Brasil. We will host the KDE Latin-American Summit - LaKademy and we need some money to put some latin-american contributors to work together face-to-face. I will focus my LaKademy work in the previously mentioned future works.

You can read more about LaKademy in this dot.KDE history. This page in English explain how to donate. There is other page with the same content in Spanish.

26 Apr 2015 9:37am GMT

Filipe Saraiva: Cantor in KDE Applications 15.04

KDE Applications 15.04 release brings a new version of the scientific programming software Cantor, with a lot of news. I am specially happy with this release because I worked in several parts of these new features. =)

Come with me™ and let's see what is new in Cantor.

Cantor ported to Qt5/KF5

cantor-kf5

Cantor Qt5/KF5 + Breeze theme. In the image it is possible to see the terminal/worksheet, variable management panel, syntax highlighting, code completion, and the standard interface

I started the Cantor port to Qt5/KF5 during previous LaKademy and I continued the development along the year. Maybe I had pushed code from 5 different countries since the beginning of this work.

The change for this new technology was successfully completed, and for the moment we don't notice any feature missed or new critical bug. All the backends and plugins were ported, and some new bugs created during this work were fixed.

We would like to ask for Cantor users to report any problem or bug in bugzilla. Anyway, the software is really very stable.

When you run Cantor Qt5/KF5 version on the first time, the software will look for Cantor Qt4 configurations and, if it exists, the configurations will be automagically migrated to Cantor Qt5/KF5.

Backend for Python 3

In Season of KDE 2014 I was the mentor of Minh Ngo in the project to create a backend for Python 3, increasing the number of backends in Cantor to 10!

cantor-backends

Backend selection screen: Python 3 and their 9 brothers

The backend developed by Minh uses D-Bus protocol to allow communication between Cantor and Python 3. This architecture is different of Python 2, but it is present in others backends, as in the backend for R.

The cool thing is Cantor can be interesting for pythonistas using Python 2 and/or Python 3 now. We would like to get feedback from you, guys!

Icon!

Cantor first release was originally in 2009, with KDE SC 4.4. Since that date the software did not have an icon.

The Cantor Qt5/KF5 release marks a substantial change in the development of the application, then it is also a good time to release an icon to the software.

Ícone do Cantor

Cantor icon

The art is excellent! It presents the idea of Cantor: a blackboard to you write and develop your equations and formulas while scratches his head and think "and now, what I need to do to solve it?". =)

Thank you Andreas Kainz and Uri Herrera, members of VDG team and authors of Cantor icon!

Other changes and bug fixes

Most bugs added in the Qt5/KF5 port were fixed before the release.

There are some small changes to be cited: in KNewStuff categories world, "Python2″ category was changed to "Python 2″ and "Python 3″ category was added; the automatic loading of pylab module in Python backends was dropped; now it is possible to run Python commands mixed with comments in the worksheet; and more.

You can see a complete log of commits, bugfixes, and new features added in this release in this page.

Future works

As future work maybe the high-priority for this moment is to drop KDELibs4Support from Cantor. Lucas developed part of this work and we would like to finish it for the next release.

I intend to test if D-Bus communication can be a good solution for Scilab backend. Another task is to redesign the graphical generation assistants of Python backends. A long-term work is to follow the creation of Jupyter project, the future of IPython notebooks. If Cantor can to be compatible with Jupyter, it will be really nice for users and to encourage the collaboration between different communities interested in scientific programming and open science.

I will take advantage of the Cantor Qt5/KF5 release to write about how to use Cantor in two different ways: the Matlab way and the IPython notebooks way. Keep your eyes in the updates from this blog! =)

If you would like to help in Cantor development, please contact me or mail kde-edu maillist and let's talk about bug fixes, development of new features, and more.

Donations to KDE Brasil - LaKademy 2015!

If you would like to support my work, please make a donation to KDE Brasil. We will host the KDE Latin-American Summit - LaKademy and we need some money to put some latin-american contributors to work together face-to-face. I will focus my LaKademy work in the previously mentioned future works.

You can read more about LaKademy in this dot.KDE history. This page in English explain how to donate. There is other page with the same content in Spanish.

26 Apr 2015 9:37am GMT

25 Apr 2015

feedPlanet Python

BangPypers: April 2015 Meetup report

The April BangPypers meetup happened at IBM, Domlur. This time we had 4 talks from 10:30AM to 1.10PM.

40+ people attended the talks. The first talk was about Enough iPython For Everyday Programming by Anand Reddy Pandikunta.
The talk covered various IPython features, integration with Emacs etc … There were lot of interaction from participants. Here is the link to
slides.

Second talk was by Bargava and Ragotham about plotting libraries in Python. The talk provoked lot of interesting discussions.

Third talk was by Anirudh about Basics of NLP. The slides for the talk can
found in slideshare.

Last talk was by Arun Ravindran about Emacs and Python. There were lot of Emacs users in our meetups :-)
Presentation was in org mode and markdown version.

There were lot of discussions during and after talks. Thanks for all speakers, participants and IBM for hosting the event.

25 Apr 2015 8:52pm GMT

BangPypers: April 2015 Meetup report

The April BangPypers meetup happened at IBM, Domlur. This time we had 4 talks from 10:30AM to 1.10PM.

40+ people attended the talks. The first talk was about Enough iPython For Everyday Programming by Anand Reddy Pandikunta.
The talk covered various IPython features, integration with Emacs etc … There were lot of interaction from participants. Here is the link to
slides.

Second talk was by Bargava and Ragotham about plotting libraries in Python. The talk provoked lot of interesting discussions.

Third talk was by Anirudh about Basics of NLP. The slides for the talk can
found in slideshare.

Last talk was by Arun Ravindran about Emacs and Python. There were lot of Emacs users in our meetups :-)
Presentation was in org mode and markdown version.

There were lot of discussions during and after talks. Thanks for all speakers, participants and IBM for hosting the event.

25 Apr 2015 8:52pm GMT

10 Nov 2011

feedPython Software Foundation | GSoC'11 Students

Benedict Stein: King Willams Town Bahnhof

Gestern musste ich morgens zur Station nach KWT um unsere Rerservierten Bustickets für die Weihnachtsferien in Capetown abzuholen. Der Bahnhof selber ist seit Dezember aus kostengründen ohne Zugverbindung - aber Translux und co - die langdistanzbusse haben dort ihre Büros.


Größere Kartenansicht




© benste CC NC SA

10 Nov 2011 10:57am GMT

09 Nov 2011

feedPython Software Foundation | GSoC'11 Students

Benedict Stein

Niemand ist besorgt um so was - mit dem Auto fährt man einfach durch, und in der City - nahe Gnobie- "ne das ist erst gefährlich wenn die Feuerwehr da ist" - 30min später auf dem Rückweg war die Feuerwehr da.




© benste CC NC SA

09 Nov 2011 8:25pm GMT

08 Nov 2011

feedPython Software Foundation | GSoC'11 Students

Benedict Stein: Brai Party

Brai = Grillabend o.ä.

Die möchte gern Techniker beim Flicken ihrer SpeakOn / Klinke Stecker Verzweigungen...

Die Damen "Mamas" der Siedlung bei der offiziellen Eröffnungsrede

Auch wenn weniger Leute da waren als erwartet, Laute Musik und viele Leute ...

Und natürlich ein Feuer mit echtem Holz zum Grillen.

© benste CC NC SA

08 Nov 2011 2:30pm GMT

07 Nov 2011

feedPython Software Foundation | GSoC'11 Students

Benedict Stein: Lumanyano Primary

One of our missions was bringing Katja's Linux Server back to her room. While doing that we saw her new decoration.

Björn, Simphiwe carried the PC to Katja's school


© benste CC NC SA

07 Nov 2011 2:00pm GMT

06 Nov 2011

feedPython Software Foundation | GSoC'11 Students

Benedict Stein: Nelisa Haircut

Today I went with Björn to Needs Camp to Visit Katja's guest family for a special Party. First of all we visited some friends of Nelisa - yeah the one I'm working with in Quigney - Katja's guest fathers sister - who did her a haircut.

African Women usually get their hair done by arranging extensions and not like Europeans just cutting some hair.

In between she looked like this...

And then she was done - looks amazing considering the amount of hair she had last week - doesn't it ?

© benste CC NC SA

06 Nov 2011 7:45pm GMT

05 Nov 2011

feedPython Software Foundation | GSoC'11 Students

Benedict Stein: Mein Samstag

Irgendwie viel mir heute auf das ich meine Blogposts mal ein bischen umstrukturieren muss - wenn ich immer nur von neuen Plätzen berichte, dann müsste ich ja eine Rundreise machen. Hier also mal ein paar Sachen aus meinem heutigen Alltag.

Erst einmal vorweg, Samstag zählt zumindest für uns Voluntäre zu den freien Tagen.

Dieses Wochenende sind nur Rommel und ich auf der Farm - Katja und Björn sind ja mittlerweile in ihren Einsatzstellen, und meine Mitbewohner Kyle und Jonathan sind zu Hause in Grahamstown - sowie auch Sipho der in Dimbaza wohnt.
Robin, die Frau von Rommel ist in Woodie Cape - schon seit Donnerstag um da ein paar Sachen zur erledigen.
Naja wie dem auch sei heute morgen haben wir uns erstmal ein gemeinsames Weetbix/Müsli Frühstück gegönnt und haben uns dann auf den Weg nach East London gemacht. 2 Sachen waren auf der Checkliste Vodacom, Ethienne (Imobilienmakler) außerdem auf dem Rückweg die fehlenden Dinge nach NeedsCamp bringen.

Nachdem wir gerade auf der Dirtroad losgefahren sind mussten wir feststellen das wir die Sachen für Needscamp und Ethienne nicht eingepackt hatten aber die Pumpe für die Wasserversorgung im Auto hatten.

Also sind wir in EastLondon ersteinmal nach Farmerama - nein nicht das onlinespiel farmville - sondern einen Laden mit ganz vielen Sachen für eine Farm - in Berea einem nördlichen Stadteil gefahren.

In Farmerama haben wir uns dann beraten lassen für einen Schnellverschluss der uns das leben mit der Pumpe leichter machen soll und außerdem eine leichtere Pumpe zur Reperatur gebracht, damit es nicht immer so ein großer Aufwand ist, wenn mal wieder das Wasser ausgegangen ist.

Fego Caffé ist in der Hemmingways Mall, dort mussten wir und PIN und PUK einer unserer Datensimcards geben lassen, da bei der PIN Abfrage leider ein zahlendreher unterlaufen ist. Naja auf jeden Fall speichern die Shops in Südafrika so sensible Daten wie eine PUK - die im Prinzip zugang zu einem gesperrten Phone verschafft.

Im Cafe hat Rommel dann ein paar online Transaktionen mit dem 3G Modem durchgeführt, welches ja jetzt wieder funktionierte - und übrigens mittlerweile in Ubuntu meinem Linuxsystem perfekt klappt.

Nebenbei bin ich nach 8ta gegangen um dort etwas über deren neue Deals zu erfahren, da wir in einigen von Hilltops Centern Internet anbieten wollen. Das Bild zeigt die Abdeckung UMTS in NeedsCamp Katjas Ort. 8ta ist ein neuer Telefonanbieter von Telkom, nachdem Vodafone sich Telkoms anteile an Vodacom gekauft hat müssen die komplett neu aufbauen.
Wir haben uns dazu entschieden mal eine kostenlose Prepaidkarte zu testen zu organisieren, denn wer weis wie genau die Karte oben ist ... Bevor man einen noch so billigen Deal für 24 Monate signed sollte man wissen obs geht.

Danach gings nach Checkers in Vincent, gesucht wurden zwei Hotplates für WoodyCape - R 129.00 eine - also ca. 12€ für eine zweigeteilte Kochplatte.
Wie man sieht im Hintergrund gibts schon Weihnachtsdeko - Anfang November und das in Südafrika bei sonnig warmen min- 25°C

Mittagessen haben wir uns bei einem Pakistanischen Curry Imbiss gegönnt - sehr empfehlenswert !
Naja und nachdem wir dann vor ner Stunde oder so zurück gekommen sind habe ich noch den Kühlschrank geputzt den ich heute morgen zum defrosten einfach nach draußen gestellt hatte. Jetzt ist der auch mal wieder sauber und ohne 3m dicke Eisschicht...

Morgen ... ja darüber werde ich gesondert berichten ... aber vermutlich erst am Montag, denn dann bin ich nochmal wieder in Quigney(East London) und habe kostenloses Internet.

© benste CC NC SA

05 Nov 2011 4:33pm GMT

31 Oct 2011

feedPython Software Foundation | GSoC'11 Students

Benedict Stein: Sterkspruit Computer Center

Sterkspruit is one of Hilltops Computer Centres in the far north of Eastern Cape. On the trip to J'burg we've used the opportunity to take a look at the centre.

Pupils in the big classroom


The Trainer


School in Countryside


Adult Class in the Afternoon


"Town"


© benste CC NC SA

31 Oct 2011 4:58pm GMT

Benedict Stein: Technical Issues

What are you doing in an internet cafe if your ADSL and Faxline has been discontinued before months end. Well my idea was sitting outside and eating some ice cream.
At least it's sunny and not as rainy as on the weekend.


© benste CC NC SA

31 Oct 2011 3:11pm GMT

30 Oct 2011

feedPython Software Foundation | GSoC'11 Students

Benedict Stein: Nellis Restaurant

For those who are traveling through Zastron - there is a very nice Restaurant which is serving delicious food at reasanable prices.
In addition they're selling home made juices jams and honey.




interior


home made specialities - the shop in the shop


the Bar


© benste CC NC SA

30 Oct 2011 4:47pm GMT

29 Oct 2011

feedPython Software Foundation | GSoC'11 Students

Benedict Stein: The way back from J'burg

Having the 10 - 12h trip from J'burg back to ELS I was able to take a lot of pcitures including these different roadsides

Plain Street


Orange River in its beginngings (near Lesotho)


Zastron Anglican Church


The Bridge in Between "Free State" and Eastern Cape next to Zastron


my new Background ;)


If you listen to GoogleMaps you'll end up traveling 50km of gravel road - as it was just renewed we didn't have that many problems and saved 1h compared to going the official way with all it's constructions sites




Freeway


getting dark


© benste CC NC SA

29 Oct 2011 4:23pm GMT

28 Oct 2011

feedPython Software Foundation | GSoC'11 Students

Benedict Stein: Wie funktioniert eigentlich eine Baustelle ?

Klar einiges mag anders sein, vieles aber gleich - aber ein in Deutschland täglich übliches Bild einer Straßenbaustelle - wie läuft das eigentlich in Südafrika ?

Ersteinmal vorweg - NEIN keine Ureinwohner die mit den Händen graben - auch wenn hier mehr Manpower genutzt wird - sind sie fleißig mit Technologie am arbeiten.

Eine ganz normale "Bundesstraße"


und wie sie erweitert wird


gaaaanz viele LKWs


denn hier wird eine Seite über einen langen Abschnitt komplett gesperrt, so das eine Ampelschaltung mit hier 45 Minuten Wartezeit entsteht


Aber wenigstens scheinen die ihren Spaß zu haben ;) - Wie auch wir denn gücklicher Weise mussten wir nie länger als 10 min. warten.

© benste CC NC SA

28 Oct 2011 4:20pm GMT