19 Aug 2018

feedPlanet Python

PyBites: PyBites Twitter Digest - Issue 27, 2018

One more day for PSF Fellowship Nominations! Get them in ASAP!

Do you know someone who has served the Python community thru extraordinary efforts? Nominate them to be a PSF Fello… https://t.co/cNzFpaaxhN

- Python Software (@ThePSF) August 18, 2018

Decorator library to configure function arguments

Submitted by @clamytoe.

https://t.co/sIYLnFT8h1

- Patrick Porto (@PatrickPorto) August 09, 2018

Mockaroo! Who knew? Create your own data to use in Sketch!

Submitted by @bohemianjack.

"Create your own #data to use in Sketch (no code required)" by @rgesulfo https://t.co/UQIjnBv3u7 - generating some… https://t.co/dqrySH0DkR

- Pybites (@pybites) August 11, 2018

Very cool use case of OpenCV

New tutorial published!🚀 Today we're building "People Counter". Learn how to count the # of people entering & leavi… https://t.co/JQYXfNxrnu

- Adrian Rosebrock (@PyImageSearch) August 13, 2018

Nice to see Netflix doing stuff like this

RT @MichelleUfford: At Netflix we're reimagining what a Jupyter notebook can be, who can use it, & what you can do with it. Check out my Ne…

- Enthought Inc. (@enthought) August 18, 2018

Deep Learning basics by Sentdex!

Just posted an updated Deep Learning basics with Python, TensorFlow and Keras tutorial: https://t.co/Vr3nvT7CRo… https://t.co/AqLE0jMaRG

- Harrison Kinsley🐍 (@Sentdex) August 11, 2018

This is a great security step. Nice going PyPI!

RT @pypi: PyPI will no longer accept passwords that have been published in data breaches. For background you can take a look at https://t.…

- Ewa Jodlowska (@ewa_jodlowska) August 14, 2018

This is why the UX is so important.

"The average user doesn't give a damn what happens, as long as (1) it works and (2) it's fast." - Daniel J. Bernstein

- Programming Wisdom (@CodeWisdom) August 17, 2018

Altair version 2.2 released

Altair version 2.2 just released! This adds support to Vega-Lite 2.6, as well as offering a number of API improveme… https://t.co/5oSmcCA5Pq

- Jake VanderPlas (@jakevdp) August 14, 2018

ripgrep is faster than {grep, ag, git grep, ucg, pt, sift} https://t.co/gTUnxyTMw2

- Regular Expression (@RegexTip) August 16, 2018

Django v Wordpress

Django vs WordPress: Learn how to decide between these 2 very different #ContentManagement systems:… https://t.co/DP1QdmFxrP

- Caktus Group (@CaktusGroup) August 17, 2018

How to use Bootstrap 4 forms in Django

RT @pybites: How to Use Bootstrap 4 Forms With #Django https://t.co/iL9gMxy8D3 via @vitorfs

- Martin Uribe (@clamytoe) August 15, 2018

PyConAU is all sold out! Who's going next week?

That's it! We're sold out! There are still some spaces left at our development sprints for main conference attende… https://t.co/WqyipT9dq6

- PyCon Australia (@pyconau) August 17, 2018

Python text-to-speech!

Text to speech conversion using python using gtts https://t.co/zyeDeIyiwJ

- Python LibHunt (@PythonLibHunt) August 18, 2018

Nice! A web UI for pdb!

Looks useful.. python-web-pdb: Web-based remote UI for Python's PDB debugger https://t.co/J4WgKJ3EuI

- Ned Batchelder (@nedbat) August 07, 2018

>>> from pybites import Bob, Julian

Keep Calm and Code in Python!

19 Aug 2018 10:44am GMT

PyBites: PyBites Twitter Digest - Issue 27, 2018

One more day for PSF Fellowship Nominations! Get them in ASAP!

Do you know someone who has served the Python community thru extraordinary efforts? Nominate them to be a PSF Fello… https://t.co/cNzFpaaxhN

- Python Software (@ThePSF) August 18, 2018

Decorator library to configure function arguments

Submitted by @clamytoe.

https://t.co/sIYLnFT8h1

- Patrick Porto (@PatrickPorto) August 09, 2018

Mockaroo! Who knew? Create your own data to use in Sketch!

Submitted by @bohemianjack.

"Create your own #data to use in Sketch (no code required)" by @rgesulfo https://t.co/UQIjnBv3u7 - generating some… https://t.co/dqrySH0DkR

- Pybites (@pybites) August 11, 2018

Very cool use case of OpenCV

New tutorial published!🚀 Today we're building "People Counter". Learn how to count the # of people entering & leavi… https://t.co/JQYXfNxrnu

- Adrian Rosebrock (@PyImageSearch) August 13, 2018

Nice to see Netflix doing stuff like this

RT @MichelleUfford: At Netflix we're reimagining what a Jupyter notebook can be, who can use it, & what you can do with it. Check out my Ne…

- Enthought Inc. (@enthought) August 18, 2018

Deep Learning basics by Sentdex!

Just posted an updated Deep Learning basics with Python, TensorFlow and Keras tutorial: https://t.co/Vr3nvT7CRo… https://t.co/AqLE0jMaRG

- Harrison Kinsley🐍 (@Sentdex) August 11, 2018

This is a great security step. Nice going PyPI!

RT @pypi: PyPI will no longer accept passwords that have been published in data breaches. For background you can take a look at https://t.…

- Ewa Jodlowska (@ewa_jodlowska) August 14, 2018

This is why the UX is so important.

"The average user doesn't give a damn what happens, as long as (1) it works and (2) it's fast." - Daniel J. Bernstein

- Programming Wisdom (@CodeWisdom) August 17, 2018

Altair version 2.2 released

Altair version 2.2 just released! This adds support to Vega-Lite 2.6, as well as offering a number of API improveme… https://t.co/5oSmcCA5Pq

- Jake VanderPlas (@jakevdp) August 14, 2018

ripgrep is faster than {grep, ag, git grep, ucg, pt, sift} https://t.co/gTUnxyTMw2

- Regular Expression (@RegexTip) August 16, 2018

Django v Wordpress

Django vs WordPress: Learn how to decide between these 2 very different #ContentManagement systems:… https://t.co/DP1QdmFxrP

- Caktus Group (@CaktusGroup) August 17, 2018

How to use Bootstrap 4 forms in Django

RT @pybites: How to Use Bootstrap 4 Forms With #Django https://t.co/iL9gMxy8D3 via @vitorfs

- Martin Uribe (@clamytoe) August 15, 2018

PyConAU is all sold out! Who's going next week?

That's it! We're sold out! There are still some spaces left at our development sprints for main conference attende… https://t.co/WqyipT9dq6

- PyCon Australia (@pyconau) August 17, 2018

Python text-to-speech!

Text to speech conversion using python using gtts https://t.co/zyeDeIyiwJ

- Python LibHunt (@PythonLibHunt) August 18, 2018

Nice! A web UI for pdb!

Looks useful.. python-web-pdb: Web-based remote UI for Python's PDB debugger https://t.co/J4WgKJ3EuI

- Ned Batchelder (@nedbat) August 07, 2018

>>> from pybites import Bob, Julian

Keep Calm and Code in Python!

19 Aug 2018 10:44am GMT

Evennia: Inline building in upcoming Evennia 0.8


Evennia, the Python MUD-server game development kit, is slowly creeping closer to its 0.8 release.

In our development branch I've just pushed the first version of the new OLC (OnLine Creator) system. This is a system to allow builders (who may have limited coding knowledge) to customize and spawn new in-game objects more easily without code access. It's started with the olc command in-game. This is a visual system for manipulating Evennia Prototypes.


Briefly on Prototypes

The Prototype is an Evennia concept that has been around a good while. The prototype is a Python dictionary that holds specific keys with values representing properties on a game object. Here's an example of a simple prototype:

{"key": "My house",

"typeclass": "typeclasses.houses.MyHouse"}


By passing this dict to the spawner, a new object named "My house" will be created. It will be set up with the given typeclass (a 'typeclass' is, in Evennia lingo, a Python class with a database backend). A prototype can specify all aspects of an in-game object - its attributes (like description and other game-specific properties), tags, aliases, location and so on. Prototypes also support inheritance - so you can expand on an existing template without having to add everything fresh every time.

There are two main reasons for the Prototypes existing in Evennia:

What's new

As said, Prototypes have been around for a good while in Evennia. But in the past they were either manually entered directly as a dict on the command line, or created in code and read from a Python module. The former solution is cumbersome and requires that you know how to build a proper-syntax Python dictionary. The latter requires server code access, making them less useful to builders than they could be.

Note: If you are visually impaired, each image is also a link to a text-only version.

OLC index


In Evennia 0.8, while you can still insert the Prototype as a raw dict, spawn/menu or the new olc command opens a new menu-driven interface.

Select a prototype to load. This will replace any prototype currently being edited! ___________________________________________________________________________________________________ Select with <num>. Other actions: examine <num> | delete <num> Back (index) | Validate prototype | Quit 1: goblin_archer 5: goblin_archwizard 2: goblin_wizard 3: goblin 4: archwizard_mixin . Other actions: examine | delete Back (index) | Validate prototype | Quit 1: goblin_archer 5: goblin_archwizard 2: goblin_wizard 3: goblin 4: archwizard_mixin " />. Other actions: examine | delete Back (index) | Validate prototype | Quit 1: goblin_archer 5: goblin_archwizard 2: goblin_wizard 3: goblin 4: archwizard_mixin " border="0" data-original-height="430" data-original-width="1550" height="110" src="https://3.bp.blogspot.com/-tdauL-B6j1E/W3f01ltKlzI/AAAAAAAAJIs/Q5-cIY6AcGU_IXXasdzPWec7cxN061WrwCLcBGAs/s400/Screenshot%2Bfrom%2B2018-08-18%2B12-27-37.png" title="Prototype loading" width="400" />


More importantly, builders can now create, save and load prototypes in the database for themselves and other builders to use. The prototypes can be tagged and searched as a joint resource. Builders can also lock prototypes if others are not to be able to read or use them to spawn things. Developers can still supply module-based "read-only" prototypes (for use as starting points or examples to their Builders, for example).

Found 1 match. (Warning: creating a prototype will overwrite the current prototype!) ____________________________________________________________________________________ Actions: examine <num> | create prototype from object <num> Back (index) | Quit 1: Griatch(#1) | create prototype from object Back (index) | Quit 1: Griatch(#1)" /> | create prototype from object Back (index) | Quit 1: Griatch(#1)" border="0" data-original-height="392" data-original-width="1280" height="122" src="https://2.bp.blogspot.com/-XFm3KqhLBwE/W3f93upyVLI/AAAAAAAAJJA/eZDWGPHM93MWvt2T5W8Ytr4cHstGg8iXACLcBGAs/s400/Screenshot%2Bfrom%2B2018-08-18%2B13-04-48.png" title="Load prototype from object" width="400" />


You can now also use the menu to search for and create a new Prototype based on an existing object (if you have access to do so). This makes it quick to start up a new prototype and tweak it for spawning other similar objects. Of course you could spawn temporary objects without saving the prototype as well.

The Typeclass defines what 'type' of object this is - the actual working code to use. All spawned objects must have a typeclass. If not given here, the typeclass must be set in one of the prototype's parents. [No typeclass set] ______________________________________________________________________________________________________________________________________________ Back (prototype-parent) | Forward (key) | Index | Validate prototype | Quit 1: evennia.contrib.tutorial_world.mob.Mob 7: evennia.contrib.tutorial_world.objects.TutorialObject 2: evennia.contrib.tutorial_world.objects.Climbable 8: evennia.contrib.tutorial_world.objects.Weapon 3: evennia.contrib.tutorial_world.objects.CrumblingWall 9: evennia.contrib.tutorial_world.objects.WeaponRack 4: evennia.contrib.tutorial_world.objects.LightSource 10: evennia.contrib.tutorial_world.rooms.BridgeRoom 5: evennia.contrib.tutorial_world.objects.Obelisk current: (1/3) 6: evennia.contrib.tutorial_world.objects.Readable next page


Builders will likely not know which typeclasses are available in the code base. There are new a few ways to list them. The menu display makes use of Evennia 0.8's new EvMenu improvements, which allows for automatically creating multi-page listings (see example above).

There is also a new switch to the typeclass command, /list, that will list all available typeclasses outside of the OLC.

Protfuncs

Another new feature are Protfuncs. Similarly to how Inlinefuncs allows for calling for the result of a function call inside a text string, Protfuncs allows for calling functions inside a prototype's values. It's given on the form $funcname(arguments), where arguments could themselves contain one or more nested Protfuncs.

As with other such systems in Evennia, only Python functions in a specific module or modules (given by settings) are available for use as Protfuncs in-game. A bunch of default ones are included out of the box. Protfuncs are called at the time of spawning. So for example, you could set the Attribute

Strength = $randint(5, 20)

to automatically spawn objects with a random strength between 5 and 20.

prototype-key: goblin, -tags: [], -locks: spawn:all();edit:all() -desc: Built from goblin prototype-parent: None key: goblin aliases: monster, mob attrs: desc = You see nothing special. strength = $randint(5,20) agility = $random(6,20) magic = 0 tags: mob (category: None) locks: call:true();control:id(1) or perm(Admin);delete:id(1) or perm(Admin);edit:perm(Admin);examine:perm(Builder);get:all();puppet:pperm(Developer) ;tell:perm(Admin);view:all() location: #2 home: #2 No validation errors found. (but errors could still happen at spawn-time) ______________________________________________________________________________________________________________________________________________ Actions: examine <num> | remove <num> Back (index) | Validate prototype | Quit 1: Spawn in prototype's defined location (#2) 2: Spawn in Griatch's location (Limbo) 3: Spawn in Griatch's inventory 4: Update 2 existing objects with this prototype | remove Back (index) | Validate prototype | Quit 1: Spawn in prototype's defined location (#2) 2: Spawn in Griatch's location (Limbo) 3: Spawn in Griatch's inventory 4: Update 2 existing objects with this prototype" /> | remove Back (index) | Validate prototype | Quit 1: Spawn in prototype's defined location (#2) 2: Spawn in Griatch's location (Limbo) 3: Spawn in Griatch's inventory 4: Update 2 existing objects with this prototype" border="0" data-original-height="1052" data-original-width="1554" height="270" src="https://1.bp.blogspot.com/-qHeRpEQTEzU/W3gFpJ9ge5I/AAAAAAAAJJw/-PVcelDk2CsIW1nJwWvMjazcJ9LyKx5pgCLcBGAs/s400/Screenshot%2Bfrom%2B2018-08-18%2B13-40-04.png" title="Spawning screen" width="400" />


When spawning, the olc will validate the prototype and run tests on any Protfunc used. For convenience you can override the spawn-location if any is hard-coded in the prototype.

https://pastebin.com/raw/K0a1z23h


The system will also allow you to try updating existing objects created from the same-named prototype earlier. It will sample the existing objects and calculate a 'diff' to apply. This is bit is still a bit iffy, with edge cases that still needs fixing.

Current status

The OLC is currently in the develop branch of Evennia - what will soon(ish) merge to become Evennia 0.8.

It's a pretty big piece of code and as such it's still a bit unstable and there are edge cases and display issues to fix. But it would be great with more people trying it out and reporting errors so the childhood issues can be ironed out before release!




Building Image: Released as Creative Commons here

19 Aug 2018 10:40am GMT

Evennia: Inline building in upcoming Evennia 0.8


Evennia, the Python MUD-server game development kit, is slowly creeping closer to its 0.8 release.

In our development branch I've just pushed the first version of the new OLC (OnLine Creator) system. This is a system to allow builders (who may have limited coding knowledge) to customize and spawn new in-game objects more easily without code access. It's started with the olc command in-game. This is a visual system for manipulating Evennia Prototypes.


Briefly on Prototypes

The Prototype is an Evennia concept that has been around a good while. The prototype is a Python dictionary that holds specific keys with values representing properties on a game object. Here's an example of a simple prototype:

{"key": "My house",

"typeclass": "typeclasses.houses.MyHouse"}


By passing this dict to the spawner, a new object named "My house" will be created. It will be set up with the given typeclass (a 'typeclass' is, in Evennia lingo, a Python class with a database backend). A prototype can specify all aspects of an in-game object - its attributes (like description and other game-specific properties), tags, aliases, location and so on. Prototypes also support inheritance - so you can expand on an existing template without having to add everything fresh every time.

There are two main reasons for the Prototypes existing in Evennia:

What's new

As said, Prototypes have been around for a good while in Evennia. But in the past they were either manually entered directly as a dict on the command line, or created in code and read from a Python module. The former solution is cumbersome and requires that you know how to build a proper-syntax Python dictionary. The latter requires server code access, making them less useful to builders than they could be.

Note: If you are visually impaired, each image is also a link to a text-only version.

OLC index


In Evennia 0.8, while you can still insert the Prototype as a raw dict, spawn/menu or the new olc command opens a new menu-driven interface.

Select a prototype to load. This will replace any prototype currently being edited! ___________________________________________________________________________________________________ Select with <num>. Other actions: examine <num> | delete <num> Back (index) | Validate prototype | Quit 1: goblin_archer 5: goblin_archwizard 2: goblin_wizard 3: goblin 4: archwizard_mixin . Other actions: examine | delete Back (index) | Validate prototype | Quit 1: goblin_archer 5: goblin_archwizard 2: goblin_wizard 3: goblin 4: archwizard_mixin " />. Other actions: examine | delete Back (index) | Validate prototype | Quit 1: goblin_archer 5: goblin_archwizard 2: goblin_wizard 3: goblin 4: archwizard_mixin " border="0" data-original-height="430" data-original-width="1550" height="110" src="https://3.bp.blogspot.com/-tdauL-B6j1E/W3f01ltKlzI/AAAAAAAAJIs/Q5-cIY6AcGU_IXXasdzPWec7cxN061WrwCLcBGAs/s400/Screenshot%2Bfrom%2B2018-08-18%2B12-27-37.png" title="Prototype loading" width="400" />


More importantly, builders can now create, save and load prototypes in the database for themselves and other builders to use. The prototypes can be tagged and searched as a joint resource. Builders can also lock prototypes if others are not to be able to read or use them to spawn things. Developers can still supply module-based "read-only" prototypes (for use as starting points or examples to their Builders, for example).

Found 1 match. (Warning: creating a prototype will overwrite the current prototype!) ____________________________________________________________________________________ Actions: examine <num> | create prototype from object <num> Back (index) | Quit 1: Griatch(#1) | create prototype from object Back (index) | Quit 1: Griatch(#1)" /> | create prototype from object Back (index) | Quit 1: Griatch(#1)" border="0" data-original-height="392" data-original-width="1280" height="122" src="https://2.bp.blogspot.com/-XFm3KqhLBwE/W3f93upyVLI/AAAAAAAAJJA/eZDWGPHM93MWvt2T5W8Ytr4cHstGg8iXACLcBGAs/s400/Screenshot%2Bfrom%2B2018-08-18%2B13-04-48.png" title="Load prototype from object" width="400" />


You can now also use the menu to search for and create a new Prototype based on an existing object (if you have access to do so). This makes it quick to start up a new prototype and tweak it for spawning other similar objects. Of course you could spawn temporary objects without saving the prototype as well.

The Typeclass defines what 'type' of object this is - the actual working code to use. All spawned objects must have a typeclass. If not given here, the typeclass must be set in one of the prototype's parents. [No typeclass set] ______________________________________________________________________________________________________________________________________________ Back (prototype-parent) | Forward (key) | Index | Validate prototype | Quit 1: evennia.contrib.tutorial_world.mob.Mob 7: evennia.contrib.tutorial_world.objects.TutorialObject 2: evennia.contrib.tutorial_world.objects.Climbable 8: evennia.contrib.tutorial_world.objects.Weapon 3: evennia.contrib.tutorial_world.objects.CrumblingWall 9: evennia.contrib.tutorial_world.objects.WeaponRack 4: evennia.contrib.tutorial_world.objects.LightSource 10: evennia.contrib.tutorial_world.rooms.BridgeRoom 5: evennia.contrib.tutorial_world.objects.Obelisk current: (1/3) 6: evennia.contrib.tutorial_world.objects.Readable next page


Builders will likely not know which typeclasses are available in the code base. There are new a few ways to list them. The menu display makes use of Evennia 0.8's new EvMenu improvements, which allows for automatically creating multi-page listings (see example above).

There is also a new switch to the typeclass command, /list, that will list all available typeclasses outside of the OLC.

Protfuncs

Another new feature are Protfuncs. Similarly to how Inlinefuncs allows for calling for the result of a function call inside a text string, Protfuncs allows for calling functions inside a prototype's values. It's given on the form $funcname(arguments), where arguments could themselves contain one or more nested Protfuncs.

As with other such systems in Evennia, only Python functions in a specific module or modules (given by settings) are available for use as Protfuncs in-game. A bunch of default ones are included out of the box. Protfuncs are called at the time of spawning. So for example, you could set the Attribute

Strength = $randint(5, 20)

to automatically spawn objects with a random strength between 5 and 20.

prototype-key: goblin, -tags: [], -locks: spawn:all();edit:all() -desc: Built from goblin prototype-parent: None key: goblin aliases: monster, mob attrs: desc = You see nothing special. strength = $randint(5,20) agility = $random(6,20) magic = 0 tags: mob (category: None) locks: call:true();control:id(1) or perm(Admin);delete:id(1) or perm(Admin);edit:perm(Admin);examine:perm(Builder);get:all();puppet:pperm(Developer) ;tell:perm(Admin);view:all() location: #2 home: #2 No validation errors found. (but errors could still happen at spawn-time) ______________________________________________________________________________________________________________________________________________ Actions: examine <num> | remove <num> Back (index) | Validate prototype | Quit 1: Spawn in prototype's defined location (#2) 2: Spawn in Griatch's location (Limbo) 3: Spawn in Griatch's inventory 4: Update 2 existing objects with this prototype | remove Back (index) | Validate prototype | Quit 1: Spawn in prototype's defined location (#2) 2: Spawn in Griatch's location (Limbo) 3: Spawn in Griatch's inventory 4: Update 2 existing objects with this prototype" /> | remove Back (index) | Validate prototype | Quit 1: Spawn in prototype's defined location (#2) 2: Spawn in Griatch's location (Limbo) 3: Spawn in Griatch's inventory 4: Update 2 existing objects with this prototype" border="0" data-original-height="1052" data-original-width="1554" height="270" src="https://1.bp.blogspot.com/-qHeRpEQTEzU/W3gFpJ9ge5I/AAAAAAAAJJw/-PVcelDk2CsIW1nJwWvMjazcJ9LyKx5pgCLcBGAs/s400/Screenshot%2Bfrom%2B2018-08-18%2B13-40-04.png" title="Spawning screen" width="400" />


When spawning, the olc will validate the prototype and run tests on any Protfunc used. For convenience you can override the spawn-location if any is hard-coded in the prototype.

https://pastebin.com/raw/K0a1z23h


The system will also allow you to try updating existing objects created from the same-named prototype earlier. It will sample the existing objects and calculate a 'diff' to apply. This is bit is still a bit iffy, with edge cases that still needs fixing.

Current status

The OLC is currently in the develop branch of Evennia - what will soon(ish) merge to become Evennia 0.8.

It's a pretty big piece of code and as such it's still a bit unstable and there are edge cases and display issues to fix. But it would be great with more people trying it out and reporting errors so the childhood issues can be ironed out before release!




Building Image: Released as Creative Commons here

19 Aug 2018 10:40am GMT

Evennia: Evennia in pictures

This article describes the MU* development system Evennia using pictures!

This article was originally written for Optional Realities.
Since it is no longer available to read on OR, I'm reposting it in full here.


Figure 1: The parts of the Evennia library



Evennia is a game development library. What you see in Figure 1 is the part you download from us. This will not run on its own, we will soon initialize the missing "jigsaw puzzle" piece on the left. But first let's look at what we've got.


Looking at Figure 1 you will notice that Evennia internally has two components, the Portal and the Server. These will run as separate processes.


The Portal tracks all connections to the outside world and understands Telnet protocols, websockets, SSH and so on. It knows nothing about the database or the game state. Data sent between the Portal and the Server is protocol-agnostic, meaning the Server sends/receives the same data regardless of how the user is connected. Hiding behind the Portal also means that the Server can be completely rebooted without anyone getting disconnected.


The Server is the main "mud driver" and handles everything related to the game world and its database. It's asynchronous and uses Twisted. In the same process of the Server is also the Evennia web server component that serves the game's website. That the Server and webserver are accessing the database in the same process allows for a consistent game state without any concerns for caching or race condition issues.


Now, let's get a game going. We'll call it mygame. Original, isn't it?


Figure 2: The full setup for mygame


After installing evennia you will have the evennia command available. Using this you create a game directory - the darker grey piece in Figure 2 that was missing previously. This is where you will create your dream game!


During initialization, Evennia will create Python module templates in mygame/ and link up all configurations to make mygame a fully functioning, if empty, game, ready to start extending. Two more commands will create your database and then start the server. From this point on, mygame is up and running and you can connect to your new game with telnet on localhost:4000 or by pointing your browser to http://localhost:4001.


Now, our new mygame world needs Characters, locations, items and more! These we commonly refer to as game entities. Let's see how Evennia handles those.


Figure 3: The Database Abstraction of Evennia entities

Evennia is fully persistent and abstracts its database in Python using Django. The database tables are few and generic, each represented by a single Python class. As seen in Figure 3, the example ObjectDB Python class represents one database table. The properties on the class are the columns (fields) of the table. Each row is an instance of the class (one entity in the game).


Among the example columns shown is the key (name) of the ObjectDB entity as well as a Foreign key-relationship for its current "location". From the above we can see that Trigger is in the Dungeon, carrying his trusty crossbow Old Betsy!


The db_typeclass_path is an important field. This is a python-style path and tells Evennia which subclass of ObjectDB is actually representing this entity.


Figure 4: Inheriting classes to customize entities


In Figure 4 we see the (somewhat simplified) Python class inheritance tree that you as an Evennia developer will see, along with the three instanced entities.

ObjectDB represents stuff you will actually see in-game and its child classes implement all the handlers, helper code and the hook methods that Evennia makes use of. In your mygame/ folder you just import these and overload the things you want to modify. In this way, the Crossbow is modified to do the stuff only crossbows can do and CastleRoom adds whatever it is that is special about rooms in the castle.


When creating a new entity in-game, a new row will automatically be created in the database table and then "Trigger" will appear in-game! If we, in code, search the database for Trigger, we will get an instance of the Character class back - a Python object we can work with normally.


Looking at this you may think that you will be making a lot of classes for every different object in the game. Your exact layout is up to you but Evennia also offers other ways to customize each individual object, as exemplified by Figure 5.


Figure 5: Adding persistent Attributes to a game entity.


The Attribute is another class directly tied to the database behind the scenes. Each Attribute basically has a key, a value and a ForeignKey relation to another ObjectDB. An Attribute serializes Python constructs into the database, meaning you can store basically any valid Python, like the dictionary of skills seen in Figure 5. The "strength" and "skills" Attributes will henceforth be reachable directly from the Trigger object. This (and a few other resources) allow you to create individualized entities while only needing to create classes for those that really behave fundamentally different.


Figure 6: Sessions, Players and Objects



Trigger is most likely played by a human. This human connects to the game via one or more Sessions, one for each client they connect with. Their account on mygame is represented by a PlayerDB entity. The PlayerDB holds the password and other account info but has no existence in the game world. Through the PlayerDB entity, Sessions can control ("puppet") one or more ObjectDB entities in-game.


In Figure 6, a user is connected to the game with three Sessions simultaneously. They are logged in to their Player account Richard. Through these Sessions they are simultaneously puppeting the in-game entities Trigger and Sir Hiss. Evennia can be configured to allow or disallow a range of different gaming styles like this.


Now, for users to be able to control their game entities and actually play the game, they need to be able to send Commands.


Figure 7: Commands are Python classes too




Commands represent anything a user can input actively to the game, such as the look command, get, quit, emote and so on.


Each Command handles both argument parsing and execution. Since each Command is described with a normal Python class, it means that you can implement parsing once and then just have the rest of your commands inherit the effect. In Figure 7, the DIKUCommand parent class implements parsing of all the syntax common for all DIKU-style commands so CmdLook and others won't have to.


Figure 8: Commands are grouped together in sets and always associated with game entities.




Commands in Evennia are always joined together in Command Sets. These are containers that can hold many Command instances. A given Command class can contribute Command instances to any number of Command Sets. These sets are always associated with game entities. In Figure 8, Trigger has received a Command Set with a bunch of useful commands that he (and by extension his controlling Player) can now use.


Figure 9: Command Sets can affect those around them



Trigger's Command Set is only available to himself. In Figure 8 we put a Command Set with three commands on the Dungeon room. The room itself has no use for commands but we configure this set to affect those inside it instead. Note that we let these be different versions of these commands (hence the different color)! We'll explain why below.

Figure 10: The name Command "Set" is not just a name



Command Sets can be dynamically (and temporarily) merged together in a similar fashion as Set Theory, except the merge priority can be customized. In Figure 10 we see a Union-type merger where the Commands from Dungeon of the same name temporarily override the commands from Trigger. While in the Dungeon, Trigger will be using this version of those commands. When Trigger leaves, his own Command Set will be restored unharmed.

Why would we want to do this? Consider for example that the dungeon is in darkness. We can then let the Dungeon's version of the look command only show the contents of the room if Trigger is carrying a light source. You might also not be able to easily get things in the room without light - you might even be fumbling randomly in your inventory!


Any number of Command Sets can be merged on the fly. This allows you to implement multiple overlapping states (like combat in a darkened room while intoxicated) without needing huge if statements for every possible combination. The merger is non-destructive, so you can remove cmdsets to get back previous states as needed.




… And that's how many illustrations I have the stamina to draw at this time. Hopefully this quick illustrated dive into Evennia helps to clarify some of the basic features of the system!

19 Aug 2018 9:30am GMT

Evennia: Evennia in pictures

This article describes the MU* development system Evennia using pictures!

This article was originally written for Optional Realities.
Since it is no longer available to read on OR, I'm reposting it in full here.


Figure 1: The parts of the Evennia library



Evennia is a game development library. What you see in Figure 1 is the part you download from us. This will not run on its own, we will soon initialize the missing "jigsaw puzzle" piece on the left. But first let's look at what we've got.


Looking at Figure 1 you will notice that Evennia internally has two components, the Portal and the Server. These will run as separate processes.


The Portal tracks all connections to the outside world and understands Telnet protocols, websockets, SSH and so on. It knows nothing about the database or the game state. Data sent between the Portal and the Server is protocol-agnostic, meaning the Server sends/receives the same data regardless of how the user is connected. Hiding behind the Portal also means that the Server can be completely rebooted without anyone getting disconnected.


The Server is the main "mud driver" and handles everything related to the game world and its database. It's asynchronous and uses Twisted. In the same process of the Server is also the Evennia web server component that serves the game's website. That the Server and webserver are accessing the database in the same process allows for a consistent game state without any concerns for caching or race condition issues.


Now, let's get a game going. We'll call it mygame. Original, isn't it?


Figure 2: The full setup for mygame


After installing evennia you will have the evennia command available. Using this you create a game directory - the darker grey piece in Figure 2 that was missing previously. This is where you will create your dream game!


During initialization, Evennia will create Python module templates in mygame/ and link up all configurations to make mygame a fully functioning, if empty, game, ready to start extending. Two more commands will create your database and then start the server. From this point on, mygame is up and running and you can connect to your new game with telnet on localhost:4000 or by pointing your browser to http://localhost:4001.


Now, our new mygame world needs Characters, locations, items and more! These we commonly refer to as game entities. Let's see how Evennia handles those.


Figure 3: The Database Abstraction of Evennia entities

Evennia is fully persistent and abstracts its database in Python using Django. The database tables are few and generic, each represented by a single Python class. As seen in Figure 3, the example ObjectDB Python class represents one database table. The properties on the class are the columns (fields) of the table. Each row is an instance of the class (one entity in the game).


Among the example columns shown is the key (name) of the ObjectDB entity as well as a Foreign key-relationship for its current "location". From the above we can see that Trigger is in the Dungeon, carrying his trusty crossbow Old Betsy!


The db_typeclass_path is an important field. This is a python-style path and tells Evennia which subclass of ObjectDB is actually representing this entity.


Figure 4: Inheriting classes to customize entities


In Figure 4 we see the (somewhat simplified) Python class inheritance tree that you as an Evennia developer will see, along with the three instanced entities.

ObjectDB represents stuff you will actually see in-game and its child classes implement all the handlers, helper code and the hook methods that Evennia makes use of. In your mygame/ folder you just import these and overload the things you want to modify. In this way, the Crossbow is modified to do the stuff only crossbows can do and CastleRoom adds whatever it is that is special about rooms in the castle.


When creating a new entity in-game, a new row will automatically be created in the database table and then "Trigger" will appear in-game! If we, in code, search the database for Trigger, we will get an instance of the Character class back - a Python object we can work with normally.


Looking at this you may think that you will be making a lot of classes for every different object in the game. Your exact layout is up to you but Evennia also offers other ways to customize each individual object, as exemplified by Figure 5.


Figure 5: Adding persistent Attributes to a game entity.


The Attribute is another class directly tied to the database behind the scenes. Each Attribute basically has a key, a value and a ForeignKey relation to another ObjectDB. An Attribute serializes Python constructs into the database, meaning you can store basically any valid Python, like the dictionary of skills seen in Figure 5. The "strength" and "skills" Attributes will henceforth be reachable directly from the Trigger object. This (and a few other resources) allow you to create individualized entities while only needing to create classes for those that really behave fundamentally different.


Figure 6: Sessions, Players and Objects



Trigger is most likely played by a human. This human connects to the game via one or more Sessions, one for each client they connect with. Their account on mygame is represented by a PlayerDB entity. The PlayerDB holds the password and other account info but has no existence in the game world. Through the PlayerDB entity, Sessions can control ("puppet") one or more ObjectDB entities in-game.


In Figure 6, a user is connected to the game with three Sessions simultaneously. They are logged in to their Player account Richard. Through these Sessions they are simultaneously puppeting the in-game entities Trigger and Sir Hiss. Evennia can be configured to allow or disallow a range of different gaming styles like this.


Now, for users to be able to control their game entities and actually play the game, they need to be able to send Commands.


Figure 7: Commands are Python classes too




Commands represent anything a user can input actively to the game, such as the look command, get, quit, emote and so on.


Each Command handles both argument parsing and execution. Since each Command is described with a normal Python class, it means that you can implement parsing once and then just have the rest of your commands inherit the effect. In Figure 7, the DIKUCommand parent class implements parsing of all the syntax common for all DIKU-style commands so CmdLook and others won't have to.


Figure 8: Commands are grouped together in sets and always associated with game entities.




Commands in Evennia are always joined together in Command Sets. These are containers that can hold many Command instances. A given Command class can contribute Command instances to any number of Command Sets. These sets are always associated with game entities. In Figure 8, Trigger has received a Command Set with a bunch of useful commands that he (and by extension his controlling Player) can now use.


Figure 9: Command Sets can affect those around them



Trigger's Command Set is only available to himself. In Figure 8 we put a Command Set with three commands on the Dungeon room. The room itself has no use for commands but we configure this set to affect those inside it instead. Note that we let these be different versions of these commands (hence the different color)! We'll explain why below.

Figure 10: The name Command "Set" is not just a name



Command Sets can be dynamically (and temporarily) merged together in a similar fashion as Set Theory, except the merge priority can be customized. In Figure 10 we see a Union-type merger where the Commands from Dungeon of the same name temporarily override the commands from Trigger. While in the Dungeon, Trigger will be using this version of those commands. When Trigger leaves, his own Command Set will be restored unharmed.

Why would we want to do this? Consider for example that the dungeon is in darkness. We can then let the Dungeon's version of the look command only show the contents of the room if Trigger is carrying a light source. You might also not be able to easily get things in the room without light - you might even be fumbling randomly in your inventory!


Any number of Command Sets can be merged on the fly. This allows you to implement multiple overlapping states (like combat in a darkened room while intoxicated) without needing huge if statements for every possible combination. The merger is non-destructive, so you can remove cmdsets to get back previous states as needed.




… And that's how many illustrations I have the stamina to draw at this time. Hopefully this quick illustrated dive into Evennia helps to clarify some of the basic features of the system!

19 Aug 2018 9:30am GMT

Kushal Das: Aadhaar, the mass surveillance system

If you are following me on Twitter, you have already seen a lot of (re)tweets related to Aadhaar. For the people first time hearing this term, it is a 12 digit unique identification number provided by the Unique Identification Authority of India (UIDAI). It is also the world's largest bio-metric ID system. It is supposed to be a voluntary service.

From the very beginning, this project tried to hide the details from the Indian citizens. Let it be privacy advocates or security researchers or human rights activists, everyone predicted that this will become a monster, a mass surveillance system, a tool of choice of the power hungry dictators.

Like any other complex system, the majority of the people only see the advertisements from the government and completely miss all the problems and horror stories this project is creating. Here are a few links below for the interested people to read.

Neither my wife, nor our daughter has an Aadhaar (I also don't have one), that means Py (our daughter) did not get admission to any school last year.

Whenever security researchers or journalists tried to report on the project, the UIDAI tried to hide behind denials and police complaints against the journalists or researchers. There are various reports on how one can get access (both read/write) to the actual production database with as little as $10-30. We now have examples of terrorist organizations having access to the same database. The UIDAI kept telling how this is an unhackable technology and for security they have a 13 feet wall outside of the data center which in turn will keep all hackers away.

They have already build 360 degree databases on top of Aadhaar, and now they are trying to link DNA to the same system.

The current government of India tried their level best to argue in the Supreme Court of India to tell that Indians don't have any rights to privacy. But, thankfully they failed in this effort, and the Supreme Court ruled privacy as a fundamental right. We are now waiting for the judgment on the Aadhaar (which will hopefully come out in the next few weeks).

Meanwhile, the evil nexus is pushing down Aadhaar to the throats of the Indian citizens and Pakistani spies and gods.

A few days ago, in an event in Jaipur, they asked Edward Snowden the following question.

How big of an issue is privacy?

The answer started with from where that argument comes from.

The answer is that Nazi Germany. The nazi minister of propaganda Joseph Goebbels did this. Because he was trying to change the conversation away from "What are your rights?" and "What evidences must the government show?" to violet them, to intrude into your private life and instead said "Why do you need your rights?", "How can you justify your rights?", "Isn't strange that you are invoking your rights? Isn't that unusual?". But, in a free society this is the opposite of the way it is supposed to work. We don't need to explain why you have a right. You don't need to explain why it is valuable, why you need it. It is for the government to explain why you don't deserve it. They go to a court, they show that you are a criminal. This is increasingly falling out of favor, because the governments and companies think that it is inefficient. It is too much work. Life would be easier, life would be more convenient for them, life would be more profitable for them if we didn't have any rights at all.

But, privacy isn't about something to hide, privacy is about something to protect. And that is the very concept of liberty. It is the idea that there can be some part of you, of your life, of your ideas that belong to you, not to society. And you get to make the decision about who you share that with. -- Edward Snowden

Why are we reading this in your blog?

This might a question for many of you. Why are reading this in a blog post or in a planet? Because we, the people with the knowledge of technology are also part of these evil plans. We now know about many private companies taking part with their local government to build 360 degree profiles, to track the citizens and to run the mass surveillance systems. For example, related to Aadhaar, for the last 4 years, Google silently pushed the Aadhaar support phone number (which now UIDAI is trying to stay away from) to every Google Android phone in India. When they got caught red handed, they claimed that they did it inadvertently. Finacle software by Infosys denies creation of bank accounts without Aadhaar. Microsoft is working to link Skype with Aadhaar. Bill Gates is trying to push the idea that Aadhaar is all good, and does not have any issues.

What can you do?

You can start by educating yourself first. Read more about the technologies which controls our lives. Have doubt about the things and try to understand how they actually work. Write about them, ask questions to the people in power. Talk about the issues to your friends and family.

This is not gong to be an easy task, but, we all should keep fighting back to make sure of a better future for our next generation.

19 Aug 2018 6:33am GMT

Kushal Das: Aadhaar, the mass surveillance system

If you are following me on Twitter, you have already seen a lot of (re)tweets related to Aadhaar. For the people first time hearing this term, it is a 12 digit unique identification number provided by the Unique Identification Authority of India (UIDAI). It is also the world's largest bio-metric ID system. It is supposed to be a voluntary service.

From the very beginning, this project tried to hide the details from the Indian citizens. Let it be privacy advocates or security researchers or human rights activists, everyone predicted that this will become a monster, a mass surveillance system, a tool of choice of the power hungry dictators.

Like any other complex system, the majority of the people only see the advertisements from the government and completely miss all the problems and horror stories this project is creating. Here are a few links below for the interested people to read.

Neither my wife, nor our daughter has an Aadhaar (I also don't have one), that means Py (our daughter) did not get admission to any school last year.

Whenever security researchers or journalists tried to report on the project, the UIDAI tried to hide behind denials and police complaints against the journalists or researchers. There are various reports on how one can get access (both read/write) to the actual production database with as little as $10-30. We now have examples of terrorist organizations having access to the same database. The UIDAI kept telling how this is an unhackable technology and for security they have a 13 feet wall outside of the data center which in turn will keep all hackers away.

They have already build 360 degree databases on top of Aadhaar, and now they are trying to link DNA to the same system.

The current government of India tried their level best to argue in the Supreme Court of India to tell that Indians don't have any rights to privacy. But, thankfully they failed in this effort, and the Supreme Court ruled privacy as a fundamental right. We are now waiting for the judgment on the Aadhaar (which will hopefully come out in the next few weeks).

Meanwhile, the evil nexus is pushing down Aadhaar to the throats of the Indian citizens and Pakistani spies and gods.

A few days ago, in an event in Jaipur, they asked Edward Snowden the following question.

How big of an issue is privacy?

The answer started with from where that argument comes from.

The answer is that Nazi Germany. The nazi minister of propaganda Joseph Goebbels did this. Because he was trying to change the conversation away from "What are your rights?" and "What evidences must the government show?" to violet them, to intrude into your private life and instead said "Why do you need your rights?", "How can you justify your rights?", "Isn't strange that you are invoking your rights? Isn't that unusual?". But, in a free society this is the opposite of the way it is supposed to work. We don't need to explain why you have a right. You don't need to explain why it is valuable, why you need it. It is for the government to explain why you don't deserve it. They go to a court, they show that you are a criminal. This is increasingly falling out of favor, because the governments and companies think that it is inefficient. It is too much work. Life would be easier, life would be more convenient for them, life would be more profitable for them if we didn't have any rights at all.

But, privacy isn't about something to hide, privacy is about something to protect. And that is the very concept of liberty. It is the idea that there can be some part of you, of your life, of your ideas that belong to you, not to society. And you get to make the decision about who you share that with. -- Edward Snowden

Why are we reading this in your blog?

This might a question for many of you. Why are reading this in a blog post or in a planet? Because we, the people with the knowledge of technology are also part of these evil plans. We now know about many private companies taking part with their local government to build 360 degree profiles, to track the citizens and to run the mass surveillance systems. For example, related to Aadhaar, for the last 4 years, Google silently pushed the Aadhaar support phone number (which now UIDAI is trying to stay away from) to every Google Android phone in India. When they got caught red handed, they claimed that they did it inadvertently. Finacle software by Infosys denies creation of bank accounts without Aadhaar. Microsoft is working to link Skype with Aadhaar. Bill Gates is trying to push the idea that Aadhaar is all good, and does not have any issues.

What can you do?

You can start by educating yourself first. Read more about the technologies which controls our lives. Have doubt about the things and try to understand how they actually work. Write about them, ask questions to the people in power. Talk about the issues to your friends and family.

This is not gong to be an easy task, but, we all should keep fighting back to make sure of a better future for our next generation.

19 Aug 2018 6:33am GMT

18 Aug 2018

feedPlanet Python

Karim Elghamrawy: Python Lambdas Explained (With Examples)

In this article, I will teach you exactly what a python lambda is. As a matter of fact if you know what functions are and how to define functions in Python then you already know what a lambda is. A Python lambda is just a Python function. But may be like a special type of [...]

The post Python Lambdas Explained (With Examples) appeared first on Afternerd.

18 Aug 2018 11:25pm GMT

Karim Elghamrawy: Python Lambdas Explained (With Examples)

In this article, I will teach you exactly what a python lambda is. As a matter of fact if you know what functions are and how to define functions in Python then you already know what a lambda is. A Python lambda is just a Python function. But may be like a special type of [...]

The post Python Lambdas Explained (With Examples) appeared first on Afternerd.

18 Aug 2018 11:25pm GMT

Codementor: How and why I built IVR enabled AWS cloud environment

Cloud infra management and automation using IVR system.

18 Aug 2018 6:02pm GMT

Codementor: How and why I built IVR enabled AWS cloud environment

Cloud infra management and automation using IVR system.

18 Aug 2018 6:02pm GMT

Marc Richter: Create your own Telegram bot with Django on Heroku – Part 4 – pull vs. push method

In the previous part of this series, we started to get familiar with telepot, a Python module to interact with Telegram bots and had a short look at how the Telegram bot API is providing messages as JSON structures.

Today we will talk about the Webhook-method (push) instead of the previously introduced getUpdate-method (pull).

What's the difference in getUpdates and a webhook? 🤔

The Default: getUpdates (pull) 📩

By default (when a new bot is registered), a bot is configured to cache messages until they are actively fetched by some request (getUpdates). This is also called "pull method".
When a message is sent to the bot while it is configured for this mode, it is cached in some kind of buffer within the infrastructure of Telegram (until they are delivered to the bot, but not longer than 24 hours, as described in the docs).

While this mode offers an easy start because you do not need to have any infrastructure or code prepared to create the bot and start exchanging messages with it, it has some downsides:

There might be a good reason to keep it like this nevertheless, like if you do not have a way to meet the technical requirements for the alternative method which is a webhook, like that you do not have a way to expose an interface to the public internet for example or if you want to prepare and test some code before firing up a deployed service. But most serious bots probably will change this to the alternate webhook (push) method sooner or later.

GET and POST HTTP methods

Before we look at the webhook push method, let's first refresh some basics about how HTTP works. This will only cover some basics to better understand what is going on in the next sections. There is more to say and explain about the methods shown here and there are more methods to the protocol as well; not because of no reason there are 656 pages to O'Reilly's book "HTTP: The Definitive Guide". Since obviously, this is beyond the scope of this article, if you want to know more, please help yourself.

GET

When you are surfing the web with your favorite browser, GET requests are sent by that browser to the web servers providing your favorite websites, asking for a specific resource (like "

GET / HTTP/1.1

" is asking for the page index). To fetch the article you are currently reading, your browser most certainly sent something like this to my web server:

"GET /create-your-own-telegram-bot-with-django-on-heroku-part-3/?somethingIWantToSet=HubbaHubba HTTP/1.1"

followed by a myriad of additional requests for each asset (like CSS files, JS, images, …). The web server then does it's magic and delivers the associated content in its answer; classically: A webpage.

The "?" in that request string separates the URL from an optional, additional section in that GET request: The query-string (

?somethingIWantToSet=HubbaHubba

). Even though GET is a method to request data from an HTTP resource, data can be sent to it, this way. Since this becomes quite unhandy soon and everyone can easily read and manipulate the content of the data sent to the server, this is only recommendable for really short things, like selecting a bright or dark theme for the page you are requesting like this:

http://www.my-great-site.com/index.php?theme=dark

To sum this up: GET is used to request data from a specified resource.

POST

Nowadays, most websites are not delivering just static content anymore. Instead, you can interact with them. You can, for example, send an email by filling a contact form or publish an answer to some forum's thread or similar. To make this work, you need to send data to the remote website instead of sending a request to the remote web server to have something sent back to you. The method in the HTTP protocol enabling you to do so is called POST.

If, for example, somebody is sending me an email using the contact form on my website, I can find a request like this in the access logs of my server:

162.158.89.81 - - [18/Aug/2018:13:11:30 +0200] "POST /contact/ HTTP/1.1" 200 21185 "https://www.marc-richter.info/contact/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:61.0) Gecko/20100101 Firefox/61.0"

There is some detail in that, like the HTTP status (

200

) indicating that the request was successful, the referer of the request (

https://www.marc-richter.info/contact/

) telling where the user came from, some details about the user agent (

Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:61.0) Gecko/20100101 Firefox/61.0

) telling what OS and Webbrowser the visitor was using, etc. But the important part for us is:

POST /contact/ HTTP/1.1

This looks pretty much the same as the GET request is, isn't it?

… wait a moment: Has somebody sent you an empty mail?

No. In fact, I sent it to myself with the following content:

To: Marc Richter
Name: Marc Richter
Email: noreply@marc-richter.info
Subject: Testmail

Message:
Hi Marc,
great site!
BR

The reason why you do not see this in the logline before is not that I cut that part. It doesn't show up there, because unlike with GET requests, the POST request is initiating the communication with calling for the destination path only, without any additional in-URL data visible. The content/payload of the request is not part of the initializing requests URL, but is appended to the header of the request.

This is not only more secure to exchange data with the server, but also it is more convenient to exchange bigger amounts of data this way. Imagine, that all data which is ever sent to your favorite social network can be read by any admin who can read the log files of that server … or your employer since you are routed over his proxy servers … 😱

To understand webhooks and to sum this up: Telegram bots are sending the JSON data we already saw in the previous part of this series to any URL we configure to them, using POST requests. The JSON data is then part of these request's payload and can easily be extracted by Django (or any other application).

Again: If you want to get additional details on this, look them up for yourself; to understand how webhooks are working, what I explained so far is already sufficient to follow this article.

Webhook (push) ⚓

Now that we know what POST is and how it is working, let's talk about how webhooks work.
There is a section in the official bot guide of Telegram, which explains how Webhooks work in three different levels of details; if you want to understand more details about Webhooks, I recommend reading at least the shortest version of it; I think it really is a great start for anybody not familiar with the concept of a webhook.

In my own words: A webhook is primarily a normal HTTP(S) socket, provided by a web server - like software listening to it for HTTP requests. Some piece of software is picking up everything which is sent towards this interface and applies some pre-defined logic on the data received. What logic applies is defined by your code, picking up that data.
This happens in real-time and is aware of the delivery result: If the message could not be forwarded to the defined interface, maybe because the server is down or there are temporary connection issues, Telegram caches the messages just like it does with the getUpdate pull method and retries to deliver it again after some time. As soon as the message was handed over to your interface successfully, Telegram forgets about the message and will never re-transmit it to your bot again.

Giving webhooks a try

Enough of the boring theory: I bet you are desperate to see this whole stuff in action, don't you? Let's change your bot to webhook-mode and have everything sent to it forwarded to examine what we need to prepare our bot application for. After that, we will switch back to the getUpdates pull method again.

Attention: Please be aware that everything that will be sent to the bot will be forwarded to a 3rd party's web service, I do have no control of. Please make sure you are OK with any data privacy policy of your country and that service before following the next lines, since I do not take responsibility about these things (if you revealed your online banking TAN lists because you felt like your bot could be interested in them or your partner is sending nudes unaware of this change, for example 😉).

First, you need to visit https://webhook.site in your favorite browser. You will be presented a page telling you all the details to utilize a unique webhook which just got created for you automatically without any registration - pretty awesome, isn't it?

Copy that link and click the "Open in a new tab"-link once. That will open a new tab, showing an empty page; but - wait! The welcome page of your webhook will have changed immediately, listing a new GET request to have just arrived. That was your browser opening that empty page.
It works the same with POST or any other HTTP method anybody makes towards that interface.

Let's now configure our bot's webhook to use that URL you just copied as a destination interface for it's webhook. You will need two things:

  1. WEBHOOK_URL: The URL of the webhook you just created
  2. BOT_TOKEN: The Token of your bot, we created in the previous sections of this series and I advised you to take a note on it because we will need that later; now is "later" 😉

In your browser, you now navigate to the following URL after you have crafted the full URL:

https://api.telegram.org/bot{BOT_TOKEN}/setWebhook?url={WEBHOOK_URL}

You need to replace the two "

{}

"-markers with your own data. You should be presented something like this:

{"ok":true,"result":true,"description":"Webhook was set"}

And: You are good to go!
Make sure the page of your webhook is opened in your browser and then, send something to your bot in Telegram. Immediately (!) you should see that new message as a POST request in the "requests" bar on the left of your webhook page. You can tick the "Format JSON" checkbox at the upper right corner of that page to make the JSON data a bit easier to read.
And that's it! This is a preview of how the messages which are sent to your bot will be forwarded to your webhook.

Telegram WebhookTelegram Webhook representation

Go on, play around with that. Have different datatypes (like images, text, vcards) sent to your bot and check out how it will arrive at the webhook.

You can rely on this format will be the same which are about to hit your Python code, soon. Better be prepared for everything ✌

Switching back to getUpdate pull method

When you are done, better switch back to the getUpdate pull method, to not accidentally expose sensitive data to this publically available service.

This can be done, sending another GET request to the bot's API like we used to enable the hook. The only difference is that we won't provide a webhook URL this time:

https://api.telegram.org/bot{BOT_TOKEN}/setWebhook?url=

Success should be indicated by the following message:

{"ok":true,"result":true,"description":"Webhook was deleted"}

Outlook for the next part of the series

So far so good! For this part of the series, that's it again!
We just learned how webhooks are working and how to enable your bot to make use of that method, how to analyze the data received by it on any webhook and how to disable it again, re-enabling getUpdate pull method.

Next time, I will show you how to prepare Heroku to host the Django app we will create in later chapters and already upload a first demo site to it.

I hope you enjoyed this part! Please let me know of all the things you either enjoyed or did not like that much and how I can do it better in the comments.

Marc Richter

Born in 1982, Marc Richter is an IT enthusiastic since 1994. He became addicted when he first put hands on their family's pc and never stopped investigating and exploring new things since then.
He is married to Jennifer Richter and proud father of two wonderful children, Lotta and Linus.
His current professional focus is DevOps and Python development.

An exhaustive bio can be found at this blog post.

Found my articles useful? Maybe you would like to support my efforts and give me a tip then?

Der Beitrag Create your own Telegram bot with Django on Heroku - Part 4 - pull vs. push method erschien zuerst auf Marc Richter's personal site.

18 Aug 2018 12:46pm GMT

Marc Richter: Create your own Telegram bot with Django on Heroku – Part 4 – pull vs. push method

In the previous part of this series, we started to get familiar with telepot, a Python module to interact with Telegram bots and had a short look at how the Telegram bot API is providing messages as JSON structures.

Today we will talk about the Webhook-method (push) instead of the previously introduced getUpdate-method (pull).

What's the difference in getUpdates and a webhook? 🤔

The Default: getUpdates (pull) 📩

By default (when a new bot is registered), a bot is configured to cache messages until they are actively fetched by some request (getUpdates). This is also called "pull method".
When a message is sent to the bot while it is configured for this mode, it is cached in some kind of buffer within the infrastructure of Telegram (until they are delivered to the bot, but not longer than 24 hours, as described in the docs).

While this mode offers an easy start because you do not need to have any infrastructure or code prepared to create the bot and start exchanging messages with it, it has some downsides:

There might be a good reason to keep it like this nevertheless, like if you do not have a way to meet the technical requirements for the alternative method which is a webhook, like that you do not have a way to expose an interface to the public internet for example or if you want to prepare and test some code before firing up a deployed service. But most serious bots probably will change this to the alternate webhook (push) method sooner or later.

GET and POST HTTP methods

Before we look at the webhook push method, let's first refresh some basics about how HTTP works. This will only cover some basics to better understand what is going on in the next sections. There is more to say and explain about the methods shown here and there are more methods to the protocol as well; not because of no reason there are 656 pages to O'Reilly's book "HTTP: The Definitive Guide". Since obviously, this is beyond the scope of this article, if you want to know more, please help yourself.

GET

When you are surfing the web with your favorite browser, GET requests are sent by that browser to the web servers providing your favorite websites, asking for a specific resource (like "

GET / HTTP/1.1

" is asking for the page index). To fetch the article you are currently reading, your browser most certainly sent something like this to my web server:

"GET /create-your-own-telegram-bot-with-django-on-heroku-part-3/?somethingIWantToSet=HubbaHubba HTTP/1.1"

followed by a myriad of additional requests for each asset (like CSS files, JS, images, …). The web server then does it's magic and delivers the associated content in its answer; classically: A webpage.

The "?" in that request string separates the URL from an optional, additional section in that GET request: The query-string (

?somethingIWantToSet=HubbaHubba

). Even though GET is a method to request data from an HTTP resource, data can be sent to it, this way. Since this becomes quite unhandy soon and everyone can easily read and manipulate the content of the data sent to the server, this is only recommendable for really short things, like selecting a bright or dark theme for the page you are requesting like this:

http://www.my-great-site.com/index.php?theme=dark

To sum this up: GET is used to request data from a specified resource.

POST

Nowadays, most websites are not delivering just static content anymore. Instead, you can interact with them. You can, for example, send an email by filling a contact form or publish an answer to some forum's thread or similar. To make this work, you need to send data to the remote website instead of sending a request to the remote web server to have something sent back to you. The method in the HTTP protocol enabling you to do so is called POST.

If, for example, somebody is sending me an email using the contact form on my website, I can find a request like this in the access logs of my server:

162.158.89.81 - - [18/Aug/2018:13:11:30 +0200] "POST /contact/ HTTP/1.1" 200 21185 "https://www.marc-richter.info/contact/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:61.0) Gecko/20100101 Firefox/61.0"

There is some detail in that, like the HTTP status (

200

) indicating that the request was successful, the referer of the request (

https://www.marc-richter.info/contact/

) telling where the user came from, some details about the user agent (

Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:61.0) Gecko/20100101 Firefox/61.0

) telling what OS and Webbrowser the visitor was using, etc. But the important part for us is:

POST /contact/ HTTP/1.1

This looks pretty much the same as the GET request is, isn't it?

… wait a moment: Has somebody sent you an empty mail?

No. In fact, I sent it to myself with the following content:

To: Marc Richter
Name: Marc Richter
Email: noreply@marc-richter.info
Subject: Testmail

Message:
Hi Marc,
great site!
BR

The reason why you do not see this in the logline before is not that I cut that part. It doesn't show up there, because unlike with GET requests, the POST request is initiating the communication with calling for the destination path only, without any additional in-URL data visible. The content/payload of the request is not part of the initializing requests URL, but is appended to the header of the request.

This is not only more secure to exchange data with the server, but also it is more convenient to exchange bigger amounts of data this way. Imagine, that all data which is ever sent to your favorite social network can be read by any admin who can read the log files of that server … or your employer since you are routed over his proxy servers … 😱

To understand webhooks and to sum this up: Telegram bots are sending the JSON data we already saw in the previous part of this series to any URL we configure to them, using POST requests. The JSON data is then part of these request's payload and can easily be extracted by Django (or any other application).

Again: If you want to get additional details on this, look them up for yourself; to understand how webhooks are working, what I explained so far is already sufficient to follow this article.

Webhook (push) ⚓

Now that we know what POST is and how it is working, let's talk about how webhooks work.
There is a section in the official bot guide of Telegram, which explains how Webhooks work in three different levels of details; if you want to understand more details about Webhooks, I recommend reading at least the shortest version of it; I think it really is a great start for anybody not familiar with the concept of a webhook.

In my own words: A webhook is primarily a normal HTTP(S) socket, provided by a web server - like software listening to it for HTTP requests. Some piece of software is picking up everything which is sent towards this interface and applies some pre-defined logic on the data received. What logic applies is defined by your code, picking up that data.
This happens in real-time and is aware of the delivery result: If the message could not be forwarded to the defined interface, maybe because the server is down or there are temporary connection issues, Telegram caches the messages just like it does with the getUpdate pull method and retries to deliver it again after some time. As soon as the message was handed over to your interface successfully, Telegram forgets about the message and will never re-transmit it to your bot again.

Giving webhooks a try

Enough of the boring theory: I bet you are desperate to see this whole stuff in action, don't you? Let's change your bot to webhook-mode and have everything sent to it forwarded to examine what we need to prepare our bot application for. After that, we will switch back to the getUpdates pull method again.

Attention: Please be aware that everything that will be sent to the bot will be forwarded to a 3rd party's web service, I do have no control of. Please make sure you are OK with any data privacy policy of your country and that service before following the next lines, since I do not take responsibility about these things (if you revealed your online banking TAN lists because you felt like your bot could be interested in them or your partner is sending nudes unaware of this change, for example 😉).

First, you need to visit https://webhook.site in your favorite browser. You will be presented a page telling you all the details to utilize a unique webhook which just got created for you automatically without any registration - pretty awesome, isn't it?

Copy that link and click the "Open in a new tab"-link once. That will open a new tab, showing an empty page; but - wait! The welcome page of your webhook will have changed immediately, listing a new GET request to have just arrived. That was your browser opening that empty page.
It works the same with POST or any other HTTP method anybody makes towards that interface.

Let's now configure our bot's webhook to use that URL you just copied as a destination interface for it's webhook. You will need two things:

  1. WEBHOOK_URL: The URL of the webhook you just created
  2. BOT_TOKEN: The Token of your bot, we created in the previous sections of this series and I advised you to take a note on it because we will need that later; now is "later" 😉

In your browser, you now navigate to the following URL after you have crafted the full URL:

https://api.telegram.org/bot{BOT_TOKEN}/setWebhook?url={WEBHOOK_URL}

You need to replace the two "

{}

"-markers with your own data. You should be presented something like this:

{"ok":true,"result":true,"description":"Webhook was set"}

And: You are good to go!
Make sure the page of your webhook is opened in your browser and then, send something to your bot in Telegram. Immediately (!) you should see that new message as a POST request in the "requests" bar on the left of your webhook page. You can tick the "Format JSON" checkbox at the upper right corner of that page to make the JSON data a bit easier to read.
And that's it! This is a preview of how the messages which are sent to your bot will be forwarded to your webhook.

Telegram WebhookTelegram Webhook representation

Go on, play around with that. Have different datatypes (like images, text, vcards) sent to your bot and check out how it will arrive at the webhook.

You can rely on this format will be the same which are about to hit your Python code, soon. Better be prepared for everything ✌

Switching back to getUpdate pull method

When you are done, better switch back to the getUpdate pull method, to not accidentally expose sensitive data to this publically available service.

This can be done, sending another GET request to the bot's API like we used to enable the hook. The only difference is that we won't provide a webhook URL this time:

https://api.telegram.org/bot{BOT_TOKEN}/setWebhook?url=

Success should be indicated by the following message:

{"ok":true,"result":true,"description":"Webhook was deleted"}

Outlook for the next part of the series

So far so good! For this part of the series, that's it again!
We just learned how webhooks are working and how to enable your bot to make use of that method, how to analyze the data received by it on any webhook and how to disable it again, re-enabling getUpdate pull method.

Next time, I will show you how to prepare Heroku to host the Django app we will create in later chapters and already upload a first demo site to it.

I hope you enjoyed this part! Please let me know of all the things you either enjoyed or did not like that much and how I can do it better in the comments.

Marc Richter

Born in 1982, Marc Richter is an IT enthusiastic since 1994. He became addicted when he first put hands on their family's pc and never stopped investigating and exploring new things since then.
He is married to Jennifer Richter and proud father of two wonderful children, Lotta and Linus.
His current professional focus is DevOps and Python development.

An exhaustive bio can be found at this blog post.

Found my articles useful? Maybe you would like to support my efforts and give me a tip then?

Der Beitrag Create your own Telegram bot with Django on Heroku - Part 4 - pull vs. push method erschien zuerst auf Marc Richter's personal site.

18 Aug 2018 12:46pm GMT

Weekly Python StackOverflow Report: (cxxxix) stackoverflow python report

These are the ten most rated questions at Stack Overflow last week.
Between brackets: [question score / answers count]
Build date: 2018-08-18 11:06:51 GMT


  1. Problems with using a rough greyscale algorithm? - [37/8]
  2. Collatz Conjecture Python - Incorrect Output Above 2 Trillion (Only!) - [28/1]
  3. if-else vs "or" operation for None-check - [15/6]
  4. Faster Python technique to count triples from a list of numbers that are multiples of each other - [14/5]
  5. Python: What happens if script stops while requests.get() is executing? - [13/1]
  6. Extending a class by parameter in Python - [12/2]
  7. How do I shorten this boolean expression? - [10/6]
  8. Getting outer environment arguments from java using graal python - [8/1]
  9. Is the File-Object iterator "broken?" - [8/1]
  10. Swapping list elements in python where the expressions contain function calls - [8/1]

18 Aug 2018 11:07am GMT

Weekly Python StackOverflow Report: (cxxxix) stackoverflow python report

These are the ten most rated questions at Stack Overflow last week.
Between brackets: [question score / answers count]
Build date: 2018-08-18 11:06:51 GMT


  1. Problems with using a rough greyscale algorithm? - [37/8]
  2. Collatz Conjecture Python - Incorrect Output Above 2 Trillion (Only!) - [28/1]
  3. if-else vs "or" operation for None-check - [15/6]
  4. Faster Python technique to count triples from a list of numbers that are multiples of each other - [14/5]
  5. Python: What happens if script stops while requests.get() is executing? - [13/1]
  6. Extending a class by parameter in Python - [12/2]
  7. How do I shorten this boolean expression? - [10/6]
  8. Getting outer environment arguments from java using graal python - [8/1]
  9. Is the File-Object iterator "broken?" - [8/1]
  10. Swapping list elements in python where the expressions contain function calls - [8/1]

18 Aug 2018 11:07am GMT

17 Aug 2018

feedPlanet Python

Codementor: Building a Discord Bot with Python and Repl.it

Find out how to build your own Discord bot using Python and Repl.it. We'll walk through all the steps needed to set your bot up on Discord and then code it using Python, all in the cloud.

17 Aug 2018 10:50pm GMT

Codementor: Building a Discord Bot with Python and Repl.it

Find out how to build your own Discord bot using Python and Repl.it. We'll walk through all the steps needed to set your bot up on Discord and then code it using Python, all in the cloud.

17 Aug 2018 10:50pm GMT

Davy Wybiral: DIY Game Controller (soldering project)

I got a new Hakko soldering iron so it's time to melt some metal!

This video shows how to make a working game controller out of only a perf board, some tactile buttons, and an Arduino Pro Micro. The code is available here.

17 Aug 2018 8:50pm GMT

Davy Wybiral: DIY Game Controller (soldering project)

I got a new Hakko soldering iron so it's time to melt some metal!

This video shows how to make a working game controller out of only a perf board, some tactile buttons, and an Arduino Pro Micro. The code is available here.

17 Aug 2018 8:50pm GMT

Codementor: Hunting Intermittent Tests

There's a sad time in testing land when you have a test suite that you can't rely on. Instead of running a suite and merging if it passes, you have to glance over the failed tests and say "Oh that one… that test is flaky."

17 Aug 2018 8:13pm GMT

Codementor: Hunting Intermittent Tests

There's a sad time in testing land when you have a test suite that you can't rely on. Instead of running a suite and merging if it passes, you have to glance over the failed tests and say "Oh that one… that test is flaky."

17 Aug 2018 8:13pm GMT

Bhishan Bhandari: Web Scraping with NodeJS

Web Scraping has been of an interest to a lot of businesses and individuals with the immense potential of the quantitative data available online. The data collected can entice the growth of an organization or a personal business. Through this post, we will see through examples on how NodeJS can be used to scrape content […]

The post Web Scraping with NodeJS appeared first on The Tara Nights.

17 Aug 2018 7:41pm GMT

Bhishan Bhandari: Web Scraping with NodeJS

Web Scraping has been of an interest to a lot of businesses and individuals with the immense potential of the quantitative data available online. The data collected can entice the growth of an organization or a personal business. Through this post, we will see through examples on how NodeJS can be used to scrape content […]

The post Web Scraping with NodeJS appeared first on The Tara Nights.

17 Aug 2018 7:41pm GMT

Christoph Zwerschke: Using GraphQL or REST, that is the question

Hamlet

This blog has been dormant for too long - it's time to post another article. In order to live up to the blog title, this article will combine some old and seasoned stuff (PostgreSQL and Shakespeare's works) with the latest trends for agile developers (GraphQL, asyncio and type hints for Python).

17 Aug 2018 4:00pm GMT

Christoph Zwerschke: Using GraphQL or REST, that is the question

Hamlet

This blog has been dormant for too long - it's time to post another article. In order to live up to the blog title, this article will combine some old and seasoned stuff (PostgreSQL and Shakespeare's works) with the latest trends for agile developers (GraphQL, asyncio and type hints for Python).

17 Aug 2018 4:00pm GMT

Stack Abuse: Using Regex for Text Manipulation in Python

Introduction

Text preprocessing is one of the most important tasks in Natural Language Processing (NLP). For instance, you may want to remove all punctuation marks from text documents before they can be used for text classification. Similarly, you may want to extract numbers from a text string. Writing manual scripts for such preprocessing tasks requires a lot of effort and is prone to errors. Keeping in view the importance of these preprocessing tasks, the Regular Expressions (aka Regex) have been developed in different languages in order to ease these text preprocessing tasks.

A Regular Expression is a text string that describes a search pattern which can be used to match or replace patterns inside a string with a minimal amount of code. In this tutorial, we will implement different types of regular expressions in the Python language.

To implement regular expressions, the Python's re package can be used. Import the Python's re package with the following command:

import re  

Searching Patterns in a String

One of the most common NLP tasks is to search if a string contains a certain pattern or not. For instance, you may want to perform an operation on the string based on the condition that the string contains a number.

To search a pattern within a string, the match and findall function of the re package is used.

The match Function

Initialize a variable text with a text string as follows:

text = "The film Titanic was released in 1998"  

Let's write a regex expression that matches a string of any length and any character:

result = re.match(r".*", text)  

The first parameter of the match function is the regex expression that you want to search. Regex expression starts with the alphabet r followed by the pattern that you want to search. The pattern should be enclosed in single or double quotes like any other string.

The above regex expression will match the text string, since we are trying to match a string of any length and any character. If a match is found, the match function returns _sre.SRE_Match object as shown below:

type(result)  

Output:

_sre.SRE_Match  

Now to find the matched string, you can use the following command:

result.group(0)  

Output:

'The film Titanic was released in 1998'  

In case if no match is found by the match function, a null object is returned.

Now the previous regex expression matches a string with any length and any character. It will also match an empty string of length zero. To test this, update the value of text variable with an empty string:

text = ""  

Now, if you again execute the following regex expression, a match will be found:

result = re.match(r".*", text)  

Since we specified to match the string with any length and any character, even an empty string is being matched.

To match a string with a length of at least 1, the following regex expression is used:

result = re.match(r".+", text)  

Here the plus sign specifies that the string should have at least one character.

Searching Alphabets

The match function can be used to find any alphabet letters within a string. Let's initialize the text variable with the following text:

text = "The film Titanic was released in 1998"  

Now to find all the alphabet letter, both uppercase and lowercase, we can use the following regex expression:

result = re.match(r"[a-zA-z]+", text)  

This regex expression states that match the text string for any alphabets from small a to small z or capital A to capital Z. The plus sign specifies that string should have at least one character. Let's print the match found by the above expression:

print(result.group(0))  

Output:

The  

In the output, you can see that the first word i.e. The is returned. This is because the match function only returns the first match found. In the regex we specified that find the patterns with both small and capital alphabets from a to z. The first match found was The. After the word The there is a space, which is not treated as an alphabet letter, therefore the matching stopped and the expression returned just The, which is the first match.

However, there is a problem with this. If a string starts with a number instead of an alphabet, the match function will return null even if there are alphabets after the number. Let's see this in action:

text = "1998 was the year when the film titanic was released"  
result = re.match(r"[a-zA-z]+", text)  
type(result)  

Output:

NoneType  

In the above script, we have updated the text variable and now it starts with a digit. We then used the match function to search for alphabets in the string. Though the text string contains alphabets, null will be returned since match function only matches the first element in the string.

To solve this problem we can use the search function.

The search Function

The search function is similar to the match function i.e. it tries to match the specified pattern. However, unlike the match function, it matches the pattern globally instead of matching only the first element. Therefore, the search function will return a match even if the string doesn't contain an alphabet at the start of the string but contains an alphabet elsewhere in the string, as shown below:

text = "1998 was the year when the film titanic was released"  
result = re.search(r"[a-zA-z]+", text)  
print(result.group(0))  

Output:

was  

The search function returns "was" since this is the first match that is found in the text string.

Matching String from the Start

To check if a string starts with a specific word, you can use the carrot key i.e. ^ followed by the word to match with the search function as shown below. Suppose we have the following string:

text = "XYZ 1998 was the year when the film titanic was released"  

If we want to find out whether the string starts with "1998", we can use the search function as follows:

result = re.search(r"^1998", text)  
type(result)  

In the output, null will be returned since the text string doesn't contain "1998" directly at the start.

Now let's change the content text variable and add "1998" at the beginning and then check if "1998" is found at the beginning or not. Execute the following script:

text = "1998 was the year when the film titanic was released"  
if re.search(r"^1998", text):  
    print("Match found")
else:  
    print("Match not found")

Output:

Match found  

Matching Strings from the End

To check whether a string ends with a specific word or not, we can use the word in the regular expression, followed by the dollar sign. The dollar sign marks the end of the statement. Take a look at the following example:

text = "1998 was the year when the film titanic was released"  
if re.search(r"1998$", text):  
    print("Match found")
else:  
    print("Match not found")

In the above script, we tried to find if the text string ends with "1998", which is not the case.

Output:

Match not found  

Now if we update the string and add "1998" at the end of the text string, the above script will return 'Match found' as shown below:

text = "was the year when the film titanic was released 1998"  
if re.search(r"1998$", text):  
    print("Match found")
else:  
    print("Match not found")

Output:

Match found  

Substituting text in a String

Till now we have been using regex to find if a pattern exists in a string. Let's move forward with another advanced regex function i.e. substituting text in a string. The sub function is used for this purpose.

Let's take a simple example of the substitute function. Suppose we have the following string:

text = "The film Pulp Fiction was released in year 1994"  

To replace the string "Pulp Fiction" with "Forrest Gump" (another movie released in 1994) we can use the sub function as follows:

result = re.sub(r"Pulp Fiction", "Forrest Gump", text)  

The first parameter to the sub function is the regular expression that finds the pattern to substitute. The second parameter is the new text that you want as a replacement for the old text and the third parameter is the text string on which the substitute operation will be performed.

If you print the result variable, you will see the new string.

Now let's substitute all the alphabets in our string with character "X". Execute the following script:

text = "The film Pulp Fiction was released in year 1994"  
result = re.sub(r"[a-z]", "X", text)  
print(result)  

Output:

TXX XXXX PXXX FXXXXXX XXX XXXXXXXX XX XXXX 1994  

It can be seen from the output that all the characters have been replaced except the capital ones. This is because we specified a-z only and not A-Z. There are two ways to solve this problem. You can either specify A-Z in the regular expression along with a-z as follows:

result = re.sub(r"[a-zA-Z]", "X", text)  

Or you can pass the additional parameter flags to the sub function and set its value to re.I which refers to case insensitive, as follows:

result = re.sub(r"[a-z]", "X", text, flags=re.I)  

More details about different types of flags can be found at Python regex official documentation page.

Shorthand Character Classes

There are different types of shorthand character classes that can be used to perform a variety of different string manipulation functions without having to write complex logic. In this section we will discuss some of them:

Removing Digits from a String

The regex expression to find digits in a string is \d. This pattern can be used to remove digits from a string by replacing them with an empty string of length zero as shown below:

text = "The film Pulp Fiction was released in year 1994"  
result = re.sub(r"\d", "", text)  
print(result)  

Output:

The film Pulp Fiction was released in year  

Removing Alphabet Letters from a String

text = "The film Pulp Fiction was released in year 1994"  
result = re.sub(r"[a-z]", "", text, flags=re.I)  
print(result)  

Output:

1994  

Removing Word Characters

If you want to remove all the word characters (letters and numbers) from a string and keep the remaining characters, you can use the \w pattern in your regex and replace it with an empty string of length zero, as shown below:

text = "The film, '@Pulp Fiction' was ? released in % $ year 1994."  
result = re.sub(r"\w","", text, flags = re.I)  
print(result)  

Output:

, '@ '  ?   % $  .

The output shows that all the numbers and alphabets have been removed.

Removing Non-Word Characters

To remove all the non-word characters, the \W pattern can be used as follows:

text = "The film, '@Pulp Fiction' was ? released in % $ year 1994."  
result = re.sub(r"\W", "", text, flags=re.I)  
print(result)  

Output:

ThefilmPulpFictionwasreleasedinyear1994  

From the output, you can see that everything has been removed (even spaces), except the numbers and alphabets.

Grouping Multiple Patterns

You can group multiple patterns to match or substitute in a string using the square bracket. In fact, we did this when we matched capital and small letters. Let's group multiple punctuation marks and remove them from a string:

text = "The film, '@Pulp Fiction' was ? released _ in % $ year 1994."  
result = re.sub(r"[,@\'?\.$%_]", "", text, flags=re.I)  
print(result)  

Output:

The film Pulp Fiction was  released  in   year 1994  

You can see that the string in the text variable had multiple punctuation marks, we grouped all these punctuations in the regex expression using square brackets. It is important to mention that with a dot and a single quote we have to use the escape sequence i.e. backward slash. This is because by default the dot operator is used for any character and the single quote is used to denote a string.

Removing Multiple Spaces

Sometimes, multiple spaces appear between words as a result of removing words or punctuation. For instance, in the output of the last example, there are multiple spaces between in and year. These spaces can be removed using the \s pattern, which refers to a single space.

text = "The film      Pulp Fiction      was released in   year 1994."  
result = re.sub(r"\s+"," ", text, flags = re.I)  
print(result)  

Output:

The film Pulp Fiction was released in year 1994.  

In the script above we used the expression \s+ which refers to single or multiple spaces.

Removing Spaces from Start and End

Sometimes we have a sentence that starts or ends with a space, which is often not desirable. The following script removes spaces from the beginning of a sentence:

text = "         The film Pulp Fiction was released in year 1994"  
result = re.sub(r"^\s+", "", text)  
print(result)  

Output:

The film Pulp Fiction was released in year 1994  

Similarly, to remove space at the end of the string, the following script can be used:

text = "The film Pulp Fiction was released in year 1994      "  
result = re.sub(r"\s+$", "", text)  
print(result)  

Removing a Single Character

Sometimes removing punctuation marks, such as an apostrophe, results in a single character which has no meaning. For instance, if you remove the apostrophe from the word Jacob's and replace it with space, the resultant string is Jacob s. Here the s makes no sense. Such single characters can be removed using regex as shown below:

text = "The film Pulp Fiction     s was b released in year 1994"  
result = re.sub(r"\s+[a-zA-Z]\s+", " ", text)  
print(result)  

Output:

The film Pulp Fiction was released in year 1994  

The script replaces any small or capital letter between one or more spaces, with a single space.

Splitting a String

String splitting is another very important function. Strings can be split using split function from the re package. The split function returns a list of split tokens. Let's split a string of words where one or more space characters are found, as shown below:

text = "The film      Pulp   Fiction was released in year 1994      "  
result = re.split(r"\s+", text)  
print(result)  

Output:

['The', 'film', 'Pulp', 'Fiction', 'was', 'released', 'in', 'year', '1994', '']

Similarly, you can use other regex expressions to split a string using the split functions. For instance, the following split function splits string of words when a comma is found:

text = "The film, Pulp Fiction, was released in year 1994"  
result = re.split(r"\,", text)  
print(result)  

Output:

['The film', ' Pulp Fiction', ' was released in year 1994']

Finding All Instances

The match function conducts a match on the first element while the search function conducts a global search on the string and returns the first matched instance.

For instance, if we have the following string:

text = "I want to buy a mobile between 200 and 400 euros"  

We want to search all the digits from this string. If we use the search function, only the first occurrence of digits i.e. 200 will be returned as shown below:

result = re.search(r"\d+", text)  
print(result.group(0))  

Output:

200  

On the other hand, the findall function returns a list that contains all the matched utterances as shown below:

text = "I want to buy a mobile between 200 and 400 euros"  
result = re.findall(r"\d+", text)  
print(result)  

Output:

['200', '400']

You can see from the output that both "200" and "400" is returned by the findall function.

Conclusion

In this article we studied some of the most commonly used regex functions in Python. Regular expressions are extremely useful for preprocessing text that can be further used for a variety of applications, such as topic modeling, text classification, sentimental analysis, and text summarization, etc.

17 Aug 2018 3:18pm GMT

Stack Abuse: Using Regex for Text Manipulation in Python

Introduction

Text preprocessing is one of the most important tasks in Natural Language Processing (NLP). For instance, you may want to remove all punctuation marks from text documents before they can be used for text classification. Similarly, you may want to extract numbers from a text string. Writing manual scripts for such preprocessing tasks requires a lot of effort and is prone to errors. Keeping in view the importance of these preprocessing tasks, the Regular Expressions (aka Regex) have been developed in different languages in order to ease these text preprocessing tasks.

A Regular Expression is a text string that describes a search pattern which can be used to match or replace patterns inside a string with a minimal amount of code. In this tutorial, we will implement different types of regular expressions in the Python language.

To implement regular expressions, the Python's re package can be used. Import the Python's re package with the following command:

import re  

Searching Patterns in a String

One of the most common NLP tasks is to search if a string contains a certain pattern or not. For instance, you may want to perform an operation on the string based on the condition that the string contains a number.

To search a pattern within a string, the match and findall function of the re package is used.

The match Function

Initialize a variable text with a text string as follows:

text = "The film Titanic was released in 1998"  

Let's write a regex expression that matches a string of any length and any character:

result = re.match(r".*", text)  

The first parameter of the match function is the regex expression that you want to search. Regex expression starts with the alphabet r followed by the pattern that you want to search. The pattern should be enclosed in single or double quotes like any other string.

The above regex expression will match the text string, since we are trying to match a string of any length and any character. If a match is found, the match function returns _sre.SRE_Match object as shown below:

type(result)  

Output:

_sre.SRE_Match  

Now to find the matched string, you can use the following command:

result.group(0)  

Output:

'The film Titanic was released in 1998'  

In case if no match is found by the match function, a null object is returned.

Now the previous regex expression matches a string with any length and any character. It will also match an empty string of length zero. To test this, update the value of text variable with an empty string:

text = ""  

Now, if you again execute the following regex expression, a match will be found:

result = re.match(r".*", text)  

Since we specified to match the string with any length and any character, even an empty string is being matched.

To match a string with a length of at least 1, the following regex expression is used:

result = re.match(r".+", text)  

Here the plus sign specifies that the string should have at least one character.

Searching Alphabets

The match function can be used to find any alphabet letters within a string. Let's initialize the text variable with the following text:

text = "The film Titanic was released in 1998"  

Now to find all the alphabet letter, both uppercase and lowercase, we can use the following regex expression:

result = re.match(r"[a-zA-z]+", text)  

This regex expression states that match the text string for any alphabets from small a to small z or capital A to capital Z. The plus sign specifies that string should have at least one character. Let's print the match found by the above expression:

print(result.group(0))  

Output:

The  

In the output, you can see that the first word i.e. The is returned. This is because the match function only returns the first match found. In the regex we specified that find the patterns with both small and capital alphabets from a to z. The first match found was The. After the word The there is a space, which is not treated as an alphabet letter, therefore the matching stopped and the expression returned just The, which is the first match.

However, there is a problem with this. If a string starts with a number instead of an alphabet, the match function will return null even if there are alphabets after the number. Let's see this in action:

text = "1998 was the year when the film titanic was released"  
result = re.match(r"[a-zA-z]+", text)  
type(result)  

Output:

NoneType  

In the above script, we have updated the text variable and now it starts with a digit. We then used the match function to search for alphabets in the string. Though the text string contains alphabets, null will be returned since match function only matches the first element in the string.

To solve this problem we can use the search function.

The search Function

The search function is similar to the match function i.e. it tries to match the specified pattern. However, unlike the match function, it matches the pattern globally instead of matching only the first element. Therefore, the search function will return a match even if the string doesn't contain an alphabet at the start of the string but contains an alphabet elsewhere in the string, as shown below:

text = "1998 was the year when the film titanic was released"  
result = re.search(r"[a-zA-z]+", text)  
print(result.group(0))  

Output:

was  

The search function returns "was" since this is the first match that is found in the text string.

Matching String from the Start

To check if a string starts with a specific word, you can use the carrot key i.e. ^ followed by the word to match with the search function as shown below. Suppose we have the following string:

text = "XYZ 1998 was the year when the film titanic was released"  

If we want to find out whether the string starts with "1998", we can use the search function as follows:

result = re.search(r"^1998", text)  
type(result)  

In the output, null will be returned since the text string doesn't contain "1998" directly at the start.

Now let's change the content text variable and add "1998" at the beginning and then check if "1998" is found at the beginning or not. Execute the following script:

text = "1998 was the year when the film titanic was released"  
if re.search(r"^1998", text):  
    print("Match found")
else:  
    print("Match not found")

Output:

Match found  

Matching Strings from the End

To check whether a string ends with a specific word or not, we can use the word in the regular expression, followed by the dollar sign. The dollar sign marks the end of the statement. Take a look at the following example:

text = "1998 was the year when the film titanic was released"  
if re.search(r"1998$", text):  
    print("Match found")
else:  
    print("Match not found")

In the above script, we tried to find if the text string ends with "1998", which is not the case.

Output:

Match not found  

Now if we update the string and add "1998" at the end of the text string, the above script will return 'Match found' as shown below:

text = "was the year when the film titanic was released 1998"  
if re.search(r"1998$", text):  
    print("Match found")
else:  
    print("Match not found")

Output:

Match found  

Substituting text in a String

Till now we have been using regex to find if a pattern exists in a string. Let's move forward with another advanced regex function i.e. substituting text in a string. The sub function is used for this purpose.

Let's take a simple example of the substitute function. Suppose we have the following string:

text = "The film Pulp Fiction was released in year 1994"  

To replace the string "Pulp Fiction" with "Forrest Gump" (another movie released in 1994) we can use the sub function as follows:

result = re.sub(r"Pulp Fiction", "Forrest Gump", text)  

The first parameter to the sub function is the regular expression that finds the pattern to substitute. The second parameter is the new text that you want as a replacement for the old text and the third parameter is the text string on which the substitute operation will be performed.

If you print the result variable, you will see the new string.

Now let's substitute all the alphabets in our string with character "X". Execute the following script:

text = "The film Pulp Fiction was released in year 1994"  
result = re.sub(r"[a-z]", "X", text)  
print(result)  

Output:

TXX XXXX PXXX FXXXXXX XXX XXXXXXXX XX XXXX 1994  

It can be seen from the output that all the characters have been replaced except the capital ones. This is because we specified a-z only and not A-Z. There are two ways to solve this problem. You can either specify A-Z in the regular expression along with a-z as follows:

result = re.sub(r"[a-zA-Z]", "X", text)  

Or you can pass the additional parameter flags to the sub function and set its value to re.I which refers to case insensitive, as follows:

result = re.sub(r"[a-z]", "X", text, flags=re.I)  

More details about different types of flags can be found at Python regex official documentation page.

Shorthand Character Classes

There are different types of shorthand character classes that can be used to perform a variety of different string manipulation functions without having to write complex logic. In this section we will discuss some of them:

Removing Digits from a String

The regex expression to find digits in a string is \d. This pattern can be used to remove digits from a string by replacing them with an empty string of length zero as shown below:

text = "The film Pulp Fiction was released in year 1994"  
result = re.sub(r"\d", "", text)  
print(result)  

Output:

The film Pulp Fiction was released in year  

Removing Alphabet Letters from a String

text = "The film Pulp Fiction was released in year 1994"  
result = re.sub(r"[a-z]", "", text, flags=re.I)  
print(result)  

Output:

1994  

Removing Word Characters

If you want to remove all the word characters (letters and numbers) from a string and keep the remaining characters, you can use the \w pattern in your regex and replace it with an empty string of length zero, as shown below:

text = "The film, '@Pulp Fiction' was ? released in % $ year 1994."  
result = re.sub(r"\w","", text, flags = re.I)  
print(result)  

Output:

, '@ '  ?   % $  .

The output shows that all the numbers and alphabets have been removed.

Removing Non-Word Characters

To remove all the non-word characters, the \W pattern can be used as follows:

text = "The film, '@Pulp Fiction' was ? released in % $ year 1994."  
result = re.sub(r"\W", "", text, flags=re.I)  
print(result)  

Output:

ThefilmPulpFictionwasreleasedinyear1994  

From the output, you can see that everything has been removed (even spaces), except the numbers and alphabets.

Grouping Multiple Patterns

You can group multiple patterns to match or substitute in a string using the square bracket. In fact, we did this when we matched capital and small letters. Let's group multiple punctuation marks and remove them from a string:

text = "The film, '@Pulp Fiction' was ? released _ in % $ year 1994."  
result = re.sub(r"[,@\'?\.$%_]", "", text, flags=re.I)  
print(result)  

Output:

The film Pulp Fiction was  released  in   year 1994  

You can see that the string in the text variable had multiple punctuation marks, we grouped all these punctuations in the regex expression using square brackets. It is important to mention that with a dot and a single quote we have to use the escape sequence i.e. backward slash. This is because by default the dot operator is used for any character and the single quote is used to denote a string.

Removing Multiple Spaces

Sometimes, multiple spaces appear between words as a result of removing words or punctuation. For instance, in the output of the last example, there are multiple spaces between in and year. These spaces can be removed using the \s pattern, which refers to a single space.

text = "The film      Pulp Fiction      was released in   year 1994."  
result = re.sub(r"\s+"," ", text, flags = re.I)  
print(result)  

Output:

The film Pulp Fiction was released in year 1994.  

In the script above we used the expression \s+ which refers to single or multiple spaces.

Removing Spaces from Start and End

Sometimes we have a sentence that starts or ends with a space, which is often not desirable. The following script removes spaces from the beginning of a sentence:

text = "         The film Pulp Fiction was released in year 1994"  
result = re.sub(r"^\s+", "", text)  
print(result)  

Output:

The film Pulp Fiction was released in year 1994  

Similarly, to remove space at the end of the string, the following script can be used:

text = "The film Pulp Fiction was released in year 1994      "  
result = re.sub(r"\s+$", "", text)  
print(result)  

Removing a Single Character

Sometimes removing punctuation marks, such as an apostrophe, results in a single character which has no meaning. For instance, if you remove the apostrophe from the word Jacob's and replace it with space, the resultant string is Jacob s. Here the s makes no sense. Such single characters can be removed using regex as shown below:

text = "The film Pulp Fiction     s was b released in year 1994"  
result = re.sub(r"\s+[a-zA-Z]\s+", " ", text)  
print(result)  

Output:

The film Pulp Fiction was released in year 1994  

The script replaces any small or capital letter between one or more spaces, with a single space.

Splitting a String

String splitting is another very important function. Strings can be split using split function from the re package. The split function returns a list of split tokens. Let's split a string of words where one or more space characters are found, as shown below:

text = "The film      Pulp   Fiction was released in year 1994      "  
result = re.split(r"\s+", text)  
print(result)  

Output:

['The', 'film', 'Pulp', 'Fiction', 'was', 'released', 'in', 'year', '1994', '']

Similarly, you can use other regex expressions to split a string using the split functions. For instance, the following split function splits string of words when a comma is found:

text = "The film, Pulp Fiction, was released in year 1994"  
result = re.split(r"\,", text)  
print(result)  

Output:

['The film', ' Pulp Fiction', ' was released in year 1994']

Finding All Instances

The match function conducts a match on the first element while the search function conducts a global search on the string and returns the first matched instance.

For instance, if we have the following string:

text = "I want to buy a mobile between 200 and 400 euros"  

We want to search all the digits from this string. If we use the search function, only the first occurrence of digits i.e. 200 will be returned as shown below:

result = re.search(r"\d+", text)  
print(result.group(0))  

Output:

200  

On the other hand, the findall function returns a list that contains all the matched utterances as shown below:

text = "I want to buy a mobile between 200 and 400 euros"  
result = re.findall(r"\d+", text)  
print(result)  

Output:

['200', '400']

You can see from the output that both "200" and "400" is returned by the findall function.

Conclusion

In this article we studied some of the most commonly used regex functions in Python. Regular expressions are extremely useful for preprocessing text that can be further used for a variety of applications, such as topic modeling, text classification, sentimental analysis, and text summarization, etc.

17 Aug 2018 3:18pm GMT

Kay Hayen: Nuitka this week #4

Contents

Goto Generators

This continues TWN #3 where I explained what is all about.

Good news is, at the time Python2 generators were largely working with the new ways, in the mean time not only did all of the Python 2.7 test suite pass with goto generators, also did the Python 3.4 test suite, i.e. also the yield from is working with it.

The way it was done is to set m_yieldfrom in generators, and then to enter a state, where the code will only be resumed, when that sub-generator that currently it is yielding from, is finished. That makes it very much like normal yield. In fact, code generation is hardly different there.

Since the whole purpose is to get rid of make/get/setcontext, the next stop is coroutines. They have async for, async with and await but at the end of the day, the implementation comes down to yield from really with only a lot of sugar applied.

Right now, I am debugging "goto coroutines". It's hard to tell when it will be finished, and then asyncgen will be waiting still.

This is easily the largest change in a long time, esp. due to the heap storage changes that I already discussed. One this is finished, I expect to turn towards C types with relative ease.

Tox Plugin

Anthony Shaw took on Tox and Nuitka and created a plugin that allows using Nuitka. I am still wrapping my head around these things. It's only a proof of concept yet. I will give it more coverage in the future.

Twitter

Follow me on twitter if you like, I will:

Follow @kayhayen

Hotfixes

So there have even more hotfixes. One addresses memory leaks found with the yield from while I was adding tests. Usually if I encounter an old issue that has a small fix, that is what I do, push out a hotfix using the git flow model. Also nested namespace packages for Python3, those are the ones without a __init__.py were not working after the original directory was removed, and that got fixed.

And right now, I have hotfixes for frames close method, which apparently was never updated to work properly for coroutines and asyncgen. That is going to be in the next hotfix.

Plans

So the heap storage seems pretty complete now, and goto generators are on the final stretch. As always, things feel right around the corner. But it's unclear how much longer I will have to debug. I am pretty sure the bare work of doing asyncgen is going to be low. Debugging that too then, that is the hard part.

A new release seems justified, but I kind of do not want to make it without that major new code used. Because apparently during the debugging, I tend to find issues that need hotfixes, so I will wait for the goto generator work to finish.

17 Aug 2018 12:55pm GMT

Kay Hayen: Nuitka this week #4

Contents

Goto Generators

This continues TWN #3 where I explained what is all about.

Good news is, at the time Python2 generators were largely working with the new ways, in the mean time not only did all of the Python 2.7 test suite pass with goto generators, also did the Python 3.4 test suite, i.e. also the yield from is working with it.

The way it was done is to set m_yieldfrom in generators, and then to enter a state, where the code will only be resumed, when that sub-generator that currently it is yielding from, is finished. That makes it very much like normal yield. In fact, code generation is hardly different there.

Since the whole purpose is to get rid of make/get/setcontext, the next stop is coroutines. They have async for, async with and await but at the end of the day, the implementation comes down to yield from really with only a lot of sugar applied.

Right now, I am debugging "goto coroutines". It's hard to tell when it will be finished, and then asyncgen will be waiting still.

This is easily the largest change in a long time, esp. due to the heap storage changes that I already discussed. One this is finished, I expect to turn towards C types with relative ease.

Tox Plugin

Anthony Shaw took on Tox and Nuitka and created a plugin that allows using Nuitka. I am still wrapping my head around these things. It's only a proof of concept yet. I will give it more coverage in the future.

Twitter

Follow me on twitter if you like, I will:

Follow @kayhayen

Hotfixes

So there have even more hotfixes. One addresses memory leaks found with the yield from while I was adding tests. Usually if I encounter an old issue that has a small fix, that is what I do, push out a hotfix using the git flow model. Also nested namespace packages for Python3, those are the ones without a __init__.py were not working after the original directory was removed, and that got fixed.

And right now, I have hotfixes for frames close method, which apparently was never updated to work properly for coroutines and asyncgen. That is going to be in the next hotfix.

Plans

So the heap storage seems pretty complete now, and goto generators are on the final stretch. As always, things feel right around the corner. But it's unclear how much longer I will have to debug. I am pretty sure the bare work of doing asyncgen is going to be low. Debugging that too then, that is the hard part.

A new release seems justified, but I kind of do not want to make it without that major new code used. Because apparently during the debugging, I tend to find issues that need hotfixes, so I will wait for the goto generator work to finish.

17 Aug 2018 12:55pm GMT

PyBites: A Python Orientation - How to Get Started

Python is a wonderful language for both beginners and expert programmers, but getting started can be daunting. Which version should I use? Which editors are best? What do you mean there are different implementations and environments? Here's a guide to help navigate these big FAQs.

tl;dr

For most people:

Which Version?

Python 2 and Python 3 are actually different languages. The differences go deeper than just print statements. The What's New in Python page on the official doc site lists all the gory details, and decent articles showcasing differences can be found here, here, and here. Although Python 3 is newer, Python 2 remains prevalent. Most popular packages use Python packaging tools to support both versions. The Python Wiki makes it clear that Python 3 is the better choice:

Python 2.x is legacy, Python 3.x is the present and future of the language

Furthermore, Python 2 will reach end-of-life in 2020. The Python team will continue to provide bug fixes for 2.7 until 2020 (PEP 373), but there will be no new language features and no 2.8 (PEP 404). (Originally, end-of-life was planned for 2015, but it was pushed back by 5 years.) There is even a Python 2.7 Countdown clock online.

Which Implementation?

In purest terms, "Python" is a language specification. An implementation provides the language processing tools (compiler, interpreter, etc.) to run Python programs. The Hitchhiker's Guide to Python has a great article entitled Picking an Interpreter that provides a good summary of available interpreters. Others are listed on python.org and the Python Wiki. The table below provides a quick overview of the big ones.

CPython

PyPy

Jython

IronPython

Python for .NET

Stackless Python

MicroPython

Unless you have a very specific reason, just use CPython. In fact, most people are referring to CPython when they say "Python." CPython has the most compatibility, the widest package library, and the richest support. If you really need speed, consider PyPy.

Managing Installations

pip is the standard tool for installing Python packages. The simplest way to install Python is to install it "globally" for the system. In fact, some operating systems like macOS and Ubuntu have Python pre-installed. However, global installation has limitations:

  1. You may want to develop packages for both versions 2 and 3.
  2. You may not have permissions to add new packages globally.
  3. Different projects may require different versions of packages.

These problems can be solved by using "virtual" environments. A virtual environment is like a local Python installation with a specific package set. For example, I have created virtual environments for Python as part of Jenkins build jobs, since I did not have permission to install special automation packages globally on the Jenkins slaves.

The standard virtual environment tool for Python is venv, which has been packaged with (C)Python since 3.3. (venv had a command line wrapper named pyvenv, but this was deprecated in 3.6.) Another older but still popular third-party tool is virtualenv. As explained in this Reddit post, venv _is the Python-sanctioned replacement for _virtualenv. However, virtualenv supports Python 2, whereas venv does not. Conda is an environment manager popular with the science and data communities, and it can support other languages in addition to Python.

That being said, there is a relatively new package manager taking the Python world by storm: pipenv. Pipenv combines pip, Pipfile, and virtualenv into an easy workflow with simple commands. Personally, I find it to be very helpful. However, it has caused some controversy (see Reddit), and it may not be applicable for all scenarios (see Chris Warrick's article). My recommendation is to use pipenv for new projects if it meets your needs.

Editors and IDEs

After setting up your Python environment, you are ready to start programming! There are two routes to take for text editing: source code editors and integrated development environments.

Source code editors are lightweight but often include basics like syntax highlighting and basic auto-completion. They're great for quick edits and light scripting. Many have plugins. Popular choices are Visual Studio Code, Sublime, Atom, and Notepad++. My current favorite is Visual Studio Code because the Python extensions are stellar and settings are simple - just remember to install the extensions you need! I use it personally for Django development.

For more intense development, I highly recommend an IDE like JetBrains PyCharm, PyDev for Eclipse, Wing Python IDE, or Eric. IDEs provide rich development support, especially for larger apps that use frameworks like Django, Pyramid, and SQLAlchemy. They also make testing easier with plugins for test frameworks like pytest, behave, and others. PyCharm and PyDev are particularly nice because they can integrate into their larger IDEs (IntelliJ IDEA and Eclipse, respectively) to handle more languages. Personally, I prefer PyCharm, but advanced features require a paid license.

Pythonese

The Python community throws around a few terms you should know:

Pythonic

Pythonista

Pythoneer

The Zen of Python

The Python Software Foundation (PSF)

PyCon

Benevolent Dictator for Life(BDFL)

This article was originally posted at AutomationPanda.com and has been reposted here with permission.


Keep Calm and Code in Python!

Andy

17 Aug 2018 7:41am GMT

PyBites: A Python Orientation - How to Get Started

Python is a wonderful language for both beginners and expert programmers, but getting started can be daunting. Which version should I use? Which editors are best? What do you mean there are different implementations and environments? Here's a guide to help navigate these big FAQs.

tl;dr

For most people:

Which Version?

Python 2 and Python 3 are actually different languages. The differences go deeper than just print statements. The What's New in Python page on the official doc site lists all the gory details, and decent articles showcasing differences can be found here, here, and here. Although Python 3 is newer, Python 2 remains prevalent. Most popular packages use Python packaging tools to support both versions. The Python Wiki makes it clear that Python 3 is the better choice:

Python 2.x is legacy, Python 3.x is the present and future of the language

Furthermore, Python 2 will reach end-of-life in 2020. The Python team will continue to provide bug fixes for 2.7 until 2020 (PEP 373), but there will be no new language features and no 2.8 (PEP 404). (Originally, end-of-life was planned for 2015, but it was pushed back by 5 years.) There is even a Python 2.7 Countdown clock online.

Which Implementation?

In purest terms, "Python" is a language specification. An implementation provides the language processing tools (compiler, interpreter, etc.) to run Python programs. The Hitchhiker's Guide to Python has a great article entitled Picking an Interpreter that provides a good summary of available interpreters. Others are listed on python.org and the Python Wiki. The table below provides a quick overview of the big ones.

CPython

PyPy

Jython

IronPython

Python for .NET

Stackless Python

MicroPython

Unless you have a very specific reason, just use CPython. In fact, most people are referring to CPython when they say "Python." CPython has the most compatibility, the widest package library, and the richest support. If you really need speed, consider PyPy.

Managing Installations

pip is the standard tool for installing Python packages. The simplest way to install Python is to install it "globally" for the system. In fact, some operating systems like macOS and Ubuntu have Python pre-installed. However, global installation has limitations:

  1. You may want to develop packages for both versions 2 and 3.
  2. You may not have permissions to add new packages globally.
  3. Different projects may require different versions of packages.

These problems can be solved by using "virtual" environments. A virtual environment is like a local Python installation with a specific package set. For example, I have created virtual environments for Python as part of Jenkins build jobs, since I did not have permission to install special automation packages globally on the Jenkins slaves.

The standard virtual environment tool for Python is venv, which has been packaged with (C)Python since 3.3. (venv had a command line wrapper named pyvenv, but this was deprecated in 3.6.) Another older but still popular third-party tool is virtualenv. As explained in this Reddit post, venv _is the Python-sanctioned replacement for _virtualenv. However, virtualenv supports Python 2, whereas venv does not. Conda is an environment manager popular with the science and data communities, and it can support other languages in addition to Python.

That being said, there is a relatively new package manager taking the Python world by storm: pipenv. Pipenv combines pip, Pipfile, and virtualenv into an easy workflow with simple commands. Personally, I find it to be very helpful. However, it has caused some controversy (see Reddit), and it may not be applicable for all scenarios (see Chris Warrick's article). My recommendation is to use pipenv for new projects if it meets your needs.

Editors and IDEs

After setting up your Python environment, you are ready to start programming! There are two routes to take for text editing: source code editors and integrated development environments.

Source code editors are lightweight but often include basics like syntax highlighting and basic auto-completion. They're great for quick edits and light scripting. Many have plugins. Popular choices are Visual Studio Code, Sublime, Atom, and Notepad++. My current favorite is Visual Studio Code because the Python extensions are stellar and settings are simple - just remember to install the extensions you need! I use it personally for Django development.

For more intense development, I highly recommend an IDE like JetBrains PyCharm, PyDev for Eclipse, Wing Python IDE, or Eric. IDEs provide rich development support, especially for larger apps that use frameworks like Django, Pyramid, and SQLAlchemy. They also make testing easier with plugins for test frameworks like pytest, behave, and others. PyCharm and PyDev are particularly nice because they can integrate into their larger IDEs (IntelliJ IDEA and Eclipse, respectively) to handle more languages. Personally, I prefer PyCharm, but advanced features require a paid license.

Pythonese

The Python community throws around a few terms you should know:

Pythonic

Pythonista

Pythoneer

The Zen of Python

The Python Software Foundation (PSF)

PyCon

Benevolent Dictator for Life(BDFL)

This article was originally posted at AutomationPanda.com and has been reposted here with permission.


Keep Calm and Code in Python!

Andy

17 Aug 2018 7:41am GMT

10 Nov 2011

feedPython Software Foundation | GSoC'11 Students

Benedict Stein: King Willams Town Bahnhof

Gestern musste ich morgens zur Station nach KWT um unsere Rerservierten Bustickets für die Weihnachtsferien in Capetown abzuholen. Der Bahnhof selber ist seit Dezember aus kostengründen ohne Zugverbindung - aber Translux und co - die langdistanzbusse haben dort ihre Büros.


Größere Kartenansicht




© benste CC NC SA

10 Nov 2011 10:57am GMT

09 Nov 2011

feedPython Software Foundation | GSoC'11 Students

Benedict Stein

Niemand ist besorgt um so was - mit dem Auto fährt man einfach durch, und in der City - nahe Gnobie- "ne das ist erst gefährlich wenn die Feuerwehr da ist" - 30min später auf dem Rückweg war die Feuerwehr da.




© benste CC NC SA

09 Nov 2011 8:25pm GMT

08 Nov 2011

feedPython Software Foundation | GSoC'11 Students

Benedict Stein: Brai Party

Brai = Grillabend o.ä.

Die möchte gern Techniker beim Flicken ihrer SpeakOn / Klinke Stecker Verzweigungen...

Die Damen "Mamas" der Siedlung bei der offiziellen Eröffnungsrede

Auch wenn weniger Leute da waren als erwartet, Laute Musik und viele Leute ...

Und natürlich ein Feuer mit echtem Holz zum Grillen.

© benste CC NC SA

08 Nov 2011 2:30pm GMT

07 Nov 2011

feedPython Software Foundation | GSoC'11 Students

Benedict Stein: Lumanyano Primary

One of our missions was bringing Katja's Linux Server back to her room. While doing that we saw her new decoration.

Björn, Simphiwe carried the PC to Katja's school


© benste CC NC SA

07 Nov 2011 2:00pm GMT

06 Nov 2011

feedPython Software Foundation | GSoC'11 Students

Benedict Stein: Nelisa Haircut

Today I went with Björn to Needs Camp to Visit Katja's guest family for a special Party. First of all we visited some friends of Nelisa - yeah the one I'm working with in Quigney - Katja's guest fathers sister - who did her a haircut.

African Women usually get their hair done by arranging extensions and not like Europeans just cutting some hair.

In between she looked like this...

And then she was done - looks amazing considering the amount of hair she had last week - doesn't it ?

© benste CC NC SA

06 Nov 2011 7:45pm GMT

05 Nov 2011

feedPython Software Foundation | GSoC'11 Students

Benedict Stein: Mein Samstag

Irgendwie viel mir heute auf das ich meine Blogposts mal ein bischen umstrukturieren muss - wenn ich immer nur von neuen Plätzen berichte, dann müsste ich ja eine Rundreise machen. Hier also mal ein paar Sachen aus meinem heutigen Alltag.

Erst einmal vorweg, Samstag zählt zumindest für uns Voluntäre zu den freien Tagen.

Dieses Wochenende sind nur Rommel und ich auf der Farm - Katja und Björn sind ja mittlerweile in ihren Einsatzstellen, und meine Mitbewohner Kyle und Jonathan sind zu Hause in Grahamstown - sowie auch Sipho der in Dimbaza wohnt.
Robin, die Frau von Rommel ist in Woodie Cape - schon seit Donnerstag um da ein paar Sachen zur erledigen.
Naja wie dem auch sei heute morgen haben wir uns erstmal ein gemeinsames Weetbix/Müsli Frühstück gegönnt und haben uns dann auf den Weg nach East London gemacht. 2 Sachen waren auf der Checkliste Vodacom, Ethienne (Imobilienmakler) außerdem auf dem Rückweg die fehlenden Dinge nach NeedsCamp bringen.

Nachdem wir gerade auf der Dirtroad losgefahren sind mussten wir feststellen das wir die Sachen für Needscamp und Ethienne nicht eingepackt hatten aber die Pumpe für die Wasserversorgung im Auto hatten.

Also sind wir in EastLondon ersteinmal nach Farmerama - nein nicht das onlinespiel farmville - sondern einen Laden mit ganz vielen Sachen für eine Farm - in Berea einem nördlichen Stadteil gefahren.

In Farmerama haben wir uns dann beraten lassen für einen Schnellverschluss der uns das leben mit der Pumpe leichter machen soll und außerdem eine leichtere Pumpe zur Reperatur gebracht, damit es nicht immer so ein großer Aufwand ist, wenn mal wieder das Wasser ausgegangen ist.

Fego Caffé ist in der Hemmingways Mall, dort mussten wir und PIN und PUK einer unserer Datensimcards geben lassen, da bei der PIN Abfrage leider ein zahlendreher unterlaufen ist. Naja auf jeden Fall speichern die Shops in Südafrika so sensible Daten wie eine PUK - die im Prinzip zugang zu einem gesperrten Phone verschafft.

Im Cafe hat Rommel dann ein paar online Transaktionen mit dem 3G Modem durchgeführt, welches ja jetzt wieder funktionierte - und übrigens mittlerweile in Ubuntu meinem Linuxsystem perfekt klappt.

Nebenbei bin ich nach 8ta gegangen um dort etwas über deren neue Deals zu erfahren, da wir in einigen von Hilltops Centern Internet anbieten wollen. Das Bild zeigt die Abdeckung UMTS in NeedsCamp Katjas Ort. 8ta ist ein neuer Telefonanbieter von Telkom, nachdem Vodafone sich Telkoms anteile an Vodacom gekauft hat müssen die komplett neu aufbauen.
Wir haben uns dazu entschieden mal eine kostenlose Prepaidkarte zu testen zu organisieren, denn wer weis wie genau die Karte oben ist ... Bevor man einen noch so billigen Deal für 24 Monate signed sollte man wissen obs geht.

Danach gings nach Checkers in Vincent, gesucht wurden zwei Hotplates für WoodyCape - R 129.00 eine - also ca. 12€ für eine zweigeteilte Kochplatte.
Wie man sieht im Hintergrund gibts schon Weihnachtsdeko - Anfang November und das in Südafrika bei sonnig warmen min- 25°C

Mittagessen haben wir uns bei einem Pakistanischen Curry Imbiss gegönnt - sehr empfehlenswert !
Naja und nachdem wir dann vor ner Stunde oder so zurück gekommen sind habe ich noch den Kühlschrank geputzt den ich heute morgen zum defrosten einfach nach draußen gestellt hatte. Jetzt ist der auch mal wieder sauber und ohne 3m dicke Eisschicht...

Morgen ... ja darüber werde ich gesondert berichten ... aber vermutlich erst am Montag, denn dann bin ich nochmal wieder in Quigney(East London) und habe kostenloses Internet.

© benste CC NC SA

05 Nov 2011 4:33pm GMT

31 Oct 2011

feedPython Software Foundation | GSoC'11 Students

Benedict Stein: Sterkspruit Computer Center

Sterkspruit is one of Hilltops Computer Centres in the far north of Eastern Cape. On the trip to J'burg we've used the opportunity to take a look at the centre.

Pupils in the big classroom


The Trainer


School in Countryside


Adult Class in the Afternoon


"Town"


© benste CC NC SA

31 Oct 2011 4:58pm GMT

Benedict Stein: Technical Issues

What are you doing in an internet cafe if your ADSL and Faxline has been discontinued before months end. Well my idea was sitting outside and eating some ice cream.
At least it's sunny and not as rainy as on the weekend.


© benste CC NC SA

31 Oct 2011 3:11pm GMT

30 Oct 2011

feedPython Software Foundation | GSoC'11 Students

Benedict Stein: Nellis Restaurant

For those who are traveling through Zastron - there is a very nice Restaurant which is serving delicious food at reasanable prices.
In addition they're selling home made juices jams and honey.




interior


home made specialities - the shop in the shop


the Bar


© benste CC NC SA

30 Oct 2011 4:47pm GMT

29 Oct 2011

feedPython Software Foundation | GSoC'11 Students

Benedict Stein: The way back from J'burg

Having the 10 - 12h trip from J'burg back to ELS I was able to take a lot of pcitures including these different roadsides

Plain Street


Orange River in its beginngings (near Lesotho)


Zastron Anglican Church


The Bridge in Between "Free State" and Eastern Cape next to Zastron


my new Background ;)


If you listen to GoogleMaps you'll end up traveling 50km of gravel road - as it was just renewed we didn't have that many problems and saved 1h compared to going the official way with all it's constructions sites




Freeway


getting dark


© benste CC NC SA

29 Oct 2011 4:23pm GMT

28 Oct 2011

feedPython Software Foundation | GSoC'11 Students

Benedict Stein: Wie funktioniert eigentlich eine Baustelle ?

Klar einiges mag anders sein, vieles aber gleich - aber ein in Deutschland täglich übliches Bild einer Straßenbaustelle - wie läuft das eigentlich in Südafrika ?

Ersteinmal vorweg - NEIN keine Ureinwohner die mit den Händen graben - auch wenn hier mehr Manpower genutzt wird - sind sie fleißig mit Technologie am arbeiten.

Eine ganz normale "Bundesstraße"


und wie sie erweitert wird


gaaaanz viele LKWs


denn hier wird eine Seite über einen langen Abschnitt komplett gesperrt, so das eine Ampelschaltung mit hier 45 Minuten Wartezeit entsteht


Aber wenigstens scheinen die ihren Spaß zu haben ;) - Wie auch wir denn gücklicher Weise mussten wir nie länger als 10 min. warten.

© benste CC NC SA

28 Oct 2011 4:20pm GMT