29 May 2020

feedPlanet Lisp

Alexander Artemenko: data-table

This library provides a few methods to work with data tables. Think about it as a limited version of Pandas Dataframe for Common Lisp.

Data-table abilities are really modest. For example, it does not provide methods to investigate data in the table.

To overcome this limitation, we'll use another library - cl-ascii-table and define a describe-object method for data tables:

POFTHEDAY> (defparameter *dt*
             (make-instance 'data-table:data-table
                            :column-names '("Name" "Birthday" "Age")
                            :rows '(("Bob" "1985-05-17" 35)
                                    ("Alice" "1989-02-23" 31)
                                    ("John" "2000-01-03" 20))))

;; This is what we'll wee by default:
POFTHEDAY> (describe *dt*)
#<DATA-TABLE:DATA-TABLE {1003D50293}>
  [standard-object]

Slots with :INSTANCE allocation:
  COLUMN-NAMES                   = ("Name" "Birthday" "Age")
  COLUMN-TYPES                   = (STRING STRING (INTEGER 0 4611686018427387903))
  ROWS                           = (("Bob" "1985-05-17" 35) ("Alice" "1989-02-23" 31))

;; But with custom method we might make it more
;; more convenient:
POFTHEDAY> (defmethod describe-object ((obj data-table:data-table) stream)
             (loop with table = (ascii-table:make-table
                                 (data-table:column-names obj))
                   for row in (data-table:rows obj)
                   do (ascii-table:add-row table row)
                   finally (ascii-table:display table)))

POFTHEDAY> (describe *dt*)
+-------+------------+-----+
| Name  | Birthday   | Age |
+-------+------------+-----+
| Bob   | 1985-05-17 |  35 |
| Alice | 1989-02-23 |  31 |
| John  | 2000-01-03 |  20 |
+-------+------------+-----+

Now let's see what capabilities for data slicing does it have.

Here is how we can retrieve columns. Pay attention, column names are case insensitive. But if case mismatch, column names of the resulting table will be different. Maybe this is a bug:

POFTHEDAY> (describe
            (data-table:select-columns *dt*
                                       '("Name" "Age")))
+-------+-----+
| Name  | Age |
+-------+-----+
| Bob   |  35 |
| Alice |  31 |
| John  |  20 |
+-------+-----+
; No values
POFTHEDAY> (describe
            (data-table:select-columns *dt*
                                       '("name" "age")))
+-------+-----+
| name  | age |
+-------+-----+
| Bob   |  35 |
| Alice |  31 |
| John  |  20 |
+-------+-----+

There is also a way to select a subtable using indices.

This is how we can select contiguous columns with "Name" and "Age" labels:

;; We have to do call this method first.
;; It will fill a list of datatypes.
;; Without filled list of datatypes, make-sub-table will fail :(

POFTHEDAY> (data-table:coerce-data-table-of-strings-to-types *dt*)

POFTHEDAY> (describe
            (data-table:make-sub-table *dt*
                                       :lci 1 :uci 3))
+------------+-----+
| Birthday   | Age |
+------------+-----+
| 1985-05-17 |  35 |
| 1989-02-23 |  31 |
| 2000-01-03 |  20 |
+------------+-----+

If we want to select only the last two rows, then we have to do some math:

;; In pandas.DataFrame this will be just: dt.tail(2)
;;
POFTHEDAY> (let ((rows-count (data-table:number-of-rows *dt*)))
             (describe
              (data-table:make-sub-table *dt*
                                         :lri (- rows-count 2)
                                         :uri rows-count)))
+-------+------------+-----+
| Name  | Birthday   | Age |
+-------+------------+-----+
| Alice | 1989-02-23 |  31 |
| John  | 2000-01-03 |  20 |
+-------+------------+-----+

Also, we can combine column selection by name with make-sub-table. But this will create an intermediate table.

There is also a method overlay-region to combine two tables. Let's use it to add email for Genry and Mary:

POFTHEDAY> (defparameter *dt*
             (data-table:plists-to-data-table
              '((:id 1 :name "Bob")
                (:id 2 :name "Genry")
                (:id 3 :name "Mary"))))

POFTHEDAY> (describe *dt*)
+----+-------+
| ID | NAME  |
+----+-------+
|  1 | Bob   |
|  2 | Genry |
|  3 | Mary  |
+----+-------+

POFTHEDAY> (data-table:overlay-region
            (data-table:plists-to-data-table
             '((:email "genry@gmail.com")
               (:email "mary@hormail.com")))
            *dt*)
NIL

POFTHEDAY> (describe *dt*)
+------------------+-------+
| ID               | NAME  |
+------------------+-------+
| genry@gmail.com  | Bob   |
| mary@hormail.com | Genry |
|                3 | Mary  |
+------------------+-------+

POFTHEDAY> ;; Oh, this method modified the original table :(((

To fix the issue, we need to recreate an original table and to specify indexes for applied overlay:

POFTHEDAY> (defparameter *dt*
             (data-table:plists-to-data-table
              '((:id 1 :name "Bob")
                (:id 2 :name "Genry")
                (:id 3 :name "Mary"))))

POFTHEDAY> (data-table:overlay-region
            (data-table:plists-to-data-table
             '((:email "genry@gmail.com")
               (:email "mary@hormail.com")))
            *dt*
            :col-idx 2
            :row-idx 1)

;; But new column has no name now :(
POFTHEDAY> (describe *dt*)
+----+-------+------------------+
| ID | NAME  | NIL              |
+----+-------+------------------+
|  1 | Bob   | NIL              |
|  2 | Genry | genry@gmail.com  |
|  3 | Mary  | mary@hormail.com |
+----+-------+------------------+

It would be nice to have support for indices and joins like in Pandas. That is what I expect from the data manipulation library.

We've viewed almost all core functions of the data-table library. There are also data coercion and comparison functions. But I think this library has lots of room for improvement.

Maybe @guicho271828 has plans to build a dataframe library as supplementary to his NumCL?

29 May 2020 12:15am GMT

28 May 2020

feedPlanet Debian

Antoine Beaupré: Upgrading my home server uplink

For more than a few decades now (!), I've been running my own server. First it was just my old Pentium 1 squatting on university networks, but eventually grew into a real server somewhere at the dawn of the millenia. Apart from the university days, the server was mostly hosted over ADSL links, first a handful of megabits, up to the current 25 Mbps down, 6 Mbps up that the Bell Canada network seems to allow to its resellers (currently Teksavvy Internet, or TSI).

Why change?

Obviously, this speed is showing its age, and especially in this age of Pandemia where everyone is on videoconferencing all the time. But it's also inconvenient when I need to upload large files on the network. I also host a variety of services on this network, and I always worry that any idiot can (rather trivially) DoS my server, so I often feel I should pack a little more punch at home (although I have no illusions about my capacity of resisting any sort of DoS attack at home of course).

Also, the idea of having gigabit links at home brings back the idea of the original internet, that everyone on the internet is a "peer". "Client" and "servers" are just a technical distinction and everyone should be able to run a server.

Requirements

So I'm shopping for a replacement. The requirements are:

  1. higher speed than 25/6, preferably 100mbps down, 30mbps up, or more. ideally 1gbps symmetric.

  2. static or near-static IP address: I run a DNS server with its IP in the glue records (although the latter could possibly be relaxed). ideally a /29 or more.

  3. all ports open: I run an SMTP server (incoming and outgoing) along with a webserver and other experiments. ideally, no firewall or policy should be blocking me from hosting stuff, unless there's an attack or security issue, obviously.

  4. clean IP address: the SMTP server needs to have a good reputation, so the IP address should not be in a "residential space" pool.

  5. IPv6 support: TSI offers IPv6 support, but it is buggy (I frequently have to restart the IPv6 interface on the router because the delegated block stops routing, and they haven't been able to figure out the problem). ideally, a /56.

  6. less than 100$/mth, ideally close to the current 60$/mth I pay.

(All amounts in $CAD.)

Contestants

I wrote a similar message asking three major ISPs in my city for those services, including business service if necessary:

I have not contacted those providers:

I might have forgotten some, let me know if you're in the area and have a good recommendation. I'll update this post with findings as they come in.

Keep in mind that I am in a major Canadian city, less than a kilometer from a major telco exchange site, so it's not like I'm in a rural community. This should just work.

TSI

First answer from TSI was "we do not provide 30mbps upload on residential services", even though they seem to have that package on their website. They confirmed that they "don't have a option more than 10 mbps upload."

Oricom

No response yet.

Ebox

No response yet.

28 May 2020 5:40pm GMT

Bits from Debian: New Debian Developers and Maintainers (March and April 2020)

The following contributors got their Debian Developer accounts in the last two months:

The following contributors were added as Debian Maintainers in the last two months:

Congratulations!

28 May 2020 4:30pm GMT

feedPlanet Grep

Frank Goossens: Google PageSpeed Insights updated, new metrics and recommendations!

If you tested your blog's performance on Google PageSpeed Insights yesterday and do so again today, you might be in for a surprise with a lower score even if not one byte (letter) got changed on your site. The reason: Google updated PageSpeed Insights to Lighthouse 6, which changes the KPI's (the lab data metrics) that are reported, adds new opportunities and recommendations and changes the way the total score is calculated.

So all starts with the changed KPI's in the lab metrics really; whereas up until yesterday First Contentful Paint, Speed Index, Time to Interactive, First Meaningful Paint, First CPU Idle and First input delay were measured, the last 3 ones are now not shown any more, having been replaced by:

The total score is calculated based on all 6 metrics, but the weight of the 3 "old" ones (FCP, SI, TTI) is significantly lowered (from 80 to 45%) and the new LCP & TBT account for a whopping 50% of your score (CLS is only 5%).

Lastly some one very interesting opportunity and two recommendations I noticed;

Summary: Google Pagespeed Insights changed a lot and it forces performance-aware users to stay on their toes. Especially sites with lots of (3rd party) JavaScript might want to reconsider some of the tools used.

Possibly related twitterless twaddle:

28 May 2020 11:36am GMT

Xavier Mertens: [SANS ISC] Flashback on CVE-2019-19781

I published the following diary on isc.sans.edu: "Flashback on CVE-2019-19781":

First of all, did you know that the Flame malware turned 8 years today! Happy Birthday! This famous malware discovered was announced on May 28th, 2012. The malware was used for targeted cyber espionage activities in the Middle East area. If this malware was probably developed by a nation-state organization. It infected a limited amount of hosts (~1000 computers) making it a targeted attack… [Read more]

[The post [SANS ISC] Flashback on CVE-2019-19781 has been first published on /dev/random]

28 May 2020 11:31am GMT

feedPlanet Debian

Elana Hashman: Presenter mode in LibreOffice Impress without an external display

I typically use LibreOffice Impress for my talks, much to some folks' surprise. Yes, you can make slides look okay with free software! But there's one annoying caveat that has bothered me for ages.

Impress makes it nearly impossible to enter presenter mode with a single display, while also displaying slides. I have never understood this limitation, but it's existed for a minimum of seven years.

I've tried all sorts of workarounds over the years, including a macro that forces LibreOffice into presenter mode, which I never was able to figure out how to reverse once I ran it...

This has previously been an annoyance but never posed a big problem, because when push came to shove I could leave my house and use an external monitor or screen when presenting at meetups. But now, everything's virtual, I'm in lockdown indefinitely, and I don't have another display available at home. And about 8 hours before speaking at a meetup today, I realized I didn't have a way to share my slides while seeing my speaker notes. Uh oh.

So I got this stupid idea.

...why don't I just placate LibreOffice with a FAKE display?

Virtual displays with xrandr

My GPU had this capability innately, it turns out, if I could just whisper the right incantations to unlock its secrets:

ehashman@red-dot:~$ cat /usr/share/X11/xorg.conf.d/20-intel.conf 
Section "Device"
    Identifier "intelgpu0"
    Driver "intel"
    Option "VirtualHeads" "1"
EndSection

After restarting X to allow this newly created config to take effect, I now could see two new virtual displays available for use:

ehashman@red-dot:~$ xrandr
Screen 0: minimum 8 x 8, current 3840 x 1080, maximum 32767 x 32767
eDP1 connected primary 1920x1080+0+0 (normal left inverted right x axis y axis) 310mm x 170mm
   1920x1080     60.01*+  59.93  
   ...
   640x360       59.84    59.32    60.00  
DP1 disconnected (normal left inverted right x axis y axis)
DP2 disconnected (normal left inverted right x axis y axis)
HDMI1 disconnected (normal left inverted right x axis y axis)
HDMI2 disconnected (normal left inverted right x axis y axis)
VIRTUAL1 disconnected (normal left inverted right x axis y axis)
VIRTUAL2 disconnected (normal left inverted right x axis y axis)

Nice. Now, to actually use it:

ehashman@red-dot:~$ xrandr --addmode VIRTUAL1 1920x1080
ehashman@red-dot:~$ xrandr --output VIRTUAL1 --mode 1920x1080 --right-of eDP1

And indeed, after running these commands, I found myself with a virtual display, very happy to black hole all my windows, available to the imaginary right of my laptop screen.

This allowed me to mash that "Present" button in LibreOffice and get my presenter notes on my laptop display, while putting my actual slides in a virtual time-out that I could still screenshare!

Wouldn't it be nice if LibreOffice just fixed the actual bug? 🤷

Well, actually...

I must forgive myself for my stage panic. The talk ended up going great, and the immediate problem was solved. But it turns out this bug has been addressed upstream! It's just... not well-documented.

A couple years ago, there was a forum post on ask.libreoffice.org that featured this exact question, and a solution was provided!

Yes, use Open Expert Configuration via Tools > Options > LibreOffice > Advanced. Search for StartAlways. You should get a node org.openoffice.Office.PresenterScreen with line Presenter. Double-click that line to toggle the boolean value to true.

I tested this out locally and... yeah that works. But it turns out the bug from 2013 had not been updated with this solution until just a few months ago.

There are very limited search results for this configuration property. I wish this was much better documented. But so it goes with free software; here's a hack and a real solution as we all try to improve things :)

28 May 2020 1:15am GMT

feedPlanet Python

Matt Layman: Bread and Butter Django - Building SaaS #58

In this episode, I worked on a views and templates. There are a number of core pages that are required to flesh out the minimal interface for the app. We're building them. I began by showing the page that we were going to work on. I outlined the changes I planned to make, then we started. The first thing we added was data about the school year, the main model on display in the page.

28 May 2020 12:00am GMT

27 May 2020

feedPlanet Python

Gal Varoquaux: Technical discussions are hard; a few tips

Note

This post discuss the difficulties of communicating while developing open-source projects and tries to gives some simple advice.

A large software project is above all a social exercise in which technical experts try to reach good decisions together, for instance on github pull requests. But communication is difficult, in particular between diverging points of view. It is easy to underestimate how much well-intended persons can misunderstand each-other and get hurt, in open source as elsewhere. Knowing why there are communication challenges can help, as well as applying a few simple rules.

Contents


The first challenge is to understand the other's point of view: the different parties see the problem differently.

Maintainer's anxiety

Open source can be anxiety-generating for the maintainers

Maintainers ensure the quality and the long-term life of an open-source project. As such, they feel responsible for any shortcoming in the product. In addition, they often do this work because they care, even though it may not bring any financial support. But they can quickly become a converging point of anxiety-generating feedback:

  • Code has bugs; the more code, the more bugs. Watching a issue tracker fill up with a long list of bugs is frightening to people who feel in charge.
  • Given that maintainers are visible and qualified, they become the target of constant requests for attention: from pleas to prioritize a specific issue to solicitations for advice.
  • A small fraction of these interactions come as plain aggressions. I have been insulted many times by unsatisfied users. Each time, it hurts me a lot. My policy is to disengage from the conversation, but I am left shaking and staring at my computer in the evening.

Related writings

Ralf Gommers discusses the cost of an open source contribution, from the point of view of the maintainer.

Ilya Grigorik suggests: Don't push your pull request.

Brett Cannon: Setting expectations for open source participation

The more popular a project, the more weight it puts on its maintainers' shoulders. A consequence is that maintainers are tired, and can sometimes approach discussions in a defensive way. Also, we may be plain scared of integrating a code that we do not fully comprehend.

Open-source developers may even, unconsciously, adopt a simple, but unfortunate, protection mechanism: being rude. The logic is flawless: if I am nasty to people, or I set unreasonnable expectations, people will let me alone. Alas, this strategy leads to toxic environments. It not only makes people unhappy but also harms the community dynamics that ground the excellence of open source.

The danger abusive gatekeeping

A maintainer quickly learns that every piece of code, no matter how cute it might be, will give him work in the long run, just like a puppy. This is unavoidable given that the complexity of code grows faster than its number of features [1], and, even for a company as rich as Google, project maintenance becomes intractable on huge projects [2].

[1] An Experiment on Unit Increase in Problem Complexity, Woodfield 1979
[2] To quote tensorflow developers "Every [code addition] takes around 16 CPU/GPU hours of [quality control]. As such, we cannot just run every [code addition] through the [quality control] infrastructure."

A maintainer's job is to say no often, to protect the project. But, as any gatekeeping, it can unfortunately become an excercise in unchecked power. Making objective choices for these difficult decisions is hard, and we all tend naturally to trust more people that we know.


Most often we are not aware of our shortcomings, let alone are we doing them on purpose.

Contributor's fatigue

A new contributor starting a conversation with a group of seasoned project maintainers may easily feel an imposter. The new contributor knows less about the project. In addition, he or she is engaging with a group of people that know each-other well, and is not yet part of that inner group.

This person does not know the code base, or the conventions, and must make extra efforts, compared to the seasoned developers, to propose a contribution suitable for the project. Often, he or she does not understand fully the reasons for the project guidelines, or for the feedback given. Request for changes can easily be seen as trifles.

Integrating the contribution can often be a lengthy process -in particular in scikit-learn. Indeed, it will involve not only shaping up the contribution, but also learning the skills and discovering the process. These long cycles can undermine motivation: humans need successes to feel enthusiasm. Also, the contributor may legitimately worry: Will all these efforts be fruitful? Will the contribution make its way to the project?

Note that for these reasons, it is recommended to start contributing with very simple features, and to seek feedback on the scope of the contribution before writing the code.

Finally, contributors are seldom paid to work on the project, and there is no single line of command that makes decisions and controls incentives for all the people on the project. No one is responsible when things go astray, which means that the weight falls on the shoulder of the individuals.


The danger behind the lengthy cycle of reviews and improvements needed to contribute is death by a thousands cuts. The contributor looses motivation, and no longer finds the energy to finish the work.

How about users?

This article is focused on developers. Yet, users are also an important part of the discussion around open source.

Often communication failures with users are due to frustration. Frustration of being unable to use the software, of hitting a bug, of seeing an important issue still not addressed. This frustration stems from incorrect expectations, which can often be traced to misunderstanding of the processes and the dynamics. Managing expectations is important to improve the dialogue, via the documentation, via notes on the issue tracker.


Communication is hard

Communication is hard: messages are sometimes received differently than we would like. Overworked people discussing very technically challenging issues only makes the matter worse. I have seen people not come across well, while I know they are absolutely lovely and caring.

We are human beings; we are limited; we misunderstand things, and we have feelings.


Emotions - My most vivid memory of a communication failure was when I was a sailing instructor. Trainees that were under my responsibility had put themselves at risk, causing me a lot of worry. During the debrief, I was angry. My failure to convey the messages without emotional loading undermined my leadership on the group, putting everybody at risk for the rest of the week.

Inability to understand the others' point of view, or to communicate ours, can bring in emotions. Emotions most often impedes technical communication.


Limited attention - We, in particular maintainers, are bombarded with email, notifications, text and code to read. As a consequence, it is easy to read things too fast, to stop in the middle, to forget.


Language barriers - Most discussions happen in English; but most of us are not native English speakers. We may hide well our difficulties, but nuances are often lost.


Clique effects - Most interactions in open source are done in writing, with low communication bandwidth. It can be much harder to convince a maintainer on the other side of the world than a colleague in the same room. Schools of thoughts naturally emerge when people work a lot together. These create bubbles, where we have the impression that everything we say is obvious and uncontroversial, and yet we fail to convince people outside of our bubble.


Little things that help

Communication can improved by continuously working on it [3]. It may be obvious to some, but it personally took me many years to learn.

[3] Training materials for managers often discuss communication, and give tricks. I am sure that there are better references than my list below. But that's the best I can do.

Hear the other: exchange

Related presentation

How can we have healthier technical discussions? by Nathaniel J. Smith

Foster multiway discussions - The goal of a technical discussion is to come up to the best solution. Better solutions emerge via confronting different points of view: a single brilliant individual probably cannot find or recognize the best solution alone.

  • Integrate input from as many perspectives as possible.
  • Make sure everyone feels heard.

Don't seek victory - Most important to keep in mind is that giving up on an argument and accepting the other point of view is a perfectly valid option. I naturally biased to think that my view on topics dear to me is the right one. However, I've learned that adopting the view of the other could bring a lot to the social dynamics of a project: we are often debating over details and the bigger benefit comes from moving forward.

In addition, if several very bright people have different conclusions than me about something that they've thought a lot, who am I to disagree?


Convey ideas well: pedagogy

Explain - Give the premises of your thoughts. Unroll your thought processes. People are not sitting in your head, and need to hear not only your conclusion, but how you got there.

Repeat things - Account for the fact that people can forget, and never hesitate to gently restate important points. Reformulating differently can also help explaining.

Keep it short - A typical reading speed is around 200 words a minute. People have limited time and attention span. The greatest help you can provide to your reader is to condense your ideas: let us avoid long threads that require several dozens of minutes to read and digest. There is a tension between this point and the above. My suggestion: remove every word that is not useful, move details to footnotes or postscriptums.


Cater for emotions: tone

Related good advice

Mozilla participation guide, expected behavior section

Stay technical - Always try to get to the technical aspect of the matter, and never the human. Give specific code and wording suggestions. When explaining a decision, give technical arguments, even if they feel obvious to you.

Be positive - Being positive in general helps people feeling happy and motivated. It is well known that positive feedback leads to quicker progress than negative, as revealed eg by studies of class rooms. I am particularly guilty of this: I always forget to say something nice, although I may be super impressed by a contribution. Likewise, avoid negative words when giving feedback (stay technical).

Avoid "you" - The mere use of the pronoun "you" puts the person we are talking to in the center of message. But the message should not be about the person, it should be about the work. It's very easy to react emotionally when it's about us. The passive voice can be useful to avoid putting people as the topic. If the topic is indeed people, sometimes "we" is an adequate substitute for "you".

Assume good faith - There are so many misunderstandings that can happen. People forget things, people make mistakes, people fail to convey their messages. Most often, all these failures are in good faith, and misunderstandings are legitimate. In the rare cases there might possibly be some bad faith, accounting for it will only make communication worse, not better. Along the same line, we should ignore when we feel assaulted or insulted, and avoid replying in kind.

Choose words wisely - The choice of words matter, because they convey implicit messages. In particular, avoid terms that carry judgement values: "good" or "bad". For example "This is done wrong" (note that this sentence already avoids "you"), could be replaced by "There might be more numerically stable / efficient way of doing it" (note also the use of precise technical wording rather than the generic term "better").

Use moderating words - Try to leave room for the other in the discussion. Statements too assertive close the door to different points of view: "this must be changed" (note the lack of "you") should be avoided while "this should be changed" is better. For this reason, this article is riddled with words such as "tend", "often", "feel", "may", "might".

Don't blame someone else - If you feel that there is some pattern that you would like to change, do not point fingers, do not blame others. Rather, point yourself at the center of the story, find an example of this pattern with you, and the message should be that "it is a pattern that we should avoid. "We" is such a powerful term. It unites; it builds a team.

Give your understanding - If you feel that there is a misunderstanding, explain how you are feeling. But do it using "I", and not "you", and acknowledge the subjectivity: "I feel ignored" rather than "you are ignoring me". Even better: only talk about the feeling: "I am loosing motivation, because this is not moving forward", or "I think that am failing to convey why this numerical problem is such an important issue" (note the use of "I think", which avoids casting the situation as necessarily true).

Implicit messages

The four sides view of communication highlights the multiple messages present even in simple statements.


I hope this can be useful. I personally try to apply these rules, because I want to work better with others.

Thanks

to many who gave me feedback: Adrin Jalali, Andreas Mueller, Elizabeth DuPre, Emmanuelle Gouillart, Guillaume Lemaitre, Joel Nothman, Joris Van den Bossche, Nicolas Hug.



PS: note how many times I've used "you" above. I can clearly get better at communication!

27 May 2020 10:00pm GMT

feedPlanet Lisp

Alexander Artemenko: cl-ppcre-unicode

This system adds Unicode support to the cl-ppcre.

What does it mean? It means that after loading cl-ppcre-unicode you'll be able to match against Unicode symbol properties.

A property matcher has a special syntax in cl-ppcre's regexps: \p{PropertyName}.

Here is an example:

;; This is how we can find out a position
;; of the first Cyrillic letter:

POFTHEDAY> (ppcre:scan "\\p{Cyrillic}"
                       "123Ю56")
3

;; Here we are extracting a
;; sequence of Emoji from the text:
POFTHEDAY> (ppcre:regex-replace
            ".*?([\\p{Emoticons}|\\p{Supplemental Symbols and Pictographs}]+).*"
            "Hello, Lisper! 🤗😃 How are you?"
            "\\1")
"🤗😃"

We are using two different Unicode classes as properties because these two characters belong to different classes.

You can use cl-unicode to discover the character's unicode class:

POFTHEDAY> (cl-unicode:code-block #\😃)
"Emoticons"

POFTHEDAY> (cl-unicode:code-block #\🤗)
"Supplemental Symbols and Pictographs"

The way, how cl-ppcre-unicode works is very interesting. It turns out that cl-ppcre has a special hook which allows you to define a property resolver.

For example, if you want to have a special property for vowels, you might do something like that:

POFTHEDAY> (defun my-property-resolver (property-name)
             (if (string-equal property-name
                               "vowel")
                 (rutils:fn vovel-p (character)
                   (member character '(#\A #\E #\I #\O #\U)
                           :test #'char-equal))
                 (cl-ppcre-unicode:unicode-property-resolver
                  property-name)))

POFTHEDAY> (setf cl-ppcre:*property-resolver*
                 #'my-property-resolver)

;; And now we can use the "Vowel" property in any
;; regular expressions!
POFTHEDAY> (ppcre:regex-replace-all
            "\\p{Vowel}"
            "Hello, Lisper! How are you?"
            "")
"Hll, Lspr! Hw r y?"

Isn't this cool!? 🤪

27 May 2020 8:15pm GMT

feedPlanet Python

Real Python: How to Write an Installable Django App

In the Django framework, a project refers to the collection of configuration files and code for a particular website. Django groups business logic into what it calls apps, which are the modules of the Django framework. There's plenty of documentation on how to structure your projects and the apps within them, but when it comes time to package an installable Django app, information is harder to find.

In this tutorial, you'll learn how to take an app out of a Django project and package it so that it's installable. Once you've packaged your app, you can share it on PyPI so that others can fetch it through pip install.

In this tutorial, you'll learn:

  • What the differences are between writing stand-alone apps and writing apps inside of projects
  • How to create a setup.cfg file for publishing your Django app
  • How to bootstrap Django outside of a Django project so you can test your app
  • How to test across multiple versions of Python and Django using tox
  • How to publish your installable Django app to PyPI using Twine

Be sure to follow along with the examples by downloading the source code at the link below:

Download the sample code: Click here to get the code you'll use to learn how to write an installable Django app in this tutorial.

Prerequisites

This tutorial requires some familiarity with Django, pip, PyPI, pyenv (or an equivalent virtual environment tool), and tox. To learn more about these topics, check out:

Starting a Sample Django App in a Project

This tutorial includes a working package to help guide you through the process of making an installable Django app. You can download the source code at the link below:

Download the sample code: Click here to get the code you'll use to learn how to write an installable Django app in this tutorial.

Even if you originally intend to make your Django app available as a package, you're likely to start inside a project. To demonstrate the process of moving from Django project to installable Django app, I've made two branches available in the repo. The project branch is the starting state of an app inside of a Django project. The master branch is the finished installable app.

You can also download the finished app at the PyPI realpython-django-receipts package page. You can install the package by running pip install realpython-django-receipts.

The sample app is a short representation of the line items on a receipt. In the project branch, you'll find a directory named sample_project that contains a working Django project. The directory looks like this:

sample_project/
│
├── receipts/
│   ├── fixtures/
│   │   └── receipts.json
│   │
│   ├── migrations/
│   │   ├── 0001_initial.py
│   │   └── __init__.py
│   │
│   ├── __init__.py
│   ├── admin.py
│   ├── apps.py
│   ├── models.py
│   ├── tests.py
│   ├── urls.py
│   └── views.py
│
├── sample_project/
│   ├── __init__.py
│   ├── asgi.py
│   ├── settings.py
│   ├── urls.py
│   └── wsgi.py
│
├── db.sqlite3
├── manage.py
├── resetdb.sh
└── runserver.sh

The most current version of Django at the time this tutorial was written was 3.0.4, and all testing was done with Python 3.7. None of the steps outlined in this tutorial should be incompatible with earlier versions of Django-I've used these techniques since Django 1.8. However, some changes are necessary if you're using Python 2. To keep the examples simple, I've assumed Python 3.7 across the code base.

Creating the Django Project From Scratch

The sample project and receipts app were created using the Django admin command and some small edits. To start, run the following code inside of a clean virtual environment:

$ python -m pip install Django
$ django-admin startproject sample_project
$ cd sample_project
$ ./manage.py startapp receipts

This creates the sample_project project directory structure and a receipts app subdirectory with template files that you'll use to create your installable Django app.

Next, the sample_project/settings.py file needs a few modifications:

  • Add '127.0.0.1' to the ALLOWED_HOSTS setting so you can test locally.
  • Add 'receipts' to the INSTALLED_APPS list.

You'll also need to register the receipts app's URLs in the sample_project/urls.py file. To do so, add path('receipts/', include('receipts.urls')) to the url_patterns list.

Exploring the Receipts Sample App

Read the full article at https://realpython.com/installable-django-app/ »


[ Improve Your Python With 🐍 Python Tricks 💌 - Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

27 May 2020 2:00pm GMT

26 May 2020

feedPlanet Lisp

Alexander Artemenko: jose

JOSE is an implementation of Javascript Object Signing and Encryption.

This @nitro_idiot's library implements a JSON Web Signature and allows to encode, decode and inspect JWT tokens.

JOSE can be useful to implement signed token exchange between microservices.

I found a great article on what JWT, JWS and JWE are. Read it you want more about them. To demonstrate, how does token inspection work, I took a JWT token from the article and parsed it with Common Lisp JOSE:

POFTHEDAY> (jose/jwt:inspect-token
            "eyJhbGciOiJSUzI1NiIsImtpZCI6Ijc4YjRjZjIzNjU2ZGMzOTUzNjRmMWI2YzAyOTA3NjkxZjJjZGZmZTEifQ.eyJpc3MiOiJhY2NvdW50cy5nb29nbGUuY29tIiwic3ViIjoiMTEwNTAyMjUxMTU4OTIwMTQ3NzMyIiwiYXpwIjoiODI1MjQ5ODM1NjU5LXRlOHFnbDcwMWtnb25ub21ucDRzcXY3ZXJodTEyMTFzLmFwcHMuZ29vZ2xldXNlcmNvbnRlbnQuY29tIiwiZW1haWwiOiJwcmFiYXRoQHdzbzIuY29tIiwiYXRfaGFzaCI6InpmODZ2TnVsc0xCOGdGYXFSd2R6WWciLCJlbWFpbF92ZXJpZmllZCI6dHJ1ZSwiYXVkIjoiODI1MjQ5ODM1NjU5LXRlOHFnbDcwMWtnb25ub21ucDRzcXY3ZXJodTEyMTFzLmFwcHMuZ29vZ2xldXNlcmNvbnRlbnQuY29tIiwiaGQiOiJ3c28yLmNvbSIsImlhdCI6MTQwMTkwODI3MSwiZXhwIjoxNDAxOTEyMTcxfQ.TVKv-pdyvk2gW8sGsCbsnkqsrS0T-H00xnY6ETkIfgIxfotvFn5IwKm3xyBMpy0FFe0Rb5Ht8AEJV6PdWyxz8rMgX2HROWqSo_RfEfUpBb4iOsq4W28KftW5H0IA44VmNZ6zU4YTqPSt4TPhyFC9fP2D_Hg7JQozpQRUfbWTJI")

(("exp" . 1401912171) ("iat" . 1401908271) ("hd" . "wso2.com")
 ("aud"
  . "825249835659-te8qgl701kgonnomnp4sqv7erhu1211s.apps.googleusercontent.com")
 ("email_verified" . T) ("at_hash" . "zf86vNulsLB8gFaqRwdzYg")
 ("email" . "prabath@wso2.com")
 ("azp"
  . "825249835659-te8qgl701kgonnomnp4sqv7erhu1211s.apps.googleusercontent.com")
 ("sub" . "110502251158920147732") ("iss" . "accounts.google.com"))

(("alg" . "RS256") ("kid" . "78b4cf23656dc395364f1b6c02907691f2cdffe1"))

#(77 82 175 250 151 114 190 77 160 91 203 6 176 38 236 158 74 172 173 45 19 248
  125 52 198 118 58 17 57 8 126 2 49 126 139 111 22 126 72 192 169 183 199 32
  76 167 45 5 21 237 17 111 145 237 240 1 9 87 163 221 91 44 115 242 179 32 95
  97 209 57 106 146 163 244 95 17 245 41 5 190 34 58 202 184 91 111 10 126 213
  185 31 66 0 227 133 102 53 158 179 83 134 19 168 244 173 225 51 225 200 80
  189 124 253 131 252 120 59 37 10 51 165 4 84 125 181 147 36)

Inspect returns payload, headers and the signature.

There are also two methods encode and decode. Here is how we can use them to issue and verify JWT token:

POFTHEDAY> (defparameter
               *secret* (ironclad:ascii-string-to-byte-array
                         "I Love Common Lisp!"))

POFTHEDAY> (defparameter
               *token*
             (jose:encode :hs256 *secret*
                          '(("user"  . "Bob")
                            ("email" . "bob@gmail.com"))))

;; Now this token can be sent to the authenticated
;; user so that user can pass it back during API calls:
POFTHEDAY> *token*
"eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1c2VyIjoiQm9iIiwiZW1haWwiOiJib2JAZ21haWwuY29tIn0.NLgg5RxlKDNqw1cqFU0_HysIu-zO7JBYUQN2IZF6c6w"

;; And when we'll receive such API call,
;; we can know who this user is.
POFTHEDAY> (jose:decode :hs256 *secret*
                        *token*)
(("user" . "Bob") ("email" . "bob@gmail.com"))
(("alg" . "HS256") ("typ" . "JWT"))

A cool feature of JWT token is that it is signed and you can trust the payload's content. Let's pretend, that Bob is the "evil hacker" who wants to get access to another account.

To do this, Bob will modify payload and use header and sign from the original token:

POFTHEDAY> (defun replace-payload (token new-payload)
             (rutils:with (((original-header rutils:_ original-sign)
                            (str:split "." token))
                           (json-payload
                            (jonathan:to-json new-payload :from :alist))
                           (new-encoded-payload
                            (jose/base64:base64url-encode json-payload)))
               (str:join "."
                         (list original-header
                               new-encoded-payload
                               original-sign))))

POFTHEDAY> (defparameter *new-token*
             (replace-payload *token*
                              '(("user"  . "Alice")
                                ("email" . "alice@wonderland.in"))))

;; Now we'll try to decode token on the server-side
;; and receive error from JOSE:
POFTHEDAY> (jose:decode :hs256 *secret*
                        *new-token*)
; Debugger entered on #<JOSE/ERRORS:JWS-VERIFICATION-ERROR {1005C37033}>
[1] POFTHEDAY> 
; Evaluation aborted on #<JOSE/ERRORS:JWS-VERIFICATION-ERROR {1005C37033}>

;; But we still can inspect bad token because
;; it's content is not encrypted:
POFTHEDAY> (jose:inspect-token *new-token*)

(("email" . "alice@wonderland.in") ("user" . "Alice"))

(("alg" . "HS256") ("typ" . "JWT"))

#(52 184 32 229 28 101 40 51 106 195 87 42 21 77 63 31 43 8 187 236 206 236 144
  88 81 3 118 33 145 122 115 172)

BTW, as you can see, tokens are not encoded, they are signed. Because of that, you should pass them only over encrypted channels.

26 May 2020 8:15pm GMT

25 May 2020

feedPlanet Grep

Mattias Geniar: What else can you stuff in a certificate chain?

I recently learned that quite a few (old) root certificates are going to expire, and many websites still send those along in the TLS handshake.

25 May 2020 12:00am GMT

27 Apr 2020

feedPlanet Sun

The Hubble Space Telescope celebrates its 30th birthday

For 30 years, the space telescope Hubble has provided the most impressive images from the vast wide space. The telescope was developed by the US space agency NASA and its European counterpart ESA. Hubble started its journey into space on 24 April 1990. With the help of the space shuttle "Discovery" it was lifted into ... Read more

27 Apr 2020 3:47pm GMT

26 Apr 2019

feedPlanet Sun

First Image of a Black Hole – Event Horizon

The Event Horizon Telescope (EHT) - a planet-scale array of 8 ground-based radio telescopes and part of an international collaboration - captured the first image of a black hole. On April 10th 2019, EHT researchers disclosed the first direct visual evidence of a supermassive black hole in the heart of the Galaxy Messier 87.

26 Apr 2019 2:32am GMT

04 Nov 2018

feedPlanet Sun

5 Budget-Friendly Telescopes You Can Choose For Viewing Planets

Socrates couldn't have been more right when he said: "I know one thing, that I know nothing." Even with all of the advancements we, as a species, have made in this world, it's still nothing compared to countless of wonders waiting to be discovered in the vast universe. If you've recently developed an interest in ... Read more

04 Nov 2018 1:27pm GMT

10 Nov 2011

feedPlanetJava

OSDir.com - Java: Oracle Introduces New Java Specification Requests to Evolve Java Community Process

From the Yet Another dept.:

To further its commitment to the Java Community Process (JCP), Oracle has submitted the first of two Java Specification Requests (JSRs) to update and revitalize the JCP.

10 Nov 2011 6:01am GMT

OSDir.com - Java: No copied Java code or weapons of mass destruction found in Android

From the Fact Checking dept.:

ZDNET: Sometimes the sheer wrongness of what is posted on the web leaves us speechless. Especially when it's picked up and repeated as gospel by otherwise reputable sites like Engadget. "Google copied Oracle's Java code, pasted in a new license, and shipped it," they reported this morning.



Sorry, but that just isn't true.

10 Nov 2011 6:01am GMT

OSDir.com - Java: Java SE 7 Released

From the Grande dept.:

Oracle today announced the availability of Java Platform, Standard Edition 7 (Java SE 7), the first release of the Java platform under Oracle stewardship.

10 Nov 2011 6:01am GMT

08 Nov 2011

feedfosdem - Google Blog Search

papupapu39 (papupapu39)'s status on Tuesday, 08-Nov-11 00:28 ...

papupapu39 · http://identi.ca/url/56409795 #fosdem #freeknowledge #usamabinladen · about a day ago from web. Help · About · FAQ · TOS · Privacy · Source · Version · Contact. Identi.ca is a microblogging service brought to you by Status.net. ...

08 Nov 2011 12:28am GMT

05 Nov 2011

feedfosdem - Google Blog Search

Write and Submit your first Linux kernel Patch | HowLinux.Tk ...

FOSDEM (Free and Open Source Development European Meeting) is a European event centered around Free and Open Source software development. It is aimed at developers and all interested in the Free and Open Source news in the world. ...

05 Nov 2011 1:19am GMT

03 Nov 2011

feedfosdem - Google Blog Search

Silicon Valley Linux Users Group – Kernel Walkthrough | Digital Tux

FOSDEM (Free and Open Source Development European Meeting) is a European event centered around Free and Open Source software development. It is aimed at developers and all interested in the Free and Open Source news in the ...

03 Nov 2011 3:45pm GMT

28 Oct 2011

feedPlanet Ruby

O'Reilly Ruby: MacRuby: The Definitive Guide

Ruby and Cocoa on OS X, the iPhone, and the Device That Shall Not Be Named

28 Oct 2011 8:00pm GMT

14 Oct 2011

feedPlanet Ruby

Charles Oliver Nutter: Why Clojure Doesn't Need Invokedynamic (Unless You Want It to be More Awesome)

This was originally posted as a comment on @fogus's blog post "Why Clojure doesn't need invokedynamic, but it might be nice". I figured it's worth a top-level post here.

Ok, there's some good points here and a few misguided/misinformed positions. I'll try to cover everything.

First, I need to point out a key detail of invokedynamic that may have escaped notice: any case where you must bounce through a generic piece of code to do dispatch -- regardless of how fast that bounce may be -- prevents a whole slew of optimizations from happening. This might affect Java dispatch, if there's any argument-twiddling logic shared between call sites. It would definitely affect multimethods, which are using a hand-implemented PIC. Any case where there's intervening code between the call site and the target would benefit from invokedynamic, since invokedynamic could be used to plumb that logic and let it inline straight through. This is, indeed, the primary benefit of using invokedynamic: arbitrarily complex dispatch logic folds away allowing the dispatch to optimize as if it were direct.

Your point about inference in Java dispatch is a fair one...if Clojure is able to infer all cases, then there's no need to use invokedynamic at all. But unless Clojure is able to infer all cases, then you've got this little performance time bomb just waiting to happen. Tweak some code path and obscure the inference, and kablam, you're back on a slow reflective impl. Invokedynamic would provide a measure of consistency; the only unforeseen perf impact would be when the dispatch turns out to *actually* be polymorphic, in which case even a direct call wouldn't do much better.

For multimethods, the benefit should be clear: the MM selection logic would be mostly implemented using method handles and "leaf" logic, allowing hotspot to inline it everywhere it is used. That means for small-morphic MM call sites, all targets could potentially inline too. That's impossible without invokedynamic unless you generate every MM path immediately around the eventual call.

Now, on to defs and Var lookup. Depending on the cost of Var lookup, using a SwitchPoint-based invalidation plus invokedynamic could be a big win. In Java 7u2, SwitchPoint-based invalidation is essentially free until invalidated, and as you point out that's a rare case. There would essentially be *no* cost in indirecting through a var until that var changes...and then it would settle back into no cost until it changes again. Frequently-changing vars could gracefully degrade to a PIC.

It's also dangerous to understate the impact code size has on JVM optimization. The usual recommendation on the JVM is to move code into many small methods, possibly using call-through logic as in multimethods to reuse the same logic in many places. As I've mentioned, that defeats many optimizations, so the next approach is often to hand-inline logic everywhere it's used, to let the JVM have a more optimizable view of the system. But now we're stepping on our own feet...by adding more bytecode, we're almost certainly impacting the JVM's optimization and inlining budgets.

OpenJDK (and probably the other VMs too) has various limits on how far it will go to optimize code. A large number of these limits are based on the bytecoded size of the target methods. Methods that get too big won't inline, and sometimes won't compile. Methods that inline a lot of code might not get inlined into other methods. Methods that inline one path and eat up too much budget might push out more important calls later on. The only way around this is to reduce bytecode size, which is where invokedynamic comes in.

As of OpenJDK 7u2, MethodHandle logic is not included when calculating inlining budgets. In other words, if you push all the Java dispatch logic or multimethod dispatch logic or var lookup into mostly MethodHandles, you're getting that logic *for free*. That has had a tremendous impact on JRuby performance; I had previous versions of our compiler that did indeed infer static target methods from the interpreter, but they were often *slower* than call site caching solely because the code was considerably larger. With invokedynamic, a call is a call is a call, and the intervening plumbing is not counted against you.

Now, what about negative impacts to Clojure itself...

#0 is a red herring. JRuby supports Java 5, 6, and 7 with only a few hundred lines of changes in the compiler. Basically, the compiler has abstract interfaces for doing things like constant lookup, literal loading, and dispatch that we simply reimplement to use invokedynamic (extending the old non-indy logic for non-indified paths). In order to compile our uses of invokedynamic, we use Rémi Forax's JSR-292 backport, which includes a "mock" jar with all the invokedynamic APIs stubbed out. In our release, we just leave that library out, reflectively load the invokedynamic-based compiler impls, and we're off to the races.

#1 would be fair if the Oracle Java 7u2 early-access drops did not already include the optimizations that gave JRuby those awesome numbers. The biggest of those optimizations was making SwitchPoint free, but also important are the inlining discounting and MutableCallSite improvements. The perf you see for JRuby there can apply to any indirected behavior in Clojure, with the same perf benefits as of 7u2.

For #2, to address the apparent vagueness in my blog post...the big perf gain was largely from using SwitchPoint to invalidate constants rather than pinging a global serial number. Again, indirection folds away if you can shove it into MethodHandles. And it's pretty easy to do it.

#3 is just plain FUD. Oracle has committed to making invokedynamic work well for Java too. The current thinking is that "lambda", the support for closures in Java 7, will use invokedynamic under the covers to implement "function-like" constructs. Oracle has also committed to Nashorn, a fully invokedynamic-based JavaScript implementation, which has many of the same challenges as languages like Ruby or Python. I talked with Adam Messinger at Oracle, who explained to me that Oracle chose JavaScript in part because it's so far away from Java...as I put it (and he agreed) it's going to "keep Oracle honest" about optimizing for non-Java languages. Invokedynamic is driving the future of the JVM, and Oracle knows it all too well.

As for #4...well, all good things take a little effort :) I think the effort required is far lower than you suspect, though.

14 Oct 2011 2:40pm GMT

07 Oct 2011

feedPlanet Ruby

Ruby on Rails: Rails 3.1.1 has been released!

Hi everyone,

Rails 3.1.1 has been released. This release requires at least sass-rails 3.1.4

CHANGES

ActionMailer

ActionPack

ActiveModel

ActiveRecord

ActiveResource

ActiveSupport

Railties

SHA-1

You can find an exhaustive list of changes on github. Along with the closed issues marked for v3.1.1.

Thanks to everyone!

07 Oct 2011 5:26pm GMT

26 Jul 2008

feedFOSDEM - Free and Open Source Software Developers' European Meeting

Update your RSS link

If you see this message in your RSS reader, please correct your RSS link to the following URL: http://fosdem.org/rss.xml.

26 Jul 2008 5:55am GMT

25 Jul 2008

feedFOSDEM - Free and Open Source Software Developers' European Meeting

Archive of FOSDEM 2008

These pages have been archived.
For information about the latest FOSDEM edition please check this url: http://fosdem.org

25 Jul 2008 4:43pm GMT

09 Mar 2008

feedFOSDEM - Free and Open Source Software Developers' European Meeting

Slides and videos online

Two weeks after FOSDEM and we are proud to publish most of the slides and videos from this year's edition.

All of the material from the Lightning Talks has been put online. We are still missing some slides and videos from the Main Tracks but we are working hard on getting those completed too.

We would like to thank our mirrors: HEAnet (IE) and Unixheads (US) for hosting our videos, and NamurLUG for quick recording and encoding.

The videos from the Janson room were live-streamed during the event and are also online on the Linux Magazin site.

We are having some synchronisation issues with Belnet (BE) at the moment. We're working to sort these out.

09 Mar 2008 3:12pm GMT