19 Jan 2020

feedPlanet Lisp

Joe Marshall: Afraid of Recursion

Here's a trick I learned several years ago for transferring time-series data. In the case in question, I needed to transfer a bunch of timestamped records, but the server had a quirk to it. If you asked for too many records at once, it would simply return an error code and give up on your request. There was no way to know beforehand how many records might exist in any given time span, so you could get an error code on nearly any request, unless it was for a very short time span. On the other hand, many requests for long time spans would succeed because they had few records in them. Despite this quirk, the code was really simple:

List<record> xfer (Timestamp start, Timestamp end) {
    try {
        return tryFetchRecords(start, end);
    } catch (TooManyRecordsException e) {
        Timestamp mid = (start + end)/2;
 List<record> firstHalf = xfer (start, mid);
 List<record> secondHalf = xfer (mid, end);
 return firstHalf.addAll(secondHalf);
    }
}

On any request, if the server returned the error code, we would simply bisect the time span, recursively ask for each half separately, and combine the two halves. Should the bisected time span still contain too many records, the time span would be bisected again. The recursion would continue until the time span was small enough that the server could honor the request and return some records. The recursion would then unwind, combining the returned records into larger and larger lists until we had all the records for our original time span. Since the time span would be cut in half on each recurrence, the depth of recursion would be proportional to the logarithm (base 2) of the total number of records, which would be a reasonably small number even with an enormous number of records.

It's certainly possible to avoid recursion and do this iteratively by some form of paging, but the code would be slightly more complex. The server is not very cooperative, so there is no easy way to determine an appropriate page size beforehand, and the server doesn't support a "paging token" to help keep track of progress. The recursive solution finds an appropriate transfer size by trial and error, and keeps track of progress more or less automatically. An iterative paging solution would have to do these things more explicitly and this would make the iterative code a bit more complex. And why add any complexity when it isn't really necessary?

I thought this solution was really cool when I first saw it. I've used this trick for transferring time series data many times. It makes the server very simple to write because the protocol requires so little of it. It simply has to refuse to answer requests that return too many results. The client code is just about the 10 lines above.

But when I first suggest this code to people I usually get "push back" (that is, until they see it work in action, then they usually get on board with it). People seem unsure about the use of recursion and want a more complex protocol where the client and server negotiate a page size or cooperatively pass a paging token back and forth on each request and response. Their eyes glaze over as soon as they see the recursive call. They want to avoid recursion just because it's recursive.

I've seen "aversion to recursion" happen in a lot of circumstances, not just this one. Recursion isn't the solution to everything. No tool solves all problems. But it is an important tool that often offers elegant solutions to many problems. Programmers shouldn't be afraid of using it when it is appropriate.

19 Jan 2020 11:52pm GMT

feedPlanet Debian

Enrico Zini: Food links

Periodic graphics: The chemistry of frozen desserts

chart food archive.org
2020-01-20

Chemical educator and Compound Interest blogger Andy Brunning samples the science behind ice cream and other icy treats

Play with your food: How to Make Sconic Sections | Evil Mad Scientist Laboratories
food archive.org
2020-01-20

The conic sections are the four classic geometric curves that can occur at the intersection between a cone and a plane: the circle, ellipse, parabola, and hyperbola.

Scott And Scurvy
health history archive.org
2020-01-20

«Writing about the first winter the men spent on the ice, Cherry-Garrard casually mentions an astonishing lecture on scurvy by one of the expedition's doctors…»

Mineral waters à la carte
chart food archive.org
2020-01-20

Cloning popular brands of mineral water is now simpler then ever before with the updated version of the mineral water calculator! When I blogged about DIY mineral water last year it was mainly a th…

Perfect egg yolks
chart food archive.org
2020-01-20

Maybe I have a hangup on soft boiled eggs, but I'm deeply fascinated by how something simple as an egg can be transformed into such a wide range of textures. I'm talking about pure eggs…

Perfect egg yolks (part 2)
chart food archive.org
2020-01-20

Egg cooked for 40 min at 63.0 °C. The pictures were taken within 6 seconds and are shown in the order they were taken. My immersion circulator is working again! And the first thing I decided to do …

Recreational kitchen mathematics: Cookie tessellations
archive.org
2020-01-20

Is there a way to avoid all that extra dough in between the cookies? (Photo: Christmas Tree Cookie Cutter from Bigstock) It should come as no surprise that food, chemistry and mathematics meet in b…

Il coriandolo che divide - Scienza in cucina - Blog - Le Scienze
food archive.org
2020-01-20

Il coriandolo (Coriandrum sativum), noto anche come cilantro, suo nome spagnolo, è una delle erbe aromatiche più antiche che si conoscano. Citato nella Bibbia, i suoi semi sono stati ritrovati in tombe egizie e in epoca romana era usato sia come erba medicinale che come condimento. L'uso culinario delle foglie fresche e dei semi essiccati si è poi diffuso in tutto il mondo e ora è molto utilizzato nelle cucine del Messico e dell'America Latina per preparare la tradizionale salsa che accompagna le tortillas, nel Medio Oriente e in alcuni paesi asiatici come la Tailandia e l'India per aromatizzare molte ricette.

19 Jan 2020 11:00pm GMT

feedPlanet Python

Codementor: How To Publish Your Own Python Package

Proper guide with demonstration of each step of HOW TO PUBLISH A PYTHON PACKAGE on PyPI.

19 Jan 2020 9:48pm GMT

feedPlanet Debian

Dirk Eddelbuettel: RPushbullet 0.3.3

RPpushbullet demo

Release 0.3.3 of the RPushbullet package just got to CRAN. RPushbullet offers an interface to the neat Pushbullet service for inter-device messaging, communication, and more. It lets you easily send (programmatic) alerts like the one to the left to your browser, phone, tablet, … - or all at once.

This release further robustifies operations via two contributed PRs. The first by Chan-Yub ensures we set UTF-8 encoding on pushes. The second by Alexandre permits to downgrade from http/2 to http/1.1 which he needed for some operations with a particular backend. I made that PR a bit more general by turning the downgrade into one driven by a new options() toggle. Special thanks also to Jeroen in help debugging this issue. See below for more details.

Changes in version 0.3.3 (2020-01-18)

  • UTF-8 encoding is now used (Chan-Yub Park in #55).

  • Curl can use HTTP/1.1 (Alexandre Shannon in #59 fixing #57, plus Dirk in #60 making it optional).

Courtesy of CRANberries, there is also a diffstat report for this release. More details about the package are at the RPushbullet webpage and the RPushbullet GitHub repo.

If you like this or other open-source work I do, you can now sponsor me at GitHub. For the first year, GitHub will match your contributions.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

19 Jan 2020 6:14pm GMT

feedPlanet Python

Simple is Better Than Complex: How to Use Chart.js with Django

Chart.js is a cool open source JavaScript library that helps you render HTML5 charts. It is responsive and counts with 8 different chart types.

In this tutorial we are going to explore a little bit of how to make Django talk with Chart.js and render some simple charts based on data extracted from our models.

Installation

For this tutorial all you are going to do is add the Chart.js lib to your HTML page:

<script src="https://cdn.jsdelivr.net/npm/chart.js@2.9.3/dist/Chart.min.js"></script>

You can download it from Chart.js official website and use it locally, or you can use it from a CDN using the URL above.

Example Scenario

I'm going to use the same example I used for the tutorial How to Create Group By Queries With Django ORM which is a good complement to this tutorial because actually the tricky part of working with charts is to transform the data so it can fit in a bar chart / line chart / etc.

We are going to use the two models below, Country and City:

class Country(models.Model):
    name = models.CharField(max_length=30)

class City(models.Model):
    name = models.CharField(max_length=30)
    country = models.ForeignKey(Country, on_delete=models.CASCADE)
    population = models.PositiveIntegerField()

And the raw data stored in the database:

cities
id name country_id population
1 Tokyo 28 36,923,000
2 Shanghai 13 34,000,000
3 Jakarta 19 30,000,000
4 Seoul 21 25,514,000
5 Guangzhou 13 25,000,000
6 Beijing 13 24,900,000
7 Karachi 22 24,300,000
8 Shenzhen 13 23,300,000
9 Delhi 25 21,753,486
10 Mexico City 24 21,339,781
11 Lagos 9 21,000,000
12 São Paulo 1 20,935,204
13 Mumbai 25 20,748,395
14 New York City 20 20,092,883
15 Osaka 28 19,342,000
16 Wuhan 13 19,000,000
17 Chengdu 13 18,100,000
18 Dhaka 4 17,151,925
19 Chongqing 13 17,000,000
20 Tianjin 13 15,400,000
21 Kolkata 25 14,617,882
22 Tehran 11 14,595,904
23 Istanbul 2 14,377,018
24 London 26 14,031,830
25 Hangzhou 13 13,400,000
26 Los Angeles 20 13,262,220
27 Buenos Aires 8 13,074,000
28 Xi'an 13 12,900,000
29 Paris 6 12,405,426
30 Changzhou 13 12,400,000
31 Shantou 13 12,000,000
32 Rio de Janeiro 1 11,973,505
33 Manila 18 11,855,975
34 Nanjing 13 11,700,000
35 Rhine-Ruhr 16 11,470,000
36 Jinan 13 11,000,000
37 Bangalore 25 10,576,167
38 Harbin 13 10,500,000
39 Lima 7 9,886,647
40 Zhengzhou 13 9,700,000
41 Qingdao 13 9,600,000
42 Chicago 20 9,554,598
43 Nagoya 28 9,107,000
44 Chennai 25 8,917,749
45 Bangkok 15 8,305,218
46 Bogotá 27 7,878,783
47 Hyderabad 25 7,749,334
48 Shenyang 13 7,700,000
49 Wenzhou 13 7,600,000
50 Nanchang 13 7,400,000
51 Hong Kong 13 7,298,600
52 Taipei 29 7,045,488
53 Dallas-Fort Worth 20 6,954,330
54 Santiago 14 6,683,852
55 Luanda 23 6,542,944
56 Houston 20 6,490,180
57 Madrid 17 6,378,297
58 Ahmedabad 25 6,352,254
59 Toronto 5 6,055,724
60 Philadelphia 20 6,051,170
61 Washington, D.C. 20 6,033,737
62 Miami 20 5,929,819
63 Belo Horizonte 1 5,767,414
64 Atlanta 20 5,614,323
65 Singapore 12 5,535,000
66 Barcelona 17 5,445,616
67 Munich 16 5,203,738
68 Stuttgart 16 5,200,000
69 Ankara 2 5,150,072
70 Hamburg 16 5,100,000
71 Pune 25 5,049,968
72 Berlin 16 5,005,216
73 Guadalajara 24 4,796,050
74 Boston 20 4,732,161
75 Sydney 10 5,000,500
76 San Francisco 20 4,594,060
77 Surat 25 4,585,367
78 Phoenix 20 4,489,109
79 Monterrey 24 4,477,614
80 Inland Empire 20 4,441,890
81 Rome 3 4,321,244
82 Detroit 20 4,296,611
83 Milan 3 4,267,946
84 Melbourne 10 4,650,000
countries
id name
1 Brazil
2 Turkey
3 Italy
4 Bangladesh
5 Canada
6 France
7 Peru
8 Argentina
9 Nigeria
10 Australia
11 Iran
12 Singapore
13 China
14 Chile
15 Thailand
16 Germany
17 Spain
18 Philippines
19 Indonesia
20 United States
21 South Korea
22 Pakistan
23 Angola
24 Mexico
25 India
26 United Kingdom
27 Colombia
28 Japan
29 Taiwan

Example 1: Pie Chart

For the first example we are only going to retrieve the top 5 most populous cities and render it as a pie chart. In this strategy we are going to return the chart data as part of the view context and inject the results in the JavaScript code using the Django Template language.

views.py

from django.shortcuts import render
from mysite.core.models import City

def pie_chart(request):
    labels = []
    data = []

    queryset = City.objects.order_by('-population')[:5]
    for city in queryset:
        labels.append(city.name)
        data.append(city.population)

    return render(request, 'pie_chart.html', {
        'labels': labels,
        'data': data,
    })

Basically in the view above we are iterating through the City queryset and building a list of labels and a list of data. Here in this case the data is the population count saved in the City model.

For the urls.py just a simple routing:

urls.py

from django.urls import path
from mysite.core import views

urlpatterns = [
    path('pie-chart/', views.pie_chart, name='pie-chart'),
]

Now the template. I got a basic snippet from the Chart.js Pie Chart Documentation.

pie_chart.html

{% extends 'base.html' %}

{% block content %}
  <div id="container" style="width: 75%;">
    <canvas id="pie-chart"></canvas>
  </div>

  <script src="https://cdn.jsdelivr.net/npm/chart.js@2.9.3/dist/Chart.min.js"></script>
  <script>

    var config = {
      type: 'pie',
      data: {
        datasets: [{
          data: {{ data|safe }},
          backgroundColor: [
            '#696969', '#808080', '#A9A9A9', '#C0C0C0', '#D3D3D3'
          ],
          label: 'Population'
        }],
        labels: {{ labels|safe }}
      },
      options: {
        responsive: true
      }
    };

    window.onload = function() {
      var ctx = document.getElementById('pie-chart').getContext('2d');
      window.myPie = new Chart(ctx, config);
    };

  </script>

{% endblock %}

In the example above the base.html template is not important but you can see it in the code example I shared in the end of this post.

This strategy is not ideal but works fine. The bad thing is that we are using the Django Template Language to interfere with the JavaScript logic. When we put {{ data|safe}} we are injecting a variable that came from the server directly in the JavaScript code.

The code above looks like this:

Pie Chart


Example 2: Bar Chart with Ajax

As the title says, we are now going to render a bar chart using an async call.

views.py

from django.shortcuts import render
from django.db.models import Sum
from django.http import JsonResponse
from mysite.core.models import City

def home(request):
    return render(request, 'home.html')

def population_chart(request):
    labels = []
    data = []

    queryset = City.objects.values('country__name').annotate(country_population=Sum('population')).order_by('-country_population')
    for entry in queryset:
        labels.append(entry['country__name'])
        data.append(entry['country_population'])
    
    return JsonResponse(data={
        'labels': labels,
        'data': data,
    })

So here we are using two views. The home view would be the main page where the chart would be loaded at. The other view population_chart would be the one with the sole responsibility to aggregate the data the return a JSON response with the labels and data.

If you are wondering about what this queryset is doing, it is grouping the cities by the country and aggregating the total population of each country. The result is going to be a list of country + total population. To learn more about this kind of query have a look on this post: How to Create Group By Queries With Django ORM

urls.py

from django.urls import path
from mysite.core import views

urlpatterns = [
    path('', views.home, name='home'),
    path('population-chart/', views.population_chart, name='population-chart'),
]

home.html

{% extends 'base.html' %}

{% block content %}

  <div id="container" style="width: 75%;">
    <canvas id="population-chart" data-url="{% url 'population-chart' %}"></canvas>
  </div>

  <script src="https://code.jquery.com/jquery-3.4.1.min.js"></script>
  <script src="https://cdn.jsdelivr.net/npm/chart.js@2.9.3/dist/Chart.min.js"></script>
  <script>

    $(function () {

      var $populationChart = $("#population-chart");
      $.ajax({
        url: $populationChart.data("url"),
        success: function (data) {

          var ctx = $populationChart[0].getContext("2d");

          new Chart(ctx, {
            type: 'bar',
            data: {
              labels: data.labels,
              datasets: [{
                label: 'Population',
                backgroundColor: 'blue',
                data: data.data
              }]          
            },
            options: {
              responsive: true,
              legend: {
                position: 'top',
              },
              title: {
                display: true,
                text: 'Population Bar Chart'
              }
            }
          });

        }
      });

    });

  </script>

{% endblock %}

Now we have a better separation of concerns. Looking at the chart container:

<canvas id="population-chart" data-url="{% url 'population-chart' %}"></canvas>

We added a reference to the URL that holds the chart rendering logic. Later on we are using it to execute the Ajax call.

var $populationChart = $("#population-chart");
$.ajax({
  url: $populationChart.data("url"),
  success: function (data) {
    // ...
  }
});

Inside the success callback we then finally execute the Chart.js related code using the JsonResponse data.

Bar Chart


Conclusions

I hope this tutorial helped you to get started with working with charts using Chart.js. I published another tutorial on the same subject a while ago but using the Highcharts library. The approach is pretty much the same: How to Integrate Highcharts.js with Django.

If you want to grab the code I used in this tutorial you can find it here: github.com/sibtc/django-chartjs-example.

19 Jan 2020 3:00pm GMT

Weekly Python StackOverflow Report: (ccxi) stackoverflow python report

These are the ten most rated questions at Stack Overflow last week.
Between brackets: [question score / answers count]
Build date: 2020-01-19 12:01:13 GMT


  1. Use data in pandas data frames to match columns together - [13/4]
  2. Filtering DataFrame on groups where count of element is different than 1 - [7/4]
  3. How to sort a list of sub-lists by the contents of sub-lists, where sub-lists contain strings and booleans? - [7/4]
  4. how can I clean this data type object and transform it to type float maintaining Null and NaN - [6/3]
  5. Finding all the combinations of free polyominoes within a specific area with a SAT-solver (Python) - [6/2]
  6. Filter numeric column by dictionary of value ranges - [6/2]
  7. Select NumPy Values Around Index - [5/4]
  8. Slicing arrays with lists - [5/4]
  9. How to extract content from <script> using Beautiful Soup - [5/4]
  10. How do you search a column and fill another column with what you find? (in pandas) - [5/3]

19 Jan 2020 12:01pm GMT

18 Jan 2020

feedPlanet Debian

Ingo Juergensmann: XMPP - Fun with Clients

As I already wrote in my last blog post there's much development in XMPP, not only on the server side, but also on the client side. It's surely not exaggerated to say that Conversations on Android is the de-facto standard client-wise. So, if you have an Android phone, that's the client you want to try&use. As I don't have Android, I can't comment on it. The situation on Linux is good as well: there are such clients as Gajim, which is an old player in the "market" and is available on other platforms as well, but there is with Dino a new/modern client as well that you may want to try out.

The situation for macOS and iOS users are not that good as for Windows, Linux or Android users. But in the end all clients have their pro and cons... I'll try to summarize a few clients on Linux, macOS and iOS...

LinuxGajim

Fully featured multiprotol client with lots of available plugins. If you want to use OMEMO with Gajim you need to enable it in your plugin settings. There is even a plugin for letting the keyboard LED blink when there are new/unread messages. I found that a little bit annoying, so I disabled that. Gajim is an old-style client with the well-known layout of a contact list window and one for the actual chats. Gajim has some nice features like service discovery on your or remote servers.

Dino

Dino is available in Debian as dino-im and is a quite new client, which you will find out at first start: it's a single window app where the focus is on the chats. There is no contact list at first glance where you can see whether or not your contacts are online. You can find your contacts when you want to start a conversation with your contact. I don't find this bad or good. It's just different and puts the chat into focus, as said, maybe similar to WhatApp, Signal or other messengers nowadays where you just sent messages back and forth and don't care if the contact is online. The contact will receive and read your message when the contact is online and will eventually answer your message.

macOSMonal

Monal is an actively developed and maintained client for macOS and iOS. If you want to try out Monal, don't waste your time with the older client, but focus on Monal Catalyst (direct download link). Catalyst shares the same code as the iOS version of Monal and will become the default Monal download within the next few weeks. It's far easier for the developers to focus on one codebase than on two different ones. Monal has great potential, but also has some issues. For some reason it seems that some messages from time to time will be sent multiple times to a contact or a MUC. The developers are very helpful and supportive. So when you find a bug or issue, please report back.

BeagleIM

BeagleIM is a free XMPP client by Tigase, which business is to sell their XMPP Communication suite and professional support for it. They provide clients for Android, macOS and iOS. Maybe for that reason their clients seems to be very mature, but of course will work best with their own server software. That doesn't mean that the clients won't work well with other 3rd party XMPP servers, just that their main focus will be their own server software, However, BeagleIM seems to work well with Prosody and ejabberd and when you have issues you can also reach out to Tigase on Mastodon, which I find a very big plus! They are really helpful there as well. The BeagleIM client is currently my main client on macOS and it works quite well. As you can see it's more or less chat-focused as well by default, but you can open a contact list window if you want to see all of available/all clients. Only issue I personally have at the moment is, that it seems to have problems with ejabberd: in the contact list and account preferences I see the accounts/contacts with ejabberd going offline/online every few minutes. There are some log entries in ejabberd that seem to be timeout related. I'm not sure whether this is an issue with ejabberd or with BeagleIM - or a rare combination of both.

iOSChatSecure

ChatSecure is one of my first XMPP clients on iOS I installed and used for a long time. It works mostly very well, supports OMEMO (like all the other clients I mention here) and it seems to be able to work well with bookmarks (i.e. use a list of MUCs to join). Only issues I have with ChatSecure currently are: 1) when ChatSecure comes to front after a deep sleep state of the client on iPhone it presents an empty screen and no accounts in settings. You need to quit ChatSecure and restart it to have it working again. Quite annoying. 2) when restarted it polls all messages again from MAM (message archives) over and over again. A small annoying where you can decide if it's better to have duplicated messages or may miss a message.

Monal

What was valid for Monal Catalyst on macOS is also (more or less) true for Monal on iOS. As said: they share the same code base. It is usuable, but has sometimes issues with joining MUCs: it appears as if you can't join a MUC, but suddenly you receive a message in the MUC and then it is listed under Chats and you can use it.

SiskinIM

Siskin is the iOS client from Tigase. It also seems to be very mature, but has some special caveats: in the settings you can configure Push Notifications and HTTP Uploads for each and every clients. Other clients make this automatically and I leave it to you to decide whether this is a nice feature that you can configure it or if it is a little bit annoying, because when you don't know that, you will be wondering why you can't upload files/fotos in a chat. Maybe uploading files will work with Tigase XMPP server, but it doesn't seem to work on my servers.

So, in the end, there are good/promising clients on iOS and macOS, but every client seem to have its own pitfalls. On iOS all three clients do support and use Apple Push Notifications, but you should choose carefully for which one you want to enable it. I can tell you it's a little bit annoying to test three clients and have Push Notifications turned on for all of them and have joined in several MUCs and get all notifications three times for every message... ;-)

MUCs are a special topic I need to investigate a little more in the future: when your server supports Bookmarks for MUCs I would assume that it should be working for all supporting clients and you only need to join MUC on one client and have that MUC at least in your list of bookmarks. If you want to join that MUC on every client might be another story. But I don't know if this is the intended behaviour of the XEP in question or if my assumption how it should work is just wrong.

In the end the situation for XMPP clients on macOS and iOS is much better than it was 1-2 years ago. Though, it is not as good as on Android, but you can help improving the situation by testing the available clients and give feedback to the developers by either joining the approrpriate MUCs or - even better - file issues on their Github pages!

Kategorie:
Debian
Tags:
Debian
XMPP
Software

18 Jan 2020 5:58pm GMT

feedPlanet Lisp

Joe Marshall: Unsyndicated blog

I've noticed that my blog posts are replicated in Planet Lisp and Planet Scheme, and here I am spamming them with random math stuff. So I'm creating a new blog, Jrm's Random Blog, where I can feel free to post about math, science, computers in general, and whatever else bugs me, without spamming the Lisp and Scheme readers. I'll keep posting to Abstract Heresies, but try to keep it more Lisp and computer language focused.

18 Jan 2020 1:33pm GMT

17 Jan 2020

feedPlanet Grep

FOSDEM organizers: Second set of speaker interviews

We have just published the second set of interviews with our main track and keynote speakers. The following interviews give you a lot of interesting reading material about various topics, from the forgotten history of early Unix to the next technological shift of computing: Daniel Riek: How Containers and Kubernetes re-defined the GNU/Linux Operating System. A Greybeard's Worst Nightmare David Stewart: Improving protections against speculative execution side channel James Bottomley: The Selfish Contributor Explained Jon 'maddog' Hall: FOSSH - 2000 to 2020 and beyond!. maddog continues to pontificate Justin W. Flory and Michael Nolan: Freedom and AI: Can Free Software舰

17 Jan 2020 3:00pm GMT

Mattias Geniar: Dumping specific tables in MySQL using mysqldump

Here's a small little trick when using mysqldump: you don't have to dump an entire database, you can dump individual tables too.

17 Jan 2020 12:00am GMT

16 Jan 2020

feedPlanet Lisp

Joe Marshall: Groups, semigroups, monoids, and computers

The day after I rant about mathematicians, I make a math post. "Do I contradict myself? Very well, then, I contradict myself, I am large, I contain multitudes." - Walt Whitman

A group is a mathematical concept. It's pretty simple. It consists of a set, G, and an operation, *, which can be used to combine any two elements of G. What the set contains is not that important. It is the * operation we're interested in, and we can usually swap out G for another set without causing too many problems other than having to change the type signature of *. There are four axioms that * must obey

Frequently you run into something that obeys all the axioms but the inverse element axiom. This is called a monoid. A monoid is very much like a group except that you can get "stuck" when manipulating it if you run into one of the non-invertible elements because there's no inverse to "undo" it. There are certain things about monoids that are true only "if the appropriate inverses exist". You run into that qualifier a lot when dealing with monoids. You don't need that qualifier if you are dealing with a group because they do exist by axiom. Or we could say that calling something a group is simply shorthand for adding "if the appropriate inverses exist" everywhere.

What does this have to do with computers? Consider the set of all subroutines with the operation of concatenation. It is closed - concatenating two subroutines gives you a third subroutine. It is associative - you just concatenate them linearly. There is an identity element, usually called no-op. And many, but not all, subroutines have inverses. So we have a monoid.

Consider the set of all strings with the operation of concatenation. It is closed, associative, the empty string is the identity element. It is a monoid.

Consider the set of functions whose input type is the same as the result type with the operation of composition. It is closed, associative, the identity function is the identity element. It is a monoid. If we consider only the subset of functions that also have inverses, we have a group. This particular monoid or group comes in especially handy because composition of functions is so useful.

Consider the set of invertible 2x2 matrices with integer components, a determinant of 1 or -1, and the operation of matrix multiply. It is closed, associative, there is an identity matrix, and I already said just consider the invertible ones. It forms a group. This group comes in handy for implementing arbitrary precision arithmetic. (Thanks to Bradley Lucier for the correction of the condition on the determinant. This makes the matrix continue to have integer components upon inversion, keeping things closed.)

The permutations of a list form a group. The integers under addition form a group.

These things are everywhere. And it isn't a coincidence. The concepts of a group, monoid, and semigroup are meant to capture the essence of what it is to have a foldable sequence of elements. (Can I complain about mathematicians here? They make up so much terminology and abstraction that it is virtually impossible to get at what they really mean. We're just talking about sequences of elements and trying to find some minimal axioms that you need to have to fold them, but try to find literature that actually says that's what we're doing is like trying to pull hen's teeth.)

So what good are groups, monoids, and semigroups? Aside from the obvious fact that foldable sequences are ubiquitous and really useful, that is. Not immediately apparent from the axioms is that in addition to folding a sequence, you can transform a sequence into a different, but equivalent one. If the appropriate inverses exist (there's that phrase), you can "unfold" some or all elements of a sequence. So by judicious folding and unfolding, you can transform a sequence.

Here's an unusual abstract example. Consider a pipeline which has a set of nodes and communicates values of the same type between the nodes. Values accumulate at the nodes until they are transmitted to the next node in the pipeline. We start with all the values in the initial node (on the right) and transmit them to the left:

(pipeline (node) (node) (node a b c))  ;; transmit the a
(pipeline (node) (node a) (node b c))  ;; transmit the b
(pipeline (node) (node a b) (node c))  ;; transmit the a
(pipeline (node a) (node b) (node c))  ;; transmit the c
(pipeline (node a) (node b c) (node))  ;; transmit the b
(pipeline (node a b) (node c) (node))  ;; transmit the c
(pipeline (node a b c) (node) (node))  ;; done

If the values we transmit are drawn from a group, we can replace each node with the group's * operator:

(* identity identity (* a b c))  ;; transmit the a
(* identity (* identity a) (* b c))  ;; transmit the b
(* identity (* a b) (* identity c))  ;; transmit the a
(* (* identity a) (* identity  b) (* identity c))  ;; transmit the c
(* (* identity a) (* b c) identity)  ;; transmit the b
(* (* a b) (* identity c) identity)  ;; transmit the c
(* (* a b c) identity identity)  ;; done

The astute reader will notice that all we're doing is making use of the associativity axiom and moving the parenthesis around so that the values seem to move between the different nodes. But we preserve the invariant that the "value" of the entire pipeline doesn't change as the values move. The * operator need not be concatenate, which would give simple queuing behavior, but can be any operator satisfying the axioms giving us much more interesting pipelines. One implementation of arbitrary precision arithmetic transmits Möbius transformations along just such a pipeline to refine the upper and lower limits of a computed approximation. In this implementation, the * operator is the composition of Möbius transformations.

Here's a more concrete example. If you have a series of nested functions: (f (g x)) and both f and g take and return the same type, rewrite it as ((compose f g) x) and use a little group theory on it.

(f (g x))
((compose f g) x)
;; or more explicitly
((fold-left compose identity (list f g)) x)

If the appropriate inverses exist, then there will be another function h such that (compose f g) is equal to (compose h f) essentially allowing you to "slide" g to the left "through" f. It is relatively easy to see that h must be equivalent to (compose f g f-1). Mathematicians say that h is conjugate to g. Conjugates always have a form like aba-1. By finding conjugates, you can take a sequence and slide the elements left and right through other elements. This also allows you to fold things out of order. (Or in the pipeline example, transmit items out of order.) If we were left folding into an accumulator, folding h before f is equivalent to folding g after f. Another way of looking at it is this. Suppose we're standing to the left of f and looking through the "lens" of f at g. h is what g "looks like" when viewed through f.

If we want, we can define slide such that (compose slide (compose f g)) is equivalent to (compose h f). slide is (compose h f g-1 f-1). (This isn't a generic slide sequence, it only works on (compose f g). It ought to be an identity because (compose f g) is equivalent to (compose h f).) I complained that mathematicians provided too few concrete examples, so here is a concrete example using list permutations:

> (reverse (rotate-left '(a b c d)))
(a d c b)

;; rewrite as explicit fold-left of compose
> ((fold-left compose identity (list reverse rotate-left)) '(a b c d))
(a d c b)

;; sliding rotate-left through reverse turns it into rotate-right
> ((fold-left compose identity (list rotate-right reverse)) '(a b c d))
(a d c b)

;; A sequence that when composed with (list reverse rotate-left) turns it into
;; (rotate-right reverse)
> (define slide 
    (fold-left compose identity (list rotate-right reverse rotate-right reverse)))
slide

> ((fold-left compose identity (list slide reverse rotate-left)) '(a b c d))
(a d c b)

;; rewrite back to direct procedure calls
> (rotate-right (reverse '(a b c d)))
(a d c b)

;; and slide ought to be an identity
> ((fold-left compose identity (list slide)) '(a b c d))
(a b c d)


Or suppose you have (f (g x)), but for some reason you want(g (f x)) (which would, in general, be a different value unless f and g happen to commute). Again, rewrite (f (g x)) as ((compose f g) x) and apply a little group theory. If the appropriate inverses exist, there will be a function commute-fg such that (compose commute-fg (compose f g)) is equivalent to (compose g f). With a little thought, you can see that commute-fg is equivalent to (compose g f g-1 f-1). (Again, this isn't a generic commute, it only causes this specific f and g to commute.) commute-fg is called a commutator because it makes f and g commute. Commutators always have the form aba-1b-1. By finding commutators and inserting them in the right place, you can take a sequence and swap adjacent elements. Again, a concrete example with lists:

;; an illustration of what swap-first two does
> (swap-first-two '(a b c d))
(b a c d)

;; we're given
> (reverse (swap-first-two '(a b c d)))
(d c a b)

;; but we want, for some reason to reverse first
> (swap-first-two (reverse '(a b c d)))
(c d b a)

;; rewrite as fold-left of compose
> ((fold-left compose identity (list reverse swap-first-two)) '(a b c d))
(d c a b)

;; define our commutator
;; note that swap-first-two and reverse are their own inverses
> (define commute-fg
    (fold-left compose identity (list swap-first-two reverse swap-first-two reverse)))

;; make f and g commute
;; observe that it returns the desired result
> ((fold-left compose identity (list commute-fg reverse swap-first-two)) '(a b c d))
(c d b a)


There's two interesting things here. First, notice that in both examples I convert (f (g x)) to ((fold-left compose identity (list f g)) x) and then proceed to ignore x and just consider (fold-left compose identity (list f g)) as if x didn't exist. I've abstracted away the x. (Of course I have to eventually supply the x if I want an answer, but it only comes back at the last moment.) Second, notice that although slide and commute-fg are foldable sequences, I use them as if they were higher order functions operating on the foldable sequence (compose f g) to transform it, first into (compose h f), second into (compose g f). This second thing is a neat trick. We're taking a function that operates on lists and treating it as if it were a higher-order function that operates on functions. This is called the "action" of slide and commute-fg because it appears as if elements of the set G of our group can "act" directly on other elements.

Every element in the underlying set G of a group has an action associated with it which operates directly on other elements in G. This is an important concept in group theory. Now earlier I said that the actual elements of G don't matter much, so the action must be more closely tied to the operator *. And if we swap out G for another set we'll still have the same actions, they'll just be associated with the elements of the new set (in an isomorphic way). The actions are pretty abstract.

There's a lot more one could say about the actions. They are a rich source of interesting math. My brain is getting fatigued with all this abstraction, so I'll leave the topic be for now.

If group theory is about the essence of what it means to have a foldable sequence, then category theory is about the essence of composition. They offer two somewhat different approaches to similar material. What do you do with sequences but compose them? What comes from composition but a sequence? Many concepts in group theory carry over into category theory. Naturally a completely different set of terminology is used, but the concepts are there.

But that's enough group theory for today and category theory can wait until later posts.

16 Jan 2020 1:48pm GMT

15 Jan 2020

feedPlanet Grep

Dries Buytaert: Happy nineteenth birthday, Drupal

Happy nineteenth birthday

Nineteen years ago today, I released Drupal 1.0.0. Every day, for the past nineteen years, the Drupal community has collaborated on providing the world with an Open Source CMS and making a difference on how the web is built and run.

It's easy to forget that software is written one line of code at the time. And that adoption is driven one website at the time. I look back on nearly two decades of Drupal, and I'm incredibly proud of our community, and how we've contributed to a more open, independent internet.

Today, many of us in the Drupal community are working towards the launch of Drupal 9. Major releases of Drupal only happen every 3-5 years. They are an opportunity to bring our community together, create something meaningful, and celebrate our collective work. But more importantly, major releases are our best opportunity to re-engage past users and attract new users, explaining why Drupal is better than closed CMS platforms.

As we mark Drupal's 19th year, let's all work together on a successful launch of Drupal 9 in 2020, including a wide-spread marketing strategy. It's the best birthday present we can give to Drupal.

15 Jan 2020 1:56pm GMT

26 Apr 2019

feedPlanet Sun

First Image of a Black Hole – Event Horizon

The Event Horizon Telescope (EHT) - a planet-scale array of 8 ground-based radio telescopes and part of an international collaboration - captured the first image of a black hole. On April 10th 2019, EHT researchers disclosed the first direct visual evidence of a supermassive black hole in the heart of the Galaxy Messier 87.

26 Apr 2019 2:32am GMT

04 Nov 2018

feedPlanet Sun

5 Budget-Friendly Telescopes You Can Choose For Viewing Planets

Socrates couldn't have been more right when he said: "I know one thing, that I know nothing." Even with all of the advancements we, as a species, have made in this world, it's still nothing compared to countless of wonders waiting to be discovered in the vast universe. If you've recently developed an interest in ... Read more

04 Nov 2018 1:27pm GMT

20 May 2012

feedPlanet Sun

Annular Solar Eclipse on Sunday, May 20th 2012

On Sunday, May 20th 2012, people in a narrow strip from Japan to the western United States will be able to see an annular solar eclipse, the first in 18 years. The moon will cover as much as 94% of the sun. An Annular Solar Eclipse is different from a Total Solar Eclipse, when the ... Read more

20 May 2012 9:51pm GMT

10 Nov 2011

feedPlanetJava

OSDir.com - Java: Oracle Introduces New Java Specification Requests to Evolve Java Community Process

From the Yet Another dept.:

To further its commitment to the Java Community Process (JCP), Oracle has submitted the first of two Java Specification Requests (JSRs) to update and revitalize the JCP.

10 Nov 2011 6:01am GMT

OSDir.com - Java: No copied Java code or weapons of mass destruction found in Android

From the Fact Checking dept.:

ZDNET: Sometimes the sheer wrongness of what is posted on the web leaves us speechless. Especially when it's picked up and repeated as gospel by otherwise reputable sites like Engadget. "Google copied Oracle's Java code, pasted in a new license, and shipped it," they reported this morning.



Sorry, but that just isn't true.

10 Nov 2011 6:01am GMT

OSDir.com - Java: Java SE 7 Released

From the Grande dept.:

Oracle today announced the availability of Java Platform, Standard Edition 7 (Java SE 7), the first release of the Java platform under Oracle stewardship.

10 Nov 2011 6:01am GMT

08 Nov 2011

feedfosdem - Google Blog Search

papupapu39 (papupapu39)'s status on Tuesday, 08-Nov-11 00:28 ...

papupapu39 · http://identi.ca/url/56409795 #fosdem #freeknowledge #usamabinladen · about a day ago from web. Help · About · FAQ · TOS · Privacy · Source · Version · Contact. Identi.ca is a microblogging service brought to you by Status.net. ...

08 Nov 2011 12:28am GMT

05 Nov 2011

feedfosdem - Google Blog Search

Write and Submit your first Linux kernel Patch | HowLinux.Tk ...

FOSDEM (Free and Open Source Development European Meeting) is a European event centered around Free and Open Source software development. It is aimed at developers and all interested in the Free and Open Source news in the world. ...

05 Nov 2011 1:19am GMT

03 Nov 2011

feedfosdem - Google Blog Search

Silicon Valley Linux Users Group – Kernel Walkthrough | Digital Tux

FOSDEM (Free and Open Source Development European Meeting) is a European event centered around Free and Open Source software development. It is aimed at developers and all interested in the Free and Open Source news in the ...

03 Nov 2011 3:45pm GMT

28 Oct 2011

feedPlanet Ruby

O'Reilly Ruby: MacRuby: The Definitive Guide

Ruby and Cocoa on OS X, the iPhone, and the Device That Shall Not Be Named

28 Oct 2011 8:00pm GMT

14 Oct 2011

feedPlanet Ruby

Charles Oliver Nutter: Why Clojure Doesn't Need Invokedynamic (Unless You Want It to be More Awesome)

This was originally posted as a comment on @fogus's blog post "Why Clojure doesn't need invokedynamic, but it might be nice". I figured it's worth a top-level post here.

Ok, there's some good points here and a few misguided/misinformed positions. I'll try to cover everything.

First, I need to point out a key detail of invokedynamic that may have escaped notice: any case where you must bounce through a generic piece of code to do dispatch -- regardless of how fast that bounce may be -- prevents a whole slew of optimizations from happening. This might affect Java dispatch, if there's any argument-twiddling logic shared between call sites. It would definitely affect multimethods, which are using a hand-implemented PIC. Any case where there's intervening code between the call site and the target would benefit from invokedynamic, since invokedynamic could be used to plumb that logic and let it inline straight through. This is, indeed, the primary benefit of using invokedynamic: arbitrarily complex dispatch logic folds away allowing the dispatch to optimize as if it were direct.

Your point about inference in Java dispatch is a fair one...if Clojure is able to infer all cases, then there's no need to use invokedynamic at all. But unless Clojure is able to infer all cases, then you've got this little performance time bomb just waiting to happen. Tweak some code path and obscure the inference, and kablam, you're back on a slow reflective impl. Invokedynamic would provide a measure of consistency; the only unforeseen perf impact would be when the dispatch turns out to *actually* be polymorphic, in which case even a direct call wouldn't do much better.

For multimethods, the benefit should be clear: the MM selection logic would be mostly implemented using method handles and "leaf" logic, allowing hotspot to inline it everywhere it is used. That means for small-morphic MM call sites, all targets could potentially inline too. That's impossible without invokedynamic unless you generate every MM path immediately around the eventual call.

Now, on to defs and Var lookup. Depending on the cost of Var lookup, using a SwitchPoint-based invalidation plus invokedynamic could be a big win. In Java 7u2, SwitchPoint-based invalidation is essentially free until invalidated, and as you point out that's a rare case. There would essentially be *no* cost in indirecting through a var until that var changes...and then it would settle back into no cost until it changes again. Frequently-changing vars could gracefully degrade to a PIC.

It's also dangerous to understate the impact code size has on JVM optimization. The usual recommendation on the JVM is to move code into many small methods, possibly using call-through logic as in multimethods to reuse the same logic in many places. As I've mentioned, that defeats many optimizations, so the next approach is often to hand-inline logic everywhere it's used, to let the JVM have a more optimizable view of the system. But now we're stepping on our own feet...by adding more bytecode, we're almost certainly impacting the JVM's optimization and inlining budgets.

OpenJDK (and probably the other VMs too) has various limits on how far it will go to optimize code. A large number of these limits are based on the bytecoded size of the target methods. Methods that get too big won't inline, and sometimes won't compile. Methods that inline a lot of code might not get inlined into other methods. Methods that inline one path and eat up too much budget might push out more important calls later on. The only way around this is to reduce bytecode size, which is where invokedynamic comes in.

As of OpenJDK 7u2, MethodHandle logic is not included when calculating inlining budgets. In other words, if you push all the Java dispatch logic or multimethod dispatch logic or var lookup into mostly MethodHandles, you're getting that logic *for free*. That has had a tremendous impact on JRuby performance; I had previous versions of our compiler that did indeed infer static target methods from the interpreter, but they were often *slower* than call site caching solely because the code was considerably larger. With invokedynamic, a call is a call is a call, and the intervening plumbing is not counted against you.

Now, what about negative impacts to Clojure itself...

#0 is a red herring. JRuby supports Java 5, 6, and 7 with only a few hundred lines of changes in the compiler. Basically, the compiler has abstract interfaces for doing things like constant lookup, literal loading, and dispatch that we simply reimplement to use invokedynamic (extending the old non-indy logic for non-indified paths). In order to compile our uses of invokedynamic, we use Rémi Forax's JSR-292 backport, which includes a "mock" jar with all the invokedynamic APIs stubbed out. In our release, we just leave that library out, reflectively load the invokedynamic-based compiler impls, and we're off to the races.

#1 would be fair if the Oracle Java 7u2 early-access drops did not already include the optimizations that gave JRuby those awesome numbers. The biggest of those optimizations was making SwitchPoint free, but also important are the inlining discounting and MutableCallSite improvements. The perf you see for JRuby there can apply to any indirected behavior in Clojure, with the same perf benefits as of 7u2.

For #2, to address the apparent vagueness in my blog post...the big perf gain was largely from using SwitchPoint to invalidate constants rather than pinging a global serial number. Again, indirection folds away if you can shove it into MethodHandles. And it's pretty easy to do it.

#3 is just plain FUD. Oracle has committed to making invokedynamic work well for Java too. The current thinking is that "lambda", the support for closures in Java 7, will use invokedynamic under the covers to implement "function-like" constructs. Oracle has also committed to Nashorn, a fully invokedynamic-based JavaScript implementation, which has many of the same challenges as languages like Ruby or Python. I talked with Adam Messinger at Oracle, who explained to me that Oracle chose JavaScript in part because it's so far away from Java...as I put it (and he agreed) it's going to "keep Oracle honest" about optimizing for non-Java languages. Invokedynamic is driving the future of the JVM, and Oracle knows it all too well.

As for #4...well, all good things take a little effort :) I think the effort required is far lower than you suspect, though.

14 Oct 2011 2:40pm GMT

07 Oct 2011

feedPlanet Ruby

Ruby on Rails: Rails 3.1.1 has been released!

Hi everyone,

Rails 3.1.1 has been released. This release requires at least sass-rails 3.1.4

CHANGES

ActionMailer

ActionPack

ActiveModel

ActiveRecord

ActiveResource

ActiveSupport

Railties

SHA-1

You can find an exhaustive list of changes on github. Along with the closed issues marked for v3.1.1.

Thanks to everyone!

07 Oct 2011 5:26pm GMT

26 Jul 2008

feedFOSDEM - Free and Open Source Software Developers' European Meeting

Update your RSS link

If you see this message in your RSS reader, please correct your RSS link to the following URL: http://fosdem.org/rss.xml.

26 Jul 2008 5:55am GMT

25 Jul 2008

feedFOSDEM - Free and Open Source Software Developers' European Meeting

Archive of FOSDEM 2008

These pages have been archived.
For information about the latest FOSDEM edition please check this url: http://fosdem.org

25 Jul 2008 4:43pm GMT

09 Mar 2008

feedFOSDEM - Free and Open Source Software Developers' European Meeting

Slides and videos online

Two weeks after FOSDEM and we are proud to publish most of the slides and videos from this year's edition.

All of the material from the Lightning Talks has been put online. We are still missing some slides and videos from the Main Tracks but we are working hard on getting those completed too.

We would like to thank our mirrors: HEAnet (IE) and Unixheads (US) for hosting our videos, and NamurLUG for quick recording and encoding.

The videos from the Janson room were live-streamed during the event and are also online on the Linux Magazin site.

We are having some synchronisation issues with Belnet (BE) at the moment. We're working to sort these out.

09 Mar 2008 3:12pm GMT