fidzu : software

Here's a trick I learned several years ago for transferring time-series data. In the case in question, I needed to transfer a bunch of timestamped records, but the server had a quirk to it. If you asked for too many records at once, it would simply return an error code and give up on your request. There was no way to know beforehand how many records might exist in any given time span, so you could get an error code on nearly any request, unless it was for a very short time span. On the other hand, many requests for long time spans would succeed because they had few records in them. Despite this quirk, the code was really simple:

List<record> xfer (Timestamp start, Timestamp end) {
    try {
        return tryFetchRecords(start, end);
    } catch (TooManyRecordsException e) {
        Timestamp mid = (start + end)/2;
 List<record> firstHalf = xfer (start, mid);
 List<record> secondHalf = xfer (mid, end);
 return firstHalf.addAll(secondHalf);
    }
}

On any request, if the server returned the error code, we would simply bisect the time span, recursively ask for each half separately, and combine the two halves. Should the bisected time span still contain too many records, the time span would be bisected again. The recursion would continue until the time span was small enough that the server could honor the request and return some records. The recursion would then unwind, combining the returned records into larger and larger lists until we had all the records for our original time span. Since the time span would be cut in half on each recurrence, the depth of recursion would be proportional to the logarithm (base 2) of the total number of records, which would be a reasonably small number even with an enormous number of records.

It's certainly possible to avoid recursion and do this iteratively by some form of paging, but the code would be slightly more complex. The server is not very cooperative, so there is no easy way to determine an appropriate page size beforehand, and the server doesn't support a "paging token" to help keep track of progress. The recursive solution finds an appropriate transfer size by trial and error, and keeps track of progress more or less automatically. An iterative paging solution would have to do these things more explicitly and this would make the iterative code a bit more complex. And why add any complexity when it isn't really necessary?

I thought this solution was really cool when I first saw it. I've used this trick for transferring time series data many times. It makes the server very simple to write because the protocol requires so little of it. It simply has to refuse to answer requests that return too many results. The client code is just about the 10 lines above.

But when I first suggest this code to people I usually get "push back" (that is, until they see it work in action, then they usually get on board with it). People seem unsure about the use of recursion and want a more complex protocol where the client and server negotiate a page size or cooperatively pass a paging token back and forth on each request and response. Their eyes glaze over as soon as they see the recursive call. They want to avoid recursion just because it's recursive.

I've seen "aversion to recursion" happen in a lot of circumstances, not just this one. Recursion isn't the solution to everything. No tool solves all problems. But it is an important tool that often offers elegant solutions to many problems. Programmers shouldn't be afraid of using it when it is appropriate.

19 Jan 2020 11:52pm GMT

Periodic graphics: The chemistry of frozen desserts

Chemical educator and Compound Interest blogger Andy Brunning samples the science behind ice cream and other icy treats

The conic sections are the four classic geometric curves that can occur at the intersection between a cone and a plane: the circle, ellipse, parabola, and hyperbola.

«Writing about the first winter the men spent on the ice, Cherry-Garrard casually mentions an astonishing lecture on scurvy by one of the expedition's doctors…»

Cloning popular brands of mineral water is now simpler then ever before with the updated version of the mineral water calculator! When I blogged about DIY mineral water last year it was mainly a th…

Maybe I have a hangup on soft boiled eggs, but I'm deeply fascinated by how something simple as an egg can be transformed into such a wide range of textures. I'm talking about pure eggs…

Egg cooked for 40 min at 63.0 °C. The pictures were taken within 6 seconds and are shown in the order they were taken. My immersion circulator is working again! And the first thing I decided to do …

Is there a way to avoid all that extra dough in between the cookies? (Photo: Christmas Tree Cookie Cutter from Bigstock) It should come as no surprise that food, chemistry and mathematics meet in b…

Il coriandolo (Coriandrum sativum), noto anche come cilantro, suo nome spagnolo, è una delle erbe aromatiche più antiche che si conoscano. Citato nella Bibbia, i suoi semi sono stati ritrovati in tombe egizie e in epoca romana era usato sia come erba medicinale che come condimento. L'uso culinario delle foglie fresche e dei semi essiccati si è poi diffuso in tutto il mondo e ora è molto utilizzato nelle cucine del Messico e dell'America Latina per preparare la tradizionale salsa che accompagna le tortillas, nel Medio Oriente e in alcuni paesi asiatici come la Tailandia e l'India per aromatizzare molte ricette.

19 Jan 2020 11:00pm GMT

Proper guide with demonstration of each step of HOW TO PUBLISH A PYTHON PACKAGE on PyPI.

19 Jan 2020 9:48pm GMT

Release 0.3.3 of the RPushbullet package just got to CRAN. RPushbullet offers an interface to the neat Pushbullet service for inter-device messaging, communication, and more. It lets you easily send (programmatic) alerts like the one to the left to your browser, phone, tablet, … - or all at once.

This release further robustifies operations via two contributed PRs. The first by Chan-Yub ensures we set UTF-8 encoding on pushes. The second by Alexandre permits to downgrade from http/2 to http/1.1 which he needed for some operations with a particular backend. I made that PR a bit more general by turning the downgrade into one driven by a new options() toggle. Special thanks also to Jeroen in help debugging this issue. See below for more details.

Changes in version 0.3.3 (2020-01-18)

• UTF-8 encoding is now used (Chan-Yub Park in #55).

• Curl can use HTTP/1.1 (Alexandre Shannon in #59 fixing #57, plus Dirk in #60 making it optional).

Courtesy of CRANberries, there is also a diffstat report for this release. More details about the package are at the RPushbullet webpage and the RPushbullet GitHub repo.

If you like this or other open-source work I do, you can now sponsor me at GitHub. For the first year, GitHub will match your contributions.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

19 Jan 2020 6:14pm GMT

Chart.js is a cool open source JavaScript library that helps you render HTML5 charts. It is responsive and counts with 8 different chart types.

In this tutorial we are going to explore a little bit of how to make Django talk with Chart.js and render some simple charts based on data extracted from our models.

Installation

For this tutorial all you are going to do is add the Chart.js lib to your HTML page:

<script src="https://cdn.jsdelivr.net/npm/chart.js@2.9.3/dist/Chart.min.js"></script>

You can download it from Chart.js official website and use it locally, or you can use it from a CDN using the URL above.

Example Scenario

I'm going to use the same example I used for the tutorial How to Create Group By Queries With Django ORM which is a good complement to this tutorial because actually the tricky part of working with charts is to transform the data so it can fit in a bar chart / line chart / etc.

We are going to use the two models below, Country and City:

class Country(models.Model):
    name = models.CharField(max_length=30)

class City(models.Model):
    name = models.CharField(max_length=30)
    country = models.ForeignKey(Country, on_delete=models.CASCADE)
    population = models.PositiveIntegerField()

And the raw data stored in the database:

---

Example 1: Pie Chart

For the first example we are only going to retrieve the top 5 most populous cities and render it as a pie chart. In this strategy we are going to return the chart data as part of the view context and inject the results in the JavaScript code using the Django Template language.

views.py

from django.shortcuts import render
from mysite.core.models import City

def pie_chart(request):
    labels = []
    data = []

    queryset = City.objects.order_by('-population')[:5]
    for city in queryset:
        labels.append(city.name)
        data.append(city.population)

    return render(request, 'pie_chart.html', {
        'labels': labels,
        'data': data,
    })

Basically in the view above we are iterating through the City queryset and building a list of labels and a list of data. Here in this case the data is the population count saved in the City model.

For the urls.py just a simple routing:

urls.py

from django.urls import path
from mysite.core import views

urlpatterns = [
    path('pie-chart/', views.pie_chart, name='pie-chart'),
]

Now the template. I got a basic snippet from the Chart.js Pie Chart Documentation.

pie_chart.html

{% extends 'base.html' %}

{% block content %}
  <div id="container" style="width: 75%;">
    <canvas id="pie-chart"></canvas>
  </div>

  <script src="https://cdn.jsdelivr.net/npm/chart.js@2.9.3/dist/Chart.min.js"></script>
  <script>

    var config = {
      type: 'pie',
      data: {
        datasets: [{
          data: {{ data|safe }},
          backgroundColor: [
            '#696969', '#808080', '#A9A9A9', '#C0C0C0', '#D3D3D3'
          ],
          label: 'Population'
        }],
        labels: {{ labels|safe }}
      },
      options: {
        responsive: true
      }
    };

    window.onload = function() {
      var ctx = document.getElementById('pie-chart').getContext('2d');
      window.myPie = new Chart(ctx, config);
    };

  </script>

{% endblock %}

In the example above the base.html template is not important but you can see it in the code example I shared in the end of this post.

This strategy is not ideal but works fine. The bad thing is that we are using the Django Template Language to interfere with the JavaScript logic. When we put {{ data|safe}} we are injecting a variable that came from the server directly in the JavaScript code.

The code above looks like this:

---

Example 2: Bar Chart with Ajax

As the title says, we are now going to render a bar chart using an async call.

views.py

from django.shortcuts import render
from django.db.models import Sum
from django.http import JsonResponse
from mysite.core.models import City

def home(request):
    return render(request, 'home.html')

def population_chart(request):
    labels = []
    data = []

    queryset = City.objects.values('country__name').annotate(country_population=Sum('population')).order_by('-country_population')
    for entry in queryset:
        labels.append(entry['country__name'])
        data.append(entry['country_population'])
    
    return JsonResponse(data={
        'labels': labels,
        'data': data,
    })

So here we are using two views. The home view would be the main page where the chart would be loaded at. The other view population_chart would be the one with the sole responsibility to aggregate the data the return a JSON response with the labels and data.

If you are wondering about what this queryset is doing, it is grouping the cities by the country and aggregating the total population of each country. The result is going to be a list of country + total population. To learn more about this kind of query have a look on this post: How to Create Group By Queries With Django ORM

urls.py

from django.urls import path
from mysite.core import views

urlpatterns = [
    path('', views.home, name='home'),
    path('population-chart/', views.population_chart, name='population-chart'),
]

home.html

{% extends 'base.html' %}

{% block content %}

  <div id="container" style="width: 75%;">
    <canvas id="population-chart" data-url="{% url 'population-chart' %}"></canvas>
  </div>

  <script src="https://code.jquery.com/jquery-3.4.1.min.js"></script>
  <script src="https://cdn.jsdelivr.net/npm/chart.js@2.9.3/dist/Chart.min.js"></script>
  <script>

    $(function () {

      var $populationChart = $("#population-chart");
      $.ajax({
        url: $populationChart.data("url"),
        success: function (data) {

          var ctx = $populationChart[0].getContext("2d");

          new Chart(ctx, {
            type: 'bar',
            data: {
              labels: data.labels,
              datasets: [{
                label: 'Population',
                backgroundColor: 'blue',
                data: data.data
              }]          
            },
            options: {
              responsive: true,
              legend: {
                position: 'top',
              },
              title: {
                display: true,
                text: 'Population Bar Chart'
              }
            }
          });

        }
      });

    });

  </script>

{% endblock %}

Now we have a better separation of concerns. Looking at the chart container:

<canvas id="population-chart" data-url="{% url 'population-chart' %}"></canvas>

We added a reference to the URL that holds the chart rendering logic. Later on we are using it to execute the Ajax call.

var $populationChart = $("#population-chart");
$.ajax({
  url: $populationChart.data("url"),
  success: function (data) {
    // ...
  }
});

Inside the success callback we then finally execute the Chart.js related code using the JsonResponse data.

---

Conclusions

I hope this tutorial helped you to get started with working with charts using Chart.js. I published another tutorial on the same subject a while ago but using the Highcharts library. The approach is pretty much the same: How to Integrate Highcharts.js with Django.

If you want to grab the code I used in this tutorial you can find it here: github.com/sibtc/django-chartjs-example.

19 Jan 2020 3:00pm GMT

These are the ten most rated questions at Stack Overflow last week.
Between brackets: [question score / answers count]
Build date: 2020-01-19 12:01:13 GMT

---

1. Use data in pandas data frames to match columns together - [13/4]
2. Filtering DataFrame on groups where count of element is different than 1 - [7/4]
3. How to sort a list of sub-lists by the contents of sub-lists, where sub-lists contain strings and booleans? - [7/4]
4. how can I clean this data type object and transform it to type float maintaining Null and NaN - [6/3]
5. Finding all the combinations of free polyominoes within a specific area with a SAT-solver (Python) - [6/2]
6. Filter numeric column by dictionary of value ranges - [6/2]
7. Select NumPy Values Around Index - [5/4]
8. Slicing arrays with lists - [5/4]
9. How to extract content from <script> using Beautiful Soup - [5/4]
10. How do you search a column and fill another column with what you find? (in pandas) - [5/3]

19 Jan 2020 12:01pm GMT

18 Jan 2020 5:58pm GMT

I've noticed that my blog posts are replicated in Planet Lisp and Planet Scheme, and here I am spamming them with random math stuff. So I'm creating a new blog, Jrm's Random Blog, where I can feel free to post about math, science, computers in general, and whatever else bugs me, without spamming the Lisp and Scheme readers. I'll keep posting to Abstract Heresies, but try to keep it more Lisp and computer language focused.

18 Jan 2020 1:33pm GMT

We have just published the second set of interviews with our main track and keynote speakers. The following interviews give you a lot of interesting reading material about various topics, from the forgotten history of early Unix to the next technological shift of computing: Daniel Riek: How Containers and Kubernetes re-defined the GNU/Linux Operating System. A Greybeard's Worst Nightmare David Stewart: Improving protections against speculative execution side channel James Bottomley: The Selfish Contributor Explained Jon 'maddog' Hall: FOSSH - 2000 to 2020 and beyond!. maddog continues to pontificate Justin W. Flory and Michael Nolan: Freedom and AI: Can Free Software舰

17 Jan 2020 3:00pm GMT

Here's a small little trick when using mysqldump: you don't have to dump an entire database, you can dump individual tables too.

17 Jan 2020 12:00am GMT

The day after I rant about mathematicians, I make a math post. "Do I contradict myself? Very well, then, I contradict myself, I am large, I contain multitudes." - Walt Whitman

A group is a mathematical concept. It's pretty simple. It consists of a set, G, and an operation, *, which can be used to combine any two elements of G. What the set contains is not that important. It is the * operation we're interested in, and we can usually swap out G for another set without causing too many problems other than having to change the type signature of *. There are four axioms that * must obey

• Closure-combining any two elements of G using * just gives you another element in G.
  Note that this means you can build an arbitrary binary tree of combinations: e.g.(* (* a b) (* (* c d) e))). These trees will always be like a tree of cons cells. In some sense, the closure axiom is equivalent to saying that all the elements of G have the same type and that the * operator operates on values of that type and produces values of that type. The closure axiom along with the binary operation means that we can reduce any tree of combinations to a single value.
• Associativity-(* (* a b) c) = (* a (* b c)) for any a, b, and c. This implies that you can take any arbitrary tree of combinations: e.g.(* (* a b) (* (* c d) e))) and simply flatten it into a list (* a b c d e), or given the flat sequence (* a b c d e) we can add parenthesis anywhere we like: (* a (* b c) d e). If we stop here and only have the closure and associativity axiom, we have what is called a "semigroup". You can use the * operation to "fold" a semigroup down to single value, or to keep an accumulator and incrementally fold elements into the accumulator.
• Identity element-There has to be an identity element id such that (* id x) = (* x id) = x for all x. It will be unique. If you see the identity object in a combination (* a b id c d), you can simply remove it: (* a b c d). The identity element also comes in handy as an initial value when you are folding a sequence. If you have some concept that would be a group except it doesn't have an identity element, then you can often just make one up and add it to the set G.
• Inverse element-For every element in G there has to be another element, that when combined with the first, gives you the identity. So if a is an element in G, there has to be some other element, call it b, such that (* a b) = (* b a) = id. The inverse element is usually notated with a little -1: a-1. If you have an element in a combination right next to it's inverse: (* a x x-1 c), you can combine the element and it's inverse to get the identity: (* a id c), and then remove the identity: (* a c)

Frequently you run into something that obeys all the axioms but the inverse element axiom. This is called a monoid. A monoid is very much like a group except that you can get "stuck" when manipulating it if you run into one of the non-invertible elements because there's no inverse to "undo" it. There are certain things about monoids that are true only "if the appropriate inverses exist". You run into that qualifier a lot when dealing with monoids. You don't need that qualifier if you are dealing with a group because they do exist by axiom. Or we could say that calling something a group is simply shorthand for adding "if the appropriate inverses exist" everywhere.

What does this have to do with computers? Consider the set of all subroutines with the operation of concatenation. It is closed - concatenating two subroutines gives you a third subroutine. It is associative - you just concatenate them linearly. There is an identity element, usually called no-op. And many, but not all, subroutines have inverses. So we have a monoid.

Consider the set of all strings with the operation of concatenation. It is closed, associative, the empty string is the identity element. It is a monoid.

Consider the set of functions whose input type is the same as the result type with the operation of composition. It is closed, associative, the identity function is the identity element. It is a monoid. If we consider only the subset of functions that also have inverses, we have a group. This particular monoid or group comes in especially handy because composition of functions is so useful.

Consider the set of invertible 2x2 matrices with integer components, a determinant of 1 or -1, and the operation of matrix multiply. It is closed, associative, there is an identity matrix, and I already said just consider the invertible ones. It forms a group. This group comes in handy for implementing arbitrary precision arithmetic. (Thanks to Bradley Lucier for the correction of the condition on the determinant. This makes the matrix continue to have integer components upon inversion, keeping things closed.)

The permutations of a list form a group. The integers under addition form a group.

These things are everywhere. And it isn't a coincidence. The concepts of a group, monoid, and semigroup are meant to capture the essence of what it is to have a foldable sequence of elements. (Can I complain about mathematicians here? They make up so much terminology and abstraction that it is virtually impossible to get at what they really mean. We're just talking about sequences of elements and trying to find some minimal axioms that you need to have to fold them, but try to find literature that actually says that's what we're doing is like trying to pull hen's teeth.)

So what good are groups, monoids, and semigroups? Aside from the obvious fact that foldable sequences are ubiquitous and really useful, that is. Not immediately apparent from the axioms is that in addition to folding a sequence, you can transform a sequence into a different, but equivalent one. If the appropriate inverses exist (there's that phrase), you can "unfold" some or all elements of a sequence. So by judicious folding and unfolding, you can transform a sequence.

Here's an unusual abstract example. Consider a pipeline which has a set of nodes and communicates values of the same type between the nodes. Values accumulate at the nodes until they are transmitted to the next node in the pipeline. We start with all the values in the initial node (on the right) and transmit them to the left:

(pipeline (node) (node) (node a b c))  ;; transmit the a
(pipeline (node) (node a) (node b c))  ;; transmit the b
(pipeline (node) (node a b) (node c))  ;; transmit the a
(pipeline (node a) (node b) (node c))  ;; transmit the c
(pipeline (node a) (node b c) (node))  ;; transmit the b
(pipeline (node a b) (node c) (node))  ;; transmit the c
(pipeline (node a b c) (node) (node))  ;; done

If the values we transmit are drawn from a group, we can replace each node with the group's * operator:

(* identity identity (* a b c))  ;; transmit the a
(* identity (* identity a) (* b c))  ;; transmit the b
(* identity (* a b) (* identity c))  ;; transmit the a
(* (* identity a) (* identity  b) (* identity c))  ;; transmit the c
(* (* identity a) (* b c) identity)  ;; transmit the b
(* (* a b) (* identity c) identity)  ;; transmit the c
(* (* a b c) identity identity)  ;; done

The astute reader will notice that all we're doing is making use of the associativity axiom and moving the parenthesis around so that the values seem to move between the different nodes. But we preserve the invariant that the "value" of the entire pipeline doesn't change as the values move. The * operator need not be concatenate, which would give simple queuing behavior, but can be any operator satisfying the axioms giving us much more interesting pipelines. One implementation of arbitrary precision arithmetic transmits Möbius transformations along just such a pipeline to refine the upper and lower limits of a computed approximation. In this implementation, the * operator is the composition of Möbius transformations.

Here's a more concrete example. If you have a series of nested functions: (f (g x)) and both f and g take and return the same type, rewrite it as ((compose f g) x) and use a little group theory on it.

(f (g x))
((compose f g) x)
;; or more explicitly
((fold-left compose identity (list f g)) x)

If the appropriate inverses exist, then there will be another function h such that (compose f g) is equal to (compose h f) essentially allowing you to "slide" g to the left "through" f. It is relatively easy to see that h must be equivalent to (compose f g f-1). Mathematicians say that h is conjugate to g. Conjugates always have a form like aba-1. By finding conjugates, you can take a sequence and slide the elements left and right through other elements. This also allows you to fold things out of order. (Or in the pipeline example, transmit items out of order.) If we were left folding into an accumulator, folding h before f is equivalent to folding g after f. Another way of looking at it is this. Suppose we're standing to the left of f and looking through the "lens" of f at g. h is what g "looks like" when viewed through f.

If we want, we can define slide such that (compose slide (compose f g)) is equivalent to (compose h f). slide is (compose h f g-1 f-1). (This isn't a generic slide sequence, it only works on (compose f g). It ought to be an identity because (compose f g) is equivalent to (compose h f).) I complained that mathematicians provided too few concrete examples, so here is a concrete example using list permutations:

> (reverse (rotate-left '(a b c d)))
(a d c b)

;; rewrite as explicit fold-left of compose
> ((fold-left compose identity (list reverse rotate-left)) '(a b c d))
(a d c b)

;; sliding rotate-left through reverse turns it into rotate-right
> ((fold-left compose identity (list rotate-right reverse)) '(a b c d))
(a d c b)

;; A sequence that when composed with (list reverse rotate-left) turns it into
;; (rotate-right reverse)
> (define slide 
    (fold-left compose identity (list rotate-right reverse rotate-right reverse)))
slide

> ((fold-left compose identity (list slide reverse rotate-left)) '(a b c d))
(a d c b)

;; rewrite back to direct procedure calls
> (rotate-right (reverse '(a b c d)))
(a d c b)

;; and slide ought to be an identity
> ((fold-left compose identity (list slide)) '(a b c d))
(a b c d)

Or suppose you have (f (g x)), but for some reason you want(g (f x)) (which would, in general, be a different value unless f and g happen to commute). Again, rewrite (f (g x)) as ((compose f g) x) and apply a little group theory. If the appropriate inverses exist, there will be a function commute-fg such that (compose commute-fg (compose f g)) is equivalent to (compose g f). With a little thought, you can see that commute-fg is equivalent to (compose g f g-1 f-1). (Again, this isn't a generic commute, it only causes this specific f and g to commute.) commute-fg is called a commutator because it makes f and g commute. Commutators always have the form aba-1b-1. By finding commutators and inserting them in the right place, you can take a sequence and swap adjacent elements. Again, a concrete example with lists:

;; an illustration of what swap-first two does
> (swap-first-two '(a b c d))
(b a c d)

;; we're given
> (reverse (swap-first-two '(a b c d)))
(d c a b)

;; but we want, for some reason to reverse first
> (swap-first-two (reverse '(a b c d)))
(c d b a)

;; rewrite as fold-left of compose
> ((fold-left compose identity (list reverse swap-first-two)) '(a b c d))
(d c a b)

;; define our commutator
;; note that swap-first-two and reverse are their own inverses
> (define commute-fg
    (fold-left compose identity (list swap-first-two reverse swap-first-two reverse)))

;; make f and g commute
;; observe that it returns the desired result
> ((fold-left compose identity (list commute-fg reverse swap-first-two)) '(a b c d))
(c d b a)

There's two interesting things here. First, notice that in both examples I convert (f (g x)) to ((fold-left compose identity (list f g)) x) and then proceed to ignore x and just consider (fold-left compose identity (list f g)) as if x didn't exist. I've abstracted away the x. (Of course I have to eventually supply the x if I want an answer, but it only comes back at the last moment.) Second, notice that although slide and commute-fg are foldable sequences, I use them as if they were higher order functions operating on the foldable sequence (compose f g) to transform it, first into (compose h f), second into (compose g f). This second thing is a neat trick. We're taking a function that operates on lists and treating it as if it were a higher-order function that operates on functions. This is called the "action" of slide and commute-fg because it appears as if elements of the set G of our group can "act" directly on other elements.

Every element in the underlying set G of a group has an action associated with it which operates directly on other elements in G. This is an important concept in group theory. Now earlier I said that the actual elements of G don't matter much, so the action must be more closely tied to the operator *. And if we swap out G for another set we'll still have the same actions, they'll just be associated with the elements of the new set (in an isomorphic way). The actions are pretty abstract.

There's a lot more one could say about the actions. They are a rich source of interesting math. My brain is getting fatigued with all this abstraction, so I'll leave the topic be for now.

If group theory is about the essence of what it means to have a foldable sequence, then category theory is about the essence of composition. They offer two somewhat different approaches to similar material. What do you do with sequences but compose them? What comes from composition but a sequence? Many concepts in group theory carry over into category theory. Naturally a completely different set of terminology is used, but the concepts are there.

But that's enough group theory for today and category theory can wait until later posts.

16 Jan 2020 1:48pm GMT

Nineteen years ago today, I released Drupal 1.0.0. Every day, for the past nineteen years, the Drupal community has collaborated on providing the world with an Open Source CMS and making a difference on how the web is built and run.

It's easy to forget that software is written one line of code at the time. And that adoption is driven one website at the time. I look back on nearly two decades of Drupal, and I'm incredibly proud of our community, and how we've contributed to a more open, independent internet.

Today, many of us in the Drupal community are working towards the launch of Drupal 9. Major releases of Drupal only happen every 3-5 years. They are an opportunity to bring our community together, create something meaningful, and celebrate our collective work. But more importantly, major releases are our best opportunity to re-engage past users and attract new users, explaining why Drupal is better than closed CMS platforms.

As we mark Drupal's 19th year, let's all work together on a successful launch of Drupal 9 in 2020, including a wide-spread marketing strategy. It's the best birthday present we can give to Drupal.

15 Jan 2020 1:56pm GMT

The Event Horizon Telescope (EHT) - a planet-scale array of 8 ground-based radio telescopes and part of an international collaboration - captured the first image of a black hole. On April 10th 2019, EHT researchers disclosed the first direct visual evidence of a supermassive black hole in the heart of the Galaxy Messier 87.

26 Apr 2019 2:32am GMT

Socrates couldn't have been more right when he said: "I know one thing, that I know nothing." Even with all of the advancements we, as a species, have made in this world, it's still nothing compared to countless of wonders waiting to be discovered in the vast universe. If you've recently developed an interest in ... Read more

04 Nov 2018 1:27pm GMT

On Sunday, May 20th 2012, people in a narrow strip from Japan to the western United States will be able to see an annular solar eclipse, the first in 18 years. The moon will cover as much as 94% of the sun. An Annular Solar Eclipse is different from a Total Solar Eclipse, when the ... Read more

20 May 2012 9:51pm GMT

From the Yet Another dept.:

To further its commitment to the Java Community Process (JCP), Oracle has submitted the first of two Java Specification Requests (JSRs) to update and revitalize the JCP.

10 Nov 2011 6:01am GMT

From the Fact Checking dept.:

ZDNET: Sometimes the sheer wrongness of what is posted on the web leaves us speechless. Especially when it's picked up and repeated as gospel by otherwise reputable sites like Engadget. "Google copied Oracle's Java code, pasted in a new license, and shipped it," they reported this morning.



Sorry, but that just isn't true.

10 Nov 2011 6:01am GMT

From the Grande dept.:

Oracle today announced the availability of Java Platform, Standard Edition 7 (Java SE 7), the first release of the Java platform under Oracle stewardship.

10 Nov 2011 6:01am GMT

papupapu39 · http://identi.ca/url/56409795 #fosdem #freeknowledge #usamabinladen · about a day ago from web. Help · About · FAQ · TOS · Privacy · Source · Version · Contact. Identi.ca is a microblogging service brought to you by Status.net. ...

08 Nov 2011 12:28am GMT

FOSDEM (Free and Open Source Development European Meeting) is a European event centered around Free and Open Source software development. It is aimed at developers and all interested in the Free and Open Source news in the world. ...

05 Nov 2011 1:19am GMT

FOSDEM (Free and Open Source Development European Meeting) is a European event centered around Free and Open Source software development. It is aimed at developers and all interested in the Free and Open Source news in the ...

03 Nov 2011 3:45pm GMT

Ruby and Cocoa on OS X, the iPhone, and the Device That Shall Not Be Named

28 Oct 2011 8:00pm GMT

This was originally posted as a comment on @fogus's blog post "Why Clojure doesn't need invokedynamic, but it might be nice". I figured it's worth a top-level post here.

Ok, there's some good points here and a few misguided/misinformed positions. I'll try to cover everything.

First, I need to point out a key detail of invokedynamic that may have escaped notice: any case where you must bounce through a generic piece of code to do dispatch -- regardless of how fast that bounce may be -- prevents a whole slew of optimizations from happening. This might affect Java dispatch, if there's any argument-twiddling logic shared between call sites. It would definitely affect multimethods, which are using a hand-implemented PIC. Any case where there's intervening code between the call site and the target would benefit from invokedynamic, since invokedynamic could be used to plumb that logic and let it inline straight through. This is, indeed, the primary benefit of using invokedynamic: arbitrarily complex dispatch logic folds away allowing the dispatch to optimize as if it were direct.

Your point about inference in Java dispatch is a fair one...if Clojure is able to infer all cases, then there's no need to use invokedynamic at all. But unless Clojure is able to infer all cases, then you've got this little performance time bomb just waiting to happen. Tweak some code path and obscure the inference, and kablam, you're back on a slow reflective impl. Invokedynamic would provide a measure of consistency; the only unforeseen perf impact would be when the dispatch turns out to *actually* be polymorphic, in which case even a direct call wouldn't do much better.

For multimethods, the benefit should be clear: the MM selection logic would be mostly implemented using method handles and "leaf" logic, allowing hotspot to inline it everywhere it is used. That means for small-morphic MM call sites, all targets could potentially inline too. That's impossible without invokedynamic unless you generate every MM path immediately around the eventual call.

Now, on to defs and Var lookup. Depending on the cost of Var lookup, using a SwitchPoint-based invalidation plus invokedynamic could be a big win. In Java 7u2, SwitchPoint-based invalidation is essentially free until invalidated, and as you point out that's a rare case. There would essentially be *no* cost in indirecting through a var until that var changes...and then it would settle back into no cost until it changes again. Frequently-changing vars could gracefully degrade to a PIC.

It's also dangerous to understate the impact code size has on JVM optimization. The usual recommendation on the JVM is to move code into many small methods, possibly using call-through logic as in multimethods to reuse the same logic in many places. As I've mentioned, that defeats many optimizations, so the next approach is often to hand-inline logic everywhere it's used, to let the JVM have a more optimizable view of the system. But now we're stepping on our own feet...by adding more bytecode, we're almost certainly impacting the JVM's optimization and inlining budgets.

OpenJDK (and probably the other VMs too) has various limits on how far it will go to optimize code. A large number of these limits are based on the bytecoded size of the target methods. Methods that get too big won't inline, and sometimes won't compile. Methods that inline a lot of code might not get inlined into other methods. Methods that inline one path and eat up too much budget might push out more important calls later on. The only way around this is to reduce bytecode size, which is where invokedynamic comes in.

As of OpenJDK 7u2, MethodHandle logic is not included when calculating inlining budgets. In other words, if you push all the Java dispatch logic or multimethod dispatch logic or var lookup into mostly MethodHandles, you're getting that logic *for free*. That has had a tremendous impact on JRuby performance; I had previous versions of our compiler that did indeed infer static target methods from the interpreter, but they were often *slower* than call site caching solely because the code was considerably larger. With invokedynamic, a call is a call is a call, and the intervening plumbing is not counted against you.

Now, what about negative impacts to Clojure itself...

#0 is a red herring. JRuby supports Java 5, 6, and 7 with only a few hundred lines of changes in the compiler. Basically, the compiler has abstract interfaces for doing things like constant lookup, literal loading, and dispatch that we simply reimplement to use invokedynamic (extending the old non-indy logic for non-indified paths). In order to compile our uses of invokedynamic, we use Rémi Forax's JSR-292 backport, which includes a "mock" jar with all the invokedynamic APIs stubbed out. In our release, we just leave that library out, reflectively load the invokedynamic-based compiler impls, and we're off to the races.

#1 would be fair if the Oracle Java 7u2 early-access drops did not already include the optimizations that gave JRuby those awesome numbers. The biggest of those optimizations was making SwitchPoint free, but also important are the inlining discounting and MutableCallSite improvements. The perf you see for JRuby there can apply to any indirected behavior in Clojure, with the same perf benefits as of 7u2.

For #2, to address the apparent vagueness in my blog post...the big perf gain was largely from using SwitchPoint to invalidate constants rather than pinging a global serial number. Again, indirection folds away if you can shove it into MethodHandles. And it's pretty easy to do it.

#3 is just plain FUD. Oracle has committed to making invokedynamic work well for Java too. The current thinking is that "lambda", the support for closures in Java 7, will use invokedynamic under the covers to implement "function-like" constructs. Oracle has also committed to Nashorn, a fully invokedynamic-based JavaScript implementation, which has many of the same challenges as languages like Ruby or Python. I talked with Adam Messinger at Oracle, who explained to me that Oracle chose JavaScript in part because it's so far away from Java...as I put it (and he agreed) it's going to "keep Oracle honest" about optimizing for non-Java languages. Invokedynamic is driving the future of the JVM, and Oracle knows it all too well.

As for #4...well, all good things take a little effort :) I think the effort required is far lower than you suspect, though.

14 Oct 2011 2:40pm GMT

Hi everyone,

Rails 3.1.1 has been released. This release requires at least sass-rails 3.1.4

CHANGES

ActionMailer

• No changes

ActionPack

• stylesheetlinktag('/stylesheets/application') and similar helpers doesn't throw Sprockets::FileOutsidePaths exception anymore [Santiago Pastorino]

• Ensure defaultassethost_protocol is respected, closes #2980. [José Valim]

  Changing rake db:schema:dump to run :environment as well as :load_config, as running :load_config alone will lead to the dumper being run without including extensions such as those included in foreigner and spatial_adapter.

  This reverses a change made here: https://github.com/rails/rails/commit/5df72a238e9fcb18daf6ab6e6dc9051c9106d7bb#L0L324

  I'm assuming here that :load_config needs to be invoked separately from :environment, as it is elsewhere in the file for db operations, if not the alternative is to go back to "task :dump => :environment do".

  [Ben Woosley]

• Update to rack-cache 1.1.

  Versions prior to 1.1 delete the If-Modified-Since and If-Not-Modified headers when config.actioncontroller.performcaching is true. This has two problems:

  • unexpected inconsistent behaviour between development & production environments
  • breaks applications that use of these headers

  [Brendan Ribera]

• Ensure that enhancements to assets:precompile task are only run once [Sam Pohlenz]

• TestCase should respect the view_assigns API instead of pulling variables on its own. [José Valim]

• javascriptpath and stylesheetpath now refer to /assets if asset pipelining is on. [Santiago Pastorino]

• button_to support form option. Now you're able to pass for example 'data-type' => 'json'. [ihower]

• imagepath and imagetag should use /assets if asset pipelining is turned on. Closes #3126 [Santiago Pastorino and christos]

• Avoid use of existing precompiled assets during rake assets:precompile run. Closes #3119 [Guillermo Iguaran]

• Copy assets to nondigested filenames too [Santiago Pastorino]

• Give precedence to config.digest = false over the existence of manifest.yml asset digests [christos]

• escape options for the stylesheetlinktag method [Alexey Vakhov]

• Re-launch assets:precompile task using (Rake.)ruby instead of Kernel.exec so it works on Windows [cablegram]

• env var passed to process shouldn't be modified in process method. [Santiago Pastorino]

• rake assets:precompile loads the application but does not initialize it.

  To the app developer, this means configuration add in config/initializers/* will not be executed.

  Plugins developers need to special case their initializers that are meant to be run in the assets group by adding :group => :assets. [José Valim]

• Sprockets uses config.assets.prefix for asset_path [asee]

• FileStore keyfilepath properly limit filenames to 255 characters. [phuibonhoa]

• Fix Hash#toquery edge case with htmlsafe strings. [brainopia]

• Allow asset tag helper methods to accept :digest => false option in order to completely avoid the digest generation. Useful for linking assets from static html files or from emails when the user could probably look at an older html email with an older asset. [Santiago Pastorino]

• Don't mount Sprockets server at config.assets.prefix if config.assets.compile is false. [Mark J. Titorenko]

• Set relative url root in assets when controller isn't available for Sprockets (eg. Sass files using asset_path). Fixes #2435 [Guillermo Iguaran]

• Fix basic auth credential generation to not make newlines. GH #2882

• Fixed the behavior of asset pipeline when config.assets.digest and config.assets.compile are false and requested asset isn't precompiled. Before the requested asset were compiled anyway ignoring that the config.assets.compile flag is false. [Guillermo Iguaran]

• CookieJar is now Enumerable. Fixes #2795

• Fixed AssetNotPrecompiled error raised when rake assets:precompile is compiling certain .erb files. See GH #2763 #2765 #2805 [Guillermo Iguaran]

• Manifest is correctly placed in assets path when default assets prefix is changed. Fixes #2776 [Guillermo Iguaran]

• Fixed stylesheetlinktag and javascriptincludetag to respect additional options passed by the users when debug is on. [Guillermo Iguaran]

• Fix ActiveRecord#exists? when passsed a nil value

• Fix assertselectemail to work on multipart and non-multipart emails as the method stopped working correctly in Rails 3.x due to changes in the new mail gem.

ActiveModel

• Remove hard dependency on bcrypt-ruby to avoid make ActiveModel dependent on a binary library. You must add the gem explicitly to your Gemfile if you want use ActiveModel::SecurePassword:

  gem 'bcrypt-ruby', '~> 3.0.0'

  See GH #2687. [Guillermo Iguaran]

ActiveRecord

• Add deprecation for the preload_associations method. Fixes #3022.

  [Jon Leighton]

• Don't require a DB connection when loading a model that uses setprimarykey. GH #2807.

  [Jon Leighton]

• Fix using select() with a habtm association, e.g. Person.friends.select(:name). GH #3030 and #2923.

  [Hendy Tanata]

• Fix belongs_to polymorphic with custom primary key on target. GH #3104.

  [Jon Leighton]

• CollectionProxy#replace should change the DB records rather than just mutating the array. Fixes #3020.

  [Jon Leighton]

• LRU cache in mysql and sqlite are now per-process caches.

  • lib/activerecord/connectionadapters/mysql_adapter.rb: LRU cache keys are per process id.
  • lib/activerecord/connectionadapters/sqlite_adapter.rb: ditto

• Database adapters use a statement pool for limiting the number of open prepared statments on the database. The limit defaults to 1000, but can be adjusted in your database config by changing 'statement_limit'.

• Fix clash between using 'preload', 'joins' or 'eager_load' in a default scope and including the default scoped model in a nested through association. (GH #2834.) [Jon Leighton]

• Ensure we are not comparing a string with a symbol in HasManyAssociation#inverseupdatescounter_cache?. Fixes GH #2755, where a counter cache could be decremented twice as far as it was supposed to be.

  [Jon Leighton]

• Don't send any queries to the database when the foreign key of a belongs_to is nil. Fixes GH #2828. [Georg Friedrich]

• Fixed findin_batches method to not include order from defaultscope. See GH #2832 [Arun Agrawal]

• Don't compute table name for abstract classes. Fixes problem with setting the primary key in an abstract class. See GH #2791. [Akira Matsuda]

• Psych errors with poor yaml formatting are proxied. Fixes GH #2645 and GH #2731

• Use the LIMIT word with the methods #last and #first. Fixes GH #2783 [Damien Mathieu]

ActiveResource

• No changes

ActiveSupport

• ruby193: String#prepend is also unsafe [Akira Matsuda]

• Fix obviously breakage of Time.=== for Time subclasses [jeremyevans]

• Added fix so that file store does not raise an exception when cache dir does not exist yet. This can happen if a delete_matched is called before anything is saved in the cache. [Philippe Huibonhoa]

• Fixed performance issue where TimeZone lookups would require tzinfo each time [Tim Lucas]

• ActiveSupport::OrderedHash is now marked as extractable when using Array#extract_options! [Prem Sichanugrist]

Railties

• Add jquery-rails to Gemfile of plugins, test/dummy app needs it. Closes #3091. [Santiago Pastorino]

• rake assets:precompile loads the application but does not initialize it.

  To the app developer, this means configuration add in config/initializers/* will not be executed.

  Plugins developers need to special case their initializers that are meant to be run in the assets group by adding :group => :assets.

SHA-1

• 9337cff7772da034b0b34b73b85cf249f1a70f52 actionmailer-3.1.1.gem
• 7bb1b8d096a6ff1ff46dcfb778bf86a5daca1b0d actionpack-3.1.1.gem
• d5dc71e1a9a0e20d819f4dff27ff0697e99a7f64 activemodel-3.1.1.gem
• 7245632cb3b38612628304c1e244855d0053f7be activerecord-3.1.1.gem
• 6d09800202c2747e84249b8646f0fd480ed4924f activeresource-3.1.1.gem
• 66df2fd144aab22f52819fd489e33a976d68a46b activesupport-3.1.1.gem
• 6a35a49948bbd9f461839a1a271def90b23a851a rails-3.1.1.gem
• 6979ef891bd03fb639b979af9fdc56781f9358d9 railties-3.1.1.gem

You can find an exhaustive list of changes on github. Along with the closed issues marked for v3.1.1.

Thanks to everyone!

07 Oct 2011 5:26pm GMT

If you see this message in your RSS reader, please correct your RSS link to the following URL: http://fosdem.org/rss.xml.

26 Jul 2008 5:55am GMT

These pages have been archived.
For information about the latest FOSDEM edition please check this url: http://fosdem.org

25 Jul 2008 4:43pm GMT

Two weeks after FOSDEM and we are proud to publish most of the slides and videos from this year's edition.

All of the material from the Lightning Talks has been put online. We are still missing some slides and videos from the Main Tracks but we are working hard on getting those completed too.

We would like to thank our mirrors: HEAnet (IE) and Unixheads (US) for hosting our videos, and NamurLUG for quick recording and encoding.

The videos from the Janson room were live-streamed during the event and are also online on the Linux Magazin site.

We are having some synchronisation issues with Belnet (BE) at the moment. We're working to sort these out.

09 Mar 2008 3:12pm GMT