This library provides a few methods to work with data tables. Think about it as a limited version of Pandas Dataframe for Common Lisp.

Data-table abilities are really modest. For example, it does not provide methods to investigate data in the table.

To overcome this limitation, we'll use another library - cl-ascii-table and define a describe-object method for data tables:

POFTHEDAY> (defparameter *dt*
             (make-instance 'data-table:data-table
                            :column-names '("Name" "Birthday" "Age")
                            :rows '(("Bob" "1985-05-17" 35)
                                    ("Alice" "1989-02-23" 31)
                                    ("John" "2000-01-03" 20))))

;; This is what we'll wee by default:
POFTHEDAY> (describe *dt*)
#<DATA-TABLE:DATA-TABLE {1003D50293}>
  [standard-object]

Slots with :INSTANCE allocation:
  COLUMN-NAMES                   = ("Name" "Birthday" "Age")
  COLUMN-TYPES                   = (STRING STRING (INTEGER 0 4611686018427387903))
  ROWS                           = (("Bob" "1985-05-17" 35) ("Alice" "1989-02-23" 31))

;; But with custom method we might make it more
;; more convenient:
POFTHEDAY> (defmethod describe-object ((obj data-table:data-table) stream)
             (loop with table = (ascii-table:make-table
                                 (data-table:column-names obj))
                   for row in (data-table:rows obj)
                   do (ascii-table:add-row table row)
                   finally (ascii-table:display table)))

POFTHEDAY> (describe *dt*)
+-------+------------+-----+
| Name  | Birthday   | Age |
+-------+------------+-----+
| Bob   | 1985-05-17 |  35 |
| Alice | 1989-02-23 |  31 |
| John  | 2000-01-03 |  20 |
+-------+------------+-----+

Now let's see what capabilities for data slicing does it have.

Here is how we can retrieve columns. Pay attention, column names are case insensitive. But if case mismatch, column names of the resulting table will be different. Maybe this is a bug:

POFTHEDAY> (describe
            (data-table:select-columns *dt*
                                       '("Name" "Age")))
+-------+-----+
| Name  | Age |
+-------+-----+
| Bob   |  35 |
| Alice |  31 |
| John  |  20 |
+-------+-----+
; No values
POFTHEDAY> (describe
            (data-table:select-columns *dt*
                                       '("name" "age")))
+-------+-----+
| name  | age |
+-------+-----+
| Bob   |  35 |
| Alice |  31 |
| John  |  20 |
+-------+-----+

There is also a way to select a subtable using indices.

This is how we can select contiguous columns with "Name" and "Age" labels:

;; We have to do call this method first.
;; It will fill a list of datatypes.
;; Without filled list of datatypes, make-sub-table will fail :(

POFTHEDAY> (data-table:coerce-data-table-of-strings-to-types *dt*)

POFTHEDAY> (describe
            (data-table:make-sub-table *dt*
                                       :lci 1 :uci 3))
+------------+-----+
| Birthday   | Age |
+------------+-----+
| 1985-05-17 |  35 |
| 1989-02-23 |  31 |
| 2000-01-03 |  20 |
+------------+-----+

If we want to select only the last two rows, then we have to do some math:

;; In pandas.DataFrame this will be just: dt.tail(2)
;;
POFTHEDAY> (let ((rows-count (data-table:number-of-rows *dt*)))
             (describe
              (data-table:make-sub-table *dt*
                                         :lri (- rows-count 2)
                                         :uri rows-count)))
+-------+------------+-----+
| Name  | Birthday   | Age |
+-------+------------+-----+
| Alice | 1989-02-23 |  31 |
| John  | 2000-01-03 |  20 |
+-------+------------+-----+

Also, we can combine column selection by name with make-sub-table. But this will create an intermediate table.

There is also a method overlay-region to combine two tables. Let's use it to add email for Genry and Mary:

POFTHEDAY> (defparameter *dt*
             (data-table:plists-to-data-table
              '((:id 1 :name "Bob")
                (:id 2 :name "Genry")
                (:id 3 :name "Mary"))))

POFTHEDAY> (describe *dt*)
+----+-------+
| ID | NAME  |
+----+-------+
|  1 | Bob   |
|  2 | Genry |
|  3 | Mary  |
+----+-------+

POFTHEDAY> (data-table:overlay-region
            (data-table:plists-to-data-table
             '((:email "genry@gmail.com")
               (:email "mary@hormail.com")))
            *dt*)
NIL

POFTHEDAY> (describe *dt*)
+------------------+-------+
| ID               | NAME  |
+------------------+-------+
| genry@gmail.com  | Bob   |
| mary@hormail.com | Genry |
|                3 | Mary  |
+------------------+-------+

POFTHEDAY> ;; Oh, this method modified the original table :(((

To fix the issue, we need to recreate an original table and to specify indexes for applied overlay:

POFTHEDAY> (defparameter *dt*
             (data-table:plists-to-data-table
              '((:id 1 :name "Bob")
                (:id 2 :name "Genry")
                (:id 3 :name "Mary"))))

POFTHEDAY> (data-table:overlay-region
            (data-table:plists-to-data-table
             '((:email "genry@gmail.com")
               (:email "mary@hormail.com")))
            *dt*
            :col-idx 2
            :row-idx 1)

;; But new column has no name now :(
POFTHEDAY> (describe *dt*)
+----+-------+------------------+
| ID | NAME  | NIL              |
+----+-------+------------------+
|  1 | Bob   | NIL              |
|  2 | Genry | genry@gmail.com  |
|  3 | Mary  | mary@hormail.com |
+----+-------+------------------+

It would be nice to have support for indices and joins like in Pandas. That is what I expect from the data manipulation library.

We've viewed almost all core functions of the data-table library. There are also data coercion and comparison functions. But I think this library has lots of room for improvement.

Maybe @guicho271828 has plans to build a dataframe library as supplementary to his NumCL?

29 May 2020 12:15am GMT

For more than a few decades now (!), I've been running my own server. First it was just my old Pentium 1 squatting on university networks, but eventually grew into a real server somewhere at the dawn of the millenia. Apart from the university days, the server was mostly hosted over ADSL links, first a handful of megabits, up to the current 25 Mbps down, 6 Mbps up that the Bell Canada network seems to allow to its resellers (currently Teksavvy Internet, or TSI).

Why change?

Obviously, this speed is showing its age, and especially in this age of Pandemia where everyone is on videoconferencing all the time. But it's also inconvenient when I need to upload large files on the network. I also host a variety of services on this network, and I always worry that any idiot can (rather trivially) DoS my server, so I often feel I should pack a little more punch at home (although I have no illusions about my capacity of resisting any sort of DoS attack at home of course).

Also, the idea of having gigabit links at home brings back the idea of the original internet, that everyone on the internet is a "peer". "Client" and "servers" are just a technical distinction and everyone should be able to run a server.

Requirements

So I'm shopping for a replacement. The requirements are:

1. higher speed than 25/6, preferably 100mbps down, 30mbps up, or more. ideally 1gbps symmetric.

2. static or near-static IP address: I run a DNS server with its IP in the glue records (although the latter could possibly be relaxed). ideally a /29 or more.

3. all ports open: I run an SMTP server (incoming and outgoing) along with a webserver and other experiments. ideally, no firewall or policy should be blocking me from hosting stuff, unless there's an attack or security issue, obviously.

4. clean IP address: the SMTP server needs to have a good reputation, so the IP address should not be in a "residential space" pool.

5. IPv6 support: TSI offers IPv6 support, but it is buggy (I frequently have to restart the IPv6 interface on the router because the delegated block stops routing, and they haven't been able to figure out the problem). ideally, a /56.

6. less than 100$/mth, ideally close to the current 60$/mth I pay.

(All amounts in $CAD.)

Contestants

I wrote a similar message asking three major ISPs in my city for those services, including business service if necessary:

• Oricom - ventes@oricom.ca
• TSI - sales@teksavvy.com
• Ebox - sales@ebox.ca

I have not contacted those providers:

• Bell Canada: i have sworn, two decades ago, never to do business with that company ever again. They have a near-monopoly on almost all telcos in Canada and I want to give them as little money as possible.

• Videotron: I know for a fact they do not allow servers on their network, and their IPv6 has been in beta for so long it has become somewhat of a joke now

I might have forgotten some, let me know if you're in the area and have a good recommendation. I'll update this post with findings as they come in.

Keep in mind that I am in a major Canadian city, less than a kilometer from a major telco exchange site, so it's not like I'm in a rural community. This should just work.

TSI

First answer from TSI was "we do not provide 30mbps upload on residential services", even though they seem to have that package on their website. They confirmed that they "don't have a option more than 10 mbps upload."

Oricom

No response yet.

Ebox

No response yet.

28 May 2020 5:40pm GMT

The following contributors got their Debian Developer accounts in the last two months:

• Paride Legovini (paride)
• Ana Custura (acute)
• Felix Lechner (lechner)

The following contributors were added as Debian Maintainers in the last two months:

• Sven Geuer
• Håvard Flaget Aasen

Congratulations!

28 May 2020 4:30pm GMT

If you tested your blog's performance on Google PageSpeed Insights yesterday and do so again today, you might be in for a surprise with a lower score even if not one byte (letter) got changed on your site. The reason: Google updated PageSpeed Insights to Lighthouse 6, which changes the KPI's (the lab data metrics) that are reported, adds new opportunities and recommendations and changes the way the total score is calculated.

So all starts with the changed KPI's in the lab metrics really; whereas up until yesterday First Contentful Paint, Speed Index, Time to Interactive, First Meaningful Paint, First CPU Idle and First input delay were measured, the last 3 ones are now not shown any more, having been replaced by:

• Largest Contentful Paint marks the point when the page's main content has likely loaded, this can generally be improved upon by removing removing render-blocking resources (JS/ CSS), optimizing images, …
• Total Blocking Time quantifies how non-interactive a page while loading, this is mainly impacted by Javascript (local and 3rd party) blocking the main thread, so improving that generally means ensuring there is less JS to execute
• Cumulative Layout Shift which measures unexpected layout shifts

The total score is calculated based on all 6 metrics, but the weight of the 3 "old" ones (FCP, SI, TTI) is significantly lowered (from 80 to 45%) and the new LCP & TBT account for a whopping 50% of your score (CLS is only 5%).

Lastly some one very interesting opportunity and two recommendations I noticed;

• GPSI already listed unused CSS, but now adds unused JS to that list, which will prove to be equally hard to control in WordPress as JS like CSS is added by almost each and every plugin. Obviously if you're using Autoptimize this will flag the Autoptimized JS, disalbe Autoptimize for the test by adding ?ao_noptimize=1 to the URL to see what original JS is unused.
• GPSI now warns about using document.write and about the impact of passive listeners on scrolling performance which can lead to Google complaining about … Google :-)

Summary: Google Pagespeed Insights changed a lot and it forces performance-aware users to stay on their toes. Especially sites with lots of (3rd party) JavaScript might want to reconsider some of the tools used.

28 May 2020 11:36am GMT

I published the following diary on isc.sans.edu: "Flashback on CVE-2019-19781":

First of all, did you know that the Flame malware turned 8 years today! Happy Birthday! This famous malware discovered was announced on May 28th, 2012. The malware was used for targeted cyber espionage activities in the Middle East area. If this malware was probably developed by a nation-state organization. It infected a limited amount of hosts (~1000 computers) making it a targeted attack… [Read more]

[The post [SANS ISC] Flashback on CVE-2019-19781 has been first published on /dev/random]

28 May 2020 11:31am GMT

I typically use LibreOffice Impress for my talks, much to some folks' surprise. Yes, you can make slides look okay with free software! But there's one annoying caveat that has bothered me for ages.

Impress makes it nearly impossible to enter presenter mode with a single display, while also displaying slides. I have never understood this limitation, but it's existed for a minimum of seven years.

I've tried all sorts of workarounds over the years, including a macro that forces LibreOffice into presenter mode, which I never was able to figure out how to reverse once I ran it...

This has previously been an annoyance but never posed a big problem, because when push came to shove I could leave my house and use an external monitor or screen when presenting at meetups. But now, everything's virtual, I'm in lockdown indefinitely, and I don't have another display available at home. And about 8 hours before speaking at a meetup today, I realized I didn't have a way to share my slides while seeing my speaker notes. Uh oh.

So I got this stupid idea.

...why don't I just placate LibreOffice with a FAKE display?

Virtual displays with xrandr

My GPU had this capability innately, it turns out, if I could just whisper the right incantations to unlock its secrets:

ehashman@red-dot:~$ cat /usr/share/X11/xorg.conf.d/20-intel.conf 
Section "Device"
    Identifier "intelgpu0"
    Driver "intel"
    Option "VirtualHeads" "1"
EndSection

After restarting X to allow this newly created config to take effect, I now could see two new virtual displays available for use:

ehashman@red-dot:~$ xrandr
Screen 0: minimum 8 x 8, current 3840 x 1080, maximum 32767 x 32767
eDP1 connected primary 1920x1080+0+0 (normal left inverted right x axis y axis) 310mm x 170mm
   1920x1080     60.01*+  59.93  
   ...
   640x360       59.84    59.32    60.00  
DP1 disconnected (normal left inverted right x axis y axis)
DP2 disconnected (normal left inverted right x axis y axis)
HDMI1 disconnected (normal left inverted right x axis y axis)
HDMI2 disconnected (normal left inverted right x axis y axis)
VIRTUAL1 disconnected (normal left inverted right x axis y axis)
VIRTUAL2 disconnected (normal left inverted right x axis y axis)

Nice. Now, to actually use it:

ehashman@red-dot:~$ xrandr --addmode VIRTUAL1 1920x1080
ehashman@red-dot:~$ xrandr --output VIRTUAL1 --mode 1920x1080 --right-of eDP1

And indeed, after running these commands, I found myself with a virtual display, very happy to black hole all my windows, available to the imaginary right of my laptop screen.

This allowed me to mash that "Present" button in LibreOffice and get my presenter notes on my laptop display, while putting my actual slides in a virtual time-out that I could still screenshare!

Wouldn't it be nice if LibreOffice just fixed the actual bug? 🤷

Well, actually...

I must forgive myself for my stage panic. The talk ended up going great, and the immediate problem was solved. But it turns out this bug has been addressed upstream! It's just... not well-documented.

A couple years ago, there was a forum post on ask.libreoffice.org that featured this exact question, and a solution was provided!

Yes, use Open Expert Configuration via Tools > Options > LibreOffice > Advanced. Search for StartAlways. You should get a node org.openoffice.Office.PresenterScreen with line Presenter. Double-click that line to toggle the boolean value to true.

I tested this out locally and... yeah that works. But it turns out the bug from 2013 had not been updated with this solution until just a few months ago.

There are very limited search results for this configuration property. I wish this was much better documented. But so it goes with free software; here's a hack and a real solution as we all try to improve things :)

28 May 2020 1:15am GMT

In this episode, I worked on a views and templates. There are a number of core pages that are required to flesh out the minimal interface for the app. We're building them. I began by showing the page that we were going to work on. I outlined the changes I planned to make, then we started. The first thing we added was data about the school year, the main model on display in the page.

28 May 2020 12:00am GMT

A large software project is above all a social exercise in which technical experts try to reach good decisions together, for instance on github pull requests. But communication is difficult, in particular between diverging points of view. It is easy to underestimate how much well-intended persons can misunderstand each-other and get hurt, in open source as elsewhere. Knowing why there are communication challenges can help, as well as applying a few simple rules.

The first challenge is to understand the other's point of view: the different parties see the problem differently.

27 May 2020 10:00pm GMT

This system adds Unicode support to the cl-ppcre.

What does it mean? It means that after loading cl-ppcre-unicode you'll be able to match against Unicode symbol properties.

A property matcher has a special syntax in cl-ppcre's regexps: \p{PropertyName}.

Here is an example:

;; This is how we can find out a position
;; of the first Cyrillic letter:

POFTHEDAY> (ppcre:scan "\\p{Cyrillic}"
                       "123Ю56")
3

;; Here we are extracting a
;; sequence of Emoji from the text:
POFTHEDAY> (ppcre:regex-replace
            ".*?([\\p{Emoticons}|\\p{Supplemental Symbols and Pictographs}]+).*"
            "Hello, Lisper! 🤗😃 How are you?"
            "\\1")
"🤗😃"

We are using two different Unicode classes as properties because these two characters belong to different classes.

You can use cl-unicode to discover the character's unicode class:

POFTHEDAY> (cl-unicode:code-block #\😃)
"Emoticons"

POFTHEDAY> (cl-unicode:code-block #\🤗)
"Supplemental Symbols and Pictographs"

The way, how cl-ppcre-unicode works is very interesting. It turns out that cl-ppcre has a special hook which allows you to define a property resolver.

For example, if you want to have a special property for vowels, you might do something like that:

POFTHEDAY> (defun my-property-resolver (property-name)
             (if (string-equal property-name
                               "vowel")
                 (rutils:fn vovel-p (character)
                   (member character '(#\A #\E #\I #\O #\U)
                           :test #'char-equal))
                 (cl-ppcre-unicode:unicode-property-resolver
                  property-name)))

POFTHEDAY> (setf cl-ppcre:*property-resolver*
                 #'my-property-resolver)

;; And now we can use the "Vowel" property in any
;; regular expressions!
POFTHEDAY> (ppcre:regex-replace-all
            "\\p{Vowel}"
            "Hello, Lisper! How are you?"
            "")
"Hll, Lspr! Hw r y?"

Isn't this cool!? 🤪

27 May 2020 8:15pm GMT

sample_project/
│
├── receipts/
│   ├── fixtures/
│   │   └── receipts.json
│   │
│   ├── migrations/
│   │   ├── 0001_initial.py
│   │   └── __init__.py
│   │
│   ├── __init__.py
│   ├── admin.py
│   ├── apps.py
│   ├── models.py
│   ├── tests.py
│   ├── urls.py
│   └── views.py
│
├── sample_project/
│   ├── __init__.py
│   ├── asgi.py
│   ├── settings.py
│   ├── urls.py
│   └── wsgi.py
│
├── db.sqlite3
├── manage.py
├── resetdb.sh
└── runserver.sh

$ python -m pip install Django
$ django-admin startproject sample_project
$ cd sample_project
$ ./manage.py startapp receipts

Read the full article at https://realpython.com/installable-django-app/ »

[ Improve Your Python With 🐍 Python Tricks 💌 - Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

27 May 2020 2:00pm GMT

JOSE is an implementation of Javascript Object Signing and Encryption.

This @nitro_idiot's library implements a JSON Web Signature and allows to encode, decode and inspect JWT tokens.

JOSE can be useful to implement signed token exchange between microservices.

I found a great article on what JWT, JWS and JWE are. Read it you want more about them. To demonstrate, how does token inspection work, I took a JWT token from the article and parsed it with Common Lisp JOSE:

POFTHEDAY> (jose/jwt:inspect-token
            "eyJhbGciOiJSUzI1NiIsImtpZCI6Ijc4YjRjZjIzNjU2ZGMzOTUzNjRmMWI2YzAyOTA3NjkxZjJjZGZmZTEifQ.eyJpc3MiOiJhY2NvdW50cy5nb29nbGUuY29tIiwic3ViIjoiMTEwNTAyMjUxMTU4OTIwMTQ3NzMyIiwiYXpwIjoiODI1MjQ5ODM1NjU5LXRlOHFnbDcwMWtnb25ub21ucDRzcXY3ZXJodTEyMTFzLmFwcHMuZ29vZ2xldXNlcmNvbnRlbnQuY29tIiwiZW1haWwiOiJwcmFiYXRoQHdzbzIuY29tIiwiYXRfaGFzaCI6InpmODZ2TnVsc0xCOGdGYXFSd2R6WWciLCJlbWFpbF92ZXJpZmllZCI6dHJ1ZSwiYXVkIjoiODI1MjQ5ODM1NjU5LXRlOHFnbDcwMWtnb25ub21ucDRzcXY3ZXJodTEyMTFzLmFwcHMuZ29vZ2xldXNlcmNvbnRlbnQuY29tIiwiaGQiOiJ3c28yLmNvbSIsImlhdCI6MTQwMTkwODI3MSwiZXhwIjoxNDAxOTEyMTcxfQ.TVKv-pdyvk2gW8sGsCbsnkqsrS0T-H00xnY6ETkIfgIxfotvFn5IwKm3xyBMpy0FFe0Rb5Ht8AEJV6PdWyxz8rMgX2HROWqSo_RfEfUpBb4iOsq4W28KftW5H0IA44VmNZ6zU4YTqPSt4TPhyFC9fP2D_Hg7JQozpQRUfbWTJI")

(("exp" . 1401912171) ("iat" . 1401908271) ("hd" . "wso2.com")
 ("aud"
  . "825249835659-te8qgl701kgonnomnp4sqv7erhu1211s.apps.googleusercontent.com")
 ("email_verified" . T) ("at_hash" . "zf86vNulsLB8gFaqRwdzYg")
 ("email" . "prabath@wso2.com")
 ("azp"
  . "825249835659-te8qgl701kgonnomnp4sqv7erhu1211s.apps.googleusercontent.com")
 ("sub" . "110502251158920147732") ("iss" . "accounts.google.com"))

(("alg" . "RS256") ("kid" . "78b4cf23656dc395364f1b6c02907691f2cdffe1"))

#(77 82 175 250 151 114 190 77 160 91 203 6 176 38 236 158 74 172 173 45 19 248
  125 52 198 118 58 17 57 8 126 2 49 126 139 111 22 126 72 192 169 183 199 32
  76 167 45 5 21 237 17 111 145 237 240 1 9 87 163 221 91 44 115 242 179 32 95
  97 209 57 106 146 163 244 95 17 245 41 5 190 34 58 202 184 91 111 10 126 213
  185 31 66 0 227 133 102 53 158 179 83 134 19 168 244 173 225 51 225 200 80
  189 124 253 131 252 120 59 37 10 51 165 4 84 125 181 147 36)

Inspect returns payload, headers and the signature.

There are also two methods encode and decode. Here is how we can use them to issue and verify JWT token:

POFTHEDAY> (defparameter
               *secret* (ironclad:ascii-string-to-byte-array
                         "I Love Common Lisp!"))

POFTHEDAY> (defparameter
               *token*
             (jose:encode :hs256 *secret*
                          '(("user"  . "Bob")
                            ("email" . "bob@gmail.com"))))

;; Now this token can be sent to the authenticated
;; user so that user can pass it back during API calls:
POFTHEDAY> *token*
"eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1c2VyIjoiQm9iIiwiZW1haWwiOiJib2JAZ21haWwuY29tIn0.NLgg5RxlKDNqw1cqFU0_HysIu-zO7JBYUQN2IZF6c6w"

;; And when we'll receive such API call,
;; we can know who this user is.
POFTHEDAY> (jose:decode :hs256 *secret*
                        *token*)
(("user" . "Bob") ("email" . "bob@gmail.com"))
(("alg" . "HS256") ("typ" . "JWT"))

A cool feature of JWT token is that it is signed and you can trust the payload's content. Let's pretend, that Bob is the "evil hacker" who wants to get access to another account.

To do this, Bob will modify payload and use header and sign from the original token:

POFTHEDAY> (defun replace-payload (token new-payload)
             (rutils:with (((original-header rutils:_ original-sign)
                            (str:split "." token))
                           (json-payload
                            (jonathan:to-json new-payload :from :alist))
                           (new-encoded-payload
                            (jose/base64:base64url-encode json-payload)))
               (str:join "."
                         (list original-header
                               new-encoded-payload
                               original-sign))))

POFTHEDAY> (defparameter *new-token*
             (replace-payload *token*
                              '(("user"  . "Alice")
                                ("email" . "alice@wonderland.in"))))

;; Now we'll try to decode token on the server-side
;; and receive error from JOSE:
POFTHEDAY> (jose:decode :hs256 *secret*
                        *new-token*)
; Debugger entered on #<JOSE/ERRORS:JWS-VERIFICATION-ERROR {1005C37033}>
[1] POFTHEDAY> 
; Evaluation aborted on #<JOSE/ERRORS:JWS-VERIFICATION-ERROR {1005C37033}>

;; But we still can inspect bad token because
;; it's content is not encrypted:
POFTHEDAY> (jose:inspect-token *new-token*)

(("email" . "alice@wonderland.in") ("user" . "Alice"))

(("alg" . "HS256") ("typ" . "JWT"))

#(52 184 32 229 28 101 40 51 106 195 87 42 21 77 63 31 43 8 187 236 206 236 144
  88 81 3 118 33 145 122 115 172)

BTW, as you can see, tokens are not encoded, they are signed. Because of that, you should pass them only over encrypted channels.

26 May 2020 8:15pm GMT

I recently learned that quite a few (old) root certificates are going to expire, and many websites still send those along in the TLS handshake.

25 May 2020 12:00am GMT

For 30 years, the space telescope Hubble has provided the most impressive images from the vast wide space. The telescope was developed by the US space agency NASA and its European counterpart ESA. Hubble started its journey into space on 24 April 1990. With the help of the space shuttle "Discovery" it was lifted into ... Read more

27 Apr 2020 3:47pm GMT

The Event Horizon Telescope (EHT) - a planet-scale array of 8 ground-based radio telescopes and part of an international collaboration - captured the first image of a black hole. On April 10th 2019, EHT researchers disclosed the first direct visual evidence of a supermassive black hole in the heart of the Galaxy Messier 87.

26 Apr 2019 2:32am GMT

Socrates couldn't have been more right when he said: "I know one thing, that I know nothing." Even with all of the advancements we, as a species, have made in this world, it's still nothing compared to countless of wonders waiting to be discovered in the vast universe. If you've recently developed an interest in ... Read more

04 Nov 2018 1:27pm GMT

From the Yet Another dept.:

To further its commitment to the Java Community Process (JCP), Oracle has submitted the first of two Java Specification Requests (JSRs) to update and revitalize the JCP.

10 Nov 2011 6:01am GMT

From the Fact Checking dept.:

ZDNET: Sometimes the sheer wrongness of what is posted on the web leaves us speechless. Especially when it's picked up and repeated as gospel by otherwise reputable sites like Engadget. "Google copied Oracle's Java code, pasted in a new license, and shipped it," they reported this morning.



Sorry, but that just isn't true.

10 Nov 2011 6:01am GMT

From the Grande dept.:

Oracle today announced the availability of Java Platform, Standard Edition 7 (Java SE 7), the first release of the Java platform under Oracle stewardship.

10 Nov 2011 6:01am GMT

papupapu39 · http://identi.ca/url/56409795 #fosdem #freeknowledge #usamabinladen · about a day ago from web. Help · About · FAQ · TOS · Privacy · Source · Version · Contact. Identi.ca is a microblogging service brought to you by Status.net. ...

08 Nov 2011 12:28am GMT

FOSDEM (Free and Open Source Development European Meeting) is a European event centered around Free and Open Source software development. It is aimed at developers and all interested in the Free and Open Source news in the world. ...

05 Nov 2011 1:19am GMT

FOSDEM (Free and Open Source Development European Meeting) is a European event centered around Free and Open Source software development. It is aimed at developers and all interested in the Free and Open Source news in the ...

03 Nov 2011 3:45pm GMT

Ruby and Cocoa on OS X, the iPhone, and the Device That Shall Not Be Named

28 Oct 2011 8:00pm GMT

This was originally posted as a comment on @fogus's blog post "Why Clojure doesn't need invokedynamic, but it might be nice". I figured it's worth a top-level post here.

Ok, there's some good points here and a few misguided/misinformed positions. I'll try to cover everything.

First, I need to point out a key detail of invokedynamic that may have escaped notice: any case where you must bounce through a generic piece of code to do dispatch -- regardless of how fast that bounce may be -- prevents a whole slew of optimizations from happening. This might affect Java dispatch, if there's any argument-twiddling logic shared between call sites. It would definitely affect multimethods, which are using a hand-implemented PIC. Any case where there's intervening code between the call site and the target would benefit from invokedynamic, since invokedynamic could be used to plumb that logic and let it inline straight through. This is, indeed, the primary benefit of using invokedynamic: arbitrarily complex dispatch logic folds away allowing the dispatch to optimize as if it were direct.

Your point about inference in Java dispatch is a fair one...if Clojure is able to infer all cases, then there's no need to use invokedynamic at all. But unless Clojure is able to infer all cases, then you've got this little performance time bomb just waiting to happen. Tweak some code path and obscure the inference, and kablam, you're back on a slow reflective impl. Invokedynamic would provide a measure of consistency; the only unforeseen perf impact would be when the dispatch turns out to *actually* be polymorphic, in which case even a direct call wouldn't do much better.

For multimethods, the benefit should be clear: the MM selection logic would be mostly implemented using method handles and "leaf" logic, allowing hotspot to inline it everywhere it is used. That means for small-morphic MM call sites, all targets could potentially inline too. That's impossible without invokedynamic unless you generate every MM path immediately around the eventual call.

Now, on to defs and Var lookup. Depending on the cost of Var lookup, using a SwitchPoint-based invalidation plus invokedynamic could be a big win. In Java 7u2, SwitchPoint-based invalidation is essentially free until invalidated, and as you point out that's a rare case. There would essentially be *no* cost in indirecting through a var until that var changes...and then it would settle back into no cost until it changes again. Frequently-changing vars could gracefully degrade to a PIC.

It's also dangerous to understate the impact code size has on JVM optimization. The usual recommendation on the JVM is to move code into many small methods, possibly using call-through logic as in multimethods to reuse the same logic in many places. As I've mentioned, that defeats many optimizations, so the next approach is often to hand-inline logic everywhere it's used, to let the JVM have a more optimizable view of the system. But now we're stepping on our own feet...by adding more bytecode, we're almost certainly impacting the JVM's optimization and inlining budgets.

OpenJDK (and probably the other VMs too) has various limits on how far it will go to optimize code. A large number of these limits are based on the bytecoded size of the target methods. Methods that get too big won't inline, and sometimes won't compile. Methods that inline a lot of code might not get inlined into other methods. Methods that inline one path and eat up too much budget might push out more important calls later on. The only way around this is to reduce bytecode size, which is where invokedynamic comes in.

As of OpenJDK 7u2, MethodHandle logic is not included when calculating inlining budgets. In other words, if you push all the Java dispatch logic or multimethod dispatch logic or var lookup into mostly MethodHandles, you're getting that logic *for free*. That has had a tremendous impact on JRuby performance; I had previous versions of our compiler that did indeed infer static target methods from the interpreter, but they were often *slower* than call site caching solely because the code was considerably larger. With invokedynamic, a call is a call is a call, and the intervening plumbing is not counted against you.

Now, what about negative impacts to Clojure itself...

#0 is a red herring. JRuby supports Java 5, 6, and 7 with only a few hundred lines of changes in the compiler. Basically, the compiler has abstract interfaces for doing things like constant lookup, literal loading, and dispatch that we simply reimplement to use invokedynamic (extending the old non-indy logic for non-indified paths). In order to compile our uses of invokedynamic, we use Rémi Forax's JSR-292 backport, which includes a "mock" jar with all the invokedynamic APIs stubbed out. In our release, we just leave that library out, reflectively load the invokedynamic-based compiler impls, and we're off to the races.

#1 would be fair if the Oracle Java 7u2 early-access drops did not already include the optimizations that gave JRuby those awesome numbers. The biggest of those optimizations was making SwitchPoint free, but also important are the inlining discounting and MutableCallSite improvements. The perf you see for JRuby there can apply to any indirected behavior in Clojure, with the same perf benefits as of 7u2.

For #2, to address the apparent vagueness in my blog post...the big perf gain was largely from using SwitchPoint to invalidate constants rather than pinging a global serial number. Again, indirection folds away if you can shove it into MethodHandles. And it's pretty easy to do it.

#3 is just plain FUD. Oracle has committed to making invokedynamic work well for Java too. The current thinking is that "lambda", the support for closures in Java 7, will use invokedynamic under the covers to implement "function-like" constructs. Oracle has also committed to Nashorn, a fully invokedynamic-based JavaScript implementation, which has many of the same challenges as languages like Ruby or Python. I talked with Adam Messinger at Oracle, who explained to me that Oracle chose JavaScript in part because it's so far away from Java...as I put it (and he agreed) it's going to "keep Oracle honest" about optimizing for non-Java languages. Invokedynamic is driving the future of the JVM, and Oracle knows it all too well.

As for #4...well, all good things take a little effort :) I think the effort required is far lower than you suspect, though.

14 Oct 2011 2:40pm GMT

Hi everyone,

Rails 3.1.1 has been released. This release requires at least sass-rails 3.1.4

CHANGES

ActionMailer

• No changes

ActionPack

• stylesheetlinktag('/stylesheets/application') and similar helpers doesn't throw Sprockets::FileOutsidePaths exception anymore [Santiago Pastorino]

• Ensure defaultassethost_protocol is respected, closes #2980. [José Valim]

  Changing rake db:schema:dump to run :environment as well as :load_config, as running :load_config alone will lead to the dumper being run without including extensions such as those included in foreigner and spatial_adapter.

  This reverses a change made here: https://github.com/rails/rails/commit/5df72a238e9fcb18daf6ab6e6dc9051c9106d7bb#L0L324

  I'm assuming here that :load_config needs to be invoked separately from :environment, as it is elsewhere in the file for db operations, if not the alternative is to go back to "task :dump => :environment do".

  [Ben Woosley]

• Update to rack-cache 1.1.

  Versions prior to 1.1 delete the If-Modified-Since and If-Not-Modified headers when config.actioncontroller.performcaching is true. This has two problems:

  • unexpected inconsistent behaviour between development & production environments
  • breaks applications that use of these headers

  [Brendan Ribera]

• Ensure that enhancements to assets:precompile task are only run once [Sam Pohlenz]

• TestCase should respect the view_assigns API instead of pulling variables on its own. [José Valim]

• javascriptpath and stylesheetpath now refer to /assets if asset pipelining is on. [Santiago Pastorino]

• button_to support form option. Now you're able to pass for example 'data-type' => 'json'. [ihower]

• imagepath and imagetag should use /assets if asset pipelining is turned on. Closes #3126 [Santiago Pastorino and christos]

• Avoid use of existing precompiled assets during rake assets:precompile run. Closes #3119 [Guillermo Iguaran]

• Copy assets to nondigested filenames too [Santiago Pastorino]

• Give precedence to config.digest = false over the existence of manifest.yml asset digests [christos]

• escape options for the stylesheetlinktag method [Alexey Vakhov]

• Re-launch assets:precompile task using (Rake.)ruby instead of Kernel.exec so it works on Windows [cablegram]

• env var passed to process shouldn't be modified in process method. [Santiago Pastorino]

• rake assets:precompile loads the application but does not initialize it.

  To the app developer, this means configuration add in config/initializers/* will not be executed.

  Plugins developers need to special case their initializers that are meant to be run in the assets group by adding :group => :assets. [José Valim]

• Sprockets uses config.assets.prefix for asset_path [asee]

• FileStore keyfilepath properly limit filenames to 255 characters. [phuibonhoa]

• Fix Hash#toquery edge case with htmlsafe strings. [brainopia]

• Allow asset tag helper methods to accept :digest => false option in order to completely avoid the digest generation. Useful for linking assets from static html files or from emails when the user could probably look at an older html email with an older asset. [Santiago Pastorino]

• Don't mount Sprockets server at config.assets.prefix if config.assets.compile is false. [Mark J. Titorenko]

• Set relative url root in assets when controller isn't available for Sprockets (eg. Sass files using asset_path). Fixes #2435 [Guillermo Iguaran]

• Fix basic auth credential generation to not make newlines. GH #2882

• Fixed the behavior of asset pipeline when config.assets.digest and config.assets.compile are false and requested asset isn't precompiled. Before the requested asset were compiled anyway ignoring that the config.assets.compile flag is false. [Guillermo Iguaran]

• CookieJar is now Enumerable. Fixes #2795

• Fixed AssetNotPrecompiled error raised when rake assets:precompile is compiling certain .erb files. See GH #2763 #2765 #2805 [Guillermo Iguaran]

• Manifest is correctly placed in assets path when default assets prefix is changed. Fixes #2776 [Guillermo Iguaran]

• Fixed stylesheetlinktag and javascriptincludetag to respect additional options passed by the users when debug is on. [Guillermo Iguaran]

• Fix ActiveRecord#exists? when passsed a nil value

• Fix assertselectemail to work on multipart and non-multipart emails as the method stopped working correctly in Rails 3.x due to changes in the new mail gem.

ActiveModel

• Remove hard dependency on bcrypt-ruby to avoid make ActiveModel dependent on a binary library. You must add the gem explicitly to your Gemfile if you want use ActiveModel::SecurePassword:

  gem 'bcrypt-ruby', '~> 3.0.0'

  See GH #2687. [Guillermo Iguaran]

ActiveRecord

• Add deprecation for the preload_associations method. Fixes #3022.

  [Jon Leighton]

• Don't require a DB connection when loading a model that uses setprimarykey. GH #2807.

  [Jon Leighton]

• Fix using select() with a habtm association, e.g. Person.friends.select(:name). GH #3030 and #2923.

  [Hendy Tanata]

• Fix belongs_to polymorphic with custom primary key on target. GH #3104.

  [Jon Leighton]

• CollectionProxy#replace should change the DB records rather than just mutating the array. Fixes #3020.

  [Jon Leighton]

• LRU cache in mysql and sqlite are now per-process caches.

  • lib/activerecord/connectionadapters/mysql_adapter.rb: LRU cache keys are per process id.
  • lib/activerecord/connectionadapters/sqlite_adapter.rb: ditto

• Database adapters use a statement pool for limiting the number of open prepared statments on the database. The limit defaults to 1000, but can be adjusted in your database config by changing 'statement_limit'.

• Fix clash between using 'preload', 'joins' or 'eager_load' in a default scope and including the default scoped model in a nested through association. (GH #2834.) [Jon Leighton]

• Ensure we are not comparing a string with a symbol in HasManyAssociation#inverseupdatescounter_cache?. Fixes GH #2755, where a counter cache could be decremented twice as far as it was supposed to be.

  [Jon Leighton]

• Don't send any queries to the database when the foreign key of a belongs_to is nil. Fixes GH #2828. [Georg Friedrich]

• Fixed findin_batches method to not include order from defaultscope. See GH #2832 [Arun Agrawal]

• Don't compute table name for abstract classes. Fixes problem with setting the primary key in an abstract class. See GH #2791. [Akira Matsuda]

• Psych errors with poor yaml formatting are proxied. Fixes GH #2645 and GH #2731

• Use the LIMIT word with the methods #last and #first. Fixes GH #2783 [Damien Mathieu]

ActiveResource

• No changes

ActiveSupport

• ruby193: String#prepend is also unsafe [Akira Matsuda]

• Fix obviously breakage of Time.=== for Time subclasses [jeremyevans]

• Added fix so that file store does not raise an exception when cache dir does not exist yet. This can happen if a delete_matched is called before anything is saved in the cache. [Philippe Huibonhoa]

• Fixed performance issue where TimeZone lookups would require tzinfo each time [Tim Lucas]

• ActiveSupport::OrderedHash is now marked as extractable when using Array#extract_options! [Prem Sichanugrist]

Railties

• Add jquery-rails to Gemfile of plugins, test/dummy app needs it. Closes #3091. [Santiago Pastorino]

• rake assets:precompile loads the application but does not initialize it.

  To the app developer, this means configuration add in config/initializers/* will not be executed.

  Plugins developers need to special case their initializers that are meant to be run in the assets group by adding :group => :assets.

SHA-1

• 9337cff7772da034b0b34b73b85cf249f1a70f52 actionmailer-3.1.1.gem
• 7bb1b8d096a6ff1ff46dcfb778bf86a5daca1b0d actionpack-3.1.1.gem
• d5dc71e1a9a0e20d819f4dff27ff0697e99a7f64 activemodel-3.1.1.gem
• 7245632cb3b38612628304c1e244855d0053f7be activerecord-3.1.1.gem
• 6d09800202c2747e84249b8646f0fd480ed4924f activeresource-3.1.1.gem
• 66df2fd144aab22f52819fd489e33a976d68a46b activesupport-3.1.1.gem
• 6a35a49948bbd9f461839a1a271def90b23a851a rails-3.1.1.gem
• 6979ef891bd03fb639b979af9fdc56781f9358d9 railties-3.1.1.gem

You can find an exhaustive list of changes on github. Along with the closed issues marked for v3.1.1.

Thanks to everyone!

07 Oct 2011 5:26pm GMT

If you see this message in your RSS reader, please correct your RSS link to the following URL: http://fosdem.org/rss.xml.

26 Jul 2008 5:55am GMT

These pages have been archived.
For information about the latest FOSDEM edition please check this url: http://fosdem.org

25 Jul 2008 4:43pm GMT

Two weeks after FOSDEM and we are proud to publish most of the slides and videos from this year's edition.

All of the material from the Lightning Talks has been put online. We are still missing some slides and videos from the Main Tracks but we are working hard on getting those completed too.

We would like to thank our mirrors: HEAnet (IE) and Unixheads (US) for hosting our videos, and NamurLUG for quick recording and encoding.

The videos from the Janson room were live-streamed during the event and are also online on the Linux Magazin site.

We are having some synchronisation issues with Belnet (BE) at the moment. We're working to sort these out.

09 Mar 2008 3:12pm GMT