So, last December I got an email from the email hosting provider for Nidelven IT that the email server would be taken down in 6 months time.

I didn't like the timing, as I was in court process, the third one in 7 years about my kids, but understand that things are expensive to maintain, a potential security hole etc. when they age.

So I wrote a little script that pretty much would do what was necessary.

Then after some thinking, it struck me that this is something others would need to do, and it wasn't completely straightforward. So I decided I could model a script based on the process I was using.

Here's the script:

https://github.com/morphex/misc/blob/master/migrate-imap.py

I found the imapsync script:

https://github.com/imapsync/imapsync

Which can be used to do the heavy lifting. I read the license file for that project, and although I'm not a lawyer, it seems straightforward enough that I can use it for my needs. It might've been a better choice to use a known license, but whatever, it is very minimalist and straightforward in its wording.

The script just lists folders for now, then I guess it could build a shell script file which calls imapsync, and that can be inspected and executed.

I was scratching my head a bit as I was writing the script, as the print() statement printed parentheses, then I saw I was running it with python 2 and not 3.

Other than that, I wasn't able to figure out a way to parse command line options for the script using just getopt, am I missing something or is there another module?

[Update on the 13th of May]

The script is now more or less complete. Gilles also responded to an email, saying imapsync imapsync also has --justfolderlists.

I couldn't quite understand the getopt module, haven't used it much before.

13 May 2021 7:25pm GMT

The PyTorch sigmoid function is an element-wise operation that squishes any real number into a range between 0 and 1. This is a very common activation function to use as the last layer of binary classifiers (including logistic regression) because it lets you treat model predictions like probabilities that their outputs are true, i.e. p(y == 1).

Mathematically, the function is 1 / (1 + np.exp(-x)). And plotting it creates a well-known curve:

y = sigmoid(x) for x in [-10, 10]

Similar to other activation functions like softmax, there are two patterns for applying the sigmoid activation function in PyTorch. Which one you choose will depend more on your style preferences than anything else.

Sigmoid Function

The first way to apply the sigmoid operation is with the torch.sigmoid() function:

import torch

torch.manual_seed(1)

x = torch.randn((3, 3, 3))
y = torch.sigmoid(x)

y.min(), y.max()

# Expected output
# (tensor(0.1667), tensor(0.9364))

There are a couple things to point out about this function. First, the operation works element-wise so x can have any input dimension you want - the output dimension will be the same. Second, torch.sigmoid() is functionally the same as torch.nn.functional.sigmoid(), which was more common in older versions of PyTorch, but has been deprecated since the 1.0 release.

Sigmoid Class

The second pattern you will sometimes see is instantiating the torch.nn.Sigmoid() class and then using the callable object. This is more common in PyTorch model classes:

class MyModel(torch.nn.Module):
    def __init__(self, input_dim):
        super().__init__()
        self.linear = torch.nn.Linear(input_dim, 1)
        self.activation = torch.nn.Sigmoid()

    def forward(self, x):
        x = self.linear(x)
        return self.activation(x)


torch.manual_seed(1)

model = MyModel(4)
x = torch.randn((10, 4))
y = model(x)

y.min(), y.max()

# Expected output
# (tensor(0.2182, grad_fn=<MinBackward1>),
#  tensor(0.5587, grad_fn=<MaxBackward1>))

The output of this snippet shows that PyTorch is keeping track of gradients for us. But the function and class-based approaches are equivalent - and you're free to call torch.sigmoid() inside the forward() method if you prefer. If you're skeptical, then prove it to yourself by copying this snippet and replacing self.activation(x) with torch.sigmoid(x).

And that's all there is to know about the PyTorch sigmoid operation. Happy coding.

The post How to Use the PyTorch Sigmoid Operation appeared first on jbencook.

13 May 2021 6:11pm GMT

We're excited to announce that Pylance, our fast and feature-rich language support for Python in Visual Studio Code, is now officially out of preview and has reached its first stable release.

Join us on Learn TV today (May 13, 2021) for a livestream event at 12:00pm PST to help us celebrate the launch and for an overview of the most performant and user-friendly editing experience for Python in VS Code ever.

Earlier this week, we announced that as of the May release of the Python extension, Pylance is also now the default Python language server in Visual Studio Code. Pylance is also now included in the Python core extension's bundle as an optional dependency, meaning that we'll automatically install it for you, but you have the option to use another language server of your choice.

Background

Last June, we released a public preview of a new Python language server, Pylance, as a commitment to providing robust and performant language features to Python users in Visual Studio Code. Since Pylance's public preview, we've received both praise and constructive feedback from the community which we have used to guide our work over the last eleven months.

Today, we are pleased to announce that the core Pylance feature set has reached its stable release in Visual Studio Code!

Overview of Pylance features

Since our initial release, we've continued to improve the performance and experience of several key features (e.g., completions, auto-imports, function signature help, docstrings), and added new features (e.g., extract variable and extract method code actions, contextual highlighting).

If you aren't already a Pylance user, here's a quick overview of some of the rich features that the language server provides for Python development in Visual Studio Code.

Auto-import completions

With auto-imports, you can get smart import suggestions in your completions list for installed and standard library modules. Upon selection, the appropriate import will be added to the top of your file, so you don't have to scroll to the top and add it in manually. Auto-import completions also include a clear preview of the import statement that will be added to your file in the completion tooltip so you know exactly what symbol you are importing. If you have already imported other submodules or functions from that module, Pylance will amend the existing statement by adding the new symbol alphabetically, helping to keep your imports organized.

Function signature help with parameter and return type information

Pylance provides a useful function signature tooltip which includes type information to help you properly invoke functions without needing to look at external documentation. Function signature help also handles functions with multiple overrides. You can navigate between signatures easily while Pylance bolds the appropriate active parameter.

Code navigation

Pylance also provides performant code navigation functionality, like go to definition and go to declaration, to allow you to quickly jump to user-defined code, source files (e.g. .py) or type stub files (.pyi).

Refactoring code actions - extract variable and extract method

Pylance also provides you to use refactoring code actions which allows you to highlight lines of code and automatically refactor and extract them into new methods or variables at the click of a button!

Extract Variable

Extract Method

Semantic colorization

Pylance's semantic colorization helps improve the readability and understanding of your code. If you are unfamiliar with semantic colorization, it's an extension on syntax highlighting. Pylance generates semantic tokens which are used by themes to apply colors based on the semantic meaning of symbols (e.g., variables, functions, modules all have distinct colors applied to them).

To see this new feature in action, you will need to apply a theme that supports semantic color. Some great themes to try out semantic colorization are the built-in Dark+ theme or One Dark Pro used below.

Contextual document highlighting

Pylance supports contextual highlighting, which helps you quickly identify where symbols are used in a particular file.

Type checking support via Pyright

Because Pylance leverages our open-source type checker called Pyright, the language server also comes with a built-in, static type checker if you are interested in writing typed Python. By default, no type checking functionality is enabled, and you'll only see diagnostics (errors and warnings) in your code when you have syntactically invalid Python. However, you can enable basic or strict type checking by configuring python.analysis.typeCheckingMode. This setting uses Pyright's type checking to apply either a basic or comprehensive set of rules over your codebase, respectively. The diagnostics produced from this mode can help improve the quality of your code and help you find edge cases more easily.

IntelliCode compatibility

Pylance is also fully compatible with IntelliCode, which brings recommended completions for your specific code context to the top of your completions list to further enhance your productivity!

Feedback

If you have any questions, comments, or feedback on your experience, please reach out to us on GitHub.

The post The Pylance language server has reached stable! appeared first on Python.

13 May 2021 5:59pm GMT

In McCarthy's early papers on Lisp, he notes that he needs a modified version of subst which needs to be aware of quoted expressions (and avoid substituting within them). He would also need a subst that was aware of lambda expressions. It would have to avoid substituting within the lambda if the name substituted matches one of the bound variables. To be useful for evaluation, it will have to deal with accidental variable capture when substituting within a lambda.

The root problem is that expressions are actually structured objects, but we are working with the list representation of those objects. Instead of substituting by operating on objects, we substitute on the list representation. We have to arrange for the syntactic substitution on the list representation to preserve the semantics of substitution on the objects they represent.

In the substitution model, we take a symbolic expression and replace some of the atoms in the expression with other expressions. We first need a way to discriminate between the different kinds of expressions. An expression is either an atomic symbol, or a list of expressions called an application. There are no other kinds of expressions.

(defun expression-dispatch (expression if-symbol if-application)
  (cond ((symbolp expression) (funcall if-symbol expression))
        ((consp expression)   (funcall if-application expression))
        (t (error "~s is not an expression." expression))))

Substitution is straightforward:

(defun xsubst (table expression)
  (expression-dispatch expression
    (lambda (symbol)
      (funcall table symbol #'identity (constantly symbol)))

    (lambda (subexpressions)
      (map 'list (lambda (subexpression) (xsubst table subexpression)) subexpressions))))

* (let ((table (table/extend (table/empty) 'x '(* a 42))))
    (xsubst table '(+ x y)))  
(+ (* A 42) Y)

We need a table of multiple substitutions so that we can substitute in parallel:

* (let ((table (table/extend
                 (table/extend (table/empty) 'x 'y)
                 'y 'x)))
    (xsubst table '(+ x y)))
(+ Y X)

So far, so good. Let's add lambda expressions. First, we need to add a new expression kind:

(defun expression-dispatch (expression if-symbol if-lambda if-application) 
  (cond ((symbolp expression) (funcall if-symbol expression))
        ((consp expression)
         (cond ((eq (car expression) 'lambda)
                (funcall if-lambda (cadr expression) (caddr expression)))
               (t (funcall if-application expression))))
        (t (error "~s is not an expression." expression))))

Substitution within a lambda expression is a bit tricky. First, you don't want to substitute a symbol if it is one of the bound variables of the lambda expression. Second, substituting a symbol may introduce more symbols. We don't want the new symbols to be accidentally captured by the bound variables in the lambda. If any new symbol has the same name as a bound variable, we have to rename the bound variable (and all its occurrances) to a fresh name so that it doesn't capture the new symbol being introduced. We'll need a helper function

(defun free-variables (expression)
   (expression-dispatch expression
     (lambda (symbol) (list symbol))
     (lambda (bound-variables body)
       (set-difference (free-variables body) bound-variables))
     (lambda (subexpressions)
       (fold-left #'union '() (map 'list #'free-variables subexpressions)))))

Now when we substitute within a lambda, we first find each free variable in the lambda, look it up in the substitution table, and collect the free variables of the substituted value:

(map 'list (lambda (var)
             (funcall table var #'free-variables (constantly '())))
      (free-variables expression))

This gives us the new free variables for each substitution. The union of all of these is the set of all the new free variables

(fold-left #'union '()
           (map 'list (lambda (var)
                        (funcall table var #'free-variables (constantly '())))
                 (free-variables expression)))

We have to rename the bound variables that are in this set:

 
(intersection 
  bound-variables
  (fold-left #'union '()
             (map 'list (lambda (var)
                          (funcall table var #'free-variables (constantly '())))
                  (free-variables expression))))

So we make a little table for renaming:

(defun make-alpha-table (variables)
  (fold-left (lambda (table variable)
               (table/extend table variable (gensym (symbol-name variable))))
             (table/empty)
             variables))

(let ((alpha-table
       (make-alpha-table
        (intersection 
          bound-variables
          (fold-left #'union '()
                     (map 'list (lambda (var)
                                  (funcall table var #'free-variables (constantly '())))
                          (free-variables expression)))))))
  …)

We rename the bound variables as necessary:

  (make-lambda
    (map 'list (lambda (symbol)
                 (funcall alpha-table symbol #'identity (constantly symbol)))
         bound-variables)
    …)

Finally, we redact the bound variables from the substitution table and append the alpha-table to make the substitutions we need for the lambda body

  (make-lambda
   (map 'list (lambda (symbol)
                (funcall alpha-table symbol #'identity (constantly symbol)))
        bound-variables)
   (xsubst (table/append alpha-table (table/redact* table bound-variables))
           body))))

The entire definition of xsubst is now this:

(defun xsubst (table expression)
  (expression-dispatch expression
    (lambda (symbol)
      (funcall table symbol #'identity (constantly symbol)))

    (lambda (bound-variables body)
      (let ((alpha-table
             (make-alpha-table
              (intersection
               bound-variables
               (fold-left #'union '()
                          (map 'list (lambda (var)
                                       (funcall table var
                                                #'free-variables
                                                (constantly '())))
                               (set-difference (free-variables body) bound-variables)))))))
        (make-lambda
         (map 'list (lambda (symbol)
                      (funcall alpha-table symbol #'identity (constantly symbol)))
              bound-variables)
         (xsubst (table/append alpha-table (table/redact* table bound-variables))
                 body))))

    (lambda (subexpressions)
      (make-application
       (map 'list (lambda (subexpression)
                    (xsubst table subexpression))
            subexpressions)))))

This is certainly more complicated than simple substitution, but we can see it does the right thing here:

* (xsubst (table/extend (table/empty) 'x '(* a y)) '(lambda (y) (+ x y)))
(LAMBDA (#:Y234) (+ (* A Y) #:Y234))

It should be obvious how to add quoted forms. This would require adding a new kind of expression to expression-dispatch and a new handling clause in xsubst that avoids substitution.

I'm not completely happy with how we've added lambda expressions to the expression syntax. Using the symbol lambda as a syntactic marker for lambda expressions causes problems if we also want to use that symbol as an argument or variable. Initially, it seems reasonable to be able to name an argument "lambda". Within the body of the function, references to the variable lambda would refer to that argument. But what about references in the operator position? By defining lambda expressions as three element lists beginning with the symbol lambda we've made it ambiguous with two-argument applications whose operator is the variable lambda. We have to resolve this ambiguity. The current behavior is that we always interpret the symbol lambda as a syntactic marker so you simply cannot use a variable named lambda as a function.

10 May 2021 11:42am GMT

Another hectic month gone by. Getting Eternia: Pet Whisperer done and published on Steam in such a short time was no small feat. There were a few hurdles along the way as well, especially when it came to getting it ready for Steam, but overall I'm really glad we managed to get it out. Having a bona fide released title is amazing!

Most of the trouble with getting it put on Steam was the manual review process they have. It turns out, real people actually check your builds before you're allowed to sell the game. And not only do they check whether it launches, they also do other tests like whether gamepad support works, whether captions are included, etc. It's a lot more extensive than I had expected, which is really nice!

Unfortunately I was also confused by a couple of options, and misunderstood others, so I had to go through the review a bunch of times, which caused a bunch of stress for me and ultimately delayed the release of Eternia by two days. Welp! At least now I know for the future that I'll have to start with the Steam review process well in advance of the actual release. With Eternia everything was so back to back that there really wasn't any more time for that than we had.

The days since the release haven't been much less hectic either. There were a few bugs that had to be ironed out that prevented people from playing the game. Most notably:

• People with surround headphones could not play as those headphones announce themselves as 7.1 surround systems, which I didn't expect anyone would have.

• Some bad versions of shared libraries snuck themselves into the release builds on Linux.

• The save menu was bugged due to a regression in the UI library

• Avast antivirus causes random Windows exceptions in the game (this is not fixed and I don't know how to fix it).

There were also some minor content fixes for typos and such along the way. In any case, the automated crash report system I instated for Kandria helped a lot as it told me when people were having issues. Still, it was heartbreaking every time I got a report, as knowing people can't even play the game and are likely very frustrated or disappointed stings a lot.

I really hope the Kandria release will be more smooth and less prone to issues now that we have a better handle on the process. Still, the amount of issues that are only uncovered by having people with various PC setups try to run your game is... well, it's not great. One of the many perils of the PC platform, I suppose.

It hasn't yet been a full week so I don't really want to go into the statistics of the Eternia release, but I'll be sure to talk about that stuff in the next weekly update!

Anyway, aside from supporting the Eternia review and taking care of the marketing around that, I also did some planning for Kandria. I asked ProHelvetia, and it turns out the submission deadline for the grant is going to be 1st of September of this year. This gives us roughly four months to graft a tutorial onto the vertical slice, and polish the fuck out of it to make it look as good as possible.

A first step in that is to iron out egregious bugs that were reported from people playing the public vertical slice, and making a new trailer that shows off all the new content we made. I've now started working on both of those things and we should be able to finish the new trailer in the next week.

I initially also wanted to put out a patch for the vertical slice already, but while we did fix a number of bugs and there's a few improvements, I'd rather wait until the rest of the other known and egregious bugs are ironed out at least. Regardless though, if you're on the Discord or the mailing list, we'll let you know when that patch hits!

Composers

In the last monthly I announced that you'll get to hear the pieces the three finalists put together, so here they are! I hope you enjoy listening to them!

• Jacob Lincke: track

• João Luís: track

• Mikel Dale: track

While all three of them did a fantastic job and the decision was a hard one, we ultimately went with...

Mikel

Hey everyone, my name is Mikel - nice to meet you! I'll be working as Kandria's composer from now on, which means I'll bang some notes on the piano and hope they sound good.

In my first week as part of the team, I've been working side by side Shinmera on the game's new trailer. Taking inspiration from OSTs like Astral Chain and Octopath Traveller (as well as using our secret weapon, Julie Elven), we're cooking up something real good!

If you have any suggestions, any soundtracks you'd like me to check out, or any weird instruments/sounds you demand I incorporate in the music, please let me know on the Discord! It'd be great to have some feedback from you too :)

Ah yes, time for shameless plug: if you'd like to listen to some other games I'm working on, feel free to peruse Underdesert (rogue-lite dungeon FPS game), Waves of Steel (naval battle!), Cryptonom (if you like monster RPGs), or Heredity (good ol' fantasy ARPG).

You can also check out my website for more deets.

Look forward to showing you what's next!

Tim

The majority of this month has been occupied by our jam game Eternia. It was actually my first game jam, and I've really enjoyed it - I'm not exaggerating when I say it's been some of the most fun I've had in game writing so far. We agreed on the broad concept early on, and then I was given a lot of creative freedom on the story and character dialogue. I'm really pleased with how much content I produced (around 20k words) in such a short time frame, which was definitely helped by the quick-iteration dialogue system, and forced by the necessary constraints.

The game is incredibly non-linear, so the challenge was writing things in a modular way, such that they made sense whichever direction you came at them from. With some crafty narrative design and plotting, I think it holds together remarkably well; players would be surprised how little, if any, conditional branching there is behind the scenes. With such a tight dev time, it was important the game wasn't a spiderweb of quest logic, for both my own sanity, and for playtesting.

Thankfully I didn't have to do too much research either, since my wife Lesleyann and I have rescued animals from shelters in the past (especially rats!). Les used to be an animal behaviourist too, so I've gained a base awareness of things like calming signals in dogs, by osmosis over the years; she was also a big help when I needed a steer on certain things. There really wasn't time for anything more than a cursory amount of research, so this was definitely a plus for this idea when Nick first suggested it.

Towards the end of the month I jumped back on Kandria, fleshing out the lore for the other world factions ready for Fred to concept. This was really beneficial to give me a feel for the wider world, and allowed some tweaking of the plot. As we progress with the content for the rest of this year, these factions will get fleshed out further with their own unique characters and quest lines.

I've also worked with Nick on the script for the new trailer, and helped in sourcing a voice actor. I've also gotten back up to speed on the vertical slice content, playtesting the quests to make sure they're still working correctly after the restructuring of the quest system. With fresh eyes, this has also let me tweak dialogue as I went.

What's next

Here's a rough roadmap for the rest of the year:

• Make the combat more flashy

• Finish a new trailer

• Design and outline the factions for the rest of the game

• Develop the soundscape for Kandria and start working on the tracks for region 1

• Add a music system that can do layering and timed transitions

• Build and add the tutorial sequence to the beginning of the game

• Polish the ProHelvetia submission material

• Polish and revise the combat design

• Explore platforming items and mechanics

• Practise platforming level design

• Start work on the horizontal slice

Of course the size of the items varies a lot, but hopefully we should be able to begin work on the horizontal slice by November. If we extrapolate the time it took for the vertical slice and cut some content, this should leave us with enough time to finish by early 2023... provided we don't run outta money.

Anyway, that's why the grant is the primary target for now. Though we were also accepted by ProHelvetia for GDC Online and the SwissNex US Game Industry Week, so that'll give us some more opportunities to look for a publisher. Having a new trailer for that should definitely help a lot!

Well, see you next time then! Remember that there's a mailing list with weekly updates, a discord for the community, and my twitter with short videos and images on the development! Oh, and do check out Eternia if you haven't yet. We've gotten some really nice reviews for it, which has been a joy to see!

09 May 2021 10:31am GMT

One of the nice new features of FreeBSD 13 is OpenZFS 2.0. OpenZFS 2.0 comes with zstd compression support. Zstd compression can have compression ratios similar to gzip with less CPU usage.

For my backups, I copy the most import data - /etc/, /home, … - first locally to a ZFS dataset. This data gets synced to a backup server. This local ZFS dataset was compressed with gzip, after upgrading the zroot pool and setting zstd as the compress method. FreeBSD failed to boot with the error message:

ZFS: unsupported feature: org.freebsd:zstd
ZFS: pool zroot is not supported
gptzfsboot: failed to mount default pool zroot

As this might help people with the same issue, I decided to create a blog post about it.

Update the boot loader

We need to update the boot loader with the newer version that has zstd compression support.

Boot from cdrom

Boot your system from FreeBSD 13 installation cdrom/dvd or USB stick and choose <Live CD>. Log in as the root account, the root account doesn't have a password on the "Live CD".

Enable ssh

I prefer to update the boot loader over ssh.

I followed this blog post to enable sshd on the live cd: https://www.krisna.net/2018/09/ssh-access-to-freebsd-live-cd-manual.html

# ifconfig
# ifconfig <net_interface> xxx.xxx.xxx.xxx up
#       mkdir /tmp/etc
#       mount_unionfs /tmp/etc/ /etc
#       passwd root
#       cd /etc/ssh/
#       vi sshd_config
#       /etc/rc.d/sshd onestart

Log on to the system remotely.

$ ssh root@xxx.xxx.xxx.xxx

Update the bootloader

The commands to install the bootloader comes from the FreeBSD wiki.

https://wiki.freebsd.org/RootOnZFS/GPTZFSBoot

The wiki page page above describes who install FreeBSD on ZFS root pool. This was very useful before the FreeBSD installer had native ZFS support.

List your partitions to get your boot device name and slice number. The example below is on FreeBSD virtual machine, the device name is vtb0 and the slice number is 1. On a physical FreeBSD system, the device name is probably ada0.

root@:~ # gpart show
=>       40  419430320  vtbd0  GPT  (200G)
         40       1024      1  freebsd-boot  (512K)
       1064        984         - free -  (492K)
       2048    8388608      2  freebsd-swap  (4.0G)
    8390656  411037696      3  freebsd-zfs  (196G)
  419428352       2008         - free -  (1.0M)

=>     33  2335913  cd0  MBR  (4.5G)
       33  2335913       - free -  (4.5G)

=>     33  2335913  iso9660/13_0_RELEASE_AMD64_DVD  MBR  (4.5G)
       33  2335913                                  - free -  (4.5G)

root@:~ # 

I use a legacy BIOS on my system. On a system with a legacy BIOS, you can use the following command to update the bootloader.

root@:~ # gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 vtbd0
partcode written to vtbd0p1
bootcode written to vtbd0
root@:~ # 

To update the bootloader on a UEFI system.

# gpart bootcode -p /boot/boot1.efi -i1 ada0

Should to the trick.

Reboot your FreeBSD 13 system and enjoy zstd compression.

root@:~ # sync
root@:~ # reboot

Have fun!

Links

• https://www.krisna.net/2018/09/ssh-access-to-freebsd-live-cd-manual.html
• https://wiki.freebsd.org/RootOnZFS/GPTZFSBoot

09 May 2021 7:04am GMT

I published the following diary on isc.sans.edu: "Alternative Ways To Perform Basic Tasks":

I like to spot techniques used by malware developers to perform basic tasks. We know the LOLBins that are pre-installed tools used to perform malicious activities. Many LOLBins are used, for example, to download some content from the Internet. Some tools are so powerful that they can also be used to perform unexpected tasks. I found an interesting blog article describing how to use curl to copy files… [Read more]

The post [SANS ISC] Alternative Ways To Perform Basic Tasks appeared first on /dev/random.

06 May 2021 10:17am GMT

You don't need a data structure to make a lookup table. You can make a table just out of the lookup function. In this example, we start with a continuation passing style lookup function:

lookup (key if-found if-not-found)
  
    Invokes (funcall if-found value) if key is in the table,
    invokes (funcall if-not-found) otherwise.

An empty table just invokes the if-not-found continuation:

(defun table/empty ()
  (lambda (key if-found if-not-found)
    (declare (ignore key if-found))
    (funcall if-not-found)))

A table can be extended by wrapping it:

(defun table/extend (table key* value)
  (lambda (key if-found if-not-found)
    (if (eql key key*)
        (funcall if-found value)
        (funcall table key if-found if-not-found))))

So let's try it out:

(defvar *table-1* (table/extend 
                    (table/extend
                      (table/empty)
                      'foo 42)
                    'bar 69))

* (funcall *table-1* 'foo #'identity (constantly 'not-found))
42

* (funcall *table-1* 'quux #'identity (constantly 'not-found))
NOT-FOUND

You can also redact an entry from a table by wrapping the table:

(defun table/redact (table redacted)
  (lambda (key if-found if-not-found)
    (if (eql key redacted)
        (funcall if-not-found)
        (funcall table key if-found if-not-found))))

(defvar *table-2* (table/redact *table-1* 'foo))

* (funcall *table-2* 'foo #'identity (constantly 'not-found))
NOT-FOUND

Are there any advantages to implementing a table in this curious manner? Building a table by nesting a series of lookup steps leads to a linear lookup in linear space, so this kind of table should be more or less comparable to an alist for individual entries. Unlike a traditional table made with a data structure, you cannot enumerate the keys and values in the table. On the other hand, you gain the ability to map keys to values without having to enumerate the keys:

(defun table/bind-predicate (table predicate value)
  (lambda (key if-found if-not-found)
    (if (funcall predicate key)
        (funcall if-found value)
        (funcall table key if-found if-not-found))))

;;; bind all even numbers to the symbol 'EVEN
(defvar *table-3* 
  (table/bind-predicate *table-2* (lambda (n) (and (numberp n) (evenp n))) 'even))

* (funcall *table-3* 6 #'identity (constantly 'not-found))
EVEN

Or you can add a default value to an existing table:

(defun table/add-default (table default-value)
  (lambda (key if-found if-not-found)
    (declare (ignore if-not-found))
    (funcall table key
      if-found
      (lambda () (funcall if-found default-value)))))

(defvar *table-4* (table/add-default *table-3* 'default))

* (funcall *table-4* 'bar #'identity (constantly 'not-found))
69
    
* (funcall *table-4* 'xyzzy #'identity (constantly 'not-found))
DEFAULT

Perhaps the biggest disadvantage of this implementation is the difficulty in inspecting a table.

* *table-4*
#<CLOSURE (LAMBDA (KEY IF-FOUND IF-NOT-FOUND) :IN TABLE/ADD-DEFAULT) {1003CD641B}>

We can use the object inspector to peek inside the closure and maybe sleuth out what this table is made out of, but it isn't just an alist where we can print out the entries.

So far, we've defined a table as being a procedure with the (key if-found if-not-found) signature, but we can flip this around and say that any procedure with a (key if-found if-not-found) signature can be thought of as a table. For example, a regular expression matcher could be considered to be a table of strings (if that were a more useful model).

03 May 2021 2:35pm GMT

Bitterzoet (maar zoals goeie fondant chocolade dat is) verhaal van een jonge man die enkel beleefd en onzichtbaar wilt zijn. Over enkel "een goed functionerend radertje" proberen zijn, maar daar niet in slagen. Over prachtig kunnen zingen, maar dat niet willen. Over liefde, verlies en vluchten. Over Noorwegen, de Faeröer eilanden en een heel klein beetje over de Caraïben. En over de Cardigans.

Aanrader; 4,5 ipv 5 sterren, voornamelijk omdat ik "Max, Micha & het Tet-offensief" nog beter vond.

03 May 2021 1:15pm GMT

For 30 years, the space telescope Hubble has provided the most impressive images from the vast wide space. The telescope was developed by the US space agency NASA and its European counterpart ESA. Hubble started its journey into space on 24 April 1990. With the help of the space shuttle "Discovery" it was lifted into ... Read more

27 Apr 2020 3:47pm GMT

The Event Horizon Telescope (EHT) - a planet-scale array of 8 ground-based radio telescopes and part of an international collaboration - captured the first image of a black hole. On April 10th 2019, EHT researchers disclosed the first direct visual evidence of a supermassive black hole in the heart of the Galaxy Messier 87.

26 Apr 2019 2:32am GMT

Socrates couldn't have been more right when he said: "I know one thing, that I know nothing." Even with all of the advancements we, as a species, have made in this world, it's still nothing compared to countless of wonders waiting to be discovered in the vast universe. If you've recently developed an interest in ... Read more

04 Nov 2018 1:27pm GMT

From the Yet Another dept.:

To further its commitment to the Java Community Process (JCP), Oracle has submitted the first of two Java Specification Requests (JSRs) to update and revitalize the JCP.

10 Nov 2011 6:01am GMT

From the Fact Checking dept.:

ZDNET: Sometimes the sheer wrongness of what is posted on the web leaves us speechless. Especially when it's picked up and repeated as gospel by otherwise reputable sites like Engadget. "Google copied Oracle's Java code, pasted in a new license, and shipped it," they reported this morning.



Sorry, but that just isn't true.

10 Nov 2011 6:01am GMT

From the Grande dept.:

Oracle today announced the availability of Java Platform, Standard Edition 7 (Java SE 7), the first release of the Java platform under Oracle stewardship.

10 Nov 2011 6:01am GMT

This is mostly for my own reference, but since it might be useful to others:

By "Properly booting" I mean "Integrating into the boot system as well as Mac OS X does". The device should be visible from the boot picker menu and should be selectable as a startup disk. For this to happen the boot should be in HFS+ format and have the following files:

• /mach_kernel (can be empty)
• /System/Library/CoreServices/boot.efi (may be booted, if so should be a symlink to the actual bootloader)
• /System/Library/CoreServices/SystemVersion.plist which should look something like

  <xml version="1.0" encoding="UTF-8"?>
  <plist version="1.0">
  <dict>
          <key>ProductBuildVersion</key>
          <string></string>
          <key>ProductName</key>
          <string>Linux</string>
          <key>ProductVersion</key>
          <string>Fedora 16</string>
  </dict>
  </plist>
  

That's enough to get it to appear in the startup disk boot pane. Getting it in the boot picker requires it to be blessed. You probably also want a .VolumeIcon.icns in / in order to get an appropriate icon.

Now all I need is an aesthetically appealing boot loader.

comments

09 Nov 2011 10:06pm GMT

From time to time i want to answer on mails on Debian mailinglists that i am not subcribed to. To have proper reply-headers set, i usually copied the archive mbox from master.debian.org to my local machine.

Now i found a much nicer way.

apt-get install fuse afuse sshfs
adduser zobel fuse
mkdir ~/fuse/
afuse -o mount_template="sshfs %r:/ %m" -o unmount_template="fusermount -u -z %m" -o timeout=60 ~/fuse
mutt -f /home/zobel/fuse/master.debian.org/home/debian/lists/debian-user/debian-user.201111

09 Nov 2011 10:04pm GMT

I recently started getting mails from no-reply@joindiaspora.com. Usually, a mail from no-reply@whatever is enough to make me believe that the admins of said whatever are clueless regarding what e-mail means and how should it work. And in this case, it really amazes me - If I get an invite to Diaspora*, right, I should not pester a hypothetical sysadmin@joindiaspora.com to get me off his list, but I should be able to reply to the person mailing me - Maybe requesting extra details on what he is inviting me to, or allowing me to tell him why I'm not interested. But yes, Diaspora* has fallen to the ease of requiring me to join their network to be able to communicate back with the "friend" who invited me.

Some of the (three?) readers of this site might not be familiar with the Diaspora* project. It is a free reimplementation (as far as I know) of something similar to Facebook - Free not only in the sense that it runs free software, but also because it is federated - Your data will not belong to a specific company (that is, you are not the value object they sell and make money with), but you can choose and switch (or become) the provider for your information. A very interesting proposal, socially and technically.

I find that a gross violation of netiquette. I should be able to reply to the mail - Even if in this case it were to (and sorry - As you are spreading my name/mail, you will excuse me if I spread your name ;-) ) fernando.estrada.invite1501335@joindiaspora.com. Such an (fictional FWIW) address would allow for mail to reach back the submitter by the same medium it was sent, without allowing open spamming into the network.

Now, what prompted me to write this mail (just before adding no-reply@joindiaspora.com to my blacklist) is the message I got (in an ugly HTML-only mail which erroneously promised to be text/plain, sigh...) is that Fernando sent me as the inviting message, «So, at least are you going to give Diaspora a chance?»

The answer is: No..

But not because of being a fundamentalist. Right, I am among what many people qualify as Free Software zealots, but many of my choices (as this one is) is in no way related to the software's freeness. I use non-free Web services, as much as many of you do. Yes, I tend to use them less, rather than more (as the tendency goes).

But the main reason I don't use Twitter is the same reason I don't use Identi.ca, its free counterpart - And the reason I'm not interested in Facebook is the same reason I will not join Diaspora* - Because I lack time for yet another stream of activity, of information, of things to do and think about.

Yes, even if I care about you and I want to follow what's going on in your life: The best way to do it is to sit over a cup of coffee, or have some dinner, or to meet once a year in the most amazing conference ever. Or we can be part of distributed projects together, and we will really interact lots. Or you can write a blog! I do follow the blogs of many of my friends (plus several planets), even if they have fallen out of fashion - A blog post pulls me to read it as it is a unit of information, not too much depending on context (a problem when I read somebody's Twitter/Identica lines: You have to hunt a lot of conversations to understand what's going on), gives a true dump of (at least one aspect of) your state of (mind|life|work), and is a referenceable unit I can forward to other people, or quote if needed.

So, yes, I might look old-fashioned, clinging to the tools of the last-decade for my Social Web presence. I will never be a Social Media Expert. I accept it - But please, don't think it is a Stallmanesque posture from me. It is just that of a person who can lose too much time, and needs to get some work done in the meantime.

(oh, of course: Blog posts also don't have to make much sense or be logically complete. But at least they allow me to post a full argument!)

09 Nov 2011 5:55pm GMT

papupapu39 · http://identi.ca/url/56409795 #fosdem #freeknowledge #usamabinladen · about a day ago from web. Help · About · FAQ · TOS · Privacy · Source · Version · Contact. Identi.ca is a microblogging service brought to you by Status.net. ...

08 Nov 2011 12:28am GMT

FOSDEM (Free and Open Source Development European Meeting) is a European event centered around Free and Open Source software development. It is aimed at developers and all interested in the Free and Open Source news in the world. ...

05 Nov 2011 1:19am GMT

FOSDEM (Free and Open Source Development European Meeting) is a European event centered around Free and Open Source software development. It is aimed at developers and all interested in the Free and Open Source news in the ...

03 Nov 2011 3:45pm GMT

Ruby and Cocoa on OS X, the iPhone, and the Device That Shall Not Be Named

28 Oct 2011 8:00pm GMT

This was originally posted as a comment on @fogus's blog post "Why Clojure doesn't need invokedynamic, but it might be nice". I figured it's worth a top-level post here.

Ok, there's some good points here and a few misguided/misinformed positions. I'll try to cover everything.

First, I need to point out a key detail of invokedynamic that may have escaped notice: any case where you must bounce through a generic piece of code to do dispatch -- regardless of how fast that bounce may be -- prevents a whole slew of optimizations from happening. This might affect Java dispatch, if there's any argument-twiddling logic shared between call sites. It would definitely affect multimethods, which are using a hand-implemented PIC. Any case where there's intervening code between the call site and the target would benefit from invokedynamic, since invokedynamic could be used to plumb that logic and let it inline straight through. This is, indeed, the primary benefit of using invokedynamic: arbitrarily complex dispatch logic folds away allowing the dispatch to optimize as if it were direct.

Your point about inference in Java dispatch is a fair one...if Clojure is able to infer all cases, then there's no need to use invokedynamic at all. But unless Clojure is able to infer all cases, then you've got this little performance time bomb just waiting to happen. Tweak some code path and obscure the inference, and kablam, you're back on a slow reflective impl. Invokedynamic would provide a measure of consistency; the only unforeseen perf impact would be when the dispatch turns out to *actually* be polymorphic, in which case even a direct call wouldn't do much better.

For multimethods, the benefit should be clear: the MM selection logic would be mostly implemented using method handles and "leaf" logic, allowing hotspot to inline it everywhere it is used. That means for small-morphic MM call sites, all targets could potentially inline too. That's impossible without invokedynamic unless you generate every MM path immediately around the eventual call.

Now, on to defs and Var lookup. Depending on the cost of Var lookup, using a SwitchPoint-based invalidation plus invokedynamic could be a big win. In Java 7u2, SwitchPoint-based invalidation is essentially free until invalidated, and as you point out that's a rare case. There would essentially be *no* cost in indirecting through a var until that var changes...and then it would settle back into no cost until it changes again. Frequently-changing vars could gracefully degrade to a PIC.

It's also dangerous to understate the impact code size has on JVM optimization. The usual recommendation on the JVM is to move code into many small methods, possibly using call-through logic as in multimethods to reuse the same logic in many places. As I've mentioned, that defeats many optimizations, so the next approach is often to hand-inline logic everywhere it's used, to let the JVM have a more optimizable view of the system. But now we're stepping on our own feet...by adding more bytecode, we're almost certainly impacting the JVM's optimization and inlining budgets.

OpenJDK (and probably the other VMs too) has various limits on how far it will go to optimize code. A large number of these limits are based on the bytecoded size of the target methods. Methods that get too big won't inline, and sometimes won't compile. Methods that inline a lot of code might not get inlined into other methods. Methods that inline one path and eat up too much budget might push out more important calls later on. The only way around this is to reduce bytecode size, which is where invokedynamic comes in.

As of OpenJDK 7u2, MethodHandle logic is not included when calculating inlining budgets. In other words, if you push all the Java dispatch logic or multimethod dispatch logic or var lookup into mostly MethodHandles, you're getting that logic *for free*. That has had a tremendous impact on JRuby performance; I had previous versions of our compiler that did indeed infer static target methods from the interpreter, but they were often *slower* than call site caching solely because the code was considerably larger. With invokedynamic, a call is a call is a call, and the intervening plumbing is not counted against you.

Now, what about negative impacts to Clojure itself...

#0 is a red herring. JRuby supports Java 5, 6, and 7 with only a few hundred lines of changes in the compiler. Basically, the compiler has abstract interfaces for doing things like constant lookup, literal loading, and dispatch that we simply reimplement to use invokedynamic (extending the old non-indy logic for non-indified paths). In order to compile our uses of invokedynamic, we use Rémi Forax's JSR-292 backport, which includes a "mock" jar with all the invokedynamic APIs stubbed out. In our release, we just leave that library out, reflectively load the invokedynamic-based compiler impls, and we're off to the races.

#1 would be fair if the Oracle Java 7u2 early-access drops did not already include the optimizations that gave JRuby those awesome numbers. The biggest of those optimizations was making SwitchPoint free, but also important are the inlining discounting and MutableCallSite improvements. The perf you see for JRuby there can apply to any indirected behavior in Clojure, with the same perf benefits as of 7u2.

For #2, to address the apparent vagueness in my blog post...the big perf gain was largely from using SwitchPoint to invalidate constants rather than pinging a global serial number. Again, indirection folds away if you can shove it into MethodHandles. And it's pretty easy to do it.

#3 is just plain FUD. Oracle has committed to making invokedynamic work well for Java too. The current thinking is that "lambda", the support for closures in Java 7, will use invokedynamic under the covers to implement "function-like" constructs. Oracle has also committed to Nashorn, a fully invokedynamic-based JavaScript implementation, which has many of the same challenges as languages like Ruby or Python. I talked with Adam Messinger at Oracle, who explained to me that Oracle chose JavaScript in part because it's so far away from Java...as I put it (and he agreed) it's going to "keep Oracle honest" about optimizing for non-Java languages. Invokedynamic is driving the future of the JVM, and Oracle knows it all too well.

As for #4...well, all good things take a little effort :) I think the effort required is far lower than you suspect, though.

14 Oct 2011 2:40pm GMT

Hi everyone,

Rails 3.1.1 has been released. This release requires at least sass-rails 3.1.4

CHANGES

ActionMailer

• No changes

ActionPack

• stylesheetlinktag('/stylesheets/application') and similar helpers doesn't throw Sprockets::FileOutsidePaths exception anymore [Santiago Pastorino]

• Ensure defaultassethost_protocol is respected, closes #2980. [José Valim]

  Changing rake db:schema:dump to run :environment as well as :load_config, as running :load_config alone will lead to the dumper being run without including extensions such as those included in foreigner and spatial_adapter.

  This reverses a change made here: https://github.com/rails/rails/commit/5df72a238e9fcb18daf6ab6e6dc9051c9106d7bb#L0L324

  I'm assuming here that :load_config needs to be invoked separately from :environment, as it is elsewhere in the file for db operations, if not the alternative is to go back to "task :dump => :environment do".

  [Ben Woosley]

• Update to rack-cache 1.1.

  Versions prior to 1.1 delete the If-Modified-Since and If-Not-Modified headers when config.actioncontroller.performcaching is true. This has two problems:

  • unexpected inconsistent behaviour between development & production environments
  • breaks applications that use of these headers

  [Brendan Ribera]

• Ensure that enhancements to assets:precompile task are only run once [Sam Pohlenz]

• TestCase should respect the view_assigns API instead of pulling variables on its own. [José Valim]

• javascriptpath and stylesheetpath now refer to /assets if asset pipelining is on. [Santiago Pastorino]

• button_to support form option. Now you're able to pass for example 'data-type' => 'json'. [ihower]

• imagepath and imagetag should use /assets if asset pipelining is turned on. Closes #3126 [Santiago Pastorino and christos]

• Avoid use of existing precompiled assets during rake assets:precompile run. Closes #3119 [Guillermo Iguaran]

• Copy assets to nondigested filenames too [Santiago Pastorino]

• Give precedence to config.digest = false over the existence of manifest.yml asset digests [christos]

• escape options for the stylesheetlinktag method [Alexey Vakhov]

• Re-launch assets:precompile task using (Rake.)ruby instead of Kernel.exec so it works on Windows [cablegram]

• env var passed to process shouldn't be modified in process method. [Santiago Pastorino]

• rake assets:precompile loads the application but does not initialize it.

  To the app developer, this means configuration add in config/initializers/* will not be executed.

  Plugins developers need to special case their initializers that are meant to be run in the assets group by adding :group => :assets. [José Valim]

• Sprockets uses config.assets.prefix for asset_path [asee]

• FileStore keyfilepath properly limit filenames to 255 characters. [phuibonhoa]

• Fix Hash#toquery edge case with htmlsafe strings. [brainopia]

• Allow asset tag helper methods to accept :digest => false option in order to completely avoid the digest generation. Useful for linking assets from static html files or from emails when the user could probably look at an older html email with an older asset. [Santiago Pastorino]

• Don't mount Sprockets server at config.assets.prefix if config.assets.compile is false. [Mark J. Titorenko]

• Set relative url root in assets when controller isn't available for Sprockets (eg. Sass files using asset_path). Fixes #2435 [Guillermo Iguaran]

• Fix basic auth credential generation to not make newlines. GH #2882

• Fixed the behavior of asset pipeline when config.assets.digest and config.assets.compile are false and requested asset isn't precompiled. Before the requested asset were compiled anyway ignoring that the config.assets.compile flag is false. [Guillermo Iguaran]

• CookieJar is now Enumerable. Fixes #2795

• Fixed AssetNotPrecompiled error raised when rake assets:precompile is compiling certain .erb files. See GH #2763 #2765 #2805 [Guillermo Iguaran]

• Manifest is correctly placed in assets path when default assets prefix is changed. Fixes #2776 [Guillermo Iguaran]

• Fixed stylesheetlinktag and javascriptincludetag to respect additional options passed by the users when debug is on. [Guillermo Iguaran]

• Fix ActiveRecord#exists? when passsed a nil value

• Fix assertselectemail to work on multipart and non-multipart emails as the method stopped working correctly in Rails 3.x due to changes in the new mail gem.

ActiveModel

• Remove hard dependency on bcrypt-ruby to avoid make ActiveModel dependent on a binary library. You must add the gem explicitly to your Gemfile if you want use ActiveModel::SecurePassword:

  gem 'bcrypt-ruby', '~> 3.0.0'

  See GH #2687. [Guillermo Iguaran]

ActiveRecord

• Add deprecation for the preload_associations method. Fixes #3022.

  [Jon Leighton]

• Don't require a DB connection when loading a model that uses setprimarykey. GH #2807.

  [Jon Leighton]

• Fix using select() with a habtm association, e.g. Person.friends.select(:name). GH #3030 and #2923.

  [Hendy Tanata]

• Fix belongs_to polymorphic with custom primary key on target. GH #3104.

  [Jon Leighton]

• CollectionProxy#replace should change the DB records rather than just mutating the array. Fixes #3020.

  [Jon Leighton]

• LRU cache in mysql and sqlite are now per-process caches.

  • lib/activerecord/connectionadapters/mysql_adapter.rb: LRU cache keys are per process id.
  • lib/activerecord/connectionadapters/sqlite_adapter.rb: ditto

• Database adapters use a statement pool for limiting the number of open prepared statments on the database. The limit defaults to 1000, but can be adjusted in your database config by changing 'statement_limit'.

• Fix clash between using 'preload', 'joins' or 'eager_load' in a default scope and including the default scoped model in a nested through association. (GH #2834.) [Jon Leighton]

• Ensure we are not comparing a string with a symbol in HasManyAssociation#inverseupdatescounter_cache?. Fixes GH #2755, where a counter cache could be decremented twice as far as it was supposed to be.

  [Jon Leighton]

• Don't send any queries to the database when the foreign key of a belongs_to is nil. Fixes GH #2828. [Georg Friedrich]

• Fixed findin_batches method to not include order from defaultscope. See GH #2832 [Arun Agrawal]

• Don't compute table name for abstract classes. Fixes problem with setting the primary key in an abstract class. See GH #2791. [Akira Matsuda]

• Psych errors with poor yaml formatting are proxied. Fixes GH #2645 and GH #2731

• Use the LIMIT word with the methods #last and #first. Fixes GH #2783 [Damien Mathieu]

ActiveResource

• No changes

ActiveSupport

• ruby193: String#prepend is also unsafe [Akira Matsuda]

• Fix obviously breakage of Time.=== for Time subclasses [jeremyevans]

• Added fix so that file store does not raise an exception when cache dir does not exist yet. This can happen if a delete_matched is called before anything is saved in the cache. [Philippe Huibonhoa]

• Fixed performance issue where TimeZone lookups would require tzinfo each time [Tim Lucas]

• ActiveSupport::OrderedHash is now marked as extractable when using Array#extract_options! [Prem Sichanugrist]

Railties

• Add jquery-rails to Gemfile of plugins, test/dummy app needs it. Closes #3091. [Santiago Pastorino]

• rake assets:precompile loads the application but does not initialize it.

  To the app developer, this means configuration add in config/initializers/* will not be executed.

  Plugins developers need to special case their initializers that are meant to be run in the assets group by adding :group => :assets.

SHA-1

• 9337cff7772da034b0b34b73b85cf249f1a70f52 actionmailer-3.1.1.gem
• 7bb1b8d096a6ff1ff46dcfb778bf86a5daca1b0d actionpack-3.1.1.gem
• d5dc71e1a9a0e20d819f4dff27ff0697e99a7f64 activemodel-3.1.1.gem
• 7245632cb3b38612628304c1e244855d0053f7be activerecord-3.1.1.gem
• 6d09800202c2747e84249b8646f0fd480ed4924f activeresource-3.1.1.gem
• 66df2fd144aab22f52819fd489e33a976d68a46b activesupport-3.1.1.gem
• 6a35a49948bbd9f461839a1a271def90b23a851a rails-3.1.1.gem
• 6979ef891bd03fb639b979af9fdc56781f9358d9 railties-3.1.1.gem

You can find an exhaustive list of changes on github. Along with the closed issues marked for v3.1.1.

Thanks to everyone!

07 Oct 2011 5:26pm GMT

If you see this message in your RSS reader, please correct your RSS link to the following URL: http://fosdem.org/rss.xml.

26 Jul 2008 5:55am GMT

These pages have been archived.
For information about the latest FOSDEM edition please check this url: http://fosdem.org

25 Jul 2008 4:43pm GMT

Two weeks after FOSDEM and we are proud to publish most of the slides and videos from this year's edition.

All of the material from the Lightning Talks has been put online. We are still missing some slides and videos from the Main Tracks but we are working hard on getting those completed too.

We would like to thank our mirrors: HEAnet (IE) and Unixheads (US) for hosting our videos, and NamurLUG for quick recording and encoding.

The videos from the Janson room were live-streamed during the event and are also online on the Linux Magazin site.

We are having some synchronisation issues with Belnet (BE) at the moment. We're working to sort these out.

09 Mar 2008 3:12pm GMT