pl6anet

Raku RSS Feeds

Roman Baumer (Freenode: rba #raku or ##raku-infra) / 2020-05-29T03:19:20


gfldex: Ungolden silence

Published by gfldex on 2020-05-27T10:03:34

On my quest to replace Bash with Raku I tried to use qx and failed. It didn’t work for me because qx fails the wrong way. Under the hood Rakudo implements it by some grammar magic that forwards to QX in core.c/Proc.pm6 as follows.

sub QX($cmd, :$cwd = $*CWD, :$env) is implementation-detail {
    my $proc := Proc.new(:out);
    $proc.shell($cmd, :$cwd, :$env);
    $proc.out.slurp(:close) // Failure.new("Unable to read from '$cmd'")
}

This will return an empty Str, an non-empty Str or a Failure object. So this can be defined but False, defined but True or undefined but False. We get the undefined case when the shell that shell starts unexpectedly closes its STDOUT descriptor. We can get something that is False even if the command succeeds and we can get something that is True if the comand fails. On Unix silence is gold. Unless you provide -v or --verbose any command that does not need to output should not. If there is something wrong a non-zero exit code is returned and if ever possible something is written to STDERR. QX ignores exitcodes and stderr is forwarded to $*ERR. So we have little chance the catch it after the fact.

This behaviour will create hard to catch bugs. Let’s make a typo.

my $s = qx!True!;
say [$s.?defined, $s.?Bool, $s.?exitcode];
dd $not-proc;
/bin/sh: 1: True: not found
[True False (Any)]
Str $s = ""

Using the defined-or operator wont help here. Nor would try. Does anybody use qx and try?

[email protected]:~/projects/raku/rakudo/src$ ack 'QX'
core.c/Process.pm6
82: once if !Rakudo::Internals.IS-WIN && try { qx/id/ } -> $id {

On Windows that might actually fail as expected. On any other platform we need something sane.

sub sane-QX($cmd, :$cwd = $*CWD, :$env, :$quiet) {
    my $proc := Proc.new(:out, :err);
    $proc.shell($cmd, :$cwd, :$env);
    my $stdout = $proc.out.slurp(:close) // Failure.new("Unable to read from '$cmd'");
    my $stderr = $proc.err.slurp(:close);
    $*ERR.print: $stderr unless $quiet;
    if $proc.exitcode != 0 {
        return "" but role QXFail {
            method defined { False }
            method exitcode { $proc.exitcode }
            method err { $stderr }
        }
    }
    $stdout
}

my $sane = sane-QX(‚True‘);
say [$sane.?defined, $sane.?Bool, $sane.?exitcode, $sane.?err];
say sane-QX(‚True‘) // ‚I has a booboo!‘;

I’m not sure if just mixing undefinedness in is the right way. Maybe an exception with the exitcode and STDERR content would be better. That way the error handling can be moved into a CATCH block.

I will ponder this a little longer and then file a bug report. If it turns out the current implementation is desired the docs will need some warning stickers.

gfldex: Watching new arrivals

Published by gfldex on 2020-05-26T09:44:47

Any boring recurring task must be as easy as possible or it will be neglected. I’m quite sure this is why we invented computers. Backups are kinda borring. In fact you want to avoid any form of excitement when it comes to backups. So they must be as easy as possible. I have a script that is triggered by a udev rule when a new device is added. This is working fine when a single disk is plugged in. (This works very well.) I got a usb hub with a few usb sticks that form a btrfs raid5 for making quick backups of my $home whenever I switch the usb hub on. This does not work fine in some cases. Getting a bash script to check if a drive in a set is missing ain’t fun. Mostly because only proper languages come with Set. We do have a proper language.

On linux it is fairly easy to find out if a drive was plugged in. All we need to do is to watch for new files to pop up in /dev/disk/by-id/. We can also learn if new partitions where found. The directory looks like the following.

$ ls -1 /dev/disk/by-id/
ata-CT120BX500SSD1_1902E16BC135
ata-CT120BX500SSD1_1902E16BC2AA
ata-TOSHIBA_HDWQ140_X83VK0GDFAYG
ata-TOSHIBA_HDWQ140_X83VK0GDFAYG-part1
ata-TOSHIBA_HDWQ140_X83VK0GDFAYG-part2
ata-TOSHIBA_HDWQ140_X83VK0GDFAYG-part3
ata-TOSHIBA_HDWQ140_Y8J9K0TZFAYG
ata-TOSHIBA_HDWQ140_Y8J9K0TZFAYG-part1
ata-TOSHIBA_HDWQ140_Y8J9K0TZFAYG-part2
ata-TOSHIBA_HDWQ140_Y8J9K0TZFAYG-part3
usb-SanDisk_Ultra_USB_3.0_4C530001160708110455-0:0
usb-SanDisk_Ultra_USB_3.0_4C530001190708111070-0:0
usb-SanDisk_Ultra_USB_3.0_4C530001220708110370-0:0
usb-SanDisk_Ultra_USB_3.0_4C530001280708111064-0:0
wwn-0x50000398dc60029a
wwn-0x50000398dc60029a-part1
wwn-0x50000398dc60029a-part2
wwn-0x50000398dc60029a-part3
wwn-0x50000398ebb01681
wwn-0x50000398ebb01681-part1
wwn-0x50000398ebb01681-part2
wwn-0x50000398ebb01681-part3

If we look for anything that doesn’t end in '-part' \d+ we got a drive. We can also tell where it’s plugged in by checking the prefix.

sub scan-drive-ids(--> Set) {
    my Set $ret;
    for '/dev/disk/by-id/'.IO.dir.grep(!*.IO.basename.match(/'part' \d+ $/)) {
        $ret ∪= .basename.Str;
        CATCH { default { warn .message } }
    }

    $ret
}

my %last-seen := scan-drive-ids;

Sets don’t got an append method. We can substitude that with ∪=. Now we got a lovely Set of drives in %last-seen that are already there. We now need to wait for new files to pop up and apply set theory to them.

react {
    whenever IO::Notification.watch-path('/dev/disk/by-id/') {
        my %just-seen := scan-drive-ids;
        my %new-drives := %just-seen ∖ %last-seen;
        my %old-drives := %last-seen ∩ %just-seen;
        my %removed-drives := %last-seen ∖ %just-seen;
        %last-seen := %just-seen;

        # say ‚old drives: ‘, %old-drives.keys.sort;
        say ‚new drives: ‘, %new-drives.keys.sort || '∅';
        say ‚removed drives: ‘, %removed-drives.keys.sort || '∅';
    }
}

By binding the Sets to an Associative container we get for and other buildins to behave. If we want to take action if certain disks are added we need to define Sets that contain the right file names.

my %usb-backup-set = Set(<usb-SanDisk_Ultra_USB_3.0_4C530001160708110455-0:0 usb-SanDisk_Ultra_USB_3.0_4C530001190708111070-0:0 usb-SanDisk_Ultra_USB_3.0_4C530001220708110370-0:0 usb-SanDisk_Ultra_USB_3.0_4C530001280708111064-0:0>);

my %root-backup-disk = Set(<ata-TOSHIBA_DT01ACA200_8443D04GS>);

my $delayed-check := Channel.new;
my Promise $timeout-promise;

react {
    whenever IO::Notification.watch-path('/dev/disk/by-id/') {
        my %just-seen := scan-drive-ids;
        my %new-drives := %just-seen ∖ %last-seen;
        my %old-drives := %last-seen ∩ %just-seen;
        my %removed-drives := %last-seen ∖ %just-seen;
        %last-seen := %just-seen;

        # say ‚old drives: ‘, %old-drives.keys.sort;
        say ‚new drives: ‘, %new-drives.keys.sort || '∅';
        say ‚removed drives: ‘, %removed-drives.keys.sort || '∅';

        if %usb-backup-set ∩ %new-drives {
            $timeout-promise = Promise.in(5).then: {
                $delayed-check.send: True;
                $timeout-promise = Nil;
            } without $timeout-promise;
        }

        if %root-backup-disk ∩ %new-drives {
            sleep 2;
            backup-root-and-home-to-disk(%root-backup-disk);
        }

        say '';
    }
    whenever $delayed-check {
        my %just-seen := scan-drive-ids;
        if %usb-backup-set ⊆ %just-seen {
            backup-home-to-usb(%usb-backup-set);
        } elsif %usb-backup-set ∩ %just-seen {
            warn 'drive missing in usb set: ' ~ (%usb-backup-set ∖ (%usb-backup-set ∩ %just-seen)).keys;
            reset-usb-hub;
        }
    }
}

I use the $delayed-check whenever-block to handle the case when one of the usb sticks refuses to come online. The vendorid and deviceid of the usb hub are hardcoded. Please note that state and start don’t mix well.

sub reset-usb-hub(--> True) {
    state $reset-attempt = 0;
    if $reset-attempt++ {
        say ‚already reset, doing nothing‘;
        $reset-attempt = 0;
    } else {
        say ‚Resetting usb hub.‘;
        my $usb_modeswitch = run <usb_modeswitch -v 0x2109 -p 0x0813 --reset-usb>;
        fail ‚resetting usb hub failed‘ unless $usb_modeswitch;
    }
}

The entire script can be found here. I believe the example of watch-path could use a modified version of this script. If you read it you can tell where Sets are used simply by spotting set operators. Making Raku a operator oriented language was a good idea. Thank you Larry.

While turning my backup script from Bash to Raku I had some more findings about shell scripting in a proper language. I shall report about them here in the next few weeks.

gfldex: Returning the right amount

Published by gfldex on 2020-05-25T08:34:17

Do you know that look your better halve gives you when you return with the wrong number of items from the groceries? Tell you what, we will write a script for that!

sub buy-eggs($amount --> Seq where *.elems == $amount) {
    my $forgotten-amount = (1..$amount).roll;
    return "egg" xx $amount - $forgotten-amount;
}

say buy-eggs 10;
Cannot do non-typename cases of type_constraint yet
at /home/dex/projects/blog/return-check.raku:3
------> $amount --> Seq where *.elems == $amount⏏) {

Raku failed us! But Raku being Raku we are allowed to help it.

multi sub trait_mod:<is>(Sub $s, :return-where(&condition)) {
    $s.wrap({
        my $ret = callsame;

        if !condition($ret) {
            fail ‚return condition does not match for sub ‘ ~ $s.name ~ ‚ for return value of ‘ ~ $ret.gist;
        }

        $ret
    });
}

Now we can check if a sub returns a certain amount of elements.

sub buy-eggs($amount --> Seq) is return-where(*.elems == 10) {...}

There is a catch though. In contrast to a where-clause a trait does not have access to variables in the scope of the signature. We can use a POST phaser for that.

sub buy-eggs($amount --> Seq) {
    my $forgotten-amount = (1..$amount).roll;
    return "egg" xx $amount - $forgotten-amount;

    POST { $_ == $amount or fail "back to the store!" }
}

A proper where clause on the return check would be nice for documentation. If we can move any input and output validation into the signature that signature becomes a description of the input and output values. And quite in contrast to hand written documentation it stays in sync with the code.

I hope that the prospect of spouse agro will convince the core devs to add where clauses to return checks.

gfldex: The mysterious infix

Published by gfldex on 2020-05-24T16:46:36

Vadim Belman informed us that he started a module for writing GUI apps in the console. That gave me a massive flashback to the nineties. Looking at the example code got me the next flashback. If you need a button you need to subclass. In fact if you need anything you need to subclass. In a dynamic language with lexical scoping that might actually work.

Looking at the code one can see lots and lots of Raku features to be used. I even found this gem.

$!id = ++⚛$sequence;

That’s clearly not from the nineties. Back then real programmers could get anything done with just one core!

There are a few examples Vadim wrote for us and the interface for his module look pretty 90ish again. Which made me think for a while how a “modern” interface could look like. In the process I didn’t come up with something better but stepped on a bug.

As it turns out is looser(&infix:<:>)should be in %categorically-won't-work. There are quite a few things that wont work but are not documented. Luckily there is a workaround for defining a new operator that got looser precedence than the colon that separates a method name from its arguments.

sub infix:«add»(Parent:D \p, Child:D \c --> Parent:D) is equiv(&infix:<and>) {
    p.add-child: c;
    p
}

$p add Child.new: :birthday("april") add Child.new: :birthday("december");

All we need to do is to be equally loose then and.

The usefulness of an add operator appeared questionalbe to me until I realised that feed operators are not. Yet Raku does not make much use of them. I believe the following should be dwimmy.

my $c = Channel.new;
$c <== "foo";

Raku is still a bit incomplete. But fear not we are only at e!

Rakudo Weekly: 2020.20 Continuously Upgraded

Published by liztormato on 2020-05-18T12:59:37

Patrick Böker has added AzureCI as a new Continuous Integration pipeline for Rakudo. It is meant to replace the different CI pipelines that Rakudo uses now. AzureCI tests all build combinations the other CI pipelines tested (plus some more). This will get rid of a good bit of redundant, wasteful testing and will make it easier to provide nightly builds and binary releases (with MacOS, Linux and Windows covered). The plan is to stop using Travis, AppVeyor and CircleCI for testing Rakudo core soon.

Conference in the Cloud Schedule

The schedule for the Conference in the Cloud has been published (/r/rakulang comments). The following presentations have Raku content:

If you want to attend the presentations interactively, you can get a ticket for just 10 US$ (or how much more you would like to donate). Presentations will also be available on YouTube.

Rakudo Star 2020.05.1

Naoum Hankache informs us that there is now a Release Candidate for Rakudo Star 2020.05.1 on Windows, while Patrick Spek informs us there is another 2020.05.1 release candidate for other operating systems as well. Kudos to both for making this happen!

Reading DNA and Count Bases

Suman Khanal has written a nice blog post about using Raku in Bioinformatics (HackerNews comments), specifically related to parsing FASTA files. It also had a challenge, which Elizabeth Mattijsen took on and resulted in the initial version of a Fasta module.

Logs, autorotating

Wenzel P. P. Peppmeyer has written a blog post about automatically rotating logs (/r/rakulang comments)

April Report

Jonathan Worthington reported on the work they did on the Raku Development Grant in April.

Renaming Progress

Weekly Challenge

These are entries for Challenge #60 that have Raku solutions:

Challenge #61 is up for your perusal!

Core Developments

Questions about Raku

Meanwhile on Twitter

Meanwhile on perl6-users

Comments about Raku

New Raku Modules

Updated Raku Modules

Winding down

A week with a lot of work on the internals (the Great Dispatcher Overhaul is coming along nicely), quite a few exciting new modules and module updates and some fine blog posts. Yours truly keeps repeating: stay safe, stay healthy, stay helpful. See you next week!

gfldex: Autorotating logs

Published by gfldex on 2020-05-14T08:56:18

While translating my backup script from bash to Raku I did a peek into /var/log/and was shocked to find 650MB of stuff I never looked at. Unsurprisingly /var/log/journal was the largest offender . Sadly I must say that systemd is very consistent in convincing me to be not very good software. The oldest offender was from 2008. Debian is very consistent in convincing me to keep it around. Even though it does not come with logrotate for xrdp.log. How hard can it be to build in rotating and compressing logs into your own software?

That turned out to be a 60-line-problem.

sub open-logfile(IO() $log-file-path = IO::Path, :$flush = False, :$max-size = 2**20 * 50, :$max-backlog = 7 --> Routine) {
    my $log-channel = Channel.new;
    my $log-promise = start {
        my $log-handle = $log-file-path ?? open $log-file-path, :a !! $*ERR;
        my $log-file-dir = $log-file-path.dirname;
        my $log-file-basename = $log-file-path.basename;

        sub compress-file($path) {
            my $gz = run <gzip -9>, $path;
            warn „could not compress $path“ unless $gz;
        }

        my $log-file-length = $log-file-path ?? $log-file-path.s !! 0;
        sub rotate-log {
            return without $log-file-path;
            note ‚rotating logfile‘;
            $log-handle.close;

            my @siblings = dir($log-file-dir).grep(*.basename.starts-with($log-file-basename));
            for @siblings».Str.sort.skip.head($max-backlog - 1).reverse -> $path {
                my $n = $path.match(/log '.' (\d+)/)[0].Int + 1;
                compress-file($path) if $n == 2;
                my $new-name = $log-file-path ~ '.' ~ ($n > 1 ?? $n ~ ‚.gz‘ !! $n);
                ($n == 2 ?? $path ~ ‚.gz‘ !! $path).IO.rename($new-name);
            }

            $log-file-path.rename($log-file-path ~ '.1');
            $log-handle = open $log-file-path, :a;
            $log-file-length = 0;
        }

        react whenever $log-channel -> Str() $line {
            my $utf8 = ($line ~ "\n").encode;
            $log-file-length += $utf8.bytes;
            rotate-log if $log-file-length > $max-size;
            $log-handle.write: $utf8;
            $log-handle.flush if $flush;
        }
    }

    my $last-message = "";
    my int $dupe-counter = 0;
    sub ($message --> Nil) {
        my $now = now.DateTime;
        my $timestamp = $now.yyyy-mm-dd ~ ' ' ~ $now.hh-mm-ss;

        if $message eq $last-message {
            $dupe-counter++;
        } else {
            if $dupe-counter > 0 {
                $log-channel.send: $timestamp ~ ' Last message repeated ' ~ $dupe-counter ~ ' times.';
            } else {
                $log-channel.send: $timestamp ~ ' ' ~ $message;
            }
            $dupe-counter = 0;
        }

        $last-message = $message;
    } but role :: { method close { $log-channel.close; await $log-promise } }
}

Without .flush after each line performance is good. With it it’s pretty bad given that we don’t have access to fancy linux flags for open and as such don’t really flush all that hard.

Since rotating logs and compressing them might take some time I put that actual writing to the file into its own thread. So usage becomes a bit indirect and we have to tell the logger to finish writing and close the file. If we don’t we can lose data. I’m not sure if the latter is a bug.

my &log = open-logfile(‚/tmp/test.log‘, :max-size(2**20));

my $lore = ‚Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.‘;

loop {
    log(100.rand > 50 ?? $lore !! 42.rand);
    last if (now - ENTER now) > 60 * 60 * 2;
}

&log.await;

After much deliberation I have come to the conclusion that renaming Perl 6 to Raku was a mistake. It should have been named Exceedingly Simple.

Rakudo Weekly: 2020.19 Release Release

Published by liztormato on 2020-05-11T15:32:48

Some weeks see one release. And then some do two! In any case, it is good to see the 2020.05 Rakudo Compiler Release, thanks to the hard work of Alexander Kiryuhin. Which was immediately followed by updated Docker Containers (by Suman Khanal and JJ Merelo) and installable Linux packages (by Claudio Ramirez). And since it has been a time since there was a Rakudo Star release, Patrick Spek immediately created a Rakudo Star Release Candidate. Rakudo Star users: please test this Release Candidate!

Testing a compiler

Andrew Shitov published Chapter 11 of their compiler book, about building control flow into the language.

RakuAST Grant Accepted

The RakuAST Grant Proposal by Jonathan Worthington has been accepted by the TPF Grant Committee.

Long Queue Short

Alexey Melezhik updates the community on RakuDist changes (/r/rakulang comments).

Confusing or What?

p6steve has written another interesting blog post, this time about Raku objects, and the different approaches for setters and getters. Which resulted in quite an extensive discussion on /r/rakulang.

Random Raku Reflections

Eric Londo has written a very nice column on their life experience in programming, and how Raku has become their personal favorite programming language to read and write (/r/rakulang comments).

Weekly Challenge

Laurent Rosenfeld did extensive weekly reviews of the Raku solutions of Challenges #56 and #57 and Luca Ferrari was the champion of April. These are entries for Challenge #59 that have Raku solutions:

Challenge #60 is up for your perusal!

Core Developments

Questions about Raku

Meanwhile on Twitter

Meanwhile on perl6-users

Comments about Raku

New Raku Modules

Updated Raku Modules

Winding down

Yet another week of every day feeling like every other day. But this week saw some nice releases, both in the compiler world, as well in the module world! Yours truly keeps repeating: stay safe, stay healthy, stay helpful. See you next week!

p6steve: Raku Objects: Confusing or What?

Published by p6steve on 2020-05-07T21:51:52

Chapter 1: The Convenience Seeker

Coming from Python, the Raku object model is recognizable, but brings a tad more structure:

Screenshot 2020-05-07 22.36.37

What works for me, as a convenience seeker, is:

These are the things you want if you are writing in a more procedural or functional style and using class as a means to define a record type.

Chapter 2: The Control Freak

Here’s the rub…

When we describe OO, terms like “encapsulation” and “data hiding” often come up. The key idea here is that the state model inside the object – that is, the way it chooses to represent the data it needs in order to implement its behaviours (the methods) – is free to evolve, for example to handle new requirements. The more complex the object, the more liberating this becomes.

However, getters and setters are methods that have an implicit connection with the state. While we might claim we’re achieving data hiding because we’re calling a method, not accessing state directly, my experience is that we quickly end up at a place where outside code is making sequences of setter calls to achieve an operation – which is a form of the feature envy anti-pattern. And if we’re doing that, it’s pretty certain we’ll end up with logic outside of the object that does a mix of getter and setter operations to achieve an operation. Really, these operations should have been exposed as methods with a names that describes what is being achieved. This becomes even more important if we’re in a concurrent setting; a well-designed object is often fairly easy to protect at the method boundary.

(source jnthn https://stackoverflow.com/questions/59671027/mixing-private-and-public-attributes-and-accessors-in-raku)

Let’s fix that:

Screenshot 2020-05-07 22.38.41
Now, I had to do a bit more lifting, but here’s what I got:

And, in contrast to Chapter 1:

Chapter 3: Who Got the Colon in the End?

I also discovered Larry’s First Law of Language Redesign: Everyone wants the colon

Apocalypse 1: The Ugly, the Bad, and the Good https://www.perl.com/pub/2001/04/02/wall.html/

I conclude that Larry’s decision was to confer the colon on the method syntax,  subtly tilting the balance towards the strict model: by preferring $p.y: 3 over $p.y = 2.

Rakudo Weekly: 2020.18 Comma Community

Published by liztormato on 2020-05-04T12:52:08

Jonathan Worthington managed to publish a new version of the Comma Community Edition, a free version of Comma, the Integrated Development Environment for Raku. Available for Windows, Linux and MacOS (or as an IDEA compatible plugin), this new release adds many new features, such as a live grammar preview, a multiline REPL (Read Evaluate Print Loop), and a formatter with configurable spacing rules. And many other fixes and improvements! (/r/rakulang comments).

Getting Chatty in a Car

Damian Conway possibly gave their most fun and unusual interview in 2018 while commuting, elaborating on linguistic diversity and programming language innovation. Worth a watch at about 15 minutes (blogs.perl.org, /r/rakulang comments).

Season of Doc ideas

JJ Merelo makes sure we don’t forget that The Perl Foundation will be applying again for participation in the Google Season of Docs. And that they’re looking for technical writers to help with improving the Raku documentation. Check out the ideas so far and add your own with a Pull Request!

Great Dispatcher Overhaul

Jonathan Worthington has published a gist with their thoughts on unifying many aspects of the current implementation of MoarVM into a “dispatch chain” like mechanism (/r/rakulang comments).

Goss in Raku

Alexey Melezhik announced their Sparrowdo::Goss module that allows you to run goss scenarios remotely.

Testing a compiler

Andrew Shitov published Chapter 10 of their compiler book, about testing the compiler.

Remembering Jeff Goff

Wendy van Dijk uploaded some more pictures of Jeff, some by Chris Jack.

Renaming Progress

Weekly Challenge

The Raku entries for Challenge #58:

Challenge #59 is up for your perusal!

Other Core Developments

Questions about Raku

Meanwhile on Twitter

Comments about Raku

New Raku Modules

Updated Raku Modules

Winding down

Yet another week has gone by. Good to see plenty of stuff happening in a time when every day feels like any other day. Yours truly keeps repeating: stay safe, stay healthy, stay helpful. See you next week!

Rakudo Weekly: 2020.17 SprinG Cleanup

Published by liztormato on 2020-04-27T14:58:58

Stefan Seifert has even done more than their usual amount of work on Rakudo. But the past week was special in that they developed a GCC-plugin that showed potential issues with objects being moved by Garbage Collection when the code assumed they couldn’t be. And this showed more than 30 of these cases that, in certain situations, could cause (very hard to reproduce) segfaults to happen. And all of these got fixed as well. A nice cleanup indeed!

Binex and Objex

Alabemenhu (aka guifa) has committed their work in progress to allow grammars to interpret binary data. Still hoping for a nice blog post. Great work so far, and already pretty useful!

Conference in the Cloud

Only a few more days for you to submit a Raku presentation for the Perl and Raku Conference in the Cloud (on 24-26 June). And there are really not enough Raku presentations yet!

“Just a chapter”

Andrew Shitov still found time to publish Chapter 9 of their compiler book, about evaluating Abstract Syntax Trees (aka ASTs). And quite a chapter it is!

Renaming Progress

Weekly Challenge

The Raku entries for Challenge #57:

Laurent Rosenfeld did a full review of all Raku solutions of week #54. And finally, Challenge #58 is up for your perusal!

Other Core Developments

Questions about Raku

Meanwhile on Twitter

Meanwhile on perl6-users

Comments about Raku

New Raku Modules

Updated Raku Modules

Winding down

A lot of work being done behind the scenes this week. Still, quite a nice catch of new and updated modules. Yours truly keeps repeating: stay safe, stay healthy, stay helpful. See you next week!

Rakudo Weekly: 2020.16 Rash In Progress

Published by liztormato on 2020-04-20T18:49:47

Arne Sommer has started publishing a number of blog posts on how to create a Raku shell (a command line interpreter like Bash). The first instalments cover paths, loops, catching interrupts and running external programs. In interesting introduction to many Raku features. And maybe the start of something really cool! (/r/rakulang comments).

Conference in the Cloud

As we know, all Perl Conferences are basically cancelled until further notice, but the organizers have not stopped: on 24-26 June there will be a Perl and Raku Conference in the Cloud. And there are really not enough Raku presentations yet! So, set aside your fears, and submit a Raku presentation! You have until May to register your presentation, so you still have a little time to mull over a subject.

Pod6 in Javascript

Alexandr Zahatski surprised yours truly, and probably many others, with an implementation of Rakudoc (aka pod6) in Javascript, and a blog post with a status update. You can even try it out in your browser! Cool stuff (/r/rakulang comments).

Grammars for binary data

Alabemenhu (aka guifa) has been working on making it possible to use grammars to interpret binary data (as opposed to strings). Yours truly is hoping for a very nice blog post about it soon. In the meantime, you can leave any comments / suggestions that you have.

Saving 70%

P6steve hit rock bottom and decided to use their Raku programming chops developed in the past years to re-invent the Physics::Unit module, and blog about it (HackerNews comments).

Grant Proposal

Last week, Jonathan Worthington surprised us with the introduction of their RakuAST plans. This week, they have submitted a RakuAST grant proposal for a TPF grant. Your comments on the proposal will be very much appreciated.

One month later

Andrew Shitov looked back on the first month of the Covid-19 Observer, which is now also available in Russian. And in the meantime Andrew still found time to publish part 2 of Chapter 8 of their compiler book.

Renaming Progress

Weekly Challenge

The Raku entries for Challenge #56:

Laurent Rosenfeld revisited the Collatz Sequence of 2 weeks ago, calculating the sequence length for all numbers up to 1_000_000 (/r/rakulang, HackerNews comments). You can also meet Alicia Bielsa, the March Champion of the Weekly Challenge. And finally, Challenge #57 is up for your perusal!

Core Developments

Questions about Raku

Meanwhile on Twitter

Meanwhile on perl6-users

Comments about Raku

New Raku Modules

Updated Raku Modules

Winding down

Quite a few blog posts, and exciting developments this week, so who’s to complain? Finally, yours truly repeats again: stay safe, stay healthy, stay helpful. See you next week!

p6steve: Raku vs. Perl – save 70%

Published by p6steve on 2020-04-17T17:36:39

Having hit rock bottom with an ‘I can’t understand my own code sufficiently enough to extend/maintain it’, I have been on a journey to review the perl5 Physics::Unit design and to use this to cut through my self made mess of raku Physics::Unit version 0.0.2.

Now I bring the perspective of a couple of years of regular raku coding to bear, so I am hoping that the bastard child of mature perl5 and raku version one will surpass both in the spirit of David Bowie’s “Pretty Things”.

One of the reasons I chose Physics::Units as a project was that, on the face of it, it seemed to have an aspect that could be approached by raku Grammars – helping me learn them. Here’s a sample of the perl5 version:

Screenshot 2020-04-17 18.40.05

Yes – a recursive descent parser written from scratch in perl5 – pay dirt! There are 215 source code lines dedicated to the parse function. 5 more screens like this…

So I took out my newly sharpened raku tools and here’s my entire port: 

Screenshot 2020-04-17 18.42.08

Instead of ranging over 215 lines, raku has refined this down to a total of 58 lines (incl. the 11 blank ones I kept in for readability) – that’s a space saving of over 70%. Partly removal of parser boilerplate code, partly the raku Grammar constructs and partly an increased focus on the program logic as opposed to the mechanism.

For my coding style, this represents a greater than a two-thirds improvement – by getting the whole parser onto a single screen, I find that I can get the whole problem into my brain’s working memory and avoid burning cycles scrolling up and down to pin down butterflies bugs.

Attentive students will have noted that the Grammar / code integration provides a very natural paradigm for loading real-world data into an OO system, the UnitAction class starts with a stub object and populates ‘has’ attributes as it goes.

Oh and the raku code does a whole lot more such as support for unicode superscripts (up to +/-4), type assignment and type checking, offsets (such as 0 K = 273.15 °C), wider tolerance of user input and so on. Most importantly Real values are kept as Rats as much as possible which helps greatly for example, when round tripping 38.5 °C to  °F and back it is still equals 38.5 °C!

One final remark – use Grammar::Tracer is a fantastic debugging tool for finding and fixing the subtle bugs that can come in and contributing to quickly getting to the optimum solution.

rakudo.org: Rakudo Star Release 2020.01

Published on 2020-02-24T00:00:00

p6steve: Raku: the firkin challenge

Published by p6steve on 2020-01-20T22:40:01

For anyone wondering where my occasional blog on raku has been for a couple of months – sorry. I have been busy wrestling with, and losing to, the first released version of my Physics::Measure module.

Of course, this is all a bit overshadowed by the name change from perl6 to raku. I was skeptical on this, but did not have a strong opinion either way. So kudos to the folks who thrashed this out and I am looking forward to a naissance. For now, I have decided to keep my nickname ‘p6steve’ – I enjoy the resonance between P6 and P–sics and that is my niche. No offence intended to either camp.

My stated aim (blogs passim) is to create a set of physical units that makes sense for high school education. To me, inspired by the perl5 Physics::Unit module, that means not just core SI units for science class, but also old style units like sea miles, furlongs/fortnight and so on for geography and even history. As I started to roll out v0.0.3 of raku Physics::Unit, I thought it would be worthwhile to track a real-world high school education resource, namely OpenStax CNX. As I came upon this passage, I had to take the firkin challenge on:

While there are numerous types of units that we are all familiar with, there are others that are much more obscure. For example, a firkin is a unit of volume that was once used to measure beer. One firkin equals about 34 liters. To learn more about nonstandard units, use a dictionary or encyclopedia to research different “weights and measures.” Take note of any unusual units, such as a barleycorn, that are not listed in the text. Think about how the unit is defined and state its relationship to SI units.

Disaster – I went back to the code for Physics::Unit and, blow me, could I figure out how to drop in a new Unit: the firkin??…. nope!! Why not? Well Physics::Unit v:0.0.3 was impenetrable even to me, the author. Statistically it has 638 lines of code alongside 380 lines of heredoc data. Practically, while it passes all the tests 100%, it is not a practical, maintainable code base.

How did we get here? Well I plead guilty to being an average perl5 coder who really loves the expressivity that Larry offers … but a newbie to raku. I wanted to take on Physics::Measure to learn raku. Finally, I have started to get raku – but it has taken me a couple of years to get to this point!

My best step now – bin the code. I have dumped my original effort, gone back to the original perl5 Physics::Unit module source and transposed it to raku. The result: 296 lines of tight code alongside the same 380 lines of heredoc – a reduction of 53%! And a new found respect for the design skill of my perl5 forbears.

I am aiming to release as v0.0.7 in April 2020.

Raku Advent Calendar: Happy new year!

Published by jjmerelo on 2019-12-26T18:48:44

This year’s advent calendar is over, and it leaves us lots of articles on metaprogramming, applications, useful Raku modules, how to migrate from Perl, programming techniques and even how to work with Raku inside containers.

No more articles until next year, and the call is already open: if you want to tell something about Raku and its surroundings, add your name to the list with possible title, and you’re in!

Meanwhile, you’ve got raku.org and docs.raku.org for all your Raku needs.

Raku means enjoyment. So we wish you all a lot of Raku for next year!

Raku Advent Calendar: Day 24: The Grinch of Raku, Part 2: Hold Your Horses

Published by kaiepi on 2019-12-24T00:01:00

In 2017, the Grinch ruined Christmas by showing off some of the naughty things you can do with Raku’s features. Unfortunately, while his heart grew by three sizes that year, there’s more than one Grinch! This Grinch will be doing something extra naughty this year, taking some inspiration from the JavaScript community.

You may have heard of JSFuck, which is a tool that allows you to write any JavaScript code using only the characters [, ], (, ), +, and !. This is something you’d only expect to be possible in a language like JavaScript, right? That’s not entirely true! To prove this, let’s port it to Raku. Since this can’t be implemented using the exact same set of characters, our restrictions will be that only non-alphanumeric ASCII characters may be used in the translated code, and string literals must not be used.

Generating Primitives

The first thing we’ll need to do is find a way to generate some primitives. The ones from JavaScript that are of interest to us are booleans, numbers, and strings; any other type of primitive can be represented through other means. These are generated mainly through type coercion on empty arrays, which also happens to be possible to do in Raku.

True and False can be generated in Raku using the ! prefix operator, similarly to how you can in JavaScript:

say ![];  # OUTPUT: True
say !![]: # OUTPUT: False

Using this in combination with the + prefix operator, we can generate any whole number, which is also the case in JavaScript:

say +[];         # OUTPUT: 0
say +![];        # OUTPUT: 1
say +![] + +![]; # OUTPUT: 2

In JavaScript, + also happens to be used to concatenate strings. When used with two empty arrays, + will coerce both to strings and concatenate them, which results in an empty string. + doesn’t behave like this in Raku, so we’ll need to use the ~ operator instead:

say (~[]).perl; # OUTPUT: ""

What about strings that aren’t empty though? In JavaScript, strings are iterable, which allows for certain characters to be used when stringifying values other than empty arrays. This isn’t the case in Raku! It’s time to start getting creative.

String bitwise operators allow you to perform the same bitwise operations you can perform on numbers on codepoints in strings. Using the ~^ infix operator, we can generate a null character given 0 and 0:

say ord +[] ~^ +[]; # OUTPUT: 0

We can’t generate the characters we need very easily with the ~+, ~|, and ~^ operators alone though. There is a way to do this using that null character, but we need a lowercase letter of some sort first. We can grab the letter "e" from "True" if we use a regex:

say ~(![] ~~ /...(.)/)[+[]]; # OUTPUT: e

Using an infinite sequence with these two characters, we can generate most of the characters in ASCII:

my Str:D @chars = (+[] ~^ +[]...~(![] ~~ /...(.)/)[+[]]...*);
say @chars[65..90];  # OUTPUT: (A B C D E F G H I J K L M N O P Q R S T U V W X Y Z)
say @chars[97..122]; # OUTPUT: (a b c d e f g h i j k l m n o p q r s t u v w x y z)

Now that we can generate the characters in the string "&chr", we’ll be able to generate any Unicode string after the next step.

Evaluating Code

Most of the JavaScript code that can be generated depends on the Function constructor in order to work. Using it, you can arbitrarily generate a function at runtime. As far as I know, it’s not possible to generate code like this in Raku without using &EVAL. There’s a problem we need to solve if we are to use it, though.

We can use string literals with &EVAL just fine:

say EVAL "'Hello, world!'"; # OUTPUT: Hello, world!

But if we try to use a value that is unknown at compile-time with it, we’ll get an exception warning about the security implications of what we’re doing, telling us to use the MONKEY-SEE-NO-EVAL pragma:

say EVAL my $ = "'Hello, world!'"; # Throws X::SecurityPolicy::Eval

That’s not good in our case! We can’t set this pragma without alphanumeric characters. It’s time to get naughty. What happens if we try to use &EVAL using an indirect symbol lookup?

say ::('&EVAL')(my $ = "'Hello world!'"); # OUTPUT: Hello, world!

Perfect! Along with this, using indirect symbol lookup we can also call the &chr routine to generate a string for any Unicode codepoint. In combination, this allows us to translate any valid Raku code.

Hold Your Horses

We’re ready to start writing code for our port of JSFuck. This will simply be a script that takes some Raku code as input and outputs its translation. All of the subroutines used (apart from &MAIN) will be pure. Now, let’s give this port a bit of a nicer name than the obvious choice and call it Hold Your Horses instead.

Our first subroutine will be &from-uint, which will translate numbers. We could just add 1 to 0 repeatedly until we get the number we’re looking for, but this will generate huge amounts of code for larger codepoints. One way we can shorten the code this generates is if we represent numbers as being products of prime numbers. This can be further shortened by representing prime numbers greater than 5 as being a sum of products of prime numbers:

use Prime::Factor;

sub from-uint(UInt:D $x, Int:D $remainder = 0 --> Str:D) is pure {
    proto sub translate(UInt:D --> Str:D) is pure {*}
    multi sub translate(0 --> '+[]') { }
    multi sub translate(1 --> '+![]') { }
    multi sub translate(UInt:D $x --> Str:D) {
        join ' + ', '+![]' xx $x
    }

    if $x <= 5 {
        my Str:D $translation = $x.&translate;
        $translation ~= ' + ' ~ $remainder.&from-uint if $remainder;
        $translation
    } elsif $x.is-prime {
        from-uint $x - 1, $remainder + 1
    } else {
        my Str:D $translation = $x.&prime-factors».&from-uint.fmt: '(%s)', ' * ';
        $translation ~= ' + ' ~ $remainder.&from-uint if $remainder;
        $translation
    }
}

Now we can implement &from-str, which will parse code input by the user. This needs to map each codepoint in the given code to a Hold Your Horses number, which can be done by looking up a character in the sequence of characters from earlier if it is within its range, otherwise &chr can be called. Since we’re using this sequence every time we see a character that is included by it, this will be stored in $_ by our next subroutine. Since translating a single codepoint can be quite intensive, let’s use the experimental is cached trait with our helper subroutine that handles this to avoid having to do it more than once for any given codepoint:

use experimental :cached;

sub from-str(Str:D $code --> Str:D) is pure {
    my Int:D constant LIMIT = 'z'.ord.succ;

    proto sub translate(UInt:D --> Str:D) is pure is cached {*}
    multi sub translate(UInt:D $codepoint where 0..^LIMIT --> Str:D) {
        sprintf '.[%s]', $codepoint.&from-uint
    }
    multi sub translate(UInt:D $codepoint where LIMIT..* --> Str:D) {
        sprintf '::(%s)(%s)',
                '&chr'.ords».&translate.join(' ~ '),
                $codepoint.&from-uint
    }

    sprintf '::(%s)(%s)',
            '&EVAL'.ords».&translate.join(' ~ '),
            $code.ords».&translate.join(' ~ ')
}

Now we can implement &hold-your-horses, which will handle the full translation of code input by the user. All this needs to do is store the sequence from earlier in $_ before calling &from-str:

sub hold-your-horses(Str:D $code --> Str:D) is pure {
    Qc:to/TRANSLATION/.chomp
    $_ := (+[] ~^ +[]...~(![] ~~ /...(.)/)[+[]]...*);
    {$code.&from-str};
    TRANSLATION
}

With &MAIN added, our script is now complete:

use v6.d;
use experimental :cached;
use Prime::Factor;
unit sub MAIN(Str:D $code) {
    say hold-your-horses $code
}

sub from-uint(UInt:D $x, Int:D $remainder = 0 --> Str:D) is pure {
    proto sub translate(UInt:D --> Str:D) is pure {*}
    multi sub translate(0 --> '+[]') { }
    multi sub translate(1 --> '+![]') { }
    multi sub translate(UInt:D $x --> Str:D) {
        join ' + ', '+![]' xx $x
    }

    if $x <= 5 {
        my Str:D $translation = $x.&translate;
        $translation ~= ' + ' ~ $remainder.&from-uint if $remainder;
        $translation
    } elsif $x.is-prime {
        from-uint $x - 1, $remainder + 1
    } else {
        my Str:D $translation = $x.&prime-factors».&from-uint.fmt: '(%s)', ' * ';
        $translation ~= ' + ' ~ $remainder.&from-uint if $remainder;
        $translation
    }
}

sub from-str(Str:D $code --> Str:D) is pure {
    my Int:D constant LIMIT = 'z'.ord.succ;

    proto sub translate(UInt:D --> Str:D) is pure is cached {*}
    multi sub translate(UInt:D $codepoint where 0..^LIMIT --> Str:D) {
        sprintf '.[%s]', $codepoint.&from-uint
    }
    multi sub translate(UInt:D $codepoint where LIMIT..* --> Str:D) {
        sprintf '::(%s)(%s)',
                '&chr'.ords».&translate.join(' ~ '),
                $codepoint.&from-uint
    }

    sprintf '::(%s)(%s)',
            '&EVAL'.ords».&translate.join(' ~ '),
            $code.ords».&translate.join(' ~ ')
}

sub hold-your-horses(Str:D $code --> Str:D) is pure {
    Qc:to/TRANSLATION/.chomp
    $_ := (+[] ~^ +[]...~(![] ~~ /...(.)/)[+[]]...*);
    {$code.&from-str};
    TRANSLATION
}

Now, does this actually work? For brevity’s sake, let’s say it works as intended if say "Hello, world! 👋" can be translated and run:

bastille% raku hold-your-horses.raku 'say "Hello, world! 👋"' > hello-world.raku
bastille% raku hello-world.raku
Hello, world! 👋

Perfect! This is the script’s output:

$_ := (+[] ~^ +[]...~(![] ~~ /...(.)/)[+[]]...*);
::(.[(+![] + +![]) * ((+![] + +![]) * (+![] + +![] + +![]) * (+![] + +![] + +![]) + +![])] ~ .[(+![] + +![] + +![]) * ((+![] + +![]) * ((+![] + +![]) * (+![] + +![] + +![] + +![] + +![]) + +![]) + +![])] ~ .[(+![] + +![]) * ((+![] + +![]) * (+![] + +![] + +![]) * ((+![] + +![]) * (+![] + +![] + +![]) + +![]) + +![])] ~ .[(+![] + +![] + +![] + +![] + +![]) * ((+![] + +![]) * (+![] + +![]) * (+![] + +![] + +![]) + +![])] ~ .[(+![] + +![]) * (+![] + +![]) * ((+![] + +![]) * (+![] + +![] + +![]) * (+![] + +![] + +![]) + +![])])(.[(+![] + +![] + +![] + +![] + +![]) * ((+![] + +![]) * ((+![] + +![]) * (+![] + +![] + +![] + +![] + +![]) + +![]) + +![])] ~ .[(+![] + +![]) * (+![] + +![]) * (+![] + +![]) * (+![] + +![]) * (+![] + +![]) * (+![] + +![] + +![]) + +![]] ~ .[((+![] + +![]) * (+![] + +![] + +![] + +![] + +![]) + +![]) * ((+![] + +![]) * (+![] + +![] + +![] + +![] + +![]) + +![])] ~ .[(+![] + +![]) * (+![] + +![]) * (+![] + +![]) * (+![] + +![]) * (+![] + +![])] ~ .[(+![] + +![]) * ((+![] + +![]) * (+![] + +![]) * (+![] + +![]) * (+![] + +![]) + +![])] ~ .[(+![] + +![]) * (+![] + +![]) * (+![] + +![]) * (+![] + +![] + +![]) * (+![] + +![] + +![])] ~ .[(+![] + +![]) * (+![] + +![]) * (+![] + +![] + +![] + +![] + +![]) * (+![] + +![] + +![] + +![] + +![]) + +![]] ~ .[(+![] + +![]) * (+![] + +![]) * (+![] + +![] + +![]) * (+![] + +![] + +![]) * (+![] + +![] + +![])] ~ .[(+![] + +![]) * (+![] + +![]) * (+![] + +![] + +![]) * (+![] + +![] + +![]) * (+![] + +![] + +![])] ~ .[(+![] + +![] + +![]) * ((+![] + +![]) * (+![] + +![]) * (+![] + +![] + +![]) * (+![] + +![] + +![]) + +![])] ~ .[(+![] + +![]) * (+![] + +![]) * ((+![] + +![]) * (+![] + +![] + +![] + +![] + +![]) + +![])] ~ .[(+![] + +![]) * (+![] + +![]) * (+![] + +![]) * (+![] + +![]) * (+![] + +![])] ~ .[((+![] + +![]) * (+![] + +![] + +![]) + +![]) * ((+![] + +![]) * (+![] + +![]) * (+![] + +![]) * (+![] + +![]) + +![])] ~ .[(+![] + +![] + +![]) * ((+![] + +![]) * (+![] + +![]) * (+![] + +![] + +![]) * (+![] + +![] + +![]) + +![])] ~ .[(+![] + +![]) * (+![] + +![] + +![]) * ((+![] + +![]) * (+![] + +![] + +![]) * (+![] + +![] + +![]) + +![])] ~ .[(+![] + +![]) * (+![] + +![]) * (+![] + +![] + +![]) * (+![] + +![] + +![]) * (+![] + +![] + +![])] ~ .[(+![] + +![]) * (+![] + +![]) * (+![] + +![] + +![] + +![] + +![]) * (+![] + +![] + +![] + +![] + +![])] ~ .[(+![] + +![] + +![]) * ((+![] + +![]) * (+![] + +![] + +![] + +![] + +![]) + +![])] ~ .[(+![] + +![]) * (+![] + +![]) * (+![] + +![]) * (+![] + +![]) * (+![] + +![])] ~ ::(.[(+![] + +![]) * ((+![] + +![]) * (+![] + +![] + +![]) * (+![] + +![] + +![]) + +![])] ~ .[(+![] + +![] + +![]) * (+![] + +![] + +![]) * ((+![] + +![]) * (+![] + +![] + +![] + +![] + +![]) + +![])] ~ .[(+![] + +![]) * (+![] + +![]) * (+![] + +![]) * ((+![] + +![]) * (+![] + +![]) * (+![] + +![] + +![]) + +![])] ~ .[(+![] + +![]) * (+![] + +![] + +![]) * ((+![] + +![]) * (+![] + +![] + +![]) * (+![] + +![] + +![]) + +![])])((+![] + +![] + +![] + +![] + +![]) * (+![] + +![] + +![] + +![] + +![]) * ((+![] + +![]) * ((+![] + +![]) * ((+![] + +![]) * (+![] + +![] + +![] + +![] + +![]) + +![]) + +![]) + +![]) * ((+![] + +![]) * (+![] + +![]) * (+![] + +![] + +![]) * (+![] + +![] + +![]) * (+![] + +![] + +![]) + +![])) ~ .[(+![] + +![]) * ((+![] + +![]) * (+![] + +![]) * (+![] + +![]) * (+![] + +![]) + +![])]);

Wrapping Up

Raku is quite a large language with an extensive set of features. These can be combined in some very interesting ways! Here, using a combination of type coercion, string bitwise operators, regexen, sequences, indirect symbol lookup, and a loophole with &EVAL, we were able to be naughty Grinches again this year and port JSFuck from JavaScript. If you’re tempted to say something is impossible to write in Raku, hold your horses; it may very well be possible to do with the right tools.

Raku Advent Calendar: Day 23 – A Raku Advent Helper

Published by Tom Browder (@tbrowder) on 2019-12-23T00:01:00

Introduction

I have been writing Raku Advent articles annually since 2016, and it’s always been a struggle for me to get a reliable transformation of my source file into the Raku Advent WordPress (WP) website without something getting changed by WP. Then, the menus are terrible and editing can be troublesome. In this article I hope to show how the situation can be improved.

Background

The great name change to Raku this year unfortunately happened late in the year and there was not a lot of time to get a new Raku Advent website ready. Consequently, theme selection and tweaking, confusion over the actual Raku Advent website link, and unfortunate article cancellations were wrinkles in the normally smoother process. However, we plan to improve the website before the 2020 Advent season, and also get commitments earlier with concrete drafts available sooner. In the meantime, in this hastily prepared stand-in article, I will go into a bit of detail on some help we hope to offer.

Article creation

Since my first experience with WordPress, I have found these things that make it awkward for me to use WP:

I’m sure most of my problems with WP are self-induced, but I do prefer a more TeX-like document production work flow.

Prior years

In past years I’ve created the articles in Gihub-flavored markdown, manually (with the assistance of my Emacs editor) converted each paragraph to single, long lines, and then posted it in a Github gist. After that, I used the tool p6advent-md2html.p6, developed by @zoffix [Note 2] and modified by @SimonProctor, to extract the html from Github’s representation of the markdown which results in a nice highlighting of code blocks. Finally, that html is copied and pasted into WP and a publishing schedule set up. That original process is outlined here:

  1. Write the post in Github-favored Markdown text
  2. Collapse each paragraph to one long line
  3. Paste the source into a Github gist
  4. Use the existing Advent tool to extract the resulting html representation to one’s local computer
  5. Copy the html and paste it into the blank, html view of the selected WP editor
  6. View the finished product and check for errors

If errors are found:

  1. Correct the errors in the WP editor

OR

  1. Correct the errors in the source
  2. Repeat steps 2 through 6 again

That process is not so bad the first time through it, but when, inevitably, errors are found, one has the choice of either manually editing it on WP or modifying the source and going through the entire process again! Neither choice is very good.

2019’s Advent goal: reduce WP pain

I decided this year to help my article-creation situation so I created a Raku tool to eliminate some of the problems. It’s available to the public as of today:

$ zef install RakuAdvent::WordPress

That module provides the tool make-wp-input. So my new steps as of this year:

  1. Write the post in raw html
  2. Run my new Advent tool (make-wp-input) to format the source into WP-acceptable html
  3. Copy the html and paste it into the blank, html view of the selected WP editor
  4. View the finished product and check and correct for errors

If errors are found:

  1. Correct the errors in the WP editor

OR, preferably,

  1. Correct the errors in the source
  2. Repeat steps 2 through 4 again

Thus, in my new process, I’ve eliminated a couple of steps, but I still have to copy/paste my clean WP source into the WordPress editor—but that’s because I have not taken advantage of the available APIs from WordPress and Github to do the drudge work.

However, in spite of other limitations, the new tool has been a huge help in easing the use of live code examples in an article. In my sandbox where I write my article, I create the code samples in their own files and then add the

<!-- insert file-name lang -->

lines as needed in the location needed. That way, I can edit the live code and test it to make sure it works, but don’t have to change the source using that code.

Tips for Raku authors

Here are some ideas I’ve found helpful while developing articles for the Raku Advent:

Wish lists

Here are some things I hope to do with make-wp-input in the New Year:

  1. Convert html source to Github-flavored markdown
  2. Handle html tables
  3. Allow paragraphs in the source html to be recognized by either blank lines above and below the text or a line with a closing tag on the line before the text or an opening tag on the line following the text
  4. Use Github’s APIs [Ref. 2] to manipulate markdown source to a Github gist and get html results back from it
  5. Use WordPress’s APIs [Ref. 3] to manipulate one’s article on WordPress (including setting or updating the publication schedule)

And here are some things I hope the community can do (or at least agree upon) for the Raku Advent website:

  1. Improve the theme and code styling.
  2. Use the old Perl 6 Advent theme?
  3. Sign up for article slots earlier in the year, and start the article (at least in skeleton form) as a scheduled one on the Raku Advent website.

Summary

This year has seen a lot of changes in the Raku community, especially with the name change, and not all are done yet. One area that still needs work is improving the new Raku Advent website. We also hope to make it easier to create and post Raku Advent articles as well as get more participation. Note the 2020 schedule is open now, so you can get your slot early and avoid last minute shopping, er, Raku Adventing!

I ❤ Raku! 😊

Merry Christmas and a Happy, Blessed New Year to all!


APPENDIX


Notes

  1. I have filed an issue with WordPress to help with time zone identification in the scheduling calendar.
  2. Names preceded by @ are IRC or Github aliases.

References

  1. wkhtmltopdf (available as a Debian package)
  2. Github API
  3. WordPress API

Raku modules used

Raku Advent Calendar: Day 22: Off Course

Published by arnesom on 2019-12-22T01:01:00

You may not have heard about my Perl 6 Courses, and I don’t blame you.

It has been quite a journey.

It started in September 2018 with my Perl6 In 45+45 Minutes introduction to Perl 6 at the Nordic Perl Workshop in Oslo. The very first time a held a presentation at a conference…

I got positive feedback, and wondered if I could build on it. The idea of a full blown course matured, and I started working on the accompanying textbook first.

The book and course is meant as an introduction to Raku, for people already familiar with programming.

I pitched the course to PerlCon 2019 in Riga, and they accepted it. The organiser asked me to promote it, and the result was my Perl 6 blog Perl 6 Musings (at the absolutely fantastic address «perl6.eu»).

Unfortunately that didn’t work out, and the course was cancelled due to too few participants.

  Beginning Raku, 1. Edition (December 2019)  

Pages: 370

File size: ~ 11 Mbyte (pdf)


arnesom.github.io/Beginning-v1.00.pdf

I am giving away this first version of the book for free. I do reserve the right to print the book and sell it. You are free to distribute the pdf file or print it. You are also free to distribute printed copies, but you may not get paid for it.

Feel free to use the code samples, either as they are or as inspiration for your own work. Atribution would be nice, but isn’t required.

I will be grateful for feedback, and do so at the Github page for the book – or by email to the address shown in the book. I intend to publish a revised version of the book if I receive feedback that warrants an update.

Not Complete

The next course, «Advanced Raku» continues where this one ends. As the book is meant as a reference, I have chosen to make a combined book for both courses, called «Raku Explained». The second half (the «Advanced Raku» part) is unfinished, but I have published a preliminary Table of Contents and Index so that you can see what the whole book intends to cover.


  Raku Explained, v0.01 (December 2019)  

Pages: 30 (Table of Contents & Index only)

File size: ~ 5 Mbyte (pdf)


arnesom.github.io/Explained-v0.01.pdf

I am also interested in feedback on the topics in the second part (chapter 18 – 32).

Raku Advent Calendar: Day 21: Searching for a Red gift

Published by SmokeMachine on 2019-12-21T00:00:00

Alabaster Snowball, the elf, was searching for a gift for the person he had drawn on the North Pole’s Secret Santa. He had the great honour to draw Santa! What to give for the one who gives everyone’s presents? So he was searching on the internet for some keywords he knew Santa would like:

Wait a minute! Is Red going to :api<2>?!! Alabaster Snowball has already read about that ORM for Raku. But it seems this new :api<2> version is taking it to the next level.

That’s it! I’ll give Santa a Red:api<2> PoC as gift! I know he has been playing with Raku, and I think it would be great to change all that collection of SQL strings on the NiceList model to a well made set of ORM classes.

Reading the documentation, Snowball learned that it would be very easy to create it’s first model:

use Red:api<2>;

unit model Child;

has UInt $!id              is id;
has Str  $.name            is column;
has Str  $.country         is column;

He started using Red:api<2> and creating a new model that represents a table child with 3 columns (id, name and country). As easy as that.

Alabaster could now just connect into a database, create the table, and start inserting children:

use Red:api<2>;
red-defaults default => database "SQLite";

Child.^create-table: :unless-exists;

Child.^create: :name<Fernanda>, :country<England> ;
Child.^create: :name<Sophia>,   :country<England> ;
Child.^create: :name<Dudu>,     :country<Scotland>;
Child.^create: :name<Rafinha>,  :country<Scotland>;
Child.^create: :name<Maricota>, :country<Brazil>  ;
Child.^create: :name<Lulu>,     :country<Brazil>  ;

And to list all children created:

.say for Child.^all.sort: *.name;

And that would run this query:

SELECT
   child.id, child.name, child.country 
FROM
   child
ORDER BY
   child.name

And prints:

Child.new(name => "Dudu", country => "Scotland")
Child.new(name => "Fernanda", country => "England")
Child.new(name => "Lulu", country => "Brazil")
Child.new(name => "Maricota", country => "Brazil")
Child.new(name => "Rafinha", country => "Scotland")
Child.new(name => "Sophia", country => "England")

If it’s needed, Santa can classify children by country:

my %by-country := Child.^all.classify: *.country;

And to discover what countries have children registered:

say %by-country.keys;

That would run:

SELECT
   DISTINCT(child.country) as "data_1"
FROM
   child

And that would return:

(England Scotland Brazil)

If he needs to get all children from England:

.say for %by-country<England>;

That would run:

SELECT
   child.id, child.name, child.country 
FROM
   child
WHERE
   child.country = ?

-- BIND: ["England"]

That would return:

Child.new(name => "Fernanda", country => "England")
Child.new(name => "Sophia", country => "England")

It’s working great! How about storing the gifts? Is there a way to store what a child asked by year?

# Gift.pm6
use Red:api<2>;

unit model Gift;

has UInt $!id            is serial;
has Str  $.name          is column{ :unique };

has      @.asked-by-year is relationship( *.gift-id, :model<ChildAskedOnYear> );

method child-asked-on-year(UInt $year = Date.today.year) {
    @!asked-by-year.grep(*.year == $year)
}

method asked-by(UInt $year) {
    self.child-asked-on-year(|($_ with $year)).map: *.child
} 
# Child.pm6
use Red:api<2>;

unit model Child;

has UInt $!id              is id;
has Str  $.name            is column;
has Str  $.country         is column;

has      @.asked-by-year   is relationship( *.child-id, :model<ChildAskedOnYear> );

method asked(UInt $year = Date.today.year) {
    @!asked-by-year.grep: *.year == $year
}
# ChildAskedOnYear.pm6
use Red:api<2>;

unit model ChildAskedOnYear;

has UInt $!id       is serial;
has UInt $.year     is column = Date.today.year;
has UInt $!child-id is referencing(*.id, :model<Child>);
has UInt $!gift-id  is referencing(*.id, :model<Gift>);

has      $.child    is relationship( *.child-id, :model<Child> );
has      $.gift     is relationship( *.gift-id,  :model<Gift>  );

Alabaster Snowball thought that way he could get all information he would need. Creating new gifts is easy!

for <doll ball car pokemon> -> $name {
    Gift.^create: :$name;
}

How about searching? Alabaster Snowball writes a new line:

.say for Gift.^all

And it returns all the gifts. But what if we want only the gifts that end with “ll”?

.say for Gift.^all.grep: *.name.ends-with: "ll"

That will run a query like:

SELECT
   gift.id, gift.name 
FROM
   gift
WHERE
   gift.name like '%ll'

Snowball wondered if it is possible to find what a child has asked:

.say for Child.^find(:name<Fernanda>).asked.map: *.gift

That runs:

SELECT
   child_asked_on_year_gift.id, child_asked_on_year_gift.name 
FROM
   child_asked_on_year
    LEFT JOIN gift as child_asked_on_year_gift ON child_asked_on_year.gift_id = child_asked_on_year_gift.id
WHERE
   child_asked_on_year.child_id = ? AND child_asked_on_year.year = 2019

And what if we want to know the last year’s gift?

.say for Child.^find(:name<Fernanda>).asked(2018).map: *.gift
SELECT
   child_asked_on_year_gift.id, child_asked_on_year_gift.name 
FROM
   child_asked_on_year
    LEFT JOIN gift as child_asked_on_year_gift ON child_asked_on_year.gift_id = child_asked_on_year_gift.id
WHERE
   child_asked_on_year.child_id = ? AND child_asked_on_year.year = '2018'

How do we know how many of each gift should be built?

say ChildAskedOnYear.^all.map(*.gift.name).Bag
SELECT
   child_asked_on_year_gift.name as "data_1", COUNT('*') as "data_2"
FROM
   child_asked_on_year
    LEFT JOIN gift as child_asked_on_year_gift ON child_asked_on_year.gift_id = child_asked_on_year_gift.id
GROUP BY
   child_asked_on_year_gift.name

The documentation for Red is on https://fco.github.io/Red/ and some examples used here can be found on https://github.com/FCO/Red/blob/join/examples/xmas/index.p6

Jo Christian Oterhals: By the way, you could replace … * with Inf or the unicode infinity symbol ∞ to make it more…

Published by Jo Christian Oterhals on 2019-11-24T19:25:11

By the way, you could replace … * with Inf or the unicode infinity symbol ∞ to make it more readable, i.e.

my @a = 1, 1, * + * … ∞;

— — or — —

my @a = 1, 1, * + * … Inf;

Jo Christian Oterhals: As I understand this, * + * … * means the following:

Published by Jo Christian Oterhals on 2019-11-24T10:20:11

As I understand this, * + * … * means the following:

First— * + * sums the two previous elements in the list. … * tells this to do this an infinite number of times; i.e.

1, 1, (1 + 1)

1, 1, 2, (1 + 2)

1, 1, 2, 3, (2 + 3)

1, 1, 2, 3, 5, (3 + 5)

1, 1, 2, 3, 5, 8, (5 + 8), etc.

… three dots means that it does it lazy, i.e. that it does not generate an element before you call it. This can be good for large lists that are computationally heavy.

my Timotimo \this: Introducing: The Heap Snapshot UI

Published by Timo Paulssen on 2019-10-25T23:12:36

Introducing: The Heap Snapshot UI

Hello everyone! In the last report I said that just a little bit of work on the heap snapshot portion of the UI should result in a useful tool.

Introducing: The Heap Snapshot UI
Photo by Sticker Mule / Unsplash

Here's my report for the first useful pieces of the Heap Snapshot UI!

Last time you already saw the graphs showing how the number of instances of a given type or frame grow and shrink over the course of multiple snapshots, and how new snapshots can be requested from the UI.

The latter now looks a little bit different:

Introducing: The Heap Snapshot UI

Each snapshot now has a little button for itself, they are in one line instead of each snapshot having its own line, and the progress bar has been replaced with a percentage and a little "spinner".

There are multiple ways to get started navigating the heap snapshot. Everything is reachable from the "Root" object (this is the norm for reachability-based garbage collection schemes). You can just click through from there and see what you can find.

Another way is to look at the Type & Frame Lists, which show every type or frame along with the number of instances that exist in the heap snapshot, and the total size taken up by those objects.

Type & Frame Lists

Introducing: The Heap Snapshot UI

Clicking on a type, or the name or filename of a frame leads you to a list of all objects of that type, all frames with the given name, or all frames from the given file. They are grouped by size, and each object shows up as a little button with the ID:

Introducing: The Heap Snapshot UI

Clicking any of these buttons leads you to the Explorer.

Explorer

Here's a screenshot of the explorer to give you an idea of how the parts go together that I explain next:

Introducing: The Heap Snapshot UI

The explorer is split into two identical panels, which allows you to compare two objects, or to explore in multiple directions from one given object.

There's an "Arrow to the Right" button on the left pane and an "Arrow to the Left" button on the right pane. These buttons make the other pane show the same object that the one pane currently shows.

On the left of each pane there's a "Path" display. Clicking the "Path" button in the explorer will calculate the shortest path to reach the object from the root. This is useful when you've got an object that you would expect to have already been deleted by the garbage collector, but for some reason is still around. The path can give the critical hint to figure out why it's still around. Maybe one phase of the program has ended, but something is still holding on to a cache that was put in as an optimization, and that still has your object in it? That cache in question would be on the path for your object.

The other half of each panel shows information about the object: Displayed at the very top is whether it is an object, a type object, an STable, or a frame.

Below that there is an input field where you can enter any ID belonging to a Collectable (the general term encompassing types, type objects, stables, and frames) to have a look.

The "Kind" field needs to have the number values replaced with human-readable text, but it's not the most interesting thing anyway.

The "Size" of the Collectable is split into two parts. One is the fixed size that every instance of the given type has. The other is any extra data an instance of this type may have attached to it, that's not a Collectable itself. This would be the case for arrays and hashes, as well as buffers and many "internal" objects.

Finally, the "References" field shows how many Collectables are referred to by the Collectable in question (outgoing references) and how many Collectables reference this object in question.

Below that there are two buttons, Path and Network. The former was explained further above, and the latter will get its own little section in this blog post.

Finally, the bottom of the panel is dedicated to a list of all references - outgoing or incoming - grouped by what the reference means, and what type it references.

Introducing: The Heap Snapshot UI

In this example you see that the frame of the function display from elementary2d.p6 on line 87 references a couple of variables ($_, $tv, &inv), the frame that called this frame (step), an outer frame (MAIN), and a code object. The right pane shows the incoming references. For incoming references, the name of the reference isn't available (yet), but you can see that 7 different objects are holding a reference to this frame.

Network View

The newest part of the heap snapshot UI is the Network View. It allows the user to get a "bird's eye" view of many objects and their relations to each other.

Here's a screenshot of the network view in action:

Introducing: The Heap Snapshot UI

The network view is split into two panes. The pane on the left lists all types present in the network currently. It allows you to give every type a different symbol, a different color, or optionally make it invisible. In addition, it shows how many of each type are currently in the network display.

The right pane shows the objects, sorted by how far they are from the root (object 0, the one in Layer 0, with the frog icon).

Each object has one three-piece button. On the left of the button is the icon representing the type, in the middle is the object ID for this particular object, and on the right is an icon for the "relation" this object has to the "selected" object:

This view was generated for object 46011 (in layer 4, with a hamburger as the icon). This object gets the little "map marker pin" icon to show that it's the "center" of the network. In layers for distances 3, 2, and 1 there is one object each with a little icon showing two map marker pins connected with a squiggly line. This means that the object is part of the shortest path to the root. The third kind of icon is an arrow pointing from the left into a square that's on the right. Those are objects that refer to the selected object.

There is also an icon that's the same but the arrow goes outwards from the square instead of inwards. Those are objects that are referenced by the selected object. However, there is currently no button to have every object referenced by the selected object put into the network view. This is one of the next steps I'll be working on.

Customizing the colors and visibility of different types can give you a view like this:

Introducing: The Heap Snapshot UI

And here's a view with more objects in it:

Introducing: The Heap Snapshot UI

Interesting observations from this image:

Next Steps

You have no doubt noticed that the buttons for collectables are very different between the network view and the type/frame lists and the explorer. The reason for that is that I only just started with the network view and wanted to display more info for each collectable (namely the icons to the left and right) and wanted them to look nicer. In the explorer there are sometimes thousands of objects in the reference list, and having big buttons like in the network view could be difficult to work with. There'll probably have to be a solution for that, or maybe it'll just work out fine in real-world use cases.

On the other hand, I want the colors and icons for types to be available everywhere, so that it's easier to spot common patterns across different views and to mark things you're interested in so they stand out in lists of many objects. I was also thinking of a "bookmark this object" feature for similar purposes.

Before most of that, the network viewer will have to become "navigable", i.e. clicking on an object should put it in the center, grab the path to the root, grab incoming references, etc.

There also need to be ways to handle references you're not (or no longer) interested in, especially when you come across an object that has thousands of them.

But until then, all of this should already be very useful!

Here's the section about the heap snapshot profiler from the original grant proposal:

Looking at the list, it seems like the majority of intended features are already available or will be very soon!

Easier Installation

Until now the user had to download nodejs and npm along with a whole load of javascript libraries in order to compile and bundle the javascript code that powers the frontend of moarperf.

Fortunately, it was relatively easy to get travis-ci to do the work automatically and upload a package with the finished javascript code and the backend code to github.

You can now visit the releases page on github to grab a tarball with all the files you need! Just install all backend dependencies with zef install --deps-only . and run service.p6!

And with that I'm already done for this report!

It looks like the heap snapshot portion of the grant is quite a bit smaller than the profiler part, although a lot of work happened in moarvm rather than the UI. I'm glad to see rapid progress on this.

I hope you enjoyed this quick look at the newest pieces of moarperf!
  - Timo

Weekly changes in and around Perl 6: 2019.42 Question

Published by liztormato on 2019-10-21T14:14:51

The Perl 6 Weekly has been renamed to the Rakudo Weekly and can be found at: https://rakudoweekly.blog.

Thank you for visiting the Perl 6 Weekly the past years. It was a blast!

Please adjust your RSS readers, email notifications. Thank you!

Hope to see you there!

Weekly changes in and around Perl 6: 2019.41 New Wineskins

Published by liztormato on 2019-10-15T16:46:05

Larry Wall emerged, almost like a deus ex machina, to give his approval of changing the name of the Perl 6 Programming Language to “Raku”. This stirred up quite some reactions on the interwebs:

Also see quite some reactions on Twitter below.

London Perl Workshop

Next weekend, on Saturday 19 October, it’s time for the London Perl Workshop again. The following presentations with Perl 6 content appear to have been planned:

An excellent opportunity to learn more about Perl 6 / Raku. And possibly the last time for a pre-Brexit visit to the UK!

Even More Video Tutorials

Yanzhan Yang, a serial tech video uploader, yet again has posted more Perl 6 introductory videos on YouTube:

Getting more amazing every week!

Andrew Shitov

It was a stressful week for Andrew Shitov: at first he was worried about the future of his Perl 6 compiler book. Then, after Larry Wall showed his approval of the renaming, he suggested having a RakuCon in May 2020 on Cyprus at the venue previously intended for the 2020 Perl Conference. Quickly followed by a blog post explaining how he sees the future of Raku, including building a new, faster Raku compiler (/r/perl6 comments). Followed by the publication of the first Raku book: Using Raku and making that available for free download (/r/perl6 comments). And to top it off, produced a little tutorial about the difference between is rw and is raw. Yours truly can only hope that future weeks will be less stressful.

Perl Weekly Challenge #29

Blog posts with Perl 6 solutions for Challenge #29:

Challenge #30 is up for your perusal.

Questions about Perl 6

Meanwhile on Twitter

Meanwhile on Facebook

Thanks to the quickly unrepairable actions of a certain individual outside of the Perl 6 community, the Perl 6 Facebook group has changed all of its URLs and/or is only accessible to people with Facebook logins. And since it is the intention of changing back to the original URLs (which will most likely take a month thanks to Facebook policies), it doesn’t seem to make sense to put deep links here now.

So, if you’re interested in happenings on Facebook, check out the Perl 6 Group, and navigate from there.

Meanwhile on perl6-users

Perl 6 in comments

Perl 6 Modules

New modules:

Updated modules:

Winding Down

The past year has been very stressful for yours truly. And the past week even more so. Some stress will remain in the near future, but it looks like stress levels will be allowed to go down in the further future, with “Perl” and “Raku” each going their own way.

This is the last Perl 6 Weekly that yours truly will write. Next week I will just announce the location of the new Rakudo Weekly blog here.

Why “Rakudo” and not “Raku”, you might ask? Well, originally the “Perl 6 Weekly” was about all implementations of Perl 6. But for the past 4 years, it has effectively been only about the “Rakudo” implementation. Now that “Perl 6” is being renamed to “Raku”, it seems like a good opportunity to not squat on the language name in a blog that is effectively dedicated to a single implementation of the Raku Programming Language.

So see you next week for directions to the new blog!

Weekly changes in and around Perl 6: 2019.40 Quick Syntaxing

Published by liztormato on 2019-10-07T14:23:14

JJ Merelo is the proud writer of the latest Perl 6 book: Perl 6 Quick Syntax Reference: A Pocket Guide to the Language, the Core Modules, and the Community. A book packed with useful information and a must-have for any developer new to Perl 6. Highly recommended! (Safari).

Sub as method

Sterling Hanenkamp shows how to use a sub as a method (/r/perl6 comments).

Perspective and FALLBACK

Greg Donald has published two blog posts in the past week: a personal one about his perspective as an outsider, and a technical one about adding an attribute to a class at runtime.

Even More Video Tutorials

Yanzhan Yang, a serial tech video uploader, yet again has posted more Perl 6 introductory videos on YouTube:

Getting more amazing every week!

Perl Weekly Challenge #28

Blog posts with Perl 6 solutions for Challenge #28:

Challenge #29 is up for your perusal.

Core Developments

Questions about Perl 6

Meanwhile on Twitter

Meanwhile on Facebook

Perl 6 in comments

Perl 6 Modules

New modules:

Updated modules:

Winding Down

A quiet week, with one more week to go on voting on the rename of Perl 6. See you all next week with more news about the Perl 6 Programming Language!

Weekly changes in and around Perl 6: 2019.39 With A Lump

Published by liztormato on 2019-09-30T18:56:54

Jonathan Worthington explains why he got a lump in his throat when he approved changing the name of “Perl 6” to “Raku”. Feelings that yours truly (like many others) only can share. (/r/perl, /r/perl6 comments). Andrew Shitov also shared his feelings about renaming the Perl 6 Programming Language (/r/perl6, Facebook comments).

Video Tutorials

Yanzhan Yang, a serial tech video uploader, has posted some more Perl 6 introductory videos on YouTube:

Amazing!

Faster Continuous Integration

Tony O’Dell has created a walk through on how to do Continuous Integration with Circle CI and Travis CI, using a Docker image.

The power of base

Shred_Alert describes the magic of base.

Perl Weekly Challenge #27

Blog posts with Perl 6 solutions for Challenge #27:

Simon Proctor is the champion of week 27! And as usual, Challenge #28 is up for your perusal.

Core Developments

Questions about Perl 6

Meanwhile on Twitter

Meanwhile on Facebook

Meanwhile on perl6-users

Perl 6 in comments

Perl 6 Modules

New modules:

Updated modules:

Winding Down

It appears there will be no Rakudo 2019.09 compiler release. Which makes sense since it’s almost October. Check out the Perl 6 Weekly next week for more news about this and many other things!

Weekly changes in and around Perl 6: 2019.38 For Else Itch

Published by liztormato on 2019-09-23T15:27:14

Damian Conway had an itch, and he scratched it in “Itch.scratch()“. An extensive treatise on how to extend the Perl 6 Programming Language, giving the for loop an else block to be executed only if no iterations were done in the for loop. In less than 25 lines! (Reddit comments).

Video Tutorials

Yanzhan Yang has posted a number of Perl 6 introductory videos on YouTube, maybe the first of many to come:

Yanzhan Yang appears to be a serial tech video uploader (Reddit comments).

Atomic Units

Steve Roe explains his thoughts on seamlessly supporting atomic units in the Physics::Measure Perl 6 module.

Perl 6 at LPW

These presentations about Perl 6 are currently planned to be given at the next London Perl Workshop:

It’s not too late to submit your presentation!

Perl Foundation News

The Perl Foundation is nominating Pete Krawczyk as Treasurer. Many thanks to Dan Wright for having filled this position for so many years.

And there is a grant proposal for curating the Perl 6 documentation, a continuation of earlier work by JJ Merelo.

Please leave your comments, if you have any of course!

What’s in a name?

Sven Gregori investigated several open source projects with naming issues, Perl 6 just being one of them (/r/perl, /r/perl6 comments).

Perl Weekly Challenge #26

Blog posts with Perl 6 solutions for Challenge #26:

Yet Ebreo is the champion of week 25! And as usual, Challenge #27 is up for your perusal.

Core Developments

Questions about Perl 6

Meanwhile on Twitter

Meanwhile on Facebook

Meanwhile on perl6-users

Perl 6 in comments

Perl 6 Modules

New modules (some of them were missed previously because of not having been uploaded to CPAN):

Updated modules:

Winding Down

A quiet week yet again, while work has started on finalizing the next Rakudo compiler release. Hope to be able to report on that next week. Until then, program safely and have fun!

p6steve: Atomic Units?

Published by p6steve on 2019-09-17T20:49:39

One of the most exciting parts of blogging about and hacking on perl6* is that there’s a community out there and there’s (always) more than one way to do it!

For Physics::Measure I have been working on a design for a ‘nano-slang’ that can provide a shortcut for the usual new Class declaration… quite long winded for my target audience of physics undergraduates and high school students.

#Instances the usual way

my Unit $u .=new(name => ‘m’, unitsof => ‘Distance’); #Unit m

my Distance $a .=new(value => 10, units => $u);           #Distance 10 m

So, duly inspired by Rakudo perl6 going atomic ⚛ applying unicode to some threading constructs, I started with the notion of the ‘libra’ operator ♎ as shorthand to declare and load Measure instances.

#Introducing the libra operator ♎ as shorthand to declare and load

my $b ♎ ’50 m’;   #Distance 50 m

$b ♎ ‘3 yards’;     #Distance 3 yards

As you can see, the gap created between ♎ and ; is a slang zone that can consume strings and numeric literals. Here’s something a bit more elaborate:

#Normalization with the .norm method

my $p ♎ ’27 kg m^2 / s^3′;   #Power 27 kg m^2 / s^3

$p .= norm;                              #Power 27 W

A few design ideas drew me in this direction:

# Resistance

[‘Ω’, ‘Ohm:s’,],       ’kg m^2 / A^2 s^3′,

[‘kilohm:s’,],          ’kilo Ohm’,

[‘megohm:s’,],       ’mega Ohm’,

HOWEVER!!

Others have proposed a much more direct approach to generate and combine Measure objects – by the use of a simple postfix syntax – thank you!

Something like:

say 500g; # –> Weight.new(grams => 500, prefix => “”)

say 2kg;  # –> Weight.new(grams => 2000, prefix => “kg”)

Watch this space! Or even better zef Physics::Measure and give it a try…

~p6steve

* soon to be Rakudo?!

my Timotimo \this: Progressing with progress.

Published by Timo Paulssen on 2019-09-12T19:50:18

Progressing with progress.

It has been a while since the last progress report, hasn't it?

Over the last few months I've been focusing on the MoarVM Heap Snapshot Profiler. The new format that I explained in the last post, "Intermediate Progress Report: Heap Snapshots", is available in the master branch of MoarVM, and it has learned a few new tricks, too.

The first thing I usually did when opening a Heap Snapshot in the heapanalyzer (the older command-line based one) was to select a Snapshot, ask for the summary, and then for the top objects by size, top objects by count, top frames by size, and/or top frames by count to see if anything immediately catches my eye. In order to make more sense of the results, I would repeat those commands for one or more other Snapshots.

Snapshot  Heap Size          Objects  Type Objects  STables  Frames  References  
========  =================  =======  ============  =======  ======  ==========  
0         46,229,818 bytes   331,212  686           687      1,285   1,146,426   
25        63,471,658 bytes   475,587  995           996      2,832   1,889,612   
50        82,407,275 bytes   625,958  1,320         1,321    6,176   2,741,066   
75        97,860,712 bytes   754,075  1,415         1,416    6,967   3,436,141   
100       113,398,840 bytes  883,405  1,507         1,508    7,837   4,187,184   

Snapshot  Heap Size          Objects    Type Objects  STables  Frames  References  
========  =================  =========  ============  =======  ======  ==========  
125       130,799,241 bytes  1,028,928  1,631         1,632    9,254   5,036,284   
150       145,781,617 bytes  1,155,887  1,684         1,685    9,774   5,809,084   
175       162,018,588 bytes  1,293,439  1,791         1,792    10,887  6,602,449 

Realizing that the most common use case should be simple to achieve, I first implemented a command summary all and later a command summary every 10 to get the heapanalyzer to give the summaries of multiple Snapshots at once, and to be able to get summaries (relatively) quickly even if there's multiple hundreds of snapshots in one file.

Sadly, this still requires the parser to go through the entire file to do the counting and adding up. That's obviously not optimal, even though this is an Embarrassingly Parallel task, and it can use every CPU core in the machine you have, it's still a whole lot of work just for the summary.

For this reason I decided to shift the responsibility for this task to MoarVM itself, to be done while the snapshot is taken. In order to record everything that goes into the Snapshot, MoarVM already differentiates between Object, Type Object, STable, and Frame, and it stores all references anyway. I figured it shouldn't have a performance impact to just add up the numbers and make them available in the file.

The result is that the summary table as shown further above is available only milliseconds after loading the heap snapshot file, rather than after an explicit request and sometimes a lengthy wait period.

The next step was to see if top objects by size and friends could be made faster in a similar way.

I decided that adding an optional "statistics collection" feature inside of MoarVM's heap snapshot profiler would be worthwhile. If it turns out that the performance impact of summing up sizes and counts on a per-type and per-frame basis makes capturing a snapshot too slow, it could be turned off.

Frontend work

> snapshot 50
Loading that snapshot. Carry on...
> top frames by size
Wait a moment, while I finish loading the snapshot...

Name                                  Total Bytes    
====================================  =============  
finish_code_object (World.nqp:2532)   201,960 bytes  
moarop_mapper (QAST.nqp:1764)         136,512 bytes  
!protoregex (QRegex.nqp:1625)         71,760 bytes   
new_type (Metamodel.nqp:1345)         40,704 bytes   
statement (Perl6-Grammar.nqp:951)     35,640 bytes   
termish (Perl6-Grammar.nqp:3641)      34,720 bytes   
<anon> (Perl6-BOOTSTRAP.c.nqp:1382)   29,960 bytes   
EXPR (Perl6-Grammar.nqp:3677)         27,200 bytes   
<mainline> (Perl6-BOOTSTRAP.c.nqp:1)  26,496 bytes   
<mainline> (NQPCORE.setting:1)        25,896 bytes   
EXPR (NQPHLL.nqp:1186)                25,760 bytes   
<anon> (<null>:1)                     25,272 bytes   
declarator (Perl6-Grammar.nqp:2189)   23,520 bytes   
<anon> (<null>:1)                     22,464 bytes   
<anon> (<null>:1)                     22,464 bytes   

Showing the top objects or frame for a single snapshot is fairly straight-forward in the commandline based UI, but how would you display how a type or frame develops its value across many snapshots?

Instead of figuring out the best way to display this data in the commandline, I switched focus to the Moarperf Web Frontend. The most obvious way to display data like this is a Line Graph, I believe. So that's what we have now!

Progressing with progress.

And of course you also get to see the data from each snapshot's Summary in graph format:

Progressing with progress.

And now for the reason behind this blog post's Title.

Progress Notifications

Using Jonathan's module Concurrent::Progress (with a slight modification) I sprinkled the code to parse a snapshot with matching calls of .increment-target and .increment. The resulting progress reports (throttled to at most one per second) are then forwarded to the browser via the WebSocket connection that already delivers many other bits of data.

The result can be seen in this tiny screencast:

Progressing with progress.

The recording is rather choppy because the heapanalyzer code was using every last drop of performance out of my CPU while it was trying to capture my screen.

There's obviously a lot still missing from the heap snapshot analyzer frontend GUI, but I feel like this is a good start, and even provides useful features already. The graphs for the summary data are nicer to read than the table in the commandline UI, and it's only in this UI that you can get a graphical representation of the "highscore" lists.

I think a lot of the remaining features will already be useful after just the initial pieces are in place, so a little work should go a long way.

Bits and Bobs

I didn't spend the whole time between the last progress report and now to work directly on the features shown here. Apart from Life Intervening™, I worked on fixing many frustrating bugs related to both of the profilers in MoarVM. I added a small subsystem I call VMEvents that allows user code to be notified when GC runs happen and other interesting bits from inside MoarVM itself. And of course I've been assisting other developers by answering questions and looking over their contributions. And of course there's the occasional video-game-development related experiment, for example with the GTK Live Coding Tool.

Finally, here's a nice little screencap of that same tool displaying a hilbert curve:

Progressing with progress.

That's already everything I have for this time. A lot has (had to) happen behind the scenes to get to this point, but now there was finally something to look at (and touch, if you grab the source code and go through the needlessly complicated build process yourself).

Thank you for reading and I hope to see you in the next one!
  - Timo

Jo Christian Oterhals: You’re right.

Published by Jo Christian Oterhals on 2019-08-25T18:51:13

You’re right. I’ll let the article stand as it is and reflect my ignorance as it was when I wrote it :-)

Jo Christian Oterhals: Perl 6 small stuff #21: it’s a date! …or: learn from an overly complex solution to a simple task

Published by Jo Christian Oterhals on 2019-07-31T13:23:17

Perl 6 small stuff #21: it’s a date! …or: learn from an overly complex solution to a simple task

This week’s Perl Weekly Challenge (#19) has two tasks. The first is to find all months with five weekends in the years from 1900 through 2019. The second is to program an implementation of word wrap using the greedy algorithm.

Both are pretty straight-forward tasks, and the solutions to them can (and should) be as well. This time, however, I’m also going to do the opposite and incrementally turn the easy solution into an unnecessarily complex one. Because in this particular case we can learn more by doing things the unnecessarily hard way. So this post will take a look at Dates and date manipulation in Perl 6, using PWC #19 task 1 as an example:

Write a script to display months from the year 1900 to 2019 where you find 5 weekends i.e. 5 Friday, 5 Saturday and 5 Sunday.

Let’s start by finding five-weekend months the easy way:

#!/usr/bin/env perl6
say join "\n", grep *.day-of-week == 5, map { Date.new: |$_, 1 }, do 1900..2019 X 1,3,5,7,8,10,12;

The algorithm for figuring this out is simple. Given the prerequisite that there must be five occurrences of not only Saturday and Sunday but also Friday, you A) *must* have 31 days to cram five weekends into. And when you know that you’ll also see that B) the last day of the month MUST be a Sunday and C) the first day of the month MUST be a Friday (you don’t have to check for both; if A is true and B is true, C is automatically true too).

The code above implements B and employs a few tricks. You read it from right to left (unless you write it from left to right, like this… say do 1900..2019 X 1,3,5,7,8,10,12 ==> map { Date.new: |$_, 1 } ==> grep *.day-of-week == 5 ==> join “\n”; )

Using the X operator I create a cross product of all the years in the range 1900–2019 and the months 1, 3, 5, 7, 8, 10, 12 (31-day months). In return I get a sequence containing all year-month pairs of the period.

The map function iterates through the Seq. There it instantiates a Date object. A little song and dance is necessary: As Date.new takes three unnamed integer parameters, year, month and day, I have to do something to what I have — a Pair with year and month. I therefore use the | operator to “explode” the pair into two integer parameters for year and month.

You can always use this for calling a sub routine with fixed parameters, using an array with parameter values rather than having separate variables for each parameter. The code below exemplifies usage:

my @list = 1, 2, 3;
sub explode-parameters($one, $two, $three) { 
…do something…
}
#traditional call 
explode-parameters(@list[0], @list[1], @list[2]);
# …or using | 
explode-parameters(|@list);

Back to the business at hand — the .grep filters out the months where the 1st is a Friday, and those are our 5 weekend months. So the output of the one-liner above looks something like this:

...
1997-08-01
1998-05-01
1999-01-01
...

This is a solution as good as any, and if a solution was all we wanted, we could have stopped here. But using this task as an example I want to explore ways to utilise the Date class. Example: The one-liner above does the job, but strictly speaking it doesn’t output the months but the first day of those months. Correcting this is easy, because the Date class supports something called formatters and use the sprintf syntax. To do this you utilise the named parameter “formatter” when instantiating the object.

say join "\n", grep *.day-of-week == 5, map { Date.new: |$_, 1, formatter => { sprintf "%04d/%02d", .year, .month } }, do 1900..2019 X 1,3,5,7,8,10,12;

Every time a routine pulls a stringified version of the date, the formatter object is invoked. In our case the output has been changed to…

...
1997/08
1998/05
1999/01
...

Formatters are powerful. Look into them.

Now to the overly complex solution. This is the unthinking programmer’s solution, as we don’t suppose anything. The program isn’t told that 5 weekend months only can occur on 31 day months. It doesn’t know that the 1st of such months must be a Friday. All it knows is that if the last day of the month is not Sunday, it figures out the date of the last Sunday (this is not very relevant when counting three-day weekends, but could be if you want to find Saturday+Sunday weekends, or only Sundays).

#!/usr/bin/env perl6
my $format-it = sub ($self) {
sprintf "%04d month %02d", .year, .month given $self;
}
sub MAIN(Int :$from-year = 1900, Int :$to-year where * > $from-year = 2019, Int :$weekend-length where * ~~ 1..3 = 3) {
my $date-loop = Date.new($from-year, 1, 1, formatter => $format-it);
while ($date-loop.year <= $to-year) {
my $date = $date-loop.later(day => $date-loop.days-in-month);
$date = $date.truncated-to('week').pred if $date.day-of-week != 7;
my @weekend = do for 0..^$weekend-length -> $w {
$date.earlier(day => $w).weekday-of-month;
};
say $date-loop if ([+] @weekend) / @weekend == 5;
$date-loop = $date-loop.later(:1month);
}
}

This code can solve the task both for three day weekends, but also for weekends consisting of Saturday + Sunday, as well as only Sundays. You control that with the command line parameter weekend-length=[1..3].

This code finds the last Sunday of each month and counts whether it has occured five times that month. It does the same for Saturday (if weekend-length=2) and Friday (if weekend-length=3). Like this:

my @weekend = do for 0..^$weekend-length -> $w { 
$date.earlier(day => $w).weekday-of-month;
};

The code then calculcates the average weekday-of-month for these three days like this:

say $date-loop if ([+] @weekend) / @weekend == 5;

This line uses the reduction operator [+] on the @weekend list to find the sum of all elements. That sum is divided by the number of elements. If the result is 5, then you have a five day weekend.

As for fun stuff to do with the Date object:

.later(day|month|year => Int) — adds the given number of time units to the current date. There’s also an earlier method for subtracting.

.days-in-months — tells you how many days there are in the current month of the Date object. The value may be 31, 30, 29 (february, leap year) or 28 (february).

.truncated-to(week|month|day|year) — rolls the date back to the first day of the week, month, day or year.

.weekday-of-month — figures out what day of week the current date is and calculates how many of that day there has been so far in that month.

Apart from this you’ll see that I added the formatter in a different way this time. This is probably cleaner looking and easier to maintain.

In the end this post maybe isn’t about dates and date manipulation at all, but rather is a call for all of us to use the documentation even more. It’s often I think that Perl 6 should have a function for x, y or z — .weekday-of-month is one such example — and the documentation tells me that it actually does!

It’s very easy to pick up Perl 6 and program it as you would have programmed Perl 5 or other languages you know well. But the documentation has lots of info of things you didn’t have before and that will make programming easier and more fun when you’ve learnt about them.

I guess you don’t need and excuse to delve into the docs, but if you do the Perl Weekly Challenge is an excellent excuse for spending time in the docs!

Jo Christian Oterhals: You have several options besides do. You could use parenthesises instead, like this:

Published by Jo Christian Oterhals on 2019-08-01T07:34:44

You have several options besides do. You could use parenthesises instead, like this:

say join "\n", grep *.day-of-week == 5, map { Date.new: |$_, 1 }, (1900..2019 X 1,3,5,7,8,10,12);

In this case I just thought the code looked better without parenthesises, so I chose to use do instead. The docs has a couple of sentences about this option here.

BTW, I chose the form Date.new: blah — the colon variant — instead of Date.new(blah) also because I wanted to avoid parenthesises. This freedom to chose is the essence of Perl’s credo “There’s more than one way to do it” I guess.

Rant: This freedom of choice is the best thing with Perl 6, but it has a downside too. Code can be harder to understand if you’re not familiar with the variants another programmer has made. This is a part of the Perls’s reputation of being write-only languages.

It won’t happen, but personally I’d like to see someone analyse real-world Perl 6 code (public repositories on Github for instance) and figure what forms are used most. The analysis could have been used to clean house — figuring out what forms to keep and which should be removed. Perl 6 would become a smaller — and arguably easier — language while staying just as powerful as it is today.

p6steve: Mind the gap

Published by p6steve on 2019-07-28T20:38:23

Observant readers looking at the dateline on the left will have noticed a gap of nearly two years between my last blog and today. As a fascinated programming hobbyist, I have only limited time in the day to devote to coding – and a startup venture put the brakes on anything but the day job for a while.

In the meantime, I have had to choose between coding and blogging – and while the ideas have been flowing ahead, I am quite keen on the idea that ‘actions speak louder than words’.

The actions have covered:

So, today, I am delighted to announce the alpha release of Physics::Measure (try zef install https://github.com/p6steve/[email protected] for now – hopefully zef install Physics::Measure within a day or two).

Hopefully the synopsis at https://github.com/p6steve/perl6-Physics-Measure makes sense. I encourage all feedback with an open mind…

Now back to the blogging!

~p6steve

my Timotimo \this: A Close Look At Controlling The MoarVM Profiler

Published by Timo Paulssen on 2019-05-22T14:41:13

A Close Look At Controlling The MoarVM Profiler

This is slightly tangential to the Rakudo Perl 6 Performance Analysis Tooling grant from The Perl Foundation, but it does interact closely with it, so this is more or less a progress report.

The other day, Jonathan Worthington of Edument approached me with a little paid Job: Profiling very big codebases can be tedious, especially if you're interested in only one specific fraction of the program. That's why I got the assignment to make the profiler "configurable". On top of that, the Comma IDE will get native access to this functionality.

The actual request was just to allow specifying whether individual routines should be included in the profile via a configuration file. That would have been possible with just a simple text file that contains one filename/line number/routine name per line. However, I have been wanting something in MoarVM that allows much more involved configuration for many different parts of the VM, not just the profiler.

A Close Look At Controlling The MoarVM Profiler
Obligatory cat photo.

That's why I quickly developed a small and simple "domain-specific language" for this and similar purposes.

The language had a few requirements:

There's also some things that aren't necessary:

While thinking about what exactly I should build – before I eventually settled on building a "programming language" for this task – I bounced back and forth between the simplest thing that could possibly work (for example, a text file with a list of file/line/name entries) and the most powerful thing that I can implement in a sensible timeframe (for example, allowing a full NQP script). A very important realization was that as long as I require the first line to identify what "version" of configuration program it is, I could completely throw away the current design and put something else instead, if the need ever arises. That allowed me to actually commit to a design that looked at least somewhat promising. And so I got started on what I call confprog.

Here's an example program. It doesn't do very much, but shows what it's about in general:

version = 1
entry profiler_static:
log = sf.name;
profile = sf.name eq "calculate-strawberries"
profile |= sf.cu.filename eq "CustomCode.pm6"

The entry decides which stage of profiling this program is applied to. In this case, the profiler_static means we're seeing a routine for the first time, before it is actually entered. That's why only the information every individual invocation of the frame in question shares is available via the variable sf, which stands for Static Frame. The Static Frame also allows access to the Compilation Unit (cu) that it was compiled into, which lets us find the filename.

The first line that actually does something assigns a value to the special variable log. This will output the name of the routine the program was invoked for.

The next line will turn on profiling only if the name of the routine is "calculate-strawberries". The line after that will also turn on profiling if the filename the routine is from is "CustomCode.pm6".

Apart from profiler_static, there are a couple more entry points available.

The syntax is still subject to change, especially before the whole thing is actually in a released version of MoarVM.

There is a whole lot of other things I could imagine being of interest in the near or far future. One place I'm taking inspiration from is where "extended Berkeley Packet Filter" (eBPF for short) programs are being used in the linux kernel and related pieces of software:

Oversimplifying a bit, BPF was originally meant for tcpdump so that the kernel doesn't have to copy all data over to the userspace process so that the decision what is interesting or not can be made. Instead, the kernel receives a little piece of code in the special BPF language (or bytecode) and can calculate the decision before having to copy anything.

eBPF programs can now also be used as a complex ruleset for sandboxing processes (with "seccomp"), to decide how network packets are to be routed between sockets (that's probably for Software Defined Networks?), what operations a process may perform on a particular device, whether a trace point in the kernel or a user-space program should fire, and so on.

So what's the status of confprog? I've written a parser and compiler that feeds confprog "bytecode" (which is mostly the same as regular moarvm bytecode) to MoarVM. There's also a preliminary validator that ensures the program won't do anything weird, or crash, when run. It is much too lenient at the moment, though. Then there's an interpreter that actually runs the code. It can already take an initial value for the "decision output value" (great name, isn't it) and it will return whatever value the confprog has set when it runs. The heap snapshot profiler is currently the only part of MoarVM that will actually try to run a confprog, and it uses the value to decide whether to take a snapshot or not.

Next up on the list of things to work on:

Apart from improvements to the confprog programming language, the integration with MoarVM lacks almost everything, most importantly installing a confprog for the profiler to decide whether a frame should be profiled (which was the original purpose of this assignment).

After that, and after building a bit of GUI for Comma, the regular grant work can resume: Flame graphs are still not visible on the call graph explorer page, and heap snapshots can't be loaded into moarperf yet, either.

Thanks for sticking with me through this perhaps a little dry and technical post. I hope the next one will have a little more excitement! And if there's interest (which you can signal by sending me a message on irc, or posting on reddit, or reaching me via twitter @loltimo, on the Perl 6 discord server etc) I can also write a post on how exactly the compiler was made, and how you can build your own compiler with Perl 6 code. Until then, you can find Andrew Shitov's presentations about making tiny languages in Perl 6 on youtube.

I hope you have a wonderful day; see you in the next one!
  - Timo

PS: I would like to give a special shout-out to Nadim Khemir for the wonderful Data::Dump::Tree module which made it much easier to see what my parser was doing. Here's some example output from another simple confprog program:

[6] @0
0 = .Label .Node @1
│ ├ $.name = heapsnapshot.Str
│ ├ $.type = entrypoint.Str
│ ├ $.position is rw = Nil
│ └ @.children = [0] @2
1 = .Op .Node @3
│ ├ $.op = =.Str
│ ├ $.type is rw = Nil
│ └ @.children = [2] @4
│   ├ 0 = .Var .Node @5
│   │ ├ $.name = log.Str
│   │ └ $.type = CPType String :stringy  @6
│   └ 1 = String Value ("we're in") @7
2 = .Op .Node @8
│ ├ $.op = =.Str
│ ├ $.type is rw = Nil
│ └ @.children = [2] @9
│   ├ 0 = .Var .Node @10
│   │ ├ $.name = log.Str
│   │ └ $.type = CPType String :stringy  §6
│   └ 1 = .Op .Node @12
│     ├ $.op = getattr.Str
│     ├ $.type is rw = CPType String :stringy  §6
│     └ @.children = [2] @14
│       ├ 0 = .Var .Node @15
│       │ ├ $.name = sf.Str
│       │ └ $.type = CPType MVMStaticFrame  @16
│       └ 1 = name.Str
3 = .Op .Node @17
│ ├ $.op = =.Str
│ ├ $.type is rw = Nil
│ └ @.children = [2] @18
│   ├ 0 = .Var .Node @19
│   │ ├ $.name = log.Str
│   │ └ $.type = CPType String :stringy  §6
│   └ 1 = String Value ("i am the confprog and i say:") @21
4 = .Op .Node @22
│ ├ $.op = =.Str
│ ├ $.type is rw = Nil
│ └ @.children = [2] @23
│   ├ 0 = .Var .Node @24
│   │ ├ $.name = log.Str
│   │ └ $.type = CPType String :stringy  §6
│   └ 1 = String Value ("  no heap snapshots for you my friend!") @26
5 = .Op .Node @27
$.op = =.Str
$.type is rw = Nil
@.children = [2] @28
0 = .Var .Node @29
    │ ├ $.name = snapshot.Str
    │ └ $.type = CPType Int :numeric :stringy  @30
1 = Int Value (0) @31

Notice how it shows the type of most things, like name.Str, as well as cross-references for things that appear multiple times, like the CPType String. Particularly useful is giving your own classes methods that specify exactly how they should be displayed by DDT. Love It!

Strangely Consistent: Refactoring the universe

Published by Carl Mäsak

I'm here to share a thing I'm working on, and chew gum; and I'm all out of gum. The purpose of this post is both to break a disagreeable silence that has befallen this blog, and to be able to geek out about a niche topic here in writing, partially sparing friends and family.

I'm currently refactoring 007's type system in a branch. Basically, ever since 007 was created, a type in 007-land has corresponded to a class/role in Perl 6, the host system that's implementing 007.

Here's what the implementation of Int looks like currently:

class Val::Int does Val {
    has Int $.value;

    method truthy {
        ?$.value;
    }
}

And here's what it looks like as I'm doing the refactor:

constant TYPE is export = {};

BEGIN {
    # ...
    TYPE<Int> = make-type "Int", :backed;
    # ...
}

So, instead of corresponding to types on the host level, all the 007 types are about to correspond to values. The former implementation was the one that felt obvious at the time (four-plus years ago), but it's become blindingly, painstakingly obvious that it really needs to be the latter.

Here's why: as soon as you want to implement class declarations in 007, in the former model you also need to bend over backwards and come up with an entirely new type in the host system. The Perl 6 code to do that looks like this:

return $.type.new(:type(EVAL qq[class :: \{
    method attributes \{ () \}
    method ^name(\$) \{ "{$name}" \}
\}]));

Which is... even someone as EVAL-positive as I wishes for a less clunky solution.

In the new model, a new class comes down to calling make-type and dropping the result in that TYPE hash. (Wait. Or not even dropping it in the TYPE hash. That hash is only for things used by 007 itself, not for user types.)

This is a refactor I've tried once before, back in 2017, but I failed back then because the code got too complicated and ran too slow. This time around I have a much better feeling.

By the way, there's also an is-type subroutine, and similarly a make-int and an is-int subroutine, and so on for every registered type. I figure why not wrap those simple things up in very short function names. So far that turns out to have been a very good decision. "Fold the language of your domain model into your code", and so on.

This is one of the things I'm doing better this time around; last time one of the problems was that each line I touched got longer and messier because there were more layers of indirection to dig through. Concerns were scattered all over the place. This time, it feels like the codebase is getting simpler thanks to those named subroutines. Maybe it can be likened to putting all your database-specific code in one place.

I sometimes get slight vertigo due to the bootstrapping aspects of this type system. One example: Object is an instance of Type, but the base class of Type is Object — a circularity. But, it turns out, if you have absolute power over the object graph, you can always bend things to your will:

BEGIN {
    TYPE<Type> = _007::Value.new(:type(__ITSELF__), slots => { name => "Type" });
    TYPE<Object> = make-type "Object";
    {
        # Bootstrap: now that we have Object, let's make it the base of Type and Object
        TYPE<Type>.slots<base> = TYPE<Object>;
        TYPE<Object>.slots<base> = TYPE<Object>;
    }
    # ...
}

I'm slightly reminded of a thing Gilad Bracha wrote once (which I can't find the exact quote for, unfortunately): that if mutual dependencies and recursive definitions are something that stump you, what you need is a healthy dose of letrec. It's twisty, yes, but it's basically a solved problem.

Like last time, I'm tackling the big task in small steps, one type at a time. I feel I've learned this from Martin Fowler's concept of asset capture. The idea is to end up back with a running system with passing tests often. I do this by replacing one old thing at a time by a new thing. Sounds obvious, but I'm actually not sure I would have been sensible enough on my own to tackle it this way, had I not known about asset capture.

One drawback is that you're sort of running the old system and the new system in parallel, as the old one is being phased out. Only once the whole thing has been asset-captured can complexity associated with the old system be completely removed.

It's a pleasant way to work. To me it's been at least a partial answer to the problem of the big rewrite. If we're free to refactor the insides, we can successively arrive at a point where the new better thing has completely replaced the old thing. The way there is allowed to be a little bit more complex (on the inside) than either endpoint. Importantly, you keep a running system throughout.

I don't have a concluding thought, except to say that I just managed to asset-capture arrays. Which is harder than it sounds, because arrays are everywhere in the compiler and the runtime.

rakudo.org: Rakudo Star Release 2019.03

Published on 2019-03-30T00:00:00

my Timotimo \this: Intermediate Progress Report: Heap Snapshots

Published by Timo Paulssen on 2019-03-22T22:22:00

Intermediate Progress Report: Heap Snapshots

Hello dear readers,

this time I don't have something finished to show off, but nevertheless I can tell you all about the work I've recently started.

The very first post on this blog already had a little bit about the heap snapshot profiler. It was about introducing a binary format to the heap snapshot profiler so that snapshots can be written out immediately after they were taken and by moarvm itself rather than by NQP code. This also made it possible to load snapshot data faster: the files contain an index that allows a big chunk of data to be split up exactly in the middle and then read and decoded by two threads in parallel.

The new format also resulted in much smaller output files, and of course reduced memory usage of turning on the profiler while running perl6 code. However, since it still captures one heap snapshot every time the GC runs, every ordinary program that runs for longer than a few minutes will accumulate quite an amount of data. Heap snapshot files (which are actually collections of multiple snapshots) can very easily outgrow a gigabyte. There would have to be another change.

Intermediate Progress Report: Heap Snapshots
Photo by Waldemar Brandt / Unsplash

Enter Compression

The new format already contained the simplest thing that you could call compression. Instead of simply writing every record to the file as it comes, records that have smaller numbers would be stored with a shorter representation. This saved a lot of space already, but not nearly as much as off-the-shelf compression techniques would.

There had to be another way to get compression than just coming up with my own compression scheme! Well, obviously I could have just implemented something that already exists. However, at the time I was discouraged by the specific requirements of the heap snapshot analyzer - the tool that reads the files to let the user interactively explore the data within it:

Normally, compression formats are not built to support easily seeking to any given spot in the uncompressed data. There was of course the possibility to compress each snapshot individually, but that would mean a whole snapshot could either only be read in with a single thread, or the compression would have to go through the whole blob and when the first splittable piece was decompressed, a thread could go off and parse it. I'm not entirely sure why I didn't go with that, perhaps I just didn't think of it back then. After all, it's already been more than a year, and my brain compresses data by forgetting stuff.

Anyway, recently I decided I'd try a regular compression format for the new moarvm heap snapshot file format. There's already a Perl 6 module named Compress::Zlib, which I first wanted to use. Writing the data out from moarvm was trivial once I linked it to zlib. Just replace fopen with gzopen, fwrite with gzwrite, fclose with gzclose and you're almost done! The compression ratio wasn't too shabby either.

When I mentioned this in the #moarvm channel on IRC, I was asked why I use zlib instead of zstd. After all, zstd usually (or always?) outperforms zlib in both compression/decompression speed and output size. The only answer I had for that was that I hadn't used the zstd C library yet, and there was not yet a Perl 6 module for it.

Figuring out zstd didn't go as smoothly as zlib, not by a long shot. But first I'd like to explain how I solved the issue of reading the file with multiple threads.

Restructuring the data

In the current binary format, there are areas for different kinds of objects that occur once per snapshot. Those are collectables and references. On top of that there are objects that are shared across snapshots: Strings that are referenced from all the other kinds of objects (for example filenames, or descriptions for references like "hashtable entry"), static frames (made up of a name like "push", an ID, a filename, and a line number), and types (made up of a repr name like VMArray, P6opaque, or CStruct and a type name like BagHash or Regex).

That resulted in a file format that has one object after the other in the given area. The heap snapshot analyzer itself then goes through these areas and splits the individual values apart, then shoves them off into a queue for another thread to actually store. Storage inside the heap analyzer consists of one array for each part of these objects. For example, there is one array of integers for all the descriptions and one array of integers for all the target collectables. The main benefit of that is not having to go through all the objects when the garbage collector runs.

The new format on the other hand puts every value of each attribute in one long string before continuing with the next attribute.

Here's how the data for static frames was laid out in the file in the previous version:

"sframes", count, name 1, uuid 1, file 1, line 1, name 2, uuid 2, file 2, line 2, name 3, uuid 3, file 3, line 3, … index data

The count at the beginning tells us how many entries we should be expecting. For collectables, types, reprs, and static frames this gives us the exact number of bytes to look for, too, since every entry has the same size. References on the other hand have a simple "compression" applied to them, which doesn't allow us to just figure out the total size by knowing the count. To offset this, the total size lives at the end in a place that can easily be found by the parser. Strings are also variable in length, but there's only a few hundred of them usually. References take up the most space in total; having four to five times as many references as there are collectables is pretty normal.

Here's how the same static frame data is laid out in the upcoming format:

"names", length, zstd(name 1, name 2, name 3, …), index data, "uuids", length, zstd(uuid 1, uuid 2, uuid 3, …), index data, "files", length, zstd(file 1, file 2, file 3, …), index data, "lines", length, zstd(line 1, line 2, line 3, …), index data

As you can see, the values for each attribute now live in the same space. Each attribute blob is compressed individually, each has a little piece of index data at the end and a length field at the start. The length field is actually supposed to hold the total size of the compressed blob, but if the heap snapshot is being output to a destination that doesn't support seeking (moving back in the file and overwriting an earlier piece of data) we'll just leave it zeroed out and rely on zstd's format being able to tell when one blob ends.

There are some benefits to this approach that I'd like to point out:

The last point, in fact, will let me put some extra fun into the files. First of all, I currently start the files with a "plain text comment" that explains what kind of file it is and how it's structured internally. That way, if someone stumbles upon this file in fifty years, they can get started finding out the contents right away!

On a perhaps more serious note, I'll put in summaries of each snapshot that MoarVM itself can just already generate while it's writing out the snapshot itself. Not only things like "total number of collectables", but also "top 25 types by size, top 25 types by count". That will actually make the heapanalyzer not need to touch the actual data until the user is interested in more specific data, like a "top 100" or "give me a thousand objects of type 'Hash'".

On top of that, why not allow the user to "edit" heap snapshots, put some comments in before sharing it to others, or maybe "bookmarking" specific objects?

All of these things will be easier to do with the new format - that's the hope at least!

Did the compression work?

I didn't actually have the patience to exhaustively measure all the details, but here's a rough ratio for comparison: One dataset I've got results in a 1.1 gigabytes big file with the current binary format, a 147 megabytes big file when using gzip and a 99 megabytes big file using zstd (at the maximum "regular" compression level - I haven't checked yet if the cpu usage isn't prohibitive for this, though).

It seems like this is a viable way forward! Allowing capture to run 10x as long is a nice thing for sure.

What comes next?

The profiler view itself in Moarperf isn't done yet, of course. I may not put in more than the simplest of features if I start on the web frontend for the heap analyzer itself.

On the other hand, there's another task that's been waiting: Packaging moarperf for simpler usage. Recently we got support for a relocatable perl6 binary merged into master. That should make it possible to create an AppImage of moarperf. A docker container should also be relatively easy to build.

We'll see what my actual next steps will be - or will have been I suppose - when I post the next update!

Thanks for reading and take care
  - Timo

brrt to the future: Reverse Linear Scan Allocation is probably a good idea

Published by Bart Wiegmans on 2019-03-21T15:52:00

Hi hackers! Today First of all, I want to thank everybody who gave such useful feedback on my last post.  For instance, I found out that the similarity between the expression JIT IR and the Testarossa Trees IR is quite remarkable, and that they have a fix for the problem that is quite different from what I had in mind.

Today I want to write something about register allocation, however. Register allocation is probably not my favorite problem, on account of being both messy and thankless. It is a messy problem because - aside from being NP-hard to solve optimally - hardware instruction sets and software ABI's introduce all sorts of annoying constraints. And it is a thankless problem because the case in which a good register allocator is useful - for instance, when there's lots of intermediate values used over a long stretch of code - are fairly rare. Much more common are the cases in which either there are trivially sufficient registers, or ABI constraints force a spill to memory anyway (e.g. when calling a function, almost all registers can be overwritten).

So, on account of this being not my favorite problem, and also because I promised to implement optimizations in the register allocator, I've been researching if there is a way to do better. And what better place to look than one of the fastest dynamic language implementations arround, LuaJIT? So that's what I did, and this post is about what I learned from that.

Truth be told, LuaJIT is not at all a learners' codebase (and I don't think it's author would claim this). It uses a rather terse style of C and lots and lots of preprocessor macros. I had somewhat gotten used to the style from hacking dynasm though, so that wasn't so bad. What was more surprising is that some of the steps in code generation that are distinct and separate in the MoarVM JIT - instruction selection, register allocation and emitting bytecode - were all blended together in LuaJIT. Over multiple backend architectures, too. And what's more - all these steps were done in reverse order - from the end of the program (trace) to the beginning. Now that's interesting...

I have no intention of combining all phases of code generation like LuaJIT has. But processing the IR in reverse seems to have some interesting properties. To understand why that is, I'll first have to explain how linear scan allocation currently works in MoarVM, and is most commonly described:

  1. First, the live ranges of program values are computed. Like the name indicates, these represent the range of the program code in which a value is both defined and may be used. Note that for the purpose of register allocation, the notion of a value shifts somewhat. In the expression DAG IR, a value is the result of a single computation. But for the purposes of register allocation, a value includes all its copies, as well as values computed from different conditional branches. This is necessary because when we actually start allocating registers, we need to know when a value is no longer in use (so we can reuse the register) and how long a value will remain in use -
  2. Because a value may be computed from distinct conditional branches, it is necessary to compute the holes in the live ranges. Holes exists because if a value is defined in both sides of a conditional branch, the range will cover both the earlier (in code order) branch and the later branch - but from the start of the later branch to its definition that value doesn't actually exist. We need this information to prevent the register allocator from trying to spill-and-load a nonexistent value, for instance.
  3. Only then can we allocate and assign the actual registers to instructions. Because we might have to spill values to memory, and because values now can have multiple definitions, this is a somewhat subtle problem. Also, we'll have to resolve all architecture specific register requirements in this step.
In the MoarVM register allocator, there's a fourth step and a fifth step. The fourth step exists to ensure that instructions conform to x86 two-operand form (Rather than return the result of an instruction in a third register, x86 reuses one of the input registers as the output register. E.g. all operators are of the form a = op(a, b)  rather than a = op(b, c). This saves on instruction encoding space). The fifth step inserts instructions that are introduced by the third step; this is done so that each instruction has a fixed address in the stream while the stream is being processed.

Altogether this is quite a bit of complexity and work, even for what is arguably the simplest correct global register allocation algorithm. So when I started thinking of the reverse linear scan algorithm employed by LuaJIT, the advantages became clear:
There are downsides as well, of course. Not knowing exactly how long a value will be live while processing it may cause the algorithm to make worse choices in which values to spill. But I don't think that's really a great concern, since figuring out the best possible value is practically impossible anyway, and the most commonly cited heuristic - evict the value that is live furthest in the future, because this will release a register over a longer range of code, reducing the chance that we'll need to evict again - is still available. (After all, we do always know the last use, even if we don't necessarily know the first definition).

Altogether, I'm quite excited about this algorithm; I think it will be a real simplification over the current implementation. Whether that will work out remains to be seen of course. I'll let you know!

brrt to the future: Something about IR optimization

Published by Bart Wiegmans on 2019-03-17T06:23:00

Hi hackers! Today I want to write about optimizing IR in the MoarVM JIT, and also a little bit about IR design itself.

One of the (major) design goals for the expression JIT was to have the ability to optimize code over the boundaries of individual MoarVM instructions. To enable this, the expression JIT first expands each VM instruction into a graph of lower-level operators. Optimization then means pattern-matching those graphs and replacing them with more efficient expressions.

As a running example, consider the idx operator. This operator takes two inputs (base and element) and a constant parameter scale and computes base+element*scale. This represents one of the operands of an  'indexed load' instruction on x86, typically used to process arrays. Such instructions allow one instruction to be used for what would otherwise be two operations (computing an address and loading a value). However, if the element of the idx operator is a constant, we can replace it instead with the addr instruction, which just adds a constant to a pointer. This is an improvement over idx because we no longer need to load the value of element into a register. This saves both an instruction and valuable register space.

Unfortunately this optimization introduces a bug. (Or, depending on your point of view, brings an existing bug out into the open). The expression JIT code generation process selects instructions for subtrees (tile) of the graph in a bottom-up fashion. These instructions represent the value computed or work performed by that subgraph. (For instance, a tree like (load (addr ? 8) 8) becomes mov ?, qword [?+8]; the question marks are filled in during register allocation). Because an instruction is always represents a tree, and because the graph is an arbitrary directed acyclic graph, the code generator projects that graph as a tree by visiting each operator node only once. So each value is computed once, and that computed value is reused by all later references.

It is worth going into some detail into why the expression graph is not a tree. Aside from transformations that might be introduced by optimizations (e.g. common subexpression elimination), a template may introduce a value that has multiple references via the let: pseudo-operator. See for instance the following (simplified) template:

(let: (($foo (load (local))))
    (add $foo (sub $foo (const 1))))

Both ADD and SUB refer to the same LOAD node


In this case, both references to $foo point directly to the same load operator. Thus, the graph is not a tree. Another case in which this occurs is during linking of templates into the graph. The output of an instruction is used, if possible, directly as the input for another instruction. (This is the primary way that the expression JIT can get rid of unnecessary memory operations). But there can be multiple instructions that use a value, in which case an operator can have multiple references. Finally, instruction operands are inserted by the compiler and these can have multiple references as well.

If each operator is visited only once during code generation, then this may introduce a problem when combined with another feature - conditional expressions. For instance, if two branches of a conditional expression both refer to the same value (represented by name $foo) then the code generator will only emit code to compute its value when it encounters the first reference. When the code generator encounters $foo for the second time in the other branch, no code will be emitted. This means that in the second branch, $foo will effectively have no defined value (because the code in the first branch is never executed), and wrong values or memory corruption is then the predictable result.

This bug has always existed for as long as the expression JIT has been under development, and in the past the solution has been not to write templates which have this problem. This is made a little easier by a feature the let: operator, in that it inserts a do operator which orders the values that are declared to be computed before the code that references them. So that this is in fact non-buggy:

(let: (($foo (load (local))) # code to compute $foo is emitted here
  (if (...)  
    (add $foo (const 1)) # $foo is just a reference
    (sub $foo (const 2)) # and here as well

The DO node is inserted for the LET operator. It ensures that the value of the LOAD node is computed before the reference in either branch


Alternatively, if a value $foo is used in the condition of the if operator, you can also be sure that it is available in both sides of the condition.

All these methods rely on the programmer being able to predict when a value will be first referenced and hence evaluated. An optimizer breaks this by design. This means that if I want the JIT optimizer to be successful, my options are:

  1. Fix the optimizer so as to not remove references that are critical for the correctness of the program
  2. Modify the input tree so that such references are either copied or moved forward
  3. Fix the code generator to emit code for a value, if it determines that an earlier reference is not available from the current block.
In other words, I first need to decide where this bug really belongs - in the optimizer, the code generator, or even the IR structure itself. The weakness of the expression IR is that expressions don't really impose a particular order. (This is unlike the spesh IR, which is instruction-based, and in which every instruction has a 'previous' and 'next' pointer). Thus, there really isn't a 'first' reference to a value, before the code generator introduces the concept. This is property is in fact quite handy for optimization (for instance, we can evaluate operands in whatever order is best, rather than being fixed by the input order) - so I'd really like to preserve it. But it also means that the property we're interested in - a value is computed before it is used in, in all possible code flow paths - isn't really expressible by the IR. And there is no obvious local invariant that can be maintained to ensure that this bug does not happen, so any correctness check may have to check the entire graph, which is quite impractical.

I hope this post explains why this is such a tricky problem! I have some ideas for how to get out of this, but I'll reserve those for a later post, since this one has gotten quite long enough. Until next time!

my Timotimo \this: Always Wear Safety Equipment When Inline Scalaring

Published by Timo Paulssen on 2019-02-21T20:44:09

Always Wear Safety Equipment When Inline Scalaring

MoarVM just recently got a nice optimization merged into the master branch. It's called "partial escape analysis" and comes with a specific optimization named "scalar replacement". In simple terms, it allows objects whose lifetime can be proven to end very soon to be created on the stack instead of in the garbage collector managed heap. More than just that, the "partial" aspect of the escape analysis even allows that when the object can escape out of our grasp, but will not always do so.

Allocation on the stack is much cheaper than allocation in the garbage collected heap, because freeing data off of the stack is as easy as leaving the function behind.

Always Wear Safety Equipment When Inline Scalaring
Photo by A. Zuhri / Unsplash

Like a big portion of optimizations in MoarVM's run-time optimizer/specializer ("spesh"), this analysis and optimization usually relies on some prior inlining of code. That's where the pun in the post's title comes from.

This is a progress report for the profiler front-end, though. So the question I'd like to answer in this post is how the programmer would check if these optimizations are being utilized in their own code.

A Full Overview

The first thing the user gets to see in the profiler's frontend is a big overview page summarizing lots of results from all aspects of the program's run. Thread creations, garbage collector runs, frame allocations made unnecessary by inlining, etc.

That page just got a new entry on it, that sums up all allocations and also shows how many allocations have been made unnecessary by scalar replacement, along with a percentage. Here's an example screenshot:

Always Wear Safety Equipment When Inline Scalaring

That on its own is really only interesting if you've run a program twice and maybe changed the code in between to compare how allocation/optimization behavior changed in between. However, there's also more detail to be found on the Allocations page:

Per-Type and Per-Routine Allocations/Replacements

The "Allocations" tab already gave details on which types were allocated most, and which routines did most of the allocating for any given type. Now there is an additional column that gives the number of scalar replaced objects as well:

Always Wear Safety Equipment When Inline Scalaring

Here's a screenshot showing the details of the Num type expanded to display the routines:

Always Wear Safety Equipment When Inline Scalaring

Spesh Performance Overview

One major part of Spesh is its "speculative" optimizations. They have the benefit of allowing optimizations even when something can't be proven. When some assumption is broken, a deoptimization happens, which is effectively a jump from inside an optimized bit of code back to the same position in the unoptimized code. There's also "on-stack-replacement", which is effectively a jump from inside unoptimized code to the same position in optimized code. The details are, of course, more complicated than that.

How can you find out which routines in your program (in libraries you're calling, or the "core setting" of all builtin classes and routines) are affected by deoptimization or by OSR? There is now an extra tab in the profiler frontend that gives you the numbers:

Always Wear Safety Equipment When Inline Scalaring

This page also has the first big attempt at putting hopefully helpful text directly next to the data. Below the table there's these sections:

Specializer Performance

MoarVM comes with a dynamic code optimizer called "spesh". It makes your code faster by observing at run time which types are used where, which methods end up being called in certain situations where there are multiple potential candidates, and so on. This is called specialization, because it creates versions of the code that take shortcuts based on assumptions it made from the observed data.

Deoptimization

Assumptions, however, are there to be broken. Sometimes the optimized and specialized code finds that an assumption no longer holds. Parts of the specialized code that detect this are called "guards". When a guard detects a mismatch, the running code has to be switched from the optimized code back to the unoptimized code. This is called a "deoptimization", or "deopt" for short.

Deopts are a natural part of a program's life, and at low numbers they usually aren't a problem. For example, code that reads data from a file would read from a buffer most of the time, but at some point the buffer would be exhausted and new data would have to be fetched from the filesystem. This could mean a deopt.

If, however, the profiler points out a large amount of deopts, there could be an optimization opportunity.

On-Stack Replacement (OSR)

Regular optimization activates when a function is entered, but programs often have loops that run for a long time until the containing function is entered again.

On-Stack Replacement is used to handle cases like this. Every round of the loop in the unoptimized code will check if an optimized version can be entered. This has the additional effect that a deoptimization in such code can quickly lead back into optimized code.

Situations like these can cause high numbers of deopts along with high numbers of OSRs.

I'd be happy to hear your thoughts on how clear this text is to you, especially if you're not very knowledgeable about the topics discussed. Check github for the current version of this text - it should be at https://github.com/timo/moarperf/blob/master/frontend/profiler/components/SpeshOverview.jsx#L82 - and submit an issue to the github repo, or find me on IRC, on the perl6-users mailing list, on reddit, on mastodon/the fediverse, twitter, or where-ever.

brrt to the future: A short post about types and polymorphism

Published by Bart Wiegmans on 2019-01-14T13:34:00

Hi all. I usually write somewhat long-winded posts, but today I'm going to try and make an exception. Today I want to talk about the expression template language used to map the high-level MoarVM instructions to low-level constructs that the JIT compiler can easily work with:

This 'language' was designed back in 2015 subject to three constraints:
Recently I've been working on adding support for floating point operations, and  this means working on the type system of the expression language. Because floating point instructions operate on a distinct set of registers from integer instructions, a floating point operator is not interchangeable with an integer (or pointer) operator.

This type system is enforced in two ways. First, by the template compiler, which attempts to check if you've used all operands correctly. This operates during development, which is convenient. Second, by instruction selection, as there will simply not be any instructions available that have the wrong combinations of types. Unfortunately, that happens at runtime, and such errors so annoying to debug that it motivated the development of the first type checker.

However, this presents two problems. One of the advantages of the expression IR is that, by virtue of having a small number of operators, it is fairly easy to analyze. Having a distinct set of operators for each type would undo that. But more importantly, there are several MoarVM instructions that are generic, i.e. that operate on integer, floating point, and pointer values. (For example, the set, getlex and bindlex instructions are generic in this way). This makes it impossible to know whether its values will be integers, pointers, or floats.

This is no problem for the interpreter since it can treat values as bags-of-bits (i.e., it can simply copy the union MVMRegister type that holds all values of all supported types). But the expression JIT works differently - it assumes that it can place any value in a register, and that it can reorder and potentially skip storing them to memory. (This saves work when the value would soon be overwritten anyway). So we need to know what register class that is, and we need to have the correct operators to manipulate a value in the right register class.

To summarize, the problem is:
There are two ways around this, and I chose to use both. First, we know as a fact for each local or lexical value in a MoarVM frame (subroutine) what type it should have. So even a generic operator like set can be resolved to a specific type at runtime, at which point we can select the correct operators. Second, we can introduce generic operators of our own. This is possible so long as we can select the correct instruction for an operator based on the types of the operands.

For instance, the store operator takes two operands, an address and a value. Depending on the type of the value (reg or num), we can always select the correct instruction (mov or movsd). It is however not possible to select different instructions for the load operator based on the type required, because instruction selection works from the bottom up. So we need a special load_num operator, but a store_num operator is not necessary. And this is true for a lot more operators than I had initially thought. For instance, aside from the (naturally generic) do and if operators, all arithmetic operators and comparison operators are generic in this way.

I realize that, despite my best efforts, this has become a rather long-winded post anyway.....

Anyway. For the next week, I'll be taking a slight detour, and I aim to generalize the two-operand form conversion that is necessary on x86. I'll try to write a blog about it as well, and maybe it'll be short and to the point. See you later!

brrt to the future: New years post

Published by Bart Wiegmans on 2019-01-06T13:15:00

Hi everybody! I recently read jnthns Perl 6 new years resolutions post, and I realized that this was an excellent example to emulate. So here I will attempt to share what I've been doing in 2018 and what I'll be doing in 2019.

In 2018, aside from the usual refactoring, bugfixing and the like:
So 2019 starts with me trying to complete the goals specified in that grant request. I've already partially completed one goal (as explained in the interim report) - ensuring that register encoding works correctly for SSE registers in DynASM. Next up is actually ensuring support for SSE (and floating point) registers in the JIT, which is surprisingly tricky, because it means introducing a type system where there wasn't really one previously. I will have more to report on that in the near future.

After that, the first thing on my list is the support for irregular register requirements in the register allocator, which should open up the possibility of supporting various instructions.

I guess that's all for now. Speak you later!

6guts: My Perl 6 wishes for 2019

Published by jnthnwrthngtn on 2019-01-02T01:35:51

This evening, I enjoyed the New Year’s fireworks display over the beautiful Prague skyline. Well, the bit of it that was between me and the fireworks, anyway. Rather than having its fireworks display at midnight, Prague puts it at 6pm on New Year’s Day. That makes it easy for families to go to, which is rather thoughtful. It’s also, for those of us with plans to dig back into work the following day, a nice end to the festive break.

Prague fireworks over Narodni Divadlo

So, tomorrow I’ll be digging back into work, which of late has involved a lot of Perl 6. Having spent so many years working on Perl 6 compiler and runtime design and implementation, it’s been fun to spend a good amount of 2018 using Perl 6 for commercial projects. I’m hopeful that will continue into 2019. Of course, I’ll be continuing to work on plenty of Perl 6 things that are for the good of all Perl 6 users too. In this post, I’d like to share some of the things I’m hoping to work on or achieve during 2019.

Partial Escape Analysis and related optimizations in MoarVM

The MoarVM specializer learned plenty of new tricks this year, delivering some nice speedups for many Perl 6 programs. Many of my performance improvement hopes for 2019 center around escape analysis and optimizations stemming from it.

The idea is to analyze object allocations, and find pieces of the program where we can fully understand all of the references that exist to the object. The points at which we can cease to do that is where an object escapes. In the best cases, an object never escapes; in other cases, there are a number of reads and writes performed to its attributes up until its escape.

Armed with this, we can perform scalar replacement, which involves placing the attributes of the object into local registers up until the escape point, if any. As well as reducing memory operations, this means we can often prove significantly more program properties, allowing further optimization (such as getting rid of dynamic type checks). In some cases, we might never need to allocate the object at all; this should be a big win for Perl 6, which by its design creates lots of short-lived objects.

There will be various code-generation and static optimizer improvements to be done in Rakudo in support of this work also, which should result in its own set of speedups.

Expect to hear plenty about this in my posts here in the year ahead.

Decreasing startup time and base memory use

The current Rakudo startup time is quite high. I’d really like to see it fall to around half of what it currently is during 2019. I’ve got some concrete ideas on how that can be achieved, including changing the way we store and deserialize NFAs used by the parser, and perhaps also dealing with the way we currently handle method caches to have less startup impact.

Both of these should also decrease the base memory use, which is also a good bit higher than I wish.

Improving compilation times

Some folks – myself included – are developing increasingly large applications in Perl 6. For the current major project I’m working on, runtime performance is not an issue by now, but I certainly feel myself waiting a bit on compiles. Part of it is parse performance, and I’d like to look at that; in doing so, I’d also be able to speed up handling of all Perl 6 grammars.

I suspect there are some good wins to be had elsewhere in the compilation pipeline too, and the startup time improvements described above should also help, especially when we pre-compile deep dependency trees. I’d also like to look into if we can do some speculative parallel compilation.

Research into concurrency safety

In Perl 6.d, we got non-blocking await and react support, which greatly improved the scalability of Perl 6 concurrent and parallel programs. Now many thousands of outstanding tasks can be juggled across just a handful of threads (the exact number chosen according to demand and CPU count).

For Perl 6.e, which is still a good way off, I’d like to having something to offer in terms of making Perl 6 concurrent and parallel programming safer. While we have a number of higher-level constructs that eliminate various ways to make mistakes, it’s still possible to get into trouble and have races when using them.

So, I plan to spend some time this year quietly exploring and prototyping in this space. Obviously, I want something that fits in with the Perl 6 language design, and that catches real and interesting bugs – probably by making things that are liable to occasionally explode in weird ways instead reliably do so in helpful ways, such that they show up reliably in tests.

Get Cro to its 1.0 release

In the 16 months since I revealed it, Cro has become a popular choice for implementing HTTP APIs and web applications in Perl 6. It has also attracted code contributions from a couple of dozen contributors. This year, I aim to see Cro through to its 1.0 release. That will include (to save you following the roadmap link):

Comma Community, and lots of improvements and features

I founded Comma IDE in order to bring Perl 6 a powerful Integrated Development Environment. We’ve come a long way since the Minimum Viable Product we shipped back in June to the first subscribers to the Comma Supporter Program. In recent months, I’ve used Comma almost daily on my various Perl 6 projects, and by this point honestly wouldn’t want to be without it. Like Cro, I built Comma because it’s a product I wanted to use, which I think is a good place to be in when building any product.

In a few months time, we expect to start offering Comma Community and Comma Complete. The former will be free of charge, and the latter a commercial offering under a subscribe-for-updates model (just like how the supporter program has worked so far). My own Comma wishlist is lengthy enough to keep us busy for a lot more than the next year, and that’s before considering things Comma users are asking for. Expect plenty of exciting new features, as well as ongoing tweaks to make each release feel that little bit nicer to use.

Speaking, conferences, workshops, etc.

This year will see me giving my first keynote at a European Perl Conference. I’m looking forward to being in Riga again; it’s a lovely city to wander around, and I remember having some pretty nice food there too. The keynote will focus on the concurrent and parallel aspects of Perl 6; thankfully, I’ve still a good six months to figure out exactly what angle I wish to take on that, having spoken on the topic many times before!

I also plan to submit a talk or two for the German Perl Workshop, and will probably find the Swiss Perl Workshop hard to resist attending once more. And, more locally, I’d like to try and track down other Perl folks here in Prague, and see if I can help some kind of Praha.pm to happen again.

I need to keep my travel down to sensible levels, but might be able to fit in the odd other bit of speaking during the year, if it’s not too far away.

Teaching

While I want to spend most of my time building stuff rather than talking about it, I’m up for the occasional bit of teaching. I’m considering pitching a 1-day Perl 6 concurrency workshop to the Riga organizers. Then we’ll see if there’s enough folks interested in taking it. It’ll cost something, but probably much less than any other way of getting a day of teaching from me. :-)

So, down to work!

Well, a good night’s sleep first. :-) But tomorrow, another year of fun begins. I’m looking forward to it, and to working alongside many wonderful folks in the Perl community.

rakudo.org: Rakudo Star Release 2018.10

Published on 2018-11-11T00:00:00

Perlgeek.de: Perl 6 Coding Contest 2019: Seeking Task Makers

Published by Moritz Lenz on 2018-11-10T23:00:01

I want to revive Carl Mäsak's Coding Contest as a crowd-sourced contest.

The contest will be in four phases:

For the first phase, development of tasks, I am looking for volunteers who come up with coding tasks collaboratively. Sadly, these volunteers, including myself, will be excluded from participating in the second phase.

I am looking for tasks that ...

This is non-trivial, so I'd like to have others to discuss things with, and to come up with some more tasks.

If you want to help with task creation, please send an email to [email protected], stating your intentions to help, and your freenode IRC handle (optional).

There are other ways to help too:

In these cases you can use the same email address to contact me, or use IRC (moritz on freenode) or twitter.

Zoffix Znet: Perl 6 Advent Calendar 2018 Call for Authors

Published on 2018-10-31T00:00:00

Write a blog post about Perl 6

Zoffix Znet: A Request to Larry Wall to Create a Language Name Alias for Perl 6

Published on 2018-10-07T00:00:00

The culmination of the naming discussion

6guts: Speeding up object creation

Published by jnthnwrthngtn on 2018-10-06T22:59:11

Recently, a Perl 6 object creation benchmark result did the rounds on social media. This Perl 6 code:

class Point {
    has $.x;
    has $.y;
}
my $total = 0;
for ^1_000_000 {
    my $p = Point.new(x => 2, y => 3);
    $total = $total + $p.x + $p.y;
}
say $total;

Now (on HEAD builds of Rakudo and MoarVM) runs faster than this roughly equivalent Perl 5 code:

use v5.10;

package Point;

sub new {
    my ($class, %args) = @_;
    bless \%args, $class;
}

sub x {
    my $self = shift;
    $self->{x}
}

sub y {
    my $self = shift;
    $self->{y}
}

package main;

my $total = 0;
for (1..1_000_000) {
    my $p = Point->new(x => 2, y => 3);
    $total = $total + $p->x + $p->y;
}
say $total;

(Aside: yes, I know there’s no shortage of libraries in Perl 5 that make OO nicer than this, though those I tried also made it slower.)

This is a fairly encouraging result: object creation, method calls, and attribute access are the operational bread and butter of OO programming, so it’s a pleasing sign that Perl 6 on Rakudo/MoarVM is getting increasingly speedy at them. In fact, it’s probably a bit better at them than this benchmark’s raw numbers show, since:

While dealing with Int values got faster recently, it’s still really making two Int objects every time around that loop and having to GC them later. An upcoming new set of analyses and optimizations will let us get rid of that cost too. And yes, startup will get faster with time also. In summary, while Rakudo/MoarVM is now winning that benchmark against Perl 5, there’s still lots more to be had. Which is a good job, since the equivalent Python and Ruby versions of that benchmark still run faster than the Perl 6 one.

In this post, I’ll look at the changes that allowed this benchmark to end up faster. None of the new work was particularly ground-breaking; in fact, it was mostly a case of doing small things to make better use of optimizations we already had.

What happens during construction?

Theoretically, the default new method in turn calls bless, passing the named arguments along. The bless method then creates an object instance, followed by calling BUILDALL. The BUILDALL method goes through the set of steps needed for constructing the object. In the case of a simple object like ours, that involves checking if an x and y named argument were passed, and if so assigning those values into the Scalar containers of the object attributes.

For those keeping count, that’s at least 3 method calls (newbless, and BUILDALL).

However, there’s a cheat. If bless wasn’t overridden (which would be an odd thing to do anyway), then the default new could just call BUILDALL directly anyway. Therefore, new looks like this:

multi method new(*%attrinit) {
    nqp::if(
      nqp::eqaddr(
        (my $bless := nqp::findmethod(self,'bless')),
        nqp::findmethod(Mu,'bless')
      ),
      nqp::create(self).BUILDALL(Empty, %attrinit),
      $bless(|%attrinit)
    )
}

The BUILDALL method was originally a little “interpreter” that went through a per-object build plan stating what needs to be done. However, for quite some time now we’ve instead compiled a per-class BUILDPLAN method.

Slurpy sadness

To figure out how to speed this up, I took a look at how the specializer was handling the code. The answer: not so well. There were certainly some positive moments in there. Of note:

However, the new method was getting only a “certain” specialization, which is usually only done for very heavily polymorphic code. That wasn’t the case here; this program clearly constructs overwhelmingly one type of object. So what was going on?

In order to produce an “observed type” specialization – the more powerful kind – it needs to have data on the types of all of the passed arguments. And for the named arguments, that was missing. But why?

Logging of passed argument types is done on the callee side, since:

The argument logging was done as the interpreter executed each parameter processing instruction. However, when there’s a slurpy, then it would just swallow up all the remaining arguments without logging type information. Thus the information about the argument types was missing and we ended up with a less powerful form of specialization.

Teaching the slurpy handling code about argument logging felt a bit involved, but then I realized there was a far simpler solution: log all of the things in the argument buffer at the point an unspecialized frame is being invoked. We’re already logging the entry to the call frame at that point anyway. This meant that all of the parameter handling instructions got a bit simpler too, since they no longer had logging to do.

Conditional elimination

Having new being specialized in a more significant way was an immediate win. Of note, this part:

      nqp::eqaddr(
        (my $bless := nqp::findmethod(self,'bless')),
        nqp::findmethod(Mu,'bless')
      ),

Was quite interesting. Since we were now specializing on the type of self, then the findmethod could be resolved into a constant. The resolution of a method on the constant Mu was also a constant. Therefore, the result of the eqaddr (same memory address) comparison of two constants should also have been turned into a constant…except that wasn’t happening! This time, it was simple: we just hadn’t implemented folding of that yet. So, that was an easy win, and once done meant the optimizer could see that the if would always go a certain way and thus optimize the whole thing into the chosen branch. Thus the new method was specialized into something like:

multi method new(*%attrinit) {
    nqp::create(self).BUILDALL(Empty, %attrinit),
}

Further, the create could be optimized into a “fast create” op, and the relatively small BUILDALL method could be inlined into new. Not bad.

Generating simpler code

At this point, things were much better, but still not quite there. I took a look at the BUILDALL method compilation, and realized that it could emit faster code.

The %attrinit is a Perl 6 Hash object, which is for the most part a wrapper around the lower-level VM hash object, which is the actual hash storage. We were, curiously, already pulling this lower-level hash out of the Hash object and using the nqp::existskey primitive to check if there was a value, but were not doing that for actually accessing the value. Instead, an .AT-KEY('x') method call was being done. While that was being handled fairly well – inlined and so forth – it also does its own bit of existence checking. I realized we could just use the nqp::atkey primitive here instead.

Later on, I also realized that we could do away with nqp::existskey and just use nqp::atkey. Since a VM-level null is something that never exists in Perl 6, we can safely use it as a sentinel to mean that there is no value. That got us down to a single hash lookup.

By this point, we were just about winning the benchmark, but only by a few percent. Were there any more easy pickings?

An off-by-one

My next surprise was that the new method didn’t get inlined. It was within the size limit. There was nothing preventing it. What was going on?

Looking closer, it was even worse than that. Normally, when something is too big to be inlined, but we can work out what specialization will be called on the target, we do “specialization linking”, indicating which specialization of the code to call. That wasn’t happening. But why?

Some debugging later, I sheepishly fixed an off-by-one in the code that looks through a multi-dispatch cache to see if a particular candidate matches the set of argument types we have during optimization of a call instruction. This probably increased the performance of quite a few calls involving passing named arguments to multi-methods – and meant new was also inlined, putting us a good bit ahead on this benchmark.

What next?

The next round of performance improvements – both to this code and plenty more besides – will come from escape analysis, scalar replacement, and related optimizations. I’ve already started on that work, though expect it will keep me busy for quite a while. I will, however, be able to deliver it in stages, and am optimistic I’ll have the first stage of it ready to talk about – and maybe even merge – in a week or so.

brrt to the future: A future for fork(2)

Published by Bart Wiegmans on 2018-10-03T22:18:00

Hi hackers. Today I want to write about a new functionality that I've been developing for MoarVM that has very little to do with the JIT compiler. But it is still about VM internals so I guess it will fit.

Many months ago, jnthn wrote a blog post on the relation between perl 5 and perl 6. And as a longtime and enthusiastic perl 5 user - most of the JIT's compile time support software is written in perl 5 for a reason - I wholeheartedly agree with the 'sister language' narrative. There is plenty of room for all sorts of perls yet, I hope. Yet one thing kept itching me:
Moreover, it’s very much the case that Perl 5 and Perl 6 make different trade-offs. To pick one concrete example, Perl 6 makes it easy to run code across multiple threads, and even uses multiple threads internally (for example, performing optimization and JIT compilation on a background thread). Which is great…except the only winning move in a game involving both threads and fork() is not to play. Sometimes one just can’t have their cake and eat it, and if you’re wanting a language that more directly gives you your POSIX, that’s probably always going to be a strength of Perl 5 over Perl 6.
(Emphasis mine). You see, I had never realized that MoarVM couldn't implement fork(), but it's true. In POSIX systems, a fork()'d child process inherits the full memory space, as-is, from its parent process. But it inherits only the forking thread. This means that any operations performed by any other thread, including operations that might need to be protected by a mutex (e.g. malloc()), will be interrupted and unfinished (in the child process). This can be a problem. Or, in the words of the linux manual page on the subject:

       *  The child process is created with a single thread—the one that
called fork(). The entire virtual address space of the parent is
replicated in the child, including the states of mutexes,
condition variables, and other pthreads objects; the use of
pthread_atfork(3) may be helpful for dealing with problems that
this can cause.

* After a fork() in a multithreaded program, the child can safely
call only async-signal-safe functions (see signal-safety(7)) until
such time as it calls execve(2).

Note that the set of signal-safe functions is not that large, and excludes all memory management functions. As noted by jnthn, MoarVM is inherently multithreaded from startup, and will happily spawn as many threads as needed by the program. It also uses malloc() and friends rather a lot. So it would seem that perl 6 cannot implement POSIX fork() (in which both parent and child program continue from the same position in the program) as well as perl 5 can.

I was disappointed. As a longtime (and enthusiastic) perl 5 user, fork() is one of my favorite concurrency tools. Its best feature is that parent and child processes are isolated by the operating system, so each can modify its own internal state without concern for concurrency,. This can make it practical to introduce concurrency after development, rather than designing it in from the start. Either process can crash while the other continues. It is also possible (and common) to run a child process with reduced privileges relative to the parent process, which isn't possible with threads. And it is possible to prepare arbitrarily complex state for the child process, unlike spawn() and similar calls.

Fortunately all hope isn't necessarily lost. The restrictions listed above only apply if there are multiple threads active at the moment that fork() is executed. That means that if we can stop all threads (except for the one planning to fork) before actually calling fork(), then the child process can continue safely. That is well within the realm of possibility, at least as far as system threads are concerned.

The process itself isn't very exciting to talk about, actually - it involves sending stop signals to system threads, waiting for these threads to join, verifying that the forking thread is the really only active thread, and restarting threads after fork(). Because of locking, it is a bit subtle (tbere may be another user thread that is also trying to fork), but not actually very hard. When I finally merged the code, it turned out that an ancient (and forgotten) thread list modification function was corrupting the list by not being aware of generational garbage collection... oops. But that was simple enough to fix (thanks to timotimo++ for the hint).

And now the following oneliner should work on platforms that support fork():(using development versions of MoarVM and Rakudo):

perl6 -e 'use nqp; my $i = nqp::fork(); say $i;'

The main limitation of this work is that it won't work if there are any user threads active. (If there are any, nqp::fork() will throw an exception). The reason why is simple: while it is possible to adapt the system threads so that I can stop them on demand, user threads may be doing arbitrary work, hold arbitrary locks and may be blocked (possibly indefinitely) on a system call. So jnthn's comment at the start of this post still applies - threads and fork() don't work together.

In fact, many programs may be better off with threads. But I think that languages in the perl family should let the user make that decision, rather than the VM. So I hope that this will find some use somewhere. If not, it was certainly fun to figure out. Happy hacking!


PS: For the curious, I think there may in fact be a way to make fork() work under a multithreaded program, and it relies on the fact that MoarVM has a multithreaded garbage collector. Basically, stopping all threads before calling fork() is not so different from stopping the world during the final phase of garbage collection. And so - in theory - it would be possible to hijack the synchronization mechanism of the garbage collector to pause all threads. During interpretation, and in JIT compiled code, each thread periodically checks if garbage collection has started. If it has, it will synchronize with the thread that started GC in order to share the work. Threads that are currently blocked (on a system call, or on acquiring a lock) mark themselves as such, and when they are resumed always check the GC status. So it is in fact possible to force MoarVM into a single active thread even with multiple user threads active. However, that still leaves cleanup to deal with, after returning from fork() in the child process. Also, this will not work if a thread is blocked on NativeCall. Altogether I think abusing the GC in order to enable a fork() may be a bit over the edge of insanity :-)

6guts: Eliminating unrequired guards

Published by jnthnwrthngtn on 2018-09-29T19:59:28

MoarVM’s optimizer can perform speculative optimization. It does this by gathering statistics as the program is interpreted, and then analyzing them to find out what types and callees typically show up at given points in the program. If it spots there is at least a 99% chance of a particular type showing up at a particular program point, then it will optimize the code ahead of that point as if that type would always show up.

Of course, statistics aren’t proofs. What about the 1% case? To handle this, a guard instruction is inserted. This cheaply checks if the type is the expected one, and if it isn’t, falls back to the interpreter. This process is known as deoptimization.

Just how cheap are guards?

I just stated that a guard cheaply checks if the type is the expected one, but just how cheap is it really? There’s both direct and indirect costs.

The direct cost is that of the check. Here’s a (slightly simplified) version of the JIT compiler code that produces the machine code for a type guard.

/* Load object that we should guard */
| mov TMP1, WORK[obj];
/* Get type table we expect and compare it with the object's one */
MVMint16 spesh_idx = guard->ins->operands[2].lit_i16;
| get_spesh_slot TMP2, spesh_idx;
| cmp TMP2, OBJECT:TMP1->st;
| jne >1;
/* We're good, no need to deopt */
| jmp >2;
|1:
/* Call deoptimization handler */
| mov ARG1, TC;
| mov ARG2, guard->deopt_offset;
| mov ARG3, guard->deopt_target;
| callp &MVM_spesh_deopt_one_direct;
/* Jump out to the interpreter */
| jmp ->exit;
|2:

Where get_spesh_slot is a macro like this:

|.macro get_spesh_slot, reg, idx;
| mov reg, TC->cur_frame;
| mov reg, FRAME:reg->effective_spesh_slots;
| mov reg, OBJECTPTR:reg[idx];
|.endmacro

So, in the case that the guard matches, it’s 7 machine instructions (note: it’s actually a couple more because of something I omitted for simplicity). Thus there’s the cost of the time to execute them, plus the space they take in memory and, especially, the instruction cache. Further, one is a conditional jump. We’d expect it to be false most of the time, and so the CPU’s branch predictor should get a good hit rate – but branch predictor usage isn’t entirely free of charge either. Effectively, it’s not that bad, but it’s nice to save the cost if we can.

The indirect costs are much harder to quantify. In order to deoptimize, we need to have enough state to recreate the world as the interpreter expects it to be. I wrote on this topic not so long ago, for those who want to dive into the detail, but the essence of the problem is that we may have to retain some instructions and/or forgo some optimizations so that we are able to successfully deoptimize if needed. Thus, the presence of a guard constrains what optimizations we can perform in the code around it.

Representation problems

A guard instruction in MoarVM originally looked like:

sp_guard r(obj) sslot uint32

Where r(obj) is an object register to read containing the object to guard, the sslot is a spesh slot (an entry in a per-block constant table) containing the type we expect to see, and the uint32 indicates the target address after we deoptimize. Guards are inserted after instructions for which we had gathered statistics and determined there was a stable type. Things guarded include return values after a call, reads of object attributes, and reads of lexical variables.

This design has carried us a long way, however it has a major shortcoming. The program is represented in SSA form. Thus, an invoke followed by a guard might look something like:

invoke r6(5), r4(2)
sp_guard r6(5), sslot(42), litui32(64)

Where r6(5) has the return value written into it (and thus is a new SSA version of r6). We hold facts about a value (if it has a known type, if it has a known value, etc.) per SSA version. So the facts about r6(5) would be that it has a known type – the one that is asserted by the guard.

The invoke itself, however, might be optimized by performing inlining of the callee. In some cases, we might then know the type of result that the inlinee produces – either because there was a guard inside of the inlined code, or because we can actually prove the return type! However, since the facts about r6(5) were those produced by the guard, there was no way to talk about what we know of r6(5) before the guard and after the guard.

More awkwardly, while in the early days of the specializer we only ever put guards immediately after the instructions that read values, more recent additions might insert them at a distance (for example, in speculative call optimizations and around spesh plugins). In this case, we could not safely set facts on the guarded register, because those might lead to wrong optimizations being done prior to the guard.

Changing of the guards

Now a guard instruction looks like this:

sp_guard w(obj) r(obj) sslot uint32

Or, concretely:

invoke r6(5), r4(2)
sp_guard r6(6), r6(5), sslot(42), litui32(64)

That is to say, it introduces a new SSA version. This means that we get a way to talk about the value both before and after the guard instruction. Thus, if we perform an inlining and we know exactly what type it will return, then that type information will flow into the input – in our example, r6(5) – of the guard instruction. We can then notice that the property the guard wants to assert is already upheld, and replace the guard with a simple set (which may itself be eliminated by later optimizations).

This also solves the problem with guards inserted away from the original write of the value: we get a new SSA version beyond the guard point. This in turn leads to more opportunities to avoid repeated guards beyond that point.

Quite a lot of return value guards on common operations simply go away thanks to these changes. For example, in $a + $b, where $a and $b are Int, we will be able to inline the + operator, and we can statically see from its code that it will produce an Int. Thus, the guard on the return type in the caller of the operator can be eliminated. This saves the instructions associated with the guard, and potentially allows for further optimizations to take place since we know we’ll never deoptimize at that point.

In summary

MoarVM does lots of speculative optimization. This enables us to optimize in cases where we can’t prove a property of the program, but statistics tell us that it mostly behaves in a certain way. We make this safe by adding guards, and falling back to the general version of the code in cases where they fail.

However, guards have a cost. By changing our representation of them, so that we model the data coming into the guard and after the guard as two different SSA versions, we are able to eliminate many guard instructions. This not only reduces duplicate guards, but also allows for elimination of guards when the broader view afforded by inlining lets us prove properties that we weren’t previously able to.

In fact, upcoming work on escape analysis and scalar replacement will allow us to start seeing into currently opaque structures, such as Scalar containers. When we are able to do that, then we’ll be able to prove further program properties, leading to the elimination of yet more guards. Thus, this work is not only immediately useful, but also will help us better exploit upcoming optimizations.