Raku RSS Feeds
Roman Baumer (Freenode: rba #raku or ##raku-infra) / 2022-05-24T03:19:32Wenzel P.P. Peppmeyer has taken last week’s Weekly Challenge to show how you can have your math look like Raku, or was it the other way around? And using a simple tweak, making the code use multiple CPU’s to get results faster (/r/rakulang comments)!
The minutes of the first online meeting of people working on the Raku Documentation have been published. Wanna get involved? Check out the rest of the documentation wiki or show yourself on the #raku-doc IRC channel!
That you can use more than one adverb on hash slices? For instance, you can use the :p and :delete adverbs at the same time to easily split a hash into two hashes:
my %h = a => 42, b => 666, c => 314, d => 737;
my %s = %h<a c>:p:delete;
say %h.kv; # (d 737 b 666)
say %s.kv; # (a 42 c 314)The :p adverb makes sure that key/value Pairs are returned, and the :delete adverb makes sure that the selected keys are removed. The result from the slice then feeds the new hash.
Weekly Challenge #166 is available for your perusal.
nqp named anon methods.Date(Time) coercion:max_threads(*|Inf) as option to ThreadPoolScheduler.head / .tail methods on native arrays between 60% and 25x faster, and fixed the BUILDPLAN helper module.A slow week ahead of the 2022.05 release. The movement of Raku community modules from the git ecosystem to the zef ecosystem is progressing nicely. And at least one new module showing a lot of promise!
This week’s image was created using two bicycle bags. The Rakudo Weekly News will continue to show Ukrainian inspired images in support of Ukrainian people, and any other people who are trying to make an end to the Russian aggression in Ukraine. Слава Україні! Героям слава!
In the mean time, please stay safe, stay healthy, keep up the good work!
If you like what I’m doing, committing to a small sponsorship would mean a great deal!
PWC 165 refers us to mathsisfun for the algorithm to be used. Let’s write it down.
my $input = '333,129 39,189 140,156 292,134 393,52 160,166 362,122 13,193
341,104 320,113 109,177 203,152 343,100 225,110 23,186 282,102
284,98 205,133 297,114 292,126 339,112 327,79 253,136 61,169
128,176 346,72 316,103 124,162 65,181 159,137 212,116 337,86
215,136 153,137 390,104 100,180 76,188 77,181 69,195 92,186
275,96 250,147 34,174 213,134 186,129 189,154 361,82 363,89';
my @points = $input.words».split(',')».Int;
my \term:<x²> := @points[*;0]».&(*²);
my \xy = @points[*;0] »*« @points[*;1];
my \Σx = [+] @points[*;0];
my \Σy = [+] @points[*;1];
my \term:<Σx²> = [+] x²;
my \Σxy = [+] xy;
my \N = [email protected];
my $m = (N * Σxy - Σx * Σy) / (N * Σx² - (Σx)²);
my $b = (Σy - $m * Σx) / N;
say [$m, $b];That was exceedingly simple. If the point cloud is large, this might also be exceedingly slow. There are no side effects and everything is nice and constant. Should be easy to get the CPU busy.
sub delayed(&c) is rw {
my $p = start { c };
Proxy.new: STORE => method (|) {}, FETCH => method { await $p; $p.result }
}
my \term:<x²> = delayed { @points[*;0]».&(*²) };
my \xy = delayed { @points[*;0] »*« @points[*;1] };
my \Σx = delayed { [+] @points[*;0] };
my \Σy = delayed { [+] @points[*;1] }
my \term:<Σx²> = delayed { [+] x² };
my \Σxy = delayed { [+] xy };
my \N = [email protected];
my $m = (N * Σxy - Σx * Σy) / (N * Σx² - (Σx)²);
my $b = (Σy - $m * Σx) / N;
say [$m, $b];Since I use sigil-less symbols, I get binding for free. This in turn makes using Proxy easy. The await $p part is called more then once per Proxy but not in a loop so there is no real need to optimise here. On my box the Proxyed version is almost twice as fast. A nice win for such a simple change.
I somehow feel this solution might please a mathematician — as Raku should.
Anton Antonov reminded us that there is a cool Raku module in the ecosystem called Chemistry::Stoichiometry which allows you to do stochiometry: calculation of molecular masses for chemical compound formulas, and perhaps more interestingly doing chemical equations balancing! With support for many (natural) languages!
A new version of Raku Navigator (a Visual Studio Code extension for the Raku Programming Language) has been announced by bscan. Looking pretty slick!
Will Coleda informs us that the initial documentation online meetup will be at noon Eastern Time on Saturday, the 21st of May (that’d be 19:00 UTC). Please contact Will by mail or on the #raku-doc IRC channel for specifics (any modern web browser should be sufficient to be able to participate).
With feedback from the community, Andrew Shitov will most likely organize an on-line only Raku Conference, probably in August. It’s a bit sad, but with only 3 months to go and people already having made holiday plans, it seemed there would be insufficient attendance to warrant an in-person event.
Only a few days left to hand in your Grant Proposals for Raku projects to the Grant Committee for the May 2022 round!
It’s that time of the year again: the 2022 StackOverflow Developer Survey would like you to participate. Let’s make sure that “Raku” and “CommaIDE” are one of the multiple choice options next year, ok?
Alexey Melezhik has given an update on the Sparky CI, continuous integration testing using the Raku Programming Language.
That many phasers actually have a return value? You can e.g. use the ENTER phaser to “freeze” an entry value, and a LEAVE phaser to give you an adhoc idea of how long a scope took to execute:
given (2..5).pick {
LEAVE say (now - ENTER now) ~ 'seconds have passed';
say "waiting $_ seconds";
sleep $_;
}Weekly Challenge #165 is available for your perusal.
cas, and has been very busy on the RakuAST branch: such as being able to export things, allowing for specification of coercion types and the is copy trait on parameters.le on native strings (which existed since 2012)!DateTime no longer accepts ISO8601 strings without the time part? by zeekar.For those of you puzzled about this week’s headline of the Rakudo Weekly News: this was the result of the popular vote for the Ukraine entry into the Eurovision Song Contest 2022. Of the maximum number of points a country could get (39 countries * 12 = 468), it received 439 (93.8%) from tele-voting (aka the public). Showing massive support from the ESC audiences!
Meanwhile the Rakudo Weekly News will continue to show Ukrainian inspired images in support of Ukrainian people, and any other people who are trying to make an end to the Russian aggression in Ukraine. Слава Україні! Героям слава!
In the mean time, please stay safe, stay healthy, keep up the good work!
If you like what I’m doing, committing to a small sponsorship would mean a great deal!
Not too long ago, someone on Twitter shared a story about the creation of Atari’s classic video game Pong — The Lies that Powered the Invention of Pong — IEEE Spectrum. I love stories about the dawn of home computing, so I curiously opened the link on my phone.
Have in mind that Pong were a game without a CPU or memory in the modern sense of the words. Later versions of Pong were ported to traditional computer systems, and could run on computers with as little as 2 KB, but the original Pong’s game logic was embedded in the circuitry.
The screen was equivalent to about 200x200 pixels with 2 colors, white and black. Had the contents of the screen been rendered out and stored digitally – which of course it neither was nor could, given the resources— it would have taken just under 10 KB uncompressed. But the console didn’t use pixel graphics; it would have been too resource hungry. It used “timing circuits used to encode the video signal and turn it on or off depending on its location” [source]. Its cleverness when it comes to conserving resources, is beyond impressive.
Since the article basically was all about conserving resources, I were even more puzzled than I normally am by one thing: After a cascade of full screen ads and modal accept-cookie windows, I finally got to the 5431 character long story. And then only to be interrupted after two mere paragraphs — 812 characters — by a button labeled “Keep reading ⬇”.
If we go back in time to the days before unlimited mobile plans and fast 4G/5G, the reason for these buttons were to save users download size and time and therefore also money. The smaller the download was, the better and cheaper it was for the user. But today it’s not as if postponing a download of 4619 characters — which all even are from the space saving lower 7 bits of the ASCII character set — saves the user measurable download time nor size. The full text of the article is equivalent to a mere 0.16 % of the total download.
With the rise of arguably bloated Javascript libraries such as React, resources seem to concern absolutely no one. The size of content is dwarfed (by orders of magnitude) by the size of libraries. It seems as if the “read more” buttons are there just because of old habits; no one seems to question why they’re there anymore.
What we’re left with in 2022 is a functionality that most of all is a hindrance: Readers visits web pages for one thing only, the article. And that one things is the only thing we limit and split. Everything else downloads and renders uninterrupted. Isn’t it strange?
But bandwidth usage is not the only a resource waste here. Take images as an example.
The screen shot I use above is a 24-bit 403x242 pixels rendering of a 2-bit equivalent game screen. Uncompressed in memory it needs 286 KB RAM. That’s almost 29 times the size compared to the original Pong system. Had the person making the screen shot reduced its bit depth to 2, the image would have taken less than 24 KB of my computer’s RAM.
If I open the article on my PC —it sports a 4 core i7 CPU and 16 GB memory, a veritable super computer compared to almost any previous computer system — a single-tab instance of Google Chrome uses a whopping 19 % of my CPU while rendering this simple layout. Simultaneously, Chrome’s total memory usage increases 253 MB, from 323 MB when idle to 576 MB, when rendering the article. That’s well over half a gigabyte spent laying out a few lines of text and a photo.
Compared to the minimal gain from splitting articles in two, this resource usage seems crazy to me.
I’ve been a web developer for 24 years. I’m a dinosaur that come from the era before even database driven web sites became popular. So I know that things are better now than then. Thus I have little nostalgia regarding how we did things before. Old timers, like me, that worked in a world of dial-up Internet, were self taught masters of none — except for three thing that we were experts at by necessity:
I worked at an online newspaper, and I remember how we the first 5–6 years tried to keep the extremely information dense front page below 100 KB at any given time. This is of course not possible today, where increasing screen resolutions demands bigger images and functionality demands stuff like videos and animation. But how much do you really need?
Using some of these tricks of yore, the article referred in the beginning — even including today’s increased demand of visual elements and interactive features — could have been a less than a 0.5 MB document if some thought was spent on optimization.
But it’s not. Let’s take a look at the article’s photo as an example. It is a 24-bit 2580x3441 pixels image. If we compare this to laptop screens, that size is ridiculous. Most laptop screens are around 1920x1080 pixels. Even newer ones, such as the Macs’ retina displays, is scaled down by the operating system to 1920x1080 or less anyway. When the article renders on a full-screen instance of Chrome on such a display, the client still scales the image down to 953x625. In other words, the client discards 93 % of the data that originally was transferred from the server.
Had there been done more optimization remotely — a one-time reduction of size, adapted to various devices, and cached server-side — we’d reduce download size by breathtaking amounts. We’d also reduce the total CPU and energy usage by astonishing amounts in addition. But giving the client responsibility for scaling and rendering, we not only transfer too much data, but we have to spent significant CPU load and RAM usage doing the exactly same scaling on thousands or millions (in the case of popular web sites) of client computers. And just to add insult to injury, this strategy uses 14 times more bandwidth than necessary.
A similar story could be told about CPU usage if we pre-generated web page elements, instead of having React on the client side receive an overload of data and render out some of it.
Something can — and should — be done about this.
And here’s why these things matter for the environment.
We talk a lot about energy usage and the corresponding cost of Bitcoin mining. The environmental impact on global warming for instance, is real and measurable.
But I see very little talk about the web or networked apps when it comes to environment issues.
Moving rendering, calculations and more, from servers to the client side, has saved many a web site owner lots of money on energy and the number of servers necessary. But multiplying rendering cost by millions by having the clients take care of it, has an enormous environmental impact as well.
So, yes, dynamic sites are great. But making more of them a little more static, sounds like a good idea to me. The environment will thank us, and maybe even our smart phones’ battery life will improve a little as well.
And that is a win-win for everyone if there ever was one.
Will Coleda has volunteered to lead Raku Programming Language documentation efforts going forward, and is looking for your feedback!
Current known efforts are documented in the wiki. In an effort to keep volunteers coordinated, please look for a monthly call starting in May, and status updates reported out on the wiki. Or you can check in on the new #raku-doc IRC channel! Or the more old-fashioned mailing-list.
The documentation effort can always use volunteers for content creation, editing, as well as front end web development, and raku scripting. So if you have suggestions and/or would like to contribute to the documentation effort in any way, this is your chance!
We live in interesting times. Andrew Shitov is trying to decide if, when and where there will be an in-person Raku Conference this year. Or that we will have an online conference like last year. Please give your thoughts and your vote in this Reddit poll (or on FaceBook).
That you can use the so-called “capture markers” <( and )> to indicate which part of the needle you actually want to capture in a match?
say "foobarbaz" ~~ / foo <( \w+ )> baz /; # ｢bar｣Weekly Challenge #164 is available for your perusal.
LEAVE phaser (and for scopes with multiple phasers), and added a :real named argument to DateTime.posix allowing for fractional second values..&foo, among many more tweaks and fixes on the RakuAST branch.
.Apart from all of the movement on the doc repository, it was a bit of a quiet week. Therefore the rest of this winding down is the same as last week :-).
Meanwhile the Rakudo Weekly News will continue to show Ukrainian inspired images in support of Ukrainian people, and any other people who are trying to make an end to the Russian aggression in Ukraine. Слава Україні! Героям слава!
In the mean time, please stay safe, stay healthy, keep up the good work!
If you like what I’m doing, committing to a small sponsorship would mean a great deal!
Once upon a time, now was the future. For me, now started on my eleventh birthday in 1983. That day I got an 8-bit 64 Kb Japanese computer called the Sharp MZ-700, an obscure home computer even when it was new. This meant that there were very few games, or other software, available for it. If you wanted to do something, you had to do it yourself.
This is where the programming language BASIC came in. Sharp included it’s variant, S-BASIC, on cassette. I knew that if only I’d manage to learn it, I could program games myself.
So the first thing I did every time I turned on the computer, was to load BASIC from this cassette. Loading was without exception a time-consuming and an excruciating experience, but the second it was loaded, a simple yet magical greeting promised a glimpse of the future.
Ready, it said. And ready I was!
The world was greeted and BASIC was ready a few years before this. The programming language was created by John G. Kemeny and Thomas E. Kurtz at Dartmouth College in 1964. They had a few design principles, one being that the code should look as much as English as possible. They also postulated that there should be only one statement per line. You could say that their philosophy was akin to the saying “as simple as possible, but not simpler”. Unlike the interpreted versions that later would dominate home computers, Kurtz and Kemeny’s BASIC compiled to binary format. As such, BASIC programs performed more or less on par with other languages.
At first, the creators didn’t imagine BASIC being used anywhere but on the Dartmouth College’s General Electric computer system, so they didn’t bother standardizing the language through an official standardization organization. But after they set BASIC free, it spread to other universities with systems such as PDP-11, Data General systems, Hewlett-Packard computers, etc.
But the more BASIC spread, minor discrepancies started to creep in. Dialects were the result. To some, this was a disaster. In 1975, the legendary pioneer computer scientist Edsger W. Dijkstra commented: “It is practically impossible to teach good programming to students that have had a prior exposure to BASIC: as potential programmers they are mentally mutilated beyond hope of regeneration”.
He was both right and an unbearable snob at the same time. The combination of the advent of good time-sharing technologies for the mainframes of the era, and BASIC, was democratizing computing faster and further than anyone could quite predict. That was just a taste of what would come.
Because Dijkstra’s quip was, ironically, perfectly aligned with the biggest computing revolution ever. In the mid-1970s, home computers suddenly appeared on the market. Intel’s first processors found their way into products such as the Altair 8800. Other microprocessors from Zilog, Motorola and Mostek, soon followed, and found their way into products like Apple II, Commodore Pet, and others.
Regardless of the microprocessor at their core, all of these systems were under powered compared to the minicomputers that preceded them. But their relative affordability still made them immediate successes. The upside of limitations was that they forced vendors to find a programming language that not only was small enough to fit within their constraints. The language also had to be immediately understandable to amateurs that didn’t have any formal training. BASIC fit both criteria perfectly.
The language had lived its first 10 years comfortably on Big Iron computers and enabled non-expert academic users to use computers as tools. But it was home computers that made BASIC a tool for millions of new computer users from all walks of life.
In most of these computers, BASIC found its way into ROM. Whoever thought of putting BASIC on ROM chips, was a genius, even if that person did or didn’t understand the would-be historical implications of their choice.
Why do I think ROMs are so important?
My computer was a blank slate when it booted. I had to choose to use BASIC. But the first thing owners of home computers from Commodore, Amstrad, Sinclair and Apple were instantly greeted by when turning on their computer, was BASIC’s friendly “Ready”. From then on, BASIC was the computer’s programming language, operating system, UI, editor and shell in one. BASIC was, for all practical purposes, the computer’s personality.
Of course, many home computer owners probably never used BASIC as a programming language beyond the mandatory “hello world” variants. But everyone used rudimentary features of BASIC to load and start games, educational software, drawing programs, and more.
And even these users probably used BASIC without knowing it, because:
In 1983 BASIC was the only computer UI home computer owners had ever used. There were no Windows back then. BASIC was the window. For most of them, it would stay that way for years to come; the Mac was an expensive niche product when it debuted in 1984, and Windows 3.x was seven years away. The upshot was that, should these users want to explore programming, they had BASIC literally at their fingertips.
You would think the fathers of BASIC were proud of the impact their little language had had. But Kurtz and Kemeny never liked the built-in BASICs much. They used the rather derogatory term “Street BASIC” to describe variants that deviated from their original. Deviations often consisted of additional commands tailor-made for vendor specific computer systems and hardware, i.e. things like graphics and control of peripherals. If the computer supported graphic sprites, BASIC probably contained custom-made commands to control them. If it had multichannel sound, you could utilize those too through custom BASIC commands. The short one-line BASIC examples, above, were from Sharps S-BASIC. As you can see that BASIC even had custom commands for using the plotter/printer.
This increased BASICs usability for owners of the various computer systems, but reduced compatibility between computer systems. This was the originators’ main gripe.
But in my opinion, this Cambrian Explosion of BASIC variants was something they should have felt extremely proud of. By 1987, 15% of American homes had a home computer, meaning that up to 36 million people were exposed to Street BASIC. If we extrapolate these number by estimating similar figures for Japan and Western Europe, the world-wide number may have been the double. This was indeed a revolutionary reach compared to the few thousands that previously had access to it on mainframes in universities and large corporations.
To criticts such as Dijkstra, and to a certain extent BASIC’s fathers, the home computer BASICs were lacking because they didn’t have any of the more advanced features of other languages.
The Street BASICs…
…had no compilers and linkers that produced optimized binary executables. Instead, BASIC was interpreted
…had no concept of modules and libraries. That meant that a Street BASIC program had to be self-contained
…did not support named functions. Instead, the program had line numbers that you could use the command GOTO to jump to (probably one of Dijkstra’s main complaints)
…did not support using a separate text editor for programming. In addition to every other role BASIC filled, it also was its own editor
…had no concept of scope or immutability, i.e. variables were always global, and their values could always be changed
In any other context, these things would be seen as major shortcomings. But to me, they are limitations with a purpose; if you let me paraphrase, in this case limitation is the mother of invention.
So the upside of the Street BASICs’ limited scope was that…
…the lack of a compiler made programs immediately test- and runnable. There was no extra step between you, the program and its resulting outcome
…the ommition of meaningful functions and sub routines meant that you had to think about your program as a recipe with step-by-step instructions. This is not only a format most people are comfortable with; this is how humans think of algorithms. Just think about how a child would describe its day: “We went to kindergarten, and then this happened, and that happened, and someone said this, and someone did that”. That’s descriptive and linear, just like old-school BASIC
…no external editors meant, again, that there was a direct connection between the computer, its user and the software being built; the use of line numbers made using the crude line editor understandable and instinctively easy to use
…global variable scope really only was a consequence of not having functions; there simply were no need for protected or local variables, making variables easier to understand for new practitioners of BASIC
The most important difference between 80’s BASICs and other languages, however, was that the former lacked the concept of modules and libraries. But this, too, was an important limitation that actually lowered the barrier of entry; as mentioned above, the computer makers had to create top-level commands for, among other things, controlling hardware that were unique to the computer. This tight coupling between hardware and software again minimized the difference between BASIC and the computer it ran on. BASIC and the computer it ran on, seemed like one and the same.
Another good thing that came out of this shortcoming was, arguably, that it was easier to get an overview over your program: Anything it did, was visible in its source code; there never was a doubt as to where functionality were defined.
For all practical purposes, BASIC is seldom used and mostly forgotten today. But given my praise for it, why is it like that? As I see there were three reasons, and as time went by, their accumulated effect was devastating.
Reason #1.
The IBM PC, and later it’s clones, completely overtook the home computer market. On the PCs, very little were in ROM. Instead, MS-DOS was usually loaded from diskettes or hard drive, and suddenly the operating system itself became the face of the computer. Not BASIC.
So from the get-go you were one degree removed from BASIC. If you wanted to use it, you had to explicitly start it after you had started DOS. BASIC was in that sense just like any other application. Within the BASIC application you typed and run your BASIC programs as you did on the previous generation of home computers. But, ironically in a world where DOS was the face of the computer, there was no immediate way to run saved BASIC programs from the MS-DOS command line. You had to load the BASIC environment first and run them from there.
All in all, BASIC on these machines felt like an afterthought.
Reason #2.
If Street BASIC had caused incompatibilities between computer systems earlier, the newer OS-based PCs could cause incompatibilities within a system.
Since BASIC for MS-DOS were run from disk, you could obtain and use any version of BASIC that you liked. Borland’s Turbo Basic might interest you more than the Microsoft offerings. You might call this freedom of choice, something most of us value. But the downside was that if you gave your program to a friend, you couldn’t be certain that your friend’s BASIC were the same as yours.
And even if you didn’t choose to use non-bundled BASICs, but exclusively used the version that came with MS-DOS, you could be in trouble anyway. Before MS-DOS 5.0, GW-BASIC was bundled with the operating system. It was more or less like the Dartmouth inspired ones you had found on home computers. But if you had MS-DOS 5.0 or newer installed, GW-BASIC was replaced by QBasic.
QBasic was an attempt at a modern BASIC. Depending on your point of view, it really was an improvement. From a usage standpoint, it was quite different. You edited your BASIC programs in a dedicated editor, and when you ran them, the editor was completely replaced by a new screen. On this screen you could interact with your programs. When you were finished, the interactive screen closed and you were back in the editor. The tight coupling between programming and usage was now gone.
Language wise, QBasic got rid of line numbers (unless you insisted on using them). Instead, it gained labels and sub-routines with named parameters. You could also differ between local and global variables. To a certain extent, you could even explicitly type variables. It looked more like BASICs to come, than the old ones it replaced. But even though these represented a maturation of the language, it was also the start of turning BASIC from being a language where you described your algorithm step by step in a linear fashion, to a non-linear one.
All of this arguably made for a better BASIC. But the changes also made the language less comprehensible for beginners. BASIC stopped being basic.
As for compatibility, GW-BASIC programs would mostly run fine in QBasic, provided that the programs were saved as ASCII files. But as this was a non-default option, few programs were. The other way around was more or less impossible. Given the improvements in and modernization of QBasic, programs written for it could not run on GW-BASIC.
But even if you and your friend’s software stack was identical, issues could still arise. Given the modularity of PCs, you couldn’t even be sure that the graphics modes supported were identical between them. That was a solvable problem, but still a nuisance.
Reason #3.
Windows, and similar graphical user interfaces, removed BASIC even further from its roots by being something very different. You could of course choose to stay text based in a windowed world. But that just wasn’t very satisfactory.
The main problem with adapting BASIC to work with windows, is that Windows programs is a collection of, well, windows and buttons, text fields, etc. Even though parts of the software can be somewhat step-by-step, most of the software must consist of fragments catering to each button, text field and window independently. The code has no obvious entry point, and can’t be read and understood as a recipe from A-Z.
We now also got a world where the environment was so function rich that it was no longer possible to have a separate command for each function of the computer. Libraries and modules became necessary, and the notation changed into something like Module.Class.Sub(). The modularity made it possible for the ecosystem to grow so big that almost no one could have a complete overview of it anymore.
This was inevitable, but the sheer size of these modern programming languages and their corresponding libraries, now is so large that the barrier of entry for newcomers are orders of magnitude bigger than they were in the 80s.
To be clear, it is with modern programming languages — anti-BASICs — I earn my living. I wouldn’t have it any other way. But BASIC was my entry point. After a few years I moved on to Pascal, Delphi (Object Pascal), Java, JavaScript, then Perl. In recent years I’ve mostly worked with Python and a few other languages depending of what kind of problem it is I’m trying to solve. Lately I’ve been dabbling with Go. Without BASIC, I probably wouldn’t have done any of this.
I just don’t agree with Dijkstra’s somewhat condescending view that it’s impossible to teach good programming practices to those who learned BASIC first. Most of the programmers around my age that I know of, has had a similar way into professional programming.
If I lament anything at all today, it’s the fact that we no longer have an easy point of entry into the profession I love. We lost something with the disappearance of the ROMs.
POSTSCRIPT
I thought I’d end this overly long article with a parting gift to you. What’s more fitting than the graphic above?
It was made using PC-BASIC, very much a non-ROM clone of GW-BASIC. It is, ironically, written in Python, a language that is often described as the BASIC for the modern era (which it — as a multi-paradigm language with modules, classes, procedures, functional programming, and beginning support of async execution — just isn’t).
Earlier in this article I wrote about incompatibility between version on the same OS; I also mentioned incompatibilities brought on by hardware differences between otherwise similar post-home computer systems.
But PC-BASIC takes this to a whole new level by having the potential of being incompatible with itself (!). If you run the program on Windows, it works as expected, even when using graphics modes. But if you try to install it on an ARM based Mac, it just won’t work. An alternative there is to install it as a module in Python and run it from the command line. But then the graphics modes won’t work. So you can’t expect everything to work similarly everywhere.
That being said, I like PC-BASIC. If for no other reason than nostalgia. Which, perhaps, all of what I have written here is.
Oleksandr Kyriukhin announced a new Comma Complete / Community release. Check out the new features and bug fixes: with Apple M1 support, support for fez and REA ecosystems, and support for lexically scoped grammars, to name but a few. In case you didn’t know, Comma is the IDE for the Raku Programming Language.
In related news, bscan has announced Raku Programming Language support in Visual Studio Code. Also pretty exciting!
After many years, JJ Merelo has decided to step back from taking responsibility for the documentation of the Raku Programming Language. On behalf of the Raku Community, the Raku Steering Council would like to thank JJ for all the work they’ve done for Raku, and its documentation in particular. For many, many years! In the hope that JJ will continue to be a pillar of the Raku Programming Language!
Ben Davies (after an absence due to ill health, good to have you back!) continues on their work on Data::Record, specifically with regards to annotating type objects with thread-safe containers (/r/rakulang comments).
Steve Roe expressed their joy about the new .are method. Nice to see!
Wenzel P.P. Peppmeyer blogged about how the Raku Programming Language could use a few more built-in functions, enigmatically.
That you can start a REPL (aka a Read Evaluate Print Loop) while running code? By simply calling the repl subroutine you get a basic debugger with introspection. For instance:
for ^5 -> $i {
say $i;
repl if $i == 2;
}The output (with user input in bold):
0
1
2
Type 'exit' to leave
[0] > $i
2
[1] > exit
3
4Weekly Challenge #163 is available for your perusal.
Code|Block).clone method in the bootstrapInts even further.Positional role.protos on the RakuAST branch.Inline::Perl5 by wingfold.Quite a few blog posts / announcements this week! Glad to see all of that!
Meanwhile the Rakudo Weekly News will continue to show Ukrainian inspired images in support of Ukrainian people, and any other people who are trying to make an end to the Russian aggression in Ukraine. Слава Україні! Героям слава!
In the mean time, please stay safe, stay healthy, keep up the good work!
If you like what I’m doing, committing to a small sponsorship would mean a great deal!
As suspicious questions on IRC and Discord revealed, there are quite a few solutions to the PWC that are not made public. One question in particular indicates that there is a build-in missing in Raku.
Nemokosch: let’s say I have a 5×5 table for some reason
PWC162-2 is asking to implement an encryption algorithm that uses a simple substitution table. Having a data-struction that allowes to turn a character into a 2-dimensional @index and then use it to @replacement[||@index] would be very helpful indeed. We can use .antipairs to turn a simple list into something we can assign to a Hash and be done with it. With 2-dimension, we have to create our own.
proto sub deepantipairs(@positional, $dimensions --> Positional) {*}
multi sub deepantipairs(@a, 2) {
@a.pairs.map({ my $outer-key = .key; .value.antipairs.map({ .key => ($outer-key, .value) }) }).flat
}
my $phrase = 'Spring has sprung!';
my @table = flat($phrase.lc.comb.grep(/\w/), 'a'..'z').unique[^25].batch(5);
my %antitable = @table.&deepantipairs(2);
# OUTPUT:
# Array @table = [("S", "p", "r", "i", "n"), ("g", "h", "a", "s", "u"), ("!", "b", "c", "d", "e"), ("f", "j", "k", "l", "m"), ("o", "q", "t", "v", "w")]
# Hash %antitable = {"!" => $(2, 0), :S($(0, 0)), :a($(1, 2)), :b($(2, 1)), :c($(2, 2)), :d($(2, 3)), :e($(2, 4)), :f($(3, 0)), :g($(1, 0)), :h($(1, 1)), :i($(0, 3)), :j($(3, 1)), :k($(3, 2)), :l($(3, 3)), :m($(3, 4)), :n($(0, 4)), :o($(4, 0)), :p($(0, 1)), :q($(4, 1)), :r($(0, 2)), :s($(1, 3)), :t($(4, 2)), :u($(1, 4)), :v($(4, 3)), :w($(4, 4))}Let’s rotate the table (the basic idea behind the Enigma) to create a much more interesting cypher-text.
sub encrypt($phrase, $text --> Str) {
my @table = flat($phrase.lc.comb.grep(/\w/), 'a'..'z').unique[^25].batch(5);
my %antitable = @table.&deepantipairs(2);
my $retval;
for $text.lc.comb.grep(/\w/) -> $char {
my @deepindex := %antitable{$char};
$retval ~= @table[||@deepindex];
@table = @table».List.flat[1..*,0].flat.batch(5);
}
$retval
}
say encrypt($phrase, 'As suspicious questions on IRC and Discord revealed, there are quite a few solutions to the PWC that are not made public. One question in particular indicates that there is a build-in in Raku missing.');
# OUTPUT:
# aprdnhbmdrodhvpkdtdxtjpppcuhjkuhmpdvfybpwannjpuychrfvdasojvvadgnwcqcwqkjmndpxexcepjcvjnagvpiopidyalietcknsbseejeqkbopsbpwypbrcuwknsejinlxjsmkxppwdasrrniboewbauejlPlease note the tripple “p” in the cypher. By rotating the replacement table, we mess up statistical properties of the natural language we encrypt. This makes limiting the key-space much harder.
As you likely spotted, I defined a proto with the argument $dimensions (our item may be an Iterable so we can’t infer this argument). Raku has many very handy methods defined in List that work very well with a single dimension. There may be more work to do when we start to support fixed size shaped arrays well.
Justin DeVuyst has done all the work to release the 2022.04 version of the Rakudo compiler of the Raku Programming Language, which contains a new will complain trait for more targeted error messages when type-checks fail (courtesy of Vadim Belman). Pre-compiled archives will be available shortly. Claudio Ramirez‘s Linux packages now also supporting the new Ubuntu 22.04 release!
Matthew Stuckwisch invites you to submit a proposal for presentation at the first in-person conference since 2019 in Houston, TX. The deadline is coming very near!
Vadim Belman wrote two blog posts: one about the new will complain trait (/r/rakulang comments), and one about the pros and cons of the various ways of dealing with race conditions.
Simon Proctor write a blog post about how to deal with updates of a very large JSON file in a server setting (/r/rakulang comments).
Alexey Melezhik provided an update on Sparky CI, continuous integration testing using Sparky.
Siavash Askari Nasr reports about their work to make flymake-rakudo, which creates an Emacs flymake checker for Raku syntax checking. Check it out, especially if you’re an Emacs user.
That you can use the Z (zip) operator to create Pairs by using Z as a meta-operator?
say "a".."c" Z=> 1..* # (a => 1 b => 2 c => 3)And that you can use that to easily create a hash with mapping of strings to numerical values?
my %hash = "a".."c" Z=> 1..*;
say %hash; # {a => 1, b => 2, c => 3}Weekly Challenge #162 is available for your perusal. And the previous Weekly Challenge #161 also, of course!
All the Pull Requests of the past 2 weeks have already been merged or closed.
IO::Path.dir-with-entries method, continued their work on speeding up compilation and module installation, and added a .Failure coercion method.will complain trait (for variables, parameters, attributes, etc.), for better error messages on typecheck failures.Ints.my :($a,$b) := 42,666 syntax legal, as per speculation, and fixed an issue with using min or max giving different results when being used as a sub, or as an infix operator.CommonMark modules to pass options to the commonmark library? by Steve Dondley.move function does not work in Windows by Suman Khanal.The Rakudo Weekly News continues to show the Ukrainian inspired images support of the people in Ukraine, and any other people who are trying to make an end to the Russian aggression in Ukraine. Слава Україні! Героям слава!
In the mean time, please stay safe, stay healthy, keep up the good work!
If you like what I’m doing, committing to a small sponsorship would mean a great deal!
These clickbaiting titles are so horrible, I couldn’t stand mocking them! But at least mine speaks truth.
My recent tasks are spinning around concurrency in one way or another. And where the concurrency is there are locks. Basically, introducing a lock is the most popular and the most straightforward solution for most race conditions one could encounter in their code. Like, whenever an investigation results in a resolution that data is being updated in one thread while used in another then just wrap both blocks into a lock and be done with it! Right? Are you sure?
They used to say about Perl that “if a problem is solved with regex then you got
two problems”. By changing ‘regex’ to ‘lock’ we shift into another domain. I
wouldn’t discuss interlocks here because it’s rather a subject for a big CS
article. But I would mention an issue that is possible to stumble upon in a
heavily multi-threaded Raku application. Did you know that Lock, Raku’s most
used type for locking, actually blocks its thread? Did you also know that
threads are a limited resource? That the default ThreadPoolScheduler has a
maximum, which depends on the number of CPU cores available to your system? It
even used to be a hard-coded value of 64 threads a while ago.
Put together, these two conditions could result in stuck code, like in this example:
BEGIN PROCESS::<$SCHEDULER> = ThreadPoolScheduler.new: max_threads => 32;
my Lock $l .= new;
my Promise $p .= new;
my @p;
@p.push: start $l.protect: { await $p; };
for ^100 -> $idx {
@p.push: start { $l.protect: { say $idx } }
}
@p.push: start { $p.keep; }
await @p;
Looks innocent, isn’t it? But it would never end because all available threads
would be consumed and blocked by locks. Then the last one, which is supposed to
initiate the unlock, would just never start in first place. This is not a bug in
the language but a side effect of its architecture. I had to create
Async::Workers module a while ago to solve a task which was hit by this issue.
In other cases I can replace Lock with Lock::Async and it would just work.
Why? The answer is in the following section. Why not always Lock::Async?
Because it is rather slow. How much slower? Read on!
Lock vs. Lock::AsyncWhat makes these different? To put it simple, Lock is based on system-level
routines. This is why it is blocking: because this is the default system
behavior.
Lock::Async is built around Promise and await. The point is that in Raku
await tries to release a thread and return it back into the scheduler pool,
making it immediately available to other jobs. So does Lock::Async too: instead
of blocking, its protect method enters into await.
BTW, it might be surprising to many, but lock method of Lock::Async doesn’t
actually lock by itself.
There is one more way to protect a block of code from re-entering. If you’re well familiar with atomic operations then you’re likely to know about it. For the rest I would briefly explain it in this section.
Let me skip the part about the atomic operations as such, Wikipedia has it. In particular we need CAS (Wikipedia again and Raku implementation). In a natural language terms the atomic approach can be “programmed” like this:
Note that 1 and 3 are both atomic ops. In Raku code this is expressed in the following slightly simplified snippet:
my atomicint $lock = 0; # 0 is unlocked, 1 is locked
while cas($lock, 0, 1) == 1 {} # lock
... # Do your work
$lock ⚛= 0; # unlock
Pretty simple, isn’t it? Let’s see what are the specs of this approach:
LockItem 2 is speculative at this moment, but guessable. Contrary to Lock, we
don’t use a system call but rather base the lock on a purely computational
trick.
Item 3 is apparent because even though Lock doesn’t release it’s thread for
Raku scheduler, it does release a CPU core to the system.
As I found myself in between of two big tasks today, I decided to make a pause
and scratch the itch of comparing different approaches to locking. Apparently,
we have three different kinds of locks at our hands, each based upon a specific
approach. But aside of that, we also have two different modes of using them. One
is explicit locking/unlocking withing the protected block. The other one is to
use a wrapper method protect, available on Lock and Lock::Async. There is
no data type for atomic locking, but this is something we can do ourselves and
have the method implemented the same way, as Lock does.
Here is the code I used:
constant MAX_WORKERS = 50; # how many workers per approach to start
constant TEST_SECS = 5; # how long each worker must run
class Lock::Atomic {
has atomicint $!lock = 0;
method protect(&code) {
while cas($!lock, 0, 1) == 1 { }
LEAVE $!lock ⚛= 0;
&code()
}
}
my @tbl = <Wrkrs Atomic Lock Async Atomic.P Lock.P Async.P>;
my $max_w = max @tbl.map(*.chars);
printf (('%' ~ $max_w ~ 's') xx [email protected]).join(" ") ~ "\n", |@tbl;
my $dfmt = (('%' ~ $max_w ~ 'd') xx [email protected]).join(" ") ~ "\n";
for 2..MAX_WORKERS -> $wnum {
$*ERR.print: "$wnum\r";
my Promise:D $starter .= new;
my Promise:D @ready;
my Promise:D @workers;
my atomicint $stop = 0;
sub worker(&code) {
my Promise:D $ready .= new;
@ready.push: $ready;
@workers.push: start {
$ready.keep;
await $starter;
&code();
}
}
my atomicint $ia-lock = 0;
my $ia-counter = 0;
my $il-lock = Lock.new;
my $il-counter = 0;
my $ila-lock = Lock::Async.new;
my $ila-counter = 0;
my $iap-lock = Lock::Atomic.new;
my $iap-counter = 0;
my $ilp-lock = Lock.new;
my $ilp-counter = 0;
my $ilap-lock = Lock::Async.new;
my $ilap-counter = 0;
for ^$wnum {
worker {
until $stop {
while cas($ia-lock, 0, 1) == 1 { } # lock
LEAVE $ia-lock ⚛= 0; # unlock
++$ia-counter;
}
}
worker {
until $stop {
$il-lock.lock;
LEAVE $il-lock.unlock;
++$il-counter;
}
}
worker {
until $stop {
await $ila-lock.lock;
LEAVE $ila-lock.unlock;
++$ila-counter;
}
}
worker {
until $stop {
$iap-lock.protect: { ++$iap-counter }
}
}
worker {
until $stop {
$ilp-lock.protect: { ++$ilp-counter }
}
}
worker {
until $stop {
$ilap-lock.protect: { ++$ilap-counter }
}
}
}
await @ready;
$starter.keep;
sleep TEST_SECS;
$*ERR.print: "stop\r";
$stop ⚛= 1;
await @workers;
printf $dfmt, $wnum, $ia-counter, $il-counter, $ila-counter, $iap-counter, $ilp-counter, $ilap-counter;
}
The code is designed for a VM with 50 CPU cores available. By setting that many workers per approach, I also cover a complex case of an application over-utilizing the available CPU resources.
Let’s see what it comes up with:
Wrkrs Atomic Lock Async Atomic.P Lock.P Async.P
2 918075 665498 71982 836455 489657 76854
3 890188 652154 26960 864995 486114 27864
4 838870 520518 27524 805314 454831 27535
5 773773 428055 27481 795273 460203 28324
6 726485 595197 22926 729501 422224 23352
7 728120 377035 19213 659614 403106 19285
8 629074 270232 16472 644671 366823 17020
9 674701 473986 10063 590326 258306 9775
10 536481 446204 8513 474136 292242 7984
11 606643 242842 6362 450031 324993 7098
12 501309 224378 9150 468906 251205 8377
13 446031 145927 7370 491844 277977 8089
14 444665 181033 9241 412468 218475 10332
15 410456 169641 10967 393594 247976 10008
16 406301 206980 9504 389292 250340 10301
17 381023 186901 8748 381707 250569 8113
18 403485 150345 6011 424671 234118 6879
19 372433 127482 8251 311399 253627 7280
20 343862 139383 5196 301621 192184 5412
21 350132 132489 6751 315653 201810 6165
22 287302 188378 7317 244079 226062 6159
23 326460 183097 6924 290294 158450 6270
24 256724 128700 2623 294105 143476 3101
25 254587 83739 1808 309929 164739 1878
26 235215 245942 2228 211904 210358 1618
27 263130 112510 1701 232590 162413 2628
28 244143 228978 51 292340 161485 54
29 235120 104492 2761 245573 148261 3117
30 222840 116766 4035 241322 140127 3515
31 261837 91613 7340 221193 145555 6209
32 206170 85345 5786 278407 99747 5445
33 240815 109631 2307 242664 128062 2796
34 196083 144639 868 182816 210769 664
35 198096 142727 5128 225467 113573 4991
36 186880 225368 1979 232178 179265 1643
37 212517 110564 72 249483 157721 53
38 158757 87834 463 158768 141681 523
39 134292 61481 79 164560 104768 70
40 210495 120967 42 193469 141113 55
41 174969 118752 98 206225 160189 2094
42 157983 140766 927 127003 126041 1037
43 174095 129580 61 199023 91215 42
44 251304 185317 79 187853 90355 86
45 216065 96315 69 161697 134644 104
46 135407 67411 422 128414 110701 577
47 128418 73384 78 94186 95202 53
48 113268 81380 78 112763 113826 104
49 118124 73261 279 113389 90339 78
50 121476 85438 308 82896 54521 510
Without deep analysis, I can make a few conclusions:
Lock. Sometimes it is even indecently faster, though
these numbers are fluctuations. But on the average it is ~1.7 times as fast as
Lock.Lock.protect is actually faster than Lock.lock/LEAVE Lock.unlock. Though
counter-intuitive, this outcome has a good explanation stemming from the
implementation details of the class. But the point is clear: use the protect
method whenever applicable.Lock::Async is not simply much slower, than the other two. It demonstrates
just unacceptable results under heavy loads. Aside of that, it also becomes
quite erratic under the conditions. Though this doesn’t mean it is to be
unconditionally avoided, but its use must be carefully justified.And to conclude with, the performance win of atomic approach doesn’t make it a clear winner due to it’s high CPU cost. I would say that it is a good candidate to consider when there is need to protect small, short-acting operations. Especially in performance-sensitive locations. And even then there are restricting conditions to be fulfilled:
In other words, the way we utilize CPU matters. If aggregated CPU time consumed
by locking loops is larger than that needed for Lock to release+acquire the
involved cores then atomic becomes a waste of resources.
By this moment I look at the above and wonder: are there any use for the atomic approach at all? Hm… 😉
By carefully considering this dilemma I would preliminary put it this way: I would be acceptable for an application as it knows the conditions it would be operated in and this makes it possible to estimate the outcomes.
But it is most certainly no go for a library/module which has no idea where and how would it be used.
It is much easier to formulate the rule of thumb for Lock::Async acceptance:
Sounds like some heavily parallelized I/O to me, for example. In such cases it
is less important to be really fast but it does matter not to hit the
max_threads limit.
This section would probably stay here for a while, until Ukraine wins the war. Until then, please, check out this page!
I have already received some donations on my PayPal. Not sure if I’m permitted to publish the names here. But I do appreciate your help a lot! In all my sincerity: Thank you!
Long time no see, my dear reader! I was planning a lot for this blog, as well as for the Advanced Raku For Beginners series. But you know what they say: wanna make the God laugh – tell him your plans!
Anyway, there is one tradition I should try to maintain however hard the times are: whenever I introduce something new into the Raku language an update has to be published. No exception this time.
So, welcome a new will complain trait!
The idea of it came to be from discussion about a
PR by @lizmat. The implementation
as such could have taken less time would I be less busy lately. Anyway, at the
moment when I’m typing these lines
PR#4861 is undergoing CI testing
and as soon as it is completed it will be merged into the master. But even
after that the trait will not be immediately available as I consider it rather
an experimental feature. Thus use experimental :will-complain; will be
required to make use of it.
The actual syntax is very simple:
<declaration> will complain <code>;
The <declaration> is anything what could result in a type check exception
thrown. I tried to cover all such cases, but not sure if something hasn’t been
left behind. See the sections below.
<code> can be any Code object which will receive a single argument: the value
which didn’t pass the type check. The code must return a string to be included
into exception message. Something stringifiable would also do.
Less words, more examples!
my enum FOO
will complain { "need something FOO-ish, got {.raku}" }
<foo1 foo2 foo3>;
my subset IntD of Int:D
will complain { "only non-zero positive integers, not {.raku}" }
where * > 0;
my class Bar
will complain -> $val { "need something Bar-like, got {$val.^name}" } {}
Basically, any type object can get the trait except for composables, i.e. –
roles. This is because there is no unambiguous way to chose the particular
complain block to be used when a type check fails:
role R will complain { "only R" } {}
role R[::T] will complain { "only R[::T]" } {}
my R $v;
$v = 13; # Which role candidate to choose from??
There are some cases when the ambiguity is pre-resolved, like my R[Int] $v;,
but I’m not ready to get into these details yet.
A variable could have specific meaning. Some like to use our to configure
modules (my heavily multi-threaded soul is grumbling, but we’re tolerant to
people’s mistakes, aren’t we?). Therefore providing them with a way to produce
less cryptic error messages is certainly for better than for worse:
our Bool:D $disable-something
will complain { "set disable-something something boolean!" } = False;
And why not to help yourself with a little luxury of easing debugging when an assignment fails:
my Str $a-lexical
will complain { "string must contain 'foo'" }
where { !.defined || .contains("foo") };
The trait works with hashes and arrays too, except that it is applied not to the actual hash or array object but to its values. Therefore it really only makes sense for their typed variants:
my Str %h will complain { "hash values are to be strings, not {.^name}" };
my Int @a will complain { "this array is all about integers, not {.^name}" };
Also note that this wouldn’t work for hashes with typed keys when a key of wrong type is used. But it doesn’t mean there is no solution:
subset IntKey of Int will complain { "hash key must be an Int" };
my %h{IntKey};
%h<a> = 13;
class Foo {
has Int $.a
is rw
will complain { "you offer me {.raku}, but with all the respect: an integer, please!" };
}
sub foo( Str:D $p will complain { "the first argument must be a string with 'foo'" }
where *.contains('foo') ) {}
By this time all CI has passed with no errors and I have merged the PR.
You all are likely to know about the Russia’s war in Ukraine. Some of you know that Ukraine is my homeland. What I never told is that since the first days of the invasion we (my family) are trying to help our friends back there who fight against the aggressor. By ‘fight’ I mean it, they’re literally at the front lines. Unfortunately, our resources are not limitless. Therefore I would like to ask for any donations you could make by using the QR code below.
I’m not asking this for myself. I didn’t even think of this when I started this post. I never took a single penny for whatever I was doing for the Raku language. Even more, I was avoiding applying for any grants because it was always like “somebody would have better use for them”.
But this time I’m asking because any help to Ukrainian militaries means saving lives, both theirs and the people they protect.
SmokeMachine reported on IRC to have found an unusual use of subset.
class Base {}
class SubClass is Base {}
multi sub trait_mod:<of>(\type, Base $base, |c) { dd type; dd c }
subset Better of SubClass where { put "where: $_"; True };
my Better $a = 42;
# OUTPUT: Better
# \()
# where: 42subset creates a new type with a metaclass of Perl6::Metamodel::SubsetHOW and registers a symbol for it. The of trait is a macro-ish sub that is called at compile time, right after the new type was created. We can get hold of both arguments of that trait and do whatever we want with those type-objects. (There are no further argument, hence the \()).
The where-clause is part of the subset and is called every time we assign a value to a container which is type-checked by the type created with subset. And that is cool. We basically get a callback into container assignment but only for types that are subclasses of a user-defined type. We can even create a fancy runtime error, based on the to be assigned value.
subset Universe of Base where { die 'wrong answer' unless $_ == 42; True }
my Universe $b = 43;
# OUTPUT: wrong answer
in block at 2021-03-08.raku line 1513Red is doing all sorts of fancy meta-programming and I can highly recommend to read its source. With this discovery we got another nice tool in box.
On IRC japhb told us that he needs a class to trust another class that can’t see the to be trusted class.
class B {…}
class A {
trusts B;
}
class B {
}This is nice and simple and allows B to call private methods on A. Sadly, that only works if A and B reside in the same file because a forward declaration will cause a compile time error, unless we define the declared type in the same compilation unit.
Method resolution is a runtime creature in Raku. By carefully looking at the code, we can learn where Rakudo stores what we need to cheat with.
class A {
has $!a = 'private';
method !a { $!a }
}
multi sub trait_mod:<is>(Mu:U \obj, :$spying!) {
A.^add_trustee(obj);
}
class B is spying(A) {
has A $.a = A.new;
method m { say $!a!A::a }
}
B.new.m;Luckily, the trait is operates quite early at compile time, so we don’t mess up method dispatch. I’m not feeling to bad poking with a trait where I should not peek. We can always move quickly into NQP-land and break things.
method cheating {
use nqp;
say nqp::getattr($.a, A, 'a');
}As shown above, privacy is a matter of politeness. What leaves the question, if forcing a forward declaration to be resolved locally, to be a good design decision. I shall ponder to file a problem solving issue.
I try to avoid to allow politics or current events to seep into my blog. The topic is the 100 year language and as such wont be history. Sadly, the head-oligarch has publicly given obvious bullshit-reasons to invade a neighbouring country. I would like to use the powers of … to shed some light into the current darkness.
The official numbers (read: favouringly rounded up) for the population of Russia is 145805947 people. If only they could get themselves a decent government, that could have been quite a few hugs. But hugging they ain’t do enough, resulting in a birthrate of 1.82 children per woman. Let’s write that down.
my $woman = 145_805_947 / 2;
my $girls-per-woman := 0.91;
role Degression { }
my \degression = (2022, $woman), -> [$y, $w] { ($y + 35, $w * $girls-per-woman) but Degression } … ∞;
multi sub pretty(Degression [$year, $woman]) {
"year: $year, Russians in RU: {$woman * 2}"
}
say degression[1000/35].&pretty;
# year: 3002, Russians in RU: 10397626.7
say "Soldiers available in {(1000/35).Int} generations: {Int(degression[1000/35][1]/$woman * 140000)}";
# Soldiers available in 28 generations: 9983Here, I use an empty role to mark each value in the lazy list. Since I start with a 2-element list, I have to use destructuring and return a list in the generator function. The role allows for multi dispatch and other type checks, as shown in sub pretty.
Russia tries to build up independent industry for more then 10 years. That doesn’t work with a negative population growth. So they utterly depend on immigration from former soviet countries. They also depend on those counties to carry the financial burden to teaching the Russian language. Ukraine wants to join the EU in hopes to repeat the success of the Czechs (Hi Jonathan!) and Polsky (lol, Brexit!) to return to their spiritual home and grow the economy of their ancestors.
If you want to avoid future wars, don’t go to Russia. No people, no soldiers, no war. Also, being paid in ₽ may not be the most solid plan.
[Health warning – the following is written by the author of raku Physics::Unit & Physics::Measure – so it is an opinion piece with a strong pro-raku Physics::Measure bias…]
The raku Physics::Unit and Physics::Measure module family (“rPM”) was built to make the most of raku’s unique blend of innovative programming language features. During the build, I had the opportunity to use a lot of raku capabilities and principles, both within the module code and in the way it is used.
These raku modules took inspiration from the perl5 Physics::Unit module on cpan written by Joel Berger, in particular the smart handling of type derivations in math operations and parsing, but otherwise were written ‘blind’ without reference to other languages’ support for units of measurement.
Now that the first wave of build is done for raku Physics::Measure, it is time to take stock:
I have chosen the Python Pint package to be the “best of breed” Python equivalent of raku Physics::Measure for the comparison examples – by way of a compliment to its authors. My detailed Pint to rPM comparison table is here.
raku is already very good at embedding perl5 and Python packages via the Inline:: modules and, since it is a direct descendant of perl5 (raku was formerly known as perl6), many current raku modules are straight translations from perl5. This is not the case with Physics::Measure which was designed and built from scratch with raku.
Here is what I set out to achieve…
’25 kg m^2 / s^3’; #25JSince my scoring “system” is focused on “what did raku bring to the party”, it is highly subjective, totally one sided and biased. But hopefully -Ofun!
Overall it awards Python Pint 30/70 whereas raku Physics::Measure wins with 62/70.
See down below for the blow-by-blow comparison…
After performing this comparison, I feel that raku Physics::Measure offers a substantial step forward in code clarity, readability and maintainability over Python Pint for anyone who wants to use units of measurement for educational, engineering or scientific problems.
This example typifies what I mean:
raku helps in three distinct ways
– language features such as Grammars and Class Types were used to develop the rPM modules. In comparison Pint has had to implement these functions without similar Python core support
– particularly in the case of the lack of core Type constructs in Python, this limits the usability and type safety of Pint in comparison to rPM
– language features such as unicode, Rats and postfix operators help to make the rPM modules more usable in the problem domain and a more natural language extension (or slang)
So…:
– yes raku brings a lot to the party
– yes raku is better than Python
Do I think that these local improvements in the “units of measurement domain” are sufficient to convince scientific programmers to switch to raku to avoid the limitations of Python? No.
Do I think that raku is a next generation language with a wide range of small, but significant improvements that, put together, amount to a real step change in the toolkit available to scientific programmers? Yes.
Do I think that raku is worth (another) look by anyone who feels constrained by Python’s emphasis on conformity? Yes.
Please do provide your feedback and comments…
~p6steve
The family of four modules – Physics::Measure, Physics::Unit, Physics::Error and Physics::Constants extend the core raku object model like this.
At the highest level, in raku code terms this looks like…
At face value this is a vast improvement in consistency and type safety versus Python – however, Pint has recognised the need for Quantity Types and implemented them at the module level.
So, many practical benefits of Raku types are also there in Python Pint
But … on scratching the surface, it becomes clear that there are many disadvantages of handling types in the Pint module:
These raku examples (there are many more) have no direct equivalent in Python / Pint:
given $measurement {
when Length { say 'long' }
when Time { say 'long' }
when Speed { say 'fast' }
}
– or –
subset Limited of Speed where 0 <= *.in('mph') <= 70;
This core raku feature is a slam dunk vs. Python. You will also note that rPM automagically knows that kilometer2 meter is m^3 (Volume).
thickness = 68 * ureg.m
area = 60 * ureg.km**2
n2g = 0.5 * ureg.dimensionless
phi = 0.2
sat = 0.7
volume = area * thickness * n2g * phi * sat
285.59999999999997 kilometer2 meter
my \thickness = 68m;
my \area = '60 sq km';
my \n2g = '0.5 ①';
my \φ = 0.2;
my \sat = 0.7;
my \volume = area * thickness * n2g * φ * sat;
285600000m^3
The rPM example below is from the Physics::Constants synopsis here … there are many other rPM areas where unicode is cool like my \θ1 = <45°30′30″>; #45°30′30″ .
Mathematicians and Physicists have spent centuries applying the art of concision to concepts in order to help humans think at a higher level. This example clearly shows the potential for raku to declutter scientific code and improve perspicacity.
I thought this would be a big win for rPM … but it turns out that using a raku Grammar to parse Unit strings is matched by the functionality of the Pint string parser.
Both handle a similar input range:
rPM output defaults to the abbreviated initial(s)
rPM goes beyond Pint by knowing that “kg m^2/s^2” is the same as “J” and automatically applies the SI Derived Unit relationship (so my Pint score is off a couple of pips)
Pint does this…
> m | meter | metre
meter
> ft | foot | feet
foot
> km | kilom | kilometer
kilometer
> J | joule | joules
joule
> kg m^2 | s^2 | kg m^2/s^2 | kg m**2/s**2 | kg·m²/s²
kilogram·meter²/second²
rPM does this…
> m | meter | metre
m
> ft | foot | feet
ft
> km | kilom | kilometer
km
> J | joule | joules
J
> kg m^2 | s^2 | kg m^2/s^2 | kg m**2/s**2 | kg·m²/s²
J
All the same I would still lean on the raku saves 70% post I wrote some time back that shows just how powerful and concise raku Grammars are compared to rolling your own recursive descent parser (in that case compared to perl)
Both Python Pint and rPM do a thorough job here. In fact during my study of Python Pint I found that they have better support for the slight difference between beer barrel and oil barrel volume … so an overall win for Pint.
The raku Physics::Error module introspects the precision and relative size of the Error value and uses this to control the rounding of the Physics::Measure value that is output.
my $val1 = 2.8275793cm ±5%;
#2.8276cm ±0.141
my $val2 = 2.8275793cm #without Error
#2.8275793cm
This is a cool trick that Python Pint does not have. However, Pint uses the Python uncertainties module that has more stats oriented standard deviation uncertainty propagation whereas raku Physics::Error is a more basic single value model. So, for now, Pint has the edge on this until the raku eco-system catches up…
And, for the record here are the raku Type Classes implemented:
On Saturday, I was honoured to be among the leaders of the raku community in the FOSDEM 2022 raku devroom. Thanks to Andrew Shitov for organising and to all those who were able to join.
Here’s the official video of my presentation for those who missed the talk (and so that I can find it again in future):
If you prefer to roll up your sleeves and try this yourself, rather than watch the video, then just open the Yacht::Navigation Jupyter Binder … eg/Chapter 9 covers Navigation Lights.
To recap on the main points:
The bit about Raku Grammar for Navigation Lights seemed like a great raku feature to present in the context of a usable (if rather controlled) real world problem and to illustrate how the combination of Grammars and OO are a big game changer for coders with intermediate skills (like me) who would not naturally reach for the Rec::Descent big guns.
~p6steve
In November (2021) I wrote a couple of posts bemoaning the headaches of the Apple Intel to ARM architecture shift (part I and part II) before coming to a solution that works for me. (raku on docker on ubuntu on vftools as set out at the end of part II).
One of the drivers to choose this option was to get the whole of my (raku) stack running native on ARM (–platform linux/arm64) to get the performance boost of the new M1 CPU architecture.
Going back to something I posted in January 2021 – where I ran through the progressive speed ups that refactoring my code had achieved – there were a couple of test timings that I can now rerun.
| secs user time Jan 2021 (1.2GHz/8GB/2 core macOS Intel core M) | first compile | pre-compiled |
| use Physics::Measure; | 10 | 1.2 |
| use Physics::Measure :ALL; | 13 | 2.8 |
| secs user time Jan 2022 (3.2GHz/8GB/8 core macOS Apple M1) | first compile | pre-compiled |
| use Physics::Measure; | 4.5 | 0.9 |
| use Physics::Measure :ALL; | 5.4 | 1.5 |
| % improvement | first compile | pre-compiled |
| use Physics::Measure; | 122% | 33% |
| use Physics::Measure :ALL; | 141% | 87% |
So – the longer running tasks are showing a throughput improvement of up to 2.4x … my config. efforts and cash have been quite beneficial, then!
I also wanted to check the load across CPUs…
So it seems that my raku toolchain is fairly evenly loading the 4 performance cores … but this quick compile does not run long enough to get above about 60% load. The even spread is a very encouraging aspect of the built in raku concurrency support.
[Also – bear in mind that the new results are running in a docker container on an ubuntu VM on vftools on macOS – so there is a new level of system capability in play.]
Finally, I ran a full on raku load to check if all 4 performance CPUs would be kept busy …
$> raku -e '.say for ^∞'
This settles at %CPU = 121+76+24+10 = 230% (where 400% is the theoretical max of 100% per CPU times 4 CPUs). Green is the user process (terminal) and red the system process (vftools). Again, it is good to see raku evenly spreading the load across all performance cores.
Hmmm – wonder if I can get something to use the GPUs 
As ever, your feedback and comments are very welcome…
~p6steve
left-pad by Daniel Sockwell (/r/ProgrammingLanguagescomments).Around this time last year, Jonathan Worthington was writing their Advent Post called Reminiscence, refinement, revolution. Today, yours truly finds themselves writing a similar blog post after what can only be called a peculiar year in the world.
The most visible highlights in the Raku Programming Language are basically:
last / next with a valueuse v6.e.PREVIEW;
say (^5).grep: { $_ == 3 ?? last False !! True } # (0 1 2)
say (^5).grep: { $_ == 3 ?? last True !! True } # (0 1 2 3)Normally, last just stops an iteration, but now you can give it a value as well, which can be handy e.g. in a grep where you know the current value is the last, but you still want to include it.
use v6.e.PREVIEW;
say (^5).map: { next if $_ == 2; $_ } # (0 1 3 4)
say (^5).map: { next 42 if $_ == 2; $_ } # (0 1 42 3 4)Similarly with map, if you want to skip a value (which was already possible), you can now replace that value by another value.
Note that you need to activate the upcoming 6.e Raku language level to enable this feature, as there were some potential issues when activated in 6.d. But that’s just one example of future proofing the Raku Programming Language.
.pick(**)The .pick(*) call will produce all possible values of the Iterable on which it is called in random order, and then stop. The .pick(**) will do the same, but then start again producing values in (another) random order until exhausted, ad infinitum.
.say for (^5).pick(* ); # 34021
.say for (^5).pick(**); # 340210214334210....Nothing essential, but it is sure nice to have 
.
is implementation-detail traitThe is implementation-detail trait indicates that something that is publicly visible, still should be considered off-limits as it is a detail of the implementation of something (whether that is your own code, or the core). This will also mark something as invisible for standard introspection:
class A {
method foo() is implementation-detail { }
method bar() { }
}
.name.say for A.^methods; # barBUILDALLSubroutines and methods in the core that are considered to be an implementation-detail, have been marked as such. This should make it more clear which parts of the Rakudo implementation are game, and which parts are off-limits for developers (knowing that they can be changed without notice). Yet another way to make sure that any Raku programs will continue to work with future versions of the Raku Programming Language.
There were many smaller and bigger fixes and improvements “under the hood” of the Raku Programming Language. Some code refactoring that e.g. made Allomorph a proper class, without changing any functionality of allomorphs in general. Or speeding up by using smarter algorithms, or by refactoring so that common hot code paths became smaller than the inlinining limit, and thus become a lot faster.
But the BIG thing in the past year, was that the so-called “new-disp” work was merged. In short, you could compare this to ripping out a gasoline engine from a car (with all its optimizations for fuel efficiency of 100+ years of experience) and replacing this by an electrical engine, while its being driven running errands. And although the electrical engine is already much more efficient to begin with, it still can gain a lot from more optimizations.
For yours truly, the notion that it is better to remove certain optimizations written in C in the virtual machine engine, and replace them by code written in NQP, was the most eye-opening one. The reason for this is that all of the optimization work that MoarVM does at runtime, can only work on the parts it understands. And C code, is not what MoarVM understands, so it can not optimize that at runtime. Simple things such as assignment had been optimized in C code and basically had become an “island” of unoptimization. But no more!
The current state of this work, is that it for now is a step forward, but also a step back in some aspects (at least for now). Some workflows most definitely have benefitted from the work so far (especially if you dispatch on anything that has a where clause in it, or use NativeCall directly, or indirectly with e.g. Inline::Perl5). But the bare startup time of Rakudo has increased. Which has its effect on the speed with which modules are installed, or testing is being done.
The really good thing about this work, is that it will allow more people to work on optimizing Rakudo, as that optimizing work can now be done in NQP, rather than in C. The next year will most definitely see one or more blog posts and/or presentations about this, to lower the already lowered threshold even further.
In any case, kudos to Jonathan Worthington, Stefan Seifert, Daniel Green, Nicholas Clark and many, many others for pulling this off! The Virtual Machine of choice for the Raku Programming Language is now ready to be taken for many more spins!
Thanks to Cro, a set of libraries for building reactive distributed systems (lovingly crafted to take advantage of all Raku has to offer), a number of ecosystem related services have come into development and production.
The new zef ecosystem has become of age and is now supported by various developer apps, such as App::Mi6, which basically reduces the distribution upload / commit process to a single mi6 release↵. Recommended by yours truly, especially if you are about to develop a Raku module from scratch. There are a number of advantages to using the zef ecosystem.
Whenever you upload a new distribution to the zef ecosystem, it becomes (almost) immediately available for download by users. This is particularly handy for CI situations, if you are first updating one or more dependencies of a distribution: the zef CLI wouldn’t know about your upload upto an hour later on the older ecosystem backends.
Distributions from the older ecosystem backends could be removed by the author without the ecosystem noticing it (p6c), or not immediately noticing it (CPAN). Distributions, once uploaded to the zef ecosystem, can not be removed.
The zef ecosystem is completely written in the Raku Programming Language itself. And you could argue that’s one more place where Raku is in production. Kudos to Nick Logan and Tony O’Dell for making this all happen!
raku.land is a place where one can browse the Raku ecosystem. A website entirely developed with the Raku Programming Language, it should be seen as the successor of the modules.raku.org website, which is not based on Raku itself. Although some of the features are still missing, it is an excellent piece of work by James Raspass and very much under continuous development.
“Those who cannot remember the past are condemned to repeat it.” George Santanaya has said. And that is certainly true in the context of the Raku Programming Language with its now 20+ year history.
Even though distributions can not be removed from the zef ecosystem, there’s of course still a chance that it may become unavailable temporarily, or more permanently. And there are still many distributions in the old ecosystems that can still disappear for whatever reason. Which is why the Raku Ecosystem Archive has been created: this provides a place where (ideally) all distributions ever to be available in the Raku ecosystem, are archived. In Perl terms: a BackPAN if you will. Before long, this repository will be able to serve as another backend for zef, in case a distribution one needs, is no longer available.
A lot of blog post have been written in the 20+ year history of what is now the Raku Programming Language. They provide sometimes invaluable insights into the development of all aspects of the Raku Programming Language. Sadly, some of these blog posts have been lost in the mists of time. To prevent more memory loss, the CCR – The Raku Collect, Conserve and Remaster Project was started. I’m pretty sure a Cro-driven website will soon emerge that will make these saved blog posts more generally available. In the meantime, if you know of any old blog posts not yet collected, please make an issue for it.
Ever since 2005, IRC has been the focal point of discussions between developers and users of the Raku Programming Language. In order to preserve all these discussions, a repository was started to store all of these logs, up to the present. Updating of the repository is not yet completey automated, but if you want to search something in the logs, or just want to keep up-to-date without using an IRC client, you can check out the experimental IRC Logs server (completely written in the Raku Programming Language).
So what will the coming year(s) bring? That is a good question.
The Raku Programming Language is an all volunteer Open Source project without any big financial backing. As such, it is dependent on the time that people put into it voluntarily. That doesn’t mean that plans cannot be made. But sometimes, sometimes even regularly, $work and Real Life take precedence and change the planning. And things take longer than expected.
If you want to make things happen in the Raku Programming Language, you can start by reporting problems in a manner that other people can easily reproduce. Or if it is a documentation problem, create a Pull Request with the way you think it should be. In other words: put some effort into it yourself. It will be recognized and appreciated by other people in the Raku Community.
Now that we’ve established that, let’s have a look at some of the developments now that we ensured the Raku Programming Language is a good base to build more things upon.
The tools that “new-disp” has made available, haven’t really been used all that much yet: the emphasis was on making things work again (after the engine had been ripped out)! So one can expect quite a few performance improvements to come to fruition now that it all works. Which in turn will make some language changes possible that were previously deemed too hard, or affecting the general performance of Raku too much.
Jonathan Worthington‘s focus has been mostly on the “new-disp” work, but the work on RakuAST will be picked up again as well. This should give the Raku Programming Language a very visible boost, by adding full blown macro and after that proper slang support. While making all applications that depend to an extent on generating Raku code and then executing it, much easier to make and maintain (e.g. Cro routing and templates, printf functionality that doesn’t depend on running a grammar every time it is called).
Cro driven websitesIt looks like most, if not all Raku related websites, will be running on Cro in the coming year. With a few new ones as well (no, yours truly will not reveal more at this moment).
After the work on RakuAST has become stable enough, a new language level (tentatively still called “6.e”) will become the default. The intent is to come with language levels more frequently than before (the last language level increase was in 2018), targeting a yearly language level increase.
The new #raku-beginner channel has become pretty active. It’s good to see a lot of new names on that channel, also thanks to a Discord bridge (kudos to Wenzel P.P. Peppmeyer for that).
The coming year will see some Raku-only events. First, there is the Raku DevRoom at FOSDEM (5-6 February), which will be an online-only event (you can still sign up for a presentation or a lightning talk!). And if all goes ok, there will be an in-person/online hybrid Raku Conference in Riga (August 11-13 2022).
And of course there are other events where Raku users are welcome: the German Perl/Raku Workshop (30 March/1 April in Leipzig), and the Perl and Raku Conference in Houston (21-25 June).
And who knows, once Covid restrictions have been lifted, how many other Raku related events will be organized!
This year saw the loss of a lot of life. Within the Raku Community, we sadly had to say goodbye to Robert Lemmen and David H. Adler. Belated kudos to them for their contributions to what is now the Raku Programming Language, and Open Source in general. You are missed!
Which brings yours truly to instill in all of you again: please be careful, stay healthy and keep up the good work!
After all Rakuing along all Christmas, Santa realizes it’s a pretty good idea to keep things packed and ready to ship whenever it’s needed. So it looks at containers. Not the containers that might or might not actually be doing all the grunt work for bringing gifts to all good boys and girls in the world, but containers that are used to pack Raku and ship it or use it for testing. Something you need to do sooner or later, and need to do real fast.
The base container needs to be clean, and tiny, and contain only what’s strictly necessary to build your application on. So it needs a bunch of binaries and that’s that. No ancillary utilities, nothing like that. Enter jjmerelo/raku, a very bare bones container, that takes 15 MBytes and contains only the Rakudo compiler, and everything it needs to work. It’s also available from GHCR, if that’s more to your taste.
You only need that to run your Raku programs. For instance, just print all environment variables that are available inside the container:
time podman run --rm -it ghcr.io/jj/raku:latest -e 'say %*ENV'
Which takes around 6 seconds in my machine, most of it devoted to downloading the container. Not a bad deal, really, all things considered.
The thing it, it comes in two flavors. Alternative is called jj/raku-gha, for obvious reasons: It’s the one that will actually work in side GitHub actions, which is where many of you will eventually use it. The difference? Well, a tiny difference, but one that took some time to discover: its default user, called raku, uses 1001 as UID, instead of the default 1000.
Right, I could have directly used 1001 as the single UID for all of them, but then I might have to do some more changes for GitHub Actions, so why bother?
Essentially, the user that runs GitHub actions uses that UID. We want our package user to be in harmony with the GHA user. We achieve harmony with that.
We will probably need zef to install new modules. And while we’re at it, we might also need to use a REPL in an easier way. Enter alpine-raku, once again in two flavors: regular and gha. Same difference as above: different UID for the user.
Also, this is the same jjmerelo/alpine-raku container I have been maintaining for some time. Its plumbing is now completely different, but its functionality is quite the same. Only it’s slimmer, so faster to download. Again
time podman run --rm -it ghcr.io/jj/raku-zef:latest -e 'say %*ENV'
Will take a bit north of 7 seconds, with exactly the same result. But we will see an interesting bit in that result:
{ENV => /home/raku/.profile, HOME => /home/raku, HOSTNAME => 2b6b1ac50f73, PATH => /home/raku/.raku/bin:/home/raku/.raku/share/perl6/site/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin, PKGS => git, PKGS_TMP => make gcc linux-headers musl-dev, RAKULIB => inst#/home/raku/.raku, TERM => xterm, container => podman}
And that’s the RAKULIB bit. What I’m saying it is that, no matter what the environment says, we’re gonna have an installation of Raku in that precise directory. Which is the home directory, and it should work, right? Only it does not, because GitHub Actions change arbitrarily the HOME variable, which is where Raku picks it from.
This was again something that required a bit of work and understanding where Rakudo picks its configuration. If we run
raku -e 'dd $*REPO.repo-chain'
We will obtain something like this:
(CompUnit::Repository::Installation.new(prefix => "/home/jmerelo/.raku"),CompUnit::Repository::Installation.new(prefix => "/home/jmerelo/.rakubrew/versions/moar-2021.10/install/share/perl6/site"), CompUnit::Repository::Installation.new(prefix => "/home/jmerelo/.rakubrew/versions/moar-2021.10/install/share/perl6/vendor"), CompUnit::Repository::Installation.new(prefix => "/home/jmerelo/.rakubrew/versions/moar-2021.10/install/share/perl6/core"), CompUnit::Repository::AbsolutePath.new(next-repo => CompUnit::Repository::NQP.new(next-repo => CompUnit::Repository::Perl5.new(next-repo => CompUnit::Repository))), CompUnit::Repository::NQP.new(next-repo => CompUnit::Repository::Perl5.new(next-repo => CompUnit::Repository)), CompUnit::Repository::Perl5.new(next-repo => CompUnit::Repository))
We’re talking about the repository chain, where Raku (through Rakudo) keeps the information or where to find the, effectively, CompUnit repositories or the libraries, precompiled (those are the CompUnit::Repository::Installation) or not (CompUnit::Repository::AbsolutePath). But let’s look at the first one, which is where it will start looking. It’s effectively our home directory, or more precisely, a subdirectory where things are installed in the normal course of things. Where does Rakudo picks that from? Let’s change the HOME environment variable and we’ll see, or rather not, because depending on the installation, it will simply hang. With the RAKULIB defined as above, however, say $*REPO.repo-chain will print
(inst#/home/raku/.raku inst#/tmp/.raku inst#/usr/share/perl6/site inst#/usr/share/perl6/vendor inst#/usr/share/perl6/core ap# nqp# perl5#)
Our CompUnit::Repository::Installation become here inst#/home/raku/.raku, but, what’s more important, the HOME environment variable gets tacked a .raku in the end and an inst# in front, implying that’s the place where Rakudo expects to find it.
This brings us back again to GitHub actions, which change that variable for no good reason, leaving our Rakudo installation effectively unusable. But no fear, a simple environment variable baked in the alpine-raku container (and its GHCR variants) will keep the actual Rakudo installation in check for GitHub actions to come.
And we can write our own GitHub actions using this image. Directly run all our stuff inside a container that has Raku. For instance, this way:
This is decevingly simply, doing exactly what you would do in your console. Install, and then test, right? That’s it. Underneath, however the fact that the container is using the right UID and Raku knows where to find its own installation despite all the moving and shaking that’s going on is what makes it run.
You can even do a bit more. Use Raku as a shell for running anything. Add this step:
- name: Use Raku to run
shell: raku {0}
run: say $*REPO.repo-chain
And with the shell magic, it will actually run that directly on the Raku interpreter. You can do anything else you want: install stuff, run Cro if you want. All within the GitHub action! For instance, do you want to chart how many files were changed in the latest commits using Raku? Here you go:
Couple of easy steps: install whatever you need, and then use Text::Chart to chart those files. This needs a bit of explaining, or maybe directly checking the source to have the complete picture: it’s using an environment variable called COMMITS, which is one more than the commits we want to chart, has been used to check out all those commits, and then, of course, we need to pop the last one since it’s a squashed commit that contains all older changes in the repo, uglifying our chart (which we don’t want). Essentially, however, is a pipe that takes the text content of the log that includes the number of file changed, extracts that number via a regex match, and feeds it into the vertical function to create the text chart. Which will show something like this (click on the > sign to show the chart):
With thousands of files at your disposal, the sky’s the limit. Do you want to install fez and upload automatically when tagging? Why not? Just do it. Upload your secret, and Bob’s your uncle. Do you want to do some complicated analytics on the source using Raku or generate thumbnails? Go ahead!
After this, Santa was incredibly happy, since all his Raku stuff was properly checked, and even containerized if needed! So he sit down to enjoy his corncob pipe, which Meta-Santa brought for him last Christmas.
And, with that, Merry Christmas to all and everyone of you!
by Tony O’Dell
Hello, world! This article is a lot about fez and how you can get started writing your first module and making it available to other users. Presuming you have rakudo and zef installed, install fez!
$ zef install fez
===> Searching for: fez
===> Updating fez mirror: https://360.zef.pm/
===> Updated fez mirror: https://360.zef.pm/
===> Testing: fez:ver<32>:auth<zef:tony-o>:api<0>
[fez] Fez - Raku / Perl6 package utility
[fez] USAGE
[fez] fez command [args]
[fez] COMMANDS
[fez] register registers you up for a new account
[fez] login logs you in and saves your key info
[fez] upload creates a distribution tarball and uploads
[fez] meta update your public meta info (website, email, name)
[fez] reset-password initiates a password reset using the email
[fez] that you registered with
[fez] list lists the dists for the currently logged in user
[fez] remove removes a dist from the ecosystem (requires fully
[fez] qualified dist name, copy from `list` if in doubt)
[fez] org org actions, use `fez org help` for more info
[fez] ENV OPTIONS
[fez] FEZ_CONFIG if you need to modify your config, set this env var
[fez] CONFIGURATION (using: /home/tonyo/.fez-config.json)
[fez] Copy this to a cool location and write your own requestors/bundlers or
[fez] ignore it and use the default curl/wget/git tools for great success.
===> Testing [OK] for fez:ver<32>:auth<zef:tony-o>:api<0>
===> Installing: fez:ver<32>:auth<zef:tony-o>:api<0>
1 bin/ script [fez] installed to:
/home/tonyo/.local/share/perl6/site/bin
Make sure that the last line is in your $PATH so the next set of commands all run smoothly. Now we can start writing the actual module, let’s write ROT13 since it’s a fairly easy problem to solve and this article really is less about module content than how to get working with fez.
Our module directory structure:
.
├── lib
│ └── ROT13.rakumod
├── t
│ ├── 00-use.rakutest
│ └── 01-tests.rakutest
└── META6.json
lib is the main content of your module, it’s where all of your module’s utilities, helpers, and organization happens. Each file corresponds to one or more modules or classes, more on in the META6.json paragraph below.
META6.json is how zef knows what the module is, it’s how fez knows what it’s uploading, and it’s how rakudo knows how to load what and from where. Let’s take a look at the structure of META6.json:
t contains all of your module’s tests. If you have “author only” tests then you’d also have a directory xt and that directory works roughly the same. For your users’ sanity WRITE TESTS!
{
"name": "ROT13",
"auth": "zef:tony-o",
"version": "0.0.1",
"api": 0,
"provides": {
"ROT13": "lib/ROT13.rakumod"
},
"depends": [],
"build-depends": [],
"test-depends": [],
"tags": [ "ROT13", "crypto" ],
"description": "ROT13 everything!"
}A quick discussion about dists. A dist is the fully qualified name of your module and it contains the name, auth, and version. It’s how zef can differentiate your ROT13 module from mine. It works in conjunction with use, such as use ROT13:auth<zef:tony-o>, and in zef: zef install ROT13:auth<tony-o>:ver<0.0.1>. The dist string is always qualified with both the :auth and the :ver internally to raku and the ecosystem, but the end user isn’t required to type the fully qualified dist if they’re less picky about what version/author of the module they’d like. In use statements you can combine auth and ver to get the author or version you’re expecting or you can omit one or both.
It’s better practice to fully qualify your use statements; as more modules hit the ecosystem with the same name, this practice will help keep your modules running smoothly.
name: this is the name of the module and becomes part of your dist, it’s what is referenced when your consumers type zef install ROT13.auth: this is how the ecosystem knows who the author is. On fez this is strict, no other rakudo ecosystem guarantees this matches the uploader’s username.version: version must be unique to the auth and name. For instance, you can’t upload two dists with the value of ROT13:auth<zef:tony-o>:ver<0.0.1>.provides: in provides is the key/value pairs of module and class names to which file they belong to. If you have two modules in one file then you should have the same file listed twice with the key for each being each class/module name. All .rakumod files in lib should be in the META6.json file. The keys here are how rakudo knows which file to look for your class/module in.depends: a list of your runtime depenciesLet’s whip up a quick ROT13 module, in lib/ROT13.rakumod dump the following
unit module ROT13;
sub rot13(Str() $text) is export {
$text.trans('a..zA..Z'=>'n..za..mN..ZA..Z')
}
Great, you can test it now (from the root of your module directory) with raku -I. -e 'use ROT13; say rot13("hello, WoRlD!");. You should get output of uryyb, JbEyQ!.
Now fill in your test files and run the tests with zef test .
If you’re not registered with fez, now’s the time!
$ fez register
>>= Email: [email protected]
>>= Username: tony-o
>>= Password:
>>= Registration successful, requesting auth key
>>= Username: tony-o
>>= Password:
>>= Login successful, you can now upload dists
$ fez checkbuild
>>= Inspecting ./META6.json
>>= meta<provides> looks OK
>>= meta<resources> looks OK
>>= ROT13:ver<0.0.1>:auth<zef:tony-o> looks OK
Oh snap, we’re lookin’ good!
$ fez upload
>>= Hey! You did it! Your dist will be indexed shortly.
Only thing to note here is that if there’s a problem indexing your module then you’ll receive an email with the gripe.
You can read more about fez here:
Perhaps you’d prefer listening:
That’s it! If there’s other things you’d like to know about fez, zef, or ecosystems then send tony-o some chat in IRC or an email!
A Christmas ditty sung to the tune of Santa Claus is Coming to Town:
| He’s making a list, | |
| He’s checking it closely, | |
| He’s gonna find out who’s Rakuing mostly, | |
| Santa Claus is Rakuing along. |
Part 1 of this article reported on the new journaling process for Santa’s employees and how they keep their individual work logs. Part 2 was a side step to show how to better manage Santa’s code by using the new Zef module repository. Part 3 was another side step because Santa was running out of time.
This article, written by yours truly, junior Elf Rodney, will attempt to showcase the use of Raku’s built-in date, time, and sorting functions along with the ease of class construction to handle the new journals in the aggregate to automate many of the reports that used to take goodly amounts of time. They can now be prepared quickly and thus free resources to research more deeply-observed problem areas.
The journals are frequently read and manipulated by system-wide programs (most found in the Raku module SantaClaus::Utils) run by root. Individual journals are periodically shortened by extracting older entries which are then concatenated onto hidden .journal.YYYY.MM files (owned by root but readable by all) in the user’s home directory.
The data in the journal files are converted to class instances which are deployed for the moment in two global Raku hashes keyed by task-id and user-id, respectively. (When the new persistent types in Raku are ready, they will be a natural fit to form a large on-disk database).
Before designing classes to use with the journals let’s take a quick look at how we want the data to be accessed and used.
First, the raw data give us, for each user and his assigned task (which may be a sub-task):
Second, the raw data give us, for each task and sub-task
It seems that the data we have so far collected don’t yield the task/sub-task relations, but that is solved with a task-id system designed with that in mind. As a start, the task-id is a two-field number with the first field being the task number and the second field being the sub-task number. Supervisors will normally use the task number and their subordinates the sub-task number.
For example, a task number might be 103458 with sub-tasks of 200 and 202. The three numbers entered by the different employees working them would enter:
The final system could be as detailed as desired, but the two-level task-id is sufficient for now.
[Sorry, this article will be finished later–I am needed for last minute jobs in the factory!]
Santa now has a personnel and task reporting system that automatically produces continuously updated reports on current tasks and the personnel resources used for them. Raku’s built-in date, time, and sorting functions help ease the work of the IT department in their job of programming and maintaining the system.
Don’t forget the “reason for the season:” 
As I always end these jottings, in the words of Charles Dickens’ Tiny Tim, “may God bless Us , Every one! [1]”
In Part I of this post, I started on my macOS tool chain rebuild, I want this to work for all my languages (Python, Perl, JavaScript, Raku and so on including specific compiler build versions and combinations such as Inline::Perl5) … coming off the disappointing Apple induced cliff edge for M1 machines upgrading from Big Sur to Monterey OS versions.
In that thrilling episode, I turned the corner and was starting to make progress – seeing how I can get the speed benefits of M1 and apply them to my raku projects:
BUT – as mentioned and commented, this way I have to install all my module dependencies on every test run … surely there must be a better, faster way…
Now before I dig in, I must say that I was feeling a bit lonely out here with my opinion that macOS Monterey is a sign that Apple is defocusing on the wider developer community …. then I found this comment that cheered my up a bit:
smoldesu 5 months ago | root | parent | prev | next [–]
It was impressive back when OSX debuted, but now it feels like Apple is lacking in both regards. Their shell utilities are all horribly outdated/mismatched, and their UI design took a nosedive with Big Sur. Modern MacOS reminds me of what Ubuntu felt like 10 years ago: it’s a confused and scared operating system that doesn’t quite know where to go from here. I mean, look at what Monterrey introduced; basically better Facetime and some new wallpapers. I’m getting the feeling that Apple has painted themselves into a corner here. They spent the past 2 decades focusing on vertical growth, only to discover that the next 50 years are going to be ruled by better interop and protocols.
https://news.ycombinator.com/item?id=27537040
And, to be fair, I think that good support for VMs is probably a fork in the road for all desktop OS providers … then (spoiler alert) a decent VM side-load means that the general (ie. non Apple) developer can have complete control and Apple can build their wall higher and need to pander to fewer requirements (goodbye perl 5 on Mac?)
First, I naively decided to add a couple of lines to the raku-test Dockerfile cloned from JJ’s repo:
RUN zef install XML::Writer SVG SVG::Plotand then rebuild from the Dockerfile pwd with:
docker build -t new-test .But, this did not work via the zsh CLI provided by Docker Desktop on macOS:
--platform=ARM64)Then, I took out my changes and just tried to clone and run an unchanged alpine-raku:
Hmmm, I dug a bit deeper into the CI commands in JJ’s github repo and realised that he was building these variants of alpine-raku on ubuntu …. so while alpine-raku is a fantastic lightweight container image for test, right now alpine-raku can’t be build on Docker Desktop on macOS Monterey M1. (Mumble mumble bloody bleeding edge.)
SO – I needed to get a clean ubuntu build environment – with a workflow that goes:
Right now I am on the free 5 day trial of the subscription based “Run Docker on Ubuntu 20.04 with latest Ubuntu Image” by Code Creator (just search for this AMI in the AWS marketplace)
Now all I have to do is prebuild the variants of zef installs that I need, tag these new Docker images and push to the Docker image repository.
Here’s an example Dockerfile (see each module eg. Physics::Unit ):
FROM jjmerelo/alpine-raku:latest
LABEL version="6.0.2" maintainer="JJ Merelo <[email protected]>"
ARG DIR="/test"
USER root
# Add raku-physics-unit dependencies
RUN zef install SVG::Plot
# Set up dirs
RUN mkdir $DIR && chown raku $DIR
COPY --chown=raku test.sh /home/raku
VOLUME $DIR
WORKDIR $DIR
# Change to non-privileged user
USER raku
# Will run this
ENTRYPOINT ["/home/raku/test.sh"]Here’s my CLI incantation to do the build:
git clone https://github.com/p6steve/raku-Physics-Unit.git
sudo docker build -t arp-unit-deps .
sudo docker run -t -v /home/ubuntu/raku-Physics-Unit:/test arp-unit-deps --verbose
sudo docker tag arp-unit-deps p6steve/alpine-raku-physics:arp-unit-deps
sudo docker push p6steve/alpine-raku-physics:arp-unit-depsAnd to run it on the macOS Docker Desktop client:
docker pull p6steve/alpine-raku-physics:arp-unit-deps
docker run -t -v //path-to/raku-Physics-Measure:/test p6steve/alpine-raku-physics:arp-unit-deps --verboseHere’s the list of images that I built – in each case, the image is built with the module dependencies since it is used for faster testing of the module…
These are all now built and pushed to https://hub.docker.com/r/p6steve/alpine-raku-physics – enjoy!
So – this has proven the point and determined a general need for a clean ubuntu build environment for adjustments to my images, I felt that I was limited by going to AWS and a paid build for this step and still felt unnecessarily hemmed in and slowed down in by the macOS Docker Desktop confection.
A bit more research turned up vftools … and this excellent recipe from Jan Mikeš… take a look to see how Jan describes 40x load speed improvements (from 160s to 3.7s).
Here’s what it has provided for me (ymmv):
~/ubuntu-vmubuntussh [email protected] without passwordssh ubuntu without passwordstart-ubuntu command on mac~/ubuntu-vm/disk-backup.img.tar.gzSo now I can start my ubuntu-vm with docker in the morning and reach for something like:
docker pull --platform linux/arm64 rakudo-star
docker run -it --entrypoint sh -l -c rakudo-star
rakuLine 2 of this takes about 1s to get a raku REPL prompt. Yessssss!
So now I am off to scratch my head with:
Please do leave any feedback and comments (click here & scroll down) …
~p6steve
There I was just being tidy and getting the latest macOS release (Monterey) for my pricey new M1 laptop – expecting the usual seamless upgrade. Then bang! <<Homebrew is not supported on ARM processors>>. This is NOT (just) a raku thing. In my case, the proximate cause seems to be lack of support for M1 for ruby <<No Homebrew ruby 2.6.8 available for arm64 processors!>>. My swag is that Monterey is stepping down rosetta mollycoddling vs BigSur and wo-betide anyone who wants to run non native via a terminal. Well what did you expect with an architecture change.
CAVEAT
This may well be the way I have set up my setup on my machine – you made never experience any of this. I would also mention that I am not a core dev and I am a bit out on a limb building from source – I can do it fine if it works… and all of what follows has a good measure of uninstall/reinstall which can leave a bit of a mess.
This made me mad. So I factory reset my machine, installed Xcode (this is safer than xcode-select --install), rosetta2 (leaving the Terminal.app “open with rosetta 2” option unchecked, since I want to go faster) and Docker Desktop and tried several things on a clean macOS machine. My failings included:
I would love it if any of these methods would work for me … and encourage you to comment if you have fared better / know the incantation that I missed!
And this is not to cast aspersions on the excellent raku toolchain – I am sure if I wait a couple of months and get off the bleeding edge, then all will be fixed. In the meantime, I realised that macOS is a bit of a sideshow for raku which is developed on ubuntu.
Since I have been wondering about the best way to test against multiple raku versions and to maintain a tight system configuration around raku to support Python::Inline and Perl5::Inline, I choose to see the silver lining in my cloud and go for Docker. So I installed the newly GA Docker Desktop for Apple Silicon.
This method is very forgiving – Docker Desktop will warn, then run AMD64 (on rosetta) using platform detection if need be, regardless of the Terminal.app settings.
Now to work out the workload … well I have been working on the various Physics::Measure modules using jnthn’s excellent CommaCP for a while, so I was very keen to get testing working for these.
I also have been using JJ’s excellent alpine-raku and raku-alpine-test on the voracious TravisCI service so it seemed natural to go with that. Here are the steps that worked for me:
Starting by running a module test from the Docker command line (raku-test is a specialised derivative, with a Dockerfile that starts FROM alpine-raku:latest):
docker run -t -v /path/to/module-dir:/test jjmerelo/raku-test
This will pull the image from the Docker repo and run it… you just specify the local path to your module.
Nice! I am alive once more with local test harness and the ability to apply multiple raku versions – BUT this is on AMD64 (aka Intel) … so all that money spent on M1 is wasted!
Stepping this up, I can use alpine-raku to run scripts directly (following the documentation):
docker run –rm -t jjmerelo/alpine-raku -e “mkdir ‘raku-app’; say ‘raku-app’.IO.absolute;”
docker run -t -v pwd:/home/raku/raku-app jjmerelo/alpine-raku /home/raku/raku-app/pell.p 6
Since I already had made a Dockerfile to run this module in interactive mode in a Jupiter notebook, I gave it a try …
git clone https://github.com/p6steve/raku-Math-Polygons
cd raku-Math-Polygons
docker build -t rmp .
This takes a while (20mins+) as it does on Jupyter binder since it is building a full raku ubuntu image from scratch – when running you can access via the browser button on Docker Desktop.
To wrap this story up, how nice it would be to get Docker and Comma working together … lo and behold, CommaIDE has a plugin just for this…
So, now all is well – Docker Desktop is up and running and I can ring all the changes from the command line:
docker run -it jjmerelo/alpine-raku
docker run -it jjmerelo/alpine-raku:2021.04
docker run -t jjmerelo/alpine-raku -e “say ‘hello þor'”
docker run -it –entrypoint sh -l -c jjmerelo/alpine-raku [container persists eg. zef modules]
docker run -v pwd:/app -it jjmerelo/alpine-raku /app/Tree.p6
BUT – this way I have to install all my module dependencies on every test run … surely there must be a better, faster way… (just wait for the next gripping instalment)
Please do leave any feedback and comments …
~p6steve
First of all, I’d like to apologize for all the errors in this post. I just haven’t got time to properly proof-read it.
A while ago I was trying to fix a problem in Rakudo which, under certain
conditions, causes some external symbols to become invisible for importing code,
even if explicit use statement is used. And, indeed, it is really confusing
when:
use L1::L2::L3::Class;
L1::L2::L3::Class.new;
fails with “Class symbol doesn’t exists in L1::L2::L3” error! It’s ok if use
throws when there is no corresponding module. But .new??
This section is needed to understand the rest of the post. A package in Raku is a typeobject which has a symbol table attached. The table is called stash (stands for “symbol table hash”) and is represented by an instance of Stash class, which is, basically, is a hash with minor tweaks. Normally each package instance has its own stash. For example, it is possible to manually create two different packages with the same name:
my $p1a := Metamodel::PackageHOW.new_type(:name<P1>);
my $p1b := Metamodel::PackageHOW.new_type(:name<P1>);
say $p1a.WHICH, " ", $p1a.WHO.WHICH; # P1|U140722834897656 Stash|140723638807008
say $p1b.WHICH, " ", $p1b.WHO.WHICH; # P1|U140722834897800 Stash|140723638818544
Note that they have different stashes as well.
A package is barely used in Raku as is. Usually we deal with packagy things like modules and classes.
Back then I managed to trace the problem down to deserialization process within
MoarVM backend. At that point I realized that somehow it pulls in packagy
objects which are supposed to be the same thing, but they happen to be
different and have different stashes. Because MoarVM doesn’t (and must not)
have any idea about the structure of high-level Raku objects, there is no way it
could properly handle this situation. Instead it considers one of the
conflicting stashes as “the winner” and drops the other one. Apparently, symbols
unique to the “loser” are lost then.
It took me time to find out what exactly happens. But not until a couple of days ago I realized what is the root cause and how to get around the bug.
What happens when we do something like:
module Foo {
module Bar {
}
}
How do we access Bar, speaking of the technical side of things? Foo::Bar
syntax basically maps into Foo.WHO<Bar>. In other words, Bar gets installed
as a symbol into Foo stash. We can also rewrite it with special syntax sugar:
Foo::<Bar> because Foo:: is a representation for Foo stash.
So far, so good; but where do we find Foo itself? In Raku there is a special
symbol called GLOBAL which is the root namespace (or a package if you wish)
of any code. GLOBAL::, or GLOBAL.WHO is where one finds all the top-level
symbols.
Say, we have a few packages like L11::L21, L11::L22, L12::L21, L12::L22.
Then the namespace structure would be represented by this tree:
GLOBAL
`- L11
`- L21
`- L22
`- L12
`- L21
`- L22
Normally there is one per-process GLOBAL symbol and it belongs to the compunit
which used to start the program. Normally it’s a .raku file, or a string
supplied on command line with -e option, etc. But each
compunit
also gets its own GLOBALish package which acts as compunit’s GLOBAL until it
is fully incorporated into the main code. Say, we declare a module in file
Foo.rakumod:
unit module Foo;
sub print-GLOBAL($when) is export {
say "$when: ", GLOBAL.WHICH, " ", GLOBALish.WHICH;
}
print-GLOBAL 'LOAD';
And use it in a script:
use Foo;
print-GLOBAL 'RUN ';
Then we can get an ouput like this:
LOAD: GLOBAL|U140694020262024 GLOBAL|U140694020262024
RUN : GLOBAL|U140694284972696 GLOBAL|U140694020262024
Notice that GLOBALish symbol remains the same object, whereas GLOBAL gets
different. If we add a line to the script which also prints GLOBAL.WHICH then
we’re going to get something like:
MAIN: GLOBAL|U140694284972696
Let’s get done with this part of the story for a while a move onto another subject.
This is going to be a shorter story. It is not a secret that however powerful Raku’s grammars are, they need some core developer’s attention to make them really fast. In the meanwhile, compilation speed is somewhat suboptimal. It means that if a project consist of many compunits (think of modules, for example), it would make sense to try to compile them in parallel if possible. Unfortunately, the compiler is not thread-safe either. To resolve this complication Rakudo implementation parallelizes compilation by spawning individual processes per each compunit.
For example, let’s refer back to the module tree example above and imagine that
all modules are used by a script. In this case there is a chance that we would
end up with six rakudo processes, each compiling its own L* module.
Apparently, things get slightly more complicated if there are cross-module
uses, like L11::L21 could refer to L21, which, in turn, refers to
L11::L22, or whatever. In this case we need to use topological sort to
determine in what order the modules are to be compiled; but that’s not the
point.
The point is that since each process does independent compilation, each compunit
needs independent GLOBAL to manage its symbols. For the time being, what we
later know as GLOBALish serves this duty for the compiler.
Later, when all pre-compiled modules are getting incorporated into the code
which uses them, symbols installed into each individual GLOBAL are getting
merged together to form the final namespace, available for our program. There
are even methods in the source, using merge_global in their names.
(Note the clickable section header; I love the guy!)
Now, you can feel the catch. Somebody might have even guessed what it is. It
crossed my mind after I was trying to implement legal symbol auto-registration
which doesn’t involve using QAST to install a phaser. At some point I got an
idea of using GLOBAL to hold a register object which would keep track of
specially flagged roles. Apparently it failed due to the parallelized
compilation mentioned above. It doesn’t matter, why; but at that point I started
building a mental model of what happens when merge is taking place. And one
detail drew my special attention: what happens if a package in a long name is
not explicitly declared?
Say, there is a class named Foo::Bar::Baz one creates as:
unit class Foo::Bar;
class Baz { }
In this case the compiler creates a stub package for Foo. The stub is used to
install class Bar. Then it all gets serialized into bytecode.
At the same time there is another module with another class:
unit class Foo::Bar::Fubar;
It is not aware of Foo::Bar::Baz, and the compiler has to create two stubs:
Foo and Foo::Bar. And not only two versions of Foo are different and have
different stashes; but so are the two versions of Bar where one is a real
class, the other is a stub package.
Most of the time the compiler does damn good job of merging symbols in such
cases. It took me stripping down a real-life code to golf it down to some
minimal set of modules which reproduces the situation where a require call
comes back with a Failure and a symbol becomes missing. The remaining part of
this post will be dedicated to this
example.
In particular, this whole text is dedicated to one
line.
Before we proceed further, I’d like to state that I might be speculating about some aspects of the problem cause because some details are gone from my memory and I don’t have time to re-investigate them. Still, so far my theory is backed by working workaround presented at the end.
To make it a bit easier to analyze the case, let’s start with namespace tree:
GLOBAL
`- L1
`- App
`- L2
`- Collection
`- Driver
`- FS
Rough purpose is for application to deal with some kind of collection which
stores its items with help of a driver which is loaded dynamically, depending,
say, on a user configuration. We have the only driver implemented: File System
(FS).
If you checkout the repository and try raku -Ilib symbol-merge.raku in the
examples/2021-10-05-merge-symbols directory, you will see some output ending
up with a line like Failure|140208738884744 (certainly true for up until
Rakudo v2021.09 and likely to be so for at least a couple of versions later).
The key conflict in this example are modules Collection and Driver. The full
name of Collection is L1::L2::Collection. L1 and L2 are both stubs.
Driver is L1::L2::Collection::Driver and because it
imports
L1::L2, L2 is a class; but L1 remains to be a stub. By commenting out the
import we’d get the bug resolved and the script would end up with something
like:
L1::L2::Collection::FS|U140455893341088
This means that the driver module was successfully loaded and the driver class symbol is available.
Ok, uncomment the import and start the script again. And then once again to get rid of the output produced by compilation-time processes. We should see something like this:
[7329] L1 in L1::L2 : L1|U140360937889112
[7329] L1 in Driver : L1|U140361742786216
[7329] L1 in Collection : L1|U140361742786480
[7329] L1 in App : L1|U140361742786720
[7329] L1 in MAIN : L1|U140361742786720
[7329] L1 in FS : L1|U140361742788136
Failure|140360664014848
We already know that L1 is a stub. Dumping object IDs also reveals that each
compunit has its own copy of L1, except for App and the script (marked as
MAIN). This is pretty much expected because each L1 symbol is installed at
compile-time into per-compunit GLOBALish. This is where each module finds it.
App is different because it is directly imported by the script and was
compiled by the same compiler process, and shared its GLOBAL with the script.
Now comes the black magic. Open lib/L1/L2/Collection/FS.rakumod and uncomment the last line in the file. Then give it a try. The output would seem impossible at first; hell with it, even at second glance it is still impossible:
[17579] Runtime Collection syms : (Driver)
Remember, this line belongs to L1::L2::Collection::FS! How come we don’t see
FS in Collection stash?? No wonder that when the package cannot see itself
others cannot see it too!
Here comes a bit of my speculation based on what I vaguely remember from the times ~2 years ago when I was trying to resolve this bug for the first time.
When Driver imports L1::L2, Collection gets installed into L2 stash, and
Driver is recorded in Collection stash. Then it all gets serialized with
Driver compunit.
Now, when FS imports Driver to consume the role, it gets the stash of L2
serialized at the previous stage. But its own L2 is a stub under L1 stub.
So, it gets replaced with the serialized “real thing” which doesn’t have FS
under Collection! Bingo and oops…
Walk through all the example files and uncomment use L1 statement. That’s it.
All compunits will now have a common anchor to which their namespaces will be
attached.
The common rule would state that if a problem of the kind occurs then make sure
there’re no stub packages in the chain from GLOBAL down to the “missing”
symbol. In particular, commenting out use L1::L2 in Driver will get our
error back because it would create a “hole” between L1 and Collection and
get us back into the situation where conflicting Collection namespaces are
created because they’re bound to different L2 packages.
It doesn’t really matter how exactly the stubs are avoided. For example, we can
easily move use L1::L2 into Collection and make sure that use L1 is still
part of L2. So, for simplicity a child package may import its parent; and
parent may then import its parent; and so on.
Sure, this adds to the boilerplate. But I hope the situation is temporary and there will be a fix.
The one I was playing with required a compunit to serialize its own GLOBALish
stash at the end of the compilation in a location where it would not be at risk
of overwriting. Basically, it means cloning and storing it locally on the
compunit (the package stash is part of the low-level VM structures). Then
compunit mainline code would invoke a method on the Stash class which would
forcibly merge the recorded symbols back right after deserialization of
compunit’s bytecode. It was seemingly working, but looked more of a kind of a
hack, than a real fix. This and a few smaller issues (like a segfault which I
failed to track down) caused it to be frozen.
As I was thinking of it lately, more proper fix must be based upon a common
GLOBAL shared by all compunits of a process. In this case there will be no
worry about multiple stub generated for the same package because each stub will
be shared by all compunits until, perhaps, the real package is found in one of
them.
Unfortunately, the complexity of implementing the ‘single GLOBAL’ approach is
such that I’m unsure if anybody with appropriate skill could fit it into their
schedule.
Around 18 months ago, I set about working on the largest set of architectural changes that Raku runtime MoarVM has seen since its inception. The work was most directly triggered by the realization that we had no good way to fix a certain semantic bug in dispatch without either causing huge performance impacts across the board or increasingly complexity even further in optimizations that were already riding their luck. However, the need for something like this had been apparent for a while: a persistent struggle to optimize certain Raku language features, the pain of a bunch of performance mechanisms that were all solving the same kind of problem but each for a specific situation, and a sense that, with everything learned since I founded MoarVM, it was possible to do better.
The result is the development of a new generalized dispatch mechanism. An overview can be found in my Raku Conference talk about it (slides, video); in short, it gives us a far more uniform architecture for all kinds of dispatch, allowing us to deliver better performance on a range of language features that have thus far been glacial, as well as opening up opportunities for new optimizations.
Today, this work has been merged, along with the matching changes in NQP (the Raku subset we use for bootstrapping and to implement the compiler) and Rakudo (the full Raku compiler and standard library implementation). This means that it will ship in the October 2021 releases.
In this post, I’ll give an overview of what you can expect to observe right away, and what you might expect in the future as we continue to build upon the possibilities that the new dispatch architecture has to offer.
The biggest improvements involve language features that we’d really not had the architecture to do better on before. They involved dispatch – that is, getting a call linked to a destination efficiently – but the runtime didn’t provide us with a way to “explain” to it that it was looking at a dispatch, let alone with the information needed to have a shot at optimizing it.
The following graph captures a number of these cases, and shows the level of improvement, ranging from a factor of 3.3 to 13.3 times faster.
Let’s take a quick look at each of these. The first, new-buf, asks how quickly we can allocate Bufs.
for ^10_000_000 {
Buf.new
}
Why is this a dispatch benchmark? Because Buf is not a class, but rather a role. When we try to make an instance of a role, it is “punned” into a class. Up until now, it works as follows:
new methodfind_method method would, if needed, create a pun of the role and cache it-> $role-discarded, |args { $pun."$name"(|args) }This had a number of undesirable consequences:
With the new dispatch mechanism, we have a means to cache constants at a given program location and to replace arguments. So the first time we encounter the call, we:
new method on the class punned from the roleFor the next thousands of calls, we interpret this dispatch program. It’s still some cost, but the method we’re calling is already resolved, and the argument list rewriting is fairly cheap. Meanwhile, after we get into some hundreds of iterations, on a background thread, the optimizer gets to work. The argument re-ordering cost goes away completely at this point, and new is so small it gets inlined – at which point the buffer allocation is determined dead and so goes away too. Some remaining missed opportunities mean we still are left with a loop that’s not quite empty: it busies itself making sure it’s really OK to do nothing, rather than just doing nothing.
Next up, multiple dispatch with where clauses.
multi fac($n where $n <= 1) { 1 }
multi fac($n) { $n * fac($n - 1) }
for ^1_000_000 {
fac(5)
}
These were really slow before, since:
where clause involvedwhere clauses twice in the event the candidate was chosen: once to see if we should choose that multi candidate, and once again when we entered itWith the new mechanism, we:
where clause, in a mode whereby if the signature fails to bind, it triggers a dispatch resumption. (If it does bind, it runs to completion)Once again, after the setup phase, we interpret the dispatch programs. In fact, that’s as far as we get with running this faster for now, because the specializer doesn’t yet know how to translate and further optimize this kind of dispatch program. (That’s how I know it currently stands no chance of turning this whole thing into another empty loop!) So there’s more to be had here also; in the meantime, I’m afraid you’ll just have to settle for a factor of ten speedup.
Here’s the next one:
proto with-proto(Int $n) { 2 * {*} }
multi with-proto(Int $n) { $n + 1 }
sub invoking-nontrivial-proto() {
for ^10_000_000 {
with-proto(20)
}
}
Again, on top form, we’d turn this into an empty loop too, but we don’t quite get there yet. This case wasn’t so terrible before: we did get to use the multiple dispatch cache, however to do that we also ended up having to allocate an argument capture. The need for this also blocked any chance of inlining the proto into the caller. Now that is possible. Since we cannot yet translate dispatch programs that resume an in-progress dispatch, we don’t yet get to further inline the called multi candidate into the proto. However, we now have a design that will let us implement that.
This whole notion of a dispatch resumption – where we start doing a dispatch, and later need to access arguments or other pre-calculated data in order to do a next step of it – has turned out to be a great unification. The initial idea for it came from considering things like callsame:
class Parent {
method m() { 1 }
}
class Child is Parent {
method m() { 1 + callsame }
}
for ^10_000_000 {
Child.m;
}
Once I started looking at this, and then considering that a complex proto also wants to continue with a dispatch at the {*}, and in the case a where clauses fails in a multi it also wants to continue with a dispatch, I realized this was going to be useful for quite a lot of things. It will be a bit of a headache to teach the optimizer and JIT to do nice things with resumes – but a great relief that doing that once will benefit multiple language features!
Anyway, back to the benchmark. This is another “if we were smart, it’d be an empty loop” one. Previously, callsame was very costly, because each time we invoked it, it would have to calculate what kind of dispatch we were resuming and the set of methods to call. We also had to be able to locate the arguments. Dynamic variables were involved, which cost a bit to look up too, and – despite being an implementation details – these also leaked out in introspection, which wasn’t ideal. The new dispatch mechanism makes this all rather more efficient: we can cache the calculated set of methods (or wrappers and multi candidates, depending on the context) and then walk through it, and there’s no dynamic variables involved (and thus no leakage of them). This sees the biggest speedup of the lot – and since we cannot yet inline away the callsame, it’s (for now) measuring the speedup one might expect on using this language feature. In the future, it’s destined to optimize away to an empty loop.
A module that makes use of callsame on a relatively hot path is OO::Monitors,, so I figured it would be interesting to see if there is a speedup there also.
use OO::Monitors;
monitor TestMonitor {
method m() { 1 }
}
my $mon = TestMonitor.new;
for ^1_000_000 {
$mon.m();
}
A monitor is a class that acquires a lock around each method call. The module provides a custom meta-class that adds a lock attribute to the class and then wraps each method such that it acquires the lock. There are certainly costly things in there besides the involvement of callsame, but the improvement to callsame is already enough to see a 3.3x speedup in this benchmark. Since OO::Monitors is used in quite a few applications and modules (for example, Cro uses it), this is welcome (and yes, a larger improvement will be possible here too).
I’ve seen some less impressive, but still welcome, improvements across a good number of other microbenchmarks. Even a basic multi dispatch on the + op:
my $i = 0;
for ^10_000_000 {
$i = $i + $_;
}
Comes out with a factor of 1.6x speedup, thanks primarily to us producing far tighter code with fewer guards. Previously, we ended up with duplicate guards in this seemingly straightforward case. The infix:<+> multi candidate would be specialized for the case of its first argument being an Int in a Scalar container and its second argument being an immutable Int. Since a Scalar is mutable, the specialization would need to read it and then guard the value read before proceeding, otherwise it may change, and we’d risk memory safety. When we wanted to inline this candidate, we’d also want to do a check that the candidate really applies, and so also would deference the Scalar and guard its content to do that. We can and do eliminate duplicate guards – but these guards are on two distinct reads of the value, so that wouldn’t help.
Since in the new dispatch mechanism we can rewrite arguments, we can now quite easily do caller-side removal of Scalar containers around values. So easily, in fact, that the change to do it took me just a couple of hours. This gives a lot of benefits. Since dispatch programs automatically eliminate duplicate reads and guards, the read and guard by the multi-dispatcher and the read in order to pass the decontainerized value are coalesced. This means less repeated work prior to specialization and JIT compilation, and also only a single read and guard in the specialized code after it. With the value to be passed already guarded, we can trivially select a candidate taking two bare Int values, which means there’s no further reads and guards needed in the callee either.
A less obvious benefit, but one that will become important with planned future work, is that this means Scalar containers escape to callees far less often. This creates further opportunities for escape analysis. While the MoarVM escape analyzer and scalar replacer is currently quite limited, I hope to return to working on it in the near future, and expect it will be able to give us even more value now than it would have been able to before.
The benchmarks shown earlier are mostly of the “how close are we to realizing that we’ve got an empty loop” nature, which is interesting for assessing how well the optimizer can “see through” dispatches. Here are a few further results on more “traditional” microbenchmarks:
The complex number benchmark is as follows:
my $total-re = 0e0;
for ^2_000_000 {
my $x = 5 + 2i;
my $y = 10 + 3i;
my $z = $x * $x + $y;
$total-re = $total-re + $z.re
}
say $total-re;
That is, just a bunch of operators (multi dispatch) and method calls, where we really do use the result. For now, we’re tied with Python and a little behind Ruby on this benchmark (and a surprising 48 times faster than the same thing done with Perl’s Math::Complex), but this is also a case that stands to see a huge benefit from escape analysis and scalar replacement in the future.
The hash read benchmark is:
my %h = a => 10, b => 12;
my $total = 0;
for ^10_000_000 {
$total = $total + %h<a> + %h<b>;
}
And the hash store one is:
my @keys = 'a'..'z';
for ^500_000 {
my %h;
for @keys {
%h{$_} = 42;
}
}
The improvements are nothing whatsoever to do with hashing itself, but instead look to be mostly thanks to much tighter code all around due to caller-side decontainerization. That can have a secondary effect of bringing things under the size limit for inlining, which is also a big help. Speedup factors of 2x and 1.85x are welcome, although we could really do with the same level of improvement again for me to be reasonably happy with our results.
The line-reading benchmark is:
my $fh = open "longfile";
my $chars = 0;
for $fh.lines { $chars = $chars + .chars };
$fh.close;
say $chars
Again, nothing specific to I/O got faster, but when dispatch – the glue that puts together all the pieces – gets a boost, it helps all over the place. (We are also decently competitive on this benchmark, although tend to be slower the moment the UTF-8 decoder can’t take it’s “NFG can’t possibly apply” fast path.)
I’ve also started looking at larger programs, and hearing results from others about theirs. It’s mostly encouraging:
Text::CSV benchmark test-t has seen roughly 20% improvement (thanks to lizmat for measuring)Cro::HTTP test application gets through about 10% more requests per secondCORE.setting, the standard library. However, a big pinch of salt is needed here: the compiler itself has changed in a number of places as part of the work, and there were a couple of things tweaked based on looking at profiles that aren’t really related to dispatch.One unpredicted (by me), but also welcome, improvement is that profiler output has become significantly smaller. Likely reasons for this include:
sink method when a value was in sink context. Now, if we see that the type simply inherits that method from Mu, we elide the call entirely (again, it would inline away, but a smaller call graph is a smaller profile).proto when the cache was missed, but would then not call an onlystar proto again when it got cache hits in the future. This meant the call tree under many multiple dispatches was duplicated in the profile. This wasn’t just a size issue; it was a bit annoying to have this effect show up in the profile reports too.To give an example of the difference, I took profiles from Agrammon to study why it might have become slower. The one from before the dispatcher work weighed in at 87MB; the one with the new dispatch mechanism is under 30MB. That means less memory used while profiling, less time to write the profile out to disk afterwards, and less time for tools to load the profiler output. So now it’s faster to work out how to make things faster.
I’m afraid so. Startup time has suffered. While the new dispatch mechanism is more powerful, pushes more complexity out of the VM into high level code, and is more conducive to reaching higher peak performance, it also has a higher warmup time. At the time of writing, the impact on startup time seems to be around 25%. I expect we can claw some of that back ahead of the October release.
Changes of this scale always come with an amount of risk. We’re merging this some weeks ahead of the next scheduled monthly release in order to have time for more testing, and to address any regressions that get reported. However, even before reaching the point of merging it, we have:
blin to run the tests of ecosystem modules. This is a standard step when preparing Rakudo releases, but in this case we’ve aimed it at the new-disp branches. This found a number of regressions caused by the switch to the new dispatch mechanism, which have been addressed.As I’ve alluded to in a number of places in this post, while there are improvements to be enjoyed right away, there are also new opportunities for further improvement. Some things that are on my mind include:
callsame one here is a perfect example! The point we do the resumption of a dispatch is inside callsame, so all the inline cache entries of resumptions throughout the program stack up in one place. What we’d like is to have them attached a level down the callstack instead. Otherwise, the level of callsame improvement seen in micro-benchmarks will not be enjoyed in larger applications. This applies in a number of other situations too.FALLBACK method could have its callsite easily rewritten to do that, opening the way to inlining.Ints (which needs a great deal of care in memory management, as they may box a big integer, not just a native integer).I would like to thank TPF and their donors for providing the funding that has made it possible for me to spend a good amount of my working time on this effort.
While I’m to blame for the overall design and much of the implementation of the new dispatch mechanism, plenty of work has also been put in by other MoarVM and Rakudo contributors – especially over the last few months as the final pieces fell into place, and we turned our attention to getting it production ready. I’m thankful to them not only for the code and debugging contributions, but also much support and encouragement along the way. It feels good to have this merged, and I look forward to building upon it in the months and years to come.
There was an interesting discussion on IRC today. In brief, it was about exposing one’s database structures over API and security implications of this approach. I’d recommend reading the whole thing because Altreus delivers a good (and somewhat emotional 🙂) point on why such practice is most definitely bad design decision. Despite having minor objections, I generally agree to him.
But I’m not wearing out my keyboard on this post just to share that discussion. There was something in it what made me feel as if I miss something. And it came to me a bit later, when I was done with my payjob and got a bit more spare resources for the brain to utilize.
First of all, a bell rang when a hash was mentioned as the mediator between a database and API return value. I’m somewhat wary about using hashes as return values primarily for a reason of performance price and concurrency unsafety.
Anyway, the discussion went on and came to the point where it touched the ground of blacklisting of a DB table fields vs. whitelisting. The latter is really worthy approach of marking those fields we want in a JSON (or a hash) rather than marking those we don’t want because blacklisting requires us to remember to mark any new sensitive field as prohibited explicitly. Apparently, it is easy to forget to stick the mark onto it.
Doesn’t it remind you something? Aren’t we talking about hashes now? Isn’t it what we sometimes blame JavaScript for, that its objects are free-form with barely any reliable control over their structure? Thanks TypeScript for trying to get this fixed in some funky way, which I personally like more than dislike.
That’s when things clicked together. I was giving this answer already on a different occasion: using a class instance is often preferable over a hash. In the light of the JSON/API safety this simple rule gets us to another rather interesting aspect. Here is an example SmokeMachine provided on IRC:
to-json %( name => "{ .first-name } { .last-name }",
password => "***" )
given $model
This was about returning basic user account information to a frontend. This is supposed to replace JSONification of a Red model like the following:
model Account {
has UInt $.id is serial is json-skip;
has Str $.username is column{ ... };
has Str $.password is column{ ... } is json-skip;
has Str $.first-name is column{ ... };
has Str $.last-name is column{ ... };
}
The model example is mine.
By the way, in my opinion, neither first name nor last name do not belong to this model and must be part of a separate table where user’s personal data is kept. In more general case, a name must either be a long single field or an array where one can fit something like “Pablo Diego José Francisco de Paula Juan Nepomuceno María de los Remedios Cipriano de la Santísima Trinidad Ruiz y Picasso”.
The model clearly demonstrates the blacklist approach with two fields marked as non-JSONifiable. Now, let’s make it the right way, as I see it:
class API::Data::User {
has Str:D $.username is required;
has Str $.first-name;
has Str $.last-name;
method !FROM-MODEL($model) {
self.new: username => .username,
first-name => .first-name,
last-name => .last-name
given $model
}
multi method new(Account:D $model) {
self!FROM-MODEL($model)
}
method COERCE(Account:D $model) {
self!FROM-MODEL($model)
}
}
And now, somewhere in our code we can do:
method get-user-info(UInt:D $id) {
to-json API::Data::User(Account.^load: :$id)
}
With Cro::RPC::JSON module this could be part of a general API class which
would provide common interface to both front- and backend:
use Cro::RPC::JSON;
class API::User {
method get-user-info(UInt:D $id) is json-rpc {
API::Data::User(Account.^load: :$id)
}
}
With such an implementation our Raku backend would get an instance of
API::Data::User. In a TypeScript frontend code of a private project of mine I
have something like the following snippet, where connection is an object
derived from jayson module:
connection.call("get-user-info", id).then(
(user: User | undefined | null) => { ... }
);
What does it all eventually give us? First, API::Data::User provides the
mechanism of whilelisting the fields we do want to expose in API. Note that
with properly defined attributes we’re as explicit about that as only possible.
And we do it declaratively one single place.
Second, the class prevents us from mistyping field names. It wouldn’t be
possible to have something like %( usrname => $model.username, ... ) somewhere
else in our codebase. Or, perhaps even more likely, to try %user<frst-name>
and wonder where did the first name go? We also get the protection against wrong
data types or undefined values.
It is also likely that working with a class instance would be faster than with a hash. I have this subject covered in another post of mine.
Heh, at some point I thought this post could fit into IRC format… 🤷
It is with great sadness that I must announce my resignation as chair of the Perl Foundation’s Community Affairs Team (CAT, the team that responds to Code of Conduct reports), effective immediately. Normally this would be a hard decision to make, but I have no choice given TPF’s recent actions. A Charter and a Code of Conduct could and should have been passed many months ago by the Board of Directors. Sadly this has not happened. The TPF Board of Directors has now unilaterally retracted all of the CAT’s transparency reports of 2021 (first, second). This includes the second transparency report that the TPF Board itself approved the contents and penalties of. Retracting the CAT’s transparency reports sends the message the Board of Directors is not willing to support the CAT, and is not prioritizing the safety of the community. I was not involved in the decision by the Board of Directors.
Remaining on the Community Affairs Team would imply I accept or support TPF’s actions. I do not.
The reason given by the Board of Directors, was that the CAT shouldn’t have acted before a Charter was passed. And since the CAT acted without such a Charter, all of its reports need to be retracted. Even the ones previously approved by the same Board of Directors!
I do not find this reasoning very compelling. While it is important to have a Charter passed and power delegated to a body that can enforce a Code of Conduct, the safety of the community should be more important! If the Board of Directors can pass a ban and transparency report, then later retract it (without involvement of the CAT), it also has the power to pass a Charter for the CAT, AND also have the power to retract the same Charter based on pubic pressure. This is what TPF’s retraction demonstrates to everyone, that even if a Charter is passed, TPF may give in to public pressure and walk back their own past statements. I find this unsettling.
I have put in a large amount of work into creating a Charter for the CAT and a Code of Conduct. I have submitted this to the Board of Directors several times, each time refining it after comments from the Board of Directors. Even if imperfect, it is important to have some kind of Charter to work with! Sadly this has not happened. What has happened instead is backtracking and now finally retracting and erasing the CAT’s past reports.
The #tpf-cat Slack channel was intended to be used by people working to improve the state of the Community Affairs Team and a Code of Conduct within the community. Instead of improving the state of the Community Affairs Team, it has instead been consumed with people trying to tear it down. I have not been on that Slack channel for several weeks now due to the bad behavior and personal attacks I have received there over a period of months. Effectively zero moderation, which I find unacceptable.
I will not do any volunteer work for The Perl Foundation again until a Charter and TPF wide Code of Conduct is passed. I would also need to be confident that the TPF communication channels (be it Slack or whatever platform TPF will use) has an enforced Code of Conduct, moderation playbook, and independent moderators. Let me be clear, a Charter, a Code of Conduct and other documents have already been presented to the Board of Directors. It’s up to the Board of Directors to get it past the finish line, in whatever form it decides to do. Or not.
In any case, it is clear that the Board of Directors is not supporting the Community Affairs Team in its current form, so it is time for me to take my leave.
I guess that for many of you what I’m about to write is fairly obvious. But I hadn’t really thought about for loops this way before. So you more or less witness my spontaneous reaction.
The other day Joelle Maslak tweeted something that made me think. Joelle pointed out that in Raku the code blocks of for loops are just lambdas — anonymous functions.
— @jmaslak
After seeing it I have to agree. Not only can I not unsee it, I find that it’s a thing of beauty.
But what it did too was to make me think about what else is possible with for loops. One idea could be to clean up for loops in general. But what’d be more fun was to see if I could create examples that are possible but not things of beauty, i.e. introduce some complexity and worst practises that no one should ever copy.
My first instinct was to check whether the lambdas could be replaced by non-anonymous functions. And they can, provided you flip the order of the for statement and the function:
Now, this isn’t unique in any way. I include the example here just to prove a point: Anonymous blocks can be replaced with named subs. Many programming languages can do this, and you have probably done this lots of times (most of what I show here can be done in, say, JavaScript; but since it was Raku that made me think of this stuff, the examples will be in Raku).
Personally, though, I’ve never thought about replacing for code blocks with subs. Mostly I’ve had a sub first and then called it from a for loop later. As I think about it, it makes sense to think about the sub and the loop simultaneously: Branching out the code loop into a sub can be a good way to shorten and clean up a piece of code. Especially when what happens in the block is a fairly long and maybe convoluted piece of code. It keeps the main code shorter and perhaps, hopefully, more readable.
This way of doing it can also be used with gather/take and similar constructs. I had to use parentheses to make it work:
OK, that was the easy — and perhaps obvious — ones. Now for the uglier stuff.
Since everything in Raku is an object, even sub routines, you can reference subs in lists and arrays. In the example below I’ve got two subs, A and B, and reference them in the array @subs. What this enables us to do, is to loop through the array and invoke the subs from there.
I include this example just as an exercise. Line 15 is basically a way to conditionally pick which sub to call. There may be some practical applications of this, although practicality is beside the point in this article.
In any case — what’s possible with named subs is also possible with anonymous functions.
But it can get even worse than this. Have a look at the following example:
What we do here is creating code conditionally and dynamically (and, honestly, you should never do that). And, again, I haven’t considered whether this has a practical application or not.
So is there a conclusion here? Not in the ordinary sense. But what it does, I guess, is to show that even if something is possible, it’s not necessarily something you should do. It’s the age old recommendation: Do as I say, not as I do.
Just have noticed that normally I have 4 editors/IDEs running at the same time:
Only Vim I could quit on occasion.
What is your state of affairs?
It’s almost 10 years since Tim Cook took the reins at Apple. A lot has happened since. But many still talk about him as if he’s just taken over, often lamenting that Apple is not as innovative now as it was under Steve Jobs.
I for one don’t understand why people would think that. It is an undeserved underestimation of him.
Yes, Cook is substantially more grey and dull than Jobs. But something good must have happened in his period as CEO — arguably even something better than under Jobs. Because at the time Cook took over Apple’s market cap was $354 billion. At the time of writing — May 14, 2021 — Apple’s worth $2048 billion. Tim’s Apple is in other words almost 6x as valuable as Steve’s Apple. You don’t get that kind of growth and valuation by only being a pencil pusher.
So why does he get so little credit? There have arguably been a couple of revolutions under him too, but they are not as easy to spot as before. It all comes down to style. Let’s start with a couple of examples.
The Apple Watch and the AirPods are Cook products. They may not have defined a new segment (Apple products seldom do), and they may not have impressed initially (more on that later), but they’ve grown to dominate the wearables segment with a 51 % market share. In 2019 Apple claimed that their wearable product segment alone was the size of a Fortune 200 company.
The difference from the Jobs days, however, is that new product launches now use a few years to find their roles. Jobs was a master at defining what something was from the get go, whereas Cook’s Apple uses time and patience to let the new products find their place in the world.
Take the Apple Watch as an example. Starting as a run-of-the-mill smart watch — although a beautiful one — the Apple Watch has iterated into a health focused power house. It is on the brink of revolutionising how we monitor and predict health issues in a way we’ve never been able to before. There were smartwatches before Apple, but the Apple product put the rest of the industry in a catch-up mode.
The AirPods have a similar story. Starting as run-of-the-mill earbuds they’ve grown into feature rich gizmos with spatial sound, Siri integration, and lots of other stuff. As was the case with Apple Watch, they once again put other dominant firms (Sony springs to mind) in catch-up mode.
Lastly their ARM based chip designs are what made these other gadgets possible. No AirPods without the S1 system-on-a-chip. No Apple Watch without the U1. Initially they built the iPhones on third-party chip sets. But they (and the iPad) wouldn’t have grown into what they are now without the A series chips. The high-end iPads and the Macs have just gotten the proprietary M1 chipset. One can just speculate as to what these products can become when things settle down a bit.
I don’t own either of these gadgets and is a casual iPhone user at best. So I can’t be accused of being a fan boy. But even so I find it hard to underestimate the Cook era Apple. As you see I think Apple is just as revolutionary now as it was under Jobs. Their products are, eventually, just as groundbreaking. It’s just that Cook plays the long game and sees the revolution play out over time.
The two couldn’t be more different. But you’d be hard pressed to find flaws in their respective results.
This article started as a comment to Erik Engheim’s article Apple is Turning Into the Next Microsoft.
I recently wrote about the new MoarVM dispatch mechanism, and in that post noted that I still had a good bit of Raku’s multiple dispatch semantics left to implement in terms of it. Since then, I’ve made a decent amount of progress in that direction. This post contains an overview of the approach taken, and some very rough performance measurements.
Of all the kinds of dispatch we find in Raku, multiple dispatch is the most complex. Multiple dispatch allows us to write a set of candidates, which are then selected by the number of arguments:
multi ok($condition, $desc) {
say ($condition ?? 'ok' !! 'not ok') ~ " - $desc";
}
multi ok($condition) {
ok($condition, '');
}
Or the types of arguments:
multi to-json(Int $i) { ~$i }
multi to-json(Bool $b) { $b ?? 'true' !! 'false' }
And not just one argument, but potentially many:
multi truncate(Str $str, Int $chars) {
$str.chars < $chars ?? $str !! $str.substr(0, $chars) ~ '...'
}
multi truncate(Str $str, Str $after) {
with $str.index($after) -> $pos {
$str.substr(0, $pos) ~ '...'
}
else {
$str
}
}
We may write where clauses to differentiate candidates on properties that are not captured by nominal types:
multi fac($n where $n <= 1) { 1 }
multi fac($n) { $n * fac($n - 1) }
Every time we write a set of multi candidates like this, the compiler will automatically produce a proto routine. This is what is installed in the symbol table, and holds the candidate list. However, we can also write our own proto, and use the special term {*} to decide at which point we do the dispatch, if at all.
proto mean($collection) {
$collection.elems == 0 ?? Nil !! {*}
}
multi mean(@arr) {
@arr.sum / @arr.elems
}
multi mean(%hash) {
%hash.values.sum / %hash.elems
}
Candidates are ranked by narrowness (using topological sorting). If multiple candidates match, but they are equally narrow, then that’s an ambiguity error. Otherwise, we call narrowest one. The candidate we choose may then use callsame and friends to defer to the next narrowest candidate, which may do the same, until we reach the most general matching one.
Raku leans heavily on multiple dispatch. Most operators in Raku are compiled into calls to multiple dispatch subroutines. Even $a + $b will be a multiple dispatch. This means doing multiple dispatch efficiently is really important for performance. Given the riches of its semantics, this is potentially a bit concerning. However, there’s good news too.
The overwhelmingly common case is that we have:
where clausesprotocallsameThis isn’t to say the other cases are unimportant; they are really quite useful, and it’s desirable for them to perform well. However, it’s also desirable to make what savings we can in the common case. For example, we don’t want to eagerly calculate the full set of possible candidates for every single multiple dispatch, because the majority of the time only the first one matters. This is not just a time concern: recall that the new dispatch mechanism stores dispatch programs at each callsite, and if we store the list of all matching candidates at each of those, we’ll waste a lot of memory too.
The situation in Rakudo today is as follows:
proto holding a “dispatch cache”, a special-case mechanism implemented in the VM that uses a search tree, with one level per argument.proto, it’s not too bad either, though inlining isn’t going to be happening; it can still use the search tree, thoughwhere clauses, it’ll be slow, because the search tree only deals in finding one candidate per set of nominal types, and so we can’t use itcallsame; it’ll be slow tooEffectively, the situation today is that you simply don’t use where clauses in a multiple dispatch if its anywhere near a hot path (well, and if you know where the hot paths are, and know that this kind of dispatch is slow). Ditto for callsame, although that’s less commonly reached for. The question is, can we do better with the new dispatcher?
Let’s start out with seeing how the simplest cases are dealt with, and build from there. (This is actually what I did in terms of the implementation, but at the same time I had a rough idea where I was hoping to end up.)
Recall this pair of candidates:
multi truncate(Str $str, Int $chars) {
$str.chars < $chars ?? $str !! $str.substr(0, $chars) ~ '...'
}
multi truncate(Str $str, Str $after) {
with $str.index($after) -> $pos {
$str.substr(0, $pos) ~ '...'
}
else {
$str
}
}
We then have a call truncate($message, "\n"), where $message is a Str. Under the new dispatch mechanism, the call is made using the raku-call dispatcher, which identifies that this is a multiple dispatch, and thus delegates to raku-multi. (Multi-method dispatch ends up there too.)
The record phase of the dispatch – on the first time we reach this callsite – will proceed as follows:
raku-invoke dispatcher with the chosen candidate.When we reach the same callsite again, we can run the dispatch program, which quickly checks if the argument types match those we saw last time, and if they do, we know which candidate to invoke. These checks are very cheap – far cheaper than walking through all of the candidates and examining each of them for a match. The optimizer may later be able to prove that the checks will always come out true and eliminate them.
Thus the whole of the dispatch processes – at least for this simple case where we only have types and arity – can be “explained” to the virtual machine as “if the arguments have these exact types, invoke this routine”. It’s pretty much the same as we were doing for method dispatch, except there we only cared about the type of the first argument – the invocant – and the value of the method name. (Also recall from the previous post that if it’s a multi-method dispatch, then both method dispatch and multiple dispatch will guard the type of the first argument, but the duplication is eliminated, so only one check is done.)
Coming up with good abstractions is difficult, and therein lies much of the challenge of the new dispatch mechanism. Raku has quite a number of different dispatch-like things. However, encoding all of them directly in the virtual machine leads to high complexity, which makes building reliable optimizations (or even reliable unoptimized implementations!) challenging. Thus the aim is to work out a comparatively small set of primitives that allow for dispatches to be “explained” to the virtual machine in such a way that it can deliver decent performance.
It’s fairly clear that callsame is a kind of dispatch resumption, but what about the custom proto case and the where clause case? It turns out that these can both be neatly expressed in terms of dispatch resumption too (the where clause case needing one small addition at the virtual machine level, which in time is likely to be useful for other things too). Not only that, but encoding these features in terms of dispatch resumption is also quite direct, and thus should be efficient. Every trick we teach the specializer about doing better with dispatch resumptions can benefit all of the language features that are implemented using them, too.
Recall this example:
proto mean($collection) {
$collection.elems == 0 ?? Nil !! {*}
}
Here, we want to run the body of the proto, and then proceed to the chosen candidate at the point of the {*}. By contrast, when we don’t have a custom proto, we’d like to simply get on with calling the correct multi.
To achieve this, I first moved the multi candidate selection logic from the raku-multi dispatcher to the raku-multi-core dispatcher. The raku-multi dispatcher then checks if we have an “onlystar” proto (one that does not need us to run it). If so, it delegates immediately to raku-multi-core. If not, it saves the arguments to the dispatch as the resumption initialization state, and then calls the proto. The proto‘s {*} is compiled into a dispatch resumption. The resumption then delegates to raku-multi-core. Or, in code:
nqp::dispatch('boot-syscall', 'dispatcher-register', 'raku-multi',
# Initial dispatch, only setting up resumption if we need to invoke the
# proto.
-> $capture {
my $callee := nqp::captureposarg($capture, 0);
my int $onlystar := nqp::getattr_i($callee, Routine, '$!onlystar');
if $onlystar {
# Don't need to invoke the proto itself, so just get on with the
# candidate dispatch.
nqp::dispatch('boot-syscall', 'dispatcher-delegate', 'raku-multi-core', $capture);
}
else {
# Set resume init args and run the proto.
nqp::dispatch('boot-syscall', 'dispatcher-set-resume-init-args', $capture);
nqp::dispatch('boot-syscall', 'dispatcher-delegate', 'raku-invoke', $capture);
}
},
# Resumption means that we have reached the {*} in the proto and so now
# should go ahead and do the dispatch. Make sure we only do this if we
# are signalled to that it's a resume for an onlystar (resumption kind 5).
-> $capture {
my $track_kind := nqp::dispatch('boot-syscall', 'dispatcher-track-arg', $capture, 0);
nqp::dispatch('boot-syscall', 'dispatcher-guard-literal', $track_kind);
my int $kind := nqp::captureposarg_i($capture, 0);
if $kind == 5 {
nqp::dispatch('boot-syscall', 'dispatcher-delegate', 'raku-multi-core',
nqp::dispatch('boot-syscall', 'dispatcher-get-resume-init-args'));
}
elsif !nqp::dispatch('boot-syscall', 'dispatcher-next-resumption') {
nqp::dispatch('boot-syscall', 'dispatcher-delegate', 'boot-constant',
nqp::dispatch('boot-syscall', 'dispatcher-insert-arg-literal-obj',
$capture, 0, Nil));
}
});
Deferring to the next candidate (for example with callsame) and trying the next candidate because a where clause failed look very similar: both involve walking through a list of possible candidates. There’s some details, but they have a great deal in common, and it’d be nice if that could be reflected in how multiple dispatch is implemented using the new dispatcher.
Before that, a slightly terrible detail about how things work in Rakudo today when we have where clauses. First, the dispatcher does a “trial bind”, where it asks the question: would this signature bind? To do this, it has to evaluate all of the where clauses. Worse, it has to use the slow-path signature binder too, which interprets the signature, even though we can in many cases compile it. If the candidate matches, great, we select it, and then invoke it…which runs the where clauses a second time, as part of the compiled signature binding code. There is nothing efficient about this at all, except for it being by far more efficient on developer time, which is why it happened that way.
Anyway, it goes without saying that I’m rather keen to avoid this duplicate work and the slow-path binder where possible as I re-implement this using the new dispatcher. And, happily, a small addition provides a solution. There is an op assertparamcheck, which any kind of parameter checking compiles into (be it type checking, where clause checking, etc.) This triggers a call to a function that gets the arguments, the thing we were trying to call, and can then pick through them to produce an error message. The trick is to provide a way to invoke a routine such that a bind failure, instead of calling the error reporting function, will leave the routine and then do a dispatch resumption! This means we can turn failure to pass where clause checks into a dispatch resumption, which will then walk to the next candidate and try it instead.
This gets us most of the way to a solution, but there’s still the question of being memory and time efficient in the common case, where there is no resumption and no where clauses. I coined the term “trivial multiple dispatch” for this situation, which makes the other situation “non-trivial”. In fact, I even made a dispatcher called raku-multi-non-trivial! There are two ways we can end up there.
where clauses. As soon as we see this is the case, we go ahead and produce a full list of possible candidates that could match. This is a linked list (see my previous post for why).callsame happens, we end up in the trivial dispatch resumption handler, which – since this situation is now non-trivial – builds the full candidate list, snips the first item off it (because we already ran that), and delegates to raku-multi-non-trivial.Lost in this description is another significant improvement: today, when there are where clauses, we entirely lose the ability to use the MoarVM multiple dispatch cache, but under the new dispatcher, we store a type-filtered list of candidates at the callsite, and then cheap type guards are used to check it is valid to use.
I did a few benchmarks to see how the new dispatch mechanism did with a couple of situations known to be sub-optimal in Rakudo today. These numbers do not reflect what is possible, because at the moment the specializer does not have much of an understanding of the new dispatcher. Rather, they reflect the minimal improvement we can expect.
Consider this benchmark using a multi with a where clause to recursively implement factorial.
multi fac($n where $n <= 1) { 1 }
multi fac($n) { $n * fac($n - 1) }
for ^100_000 {
fac(10)
}
say now - INIT now;
This needs some tweaks (and to be run under an environment variable) to use the new dispatcher; these are temporary, until such a time I switch Rakudo over to using the new dispatcher by default:
use nqp;
multi fac($n where $n <= 1) { 1 }
multi fac($n) { $n * nqp::dispatch('raku-call', &fac, $n - 1) }
for ^100_000 {
nqp::dispatch('raku-call', &fac, 10);
}
say now - INIT now;
On my machine, the first runs in 4.86s, the second in 1.34s. Thus under the new dispatcher this runs in little over a quarter of the time it used to – a quite significant improvement already.
A case involving callsame is also interesting to consider. Here it is without using the new dispatcher:
multi fallback(Any $x) { "a$x" }
multi fallback(Numeric $x) { "n" ~ callsame }
multi fallback(Real $x) { "r" ~ callsame }
multi fallback(Int $x) { "i" ~ callsame }
for ^1_000_000 {
fallback(4+2i);
fallback(4.2);
fallback(42);
}
say now - INIT now;
And with the temporary tweaks to use the new dispatcher:
use nqp;
multi fallback(Any $x) { "a$x" }
multi fallback(Numeric $x) { "n" ~ new-disp-callsame }
multi fallback(Real $x) { "r" ~ new-disp-callsame }
multi fallback(Int $x) { "i" ~ new-disp-callsame }
for ^1_000_000 {
nqp::dispatch('raku-call', &fallback, 4+2i);
nqp::dispatch('raku-call', &fallback, 4.2);
nqp::dispatch('raku-call', &fallback, 42);
}
say now - INIT now;
On my machine, the first runs in 31.3s, the second in 11.5s, meaning that with the new dispatcher we manage it in a little over a third of the time that current Rakudo does.
These are both quite encouraging, but as previously mentioned, a majority of multiple dispatches are of the trivial kind, not using these features. If I make the most common case worse on the way to making other things better, that would be bad. It’s not yet possible to make a fair comparison of this: trivial multiple dispatches already receive a lot of attention in the specializer, and it doesn’t yet optimize code using the new dispatcher well. Of note, in an example like this:
multi m(Int) { }
multi m(Str) { }
for ^1_000_000 {
m(1);
m("x");
}
say now - INIT now;
Inlining and other optimizations will turn this into an empty loop, which is hard to beat. There is one thing we can already do, though: run it with the specializer disabled. The new dispatcher version looks like this:
use nqp;
multi m(Int) { }
multi m(Str) { }
for ^1_000_000 {
nqp::dispatch('raku-call', &m, 1);
nqp::dispatch('raku-call', &m, "x");
}
say now - INIT now;
The results are 0.463s and 0.332s respectively. Thus, the baseline execution time – before the specializer does its magic – is less using the new general dispatch mechanism than it is using the special-case multiple dispatch cache that we currently use. I wasn’t sure what to expect here before I did the measurement. Given we’re going from a specialized mechanism that has been profiled and tweaked to a new general mechanism that hasn’t received such attention, I was quite ready to be doing a little bit worse initially, and would have been happy with parity. Running in 70% of the time was a bigger improvement than I expected at this point.
I expect that once the specializer understands the new dispatch mechanism better, it will be able to also turn the above into an empty loop – however, since more iterations can be done per-optimization, this should still show up as a win for the new dispatcher.
With one relatively small addition, the new dispatch mechanism is already handling most of the Raku multiple dispatch semantics. Furthermore, even without the specializer and JIT really being able to make a good job of it, some microbenchmarks already show a factor of 3x-4x improvement. That’s a pretty good starting point.
There’s still a good bit to do before we ship a Rakudo release using the new dispatcher. However, multiple dispatch was the biggest remaining threat to the design: it’s rather more involved than other kinds of dispatch, and it was quite possible that an unexpected shortcoming could trigger another round of design work, or reveal that the general mechanism was going to struggle to perform compared to the more specialized one in the baseline unoptimized, case. So far, there’s no indication of either of these, and I’m cautiously optimistic that the overall design is about right.
I love Perl 6 asynchronous features. They are so easy to use and can give instant boost by changing few lines of code that I got addicted to them. I became asynchronous junkie. And finally overdosed. Here is my story...
I was processing a document that was divided into chapters, sub-chapters, sub-sub-chapters and so on. Parsed to data structure it looked like this:
my %document = (
'1' => {
'1.1' => 'Lorem ipsum',
'1.2' => {
'1.2.1' => 'Lorem ipsum',
'1.2.2' => 'Lorem ipsum'
}
},
'2' => {
'2.1' => {
'2.1.1' => 'Lorem ipsum'
}
}
);
Every chapter required processing of its children before it could be processed. Also processing of each chapter was quite time consuming - no matter which level it was and how many children did it have. So I started by writing recursive function to do it:
sub process (%chapters) {
for %chapters.kv -> $number, $content {
note "Chapter $number started";
&?ROUTINE.($content) if $content ~~ Hash;
sleep 1; # here the chapter itself is processed
note "Chapter $number finished";
}
}
process(%document);
So nothing fancy here. Maybe except current &?ROUTINE variable which makes recursive code less error prone - there is no need to repeat subroutine name explicitly. After running it I got expected DFS (Depth First Search) flow:
$ time perl6 run.pl
Chapter 1 started
Chapter 1.1 started
Chapter 1.1 finished
Chapter 1.2 started
Chapter 1.2.1 started
Chapter 1.2.1 finished
Chapter 1.2.2 started
Chapter 1.2.2 finished
Chapter 1.2 finished
Chapter 1 finished
Chapter 2 started
Chapter 2.1 started
Chapter 2.1.1 started
Chapter 2.1.1 finished
Chapter 2.1 finished
Chapter 2 finished
real 0m8.184s
It worked perfectly, but that was too slow. Because 1 second was required to process each chapter in serial manner it ran for 8 seconds total. So without hesitation I reached for Perl 6 asynchronous goodies to process chapters in parallel.
sub process (%chapters) {
await do for %chapters.kv -> $number, $content {
start {
note "Chapter $number started";
&?ROUTINE.outer.($content) if $content ~~ Hash;
sleep 1; # here the chapter itself is processed
note "Chapter $number finished";
}
}
}
process(%document);
Now every chapter is processed asynchronously in parallel and first waits for its children to be also processed asynchronously in parallel. Note that after wrapping processing in await/start construct &?ROUTINE must now point to outer scope.
$ time perl6 run.pl
Chapter 1 started
Chapter 2 started
Chapter 1.1 started
Chapter 1.2 started
Chapter 2.1 started
Chapter 1.2.1 started
Chapter 2.1.1 started
Chapter 1.2.2 started
Chapter 1.1 finished
Chapter 1.2.1 finished
Chapter 1.2.2 finished
Chapter 2.1.1 finished
Chapter 2.1 finished
Chapter 1.2 finished
Chapter 1 finished
Chapter 2 finished
real 0m3.171s
Perfect. Time dropped to expected 3 seconds - it was not possible to go any faster because document had 3 nesting levels and each required 1 second to process. Still smiling I threw bigger document at my beautiful script - 10 chapters, each with 10 sub-chapters, each with 10 sub-sub-chapters. It started processing, run for a while... and DEADLOCKED.
Friedrich Nietzsche said that "when you gaze long into an abyss the abyss also gazes into you". Same rule applies to code. After few minutes me and my code were staring at each other. And I couldn't find why it worked perfectly for small documents but was deadlocking in random moments for big ones. Half an hour later I blinked and got defeated by my own code in staring contest. So it was time for debugging.
I noticed that when it was deadlocking there was always constant amount of 16 chapters that were still in progress. And that number looked familiar to me - thread pool!
$ perl6 -e 'say start { }'
Promise.new(
scheduler => ThreadPoolScheduler.new(
initial_threads => 0,
max_threads => 16,
uncaught_handler => Callable
),
status => PromiseStatus::Kept
)
Every asynchronous task that is planned needs free thread so it can be executed. And on my system only 16 concurrent threads are allowed as shown above. To analyze what happened let's use document from first example but also assume thread pool is limited to 4:
$ perl6 run.pl # 4 threads available by default
Chapter 1 started # 3 threads available
Chapter 1.1 started # 2 threads available
Chapter 2 started # 1 thread available
Chapter 1.1 finished # 2 threads available again
Chapter 1.2 started # 1 thread available
Chapter 1.2.1 started # 0 threads available
# deadlock!
At this moment chapter 1 subtree holds three threads and waits for one more for chapter 1.2.2 to complete everything and start ascending from recursion. And subtree of chapter 2 holds one thread and waits for one more for chapter 2.1 to descend into recursion. In result processing gets to a point where at least one more thread is required to proceed but all threads are taken and none can be returned to thread pool. Script deadlocks and stops here forever.
How to solve this problem and maintain parallel processing? There are many ways to do it :)
The key to the solution is to process asynchronously only those chapters that do not have unprocessed chapters on lower level.
Luckily Perl 6 offers perfect tool - promise junctions. It is possible to create a promise that waits for other promises to be kept and until it happens it is not sent to thread pool for execution. Following code illustrates that:
my $p = Promise.allof( Promise.in(2), Promise.in(3) );
sleep 1;
say "Promise after 1 second: " ~ $p.perl;
sleep 3;
say "Promise after 4 seconds: " ~ $p.perl;
Prints:
Promise after 1 second: Promise.new(
..., status => PromiseStatus::Planned
)
Promise after 4 seconds: Promise.new(
..., status => PromiseStatus::Kept
)
Let's rewrite processing using this cool property:
sub process (%chapters) {
return Promise.allof(
do for %chapters.kv -> $number, $content {
my $current = {
note "Chapter $number started";
sleep 1; # here the chapter itself is processed
note "Chapter $number finished";
};
if $content ~~ Hash {
Promise.allof( &?ROUTINE.($content) )
.then( $current );
}
else {
Promise.start( $current );
}
}
);
}
await process(%document);
It solves the problem when chapter was competing with its sub-chapters for free threads but at the same time it needed those sub-chapters before it can process itself. Now awaiting for sub-chapters to complete does not require free thread. Let's run it:
$ perl6 run.pl
Chapter 1.1 started
Chapter 1.2.1 started
Chapter 1.2.2 started
Chapter 2.1.1 started
-
Chapter 1.1 finished
Chapter 1.2.1 finished
Chapter 1.2.2 finished
Chapter 1.2 started
Chapter 2.1.1 finished
Chapter 2.1 started
-
Chapter 1.2 finished
Chapter 1 started
Chapter 2.1 finished
Chapter 2 started
-
Chapter 1 finished
Chapter 2 finished
real 0m3.454s
I've added separator for each second passed so it is easier to understand. When script starts chapters 1.1, 1.2.1, 1.2.2 and 2.1.1 do not have sub-chapters at all. So they can take threads from thread pool immediately. When they are completed after one second then Promises that were awaiting for all of them are kept and chapters 1.2 and 2.1 can be processed safely on thread pool. It keeps going until getting out of recursion.
After trying big document again it was processed flawlessly in 72 seconds instead of linear 1000.
I'm high on asynchronous processing again!
You can download script here and try different data sizes and algorithms for yourself (params are taken from command line).
My goodness, it appears I’m writing my first Raku internals blog post in over two years. Of course, two years ago it wasn’t even called Raku. Anyway, without further ado, let’s get on with this shared brainache.
I use “dispatch” to mean a process by which we take a set of arguments and end up with some action being taken based upon them. Some familiar examples include:
$basket.add($product, $quantity). We might traditionally call just $product and $qauntity the arguments, but for my purposes, all of $basket, the method name 'add', $product, and $quantity` are arguments to the dispatch: they are the things we need in order to make a decision about what we’re going to do.uc($youtube-comment). Since Raku sub calls are lexically resolved, in this case the arguments to the dispatch are &uc (the result of looking up the subroutine) and $youtube-comment.At first glance, perhaps the first two seem fairly easy and the third a bit more of a handful – which is sort of true. However, Raku has a number of other features that make dispatch rather more, well, interesting. For example:
wrap allows us to wrap any Routine (sub or method); the wrapper can then choose to defer to the original routine, either with the original arguments or with new argumentsproto routine that gets to choose when – or even if – the call to the appropriate candidate is madecallsame in order to defer to the next candidate in the dispatch. But what does that mean? If we’re in a multiple dispatch, it would mean the next most applicable candidate, if any. If we’re in a method dispatch then it means a method from a base class. (The same thing is used to implement going to the next wrapper or, eventually, to the originally wrapped routine too). And these can be combined: we can wrap a multi method, meaning we can have 3 levels of things that all potentially contribute the next thing to call!Thanks to this, dispatch – at least in Raku – is not always something we do and produce an outcome, but rather a process that we may be asked to continue with multiple times!
Finally, while the examples I’ve written above can all quite clearly be seen as examples of dispatch, a number of other common constructs in Raku can be expressed as a kind of dispatch too. Assignment is one example: the semantics of it depend on the target of the assignment and the value being assigned, and thus we need to pick the correct semantics. Coercion is another example, and return value type-checking yet another.
Dispatch is everywhere in our programs, quietly tieing together the code that wants stuff done with the code that does stuff. Its ubiquity means it plays a significant role in program performance. In the best case, we can reduce the cost to zero. In the worst case, the cost of the dispatch is high enough to exceed that of the work done as a result of the dispatch.
To a first approximation, when the runtime “understands” the dispatch the performance tends to be at least somewhat decent, but when it doesn’t there’s a high chance of it being awful. Dispatches tend to involve an amount of work that can be cached, often with some cheap guards to verify the validity of the cached outcome. For example, in a method dispatch, naively we need to walk a linearization of the inheritance graph and ask each class we encounter along the way if it has a method of the specified name. Clearly, this is not going to be terribly fast if we do it on every method call. However, a particular method name on a particular type (identified precisely, without regard to subclassing) will resolve to the same method each time. Thus, we can cache the outcome of the lookup, and use it whenever the type of the invocant matches that used to produce the cached result.
When one starts building a runtime aimed at a particular language, and has to do it on a pretty tight budget, the most obvious way to get somewhat tolerable performance is to bake various hot-path language semantics into the runtime. This is exactly how MoarVM started out. Thus, if we look at MoarVM as it stood several years ago, we find things like:
where comes at a very high cost)Sub object has a private attribute in it that holds the low-level code handle identifying the bytecode to run)These are all still there today, however are also all on the way out. What’s most telling about this list is what isn’t included. Things like:
$obj.SomeType::method-name())A few years back I started to partially address this, with the introduction of a mechanism I called “specializer plugins”. But first, what is the specializer?
When MoarVM started out, it was a relatively straightforward interpreter of bytecode. It only had to be fast enough to beat the Parrot VM in order to get a decent amount of usage, which I saw as important to have before going on to implement some more interesting optimizations (back then we didn’t have the kind of pre-release automated testing infrastructure we have today, and so depended much more on feedback from early adopters). Anyway, soon after being able to run pretty much as much of the Raku language as any other backend, I started on the dynamic optimizer. It gathered type statistics as the program was interpreted, identified hot code, put it into SSA form, used the type statistics to insert guards, used those together with static properties of the bytecode to analyze and optimize, and produced specialized bytecode for the function in question. This bytecode could elide type checks and various lookups, as well as using a range of internal ops that make all kinds of assumptions, which were safe because of the program properties that were proved by the optimizer. This is called specialized bytecode because it has had a lot of its genericity – which would allow it to work correctly on all types of value that we might encounter – removed, in favor of working in a particular special case that actually occurs at runtime. (Code, especially in more dynamic languages, is generally far more generic in theory than it ever turns out to be in practice.)
This component – the specializer, known internally as “spesh” – delivered a significant further improvement in the performance of Raku programs, and with time its sophistication has grown, taking in optimizations such as inlining and escape analysis with scalar replacement. These aren’t easy things to build – but once a runtime has them, they create design possibilities that didn’t previously exist, and make decisions made in their absence look sub-optimal.
Of note, those special-cased language-specific mechanisms, baked into the runtime to get some speed in the early days, instead become something of a liability and a bottleneck. They have complex semantics, which means they are either opaque to the optimizer (so it can’t reason about them, meaning optimization is inhibited) or they need special casing in the optimizer (a liability).
So, back to specializer plugins. I reached a point where I wanted to take on the performance of things like $obj.?meth (the “call me maybe” dispatch), $obj.SomeType::meth() (dispatch qualified with a class to start looking in), and private method calls in roles (which can’t be resolved statically). At the same time, I was getting ready to implement some amount of escape analysis, but realized that it was going to be of very limited utility because assignment had also been special-cased in the VM, with a chunk of opaque C code doing the hot path stuff.
But why did we have the C code doing that hot-path stuff? Well, because it’d be too espensive to have every assignment call a VM-level function that does a bunch of checks and logic. Why is that costly? Because of function call overhead and the costs of interpretation. This was all true once upon a time. But, some years of development later:
I solved the assignment problem and the dispatch problems mentioned above with the introduction of a single new mechanism: specializer plugins. They work as follows:
The vast majority of cases are monomorphic, meaning that only one set of guards are produced and they always succeed thereafter. The specializer can thus compile those guards into the specialized bytecode and then assume the given target invocant is what will be invoked. (Further, duplicate guards can be eliminated, so the guards a particular plugin introduces may reduce to zero.)
Specializer plugins felt pretty great. One new mechanism solved multiple optimization headaches.
The new MoarVM dispatch mechanism is the answer to a fairly simple question: what if we get rid of all the dispatch-related special-case mechanisms in favor of something a bit like specializer plugins? The resulting mechanism would need to be a more powerful than specializer plugins. Further, I could learn from some of the shortcomings of specializer plugins. Thus, while they will go away after a relatively short lifetime, I think it’s fair to say that I would not have been in a place to design the new MoarVM dispatch mechanism without that experience.
All the method caching. All the multi dispatch caching. All the specializer plugins. All the invocation protocol stuff for unwrapping the bytecode handle in a code object. It’s all going away, in favor of a single new dispatch instruction. Its name is, boringly enough, dispatch. It looks like this:
dispatch_o result, 'dispatcher-name', callsite, arg0, arg1, ..., argN
Which means:
dispatcher-nameresult(Aside: this implies a new calling convention, whereby we no longer copy the arguments into an argument buffer, but instead pass the base of the register set and a pointer into the bytecode where the register argument map is found, and then do a lookup registers[map[argument_index]] to get the value for an argument. That alone is a saving when we interpret, because we no longer need a loop around the interpreter per argument.)
Some of the arguments might be things we’d traditionally call arguments. Some are aimed at the dispatch process itself. It doesn’t really matter – but it is more optimal if we arrange to put arguments that are only for the dispatch first (for example, the method name), and those for the target of the dispatch afterwards (for example, the method parameters).
The new bootstrap mechanism provides a small number of built-in dispatchers, whose names start with “boot-“. They are:
boot-value– take the first argument and use it as the result (the identity function, except discarding any further arguments)boot-constant – take the first argument and produce it as the result, but also treat it as a constant value that will always be produced (thus meaning the optimizer could consider any pure code used to calculate the value as dead)boot-code – take the first argument, which must be a VM bytecode handle, and run that bytecode, passing the rest of the arguments as its parameters; evaluate to the return value of the bytecodeboot-syscall – treat the first argument as the name of a VM-provided built-in operation, and call it, providing the remaining arguments as its parametersboot-resume – resume the topmost ongoing dispatchThat’s pretty much it. Every dispatcher we build, to teach the runtime about some other kind of dispatch behavior, eventually terminates in one of these.
Teaching MoarVM about different kinds of dispatch is done using nothing less than the dispatch mechanism itself! For the most part, boot-syscall is used in order to register a dispatcher, set up the guards, and provide the result that goes with them.
Here is a minimal example, taken from the dispatcher test suite, showing how a dispatcher that provides the identity function would look:
nqp::dispatch('boot-syscall', 'dispatcher-register', 'identity', -> $capture {
nqp::dispatch('boot-syscall', 'dispatcher-delegate', 'boot-value', $capture);
});
sub identity($x) {
nqp::dispatch('identity', $x)
}
ok(identity(42) == 42, 'Can define identity dispatch (1)');
ok(identity('foo') eq 'foo', 'Can define identity dispatch (2)');
In the first statement, we call the dispatcher-register MoarVM system call, passing a name for the dispatcher along with a closure, which will be called each time we need to handle the dispatch (which I tend to refer to as the “dispatch callback”). It receives a single argument, which is a capture of arguments (not actually a Raku-level Capture, but the idea – an object containing a set of call arguments – is the same).
Every user-defined dispatcher should eventually use dispatcher-delegate in order to identify another dispatcher to pass control along to. In this case, it delegates immediately to boot-value – meaning it really is nothing except a wrapper around the boot-value built-in dispatcher.
The sub identity contains a single static occurrence of the dispatch op. Given we call the sub twice, we will encounter this op twice at runtime, but the two times are very different.
The first time is the “record” phase. The arguments are formed into a capture and the callback runs, which in turn passes it along to the boot-value dispatcher, which produces the result. This results in an extremely simple dispatch program, which says that the result should be the first argument in the capture. Since there’s no guards, this will always be a valid result.
The second time we encounter the dispatch op, it already has a dispatch program recorded there, so we are in run mode. Turning on a debugging mode in the MoarVM source, we can see the dispatch program that results looks like this:
Dispatch program (1 temporaries)
Ops:
Load argument 0 into temporary 0
Set result object value from temporary 0
That is, it reads argument 0 into a temporary location and then sets that as the result of the dispatch. Notice how there is no mention of the fact that we went through an extra layer of dispatch; those have zero cost in the resulting dispatch program.
Argument captures are immutable. Various VM syscalls exist to transform them into new argument captures with some tweak, for example dropping or inserting arguments. Here’s a further example from the test suite:
nqp::dispatch('boot-syscall', 'dispatcher-register', 'drop-first', -> $capture {
my $capture-derived := nqp::dispatch('boot-syscall', 'dispatcher-drop-arg', $capture, 0);
nqp::dispatch('boot-syscall', 'dispatcher-delegate', 'boot-value', $capture-derived);
});
ok(nqp::dispatch('drop-first', 'first', 'second') eq 'second',
'dispatcher-drop-arg works');
This drops the first argument before passing the capture on to the boot-value dispatcher – meaning that it will return the second argument. Glance back at the previous dispatch program for the identity function. Can you guess how this one will look?
Well, here it is:
Dispatch program (1 temporaries)
Ops:
Load argument 1 into temporary 0
Set result string value from temporary 0
Again, while in the record phase of such a dispatcher we really do create capture objects and make a dispatcher delegation, the resulting dispatch program is far simpler.
Here’s a slightly more involved example:
my $target := -> $x { $x + 1 }
nqp::dispatch('boot-syscall', 'dispatcher-register', 'call-on-target', -> $capture {
my $capture-derived := nqp::dispatch('boot-syscall',
'dispatcher-insert-arg-literal-obj', $capture, 0, $target);
nqp::dispatch('boot-syscall', 'dispatcher-delegate',
'boot-code-constant', $capture-derived);
});
sub cot() { nqp::dispatch('call-on-target', 49) }
ok(cot() == 50,
'dispatcher-insert-arg-literal-obj works at start of capture');
ok(cot() == 50,
'dispatcher-insert-arg-literal-obj works at start of capture after link too');
Here, we have a closure stored in a variable $target. We insert it as the first argument of the capture, and then delegate to boot-code-constant, which will invoke that code object and pass the other dispatch arguments to it. Once again, at the record phase, we really do something like:
And the resulting dispatch program? It’s this:
Dispatch program (1 temporaries)
Ops:
Load collectable constant at index 0 into temporary 0
Skip first 0 args of incoming capture; callsite from 0
Invoke MVMCode in temporary 0
That is, load the constant bytecode handle that we’re going to invoke, set up the args (which are in this case equal to those of the incoming capture), and then invoke the bytecode with those arguments. The argument shuffling is, once again, gone. In general, whenever the arguments we do an eventual bytecode invocation with are a tail of the initial dispatch arguments, the arguments transform becomes no more than a pointer addition.
All of the dispatch programs seen so far have been unconditional: once recorded at a given callsite, they shall always be used. The big missing piece to make such a mechanism have practical utility is guards. Guards assert properties such as the type of an argument or if the argument is definite (Int:D) or not (Int:U).
Here’s a somewhat longer test case, with some explanations placed throughout it.
# A couple of classes for test purposes
my class C1 { }
my class C2 { }
# A counter used to make sure we're only invokving the dispatch callback as
# many times as we expect.
my $count := 0;
# A type-name dispatcher that maps a type into a constant string value that
# is its name. This isn't terribly useful, but it is a decent small example.
nqp::dispatch('boot-syscall', 'dispatcher-register', 'type-name', -> $capture {
# Bump the counter, just for testing purposes.
$count++;
# Obtain the value of the argument from the capture (using an existing
# MoarVM op, though in the future this may go away in place of a syscall)
# and then obtain the string typename also.
my $arg-val := nqp::captureposarg($capture, 0);
my str $name := $arg-val.HOW.name($arg-val);
# This outcome is only going to be valid for a particular type. We track
# the argument (which gives us an object back that we can use to guard
# it) and then add the type guard.
my $arg := nqp::dispatch('boot-syscall', 'dispatcher-track-arg', $capture, 0);
nqp::dispatch('boot-syscall', 'dispatcher-guard-type', $arg);
# Finally, insert the type name at the start of the capture and then
# delegate to the boot-constant dispatcher.
nqp::dispatch('boot-syscall', 'dispatcher-delegate', 'boot-constant',
nqp::dispatch('boot-syscall', 'dispatcher-insert-arg-literal-str',
$capture, 0, $name));
});
# A use of the dispatch for the tests. Put into a sub so there's a single
# static dispatch op, which all dispatch programs will hang off.
sub type-name($obj) {
nqp::dispatch('type-name', $obj)
}
# Check with the first type, making sure the guard matches when it should
# (although this test would pass if the guard were ignored too).
ok(type-name(C1) eq 'C1', 'Dispatcher setting guard works');
ok($count == 1, 'Dispatch callback ran once');
ok(type-name(C1) eq 'C1', 'Can use it another time with the same type');
ok($count == 1, 'Dispatch callback was not run again');
# Test it with a second type, both record and run modes. This ensures the
# guard really is being checked.
ok(type-name(C2) eq 'C2', 'Can handle polymorphic sites when guard fails');
ok($count == 2, 'Dispatch callback ran a second time for new type');
ok(type-name(C2) eq 'C2', 'Second call with new type works');
# Check that we can use it with the original type too, and it has stacked
# the dispatch programs up at the same callsite.
ok(type-name(C1) eq 'C1', 'Call with original type still works');
ok($count == 2, 'Dispatch callback only ran a total of 2 times');
This time two dispatch programs get produced, one for C1:
Dispatch program (1 temporaries)
Ops:
Guard arg 0 (type=C1)
Load collectable constant at index 1 into temporary 0
Set result string value from temporary 0
And another for C2:
Dispatch program (1 temporaries)
Ops:
Guard arg 0 (type=C2)
Load collectable constant at index 1 into temporary 0
Set result string value from temporary 0
Once again, no leftovers from capture manipulation, tracking, or dispatcher delegation; the dispatch program does a type guard against an argument, then produces the result string. The whole call to $arg-val.HOW.name($arg-val) is elided, the dispatcher we wrote encoding the knowledge – in a way that the VM can understand – that a type’s name can be considered immutable.
This example is a bit contrived, but now consider that we instead look up a method and guard on the invocant type: that’s a method cache! Guard the types of more of the arguments, and we have a multi cache! Do both, and we have a multi-method cache.
The latter is interesting in so far as both the method dispatch and the multi dispatch want to guard on the invocant. In fact, in MoarVM today there will be two such type tests until we get to the point where the specializer does its work and eliminates these duplicated guards. However, the new dispatcher does not treat the dispatcher-guard-type as a kind of imperative operation that writes a guard into the resultant dispatch program. Instead, it declares that the argument in question must be guarded. If some other dispatcher already did that, it’s idempotent. The guards are emitted once all dispatch programs we delegate through, on the path to a final outcome, have had their say.
Fun aside: those being especially attentive will have noticed that the dispatch mechanism is used as part of implementing new dispatchers too, and indeed, this ultimately will mean that the specializer can specialize the dispatchers and have them JIT-compiled into something more efficient too. After all, from the perspective of MoarVM, it’s all just bytecode to run; it’s just that some of it is bytecode that tells the VM how to execute Raku programs more efficiently!
A resumable dispatcher needs to do two things:
When a resumption happens, the resume callback will be called, with any arguments for the resumption. It can also obtain the resume initialization state that was set in the dispatch callback. The resume initialization state contains the things needed in order to continue with the dispatch the first time it is resumed. We’ll take a look at how this works for method dispatch to see a concrete example. I’ll also, at this point, switch to looking at the real Rakudo dispatchers, rather than simplified test cases.
The Rakudo dispatchers take advantage of delegation, duplicate guards, and capture manipulations all having no runtime cost in the resulting dispatch program to, in my mind at least, quite nicely factor what is a somewhat involved dispatch process. There are multiple entry points to method dispatch: the normal boring $obj.meth(), the qualified $obj.Type::meth(), and the call me maybe $obj.?meth(). These have common resumption semantics – or at least, they can be made to provided we always carry a starting type in the resume initialization state, which is the type of the object that we do the method dispatch on.
Here is the entry point to dispatch for a normal method dispatch, with the boring details of reporting missing method errors stripped out.
# A standard method call of the form $obj.meth($arg); also used for the
# indirect form $obj."$name"($arg). It receives the decontainerized invocant,
# the method name, and the the args (starting with the invocant including any
# container).
nqp::dispatch('boot-syscall', 'dispatcher-register', 'raku-meth-call', -> $capture {
# Try to resolve the method call using the MOP.
my $obj := nqp::captureposarg($capture, 0);
my str $name := nqp::captureposarg_s($capture, 1);
my $meth := $obj.HOW.find_method($obj, $name);
# Report an error if there is no such method.
unless nqp::isconcrete($meth) {
!!! 'Error reporting logic elided for brevity';
}
# Establish a guard on the invocant type and method name (however the name
# may well be a literal, in which case this is free).
nqp::dispatch('boot-syscall', 'dispatcher-guard-type',
nqp::dispatch('boot-syscall', 'dispatcher-track-arg', $capture, 0));
nqp::dispatch('boot-syscall', 'dispatcher-guard-literal',
nqp::dispatch('boot-syscall', 'dispatcher-track-arg', $capture, 1));
# Add the resolved method and delegate to the resolved method dispatcher.
my $capture-delegate := nqp::dispatch('boot-syscall',
'dispatcher-insert-arg-literal-obj', $capture, 0, $meth);
nqp::dispatch('boot-syscall', 'dispatcher-delegate',
'raku-meth-call-resolved', $capture-delegate);
});
Now for the resolved method dispatcher, which is where the resumption is handled. First, let’s look at the normal dispatch callback (the resumption callback is included but empty; I’ll show it a little later).
# Resolved method call dispatcher. This is used to call a method, once we have
# already resolved it to a callee. Its first arg is the callee, the second and
# third are the type and name (used in deferral), and the rest are the args to
# the method.
nqp::dispatch('boot-syscall', 'dispatcher-register', 'raku-meth-call-resolved',
# Initial dispatch
-> $capture {
# Save dispatch state for resumption. We don't need the method that will
# be called now, so drop it.
my $resume-capture := nqp::dispatch('boot-syscall', 'dispatcher-drop-arg',
$capture, 0);
nqp::dispatch('boot-syscall', 'dispatcher-set-resume-init-args', $resume-capture);
# Drop the dispatch start type and name, and delegate to multi-dispatch or
# just invoke if it's single dispatch.
my $delegate_capture := nqp::dispatch('boot-syscall', 'dispatcher-drop-arg',
nqp::dispatch('boot-syscall', 'dispatcher-drop-arg', $capture, 1), 1);
my $method := nqp::captureposarg($delegate_capture, 0);
if nqp::istype($method, Routine) && $method.is_dispatcher {
nqp::dispatch('boot-syscall', 'dispatcher-delegate', 'raku-multi', $delegate_capture);
}
else {
nqp::dispatch('boot-syscall', 'dispatcher-delegate', 'raku-invoke', $delegate_capture);
}
},
# Resumption
-> $capture {
... 'Will be shown later';
});
There’s an arguable cheat in raku-meth-call: it doesn’t actually insert the type object of the invocant in place of the invocant. It turns out that it doesn’t really matter. Otherwise, I think the comments (which are to be found in the real implementation also) tell the story pretty well.
One important point that may not be clear – but follows a repeating theme – is that the setting of the resume initialization state is also more of a declarative rather than an imperative thing: there isn’t a runtime cost at the time of the dispatch, but rather we keep enough information around in order to be able to reconstruct the resume initialization state at the point we need it. (In fact, when we are in the run phase of a resume, we don’t even have to reconstruct it in the sense of creating a capture object.)
Now for the resumption. I’m going to present a heavily stripped down version that only deals with the callsame semantics (the full thing has to deal with such delights as lastcall and nextcallee too). The resume initialization state exists to seed the resumption process. Once we know we actually do have to deal with resumption, we can do things like calculating the full list of methods in the inheritance graph that we want to walk through. Each resumable dispatcher gets a single storage slot on the call stack that it can use for its state. It can initialize this in the first step of resumption, and then update it as we go. Or more precisely, it can set up a dispatch program that will do this when run.
A linked list turns out to be a very convenient data structure for the chain of candidates we will walk through. We can work our way through a linked list by keeping track of the current node, meaning that there need only be a single thing that mutates, which is the current state of the dispatch. The dispatch program mechanism also provides a way to read an attribute from an object, and that is enough to express traversing a linked list into the dispatch program. This also means zero allocations.
So, without further ado, here is the linked list (rather less pretty in NQP, the restricted Raku subset, than it would be in full Raku):
# A linked list is used to model the state of a dispatch that is deferring
# through a set of methods, multi candidates, or wrappers. The Exhausted class
# is used as a sentinel for the end of the chain. The current state of the
# dispatch points into the linked list at the appropriate point; the chain
# itself is immutable, and shared over (runtime) dispatches.
my class DeferralChain {
has $!code;
has $!next;
method new($code, $next) {
my $obj := nqp::create(self);
nqp::bindattr($obj, DeferralChain, '$!code', $code);
nqp::bindattr($obj, DeferralChain, '$!next', $next);
$obj
}
method code() { $!code }
method next() { $!next }
};
my class Exhausted {};
And finally, the resumption handling.
nqp::dispatch('boot-syscall', 'dispatcher-register', 'raku-meth-call-resolved',
# Initial dispatch
-> $capture {
... 'Presented earlier;
},
# Resumption. The resume init capture's first two arguments are the type
# that we initially did a method dispatch against and the method name
# respectively.
-> $capture {
# Work out the next method to call, if any. This depends on if we have
# an existing dispatch state (that is, a method deferral is already in
# progress).
my $init := nqp::dispatch('boot-syscall', 'dispatcher-get-resume-init-args');
my $state := nqp::dispatch('boot-syscall', 'dispatcher-get-resume-state');
my $next_method;
if nqp::isnull($state) {
# No state, so just starting the resumption. Guard on the
# invocant type and name.
my $track_start_type := nqp::dispatch('boot-syscall', 'dispatcher-track-arg', $init, 0);
nqp::dispatch('boot-syscall', 'dispatcher-guard-type', $track_start_type);
my $track_name := nqp::dispatch('boot-syscall', 'dispatcher-track-arg', $init, 1);
nqp::dispatch('boot-syscall', 'dispatcher-guard-literal', $track_name);
# Also guard on there being no dispatch state.
my $track_state := nqp::dispatch('boot-syscall', 'dispatcher-track-resume-state');
nqp::dispatch('boot-syscall', 'dispatcher-guard-literal', $track_state);
# Build up the list of methods to defer through.
my $start_type := nqp::captureposarg($init, 0);
my str $name := nqp::captureposarg_s($init, 1);
my @mro := nqp::can($start_type.HOW, 'mro_unhidden')
?? $start_type.HOW.mro_unhidden($start_type)
!! $start_type.HOW.mro($start_type);
my @methods;
for @mro {
my %mt := nqp::hllize($_.HOW.method_table($_));
if nqp::existskey(%mt, $name) {
@methods.push(%mt{$name});
}
}
# If there's nothing to defer to, we'll evaluate to Nil (just don't set
# the next method, and it happens below).
if nqp::elems(@methods) >= 2 {
# We can defer. Populate next method.
@methods.shift; # Discard the first one, which we initially called
$next_method := @methods.shift; # The immediate next one
# Build chain of further methods and set it as the state.
my $chain := Exhausted;
while @methods {
$chain := DeferralChain.new(@methods.pop, $chain);
}
nqp::dispatch('boot-syscall', 'dispatcher-set-resume-state-literal', $chain);
}
}
elsif !nqp::istype($state, Exhausted) {
# Already working through a chain of method deferrals. Obtain
# the tracking object for the dispatch state, and guard against
# the next code object to run.
my $track_state := nqp::dispatch('boot-syscall', 'dispatcher-track-resume-state');
my $track_method := nqp::dispatch('boot-syscall', 'dispatcher-track-attr',
$track_state, DeferralChain, '$!code');
nqp::dispatch('boot-syscall', 'dispatcher-guard-literal', $track_method);
# Update dispatch state to point to next method.
my $track_next := nqp::dispatch('boot-syscall', 'dispatcher-track-attr',
$track_state, DeferralChain, '$!next');
nqp::dispatch('boot-syscall', 'dispatcher-set-resume-state', $track_next);
# Set next method, which we shall defer to.
$next_method := $state.code;
}
else {
# Dispatch already exhausted; guard on that and fall through to returning
# Nil.
my $track_state := nqp::dispatch('boot-syscall', 'dispatcher-track-resume-state');
nqp::dispatch('boot-syscall', 'dispatcher-guard-literal', $track_state);
}
# If we found a next method...
if nqp::isconcrete($next_method) {
# Call with same (that is, original) arguments. Invoke with those.
# We drop the first two arguments (which are only there for the
# resumption), add the code object to invoke, and then leave it
# to the invoke or multi dispatcher.
my $just_args := nqp::dispatch('boot-syscall', 'dispatcher-drop-arg',
nqp::dispatch('boot-syscall', 'dispatcher-drop-arg', $init, 0),
0);
my $delegate_capture := nqp::dispatch('boot-syscall',
'dispatcher-insert-arg-literal-obj', $just_args, 0, $next_method);
if nqp::istype($next_method, Routine) && $next_method.is_dispatcher {
nqp::dispatch('boot-syscall', 'dispatcher-delegate', 'raku-multi',
$delegate_capture);
}
else {
nqp::dispatch('boot-syscall', 'dispatcher-delegate', 'raku-invoke',
$delegate_capture);
}
}
else {
# No method, so evaluate to Nil (boot-constant disregards all but
# the first argument).
nqp::dispatch('boot-syscall', 'dispatcher-delegate', 'boot-constant',
nqp::dispatch('boot-syscall', 'dispatcher-insert-arg-literal-obj',
$capture, 0, Nil));
}
});
That’s quite a bit to take in, and quite a bit of code. Remember, however, that this is only run for the record phase of a dispatch resumption. It also produces a dispatch program at the callsite of the callsame, with the usual guards and outcome. Implicit guards are created for the dispatcher that we are resuming at that point. In the most common case this will end up monomorphic or bimorphic, although situations involving nestings of multiple dispatch or method dispatch could produce a more morphic callsite.
The design I’ve picked forces resume callbacks to deal with two situations: the first resumption and the latter resumptions. This is not ideal in a couple of ways:
Only the second of these really matters. The reason for the non-uniformity is to make sure that the overwhelming majority of calls, which never lead to a dispatch resumption, incur no per-dispatch cost for a feature that they never end up using. If the result is a little more cost for those using the feature, so be it. In fact, early benchmarking shows callsame with wrap and method calls seems to be up to 10 times faster using the new dispatcher than in current Rakudo, and that’s before the specializer understands enough about it to improve things further!
Everything I’ve discussed above is implemented, except that I may have given the impression somewhere that multiple dispatch is fully implemented using the new dispatcher, and that is not the case yet (no handling of where clauses and no dispatch resumption support).
Getting the missing bits of multiple dispatch fully implemented is the obvious next step. The other missing semantic piece is support for callwith and nextwith, where we wish to change the arguments that are being used when moving to the next candidate. A few other minor bits aside, that in theory will get all of the Raku dispatch semantics at least supported.
Currently, all standard method calls ($obj.meth()) and other calls (foo() and $foo()) go via the existing dispatch mechanism, not the new dispatcher. Those will need to be migrated to use the new dispatcher also, and any bugs that are uncovered will need fixing. That will get things to the point where the new dispatcher is semantically ready.
After that comes performance work: making sure that the specializer is able to deal with dispatch program guards and outcomes. The goal, initially, is to get steady state performance of common calling forms to perform at least as well as in the current master branch of Rakudo. It’s already clear enough there will be some big wins for some things that to date have been glacial, but it should not come at the cost of regression on the most common kinds of dispatch, which have received plenty of optimization effort before now.
Furthermore, NQP – the restricted form of Raku that the Rakudo compiler and other bits of the runtime guts are written in – also needs to be migrated to use the new dispatcher. Only when that is done will it be possible to rip out the current method cache, multiple dispatch cache, and so forth from MoarVM.
An open question is how to deal with backends other than MoarVM. Ideally, the new dispatch mechanism will be ported to those. A decent amount of it should be possible to express in terms of the JVM’s invokedynamic (and this would all probably play quite well with a Truffle-based Raku implementation, although I’m not sure there is a current active effort in that area).
While my current focus is to ship a Rakudo and MoarVM release that uses the new dispatcher mechanism, that won’t be the end of the journey. Some immediate ideas:
handles (delegation) and FALLBACK (handling missing method call) mechanisms can be made to perform better using the new dispatcherassuming – used to curry or otherwise prime arguments for a routine – is not ideal, and an implementation that takes advantage of the argument rewriting capabilities of the new dispatcher would likely perform a great deal betterSome new language features may also be possible to provide in an efficient way with the help of the new dispatch mechanism. For example, there’s currently not a reliable way to try to invoke a piece of code, just run it if the signature binds, or to do something else if it doesn’t. Instead, things like the Cro router have to first do a trial bind of the signature, and then do the invoke, which makes routing rather more costly. There’s also the long suggested idea of providing pattern matching via signatures with the when construct (for example, when * -> ($x) {}; when * -> ($x, *@tail) { }), which is pretty much the same need, just in a less dynamic setting.
Working on the new dispatch mechanism has been a longer journey than I first expected. The resumption part of the design was especially challenging, and there’s still a few important details to attend to there. Something like four potential approaches were discarded along the way (although elements of all of them influenced what I’ve described in this post). Abstractions that hold up are really, really, hard.
I also ended up having to take a couple of months away from doing Raku work at all, felt a bit crushed during some others, and have been juggling this with the equally important RakuAST project (which will be simplified by being able to assume the presence of the new dispatcher, and also offers me a range of softer Raku hacking tasks, whereas the dispatcher work offers few easy pickings).
Given all that, I’m glad to finally be seeing the light at the end of the tunnel. The work that remains is enumerable, and the day we ship a Rakudo and MoarVM release using the new dispatcher feels a small number of months away (and I hope writing that is not tempting fate!)
The new dispatcher is probably the most significant change to MoarVM since I founded it, in so far as it sees us removing a bunch of things that have been there pretty much since the start. RakuAST will also deliver the greatest architectural change to the Rakudo compiler in a decade. Both are an opportunity to fold years of learning things the hard way into the runtime and compiler. I hope when I look back at it all in another decade’s time, I’ll at least feel I made more interesting mistakes this time around.
Many years back, Larry Wall shared his thesis on the nature of scripting. Since recently even Java gained 'script' support I thought it would be fitting to revisit the topic, and hopefully relevant to the perl and raku language community.
The weakness of Larry's treatment (which, to be fair to the author, I think is more intended to be enlightening than to be complete) is the contrast of scripting with programming. This contrast does not permit a clear separation because scripts are programs. That is to say, no matter how long or short, scripts are written commands for a machine to execute, and I think that's a pretty decent definition of a program in general.
A more useful contrast - and, I think, the intended one - is between scripts and other sorts of programs, because that allows us to compare scripting (writing scripts) with 'programming' (writing non-script programs). And to do that we need to know what other sorts of programs there are.
The short version of that answer is - systems and applications, and a bunch of other things that aren't really relevant to the working programmer, like (embedded) control algorithms, spreadsheets and database queries. (The definition I provided above is very broad, by design, because I don't want to get stuck on boundary questions). Most programmers write applications, some write systems, virtually all write scripts once in a while, though plenty of people who aren't professional programmers also write scripts.
I think the defining features of applications and systems are, respectively:
Consider for instance a mail client (like thunderbird) in comparison to a mailer daemon (like sendmail) - one provides an interface to read and write e-mails (the model) and the other provides functionality to send that e-mail to other servers.
Note that under this (again, broad) definition, libraries are also system software, which makes sense, considering that their users are developers (just as for, say, PostgreSQL) who care about things like performance, reliability, and correctness. Incidentally, libraries as well as 'typical' system software (such as database engines and operating system kernels) tend to be written in languages like C and C++ for much the same reasons.
What then, are the differences between scripts, applications, and systems? I think the following is a good list:
Obviously these distinctions aren't really binary - 'short' versus 'long', 'ad-hoc' versus 'general purpose' - and can't be used to conclusively settle the question whether something is a script or an application. (If, indeed, that question ever comes up). More important is that for the 10 or so scripts I've written over the past year - some professionally, some not - all or most of these properties held, and I'd be surprised if the same isn't true for most readers.
And - finally coming at the point that I'm trying to make today - these features point to a specific niche of programs more than to a specific technology (or set of technologies). To be exact, scripts are (mostly) short, custom programs to automate ad-hoc tasks, tasks that are either to specific or too small to develop and distribute another program for.
This has further implications on the preferred features of a scripting language (taken to mean, a language designed to enable the development of scripts). In particular:
This niche doesn't always exist. In computing environments where everything of interest is adequately captured by an application, or which lacks the ability to effectively automate ad-hoc tasks (I'm thinking in particular of Windows before PowerShell), the practice of scripting tends to not develop. Similarily, in a modern 'cloud' environment, where system setup is controlled by a state machine hosted by another organization, scripting doesn't really have much of a future.
To put it another way, scripting only thrives in an environment that has a lot of 'scriptable' tasks; meaning tasks for which there isn't already a pre-made solution available, environments that have powerful facilities available for a script to access, and whose users are empowered to automate those tasks. Such qualities are common on Unix/Linux 'workstations' but rather less so on smartphones and (as noted before) cloud computing environments.
Truth be told I'm a little worried about that development. I could point to, and expound on, the development and popularity of languages like go and rust, which aren't exactly scripting languages, or the replacement of Javascript with TypeScript, to make the point further, but I don't think that's necessary. At the same time I could point to the development of data science as a discipline to demonstrate that scripting is alive and well (and indeed perhaps more economically relevant than before).
What should be the conclusion for perl 5/7 and raku? I'm not quite sure, mostly because I'm not quite sure whether the broader perl/raku community would prefer their sister languages to be scripting or application languages. (As implied above, I think the Python community chose that they wanted Python 3 to be an application language, and this was not without consequences to their users).
Raku adds a number of features common to application languages (I'm thinking of it's powerful type system in particular), continuing a trend that perl 5 arguably pioneered. This is indeed a very powerful strategy - a language can be introduced for scripts and some of those scripts are then extended into applications (or even systems), thereby ensuring its continued usage. But for it to work, a new perl family language must be introduced on its scripting merits, and there must be a plentiful supply of scriptable tasks to automate, some of which - or a combination of which - grow into an application.
For myself, I would like to see scripting have a bright future. Not just because scripting is the most accessible form of programming, but also because an environment that permits, even requires scripting, is one were not all interesting problems have been solved, one where it's users ask it to do tasks so diverse that there isn't an app for that, yet. One where the true potential of the wonderful devices that surround is can be explored.
In such a world there might well be a bright future for scripting.
In this post, I’d like to demonstrate a few ways of computing factorials using the Raku programming language.
Let me start with the basic and the most effective (non necessarily the most efficient) form of computing the factorial of a given integer number:
say [*] 1..10; # 3628800
In the below examples, we mostly will be dealing with the factorial of 10, so remember the result. But to make the programs more versatile, let us read the number from the command line:
unit sub MAIN($n); say [*] 1..$n;
To run the program, pass the number:
$ raku 00-cmd.raku 10 3628800
The program uses the reduction meta-operator [ ] with the main operator * in it.
You can also start with 2 (you can even compute 0! and 1! this way).
unit sub MAIN($n); say [*] 2..$n;
The second solution is using a postfix for loop to multiply the numbers in the range:
unit sub MAIN($n); my $f = 1; $f *= $_ for 2..$n; say $f;
This solution is not that expressive but still demonstrates quite a clear code.
You can also use map that is applied to a range:
unit sub MAIN($n); my $f = 1; (2..$n).map: $f *= *; say $f;
Refer to my article All the stars of Perl 6, or * ** * to learn more about how to read *= *.
Let’s implement a recursive solution.
unit sub MAIN($n);
sub factorial($n) {
if $n < 2 {
return 1;
}
else {
return $n * factorial($n - 1);
}
}
say factorial(n);
There are two branches, one of which terminates recursion.
The previous program can be rewritten to make a code with less punctuation:
unit sub MAIN($n);
sub factorial($n) {
return 1 if $n < 2;
return $n * factorial($n - 1);
}
say factorial($n);
Here, the first return is managed by a postfix if, and the second return can only be reached if the condition in if is false. So, neither an additional Boolean test nor else is needed.
What if you need to compute a factorial of a relatively big number? No worries, Raku will just do it:
say [*] 1..500;
The speed is more than acceptable for any practical application:
raku 06-long-factorial.raku 0.14s user 0.02s system 124% cpu 0.127 total
Let’s try something opposite and compute a factorial, which can fit a native integer:
unit sub MAIN($n); my int $f = 1; $f *= $_ for 2..$n; say $f;
I am using a for loop here, but notice that the type of $f is a native integer (thus, 4 bytes). This program works with the numbers up to 20:
$ raku 07-int-factorial.raku 20 2432902008176640000
The fun fact is that you can add a dot to the first program 
unit sub MAIN($n); say [*] 1 ... $n;
Now, 1 ... $n is a sequence. You can start it with 2 if you are not planning to compute a factorials of 0 and 1.
Unlike the solution with a range, it is possible to swap the ends of the sequence:
unit sub MAIN($n); say [*] $n ... 1;
Nothing stops us from defining the elements of the sequence with a code block. The next program shows how you do it:
unit sub MAIN($n); my @f = 1, * * ++$ ... *; say @f[$n];
This time, the program generates a sequence of factorials from 1! to $n!, and to print the only one we need, we take the value from the array as @f[$n]. Notice that the sequence itself is lazy and its right end is undefined, so you can’t use @f[*-1], for example.
The rule here is * * ++$ (multiply the last computed value by the incremented index); it is using the built-in state variable $.
The idea of the solutions 4 and 5 with two branches can be further transformed to using multi-functions:
unit sub MAIN($n);
multi sub factorial(1) { 1 }
multi sub factorial($n) { $n * factorial($n - 1) }
say factorial($n);
For the numbers above 1, Raku calls the second variant of the function. When the number comes down to 1, recursion stops, because the first variant is called. Notice how easily you can create a variant of a function that only reacts to the given value.
The previous program loops infinitely if you try to set $n to 0. One of the simplest solution is to add a where clause to catch that case too.
unit sub MAIN($n);
multi sub factorial($n where $n < 2) { 1 }
multi sub factorial($n) { $n * factorial($n - 1) }
say factorial($n);
Here’s another classical Raku solution: modifying its grammar to allow mathematical notation $n!.
unit sub MAIN($n);
sub postfix:<!>($n) {
[*] 1..$n
}
say $n!;
A rarely seen Raku’s feature called methodop (method operator) that allows you to call a function as it if was a method:
unit sub MAIN($n);
sub factorial($n) {
[*] 1..$n
}
say $n.&factorial;
Recursive solutions are perfect subjects for result caching. The following program demonstrates this approach.
unit sub MAIN($n);
use experimental :cached;
sub f($n) is cached {
say "Called f($n)";
return 1 if $n < 2;
return $n * f($n - 1);
}
say f($n div 2);
say f(10);
This program first computes a factorial of the half of the input number, and then of the number itself. The program logs all the calls of the function. You can clearly see that, say, the factorial of 10 is using the results that were already computed for the factorial of 5:
$ raku 15-cached-factorial.raku 10 Called f(5) Called f(4) Called f(3) Called f(2) Called f(1) 120 Called f(10) Called f(9) Called f(8) Called f(7) Called f(6) 3628800
Note that the feature is experimental.
The reduction operator that we already used has a special variant [\ ] that allows to keep all the intermediate results. This is somewhat similar to using a sequence in the example 10.
unit sub MAIN($n); my @f = [\*] 1..$n; say @f[$n - 1];
Now a few programs that go beyond the factorials themselves. The first program computes the value of the expression a! / b!, where both a and b are integer numbers, and a is not less than b.
The idea is to optimise the solution to skip the overlapping parts of the multiplication sequences. For example, 10! / 5! is 6 * 7 * 8 * 9 * 10.
To have more fun, let us modify Raku’s grammar so that it really parses the above expression.
unit sub MAIN($a, $b where $a >= $b);
class F {
has $.n;
}
sub postfix:<!>(Int $n) {
F.new(n => $n)
}
sub infix:</>(F $a, F $b) {
[*] $b.n ^.. $a.n
}
say $a! / $b!;
We already have seen the postfix:<!> operator. To catch division, another operator is defined, but to prevent catching the division of data of other types, a proxy class F is introduced.
To keep proper processing of expression such as 4 / 5, define another / operator that catches things which are not F. Don’t forget to add multi to both options. The callsame built-in routine dispatches control to built-in operator definitions.
. . .
multi sub infix:</>(F $a, F $b) {
[*] $b.n ^.. $a.n
}
multi sub infix:</>($a, $b) {
callsame
}
say $a! / $b!;
say 4 / 5;
Let’s try to reduce the number of multiplications. Take a factorial of 10:
10 * 9 * 8 * 7 * 6 * 5 * 4 * 3 * 2 * 1
Now, take one number from each end, multiply them, and repeat the procedure:
10 * 1 = 10 9 * 2 = 18 8 * 3 = 24 7 * 4 = 28 6 * 5 = 30
You can see that every such result is bigger than the previous one by 8, 6, 4, and 2. In other words, the difference reduces by 2 on each iteration, starting from 10, which is the input number.
The whole program that implements this algorithm is shown below:
unit sub MAIN(
$n is copy where $n %% 2 #= Even numbers only
);
my $f = $n;
my $d = $n - 2;
my $m = $n + $d;
while $d > 0 {
$f *= $m;
$d -= 2;
$m += $d;
}
say $f;
It only works for even input numbers, so it contains a restriction reflected in the where clause of the MAIN function. As homework, modify the program to accept odd numbers too.
Before wrapping up, let’s look at a couple of exotic methods, which, however, can be used to compute factorials of non-integer numbers (or, to be stricter, to compute what can be called extended definition of it).
The proper way would be to use the Gamma function, but let me illustrate the method with a simpler formula:
An integral is a sum by definition, so let’s make a straightforward loop:
unit sub MAIN($n);
my num $f = 0E0;
my num $dx = 1E-6;
loop (my $x = $dx; $x <= 1; $x += $dx) {
$f += (-log($x)) ** $n;
}
say $f * $dx;
With the given step of 1E-6, the result is not that exact:
$ raku 19-integral-factorial.raku 10 3086830.6595557937
But you can compute a ‘factorial’ of a floating-point number. For example, 5! is 120 and 6! is 720, but what is 5.5!?
$ raku 19-integral-factorial.raku 5.5 285.948286477563
And finally, the Stirling’s formula for the rescue. The bigger the n, the more correct is the result.
The implementation can be as simple as this:
unit sub MAIN($n); # τ = 2 * π say (τ * $n).sqrt * ($n / e) ** $n;
But you can make it a bit more outstanding if you have a fixed $n:
say sqrt(τ * 10) * (10 / e)¹⁰;
* * *
And that’s it for now. You can find the source code of all the programs shown here in the GitHub repository github.com/ash/factorial.
I am happy to report that the first part of the Raku course is completed and published. The course is available at course.raku.org.
The grant was approved a year and a half ago right before the PerlCon conference in Rīga. I was the organiser of the event, so I had to postpone the course due to high load. During the conference, it was proposed to rename Perl 6, which, together with other stuff, made me think if the course is needed.
After months, the name was settled, the distinction between Perl and Raku became clearer, and, more importantly, external resourses and services, e.g., Rosettacode and glot.io started using the new name. So, now I think it is still a good idea to create the course that I dreamed about a couple of years ago. I started the main work in the middle of November 2020, and by the beginning of January 2021, I had the first part ready.
The current plan includes five parts:
It differs a bit from the original plan published in the grant proposal. While the material stays the same, I decided to split it differently. Initially, I was going to go through all the topics one after another. Now, the first sections reveal the basics of some topics, and we will return to the same topics on the next level in the second part.
For example, in the first part, I only talk about the basic data types: Int, Rat, Num, Str, Range, Array, List, and Hash and basic usage of them. The rest, including other types (e.g., Date or DateTime) and the methods such as @array.rotate or %hash.kv is delayed until the second part.
Contrary, functions were a subject of the second part initially, but they are now discussed in the first part. So, we now have Part 1 “Raku essentials” and Part 2 “Advanced Raku topics”. This shuffling allowed me to create a liner flow in such a way that the reader can start writing real programs already after they finish the first part of the course.
I must say that it is quite a tricky task to organise the material without backward links. In the ideal course, any topic may only be based on the previously explained information. A couple of the most challenging cases were ranges and typed variables. They both causes a few chicken-and-egg loops.
During the work on the first part, I also prepared a ‘framework’ that generates the navigation through the site and helps with quiz automation. It is hosted as GitHub Pages and uses Jekyll and Liquid for generating static pages, and a couple of Raku programs to automate the process of adding new exercises and highlighting code snippets. Syntax highlighting is done with Pygments.
Returning the to course itself, it includes pages of a few different types:
The quizzes were not part of the grant proposal, but I think they help making a better user experience. All the quizzes have answers and comments. All the exercises are solved and published with the comments to explain the solution, or even to highlight some theoretical aspects.
The first part covers 91 topics and includes 73 quizzes and 65 exercises (with 70 solutions :-). There are about 330 pages in total. The sources are kept in a GitHub repository github.com/ash/raku-course, so people can send pull requiest, etc.
At this point, the first part is fully ready. I may slightly update it if the following parts require additional information about the topics covered in Part 1.
This text is a grant report, and it is also (a bit modified) published at https://news.perlfoundation.org/post/rakucourse1 on 13 January 2021.
C omputer security has seen its share of mind-boggling news lately. None more mind boggling than the news about how alleged Russian hackers installed a backdoor into the IT monitoring product Solarwind Orion. Through this they got got entrance into the computer systems of several US agencies and departments — ironically even into the systems of a cyber security company (Fireeye) and Microsoft itself . The news made me think of my own history with computer security, and down memory lane I went.
One particular day in late July or early August 1989 my parents, sister and me were driving home from a short summer vacation. At a short stop in a largish city, I had found a newsstand carrying foreign magazines. There I’d bought a copy of PC/Computing’s September issue (to this day I don’t understand why American magazines are on sale a couple of months before the cover date) so that I had something to make time in the backseat pass faster.
Among articles about the relatively new MS-DOS replacement OS/2 (an OS co-developed with IBM that Microsoft would come to orphan the minute they launched Windows 3.0 and understood the magnitude of the success they had on their hands) and networking solutions from Novell (which Windows would kill as well, albeit more indirectly), the magazine brought an excerpt of the book “The Cuckoo’s Egg” by a guy named Clifford Stoll. Although I had bought the magazine for the technical information such as the stuff mentioned above, this excerpt stood out. It was a mesmerising story about how an astronomer-turned-IT-guy stumbled over a hacker, and how he, aided by interest but virtually no support from the FBI, CIA and NSA, almost single handedly traced the hacker’s origins back to a sinister government sponsored organisation in the then communist East Germany.
This is the exact moment I discovered that my passion — computers — could form the basis of a great story.
Coastal Norway where I grew up is probably as far from the author’s native San Francisco as anything; at least our book stores were. So it wasn’t until the advent of Amazon.com some years later that I was able to order a copy of the book. Luckily, the years passed had not diminished the story. Granted, the Internet described by the author Clifford Stoll was a little more clunky than the slightly more modern dial-up internet I ordered the book on. But subtract the World Wide Web from the equation and the difference between his late eighties internet and my mid-nineties modem version weren’t all that big. My Internet was as much a monochrome world of telnet and text-only servers as it was a colourful web. Email, for instance, was something I managed by telnetting into a HP Unix server and using the command line utility pine to read and send messages.
What struck me with the story was that the hacker’s success very often was enabled by sloppy system administration; one could arguably say that naïve or ignorant assumptions by sysadmins all across the US made the hack possible. Why sloppy administration and ignorant assumptions? Well, some of the reason was that the Internet was largely run by academia back then. Academia was (and is) a culture of open research and sharing of ideas and information. As such it’s not strange that sysadmins of that time assumed that users of the computer systems had good intentions too.
But no one had considered that the combination of several (open) sources of information and documents stored on these servers, could end in very comprehensive insight into, say, the Space Shuttle program or military nuclear research. Actually, the main downside to unauthorised usage had to do with cost: processing power was expensive at the time and far from a commodity. Users were billed by for their computer usage. So billing was actually the reason why Stoll started his hacker hunt. There was a few cents worth of computer time that couldn’t be accounted for. Finding out whether this was caused by a bug or something else, was the primary goal of Mr. Stoll’s hunt. What the hacker had spent this time on was — at first — a secondary issue at best.
With that in mind it’s maybe not so strange that one of the most common errors made was not changing default passwords on multi-user computers connected to the internet. One of the systems having a default password was the now virtually extinct VAX/VMS operating system for Digital’s microcomputer series VAX. This was one of the things Mr. Stoll found out by logging each and every interaction the hacker, using Stoll’s compromised system as a gateway, had with other systems (the description of how he logged all this by wiring printers up to physical ports on the microcomputer, rewiring the whole thing every time the hacker logged on through another port, is by itself worth reading the book for). Using the backdoor, the hacker did not only gain access to that computer — they got root privileges as well.
In the 30+ years passed since I read the book I’ve convinced myself about two things: 1) we’ve learned to not use default passwords anymore, and 2) that VMS systems exhibiting this kind of backdoor are long gone.
Well, I believed these things until a few weeks ago. That’s when I stumbled on to a reddit post — now deleted, but there still is a cached version available on Waybackmachine. Here the redditor explained how he’d identified 33 remaining VAX/VMS systems still on the Internet:
About a year ago I read the book “A Cuckoo’s Egg”, written in 1989. It included quite a bit of information pertaining to older systems such as the VAX/VMS. I went to Censys (when their big data was still accessible easily and for free) and downloaded a set of the telnet (port 23) data. A quick grep later and I had isolated all the VAX/VMS targets on the net. Low and behold, of the 33 targets (I know, really scraping the bottom of the barrel here) more than half of them were still susceptible to default password attacks literally listed in a book from almost 3 decades ago. After creating super user accounts I contacted these places to let them know that that they were sporting internet facing machines using default logins from 1989. All but one locked me out. This is 31 years later… The future will be a mess, kids
I applaud the redditor that discovered this. Because isn’t what he found a testament of something breathtakingly incompetent and impressive at the same time? Impressive in the sense that someone’s been able to keep these ancient systems alive on the internet for decades; incompetent because the sysadmins has ignored patching the most well documented security flaw of those systems for well over a quarter century?
So maybe this starts to answer question posed in the title: Did we learn anything from this?
Yes, of course we did. If we look past the VMS enthusiast out there, computer security is very different now than back then. Unencrypted communication is almost not used anywhere anymore. Security is provided by multilayered hardware and software solutions. In addition are not only password policies widely enforced on users, but two-factor and other extra layers of authentication is used as well.
But the answer is also No. While my organisations such as my workplace — which is not in the business of having secrets — has implemented lots of the newest security measures, this autumn we learned that the Norwegian parliament — which is in the business of having secrets — haven’t. They had weak password policies and no two-factor authentication for their email system.
Consequently they recently became an easy target for Russian hackers. I obviously don’t know what was the reasoning behind having weak security implemented. But my guess is that the IT department assessed the digital competence of the parliament members and concluded that it was too low for them to handle strong passwords and managing two-factor authentication.
And this is perhaps the point where the security of yesteryear and security today differs the most: As we’re closing in on 2021, weak security is a conscious choice; but it is the same as leaving the door wide open, and any good sysadmin knows it.
The ignorance exhibited in the case of the Norwegian parliament borders, in my opinion, on criminal ignorance — although I guess no one will ever have to take the consquence. What it does prove, however, is that while systems can be as good as anything, people are still the weakest link in any such system.
In sum I think my answer to the initial question is an uneasy Maybe. We still have some way to go before what Cliff Stoll taught us 32 years ago has become second nature.
This week’s task has an interesting solution in Raku. So, here’s the task:
You are given two strings $A and $B. Write a script to check if the given strings are Isomorphic. Print 1 if they are otherwise 0.
OK, so if the two strings are isomorphic, their characters are mapped: for each character from the first string, the character at the same position in the second string is always the same.
In the stings abc and def, a always corresponds to d, b to e, and c to f. That’s a trivial case. But then for the string abca, the corresponding string must be defd.
The letters do not need to go sequentially, so the strings aeiou and bcdfg are isomorphic too, as well as aeiou and gxypq. But also aaeeiioouu and bbccddffgg, or the pair aeaieoiuo and gxgyxpyqp.
The definition also means that the number of different characters is equal in both strings. But it also means that if we make the pairs of corresponding letters, the number of unique pairs is also the same, right? If a matches x, there cannot be any other pair with the first letter a.
Let’s exploit these observation:
sub is-isomorphic($a, $b) {
+(([==] ($a, $b)>>.chars) &&
([==] ($a.comb, $b.comb, ($a.comb Z~ $b.comb))>>.unique));
}
First of all, the strings must have the same length.
Then, the strings are split into characters, and the number of unique characters should also be equal. But the collection of the unique pairs from the corresponding letters from both strings should also be of the same size.
Test it:
use Test;
# . . .
is(is-isomorphic('abc', 'def'), 1);
is(is-isomorphic('abb', 'xyy'), 1);
is(is-isomorphic('sum', 'add'), 0);
is(is-isomorphic('ACAB', 'XCXY'), 1);
is(is-isomorphic('AAB', 'XYZ'), 0);
is(is-isomorphic('AAB', 'XXZ'), 1);
is(is-isomorphic('abc', 'abc'), 1);
is(is-isomorphic('abc', 'ab'), 0);
* * *
→ GitHub repository
→ Navigation to the Raku challenges post series
Today there’s a chance to demonstrate powerful features of Raku on the solution of Day 18 of this year’s Advent of Code.
The task is to print the sum of a list of expressions with +, *, and parentheses, but the precedence of the operations is equal in the first part of the problem, and is opposite to the standard precedence in the second part.
In other words, 3 + 4 * 5 + 6 is (((3 + 4) * 5) + 6) in the first case and (3 + 4) * (5 + 6) in the second.
Here is the solution. I hope you are impressed too.
use MONKEY-SEE-NO-EVAL;
sub infix:<m>($a, $b) { $a * $b }
say [+] ('input.txt'.IO.lines.race.map: *.trans('*' => 'm')).map: {EVAL($_)}
The lines with the expressions come from the file input.txt. For each line, I am replacing * with m, which I earlier made an infix operator that actually does multiplication.
For the second part, we need our m to have lower precedence than +. There’s nothing simpler:
sub infix:<m>($a, $b) is looser<+> { $a * $b }
Parsing and evaluation are done using EVAL.
* * *
→ Browse the code on GitHub
→ See all blog posts on Advent of Code 2020
When I started covid.observer about seven months ago, I thought there would be no need to update it after about 3-4 months. In reality, we are approaching to the end of the year, and I will have to fix the graphs which display data per week, as the week numbers will very soon make a loop.
All this time, more data arrived, and I also made it even more by adding a separate statistics for the regions of Russia, with its 85 subdivisions, which brought the total count of countries and regions up to almost 400.
mysql> select count(distinct cc) from totals; +--------------------+ | count(distinct cc) | +--------------------+ | 392 | +--------------------+ 1 row in set (0.00 sec)
Due to frequent updates that changes data in the past, it is not that easy to make incremental update of statistics, and again, I did not expect that I’ll run the site for so long.
mysql> select count(distinct date) from daily_totals; +----------------------+ | count(distinct date) | +----------------------+ | 329 | +----------------------+ 1 row in set (0.00 sec)
The bottom line is that daily generation became clumsy and not smooth. Before summer, the whole website could be regenerated in less than 15 minutes, but now it turned to 40-50 minutes. And I tend to do it twice a day, as a fresh portion of today’s Russian data arrives a few hours after we’ve got a daily update by the Johns Hopkins University (for yesterday’s stats).
But the most scary signals began after the program started crashing with quite unpleasant errors.
Latest JHU data on 12/12/20 Latest RU data on 12/13/20 Generating impact timeline... Generating world data... MoarVM panic: Unable to initialize event loop
Failed to open file /Users/ash/Projects/covid.observer/COVID-19/csse_covid_19_data/csse_covid_19_daily_reports_us/12-10-2020.csv: Too many open files
Generating impact timeline... Generating world data... Not enough positional arguments; needed at least 4 in sub per-capita-data at /Users/ash/Projects/covid.observer/lib/CovidObserver/Statistics.rakumod (CovidObserver::Statistics) line 1906
The errors were not consistent, and I managed to re-run the program by pieces to get the update. But none of the errors were easily explainable.
MoarVM panic gives no explanation, but it completely disappears if I run the program in two parts:
$ ./covid.raku fetch $ ./covid.raku generate
instead of a combined run that both fetches the data and generates the statistics:
$ ./covid.raku update
The Too many open files is a strange one as while I process the files in loops, I do not intentionally keep them open. But that error seems to be solved by changing system settings:
$ ulimit -n 10000
The final error, Not enough positional arguments; needed at least 4, is the weirdest. Such thing happens when you call a function that expects a different number of arguments. That never occurred for months after all bugs were found and fixed. It can only be explained by the new piece of data. Indeed, it may happen that some data is missing, but I believe I already found all the cases where I need to provide the function calls with default zero values.
Having all that, and the fact that the program run takes dozens of minutes before you can catch an error, it was quite frustrating.
And here comes Liz!
She proposed to look into the things and actually spent the whole day by first installing the code and all its requirements and then by actually doing that job to run, debug, and re-run. By the end of the day she created a pull request, which made the program twice as fast!
Let’s look at the changes. There are three of them (but no, they do not directly answer the above-mentioned three error messages).
The first two changes introduce parallel processing of countries (remember, there are about 400 of what is considered a unique $cc in the program).
my %country-stats = get-known-countries<>.race(:1batch,:8degree).map: -> $cc {
$cc => generate-country-stats($cc, %CO, :%mortality, :%crude, :$skip-excel)
}
Calling .race on the result of get-known-countries() function improves the previously sequential processing of countries. Indeed, their stats are computed independently, so there’s no reason for one country to wait for another. The parameters of race, the batch size and the number of workers, can probably be tuned to fit your hardware.
The second change is similar, but for another part of the code where the continents are processed in a loop:
for %continents.keys.race(:1batch,:8degree) -> $cont {
generate-continent-stats($cont, %CO, :$skip-excel);
}
Finally, the third change is to make some counters native integers instead of Raku Ints:
my int $c = $confirmed[$index] // 0; my int $f = $failed[$index] // 0; my int $r = $recovered[$index] // 0; my int $a = $active[$index] // 0;
I understand that this reduces both the memory and the processing time of these variables, but for some reason it also eliminated the error in counting function parameters.
And finally, I want to mention the <> thing that you may have noticed in the first code change. This is the so-called decontainerization operator. What it does is illustrated by this example from the documentation:
use JSON::Tiny;
my $config = from-json('{ "files": 3, "path": "/home/some-user/raku.pod6" }');
say $config.raku;
# OUTPUT: «${:files(3), :path("/home/some-user/raku.pod6")}»
my %config-hash = $config<>;
say %config-hash.raku;
# OUTPUT: «{:files(3), :path("/home/some-user/raku.pod6")}»
The $config variable is a scalar variable that keeps a hash. To work with it as with a hash, the variable is decontainerized as $config<>. This gives us a proper hash %config-hash.
I think that’s it for now. The main advantage of the above changes is that the program now needs less than 25 minutes to re-generate the whole site and it does not fail.
Well, but it became a bit louder too as Rakudo uses more cores
Thanks, Liz!
I’d like to thank everyone who voted for me in the recent Raku Steering Council elections. By this point, I’ve been working on the language for well over a decade, first to help turn a language design I found fascinating into a working implementation, and since the Christmas release to make that implementation more robust and performant. Overall, it’s been as fun as it has been challenging – in a large part because I’ve found myself sharing the journey with a lot of really great people. I’ve also tried to do my bit to keep the community around the language kind and considerate. Receiving a vote from around 90% of those who participated in the Steering Council elections was humbling.
Alas, I’ve today submitted my resignation to the Steering Council, on personal health grounds. For the same reason, I’ll be taking a step back from Raku core development (Raku, MoarVM, language design, etc.) Please don’t worry too much; I’ll almost certainly be fine. It may be I’m ready to continue working on Raku things in a month or two. It may also be longer. Either way, I think Raku will be better off with a fully sized Steering Council in place, and I’ll be better off without the anxiety that I’m holding a role that I’m not in a place to fulfill.
 |
| Both ADD and SUB refer to the same LOAD node |
 |
| The DO node is inserted for the LET operator. It ensures that the value of the LOAD node is computed before the reference in either branch |