Larry Wall emerged, almost like a deus ex machina, to give his approval of changing the name of the Perl 6 Programming Language to “Raku”. This stirred up quite some reactions on the interwebs:
Next weekend, on Saturday 19 October, it’s time for the London Perl Workshop again. The following presentations with Perl 6 content appear to have been planned:
Thanks to the quickly unrepairable actions of a certain individual outside of the Perl 6 community, the Perl 6 Facebook group has changed all of its URLs and/or is only accessible to people with Facebook logins. And since it is the intention of changing back to the original URLs (which will most likely take a month thanks to Facebook policies), it doesn’t seem to make sense to put deep links here now.
So, if you’re interested in happenings on Facebook, check out the Perl 6 Group, and navigate from there.
The past year has been very stressful for yours truly. And the past week even more so. Some stress will remain in the near future, but it looks like stress levels will be allowed to go down in the further future, with “Perl” and “Raku” each going their own way.
This is the last Perl 6 Weekly that yours truly will write. Next week I will just announce the location of the new Rakudo Weekly blog here.
Why “Rakudo” and not “Raku”, you might ask? Well, originally the “Perl 6 Weekly” was about all implementations of Perl 6. But for the past 4 years, it has effectively been only about the “Rakudo” implementation. Now that “Perl 6” is being renamed to “Raku”, it seems like a good opportunity to not squat on the language name in a blog that is effectively dedicated to a single implementation of the Raku Programming Language.
So see you next week for directions to the new blog!
Ben Davies saw his work of the past months on improving hostname resolution (which is part of his IP6NS grant) finally merged.
Stefan Seifert fixed an issue in the profiler that could cause a segfault.
Daniel Green fixed an issue in the profiler with exclusive times potentially underflowing.
Jan-Olof Hendig fixed an initialization issue caught with valgrind.
Paweł Murias fixed various issues on the Javascript backend, in preparation for the final report on his TPF grant.
Elizabeth Mattijsen made all of the Buf.write- methods return the invocant, for easier chaining. And added support for calling these methods on Buf type objects, for easier creation of Bufs from scratch.
Jan-Olof Hendig fixed a few C-scoping issues in MoarVM and a potential segfault when closing STDIN.
Stefan Seifert fixed a potential memory corruption issue with spesh plugins and Garbage Collection.
Timo Paulssen fixed an interface problem between the profiler and Comma.
Elizabeth Mattijsen made it possible to subclass the Proxy, allowing one to more easily create custom containers. And she sped up a lot of Cool methods.
It appears there will be no Rakudo 2019.09 compiler release. Which makes sense since it’s almost October. Check out the Perl 6 Weekly next week for more news about this and many other things!
Damian Conway had an itch, and he scratched it in “Itch.scratch()“. An extensive treatise on how to extend the Perl 6 Programming Language, giving the for loop an else block to be executed only if no iterations were done in the for loop. In less than 25 lines! (Reddit comments).
Video Tutorials
Yanzhan Yang has posted a number of Perl 6 introductory videos on YouTube, maybe the first of many to come:
A quiet week yet again, while work has started on finalizing the next Rakudo compiler release. Hope to be able to report on that next week. Until then, program safely and have fun!
One of the most exciting parts of blogging about and hacking on perl6* is that there’s a community out there and there’s (always) more than one way to do it!
For Physics::Measure I have been working on a design for a ‘nano-slang’ that can provide a shortcut for the usual new Class declaration… quite long winded for my target audience of physics undergraduates and high school students.
#Instances the usual way
my Unit $u .=new(name => ‘m’, unitsof => ‘Distance’); #Unit m
my Distance $a .=new(value => 10, units => $u); #Distance 10 m
So, duly inspired by Rakudo perl6 going atomic applying unicode to some threading constructs, I started with the notion of the ‘libra’ operator as shorthand to declare and load Measure instances.
#Introducing the libra operator as shorthand to declare and load
my $b ’50 m’; #Distance 50 m
$b ‘3 yards’; #Distance 3 yards
As you can see, the gap created between and ; is a slang zone that can consume strings and numeric literals. Here’s something a bit more elaborate:
#Normalization with the .norm method
my $p ’27 kg m^2 / s^3′; #Power 27 kg m^2 / s^3
$p .= norm; #Power 27 W
A few design ideas drew me in this direction:
Physics::Measure overloads the main arithmetic operators (+,-,*,/,**,sqrt) and I felt that the use of unicode would be a good convention to warn coders that there is something substantive happening to the core raku language.
It is verboten in Rakudo to overload assignment ‘=’ … quite right too! But I wanted something like a combined variable declaration and (default) assignment for the congenitally lazy coder (and the libra kinda looks like an equals sign).
Resistance is one of the Units implemented by Physics::Measure – and the Ohm symbol (Ω) looks a lot like .
There are a couple hundred definition lines in Physics::Measure covering metric SI units, cgi, feet and inches, US and imperial, plus some fun ones, so it wasn’t practical to inject infix/postfix equivalents 1:1 without overwhelming the raku module namespace.
I was born in October
# Resistance
[‘Ω’, ‘Ohm:s’,], ’kg m^2 / A^2 s^3′,
[‘kilohm:s’,], ’kilo Ohm’,
[‘megohm:s’,], ’mega Ohm’,
HOWEVER!!
Others have proposed a much more direct approach to generate and combine Measure objects – by the use of a simple postfix syntax – thank you!
Something like:
say 500g; # –> Weight.new(grams => 500, prefix => “”)
say 2kg;# –> Weight.new(grams => 2000, prefix => “kg”)
Watch this space! Or even better zef Physics::Measure and give it a try…
Timo Paulssen published a report about his work on the MoarVM Heap Snapshot Profiler of the past months. This is an excellent tool that is already being used by MoarVM / Rakudo core developers to track resource usage in Perl 6 programs. Glad to see such progress!
Jan-Olof Hendig found a missing break statement in MoarVM with Coverity.
JJ Merelo supervised / helped a lot with upgrading the documentation processing and generation system, mainly around this issue.
Vadim Belman continued his extensive work on re-imagining the MoarVM / NQP / Rakudo build system, making sure the different backends share as much as possible. He also improved the handling of multiple language versions in Perl 6.
Timo Paulssen introduced “VM Event Subscription” that allows monitoring of VM events as they happen such as a GC, and Spesh Status. And did quite some work on the profiler and the heap-snapshotter.
Stefan Seifert (again) fixed a number of possible memory corruptions in MoarVM.
Jonathan Worthington fixed and tweaked spesh logging to regain some lost execution speed. He also made sure that when running with MVM_HASH_RANDOMIZE=0 that really no hash randomization will occur. And fixed issues with exception propagation in threaded programs and external programs segfaulting.
Daniel Green fixed a memory leak when reading invalid UTF8 data and silenced some clang warnings.
Bart Wiegmans optimized de-optimizations and fixed some failing qregex testcases.
Christian Bartolomäus improved some error messages on the JVM backend.
Paweł Murias fixed some bugs on the Javascript backend.
Peter du Marchie van Voorthuysen fixed various issues with vows on Promises.
A week with a lot less happening for yours truly. Meanwhile, developments on the build system are simply staggering. As well as stability improvements. Looking forward to reporting more of these next week!
It has been a while since the last progress report, hasn't it?
Over the last few months I've been focusing on the MoarVM Heap Snapshot Profiler. The new format that I explained in the last post, "Intermediate Progress Report: Heap Snapshots", is available in the master branch of MoarVM, and it has learned a few new tricks, too.
The first thing I usually did when opening a Heap Snapshot in the heapanalyzer (the older command-line based one) was to select a Snapshot, ask for the summary, and then for the top objects by size, top objects by count, top frames by size, and/or top frames by count to see if anything immediately catches my eye. In order to make more sense of the results, I would repeat those commands for one or more other Snapshots.
Realizing that the most common use case should be simple to achieve, I first implemented a command summary all and later a command summary every 10 to get the heapanalyzer to give the summaries of multiple Snapshots at once, and to be able to get summaries (relatively) quickly even if there's multiple hundreds of snapshots in one file.
Sadly, this still requires the parser to go through the entire file to do the counting and adding up. That's obviously not optimal, even though this is an Embarrassingly Parallel task, and it can use every CPU core in the machine you have, it's still a whole lot of work just for the summary.
For this reason I decided to shift the responsibility for this task to MoarVM itself, to be done while the snapshot is taken. In order to record everything that goes into the Snapshot, MoarVM already differentiates between Object, Type Object, STable, and Frame, and it stores all references anyway. I figured it shouldn't have a performance impact to just add up the numbers and make them available in the file.
The result is that the summary table as shown further above is available only milliseconds after loading the heap snapshot file, rather than after an explicit request and sometimes a lengthy wait period.
The next step was to see if top objects by size and friends could be made faster in a similar way.
I decided that adding an optional "statistics collection" feature inside of MoarVM's heap snapshot profiler would be worthwhile. If it turns out that the performance impact of summing up sizes and counts on a per-type and per-frame basis makes capturing a snapshot too slow, it could be turned off.
Frontend work
> snapshot 50
Loading that snapshot. Carry on...
> top frames by size
Wait a moment, while I finish loading the snapshot...
Name Total Bytes
==================================== =============
finish_code_object (World.nqp:2532) 201,960 bytes
moarop_mapper (QAST.nqp:1764) 136,512 bytes
!protoregex (QRegex.nqp:1625) 71,760 bytes
new_type (Metamodel.nqp:1345) 40,704 bytes
statement (Perl6-Grammar.nqp:951) 35,640 bytes
termish (Perl6-Grammar.nqp:3641) 34,720 bytes
<anon> (Perl6-BOOTSTRAP.c.nqp:1382) 29,960 bytes
EXPR (Perl6-Grammar.nqp:3677) 27,200 bytes
<mainline> (Perl6-BOOTSTRAP.c.nqp:1) 26,496 bytes
<mainline> (NQPCORE.setting:1) 25,896 bytes
EXPR (NQPHLL.nqp:1186) 25,760 bytes
<anon> (<null>:1) 25,272 bytes
declarator (Perl6-Grammar.nqp:2189) 23,520 bytes
<anon> (<null>:1) 22,464 bytes
<anon> (<null>:1) 22,464 bytes
Showing the top objects or frame for a single snapshot is fairly straight-forward in the commandline based UI, but how would you display how a type or frame develops its value across many snapshots?
Instead of figuring out the best way to display this data in the commandline, I switched focus to the Moarperf Web Frontend. The most obvious way to display data like this is a Line Graph, I believe. So that's what we have now!
And of course you also get to see the data from each snapshot's Summary in graph format:
And now for the reason behind this blog post's Title.
Progress Notifications
Using Jonathan's module Concurrent::Progress (with a slight modification) I sprinkled the code to parse a snapshot with matching calls of .increment-target and .increment. The resulting progress reports (throttled to at most one per second) are then forwarded to the browser via the WebSocket connection that already delivers many other bits of data.
The result can be seen in this tiny screencast:
The recording is rather choppy because the heapanalyzer code was using every last drop of performance out of my CPU while it was trying to capture my screen.
There's obviously a lot still missing from the heap snapshot analyzer frontend GUI, but I feel like this is a good start, and even provides useful features already. The graphs for the summary data are nicer to read than the table in the commandline UI, and it's only in this UI that you can get a graphical representation of the "highscore" lists.
I think a lot of the remaining features will already be useful after just the initial pieces are in place, so a little work should go a long way.
Bits and Bobs
I didn't spend the whole time between the last progress report and now to work directly on the features shown here. Apart from Life Intervening™, I worked on fixing many frustrating bugs related to both of the profilers in MoarVM. I added a small subsystem I call VMEvents that allows user code to be notified when GC runs happen and other interesting bits from inside MoarVM itself. And of course I've been assisting other developers by answering questions and looking over their contributions. And of course there's the occasional video-game-development related experiment, for example with the GTK Live Coding Tool.
Finally, here's a nice little screencap of that same tool displaying a hilbert curve:
That's already everything I have for this time. A lot has (had to) happen behind the scenes to get to this point, but now there was finally something to look at (and touch, if you grab the source code and go through the needlessly complicated build process yourself).
Thank you for reading and I hope to see you in the next one!
- Timo
Published by Jo Christian Oterhals
on 2019-07-31T13:23:17
Perl 6 small stuff #21: it’s a date! …or: learn from an overly complex solution to a simple task
This week’s Perl Weekly Challenge (#19) has two tasks. The first is to find all months with five weekends in the years from 1900 through 2019. The second is to program an implementation of word wrap using the greedy algorithm.
Both are pretty straight-forward tasks, and the solutions to them can (and should) be as well. This time, however, I’m also going to do the opposite and incrementally turn the easy solution into an unnecessarily complex one. Because in this particular case we can learn more by doing things the unnecessarily hard way. So this post will take a look at Dates and date manipulation in Perl 6, using PWC #19 task 1 as an example:
Write a script to display months from the year 1900 to 2019 where you find 5 weekends i.e. 5 Friday, 5 Saturday and 5 Sunday.
Let’s start by finding five-weekend months the easy way:
#!/usr/bin/env perl6
say join "\n", grep *.day-of-week == 5, map { Date.new: |$_, 1 }, do 1900..2019 X 1,3,5,7,8,10,12;
The algorithm for figuring this out is simple. Given the prerequisite that there must be five occurrences of not only Saturday and Sunday but also Friday, you A) *must* have 31 days to cram five weekends into. And when you know that you’ll also see that B) the last day of the month MUST be a Sunday and C) the first day of the month MUST be a Friday (you don’t have to check for both; if A is true and B is true, C is automatically true too).
The code above implements B and employs a few tricks. You read it from right to left (unless you write it from left to right, like this… say do 1900..2019 X 1,3,5,7,8,10,12 ==> map { Date.new: |$_, 1 } ==> grep *.day-of-week == 5 ==> join “\n”; )
Using the X operator I create a cross product of all the years in the range 1900–2019 and the months 1, 3, 5, 7, 8, 10, 12 (31-day months). In return I get a sequence containing all year-month pairs of the period.
The map function iterates through the Seq. There it instantiates a Date object. A little song and dance is necessary: As Date.new takes three unnamed integer parameters, year, month and day, I have to do something to what I have — a Pair with year and month. I therefore use the | operator to “explode” the pair into two integer parameters for year and month.
You can always use this for calling a sub routine with fixed parameters, using an array with parameter values rather than having separate variables for each parameter. The code below exemplifies usage:
my @list = 1, 2, 3;
sub explode-parameters($one, $two, $three) { …do something… }
Back to the business at hand — the .grep filters out the months where the 1st is a Friday, and those are our 5 weekend months. So the output of the one-liner above looks something like this:
... 1997-08-01 1998-05-01 1999-01-01 ...
This is a solution as good as any, and if a solution was all we wanted, we could have stopped here. But using this task as an example I want to explore ways to utilise the Date class. Example: The one-liner above does the job, but strictly speaking it doesn’t output the months but the first day of those months. Correcting this is easy, because the Date class supports something called formatters and use the sprintf syntax. To do this you utilise the named parameter “formatter” when instantiating the object.
say join "\n", grep *.day-of-week == 5, map { Date.new: |$_, 1, formatter => { sprintf "%04d/%02d", .year, .month } }, do 1900..2019 X 1,3,5,7,8,10,12;
Every time a routine pulls a stringified version of the date, the formatter object is invoked. In our case the output has been changed to…
Now to the overly complex solution. This is the unthinking programmer’s solution, as we don’t suppose anything. The program isn’t told that 5 weekend months only can occur on 31 day months. It doesn’t know that the 1st of such months must be a Friday. All it knows is that if the last day of the month is not Sunday, it figures out the date of the last Sunday (this is not very relevant when counting three-day weekends, but could be if you want to find Saturday+Sunday weekends, or only Sundays).
#!/usr/bin/env perl6
my $format-it = sub ($self) { sprintf "%04d month %02d", .year, .month given $self; }
sub MAIN(Int :$from-year = 1900, Int :$to-year where * > $from-year = 2019, Int :$weekend-length where * ~~ 1..3 = 3) { my $date-loop = Date.new($from-year, 1, 1, formatter => $format-it); while ($date-loop.year <= $to-year) { my $date = $date-loop.later(day => $date-loop.days-in-month); $date = $date.truncated-to('week').pred if $date.day-of-week != 7; my @weekend = do for 0..^$weekend-length -> $w { $date.earlier(day => $w).weekday-of-month; }; say $date-loop if ([+] @weekend) / @weekend == 5; $date-loop = $date-loop.later(:1month); } }
This code can solve the task both for three day weekends, but also for weekends consisting of Saturday + Sunday, as well as only Sundays. You control that with the command line parameter weekend-length=[1..3].
This code finds the last Sunday of each month and counts whether it has occured five times that month. It does the same for Saturday (if weekend-length=2) and Friday (if weekend-length=3). Like this:
my @weekend = do for 0..^$weekend-length -> $w { $date.earlier(day => $w).weekday-of-month; };
The code then calculcates the average weekday-of-month for these three days like this:
say $date-loop if ([+] @weekend) / @weekend == 5;
This line uses the reduction operator [+] on the @weekend list to find the sum of all elements. That sum is divided by the number of elements. If the result is 5, then you have a five day weekend.
As for fun stuff to do with the Date object:
.later(day|month|year => Int) — adds the given number of time units to the current date. There’s also an earlier method for subtracting.
.days-in-months — tells you how many days there are in the current month of the Date object. The value may be 31, 30, 29 (february, leap year) or 28 (february).
.truncated-to(week|month|day|year) — rolls the date back to the first day of the week, month, day or year.
.weekday-of-month — figures out what day of week the current date is and calculates how many of that day there has been so far in that month.
Apart from this you’ll see that I added the formatter in a different way this time. This is probably cleaner looking and easier to maintain.
In the end this post maybe isn’t about dates and date manipulation at all, but rather is a call for all of us to use the documentation even more. It’s often I think that Perl 6 should have a function for x, y or z — .weekday-of-month is one such example — and the documentation tells me that it actually does!
It’s very easy to pick up Perl 6 and program it as you would have programmed Perl 5 or other languages you know well. But the documentation has lots of info of things you didn’t have before and that will make programming easier and more fun when you’ve learnt about them.
I guess you don’t need and excuse to delve into the docs, but if you do the Perl Weekly Challenge is an excellent excuse for spending time in the docs!
Published by Jo Christian Oterhals
on 2019-08-01T07:34:44
You have several options besides do. You could use parenthesises instead, like this:
say join "\n", grep *.day-of-week == 5, map { Date.new: |$_, 1 }, (1900..2019 X 1,3,5,7,8,10,12);
In this case I just thought the code looked better without parenthesises, so I chose to use do instead. The docs has a couple of sentences about this option here.
BTW, I chose the form Date.new: blah — the colon variant — instead of Date.new(blah) also because I wanted to avoid parenthesises. This freedom to chose is the essence of Perl’s credo “There’s more than one way to do it” I guess.
Rant: This freedom of choice is the best thing with Perl 6, but it has a downside too. Code can be harder to understand if you’re not familiar with the variants another programmer has made. This is a part of the Perls’s reputation of being write-only languages.
It won’t happen, but personally I’d like to see someone analyse real-world Perl 6 code (public repositories on Github for instance) and figure what forms are used most. The analysis could have been used to clean house — figuring out what forms to keep and which should be removed. Perl 6 would become a smaller — and arguably easier — language while staying just as powerful as it is today.
Observant readers looking at the dateline on the left will have noticed a gap of nearly two years between my last blog and today. As a fascinated programming hobbyist, I have only limited time in the day to devote to coding – and a startup venture put the brakes on anything but the day job for a while.
In the meantime, I have had to choose between coding and blogging – and while the ideas have been flowing ahead, I am quite keen on the idea that ‘actions speak louder than words’.
The actions have covered:
learning baby perl6 (highly recommend Think Perl6 by Laurent Rosenfeld)
learning perl6 OO
learning perl6 Grammars
learning about Unit systems
writing Physics::Measure v0.0.1
figuring out the point of it all
writing and publishing Math::Polygons
asking a bunch of questions under the perl6 tag on Stack Overflow
learning a bit more about perl6
writing and publishing Physics::Measure v0.0.2 (~2k lines)
So, today, I am delighted to announce the alpha release of Physics::Measure (try zef install https://github.com/p6steve/[email protected] for now – hopefully zef install Physics::Measure within a day or two).
I have a logfile of the following form that I would like to parse.
[ 2016.03.09 20:40:28 ] (MessageType) Some message text that depends on the <MessageType>
Since the form of the text depends on the message type I need a rule to identify the message type and a rule to parse the message body itself. To aid my struggle through the Grammar in question I use Grammar::Tracer from jnthn’s Grammer::Debugger module. It’s a fine module that will tell until where a match was OK and at which point the Grammar gave up parsing. In the case of a successful match it shows part of the substring that was successfully parsed. If parsing a rule or token fails it will tell you but wont show the offending string. The whole purpose of Grammar wrangling is to identify the bits that wont match and change the Grammar until they go away. Not showing the offending string is not overly helpful.
But fear not as Grammars are classes and as such can have methods. Let’s define one and add it to a chain of options.
method parse-fail {
# self is a subclass of Grammar
say self.postmatch.substr(0, 100);
exit 0;
}
rule body-line { '[' <timestamp> ']' [ <body-notify> | <body-question> | <body-info> | <body-warning> || <parse-fail> ] }
So when none of the known message types match the Grammar stops and shows the string that still needs to be handled. With that I could parse all 8768 files until I got them all covered. Also this is much faster then running with Grammar::Tracer.
It seems to be very useful to have folk implement a language they would like to use to implement that language.
There’s been a few weeks where I haven’t followed Perl Weekly Challenge, but luckily I found some time for Challenge #11 this week. Inititally both exercises looked they quite a bit of coding. But after a while it became apparaent that both could be solved with relatively simple one (or two) liners.
But I also found that one-liners aren’t always very robust when it comes to error handling, so that gave me an opportunity to explore Perl6’s built-in type (and content) checking of command line input parameters, as well as how to use Perl 6’s built-in methods of generating friendly and readable Usage output.
But before all of that we’ll start with the second excersise.
Write a script to create an Indentity Matrix for the given size. For example, if the size is 4, then create Identity Matrix 4x4.
I admit that I really don’t know what an Identity Matrix is or why it’s important, but the challenge provided a link to the Wikipedia page so that I’d at least understand how one would look. The look, simply told, is a square matrix where the main diagonal is filled with 1’s and the rest is filled with 0's.
There are Perl 6 modules that are helpful, but I though I’d write a solution from the bottom up. Given the many cool features of Perl 6, generating an identity matrix is one line of code:
my &idm = -> $s { gather for ^$s -> $y { take map { Int($_ == $y) }, ^$s } };
Call the function like this: my @identity-matrix = idm(4); — and an array containing the identity matrix for your given size is returned.
Note the & sigil. This is the sigil for Code, just as @ is for arrays and % is for hashes. This code utilises Perl 6’s anonymous functions. I could have written this as an ordinary sub, but in the context of a one-liner I think this functional variant looks better: -> $s defines an anonymous function with one parameter ($s).
The interesting thing with the code sigil is that it’s optional when you call it. So you can call it both like my @id-mat = idm(4) and my @id-mat = &idm(4) . I think the former looks better.
As you’ll have seen many times before on this blog I once again use the gather… take… functionality, which gives me the ability to easily build an array (I could have used push on an array too, but again — I feel gather-take looks better and is more concise).
^$s generates a list from 0 to the requested size, and the for loops trough that list. For row number one I loop through ^$s once more, and use map populate the identity matrix. If row number and column number is the same, the value is set to 1. If not 0. Instead of using an if sentence to check and set this, I combine everything into one with Int($_ == $y). $_ == $y returns True or False; Int() casts this into 1 or 0.
The end result is an array of arrays containing your identity matrix.
I could have stopped there, but since this is a Perl Weekly Challenge, one I thought I should prove that the function works. I do that by adding a second line of code that prints creates and prints a matrix:
.join(' ').say for $idm(4);
The output is:
1 0 0 0 0 1 0 0 0 0 1 0 0 0 0 1
That’s it. Short and simple.
Having solved this second excersise, I returned to the first. That one looked like it’d need more code.
Write a script that computes the equal point in the Fahrenheit and Celsius scales, knowing that the freezing point of water is 32 °F and 0 °C, and that the boiling point of water is 212 °F and 100 °C.
I guess you could explain equal points in many ways. I think of it as the point where two lines cross. In this case: What’s the point where the celcius # and the fahrenheit # is identical and indicates the same absolute temperature?
Before we begin: The answer is 40 below zero (-40). But how do we compute this easily? We have to use the formula for conversion. If x represents the temperature in celcius, fahrenheit — y — is calculated like this:
y = 9/5*x + 32
When you look at it like this you’ll see that this looks exactly like a formula for a line, the y = mx + b we know from school — where m is the gradient or the slope of the line and b the y intercept, i.e. the value of y when x = 0. In this case m = 9/5 and b = 32.
The celcius scale is easy — y is equal to x always, so the formula looks like this:
y = x
To calculate the equal point we have to find the value of x where y is the same for both formulas. We can build the equation for solving this by taking the right sides of the equation for fahrenheit and celcius lines and put them together like this…
(the right hand side of y = x) = (the right hand side of y = 9/5 * x + 32) x = 9/5 * x + 32
…and solve the equation. Since 9/5 is 1.8, let’s use that instead. So…
x = 1.8x + 32 x-1.8x = 32 # or 1.0x-1.8x = 32 -0.8x = 32 # -0,8x because 1.0–1,8 x = 32 / -0.8 x = -40
Now we know enough to not only solve a comparision between celcius and fahrenheit, but between celcius and whatever scale you’d throw at it. We have to implement x = 32 / -0.8 in some way. Or, figure out a way to make -0.8 generic, i.e. put in any scale and get this to work.
Here’s the key: the grade 9/5 or 1.8 is really just the fahrenheit degrees for the boiling point of water subtracted by the same for the freezing point, and then divided by the same for celcius (which happens to be 100). I.e. (212-32)/100 = 9/5 = 1.8. So if we have the freezing and the boiling point of any temperature scale other than celcius itself, the grade for that scale is:
(freezing point-boiling point) / 100
Now, if we peek at on of the steps of the calculation above, 1.0x-1.8x = 32, we see that vi have to subtract the grade from 1.0, so that we end up with the final
y-intercept (or freezing point) / 1.0-grade
…or more verbosely
freezing point / (1.0-((freezing point-boiling point) / 100))
If we assume that we have a variable $a for the freezing point and a variable $b for the boiling point, a Perl 6 one-liner would look like this:
Substitute 32, 212 with anything you’d like. Let’s say you invented a temperature scale where 20 is freezing and 80 is boiling, swap (32,212) with (20,80). The result? 50.
Quick and dirty — but effective. However the code above will crash under certain circumstances, and in many scenarios calculate that the equal point is at a temperature that simply does not exists (temperatures below absolute zero). So I thought that I’d implement this with a little error handling as well. And I think that version showcase some Perl 6 greatness that I haven’t touched upon earlier:
multi MAIN( #= Compares the C. scale to any scale you dream up Real $f, #= freezing point in custom scale Real $b #= boiling point in custom scale ) { say "There is no equal point for this scale." and exit if $b - $f == 100; my $equal-point = $f / (1 - (($b - $f) / 100)); say "The calculated equal point is only theoretical as it is below absolute zero." if $equal-point < -273.15; say "Equal point: $equal-point"; }
multi MAIN( #= Compares the C. scale to a named temperature scale Str $scale where { $_ ~~ m:i/^(fahrenheit|kelvin|rankin)$/ } #= Name of scale (Fahrenheit, Kelvin or Rankin) ) { given $scale.fc { when "fahrenheit" { MAIN( 32 , 212 ); } when "kelvin" { MAIN(273.15, 373.15); } when "rankin" { MAIN(491.67, 671.67); } } }
The use of the MAIN subroutines gives us some command line parsing for free. Not only that, but Perl 6 does type checking for us — in the case of the first multi MAIN, it ensures that the two parameters it receives actually are numbers. The second multi MAIN looks only for a single string, and only three strings in particular: fahrenheit, kelvin and rankin.
The program won’t run if you try to start it with any other parameters. Instead it spits out a Usage text like this:
Usage: eqpoint.p6 <f> <b> -- Compares the Celcius scale to whatever scale you dream up eqpoint.p6 <scale> -- Compares the Celcius scale to a given temperature scale
<f> freezing point in this scale <b> boiling point in this scale <scale> Name of scale (Fahrenheit, Kelvin or Rankin)
You’ll see that this custom usage text is caused by my use of #= in the code above. #= behind the sub routine name is the generic description of the usage. Behind parameter variables they describe that particular parameter.
Now, when calculating the grade I do some value checking to avoid certain scenarios. First I check whether the difference between boiling and freezing in the user’s custom sale equals 100. If it does, that scale runs in parallell with the celcius scale. That means that the lines never cross and there is no equal point. If that’s the case I print out a message about that and exit. (The equation would have cause a divide by zero error had I move on)
In addition I check whether the calulcated equation point is below -273.15 celcius. If it is, the point is below absolute zero and would never ever occur in this universe. In those scenarios I point out that the equal point is only theoretical.
I think it’s extra fun when the challenges, well, challenge me to explore little-used areas of Perl 6. I got to do lots of that today. If you haven’t tried your hand at the Perl Weekly Challenge yet, I highly recommend it.
Jnthn answered the question why $*IN.lines blocks in a react block. What isn’t explained is what whenever actually does before it starts blocking.
react {
whenever $*IN.lines { .say }
}
Looking at the syntax of a whenever block, we see that whenever takes a variable immediatly followed by a block. The only place where a structure like that can be defined is Grammar.nqp.
Here the grammar just checks a few things without actually generating any code. So we head to Actions.nqp.
method statement_control:sym<whenever>($/) {
my $xblock := $<xblock>.ast;
make QAST::Op.new(
:op<call>, :name<&WHENEVER>, :node($/),
$xblock[0], block_closure($xblock[1])
);
}
The whenever block is converted to a call to sub WHENEVER which we find in Supply.pm6.
sub WHENEVER(Supply() $supply, &block) {
There we go. A whenever block takes its first argument of any type and calles .Supply on it, as long as Any is a parent of that type. In the case of $*IN that type will typically be whatever IO::Handle.lines returns.
Seq.new(self!LINES-ITERATOR($close))
To turn a Seq into a SupplyAny.Supply calls self.list.Supply. Nowhere in this fairly long chain of method lookups (this can’t be fast) are there any threads to be found. If we want to fix this we need to sneak a Channel into $*IN.lines which does exactly that.
$*IN.^can('lines')[1].wrap(my method {
my $channel = Channel.new;
start {
for callsame() {
last if $channel.closed;
$channel.send($_)
}
LEAVE $channel.close unless $channel.closed;
}
$channel
});
use Concurrent::Channelify;
react {
whenever signal(SIGINT) {
say "Got signal";
exit;
}
whenever $*IN.lines⇒ {
say "got line";
}
}
We already use to indicate atomic operations. Maybe using prefix:<∥> to indicate concurrency makes sense. Anyway, we went lucky once again that Rakudo is implemented (mostly) in Perl 6 so we can find out where we need to poke it whenever we want to change it.
This is slightly tangential to the Rakudo Perl 6 Performance Analysis Tooling grant from The Perl Foundation, but it does interact closely with it, so this is more or less a progress report.
The other day, Jonathan Worthington of Edument approached me with a little paid Job: Profiling very big codebases can be tedious, especially if you're interested in only one specific fraction of the program. That's why I got the assignment to make the profiler "configurable". On top of that, the Comma IDE will get native access to this functionality.
The actual request was just to allow specifying whether individual routines should be included in the profile via a configuration file. That would have been possible with just a simple text file that contains one filename/line number/routine name per line. However, I have been wanting something in MoarVM that allows much more involved configuration for many different parts of the VM, not just the profiler.
Obligatory cat photo.
That's why I quickly developed a small and simple "domain-specific language" for this and similar purposes.
The language had a few requirements:
It should not be possible to write a program that loops infinitely
It should have access to MoarVM internal data that is normally not accessible
It shouldn't allow changing internal data so that things could break
It should allow reasonably complex calculations and comparisons
There's also some things that aren't necessary:
It doesn't need to be very pleasant to write
It doesn't have to be succinct
It doesn't have to have every feature under the sun
While thinking about what exactly I should build – before I eventually settled on building a "programming language" for this task – I bounced back and forth between the simplest thing that could possibly work (for example, a text file with a list of file/line/name entries) and the most powerful thing that I can implement in a sensible timeframe (for example, allowing a full NQP script). A very important realization was that as long as I require the first line to identify what "version" of configuration program it is, I could completely throw away the current design and put something else instead, if the need ever arises. That allowed me to actually commit to a design that looked at least somewhat promising. And so I got started on what I call confprog.
Here's an example program. It doesn't do very much, but shows what it's about in general:
The entry decides which stage of profiling this program is applied to. In this case, the profiler_static means we're seeing a routine for the first time, before it is actually entered. That's why only the information every individual invocation of the frame in question shares is available via the variable sf, which stands for Static Frame. The Static Frame also allows access to the Compilation Unit (cu) that it was compiled into, which lets us find the filename.
The first line that actually does something assigns a value to the special variable log. This will output the name of the routine the program was invoked for.
The next line will turn on profiling only if the name of the routine is "calculate-strawberries". The line after that will also turn on profiling if the filename the routine is from is "CustomCode.pm6".
Apart from profiler_static, there are a couple more entry points available.
profiler_static, which runs the first time any routine is encountered, and stores the decision until the profile run is finished.
profiler_dynamic will be evaluated every time a routine gets entered that would potentially be profiled otherwise (for example because a profiler_static program turned it on, or because there is no profiler_static program, which means that every routine is eligible for profiling).
Anything about the current execution context is available here, including the name of the current routine, the count and types of arguments passed, the routine that called this routine, and so on.
Information that seems less immediately useful like how many times has the GC run yet? or how many threads are there right now? is also available for very special cases.
heapsnapshot will be run every time a GC run happens. It lets the user decide whether a GC run should result in a heap snapshot being taken or not, based on whether the run has been a minor or major collection, and arbitrary other factors, including time since start of the program, or time since the last heap snapshot was taken.
spesh and jit I'm not entirely sure about yet. I want to allow turning the specializer or the jit off completely for certain routines, but I also want to offer control over individual optimizations in spesh (turning off inlining for one particular routine, preventing scalar replacement but only for Rat types, ...) and behaviour of the jit (use the template jit for everything except one specific piece of one particular routine, ...).
How exactly this should look in the program, I do not know yet. It's also probably not going to be very interesting for end-users, but finding bugs in spesh and the JIT could be easier if you can pinpoint (even automatically!) the exact place something goes wrong down to "which optimization at what position makes the difference".
The syntax is still subject to change, especially before the whole thing is actually in a released version of MoarVM.
There is a whole lot of other things I could imagine being of interest in the near or far future. One place I'm taking inspiration from is where "extended Berkeley Packet Filter" (eBPF for short) programs are being used in the linux kernel and related pieces of software:
Oversimplifying a bit, BPF was originally meant for tcpdump so that the kernel doesn't have to copy all data over to the userspace process so that the decision what is interesting or not can be made. Instead, the kernel receives a little piece of code in the special BPF language (or bytecode) and can calculate the decision before having to copy anything.
eBPF programs can now also be used as a complex ruleset for sandboxing processes (with "seccomp"), to decide how network packets are to be routed between sockets (that's probably for Software Defined Networks?), what operations a process may perform on a particular device, whether a trace point in the kernel or a user-space program should fire, and so on.
So what's the status of confprog? I've written a parser and compiler that feeds confprog "bytecode" (which is mostly the same as regular moarvm bytecode) to MoarVM. There's also a preliminary validator that ensures the program won't do anything weird, or crash, when run. It is much too lenient at the moment, though. Then there's an interpreter that actually runs the code. It can already take an initial value for the "decision output value" (great name, isn't it) and it will return whatever value the confprog has set when it runs. The heap snapshot profiler is currently the only part of MoarVM that will actually try to run a confprog, and it uses the value to decide whether to take a snapshot or not.
Next up on the list of things to work on:
There's barely any operators supported, like for comparing numbers, or arithmetic.
There is not yet anything for floating point numbers, but I would like to offer support for that, since it's how we usually represent time.
Values can not yet be persisted between runs, so you can't, for example, store the time of the last run, or how often a given decision was taken, etc.
Printing to a log by assigning to a variable named "log" looks very odd, so the language will want to have syntax for function calls.
Random decision making would be nice, for stochastic things.
Apart from improvements to the confprog programming language, the integration with MoarVM lacks almost everything, most importantly installing a confprog for the profiler to decide whether a frame should be profiled (which was the original purpose of this assignment).
After that, and after building a bit of GUI for Comma, the regular grant work can resume: Flame graphs are still not visible on the call graph explorer page, and heap snapshots can't be loaded into moarperf yet, either.
Thanks for sticking with me through this perhaps a little dry and technical post. I hope the next one will have a little more excitement! And if there's interest (which you can signal by sending me a message on irc, or posting on reddit, or reaching me via twitter @loltimo, on the Perl 6 discord server etc) I can also write a post on how exactly the compiler was made, and how you can build your own compiler with Perl 6 code. Until then, you can find Andrew Shitov's presentations about making tiny languages in Perl 6 on youtube.
I hope you have a wonderful day; see you in the next one!
- Timo
PS: I would like to give a special shout-out to Nadim Khemir for the wonderful Data::Dump::Tree module which made it much easier to see what my parser was doing. Here's some example output from another simple confprog program:
Notice how it shows the type of most things, like name.Str, as well as cross-references for things that appear multiple times, like the CPTypeString. Particularly useful is giving your own classes methods that specify exactly how they should be displayed by DDT. Love It!
As announced earlier I went to write a module to make Nil.list behave a little better. There are basicly two way Nil could be turned into a list. One should warn the same way as Nil.Str does and the other should end the program loudly. Doing both at the same time however does not make sense.
There are a few ways this could be done. One is augmenting Nil with a list method and have this method check a dynamic variable to pick the desired behaviour. That would be slow and might hurt if Nil.list is called in a loop. The other is by using a custom sub EXPORT and a given switch.
# lib/NoNilList/Warning.pm6
use NoNilList 'Warning';
# lib/NoNilList/Fatal.pm6
use NoNilList 'Fatal';
# lib/NoNilList.pm6
sub EXPORT($_?) {
given $_ {
when 'Warning' {
# augment Nil with a warning .list
}
when 'Fatal' {
# augment Nil with a failing .list
}
default {
die 'Please use NoNilList::Warning or NoNilList::Fatal.';
}
}
%() # Rakudo complains without this
}
Now use NoNilList; will yield a compile time error with a friedly hint how to avoid it.
I left the augmenting part out because it does not work. I thought I stepped on #2779 again but was corrected that this is acually a different bug. Jnthn++ fixed part of that new bug (Yes, Perl 6 bugs are so advanced they come in multiple parts.) and proposed the use of the MOP instead. That resulted in #2897. The tricky bit is that I have to delay augmentation of Nil to after the check on $_ because augment is a declarator and as such executed at compile time — in a module that can be months before the program starts to run. Both an augment in an EVAL string and the MOP route would lead there. I wanted to use this module as my debut on 6PAN. That will have to wait for another time.
If you find a bug please file it. It will lead to interresting discoveries for sure.
Published by Jo Christian Oterhals
on 2019-05-10T12:05:02
After a week’s hiatus I’ve returned to the Perl Weekly Challenge. This is the seventh challenge so far. As before there are two excercises. The first one is to calculate all niven numbers between 0 and 50. Niven numbers are integers that are divisible by the sum of its digits. I.e. 47 is a niven number if 47 / (4 + 7) is an integer without remainder (it’s not, as the result is 4.2727; 48 is, as 48/(4+8) is 4).
These kinds of operations are called modulo operations or mods. Most programming languages use the operator % for this, so that you can check for this by testing whether 47 % (4 + 7) == 0. But Perl6 has an additional operator, a shorthand for the former, and is written as 47 %% (4 + 7). %% returns True or False and lets you ignore the == 0 part.
To find the individual digits of a number we use the .comb method. That returns a list with the individual digits (I’ve used .comb on earlier challenges as well, but then to divide strings into their individual characters). On the combed list we use something called a reduction operator, specifically [+]. What this does is that it loops through the list and adds all of the numbers within. Where you once would have written something like my $result = 0; $result += $_ for 1, 2, 3, 4, 5; say $result; you can now do the same with a simple say [+] 1, 2, 3, 4, 5 . There are lots of these reduction operators. They save time, so look into them.
Knowing these two things it becomes apparaent that calculating Niven numbers with Perl 6 can be done with a simple and reasonably readable one-liner.
.say if $_ %% [+] .comb for 0..50;
That’s it.
The second excercise is a little harder: “A word ladder is a sequence of words [w0, w1, …, wn] such that each word wi in the sequence is obtained by changing a single character in the word wi-1. All words in the ladder must be valid English words. Given two input words and a file that contains an ordered word list, implement a routine (e.g., find_shortest_ladder(word1, word2, wordlist)) that finds the shortest ladder between the two input words.”
There are a lot of givens and requirements, so you should look at the web page of challenge 7 to see them all. But the most important of them is that the two words you want to create a ladder between, has to be of the same length. And the list of words you use for this must also only contain words of that length. Additionally you’re also only supposed to compare lowercase words.
Knowing all that, here’s my attempt at excercise 2.
File: challenge2.p6
#!/usr/bin/env perl6
# This script needs a plain-text dictionary of words to work. # MacOS has one built-in here. May have different location # or not be present at all on other systems.
constant $DICTIONARY = "/usr/share/dict/words";
subset Str-lc of Str where * ~~ /^<lower>+$/;
sub MAIN(Str-lc $word1, Str-lc $word2 where { sprintf("%s", $word1).chars == sprintf("%s", $word2).chars }, Bool :$list-all = False) {
my @ladders = []; my @words = $DICTIONARY.IO.lines.grep( { $_ eq $_.lc && $_.chars == $word1.chars } );
gen-word-ladder($word1, [ $word1 ], {});
for @ladders.sort({ $^a.elems cmp $^b.elems }).kv -> $i, @ladder { say @ladder.join(" -> "); exit if !$list-all && $i == 0; }
sub gen-word-ladder(Str $word, @ladder is copy, %seen is copy) { for $word.comb.kv -> $index, $character { my $r3 = $word.substr(0, $index) ~ "." ~ $word.substr(*-($word.chars - ($index + 1))); my $r5 = ( '.' x $index ) ~ $word2.substr($index,1) ~ ( '.' x $word.chars - 1 - $index) ; for @words.grep( / <$r3> /).grep( / <$r5> / ) -> $x { if ! %seen{$x}.defined && $x ne $word { %seen{$x} = True; @ladder.push($x); @ladders.push(@ladder) if $x eq $word2; gen-word-ladder($x, @ladder, %seen); } } } }
}
Basically, this is a solution using recursion. I won’t go into detail about what the code does, I’m just going to comment on Perl6-isms that may be interesting for newcomers.
The main sub routine here is MAIN. Subroutines named MAIN is a special kind of subroutine. This routine is automatically called when a script is invoked. Also, the parameters you define in the sub routine are parameters that will be required on the command line when you run the script. This definition says that two words are required. Additionally there’s an optional flag that can be used (--list-all) if you also want to the rest of the possible word ladders too and not only the shortest. (If you want several different signatures to be possible, use the multi keyword and define multiple MAIN’s)
Note the use of subset and where here. They define constraints. Here I define a subset of Str, called Str-lc, that requires the string to be lowercase. You can check for all sorts of things. That all this is built-in saves you from writing lots of error checking code.
The nice thing about this, if you use the subset types in the context of MAIN, is that Perl6 will check your CLI parameters for you. If you’ve made any errors it will exit and print an automatically generated Usage message. Again, less typing and more done for you automatically!
Now, in the declaration of MAIN itself I also use the whereclause. Use that to define constraints that are ad-hoc and not repeated or constraints that can’t be defined as subsets due to a dynamic nature.
You should note my use of sprintf here. You may find it redundant that I use sprintf to convert a string to a string here. Due to an error in the Rakudo Star 2019.03.01 distribution of Perl 6, the comparison $word1.chars == $word2.chars doesn't work. It throws this error:
Cannot call method 'chars' on a null object
The only time I don’t get this error is when the two strings actually are of equal length. A workaround is sprintf. If you run a string through sprintf(“%s”), a brand new Str object is returned. And on them we can run instance methods. So then I also get checks for length equality built into the MAIN declaration. As the above mentioned error isn’t present in earlier versions of Rakudo, it’ll be fixed again in the future so that this absurd conversion round-robin is not necessary.
In any case, with very little code you get lots of fancy functionality. So when you run the script with correct parameters, all runs as expected…
# ./challenge2.p6 --list-all horse lousy horse -> house -> louse -> lousy horse -> house -> louse -> housy -> lousy horse -> house -> horsy -> housy -> lousy
…while a reasonably understandable usage report is printed if not…
Do you see the .kv used in a couple of the for loops? Normally .kv returns the keys and values of a hash interleaved [1]. But used in the context of a list/an array, the key is an integer representing the index while the value is, as you’d expect, the value. It’s as if you’ve got an automatic $counter. As it is. Now, it’s just a little thing, but I’ve programmed enough $counter++ lines in my life to appreciate this.
Another handy shorthand is this: $DICTIONARY.IO — it’s the easiest way to open a file for reading. If your string correspons to a file, adding .IO behind the Str variable name converts it to an IO::Path object which adds lots of stuff, including the .lines method that opens the file and returns the contents line by line [2].
Lastly I’d like to point you to / <$r3> / and / <$r5> /. The < and > surronding a variable tells Perl6 that you want whatever’s in them to be interpolated as a regexp. So that…
my $r5 = "al.ha";
say "alpha" ~~ /$r5/; # Output: Nil say "alpha" ~~ /<$r5>/; # Output: ｢alpha｣ (match)
This means you can build regexpes dynamically and very simply. It’s a nice function, although — maybe — a function with EVAL like downsides.
NOTES
[1] Over on Reddit /u/liztormato pointed out that my explanation about what the .kv method of an Array/List does, was wrong. I.e. I understood its effects, but misunderstood what it actually did to achieve those effects. So this explanation is Liz’s not mine.
[2] This too has been clarified by Liz on Reddit. I have to say that this is the nice thing about blogging about Perl 6: The community is very welcoming, so it’s no stress to lay out all my misunderstandings. I’ve really learned so much by writing these articles. Thanks to everyone who has pointed out errors and misunderstandings and maybe even suggested improvements.
I'm here to share a thing I'm working on, and chew gum; and I'm all out of gum. The purpose of this post is both to break a disagreeable silence that has befallen this blog, and to be able to geek out about a niche topic here in writing, partially sparing friends and family.
I'm currently refactoring 007's type system in a branch. Basically, ever since 007 was created, a type in 007-land has corresponded to a class/role in Perl 6, the host system that's implementing 007.
Here's what the implementation of Int looks like currently:
class Val::Int does Val {
has Int $.value;
method truthy {
?$.value;
}
}
And here's what it looks like as I'm doing the refactor:
constant TYPE is export = {};
BEGIN {
# ...
TYPE<Int> = make-type "Int", :backed;
# ...
}
So, instead of corresponding to types on the host level, all the 007 types are about to correspond to values. The former implementation was the one that felt obvious at the time (four-plus years ago), but it's become blindingly, painstakingly obvious that it really needs to be the latter.
Here's why: as soon as you want to implement class declarations in 007, in the former model you also need to bend over backwards and come up with an entirely new type in the host system. The Perl 6 code to do that looks like this:
Which is... even someone as EVAL-positive as I wishes for a less clunky solution.
In the new model, a new class comes down to calling make-type and dropping the result in that TYPE hash. (Wait. Or not even dropping it in the TYPE hash. That hash is only for things used by 007 itself, not for user types.)
This is a refactor I've tried once before, back in 2017, but I failed back then because the code got too complicated and ran too slow. This time around I have a much better feeling.
By the way, there's also an is-type subroutine, and similarly a make-int and an is-int subroutine, and so on for every registered type. I figure why not wrap those simple things up in very short function names. So far that turns out to have been a very good decision. "Fold the language of your domain model into your code", and so on.
This is one of the things I'm doing better this time around; last time one of the problems was that each line I touched got longer and messier because there were more layers of indirection to dig through. Concerns were scattered all over the place. This time, it feels like the codebase is getting simpler thanks to those named subroutines. Maybe it can be likened to putting all your database-specific code in one place.
I sometimes get slight vertigo due to the bootstrapping aspects of this type system. One example: Object is an instance of Type, but the base class of Type is Object — a circularity. But, it turns out, if you have absolute power over the object graph, you can always bend things to your will:
BEGIN {
TYPE<Type> = _007::Value.new(:type(__ITSELF__), slots => { name => "Type" });
TYPE<Object> = make-type "Object";
{
# Bootstrap: now that we have Object, let's make it the base of Type and Object
TYPE<Type>.slots<base> = TYPE<Object>;
TYPE<Object>.slots<base> = TYPE<Object>;
}
# ...
}
I'm slightly reminded of a thing Gilad Bracha wrote once (which I can't find the exact quote for, unfortunately): that if mutual dependencies and recursive definitions are something that stump you, what you need is a healthy dose of letrec. It's twisty, yes, but it's basically a solved problem.
Like last time, I'm tackling the big task in small steps, one type at a time. I feel I've learned this from Martin Fowler's concept of asset capture. The idea is to end up back with a running system with passing tests often. I do this by replacing one old thing at a time by a new thing. Sounds obvious, but I'm actually not sure I would have been sensible enough on my own to tackle it this way, had I not known about asset capture.
One drawback is that you're sort of running the old system and the new system in parallel, as the old one is being phased out. Only once the whole thing has been asset-captured can complexity associated with the old system be completely removed.
It's a pleasant way to work. To me it's been at least a partial answer to the problem of the big rewrite. If we're free to refactor the insides, we can successively arrive at a point where the new better thing has completely replaced the old thing. The way there is allowed to be a little bit more complex (on the inside) than either endpoint. Importantly, you keep a running system throughout.
I don't have a concluding thought, except to say that I just managed to asset-capture arrays. Which is harder than it sounds, because arrays are everywhere in the compiler and the runtime.
In a for loop Nil is turned into a List with one Element that happens to be Any. This really buged me so I went to find out why. As it turns out the culprit is the very definition of Nil is Cool. To be able to turn any single value into a ListCool implements method list(). Which takes a single values and turns that value into a List with that one value. Nil indicates the absense of a value and turning it into a value doesn’t make sense. Luckily we can change that.
use MONKEY-TYPING;
augment class Nil {
method list() {
note 'Trying to turn Nil into a list.';
note Backtrace.new.list.tail.Str;
Empty
}
}
Nil.HOW.compose(Nil);
sub niler() { Nil }
for niler() { say 'oi‽' }
We can’t just warn because that would show the wrong point in the stack trace. So we note (which also goes to $*ERR) and pull the last value out of the Backtrace.
Interestingly Failure throws both in .list and in .iterator. Nil implements push, append, unshift and prepend by immediatly die-ing. Adding more to nothing is deadly but turning nothing first into something vaguely undefined and then allowing to add more stuff to it is inconsistent at best. What leads me to believe that Nil.list as it is specced today is just an oversight.
At least I can now write a simple module to protect my code from surprising Nils.
Jo Christian Oterhals asked for a parallel solution for challenge 2. I believe he had problems to find one himself, because his code sports quite a few for loops. By changing those to method call chains, we can use .hyper to run at lease some code concurrently.
use v6.d;
constant CORES = $*KERNEL.cpu-cores;
# workaround for #1210
sub prefix:<[max]>(%h){ %h.sort(-*.value).first }
my %dict = "/usr/share/dict/words".IO.lines.map({ .lc => True });
my %seen;
%dict.keys».&{ %seen{.comb.sort.join}++; };
with [max] %seen {
say .value, .key.comb.hyper(:batch(1024), :degree(CORES)).permutations».join.grep({ %dict{$_}:exists }).Str
}
My approach is a little different then Jo’s. I don’t try to keep all combinations around but just count the anagrams for each entry in the word list. Then I find a word with the most anagrams (there are more candidates with the same amount that I skip) and reconstruct the anagrams for that word.
The only operation where any form of computation happens is the generation of permutations. Anything else is just too memory bound to get a boost by spinning up threads. With the .hyper-call the program is a tiny wee bit faster then with just one thread on my Threadripper box. A system with slow cores/smaller caches should benefit a little more. The main issue is that the entire word list fits into the 3rd level cache. With a bigger dataset a fast system might benefit as well.
In many cases multi-core systems are fairy dust, which makes the wallets of chip makers sparkle. Wrangling Hashs seams to be one of those.
this time I don't have something finished to show off, but nevertheless I can tell you all about the work I've recently started.
The very first post on this blog already had a little bit about the heap snapshot profiler. It was about introducing a binary format to the heap snapshot profiler so that snapshots can be written out immediately after they were taken and by moarvm itself rather than by NQP code. This also made it possible to load snapshot data faster: the files contain an index that allows a big chunk of data to be split up exactly in the middle and then read and decoded by two threads in parallel.
The new format also resulted in much smaller output files, and of course reduced memory usage of turning on the profiler while running perl6 code. However, since it still captures one heap snapshot every time the GC runs, every ordinary program that runs for longer than a few minutes will accumulate quite an amount of data. Heap snapshot files (which are actually collections of multiple snapshots) can very easily outgrow a gigabyte. There would have to be another change.
The new format already contained the simplest thing that you could call compression. Instead of simply writing every record to the file as it comes, records that have smaller numbers would be stored with a shorter representation. This saved a lot of space already, but not nearly as much as off-the-shelf compression techniques would.
There had to be another way to get compression than just coming up with my own compression scheme! Well, obviously I could have just implemented something that already exists. However, at the time I was discouraged by the specific requirements of the heap snapshot analyzer - the tool that reads the files to let the user interactively explore the data within it:
Reading the file has to be fast
Any snapshot in the file must be accessible equally quickly and in any order
In order to get decent speed out of the reading process, it had to be multithreadable.
Normally, compression formats are not built to support easily seeking to any given spot in the uncompressed data. There was of course the possibility to compress each snapshot individually, but that would mean a whole snapshot could either only be read in with a single thread, or the compression would have to go through the whole blob and when the first splittable piece was decompressed, a thread could go off and parse it. I'm not entirely sure why I didn't go with that, perhaps I just didn't think of it back then. After all, it's already been more than a year, and my brain compresses data by forgetting stuff.
Anyway, recently I decided I'd try a regular compression format for the new moarvm heap snapshot file format. There's already a Perl 6 module named Compress::Zlib, which I first wanted to use. Writing the data out from moarvm was trivial once I linked it to zlib. Just replace fopen with gzopen, fwrite with gzwrite, fclose with gzclose and you're almost done! The compression ratio wasn't too shabby either.
When I mentioned this in the #moarvm channel on IRC, I was asked why I use zlib instead of zstd. After all, zstd usually (or always?) outperforms zlib in both compression/decompression speed and output size. The only answer I had for that was that I hadn't used the zstd C library yet, and there was not yet a Perl 6 module for it.
Figuring out zstd didn't go as smoothly as zlib, not by a long shot. But first I'd like to explain how I solved the issue of reading the file with multiple threads.
Restructuring the data
In the current binary format, there are areas for different kinds of objects that occur once per snapshot. Those are collectables and references. On top of that there are objects that are shared across snapshots: Strings that are referenced from all the other kinds of objects (for example filenames, or descriptions for references like "hashtable entry"), static frames (made up of a name like "push", an ID, a filename, and a line number), and types (made up of a repr name like VMArray, P6opaque, or CStruct and a type name like BagHash or Regex).
That resulted in a file format that has one object after the other in the given area. The heap snapshot analyzer itself then goes through these areas and splits the individual values apart, then shoves them off into a queue for another thread to actually store. Storage inside the heap analyzer consists of one array for each part of these objects. For example, there is one array of integers for all the descriptions and one array of integers for all the target collectables. The main benefit of that is not having to go through all the objects when the garbage collector runs.
The new format on the other hand puts every value of each attribute in one long string before continuing with the next attribute.
Here's how the data for static frames was laid out in the file in the previous version:
"sframes", count, name 1, uuid 1, file 1, line 1, name 2, uuid 2, file 2, line 2, name 3, uuid 3, file 3, line 3, … index data
The count at the beginning tells us how many entries we should be expecting. For collectables, types, reprs, and static frames this gives us the exact number of bytes to look for, too, since every entry has the same size. References on the other hand have a simple "compression" applied to them, which doesn't allow us to just figure out the total size by knowing the count. To offset this, the total size lives at the end in a place that can easily be found by the parser. Strings are also variable in length, but there's only a few hundred of them usually. References take up the most space in total; having four to five times as many references as there are collectables is pretty normal.
Here's how the same static frame data is laid out in the upcoming format:
"names", length, zstd(name 1, name 2, name 3, …), index data, "uuids", length, zstd(uuid 1, uuid 2, uuid 3, …), index data, "files", length, zstd(file 1, file 2, file 3, …), index data, "lines", length, zstd(line 1, line 2, line 3, …), index data
As you can see, the values for each attribute now live in the same space. Each attribute blob is compressed individually, each has a little piece of index data at the end and a length field at the start. The length field is actually supposed to hold the total size of the compressed blob, but if the heap snapshot is being output to a destination that doesn't support seeking (moving back in the file and overwriting an earlier piece of data) we'll just leave it zeroed out and rely on zstd's format being able to tell when one blob ends.
There are some benefits to this approach that I'd like to point out:
The compression algorithm can perhaps figure patterns out better, because more "random" data isn't interspersed to break up patterns. I have not actually tested this yet, though!
The individual attributes can be uncompressed and stored away in parallel, since there's one count shared between many attributes. Compare this to decoding 500 thousand collectables on one thread and 1.7 million on another.
Need to add another attribute for some reason? You can just put it next to the other blobs, give it a new name that the older versions don't know and they'll just skip it!
The last point, in fact, will let me put some extra fun into the files. First of all, I currently start the files with a "plain text comment" that explains what kind of file it is and how it's structured internally. That way, if someone stumbles upon this file in fifty years, they can get started finding out the contents right away!
On a perhaps more serious note, I'll put in summaries of each snapshot that MoarVM itself can just already generate while it's writing out the snapshot itself. Not only things like "total number of collectables", but also "top 25 types by size, top 25 types by count". That will actually make the heapanalyzer not need to touch the actual data until the user is interested in more specific data, like a "top 100" or "give me a thousand objects of type 'Hash'".
On top of that, why not allow the user to "edit" heap snapshots, put some comments in before sharing it to others, or maybe "bookmarking" specific objects?
All of these things will be easier to do with the new format - that's the hope at least!
Did the compression work?
I didn't actually have the patience to exhaustively measure all the details, but here's a rough ratio for comparison: One dataset I've got results in a 1.1 gigabytes big file with the current binary format, a 147 megabytes big file when using gzip and a 99 megabytes big file using zstd (at the maximum "regular" compression level - I haven't checked yet if the cpu usage isn't prohibitive for this, though).
It seems like this is a viable way forward! Allowing capture to run 10x as long is a nice thing for sure.
What comes next?
The profiler view itself in Moarperf isn't done yet, of course. I may not put in more than the simplest of features if I start on the web frontend for the heap analyzer itself.
On the other hand, there's another task that's been waiting: Packaging moarperf for simpler usage. Recently we got support for a relocatable perl6 binary merged into master. That should make it possible to create an AppImage of moarperf. A docker container should also be relatively easy to build.
We'll see what my actual next steps will be - or will have been I suppose - when I post the next update!
Hi hackers! Today First of all, I want to thank everybody who gave such useful feedback on my last post. For instance, I found out that the similarity between the expression JIT IR and the Testarossa Trees IR is quite remarkable, and that they have a fix for the problem that is quite different from what I had in mind.
Today I want to write something about register allocation, however. Register allocation is probably not my favorite problem, on account of being both messy and thankless. It is a messy problem because - aside from being NP-hard to solve optimally - hardware instruction sets and software ABI's introduce all sorts of annoying constraints. And it is a thankless problem because the case in which a good register allocator is useful - for instance, when there's lots of intermediate values used over a long stretch of code - are fairly rare. Much more common are the cases in which either there are trivially sufficient registers, or ABI constraints force a spill to memory anyway (e.g. when calling a function, almost all registers can be overwritten).
So, on account of this being not my favorite problem, and also because I promised to implement optimizations in the register allocator, I've been researching if there is a way to do better. And what better place to look than one of the fastest dynamic language implementations arround, LuaJIT? So that's what I did, and this post is about what I learned from that.
Truth be told, LuaJIT is not at all a learners' codebase (and I don't think it's author would claim this). It uses a rather terse style of C and lots and lots of preprocessor macros. I had somewhat gotten used to the style from hacking dynasm though, so that wasn't so bad. What was more surprising is that some of the steps in code generation that are distinct and separate in the MoarVM JIT - instruction selection, register allocation and emitting bytecode - were all blended together in LuaJIT. Over multiple backend architectures, too. And what's more - all these steps were done in reverse order - from the end of the program (trace) to the beginning. Now that's interesting...
I have no intention of combining all phases of code generation like LuaJIT has. But processing the IR in reverse seems to have some interesting properties. To understand why that is, I'll first have to explain how linear scan allocation currently works in MoarVM, and is most commonly described:
First, the live ranges of program values are computed. Like the name indicates, these represent the range of the program code in which a value is both defined and may be used. Note that for the purpose of register allocation, the notion of a value shifts somewhat. In the expression DAG IR, a value is the result of a single computation. But for the purposes of register allocation, a value includes all its copies, as well as values computed from different conditional branches. This is necessary because when we actually start allocating registers, we need to know when a value is no longer in use (so we can reuse the register) and how long a value will remain in use -
Because a value may be computed from distinct conditional branches, it is necessary to compute the holesin the live ranges. Holes exists because if a value is defined in both sides of a conditional branch, the range will cover both the earlier (in code order) branch and the later branch - but from the start of the later branch to its definition that value doesn't actually exist. We need this information to prevent the register allocator from trying to spill-and-load a nonexistent value, for instance.
Only then can we allocate and assign the actual registers to instructions. Because we might have to spillvalues to memory, and because values now can have multiple definitions, this is a somewhat subtle problem. Also, we'll have to resolve all architecture specific register requirements in this step.
In the MoarVM register allocator, there's a fourth step and a fifth step. The fourth step exists to ensure that instructions conform to x86 two-operand form (Rather than return the result of an instruction in a third register, x86 reuses one of the input registers as the output register. E.g. all operators are of the form a = op(a, b) rather than a = op(b, c). This saves on instruction encoding space). The fifth step inserts instructions that are introduced by the third step; this is done so that each instruction has a fixed address in the stream while the stream is being processed.
Altogether this is quite a bit of complexity and work, even for what is arguably the simplest correct global register allocation algorithm. So when I started thinking of the reverse linear scan algorithm employed by LuaJIT, the advantages became clear:
In LuaJIT, the IR maintains its SSA form - there is only a single definition of a value. This means that when iterating in reverse order, computing the live range becomes trivial. When we first encounter a use of a value, then by definition that is the last use. And when we encounter a definition, that is the only and single definition, and we can release the register. So there's no need to compute the live range in advance of allocation.
Furthermore, rather than merging the values of multiple branches into the same live range, each value on either side becomes an individual live range. As a result, the live range of a value never has a hole, further simplifying code.
LuaJIT uses register hints to indicate which registers could best be picked for a specific value. This is often determined by how a value is used (e.g., the divisor in a div instruction must be in the rcx register). If the preferred register can't be allocated, the register allocator inserts code to move it to the right place where needed. Having hints can be expected to greatly reduce the need for such code.
There are downsides as well, of course. Not knowing exactly how long a value will be live while processing it may cause the algorithm to make worse choices in which values to spill. But I don't think that's really a great concern, since figuring out the best possible value is practically impossible anyway, and the most commonly cited heuristic - evict the value that is live furthest in the future, because this will release a register over a longer range of code, reducing the chance that we'll need to evict again - is still available. (After all, we do always know the last use, even if we don't necessarily know the first definition).
Altogether, I'm quite excited about this algorithm; I think it will be a real simplification over the current implementation. Whether that will work out remains to be seen of course. I'll let you know!
Hi hackers! Today I want to write about optimizing IR in the MoarVM JIT, and also a little bit about IR design itself.
One of the (major) design goals for the expression JIT was to have the ability to optimize code over the boundaries of individual MoarVM instructions. To enable this, the expression JIT first expands each VM instruction into a graph of lower-level operators. Optimization then means pattern-matching those graphs and replacing them with more efficient expressions.
As a running example, consider the idx operator. This operator takes two inputs (base and element) and a constant parameter scale and computes base+element*scale. This represents one of the operands of an 'indexed load' instruction on x86, typically used to process arrays. Such instructions allow one instruction to be used for what would otherwise be two operations (computing an address and loading a value). However, if the element of the idx operator is a constant, we can replace it instead with the addr instruction, which just adds a constant to a pointer. This is an improvement over idx because we no longer need to load the value of element into a register. This saves both an instruction and valuable register space.
Unfortunately this optimization introduces a bug. (Or, depending on your point of view, brings an existing bug out into the open). The expression JIT code generation process selects instructions for subtrees (tile) of the graph in a bottom-up fashion. These instructions represent the value computed or work performed by that subgraph. (For instance, a tree like (load (addr ? 8) 8) becomes mov ?, qword [?+8]; the question marks are filled in during register allocation). Because an instruction is always represents a tree, and because the graph is an arbitrary directed acyclic graph, the code generator projects that graph as a tree by visiting each operator node only once. So each value is computed once, and that computed value is reused by all later references. It is worth going into some detail into why the expression graph is not a tree. Aside from transformations that might be introduced by optimizations (e.g. common subexpression elimination), a template may introduce a value that has multiple references via the let: pseudo-operator. See for instance the following (simplified) template:
In this case, both references to $foo point directly to the sameload operator. Thus, the graph is not a tree. Another case in which this occurs is during linking of templates into the graph. The output of an instruction is used, if possible, directly as the input for another instruction. (This is the primary way that the expression JIT can get rid of unnecessary memory operations). But there can be multiple instructions that use a value, in which case an operator can have multiple references. Finally, instruction operands are inserted by the compiler and these can have multiple references as well.
If each operator is visited only once during code generation, then this may introduce a problem when combined with another feature - conditional expressions. For instance, if two branches of a conditional expression both refer to the same value (represented by name $foo) then the code generator will only emit code to compute its value when it encounters the first reference. When the code generator encounters $foo for the second time in the other branch, no code will be emitted. This means that in the second branch, $foo will effectively have no defined value (because the code in the first branch is never executed), and wrong values or memory corruption is then the predictable result.
This bug has always existed for as long as the expression JIT has been under development, and in the past the solution has been not to write templates which have this problem. This is made a little easier by a feature the let: operator, in that it inserts a do operator which orders the values that are declared to be computed before the code that references them. So that this is in fact non-buggy: (let: (($foo (load (local))) # code to compute $foo is emitted here (if (...) (add $foo (const 1)) # $foo is just a reference (sub $foo (const 2)) # and here as well
The DO node is inserted for the LET operator. It ensures that the value of the LOAD node is computed before the reference in either branch
Alternatively, if a value $foo is used in the condition of the if operator, you can also be sure that it is available in both sides of the condition.
All these methods rely on the programmer being able to predict when a value will be first referenced and hence evaluated. An optimizer breaks this by design. This means that if I want the JIT optimizer to be successful, my options are:
Fix the optimizer so as to not remove references that are critical for the correctness of the program
Modify the input tree so that such references are either copied or moved forward
Fix the code generator to emit code for a value, if it determines that an earlier reference is not available from the current block.
In other words, I first need to decide where this bug really belongs - in the optimizer, the code generator, or even the IR structure itself. The weakness of the expression IR is that expressions don't really impose a particular order. (This is unlike the spesh IR, which is instruction-based, and in which every instruction has a 'previous' and 'next' pointer). Thus, there really isn't a 'first' reference to a value, before the code generator introduces the concept. This is property is in fact quite handy for optimization (for instance, we can evaluate operands in whatever order is best, rather than being fixed by the input order) - so I'd really like to preserve it. But it also means that the property we're interested in - a value is computed before it is used in, in all possible code flow paths - isn't really expressible by the IR. And there is no obvious local invariant that can be maintained to ensure that this bug does not happen, so any correctness check may have to check the entire graph, which is quite impractical.
I hope this post explains why this is such a tricky problem! I have some ideas for how to get out of this, but I'll reserve those for a later post, since this one has gotten quite long enough. Until next time!
MoarVM just recently got a nice optimization merged into the master branch. It's called "partial escape analysis" and comes with a specific optimization named "scalar replacement". In simple terms, it allows objects whose lifetime can be proven to end very soon to be created on the stack instead of in the garbage collector managed heap. More than just that, the "partial" aspect of the escape analysis even allows that when the object can escape out of our grasp, but will not always do so.
Allocation on the stack is much cheaper than allocation in the garbage collected heap, because freeing data off of the stack is as easy as leaving the function behind.
Like a big portion of optimizations in MoarVM's run-time optimizer/specializer ("spesh"), this analysis and optimization usually relies on some prior inlining of code. That's where the pun in the post's title comes from.
This is a progress report for the profiler front-end, though. So the question I'd like to answer in this post is how the programmer would check if these optimizations are being utilized in their own code.
A Full Overview
The first thing the user gets to see in the profiler's frontend is a big overview page summarizing lots of results from all aspects of the program's run. Thread creations, garbage collector runs, frame allocations made unnecessary by inlining, etc.
That page just got a new entry on it, that sums up all allocations and also shows how many allocations have been made unnecessary by scalar replacement, along with a percentage. Here's an example screenshot:
That on its own is really only interesting if you've run a program twice and maybe changed the code in between to compare how allocation/optimization behavior changed in between. However, there's also more detail to be found on the Allocations page:
Per-Type and Per-Routine Allocations/Replacements
The "Allocations" tab already gave details on which types were allocated most, and which routines did most of the allocating for any given type. Now there is an additional column that gives the number of scalar replaced objects as well:
Here's a screenshot showing the details of the Num type expanded to display the routines:
Spesh Performance Overview
One major part of Spesh is its "speculative" optimizations. They have the benefit of allowing optimizations even when something can't be proven. When some assumption is broken, a deoptimization happens, which is effectively a jump from inside an optimized bit of code back to the same position in the unoptimized code. There's also "on-stack-replacement", which is effectively a jump from inside unoptimized code to the same position in optimized code. The details are, of course, more complicated than that.
How can you find out which routines in your program (in libraries you're calling, or the "core setting" of all builtin classes and routines) are affected by deoptimization or by OSR? There is now an extra tab in the profiler frontend that gives you the numbers:
This page also has the first big attempt at putting hopefully helpful text directly next to the data. Below the table there's these sections:
Specializer Performance
MoarVM comes with a dynamic code optimizer called "spesh". It makes your code faster by observing at run time which types are used where, which methods end up being called in certain situations where there are multiple potential candidates, and so on. This is called specialization, because it creates versions of the code that take shortcuts based on assumptions it made from the observed data.
Deoptimization
Assumptions, however, are there to be broken. Sometimes the optimized and specialized code finds that an assumption no longer holds. Parts of the specialized code that detect this are called "guards". When a guard detects a mismatch, the running code has to be switched from the optimized code back to the unoptimized code. This is called a "deoptimization", or "deopt" for short.
Deopts are a natural part of a program's life, and at low numbers they usually aren't a problem. For example, code that reads data from a file would read from a buffer most of the time, but at some point the buffer would be exhausted and new data would have to be fetched from the filesystem. This could mean a deopt.
If, however, the profiler points out a large amount of deopts, there could be an optimization opportunity.
On-Stack Replacement (OSR)
Regular optimization activates when a function is entered, but programs often have loops that run for a long time until the containing function is entered again.
On-Stack Replacement is used to handle cases like this. Every round of the loop in the unoptimized code will check if an optimized version can be entered. This has the additional effect that a deoptimization in such code can quickly lead back into optimized code.
Situations like these can cause high numbers of deopts along with high numbers of OSRs.
Just as I experienced with this very blog post you're reading right now, knowing how to start may just be the hardest part of a great many things.
When you've just run your profile – which I'm planning to make easier in the future as well – and you're looking at the overview page, you're really getting an overview of the very broadest kind. How long did the program run? Did it spend a lot of time running the GC? Are many routines not optimized or jitted?
However, profiling is usually used to find the critical little piece of code that takes an extraordinary amount of time. This is where your optimization attempts should usually begin.
Until now, the overview page didn't mention any piece of code by name at all. This changed when I brought in a flame graph (well, icicle graph in this case).
Here's two screenshots to give you an idea what I'm talking about:
Clicking on a box in the flame graph will expand the node to be 100% the width of the full graph so you can inspect the descendant nodes more easily. Here I've selected the step function:
Selecting one of the nodes gives the name of the routine, the source code line and links to the node in the call graph explorer (the rightwards arrow button) and to the routine in the routine list. On top of that, the filename and line number below the routine name are clickable if they are from the core setting, and they take you right to the implementation file on github.
The next step is, of course, to also put flame graphs into the call graph explorer. I'm not entirely sure how to make it behave when navigating upwards to the root or downwards to the selected children, i.e. whether it should keep nodes towards the root or how many.
That's already everything for today. I couldn't invest terribly much time into moarperf this and last month, but I'll continue working :)
Hi all. I usually write somewhat long-winded posts, but today I'm going to try and make an exception. Today I want to talk about the expression template language used to map the high-level MoarVM instructions to low-level constructs that the JIT compiler can easily work with:
This 'language' was designed back in 2015 subject to three constraints:
It should make it easy to develop 'templates' for MoarVM instructions, so we can map the ~800 or so different instructions supported by the interpreter to something the JIT compiler can work with.
It should be simple to process and analyze; specifically, it should be suitable as input to the instruction selection process (the tiler).
It should be simple to implement, both from the frontend (meaning the perl program that compiles a template file to a C header) and the backend (meaning the C code that combines templates into the IR that is compiled).
Recently I've been working on adding support for floating point operations, and this means working on the type system of the expression language. Because floating point instructions operate on a distinct set of registers from integer instructions, a floating point operator is not interchangeable with an integer (or pointer) operator.
This type system is enforced in two ways. First, by the template compiler, which attempts to check if you've used all operands correctly. This operates during development, which is convenient. Second, by instruction selection, as there will simply not be any instructions available that have the wrong combinations of types. Unfortunately, that happens at runtime, and such errors so annoying to debug that it motivated the development of the first type checker.
However, this presents two problems. One of the advantages of the expression IR is that, by virtue of having a small number of operators, it is fairly easy to analyze. Having a distinct set of operators for each type would undo that. But more importantly, there are several MoarVM instructions that are generic, i.e. that operate on integer, floating point, and pointer values. (For example, the set, getlex and bindlex instructions are generic in this way). This makes it impossible to know whether its values will be integers, pointers, or floats.
This is no problem for the interpreter since it can treat values as bags-of-bits (i.e., it can simply copy the union MVMRegister type that holds all values of all supported types). But the expression JIT works differently - it assumes that it can place any value in a register, and that it can reorder and potentially skip storing them to memory. (This saves work when the value would soon be overwritten anyway). So we need to know what register class that is, and we need to have the correct operators to manipulate a value in the right register class.
To summarize, the problem is:
We need to know the type of each value, both to ensure we use the correct instructions and the right registers.
There are several cases in which we don't really know (for the template) what type each value has.
There are two ways around this, and I chose to use both. First, we know as a fact for each local or lexical value in a MoarVM frame (subroutine) what type it should have. So even a generic operator like set can be resolved to a specific type at runtime, at which point we can select the correct operators. Second, we can introduce generic operators of our own. This is possible so long as we can select the correct instruction for an operator based on the types of the operands.
For instance, the store operator takes two operands, an address and a value. Depending on the type of the value (reg or num), we can always select the correct instruction (mov or movsd). It is however not possible to select different instructions for theload operator based on the type required, because instruction selection works from the bottom up. So we need a special load_num operator, but a store_num operator is not necessary.And this is true for a lot more operators than I had initially thought. For instance, aside from the (naturally generic) do and if operators, all arithmetic operators and comparison operators are generic in this way.
I realize that, despite my best efforts, this has become a rather long-winded post anyway.....
Anyway. For the next week, I'll be taking a slight detour, and I aim to generalize the two-operand form conversion that is necessary on x86. I'll try to write a blog about it as well, and maybe it'll be short and to the point. See you later!
Hi everybody! I recently read jnthns Perl 6 new years resolutions post, and I realized that this was an excellent example to emulate. So here I will attempt to share what I've been doing in 2018 and what I'll be doing in 2019.
In 2018, aside from the usual refactoring, bugfixing and the like:
I added support for the fork() system call in the MoarVM backend.
I removed the 'invokish' control flow mechanism and replaced it with controlled return address modification.
I requested a grant from the Perl Foundation aiming to complete the expression JIT compiler with floating point support, irregular registers and the like.
So 2019 starts with me trying to complete the goals specified in that grant request. I've already partially completed one goal (as explained in the interim report) - ensuring that register encoding works correctly for SSE registers in DynASM. Next up is actually ensuring support for SSE (and floating point) registers in the JIT, which is surprisingly tricky, because it means introducing a type system where there wasn't really one previously. I will have more to report on that in the near future.
After that, the first thing on my list is the support for irregular register requirements in the register allocator, which should open up the possibility of supporting various instructions.
This evening, I enjoyed the New Year’s fireworks display over the beautiful Prague skyline. Well, the bit of it that was between me and the fireworks, anyway. Rather than having its fireworks display at midnight, Prague puts it at 6pm on New Year’s Day. That makes it easy for families to go to, which is rather thoughtful. It’s also, for those of us with plans to dig back into work the following day, a nice end to the festive break.
So, tomorrow I’ll be digging back into work, which of late has involved a lot of Perl 6. Having spent so many years working on Perl 6 compiler and runtime design and implementation, it’s been fun to spend a good amount of 2018 using Perl 6 for commercial projects. I’m hopeful that will continue into 2019. Of course, I’ll be continuing to work on plenty of Perl 6 things that are for the good of all Perl 6 users too. In this post, I’d like to share some of the things I’m hoping to work on or achieve during 2019.
Partial Escape Analysis and related optimizations in MoarVM
The MoarVM specializer learned plenty of new tricks this year, delivering some nice speedups for many Perl 6 programs. Many of my performance improvement hopes for 2019 center around escape analysis and optimizations stemming from it.
The idea is to analyze object allocations, and find pieces of the program where we can fully understand all of the references that exist to the object. The points at which we can cease to do that is where an object escapes. In the best cases, an object never escapes; in other cases, there are a number of reads and writes performed to its attributes up until its escape.
Armed with this, we can perform scalar replacement, which involves placing the attributes of the object into local registers up until the escape point, if any. As well as reducing memory operations, this means we can often prove significantly more program properties, allowing further optimization (such as getting rid of dynamic type checks). In some cases, we might never need to allocate the object at all; this should be a big win for Perl 6, which by its design creates lots of short-lived objects.
There will be various code-generation and static optimizer improvements to be done in Rakudo in support of this work also, which should result in its own set of speedups.
Expect to hear plenty about this in my posts here in the year ahead.
Decreasing startup time and base memory use
The current Rakudo startup time is quite high. I’d really like to see it fall to around half of what it currently is during 2019. I’ve got some concrete ideas on how that can be achieved, including changing the way we store and deserialize NFAs used by the parser, and perhaps also dealing with the way we currently handle method caches to have less startup impact.
Both of these should also decrease the base memory use, which is also a good bit higher than I wish.
Improving compilation times
Some folks – myself included – are developing increasingly large applications in Perl 6. For the current major project I’m working on, runtime performance is not an issue by now, but I certainly feel myself waiting a bit on compiles. Part of it is parse performance, and I’d like to look at that; in doing so, I’d also be able to speed up handling of all Perl 6 grammars.
I suspect there are some good wins to be had elsewhere in the compilation pipeline too, and the startup time improvements described above should also help, especially when we pre-compile deep dependency trees. I’d also like to look into if we can do some speculative parallel compilation.
Research into concurrency safety
In Perl 6.d, we got non-blocking await and react support, which greatly improved the scalability of Perl 6 concurrent and parallel programs. Now many thousands of outstanding tasks can be juggled across just a handful of threads (the exact number chosen according to demand and CPU count).
For Perl 6.e, which is still a good way off, I’d like to having something to offer in terms of making Perl 6 concurrent and parallel programming safer. While we have a number of higher-level constructs that eliminate various ways to make mistakes, it’s still possible to get into trouble and have races when using them.
So, I plan to spend some time this year quietly exploring and prototyping in this space. Obviously, I want something that fits in with the Perl 6 language design, and that catches real and interesting bugs – probably by making things that are liable to occasionally explode in weird ways instead reliably do so in helpful ways, such that they show up reliably in tests.
Get Cro to its 1.0 release
In the 16 months since I revealed it, Cro has become a popular choice for implementing HTTP APIs and web applications in Perl 6. It has also attracted code contributions from a couple of dozen contributors. This year, I aim to see Cro through to its 1.0 release. That will include (to save you following the roadmap link):
Providing flexible HTTP reverse proxying support
Providing a number of robustness patterns (timeout, retry, circuit breaker, and so forth)
Providing Cro support some message queue protocols
Ensuring Cro’s features work on MacOS and Windows
Providing some further help for those building server-side web applications (templating, an easier time setting up common login flows, etc.)
Conducting a security review
Specifying a stability/deprecation policy
Lots of polishing
Comma Community, and lots of improvements and features
I founded Comma IDE in order to bring Perl 6 a powerful Integrated Development Environment. We’ve come a long way since the Minimum Viable Product we shipped back in June to the first subscribers to the Comma Supporter Program. In recent months, I’ve used Comma almost daily on my various Perl 6 projects, and by this point honestly wouldn’t want to be without it. Like Cro, I built Comma because it’s a product I wanted to use, which I think is a good place to be in when building any product.
In a few months time, we expect to start offering Comma Community and Comma Complete. The former will be free of charge, and the latter a commercial offering under a subscribe-for-updates model (just like how the supporter program has worked so far). My own Comma wishlist is lengthy enough to keep us busy for a lot more than the next year, and that’s before considering things Comma users are asking for. Expect plenty of exciting new features, as well as ongoing tweaks to make each release feel that little bit nicer to use.
Speaking, conferences, workshops, etc.
This year will see me giving my first keynote at a European Perl Conference. I’m looking forward to being in Riga again; it’s a lovely city to wander around, and I remember having some pretty nice food there too. The keynote will focus on the concurrent and parallel aspects of Perl 6; thankfully, I’ve still a good six months to figure out exactly what angle I wish to take on that, having spoken on the topic many times before!
I also plan to submit a talk or two for the German Perl Workshop, and will probably find the Swiss Perl Workshop hard to resist attending once more. And, more locally, I’d like to try and track down other Perl folks here in Prague, and see if I can help some kind of Praha.pm to happen again.
I need to keep my travel down to sensible levels, but might be able to fit in the odd other bit of speaking during the year, if it’s not too far away.
Teaching
While I want to spend most of my time building stuff rather than talking about it, I’m up for the occasional bit of teaching. I’m considering pitching a 1-day Perl 6 concurrency workshop to the Riga organizers. Then we’ll see if there’s enough folks interested in taking it. It’ll cost something, but probably much less than any other way of getting a day of teaching from me. :-)
So, down to work!
Well, a good night’s sleep first. :-) But tomorrow, another year of fun begins. I’m looking forward to it, and to working alongside many wonderful folks in the Perl community.
This school semester I took my first proof-based class titled “Intro to Mathematical Proof Workshop”. After having taken other math classes (Calculus, Matrix Algebra, etc.), I felt that I didn’t have that much of a mathematical foundation and up to this point, all I had been doing was purely computational mathematics sprinkled with some proofs here and there. Looking back, I found the class quite enjoyable and learning about different theorems and their proofs, mostly from number theory, has given me a new perspective of mathematics.
“How is this related to Perl 6?”, you might be asking. As I mentioned, most of the proofs that were discussed either in class or left for homework were related to number theory. If there’s one thing Perl 6 and number theory have in common is their accessibility. Similar to how the content of the elementary theory of numbers can be tangible and familiar, Perl 6 can be quite approachable to beginners. In fact, beginners are encouraged to write what’s known as “baby Perl”.
Another thing they seem to share is their vastness. For example, in Perl 6 one can find many operators while in number theory one can find a plethora of different types of numbers from even numbers to cute numbers. For most purposes, these numbers are easy to understand and if one has the definition of a number, then it’s quite easy to check if a given integer follows in that category. For example, a prime number is formally defined as follows:
An integer p > 1 is called a prime number, or simply a prime, if its only positive divisors are 1 and p. Otherwise, the integer p is termed composite.
By using this definition, we can quite simply figure out if a certain number is a prime. For example, among the first ten positive integers, 2, 3, 5 and 7 are primes. This is quite trivial for small numbers but doing it by hand with larger numbers would become tedious in no time. This is where Perl 6 comes into the picture. Perl 6 offers many constructs/features that even if they don’t make the job easy, they certainly simplify it. For instance, with the definition of a prime in mind, we could easily implement an algorithm that tests for primality in Perl 6:
Please bear in mind that this is not about writing performant code. If the code turns down that way, then it would be excellent but it is not the goal. My aim is to showcase the easiness with which a beginner can express mathematical constructs in Perl 6. It’s worth mentioning that Perl 6 already includes the subroutine (or method) is-prime that tests for primality. However, although this is true for prime numbers, it might not be the case for another type of number you might come across such as a factorial, a factorion or even a Catalan number. And in cases like this, Perl 6 will be helpful.
After learning about different types of numbers, I set out to look for
some peculiar numbers and see how I could implement them using Perl 6. In the process, I found this useful website that lists a bunch of numbers, their definitions and some examples. From all of these, I’ve chosen four types of numbers that aren’t stupidly difficult to implement (I still write baby Perl ) while being enough to illustrate some Perl 6 constructs. On the other hand, I’ve avoided those which might be too straightforward.
Let us start with…
Amicable numbers
Amicable numbers are pairs of numbers, also known as friendly numbers, each of whose aliquot parts add to give the other number.
sub aliquot-parts( $number ) {
(^$number).grep: $number %% *;
}
sub infix:<amic>( $m, $n ) {
$m == aliquot-parts($n).sum &&
$n == aliquot-parts($m).sum;
}
say 12 amic 28; # False, 12 and 28 aren't amicables.
say 220 amic 284; # True, 220 and 284 are though.
A number’s aliquot parts are its factors excluding the number itself. To find the aliquot parts of a number, I’ve a created the subroutine aliquot-parts which uses 1..^$number to create a list of numbers from 1 up to $numbers (exclusive). This list is subsequently grepped to find out those numbers in the list that evenly divide $number. In this snippet it’s achieved by using the infix operator %% which returns True if a first operand is divisible by a second operand. Otherwise, it returns False. The second operand stands for any number in the list aforementioned so I’ve used *, which in this context is known as the whatever star and creates a closure over the expression $number %% *. Thus the whole expression in the subroutine is equivalent to (^$number).grep: { $number %% $_ };. At the end, the subroutine returns a list of factors of $number excluding $number itself.
To find out if two numbers are amicable, we could have used just a subroutine. However, Perl 6 allows for the creation of new operators, which are just subroutines with funny names themselves, and I’ve done just that. I created the infix operator (meaning between two operands) amic that returns True if two numbers are amicable. Otherwise, False. As you can see, the syntax to create a new operator is straightforward: the keyword sub, followed by the type of the operator (prefix, infix, postfix, etc.), the name of the operator inside quoting constructs, the expected parameters and a code block.
Factorion
A factorion is a natural number that equals the sum of the factorials of its digits in a given base.
subset Whole of Int where * ≥ 0;
sub postfix:<!>( Whole $N --> Whole ) {
[*] 1..$N;
}
sub is-factorion( Whole $number --> Bool ) {
$number == $number.comb.map({ Int($_)! }).sum
}
say is-factorion(25); # False
say is-factorion(145); # True
Recall that a factorial of a number N, which is usually denoted by N!, is the product 1 x 2 x ... x N. For example, 3! = 1 x 2 x 3 = 6. In the code snippet, I created the postfix operator ! to return the factorial of an integer operand. Thus say 3!; will work just fine in the code snippet and prints 6. How the factorial is calculated is straightforward: The range 1..$N creates a list of numbers from 1 to $N (inclusive) and then I use [...], which is known as the reduction meta-operator, with the operator * to reduce the created list to 1 x 2 x ... $N which effectively gives me the factorial of $N. There are many operators in Perl 6 and the meta-operator [...] can work with many of them.
As for the factorion, I want to know if a number is a factorion so I created a subroutine that takes an integer and returns a Boolean. Perl 6 is gradually typed so it allows to type variables explicitly, specify the return type of a sub, etc. I’ve decided to type the subroutines’ parameters and the subroutine’s return type.
In the section about the amicable numbers, I was quite liberal regarding the subroutines’ arguments. However, here I’ve decided to comply with the definition of a factorial and only allow for whole numbers, hence the definition and use of the Whole type. In Perl 6, the operator subset declares a new type using a base type. However if I hadn’t used the where clause, I’d have ended up with just another name for the Int type which would be redundant. So I used the where clause to constraint the type of any assignment to the desired input. In this case, the assignment to a variable of type Whole.
With the is-factorion sub, I used the method comb to break up $number into its digits and then use map to find their respective factorials and sum them up. The sub returns True if $number is equal to the sum of the factorials of its digits. It returns False otherwise.
Cyclic Numbers
A cyclic number is a number with N digits, which, when multiplied by 1, 2, 3, ..., N produces the same digits in a different order.
sub is-cyclic( Int $n --> Bool ) {
for 1..$n.chars -> $d {
return False if $n.comb.Bag != ($n * $d).comb.Bag;
}
return True;
}
say is-cyclic(142857); # True
say is-cyclic(95678); # False
Here I created the subroutine is-cyclic that takes an integer and returns a Boolean value. I use a for loop to iterate over the places of the number’s digits (1st, 2nd, etc.) and use them to multiply the number in each iteration. Afterward I use the previously seen comb method followed by the Bag method. In Perl 6, a Bag is an immutable collection of distinct elements in no particular order where each element is weighted by the number of copies in the collection. This is the kind of structure I need since only the number’s digits and their amounts are important, not their order and a Bag accomplishes exactly this. The subroutine returns False if the bags don’t have the same numbers or have the same numbers but are weighted differently. Otherwise, True is returned indicating the number’s cyclic-ness.
Happy numbers
A happy number is defined by the following process: Starting with any positive integer, replace the number by the sum of the squares of its digits in base-ten, and repeat the process until the number either equals 1 (where it will stay), or it loops endlessly in a cycle that does not include 1.
sub is-happy( $n is copy ) {
my $seen-numbers = :{};
while $n > 1 {
return False if $n ∈ $seen-numbers;
$seen-numbers{$n} = True;
$n = $n.comb.map(*²).sum
}
return True;
}
say is-happy(7); # True
say is-happy(2018); # False
After going through the process described in the definition, a happy number ends being equal to 1. On the other hand, a non-happy number follows a sequence that reaches the cycle 4, 16, 37, 58, 89, 145, 42, 20, 4,… which excludes 1. Armed with this fact, I created the hash $seen-numbers to keep track of such numbers. As illustrated by the while loop, the process is repeated over and over while $n is greater than 1 or until a number has been seen. Here the line that stands out is the one containing the symbol ∈. In set theory, if an element p is a member (or element) of a set A, then it’s denoted by p ∈ A and this exactly what’s being tested here. If the number $n is an element of the hash, then the sub returns False. Otherwise, it returns True which indicates the number’s happiness.
Summary
In this post, I slightly went over gradual typing, how to define a new operator, sub-classing by using the subset keyword and the set and bag data structures. As you may have realized, Perl 6 offers many constructs that facilitate many different tasks. In this instance, it was my desire to express definitions of numbers in a more programmatic way. Yours could be totally different but you can rest assured that Perl 6 is there to make the job easier and immensely more enjoyable.
Well…that’s all folks! Happy Christmas and a wonder-ofun New Year!
Today, let me introduce Algorithm::LDA.
This module is a Latent Dirichlet Allocation (i.e., LDA) implementation for topic modeling.
Introduction
What’s LDA? LDA is one of the popular unsupervised machine learning methods.
It models document generation process and represents each document as a mixture of topics.
So, what does “a mixture of topics” mean? Fig. 1 shows an article in which some of the words are highlighted in three colors: yellow, pink, and blue. Words about genetics are marked in yellow; words about evolutionary biology are marked in pink; words about data analysis are marked in blue. Imagine all of the words in this article are colored, then we can represent this article as a mixture of topics (i.e., colors).
Fig. 1:
(This image is from “Probabilistic topic models.” (David Blei 2012))
OK, then I’ll demonstrate how to use Algorithm::LDA in the next section.
Modeling Quotations
In this article, we explore Wikiquote. Wikiquote is a cloud-sourced platform providing sourced quotations.
By using Wikiquote API, we get quotations that are used for LDA estimation. After that, we execute LDA and plot the result.
Finally, we create an information retrieval application using the resulting model.
Preliminary
Wikiquote API
Wikiquote has action API that provides means for getting Wikiquote resources.
For example, you can get content of the Main Page as follows:
{"batchcomplete":"","warnings":{"main":{"*":"Subscribe to the mediawiki-api-announce mailing list at <https://lists.wikimedia.org/mailman/listinfo/mediawiki-api-announce> for notice of API deprecations and breaking changes. Use [[Special:ApiFeatureUsage]] to see usage of deprecated features by your application."},"revisions":{"*":"Because \"rvslots\" was not specified, a legacy format has been used for the output. This format is deprecated, and in the future the new format will always be used."}},"query":{"pages":{"1":{"pageid":1,"ns":0,"title":"Main Page","revisions":[{"contentformat":"text/x-wiki","contentmodel":"wikitext","*":"
WWW by Zoffix Znet is a library which provides easy-to-use API for fetching and parsing json very simply.
For instance, as the README says, you can easily get content by jget(URL)<HASHKEY> style:
say jget('https://httpbin.org/get?foo=42&bar=x')<args><foo>;
NLTK is a toolkit for natural language processing.
Not only APIs, it also provides corpus.
You can get stopwords for English via “70. Stopwords Corpus”: http://www.nltk.org/nltk_data/
Exercise 1: Get Quotations and Create Cleaned Documents
At the beginning, we have to get quotations from Wikiquote and create clean documents.
The main goal of this section is to create documents according to the following format:
where get-members-from-category gets members via Wikiquote API:
sub get-members-from-category(Str $category -->List) {
my $member-url ="https://en.wikiquote.org/w/api.php?action=query&list=categorymembers&cmtitle={$category}&cmlimit=100&format=json";
@(jget($member-url)<query><categorymembers>.map(*<title>));
}
Next, call get-pages:
my @pages = get-pages(@members);
get-pages is a subroutine that gets pages of the given titles (i.e., members):
sub get-pages(Str @members, Int $batch =50-->List) {
myInt $start=0;
my @pages;
while $start< @members {
my $list = @members[$start..^List($start+ $batch, +@members).min].map({ uri_escape($_) }).join('%7C');
my $url ="https://en.wikiquote.org/w/api.php?action=query&prop=revisions&rvprop=content&format=json&formatversion=2&titles={$list}";
@pages.push($_) for jget($url)<query><pages>.map({ %(body =>.<revisions>[0]<content>, title =>.<title>) });
$start+= $batch;
}
@pages;
}
where @members[$start..^List($start + $batch, [email protected]).min] is a slice of length $batch, and the elements of the slice are percent encoded by uri_escase and joint by %7C (i.e., percent encoded pipe symbol).
In this case, one of the resulting $list is:
Note that get-pages subroutine uses hash contextualizer %() for creating a sequence of hash:
@pages.push($_) for jget($url)<query><pages>.map({ %(body =>.<revisions>[0]<content>, title =>.<title>) });
After that, we call create-documents-from-pages:
my @documents = create-documents-from-pages(@pages, @members);
create-documents-from-pages creates documents from each page:
sub create-documents-from-pages(@pages, @members -->List) {
my @documents;
for @pages -> $page {
my @quotations = $page<body>.split("\n")\
.map(*.subst(/\[\[$<text>=(<-[\[\]|]>+?)\|$<link>=(<-[\[\]|]>+?)\]\]/, { $<text> }, :g))\
.map(*.subst(/\[\[$<text>=(<-[\[\]|]>+?)\]\]/, { $<text> }, :g))\
.map(*.subst("[", "[", :g))\
.map(*.subst("]", "]", :g))\
.map(*.subst("&", "&", :g))\
.map(*.subst(" ", "", :g))\
.map(*.subst(/:i [ \<\/?\s?br\> | \<br\s?\/?\> ]/, "", :g))\
.grep(/^\*<-[*]>/)\
.map(*.subst(/^\*\s+/, ""));
# Note: The order of array wikiquote API returned is agnostic.myInt $index= @members.pairs.grep({ .valueeq $page<title> }).map(*.key).head;
@documents.push(%(body =>$_, personid => $index)) for @quotations;
}
@documents.sort({ $^a<personid> <=> $^b<personid> }).pairs.map({ %(docid =>.key, personid =>.value<personid>, body =>.value<body>) }).list
}
where .map(*.subst(/\[\[$<text>=(<-[\[\]|]>+?)\|$<link>=(<-[\[\]|]>+?)\]\]/, { $<text> }, :g)) and .map(*.subst(/\[\[$<text>=(<-[\[\]|]>+?)\]\]/, { $<text> }, :g)) are coverting commands that extract texts for displaying and delete texts for internal linking from anchor texts. For example, [[Perl]] is reduced into Perl. For more syntax info, see: https://docs.perl6.org/language/regexes#Named_captures or https://docs.perl6.org/routine/subst
After some cleaning operations (.e.g., .map(*.subst("[", "[", :g))), we extract quotation lines. .grep(/^\*<-[*]>/) finds lines starting with single asterisk because most of the quotations appear in such kind of lines.
Next, .map(*.subst(/^\*\s+/, "")) deletes each asterisk since asterisk itself isn’t a constituent of each quotation.
Finally, we save the documents and members (i.e., titles):
my $docfh =open"documents.txt", :w;
$docfh.say((.<docid>, .<personid>, .<body>).join("")) for @documents;
$docfh.close;
my $memfh =open"members.txt", :w;
$memfh.say($_) for @members;
$memfh.close;
Exercise 2: Execute LDA and Visualize the Result
In the previous section, we saved the cleaned documents.
In this section, we use the documents for LDA estimation and visualize the result.
The goal of this section is to plot a document-topic distribution and write a topic-word table.
The whole source code is:
usev6.c;
use Algorithm::LDA;
use Algorithm::LDA::Formatter;
use Algorithm::LDA::LDAModel;
use Chart::Gnuplot;
use Chart::Gnuplot::Subset;
sub create-model(@documents --> Algorithm::LDA::LDAModel) {
my $stopwords ="stopwords/english".IO.lines.Set;
my &tokenizer =-> $line { $line.words.map(*.lc).grep(-> $w { ($stopwords !(cont) $w) and $w !~~ /^[ <:S> | <:P> ]+$/ }) };
my ($documents, $vocabs) = Algorithm::LDA::Formatter.from-plain(@documents.map({ my ($, $, *@body) =.words; @body.join("") }), &tokenizer);
my Algorithm::LDA $lda .=new(:$documents, :$vocabs);
my Algorithm::LDA::LDAModel $model = $lda.fit(:num-topics(10), :num-iterations(500), :seed(2018));
$model
}
sub plot-topic-distribution($model, @members, @documents, $search-regex =rx/Larry/) {
my $target-personid = @members.pairs.grep({ .value~~ $search-regex }).map(*.key).head;
my $docid = @documents.map({ my ($docid, $personid, *@body) =.words; %(docid => $docid, personid => $personid, body => @body.join("")) })\
.grep({ .<personid> == $target-personid and.<body> ~~ /:i<< perl >>/}).map(*<docid>).head;
note("@documents[$docid] is selected");
my ($row-size, $col-size) = $model.document-topic-matrix.shape;
my @doc-topic =gatherfor ($docid X^$col-size) -> ($i, $j) { take $model.document-topic-matrix[$i;$j]; }
my Chart::Gnuplot $gnu .=new(:terminal("png"), :filename("topics.png"));
$gnu.command("set boxwidth 0.5 relative");
my AnyTicsTic @tics = @doc-topic.pairs.map({ %(:label(.key), :pos(.key)) });
$gnu.legend(:off);
$gnu.xlabel(:label("Topic"));
$gnu.ylabel(:label("P(z|theta,d)"));
$gnu.xtics(:tics(@tics));
$gnu.plot(:vertices(@doc-topic.pairs.map({ @(.key, .value.exp) })), :style("boxes"), :fill("solid"));
$gnu.dispose;
}
sub write-nbest($model) {
my $topics := $model.nbest-words-per-topic(10);
for^(10/5) -> $part-i {
say"|"~ (^5).map(-> $t { "topic { $part-i *5+ $t }" }).join("|") ~"|";
say"|"~ (^5).map({ "----" }).join("|") ~"|";
for^10-> $rank {
say"|"~gatherfor ($part-i *5)..^($part-i *5+5) -> $topic {
take@($topics)[$topic;$rank].key;
}.join("|") ~"|";
}
"".say;
}
}
sub save-model($model) {
my @document-topic-matrix := $model.document-topic-matrix;
my ($document-size, $topic-size) = @document-topic-matrix.shape;
my $doctopicfh =open"document-topic.txt", :w;
$doctopicfh.say: ($document-size, $topic-size).join("");
for^$document-size -> $doc-i {
$doctopicfh.say:gatherfor^$topic-size -> $topic { take @document-topic-matrix[$doc-i;$topic] }.join("");
}
$doctopicfh.close;
my @topic-word-matrix := $model.topic-word-matrix;
my ($, $word-size) = @topic-word-matrix.shape;
my $topicwordfh =open"topic-word.txt", :w;
$topicwordfh.say: ($topic-size, $word-size).join("");
for^$topic-size -> $topic-i {
$topicwordfh.say:gatherfor^$word-size -> $word { take @topic-word-matrix[$topic-i;$word] }.join("");
}
$topicwordfh.close;
my @vocabulary := $model.vocabulary;
my $vocabfh =open"vocabulary.txt", :w;
$vocabfh.say($_) for @vocabulary;
$vocabfh.close;
}
my @documents ="documents.txt".IO.lines;
my $model = create-model(@documents);
my @members ="members.txt".IO.lines;
plot-topic-distribution($model, @members, @documents);
write-nbest($model);
save-model($model);
First, we load the cleaned documents and call create-model:
my @documents ="documents.txt".IO.lines;
my $model = create-model(@documents);
create-model creates a LDA model by loading given documents:
sub create-model(@documents --> Algorithm::LDA::LDAModel) {
my $stopwords ="stopwords/english".IO.lines.Set;
my &tokenizer =-> $line { $line.words.map(*.lc).grep(-> $w { ($stopwords !(cont) $w) and $w !~~ /^[ <:S> | <:P> ]+$/ }) };
my ($documents, $vocabs) = Algorithm::LDA::Formatter.from-plain(@documents.map({ my ($, $, *@body) =.words; @body.join("") }), &tokenizer);
my Algorithm::LDA $lda .=new(:$documents, :$vocabs);
my Algorithm::LDA::LDAModel $model = $lda.fit(:num-topics(10), :num-iterations(500), :seed(2018));
$model
}
where $stopwords is a set of English stopwords from NLTK (I mentioned preliminary section), and &tokenizer is a custom tokenizer for Algorithm::LDA::Formatter.from-plain. The tokenizer converts given sentence as follows:
Splits given sentence by whitespace and makes a list of tokens.
Replaces each characters of the token with lower-case characters.
Deletes token that exists in the stopwords list or one-length token categorized as Symbol or Punctuation.
Algorithm::LDA::Formatter.from-plain creates numerical native documents (i.e., each word in a document is mapped to its corresponding vocabulary id, and this id is represented by C int32) and vocabulary from a list of texts.
After creating an Algorithm::LDA instance using the above numerical documents, we can start LDA estimation by Algorithm::LDA.fit. In this example, we set the number of topics to 10, and the number of iterations to 100, and the seed for srand to 2018.
Next, we plot a document-topic distribution. Before this plotting, we load the saved members:
my @members ="members.txt".IO.lines;
plot-topic-distribution($model, @members, @documents);
plot-topic-distribution plots topic distribution with Chart::Gnuplot:
sub plot-topic-distribution($model, @members, @documents, $search-regex =rx/Larry/) {
my $target-personid = @members.pairs.grep({ .value~~ $search-regex }).map(*.key).head;
my $docid = @documents.map({ my ($docid, $personid, *@body) =.words; %(docid => $docid, personid => $personid, body => @body.join("")) })\
.grep({ .<personid> == $target-personid and.<body> ~~ /:i<< perl >>/}).map(*<docid>).head;
note("@documents[$docid] is selected");
my ($row-size, $col-size) = $model.document-topic-matrix.shape;
my @doc-topic =gatherfor ($docid X^$col-size) -> ($i, $j) { take $model.document-topic-matrix[$i;$j]; }
my Chart::Gnuplot $gnu .=new(:terminal("png"), :filename("topics.png"));
$gnu.command("set boxwidth 0.5 relative");
my AnyTicsTic @tics = @doc-topic.pairs.map({ %(:label(.key), :pos(.key)) });
$gnu.legend(:off);
$gnu.xlabel(:label("Topic"));
$gnu.ylabel(:label("P(z|theta,d)"));
$gnu.xtics(:tics(@tics));
$gnu.plot(:vertices(@doc-topic.pairs.map({ @(.key, .value.exp) })), :style("boxes"), :fill("solid"));
$gnu.dispose;
}
In this example, we plot topic distribution of a Larry Wall’s quotation (“Although the Perl Slogan is There’s More Than One Way to Do It, I hesitate to make 10 ways to do something.”):
After the plotting, we call write-nbest:
write-nbest($model);
In LDA, what topic XXX represents is expressed as a list of words. write-nbest writes a markdown style topic-word distribution table:
As you can see, the quotation of “Although the Perl Slogan is There’s More Than One Way to Do It, I hesitate to make 10 ways to do something.” contains “one”, “way” and “perl”. This is the reason why this quotation is mainly composed of topic 8.
For the next section, we save the model by save-model subroutine:
sub save-model($model) {
my @document-topic-matrix := $model.document-topic-matrix;
my ($document-size, $topic-size) = @document-topic-matrix.shape;
my $doctopicfh =open"document-topic.txt", :w;
$doctopicfh.say: ($document-size, $topic-size).join("");
for^$document-size -> $doc-i {
$doctopicfh.say:gatherfor^$topic-size -> $topic { take @document-topic-matrix[$doc-i;$topic] }.join("");
}
$doctopicfh.close;
my @topic-word-matrix := $model.topic-word-matrix;
my ($, $word-size) = @topic-word-matrix.shape;
my $topicwordfh =open"topic-word.txt", :w;
$topicwordfh.say: ($topic-size, $word-size).join("");
for^$topic-size -> $topic-i {
$topicwordfh.say:gatherfor^$word-size -> $word { take @topic-word-matrix[$topic-i;$word] }.join("");
}
$topicwordfh.close;
my @vocabulary := $model.vocabulary;
my $vocabfh =open"vocabulary.txt", :w;
$vocabfh.say($_) for @vocabulary;
$vocabfh.close;
}
Exercise 3: Create Quotation Search Engine
In this section, we create a quotation search engine which uses the model created in the previous section.
More specifically, we create LDA-based document model (Xing Wei and W. Bruce Croft 2006) and make a CLI tool that can search quotations. (Note that the words “token” and “word” are interchangable in this section)
The whole source code is:
usev6.c;
sub MAIN(Str:$query!) {
my \doc-topic-iter ="document-topic.txt".IO.lines.iterator;
my \topic-word-iter ="topic-word.txt".IO.lines.iterator;
my ($document-size, $topic-size) = doc-topic-iter.pull-one.words;
my ($, $word-size) = topic-word-iter.pull-one.words;
myNum @document-topic[$document-size;$topic-size];
myNum @topic-word[$topic-size;$word-size];
for^$document-size -> $doc-i {
my \maybe-line := doc-topic-iter.pull-one;
die"Error: Something went wrong"if maybe-line =:= IterationEnd;
myNum @line =@(maybe-line).words>>.Num;
for^@line {
@document-topic[$doc-i;$_] = @line[$_];
}
}
for^$topic-size -> $topic-i {
my \maybe-line := topic-word-iter.pull-one;
die"Error: Something went wrong"if maybe-line =:= IterationEnd;
myNum @line =@(maybe-line).words>>.Num;
for^@line {
@topic-word[$topic-i;$_] = @line[$_];
}
}
my%vocabulary ="vocabulary.txt".IO.lines.pairs>>.antipair.hash;
my @members ="members.txt".IO.lines;
my @documents ="documents.txt".IO.lines;
my @docbodies = @documents.map({ my ($, $, *@body) =.words; @body.join("") });
my%doc-to-person = @documents.map({ my ($docid, $personid, $) =.words; %($docid => $personid) }).hash;
my @query = $query.words.map(*.lc);
my @sorted-list =gatherfor^$document-size -> $doc-i {
myNum $log-prob =gatherfor @query -> $token {
my Num $log-ml-prob = Pml(@docbodies, $doc-i, $token);
my Num $log-lda-prob = Plda($token, $topic-size, $doc-i, %vocabulary, @document-topic, @topic-word);
take log-sum(log(0.2) +$log-ml-prob, log(0.8) +$log-lda-prob);
}.sum;
take%(doc-i => $doc-i, log-prob => $log-prob);
}.sort({ $^b<log-prob> <=> $^a<log-prob> });
for^10 {
my $docid = @sorted-list[$_]<doc-i>;
sprintf("\"%s\" by %s %f", @docbodies[$docid], @members[%doc-to-person{$docid}], @sorted-list[$_]<log-prob>).say;
}
}
sub Pml(@docbodies, $doc-i, $token --> Num) {
my Int $num-tokens = @docbodies[$doc-i].words.grep({/:i^ $token $/ }).elems;
myInt $total-tokens = @docbodies[$doc-i].words.elems;
return-100e0if $total-tokens ==0or $num-tokens ==0;
log($num-tokens) -log($total-tokens);
}
sub Plda($token, $topic-size, $doc-i, %vocabulary is raw, @document-topic is raw, @topic-word is raw --> Num) {gatherfor^$topic-size -> $topic {
if%vocabulary{$token}:exists {
take @document-topic[$doc-i;$topic] + @topic-word[$topic;%vocabulary{$token}];
} else {
take-100e0;
}
}.reduce(&log-sum);
}
sub log-sum(Num $log-a, Num $log-b --> Num) {if $log-a < $log-b {
return $log-b +log(1+exp($log-a - $log-b))
} else {
return $log-a +log(1+exp($log-b - $log-a))
}
}
At the beginning, we load the saved model and prepare @document-topic, @topic-word, %vocabulary, @documents, @docbodies, %doc-to-person and @members:
my \doc-topic-iter ="document-topic.txt".IO.lines.iterator;
my \topic-word-iter ="topic-word.txt".IO.lines.iterator;
my ($document-size, $topic-size) = doc-topic-iter.pull-one.words;
my ($, $word-size) = topic-word-iter.pull-one.words;
myNum @document-topic[$document-size;$topic-size];
myNum @topic-word[$topic-size;$word-size];
for^$document-size -> $doc-i {
my \maybe-line = doc-topic-iter.pull-one;
die"Error: Something went wrong"if maybe-line =:= IterationEnd;
myNum @line =@(maybe-line).words>>.Num;
for^@line {
@document-topic[$doc-i;$_] = @line[$_];
}
}
for^$topic-size -> $topic-i {
my \maybe-line = topic-word-iter.pull-one;
die"Error: Something went wrong"if maybe-line =:= IterationEnd;
myNum @line =@(maybe-line).words>>.Num;
for^@line {
@topic-word[$topic-i;$_] = @line[$_];
}
}
my%vocabulary ="vocabulary.txt".IO.lines.pairs>>.antipair.hash;
my @members ="members.txt".IO.lines;
my @documents ="documents.txt".IO.lines;
my @docbodies = @documents.map({ my ($, $, *@body) =.words; @body.join("") });
my%doc-to-person = @documents.map({ my ($docid, $personid, $) =.words; %($docid => $personid) }).hash;
Next, we set @query using option :$query:
my @query = $query.words.map(*.lc);
After that, we compute the probability of P(query|document) based on Eq. 9 of the aforementioned paper (Note that we use logarithm to avoid undeflow and set the parameter mu to zero) and sort them.
my @sorted-list =gatherfor^$document-size -> $doc-i {
myNum $log-prob =gatherfor @query -> $token {
my Num $log-ml-prob = Pml(@docbodies, $doc-i, $token);
my Num $log-lda-prob = Plda($token, $topic-size, $doc-i, %vocabulary, @document-topic, @topic-word);
take log-sum(log(0.2) +$log-ml-prob, log(0.8) +$log-lda-prob);
}.sum;
take%(doc-i => $doc-i, log-prob => $log-prob);
}.sort({ $^b<log-prob> <=> $^a<log-prob> });
Plda adds logarithmic topic given document probability (i.e., lnP(topic|theta,document)) and word given topic probability (i.e., lnP(word|phi,topic)) for each topic, and sums them by .reduce(&log-sum);:
and Pml (ml means Maximum Likelihood) counts $token and normalizes it by the number of the total tokens in the document (Note that this computation is also conducted in log space):
sub Pml(@docbodies, $doc-i, $token --> Num) {
my Int $num-tokens = @docbodies[$doc-i].words.grep({/:i^ $token $/ }).elems;
myInt $total-tokens = @docbodies[$doc-i].words.elems;
return-100e0if $total-tokens ==0or $num-tokens ==0;
log($num-tokens) -log($total-tokens);
}
OK, then let’s execute!
query “perl”:
$ perl6 search-quotation.p6 --query="perl""Perl will always provide the null." by Larry Wall -3.301156
"Perl programming is an *empirical* science!" by Larry Wall -3.345189
"The whole intent of Perl 5's module system was to encourage the growth of Perl culture rather than the Perl core." by Larry Wall -3.490238
"I dunno, I dream in Perl sometimes..." by Larry Wall -3.491790
"At many levels, Perl is a 'diagonal' language." by Larry Wall -3.575779
"Almost nothing in Perl serves a single purpose." by Larry Wall -3.589218
"Perl has a long tradition of working around compilers." by Larry Wall -3.674111
"As for whether Perl 6 will replace Perl 5, yeah, probably, in about 40 years or so." by Larry Wall -3.684454
"Well, I think Perl should run faster than C." by Larry Wall -3.771155
"It's certainly easy to calculate the average attendance for Perl conferences." by Larry Wall -3.864075
query “apple”:
$ perl6 search-quotation.p6 --query="apple""Steve Jobs is the"With phones moving to technologies such as Apple Pay, an unwillingness to assure security could create a Target-like exposure that wipes Apple out of the market." by Rob Enderle -3.841538"*:From Joint Apple / HP press release dated 1 January 2004 available [http://www.apple.com/pr/library/2004/jan/08hp.html here]." by Carly Fiorina -3.904489"Samsung did to Apple what Apple did to Microsoft, skewering its devoted users and reputation, only better. ... There is a way for Apple to fight back, but the company no longer has that skill, and apparently doesn't know where to get it, either." by Rob Enderle -3.940359"[W]hen it came to the iWatch, also a name that Apple didn't own, Apple walked away from it and instead launched the Apple Watch. Certainly, no risk of litigation, but the product's sales are a fraction of what they otherwise might have been with the proper name and branding." by Rob Enderle -4.152145"[W]hen Apple wanted the name "iPhone" and it was owned by Cisco, Steve Jobs just took it, and his legal team executed so he could keep it. It turned out that doing this was surprisingly inexpensive. And, as the Apple Watch showcased, the Apple Phone likely would not have sold anywhere near as well as the iPhone." by Rob Enderle -4.187223"The cause of [Apple v. Qualcomm] appears to be an effort by Apple to pressure Qualcomm into providing a unique discount, largely because Apple has run into an innovation wall, is under increased competition from firms like Samsung, and has moved to a massive cost reduction strategy. (I've never known this to end well, as it causes suppliers to create unreliable components and outright fail.)" by Rob Enderle -4.318575"Apple tends to aggressively work to not discover problems with products that are shipped and certainly not talk about them." by Rob Enderle -4.380863"Apple no longer owns the tablet market, and will likely lose dominance this year or next. ... this level of sustained dominance doesn't appear to recur with the same vendor even if it launched the category." by Rob Enderle -4.397954"Apple is becoming more and more like a typical tech firm — that is, long on technology and short on magic. ... Apple is drifting closer and closer to where it was back in the 1990s. It offers advancements that largely follow those made by others years earlier, product proliferation, a preference for more over simple elegance, and waning excitement." by Rob Enderle -4.448473"[T]he litigation between Qualcomm and Apple/Intel ... is weird. What makes it weird is that Intel appears to think that by helping Apple drive down Qualcomm prices, it will gain an advantage, but since its only value is as a lower cost, lower performing, alternative to Qualcomm's modems, the result would be more aggressively priced better alternatives to Intel's offerings from Qualcomm/Broadcom, wiping Intel out of the market. On paper, this is a lose/lose for Intel and even for Apple. The lower prices would flow to Apple competitors as well, lowering the price of competing phones. So, Apple would not get a lasting benefit either." by Rob Enderle -4.469852 Ronald McDonald of Apple, he is the face." by Rob Enderle -3.822949"With phones moving to technologies such as Apple Pay, an unwillingness to assure security could create a Target-like exposure that wipes Apple out of the market." by Rob Enderle -3.849055"*:From Joint Apple / HP press release dated 1 January 2004 available [http://www.apple.com/pr/library/2004/jan/08hp.html here]." by Carly Fiorina -3.895163"Samsung did to Apple what Apple did to Microsoft, skewering its devoted users and reputation, only better. ... There is a way for Apple to fight back, but the company no longer has that skill, and apparently doesn't know where to get it, either." by Rob Enderle -4.052616"*** The previous line contains the naughty word '$&'.\nif /(ibm|apple|awk)/;# :-)" by Larry Wall -4.088445"The cause of [Apple v. Qualcomm] appears to be an effort by Apple to pressure Qualcomm into providing a unique discount, largely because Apple has run into an innovation wall, is under increased competition from firms like Samsung, and has moved to a massive cost reduction strategy. (I've never known this to end well, as it causes suppliers to create unreliable components and outright fail.)" by Rob Enderle -4.169533
"[T]he litigation between Qualcomm and Apple/Intel ... is weird. What makes it weird is that Intel appears to think that by helping Apple drive down Qualcomm prices, it will gain an advantage, but since its only value is as a lower cost, lower performing, alternative to Qualcomm's modems, the result would be more aggressively priced better alternatives to Intel's offerings from Qualcomm/Broadcom, wiping Intel out of the market. On paper, this is a lose/lose for Intel and even for Apple. The lower prices would flow to Apple competitors as well, lowering the price of competing phones. So, Apple would not get a lasting benefit either." by Rob Enderle -4.197869
"Apple tends to aggressively work to not discover problems with products that are shipped and certainly not talk about them." by Rob Enderle -4.204618
"Today's tech companies aren't built to last, as Apple's recent earnings report shows all too well." by Rob Enderle -4.209901
"[W]hen it came to the iWatch, also a name that Apple didn't own, Apple walked away from it and instead launched the Apple Watch. Certainly, no risk of litigation, but the product's sales are a fraction of what they otherwise might have been with the proper name and branding." by Rob Enderle -4.238582
Conclusions
In this article, we explored Wikiquote and created a LDA model using Algoritm::LDA.
After that we built an information retrieval application.
Thanks for reading my article! See you next time!
Citations
Blei, David M. “Probabilistic topic models.” Communications of the ACM 55.4 (2012): 77-84.
Wei, Xing, and W. Bruce Croft. “LDA-based document models for ad-hoc retrieval.” Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval. ACM, 2006.
I’ve already mentioned Bisectable in one of the advent posts two years ago, but since then a lot has changed, so I think it’s time to give a brief history of the bisectable bot and its friends.
First of all, let’s define the problem that is being solved. Sometimes it happens that a commit introduces an unintended change in behavior (a bug). Usually we call that a regression, and in some cases the easiest way to figure out what went wrong and fix it is to first find which commit introduced the regression.
There are exactly 9000 commits between Rakudo 2015.12 and 2018.12, and even though it’s not over 9000, that’s still a lot.
Luckily, we don’t need to test all of the revisions. Assuming that the behavior wasn’t changing back and forth all the time, we can use binary search.
git bisect and binary search
Basically, given any commit range, we take a commit in the “middle” of the range and test it. If it’s “bad” or if it shows the “new” (now incorrect) behavior, then we can throw away the second half of our range (because we know that the change must have happened before that commit or exactly on that commit). Similarly we throw away the other half if it is “good” (or “old”). So instead of testing all 9000 commits we can just check about log n revisions (≈13).
Git comes with git bisect command which implements the binary search logic for you. All you have to do is give it some starting points and then for every commit it jumps to, tell if it is good/bad. If you do that enough times, it’ll tell you which commit is at fault.
That’s all good, but there are two problems with it.
Problem ❶: Skipping
Let’s imagine a situation where 2 + 2 used to return 4 (correct!), but now returns 42 (… also right, but not quite).
So you kick off the bisection process, git jumps between revisions, you test them. If it’s 4 then it’s good (or old), if it’s 42 then it is bad (or new). But then you stumble upon this behavior:
> 2 + 2
Merry Christmas!
… Now what? Clearly that specific revision is somewhat special. We can’t tell if our bug is present or not, we simply can’t know. Yes, it doesn’t print 4, but we are looking for a very specific bug, so it doesn’t classify as “new” behavior either. Of course, we can toss a coin and mark it randomly as old or new, and hope for a Christmas miracle… but that has a 50% probability (if we see only one of these) to divert the binary search into the wrong direction.
For these cases git provides a special skip command.
If you are testing manually, then it is somewhat straightforward to handle these revisions (as long as you remember that you should skip them). However, because of problem ❷, a lot of people are tempted to use git bisect run which automates the process with a script. It is possible to skip revisions using a script too (use exit code 125), but it is not that obvious how to figure out which revisions should be skipped.
Problem ❷: Build times
Let’s take the optimistic figure of 13 to estimate the amount of revisions that we are going to test. Remember that it doesn’t include commits that we will have to skip, and possibly other extra builds that we might want to test.
The amount of time it takes to build rakudo varies depending on the hardware, but let’s optimistically say that it takes us 2 minutes to build rakudo on a particular commit and test it.
13 × 2 = 26 (minutes)
That’s not very convenient, right? And if something goes wrong during the process… you start over, and then you wait.
Bisectable
In 2016, after seeing the pain of those who have to run git bisect manually (actually, mostly myself), I wondered:
<AlexDaniel> has anybody thought about building rakudo for every single commit, so that you can quickly run git bisect?
The cost-benefit analysis of the idea was promptly questioned:
<perlpilot> AlexDaniel: do you believe that bisects will be common in the future?
To which I provided a very detailed justification:
<AlexDaniel> perlpilot: yes
Three days later, the bot joined the channel. The reactions were quite interesting to see:
Little did we know back then. Even I had no idea how useful it will turn out. Fast forward 2 years:
<lizmat> with regards to size of commits: I try to keep them as small and contained as possible, to allow for easier bisecting
<lizmat> in that sense, bisectable6 has changed the way I code
<lizmat> also: bisectable6 has made me worry less about changes I commit
<lizmat> because it usually limits the places to look for fixing an issue so much, that they can be fixed within minutes rather than hours
<lizmat> or at least show the cause very quickly (so the short-time fix may mean a revert)
<AlexDaniel> \o/
But it wasn’t always perfect. About one hour after the introduction of the bot, it was used for its purpose:
However, because of an off-by-one, it returned the wrong commit. The actual commit was e2f1fa7, and 949a7c7 is its parent.
Honestly, the bot was very bad back then. For example, it fully relied on the exit code, so you couldn’t just throw 2 + 2 into it and expect it to check the output. Eventually, different modes were implemented, and nowadays the bot first checks the behavior on the starting points (e.g. 2015.12 and HEAD), and determines the best strategy to perform the bisection. For example, if the signal is different (e.g. a SEGV), then it bisects based on the signal. If the signal is same, but the exit code is different, then it uses the exit code. If all else can’t be used, it bisects using the output.
Keep in mind that bisectable checks for you if perl6 binary can’t be built. This means that in most cases you don’t need to add your own logic for skipping. Not only it brought the bisection time from tens of minutes to a few seconds, it also gives results that are more reliable/correct.
Storage
Some time later the commit range was expanded to 2014.01 … HEAD, meaning all commits starting from the first ever Rakudo on Moar release. Currently it has over 17000 builds. It may sound like a lot, but with every rakudo installation taking just ≈28 MB, that’s not too much. Having a few TB of storage should get you going for a few years to come.
That being said, I don’t have that luxury on my server. It has a RAID of 120 GB SSDs, so the whole thing not only has to fit into that little amount of space, but it should also leave enough space for the rest of the system.
There was a lot of experimentation (one, two) involved in figuring out the best strategy to save space, but long story short, we can go as low as about half a megabyte per build! Of course, it is always a tradeoff between the compression ratio and decompression speed, but using modern compression algorithms (zstd, lrzip) everything is relatively easy.
More bots, more better
Shortly after Bisectable was released, people saw an opportunity for other tools. Want to run some code on a specific commit? Sure, here’s a bot for that. Want to download a prebuilt rakudo archive instead of wasting your own cpu time? Yes, there’s another bot. Want to graph some info about rakudo? Of course there’s a bot for that!
And it continued until we reached the total of 17 bots! Some argue that these bots should stop multiplying like that, and perhaps people are right. But I guess the point is that now it is extremely easy to build upon Whateverable to create more tools for developers, which is of course great.
OK, now what?
So bisectable can bisect across thousands of commits in no time. It consumes very little storage space, and it doesn’t require full understanding of the bisection process from the user. Now that the bisection is free and easy, can we do more?
Yes, Blin!
You may have heard about Toaster. Toaster is a tool that attempts to install every module in the ecosystem on two or more revisions. For example, let’s say that the last release was 2018.12 and the release manager is about to cut a rakudo release from master HEAD. You can then run toaster on 2018.12 and master, and it will show which modules used to install cleanly but no longer do.
That gives us the information that something is likely wrong in Rakudo, but doesn’t tell what exactly. Given that this post was mostly about Bisectable, you can probably guess where this is going.
Project Blin – Toasting Reinvented
Blin is a quality assurance tool for Rakudo releases. It is used to find regressions in rakudo, but unlike Toaster, not only it tells which modules are no longer installable, it also bisects rakudo to find out which commit caused the issue. Of course, it is built around Whateverable, so that extra functionality doesn’t cost much (and doesn’t even require a lot of code). As a bonus, it generates nice graphs to visualize how the issue propagates from module dependencies (though that is not very common).
One important feature of Blin is that it tries to install every module just once. So if module B depends on module A, A will be tested and installed once, and then reused for the testing of B. Because this process is parallelized, you may wonder how it was implemented. Basically, it uses the underrated react/whenever feature:
For every module (we have more than 1200 now) it creates its own whenever block which fires when its dependencies are satisfied. In my opinion, that’s the whole implementation of the main logic in Blin, everything else is just glue to get Whateverable and Zef working together to achieve what we need, + some output generation.
In some way, Blin didn’t change much in the way we do quality assurance for Rakudo. Toaster was already able to give us some basic info (albeit slower) so that we could start the investigation, and in the past I was known for shoving weird things (e.g. full modules with dependencies) into bisectable. It’s just that now it is much easier, and when The Day comes, I won’t be punished for robot abuse.
Future
Whateverable and Blin together have 243 open issues. Both projects work great and are very useful, but as we say, they are Less Than Awesome. Most issues are relatively easy and fun to work with, but they require time. If there’s anything you can help with or if you want to maintain these projects, please feel free to do so. And if you want to build your own tools based on Whateverable (which we probably need a lot!), see this hello world gist.
A good amount of my work time this year has been spent on building a couple of Perl 6 applications. After a decade of contributing to Perl 6 compiler and runtime development, it feels great to finally be using it to deliver production solutions solving real-world problems. I’m still not sure whether writing code in an IDE I founded, using a HTTP library I designed, compiled by a compiler I implemented large parts of, and running on a VM that I play architect for, makes me one of the world’s worst cases of “Not Invented Here”, or just really Full Stack.
Whatever I’m working on, I highly value automated testing. Each passing test is something I know works – and something that I won’t break as I evolve the software in question. Even with automated tests, bugs happen, but adding a test to cover the bug at least means I’ll make different bugs in the future, which is perhaps a bit more forgivable.
Most of the code, and complexity, in the system I’m currently working on is in its domain objects. Those are reached through a HTTP API, implemented using Cro – and like the rest of the system, this HTTP API has automated tests. They use one old module of mine – Test::Mock – along with a new module released this year, Cro::HTTP::Test. In today’s advent post, I’ll discuss how I’ve been using them together, with results that I find quite pleasing.
A sample problem
It’s the advent calendar, so of course I need a sufficiently festive example problem. For me, one of the highlights of Christmas time in Central Europe is the Christmas markets, many set on beautiful historic city squares. And what, aside from sausage and mulled wine, do we need on that square? A tall, handsome Christmas tree, of course! But how to find the best tree? Well, we get the internet to help, by building a system where they can submit suggestions of trees they’ve seen that might be suitable. What could possibly go wrong?
One can PUT to a route /trees/{latitude}/{longitude} to submit a candidate tree at that location. The expected payload is a JSON blob with a tree height, and a text description of 10-200 characters explaining why the tree is so awesome. If a tree in the same location has already been submitted, a 409 Conflict response should be returned. If the tree is accepted, then a simple 200 OK response will be produced, with a JSON body describing the tree.
A GET of the same URI will return a description of the tree in question, while a GET to /trees will return the submitted trees, tallest first.
Testability
Back in highschool, science classes were certainly among my favorite. Now and then, we got to do experiments. Of course, each experiment needed writing up – both the planning before, the results, and an analysis of them. One of the most important parts of the planning was about how to ensure a “fair test”: how would we try control all of the things we weren’t trying to test, so that we could trust in our observations and draw conclusions from them?
Testing in software involves much the same thought process: how do we exercise the component(s) we’re interested in, while controlling the context they operate in? Sometimes, we get lucky, and we’re testing pure logic: it doesn’t depend on anything other than the things we give it to work with. In fact, we can create our own luck in this regard, spotting parts of our system that can be pure functions or immutable objects. To take examples from the current system I’m working on:
We have an object model built up from a bunch of specification files.The process of building it is pretty involved, with a bunch of sanity checks, a few graph algorithms, and so forth. But the result is a bunch of immutable objects. Once constructed, they never change. Testing is easy: throw a bunch of test input in, and check that it builds the expected objects.
We have a small language with an evaluator. The data used by expressions in the language is passed in as an argument to the evaluator, and then we can check the result is what is expected. Thus, the evaluator is a pure function.
So, the first thing to do for testability is to find the bits of the system that can be like this and build them that way. Alas, not all things are so simple. HTTP APIs are often a gateway to mutable state, database operations, and so forth. Further, a good HTTP API will map error conditions from the domain level into appropriate HTTP status codes. We’d like to be able to create such situations in our tests, so as to cover them. This is where a tool like Test::Mock comes in – but to use it, we need to factor our Cro service in a way that is test-friendly.
Stubbing a service
For those new to Cro, let’s take a look at the bare minimum we can write to get a HTTP service up and running, serving some fake data about trees.
use Cro::HTTP::Router;
use Cro::HTTP::Server;
my $application = route {
get->'trees' {
content 'application/json', [
{
longitude =>50.4311548,
latitude =>14.586079,
height =>4.2,
description =>'Nice color, very bushy'
},
{
longitude =>50.5466504,
latitude =>14.8438714,
height =>7.8,
description =>'Really tall and wide'
},
]
}
}
my $server = Cro::HTTP::Server.new(:port(10000), :$application);
$server.start;
reactwhenever signal(SIGINT) {
$server.stop;
exit;
}
This isn’t a great setup for being able to test our routes, however. Better would be to put the routes into a subroutine in a module lib/BestTree.pm6:
unit moduleBestTree;
use Cro::HTTP::Router;
sub routes() isexport {
route {
get->'trees' {
content 'application/json', [
{
longitude =>50.4311548,
latitude =>14.586079,
height =>4.2,
description =>'Nice color, very bushy'
},
{
longitude =>50.5466504,
latitude =>14.8438714,
height =>7.8,
description =>'Really tall and wide'
},
]
}
}
}
And use it from the script:
use BestTree;
use Cro::HTTP::Server;
my $application = routes();
my $server = Cro::HTTP::Server.new(:port(10000), :$application);
$server.start;
reactwhenever signal(SIGINT) {
$server.stop;
exit;
}
Now, if we had something that could be used to test that route blocks do the right thing, we could use this module, and get on with our testing.
Stores, models, etc.
There’s another problem, however. Our Christmas tree service will be stashing the tree information away in some database, as well as enforcing the various rules. Where should this logic go?
There’s many ways we might choose to arrange this code, but the key thing is that this logic does not belong in our Cro route handlers. Their job is to map between the domain objects and the world of HTTP, for example turning domain exceptions into appropriate HTTP error responses. That mapping is what we’ll want to test.
So, before we continue, let’s define how some of those things look. We’ll have a BestTree::Tree class that represents a tree:
But how can we arrange things so we can take control of the store that is used by the routes, for testing purposes? One easy way is to make it a parameter to our routes subroutine, meaning it will be available in the route block:
sub routes(BestTree::Store $store) isexport {
...
}
This is a functional factoring. Some folks may prefer to use some kind of OO-based Dependency Injection, using some kind of container. That can work fine with Cro too: just have a method that returns the route block. (If building something non-tiny with Cro, check out the documentation on structuring services for some further advice on this front.)
Getting a list of trees
Now we’re ready to start writing tests! Let’s stub the test file:
use BestTree;
use BestTree::Store;
use Cro::HTTP::Test;
useTest::Mock;
useTest;
# Tests will go heredone-testing;
We use BestTree, which contains the routes we want to test, along with:
Cro::HTTP::Test, which we will use to easily write tests for our routes
Test::Mock, which we’ll use to fake the store
Test, which we don’t strictly need, but having access to subtest will
let us produce more organized test output
Next, we’ll make a couple of tree objects to use in our tests:
my $fake-tree-a = BestTree::Tree.new:latitude =>50.4311548,
longitude =>14.586079,
height =>4.2,
description =>'Nice color, very bushy';
my $fake-tree-b = BestTree::Tree.new:latitude =>50.5466504,
longitude =>14.8438714,
height =>7.8,
description =>'Really tall and wide';
And here comes the first test:
subtest 'Get all trees'=> {
my $fake-store = mocked BestTree::Store, returning => {
all-trees => [$fake-tree-a, $fake-tree-b]
};
test-service routes($fake-store), {
test get('/trees'),
status =>200,
json => [
{
latitude =>50.4311548,
longitude =>14.586079,
height =>4.2,
description =>'Nice color, very bushy'
},
{
latitude =>50.5466504,
longitude =>14.8438714,
height =>7.8,
description =>'Really tall and wide'
}
];
check-mock $fake-store,
*.called('all-trees', times =>1, with=> \());
}
}
First, we make a fake of BestTree::Store that, whenever all-trees is called, will return the fake data we specify. We then use test-service, passing in the route block created with the fake store. All test calls within the block that follows will be executed against that route block.
Notice that we don’t need to worry about running a HTTP server here to host the routes we want to test. In fact, due to the pipeline architecture of Cro, it’s easily possible for us to take the Cro HTTP client, wire its TCP message output to put the data it would send into a Perl 6 Channel, and then have that data pushed into the server pipeline’s TCP message input pipeline, and vice versa. This means that we test things all the way down to the bytes that are sent and received, but without actually having to hit even the local network stack. (Aside: you can also use Cro::HTTP::Test with a URI, which means if you really wanted to spin up a test server, or even wanted to write tests against some other service running in a different process, you could do it.)
The test routine specifies a test case. Its first argument describes the request that we wish to perform – in this case, a get to /trees. The named arguments then specify how the response should look. The status check ensures we get the expected HTTP status code back. The json check is really two in one:
It checks that the HTTP content-type is a JSON one
It checks that the body deserializes to the supplied JSON (if you don’t want to
test every single piece of it, pass a block there, which should evaluate to True)
If that’s all we did, and we ran our tests, we’d find they mysteriously pass, even though we didn’t yet edit our route block’s get handler to actually use the store! Why? Because it turns out I was lazy and used the data from my earlier little server example as my test data here. No worries, though: to make the test stronger, we can add a call to check-mock, and then assert that our fake store really did have the all-trees method called once, and with no arguments passed.
That just leaves us to make the test pass, by implementing the handler properly:
Time for the next test: getting a tree. There are two cases to consider here: the one where the tree is found, and the one where the tree is not found. Here’s a test for the case where a tree is found:
subtest 'Get a tree that exists'=> {
my $fake-store = mocked BestTree::Store, returning => {
find-tree => $fake-tree-b
};
test-service routes($fake-store), {
test get('/trees/50.5466504/14.8438714'),
status =>200,
json => {
latitude =>50.5466504,
longitude =>14.8438714,
height =>7.8,
description =>'Really tall and wide'
};
check-mock $fake-store,
*.called('find-tree', times =>1, with=> \(50.5466504, 14.8438714));
}
}
Running this now fails. In fact, the status code check fails first, because we didn’t implement the route yet, and so get 404 back, not the expected 200. So, here’s an implementation to make it pass:
And the tests pass, and we know our refactor is good. But wait, what about if there is no tree there? In that case, the store will return Nil. We’d like to map that into a 404. Here’s another test:
subtest 'Get a tree that does not exist'=> {
my $fake-store = mocked BestTree::Store, returning => {
find-tree =>Nil
};
test-service routes($fake-store), {
test get('/trees/50.5466504/14.8438714'),
status =>404;
check-mock $fake-store,
*.called('find-tree', times =>1, with=> \(50.5466504, 14.8438714));
}
}
Which fails, in fact, with a 500 error, since we didn’t consider that case in our route block. Happily, this one is easy to deal with: turn out given into a with, which checks we got a defined object, and then add an else and produce the 404 Not Found response.
Last but not least, let’s test the route for suggesting a new tree. Here’s the successful case:
subtest 'Suggest a tree successfully'=> {
my $fake-store = mocked BestTree::Store;
test-service routes($fake-store), {
my%body=description =>'Awesome tree', height =>4.25;
test put('/trees/50.5466504/14.8438714', json =>%body),
status =>200,
json => {
latitude =>50.5466504,
longitude =>14.8438714,
height =>4.25,
description =>'Awesome tree'
};
check-mock $fake-store,
*.called('suggest-tree', times =>1, with=>:(
BestTree::Tree $treewhere {
.latitude ==50.5466504&&.longitude ==14.8438714&&.height ==4.25&&.description eq'Awesome tree'
}
));
}
}
This is mostly familiar, except the check-mock call looks a little different this time. Test::Mock lets us test the arguments in two different ways: with a Capture (as we’ve done so far) or with a Signature. The Capture case is great for all of the simple cases, where we’re just dealing with boring values. However, once we get in to reference types, or if we don’t actually care about exact values and just want to assert the things we care about, a signature gives us the flexibility to do that. Here, we use a where clause to check that the tree object that the route handler has constructed contains the expected data.
Notice how Cro lets us use Perl 6 signatures to destructure the request body. In one line, we’ve said:
The request body must have height and description
That we want the height to be a Real number
That we want the description to be a string
Should any of those fail, Cro will automatically produce a 400 bad request for us. In fact, we can write tests to cover that – along with a new test to make sure a conflict will result in a 409.
subtest 'Problems suggesting a tree'=> {
my $fake-store = mocked BestTree::Store, computing => {
suggest-tree => {
dieX::BestTree::Store::AlreadySuggested.new;
}
}
test-service routes($fake-store), {
# Missing or bad data.
test put('/trees/50.5466504/14.8438714', json => {}),
status =>400;
my%bad-body =description =>'ok';
test put('/trees/50.5466504/14.8438714', json =>%bad-body),
status =>400;
%bad-body<height> ='grinch';
test put('/trees/50.5466504/14.8438714', json =>%bad-body),
status =>400;
# Conflict.my%body=description =>'Awesome tree', height =>4.25;
test put('/trees/50.5466504/14.8438714', json =>%body),
status =>409;
}
}
The main new thing here is that we’re using computing instead of returning with mocked. In this case, we pass a block, and it will be executed. (The block does not get the method arguments, however. If we want to get those, there is a third option, overriding, where we get to take the arguments and write a fake method body.)
And how to handle this? By making our route handler catch and map the typed exception:
With Cro::HTTP::Test, there’s now a nice way to write HTTP tests in Perl 6. Put together with a testable design, and perhaps a module like Test::Mock, we can also isolate our Cro route handlers from everything else, easing their testing.
The logic in our route handlers here is relatively straightforward; small sample problems usually are. Even here, however, I find there’s value in the journey, rather than only in the destination. The act of writing tests for a HTTP API puts me in the frame of mind of whoever will be calling the API, which can be a useful perspective to have. Experience also tells that tests “too simple to fail” do end up catching mistakes: the kinds of mistakes I might assume I’m too smart to make. Discipline goes a long way. On which note, I’ll now be disciplined about taking a break from the keyboard now and then, and go enjoy a Christmas market. -Ofun!
The year is ending and we have a lot to celebrate! What is a better way to celebrate the end of the year than with our family and friends? To help achieve that, here at my home, we decided to run a Secret Santa Game! So, my goal is to write a Secret Santa Program! That’s something where I can use this wonderful project called Red.
Red is an ORM (Object Relational Model) for perl6 still under development and not published as a module yet. But it’s growing and it is close to a release.
So let’s create our first table: a table that will store the people participating in our Secret Santa. To the code:
Red maps relational databases to OOP. Each table is mapped to a Red class (model), each of whose objects represents a row.
The way we a create a model is by using the model special word. A model is just a normal class that extends Red::Model and has a MetamodelX::Red::Model‘s object as its metaclass. Red does not add any methods you didn’t explicitly create to its models. So to interact with the database you should use the metaclass.
But let’s continue.
The code creates a new model called Person. The name of the table this model represents will be the same name as the model: “Person”. If necessary, you can change the name of the table with the is table<...>trait (for example: model Person is table<another_name> {...}).
This model has 3 attributes:
$.name have an is columntrait;
$.email have an is column{ :nullable };
and $.id has an is serial. That means the same as is column{ :id, :auto-increment }.
Red uses not null columns by default, so if you want to create a nullable columns you should use is column{ :nullable }.
So all attributes on Person are columns. The is serial (I mean the :id part) means that it’s the table’s primary key.
After that it’s setting a dynamic variable ($*RED-DB) for the result of database "SQLite". The databasesub receives the driver‘s name and the parameters it expects.
In this case it uses the SQLite driver and if you don’t pass any argument, it will use it as an in memorydatabase. If you want to use a file named secret-santa.db as the database file, you can do database "SQLite", :database<secret-santa.db>. Or, if you want to use a local Postgres, just use database "Pg". Red uses the variable $*RED-DB to know what database to use.
OK, now lets create the table! As I said before, Red did not add any methods you didn’t explicitly ask for. So, to create the table a metaclass‘ method is used. Person.^create-table is how you create the table.
This will run:
Now we should insert some data. We do that with another meta method (.^create). The .^createmeta method expects the same arguments.new expects. Each named argument will set an attribute with the same name. .^create will create a new Person object, save it in the database (with .^save: :insert), and return it.
It runs:
Every model has a ResultSeq. That is a sequence that represents every row on the table. We can get its ResultSeq with .^all (or .^rs). ResultSeq has some methods to help you to get information from the table, for example: .grep will filter the rows (as it does in a normal Seq) but it doesn’t do that in memory, it returns a new ResultSeq with that filter set. When its iterator is retrieved, it runs a SQL query using everything set on the ResultSeq.
In our example, Person.^all.grep(*.email.defined).map: *.name will run a query like:
And it’ll print:
Fernando Aline
Okay, we have a code that can save who is entered in our Secret Santa game. But each one on it want different gifts. How can we know the wishes of each one?
Let’s modify the code to make it save the wishlist for everyone participating in the secret santa:
That prints:
Fernando
Comma => https://commaide.com
perl6 books => https://perl6book.com
mac book pro => https://www.apple.com/shop/buy-mac/macbook-pro/15-inch-space-gray-2.6ghz-6-core-512gb#
Aline
a new closet => https://i.pinimg.com/474x/02/05/93/020593b34c205792a6a7fd7191333fc6--wardrobe-behind-bed-false-wall-wardrobe.jpg
Fernanda
mimikyu plush => https://www.pokemoncenter.com/mimikyu-poké-plush-%28standard-size%29---10-701-02831
camelia plush => https://farm9.static.flickr.com/8432/28947786492_80056225f3_b.jpg
Sophia
baby alive => https://www.target.com/p/baby-alive-face-paint-fairy-brunette/-/A-51304817
Now we have a new modelWishlist that refers to a table named wishlist. It has $!id as id, $!name and $!link are columns, and there are some things new! has UInt $!wisher-id is referencing{ Person.id }; is the same as has UInt $!wisher-id is column{ :references{ Person.id } }; that means it’s a column that’s a foreign key that references the idPerson‘s column. It also has a has Person $.wisher is relationship{ .wisher-id }; it’s not a column, it’s a “virtual” field. the $sigil means that there is only 1 wisher for a wish. And is relationship expects a Callable that will receive a model. If it’s Scalar it will receive the current model as the only argument. So, in this case, it will be Wishlist. The return of the relationsip’s Callable must be a column that references some other column.
Lets see how this table is created:
As you can see, no wisher column is created.
The Personmodel has changed too! Now it has a @.wishesrelationship (has Wishlist @.wishes is relationship{ .wisher-id }). It uses a @sigil so each Person can have more than one wish. The Callable passed will receive the type of the Positionalattribute (Wishlist on this case) and must return a column that references some other column.
The table created is the same as before.
We created a new Person as we did before: my \fernando = Person.^create: :name<Fernando>, :email<[email protected]>; and now we can use the relationship (wishes) to create a new wish (fernando.wishes.create: :name<Comma>, :link<https://commaide.com>). That creates a new wish for Fernando running the following SQL:
Had you seen? wisher_id is 1… 1 is Fernando’s id. Once you have created the wish from Fernando’s .wishes(), it already knows that it belongs to Fernando.
And then we define wishes for every person we create.
Then we loop over every Person in the database (Person.^all) and print its name and loop over that person’s wishes and print its name and link.
Okey, we can save who is participating… get what they want… but the draw? Who should I give a gift to? To do that we change our program again:
Now Person has two new attributes ($!pair-id and $.pair) and a new method (draw). $!pair-id is a foreign key that references the field id on the same table (Person) so we have to use an alias (.^alias). The other one is the relationship ($.pair) that uses that foreign key.
The new method (draw) is where the magic happens. It uses the method .pick: * that on a normal Positional would shuffle the list. And it does the same here, with the query:
Once we have the shuffled list, we use .rotor to get two items and go one back, so we save what is the pair of each person giving to the next person, and the last person in the list will give to the first person.
And this is the output of our final code:
Fernando -> Sophia Wishlist: baby alive Aline -> Fernanda Wishlist: mimikyu plush, camelia plush Fernanda -> Fernando Wishlist: COMMA, perl6 books, mac book pro Sophia -> Aline Wishlist: a new closet
As a bonus, let’s check out the track Red is going to follow. This is a current working code:
For the first phase, development of tasks, I am looking for volunteers who
come up with coding tasks collaboratively. Sadly, these volunteers, including
myself, will be excluded from participating in the second phase.
I am looking for tasks that ...
are well-worded
have a (private) implementation that shows it is solvable, preferably
in under 500 lines of code
have a few public tests to help the participants, and a more tests
tests that help the graders and commenters later on
This is non-trivial, so I'd like to have others to discuss things with,
and to come up with some more tasks.
If you want to help with task creation, please send an email to [email protected], stating your intentions to help, and your freenode IRC handle (optional).
There are other ways to help too:
You can pledge a prize (some ideas: Comma IDE subscription,
books, Camelia plushies, Amazon vouchers)
You can help design a small website for the contest
You can help iron out the exact rules for the contest
... or any other way that I didn't think of :-)
In these cases you can use the same email address to contact me,
or use IRC (moritz on freenode) or twitter.
Recently, a Perl 6 object creation benchmark result did the rounds on social media. This Perl 6 code:
class Point {
has $.x;
has $.y;
}
my $total = 0;
for ^1_000_000 {
my $p = Point.new(x => 2, y => 3);
$total = $total + $p.x + $p.y;
}
say $total;
Now (on HEAD builds of Rakudo and MoarVM) runs faster than this roughly equivalent Perl 5 code:
use v5.10;
package Point;
sub new {
my ($class, %args) = @_;
bless \%args, $class;
}
sub x {
my $self = shift;
$self->{x}
}
sub y {
my $self = shift;
$self->{y}
}
package main;
my $total = 0;
for (1..1_000_000) {
my $p = Point->new(x => 2, y => 3);
$total = $total + $p->x + $p->y;
}
say $total;
(Aside: yes, I know there’s no shortage of libraries in Perl 5 that make OO nicer than this, though those I tried also made it slower.)
This is a fairly encouraging result: object creation, method calls, and attribute access are the operational bread and butter of OO programming, so it’s a pleasing sign that Perl 6 on Rakudo/MoarVM is getting increasingly speedy at them. In fact, it’s probably a bit better at them than this benchmark’s raw numbers show, since:
The measurements include startup time. Probably 10%-15% of the time Rakudo spends executing the program is startup, parsing, compilation, etc. (noting the compiler is itself written in Perl 6). By contrast, Perl 5 has really fast startup and a parser written in C.
The math in Perl 6 is arbitrary precision (that is, will upgrade to a big integer if needed); it costs something to handle that, even if it doesn’t happen
Every math operation allocates a new Int object; there’s two uses of + here, so that means two new Int allocations per iteration of the loop, which then have to be garbage collected
While dealing with Int values got faster recently, it’s still really making two Int objects every time around that loop and having to GC them later. An upcoming new set of analyses and optimizations will let us get rid of that cost too. And yes, startup will get faster with time also. In summary, while Rakudo/MoarVM is now winning that benchmark against Perl 5, there’s still lots more to be had. Which is a good job, since the equivalent Python and Ruby versions of that benchmark still run faster than the Perl 6 one.
In this post, I’ll look at the changes that allowed this benchmark to end up faster. None of the new work was particularly ground-breaking; in fact, it was mostly a case of doing small things to make better use of optimizations we already had.
What happens during construction?
Theoretically, the default new method in turn calls bless, passing the named arguments along. The bless method then creates an object instance, followed by calling BUILDALL. The BUILDALL method goes through the set of steps needed for constructing the object. In the case of a simple object like ours, that involves checking if an x and y named argument were passed, and if so assigning those values into the Scalar containers of the object attributes.
For those keeping count, that’s at least 3 method calls (new, bless, and BUILDALL).
However, there’s a cheat. If bless wasn’t overridden (which would be an odd thing to do anyway), then the default new could just call BUILDALL directly anyway. Therefore, new looks like this:
The BUILDALL method was originally a little “interpreter” that went through a per-object build plan stating what needs to be done. However, for quite some time now we’ve instead compiled a per-class BUILDPLAN method.
Slurpy sadness
To figure out how to speed this up, I took a look at how the specializer was handling the code. The answer: not so well. There were certainly some positive moments in there. Of note:
The x and y accessor methods were being inlined
The + operators were being inlined
Everything was JIT-compiled into machine code
However, the new method was getting only a “certain” specialization, which is usually only done for very heavily polymorphic code. That wasn’t the case here; this program clearly constructs overwhelmingly one type of object. So what was going on?
In order to produce an “observed type” specialization – the more powerful kind – it needs to have data on the types of all of the passed arguments. And for the named arguments, that was missing. But why?
Logging of passed argument types is done on the callee side, since:
If the callee is already specialized, then we don’t want to waste time in the caller doing throwaway logging
If the callee is not specialized, but the caller is, then we don’t want to miss out on logging types (and we only log when running unspecialized code)
The argument logging was done as the interpreter executed each parameter processing instruction. However, when there’s a slurpy, then it would just swallow up all the remaining arguments without logging type information. Thus the information about the argument types was missing and we ended up with a less powerful form of specialization.
Teaching the slurpy handling code about argument logging felt a bit involved, but then I realized there was a far simpler solution: log all of the things in the argument buffer at the point an unspecialized frame is being invoked. We’re already logging the entry to the call frame at that point anyway. This meant that all of the parameter handling instructions got a bit simpler too, since they no longer had logging to do.
Conditional elimination
Having new being specialized in a more significant way was an immediate win. Of note, this part:
Was quite interesting. Since we were now specializing on the type of self, then the findmethod could be resolved into a constant. The resolution of a method on the constant Mu was also a constant. Therefore, the result of the eqaddr (same memory address) comparison of two constants should also have been turned into a constant…except that wasn’t happening! This time, it was simple: we just hadn’t implemented folding of that yet. So, that was an easy win, and once done meant the optimizer could see that the if would always go a certain way and thus optimize the whole thing into the chosen branch. Thus the new method was specialized into something like:
multi method new(*%attrinit) {
nqp::create(self).BUILDALL(Empty, %attrinit),
}
Further, the create could be optimized into a “fast create” op, and the relatively small BUILDALL method could be inlined into new. Not bad.
Generating simpler code
At this point, things were much better, but still not quite there. I took a look at the BUILDALL method compilation, and realized that it could emit faster code.
The %attrinit is a Perl 6 Hash object, which is for the most part a wrapper around the lower-level VM hash object, which is the actual hash storage. We were, curiously, already pulling this lower-level hash out of the Hash object and using the nqp::existskey primitive to check if there was a value, but were not doing that for actually accessing the value. Instead, an .AT-KEY('x') method call was being done. While that was being handled fairly well – inlined and so forth – it also does its own bit of existence checking. I realized we could just use the nqp::atkey primitive here instead.
Later on, I also realized that we could do away with nqp::existskey and just use nqp::atkey. Since a VM-level null is something that never exists in Perl 6, we can safely use it as a sentinel to mean that there is no value. That got us down to a single hash lookup.
By this point, we were just about winning the benchmark, but only by a few percent. Were there any more easy pickings?
An off-by-one
My next surprise was that the new method didn’t get inlined. It was within the size limit. There was nothing preventing it. What was going on?
Looking closer, it was even worse than that. Normally, when something is too big to be inlined, but we can work out what specialization will be called on the target, we do “specialization linking”, indicating which specialization of the code to call. That wasn’t happening. But why?
Some debugging later, I sheepishly fixed an off-by-one in the code that looks through a multi-dispatch cache to see if a particular candidate matches the set of argument types we have during optimization of a call instruction. This probably increased the performance of quite a few calls involving passing named arguments to multi-methods – and meant new was also inlined, putting us a good bit ahead on this benchmark.
What next?
The next round of performance improvements – both to this code and plenty more besides – will come from escape analysis, scalar replacement, and related optimizations. I’ve already started on that work, though expect it will keep me busy for quite a while. I will, however, be able to deliver it in stages, and am optimistic I’ll have the first stage of it ready to talk about – and maybe even merge – in a week or so.
Hi hackers. Today I want to write about a new functionality that I've been developing for MoarVM that has very little to do with the JIT compiler. But it is still about VM internals so I guess it will fit.
Many months ago, jnthn wrote a blog post on the relation between perl 5 and perl 6. And as a longtime and enthusiastic perl 5 user - most of the JIT's compile time support software is written in perl 5 for a reason - I wholeheartedly agree with the 'sister language' narrative. There is plenty of room for all sorts of perls yet, I hope. Yet one thing kept itching me:
Moreover, it’s very much the case that Perl 5 and Perl 6 make different trade-offs. To pick one concrete example, Perl 6 makes it easy to run code across multiple threads, and even uses multiple threads internally (for example, performing optimization and JIT compilation on a background thread). Which is great…except the only winning move in a game involving both threads and fork() is not to play. Sometimes one just can’t have their cake and eat it, and if you’re wanting a language that more directly gives you your POSIX, that’s probably always going to be a strength of Perl 5 over Perl 6.
(Emphasis mine). You see, I had never realized that MoarVM couldn't implement fork(), but it's true. In POSIX systems, a fork()'d child process inherits the full memory space, as-is, from its parent process. But it inherits only the forking thread. This means that any operations performed by any other thread, including operations that might need to be protected by a mutex (e.g. malloc()), will be interrupted and unfinished (in the child process). This can be a problem. Or, in the words of the linux manual page on the subject:
* The child process is created with a single thread—the one that called fork(). The entire virtual address space of the parent is replicated in the child, including the states of mutexes, condition variables, and other pthreads objects; the use of pthread_atfork(3) may be helpful for dealing with problems that this can cause.
* After a fork() in a multithreaded program, the child can safely call only async-signal-safe functions (see signal-safety(7)) until such time as it calls execve(2).
Note that the set of signal-safe functions is not that large, and excludes all memory management functions. As noted by jnthn, MoarVM is inherently multithreaded from startup, and will happily spawn as many threads as needed by the program. It also uses malloc() and friends rather a lot. So it would seem that perl 6 cannot implement POSIX fork() (in which both parent and child program continue from the same position in the program) as well as perl 5 can.
I was disappointed. As a longtime (and enthusiastic) perl 5 user, fork() is one of my favorite concurrency tools. Its best feature is that parent and child processes are isolated by the operating system, so each can modify its own internal state without concern for concurrency,. This can make it practical to introduce concurrency after development, rather than designing it in from the start. Either process can crash while the other continues. It is also possible (and common) to run a child process with reduced privileges relative to the parent process, which isn't possible with threads. And it is possible to prepare arbitrarily complex state for the child process, unlike spawn() and similar calls.
Fortunately all hope isn't necessarily lost. The restrictions listed above only apply if there are multiple threads active at the moment that fork() is executed. That means that if we can stop all threads (except for the one planning to fork) before actually calling fork(), then the child process can continue safely. That is well within the realm of possibility, at least as far as system threads are concerned.
The process itself isn't very exciting to talk about, actually - it involves sending stop signals to system threads, waiting for these threads to join, verifying that the forking thread is the really only active thread, and restarting threads after fork(). Because of locking, it is a bit subtle (tbere may be another user thread that is also trying to fork), but not actually very hard. When I finally merged the code, it turned out that an ancient (and forgotten) thread list modification function was corrupting the list by not being aware of generational garbage collection... oops. But that was simple enough to fix (thanks to timotimo++ for the hint).
And now the following oneliner should work on platforms that support fork():(using development versions of MoarVM and Rakudo):
perl6 -e 'use nqp; my $i = nqp::fork(); say $i;'
The main limitation of this work is that it won't work if there are any user threads active. (If there are any, nqp::fork() will throw an exception). The reason why is simple: while it is possible to adapt the system threads so that I can stop them on demand, user threads may be doing arbitrary work, hold arbitrary locks and may be blocked (possibly indefinitely) on a system call. So jnthn's comment at the start of this post still applies - threads and fork() don't work together.
In fact, many programs may be better off with threads. But I think that languages in the perl family should let the user make that decision, rather than the VM. So I hope that this will find some use somewhere. If not, it was certainly fun to figure out. Happy hacking!
PS: For the curious, I think there may in fact be a way to make fork() work under a multithreaded program, and it relies on the fact that MoarVM has a multithreaded garbage collector. Basically, stopping all threads before calling fork() is not so different from stopping the world during the final phase of garbage collection. And so - in theory - it would be possible to hijack the synchronization mechanism of the garbage collector to pause all threads. During interpretation, and in JIT compiled code, each thread periodically checks if garbage collection has started. If it has, it will synchronize with the thread that started GC in order to share the work. Threads that are currently blocked (on a system call, or on acquiring a lock) mark themselves as such, and when they are resumed always check the GC status. So it is in fact possible to force MoarVM into a single active thread even with multiple user threads active. However, that still leaves cleanup to deal with, after returning from fork() in the child process. Also, this will not work if a thread is blocked on NativeCall. Altogether I think abusing the GC in order to enable a fork() may be a bit over the edge of insanity :-)
MoarVM’s optimizer can perform speculative optimization. It does this by gathering statistics as the program is interpreted, and then analyzing them to find out what types and callees typically show up at given points in the program. If it spots there is at least a 99% chance of a particular type showing up at a particular program point, then it will optimize the code ahead of that point as if that type would always show up.
Of course, statistics aren’t proofs. What about the 1% case? To handle this, a guard instruction is inserted. This cheaply checks if the type is the expected one, and if it isn’t, falls back to the interpreter. This process is known as deoptimization.
Just how cheap are guards?
I just stated that a guard cheaply checks if the type is the expected one, but just how cheap is it really? There’s both direct and indirect costs.
The direct cost is that of the check. Here’s a (slightly simplified) version of the JIT compiler code that produces the machine code for a type guard.
/* Load object that we should guard */
| mov TMP1, WORK[obj];
/* Get type table we expect and compare it with the object's one */
MVMint16 spesh_idx = guard->ins->operands[2].lit_i16;
| get_spesh_slot TMP2, spesh_idx;
| cmp TMP2, OBJECT:TMP1->st;
| jne >1;
/* We're good, no need to deopt */
| jmp >2;
|1:
/* Call deoptimization handler */
| mov ARG1, TC;
| mov ARG2, guard->deopt_offset;
| mov ARG3, guard->deopt_target;
| callp &MVM_spesh_deopt_one_direct;
/* Jump out to the interpreter */
| jmp ->exit;
|2:
So, in the case that the guard matches, it’s 7 machine instructions (note: it’s actually a couple more because of something I omitted for simplicity). Thus there’s the cost of the time to execute them, plus the space they take in memory and, especially, the instruction cache. Further, one is a conditional jump. We’d expect it to be false most of the time, and so the CPU’s branch predictor should get a good hit rate – but branch predictor usage isn’t entirely free of charge either. Effectively, it’s not that bad, but it’s nice to save the cost if we can.
The indirect costs are much harder to quantify. In order to deoptimize, we need to have enough state to recreate the world as the interpreter expects it to be. I wrote on this topic not so long ago, for those who want to dive into the detail, but the essence of the problem is that we may have to retain some instructions and/or forgo some optimizations so that we are able to successfully deoptimize if needed. Thus, the presence of a guard constrains what optimizations we can perform in the code around it.
Representation problems
A guard instruction in MoarVM originally looked like:
sp_guard r(obj) sslot uint32
Where r(obj) is an object register to read containing the object to guard, the sslot is a spesh slot (an entry in a per-block constant table) containing the type we expect to see, and the uint32 indicates the target address after we deoptimize. Guards are inserted after instructions for which we had gathered statistics and determined there was a stable type. Things guarded include return values after a call, reads of object attributes, and reads of lexical variables.
This design has carried us a long way, however it has a major shortcoming. The program is represented in SSA form. Thus, an invoke followed by a guard might look something like:
Where r6(5) has the return value written into it (and thus is a new SSA version of r6). We hold facts about a value (if it has a known type, if it has a known value, etc.) per SSA version. So the facts about r6(5) would be that it has a known type – the one that is asserted by the guard.
The invoke itself, however, might be optimized by performing inlining of the callee. In some cases, we might then know the type of result that the inlinee produces – either because there was a guard inside of the inlined code, or because we can actually prove the return type! However, since the facts about r6(5) were those produced by the guard, there was no way to talk about what we know of r6(5) before the guard and after the guard.
More awkwardly, while in the early days of the specializer we only ever put guards immediately after the instructions that read values, more recent additions might insert them at a distance (for example, in speculative call optimizations and around spesh plugins). In this case, we could not safely set facts on the guarded register, because those might lead to wrong optimizations being done prior to the guard.
That is to say, it introduces a new SSA version. This means that we get a way to talk about the value both before and after the guard instruction. Thus, if we perform an inlining and we know exactly what type it will return, then that type information will flow into the input – in our example, r6(5) – of the guard instruction. We can then notice that the property the guard wants to assert is already upheld, and replace the guard with a simple set (which may itself be eliminated by later optimizations).
This also solves the problem with guards inserted away from the original write of the value: we get a new SSA version beyond the guard point. This in turn leads to more opportunities to avoid repeated guards beyond that point.
Quite a lot of return value guards on common operations simply go away thanks to these changes. For example, in $a + $b, where $a and $b are Int, we will be able to inline the + operator, and we can statically see from its code that it will produce an Int. Thus, the guard on the return type in the caller of the operator can be eliminated. This saves the instructions associated with the guard, and potentially allows for further optimizations to take place since we know we’ll never deoptimize at that point.
In summary
MoarVM does lots of speculative optimization. This enables us to optimize in cases where we can’t prove a property of the program, but statistics tell us that it mostly behaves in a certain way. We make this safe by adding guards, and falling back to the general version of the code in cases where they fail.
However, guards have a cost. By changing our representation of them, so that we model the data coming into the guard and after the guard as two different SSA versions, we are able to eliminate many guard instructions. This not only reduces duplicate guards, but also allows for elimination of guards when the broader view afforded by inlining lets us prove properties that we weren’t previously able to.
In fact, upcoming work on escape analysis and scalar replacement will allow us to start seeing into currently opaque structures, such as Scalar containers. When we are able to do that, then we’ll be able to prove further program properties, leading to the elimination of yet more guards. Thus, this work is not only immediately useful, but also will help us better exploit upcoming optimizations.
My work on Perl 6 performance continues, thanks to a renewal of my grant from The Perl Foundation. I’m especially focusing on making common basic operations faster, the idea being that if those go faster than programs composed out of them also should. This appears to be working out well: I’ve not been directly trying to make the Text::CSV benchmark run faster, but that’s resulted from my work.
I’ll be writing a few posts in on various of the changes I’ve done. This one will take a look at some related optimizations around boxing, unboxing, and common mathematical operations on Int.
Boxing and unboxing
Boxing is taking a natively typed value and wrapping it into an object. Unboxing is the opposite: taking an object that wraps a native value and getting the native value out of it.
In Perl 6, these are extremely common. Num and Str are boxes around num and str respectively. Thus, unless dealing with natives explicitly, working with floating point numbers and strings will involve lots of box and unbox operations.
There’s nothing particularly special about Num and Str. They are normal objects with the P6opaque representation, meaning they can be mixed into. The only thing that makes them slightly special is that they have attributes marked as being a box target. This indicates the attribute out as being the one to write to or read from in a box or unbox operation.
Thus, a box operations is something like:
Create an instance of the box type
Find out where in that object to write the value to
Write the value there
While unbox is:
Find out where in the object to read a value from
Read it from there
Specialization of box and unbox
A box is actually two operations: an allocation and a store. We know how to fast-path allocations and JIT them relatively compactly, however that wasn’t being done for box. So, step one was to decompose this higher-level op into the allocation and the write into the allocated object. The first step could then be optimized in the usual way allocations are.
In the unspecialized path, we first find out where to write the native value to, and then write it. However, when we’re producing a specialized version, we almost always know the type we’re boxing into. Therefore, the object offset to write to can be calculated once, and a very simple instruction to do a write at an offset into the object produced. This JITs extremely well.
There are a couple of other tricks. Binds into a P6opaque generally have to check that the object wasn’t mixed in to, however since we just allocated it then we know that can’t be the case and can skip that check. Also, a string is a garbage-collectable object, and when assigning one GC-able object into another one, we need to perform a write barrier for the sake of generational GC. However, since the object was just allocated, we know very well that it is in the nursery, and so the write barrier will never trigger. Thus, we can omit it.
Unboxing is easier to specialize: just calculate the offset, and emit a simpler instruction to load the value from there.
I’ve also started some early work (quite a long way from merge) on escape analysis, which will allow us to eliminate many box object allocations entirely. It’s a great deal easier to implement this if allocations, reads, and writes to an object have a uniform representation. By lowering box and unbox operations into these lower level operations, this eases the path to implementing escape analysis for them.
What about Int?
Some readers might have wondered why I talked about Num and Str as examples of boxed types, but not Int. It is one too – but there’s a twist. Actually, there’s two twists.
The first is that Int isn’t actually a wrapper around an int, but rather an arbitrary precision integer. When we first implemented Int, we had it always use a big integer library. Of course, this is slow, so later on we made it so any number fitting into a 32-bit range would be stored directly, and only allocate a big integer structure if it’s outside of this range.
Thus, boxing to a big integer means range-checking the value to box. If it fits into the 32-bit range, then we can write it directly, and set the flag indicating that it’s a small Int. Machine code to perform these steps is now spat out directly by the JIT, and we only fall back to a function call in the case where we need a big integer. Once again, the allocation itself is emitted in a more specialized way too, and the offset to write to is determined once at specialization time.
Unboxing is similar. Provided we’re specializing on the type of the object to unbox, then we calculate the offset at specialization time. Then, the JIT produces code to check if the small Int flag is set, and if so just reads and sign extends the value into a 64-bit register. Otherwise, it falls back to the function call to handle the real big integer case.
For boxing, however, there was a second twist: we have a boxed integer cache, so for small integers we don’t have to repeatedly allocate objects boxing them. So boxing an integer is actually:
Check if it’s in the range of the box cache
If so, return it from the cache
Otherwise, do the normal boxing operation
When I first did these changes, I omitted the use of the box cache. It turns out, however, to have quite an impact in some programs: one benchmark I was looking at suffered quite a bit from the missing box cache, since it now had to do a lot more garbage collection runs.
So, I reinstated use of the cache, but this time with the JIT doing the range checks in the produced machine code and reading directly out of the cache in the case of a hit. Thus, in the cache hit case, we now don’t even make a single function call for the box operation.
Faster Int operations
One might wonder why we picked 32-bit integers as the limit for the small case of a big integer, and not 64-bit integers. There’s two reasons. The most immediate is that we can then use the 32 bits that would be the lower 32 of a 64-bit pointer to the big integer structure as our “this is a small integer” flag. This works reliably as pointers are always aligned to at least a 4-byte boundary, so a real pointer to a big integer structure would never have the lowest bits set. (And yes, on big-endian platforms, we swap the order of the flag and the value to account for that!)
The second reason is that there’s no portable way in C to detect if a calculation overflowed. However, out of the basic math operations, if we have two inputs that fit into a 32-bit integer, and we do them at 64-bit width, we know that the result can never overflow the 64-bit integer. Thus we can then range check the result and decide whether to store it back into the result object as 32-bit, or to store it as a big integer.
Since Int is immutable, all operations result in a newly allocated object. This allocation – you’ll spot a pattern by now – is open to being specialized. Once again, finding the boxed value to operate on can also be specialized, by calculating its offset into the input objects and result object. So far, so familiar.
However, there’s a further opportunity for improvement if we are JIT-compiling the operations to machine code: the CPU has flags for if the last operation overflowed, and we can get at them. Thus, for two small Int inputs, we can simply:
Grab the values
Do the calculation at 32-bit width
Check the flags, and store it into the result object if no overflow
If it overflowed, fall back to doing it wider and storing it as a real big integer
I’ve done this for addition, subtraction, and multiplication. Those looking for a MoarVM specializer/JIT task might like to consider doing it for some of the other operations. :-)
In summary
Boxing, unboxing, and math on Int all came with various indirections for the sake of generality (coping with mixins, subclassing, and things like IntStr). However, when we are producing type-specialized code, we can get rid of most of the indirections, resulting in being able to perform them faster. Further, when we JIT-compile the optimized result into machine code, we can take some further opportunities, reducing function calls into C code as well as taking advantage of access to the overflow flags.
Encodings. They may seem to some as horribly boring and bland.
Nobody really wants to have to worry about the details.
Encodings are how we communicate, transmit data.
Their purpose is to be understood by others.
When they work, nobody should even know they
are there. When they don’t — and everyone can
probably remember a time when they have tried to open
a poorly encoded or corrupted file and been disappointed — they cannot be ignored.
Here I talk about the work I have done in the past
and work still in progress to improve the support
of current encodings, add new encodings, and add
new features and new options to allow Perl 6 and MoarVM to support more encodings
and in a way which better achieves the goals encodings were designed to solve.
I started looking at the most common encodings on the Web.
We supported the two most common, UTF-8 and ISO-8859-1
but we did not support windows-1251 (Cyrillic) or ShiftJIS.
This seemed like the makings of a new project so I got to work.
I decided to start with windows-1251 since it was an 8 bit encoding while
Shift-JIS was a variable length one or two byte encoding (the shift in the name comes from
the second byte 'shift'). While the encoding itself was simpler than
Shift-JIS, I was soon realizing some issues with both windows-1251 and
our already supported windows-1252 encoding.
One of the first major issues I found in windows-1252,
you could create files which could not be decoded by other programs. An example of this is
codepoint 0x81 (129) which does not have a mapping in the windows-1252 specification.
This would cause many programs to not detect the file as
windows-1252, or to fail saying the file was corrupted and could not be decoded.
While our non-8bit encodings would throw an error if asked to encode
text which did not exist in that codemapping, our 8 bit encodings would
happily pass through invalid codepoints, as long as they fit in 8 bits.
As I said at the start, encodings are a way for us to communicate, to exchange data
and to be understood. This to me indicated a major failing. While the solution could be
to never attempt to write codepoints that don’t exist in the target encoding, that was
not an elegant solution and would just be a workaround.
To remedy this I had to add new ops, such as decodeconf to replace decode op,
and encodeconf to replace encode (plus many others). This would allow us to
specify a configuration for our encodings and allow Perl 6 to tell
MoarVM if it wanted to encode strictly according to the standard or
to be more lenient.
I added a new :strict option to open, encode and a few others to allow you
to specify if it should be encoding strict or not strict. In the needs of
backwards compatibility it still defaults to leniently encoding and decoding.
Strict is planned to become the default for Perl 6 specification 6.e.
Replacements
Some of our other encodings had support for 'replacements'. If you
tried to encode something that would not fit in the target encoding,
this allowed you to supply a string that could be one or more
characters which would but substituted instead of causing MoarVM to throw an
exception.
Once I had the strictness nailed down I was able to add support for replacements
so one could have the ability to write data in a strict mode while still ensuring
all compatible characters would get written properly, and you did not have to choose
between unstrict and writing incompatible files and strict and having the encoding or
file write failing when you really needed it not to.
Shift-JIS
Encodings are not without difficulty. As with the previous encodings I talked about,
Shift-JIS would not be without decisions that had to be made. With Shift-JIS, the
question became, "which Shift-JIS?".
You see, there are dozens of different extensions to Shift-JIS, in addition to
original Shift-JIS encoding. As a variable length encoding, most of the one byte
codepoints had been taken while there were many hundreds of open and unallocated
codepoints in the two byte range. This saw the wide proliferation of manufacturer
specific extensions which other manufacturers may adopt while adding their own
extensions to the format.
Which of the dozens of different Shift-JIS encodings should we support? I eventually decided on
windows-932 because that is the standard that is used by browsers
when they encounter Shift-JIS text and encompasses a great many symbols. It is the
most widely used Shift-JIS format. This would allow us to support the majority of
Shift-JIS encoded documents out there, and the one which web browsers use on the web.
Most but not all the characters in it are compatible with the other extensions, so
it seemed like it was the sensible choice.
The only notable exception to this is that windows-932 is not totally compatible with the
original Shift-JIS encoding. While the original windows-932 encoding (and some extensions to it)
map the ASCII’s backslash to the yen symbol ¥, windows-932 keeps it to the backslash. This was to retain
better compatibility with ASCII and other formats.
UTF-16
While MoarVM had a UTF-16 encoder and decoder, it was not fully featured.
That was about it. You couldn’t open a filehandle as UTF-16
and write to it. You couldn’t even do $fh.write: "hello".encode('utf16')
as the write function did not know how to deal with writing a 16 bit buffer
(it expected an 8-bit buffer).
In addition there was no streaming UTF-16 decoder, so there was no way to read a
UTF-16 file. So I set out to work, first allowing us to write 16-bit buffers to
a file and then being able to .print a filehandle and write text to it.
At this point I knew that I would have to confront the BOM in the room, but decided
to first implement the streaming decoder and ensure all of our file handle operations
worked.
The BOM in the Room
You may be noticing a theme here. You do not just implement an encoding. It
does not exist in a vacuum. While there may be standards, there may be multiple
standards. And those standards may or may not be followed in practice or it may
not be totally straightforward.
Endianess defines which order the bytes are in a number. Big endian machines
will store a 16-bit number with the larger section first, similar to how we write
numbers, while little endian machines will put the smallest section first.
This only matters for encoding numbers that are more than one byte. UTF-16
can be either big endian or little endian. Meaning big and little endian
files will contain the same bytes, but the order is different.
Since swapping the two bytes of a 16-bit integer causes a different integer
instead of an invalid one, the creators of UTF-16 knew it was not possible to determine with certainty
the endianess of a 16 bit number. And so the byte order
mark was created, a codepoint that would be added at the start of a file to
signify which endianess the file was. Since they didn’t want to break already
existing software, they used a "Zero width no-break space" (ZWNBSP) and designated that
a ZWNBSP with bytes reversed would be invalid in UTF-16, a "non-character".
If the BOM was not removed, it would not be visible since it is zero width.
If your program opens a file and sees a byte order mark it knew it was in the correct endianess.
If it’s the "non-character" then it knew it had to swap the bytes when it read the file.
The standard states that utf16 should always use a BOM and that when reading
a file as utf16 the BOM should be used to detect the endianess. It states the BOM is not passed
through as it is assumed to not be part of the actual data. If it doesn’t exist then
the system is supposed to decode as the endianess that the context suggests.
So you get a format where you may lose the first codepoint
of your file if that codepoint happens to be a zero width non-break space.
For utf16be (big endian) and utf16le (little endian) the standard states the byte order mark should
not exist, and that any zero width no-break space at the start of a file is just
that, and should be passed through.
Standards and practice are not identical and in practice many programs will remove
a leading zero width non-break space even if set explicitly to utf16be or utf16le.
I was unsure which route I should take.
I looked at how Perl 5 handled it and after thinking for a long time I decided
that we would follow the spec. Even though other software will get rid of any
detected byte order mark, I think it’s important that if a user specifies utf16be
or utf16le, they will always get back the same codepoints that they write to a file.
Current Work
I am currently working on adding support so that when you write a file in 'utf16'
it will add a byte order mark, as utf16 is supposed to always have one. Until this
is added you can write a file with Perl 6 on one computer
using the 'utf16' encoding which will not decode correctly using the same 'utf16' setting
if the two computers are of different endianess.
Since there was no functionality for writing utf-16 to a file previously, there
should be no backwards compatibility issues to worry about there and we can set
it to add a byte order mark if the utf16 encoder is used.
Release 2018.09
In the last release utf16, utf16le and utf16be should work, aside from utf16 not
adding a byte order mark on file write. Anyone who uses utf16 and finds any issues
or comments are free to email me or make an issue on the MoarVM or Rakudo issue
trackers.
What’s the most common type your Perl 6 code uses? I’ll bet you that in most programs you write, it’ll be Scalar. That might come as a surprise, because you pretty much never write Scalar in your code. But in:
my $a = 41;
my $b = $a + 1;
Then both $a and $b point to Scalar containers. These in turn hold the Int objects. Contrast it with:
my $a := 42;
my $b := $a + 1;
Where there are no Scalar containers. Assignment in Perl 6 is an operation on a container. Exactly what it does depending on the type of the container. With an Array, for example, it iterates the data source being assigned, and stores each value into the target Array. Assignment is therefore a copying operation, unlike binding which is a referencing operation. Making assignment the shorter thing to type makes it more attractive, and having the more attractive thing decrease the risk of action at a distance is generally a good thing.
Having Scalar be first-class is used in a number of features:
Lazy vivification, so if %a{$x} { ... } will not initialize the hash slot in question, but %a{$x} = 42 will do so (this also works many levels deep)
The is rw trait on parameters being able to work together with late-bound dispatch
Making l-value routines possible, including every is rw accessor
List assignment
Using meta-ops on assignment, for example Z=
And probably some more that I forgot. It’s powerful. It’s also torture for those of us building Perl 6 implementations and trying to make them run fast. The frustration isn’t so much the immediate cost of the allocating all of those Scalar objects – that of course costs something, but modern GC algorithms can throw away short-lived objects pretty quickly – but also because of the difficulties it introduces for program analysis.
Despite all the nice SSA-based analysis we do, tracking the contents of Scalar containers is currently beyond that. Rather than any kind of reasoning to prove properties about what a Scalar holds, we instead handle it through statistics, guards, and deoptimization at the point that we fetch a value from a Scalar. This still lets us do quite a lot, but it’s certainly not ideal. Guards are cheap, but not free.
Looking ahead
Over the course of my current grant from The Perl Foundation, I’ve been working out a roadmap for doing better with optimization in the presence of Scalar containers. Their presence is one of the major differences between full Perl 6 and the restricted NQP (Not Quite Perl), and plays a notable part in the performance difference between the two.
I’ve taken the first big step towards improving this situation by significantly re-working the way Scalar containers are handled. I’ll talk about that in this post, but first I’d like to provide an idea of the overall direction.
In the early days of MoarVM, when we didn’t have specialization or compilation to machine code, it made sense to do various bits of special-casing of Scalar. As part of that, we wrote code handling common container operations in C. We’ve by now reached a point where the C code that used to be a nice win is preventing us from performing the analyses we need in order to do better optimizations. At the end of the day, a Scalar container is just a normal object with an attribute $!value that holds its value. Making all operations dealing with Scalar container really be nothing more than some attribute lookups and binds would allow us to solve the problem in terms of more general analyses, which stand to benefit many other cases where programs use short-lived objects.
The significant new piece of analysis we’ll want to do is escape analysis, which tells us which objects have a lifetime bounded to the current routine. We understand “current routine” to incorporate those that we have inlined.
If we know that an object’s usage lies entirely within the current routine, we can then perform an optimization known as scalar replacement, which funnily enough has nothing much to do with Scalar in the Perl 6 sense, even if it solves the problems we’re aiming to solve with Scalar! The idea is that we allocate a local variable inside of the current frame for each attribute of the object. This means that we can then analyze them like we analyze other local variables, subject them to SSA, and so forth. This for one gets rid of the allocation of the object, but also lets us replace attribute lookups and binds with a level of indirection less. It will also let us reason about the contents of the once-attributes, so that we can eliminate guards that we previously inserted because we only had statistics, not proofs.
So, that’s the direction of travel, but first, Scalar and various operations around it needed to change.
Data structure redesign
Prior to my recent work, a Scalar looked something like:
class Scalar {
has $!value; # The value in the Scalar
has $!descriptor; # rw-ness, type constraint, name
has $!whence; # Auto-vivification closure
}
The $!descriptor held the static information about the Scalar container, so we didn’t have to hold it in every Scalar (we usually have many instances of the same “variable” over a programs lifetime).
The $!whence was used when we wanted to do some kind of auto-vivification. The closure attached to it was invoked when the Scalar was assigned to, and then cleared afterwards. In an array, for example, the callback would bind the Scalar into the array storage, so that element – if assigned to – would start to exist in the array. There are various other forms of auto-vivification, but they all work in roughly the same way.
This works, but closures aren’t so easy for the optimizer to deal with (in short, a closure has to have an outer frame to point to, and so we can’t inline a frame that takes a closure). Probably some day we’ll find a clever solution to that, but since auto-vivification is an internal mechanism, we may as well make it one that we can see a path to making efficient in the near term future.
So, I set about considering alternatives. I realized that I wanted to replace the $!whence closure with some kind of object. Different types of object would do different kinds of vivification. This would work very well with the new spesh plugin mechanism, where we can build up a set of guards on objects. It also will work very well when we get escape analysis in place, since we can then potentially remove those guards after performing scalar replacement. Thus after inlining, we might be able to remove the “what kind of vivification does this assignment cause” checking too.
So this seemed workable, but then I also realized that it would be possible to make Scalar smaller by:
Placing the new auto-vivification objects in the $!descriptor slot instead
Having the vivification objects point to the original descriptor carrying the name, type, etc.
Upon first assignment, running the vivification logic and then replacing the Scalar‘s $!descriptor with the simple one carrying the name and value, thus achieving the run-once semantics
This not only makes Scalar smaller, but it means that we can use a single guard check to indicate the course of action we should take with the container: a normal assignment, or a vivification.
The net result: vivification closures go away giving more possibility to inline, assignment gets easier to specialize, and we get a memory saving on every Scalar container. Nice!
C you later
For this to be really worth it from an optimization perspective, I needed to eliminate various bits of C special-case code around Scalar and replace it with standard MoarVM ops. This implicated:
Assignment
Atomic compare and swap
Atomic store
Handling of return values, including decontainerization
Creation of new Scalar containers from a given descriptor
The first 3 became calls to code registered to perform the operations, using the 6model container API. The second two cases were handled by replacing the calls to C extops with desugars, which is a mechanism that takes something that is used as an nqp::op and rewrites it, as it is compiled, into a more interesting AST, which is then in turn compiled. Happily, this meant I could make all of the changes I needed to without having to go and do a refactor across the CORE.setting. That was nice.
So, now those operations were compiled into bytecode operations instead of ops that were really just calls to C code. Everything was far more explicit. Good! Alas, the downside is that the code we generate gets larger in size.
Optimization with spesh plugins
I talked about specializer plugins in a recent post, where I used them to greatly speed up various forms of method dispatch. However, they are also applicable to optimizing operations on Scalar containers.
The change to decontainerizing return values was especially bad at making the code larger, since it had to do quite a few checks. However, with a spesh plugin, we could just emit a use of the plugin, followed by calling whatever the plugin produces.
Here’s a slightly simplified version of the the plugin I wrote, annotated with some comments about what it is doing. The key thing to remember about a spesh plugin is that it is not doing an operation, but rather it’s setting up a set of conditions under which a particular implementation of the operation applies, and then returning that implementation.
nqp::speshreg('perl6', 'decontrv', sub ($rv) {
# Guard against the type being returned; if it's a Scalar then that
# is what we guard against here (nqp::what would normally look at
# the type inside such a container; nqp::what_nd does not do that).
nqp::speshguardtype($rv, nqp::what_nd($rv));
# Check if it's an instance of a container.
if nqp::isconcrete_nd($rv) && nqp::iscont($rv) {
# Guard that it's concrete, so this plugin result only applies
# for container instances, not the Scalar type object.
nqp::speshguardconcrete($rv);
# If it's a Scalar container then we can optimize further.
if nqp::eqaddr(nqp::what_nd($rv), Scalar) {
# Grab the descriptor.
my $desc := nqp::speshguardgetattr($rv, Scalar, '$!descriptor');
if nqp::isconcrete($desc) {
# Has a descriptor, so `rw`. Guard on type of value. If it's
# Iterable, re-containerize. If not, just decont.
nqp::speshguardconcrete($desc);
my $value := nqp::speshguardgetattr($rv, Scalar, '$!value');
nqp::speshguardtype($value, nqp::what_nd($value));
return nqp::istype($value, $Iterable) ?? &recont !! &decont;
}
else {
# No descriptor, so it's already readonly. Return as is.
nqp::speshguardtypeobj($desc);
return &identity;
}
}
# Otherwise, full slow-path decont.
return &decontrv;
}
else {
# No decontainerization to do, so just produce identity.
return &identity;
}
});
Where &identity is the identity function, &decont removes the value from its container, &recont wraps the value in a new container (so an Iterable in a Scalar stays as a single item), and &decontrv is the slow-path for cases that we do not know how to optimize.
The same principle is also used for assignment, however there are more cases to analyze there. They include:
When the type constraint is Mu, and there is a normal (non-vivify) descriptor, then we do a specialization based on the value being the Nil object (in which case we produce the operation that set $!value back to the default value from the descriptor) or non-Nil (just assign a value, with no need to type check)
When the type constraint is something else, and there is a normal (non-vivify) descriptor, then we do a specialization based on the type of the descriptor being assigned. Since the optimizer will often know this already, then we can optimize out the type check
When it is an array auto-viv, we produce the exact sequence of binds needed to effect the operation, again taking into account a Mu type constraint and a type constraint that needs to be checked
Vivifying hash assignments are not yet optimized by the spesh plugin, but will be in the near future.
The code selected by the plugin is then executed to perform the operation. In most cases, there will only be a single specialization selected. In that case, the optimizer will inline that specialization result, meaning that the code after optimization is just doing the required set of steps needed to do the work.
Next steps
Most immediately, a change to such a foundational part of the the Rakudo Perl 6 implementation has had some fallout. I’m most of the way through dealing with the feedback from toaster (which runs all the ecosystem module tests), being left with a single issue directly related to this work to get to the bottom of. Beyond that, I need to spend some time re-tuning array and hash access to better work with these changes.
Then will come the step that this change was largely in aid of: implementing escape analysis and scalar replacement, which for much Perl 6 code will hopefully give a quite notable performance improvement.
This brings me to the end of my current 200 hours on my Perl 6 Performance and Reliability Grant. Soon I will submit a report to The Perl Foundation, along with an application to continue this work. So, all being well, there will be more to share soon. In the meantime, I’m off to enjoy a week’s much needed vacation.
Hashes are very useful data structures and underlie many internal representations in Perl 6
as well as being used as themselves. These data structures are very nice since they offer
O(1) insertion time and O(1) lookup time on average. Hashes have long been considered
an essential feature for Perl, much loved by users. Though when exploited, hashes
can cause servers to grind to a halt. New in Rakudo Perl 6 2018.5 will be a feature
called hash randomization which does much to help protect against this attack.
In this article I explain some hashing basics as well as how the attack against
non-randomized hashing can work.
Some hashing basics: when we use a hash, we take a string and come up with a unique
integer to represent the string. Similar to how md5 or sha1 sums take an
arbitrary amount of data and condense it into a shorter number which can identify
it, we do a similar
thing for strings.
my%hash; %hash<foo> = 10
In this code, MoarVM takes the string foo and performs a hashing function on it
using a series of bitwise operations. The goal is to create a shorter number which
allows us to put the foo key into one of the 8 buckets that MoarVM initializes
when a hash is created.
Our hashing code sets up a predefined number of buckets . When a bucket fills up to have 10 items it doubles the number of
buckets. In normal operation the hashes will be randomly distributed, so it would
take ≈47 keys added (≈47 is the average number of items to result in one bucket
being filled to 10 items) before we have to expand the buckets the first time.
When the buckets are expanded, we will now have 16 buckets. In normal operation
our previous ≈47 items should be evenly distributed into those 16 buckets.
The Attack
Without a random hash seed it is easy for an attacker to generate
strings which will result in the same hash. This devolves to O(n️2)
time for the hash lookup. This O(n2) is actually O(string_length * num_collisions).
When we have hash collisions, that means that no matter how many times we double
the number of buckets we have, the strings which have hash collisions will always
remain in the same bucket as each other. To locate the correct string, MoarVM
must go down the chain and compare each hash value with the one we’re looking for.
Since they are all the same, we must fall back to also checking each string itself
manually until we find the correct string in that bucket.
This attack is done by creating a function that essentially is our hashing
function backward (for those curious see here for an example of
code which does forward and backward hashing for Chrome V8 engine’s former
hashing function). We hash our target string, t. We then use random 3
character sequences (in our case graphemes) and plug them into our backward
hashing function along with the hash for our target t. The backward hash and
the random character sequence are stored in the dictionary and the process is
repeated until we have a very large number of backward hash’s and random 3
grapheme prefixes.
We can then use this dictionary to construct successively longer strings
(or short if we so desire) which are the same hash as our target string
t. This is a simplification of how the Meet-In-The-Middle attack works.
This has been fixed in most programming languages (Python, Ruby, Perl), and
several CVE’s have been issued over the years for this exploit (See CVE’s for
PHP, OCaml, Perl, Ruby and
Python).
Assuming everything is fine for the next release I will also merge changes which
introduce a stronger hashing function called SipHash. SipHash is meant to
protect against an attacker discovering a hash secret
remotely. While randomizing
the seed makes this attack much harder, a determined attacker could discover the
hash and if that is done they can easily perform a meet in the middle attack.
SipHash was designed to solve the
vulnerability of the hash function itself to meet-in-the-middle attacks. Both
the randomization of the hash secret in addition with a non-vulnerable hashing
function work work together to avert hash collision denial of service attacks.
While the hash secret randomization will be out in Rakudo 2018.05, SipHash is
planned to be introduced in Rakudo 2018.06.
Randomness Source
On Linux and Unix we prefer function calls rather than reading from /dev/urandom.
There are some very important reasons for this.
Relying on an external file existing is potentially problematic.
If we are in a chroot and /dev is not mounted we will not have access to /dev/urandom.
/dev/urandom is not special, it can be deleted by accident (or on purpose) or
a sparse data file mounted in its place undetectable by programs. Trusting it
simply because of its path is not ideal. Also, if we exhaust the maximum number of open file
descriptors we will be unable to open /dev/urandom as well.
System Functions
On Windows we use the pCryptGenRandom which is provided by advapi32.dll since
Windows XP.
Linux, FreeBSD, OpenBSD and MacOS all use system provided random calls (if available)
to get the data rather than having to open /dev/urandom. All these OS’s
guarantee these calls to be non-blocking, though MacOS’s documentation does
not comment on it. This is mostly important in very early userspace, which bit
Python when a developer accidentally changed the randomness source causing
systems which relied on very early Python scripts to stop booting due to waiting
for randomness source to initialize.
If the function doesn’t exist we fall back to using /dev/urandom. If opening or
reading it fails, on BSD’s we will use the arc4random() function. In many BSD’s
this is seeded from the system’s random entropy pool, providing us with a back up in case
/dev/urandom doesn’t exist.
On Linux we use the getrandom() system call which was added to kernel 3.17
instead of using the glibc wrapper since the glibc wrapper was added
much later than to the kernel.
On MacOS, Solaris and FreeBSD we use getrandom() while on OpenBSD we use getentropy().
User Facing Changes
From Rakudo Perl 6 2018.05, the order that keys are returned will be random between
each perl6 instance.
perl6 -e 'my %hash = <a 1 b 1 c 1 d 1 e 1 f 1>; say %hash.keys'(d f c a b e)
perl6 -e 'my %hash = <a 1 b 1 c 1 d 1 e 1 f 1>; say %hash.keys'(e f a d c b)
This will also effect iterating a hash without sorting: for %hash { }
What Do I Have To Do?
Users and module developers should make sure that they explicitly sort hashes
and not rely on a specific order being constant. If you have a module, take a look
at the code and see where you are iterating on a hash’s keys and whether or not
the order of processing the hash’s keys affects the output of the program.
# This should be okay since we are putting the hash into another hash, order# does not matter.for%hash.keys -> $key {
%stuff{$key} = $i++;
}
# This can potentially cause issues, depending on where `@stuff` is used.for%hash.keys -> $key {
@stuff.push:$key;
}
# This should be OK since we are using is-deeply and comparing a hash with another# hashis-deeplymy-cool-hash-returning-function($input), %( foo => 'text', bar => 'text', baz => 'text');
# Probably best to avoid using `is`. The `is` test function converts the input to a string before# checking for equality, but works since we stringify it in sorted order.is%hash, %( foo => 'text', bar => 'text', baz => 'text');
# NO. Keys are not guaranteed to be in the same order on each invocationis%hash.keys, <a b c d>;
Module Developers
Module developers should check out the git master of Rakudo, or if 2018.05 has
been released, use that to run the tests of your module. Make sure to run
the tests multiple times, ideally at least 10 times or use a loop:
while prove -e 'perl6 -Ilib';do true;done
This loop will run again and again until it encounters a test failure, in which
case it will stop.
You must run your tests many times because the hash order will be different on each run.
For hashes will a small number of items, it may not fail on every run. Make sure
that you also look at the source to identify items that need fixing; don’t just
rely on the test’s to tell you if you must make changes to your module.
.
.
.
.
.
by Andrew Ruthven.
.
Braindead MacAbre 
🅼 
🆂.
.
.
.
.
.
John Steed.
Pandy Knight 
.
んまいから食べ バブー.
applying unicode to some threading constructs, I started with the notion of the ‘libra’ operator 
as shorthand to declare and load Measure instances.
eevee 
Mishka }}.
.
.
おおやけハジメ@node.js.
.
) while being enough to illustrate some Perl 6 constructs. On the other hand, I’ve avoided those which might be too straightforward.
s
, 
