I implemented the Boyer–Moore–Horspool substring-search algorithm in Arc[1].
I then plugged it into my playlist program, and, well... see for yourself:
;; Arc posmatch: 38,236 ms
;; Boyer–Moore–Horspool: 6,396 ms
Wow. I think this confirms that the biggest bottleneck in my program was posmatch[2].
Keep in mind, though, Boyer-Moore isn't always faster than posmatch[3]: due to the preprocessing time, long pattern strings will cause the algorithm to perform slower.
This only applies when the string is preprocessed multiple times: if it's only used once, then the overhead should be negligible. And if you want to use the same pattern string multiple times, you can preprocess it once and then reuse it:
(let x (boyer-moore-process "foobarquxcorgenou")
(boyer-moore-search x "adadadadadadadadadadadad")
(boyer-moore-search x "adadadadadadadadadadadad")
...
(boyer-moore-search x "adadadadadadadadadadadad"))
In general, though, Boyer-Moore is faster. Especially for multi-string searches, which is what my playlist program does.
I added in Arubic-style namespace support, and also implemented dynamic and parameter variables better. If you're interested in reading more, check out these two links:
I finally added in Arubic to Nu. Now all you have to do is `./arc -i arubic` and you'll get a REPL with Arubic functionality. You can also use `(import arubic)` to write a file using Arubic.
Why would you want to do that? Well, here are the current changes between Arc and Arubic, though I plan to add more over time:
[1 2 3] -> (list 1 2 3)
{a 1 b 2} -> (obj a 1 b 2)
Because of the changes above, you no longer need the [] syntax for functions most of the time, so I'm repurposing them to create lists, which I've found to be far more common.
`isa` accepts multiple arguments:
(isa x 'foo 'bar 'qux) ;; Arubic
(in type.x 'foo 'bar 'qux) ;; Arc 3.1
`prn` prints spaces between each argument:
(prn "foo" "bar" "qux") -> "foo bar qux\n"
You can also use `str` and `str?` as aliases for `string` and `string?`.
Small changes, but they do make writing code more pleasant.
2) This helps out a ton with arc2js. In arc2js, now all I need to do is provide an `=` macro and I'll get zap, or=, push, pull, swap, etc. all for free. Without this change, I'd have to define my own custom versions of zap, or=, etc...
"It truly does not make any sense to me why zap is defined like this"
'zap is the only use I typically have for the 'setforms "binds" list (which ensures the subexpressions of 'place are only evaluated once).
Still, 'zap doesn't need to work that way: as long as I'm using a language where I know 'zap evaluates its place twice, I'm pretty much okay with it. It's a wart, but it's not an impediment.
To explore Arc-3.1-like options for a bit, here's a cleanup of Arc 3.1's definition of 'zap:
(mac zap (op place . args)
(with (gop (uniq)
gargs (map [uniq] args)
(binds val setter) setforms.place)
`(atwiths (,@binds ,gop ,op ,@(mappend list gargs args))
(,setter (,gop ,val ,@gargs)))))
If we allow 'setter and 'val to compile and evaluate before 'op and 'args, it gets shorter:
(mac zap (op place . args)
(let (binds val setter) setforms.place
`(atwiths ,binds
(,setter (,op ,val ,@args)))))
I prefer to arrange the compilation and evaluation orders from left to right ('op, 'place, 'args), using a technique like this:
(mac place (place)
(let (binds val setter) setforms.place
`(withs ,binds
(list (fn () ,val) ,setter))))
(mac zap (op place . args)
`(atomic:fn-zap ,op (place ,place) (list ,@args)))
(def fn-zap (op (getter setter) args)
(setter:apply op (getter) args))
"'zap is the only use I typically have for the 'setforms "binds" list (which ensures the subexpressions of 'place are only evaluated once)."
Hm... yes, you're right, `(zap + (foo (bar qux)) 1)` evaluates `(bar qux)` twice, and I don't see an easy/obvious way to fix that in `=`. I'll need to think about this.
It would appear that the `=` operator is already thread-safe even without `atomic-invoke`. In any case, if you're terribly worried, you can always wrap it yourself.
I just got rid of `setforms` completely[1]: not even `=` uses it anymore. I then reimplemented `=` in a much shorter and clearer way[2].
Arc 3.1 takes 80 lines to implement expand=, but Nu takes only 31. And Nu is much clearer and easier to understand as well. In addition, Nu's output is much shorter and is faster:
All constant-time procedures and operations provided by Racket are
thread-safe because they are atomic. For example, set! assigns to a variable
as an atomic action with respect to all threads, so that no thread can see a
“half-assigned” variable. Similarly, vector-set! assigns to a vector
atomically. The hash-set! procedure is not atomic, but the table is
protected by a lock; see Hash Tables for more information. Port operations
are generally not atomic, but they are thread-safe in the sense that a byte
consumed by one thread from an input port will not be returned also to
another thread, and procedures like port-commit-peeked and write-bytes-avail
offer specific concurrency guarantees.
It mentions that hash table assignment is not thread-safe, however if you then go to the hash table page[1], it says this:
A mutable hash table can be manipulated with hash-ref, hash-set!, and
hash-remove! concurrently by multiple threads, and the operations are
protected by a table-specific semaphore as needed. Three caveats apply,
however [...]
In other words, Racket already handles everything, according to the docs. If you're ever worried enough, or run into any problems, it's not hard to wrap it in `atomic` yourself. I'd rather not have the cost of `atomic-invoke` for every assignment, especially if you run all your code in one thread (like I do).
By the way, Nu doesn't use `set!` for global assignment, so I'm not sure if global assignment in Nu is thread-safe or not. But I'd assume it is, since I think `namespace-set-variable-value!` is constant-time.
This ensures that car.foo, (bar), and (scar gs1 val) all happen without interference in between. I suspect Racket at most protects those on an individual basis.
"In pg-Arc, '= on a variable is 'assign without 'atomic. Where 'atomic comes in is when there's a setforms thing to worry about."
I am aware. It still seems to me that if you're dealing with threads, you should wrap assignment in atomic yourself if you're worried about such things. Code that doesn't deal with threads shouldn't have to use atomic.
Perhaps there should be an `a=` macro that's just like `=` but it calls `atomic`. Hm... I wonder... would it be possible to detect whether code is running in the default thread and if not, automatically wrap it in atomic...? May be more trouble than it's worth, though.
That's what I think. Anyone who cares can say (atomic:= ...) or (atomic:zap ...), so I only see a couple of reasons why we'd want to have the 'atomic implicit:
- We want to use it all the time anyway. (I doubt it, but it's hard to tell. I haven't used threads, and therefore I've never bothered to find a way to squeeze utility out of it.)
- There are people who do care, and they'd be better off if the people who didn't care still used 'atomic by accident. (Again, it's hard for me to tell if this is true.)
I just implemented `eachfn` and then changed Arc so the `each` macro just calls `eachfn`. This had a significant speedup, with no cost at all in functionality. At this rate, Nu will end up being just as fast as Arc 3.1, with all the shiny extra features and bug fixes.
So, the rule of thumb is: don't write macros that do a lot of work. Instead, write a function that does the work, and then write a macro that just calls the function. This goes double for macros like `each` that do a type-check at runtime.
...because each does not map. Nor does it some. Nor does it mappend. Nor does it all. Nor does it keep. Nor does it rem.
I don't think you understand what I'm talking about. In Arubic, map is just a macro that expands into mapfn. The semantics are exactly the same as Arc 3.1, it's just easier to say...
(map x foo ...)
...rather than:
(map (fn (x) ...) foo)
The only point of it is to remove the (fn ...) bit, resulting in shorter and easier to read code.
"Focus on just map. Are you saying it doesn't return a transformed list?"
No. You are still misunderstanding me. I will be as clear and precise as I possibly can. In Arc 3.1, you call `(map x foo)` where `x` is a function and `foo` is a list.
In Arubic, you would say that as `(mapfn x foo)`. It's exactly the same, except you use the name `mapfn` rather than `map`. Now, Arubic also provides a macro called `map`, which expands into `mapfn`. In other words:
The sole purpose of this macro is so that you don't need to type out the `(fn ...)` part in `(mapfn (fn (x) ...) foo)`. That's it. The meaning of `map` is the same, it's just that now that it's a macro, it's a lot more convenient to use. But you can still pass in a function directly by using `mapfn`.
---
"You'll have to pry (map car xs) from my cold dead hands :)"
If you dislike any parts of Arubic (whether misunderstood or not), simply do not use those parts. The point of the namespace system is that you can change what you want and use what you want, rather than being tied down to one semantic/syntax/meaning/language. Thus, it is possible (and relatively easy) to use some of the things from Arubic that you like, while not using the parts you dislike.
In this particular case, however, you are misunderstanding me. Your example would be `(mapfn car foo)` in Arubic.
In this particular case, however, you are misunderstanding me. Your example would be `(mapfn car foo)` in Arubic.
No, I'm now pretty sure I'm not misunderstanding you.
My entire objection is to the name switch. Of course I know I can change the default. It's not a sensible default, it's like defining 1 to be 0.99. You're fragmenting the semantics of a name that has thus far had a pretty clear meaning. You're welcome to do this, but it's going to hinder users with any programming background.
I don't understand why the thing you want must be called map. Why not mapmc, for example? Or some other 3-letter name?
"My entire objection is to the name switch. Of course I know I can change the default. It's not a sensible default, it's like defining 1 to be 0.99."
It's not the default. The default is Arc 3.1. You have to opt-into Arubic. In any case, how is it not sensible? It provides the shortest, easiest to read, and fastest code.
---
"You're fragmenting the semantics of a name that has thus far had a pretty clear meaning."
And this is why I think you're still misunderstanding me. The meaning of map is the same. It's just that now you can write (map x foo ...) rather than (map (fn (x) ...) foo), which is shorter and easier to read.
If you're worried that it'll mess up Arc programmers who are used to the naming of map, then as already said, Arc 3.1 is the default. You only get the new syntax if you import Arubic.
I do not see how changing the meaning of names in a new language is a bad thing. That's like me saying that wart should use exactly the same names as Arc, which is absurd: they're different languages. If you want to use a different name for something in wart, then go for it.
Perhaps you still do not grasp that Arubic is a different language from Arc, implemented in a separate namespace so as not to mess up Arc's namespace.
---
"I don't understand why the thing you want must be called map. Why not mapmc, for example? Or some other 3-letter name?"
Suit yourself. I don't understand what I have said to merit hostility.
I know we can all do what we want. I also am aware of lots of things I can try if I don't like something you do. I don't understand these defensive reactions. Was I rude somewhere?
All this while you keep insisting you know what I am misunderstanding. Is that not rude?
Arubic's map is a macro, not a function. Until it grows fexprs the meaning is by definition not the same.
"This is how I like it" is a perfectly valid reason (and would have ended this conversation eons ago). Just don't insist it's all still the same when it isn't.
No, the default is racket. No, it's machine code. Or is it cosmic background radiation? WTF, are you seriously saying that creating a new language isn't an act of choosing defaults?
In Nu, the default namespace is the one corresponding closely with Arc 3.1. And in this namespace, the default definition of 'map is pretty much Arc 3.1's. Therefore, "it's not the default" is true in Nu.
Another namespace in Nu is Arubic. Using this namespace is like using a language other than Arc 3.1, where 'map is a macro. If your problem is with Arubic itself, you're free to ignore the parts of Arubic you don't like (even if you want to use an library written in full Arubic). If the point is you think people shouldn't dilute the overall meaning of "map" with the meaning Pauan gives it in Arubic, that's a moral position, with Pauan as an unintentional villain!
---
What I think is that the meaning of "map" isn't tainted here, at least not any more than usual. In fact, I wanted 'map to be an 'each -like macro before I knew Pauan had the same thing in mind.
If I don't know what language we're talking about and I hear "map," I assume it'll be an abstraction that applies a given transformation to elements of a given data structure (whatever "abstraction," "transformation," "data structure," and "given" mean), along with some accidental complexity suitable to the language, such as the timing of computation, side effects permitted in the transformation, or late-bound dependencies (like 'map calling out to 'cons, such that rebinding 'cons changes what happens).
In a language where macros like 'after and 'each are more convenient to use than higher-order functions like 'protect and Anarki's 'trav, a macro for map is to be expected.
In Arc, almkglor[1] calls this macro "mapeach" in Anarki, and I call it "maplet" in Lathe. Pauan and I call it "map" as long as we don't have naming conflicts to worry about. As long as this macro is the primary way we use the map concept, using that name keeps our programs brief and frank.
Notice that the namespace is now length 2, because it inherits from Arc's namespace. But you don't want to use Arubic, you want to use Arc! But because of the malleability and inspectability of Nu namespaces, it's easy to fix that. Just do this:
The above just reversed the order of the namespace, so now the Arc namespace inherits from the Arubic namespace, rather than vice versa.
This effectively "undoes" the changes that Arubic made, which is exactly what you want: now you can use the functions and macros defined in the Arubic library from within Arc, and everything works perfectly.
I just did some timing tests. As I suspected, the namespace system is quite costly in terms of performance:
1 2 3 4 6 6 7
0% 1732% 31% 22% 20% 17% 12%
What the above means is that when going from 1 namespace to 2, you lose 1732% of speed, meaning your code will now run ~17 times slower. But going from 2 namespaces to 3 has only a 31% drop in speed, and 3 to 4 only has 22%, etc.
What this means is that the initial hurdle of namespaces is quite large, but adding more namespaces after the 2nd is quite decent in terms of performance.
The reason for this is (ironically) speed optimizations. When executing code in the Arc namespace, I actually completely bypass the namespace system entirely, so it's relying on Racket to do global lookups, and Racket is far faster at looking up global variables than Nu is.
That's why going from 1 to 2 namespaces is so slow: Racket is so fast that when it switches to the Nu code, Nu looks horribly slow by comparison. But when going from 2 to 3 namespaces, we're already using the Nu code, so the difference in speed is tiny.
In other words, Racket is so fast it makes Nu look bad.
---
By the way, this only applies to code that is evaluated in a Nu namespace: if you evaluate everything in Arc's namespace, there's no cost in speed at all. So this drop in speed only applies while using things like Arubic, and not Arc itself.
I just added in an interesting feature: automatic namespace creation.
When using the `(import ...)` form, if the file tries to overwrite an already-existing variable, it will automatically create a new namespace. Otherwise, it won't.
This is mostly a performance optimization, since it's entirely possible to load every single file into a different namespace, but using this technique I avoid creating namespaces except when absolutely necessary, while still providing a reasonable amount of library isolation.
I'm still anti-namespaces -- I'd rather work on codebases that don't need them, and I'd rather just rely on tests to catch collisions. But your article makes an eloquent case for them that has had more effect than anything else I've read.
I would like to note something. If all you want to do is write programs and get stuff done then namespaces are indeed a luxury. It is not particularly difficult to use a language (including Arc) to write useful programs, despite the lack of namespaces.
Where namespaces start to become necessary is when you want to change the language itself beyond what can be easily done by adding macros. Then you are in the predicament that you want to write your code in the new (and presumably better) language that you created, but now you have to give up libraries written in the old language.
With namespaces, it's no longer an either/or thing: you can have your cake and eat it too (at the cost of some performance). So, for writing code, namespaces aren't important. But for language experimentation, namespaces are so incredibly useful that I consider them necessary.
The only reason I put namespace support into Nu's core is because I really want to use Arubic, but doing so practically requires decent namespace support. I invented this system out of necessity, which should (hopefully) demonstrate its usefulness.
"Would it be hard for Nu to permit lifting arc primitives?"
Not at all. Arc lifting happens automatically and transparently with namespace inheritance. It works right now. In fact, that's the entire point of the namespace system.
And if you ever want to directly evaluate something in Arc (bypassing the inheritance system), you can call `(w/arc3 ...)`
Okay, I'm still not sure what you mean by "lifting Arc primitives" but I'll do my best to explain how Nu works.
The compiler itself is written in Arc's namespace. There is no separation at all between Arc, the Arc compiler, and Racket: everything is cobbled into one giant namespace. So changing the compiler is as simple as writing an Arc function/macro. This is how ar works, and I liked it enough that I copied the idea for Nu.
That also means no need for the $ primitive because all Racket stuff is prefixed with `racket-`. Thus you would say `racket-regexp-match` rather than `$.regexp-match`.
Note: it's possible to import Racket modules with any prefix you like (including no prefix), but it's there mostly to avoid namespace collisions with Arc stuff.
---
As for doing the same thing with Arc primitives, I assume you mean that if you create a new namespace, you might sometimes want to use the definitions in the new namespace, and sometimes the definitions in Arc's namespace.
You can do that too, by calling eval-w/ with arc3-namespace:
(eval-w/ arc3-namespace
foo)
Because it's common enough to want to load things into Arc's namespace, the above can also be expressed as w/arc3:
(w/arc3 foo)
That will give you the definition of "foo" from Arc's namespace. This shouldn't be necessary most of the time because it's possible to have a namespace inherit from another. In that case, if the variable isn't found, it'll search in the parent.
Thus, if you make a new namespace that inherits from Arc's namespace, you'll automatically get all the Arc stuff for free. That's how Arubic works: it makes a namespace that inherits from Arc, and then changes things. Anything that Arubic doesn't change is automatically taken from Arc.
Ok perhaps your namespaces are the simplest way to 'lift' arc into Nu.
If you implement language A in language B, A is above B in the stack. It's meaningless to call A's code from B, but useful to call B from A. The latter is what I call lifting. But I realize this terminology doesn't work for anything but simple static compilation.
If I wrote an awesome library in Arubic, especially a library that would be difficult to port to Arc, and you wanted to use the library, but didn't want to use Arubic, it would be very useful to write Arc code that used said Arubic library.
As another example, you may want to write a module in Racket that imports a library written in Arc. Even though Arc is implemented in Racket, I don't see how that counts as "meaningless". This example isn't even theoretical, there was somebody on this forum earlier who wanted to do exactly that:
Okay, some interesting info: Racket is apparently quite slow at applying a function at runtime. I ran this code in Arc 3.1, ar, and Nu:
(time (repeat 1000000 (let (a b) (list 1 2) a)))
Here are my results:
Nu time: 3165 msec.
Arc 3.1 time: 1830 msec.
ar time: 1391 msec.
Very surprising! Arc 3.1 expands into a Racket let* and Nu expands into a Racket lambda... but ar's code is implemented in Arc, so it expands into multiple nested Arc lambdas... yet somehow manages to be faster than Arc 3.1 and Nu.
I am not sure how plain lambdas are faster than Racket's built-in let* . In any case, I changed my code so it expands into let* and now Nu gets 1649 msec, which is comparable to ar. Thus the problem is clearly applying a function at runtime.
Just to remove some more variables, I changed the function so it didn't use rest or optional args, but it ended up being almost as slow, so I know it's not that.
Hm... perhaps it's not the fact that it's applying, per se. Perhaps it's because Racket has to create a function every single time. In other words, if Racket sees this:
((fn (x) ...) ...)
It can try and optimize it away so a function is never actually created. But when Racket sees this:
(apply (fn (x) ...) ...)
It can't do that optimization... thus it's forced to create a function. Creating and destroying a function every time the let is called could indeed cause a performance problem.
---
EDIT: to test my hypothesis, I used this code:
(let foo (fn ((o a) (o b) . rest) a)
(time (repeat 1000000 (let x (list 1 2) (apply foo x)))))
...unfortunately, it ended up getting 2966 msec, so it seems the problem is indeed Racket's apply, or perhaps my compiler's apply. I'll see if I can optimize it a bit.
Racket seems fairly good at "lambda lifting", if that is the correct term. To demonstrate with adding the numbers from 1 to n: version 0 should pretty obviously be compiled into a loop; versions 1 and 2 are less obvious. In 64-bit "racket", versions 1 and 2 seem to take twice as long as the loop in version 0, but that's still much better than allocating lambdas; in 32-bit, the "twice as long" difference is smothered by the cost of allocating/GCing bignums. The last version is actually written in Arc, and 25x slower in arc3.1; though in 32-bit, the cost of handling bignums makes arc3.1 only 2x slower. The results of the 64-bit version seem to demonstrate that Racket successfully avoided allocating closures at runtime in all cases.
(define (sum0 n)
(let loop ((n n) (tt 0))
(if (zero? n)
tt
(loop (- n 1) (+ n tt)))))
(define sum1
(λ (n)
((λ (f n tt)
(if (zero? n)
tt
(f f (- n 1) (+ n tt))))
(λ (f n tt)
(if (zero? n)
tt
(f f (- n 1) (+ n tt))))
n
0)))
(define sum2
(λ (n)
((λ (f) (f f n 0))
(λ (f n tt)
(if (zero? n)
tt
(f f (- n 1) (+ n tt)))))))
(= sum3
(fn (n)
((fn (f n tt)
(if (is n 0)
tt
(f f (- n 1) (+ n tt))))
(fn (f n tt)
(if (is n 0)
tt
(f f (- n 1) (+ n tt))))
n
0)))
;Paste this command in, but copy the above to clipboard before running:
arc> (let xs (readall:pbpaste) (map [eval (list '$ _)] butlast.xs) (eval last.xs)
(each x '(sum0 sum1 sum2 _sum3) (repeat 2 (time:eval `(($ ,x) 10000000)))))
;64-bit racket v5.2.0.3: no mallocing beyond initial compilation
time: 41 cpu: 41 gc: 0 mem: 25720
time: 40 cpu: 41 gc: 0 mem: 6096
time: 80 cpu: 80 gc: 0 mem: 7576
time: 80 cpu: 80 gc: 0 mem: 6136
time: 81 cpu: 80 gc: 0 mem: 7576
time: 80 cpu: 80 gc: 0 mem: 6096
time: 1026 cpu: 1027 gc: 0 mem: 7408
time: 1018 cpu: 1019 gc: 0 mem: 6112
;32-bit racket v5.1.3.10: runtime is dominated by consing bignums
time: 894 cpu: 892 gc: 24 mem: 1478560
time: 872 cpu: 872 gc: 16 mem: 1236500
time: 841 cpu: 841 gc: 15 mem: 1238156
time: 844 cpu: 843 gc: 17 mem: 1236476
time: 839 cpu: 839 gc: 15 mem: 1237300
time: 838 cpu: 837 gc: 15 mem: -15541124
time: 1857 cpu: 1857 gc: 18 mem: 1237784
time: 1864 cpu: 1864 gc: 17 mem: 1236436
I haven't fixed `apply` yet, but I did put in a workaround. Using Racket's `apply`, Nu is actually faster than Arc 3.1 and ar!
(timeit (let a 1 a))
ar time: 8.101 gc: 0.268 mem: 973.008
Nu time: 6.923 gc: 0.0 mem: 88.736
Arc 3.1 time: 4.77 gc: 0.28 mem: -3305.9
(timeit (let a (list 1 2) (car a)))
Arc 3.1 time: 17.3 gc: 0.86 mem: -7,759.58
ar time: 10.303 gc: 0.552 mem: 1258.64
Nu time: 8.158 gc: 0.196 mem: -515.648
(timeit (let (a b) (list 1 2) a))
Arc 3.1 time: 17.47 gc: 1.0 mem: -6997.07
ar time: 13.166 gc: 0.696 mem: -16510.112
Nu time: 12.102 gc: 0.512 mem: -10028.488
So, it seems my idea of applying nested functions to implement destructuring is good in essentially every way: faster, shorter, and easier to implement.
Interestingly, judging by the data above, it would seem Arc 3.1 is very slow at creating lists, probably because `list` is implemented in arc.arc, whereas ar and Nu provide faster implementations.
---
Now let's test optional args:
(timeit ((fn (a) a) 1))
ar time: 7.534 gc: 0.352 mem: 866.232
Nu time: 6.976 gc: 0.0 mem: 88.368
Arc 3.1 time: 4.78 gc: 0.28 mem: -3295.58
(timeit ((fn (a (o b)) a) 1))
ar time: 14.493 gc: 0.464 mem: 1639.648
Nu time: 7.903 gc: 0.248 mem: -1664.792
Arc 3.1 time: 5.84 gc: 0.36 mem: -2097.19
Overhead
ar - 6.959
Arc 3.1 - 1.06
Nu - 0.927
As you can see, in Nu and Arc 3.1, there's very little overhead from optional args, but in ar, optional args are quite costly.
Update: I didn't want to be unfair to Arc 3.1 because of its slow implementation of `list`, so I redid the tests using `quote` instead:
(timeit (let a '(1 2) (car a)))
Nu time: 10.628 gc: 0.196 mem: -1747.08
ar time: 8.529 gc: 0.252 mem: 967.432
Arc 3.1 time: 5.26 gc: 0.34 mem: 4952.98
(timeit (let (a b) '(1 2) a))
Nu time: 14.066 gc: 0.52 mem: 90.504
ar time: 13.305 gc: 0.376 mem: -9236.904
Arc 3.1 time: 6.79 gc: 0.35 mem: -2,093.93
Overhead
ar - 4.776
Nu - 3.438
Arc 3.1 - 1.53
As expected, Arc 3.1 is miles ahead of both ar and Nu. Interestingly, Nu is now listed as slower than ar... it would appear that either Nu has a faster implementation of `list`, a slower implementation of `quote`, or possibly both. In any case, this demonstrates that applying nested functions should be approximately the same as complex fns in terms of speed.
One thing I noticed is that Nu has drastically greater overhead than Arc 3.1, but less than ar.
It seems the problem was that quote was slow in Nu. I've fixed that, so here's the new times:
(timeit (let a '(1 2) (car a)))
ar time: 8.613 gc: 0.308 mem: 460.696
Nu time: 7.671 gc: 0.0 mem: 88.976
Arc 3.1 time: 5.33 gc: 0.35 mem: 5050.25
(timeit (let (a b) '(1 2) a))
ar time: 12.111 gc: 0.436 mem: -19278.128
Nu time: 11.438 gc: 0.324 mem: 1435.016 (apply fn)
Nu time: 8.96 gc: 0.0 mem: 125.352 (Racket let*)
Arc 3.1 time: 7.0 gc: 0.35 mem: -2124.82
Overhead
ar - 3.498
Arc 3.1 - 1.67
Nu - 1.289
Nu now has the lowest overhead out of the three...! Also note that Nu does not spend any time in garbage collection, unlike ar and Arc 3.1.
This seems to be a common trend: Nu either spending no time in garbage collection, or less time than ar and Arc 3.1. Not sure how important that is compared to raw speed, but it's nice.
Unfortunately, this also demonstrates that applying nested functions is slower than using a Racket let*. So the reason Nu won the speed contest earlier wasn't because of my destructuring idea: Nu was just plain faster than ar in general.
And since I'm in a timing mood, here's the times for optional args:
(time (repeat 1000000 ((fn (a (o b 3)) (list a b)) 1 2)))
Arc 3.1 time: 1828 msec.
ar time: 1814 msec.
Nu time: 1554 msec.
So, it does make a difference that Nu uses plain lambdas, rather than complex fns! Now I just need to get apply to be faster.
---
On a related note: racket-set! is slow. Using racket-let or Arc's let is faster, by a fairly significant amount. So I'll be changing my compiler so it doesn't use mutation.
Awesome! I've been looking forward to seeing some development on ar. ^_^ I like what you're doing with the argument lists. Have you found your arc2js progress useful for this? (Part of me wonders how Racket's local (define ...) forms compare against its other options in terms of performance. If they're faster, that could pose the same compilation annoyances as arc2js deals with. ;) )
As far as the name "Nu" goes, it's actually been taken by another lisp (and I'm not even talking about NewLISP :-p ): https://github.com/timburks/nu. Early this year I did a couple of months of Objective-C programming for work, and I GitHub-followed Nu in envy. XD
"I like what you're doing with the argument lists."
You mean the "destructuring is just lambdas" thing?
---
"Have you found your arc2js progress useful for this?"
No, except insofar as writing arc2js taught me some useful compiler things. For the most part, it's been pretty simple and easy, so I don't think arc2js helped much, if at all.
---
"Part of me wonders how Racket's local (define ...) forms compare against its other options in terms of performance."
(let a nil (%nocompile (racket-define a 5)) a)
(let a nil (= a 5) a)
Time: 1500-1600 ms
(let a nil (%nocompile (racket-let* ((a 5)) a)))
(let a nil (%nocompile (racket-let ((a 5)) a)))
(let a nil (let a 5 a))
(let a nil a)
Time: 1400-1500 ms
It would seem Racket optimizes lambdas very heavily.
---
"As far as the name "Nu" goes, it's actually been taken by another lisp (and I'm not even talking about NewLISP :-p )"
Darn. I still like the name "Nu" enough that I want to keep it, though.
It took me a while to reply because I was planning to do some of my own testing before posting, but I haven't found the time.
---
"You mean the "destructuring is just lambdas" thing?"
No, I don't know what to think about that yet. I like it if and only if this works:
(let (a . b) '(1 . 2)
...)
Even if that works, I almost expect Racket to copy a mutable argument list into an immutable one for use inside the lambda, so here's a second test case:
(withs (foo (list 1 2 3)
(a . b) foo)
(= b.0 4)
(is foo.1 4))
What I definitely do like is the use of Racket's optional arg syntax where possible. ^_^
---
"It would seem Racket optimizes lambdas very heavily."
IMO, those examples aren't normal uses of (define ...), since they're contexts where the variable is already defined. I'd like to test these cases:
; Racket code
(let ((a 5)) a)
((lambda (a) a) 5)
(begin (define a 5) a)
; NOTE: Make sure this (begin ...) isn't at the top level, or the
; definition will be global.
(let () (define a 5) a)
((lambda () (define a 5) a))
* Internal-definition expansion has changed to use `let*' semantics
for sequences that contain no back references. This change
removes a performance penalty for using internal definitions
instead of `let' in common cases, and it only changes the meaning
of programs that capture continuations in internal definitions.
Internal definitions are now considered preferable in style to
`let'.
Then again, if they're going to make more semantic changes like this one, it might be better to avoid compiling to an internal-definition style. :-p
Anyway, from the phrasing in the changelog, it sounds like internal definitions are implemented in ways the programmer can already control at a lower level with things like 'let*, so you'll probably continue to find that internal definitions are at least as slow as other options.
The reason is because racket-mlist->list expects a proper list. I can fix that easily.
Update: I fixed racket-mlist->list, but Racket's apply absolutely positively requires a proper list, so it looks like I can't use nested lambdas if I want to fix that: I'll have to use a Racket let*
---
"I almost expect Racket to copy a mutable argument list into an immutable one for use inside the lambda"
It's true, it does copy the list, so that returns nil. I wonder if I can work around that...
---
"One reason it's on my mind is the Racket 5.2 changelog"
I'm not using Racket 5.2. And even if I were, the reason for the speed tests was to see how it would perform within the (fn ...) expansion, where it does indeed overwrite an already-existing variable. (Though I could change it to use a gensym...)
This is very similar to a macro in Arubic-style namespaces:
(eval-w/ foo ...)
The above evaluates the expression "..." in the "foo" namespace, which may potentially be a table. That's not quite right, though. Here's the correct version:
(eval-w/ (new-namespace foo namespace)
...)
The above could then be easily wrapped in a macro like "w/table", or even overloading the "with" macro as you did.
---
"..it cranks for a good second and a half, and conses over a million cells."
I'm not entirely sure how such a simple expression could cause so much consing. What is wart doing with all those cells?
Because that would require hash table's keys being sorted, and then the results would vary depending on what keys the hash table has. For instance:
(let (a b c) (obj d 4 c 3 b 2 a 1)
(list a b c))
Should that be (3 2 1) or (1 2 3)? Or maybe (4 3 2)? But, that's assuming hash table keys are sorted in the first place, when they aren't.
In addition, that would require a runtime check to determine whether the argument is a hash table or a list. With my method, the check is done at compile-time: no runtime costs.
Also, it works on more than just hash tables. In fact, it works on anything that accepts 1-2 arguments, where the first is a symbol. Thus, if you created a new data type, you could use keyword destructuring on it. Basically, this:
(let (:a :b :c) (obj a 1 b 2 c 3)
(list a b c))
Compiles into the equivalent of this:
(withs (g1 (obj a 1 b 2 c 3)
a (g1 'a)
b (g1 'b)
c (g1 'c))
(list a b c))
Compare that to the following:
(let (a b c) (list 1 2 3)
(list a b c))
Which compiles into the equivalent of this:
(withs (g1 (list 1 2 3)
a (car g1)
g1 (cdr g1)
b (car g1)
g1 (cdr g1)
c (car g1))
(list a b c))
Interestingly, that means you could create a data type that worked with both kinds of destructuring. For instance, an alist might support the `foo!bar` syntax, and then you would need a way to distinguish between "look this up based on the key" and "look this up based on the index".
As a final note, from a conceptual standpoint, I view keyword arguments as being implemented with an implicit hash table. Thus using keyword syntax to destructure a hash table makes lots of sense to me.
In other words, if these two are equivalent:
(let (a b c) foo ...)
(apply (fn (a b c) ...) foo)
"I view keyword arguments as being implemented with an implicit hash table. Thus using keyword syntax to destructure a hash table makes lots of sense to me."
Interesting. If you make keyword args pervasive and optional like I do, you need a sense of ordering as well in the table, which weakens that imagery.
"Should that be (3 2 1) or (1 2 3)?"
It should be whatever you define it to be :) Making the symbols keywords didn't actually help me avoid that question at first glance.
"With my method, the check is done at compile-time."
But how does it perform the check for this expression?
(let (:a :b :c) h
..)
Are you using the presence of keywords to disambiguate whether or not to insert the g1 (cdr g1) pairs in the withs?
"Interesting. If you make keyword args pervasive and optional like I do, you need a sense of ordering as well in the table, which weakens that imagery."
Oh I'd love to have pervasive keywords like Python (or wart), but it seems Racket doesn't support that. And I don't plan for Nu to become an interpreter. Keywords are optional in Nu, though (or will be, once I've implemented them).
---
"Making the symbols keywords didn't actually help me avoid that question at first glance."
I just like how there's a clean (and visible) separation between "lookup by key" and "lookup by index". It also avoids a runtime check as well, which is nice.
---
"Are you using the presence of keywords to disambiguate whether or not to insert the g1 (cdr g1) pairs in the withs?"
Yes. The code the compiler outputs depends on whether the argument is a keyword or not.
Pros: it's shorter to use because I don't need to repeat the keys.
Cons: it only works with syms. You can't eval an expr that returns a hash. It replaces the runtime check you're concerned about with that super ugly eval-inside-unquote. You can't import just a few of the keys so it feels a little like 'using namespace std' in C++ -- you may not know what vars you're going to get, and you may end up overriding bindings.
Perhaps I should just make it a new term. That would address the first limitation:
You're not going to be able to use (with hash ...) with a local variable 'hash unless you have at least one of these:
- Fexprs. An fexpr implementation of 'with can use the complete value of 'hash as it determines how to treat the unparsed body.
- Static typing with record types, so that a macro can use the type of 'hash as it determines how to treat the unparsed body.
- Some variant of JavaScript-style scope chain semantics, in the sense that a bare variable reference means (or can mean) a field lookup in general. IMO, this would be the most straightforward to add to an Arc-3.1-like compiler, since it's a matter of compiling foo to (scope 'foo), (with foo ...) to (let ((scope (shadow (scope 'foo) scope))) ...), and other scope-related things in their own analogous ways.
(def foo (a b)
(list a b))
(foo 1 2) -> (1 2)
(foo :b 2 :a 1) -> (1 2)
That doesn't work in Racket. You need to explicitly say that the arguments are keywords:
(def foo (:a :b)
(list a b))
In other words, arguments are either positional or keyword-based, but not both at the same time. This is different from Python, which lets you treat an argument as either one:
Nu should be feature-complete, which means it implements everything that Arc 3.1 does.
However, seeing as how I've only spent the past 4-5 days working on it, obviously it may have bugs, might be slow, and is missing some new additions I want to add to it.
The "Details" section of the README should cover the stuff you want to know, but in case it doesn't, feel free to ask me.
---
Thank you to awwx for writing ar, which was both an inspiration and the physical basis for large parts of Nu.
I figured out why `(+ 1 2)` is so slow in Nu. It was calling Arc's `or` which is pretty slow (compared to not calling it, anyways). So here's the new numbers, this time using the time library[1]:
As you can see, Nu is now faster than ar, but still slower than Arc 3.1. On the other hand, Nu's + is implemented in terms of case-lambda, so it should be better on memory in the case of only 2 arguments.
