I totally agree with you. I do a lot of system scripts (log analysis, job management on a cluster, etc.) and I usually use Perl or Python for these. I'd love switching to Arc for that, but as for now speed is really an issue. Especially startup time : a script is something you possibly run very frequently (sometimes many thousands instances at a time), and even two small seconds on startup is a killer. That's why I spent a few time trying to fix this in arc-exe, btw.
I will soon have to try classification algorithms (you know, bayesian nets, neural nets, these kinds of things). It will be prototypes, so full speed is not required yet. Well even there, something a little faster would be good.
Well, pg promised a profiler and proposed (in a recent poll) to work on an FFI a day or another. I guess that,with these two, we will have what we need in the future...
The FFI is certainly important and I wouldn't use Arc if it didn't have a well-defined standard FFI in its eventual form (to my mind that was one of the biggest mistakes of CL).
I don't think that a FFI is a good substitute for a fast Arc implementation though. It's true that I can re-code any time-critical parts of my application in C, but why should I have to? What you're saying amounts to an unpalatable compromise, and to my mind it's not a compromise that a hundred year language can afford to make.
It will be good to have a profiler for Arc.
I don't agree with your argument that a profiler and FFI will gives us what we need for the future. Python has both a profiler and an FFI. I've used both, in combination, for numerical simulation. It's a highly inelegant and ultimately unsatisfying solution. The python part of the app was never fast enough, and whilst profiling helped, it also lead to some very ugly code. The C-coded parts were... well... C code! I think I would be preaching to the converted if I started in on C's deficiencies in this forum :-)
CL for all its faults shows us that a lisp can be fast. I can't see any reason why Arc implementations won't be fast eventually too. If someone more knowledgeable can see one I'd be very interested to learn about it, but this thread doesn't seem to have come up with anything concrete.
[A word of explanation: the Python/C combination wasn't my decision, and in hindsight I wouldn't recommend it to anyone starting a new project involving numerical simulation.]
You make a good point there. However, look at the very-efficient CL code : it looks like C code. Written in CL, sure, but it often deals with calculations on vectors of fixnums where the type of everything was predefined and type checking reduced to nothing. No dynamic dispatch of generic functions on heterogenous lists here.
But an FFI does not necessarily mean you have to write C code everytime you need an efficient calculation. Have a look at the following code I've just tested :
(def fib (n)
(if (<= n 2)
n
(+ (fib (- n 1)) (fib (- n 2)))))
(time:fib 30)
-> time: 47498 msec.
<< Hidden code here >>
(def ffib (n)
(if (<= n 2)
n
(+ (ffib (- n 1)) (ffib (- n 2)))))
(time:ffib 30)
-> time: 4635 msec.
More than 10 times faster, and they are both pure Arc code. What's the magic part ? Well, it just imports new declarations of +, - and <= I previously wrote in C (with the FFI I worked on recently). These definitions only deal with fixnums, that's why they are fast. Here is the full code :
(w/inline "
char inf (long a, long b){
return a <= b;
}
long minus (long a, long b){
return a - b;
}
long plus (long a, long b){
return a + b;
}"
(cdef _<= "inf" cbyte (clong clong))
(cdef - "minus" clong (clong clong))
(cdef + "plus" clong (clong clong))
(def <= (a b) (is 1 (_<= a b))))
Well, that declaration could be encapsulated into something like :
(declare-numeric
(def ffib ...
And you would end up with something as fast as CL (and looking like optimized CL code). Well, not really, but this is still alpha version.
Actually, if you are using Anarki, a lot of the slowness comes from infix.arc redefining math operators. Just removing infix.arc from libs.arc will increase the speed of math operations by about an order of magnitude. In fact, removing infix.arc actually produced more of a speedup for me than using your ffi.
With infix.arc:
arc> (def fib (n) (if (< n 2) 1 (+ (fib (- n 1)) (fib (- n 2)))))
#<procedure: fib>
arc> (time (fib 30))
time: 34672 msec.
1346269
Without infix.arc:
arc> (def fib (n) (if (< n 2) 1 (+ (fib (- n 1)) (fib (- n 2)))))
#<procedure: fib>
arc> (time (fib 30))
time: 4219 msec.
1346269
Yeah, sorry about that. I wasn't thinking about performance when I originally put infix.arc up on Anarki, and since I was doing a lot of testing at the time I found it convenient to add it to libs.arc. Perhaps it would be best to leave it out by default though.
Oh that's right... On my machine, the FFI version is still a little faster (about 20% faster), but not that much. It now deals more efficiently with boolean values however, that might explain it... Anyway, I don't know if it's worth using FFI this way now... Not until we can compile Arc code to efficient C code at least :)
Thanks for the counter example, you've made a very elegant point. It's quite encouraging in fact. It seems to point to the strength of Arc. I think this would require a lot more effort to achieve in Python.
Of course in this case one would hope that the fundamental arithmetic and relational operators would already be implemented efficiently in an Arc native binary compiler, but the principle you've illustrated still holds regardless.
