Arc Forum | Feature Suggestion: Dynamic (fluid) binding

Arc Forum

Feature Suggestion: Dynamic (fluid) binding

2 points by jazzdev 6073 days ago | 20 comments

Dear Paul,

I think adding dynamic (fluid) binding to Arc would make code smaller. Consider a debugger:

  (local (help backtrace up down print eval-in goto-frame)
  
    (def debugger (stack env)
      (with (run t framenum 0 framecnt (1- (len stack)))
        (while run
          (print)
          (let it (read)
            (if (is it 'q) (set run nil)
                (is it 'u) (up)
                (is it 'd) (down)
                (is it 'f) (goto-frame (read))
                (is it 'e) (eval-in (read))
                (is it 'b) (backtrace)
                           (pr "Unknown command: " it))
  
    (def print ()
      (pr "\n" framenum ":\t" (car (nth framenum stack))))
  
    (def backtrace ()
      (pr "    Backtrace:\n")
      (let n -1
        (each frame stack
          (prn (++ n) ":\t" (car frame)))))
  
    (def up ()
      (if (is framenum framecnt) (prn "\tAt top")
          (++ framenum)))
    ...

What seems intuitive to me is that stack, env, framenum and framecnt should be dynamically bound. print is called in a (dynamic) environment where framenum is defined, so it should just be able to reference it.

Possible fix #1: create fluid-with macro

      (fluid-with (run t framenum 0 framecnt (1- (len stack)))

This was suggested in http://arclanguage.org/item?id=4339, but to make this thread-safe you have to ensure that only one thread at a time can run debugger.

It doesn't make the code longer, but it also doesn't allow multiple threads to run debugger concurrently I don't see how this solution could work.

Possible fix #2: thread-local variables

Create a thread-local macro that is like "with" but makes all the variables thread-local. The bang syntax could be used to make referencing easier, but it still makes the code longer.

      (thread-local (run t framenum 0 framecnt (1- (len stack)))
  ...
    (def print ()
      (pr "\n" framenum ":\t" (car (nth tl!framenum tl!stack))))

Assume (tl 'framenum) gets the thread-local value for framenum. Acceptable, but it makes the code bigger.

Possible fix #3: just pass arguments to functions

          (print framenum stack)
  ...
    (def print (framenum stack)
      (pr "\n" framenum ":\t" (car (nth framenum stack))))

This seems straight-forward. Not bad for print, but up has to be called as:

                (is it 'u) (= framenum (up framenum framecnt))

Acceptable, but makes the code even bigger.

Possible fix #4: macros

Could define 'up' as a macro

    (mac up ()
      (if (is framenum framecnt) (prn "\tAt top")
          (++ framenum)))

Now it has access to those variables. The only downside is that now I have write the code bottom up.

    (mac up ()
      (if (is framenum framecnt) (prn "\tAt top")
          (++ framenum)))
    ...
  
    (def debugger ...

I guess using macros to deal with variable scoping is elegant enough. My only beef with this is that I think the code is less readable bottom-up. If I'm reading the debugger code. Don't I want to read it top down? I can't understand what up does until I read debugger. So I don't like this solution either because "Programs must be written for people to read," right?

I ran into this again wanting to write a sql module that could be used something like this:

  (with-open conn* (sql-conn jdbc_url username password)
    (with-open stmt* (sql-stmt)
      (let rs (sql-select "BlahID from BlahTable")
        (while (next rs)
          ...
        ))))

sql-stmt uses conn* and sql-select uses stmt.
Now alternative #2 (thread-local variables) is even less appealing because I need a special version of with-open that calls thread-local instead let.
Alternative #3 isn't bad, but does make the code bigger:
(with-open conn* (sql-conn jdbc_url username password) (with-open stmt* (sql-stmt conn*) (let rs (sql-select stmt* "BlahID from BlahTable") (while (next rs) ... ))))
Dynamic binding seems so intuitive to me. Why isn't it the default binding? Is this another onion in the varnish? The logic would seem to be that it makes code more robust to only have lexical binding. But that's just another way of saying it makes it harder to write bad programs. And Arc isn't designed to keep people from writing bad programs, right?
Have I missed an elegant solution that doesn't make the code bigger, harder to read or not thread-safe?

4 points by jazzdev 6072 days ago | link

Of course! It came to me in my sleep. The standard scheme solution:

Possible fix #5: local methods

  (def debugger (stack env)
    (with (run t framenum 0 framecnt (1- (len stack)))
  
      (def repl ()
        (while run
          (print)
          (let it (read)
            (if (is it 'q) (set run nil)
                (is it 'u) (up)
                (is it 'd) (down)
                (is it 'f) (goto-frame (read))
                (is it 'e) (eval-in (read))
                (is it 'b) (backtrace)
                           (pr "Unknown command: " it)))))
  
      (def print ()
        (pr "\n" framenum ":\t" (car (nth framenum stack))))
  
      ...
  
      (repl)))

This is even shorter because I don't need the outer environment to hide the helper functions from the global environment. Duh! Why didn't I see this sooner?

-----

2 points by absz 6072 days ago | link

There we are, closures to the rescue :) Careful, though; remember that what you define with def will be accessible throughout the entire program. Use let or with for local functions. (I can't tell what you want there, but that's tripped people up on these fora before.)

-----

1 point by absz 6073 days ago | link

I have never liked dynamic binding, and here (adapted from http://en.wikipedia.org/wiki/Scope_(programming)) is a classic problem with it. With both dynamic and lexical binding, the following code executes with the same result:

  arc> (= x 0)
  0
  arc> (def f ()
         x)
  #<procedure: f>
  arc> (def g ()
         (let x 1
           (f)))
  #<procedure: g>
  arc> x
  0
  arc> (f)
  0

However, the result of g is different. With lexical binding, as we have things now

  arc> (g)
  0

With dynamic binding

  arc> (g)
  1

Closures no longer seem very closed, do they? This behaviour is entirely counter-intuitive. Or suppose that we have the following with lexical binding:

  arc> (def mymap (f xs)
         (if (and (alist xs) xs)
           (cons (f (car xs)) (mymap (cdr xs)))
           xs))
  #<procedure: mymap>
  ; ... time passes ...
  arc> (= f [* _ _])
  #<procedure>
  arc> (mymap [f _] '(1 2 3 4 5))
  (1 4 9 16 25)

With dynamic binding, the last call ((mymap [f _] '(1 2 3 4 5))) would instead be an infinite loop: let F be [f _]. Then mymap's (f (car xs)) would be (F (car xs)). But the f referenced in F is actually F, as that's what f means in the local scope of mymap. So this is equivalent to ([F _] (car xs)), which is thus equivalent to ([[F _] _] (car xs)), which is equivalent to...

Dynamic binding is leaky, opens closures, and enables spooky action at a distance. That is why it is not the default binding.

-----

1 point by jazzdev 6072 days ago | link

> Closures no longer seemed closed, do they?

I would argue that

  (def f ()
    x)

Is not closed over x. x is a free-variable.

> This behavior is entirely counter-intuitive

What's counter intuitive to me is:

  arc> (= x 10)
  10
  arc> (f)
  10
  arc> (let x 12 (f))
  10

How come I can set x globally and affect f, but I can't set x locally and affect f? That seems inconsistent to me.

-----

3 points by almkglor 6072 days ago | link

The problem is the inherent difference between binding x in f where it is defined as opposed to binding x in f where it is called.

Dynamic binding fails in this condition:

  (let x 42
   (def set-x (new-x)
     (= x (+ new-x 1)))
   (def f ()
     x))

The point is that x does not have a local or a global lifetime. Instead, it has a local or a global scope. The lifetime of every variable is like that of any object: when everything that can access it is lost, it gets garbage collected. That's even more consistent: not only are values automatically managed, locations are automatically managed too.

Global variables are always accessible and therefore will never be destroyed. But local variables might die as soon as the function exits, or it might live on. The only functions that have the right to access them, however, are those that are defined specifically to be within its scope. This means we have automatic protection.

Granted, you want to release this protection; presumably you have some plan to have programmers explicitly ask for the protection. I've programmed in a language which supports both dynamic and syntactic binding, and I always prefer using the syntactic binding. This is because while hacking into some other function's global is cool, some day you are going to do this accidentally, and when that day comes, you'll know why a library that used to work suddenly fails.

-----

1 point by jazzdev 6072 days ago | link

> This is because while hacking into some other function's global is cool, some day you are going to do this accidentally, and when that day comes, you'll know why a library that used to work suddenly fails.

Isn't that one of the goals of Arc? A hackable language?

That's a good reason if Arc is designed for average programmers, but pg says it is explicitly designed for good programmers.

-----

3 points by absz 6072 days ago | link

This, it seems to me, is a very different use of "hack." It's more akin to forbidding (scar 'blue 'red), though not quite as severe: the latter is something nonsensical, and though you could hack together a meaning for it, it would certainly break down. The former case hacks around the definition of a function, but is also certain to break. These uses of "hack" are the ugly kind, closer to "kludge", not the beautiful or powerful kind.

-----

2 points by almkglor 6072 days ago | link

LOL. This is where the semi-deliberate delicious ambiguity of "hack is kludge! what you mean? xy is x times y but 23 is not 2 times 3?" vs "hack is elegant! what you mean? e^(i*pi) = -1?" rears its ugly head.

-----

1 point by jazzdev 6072 days ago | link

>Dynamic binding fails in this condition:

  (let x 42
   (def set-x (new-x)
     (= x (+ new-x 1)))
   (def f ()
     x))

There are no free variables in set-x or f (except for +) so dynamic binding can't affect this (except for +).

I'm not advocating releasing any protection on lexical variables. I'm advocating a change in the semantics of free variables.

-----

4 points by almkglor 6072 days ago | link

Right. So tell me, suppose I have a very, very old piece of code, a classic, like this:

   (def map1 (f xs)
    " Return a sequence with function f applied to every element in sequence xs.
      See also [[map]] [[each]] [[mappend]] [[andmap]] [[ormap]] "
    (if (no xs)
        nil
        (cons (f (car xs)) (map1 f (cdr xs)))))

Now suppose in the future some random guy creates an application focused on cars. Since the guy knows he won't use 'car and 'cdr (car? cdr? pfft. that's so low-level! use proper let destructuring, foo!), he's quite willing to reuse the name 'car as a variable:

  (let car (get-user-car)
    (map1 color-wheel car!wheels))

Oops! map1's car function got overridden!

This is a rather contrived example, but hey, believe me: it'll happen, and it won't be as obvious as that. That's why dynamic variables are explicit in CL. That's why the default binding will be static.

Now if someone wants to explicitly have dynamic variables, it's already done. http://arclanguage.com/item?id=2497

-----

1 point by jazzdev 6072 days ago | link

>Now suppose...

  (let car (get-user-car)
    (map1 color-wheel car!wheels))

>Oops! map1's car function got overridden!

That's a good argument. I can see the havoc that would result.

Still, I'm willing to risk that so that I have the option to do something like:

  (with (ocar car car-counter 0)
    (let car (fn (xs) (++ car-counter) (ocar xs))
      (map1 color-wheel car!wheels))
    (prn "map1 called car " car-counter " times"))

-----

1 point by absz 6072 days ago | link

The problem is that you then have a function whose return value depends on where it is called. Not when, but physically where. Also, imagine the havoc that would be reached with a memoized function: if f were memoized, then its return value would change depending on where it was called first. You could certainly argue that you shouldn't memoize f, but I think that you would find that this cropped up with functions that were actually a good idea to memoize.

The problem with being able to change f's return value by wrapping it in a lexical scope is that it destroys the "black box" abstraction of a function. You have to know how f is implemented to get a certain effect; alternatively, if you don't know how f is implemented, a simple change of variable name could wreak havoc with your program.

Actually, that name-change issue reveals another problem with dynamic binding: dynamic binding, I think, destroys alpha-equivalence (a fundamental property of the lambda calculus: http://en.wikipedia.org/wiki/Lambda_calculus#.CE.B1-conversi...). It might even play with the ability to do beta-reduction, but I'd have to think about that one more.

-----

2 points by jazzdev 6072 days ago | link

>dynamic binding, I think, destroys alpha-equivalence

Alpha-equivalence applies to bound variables.

Dynamic binding applies to free variables.

-----

1 point by absz 6072 days ago | link

Gah, same gotcha as before. Mea culpa. I still stand by the first two paragraphs, though; it may not be alpha-equivalence, but changing names should rarely, if ever, affect distinct parts of the program.

-----

1 point by jazzdev 6072 days ago | link

> With dynamic binding, the last call ((mymap [f _] '(1 2 3 4 5))) would instead be an infinite loop

No, the parameter names in a function are in the lexical scope so they can't be modified by a dynamic scope.

That is, f is not a free variable in mymap. It get's lexically bound to the value of the first argument.

-----

1 point by absz 6072 days ago | link

Fair enough. That's my fault: I wasn't entirely clear on dynamic binding. The rest of my objections do still stand, however :)

-----

4 points by cchooper 6072 days ago | link

Does anyone use dynamic scoping enough to make it the default behaviour? I know I don't.

-----

1 point by eds 6073 days ago | link

> Dynamic binding seems so intuitive to me. Why isn't it the default binding?

From http://www.findinglisp.com/blog/2004/05/special-vs-lexical-v... :

"In the old days, they say (I wasn't there), many Lisps had only special variables (are they really that special when they're all you have?). The problem is that bugs associated with the dynamic/special behavior are quite difficult to debug. You can have one part of the program affect another part and it isn't clear what the connection between the two is unless you're looking at the call graph."

> Have I missed an elegant solution that doesn't make the code bigger, harder to read or not thread-safe?

Perhaps CL style special variables? You can reference them without extra code (although you need naming conventions to prevent mixing lexical and special variables), and I think most CL implementations automatically make thread-local copies when using special variables in a thread. (See the rest of that article I linked above.)

-----

1 point by jazzdev 6072 days ago | link

>The problem is that bugs associated with the dynamic/special behavior are quite difficult to debug. You can have one part of the program affect another part and it isn't clear what the connection between the two is unless you're looking at the call graph.

Yep, that's a reasonable argument for a language designed for average programmers. But if Arc is designed for expert programmers then I think expert programmers will want to make a different trade-off. I'll risk shooting myself in the foot to get a more powerful language.

> Perhaps CL style special variables?

Yep, that would be good. That would be adding dynamic binding to Arc.

-----

1 point by almkglor 6072 days ago | link

http://arclanguage.com/item?id=2497

-----