A possible problem is that I can see myself wanting to use this syntax a lot for accessing hash tables with symbols as keys. But ! is a lot more annoying to type than . is.
My alternative suggestion (which I'm sure has worse problems of its own) would be
x.y -> (x 'y) if y is a symbol, (x y) otherwise
x,y -> (x y) (by analogy with , inside backquote)
x!y -> (x 'y) (if there should ever be a need to quote a non-symbol?)
Edit: Also in favour of my proposal, ! looks too much like a letter l:
