Okay, forgive me if I'm retarded, but it's been a while since I've done any lisp and reading someone's source always takes a while.
I was trying to go through news.arc and figure out how it deals with memory, and as best I can tell, it doesn't seem to. It looks like it just loads some n most recent stories/posts on startup, and accumulates new ones in memory while simultaneously saving them.
This seems to me that if left running, the app will eventually suck up memory and need to be restarted. Is this the case, or did I miss the line in the implementation where it flushes old stories out?
I'm not convinced of that. It appears to be a macro for building a cache of various things with a timeout. It doesn't appear to touch the stories* table.
On an unrelated note, can anyone explain to me the memory model/threading mechanism interaction? It seemed at first that it was a single-threaded server, but after looking at srv.arc, it's clear that multiple threads are being spawned for the server process, yet stories, profs, and votes* aren't don't seem to be rebuilt from permanent storage, which implies the tables can be shared between processes. Which, I guess, goes back to my original question. So this isn't unrelated at all.
Hmm. After another cursory glance, yes, this appears about correct.
As for memory model: global variables are always shared between threads. The underlying implementation automatically locks tables when mutating them AFAIK.
As for suck up memory: I suppose the thinking is that a single post may very well take less than half a kilobyte, and modern computers have gigabytes of memory, so...
That was my thought too. I suppose I may have been overestimating the traffic to HN, or the size of the server or both. I guess I wanted a definitive answer about having to restart the server periodically. That seems to be the case, but it doesn't seem to be too bad.
This seems to be a pretty good model for a moderate-traffic app with data that gets modified infrequently. I don't think I'm quite ready to arc just for that, but I was wondering if this is a sensible approach in any other language, say, python? Any thoughts?
Well, Arc is still starting. In more-established languages, such approaches are unnecessary: there are libraries which will handle caching of data etc. properly, releasing unused memory for garbage collection once they are no longer accessed for some time, and rebuilding objects from permanent store if they have been disposed. In Arc it's still not yet implemented, so that approach works fine.
That said you might be interested in the so-called "Anarki" repository, which contains some of the elements I and others have built so that the server works a little better. For example: being able to serve files in subdirectories of your Arc installation, instead of the Arc installation; table-like data structures for caching data, or for persistent disk-based data; a slightly more extensible language, with some of the more common methods of extension already prepackaged in macros; etc.
Can you give an example of such libraries, off the top of your head?
I am basically interested in small multi-user applications that don't sit on top of relational databases. There really isn't too much information out there on the matter. Everyone seem to want to use a database, even for the simplest things. I suppose that given the pedigree of arc, this flat file storage business seems sensible enough.
Anyway, thanks for the pointer to anarki. I'll take a look at it.
For that matter, most languages prefer db's because of the fact that file storage operations don't have, umm, structure.
In fact the canonical Arc web app, news.arc, has a list structure to store in the "flat" file. Thus for simple apps where entities only have a few not very complex fields, textual representations of lists seem to be enough.
In other languages however their "array" syntax (which is approximately what lists are in Arc) is usually not readable by a built-in function a la lisp 'read. Also, their array syntax is usually not the center of attention, unlike in Lisp where the code syntax is itself the "array" syntax.
