Experimental IRC log happs-2007-11-06

Available formats: content-negotiated html turtle (see SIOC for the vocabulary)

Back to channel and daily index: content-negotiated html turtle

These logs are provided as an experiment in indexing discussions using IRCHub.py, Irc2RDF.hs, and SIOC.

05:11:07<ac>hello
08:19:25<kc5tja>ΒΆ/window close
09:08:12<ac>I just installed HAppS about 5 minutes ago, but it seems to me at least part of the framework is far more general than web applications
09:09:28<ac>Seems to me that recording state changes to disk and regenerating that state, and having the state be represented by data structures that are automatically indexed could be applied to all sorts of applications, like a word processor for example
15:16:42<shapr>@yow !
15:27:19<ac>Is this a reasonable observation? Or do I have no clue what that part of HAppS does?
15:35:50<shapr>ac: Yes, I agree.
15:35:58<shapr>I think it's a very general framework.
15:36:19<ac>I have a number of questions about HAppS
15:37:03<ac>from the general to the specific. I've already started with the general ;)
15:37:55<ac>If HAppS stores everything in memory, what happens when your state exceeds the size of your memory?
15:38:46<ac>I assume the answer is "get more memory", "use swap", or "scale to multiple machines"
16:02:53<shapr>We're working on the 'scale to multiple machines' part.
16:03:09<shapr>But yes, the other answers are currently valid.
16:03:57<ac>swap isn't very attractive though because the OS isn't very intelligent about what it swaps, or is that in practice not actually true?
16:05:20<ac>in practice it's quite disasterous to actually run out of memory and start using swap space (in fact, I don't even have swap). It would be nice if you could flag portions of your data structure as low priority and have the OS swap it out before more important things get swapped out
16:05:33<ac>unfortunately, AFAIK, you can't do that with current OSs
16:06:05<shapr>Yeah, that's true.
16:07:07<ac>you could compensate for OSs lack of intelligence by automatically storing portions to disk, but then you'd end up with something suspiciously like an existing RDBMS
16:07:30<ac>except it'd be native to Haskell
16:08:34<shapr>Yeah, that would be nifty.
16:08:53<shapr>We're working on multimaster right now, but sharding will be next.
16:09:06<shapr>With sharding, different machines have different chunks of the state.
16:10:59<ac>sounds like then you'll have something pretty similar to mnesia (though I haven't actually written a line of Erlang code)
16:11:12<shapr>Yeah, pretty much.
16:11:16<shapr>We've noticed that resemblance :-)
16:11:24<ac>I figured.
16:12:03<shapr>We're also working on zero-downtime upgrading of binaries, and that's ending up looking like Erlang's hot code upgrading.
16:12:21<shapr>Though I don't see how we can get all of Erlang's benefits even with the ghc-api.
16:12:29<ac>How could I use just the state storage and indexing portion of HAppS. Specifically I currently want to use the fastcgi library from hackage for HTTP stuff, but I'm also interested in things completely different from web apps
16:13:02<shapr>I know lemmih got HAppS working with FastCGI
16:13:16<ac>oh nice. I'd like to see the code
16:13:19<shapr>IxSet is the indexing portion, I think it shouldn't be hard to separate.
16:13:26<ac>which is the state part?
16:13:39<shapr>I don't know how hard it would be to run HAppS-State separately.
16:14:17<shapr>I do wish my desktop apps used transactions. I wouldn't lose work that way...
16:14:47<ac>Yeah, I'm actually interested in writing something like a text editor that uses a HAppS-like framework
16:15:56<ac>What do you mean by "HAppS-State"? I don't see a module I can import that corresponds to that
16:16:17<ac>Would that be HAppS.Agents.*?
16:26:20<ac>BTW, where would I find lemmih, or lemmih's code that uses FastCGI and HAppS?
16:26:27<shapr>HAppS is split into several repositories, HAppS-State is one of them.
16:26:41<shapr>Lemmih will be back online when he's fixed his internet problems.
16:26:52<shapr>He's one of the people paid to work on HAppS (like me).
16:27:05<ac>interesting. Who pays y'all?
16:27:17<shapr>alexj is the man behind it all.
16:27:46<shapr>Check out the talk and slides link from the top right of http://happs.org/
16:28:25<ac>shapr: already watched one (the one you can find everything I know about HAppS in)
16:32:38<ac>should I be experimenting with version .8.8, or .9.1?
16:32:46<shapr>0.9.1
16:33:07<ac>I pasted a darcs command from a tutorial and that grabbed .8.8
16:33:11<ac>How do I get .9.1?
16:34:03<shapr>http://happs.org/HAppS-Begin/public/index.html
16:36:03<ac>should I uninstall the current version, with something like "ghc-pkg unregister HAppS"?
16:36:10<shapr>hiya alexj !
16:36:32<alexj>good morning
16:37:24<shapr>ac: What tutorial pointed you to 0.8.8 ?
16:37:27<shapr>I'd like to fix that.
16:38:21<ac>http://bluebones.net/2007/09/how-to-install-happs/
16:38:40<ac>darcs get --partial http://happs.org/HAppS
16:41:29<shapr>Thanks, I'll put up a comment.
16:42:08<ac>I was wondering about that. I figured I should be using .9.1
16:43:09<shapr>Yup, any other questions?
16:43:26<ac>I think I just need to read a couple tutorials and experiment with it
16:43:59<shapr>Feel free to ask questions here on #happs, if I'm not deeply focussed on something I'll answer what I can.
16:44:05<ac>And see if it solves all my web programming woes ;)
16:44:35<ac>ACTION dreads going back to work where he'll be forced to write PHP
16:45:26<shapr>ick
16:46:22<shapr>I used to do J2EE. J2EE does more that PHP, but the overhead is painful.
16:47:03<ac>I think I'd be a good deal happier writing Java
16:47:22<shapr>Yeah, I can see that.
16:47:46<shapr>But there's a lot of, um, 'unnecessary effort' (read feces) involved in both.
16:47:49<alexj>ac, you in nyc?
16:47:58<ac>alexj: no, why do you ask?
16:48:08<alexj>speakeasy.net I think started in nyc.
16:48:13<shapr>ac: Your hostname implies Seattle, is that correct?
16:48:17<ac>yep
16:48:24<shapr>I used to live in First Hill.
16:48:24<alexj>ah ok.
16:48:34<ac>I'm right next to Green Lake now :)
16:48:38<shapr>Ah, cool!
16:48:45<ac>shapr: don't tell me you're in Seattle too...
16:48:56<shapr>No, I'm in Alabama now.
16:49:28<shapr>ac: Are you part of the PDX Haskellers?
16:49:44<ac>never heard of 'em. Perhaps I should be?
16:50:10<ac>I started learning Haskell about 3 weeks ago
16:51:06<shapr>The biggest concentration of commercial Haskellers is in Portland, Oregon.
16:51:53<shapr>That whole thing started with Portland State University, and spawned at least one Haskell company, Galois Connections @ galois.com
16:53:13<ac>galois sounds like a fun company to work for
16:53:42<ac>problem is, I think my job is really awesome, even though currently I'm technically unemployed. I guess it's kind of a long story
16:54:18<shapr>Are you getting paid?
16:54:37<ac>Not at the moment, which is why I'm technically unemployed, and learning Haskell ;)
16:55:40<ac>Next Friday I'll most likely know for sure if my job still exists
16:58:28<shapr>Ah, I see.
16:58:46<ac>why do you ask though?
17:14:20<shapr>I guess that's how I divide having a job from not having a job.
17:14:24<shapr>If I still get paid, I have a job.
17:14:55<ac>I'm thinking of my current situation as a vacation
17:20:42<ac>(with an extra bonus of possibly not having a job at the end of it)
17:24:36<shapr>heh
18:14:33<ac>"darcs get" grabs the entire history of the repository right? Isn't there an option to only get what's current?
18:15:15<ac>I guess that would be "darcs get --partial"
18:17:20<ac>hmm. Compiling Data failed with "No instance for (Data DefaultD BSC.ByteString)" in src/HAppS/Data/Default.hs:114:0
18:41:30<Saizan_>ac: you need darcs version of syb-with-class
18:42:39<ac>thought I already installed that
18:42:55<ac>yeah, I have syb-with-class-0.3
18:48:14<ac>maybe that's not a recent enough version, or perhaps I'm missing some hidden dependency
19:13:32<Saizan_>0.3 from hackage is different than 0.3 from darcs
19:40:06<ac>I see. Hopefully by tomorrow I'll have exhausted my stupid questions
19:40:13<ac>:)
19:42:15<ac>I have Data, IxSet, and State installed
19:46:16<ac>looks like the API was restructured from .8.8
19:54:50<Saizan_>yeah
21:02:22<ac>how do I forget old state?
21:03:58<ac>well, I guess I don't want to forget it entirely. I want to store a lot of small messages that over time (like a year) will eventually be large enough to not want to all store in memory
21:04:17<ac>but the older they are, the less likely they are to be read
21:05:51<ac>it would be nice to have them in a full text index somewhere, but not absolutely necessary
21:31:27<ac>just swapping things intelligently seems like it would be by far the most elegant solution :-/
21:39:53<shapr>Some sort of least recently used setup?
21:49:36<ac>shapr: or even simpler, date they were entered originally
22:13:52<ac>so, old stuff I want to be randomly accessed from disk (as in a database like sqlite, berklydb, etc), new stuff I want in memory because it gets queried all the time
22:14:42<ac>might as well stick everything in the DB to begin with and do away with the default storage. Which means all I have to do is make a new Read and Show
22:16:03<ac>Then forget anything beyond, say 5000 records, and then save the lowest ID in memory. Fetch would then check if the ID is less than the lowest, and if so, fetch it from the database
22:16:57<ac>seems like you get the best of both worlds: fast, scalable, painless in-memory data but everything is in a database if you need it there
22:17:25<ac>writes to the database can be as low a priority as you want
22:18:24<ac>the worst drawback is you have to add some extra configuration crap for the database
22:27:51<ac>I'd imagine someone has made a "haskell type -> sql table definition" function. Even something rather unsophisticated would be fine, like a table with ID, entry date, and a string containing the serialized type
22:28:16<ac>(at least for my purposes)
22:31:41<ac>Does this sound reasonable, or is this completely contrary to HAppS as it exists?
22:32:36<alexj>ac: my general belief is that you want to actively manage what is in disk and what is in memory.
22:33:03<alexj>if you are archiving stuff, in general you can save it as a blob in s3 and then just store a URL in memory.
22:33:30<alexj>read/show and their equivalents are good enough for getting the data back when you want it without much pain.
22:37:19<ac>alexj: but then you have to roll your own on-disk indexing, or just do without it
22:39:17<alexj>what sort of on disk indexing do you want?
22:39:42<ac>just a simple tree
22:40:38<ac>and full text I guess
22:42:55<ac>To be honest, whenever I have a medium amount of data, I always drag out a database system
22:43:41<ac>Even when it's not suited for the problem at hand (like one time I did lots of sequential processing, and MySQL basically sucked for that :P)
22:52:31<ac>I can easily imagine storing an index of files in memory that contain chunks of serialized data structure, but that's just a hack that accomplishes the same thing as BerklyDB, or haskelldb-flat
22:55:15<ac>scratch the full text though. there's no reason to store that on disk. that stays in memory
22:56:48<alexj>not sure I understand your use case.
23:00:39<ac>alexj: I don't entirely either. I'll shut up for the time being

Back to channel and daily index: content-negotiated html turtle