Available formats: content-negotiated html turtle (see SIOC for the vocabulary)
Back to channel and daily index: content-negotiated html turtle
These logs are provided as an experiment in indexing discussions using IRCHub.py, Irc2RDF.hs, and SIOC.
| 07:42:57 | <mae_> | hola, just uploaded hopefully a much more robust version of build auto |
| 07:43:40 | <mae_> | if you setup a new guestbook app, try: happstack build auto "cabal build" dist/build/guestbook-server/guestbook-server |
| 14:01:00 | <mae_> | good morning! |
| 19:31:25 | <gcollins> | is anyone around? |
| 19:37:34 | <abuiles> | Hi |
| 19:38:15 | <abuiles> | gcollins: How is it going ? |
| 19:38:38 | <gcollins> | not bad! i'm getting married in a couple of weeks so i'm pretty busy |
| 19:39:01 | <gcollins> | i was wondering if anyone had anything else to say about re-doing the http stack for 0.4 |
| 19:39:36 | <gcollins> | there are a couple of things everyone's been asking for (sendfile support, simpler monad stack, etc) |
| 19:39:56 | <abuiles> | have you checked build auto ? |
| 19:40:25 | <gcollins> | no, not yet |
| 19:40:28 | <abuiles> | I'm dunno about the formers. |
| 19:40:37 | <gcollins> | i'm personally not super-interested in that feature |
| 19:41:02 | <abuiles> | I see |
| 19:42:14 | <abuiles> | Did you know something about lockfile ? |
| 19:42:30 | <abuiles> | I mean, I don't know what is that about .... |
| 19:43:07 | <gcollins> | I know a little about lockfile stuff, just from programming on unix for years |
| 19:45:43 | <abuiles> | any reference ? |
| 19:47:00 | <abuiles> | never mind,, I just went to wikipedia |
| 19:48:50 | <mae_work> | gcollins: you don't find "build auto" useful? :) |
| 19:49:08 | <mae_work> | i was thinking it quite handy |
| 19:49:29 | <gcollins> | I haven't tried it yet, actually -- right now I'm mostly treating happstack like a convenient http library |
| 19:49:37 | <gcollins> | i don't use the state stuff either :( |
| 19:49:54 | <mae_work> | heh |
| 19:50:02 | <mae_work> | well basically it can be used for anything that you build with ghc |
| 19:50:16 | <mae_work> | it abuses ghc to find out if there are changes (because ghc only links a new binary if there is) |
| 19:50:33 | <gcollins> | i just skimmed the code -- it runs that periodically in a loop, right? |
| 19:50:36 | <mae_work> | so anything that you are creating with either a build script or is using something like cabal, can use it |
| 19:50:40 | <mae_work> | yeah |
| 19:50:51 | <mae_work> | builds then waits 5 secs |
| 19:50:57 | <gcollins> | a nice-to-have there would be a binding to inotify |
| 19:51:05 | <mae_work> | not a mac user :) |
| 19:51:13 | <mae_work> | "i'm not super interested in that feature" |
| 19:51:20 | <gcollins> | :) |
| 19:51:21 | <mae_work> | you mean for growl right |
| 19:51:59 | <gcollins> | no -- inotify is a filesystem-watching api. so you can watch a directory and get a notification if the contents change |
| 19:52:00 | <mae_work> | but yeah basically the way i use it in the guestbook app (it has a cabal file) |
| 19:52:15 | <gcollins> | on osx i think you'd use epoll |
| 19:52:22 | <mae_work> | happstack build auto "cabal build" dist/build/path/to/my/output/bin |
| 19:52:27 | <gcollins> | and i'm writing you from my macbook :) |
| 19:52:31 | <mae_work> | and then tail -f build.err.log |
| 19:52:41 | <gcollins> | what's the use case for this thing, btw? |
| 19:52:45 | <mae_work> | (stdout and stderr from the build command go to build.out.log and build.err.log) |
| 19:52:54 | <gcollins> | it's supposed to make development more convenient? |
| 19:52:58 | <mae_work> | um use case is, i don't wanna have to type in cabal build every time |
| 19:53:02 | <mae_work> | yeah basically |
| 19:53:06 | <mae_work> | at first i mucked around with ghci |
| 19:53:19 | <mae_work> | but the dependency check is a completely different implementation from cabal |
| 19:53:29 | <mae_work> | and it still has weird issues with template haskell |
| 19:53:36 | <gcollins> | i usually do most of my haskell development from inside emacs |
| 19:53:47 | <mae_work> | emacs does this? |
| 19:53:54 | <mae_work> | i am so used to vi i could never get into emacs |
| 19:53:54 | <gcollins> | C-c C-l punts the current buffer over to ghci |
| 19:54:10 | <mae_work> | there are disadvantages with ghci as well |
| 19:54:19 | <mae_work> | ie performance will seriously drag depending on how fat your stuff is |
| 19:54:25 | <mae_work> | you fix that by building some of the modules |
| 19:54:28 | <mae_work> | so they aren't interpreted |
| 19:54:31 | <gcollins> | also emacs has a neat integration with the interpreter so if you put the cursor over a symbol it'll go look up the type in the minibuffer |
| 19:54:31 | <mae_work> | in bytecode |
| 19:54:45 | <mae_work> | yeah that kicks ass |
| 19:54:51 | <gcollins> | yeah i wouldn't run a production system from ghci |
| 19:54:52 | <mae_work> | i guess it really depends |
| 19:55:05 | <mae_work> | if your dev cycle is code/build/check browser |
| 19:55:11 | <mae_work> | i think that is definitely the use case |
| 19:55:15 | <mae_work> | but it is not limited to that |
| 19:55:34 | <gcollins> | yeah it doesn't work so well with my workflow |
| 19:55:56 | <mae_work> | what is your workflow? :) |
| 19:56:01 | <gcollins> | which is write a function, punt to interpreter, clean up errors/warnings, THEN build a binary |
| 19:56:39 | <gcollins> | emacs makes the repl-loop style really convenient |
| 19:56:54 | <gcollins> | actually i've been meaning to talk to you about the http stuff |
| 19:57:16 | <gcollins> | i have a feeling that something is pretty broken with http pipelining inside happstack |
| 19:58:06 | <gcollins> | at work i'm writing an analytics-gathering engine using happstack -- when i connected my test program to the server using pipelining (on linux) performance went through the floor |
| 19:58:26 | <gcollins> | strace said that it was spending 99% of its time in "futex" |
| 19:58:47 | <gcollins> | waiting on some lock, either in happstack (not so sure about that) or in the runtime system (more likely) |
| 19:59:55 | <gcollins> | honestly i'm a little bit leery of the http stuff in happstack -- the lazy i/o worries me |
| 20:02:12 | <abuiles> | mae_work: Hey! ,, yeah I'm sure I rebuild again.. however, I'm doing it again. |
| 20:03:49 | <gcollins> | brb |
| 20:08:58 | <gcollins> | back |
| 20:30:59 | <mae_work> | gcollins: so you think the ghc runtime might be to blame? |
| 20:31:07 | <mae_work> | or rather the lazy-io impl |
| 20:31:15 | <mae_work> | i do see strange things happening with handles all the time |
| 20:31:31 | <gcollins> | it's possible -- i found a temporary workaround and didn't investigate further |
| 20:32:05 | <gcollins> | i tried to build it with GHC head (because people suggested that they'd fixed a lot of parallelism bugs) but the program segfaulted |
| 20:32:10 | <mae_work> | i would really like to look into using a tighter allocation loop, something like the enumerator in hyena |
| 20:32:24 | <mae_work> | not relying on lazy io |
| 20:32:27 | <mae_work> | too unpredictable |
| 20:32:49 | <gcollins> | i think that oleg guy made a good argument for that |
| 20:33:16 | <gcollins> | the question is whether to go with an enumerator-style or imperative-style implementation |
| 20:33:21 | <mae_work> | well really, i think that the actual request loop is very small |
| 20:33:28 | <mae_work> | i wonder how hard it would be to swap it out |
| 20:33:35 | <gcollins> | not hard at all i think |
| 20:33:52 | <gcollins> | i've been thinking about doing that |
| 20:33:52 | <mae_work> | what are the pros and cons of each? |
| 20:34:12 | <mae_work> | i know imperative is more error-prone, but probably more idiomatic to most people |
| 20:34:26 | <gcollins> | honestly i don't know enough about that left-fold enumerator stuff to be able to comment |
| 20:34:50 | <mae_work> | i know little about enumerators except that I guess it would use linear memory in theory right? using tail recursion? |
| 20:34:56 | <gcollins> | you know what we should do: use the server functions from the HTTP library |
| 20:35:19 | <gcollins> | yeah, i think it allows you to process things in chunks |
| 20:35:28 | <mae_work> | http://github.com/tibbe/hyena/blob/c981126f941ced0827dcef7eedff45842cd5bd91/Hyena/Http.hs |
| 20:35:42 | <gcollins> | the downside of that approach is that it's a bit of a mind-bender at first |
| 20:35:57 | <mae_work> | one thing i have considered is to simply not support http 1.0 (1.1 + only) and then we can treat everything as chunked encoding |
| 20:36:01 | <mae_work> | alleviating a number of problems |
| 20:36:07 | <mae_work> | this sort of assumes the app server pattern though |
| 20:36:12 | <gcollins> | yes |
| 20:36:14 | <mae_work> | some people want to use it as a front facing server |
| 20:36:20 | <gcollins> | i think http/1.0 support is a must |
| 20:36:37 | <tibbe> | mae_work: you called? ;) |
| 20:36:40 | <gcollins> | personally i'm proxying it through lighttpd right now |
| 20:36:48 | <mae_work> | tibbe: hehe how is hyena coming? |
| 20:36:58 | <tibbe> | mae_work: slowly but surely |
| 20:37:02 | <mae_work> | gonna release a cabal soon? |
| 20:37:21 | <tibbe> | mae_work: I'm reimplementing the internals, it'll support pipelining and all sorts of cool stuff |
| 20:37:41 | <gcollins> | that's nice code, kudos |
| 20:37:47 | <gcollins> | very clean |
| 20:37:49 | <mae_work> | gcollins: i have sort of assumed in the back of my mind that i would use hyena eventually, and maintain what we have for now (unless someone comes up with something better that is) |
| 20:37:51 | <tibbe> | mae_work: I'm doing some tricky CPS stuff now, I hope that once I sort that out I can make a release |
| 20:38:01 | <mae_work> | CPS? |
| 20:38:38 | <tibbe> | mae_work: let me put the code somewhere |
| 20:38:41 | <tibbe> | mae_work: sec |
| 20:39:05 | <gcollins> | i don't think it matters which http engine to use, as long as it's fast and will let us plumb in sendfile() |
| 20:39:14 | <mae_work> | thats true. |
| 20:39:19 | <gcollins> | CPS = continuation-passing style |
| 20:39:22 | <mae_work> | and that its compatible ?:) |
| 20:39:40 | <mae_work> | that is what i like about the engine currently |
| 20:39:46 | <tibbe> | mae_work: http://www.hpaste.org/fastcgi/hpaste.fcgi/view?id=5008#a5008 |
| 20:39:51 | <mae_work> | it has some bugs, but a lot of compatibility stuff has already been tested |
| 20:39:58 | <gcollins> | we're working to an interface here, any engine can be made to work |
| 20:40:03 | <tibbe> | mae_work: implementing all the parsing combinators in this style seems to be faster |
| 20:40:14 | <mae_work> | gcollins: right |
| 20:40:30 | <tibbe> | gcollins: to be honest I'm not sure how to expose sendfile in hyena |
| 20:40:37 | <gcollins> | i have a feeling that the HTTP library implementation is pretty good |
| 20:40:50 | <abuiles> | guys , is any way of doing unregister in cascade? I Have happstack-data-0.2.1 happstack-ixset-0.2.1 happstack-server-0.2.1 happstack-state-0.2.1 happstack-0.2.1 happstack-helpers-0.22 , but I have the latests as well. I'm just gonna leave 0.3 |
| 20:40:51 | <gcollins> | tibbe: allow access to the raw socket |
| 20:40:57 | <gcollins> | i mean |
| 20:41:02 | <gcollins> | that's a little facile |
| 20:41:04 | <mae_work> | tibbe: sendfile needs a raw socket and a filename, the rest is operating-system-level magic |
| 20:41:12 | <tibbe> | gcollins: yes that's probably the way to go |
| 20:41:25 | <tibbe> | gcollins: network-bytestring might come with a wrapper for it soon |
| 20:41:26 | <gcollins> | just advertise it as "unsafe" |
| 20:41:34 | <tibbe> | gcollins: so it works on Windows |
| 20:41:40 | <gcollins> | nice! |
| 20:41:52 | <gcollins> | i mean, the C shim for that wouldn't be too nasty to write |
| 20:41:53 | <mae_work> | wrapper for what? |
| 20:41:58 | <tibbe> | ACTION needs a windows hacker for network-bytestring |
| 20:42:10 | <mae_work> | windows nt supports sendfile (but its call transmitfile) |
| 20:42:16 | <tibbe> | mae_work: for the unix and windows variations of sendfile |
| 20:42:20 | <mae_work> | ah ok |
| 20:42:26 | <mae_work> | so network bytestring has this licked already? |
| 20:42:28 | <gcollins> | tibbe: i finally scrubbed my house of windows machines about six months ago |
| 20:42:30 | <mae_work> | or rather will |
| 20:42:37 | <tibbe> | network-bytestring now supports scatter/gather I/O |
| 20:42:49 | <tibbe> | mae_work: it will |
| 20:42:58 | <mae_work> | ok |
| 20:43:13 | <tibbe> | mae_work: been short on time lately |
| 20:43:24 | <gcollins> | i know the feeling |
| 20:43:30 | <mae_work> | gcollins: so then, http library in core sounds reasonable, but does it / will it support sendfile? |
| 20:43:32 | <tibbe> | the iteratee stuff is a bit mind boggeling on the implementation side |
| 20:44:07 | <gcollins> | well -- i haven't actually tested how robust it is, but i've poked in there |
| 20:44:18 | <gcollins> | it'll definitely support sendfile |
| 20:44:19 | <tibbe> | I've been staring at the implementation of one of the parsing combinators for a while now, I have it working in direct style but converting it to CPS is tricky |
| 20:44:48 | <mae_work> | heh |
| 20:45:01 | <mae_work> | tibbe: what are you developing hyena for? |
| 20:45:10 | <gcollins> | receiveHTTP :: HStream ty => HandleStream ty -> IO (Result (Request ty)) |
| 20:45:18 | <gcollins> | respondHTTP :: HStream ty => HandleStream ty -> Response ty -> IO () |
| 20:45:37 | <mae_work> | gcollins: so you see happstack-server as a higher level library that can run on any http engine potentially right? (this is how i see it) |
| 20:45:42 | <gcollins> | it's pretty low-level --- instead of respondHTTP you could write a custom handler that used sendfile |
| 20:45:45 | <tibbe> | mae_work: I want a web application server "to rule them all" so web framework writers don't have to roll their own every time |
| 20:45:49 | <gcollins> | mae_work: pretty much |
| 20:46:08 | <tibbe> | mae_work: I plan to standardize some interface (ala Python's WSGI and Ruby's Rack) |
| 20:46:15 | <mae_work> | gcollins: still has a way to go before it deserves that place :) like the stuff that was on the ml |
| 20:46:35 | <mae_work> | tibbe: great |
| 20:46:40 | <mae_work> | tibbe: you must support sendfile! |
| 20:46:45 | <mae_work> | then we won't have to do that work :) |
| 20:46:48 | <gcollins> | yeah it's pretty alpha still but so is everything else |
| 20:47:14 | <gcollins> | i settled on happstack because it looked more complete than the other options |
| 20:47:33 | <mae_work> | tibbe: what windows questions do you have (regarding network-bytestring) I wouldn't call myself a "windows hacker" but I actually implemented several kernel32 ffi interfaces w/ c2hs recently |
| 20:47:47 | <gcollins> | i think we should deprecate ServerPartT honestly |
| 20:48:42 | <mae_work> | right, very few cases where you would want a WebPartT but not a ServerPartT I g uess |
| 20:48:43 | <mae_work> | guess * |
| 20:49:31 | <mae_work> | and with the typeclasses we added in 0.2, it almost becomes transient now |
| 20:50:01 | <gcollins> | the whole thing can be collapsed down to something much less complicated |
| 20:50:46 | <gcollins> | and i'd prefer the datatype to be more abstract honestly -- ServerPartT makes you know way too much about its "guts" |
| 20:52:54 | <tibbe> | mae_work: I basically need someone to wrap the windows equivalents of the readv/writev system calls |
| 20:53:10 | <tibbe> | mae_work: I don't have a machine to test on |
| 20:53:27 | <mae_work> | tibbe: i can probably do that |
| 20:53:57 | <mae_work> | you have a test suite or something? so i can make sure the patch works before i send it to you? |
| 20:54:44 | <tibbe> | mae_work: there are some tests |
| 20:54:57 | <tibbe> | mae_work: if you have something that kinda works I could try to expand the test suite |
| 20:55:09 | <tibbe> | (testing socket code sucks in comparison to pure code) |
| 20:55:28 | <abuiles> | mae_work: I just checked. I rebuild and reinstall, and the process continues running. in the other hand, lockfile is working as expected :) |
| 20:55:34 | <mae_work> | tibbe: ok, well i mean I just don't know what all your looking for in your test (to make sure it behaves similarly to what you expecT) |
| 20:55:47 | <mae_work> | hmm |
| 20:55:48 | <mae_work> | thats strange |
| 20:55:51 | <tibbe> | mae_work: http://github.com/tibbe/network-bytestring/tree/master |
| 20:55:52 | <mae_work> | abuiles: what platform |
| 20:56:01 | <abuiles> | linux mint |
| 20:56:04 | <abuiles> | felicia |
| 20:56:18 | <mae_work> | thats right |
| 20:56:19 | <tibbe> | mae_work: the test suite basically creates a server/client pair and sends a string between them |
| 20:56:36 | <tibbe> | mae_work: check the tests/ directory |
| 20:56:46 | <abuiles> | if you want I can show you my screen,, vnc |
| 20:57:23 | <tibbe> | mae_work: you could cut-n-paste the testSendAll and replace sendAll by sendMany |
| 20:58:03 | <mae_work> | um |
| 20:58:08 | <mae_work> | abuiles: can't right now |
| 20:58:41 | <mae_work> | abuiles: what is the exact command you are entering? |
| 20:59:05 | <mae_work> | tibbe: so you already have the windows stuff in here no? http://github.com/tibbe/network-bytestring/blob/3f33f16bb846458b28ad6453e40de22aa8ad00ba/Network/Socket/ByteString/Internal.hs |
| 20:59:13 | <abuiles> | mae_work: happstack build auto "cabal configure & cabal install" ./dist/build/guestbook-server/guestbook-server |
| 20:59:28 | <tibbe> | mae_work: yes, contributed by someone else |
| 20:59:33 | <mae_work> | tibbe: oh |
| 21:00:09 | <mae_work> | so err, it is complete, but not tested/ |
| 21:00:09 | <mae_work> | ? |
| 21:00:32 | <tibbe> | mae_work: it's incomplete |
| 21:00:56 | <tibbe> | mae_work: check the implementation of Network.Socket.ByteString.sendMany |
| 21:01:30 | <tibbe> | mae_work: there are instructions for contributing in the README, I accept plain diffs if you don't want to set up git |
| 21:03:44 | <mae_work> | hm ok |
| 21:04:00 | <abuiles> | mae_work: In order to kill the app, I have to do 2 times ctrl-c, dunno if that is a strange behaviour . |
| 21:04:27 | <mae_work> | abuiles: so you don't get the prompt back until a second ctrl-c? |
| 21:04:53 | <abuiles> | mae_work: exactly . |
| 21:06:28 | <abuiles> | could be something of the distro ? |
| 21:08:08 | <mae_work> | tibbe: can't find a winsock function that looks similar to readv and writev, http://msdn.microsoft.com/en-us/library/ms741394(VS.85).aspx |
| 21:08:31 | <mae_work> | abuiles: nah, what version of ghc? |
| 21:09:28 | <abuiles> | 6.10.2 |
| 21:13:08 | <tibbe> | mae_work: WSARecv |
| 21:13:09 | <mae_work> | nm found it |
| 21:13:11 | <mae_work> | http://msdn.microsoft.com/en-us/library/aa365469.aspx |
| 21:13:23 | <mae_work> | ReadFileScatter Function |
| 21:13:37 | <tibbe> | mae_work: or so I've read |
| 21:13:40 | <mae_work> | win2k + |
| 21:14:00 | <mae_work> | WriteFileGather |
| 21:14:01 | <mae_work> | http://msdn.microsoft.com/en-us/library/aa365749(VS.85).aspx |
| 21:14:08 | <tibbe> | mae_work: http://msdn.microsoft.com/en-us/library/ms741688(VS.85).aspx |
| 21:14:14 | <tibbe> | mae_work: has several buffers |
| 21:14:49 | <mae_work> | hmm |
| 21:15:12 | <mae_work> | wonder what the difference is |
| 21:15:20 | <mae_work> | between WSARecv and ReadFileScatter |
| 21:16:12 | <abuiles> | mae_work : which version of ghc are you using ? |
| 21:16:25 | <mae_work> | abuiles: 6.10.1 last i checked i think |
| 21:16:37 | <mae_work> | it could be that the new ghc handles ctrl-c differently |
| 21:16:44 | <mae_work> | because there are two threads which need to die |
| 21:17:46 | <abuiles> | mae_work: I'm going to install 6.10.1 to check. whether it works or continues the same. |
| 21:17:59 | <tibbe> | mae_work: not sure, I think the WSA one is the one to use as the other WSA functions are socket functions |
| 21:18:24 | <mae_work> | ah |
| 21:18:35 | <mae_work> | so ReadFileScatter is probably more high level |
| 21:18:51 | <mae_work> | ah ok |
| 21:18:58 | <mae_work> | ReadFileScatter takes a file handle |
| 21:19:05 | <mae_work> | WSA takes a socket |
| 21:19:12 | <mae_work> | (these are the same on linux no? |
| 21:21:06 | <mae_work> | tibbe: when will you implement sendfile for network bytestring? |
| 21:21:23 | <tibbe> | mae_work: patches welcome ;) |
| 21:21:36 | <tibbe> | mae_work: otherwise as soon as I have time |
| 21:21:47 | <tibbe> | mae_work: I really want to get a first version of hyena out there |
| 21:21:48 | <mae_work> | tibbe: if you implement sendfile, I will work on the windows portion of readv/sendv and also sendfile |
| 21:21:57 | <mae_work> | tibbe: 10-4 |
| 21:22:09 | <tibbe> | mae_work: I can't promise you a date but I'll try to get to it |
| 21:22:24 | <tibbe> | mae_work: if you "follow" me on github you'll see when it's done |
| 21:22:32 | <mae_work> | heh |
| 21:22:33 | <mae_work> | ok |
| 21:23:28 | <mae_work> | gcollins: ok so stepping up a few semantic levels higher, can you organize your proposals for overhauling the http stack into a wiki page? |
| 21:24:03 | <gcollins> | sure. but not for a while, my wedding & honeymoon are coming up soon |
| 21:24:18 | <mae_work> | gcollins: ok |
| 21:24:20 | <mae_work> | congratulations :) |
| 21:24:25 | <gcollins> | i actually will probably have some time to code in budapest |
| 21:24:37 | <mae_work> | right |
| 21:24:53 | <gcollins> | thanks! you have a new child coming soon too, right -- or has he/she arrived yet? |
| 21:25:05 | <mae_work> | very soon |
| 21:25:11 | <mae_work> | in a week or so |
| 21:25:19 | <mae_work> | trying to release 0.3.1 before that :) |
| 21:25:42 | <mae_work> | apparently alex jacobson just added cabal support to sp, somewhat negating the need for my "auto build" tool |
| 21:25:44 | <mae_work> | but oh well |
| 21:26:43 | <mae_work> | I really see happstack evolving with the http stuff for awhile |
| 21:26:58 | <mae_work> | until it feels tight and nice I don't think I wanna delve into the state stuff too deeply |
| 21:27:17 | <abuiles> | sorry to ask, but what is sp ? |
| 21:28:18 | <mae_work> | searchpath |
| 21:29:01 | <abuiles> | than |
| 21:29:04 | <abuiles> | *ks |
| 21:31:00 | <mae_work> | gcollins: ok well, if you do have time, I would love to see a wiki page, lets try to be as concrete as possible though, so we can move from idea to patches without too much fuss |
| 21:31:10 | <mae_work> | I want to clean house |
| 21:31:31 | <gcollins> | i like the state stuff in theory but databases are really tricky and it'd take a lot to get me to trust it with anything important |
| 21:31:54 | <gcollins> | ok -- i'll spend some time thinking about it for sure |
| 21:32:06 | <mae_work> | i wonder if creighton ever finished porting all that crypto stuff so we can move that dep outwards |
| 21:33:09 | <mae_work> | gcollins: yeah thats the problem, it will take a lot of work to become "production ready", but a native haskell database could definitely cut the time of development down if done right, it is still in the "experimental" stage |
| 21:33:24 | <mae_work> | hell, we are using it for patch-tag though, hasn't had any problems yet |
| 21:34:02 | <gcollins> | do you think it'll scale to millions of transactions daily? |
| 21:34:34 | <gcollins> | i don't trust ORMs either for what it's worth -- i've seen too much rotten SQL come out of things like hibernate et al |
| 21:36:20 | <mae_work> | same here |
| 21:36:24 | <mae_work> | i used rails for awhile |
| 21:36:34 | <mae_work> | if i went back to sql dev, i'd probably handcode the routines |
| 21:36:40 | <gcollins> | oh man --- don't get me started on that stuff |
| 21:36:50 | <mae_work> | heh |
| 21:37:19 | <mae_work> | the whole idea of RAD (rapid application development) is a pipe dream, because you always have to end up coming full circle anyways, and realizing the tool has given you dogshit |
| 21:38:29 | <gcollins> | yeah -- the people involved in that stuff want to make out like they've invented this mindblowing new thing that's soooo revolutionary |
| 21:39:07 | <gcollins> | meanwhile it's 200x slower than C |
| 21:39:24 | <mae_work> | the compelling thing for me about MACID |
| 21:39:30 | <mae_work> | is that you cut out the query bottleneck |
| 21:39:42 | <mae_work> | so for large, flexible, accessible, volumes of data, sql is very good |
| 21:39:56 | <gcollins> | and those of us who have been around computers for more than a couple of years are like "we've read this script before" |
| 21:40:02 | <mae_work> | but for serving up queries with raw speed, you can't beat in-memory |
| 21:40:25 | <gcollins> | that's definitely true -- your dataset has to fit in memory though |
| 21:40:28 | <mae_work> | hehe |
| 21:40:32 | <mae_work> | memory is cheap now |
| 21:40:34 | <mae_work> | : ) |
| 21:40:43 | <mae_work> | + most of your data is thin |
| 21:40:47 | <mae_work> | and binary data belongs on a filesystem |
| 21:40:49 | <mae_work> | or s3 or something |
| 21:40:52 | <mae_work> | (if its lots of it) |
| 21:41:04 | <mae_work> | same argument applies there too |
| 21:41:10 | <mae_work> | "all your data has to fit on the filesystem" |
| 21:41:18 | <gcollins> | that'd work for the majority of apps but there are datasets it'd break for in a couple of days |
| 21:41:18 | <mae_work> | you just need to fix your perception :) |
| 21:41:53 | <tibbe> | ACTION is of the believe that relational databases don't scale. |
| 21:42:07 | <gcollins> | tibbe: i agree |
| 21:42:21 | <tibbe> | but then I work on YouTube |
| 21:42:21 | <gcollins> | i'm actually working on something pretty neat in this area |
| 21:42:37 | <tibbe> | I want something like Amazon's Dynamo written in Haskell |
| 21:43:10 | <gcollins> | tibbe: i'm working on something similar to that (not so fancy, just key-value-pair) backed with tokyo tyrant |
| 21:43:10 | <tibbe> | Dynamo is essentially a persistent memcache with some extra bells and whistles |
| 21:43:20 | <tibbe> | gcollins: sounds interesting |
| 21:43:24 | <abuiles> | ACTION feel like a kid listening to adults conversations. |
| 21:44:08 | <gcollins> | it'll be very similar to "lightcloud", just a thin consistent-hashing frontend |
| 21:45:00 | <gcollins> | the consistent-hashing stuff was super-simple, i've finished it already -- the "real work" begins with the server part of it |
| 21:46:50 | <gcollins> | ideally it should be transparent to the user that you're talking to a distributed key-value store, i'll probably implement the tokyo-tyrant protocol (with an authenticating frontend) |
| 21:47:00 | <tibbe> | ok |
| 21:47:09 | <gcollins> | the tricky thing is writing good tools to add/remove nodes from the cluster |
| 21:47:11 | <tibbe> | it needs to be reliable and fast |
| 21:47:19 | <gcollins> | agreed |
| 21:47:29 | <tibbe> | there are some nice OS implementations out there |
| 21:47:43 | <gcollins> | tokyo tyrant is pretty much the fastest key-value store i've found |
| 21:47:57 | <gcollins> | i'm a really big fan |
| 21:48:06 | <tibbe> | gcollins: http://project-voldemort.com/ |
| 21:48:12 | <tibbe> | gcollins: I'll check it out tomorrow |
| 21:48:20 | <gcollins> | yep, i've looked into that one |
| 21:48:37 | <gcollins> | the java dependency makes me turn up my nose a bit |
| 21:48:44 | <gcollins> | but the design looks solid to me |
| 21:49:24 | <tibbe> | I meant the design more than the implementation |
| 21:49:29 | <tibbe> | something to use as inspiration |
| 21:49:30 | <tibbe> | bed time |
| 21:49:31 | <tibbe> | gnight |
| 21:50:12 | <gcollins> | night |
| 21:51:00 | <abuiles> | night |
| 22:02:22 | <abuiles> | gcollins: how do you run your site, with cgi or something like that ? |
| 22:02:37 | <gcollins> | reverse-proxied through lighttpd |
| 22:05:53 | <mae_work> | yeah |
| 22:05:57 | <mae_work> | thats my weapon of choice |
| 22:06:04 | <mae_work> | (if i'm doing it end-to-end) |
| 22:06:14 | <mae_work> | because of the nice proxy balancer |
| 22:06:30 | <mae_work> | but I think AWS now has an ip-level load balancer service |
| 22:06:32 | <mae_work> | which is nice |
| 22:07:01 | <gcollins> | we use amazon at work, i should bring that up -- we've been using haproxy on our frontend |
| 22:07:24 | <gcollins> | all of that stuff is apache/mod_wsgi/python |
| 22:20:14 | <mae_work> | what is |
| 22:20:16 | <mae_work> | the amazon stuff? |
| 22:23:52 | <gcollins> | no, our stuff |
| 22:24:04 | <mae_work> | ah |
| 22:24:08 | <gcollins> | although we're rolling out our first haskell server soon |
| 22:24:18 | <mae_work> | neat. |
| 22:24:37 | <mae_work> | where do you work? |
| 22:24:44 | <gcollins> | given that we'll be using this in production, i may be able to get some 9-to-5 time into happstack in the next few months |
| 22:24:49 | <gcollins> | www.polarmobile.com |
| 22:25:07 | <gcollins> | we do content delivery to smartphones |
| 22:26:13 | <mae_work> | gcollins: re 9-5 time in happstack, great! |
| 22:26:25 | <mae_work> | gcollins: how do you feel about moving towards hyena as the core? |
| 22:27:14 | <abuiles> | mae_work: indeed, is ghc-10.0.2 problem.. |
| 22:27:19 | <gcollins> | i'd want to see benchmarks |
| 22:27:32 | <mae_work> | i mean really there are two main directions for Happstack.HTTP, one is to clean house, simplify the monad stack, have nice error handling etc.. |
| 22:27:44 | <mae_work> | two is to move to a http core that doesn't use lazy io |
| 22:27:51 | <gcollins> | right |
| 22:27:53 | <mae_work> | would one or two be the 9-5 project |
| 22:27:58 | <gcollins> | and those are orthogonal to each other |
| 22:28:13 | <gcollins> | i'd probably want to improve the http/network stuff first |
| 22:28:14 | <abuiles> | mae_work: with ghc-10,0,1 works perfect . |
| 22:28:22 | <mae_work> | um |
| 22:28:29 | <mae_work> | meaning the core, right? |
| 22:28:37 | <mae_work> | abuiles: ok, i'll have to take that into account |
| 22:28:43 | <mae_work> | abuiles: thanks for testing that out |
| 22:29:00 | <mae_work> | probably there is a slightly different behavior that I have to cater to |
| 22:29:16 | <mae_work> | i.e. i might need to explicitly handle a sigterm, for instance |
| 22:29:21 | <mae_work> | (want to avoid that though) |
| 22:29:26 | <gcollins> | right, the part that renders a Response out the socket |
| 22:29:44 | <mae_work> | gcollins: ok |
| 22:30:08 | <abuiles> | mae_wokr : not problem man .... Hope I can continue helping out the project ( although I not have the same experience as you guys,,, just a college kid :) ... but very enthusiastic :) ) |
| 22:30:28 | <gcollins> | re: simplifying/reworking ServerPart -- that's an aesthetic vs. a correctness issue |
| 22:30:41 | <mae_work> | gcollins: i mean, i think hyena in the long run would be beneficial in the sense that there is less maintenance overhead for the project |
| 22:31:12 | <gcollins> | I can see it having multiple backends actually |
| 22:31:32 | <mae_work> | a more conservative approach would bring better performance and/or behavior (with things like pipe-lining and large files) -- carefully replace a few key items in the http loop |
| 22:32:00 | <gcollins> | that's a no-brainer to start at least |
| 22:32:07 | <gcollins> | that http inner loop is not a ton of code |
| 22:32:40 | <mae_work> | right |
| 22:32:48 | <mae_work> | so i was thinking multiple backend was a good idea |
| 22:32:54 | <mae_work> | because it gives more time for people to test |
| 22:32:57 | <mae_work> | before we switch the "default" |
| 22:33:02 | <mae_work> | and we can bench each one independently |
| 22:33:04 | <gcollins> | well, it's necessary anyways --- you need to race them :) |
| 22:33:08 | <mae_work> | yep |
| 22:33:28 | <mae_work> | so step 1, abstract the core from the rest of the bits, make some sort of shim |
| 22:33:40 | <mae_work> | step 2, make sure nothing broke with the "legacy" core |
| 22:33:42 | <abuiles> | ok guys,, good night,, will be back to irc by thursday.. this time from sweet home.... |
| 22:33:48 | <mae_work> | step 3, implement new cores |
| 22:33:55 | <gcollins> | well, from simpleHTTP' to runServerPartT is not a ton of code |
| 22:34:02 | <gcollins> | abuiles: bye |
| 22:34:08 | <mae_work> | true |
| 22:34:29 | <gcollins> | i think we can just delineate at that boundary |
| 22:34:49 | <gcollins> | i imagine the next version will run other things besides ServerPartTs |
| 22:35:01 | <mae_work> | yeah, i forget sometimes I am working an a language that facilitates this types of things much easier |
| 22:35:13 | <gcollins> | haskell FTW |
| 22:35:22 | <mae_work> | gcollins: what do you mean "other things" |
| 22:35:39 | <gcollins> | well, you need a mode that gives you access to the raw socket |
| 22:35:47 | <mae_work> | ServerPartT was from when CPS was the one true way (tm) |
| 22:35:55 | <mae_work> | but now we generalized the transformers and can use a do-block style |
| 22:36:01 | <gcollins> | i wrote a mailing list post about it a while back |
| 22:36:07 | <mae_work> | i am still not sure that CPS is easy for most people to understand |
| 22:36:20 | <gcollins> | http://groups.google.com/group/HAppS/tree/browse_frm/thread/1913f4fea4fb9c65/3a0924db62f7fc80?rnum=1&_done=%2Fgroup%2FHAppS%2Fbrowse_frm%2Fthread%2F1913f4fea4fb9c65%2F3a0924db62f7fc80%3Ftvc%3D1%26q%3Dgregory%2Bcollins%26#doc_969da8b5abb40545 |
| 22:36:23 | <mae_work> | gcollins: is that really necessary? |
| 22:36:43 | <gcollins> | if you wanna be able to use sendfile |
| 22:36:45 | <mae_work> | couldn't we simply pass a fun and have that handed a socket at a lower level |
| 22:36:57 | <mae_work> | and likewise for sendfile, pass a file path down |
| 22:38:00 | <gcollins> | you mean expose "SendFile f" as a datatype constructor on Response? |
| 22:38:05 | <gcollins> | and then have the backend deal with it? |
| 22:38:06 | <mae_work> | something like that |
| 22:38:07 | <mae_work> | yeah |
| 22:38:14 | <gcollins> | that's another idea i hadn't considered |
| 22:38:21 | <gcollins> | that might actually be better |
| 22:38:24 | <mae_work> | i like that better than exposing the raw socket |
| 22:38:28 | <mae_work> | at the app level |
| 22:39:15 | <mae_work> | it also allows us to not have to "fake" using the filter etc machinery |
| 22:39:15 | <gcollins> | hm |
| 22:39:19 | <mae_work> | when we are going to ignore it anyways |
| 22:39:50 | <gcollins> | Would the backend be a typeclass then? |
| 22:39:55 | <mae_work> | perhaps |
| 22:40:08 | <gcollins> | either that or a wrapping function |
| 22:40:14 | <mae_work> | let me take a look here |
| 22:41:52 | <gcollins> | actually yeah, I like the idea of adding a constructor to Response a lot |
| 22:42:12 | <mae_work> | yep |
| 22:43:31 | <gcollins> | we can just fix fileServe then |
| 22:44:45 | <mae_work> | take a look at this |
| 22:44:46 | <mae_work> | http://haskell.pastebin.com/m469ed012 |
| 22:45:08 | <mae_work> | only diff is |
| 22:45:12 | <mae_work> | instead of rsBody |
| 22:45:14 | <mae_work> | rsFilePath |
| 22:45:39 | <gcollins> | i like it -- except won't that be a name clash? |
| 22:45:45 | <mae_work> | ? |
| 22:45:51 | <mae_work> | with what |
| 22:46:08 | <mae_work> | oh i see |
| 22:46:10 | <mae_work> | um |
| 22:46:10 | <mae_work> | shouldn't be |
| 22:46:12 | <mae_work> | since they are the same type |
| 22:46:23 | <mae_work> | good q |
| 22:46:54 | <mae_work> | see if you can run it through ghci :) |
| 22:47:09 | <gcollins> | yep |
| 22:47:25 | <gcollins> | i still don't know all of the rules by heart :) |
| 22:49:56 | <mae_work> | gcollins: nope it works fine |
| 22:50:00 | <mae_work> | so i guess the only issue here is that |
| 22:50:09 | <mae_work> | someone could feasibly set the header ContentLength |
| 22:50:25 | <mae_work> | in this instance, we must ignore it |
| 22:50:42 | <mae_work> | or rather |
| 22:50:43 | <mae_work> | overwrite it |
| 22:50:45 | <gcollins> | the backend that handles SendFile will set its own content-length based on fstat() |
| 22:50:50 | <gcollins> | right |
| 22:51:04 | <mae_work> | hmm |
| 22:51:22 | <mae_work> | so we have to open the file handle |
| 22:51:27 | <mae_work> | to check the size? |
| 22:51:29 | <mae_work> | low overhead? |
| 22:51:33 | <gcollins> | no |
| 22:51:41 | <gcollins> | very low overhead |
| 22:51:55 | <mae_work> | http://www.opengroup.org/onlinepubs/009695399/functions/fstat.html |
| 22:52:04 | <mae_work> | k |
| 22:52:08 | <mae_work> | so given this idea |
| 22:52:14 | <mae_work> | this should actually super simple to implement |
| 22:52:15 | <gcollins> | or stat() |
| 22:52:17 | <gcollins> | rather |
| 22:52:23 | <mae_work> | (ignoring the integration into fileServe for now) |
| 22:52:23 | <gcollins> | right |
| 22:52:33 | <mae_work> | and aside from the C pushups |
| 22:52:37 | <mae_work> | and if we move to something like hyena later |
| 22:52:39 | <mae_work> | which has sendfile support |
| 22:52:42 | <mae_work> | we make the translation |
| 22:52:58 | <gcollins> | well for round one you can just send the file out the socket by hand |
| 22:52:59 | <mae_work> | but the interface can stay the same |
| 22:53:08 | <mae_work> | right |
| 22:53:14 | <gcollins> | optimize it later |
| 22:53:31 | <mae_work> | oh i see |
| 22:53:32 | <gcollins> | like after it gets integrated into network-bytestring |
| 22:53:45 | <mae_work> | well |
| 22:53:54 | <mae_work> | right ok |
| 22:54:12 | <gcollins> | i mean, that'll be the "canonical" way of doing that within a month or two |
| 22:54:16 | <mae_work> | i don't see why sendfile semantically even belongs in network-bytestring |
| 22:54:22 | <mae_work> | as you will never ever create a bytestring |
| 22:54:33 | <gcollins> | and it'll be a 10-line patch to wire it in |
| 22:54:41 | <gcollins> | that's a good point :) |
| 22:54:53 | <gcollins> | it should probably go into network proper |
| 22:55:01 | <mae_work> | before i talked to tibbe today I was simply gonna create a portable sendfile cabal package |
| 22:55:03 | <mae_work> | and throw it on hackage |
| 22:55:19 | <mae_work> | network seems like the right place |
| 22:55:24 | <mae_work> | but acceptance is a pain in the ass |
| 22:55:35 | <gcollins> | i'd do that |
| 22:55:40 | <mae_work> | besides, the haskell platform will start using hackage libs anyways |
| 22:55:42 | <gcollins> | actually |
| 22:55:49 | <mae_work> | which? |
| 22:55:59 | <gcollins> | release a hackage lib |
| 22:56:03 | <mae_work> | yeah |
| 22:56:07 | <mae_work> | it can always move |
| 22:56:28 | <gcollins> | because you'll get bug reports, it'll be easier to convince people to put it into network later |
| 22:56:38 | <mae_work> | yeppers |
| 22:56:59 | <gcollins> | it's cross-platform? |
| 22:57:05 | <mae_work> | yeah |
| 22:57:08 | <mae_work> | windows and linux |
| 22:57:12 | <mae_work> | (windows 2000 or later) |
| 22:57:18 | <gcollins> | osx? |
| 22:57:20 | <mae_work> | only in windows its called TransmitFile |
| 22:57:24 | <mae_work> | yeah osx should have it |
| 22:57:39 | <mae_work> | read here |
| 22:57:40 | <mae_work> | no clue |
| 22:57:40 | <mae_work> | http://sendfile.darwinports.com/ |
| 22:57:44 | <gcollins> | int |
| 22:57:44 | <gcollins> | sendfile(int fd, int s, off_t offset, off_t *len, struct sf_hdtr *hdtr, |
| 22:57:44 | <gcollins> | int flags); |
| 22:57:56 | <gcollins> | is the interface the same? |
| 22:58:05 | <mae_work> | for what |
| 22:58:07 | <mae_work> | TransmitFile? |
| 22:58:26 | <mae_work> | not sure |
| 22:58:31 | <mae_work> | but i can make it match |
| 22:58:39 | <mae_work> | with a c shim |
| 22:58:41 | <mae_work> | http://msdn.microsoft.com/en-us/library/ms740565.aspx |
| 22:58:53 | <gcollins> | no i mean that's the osx sendfile call |
| 22:59:29 | <mae_work> | i think it is probably the same as linux |
| 22:59:36 | <mae_work> | since it seems to be part of standard c lib |
| 23:00:59 | <gcollins> | ssize_t sendfile(int out_fd, int in_fd, off_t *offset, size_t count); |
| 23:01:09 | <gcollins> | it's different |
| 23:01:41 | <mae_work> | that seems to be the std one |
| 23:01:54 | <gcollins> | the osx one is different |
| 23:01:56 | <mae_work> | dammit man |
| 23:02:06 | <mae_work> | someone has to have created a portable sendfile shim already |
| 23:02:24 | <gcollins> | it's not a ton of work actually |
| 23:02:27 | <mae_work> | i don't even begin to know the keywords that would unhide such a diamond in the rough |
| 23:02:42 | <mae_work> | gcollins: i know, but then you have to worry about cross platform testing etc |
| 23:02:48 | <mae_work> | its not about difficulty |
| 23:03:02 | <mae_work> | its about someone whose already went through this pain and tested it for me |
| 23:03:03 | <mae_work> | : ) |
| 23:04:02 | <gcollins> | yeah |
| 23:04:07 | <gcollins> | i don't see anything |
| 23:05:11 | <mae_work> | auto tools has something i bet :) |
| 23:05:15 | <mae_work> | but i don't wanna mess with that |
| 23:06:01 | <mae_work> | so ok |
| 23:06:04 | <gcollins> | sweet jesus no |
| 23:06:23 | <mae_work> | i can't assume that the *nixes have a same impl |
| 23:06:28 | <gcollins> | right |
| 23:06:30 | <mae_work> | how do i even begin to target them |
| 23:06:33 | <mae_work> | arg |
| 23:06:39 | <gcollins> | i would pick linux / osx |
| 23:06:46 | <gcollins> | or wait for contributors |
| 23:06:53 | <mae_work> | i will pick linux :) |
| 23:06:56 | <gcollins> | i'd write an osx version, no problem |
| 23:07:03 | <mae_work> | yeah |
| 23:07:11 | <gcollins> | that's 15 minutes work |
| 23:07:14 | <mae_work> | just need to figure out the build flags to target |
| 23:07:15 | <mae_work> | right |
| 23:07:26 | <mae_work> | ok so i have your signature in blood |
| 23:07:33 | <mae_work> | you will work on osx should i create such a library |
| 23:07:39 | <mae_work> | i will work on windows and linux |
| 23:07:46 | <mae_work> | : ) |
| 23:07:55 | <gcollins> | i'd also provide a "default" implementation just in case the others fail |
| 23:08:05 | <gcollins> | ok |
| 23:08:06 | <mae_work> | ie send by hand |
| 23:08:08 | <mae_work> | well |
| 23:08:11 | <gcollins> | right |
| 23:08:13 | <gcollins> | send by hand |
| 23:08:41 | <mae_work> | or expect that the person should know what they are doing |
| 23:08:54 | <mae_work> | and just throw an exception on a non-supported platform |
| 23:08:55 | <mae_work> | : ) |
| 23:09:00 | <mae_work> | we can create a higher level fun |
| 23:09:01 | <mae_work> | like |
| 23:09:14 | <gcollins> | speaking as a consumer of this product i really would prefer a unified, no-hassle solution |
| 23:09:33 | <mae_work> | well |
| 23:09:34 | <gcollins> | i.e. sendFile :: FilePath -> Handle -> IO () |
| 23:09:38 | <mae_work> | right |
| 23:09:41 | <mae_work> | what i mean is |
| 23:09:49 | <mae_work> | no one should be constructing SendFile |
| 23:09:54 | <mae_work> | unless they really know what they are doing |
| 23:10:21 | <mae_work> | the type would probably be |
| 23:10:39 | <mae_work> | sendFile :: FilePath -> IO Response |
| 23:10:47 | <mae_work> | (Response being the Datatype proper, not the instance) |
| 23:11:01 | <mae_work> | or |
| 23:11:02 | <gcollins> | for happstack, yes |
| 23:11:23 | <mae_work> | sendFile :: FilePath -> MimeType -> IO Response |
| 23:11:32 | <gcollins> | i meant for a "Network.SendFile" package on hackage |
| 23:11:39 | <mae_work> | gcollins: ahh ok so your saying sendfile itself should have a fallback if the platform is unsupported |
| 23:11:45 | <gcollins> | right |
| 23:12:09 | <gcollins> | so you can still compile and use it on hp-ux (or whatever) |
| 23:12:15 | <gcollins> | VMS |
| 23:12:44 | <gcollins> | PDP-11 |
| 23:13:32 | <mae_work> | oh good god |
| 23:13:33 | <mae_work> | pdp-11 |
| 23:13:39 | <mae_work> | you must have read the history of hackerdom |
| 23:13:40 | <mae_work> | : ) |
| 23:13:45 | <mae_work> | or just be a real big geek |
| 23:13:51 | <gcollins> | ehm |
| 23:13:59 | <mae_work> | = ) |
| 23:14:00 | <gcollins> | my fiance and i are watching star trek right now |
| 23:14:05 | <mae_work> | ahh |
| 23:14:08 | <mae_work> | that was a good flick |
| 23:14:14 | <mae_work> | wait your in the theatre? |
| 23:14:18 | <gcollins> | no, i mean next generation back episodes |
| 23:14:21 | <mae_work> | oh oh |
| 23:14:33 | <gcollins> | giant dork, yes |
| 23:14:52 | <gcollins> | does solaris have sendfile() yet? |
| 23:14:59 | <gcollins> | that's the only other relevant platform |
| 23:15:01 | <mae_work> | who knows |
| 23:15:11 | <mae_work> | who knew that sendfile would be such a hit |
| 23:15:16 | <mae_work> | in the end its all about throughput of flat files |
| 23:15:24 | <gcollins> | free/open bsd |
| 23:15:24 | <mae_work> | because we can't stick it all in memory |
| 23:15:37 | <mae_work> | and *nix filesystem is still the best damn way to handle it |
| 23:15:46 | <gcollins> | zero copy from kernel/userland is a really good idea |
| 23:16:05 | <gcollins> | that i think is the main motivation |
| 23:16:41 | <gcollins> | straight from the buffer cache out to the network interface using DMA |
| 23:17:08 | <gcollins> | vs. two kernel-userland memory copies and a couple of context switches |
| 23:17:38 | <mae_work> | ok so |
| 23:17:41 | <mae_work> | once upon a time |
| 23:17:45 | <mae_work> | i was watching lighttpd |
| 23:18:35 | <mae_work> | is this epoll / linux-specific / asynchronous fd stuff really that great |
| 23:18:58 | <mae_work> | or is it just rice for http server designers |
| 23:19:07 | <gcollins> | no, it's that good |
| 23:19:16 | <mae_work> | is it portable? at all? |
| 23:19:57 | <gcollins> | not really. OSX and Windows have their own proprietary interfaces to this kind of I/O |
| 23:19:58 | <mae_work> | graceful degradation.. |
| 23:20:11 | <mae_work> | gcollins: but windows and osx have their own version? |
| 23:20:14 | <gcollins> | there's something called "libevent" which abstracts that stuff |
| 23:20:24 | <mae_work> | ah |
| 23:20:38 | <gcollins> | osx has something called "kqueue" which i think is similar |
| 23:20:57 | <mae_work> | ok so |
| 23:21:08 | <mae_work> | what do you think of using libevent |
| 23:21:13 | <mae_work> | worthit or not so worthit |
| 23:21:23 | <mae_work> | probably needs auto tools.. |
| 23:21:44 | <gcollins> | i think it might be worth it |
| 23:21:49 | <gcollins> | actually |
| 23:22:17 | <gcollins> | there's this: |
| 23:22:17 | <gcollins> | http://hackage.haskell.org/cgi-bin/hackage-scripts/package/hlibev |
| 23:22:21 | <mae_work> | yeah, i just hate the idea of not being able to use "cabal install" |
| 23:22:52 | <gcollins> | based on this: |
| 23:22:53 | <gcollins> | http://libev.schmorp.de/bench.html |
| 23:22:55 | <mae_work> | yeah but libev will require manual compilation etc |
| 23:23:11 | <gcollins> | maybe it can be an optional backend |
| 23:24:54 | <mae_work> | yeah |
| 23:25:06 | <mae_work> | but again, just depends how hard you wanna press the scalability pedal |
| 23:25:07 | <gcollins> | in theory that would be pretty fast |
| 23:25:41 | <mae_work> | what is beautiful about this though is, the power of haskell in the app, with the raw scalability potential by having a very thin layer between the app and the kernel |
| 23:25:51 | <gcollins> | yep |
| 23:26:04 | <mae_work> | and since haskell is known to be pretty damn fast with hashmaps etc in memory |
| 23:26:09 | <mae_work> | combining these two would be awesome |
| 23:26:36 | <mae_work> | usually you just get either a really optimized "plain" http server with which you have to hook up cgi (killing your perf) or an app server which is portable with no rice |
| 23:26:37 | <mae_work> | : ) |
| 23:26:44 | <gcollins> | i mean, we've been doing web in languages like php and python and tcl for ages -- getting within 2-3x of C would be fine for 99% of applications |
| 23:27:12 | <gcollins> | here's my opinion on that |
| 23:27:25 | <gcollins> | it's not a "performance" issue -- it's really a matter of cash money |
| 23:27:55 | <gcollins> | if server X can handle double the load of server Y then that's $60/mo I don't have to spend on application servers |
| 23:28:26 | <mae_work> | yep |
| 23:28:43 | <mae_work> | if your problem is scalability this is true |
| 23:28:43 | <gcollins> | of course satisfying performance guarantees is also really important for other reasons |
| 23:28:54 | <gcollins> | that's why facebook killed myspace |
| 23:29:10 | <mae_work> | whats facebook running on |
| 23:29:45 | <gcollins> | php i think? |
| 23:29:49 | <gcollins> | lighthttpd maybe? |
| 23:29:52 | <mae_work> | heh |
| 23:29:59 | <gcollins> | god i can never type that |
| 23:30:14 | <gcollins> | youtube runs lighttpd, i think |
| 23:30:21 | <mae_work> | i got past that crutch by mouthing it in my head "light-tpd" |
| 23:30:28 | <mae_work> | it does |
| 23:30:31 | <mae_work> | for video streaming |
| 23:30:38 | <mae_work> | bazillions and bazillions of movies |
| 23:30:51 | <gcollins> | they definitely have a scalability problem |
| 23:31:02 | <gcollins> | which they've solved really really well actually |
| 23:31:07 | <gcollins> | that's so highly impressive |
| 23:31:11 | <mae_work> | yeah |
| 23:31:16 | <mae_work> | then again, they were a startup |
| 23:31:20 | <mae_work> | and bought later |
| 23:31:45 | <mae_work> | not a child of herman-miller cush blood-sucking startup cash. |
| 23:31:52 | <gcollins> | :) |
| 23:31:59 | <mae_work> | i am not sure if that even made any sense |
| 23:32:01 | <mae_work> | well they were a startup |
| 23:32:08 | <mae_work> | they just weren't a megacorp with lots of money to spend on BS |
| 23:32:24 | <gcollins> | efficiency is important for sure |
| 23:32:36 | <gcollins> | even for medium-traffic sites |
| 23:32:43 | <mae_work> | my company tries to throw hardware at everything |
| 23:32:52 | <gcollins> | that never works |
| 23:32:57 | <gcollins> | well |
| 23:33:02 | <mae_work> | so true |
| 23:33:04 | <mae_work> | you know |
| 23:33:05 | <gcollins> | it temporarily works sometimes |
| 23:33:11 | <mae_work> | we have like 500 consumers of an ERP product |
| 23:33:17 | <mae_work> | we have 2 netscalers man |
| 23:33:24 | <mae_work> | we have several warehouses |
| 23:33:45 | <mae_work> | we have a 3-node oracle cluster with each node containing quad-dual cores and 32gb ram |
| 23:33:51 | <mae_work> | mind you this was spec'd out about 3 years ago |
| 23:33:58 | <mae_work> | and purchased 3 years ago |
| 23:34:01 | <gcollins> | yep -- megabucks |
| 23:34:12 | <mae_work> | the database is literally only 32gb |
| 23:34:17 | <mae_work> | it could fit in the ram of one of them! |
| 23:34:32 | <gcollins> | Load average: 0.04, 0.08, 0.03 |
| 23:34:37 | <mae_work> | yeah heh |
| 23:34:45 | <mae_work> | the netscalers are active/passive |
| 23:34:53 | <mae_work> | literally at peak times it is like 1.3 percent cpu |
| 23:34:58 | <mae_work> | the rest of the time maybe it cracks 0.5 |
| 23:35:13 | <gcollins> | man -- i've always worked at startups |
| 23:35:28 | <mae_work> | but all their performance problems stem from fundamental issues that can't be fixed by throwing hardware at it |
| 23:35:34 | <mae_work> | ridiculous queries |
| 23:35:37 | <gcollins> | exactly |
| 23:36:01 | <gcollins> | i'm a firm believer in the judicious application of computer science |
| 23:36:19 | <mae_work> | right now i'm dealing with some SAP a-holes for business intelligence |
| 23:36:25 | <mae_work> | hehe |
| 23:36:54 | <gcollins> | that's an oxymoron if i've ever heard one |
| 23:37:02 | <mae_work> | : ) |
| 23:44:09 | <mae_work> | so yeah err, i gotta roll but uh the sendfile is one of my definite goals for 0.4 |
| 23:44:19 | <mae_work> | so i will get that lib up there |
| 23:44:29 | <mae_work> | and uh i think 0.3.1 is pretty much ready to roll |
| 23:44:32 | <mae_work> | no reason to hold it back |
| 23:45:01 | <gcollins> | it's got a lot of fixes vs 0.2 |
| 23:45:04 | <mae_work> | just need to make sure that 1. haddocks all build and 2. they pass the hackage checker script |
| 23:45:16 | <mae_work> | and other various administrative bits |
| 23:45:45 | <mae_work> | i'm almost thinking for how long i left it at 0.3.1 |
| 23:45:48 | <mae_work> | that i should bump it again to 0.3.2 |
| 23:45:57 | <mae_work> | so people don't end up saying "i installed 0.3.1 and its not working" |
| 23:46:06 | <mae_work> | (that built it from the repo) |
Back to channel and daily index: content-negotiated html turtle