00:00:36 Hmm. I'm not actually measuring some things correctly. 00:00:52 That seems somewhat dependan on the math lib 00:03:20 long time: I could time it, but I'm trying to time Eep at the mo'. It's being a bit barfy 00:05:19 Heh. 00:05:50 I'd really ahve to compile that sucker by hand. I'm not sure I *want* to :) 00:06:44 heh, heh 00:17:57 * AaronSw waves 00:18:02 hey aaron. 00:18:11 Hey bijan 00:19:03 Hm, lots of new people 00:20:09 I'm in the middle of "The Star Fraction" now, so I'll catch up on stuff later. 00:20:13 oh aaron. 00:20:19 yeah? 00:20:24 Any long running n3 things not involving builtins? 00:21:13 * bijan still looking for tests. 00:21:24 None come to mind. 00:21:55 Oh well. 00:22:51 Wow, Lessig wrote me a personal thank you note! That's awesome. 00:22:59 For? 00:23:24 I guess it's sort of a secret. 00:23:39 Some RDF stuff I did for him... 00:24:31 Ooh, neat: http://apple.slashdot.org/ 00:27:23 * sbp returns 00:30:02 1.52 0.11 for that test? Hmm... something must have borked 00:31:23 1.52? 00:31:49 seconds to parse the NTriples files (800KB), seconds to perform the query 00:32:16 Which ntriples file. the geo thing? 00:32:35 yeah. I converted it to NTriples since I didn't want to crash test my N3 parser... :-) 00:32:40 Heheheh. 00:32:45 Got ya beat *there*. 00:34:42 wow! it wasn't kidding! 00:34:43 1.42999994755 00:34:43 0.160000085831 00:34:53 This is eep? 00:34:56 yep 00:35:00 Did you just smoke both me and cwm? 00:35:01 You pig. 00:35:05 bwahahahaha! 00:35:15 I think the problem with rm's test is IDLE 00:35:26 if IDLE has to print something really huge, it just stands there 00:35:37 OTOH, I might be wrong - it might be a problem with the inference module 00:35:43 but the query module is... well, rather fast 00:36:07 * sbp goes to get a drink 00:38:24 Hmm... and the output file is 0.99MB 00:38:32 Python is an awesome thing 00:38:57 Hmm. 00:39:03 Something seems up with that. 00:39:27 I can post you the data, if you like 00:39:34 sure. 00:40:11 What is your query engine doing, btw? 00:40:26 I guess if it's just a loop over the list, that could be nippy. 00:42:19 it's querying for the triples, plus matching the variables 00:42:26 [sent, BWT] 00:42:32 or BTW... 00:42:37 "querying...*plus* mathcing?" 00:42:45 yeah 00:42:57 ?x :y ?z is not the same as ?x :y ?x 00:44:24 ouch, y'know, *that's* something that bites me. 00:44:27 I think. 00:44:35 what's that? 00:44:50 oh, does it not match the variables? 00:45:04 er..hmm. 00:45:46 test case: store: :p :q :p . :p :q :r . query: { ?x :q ?x } log:implies { ?x :q ?x } . 00:45:47 in {?x :parent ?z. ?y :parent ?z} => {?x :co-parent ?y} and 00:46:06 :bob :parent :sally. :mary :parent :sally. 00:46:13 I will get that bob and mary are their own co-parents 00:47:03 Ugh 00:47:04 Oh, this isnt' the inference? This is the query? 00:47:17 yep, just the query 00:47:38 D'oh! 00:47:43 I thought this was the *inference*. 00:47:46 heh, heh 00:47:53 I'll benchmark the inference... 00:48:05 AlexMax has quit ("ButchX-p6 by conio Accept no limitations") 00:50:02 *Much* less than a second. 00:50:08 Doing query test... 00:50:08 0.160000085831 00:50:08 Doing inference test... 00:50:08 0.27999997139 00:50:12 Hmm. 00:50:33 How many hits do you get? 00:50:41 still pretty quick. that's using infer.filter, which is the main inference function 00:50:42 hits? 00:50:47 1211 00:50:54 I get 1211 00:50:58 From your query. 00:50:59 7266 1211 1211 00:51:18 7266 is the length of the input, in triples 00:51:52 0.02 for the query 00:52:00 For some varient of the query. 00:52:27 7269 for the triples 00:52:39 ooh, slight discrepancy 00:52:45 The rules. 00:53:21 Still. 00:53:37 I'm not sure what your inference thing is. 00:53:44 Are you doing the rule we used? 00:53:47 yep 00:54:00 it does { ?x :blargh ?y } => { ?x :blargh ?y } . 00:54:13 Do you optimize for that/ 00:54:13 ? 00:54:32 Where do you accumulate results? 00:54:42 * bijan suspects that writing to rdf_db is *really* killing him. 00:54:48 no - it's pretty general. But then, once you have a query engine, it's just a matter of plugging the variables back in at the other end 00:55:01 accumulate results? 00:55:17 it can print them out as a Python list, or as NTriples 00:55:25 Ok, so in a list. 00:55:33 yes, accumulating in a list is faster for me, I'll bet. 00:55:40 Writing each to the database is slow. 00:55:48 Interleaving the writes is prolly slow. 00:56:04 yeah... there are a number of reasons why it would be slower 00:56:39 I wonder why CWM is slower. 00:56:42 So much slower. 00:56:48 And I wonder if this is a corner case where you do well. 00:56:56 Or whether you do well in general. 00:56:58 the new CWM? Tim is interning formulae now 00:57:07 well in general: I doubt it :-) 00:57:18 I'm just trying to set up rm's test case right now... 00:57:32 And I have to run! Hurry up! 00:57:52 I'm trying, I'm trying! 00:59:17 it seems to just sit there 00:59:22 let's reduce the test case a bit... 01:00:18 Hmm. I bet you have to traverse the list for each additional statement? 01:00:25 You might be better off shrinkin the rules. 01:00:46 yep, I just did... still running 01:00:48 If you have big_list_of_triples (datastore) 01:01:05 Then for each antecedent statemetn: search list until get hit. 01:01:21 And having to start over if you get a miss down the line... 01:01:24 yeah, that will be slow. 01:01:48 Hmm... 01:02:18 (Assuming that I have your inference strategy correct) 01:02:25 even if that's fast for the once through case. 01:02:52 I added one triple to the ante and desc, and it added a second to the query... 01:02:57 s/query/inference/ 01:03:08 Which one? 01:03:12 music one? 01:03:14 Or the geo one? 01:03:16 yep 01:03:18 the music one 01:03:28 ah, add one more and it just sits there 01:03:32 bug? 01:03:39 it seems to be 01:03:47 * bijan is disappointed. 01:04:04 * bijan shifts to being superior because of correctness, rather than performance. 01:04:29 difficult to tell, though. The new line is a ?x ?y ?z type of thing, which might be causing it grief - but it already had one to handle 01:04:30 Is it any three triples or a specific third triple? 01:04:35 correctness: heh, heh 01:04:49 dunno, this one is still hanging. I'll cut it and try again 01:04:50 Oh, I'm *creaming* you on documentation! 01:05:05 yeah. There isn't any Eep documentation :-) 01:05:09 And Looks. I'm Much Cuter. ;) 01:05:11 Yes, I know :) 01:05:17 That's why I'm killing you :) 01:05:22 heh, heh 01:05:35 but "Cuter"? nah 01:05:42 *Definitely*. 01:05:44 No question. 01:05:48 Everyone agrees! 01:05:53 I don't 01:05:58 so it can't be everyone 01:06:02 "Everyone who counts agrees!" 01:06:12 I count! 1, 2, 3, 4, 5, 6, 7... 01:06:16 You do too, actually; you just lie about it. 01:06:28 Perhaps 01:06:29 ok I've really got to run. ta 01:06:32 c'ya! 01:06:36 bijan has quit ("Leaving") 01:09:57 ooh, it seems to be a problem in the query module, in fact 01:10:42 without this one particular line, it's really quick:- 01:10:43 Doing parse test... 01:10:43 0.0499999523163 134 01:10:43 Doing query test... 01:10:43 1.05000007153 134 01:12:33 ooh, it just did a slightly different test in about half a minute:- 01:12:33 Doing query test... 01:12:33 36.4199999571 134 01:13:52 and again, with a proper "length of results" count: 36.25 31 01:14:38 ah, I have an idea 01:17:19 * sbp tries it with a new cp function 01:17:44 Pff, slower: 37.6799999475 31 01:20:45 Hmm... it's the last bit of a particular function that's taking so long 01:21:12 argh! no wonder 01:21:12 1014513652.06 265856 01:21:32 265856 cartesian paths! not efficient going through all of them... 01:22:00 s/paths/products/ 01:25:23 so, it is the cp function doing it... ugh 01:27:54 ooh, with the new function, I get a MemoryError 01:27:56 that's a new one 01:30:20 bijan will be so happy :-) 01:34:45 heh, the test that crashes out all of the time is trying to compute 35,624,704 cartesian products 01:40:11 * sbp mails bijan the good news 01:42:29 heh, heh:- 01:42:30 File "query.py", line 61, in xtquery 01:42:30 if n > 500000: raise "You want me to find "+str(n)+" cartesian products? Pff" 01:42:30 You want me to find 35624704 cartesian products? Pff 01:46:38 http://www.python.org/doc/current/lib/profile.html 01:47:28 ooh, neat 01:47:42 * sbp says "ooh" a lot lately 01:47:57 Me too, I think. 01:48:10 'tis a good word 01:48:35 Heh, google bought the word "python" on Google. 01:48:46 -- 01:48:47 python=$10K? 01:48:47 Enter Google's programming contest 01:48:47 Win fame, fortune and a trip to CA 01:48:47 google.com/programming-contest/ 01:48:48 -- 01:48:59 heh 01:50:03 They have a lot of different versions of it, but they don't seem to have bought other languages. 01:50:26 Ah, they bought C++ 02:09:19 bijan (bparsia@login6.isis.unc.edu) has joined #swhack 02:09:35 uh oh 02:09:44 I came on to giggle! 02:09:50 damnit! 02:10:13 I am curious about the huge discrepancy in your times :) 02:10:24 me too 02:10:45 well, for smaller queries, you don't get many results, so it's stunningly fast 02:11:00 Eh, that alone isn't good reasoning. 02:11:10 but for larger ones, you get more results... for each result, your cp value goes up by the length of that result 02:11:18 it skyrockets quite quickly 02:11:33 "many results", you mean consequences per application? 02:11:35 goes up: as in *multiplied* by the length 02:11:47 no, not consequences... 02:11:51 this is in the query module 02:12:03 I don't understand. 02:12:08 The geo example had lots of results... 02:12:13 OHOHOH. 02:12:17 YOu mean anding the results/ 02:12:19 ? 02:12:34 (That's the same as what I meant by consequences0 02:12:49 Hm... yeah, I guess so 02:12:53 (A query is just an inference to the matches in the antecedent) 02:13:10 Like the sample one you gave. 02:13:30 So, filtering a list is fast. 02:13:40 (Hmm. I wonder if list comprehensions would be nippy..) 02:14:00 SOrry. 02:14:07 well, it just performs simple matching, and then ponders over the paths through each of the results 02:14:08 Say I have a query statemetn A. 02:14:13 O.K. 02:14:33 To match A against the store, I wall the list examining each statement. 02:14:42 If it matches, it goes into the results. 02:14:48 When I read the end of the list I'm done. 02:14:49 Yes? 02:15:11 yep 02:15:13 If I have two query statements, A and B 02:15:19 I wall the list gathering all the As? 02:15:25 Then walk the list gathering all the Bs? 02:15:31 [(a results), (b results)] 02:15:36 and then find the paths 02:15:58 "paths'? 02:16:13 cartesian product 02:16:28 here's how I caught it:- 02:16:28 for item in all: n *= len(item) 02:16:29 if n > 500000: raise "You want me to find "+str(n)+" cartesian products? Pff" 02:16:41 "all" is the list of results, per above 02:16:49 [n is initially 1] 02:17:12 Hmm. I'm not sure I understand. 02:17:16 Which N hurts? 02:17:25 The length of the query? (A & B &...n) 02:17:39 Or the length of the results (rA & rB &....) 02:17:45 the product of the lengths of the items in the results 02:17:55 (rA * rB *....) 02:17:59 Ah. 02:18:09 Because you cross compare all the permutaitons. 02:18:14 yep 02:18:20 *that's* dumb. 02:18:23 yep 02:18:45 rather: that's *dumb*. 02:19:24 Ouch. i"m trying to generate a simple unification algo for that. 02:19:34 For your walk the list technqiue. 02:19:36 Not trivial. 02:19:46 especially without backtracking. 02:20:03 indeed. the approach seemed very intuitive to me, but I didn't consider what'd happen when I scaled it up a little bit... :-) 02:21:56 the only way to beat it would be to totally scrap the algorithm, and start again 02:22:05 Well, yes. 02:23:06 Yay! You're again behind me in completeness too! :) 02:23:45 I'm a fair way behind in all places... except for small queries :-) 02:24:00 Well, actually, there too. 02:24:07 Sorry. 02:24:12 how's that? 02:24:38 Well, you kill me if I use an n3 rule writing to a "heavy" context. 02:24:56 But if I use a findall I'm 2 order of mags faster :) 02:25:56 * sbp ponders re-writing vs. scrapping 02:26:17 In general, SWI queries are going to be reasonable quick. 02:26:49 I mean, I don't *have* to use an N3 rule to find all the foos that match bar. 02:26:55 That's part of the reason to do in SWI :) 02:27:07 yep... I can see the logic behind using Prolog 02:27:16 SO that I can write prolog programs to manipulate the n3 store. 02:27:59 Come over to the dark side. 02:28:06 Learn prolog... 02:28:09 Work on mine... 02:28:42 by the time I would have learned Prolog to the extent that I can help you with CWMClone, you'll have finished it and gone on to better things 02:29:16 Ha! I doubt it :) 02:29:26 But I would be happy to turn it over to you ;) 02:29:37 Hey, your other choice is to rewrite query... 02:29:47 Learning prolog is looking good, eh? 02:30:20 blargh. I suppose. Any good manuals online? 02:30:26 Yes. 02:30:34 .google Prolog primer 02:30:35 Prolog primer: http://www.ibc.wustl.edu/moirai/cs_magenta/prolog.html 02:30:36 SEveral *Excellent* tutorials, actually. 02:31:33 well, that one is weak 02:31:37 I link to a few from my first article 02:31:42 .google Prolog tutorial, that bijan approves 02:31:43 no results found. 02:31:46 ah, good point 02:33:32 Hmm. WHy do you get the cartesian product? 02:34:08 so that I can match the variables against all of the paths 02:34:24 Ok, A & B 02:34:37 I'm walking for A, find a match. 02:34:42 Why not check B right then? 02:34:47 Will that help? 02:35:04 yeah, but that'd be a totally different way of querying 02:35:13 in fact, it's the original way that I wrote it, as I recall 02:35:22 Sure, it's a kind of unification. 02:35:28 Why did you abandon it? 02:35:50 the latter way seemed easier (it was much easier to program), and faster for the small tests that I ran 02:36:36 Well, what about filter on A, then walk rA to see if B matches? 02:36:49 Hmm... 02:36:49 I don't understand the "paths' :) 02:37:05 perhaps I could print out debug information for a sample query? 02:37:08 The cartesian product doesn't maek *sense* 02:37:16 Sure. 02:38:46 the query:- 02:39:50 print tquery(eep.parse("""?x <#sonOf> ?y . 02:39:50 ?y <#sonOf> ?z ."""), 02:39:50 eep.parse("""<#Bob> <#sonOf> <#Fred> . 02:39:50 <#Fred> <#sonOf> <#John> . 02:39:50 <#John> <#sonOf> <#Wayne> ."""), 1) 02:40:23 Ok, can, both match all of them. 02:40:46 You store bindings with the results? 02:41:15 i.e., {(?x, #bob) (?y, #Fred)... 02:42:01 argh, sorry, I got disconnected 02:42:03 actually, that's not a good example because the query triples are basically identical 02:42:04 Or somthing. 02:42:06 here's a better one:- 02:42:12 print tquery(eep.parse("?x <#sonOf> ?y .\n?x <#name> ?z ."), 02:42:12 eep.parse("""<#Bob> <#sonOf> <#Fred> . 02:42:12 <#Fred> <#sonOf> <#John> . 02:42:13 <#Bob> <#name> "Bob" . 02:42:15 <#Fred> <#name> "Fred" ."""), 1) 02:42:29 and the debug information:- 02:42:29 all results: [[[<#Bob>, <#sonOf>, <#Fred>], [<#Fred>, <#sonOf>, <#John>]], [[<#Bob>, <#name>, "Bob"], [<#Fred>, <#name>, "Fred"]]] 02:42:30 dict of results: {'?z': None, '?y': None, '?x': None} 02:42:46 I store the bindings only at the very end, when it does the variable matching 02:42:54 I don't understand. 02:43:12 You found all the #sonOfs 02:43:20 THen all the #names 02:43:22 and all the names 02:43:32 then, for each sonOf. 02:43:37 no 02:43:50 not for each sonOf. This when I get the cp 02:43:58 which turns out to be: [[[<#Bob>, <#sonOf>, <#Fred>], [<#Bob>, <#name>, "Bob"]], [[<#Fred>, <#sonOf>, <#John>], [<#Bob>, <#name>, "Bob"]], [[<#Bob>, <#sonOf>, <#Fred>], [<#Fred>, <#name>, "Fred"]], [[<#Fred>, <#sonOf>, <#John>], [<#Fred>, <#name>, "Fred"]]] 02:44:18 then I apply matching 02:44:30 Ergh. 02:44:32 Pretty print that? 02:44:36 sure... 02:44:47 Or pretty paste, whatever :) 02:45:02 [[<#Bob>, <#sonOf>, <#Fred>], [<#Bob>, <#name>, "Bob"]] 02:45:02 [[<#Fred>, <#sonOf>, <#John>], [<#Bob>, <#name>, "Bob"]] 02:45:03 [[<#Bob>, <#sonOf>, <#Fred>], [<#Fred>, <#name>, "Fred"]] 02:45:03 [[<#Fred>, <#sonOf>, <#John>], [<#Fred>, <#name>, "Fred"]] 02:45:39 the original query is ?x <#sonOf> ?y .\n?x <#name> ?z . 02:45:40 Then you see if ?x is the same in the first one? 02:45:48 yep 02:45:58 Ok, yes, that's insane :) 02:46:07 well, it's a bit more generic than that, because it has to work for all queries... 02:46:41 Hmm. But hte originalist is just a list of triples yes/ 02:46:42 ? 02:46:48 Are you using a >2.1 python? 02:46:49 the original list of what? 02:46:52 Python 2.2 02:46:53 triples 02:46:57 Ooo! Ok 02:47:00 yeah, that's just a list 02:47:04 Try a list comprehension on it! 02:47:15 and how would I be doing that? :-) 02:47:20 (Taking a leaf from mnesia) 02:47:26 Hit the manual. 02:47:40 * sbp prints out a copy and goes to find a hammer 02:47:57 A list comprehension is basically a query of the form "All items where item is blah blah blah 02:48:48 I don't know the python syntax for them. 02:49:29 Mnesia (the erlang distributed db) uses them for it's query language (mnesymone) 02:49:31 oh, right: [x(s) for s in list] 02:49:44 There you go. 02:50:05 I use that all over the place for stuff. What do I do with in in the query, though? 02:50:06 Ok, what would be the transform of the query... 02:50:41 Well, look at your query... 02:51:06 ?x :sonOf ?y. ?x :name ?z. 02:51:20 Hmm. 02:51:23 No. 02:51:52 Damn, I'm still binding the variables paritally :) 02:52:12 End up with your cross product again. 02:52:53 When I was writing the first query engine, I kept doing partial bindings... quite frustrating. Then I thought I had it figured out for Eep :-) 02:53:15 well, it does *work* 02:53:17 Partial bindings are the right why. 02:53:19 way 02:55:17 What's your break over point? 02:55:31 3 query statements and 100 items? 02:55:31 :) 02:55:36 *ahem* 02:55:48