2008-09-08T01:35:29 *** Mark__T has joined #openils-evergreen 2008-09-08T04:46:01 *** Mark__T has quit IRC 2008-09-08T07:29:50 *** Mark__T has joined #openils-evergreen 2008-09-08T08:02:21 *** kgs_away has joined #openils-evergreen 2008-09-08T08:02:57 *** kgs_away is now known as kgs 2008-09-08T08:46:31 *** rsinger has joined #OpenILS-Evergreen 2008-09-08T08:50:13 *** dbs has joined #openils-evergreen 2008-09-08T08:51:08 kgs: I see you haven't posted your 1.4 release update yet 2008-09-08T08:51:25 dbs: I was busy upgrading the blog :-) 2008-09-08T08:51:31 so it will be soooooooooo pretty 2008-09-08T08:51:39 actually the ESI blog, but never mind 2008-09-08T08:51:44 Not sure if you want to point out the Trac roadmap 2008-09-08T08:52:03 http://svn.open-ils.org/trac/ILS/ -> click on Roadmap 2008-09-08T08:52:22 then click "Milestone: 1.4" to see the list of open tickets 2008-09-08T08:52:41 (which theoretically represents the big pieces that we need to finish off) 2008-09-08T08:52:46 I see that still doesn't work with evergreen-ils (just noting) 2008-09-08T08:53:49 yeah - is dmcmorris_esi rolling that out or is that on someone else's (my?) plate? 2008-09-08T08:54:30 I will ask dmcmorris_esi 2008-09-08T08:54:40 he just fixed the MTA for the email :) 2008-09-08T08:54:45 looking forward to the pretty 2008-09-08T08:54:53 pretty? 2008-09-08T08:55:23 to summarize: yes, that's a good thing to add -- thank you 2008-09-08T08:58:44 "the pretty" as in new blog style 2008-09-08T08:59:20 kgs: also good because it will hold our feet to the fire in terms of concrete things that need to be done - even though it introduces yet another roadmap 2008-09-08T08:59:48 dbs: I was indeed having a sinking feeling, "oh goody... another roadmap" 2008-09-08T08:59:58 and yet also thinking this had more detail 2008-09-08T09:00:14 but one that (theoretically, at least) keeps itself in sync with reality 2008-09-08T09:01:06 I have also asked for Debian to be upgraded on evergreen-ils.org so that Dokuwiki and Wordpress would at least get in the ballpark range 2008-09-08T09:03:15 I'm also going to post about the acquisitions documentation work because I'm not getting anywhere and it has been a month 2008-09-08T09:03:40 ahh 2008-09-08T09:04:20 I exchanged phone messages with my professor of technical writers friend, but failed to connect 2008-09-08T09:04:42 I posted something to ACQNET last week... no nibbles 2008-09-08T09:10:37 dbs: also, the blog platform at evergreen-ils.org is so kludgey that my inclination is to blog exclusively at the new ESI blog... it's just so much faster 2008-09-08T10:03:00 *** Mark__T has left #openils-evergreen 2008-09-08T10:09:05 *** skmurphy has quit IRC 2008-09-08T10:16:21 *** skmurphy has joined #openils-evergreen 2008-09-08T10:23:06 *** skmurphy has quit IRC 2008-09-08T10:25:34 *** skmurphy has joined #openils-evergreen 2008-09-08T11:31:29 I can haz vandelay import now, yay! 2008-09-08T11:31:56 yay 2008-09-08T11:32:30 yay! 2008-09-08T11:33:46 s/yay/YAYZ/ 2008-09-08T11:53:51 well, hmm. sorta. https://comet.cs.uoguelph.ca/vandelay/vandelay.html?ses=6281b3f6e20aa33e57168902f92c389d 2008-09-08T11:54:58 dbs: what's the prob? 2008-09-08T11:55:48 I thought life was better because one of the previously created queues ("freaky") actually displayed some data, however corrupted 2008-09-08T11:56:09 but in running an import of a new queue, I don't get past the "Processing..." stage 2008-09-08T11:56:53 and no records show up in queued_bib_record 2008-09-08T11:56:57 bah. 2008-09-08T11:57:55 dbs: ok if i load records? 2008-09-08T11:57:59 berick: sure 2008-09-08T11:58:10 got some vandelay_unix.log results this time 2008-09-08T11:58:14 heh, good, they're your records! ;) 2008-09-08T11:58:54 bah, same thing here. 2008-09-08T11:59:19 dbs pasted "hunting down vandelay" at http://paste.lisp.org/display/66495 2008-09-08T12:00:23 dbs: grep "editor\[" in osrfsys. there will be a call that returned no results 2008-09-08T12:00:25 dbs annotated #66495 with "Corresponds with this in osrfsys.log" at http://paste.lisp.org/display/66495#1 2008-09-08T12:00:33 which is the results of a DB error, almost certainly 2008-09-08T12:00:59 right there with you 2008-09-08T12:01:41 hrm, well, it should have stopped processing at the first UPDATE_FAILED event... 2008-09-08T12:01:52 instead of getting all to the way to the create failure 2008-09-08T12:02:20 I'll poke at the db server log to see what's getting generated 2008-09-08T12:04:10 dbs annotated #66495 with "and the database log is interesting" at http://paste.lisp.org/display/66495#2 2008-09-08T12:07:13 running one of those insert statements from psql just seems to hang - interesting 2008-09-08T12:09:51 and ps wax on the db server shows 5 "INSERT" processes 2008-09-08T12:10:52 is it going crazy trying to match against our 3.7 million records? 2008-09-08T12:15:50 hmm. I could see vandelay.match_bib_record() taking a long time to process, given the many "SELECT * FROM biblio.record_entry" calls; need to check my indexes on those columns too 2008-09-08T12:21:38 hmm, bre.id and bre.tcn_value have indexes, mfr.value is using the expression index, so probably not that 2008-09-08T12:27:37 huh 2008-09-08T12:28:12 the INSERTs all finished, and inspecting the queues shows that only my manually issued INSERT actually worked: http://comet.cs.uoguelph.ca/vandelay/vandelay.html?ses=f3d4760b5939ec52875f738ddaad3086 2008-09-08T12:28:49 (queue "testfour") 2008-09-08T12:36:32 it's like the opensrf call times out and kills the insert 2008-09-08T12:43:12 hmm - appending to an existing queue seemed to work 2008-09-08T12:57:12 dbs: arg, sorry, had to step away 2008-09-08T12:57:14 cool, once I get that far, the actual import works 2008-09-08T12:57:18 * berick reads up 2008-09-08T12:58:10 s'ok, probably too early to have a piker like me poking at vandelay anyways 2008-09-08T12:58:41 right, if the call times out, the ML will send a rollback to the DB transaction 2008-09-08T12:58:45 so, that's interesting 2008-09-08T12:58:56 how long did your manual insert take? 2008-09-08T12:59:31 I didn't time it, I could try again to see 2008-09-08T12:59:41 but have to go meet lynn for lunch first 2008-09-08T13:00:03 no biggie 2008-09-08T13:00:09 thanks, dbs 2008-09-08T13:25:10 *** mrpeters-isl has joined #openils-evergreen 2008-09-08T13:25:29 Bill, you around? If so, could I have 5 minutes of your time? THis is Mike Peters from Indiana State Library 2008-09-08T13:30:50 *** mrpeters-isl has quit IRC 2008-09-08T13:33:44 *** skmurphy_lt__ has joined #openils-evergreen 2008-09-08T13:33:47 *** skmurphy_lt__ is now known as skmurphy_lt 2008-09-08T13:34:31 *** phase_bb has quit IRC 2008-09-08T13:47:14 berick: back, if you want to pursue anything 2008-09-08T13:47:41 i want to pursue edited documents 2008-09-08T13:47:45 * skmurphy_lt is not berick 2008-09-08T13:48:20 dbs: well, i take it you've narrowed it down a DB speed issue... 2008-09-08T13:49:12 berick: I suspect so, anyway. maybe we'll need a "ingest this crap server-side, and email me when you're done" mode for large databases / large imports 2008-09-08T13:49:34 skmurphy_lt: I want to generate edited documents! 2008-09-08T13:52:14 dbs: what's the expression index on mfr ... er ... on 2008-09-08T13:52:46 eeevil: it's the 1024 substr mfr (vs. mfrr) 2008-09-08T13:52:59 right ... duh 2008-09-08T13:53:48 for mrfr: "metabib_full_rec_value_idx" btree ("substring"(value, 1, 1024)) 2008-09-08T13:56:15 explain tells me it would use bitmap heap scan / bitmap index scan for a query against mfr.value, so that sounds right 2008-09-08T13:56:15 dbs: can you grab an ISBN from one of the records? 2008-09-08T13:56:27 shore 2008-09-08T13:56:29 ahh... 2008-09-08T13:56:33 well, you got it 2008-09-08T13:56:41 well 2008-09-08T13:56:45 yeah, grab an isbn 2008-09-08T13:57:13 *** skmurphy_lt__ has joined #openils-evergreen 2008-09-08T13:59:13 dbs pasted "ISBNs from last record added" at http://paste.lisp.org/display/66500 2008-09-08T13:59:45 ah, aacr2 - how I loathe thee 2008-09-08T14:00:19 what does explain say about ... SELECT * FROM metabib.full_rec fr WHERE fr.value LIKE LOWER('9780822342427%') AND fr.tag = '020' AND fr.subfield = 'a'; 2008-09-08T14:00:35 btw, the crap at the end should be stripped from the attr value 2008-09-08T14:01:41 dbs annotated #66500 with "explain THIS!" at http://paste.lisp.org/display/66500#1 2008-09-08T14:01:47 because of v.b_a_d.remove = (?:-|\s.+$) 2008-09-08T14:02:06 should "remove dashes and crap after (and including) a space" 2008-09-08T14:02:13 *** skmurphy_lt has quit IRC 2008-09-08T14:02:37 wow ... those are some crapy estimates 2008-09-08T14:03:08 shouldn't be using a bitmap scan on that query ... what does EXPLAIN ANALYZE say? 2008-09-08T14:03:35 *** skmurphy_lt___ has joined #openils-evergreen 2008-09-08T14:03:37 *** skmurphy_lt___ is now known as skmurphy_lt 2008-09-08T14:03:54 * dbs does some unicorn administration while that runs 2008-09-08T14:04:05 oh noes 2008-09-08T14:04:16 that should have been really fast... 2008-09-08T14:05:29 * dbs twiddles thumbs, and unicorn config knobs 2008-09-08T14:05:57 note that I haven't automated the cache-warmer 2008-09-08T14:06:20 so some of this would be cold, and also still have lots of performance work to do 2008-09-08T14:07:12 not unless you have the same record in there a lot of times ... a LOT of times 2008-09-08T14:07:52 I have indexes on (tag, subfield), value, and id for mfr 2008-09-08T14:08:49 * dbs wonders whether the LOWER pushes it through a different path due to en-CA locale for the cluster 2008-09-08T14:09:11 dbs annotated #66500 with "analyze THAT" at http://paste.lisp.org/display/66500#2 2008-09-08T14:09:37 hrm... 2008-09-08T14:10:10 so osrf_system.c needs an #ifndef block to set HOST_NAME_MAX 2008-09-08T14:10:11 would you mind creating an index on mfr? (def to follow) 2008-09-08T14:10:18 but i have no idea what an appropriate value would be 2008-09-08T14:10:20 i have an identical server that I could test this with 2008-09-08T14:10:25 eeevil: not at all 2008-09-08T14:10:42 256 seems really common 2008-09-08T14:10:52 well.. nevemind 2008-09-08T14:11:32 sboyette: i imagine that's fine 2008-09-08T14:11:40 ok 2008-09-08T14:11:57 eeevil annotated #66500 with "dev analyze" at http://paste.lisp.org/display/66500#3 2008-09-08T14:12:18 odd that mine uses the value index ... 2008-09-08T14:13:26 ahh... locale strikes again 2008-09-08T14:13:33 sboyette: what needs that? 2008-09-08T14:13:34 text pattern ops index? 2008-09-08T14:13:45 oh 2008-09-08T14:14:24 criminy crud, sounds like that's going to be a pattern throughout the schema eh? 2008-09-08T14:15:13 or use locale C ;) 2008-09-08T14:15:26 just like http://svn.open-ils.org/trac/ILS/changeset/10440 yeah? 2008-09-08T14:15:47 eeevil: heh - you know the story: not so bad for canucks, real bad for the finns 2008-09-08T14:15:48 yeah 2008-09-08T14:16:01 well, until 2.0, yeah 2008-09-08T14:16:18 when we'll incorporate NVL() sorting 2008-09-08T14:16:47 berick: 'make'. i've lost the error trace to compilation scroll now, but i can take the #define out and make it blow up again if you'd like 2008-09-08T14:17:09 berick: though it now detonates much later on at 2008-09-08T14:17:12 ld: duplicate symbol _osrf_stack_entry_point in .libs/libopensrf_la-osrf_stack.o and .libs/libopensrf_la-osrf_app_session.o 2008-09-08T14:17:14 *** skmurphy_lt__ has quit IRC 2008-09-08T14:17:51 wheee porting 2008-09-08T14:18:19 sboyette: os/2 eh? 2008-09-08T14:18:35 VMS 4.2 2008-09-08T14:26:14 *** skmurphy_lt has quit IRC 2008-09-08T14:32:32 berick: I finished my work on the status filtered storage method (ticket #19) 2008-09-08T15:13:14 dbs annotated #66500 with "Ahhh, much better after TPO" at http://paste.lisp.org/display/66500#4 2008-09-08T15:14:43 still no match for dev.gapines.org, but much, much better than before 2008-09-08T15:22:09 dbs: can i take it you are able to upload? 2008-09-08T15:22:14 without the timeouts 2008-09-08T15:22:15 yay, import creates queue and goes directly to displaying records 2008-09-08T15:22:22 ah, cool 2008-09-08T15:22:37 gotta figure out how to consistently get the "Select" column to show though 2008-09-08T15:22:59 ah, click in a box and SHIFT-TAB 2008-09-08T15:29:00 *** kgs has quit IRC 2008-09-08T15:54:34 esi people: your new blog site seems kinda disconnected from the rest of the site. might want to link the header to http://esilibrary.com instead of http://blog.esilibrary.com 2008-09-08T16:06:46 *** kgs has joined #openils-evergreen 2008-09-08T16:17:10 berick: re vandelay queue population, IIRC, the spool processor sends status messages (not STATUS, just "did 1") to the client ... having some feedback during the initial ingest would be good ... could we update the progress bar with something (or update something with something) using onrecv or whatever the streaming response event is called? 2008-09-08T16:18:54 *** kgs is now known as kgs_onroad 2008-09-08T16:24:14 phasefx et al: what do you think about using the "setRequestHeader()" method of XHR for staff client locale switching? at least for the XML/DTD stuff 2008-09-08T16:25:10 YAY! (X-OpenSRF-Language or whatever?) 2008-09-08T16:26:02 eeevil: right, I could reuse X-OpenSRF-whatever 2008-09-08T16:27:16 basically, just so I can add "SetEnvIf X-OpenSRF-whatever fr-CA locale=fr-CA" to eg_vhost.conf 2008-09-08T16:27:49 hrm 2008-09-08T16:28:14 but that won't automatically get set on every http request; just on XHR 2008-09-08T16:28:48 dbs: I'm not sure that we're pulling anything with xml or dtd's using XHR...? 2008-09-08T16:28:56 phasefx: right 2008-09-08T16:29:04 that was the "hrm", I got off track 2008-09-08T16:29:13 I think that pref we found is a good tactic 2008-09-08T16:31:53 more like this, probably: http://developer.mozilla.org/en/Setting_HTTP_request_headers 2008-09-08T16:32:03 phasefx: you mean general.useragent.locale? 2008-09-08T16:33:21 eeevil: we can, but we need some sane default of how many total items we are processing (since marc::batch doesn't read ahead to see how many recs are in the batch, at least, not that i know of) 2008-09-08T16:34:11 how about a secondary indicator of how many have been processed? 2008-09-08T16:34:43 dbs: yeah.. I was poking around with such things in http://open-ils.org/~phasefx/locale.js 2008-09-08T16:34:58 dbs: and reloading the chrome after changing the pref 2008-09-08T16:35:40 berick: just $dojo.byId('import-count-span').innerHTML = count_received; or something? 2008-09-08T16:41:16 ok ... /me runs away 2008-09-08T16:45:57 eeevil: yep, that'll do 2008-09-08T16:46:48 phasefx: right, that would take care of the properties, but we need to deal with the remote DTDs that are being driven by apache env sniffing 2008-09-08T16:47:11 * dbs is looking at a fr-CA staff client (somewhat) 2008-09-08T16:47:56 dbs: if the pref is set, then won't any http requests (whether from XHR or that pref sets a header? 2008-09-08T16:48:31 I'm pretty sure it gets transmitted in http headers 2008-09-08T16:48:59 hmm 2008-09-08T16:49:25 as part of User-Agent? 2008-09-08T16:49:26 testing now 2008-09-08T16:50:39 hrmm.. tried a random value like foobar.. it shows up in the User-Agent, but not in Accept-Language 2008-09-08T16:51:28 ah, intl.accept_languages 2008-09-08T16:51:28 yeah, I just tried that too 2008-09-08T16:51:57 I'm really worried about anything quasi-standard, to be honest 2008-09-08T16:52:13 intl.accept_languages does affect the Accept-Language header 2008-09-08T16:52:43 (which sounds insane, but I mean that from a "don't want the OPAC locale-switcher to get confused by this") 2008-09-08T16:53:12 you want different locales in embedded opac and staff client? 2008-09-08T16:54:04 No, I want the same locales - as long as the existing locale switcher for the opac can override accept-languages 2008-09-08T16:55:33 gah. gotta run. back on tonight 2008-09-08T16:55:41 *** dbs has quit IRC 2008-09-08T17:27:24 *** mjg_ has joined #openils-evergreen 2008-09-08T18:31:23 *** phase_bb has joined #openils-evergreen 2008-09-08T19:02:29 *** rsinger_ has joined #OpenILS-Evergreen 2008-09-08T19:19:21 *** rsinger has quit IRC 2008-09-08T20:45:11 *** dbs has joined #openils-evergreen 2008-09-08T20:51:57 * dbs spies mjg_ 2008-09-08T20:53:12 * mjg_ bows and flourishes 2008-09-08T20:56:32 uh-oh - more gin and tonics? 2008-09-08T21:01:59 Just diet soda tonight. Gotta show restraint every now and again so folks don't envy me. You know, too much. 2008-09-08T21:02:23 * mjg_ coughs in self-amusement. 2008-09-08T21:02:40 But I am very sleep-deprived, which is like being drunk but less costly. 2008-09-08T21:04:21 I know that state well 2008-09-08T22:29:26 *** mjg_ has quit IRC