2008-08-12T00:37:29 *** dbs has quit IRC 2008-08-12T01:26:00 *** Mark__T has joined #openils-evergreen 2008-08-12T03:10:10 *** phasefx has quit IRC 2008-08-12T03:10:40 *** berick has quit IRC 2008-08-12T03:10:46 *** bradl has quit IRC 2008-08-12T03:10:49 *** bradl has joined #openils-evergreen 2008-08-12T03:18:32 *** phasefx has joined #openils-evergreen 2008-08-12T03:18:54 *** berick has joined #openils-evergreen 2008-08-12T07:04:23 *** phasefx2_ has quit IRC 2008-08-12T07:04:23 *** gmcharlt_away has quit IRC 2008-08-12T07:04:52 *** gmcharlt_away has joined #openils-evergreen 2008-08-12T07:04:53 *** phasefx2_ has joined #openils-evergreen 2008-08-12T07:15:05 *** rsinger has quit IRC 2008-08-12T08:07:05 *** gmcharlt_away is now known as gmcharlt 2008-08-12T08:14:40 *** Slazer has joined #openils-evergreen 2008-08-12T08:19:51 *** kbeswick has joined #openils-evergreen 2008-08-12T08:21:24 *** rsinger has joined #OpenILS-Evergreen 2008-08-12T08:26:19 @later tell dbs re that full_rec patch ... I'm still scared of it. I'll push it, though, if that's concensus 2008-08-12T08:26:19 miker_: The operation succeeded. 2008-08-12T09:31:23 *** dbs has joined #openils-evergreen 2008-08-12T10:08:15 *** Mark__T has quit IRC 2008-08-12T10:42:40 *** dbs has quit IRC 2008-08-12T11:49:08 *** dbs has joined #openils-evergreen 2008-08-12T13:17:25 oh, good golly - I guess our sample fm_IDL.xml pretty much demands that we add example-reporter-extension to our schema, or suffer lots of postgresql error messages when cstore fires up 2008-08-12T13:18:52 ah - re: mfr index patch - I'm willing to test it out on our servers for a while if you want it to bake 2008-08-12T13:20:35 I need to figure out why our server has such horrible performance though. I can run an UPDATE and the process will show 2 minutes of CPU time over the span of an hour. 2008-08-12T13:20:59 dbs: UPDATE of what? 2008-08-12T13:21:36 in this case, updating biblio.record_entry source column where id < 1000000 (should be about 500K records) 2008-08-12T13:22:01 hrm... checking fkeys? 2008-08-12T13:22:42 there's no fkey on source, it's nullable 2008-08-12T13:23:20 there's an fkey to id, and the row is being updated 2008-08-12T13:24:00 or, that's the first thing that pops into my head 2008-08-12T13:24:11 huh - I just expected pgsql to recognize that the update isn't touching that column, but maybe. 2008-08-12T13:25:13 things do seem pathologically slow on this server, though, given that it's an 8-core 16GB RAIDed (5, yes, sigh) system dedicated to pgsql 2008-08-12T13:25:45 may I poke at it somehow? 2008-08-12T13:27:22 eeevil: eventually, perhaps - vpn is involved and the systems are hosted by a third party 2008-08-12T13:28:40 first thing I'd check is indexes on metabib.metarecord_source_map (both source and metarecord columns) ... it's possible they're not there by default, which would be non-good for staged search 2008-08-12T13:28:57 also, metabib.rec_descriptor.record 2008-08-12T13:29:04 well, not the first thing 2008-08-12T13:29:18 first thing would be a nice VACUUM ANALYZE 2008-08-12T13:34:20 righto - statistics statistics statistics 2008-08-12T13:37:18 we're up to about 4M bib records loaded now, so there's that. the plan is to add sources so that we can control transparency first, then clean up the org_unit hierarchy and add holdings 2008-08-12T13:44:18 those indices are all there, so that's goodness 2008-08-12T13:45:16 hah: a.re contains more than max_fsm_pages with useful free space - that's what I get for loading auths I guess 2008-08-12T13:50:56 *** rsinger is now known as Talis_US_Employe 2008-08-12T13:51:35 *** Talis_US_Employe is now known as Vendor_Lackey 2008-08-12T13:51:54 *** Vendor_Lackey is now known as rsinger 2008-08-12T14:01:01 dbs: vac full will help with that (though it will also take forever) 2008-08-12T14:01:28 as you well know, I'm sure 2008-08-12T14:10:10 eeevil: it's a delicate balancing act :) this is the first time I've had all of the records in place, so it's a new height for me 2008-08-12T14:12:06 i suppose an alternative to your mfr index patch would be to apply your pgsql source patch 2008-08-12T14:12:50 this is what was actually leading me towards compiling & installing pgsql as part of Makefile.install - but I doubt we want to deal with the lack of init scripts, etc 2008-08-12T15:10:33 *** rsinger_ has joined #OpenILS-Evergreen 2008-08-12T15:11:02 *** Slazer has quit IRC 2008-08-12T15:23:00 *** rsinger has quit IRC 2008-08-12T15:30:10 *** phase_bb has quit IRC 2008-08-12T15:38:05 *** _bott_ has quit IRC 2008-08-12T15:39:28 *** _bott_ has joined #OpenILS-Evergreen 2008-08-12T15:41:43 *** phase_bb has joined #openils-evergreen 2008-08-12T15:44:08 *** kbeswick has quit IRC 2008-08-12T16:00:04 *** agJohn has joined #openils-evergreen 2008-08-12T16:49:07 *** dbs has quit IRC 2008-08-12T20:25:43 @later tell dbs I'm working on a script to upgrade a DB to the same state as the patch would do for a new installation (the shadow full_rec patch) 2008-08-12T20:25:43 miker_: The operation succeeded. 2008-08-12T20:47:34 *** dbs has joined #openils-evergreen 2008-08-12T20:54:32 *** kgs is now known as kgs_away 2008-08-12T21:00:52 *** phase_bb has quit IRC 2008-08-12T21:07:14 dbs: Open-ILS/src/sql/Pg/1.4-shadow_full_rec-upgade-db.sql 2008-08-12T21:07:42 took less than 10min on dev (0.25M records) 2008-08-12T21:07:55 miker_: cool, in easy upgrade form even 2008-08-12T21:08:05 indeed 2008-08-12T21:08:27 it'll bomb out if anything else is going on (quite on purpose) 2008-08-12T21:08:29 well... 2008-08-12T21:08:34 "going on" 2008-08-12T21:08:54 if metabib.[real_]full_rec changes during the work 2008-08-12T21:09:11 easy enough to avoid that on our systems right now 2008-08-12T21:09:42 yeah ... and dev is a slow box, relatively speaking 2008-08-12T21:09:52 I'm still pgsql tuning 2008-08-12T21:11:36 btw - haven't dug into it too heavily yet, but does staged search perhaps want a coalesce() around date value for sort by date option? 2008-08-12T21:12:17 "too heavily" == "saw the error in the log and thought, hmm, bet that's because my data set includes a record without a pubdate" 2008-08-12T21:13:29 oh, there's a coalesce in there 2008-08-12T21:14:10 :) ... mine too, but surprisingly few are missing, actually 2008-08-12T21:14:24 I think there's even a NULLIF ;) 2008-08-12T21:16:01 there is 2008-08-12T21:17:53 oh, heh 2008-08-12T21:18:03 eh? 2008-08-12T21:18:05 I'm running trunk from about a week ago - before the coalesce/nullif 2008-08-12T21:18:14 ha 2008-08-12T21:18:20 brb 2008-08-12T21:18:27 my slow mind thinks like your great mind, just... slower 2008-08-12T21:24:09 *** _bott1 has joined #openils-evergreen 2008-08-12T21:24:34 _bott1: how's things? 2008-08-12T21:25:00 <_bott1> miker_: not bad, looking to wrap up the Z things 2008-08-12T21:25:17 dbs: I just stare at this stuff all day, so ... a bit of an advantage 2008-08-12T21:25:59 <_bott1> miker_: should I just grab the Z3950.pm that you already patched? 2008-08-12T21:26:00 _bott1: gooood ... I'd happily apply the changes if you'd like, but if you want to do it you'll want to grab a copy of Z3950.pm from rel_1_2_3 2008-08-12T21:27:16 <_bott1> K, I'll bring down one brick and test it there first 2008-08-12T21:28:16 ok ... so ... am I putting the changes in or are you? ;) 2008-08-12T21:28:45 dbs: ahh... it feels good to have 0 Ms in my 'svn up' :) 2008-08-12T21:29:15 <_bott1> miker_: I can do it ...I don't anticipate any problems, but are you going to be around? 2008-08-12T21:29:43 I will be here until you confirm that there are no problems. 2008-08-12T21:29:57 <_bott1> ...off to update 2008-08-12T21:30:12 k 2008-08-12T21:30:32 miker_: happy to help prod :) 2008-08-12T21:31:02 ahhh... 675K duplicate ISBNs out of 4M records 2008-08-12T21:33:22 just imagine the number of ISBN-less records that we could match too (well, do you have many ISxN-less records?) 2008-08-12T21:35:39 miker_: oh, I'm sure. let's run a query and find out.. 2008-08-12T21:36:23 * dbs ups max_fsm_pages to 750000 2008-08-12T21:37:00 * miker_ imagines the 3-way left self-join on full_rec... then remembers the days before the materialized reporting view ... and weeps a little 2008-08-12T21:37:26 dbs: you know, I just stick it at 2000000 and forget it 2008-08-12T21:37:39 probably wise 2008-08-12T21:38:46 hah - thanks for heading me off from the self-join route 2008-08-12T21:39:13 well, it'll work ... :) 2008-08-12T21:39:22 but, you could just use the mat view 2008-08-12T21:39:31 self-joined 2008-08-12T21:39:47 but 4M rows instead of (just guessing here) 100M 2008-08-12T21:42:01 would you believe 1.8M rows have no ISxNs? 2008-08-12T21:42:17 select count(*) from reporter.materialized_simple_record where isbn = '{NULL}' and issn = '{NULL}'; 2008-08-12T21:42:33 (if that's not too retarded a query) 2008-08-12T21:45:07 seems fine to me 2008-08-12T21:46:34 1.5M duplicate fingerprints 2008-08-12T21:47:31 oooo nice 2008-08-12T21:47:54 metarecord searches will do a body good in conifer 2008-08-12T21:48:35 <_bott1> miker_: Little error. Query returns result like "Showing 0 of 2". Log has some details... 2008-08-12T21:48:41 you can make that the default with on constant in the opac 2008-08-12T21:55:36 miker_: right - I've seen that - of course, when we dedupe the ISxNs, we'll probably have far fewer dupe fingerprints :) 2008-08-12T21:55:56 probably so 2008-08-12T21:56:49 though ... you'll want to dedup on more the ISxN probably ... we usually treat that as a secondary optional match (must match one of isbn,issn,title+author(last name), etc) 2008-08-12T21:57:19 oh yeah, it's far from trustworthy! 2008-08-12T21:58:20 i imagine doing something like building staging tables with fingerprints on steroids to more solidly identify dupes 2008-08-12T21:58:54 that would work well for records already in there, yeah 2008-08-12T21:59:08 though SQL is far from the best mechanism for manipulating MARC ;) 2008-08-12T21:59:20 not that there's a good mechanism, mind you 2008-08-12T22:00:29 (just dumping the db now so I can update staged search and put your real_full_rec patch to work) 2008-08-12T22:01:37 dumping to nfs... gah. 2008-08-12T22:02:29 oh dear 2008-08-12T22:02:45 time for wal archiving and hot backups 2008-08-12T22:04:08 eventually... third party servers, man, I'm just working with what's there for now 2008-08-12T22:04:18 oh, I understand 2008-08-12T22:11:28 mmm... beer. 2008-08-12T22:44:50 *** _bott1 has left #openils-evergreen 2008-08-12T22:57:59 hmm - I think in a vanilla EG install, metabib_full_rec_value_idx won't exist 2008-08-12T22:58:36 yeah, commented out 2008-08-12T22:58:41 (or it was) 2008-08-12T22:58:53 * jeff watches a backup run 2008-08-12T22:59:02 * jeff gets more work done 2008-08-12T22:59:37 (time to export some data) 2008-08-12T22:59:39 change to DROP IF EXISTS and screw 8.1? 2008-08-12T23:19:42 *** kgs_away has quit IRC 2008-08-12T23:22:52 dbs: hrm... you'll want that if you sort on title or author, or do isbn searche 2008-08-12T23:22:55 searches 2008-08-12T23:23:00 ok .. /me out 2008-08-12T23:23:28 gnite 2008-08-12T23:23:53 (and thanks for the typo fix! one day I'll get hooked on phonics ;) ) 2008-08-12T23:27:09 hah 2008-08-12T23:27:23 *** dbs has quit IRC