2008-08-25T01:23:22 *** Mark__T has joined #openils-evergreen 2008-08-25T02:05:25 *** jeff_ has quit IRC 2008-08-25T02:06:16 *** jeff has joined #openils-evergreen 2008-08-25T07:41:27 *** kgs_onroad has joined #openils-evergreen 2008-08-25T08:08:20 *** kgs_onroad has quit IRC 2008-08-25T08:09:28 *** kbeswick has joined #openils-evergreen 2008-08-25T08:12:19 *** eeevil has joined #openils-evergreen 2008-08-25T08:17:34 *** kgs_onroad has joined #openils-evergreen 2008-08-25T09:04:00 *** phasefx has joined #openils-evergreen 2008-08-25T09:12:10 *** sylvar has joined #openils-evergreen 2008-08-25T09:12:54 Good morning, y'all. We finally have a real live physical server for a database server! At the moment it seems to be running postgres 8.3 -- is that unusual for an Evergreen db server? 2008-08-25T09:13:37 Mind you, we're not using it yet, I'm trying to set it up. 2008-08-25T09:15:54 *** dbs has joined #openils-evergreen 2008-08-25T09:17:48 Good morning, Dan. We finally have a real live physical server for a database server! At the moment it seems to be running postgres 8.3 -- is that unusual for an Evergreen db server? 2008-08-25T09:17:53 sylvar: it's impossible for an evergreen server :) 2008-08-25T09:18:08 ah, okay. I'll get 8.2 installed then. 2008-08-25T09:18:38 right on 2008-08-25T09:20:11 fortunately it's about as complicated as apt-get remove postgresql-8.3 and apt-get install postgresql-8.2 2008-08-25T09:20:52 getting the data loaded might be trickier; I've never done that before 2008-08-25T09:20:53 sylvar: well, it's not _quite_ impossible - some parts will run with pg 8.3 at the moment, on trunk 2008-08-25T09:22:20 berick: special ILS-Contrib auth handler... you mean using auth_digest instead of auth_basic? 2008-08-25T09:22:33 sylvar: but of course the other parts will explode in a righteous fury 2008-08-25T09:22:45 dbs: i mean, why the Apache handler at all? 2008-08-25T09:24:42 berick: I think it matched the docs I was following - and I wanted MD5 digest because it was my first time setting up a piece of open-ils.org infrastructure 2008-08-25T09:25:44 interesting. for the other trac projects, i'm just relying on trac's auth module.. no special apache handling required 2008-08-25T09:25:44 dbs: happy belated birthday! 2008-08-25T09:25:55 merci, kbeswick 2008-08-25T09:27:15 berick: see http://trac.edgewall.org/wiki/TracAuthenticationIntroduction - specially the bottom paragraph about passwords going in cleartext over the network 2008-08-25T09:28:29 Okay, I've got postgresql-8.2 running now, and I'm feeding it the results of pg_dumpall. I'm not sure why it's saying 2008-08-25T09:28:31 psql:/data/alldbs_dump:10573: ERROR: relation "author_field_entry" does not exist 2008-08-25T09:28:32 IIRC, I remember thinking to myself "No way in hell am I going to be responsible for setting up an auth system with cleartext passwords" 2008-08-25T09:28:32 psql:/data/alldbs_dump:10580: ERROR: relation "full_rec" does not exist 2008-08-25T09:28:34 psql:/data/alldbs_dump:10587: ERROR: relation "keyword_field_entry" does not exist 2008-08-25T09:28:35 but I'll see how it goes. 2008-08-25T09:29:27 dbs: ah, ok 2008-08-25T09:29:32 * berick wonders if we should force SSL 2008-08-25T09:30:47 *** sylvar has quit IRC 2008-08-25T09:31:33 *** sylvar has joined #openils-evergreen 2008-08-25T09:39:09 *** phase_bb has quit IRC 2008-08-25T10:11:57 *** Mark__T has quit IRC 2008-08-25T10:19:54 so... what should we be doing with kbeswick's autotools work for Evergreen - let him adjust to the pruning and shifting of the ./Evergreen directory, then retest and apply if all is well? 2008-08-25T10:20:17 dbs: that sounds like a plan to me 2008-08-25T10:20:52 as far as the perfect set of configure switches etc, we'll probably be better served adjusting it after the fact (and after running into the rough edges enough times) 2008-08-25T10:21:00 grand 2008-08-25T10:21:28 agreed 2008-08-25T10:21:29 btw, slazer's last day was Thursday... *sniff* although he may be back for the fall 2008-08-25T10:21:53 just in time for phasefx to jump back into the SC ;) 2008-08-25T10:22:49 man that xul stack is a beast to debug 2008-08-25T10:23:18 yeah 2008-08-25T10:36:23 Oh dear. 2008-08-25T10:36:25 psql:/data/alldbs_dump:44860830: invalid command \N 2008-08-25T10:36:26 psql:/data/alldbs_dump:44860831: invalid command \N 2008-08-25T10:36:28 psql:/data/alldbs_dump:44860832: invalid command \N 2008-08-25T10:36:29 etc. 2008-08-25T10:36:39 *** phase_bb has joined #openils-evergreen 2008-08-25T10:37:29 Is there a straightforward command or procedure that I should be using in order to move my database from the virtual server to the physical one? 2008-08-25T10:38:24 Hmm. does pg_dumpall include the createlang / include contrib directives for the new databases? 2008-08-25T10:40:12 Would it be sufficient to grep the dumpfile for 'createlang'? It's 55gb, so it'll take a bit of time to grep it. 2008-08-25T10:41:10 sylvar: I would expect those commands to be in the first 1000 lines or so 2008-08-25T10:44:07 so maybe something like "head -n 10000 dumpfile | grep -i plperl" ? 2008-08-25T11:08:43 CREATE PROCEDURAL LANGUAGE plperl; 2008-08-25T11:08:45 ALTER PROCEDURAL LANGUAGE plperl OWNER TO postgres; 2008-08-25T11:08:46 -- Name: plperlu; Type: PROCEDURAL LANGUAGE; Schema: -; Owner: postgres 2008-08-25T11:08:48 CREATE PROCEDURAL LANGUAGE plperlu; 2008-08-25T11:08:49 ALTER PROCEDURAL LANGUAGE plperlu OWNER TO postgres; 2008-08-25T11:08:51 LANGUAGE plperlu IMMUTABLE STRICT; 2008-08-25T11:08:52 LANGUAGE plperlu; 2008-08-25T11:09:46 if it's the same major version (8.2, say), platform and alignment (32bit vs 64bit) you can just copy the data directory 2008-08-25T11:09:56 and now I'm really out for about an hour. biab 2008-08-25T11:13:54 *** _bott_ has joined #OpenILS-Evergreen 2008-08-25T11:16:04 hmm, "Linux evergreen-test 2.6.22-14-server #1 SMP Tue Feb 12 08:27:05 UTC 2008 i686 GNU/Linux" vs. "Linux evergreen 2.6.24-19-server #1 SMP Sat Jul 12 00:40:01 UTC 2008 i686 GNU/Linux"... does that looks like the same platform and alignment to y'all? or sufficiently similar? 2008-08-25T11:24:44 ubuntu gutsy -> ubuntu hardy 2008-08-25T11:29:07 hmm, so would it make sense to upgrade the older version and then copy the data directory? 2008-08-25T11:31:47 sylvar: that won't help you, as there is no auto-upgrade of pg 8.2 database clusters to pg 8.3 in the ubuntu gutsy -> hardy upgrade 2008-08-25T11:31:54 although 2008-08-25T11:32:10 ugh, you don't want that anyways - never mind 2008-08-25T11:32:30 yeah, I've already downgraded postgres on the new server to 8.2 2008-08-25T11:38:21 given that, does the kernel version matter? are there other differences that would matter? 2008-08-25T11:39:34 sylvar: no, I wouldn't think so 2008-08-25T11:41:42 OK, so as long as the data directories are the same path from the root, I should just be able to copy 'em over. awesome. I'll wait until after lunch to see if anyone sees a glaring flaw with the plan, but this sounds like a win. thanks! 2008-08-25T11:41:44 dbs++ 2008-08-25T11:41:49 miker++ 2008-08-25T11:56:12 miker_: I'm trying your "cat _pertinent-db-files_ > /dev/null" hack as a temporary countermeasure for our sysadmin's nasty "tar czf backup.tar.gz /" nightly fs-cache-killing strategy 2008-08-25T11:56:56 focusing on indexes... should be interesting 2008-08-25T12:05:55 that's an awesome backup strategy right there :) 2008-08-25T12:06:28 is the resulting tarball split(1) over a stack of 7,200 floppies? :) 2008-08-25T12:09:18 tornado warning all over Equinoxia until 4pm... nooo it mah birthday! 2008-08-25T12:09:47 sboyette: it's... less than optimal, yes. debian doesn't support their backup system of choice, so this was their stopgap approach until something better (rsync-based, methinks) gets set up 2008-08-25T12:09:54 (http://icanhascheezburger.com/2007/05/02/nom-nom-nom-2/) 2008-08-25T12:17:31 hmm - throwing the fts indexes into the fs cache seems to help significantly, surprise surprise 2008-08-25T12:17:34 miker_++ 2008-08-25T12:18:52 maybe I'll put together a stupid perl script "cache-warmer.pl" that queries pg_class for just those data files and cats them out to /dev/null; I could see that being useful after big data loads 2008-08-25T12:59:08 miker_: http://coffeecode.net/archives/165-Heating-up-Evergreen-search.html if you're interested 2008-08-25T13:00:58 miker_, we fired our legacy box so you can look for the historical data. or, that's what I'm told to tell you anyway :) 2008-08-25T13:03:45 *** rsinger has quit IRC 2008-08-25T13:06:00 atheos: thanks :) 2008-08-25T13:06:38 dbs: you want to try out GIN indexes too, instead of GiST? 2008-08-25T13:06:45 they'll be smaller, and build much faster 2008-08-25T13:07:02 but can be caused to throw ugly errors if empty searches slip through 2008-08-25T13:07:49 eeevil: maybe on the next iteration - got to have something else to blog about :) 2008-08-25T13:08:43 haha 2008-08-25T13:15:59 ah, so. authorities are slow because we have an en-CA locale for the database cluster, rather than C or POSIX, and authorities depend on LIKE. 2008-08-25T13:18:51 so theoretically I could create another index using text_pattern_ops to sidestep that (not so worried about en-CA, more worried about no-NO and the like) 2008-08-25T13:22:10 *** pmurray_away is now known as pmurray 2008-08-25T13:24:46 dbs: indeed ... I'd have no problem making the text_pat_opts the default index type on are.value 2008-08-25T13:25:49 ooh, that sped up the authorities dramatically. nice 2008-08-25T13:26:48 eeevil: cool, shall I add just a simple "create index "authority_full_rec_value_tpo_index" on authority.full_rec (value text_pattern_ops);"? 2008-08-25T13:27:11 there's an fts index on value, but no other index 2008-08-25T13:31:33 an fts index on index_vector, you mean? 2008-08-25T13:32:08 yes, that is what I mean 2008-08-25T13:33:08 the PROCEED SIR! :) 2008-08-25T13:45:02 also, for now I'll create an upgrade file named "1.2.3-1.4-upgrade-db.sql" - assuming that there will be no 1.2.4 or 1.3 in between 2008-08-25T14:52:55 *** kgs_ has joined #openils-evergreen 2008-08-25T15:13:13 <_bott_> berick told me when he was in town that a delay for opportunistic hold capture was on the way. Has this made it into a release yet? 2008-08-25T15:13:40 thas been around for a bit 2008-08-25T15:15:06 <_bott_> what's the default delay? 2008-08-25T15:15:48 <_bott_> nevermind, looks like 24h 2008-08-25T15:16:37 _bott_: there is no default delay, actually. it's trivial to add to the database, though (no UI for controllling it yet) 2008-08-25T15:17:32 *** sylvar has quit IRC 2008-08-25T15:18:20 <_bott_> berick: looks like 24h is set in the example hold_targeter.pl though ? 2008-08-25T15:19:02 _bott_: that specifies how long a hold will sit targeted, but un-captured before the targeter looks for a new copy 2008-08-25T15:19:25 i'm looking for the org setting now... 2008-08-25T15:19:28 <_bott_> oh yeah, gotcha 2008-08-25T15:21:22 _bott_: something like this is what you need: insert into actor.org_unit_setting (org_unit, name, value) values (1, 'circ.hold_stalling.soft', '"1 day"'); 2008-08-25T15:21:30 the value is JSON-encoded, hence the extra quotes 2008-08-25T15:21:46 <_bott_> wonderful, thanks! 2008-08-25T15:21:56 assuming 1 is the ID of the root org unit, this setting will apply globally 2008-08-25T15:24:40 skmurphy! 2008-08-25T15:24:45 dbs! 2008-08-25T15:25:15 hows it going? 2008-08-25T15:25:40 good, good - my attention is fractured, but occasionally my synapses fire in a useful way 2008-08-25T15:26:44 i think that used to happen with me 2008-08-25T15:27:05 did the esi guys tell you i'm leaving PINES? 2008-08-25T15:27:09 I think I might have dreamed that you were no longer in GA 2008-08-25T15:27:14 well there ya go 2008-08-25T15:27:24 prophetic! 2008-08-25T15:27:41 thanks for thinking of me =) 2008-08-25T15:27:50 where are you going? 2008-08-25T15:28:00 starting a competing business 2008-08-25T15:28:03 that is not true 2008-08-25T15:28:07 hah 2008-08-25T15:28:11 i'm going into contract work, though 2008-08-25T15:28:20 if you want a job north of the border, I could find you a steady gig 2008-08-25T15:28:39 i've long assumed expat status was in my future 2008-08-25T15:28:59 canada is so reasonable and cold 2008-08-25T15:29:12 cold logic 2008-08-25T15:47:04 *** kbeswick has quit IRC 2008-08-25T15:56:53 damn, missed talking to kbeswick in person again 2008-08-25T16:06:15 yay, new Perl tests caught a compile error.. saving me some head scratching later on 2008-08-25T16:06:25 for opensrf, that is 2008-08-25T16:06:40 a winrar is us! 2008-08-25T16:07:18 indubitably 2008-08-25T16:07:39 sboyette++ tests++ 2008-08-25T16:09:30 damn! I'm having too much fun just running searches on our bibs + authorities test server, now that the speed is tolerable 2008-08-25T16:09:50 does your library have things in it? 2008-08-25T16:10:42 ha! 2008-08-25T16:11:38 dbs: try GIN indexes ... just create the same index, but say "USING GIN" instead of "USING GIST" ... then drop the gist index and vacuum the table ... you can even "CREATE INDEX CONCURRENTLY" to allow updates to the table while the index builds 2008-08-25T16:12:43 eeevil: cool, thanks for the hints 2008-08-25T16:14:01 * eeevil wants someone else test out gin ... badly :) 2008-08-25T16:18:55 dang, looks like our added content provider really is using IP address authentication :( 2008-08-25T16:20:18 eeevil: I have so many items on my personal wishlist of round tuits. I think I need to order some Jolt. Or Red Bull. Or whatever the hacker kids are drinking these days. 2008-08-25T16:21:34 premium gin 2008-08-25T16:22:08 and single-malts. and guinness. the ones i know, anyhow :) 2008-08-25T16:39:38 mmmm, Guinness 2008-08-25T16:46:54 atheos: yo. we're done with the old box for now, btw 2008-08-25T16:48:06 awesome, thanks 2008-08-25T16:48:46 no no, thank you sur 2008-08-25T16:48:47 sir 2008-08-25T16:51:46 okay, eeevil - running "index_gin.sql" now. going to go change into bike clothes and head home. 2008-08-25T16:56:00 *** eeevil has quit IRC 2008-08-25T16:56:54 no, no, no. Thank You! 2008-08-25T16:57:10 *** eeevil has joined #openils-evergreen 2008-08-25T16:57:20 so far, our migration has been very smooth 2008-08-25T16:57:42 nothing like dream I woke up from last night 2008-08-25T17:03:29 *** pmurray is now known as pmurray_away 2008-08-25T17:13:11 home time 2008-08-25T17:13:12 *** dbs has quit IRC 2008-08-25T17:13:57 *** kgs_ has quit IRC 2008-08-25T18:38:45 *** kgs_ has joined #openils-evergreen 2008-08-25T19:40:41 *** dbs has joined #openils-evergreen 2008-08-25T19:41:40 berick: you are a coding FIEND 2008-08-25T19:41:45 berick++ 2008-08-25T20:03:58 *** rsinger has joined #OpenILS-Evergreen 2008-08-25T20:32:51 *** rsinger has quit IRC 2008-08-25T21:41:29 *** rsinger has joined #OpenILS-Evergreen 2008-08-25T21:45:07 miker_: 300 minutes so far building the authority GIN index (750,000 rows) and counting 2008-08-25T21:53:03 *** rsinger has quit IRC 2008-08-25T21:59:36 note to self: obsessively checking indexing progress every five minutes will not speed it up. nor will it let you get anything else done. dumbass. 2008-08-25T22:20:11 dbs: hrmm... that seems like an awefully long time 2008-08-25T22:20:18 *** rsinger has joined #OpenILS-Evergreen 2008-08-25T22:20:22 miker_: yes, yes it does 2008-08-25T22:20:50 did you use CONCURRENTLY? 2008-08-25T22:21:33 well 2008-08-25T22:21:38 that shouldn't matter, of course 2008-08-25T22:21:58 do you recall how long it took to build the GIST index? 2008-08-25T22:22:23 or, if the index_vector index was already there, how long it took to load the authority files? 2008-08-25T22:22:26 file 2008-08-25T22:32:24 miker_: sorry, was off in email response land 2008-08-25T22:32:41 so: 1) no, no CONCURRENTLY because the data ain't a-changing 2008-08-25T22:33:19 2) building the GIST indexes took forever too, IIRC 2008-08-25T22:33:43 lots of IO wait? 2008-08-25T22:33:45 all CPU? 2008-08-25T22:35:52 well, for one thing the "tar is my backup!" process kicked in; I killed that off permanently and replaced it with rdiff-backup (which is a lesser of two evils, I realize) 2008-08-25T22:36:08 heh 2008-08-25T22:36:31 dbs pasted "vmstat" at http://paste.lisp.org/display/65857 2008-08-25T22:37:25 ok... all CPU then :) 2008-08-25T22:37:36 8-core system, 1 core is running the create index at 100%, so it seems to be doing the right thing 2008-08-25T22:37:38 *** rsinger has quit IRC 2008-08-25T22:38:07 right 2008-08-25T22:38:43 we'll see what happens overnight 2008-08-25T22:39:32 hmmm... "Optimized index creation (Makes use of maintenance_work_mem to accumulate postings in memory.)" 2008-08-25T22:39:53 is m_w_m relatively small? 2008-08-25T22:41:59 I think work_mem is at 1GB... one sec 2008-08-25T22:42:33 no, work_mem is 256 MB, m_w_m is 1GB 2008-08-25T22:42:39 maint_work_mem 2008-08-25T22:42:47 ahh 2008-08-25T22:42:49 k 2008-08-25T22:43:59 man ... it took just a fiew minutes to build a gin index over an xpath expression on 250k bib records ... I don' t know 2008-08-25T22:47:14 you think I should upgrade from the C64 tape drive to a 1541 disk? 2008-08-25T22:47:55 heh 2008-08-25T22:48:28 or at least stop using the Def Leopard tape to store the authority data 2008-08-25T22:48:46 But I love Animals! 2008-08-25T22:54:03 one of these days, I'll init a cluster in the C locale and try this indexing stuff 2008-08-25T22:54:03 *** _dkyle_ has quit IRC